Should this feature use AI at all? Deciding from practitioner evidence
The question is not whether a model could do the step. It is whether the step involves judgement over messy input, or a rule somebody could simply have written down.
The short answer: put the task to the five-practitioner test. If several competent people reach the same answer from the same input, the step is a rule and belongs in code. If they disagree and each can defend their reasoning, it is judgement over messy input, and that is the only profile where a model reliably pays for itself. Your corpus already contains the experiment.
In 2026 the default answer to every product question has drifted toward yes, use a model, which makes the question worth asking properly. A model in the path adds cost per call, latency, non-determinism, a vendor dependency and a class of failure that is hard to explain to a customer. Those are acceptable prices for judgement. They are a bad trade for a lookup.
The five-practitioner test
Take a step from your workflow map and find it performed in several independent videos with roughly comparable inputs. Then compare outcomes. Convergence means the task has a correct answer that a rule can reach. Divergence with defensible reasoning on each side means the task is genuinely ambiguous.
The test is cheap because the material is already gathered, and it is more honest than asking yourself whether a task “feels AI-ish”. It is also the same evidence base used for resolving contradictions in a corpus, described in resolving contradictions in video research— except here the contradiction is the finding rather than the problem.
A step where practitioners disagree is a step where a tool can add value beyond speed, because the user cannot simply do it faster themselves. A step where everyone agrees is a step where your competition is a well-written function.
Four task profiles and what each deserves
| Task profile | Evidence in the corpus | Right mechanism |
|---|---|---|
| Deterministic rule | Everyone reaches the same answer | Code, with the rule stated in the UI |
| Judgement over messy input | Competent people disagree, defensibly | Model, with the working shown |
| Bulk tedium, low stakes | People complain about volume, not difficulty | Model with cheap review, or plain automation |
| High-stakes judgement | People double-check and escalate | Assistive draft, human decides, always |
The fourth row is where most of the regret lives. Domains where mistakes are expensive do not want an autonomous answer at any accuracy level that is currently achievable, and shipping one anyway produces a product people admire in demos and disable in practice. The cost signals that identify these domains are the same ones covered in estimating support burden from practitioner complaints.
Reading scepticism precisely
- ✗Objections dismissed as resistance to change
- ✗Accuracy complaints answered with a better model
- ✗Control concerns answered with reassurance
- ✗Demo optimised for the impressive step
- ✓Accuracy complaints become a measurable bar
- ✓Control concerns become an interface requirement
- ✓Verifiability designed in from the first version
- ✓Demo optimised for the tedious step people hate
The two kinds of objection need completely different responses. “It gets it wrong too often” is a threshold you can measure against the tolerance benchmarks in your corpus. “I would not know if it was wrong” is a design problem that no accuracy improvement fixes, because the user is objecting to opacity rather than to error rate.
Distinguishing those two is easier when you know which voices are independent and which are promotional, since vendor-adjacent content systematically understates the second objection — the correction described in vendor content versus independent creators.
The common failure: automating the fun part
Practitioners narrate the creative, visible steps with enthusiasm and rush past the drudgery. A product team watching the same footage will naturally build toward the parts that were fun to watch, and end up automating the step the user actually wanted to keep while leaving the one they wanted removed.
The corrective is to rank candidate steps by the four cost properties — frequency, error-proneness, time pressure, cost of a mistake — rather than by narrative prominence, which is the method in spotting manual workarounds in creator content.
Separating the capability from the fashion
A large fraction of creator enthusiasm for any model-based feature is about the technology rather than about the outcome, and it decays. Enthusiasm that survives into the follow-up video, the one made three months later about what they kept using, is worth far more than the launch-week reaction.
Separating durable signal from the launch cycle is the whole discipline in spotting hype versus signal in creator content, and the specific ways model-based research itself goes wrong are catalogued in AI video research failure modes.
A yes still has to earn its place in version one
Deciding that a step deserves a model is not the same as deciding it belongs in the first release. Model-based features carry the heaviest support load of anything you can ship, because every disputed output becomes a conversation, and that load lands before you have the revenue to absorb it.
The safer sequence is usually to ship the deterministic parts first, earn the right to be trusted with judgement, then add the model where the evidence said it belonged. Whether the AI step is the product or a feature of something larger is worth settling explicitly, using the feature-or-product test, and prioritising against the rest of the backlog with research evidence rather than intuition.
The dependency you are taking on
Even when the evidence says judgement, the decision is not free, and the costs land in places that do not show up in a prototype. Four are worth pricing before committing.
| Cost | Where it shows up | Mitigation |
|---|---|---|
| Unit economics | Cost per call against your price per seat | Cache, batch, or gate behind a higher tier |
| Latency | A step that used to feel instant no longer does | Stream, or keep the model off the hot path |
| Non-determinism | Support tickets you cannot reproduce | Log inputs and outputs, show the working |
| Vendor dependency | Their outage, their deprecation, their price change | Abstract the call, keep a degraded fallback path |
The unit-economics row is the one that decides most cases. A feature that costs meaningful money on every invocation cannot sit on an unlimited entry plan, and discovering that after launch means either re-pricing or rationing something customers were already using.
Non-determinism is the underrated one. A deterministic bug is reproduced, fixed and closed; a model that answers differently on Tuesday produces a ticket that never fully resolves, and the accumulation of those is how a small support function stops coping.
What this costs
The read is a synthesis over material you already have. As of September 2026, Hobby is $19 a month with 25 videos and 2 projects, Pro is $59 with 80 videos and 8 projects, and Studio is $199 with 250 videos, 20 projects and 3 seats, each with a 7-day free trial — see the pricing page.
Run a synthesis that shows where competent practitioners agree, where they disagree, and which objections are about accuracy versus control. 7-day free trial.
Closing thought
The best argument for putting a model in your product is that the people doing the job today cannot agree on the answer. The best argument against it is that they can.
Frequently asked
How do I decide whether a feature should use AI at all?
Ask whether the step practitioners perform involves judgement over messy input, or a rule they could write down. Judgement over messy input is where a model earns its cost. A rule someone can state in a sentence should be code, because code is cheaper, faster, explainable and never has a bad day.
What does practitioner content tell me that a spec cannot?
Whether the step is genuinely ambiguous. If five competent practitioners reach the same answer from the same input, the task is deterministic and you have found a rule. If they disagree and each can defend their answer, it is judgement, and that is the profile where a model helps.
Is practitioner scepticism about AI features a reason not to build one?
Not by itself, but the reasons matter. Scepticism about accuracy is a bar to clear. Scepticism about losing control or being unable to check the work is a design constraint, and products that ignore it get abandoned quietly rather than reviewed badly.
What is the cheapest way to fail at this?
Automating the step people enjoy or get credit for, while leaving intact the tedious step they actually wanted removed. It is a surprisingly common outcome, because the enjoyable step is usually the one that looks impressive in a demo.
How should uncertainty be presented?
The way the domain already presents it. If practitioners annotate confidence, flag assumptions or show their working, users expect the same from software. A confident wrong answer in a domain where humans hedge reads as incompetence rather than as convenience.
Does this apply to using AI internally, not just in the product?
Yes, and the test is the same. Internal use has a lower accuracy bar because a human is in the loop, which is precisely why the judgement-versus-rule question still decides whether it is worth the dependency.
What does this analysis cost?
As of September 2026, Hobby is $19 a month for 25 videos and 2 projects, Pro is $59 for 80 videos and 8 projects, and Studio is $199 for 250 videos, 20 projects and 3 seats, with a 7-day free trial on every plan. The read runs over an existing corpus as an extra synthesis.