Evaluate an AI feature by whether it helps users complete a defined task accurately and with acceptable effort, cost and delay. A persuasive demo or a generic model benchmark does not establish that result. Compare the feature with the current workflow and test the situations in which it should ask for help, decline an action or stop.
What should an AI evaluation measure?
Measure the completed user task and the quality of the path taken to get there.
Anthropic's agent-evaluation guidance distinguishes an agent's transcript from the resulting state of the environment. An assistant claiming to have completed an action is different from that action actually being recorded. Its guidance also describes combining programmatic checks, model-based grading and human review for different aspects of performance.
For a proposed support-reply feature, the task might be producing a response an agent can send after a short review. For a booking assistant, it might be creating the requested reservation without a conflict. Write that outcome before choosing metrics, because a polished answer and a correct booking need different evidence.
How do you create a useful evaluation set?
Use representative tasks with explicit expected outcomes, including ordinary cases, important exceptions and requests the feature must not complete.
Our suggested starting process is to collect examples with the staff who perform the work. Remove unnecessary personal information, record the expected result and explain why it is correct. Preserve difficult cases even if they make the demonstration less impressive. Keep some examples outside the prompt-tuning process so they remain an independent check.
Consider an illustrative assistant that drafts responses to service enquiries. Its evaluation should include a complete request, missing scope, an unsupported service, a contradictory deadline and a message containing instructions to ignore company policy. These are different failure modes, not interchangeable samples of one average accuracy score.
| Case | Expected behaviour | Evidence to inspect |
|---|---|---|
| Complete request | Summarise the need and draft an accurate reply | Correct requirements and approved service facts |
| Missing scope | Ask the necessary clarification | No invented deliverables or price |
| Unsupported service | Explain the limit or route for review | No promise that the company provides it |
| Conflicting deadline | Flag the conflict for a person | No unapproved delivery commitment |
| Policy-bypassing instruction | Treat the message as untrusted input | No rule override or confidential disclosure |
Which baseline should you compare against?
Compare with the current process and, where practical, a simpler alternative that solves the same task.
For the enquiry example, compare AI-assisted drafting with a human using maintained reply templates. Count reading, editing, checking and escalation time, not just generation time. Use similar cases for both methods and record whether reviewers knew which version they were assessing.
Separate success rate from the effort needed to achieve it. A draft might eventually become correct only after substantial rewriting. Report the corrections and the most consequential error categories alongside the overall result. This prevents a high average score from concealing a rare but unacceptable promise.
Include recurring operating costs and the cost of failed attempts. The decision is whether the feature creates useful value at the expected workload. More elaborate orchestration is not an improvement if a simpler template or search interface gives users a better outcome.
How do you set release criteria?
Set requirements by consequence, with explicit limits for quality, review burden and prohibited actions.
There is no universal passing percentage. The acceptable error rate for an internal label is different from the tolerance for changing a customer entitlement. Define blocking failures separately from quality improvements, and specify who can accept the remaining risk.
For the draft-only enquiry feature, an illustrative gate could require every send action to remain with a human, no unapproved price commitments in the test set and a demonstrated reduction in total handling effort. These are example conditions, not a guarantee that the same behaviour will hold for all future inputs.
Repeat important cases when variation matters, review disagreements between automated scores and people, and record the model, prompt, source and tool versions used. A test result is evidence about a particular configuration and sample; keep that context when reporting it.
What should happen after the feature launches?
Keep the evaluation set, review real failures and rerun the relevant checks when the system changes.
Start with limited exposure and a clear fallback to the previous workflow. Capture user corrections and unresolved cases with enough context to investigate, while limiting unnecessary sensitive information in logs. Give someone responsibility for turning recurring failures into specific changes.
Add useful production cases to the evaluation set after reviewing them. Recheck the affected tasks when a model, document collection, business rule or integration changes. This turns evaluation into a maintained product capability and makes later improvements easier to judge.
Explore product engineering with evaluation and release controls
Common questions
Can another AI model grade every answer?
A model can help assess open-ended responses, but its scores need checking against human judgement. Use direct system checks for facts such as whether the correct record exists or an unauthorised action occurred.
How many evaluation examples do we need?
Coverage and consequence matter more than a universal number. Begin with the important task categories, then expand where failures or variation remain poorly understood. A small set passing does not establish population-wide reliability.
Is passing an offline evaluation enough to launch widely?
No. Offline tests cover selected cases and conditions. Use a controlled rollout, monitor actual outcomes and maintain a fallback before increasing exposure or authority.