Your AI evaluation dashboard is green, yet users rewrite the output or abandon the task. Before changing the model, check whether the evaluation measures the job users are trying to complete. Passing a rubric proves compliance with that rubric; it does not automatically prove that the workflow is useful.
Trace the gap across five places: the success definition, the dataset, the context supplied to the model, the generated answer, and the interface where people act on it. Change the layer supported by the evidence, then test whether the task becomes easier to complete.
The example below is an illustrative prioritization assistant. Its scores and pilot plan are invented to explain the method, not results from a customer or StoriesOnBoard experiment.
Inspect one green result that failed the user
A product owner asks which of six backlog items to bring into a planning meeting. The assistant returns a polished ranking with a rationale for every item. The existing rubric scores structure, clarity, and whether each item has a reason. It awards 9 out of 10.
The product owner still cannot use it. The top item depends on a platform change that will not be ready. Two other items serve a segment excluded from the upcoming pilot. The recommendation includes no source links, and the interface offers only a large “Apply all” action.
The failure is not simply that the model lacks intelligence. The evaluation rewards presentation, the dataset omits blocked dependencies, the context excludes the pilot definition, and the interface makes partial adoption awkward. A stronger model might produce a more persuasive version of the same unusable recommendation.
| Current check | Why it passes | Missing outcome check |
|---|---|---|
| Every item has a rationale. | Six fluent explanations are present. | Does each recommendation respect current dependencies and target users? |
| The list is clearly ranked. | The output uses a consistent numbered format. | Can the product owner understand and challenge the ranking evidence? |
| The answer is concise. | The response stays below the length target. | Does it retain the limitations needed for a planning decision? |
| The UI displays the result. | The panel renders successfully. | Can a user accept one item, reject another, and preserve the original backlog? |
Observe the task, not only the rating
Ask a few representative users to complete a real, authorized task with the assistant. Watch where they pause, consult another source, rewrite a sentence, or abandon the output. Do not coach them toward the intended result. Afterward, ask what information was missing and what they needed to do next.
Capture the task context alongside the feedback. “Unhelpful” means different things for a product owner preparing a meeting and a business analyst checking traceability. Record role, experience, task type, available sources, and whether the user had authority to act.
Use behavioral evidence carefully. Copying a response does not prove that it was correct; rejecting it does not prove it was wrong. A user may copy text to investigate it or reject a good suggestion because the interface makes it difficult to adapt. Combine events with observation and interviews.
A useful observation note reads: “The participant opened the dependency register before accepting any recommendation and removed the first-ranked item.” It is more actionable than “confidence seemed low.” Keep interpretations separate from what was observed.
Use a decision tree to find the layer to change

- Does the rubric describe task success? If it only checks tone or formatting, add the decision constraints and required next action.
- Does the dataset represent the struggling users? If it contains clean expert prompts but production users provide incomplete requests, add realistic cases and report them as a separate cohort.
- Did the system receive the necessary context? If the pilot scope or dependency record was missing, inspect retrieval and context assembly before changing generation.
- Did the answer use the supplied context correctly? If the evidence was present but ignored, test instructions, examples, model behavior, and grading anchors.
- Could the user act on a good answer? If the output was sound but hard to inspect or apply, test the interface and handoff.
Do not let the tree become a way to assign blame. Product may need to define a missing policy; engineering may need to expose the right source; design may need to make review possible. The point is to choose an intervention that matches the observed failure.
Revise the rubric and dataset without hiding the old result
For the prioritization assistant, keep the existing presentation checks but stop treating them as sufficient. Add supported scope, dependency awareness, explicit unknowns, and reviewability. A recommendation that violates a hard release constraint fails even if the prose is excellent.
Build paired cases that reveal the missing distinction. In one, the platform dependency is complete; in another, it is blocked. In one, the pilot includes the relevant segment; in another, it excludes it. The expected recommendation should change when those facts change.
Include cases with absent context. The appropriate behavior may be to request the pilot definition or produce a conditional recommendation. An evaluation that always supplies perfect context will not reveal whether the assistant handles uncertainty responsibly.
Version the revised rubric and re-score the baseline. Report “9/10 under presentation rubric v1; fails dependency gate under task rubric v2,” rather than retroactively pretending the old score measured the new standard. A changed definition of success is a product decision worth recording.
Choose a focused experiment
The fictional team decides to test a context and review-flow change before replacing the model. It adds the approved pilot definition and dependency status to the input, then presents recommendations individually with their supporting evidence. Because this changes two layers, it first tests context quality offline and then tests the review interaction separately.
| Experiment field | Filled example plan |
|---|---|
| Hypothesis | Showing current scope and dependency evidence will reduce unsupported recommendations and help users make selective planning decisions. |
| Offline comparison | Same model and prompts, frozen tasks, with and without the missing context. Score constraint adherence and unsupported claims. |
| Interaction comparison | Use equivalent reviewed recommendations; compare Apply all with item-by-item review. Counterbalance task order. |
| Primary task outcome | A planning shortlist that the participant can justify and that violates no agreed pilot constraint. |
| Supporting observations | Corrections, evidence lookups, abandoned attempts, and explanation of acceptance or rejection. |
| Guardrails | No automatic backlog writes; no unauthorized data; preserve the original shortlist for comparison. |
| Decision rule | Advance only if the targeted failures improve without new constraint violations; investigate mixed results by role and task. |
For an early usability study, a small sample can reveal mechanisms but should not be presented as a reliable population-level uplift. For a larger controlled rollout, agree the analysis and stopping criteria with someone who can assess the required sample and uncertainty.
Keep the original baseline available. If you change the model, prompt, retrieval, and interface at the same time, you may learn that the package works better, but you will not know which change caused the difference.
Connect offline scores to live feedback
After a controlled rollout, review whether the same failure categories recur. Monitor corrections and escalations alongside output quality, latency, and task completion. Break down the results by relevant segments instead of averaging away a weak experience for one role.
Anthropic's evaluation guidance presents offline evaluation as complementary to production feedback and other quality signals. The practical implication is to keep a route from a user complaint to a reproducible case, while retaining observation of the broader workflow.
When the new task-based evaluation passes but users still struggle, repeat the investigation. Perhaps the source information is correct but arrives too late, or the recommendation answers a question the user did not need answered. Resist adding more rubric criteria until you understand that next gap.
Take one recent “green but unusable” answer to your next product review. Put the score beside the actual task, the supplied evidence, and the user's next action. That comparison will usually give you a better experiment than another round of generic prompt polishing.




