“The assistant gave me the wrong answer” is a useful warning, but it is not yet a regression test. To make the complaint repeatable, capture the task, the evidence the assistant actually saw, the output, and the behavior the user expected. Then turn the failure into a case that another person can run and grade.
A living AI regression suite preserves important lessons from production without becoming a pile of disconnected anecdotes. It needs stable case IDs, reviewed expectations, representative sampling, version history, and a clear relationship to production monitoring.
This guide follows three fictional complaints about a product-planning assistant. The examples demonstrate a process; they are not customer incidents or measured failure rates.
Capture the incident before changing the prompt
When a complaint arrives, save the smallest authorized record that explains the failure. Record the user's role and task, input, source revisions, retrieved context, tool calls, model and prompt versions, output, expected behavior, and consequence. If some of these are unavailable, state that instead of reconstructing them as facts.
Separate the user's wording from the investigation. “It ignored our decision” is the complaint. “The retrieval step returned an older workshop note but not the approved decision” is a possible diagnosis that requires evidence. Preserving both helps the team avoid solving the wrong problem.
Remove unnecessary customer identifiers and secrets. If a case requires private data to reproduce, keep it in an appropriately controlled test environment. A sanitized version should preserve the relationship that caused the failure—such as conflicting dates or permissions—without retaining irrelevant personal detail.
Do not immediately add the entire conversation to the permanent suite. Long transcripts can obscure the relevant condition and make future failures difficult to interpret. First establish what happened, then create a focused case with a link back to the restricted incident record.
Turn three complaints into repeatable cases
| Complaint | Minimal reproduction | Expected behavior | Grader and gate |
|---|---|---|---|
| C17: “It put a deferred feature into our pilot.” | Ask for pilot scope. Supply an older workshop proposal for bulk import and a newer approved decision deferring it. | Exclude bulk import, identify the later decision, and explain the conflict if relevant. | Source review. An invented commitment fails the case. |
| C18: “The invitation summary exposed another team's plan.” | Use an identity with access to Board A only. An isolated test fixture contains Board B but must not return it to that identity. | Retrieve and summarize only authorized records; state the scope of the answer. | Access-control integration test plus output inspection. Unauthorized retrieval or disclosure fails. |
| C19: “The answer looked complete, but support still had no recovery steps.” | Ask for support guidance using a feature packet that documents invitation creation but leaves recovery undecided. | Explain the documented path and explicitly flag the missing recovery decision. | Anchored rubric. Claiming complete coverage without that limitation fails. |
C17 is a source-precedence problem. C18 crosses an authorization boundary and cannot be adequately tested by asking a model to promise discretion. C19 is a completeness and uncertainty problem. A single generic instruction to “be more accurate” does not specify a fix for any of them.
For C18, use deliberately created test identities and fixtures. Do not probe another team's real data to demonstrate the failure. Test the retrieval boundary as well as the final text: a system that retrieves forbidden content and happens not to quote it has not passed the access test.
Write a test record another person can rerun
Case ID: scope-conflict-c17
Origin: sanitized incident C17
Task: summarize the approved pilot release
Segment: product owner; conflicting source versions
Sources: workshop-v2 and approved-scope-v4, frozen copies
Prompt/model: recorded separately for each candidate run
Expected: bulk import excluded; approved-scope-v4 identified
Forbidden: presenting bulk import as committed
Grader: source check with reviewer evidence
Severity: release-commitment error
Owner: product quality owner
Status: active regression
Known limitation: does not test retrieval ranking on the full corpus
The final line matters. A minimal reproduction can prove whether the model handles supplied conflicting evidence, but it may not reproduce why the right document was absent from the original retrieval. Keep a separate integration case if the incident depended on retrieval, tool permissions, or the application workflow.
Run the case against the recorded failing version when feasible. If the case passes, investigate why. You may have removed the decisive condition while simplifying it, changed the source order, or used a different model configuration. Do not describe an unconfirmed reproduction as a deterministic test of the original incident.
Keep the suite representative as failures accumulate
Failure-driven cases are intentionally skewed toward problems. They protect against recurrence, but they do not estimate how often users experience those problems. Maintain a separate representative sample of normal work alongside the regression set.
Tag cases by task, role, language, source quality, failure mechanism, and severity. Review whether one prolific customer or one heavily reported workflow dominates the suite. Ten complaints caused by the same stale document may justify one core regression case and a frequency record, rather than ten nearly identical tests.
Deduplicate by causal condition, not just text similarity. Two identical questions from different permission contexts can be meaningfully different tests. Conversely, ten paraphrases with the same context and expected answer may add little independent coverage.
Keep untouched evaluation cases outside day-to-day prompt tuning. Once a held-out case has been repeatedly examined and used to improve the system, treat it as development knowledge and replenish the held-out set. Renaming the file does not restore its independence.

Version additions, changes, and retirements
A living suite changes because the product and its users change. Treat expectation changes as reviewed editorial decisions. If a feature moves from deferred to approved, the old expectation should not remain an unexplained permanent failure.
| Suite version | Change | Reason and comparison rule |
|---|---|---|
| 1.1 | Add C17 conflict case and a normal single-source scope case. | Test the incident and preserve a simple control. Compare both candidates on version 1.1. |
| 1.2 | Add C18 permission fixture and integration check. | Cover unauthorized retrieval, not only output wording. Report this new gate separately. |
| 1.3 | Add C19 missing-recovery case; clarify the uncertainty rubric. | Re-score earlier uncertainty cases under the revised rubric. |
| 1.4 | Retire an obsolete role label; retain the original record and replacement link. | The product terminology changed. Do not compare raw totals across different suites. |
Never delete an awkward failure merely to improve the headline score. Retire it when the scenario is no longer supported, merge it when it duplicates another mechanism, or correct it when the expectation was wrong. Record which of those happened and who reviewed the decision.
When comparing a new system to the baseline, run both on the same active suite revision. If that is impossible, report the overlapping cohort and the newly added cases separately. A higher pass rate on an easier set is not evidence of improvement.
Separate offline evaluation from production monitoring
Offline evaluation asks whether a candidate meets known expectations on controlled cases. Production monitoring asks what happens in current use: errors, latency, rejected suggestions, tool failures, escalations, and changing task patterns. Neither replaces the other.
A regression suite can pass while a new source format breaks retrieval in production. Monitoring can show rising rejection without explaining whether the cause is poor answers, a confusing interface, or a change in who is using the feature. Investigate the signal, then add a reviewed case when you understand the mechanism.
Anthropic's agent evaluation guidance treats evaluations as one part of a broader quality process. For a product team, the practical loop is observation, investigation, a repeatable case, a candidate change, and a monitored rollout.
Be explicit about denominators. “Three permission cases failed in a 40-case regression suite” describes suite behavior. It does not mean 7.5% of users experienced a permission failure. Production incidence needs an appropriate count of affected events or users and a clearly defined observation period.
Make the next incident improve the system
Assign one owner to the intake-to-test handoff. Their job is to ensure that the expected behavior is agreed, the sensitive data is handled appropriately, and the case runs in the right layer. Engineering owns the technical reproduction; product owns the user outcome; reviewers own the grading evidence.
At the next quality review, bring one complaint, its original evidence, the minimal test, and the proposed correction. If the team cannot explain what the test proves, keep investigating. A small, understood regression case is more durable than a large collection of transcripts nobody can confidently grade.




