A product owner changes a prompt, likes the next answer, and ships it. A week later, another team discovers that the improved answer depends on information their users rarely provide. The missing artifact was an AI evaluation dataset: a repeatable collection of inputs, expected behavior, and scoring rules that existed before the change.
Build the baseline around a specific product task. Collect representative cases, freeze the context used for each case, define a small anchored rubric, and compare the old and new versions on the same inputs. Start with human scoring so you know what the numbers mean. A model-generated grade can come later.
This guide uses an illustrative backlog assistant that answers questions about release scope. Its eight examples are a teaching dataset, not real customer records or evidence that eight cases are enough for a production release.
Define the task before collecting answers
Write one sentence describing the decision the assistant supports: “Help a product owner identify which invitation stories belong to the agreed first release, using the current story map and its linked decisions.” That is narrower and more testable than “give helpful product advice.”
Then state the boundaries. The assistant may summarize approved scope and identify unanswered questions. It may not invent commitments, move stories, or treat a proposed requirement as approved. A correct answer should let the product owner take the next step without rereading the entire source collection.
For each run, retain the input, source snapshot or revision, permission context, prompt version, model identifier, output, and reviewer notes. Store a stable case ID separately from the wording. If the story map changes, you need to distinguish a better answer from a different source.
Structured product context for AI agents helps define those inputs. It does not supply an evaluation framework automatically; the dataset and scoring process remain your team's responsibility.
Build a dataset with successes, failures, and missing information
Start with authorized examples from actual work: accepted answers, corrected drafts, abandoned tasks, and support questions. Remove unnecessary personal data. Sample across roles, source quality, task difficulty, and permissions instead of selecting only memorable failures. Keep the original case and any simplified reproduction linked.
The table below is fully fictional. Each row includes enough source context to judge the expected behavior. In a real dataset, attach the underlying evidence rather than asking reviewers to trust a summary.
| Case and split | Input and source facts | Expected behavior | Failure to catch |
|---|---|---|---|
| E01 · Development | “What is in release one?” R1 contains single invitation and Viewer/Editor choice; bulk import is R2. | Name the two R1 capabilities; exclude bulk import. | Promoting a later item into R1. |
| E02 · Development | “Can a Member invite someone?” The approved permission rule says Admin only. | Answer no; identify the Admin prerequisite and the rule. | Inferring access from a visible button. |
| E03 · Development | “When does a link expire?” No approved duration appears in the sources. | Say the duration is unspecified and ask for the policy owner. | Inventing a familiar 24-hour default. |
| E04 · Development | “Is bulk import committed?” Workshop notes propose it; the later scope decision explicitly defers it. | Prefer the later approved decision and explain the disagreement. | Treating brainstorming as a commitment. |
| E05 · Held out | “Show the invitation plan.” The caller can read Board A; Board B is outside their access. | Use only authorized Board A evidence; state that the answer is limited to that scope. | Including inaccessible material. |
| E06 · Held out | “What changed since the previous review?” Snapshot 8 adds expired-link recovery to R1; snapshot 7 lacks it. | Report the addition and identify both snapshots. | Restating the whole map without identifying change. |
| E07 · Held out | “Is the flow complete?” Invite and accept stories exist; no recovery behavior is specified. | Flag recovery as an unanswered scope question, not an approved feature. | Claiming completeness or silently adding work. |
| E08 · Held out | “Summarize R1 for support.” R1 includes invitations, role choice, and recovery; email delivery is outside this prototype. | Give a concise task summary and clearly retain the delivery limitation. | Turning a prototype into a production promise. |
Development cases are available while improving instructions. Held-out cases are reserved for a later comparison. This published example cannot be anyone's secret test set: create your own unseen cases for that purpose. Separate near-duplicate conversations and source documents before splitting; a paraphrase of a development answer is weak evidence of generalization.
Keep a coverage inventory next to the dataset. E03 checks missing information, E04 conflicting evidence, E05 access boundaries, and E06 change detection. Add ordinary successful tasks too. A suite made only of difficult edge cases answers a different question from a sample of everyday use.
Copy a rubric with anchored 0–2 scores
A rubric translates “good” into observable judgments. The reviewer role here is a product owner checking a release-scope answer against supplied evidence. Reviewers must use the same source packet and task definition.
| Criterion | 0 | 1 | 2 |
|---|---|---|---|
| Groundedness | A material claim contradicts or invents source facts. | Core claims are supported, but attribution or an important uncertainty is unclear. | Material claims trace to authorized evidence; uncertainties and conflicts are explicit. |
| Completeness | Misses the main requested decision or a required constraint. | Answers the main question but omits a relevant secondary item. | Covers the requested decision and all case-specific required constraints. |
| Usefulness | Misleads the next action or gives no usable answer. | Usable after an identifiable clarification or rewrite. | Clear enough for the intended reader to take the appropriate next step. |
Add hard gates outside the average: unauthorized disclosure, unapproved changes, and invented release commitments fail the case regardless of other scores. A polished answer must not compensate for a permission violation.
Reviewer role: Product owner validating release-scope assistance.
Task: Answer the user's question from the supplied authorized source packet.
Score groundedness, completeness, and usefulness from 0 to 2.
For each score, quote the relevant answer span and identify source evidence.
Record hard-gate failures separately.
If the source packet cannot resolve the judgment, mark ungradable
and explain the missing evidence. Do not turn uncertainty into a pass.
Return: case ID, three scores, evidence, hard gates, required correction.“Ungradable” is a review state, not a fourth quality score. Fix the broken case or report it separately. Silently excluding it from the denominator makes comparisons difficult to interpret.
See how two answers receive different scores
For E03, answer A says: “Invitation links expire after 24 hours. Ask an admin to resend one.” The duration is invented. Groundedness is 0, completeness is 0 because the answer misses the essential uncertainty, and usefulness is 0 because it directs the reader using a false rule. The invented-policy gate fails.
Answer B says: “The supplied invitation requirements do not specify an expiry duration. Confirm that policy with the feature owner before documenting a deadline. The recovery story covers replacement of an expired link but does not define the timeout.” This earns 2/2/2 against the stated case. It answers the question honestly and identifies the next decision.
For E01, an answer naming single invitation but omitting Viewer/Editor choice can receive 2/1/1. It is grounded but incomplete and needs editing before support can use it. This distinction prevents every imperfection from becoming a vague “hallucination.” These are hand-scored illustrative judgments, not model benchmark results.
Calibrate two reviewers before scaling
Ask two reviewers to independently score the same small batch, with candidate identities hidden where practical. Compare each criterion, not just the total. For E01, one reviewer might treat role choice as optional detail while another considers it required. Resolve that by checking the task definition and adding the missing anchor, not by averaging away disagreement.
Record the original scores, the adjudicated score, the reason, and the rubric revision. Re-score affected cases after a material rule change. For a larger program, measure agreement and examine recurring disagreement categories; an agreement number alone does not show that both reviewers are right.
Anthropic's guide to agent evaluations distinguishes code, model, and human graders and recommends calibrating model-based grading against human judgment. Here, that means preserving reviewer evidence before automating any scoring.
Compare the old and new versions on identical cases
Choose the decision rule before looking at candidate results. For this fictional assistant, a starting gate could require zero hard-gate failures, no new failures on previously passing cases, and improvement on the specifically targeted missing-information cases. This is an example policy, not a universal numerical threshold.
Run both versions against the same evidence snapshots and permissions. Retain outputs so reviewers can inspect paired changes. If outputs vary across repeated runs, report that variation rather than choosing the best attempt. Keep generation settings and tool access consistent unless one is the variable being tested.
A useful comparison row reads: “E03, baseline 0/0/0 with invented timeout; candidate 2/2/2 with explicit unknown; no tool call; reviewer agreed.” A decision summary should also show regressions, ungradable cases, run count, rubric version, and the segments absent from the dataset. Avoid presenting a single mean as the release decision.
The next action is practical: take one recurring product question, collect a first batch of authorized examples, and have two people score them independently. You now have a baseline that can explain why a new prompt is better—and where it is still unproven.
