Prompting, RAG, or Fine-Tuning? A Product Manager’s Experiment Plan

Choose the next experiment from the failure mechanism. Compare prompt changes, curated context, retrieval, and training against a common product task and evaluation set.

Three experiment branches isolate instructions, current product facts, and learned behavior.
Table of contents
  1. Diagnose the failure before choosing an architecture
  2. Separate prompt engineering from context engineering
  3. Use RAG when answers need the right current evidence
  4. Consider fine-tuning for a stable behavior problem
  5. Run a controlled experiment with a diagnostic control
  6. Write a decision record before scaling

A product assistant gives inconsistent answers. Should you improve the prompt, add retrieval-augmented generation, or fine-tune a model? Start by locating the missing ingredient. Instructions, access to current facts, and learned behavior are different problems, even when they produce similarly disappointing answers.

Prompting changes the instructions and examples supplied at inference time. Retrieval-augmented generation, or RAG, retrieves relevant material and includes it in the model's context. Fine-tuning changes model parameters through training examples. These approaches can be combined, but they should not be selected by fashion or by a single impressive demo.

This guide proposes an experiment plan for a fictional release-support assistant. The examples and decision record are illustrative; no performance results are claimed.

Diagnose the failure before choosing an architecture

Read a sample of failed outputs together with the exact inputs the model received. Ask whether the required fact was absent, present but ignored, contradictory, stale, or outside the user's access. Then ask whether the desired behavior was actually specified.

If the latest release decision was missing from the input, a stronger instruction to “be accurate” cannot supply it. If the decision was present but the output ignored a required format, building a search index may not address the problem. If the task changes every week, training on yesterday's examples may be an expensive way to encode a moving target.

Observed failureLeading hypothesisFirst experimentEvidence that would change the choice
The assistant invents a deadline when none is stated.The instructions do not define behavior for missing facts.Add an explicit unknown-answer rule and contrasting examples.It still invents deadlines despite clear instructions and supplied evidence.
The answer quotes a superseded release decision.Current authoritative material is absent or poorly selected.Test retrieval with revision and approval metadata.The correct document is already present but is consistently misinterpreted.
The output format varies across an otherwise stable task.The model does not reliably follow the target behavior.Test a clearer schema, examples, and validation before considering training.Validated prompting remains inadequate on a sufficiently varied task set.
The answer includes another team's confidential plan.The access boundary is defective.Fix and test authorization in the data path.No prompt or fine-tuning result can substitute for that boundary.

Separate prompt engineering from context engineering

A prompt says what to do with information. Context engineering determines which instructions, documents, tool results, conversation history, and state are available for that task. A well-written instruction can still fail when surrounded by stale or irrelevant material.

For the release assistant, the prompt might require a concise answer, source references, and explicit unknowns. The context packet supplies the approved release scope, relevant stories, the caller's permitted workspace, and the source revision. The packet should also identify conflicting or superseded notes.

Anthropic's context engineering guidance treats context selection as a broader concern than prompt wording. In product work, a useful starting point is a small, reviewable product-context packet rather than a large undifferentiated document dump.

Keep a known-good, manually assembled packet as a diagnostic control. If the model succeeds with that packet but fails with retrieved context, investigate retrieval. If it fails with both, inspect the instructions, reasoning demands, source ambiguity, and model behavior.

Use RAG when answers need the right current evidence

RAG can help when the answer depends on a body of documents that changes or is too large to include in every request. The system retrieves candidate material, selects useful passages, and supplies them for generation. Microsoft's RAG overview describes this retrieval-and-generation pattern.

Evaluate retrieval separately from the final answer. Did it find the approved decision? Did it include the right revision? Did it preserve enough surrounding text to interpret an exception? Did access filtering happen before content reached the model?

Then evaluate the generated answer. A correct source can still be misread, and a citation can point to a passage that does not support the claim. Ask the assistant to distinguish evidence from inference, and verify that distinction on held-out tasks.

RAG introduces its own operations: ingestion, indexing, freshness, permissions, retrieval ranking, and failure handling. When a source is unavailable, the system should explain the limitation or route the task appropriately. Falling back to an unsupported confident answer defeats the purpose of grounded retrieval.

Consider fine-tuning for a stable behavior problem

Fine-tuning is worth investigating when a stable task requires behavior that remains inconsistent after a reasonable prompting baseline, and the team has enough high-quality examples to train and evaluate it. Examples might demonstrate a specialized output convention or a repeated transformation with well-defined acceptance criteria.

Training data needs reviewed inputs and desired outputs, clear rights to use the material, and separation from the final evaluation set. A larger collection of unreviewed responses can teach the wrong habits. Changes to the desired behavior may require another training and evaluation cycle.

AWS's comparison of RAG and fine-tuning discusses their different maintenance tradeoffs and the possibility of combining them. For this product workflow, the key inference is that training does not replace runtime access to current, authorized product records.

Do not use fine-tuning as a permission system or as proof that the model will never invent facts. A tuned model still needs the correct context, access controls, tests, and a route for uncertainty. Verify that the chosen provider and model support the required training method before committing engineering effort.

Experiment branches distinguish instructions, retrieved product facts, and learned behavior while sharing one held-out evaluation set.
Keep the task and evaluation constant while changing the mechanism under investigation.

Run a controlled experiment with a diagnostic control

Define the task as answering release-support questions using approved product decisions. Freeze a dataset containing ordinary questions, missing facts, conflicting revisions, permission boundaries, and specialized output requirements. Agree the scoring rubric and hard gates before running candidates.

CandidateWhat changesWhat it testsWhat to record
B0 · BaselineNothing; current system and sources.Existing quality and failure categories.Outputs, source packet, latency, cost, corrections.
P1 · PromptExplicit task boundaries, unknown-answer rule, and reviewed examples.Whether better instructions solve the observed failures.Paired gains and regressions on identical cases.
C1 · Curated contextA reviewer supplies the relevant authorized evidence to P1.Whether correct context is sufficient.Residual failures despite an adequate source packet.
R1 · RetrievalAutomated retrieval supplies context to the same P1 instructions.Whether the system can find the evidence C1 needed.Source recall, stale results, unsupported claims, access failures.
F1 · Fine-tuning trialA tuned model uses a defined, comparable context setup.Whether training improves the remaining stable behavior problem.Held-out quality, new regressions, training and maintenance costs.

Run F1 only if the earlier results justify it and the data is suitable. Keep training examples separate from final evaluation cases. If the tuned model differs in more than its training, record those differences; the comparison then tests a package, not a single causal change.

Measure costs at the task level, including retrieval, longer context, retries, human correction, and training maintenance. A lower per-call price can still produce a more expensive accepted answer. Report latency distributions and failure cases rather than only a best-case demo.

Write a decision record before scaling

A filled decision record can be useful even before results exist. For this illustrative team it reads: “We will test P1 and C1 first because missing-source and missing-instruction failures dominate the reviewed examples. R1 proceeds if C1 resolves the factual gaps. Fine-tuning remains a candidate only for residual stable behavior failures. No candidate can pass with unauthorized retrieval or invented release commitments.”

After the experiment, replace the hypotheses with observed results, including what did not improve. Name the owner, evaluation revision, source snapshot, remaining risks, and next review trigger. If the product task changes, revisit the decision rather than treating the architecture as permanent.

Choose the smallest experiment that can disprove your current diagnosis. You will learn more from showing that the right context fixes one class of failure than from building three architectures and comparing their most flattering answers.