Prompting vs RAG vs fine-tuning

The interview question: "the model does not know our data and the tone is off, so let us fine-tune". The answer is a ladder, and fine-tuning is the last rung, not the first.

Correction worth remembering: fine-tuning is a poor way to add knowledge. Facts baked into the weights are unreliable, go stale and cannot be cited. For facts, use RAG.

• Brand voice: thousands of past support replies, and "warm but concise" never quite captures it.
• Strict output format: your exact JSON schema or DSL, every time (95% with prompting, 99.9% tuned).
• Classification or routing at scale: ticket categories, intent, moderation, with a small tuned model instead of a big model plus a long prompt.
• Extraction from messy invoices or clinical notes in your house style.
• Text-to-SQL or code for your schema and conventions.
• Distillation: a big model writes good answers, a small model is trained on them for similar quality at a fraction of the cost.
• Domain style: legal clause rewriting, clinical summaries in your template.
• Refusal and escalation behavior for a specific class of requests.

A 2,000-token prompt of rules works but is slow and costly on every call. Fine-tuning a small model bakes the rules in, so the prompt becomes short. But try prompt caching first, since it is free to try. Fine-tuning also needs a few hundred to a few thousand good examples, an eval set to prove the tuned model is no worse, and a retrain whenever the rules change.