RAG vs. fine-tuning: how to choose for enterprise knowledge

"Should we fine-tune a model on our data?" is one of the first questions teams ask when they want an assistant that knows their business. Usually the honest answer is: not yet — and maybe not for that.

The one-line answer

• Use retrieval (RAG) when the model needs to know things: policies, contracts, product data, tickets — information that changes and that people will want sources for.
• Use fine-tuning when the model needs to behave a certain way: a strict output format, a house style, a narrow classification task, or a smaller, cheaper model doing one job well.

Most enterprise knowledge problems are the first kind.

What each one actually changes

	Prompting	RAG	Fine-tuning
Adds new knowledge	Only what fits in the prompt	Yes, from your sources at query time	Weakly, and frozen at training time
Stays current	Manual	Re-index and it's current	Retrain to update
Can cite sources	No	Yes	No
Respects per-user permissions	No	Yes, if retrieval filters by access	No
Changes style and format	Somewhat	Somewhat	Strongly
Effort to update	Low	Low–medium	Medium–high

Retrieval keeps your knowledge outside the model, in a store you control. That is why it can stay fresh, show where an answer came from, and filter what each user is allowed to see.

When RAG is the right call

Choose retrieval when:

• answers must be grounded in specific documents and cite them;
• the content changes — prices, policies, product specs, open tickets;
• different people may see different data;
• you need to explain an answer to an auditor, a customer or a colleague.

A well-built RAG system pairs a capable general model with a search layer that finds the right passages, and instructs the model to answer only from them — and to say so when the sources don't contain the answer.

When fine-tuning earns its place

Fine-tuning is worth it when prompting can't get you a consistent result:

• Strict formats — always producing a specific JSON schema or document structure.
• Narrow, repeated tasks — routing tickets, extracting fields, classifying documents.
• Cost and latency — teaching a smaller model one job so you don't pay for a large one on every call.
• Specialist language — domain shorthand that general models misread.

Note what is not on that list: memorising your knowledge base. Facts learned in training can't be cited, can't be filtered by permission, and go stale the day after training.

Where RAG projects usually go wrong

RAG is the right default, but it isn't automatic. The common failure points:

1. Chunking that splits tables, clauses or steps in the middle, so the right answer is never retrieved whole.
2. Weak retrieval — relying on a single embedding search when keyword search, metadata filters or re-ranking would find better passages.
3. Ignoring permissions — indexing everything and filtering nothing, so the assistant can surface documents a user should never see.
4. Stale indexes — no pipeline to re-index when source documents change.
5. No evals — without a set of real questions with known answers, nobody can tell whether a change helped.

Most "the model hallucinates" complaints turn out to be retrieval problems: the right passage never reached the model.

A pragmatic path

For most teams, this order works:

1. Start with prompting on a strong general model to learn what users actually ask.
2. Add retrieval over the sources that matter, with permissions and citations, and build an eval set from real questions.
3. Improve retrieval — hybrid search, better chunking, re-ranking — measuring each change against your evals.
4. Fine-tune selectively for the specific sub-tasks where consistency, cost or latency justify it — often a small model for classification or extraction inside the larger system.

RAG and fine-tuning aren't rivals. The strongest systems use retrieval for knowledge and, where it pays off, fine-tuned models for behaviour.