RAG hands a model your documents at question time; fine-tuning changes how it answers. They fix different problems. If the trouble is knowledge (facts the model never saw, facts that change, answers that need a source), use retrieval-augmented generation. If the trouble is behavior (a format, a set of labels or a tone the model keeps missing even with a careful prompt), fine-tuning is the tool.
Teams with both problems can use both, and one study below found the gains add up. That rule comes from what each method touches, from studies that ran both on the same questions, and from one date in OpenAI’s documentation that anyone planning to fine-tune should know about.
What each method changes
Fine-tuning trains an existing model further on your examples, so its weights change. OpenAI’s supervised fine-tuning guide asks for pairs of prompts and ideal answers. It accepts as few as 10 examples, says improvements usually show up with 50 to 100, and suggests starting with 50 well-made ones. Google’s tuning overview for Gemini says to think of 100 examples or more, and to start with prompting before moving to tuning.
Many teams do not retrain every weight. The LoRA paper froze the original weights and trained small added matrices instead. On GPT-3 175B, the authors report 10,000 times fewer trainable parameters and a third of the GPU memory of full fine-tuning, with similar quality. Either way, the result is a new model version you have to host, evaluate and replace when something changes.
RAG leaves the model alone. A search step finds passages in your documents and places them in the prompt next to the question. Our RAG explainer covers the pipeline, and the embeddings guide covers how the search ranks passages. To change what the system knows, you edit the index, not the model.
The vendors’ own use lists follow the same split. OpenAI’s model optimization guide names classification, nuanced translation, output in a specific format and fixing instruction-following failures as the jobs supervised fine-tuning suits. For facts outside the training data, the same guide says to put the relevant context in the prompt, which is what retrieval automates.
What happened when both were tested on new facts
Four Microsoft researchers built the cleanest test of the knowledge question in Fine-Tuning or Retrieval? (first posted December 10, 2023, revised January 30, 2024). They wrote 910 multiple-choice questions about US events from August to November 2023, after the training data of the three 7-billion-parameter models they tested. Then they either fine-tuned each model on the Wikipedia text about those events or gave the base model that text through retrieval.
The fine-tuning here was unsupervised: the models were trained on the articles themselves, not on question-and-answer pairs. Accuracy on the current-events questions:
- Mistral 7B: 48.1% with no help, 50.4% after fine-tuning on the articles, 58.8% after fine-tuning on ten paraphrased versions of each passage, and 87.5% with retrieval.
- Llama 2 7B: 35.3% with no help, 21.9% after fine-tuning (worse than doing nothing), 39.2% with paraphrases, and 58.5% with retrieval.
- Orca 2 7B: 45.6% with no help, 51.1% after fine-tuning, 56.6% with paraphrases, and 87.6% with retrieval.
For Mistral, retrieval added 39.4 percentage points and plain fine-tuning added 2.3, about 17 times less. Paraphrasing helped fine-tuning, which the authors read as a sign that a model needs to see a fact many times, in different words, before it sticks. Pairing the paraphrase-tuned Mistral with retrieval scored 83.0%, below retrieval on the untouched model.
A second study points the same way for niche facts. Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge (Soudani, Kanoulas and Hasibi, 2024) tested twelve models. Fine-tuning on generated question data helped across the board, but retrieval beat it by a large margin on the least popular entities, the kind of thing a company’s own product names or internal codes resemble.
The combined case also has evidence. In RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture (January 2024), another Microsoft team fine-tuned models on question-and-answer pairs generated from farming documents. They report accuracy up more than 6 percentage points from fine-tuning, plus 5 more when retrieval was added on top. The fine-tuned model also drew on information from other regions, raising answer similarity from 47% to 72% in one experiment.
These are research results on small open models with multiple-choice or generated questions, mostly from 2023 and 2024. They show which way the effect runs, not the score your own system will get.
A worked example: one help desk, two problems
Take a support team with a 300-page policy manual that changes most weeks. Agents want draft replies that quote the right policy, and every draft has to follow a fixed layout and carry one of 12 category labels.
The policy facts. Anthropic’s retrieval guide says a knowledge base under 200,000 tokens, about 500 pages, can go into the prompt whole, with no retrieval step. That works out to roughly 400 tokens a page, so the 300-page manual is about 120,000 tokens and fits under that line. When the manual grows past about 500 pages, retrieval takes over. In both setups, a policy change means swapping the text that is sent or indexed.
Fine-tuning the same facts in would mean a new training run after each significant change. If the team copied the method that worked best for fine-tuning in the Microsoft test, ten paraphrases per passage, then 120,000 tokens split into 300-token chunks gives 400 chunks and 4,000 paraphrased passages to generate and check per run. And on that test, the result would still trail retrieval.
The layout and the labels. This is a behavior problem, and it is where fine-tuning earns a place. The team should first put the layout and a few labeled examples in the prompt, then measure. If drafts still drift, 50 to 100 tickets labeled the way the team wants are a realistic starting set under OpenAI’s and Google’s guidance.
The split: the manual goes in the prompt or behind retrieval, and fine-tuning is held back for the format, only if prompting fails.
Check the platform before planning a fine-tune
On OpenAI, fine-tuning is closing. Its deprecations page gives the timeline:
- May 7, 2026: organizations that had never run fine-tuning could no longer create fine-tuning jobs.
- July 2, 2026: organizations with no inference on a fine-tuned model in the previous 60 days lost job creation too.
- January 6, 2027: remaining customers can no longer create new fine-tuning jobs.
Fine-tuned models keep running until their base model shuts down. The supervised fine-tuning guide lists October 23, 2026 for gpt-4.1-nano and its fine-tuned versions, and January 14, 2027 for gpt-4.1 and gpt-4.1-mini and theirs. A model trained in December 2026 on gpt-4.1 would therefore have weeks of life left.
Google still offers supervised fine-tuning, preference tuning and reinforcement-learning fine-tuning for Gemini, according to its tuning overview (last updated November 13, 2026). The general point holds across vendors: a fine-tuned model is tied to the base model it came from. A retrieval index survives a model change, although the vectors must be rebuilt if you switch embedding models.
How to choose
- The facts change, or answers need a citation: use retrieval, or put the whole source in the prompt if it is under about 500 pages.
- The facts are rare or company-specific: retrieval, per the less-popular-knowledge results above.
- The model knows enough but answers in the wrong shape: improve the prompt and add examples first. Fine-tune with 50 to 100 good examples only if that fails.
- You have both problems: use both, and measure each change separately.
- Before either: write a test set of real questions, including some your documents cannot answer, and score every version against it. Our guide to why AI hallucinates explains why the unanswerable ones matter.




