Prompt engineering changes a model's instructions, RAG changes the information it sees on each request, and fine-tuning changes the model itself. Start with prompting, because it is the cheapest to try and change. Add RAG when answers must come from your own documents, especially if they change or need citations - that is what most people mean by training the AI on their data. Consider fine-tuning only for a narrow, high-volume task where a measured gap in format, style or cost remains and you have labelled examples to prove the tuned model is better. Decide with a test set of 30-50 real cases.
What is the difference between fine-tuning, RAG and prompt engineering?
They change different things. Prompt engineering changes the instructions you give a model on each request. Retrieval-augmented generation (RAG) changes the information the model can see on each request, by finding the relevant passages in your documents and handing them over with the question. Fine-tuning changes the model itself, by training it further on examples of the input and output you want.
The useful way to remember it: prompting shapes behaviour, RAG supplies knowledge, and fine-tuning bakes in a habit. Most business problems are a knowledge problem or a behaviour problem, which is why most of them are solved without fine-tuning at all.
How do fine-tuning, RAG and prompt engineering compare?
| Prompt engineering | RAG | Fine-tuning | |
|---|---|---|---|
| What it changes | The instructions on each request | The information on each request | The model's weights |
| Best for | Tone, format, steps, rules | Answering from your own, changing documents | A consistent style or narrow task at high volume |
| Time to first result | Hours | Days to weeks | Weeks, including preparing data |
| Up-front cost | Lowest | Moderate: parsing, indexing, search | Highest: labelled data, training, evaluation |
| Keeping it current | Edit the prompt | Re-index changed documents | Retrain when behaviour must change |
| Can cite its source | No | Yes | No |
| Typical failure | Instructions ignored on edge cases | Wrong or stale passage retrieved | Learns the wrong lesson from the data |
When is prompt engineering enough?
More often than people expect, and it should always be tried first because it is the cheapest to change. Current models follow detailed instructions well, and a few worked examples inside the prompt often fix what looks like a need for training.
Prompt engineering is usually enough when:
- The knowledge needed is general, or short enough to include in the prompt itself.
- The problem is format or tone: always reply in this structure, never promise delivery dates.
- The task is a sequence of steps you can write down.
- You can show what good looks like with a handful of examples.
Prompts are also where safety rules start, though anything that must never happen belongs in code as well. A prompt is an instruction, not a guarantee.
When do you need RAG?
When the answer lives in your own documents and those documents are too many, too long or too frequently updated to put in the prompt. Policies, product documentation, contracts, support history, internal wikis. RAG is the right tool when you need the model to answer from current facts and show where each answer came from.
RAG is also the honest answer to "can we train the AI on our documents?" In most cases what the person means is that they want answers grounded in those documents, with citations, that update when the documents change. That is retrieval, not training. Fine-tuning a model on documents does not reliably teach it the facts in them, and it cannot tell you which document an answer came from.
We have written about the build and the testing separately: how to implement retrieval augmented generation, QA for RAG systems, and 7 signs your RAG chatbot is giving wrong answers.
When does fine-tuning actually make sense?
When you need a model to do one narrow thing consistently, at a volume where it matters that each request is short, cheap or fast - and prompting has already been tried and measured. Typical cases:
- A fixed output format or house style that long prompts still get wrong on a measurable share of cases.
- A narrow classification or extraction task at very high volume, where a smaller fine-tuned model can replace a larger general one.
- Domain language the general model handles poorly, where you have thousands of good labelled examples.
- Removing a long prompt from every request, when the prompt itself has become a large part of the bill.
What fine-tuning costs that is easy to miss
The training run is rarely the expensive part. The expensive parts are assembling and cleaning enough labelled examples, building an evaluation set to prove the fine-tuned model beats the prompted one, and repeating the work when the base model you tuned is retired or superseded. A fine-tuned model is a model you now maintain.
What fine-tuning does not fix
It does not keep knowledge current, it does not provide citations, and it does not stop a model making things up. If those are the problems, fine-tuning is the wrong tool however much data you have.
Can you combine fine-tuning, RAG and prompt engineering?
Yes, and production systems usually combine at least two. Almost every RAG system is also a prompt engineering exercise: the retrieved passages arrive inside a carefully written prompt that tells the model to answer only from them and to say when they do not contain the answer. A smaller number of systems add a fine-tuned model on top, typically for one high-volume step such as classifying the incoming question before retrieval.
The order matters. Each layer should be added only once the one below it has been measured and found short.
Which approach fits common business use cases?
A starting point for the cases we are asked about most. Each is a first guess to test, not a verdict - your own test set decides.
| Use case | Start with | Why |
|---|---|---|
| Support chatbot answering from help articles | RAG + prompting | Answers must come from current articles, with links to them |
| Drafting replies in your brand voice | Prompting with examples | Tone is a behaviour problem; a few good examples usually fix it |
| Searching contracts or policies | RAG | Needs exact passages and citations, and documents change |
| Sorting incoming emails or tickets into categories | Prompting, then fine-tuning at high volume | A prompt works first; a small tuned model can cut cost at scale |
| Extracting fields from invoices or forms | Prompting with structured output | Format is enforceable in code; tune only if errors persist at volume |
| Internal assistant over company knowledge | RAG + prompting | Knowledge is private and changing; answers need sources |
| Writing in a highly specialised house style | Prompting, then fine-tuning | Tune only if long prompts still miss on a measured share of cases |
The pattern is consistent: knowledge problems point to retrieval, behaviour problems to prompting, and fine-tuning earns its place only when volume or consistency makes the measured difference worth maintaining a model of your own.
How do you decide which one your use case needs?
- 01
Build a test set first
Thirty to fifty real inputs with the correct output written down. Without it, you cannot tell whether any of the three approaches worked.
- 02
Try a well-written prompt
With clear instructions and a few examples. Score it on the test set. Many projects stop here.
- 03
Add retrieval if knowledge is missing
If wrong answers come from not knowing your facts, add RAG and score again - retrieval and answers separately.
- 04
Consider fine-tuning last
Only if a measured gap remains in format, style or cost at volume, and you have the labelled data to train and evaluate with.
The test set is the step people skip and the one that makes the decision obvious. How we build one is in evaluating LLM output in production.
Fine-tuning vs RAG vs prompt engineering: which do you need?
Start with prompt engineering, because it is the cheapest to try and change. Add RAG when the model needs to answer from your own documents, especially if they change or you need citations - that is what most people mean by "training the AI on our data". Consider fine-tuning only for a narrow, high-volume task where a measured gap in format, style or cost remains, and you have the labelled examples to prove the tuned model is better. Decide with a test set of real cases, not by preference.
If your use case points to retrieval, see RAG development for how we build it.