0%

0000000

0x00

Fine-Tuning vs RAG for a Self-Hosted LLM: Facts or Behaviour

Abdullah ChaudarySelf-Hosted AI12 min read

Black card: Retrieve for facts. Fine-tune for behaviour. Two lanes into one model: retrieved documents and a fine-tune lane into the weights. Looping GIF.

Retrieve for facts, fine-tune for behaviour. In a Qwen3 demo, retrieval took a changed fact in 0.014 s while LoRA missed 3 of 8 facts, yet LoRA got a JSON contract right 10 of 10.

On 19 October 2026 I changed one line in an invented document, "kept for 5 years" to "kept for 7 years", and asked the same question again. The retrieval index rebuilt in 0.014 seconds and the model answered "after 7 years". The LoRA adapter, trained on the same facts for 600 seconds, answered "30 days": neither the old value nor the new one.

Fine-tuning vs RAG for a self-hosted LLM comes down to that exchange: facts change, and weights do not change with them.

I led an AI/pharma data platform running self-hosted open-weight models (GPT-OSS, Llama, Qwen), where I fine-tuned them and built a custom chat and evaluation engine. I am not reporting that platform's results here. The failures and numbers below come from a demo with invented, labelled sample data, run on Qwen3-0.6B on a CPU on 19 October 2026.

TL;DR: Retrieve for facts. In the demo, TF-IDF retrieval answered 8 of 8 reworded questions and took a changed fact in 0.014 s, while a LoRA adapter trained on those facts got 5 of 8 and would need another 600-second run to learn the change. Fine-tune for behaviour: a LoRA adapter trained on 60 labelled notes got all four fields right on 10 of 10 held-out notes, where the best prompt managed 2 of 10. Decide with an evaluation set, not a hunch.

Should a Private Data Platform Fine-Tune an LLM or Use RAG?

Start with retrieval for anything the model must state as a fact, and keep fine-tuning for behaviour: an output contract, a tone, a narrow judgement the prompt cannot pin down. Lewis et al. (2020) built RAG because parametric models struggle to show where an answer came from and to update what they know.

A private document set demands both: documents get revised, and a regulated reader wants the source. A self-hosted platform keeps data on hardware you own, which rules out hosted APIs but not either technique. On the platform I led, retrieval ran on Qdrant with BGE-M3, written up in full in hybrid search with Qdrant and BGE-M3, so this post does not repeat it.

Fine-tuning is the answer less often than it is the question, and the demo below shows why retrieval comes first.

How Did I Test Fine-Tuning Against Retrieval?

I ran two experiments on Qwen3-0.6B (revision c1899de) on an 8-core CPU with no GPU, using greedy decoding and fixed seeds. A 0.6B model trains in minutes on a CPU, which is the point: the demo tests the techniques, not the model's ceiling.

  1. Facts. Twelve invented documents for a fictional lab, "Northwind Labs": storage temperatures, an SOP number, a courier, a records retention period. Eight held-out questions, each a rewording that never appears in training. Three setups, then one fact changed:
  • the base model alone
  • the base model with retrieval (scikit-learn TF-IDF, top 2 documents placed in the prompt)
  • a LoRA adapter trained on 24 question-and-answer pairs (8 facts, 3 phrasings each)
  1. Behaviour. A strict JSON contract: one object, four keys (category, priority, site, needs_followup), fixed enum values, no prose and no code fences. Ten held-out incident notes, scored by a deterministic validator, so no model judged another. Three setups: the base model with a careful instruction prompt, the base model with three retrieved examples, and a LoRA adapter trained on 60 example notes.

The host was shared with other jobs during the run, so wall times are slow; only the ratios inside one run mean anything.

What Happens When a Fine-Tuned Model Meets a Changed Fact?

The fine-tuned model keeps what it was trained on, or worse, blends it. Retrieval answered all 8 reworded questions and picked up the changed fact after a 0.014-second re-index. The LoRA adapter answered 5 of 8 rewordings and gave a wrong value after the change.

These are the scores on the 8 held-out questions:

  • Base model alone: 0 of 8. It had never seen the facts and guessed or deflected.
  • Base model plus TF-IDF retrieval: 8 of 8. The right document came back for every question.
  • LoRA adapter, asked its training phrasings: 8 of 8, word for word.
  • LoRA adapter, asked the rewordings: 5 of 8.

The adapter trained for 48 steps over 8 epochs. Training loss fell from 1.7077 to 0.0003, yet the adapter had learned the training strings rather than the facts. Its misses were confident and borrowed from other facts:

Q: What percentage must a new QC analyst reach on TM-204?
A: 'QC lab workstations are tested every 90 days.'

Q: Who carries Northwind's outgoing clinical samples?
A: 'Outgoing clinical samples ship with blue boxes.'

Ovadia et al. (2023) saw the same pattern with larger models and unsupervised fine-tuning: RAG "consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge", and exposing a model to "numerous variations of the same fact" during training could ease the problem. The demo trained on three phrasings per fact, and that was not enough for 3 of 8 rewordings.

Then the update. The retention line changed from 5 years to 7 years, the index was rebuilt, and both setups got the same question:

[update rag] reindex 0.014s
A: 'After a Harlow batch is released, its batch
    records can be destroyed under supervision
    after 7 years.'
[update lora]
A: 'Harlow plant batch records are destroyed
    30 days after batch release.'

The "30 days" came from a different document, the shelf life of an opened buffer. Before the change, asked in its training wording, the adapter had answered "kept for 5 years". Teaching it "7 years" means new training pairs and another run like the first, which took 600.106 seconds against 0.014 seconds to re-index.

The retrieval here is deliberately naive, a TF-IDF ranker over short clean documents. That makes the result a lower bound for retrieval. This is all of it:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel

class TfidfIndex:
    def __init__(self, texts):
        self.texts = list(texts)
        self.vec = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)
        self.mat = self.vec.fit_transform(self.texts)

    def search(self, query, k=2):
        sims = linear_kernel(self.vec.transform([query]), self.mat)[0]
        order = sims.argsort()[::-1][:k]
        return [(int(i), float(sims[i])) for i in order]

Can Fine-Tuning Fix Behaviour That Prompting Cannot?

Yes, when the hard part is judgement rather than syntax. In the demo a careful prompt already produced valid JSON 9 times in 10, but got all four values right only once. A LoRA adapter trained on 60 labelled notes returned valid JSON with every value right on all 10 held-out notes.

These are the results on the 10 held-out notes. "Strict valid" means the whole reply parses and matches the contract; "all four right" also needs every value to match the label:

  • Careful instruction prompt: 9 of 10 strict valid, 1 of 10 with all four fields right, 25 of 40 fields right.
  • Prompt plus three retrieved examples: 10 of 10 strict valid, 2 of 10 with all four fields right, 25 of 40 fields right.
  • LoRA adapter on 60 notes: 10 of 10 strict valid, 10 of 10 with all four fields right, 40 of 40 fields right. It trained for 45 steps over 3 epochs (loss 0.0681 down to 0.0028) in 1,531.898 seconds on the shared CPU.

The one format failure from the prompt alone was valid JSON wrapped in a markdown code fence, which the contract forbids:

[base] #7 strict=False (invalid JSON: JSONDecodeError) fields=0/4
RAW, inside a markdown fence labelled json:
{
  "category": "equipment",
  "priority": "low",
  "site": "dundee",
  "needs_followup": false
}

The bigger failure was quieter: valid JSON with wrong values. The base model labelled 9 of 10 notes "equipment", including a missing signature in a logbook, which is a documentation issue. Retrieved examples removed the fence but did not raise field accuracy, and in one case the model copied the site from an example instead of reading the note.

The adapter fixed both. On the logbook note it answered "documentation", and it never added a fence:

[lora] #0 strict=True (ok) fields=4/4
    RAW: '{"category": "documentation", "priority": "medium",
          "site": "harlow", "needs_followup": true}'

This is the case fine-tuning is for. The contract and the labelling rules were stable, 60 examples taught them, and no document had to change for the answer to be right. The caution is the test set: the 10 notes were written from the same templates as the training notes, with no exact overlap, so they are easier than real incident reports.

Which Open-Weight Model Fits: GPT-OSS, Llama or Qwen3?

Choose GPT-OSS, Llama 3.3 or Qwen3 by licence, size and format rules before quality, because those decide what you are allowed to run and whether it fits your hardware. GPT-OSS and Qwen3 are Apache 2.0. Llama 3.3 has a community licence with a 700 million monthly active user threshold and naming rules. Quality on your task is something you measure.

  • GPT-OSS. Two mixture-of-experts reasoning models under Apache 2.0, released in August 2025 (model card, arXiv:2508.10925). gpt-oss-20b has 21B total and 3.6B active parameters and runs within 16 GB; gpt-oss-120b has 117B total and 5.1B active and fits one 80 GB GPU. Its model card says it should only be used with the harmony format, "as it will not work correctly otherwise", so training data has to be built in that format.
  • Llama. Llama 3.3 70B Instruct was released on 6 December 2024 with a 128k context, and fine-tuning is permitted. The Llama 3.3 Community License requires a separate licence above 700 million monthly active users, the "Built with Llama" notice, and a derived model name that begins with "Llama". Check the licence of the exact version you tune.
  • Qwen3. Released on 29 April 2025 as six dense models (0.6B to 32B) and two mixture-of-experts models, with thinking and non-thinking modes. The technical report (arXiv:2505.09388) says all are available under Apache 2.0. The Qwen3-14B model card recommends serving with vLLM 0.8.5 or later, or SGLang 0.4.6.post1 or later.

The demo used Qwen3-0.6B only because it trains on a CPU; it says nothing about how GPT-OSS or Llama would score. Whether owning the GPUs for the larger models pays off is a separate sum, covered in what self-hosting really costs against a cloud bill.

What Do LoRA and QLoRA Change About the Cost of Fine-Tuning?

LoRA freezes the base weights and trains small low-rank matrices beside them, so a fine-tune touches a fraction of the model. Hu et al. (2021) report about 10,000 times fewer trainable parameters and about 3 times less GPU memory than full fine-tuning of GPT-3 175B, with no added inference latency.

In the demo, rank 16 on every linear layer meant 10,092,544 trainable parameters out of 606,142,464, or 1.665%. This is the configuration:

from peft import LoraConfig, get_peft_model

cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.0, task_type="CAUSAL_LM",
                 target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                 "gate_proj", "up_proj", "down_proj"])
model = get_peft_model(base_model, cfg)   # AdamW, lr 2e-4, batch 4, loss on answer tokens only

QLoRA (Dettmers et al., 2023) trains the same adapters on top of a frozen 4-bit model and fine-tuned a 65B model on one 48 GB GPU; the paper reports no loss against 16-bit fine-tuning on its benchmarks. I did not use QLoRA here. The cost that LoRA does not shrink is the data: every fact or behaviour you want in the weights needs examples written, checked and kept current.

How Do You Evaluate a Fine-Tuned Model Against RAG?

Write the evaluation set first, score the base model with retrieval as the baseline, then score the tuned model on the same items. OpenAI's model optimisation guide says to "consider writing evals before you start writing prompts", and its loop runs evals, prompt, fine-tune, evaluate, adjust.

Four habits made the demo's comparison usable:

  1. Held-out rewordings, not training questions. On its training phrasings the adapter scored 8 of 8. On rewordings it scored 5 of 8. Testing on training questions would have reported a fix that was not there.
  2. A validator, not a judge model. Parsing the reply and checking keys and enums is cheap and gives the same verdict every time.
  3. Greedy decoding and fixed seeds. With 8 or 10 items, sampling noise can be larger than the gap you are measuring.
  4. A second run after the data changes. The fact update was the evaluation that separated the setups most clearly.

On the platform I led, the custom evaluation engine did this job; its results are not in this post. For a different system scored against hand-checked documents, see evaluating LLM extraction on hand-checked documents.

Fine-Tuning vs RAG for a Self-Hosted LLM: How Do You Decide?

Name the failure first. A wrong, missing or outdated fact points to retrieval. A broken format, tone or narrow judgement points to prompting, then retrieved examples, then fine-tuning, in that order. Let the evaluation set decide when to move to the next step.

  1. The answer must quote or cite a document. Retrieve. Weights cannot point to a source.
  2. The facts change. Retrieve. In the demo, absorbing a change cost 0.014 s by re-index against a 600-second training run.
  3. The output must match a strict schema. Write the validator, then try a careful prompt and retrieved examples. In the demo those two reached 9 and 10 of 10 valid JSON.
  4. The values are wrong even when the format is right. That is a judgement the model lacks, and the case where fine-tuning on labelled examples is worth testing. In the demo, 60 labelled notes took field accuracy from 25 of 40 to 40 of 40.
  5. The prompt carries many examples on every call. Fine-tuning can shorten it; OpenAI's guide lists shorter prompts and a smaller, cheaper model among the reasons to tune.
  6. There is no evaluation set yet. Build one before anything else on this list.

Limitations: What Does This Demo Not Cover?

Every figure comes from one demo run on 19 October 2026: invented documents, 8 questions and 10 notes, one small model. None of it is a production result or the platform's code.

  • One 0.6B model on a CPU. Larger models score higher in every setup. The ranking between setups is the point, not the absolute numbers. I did not run GPT-OSS or Llama.
  • Easy retrieval. Short, clean documents suit TF-IDF. Real document sets have near-duplicates, tables and conflicting versions.
  • Template-made test notes. The 10 behaviour notes came from the same templates as the 60 training notes (no exact overlap). Real incident reports vary more, so 10 of 10 is an upper bound.
  • Tiny evaluation sets. One answer moves a score by 10 or 12.5 points. Nothing here is statistically significant.
  • The update was asked once. After the change, the adapter was asked only the reworded question; its training wording was not re-asked.
  • No QLoRA, adapter merge or serving stack. Nothing was quantised, merged or served through vLLM.
  • Hosted fine-tuning. OpenAI's guide, as read on 19 October 2026, says OpenAI is winding down its hosted fine-tuning platform for new users. That change concerns OpenAI's hosted service only.
  • A shared host. Other jobs ran during the demo, so wall times are slow; only ratios within the run are meaningful.

Retrieval fixes what the model does not know. Fine-tuning changes how it answers, and only a held-out evaluation tells you which one you need.

If you have seen a fine-tuned model beat retrieval on factual questions and keep winning after the documents changed, how did you keep the training data current? In this demo the fact moved and the weights did not.

More in Self-Hosted AI field notes.

References

next up

Postgres Schemas Behind FastAPI: Nine Namespaces, Not Walls