Fine-Tuning

Watch how additional training can adapt behavior, style, or domain performance — and where that differs from retrieval or prompt design.

LLM fine-tuning continues training a pretrained model on a smaller, curated dataset so that its weights, or a small set of added adapter weights, change to produce a desired behaviour more reliably: a consistent output format, a house style, a specialised task. So when should you fine-tune instead of using RAG or prompting? Fine-tune when the problem is inconsistent behaviour that prompts and examples cannot fix reliably; prompt when the instruction is simply missing; and use RAG when the model lacks current or private facts, which are better supplied at request time (see Why RAG Exists).

A SaaS company wants its support assistant to read a ticket and produce a JSON triage record: category (one of 14), urgency (1 to 4), a suggested owner team, and a two-sentence customer-facing summary in the company's voice. Prompted with a detailed instruction and three examples, the general model gets the shape right most of the time. But the engineers observe two things: some tickets come back with invented category names or an extra paragraph before the JSON, and each request carries a 1,800-token prompt of rules and examples that they pay for on every call.

Before deciding anything, they establish a baseline: 500 historical tickets with agreed correct triage records, run through the prompted model, scored on valid-JSON rate, category accuracy, urgency agreement, and reviewer-rated summary quality. Fine-tuning is a candidate only if that baseline shows a gap that better prompting, retrieval of examples, or constrained output does not close. The question to keep asking is: is the problem missing knowledge, missing instruction, or inconsistent behaviour? Only the last is typically a fine-tuning problem.

These are often listed as alternatives, but they act at different points and change different things.

Pretraining learns general language ability from huge corpora and changes all weights, at great cost (see the LLM Overview). Fine-tuning continues training on a small, targeted dataset and changes weights (all or adapter). Prompting changes only the input text; weights are untouched. In-context learning is what happens when examples in the prompt steer the model's output without any weight change; it vanishes when the prompt does. RAG retrieves documents at request time and puts them in the prompt, so it changes what the model can see, not what it is (see RAG Architecture). Only pretraining and fine-tuning change weights; everything else changes inputs. That is the decisive distinction when deciding how to update a system: weight changes are slow to make, harder to reverse, and need a new evaluation; input changes can be deployed in minutes.

Fine-tuning quality is dominated by data. A common format is one JSON object per line (JSONL), each pairing an input with the ideal output. For an instruction-style dataset:

{"instruction": "Triage this support ticket.", "input": "Ticket: Our nightly export has failed 3 times since the 14th release. Error 502.", "output": "{\"category\":\"data_export\",\"urgency\":2,\"owner\":\"platform-team\",\"summary\":\"Your nightly export is failing with a 502 error since the latest release. We have routed this to our platform team and will update you shortly.\"}"}

{"instruction": "Triage this support ticket.", "input": "Ticket: How do I change the billing email?", "output": "{\"category\":\"billing_howto\",\"urgency\":1,\"owner\":\"support-tier1\",\"summary\":\"You can update the billing email under Settings > Billing. Let us know if you cannot find it.\"}"}

{"messages": [
  {"role": "system",    "content": "You triage tickets for ShopCo and reply with JSON only."},
  {"role": "user",      "content": "Ticket: Charged twice for order 88213."},
  {"role": "assistant", "content": "{\"category\":\"billing_dispute\",\"urgency\":3,\"owner\":\"billing-team\",\"summary\":\"...\"}"}
]}

Good dataset practice includes: recording provenance for every example and checking licence and privacy constraints (are customer messages allowed in training data under your terms; have identifiers been removed?); deduplicating near-identical tickets so frequent cases do not dominate; checking label consistency (two reviewers triaging the same ticket differently teaches the model noise); and balancing classes so rare, high-urgency cases are present. Split the data into train, validation, and test sets before any training, ideally by time or by customer rather than at random, so that the test set resembles the future and so that near-duplicates (the same incident reported by five users) do not leak across splits. Keep the test set untouched until the release decision.

Supervised fine-tuning (SFT) uses the same next-token objective as pretraining, with differences in what is counted. Each record is rendered with the chat template and tokenized. The loss is computed only on the target tokens (the assistant response); the instruction and input tokens are in the context but are masked from the loss, so the model learns to produce answers, not to reproduce the ticket text. Records are batched (minibatches, with padding or packing), the optimiser (commonly AdamW) updates the trainable weights using a learning rate that is usually much lower than in pretraining, and training runs for a small number of epochs, watching validation loss.

tokens   : [system][user: ticket text ........][assistant: {"category": ... }]<end>
loss mask:   0        0 0 0 0 0 0 0 0           1 1 1 1 1 1 1 1 1 1 1 1 1 1 1     <- loss only on the answer

Epoch count and learning rate trade off underfitting and overfitting. Too few steps and format errors persist; too many and the model memorises training answers, and validation loss starts rising while training loss keeps falling. Save checkpoints periodically and choose using validation metrics (format validity, accuracy), not only loss. Because SFT teaches by imitation, label quality is the ceiling: the model learns the average behaviour of your examples, including their mistakes and inconsistencies.

Move the checkpoint below to see how the gap between training and validation loss opens up.

Full fine-tuning updates every weight. For a multi-billion-parameter model, training needs memory not only for the weights, but for gradients and optimiser state (AdamW keeps two extra values per trainable parameter), so it is much more than inference needs, and you end with a complete new copy of the model per task. Parameter-efficient fine-tuning (PEFT) instead freezes the base weights and trains a small number of added parameters, a small fraction of the full model's weights.

LoRA (Low-Rank Adaptation; Hu et al., 2021) rests on the observation that useful weight updates often have low intrinsic rank. For a frozen weight matrix W (d×k), LoRA learns two small matrices B (d×r) and A (r×k) with rank r much smaller than d and k, and uses

W' = W + (alpha / r) * B @ A          # W frozen; only A and B are trained; (alpha / r) is an optional scaling

The arithmetic

One attention projection, 4096 x 4096:
   full matrix                = 4096 x 4096                = 16,777,216 weights
LoRA with rank r = 8:
   A (8 x 4096) + B (4096 x 8) = 32,768 + 32,768             =     65,536 trainable weights
   fraction                    = 65,536 / 16,777,216          =   0.39 %   (1/256)

Change the rank to see how the trainable count for this one matrix scales.

This number is for one matrix. A model applies adapters to selected matrices in many layers, so the total trainable count is the per-matrix count multiplied by the number of adapted matrices. It is also not the total memory or compute for training: the full base model must still be loaded (and its activations stored for backpropagation), so peak memory depends on model size, sequence length, batch size, and precision, not just the adapter size. What shrinks is optimiser state and gradient storage, and the artifact you ship: an adapter of megabytes rather than a full model copy. At inference the adapter can be merged into W (no added latency) or kept separate so many adapters share one base.

QLoRA (Dettmers et al., 2023) loads the frozen base model in 4-bit quantised form and trains LoRA adapters in higher precision on top, which makes fine-tuning of larger models feasible on much smaller GPUs. Caveats: quantisation can slightly affect quality; the implementation depends on specific library and hardware support; and an adapter trained against a quantised base should be evaluated in the configuration you will actually serve.

1. 1

   Baseline

   Measure the prompted (and, if relevant, retrieval-augmented) model on a held-out set. Decide what improvement would justify the effort.
2. 2

   Data quality

   Clean, deduplicate, resolve label disagreements, remove personal data, and split.
3. 3

   Formatting

   Convert to the target chat template; verify with a few decoded examples that loss masks sit where intended.
4. 4

   Train

   Start with a small run, monitor loss curves, save checkpoints.
5. 5

   Validate

   Pick a checkpoint on validation metrics.
6. 6

   Held-out evaluation

   Run the test set once, comparing with the baseline on the same metrics.
7. 7

   Safety and regression tests

   Check refusal behaviour, injection resistance, and unrelated capabilities that you rely on.
8. 8

   Stage, canary, monitor, roll back

   Serve to a small percentage of traffic, compare live metrics, and keep the previous version deployable.

The same workflow as a data and artifact flow, including where a failed gate sends you back:

Overfitting: training loss falls, validation loss rises, and outputs on new phrasings degrade; visible when validation or test metrics lag behind training metrics. Catastrophic forgetting: the model gains the narrow skill and loses general ability, such as following unrelated instructions or maintaining safety behaviours; visible only if you run a general regression suite, so include one. Distribution shift: the tickets a year from now differ from the training set (new products); visible as slow metric decay in monitoring. Privacy leakage: models can reproduce rare training strings, particularly when trained for many epochs on small data; test by prompting for training-record fragments, and keep personal data out of training sets. Benchmark contamination: if test items leak into training data (including near-duplicates), reported scores are inflated; avoid by splitting before any processing and checking overlap.

The team wants a slightly friendlier tone. Change the prompt; test; ship today. The assistant must answer about the product's pricing, updated monthly. Retrieval (or a pricing tool), because facts change faster than any training cycle and need citations; baking prices into weights is both slow and unverifiable. All outputs must follow a 14-category schema with a strict style, and prompting still fails on 4 percent of tickets while each call carries 1,800 tokens of instructions. Fine-tune: behaviour is stable, examples are plentiful, and a tuned model can use a short prompt. Many real systems are hybrid: a fine-tuned model for format and style, retrieval for current facts, a prompt for the per-request task, and code for authorisation (see System Prompts vs User Prompts).

When not to train: you have fewer than a few dozen reliable examples; the task changes weekly; the failure is missing information rather than inconsistent behaviour; you cannot evaluate rigorously (you would not know if it improved); or the base model already meets the target once prompted well. A tuned model that you cannot evaluate is a risk, not an asset.

The sketch below uses the Hugging Face transformers, datasets, peft and trl libraries, and assumes a CUDA GPU with enough memory for your chosen base model, a recent Python environment with these packages installed, a model whose licence allows your use, and a dataset in the chat-message format shown above. Library APIs change between versions (the trl trainer arguments in particular); this sketch was not executed as part of this lesson, so treat it as an outline to adapt and test against the documentation of your installed versions (PEFT and TRL).

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "your-org/base-model"                         # an open-weight model you are licensed to use
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="auto", device_map="auto")

ds = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

lora = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                  target_modules=["q_proj", "v_proj"],          # module names vary by architecture
                  task_type="CAUSAL_LM")

cfg = SFTConfig(output_dir="out/triage-lora-v1", num_train_epochs=2, learning_rate=2e-4,
                per_device_train_batch_size=4, gradient_accumulation_steps=4,
                eval_strategy="steps", eval_steps=100, save_steps=100, logging_steps=20,
                assistant_only_loss=True)                         # option name/availability depends on trl version

trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds["train"], eval_dataset=ds["validation"],
                     peft_config=lora, processing_class=tok)
trainer.train()
trainer.save_model("out/triage-lora-v1")                         # adapter weights only

# Later, for inference:
#   base = AutoModelForCausalLM.from_pretrained(BASE, ...)
#   model = PeftModel.from_pretrained(base, "out/triage-lora-v1")
# Then run the held-out test set and compare with the baseline before any rollout.

An adapter is only meaningful relative to its inputs, so version them together: base model ID and revision, tokenizer and chat template, dataset version and hash, training configuration (rank, learning rate, seeds, library versions), the adapter artifact, the evaluation results, and the serving configuration. When something regresses, this record lets you reproduce or roll back.

Deployment compatibility checks matter: the serving stack must support the base architecture and the adapter format; serving merged weights versus dynamic adapters affects latency and memory. GPU memory is driven by the base model's weights (a function of parameter count and precision), the KV cache (grows with context length and concurrent requests; see Context Window), and batch size. Estimate cost by measuring tokens per second per GPU on your hardware at your realistic batch size and context, then multiplying by your provider's current GPU price; do not copy a price from an article, because they change.

GATE 1  held-out test set        : valid-JSON rate >= baseline + margin; category accuracy >= baseline
GATE 2  slice analysis           : no category below its minimum recall (especially urgency 4 tickets)
GATE 3  general regression suite : refusal behaviour and unrelated tasks within tolerance of base model
GATE 4  privacy probe            : no verbatim training-record fragments from extraction prompts
GATE 5  latency / cost           : p95 latency and tokens per ticket within budget
GATE 6  canary (5% traffic, 48h) : live valid-JSON rate, human-override rate; automatic rollback on breach

If the model passes, it ships with its provenance record and a rollback to the previous version. The next lesson looks at the other post-training stage that determines how all assistant models, tuned or not, behave: alignment.