Model Alignment

Watch how alignment turns a capable language model into a more useful assistant, then use the written lesson below to separate behavior shaping from truthfulness, security, and product reliability.

For language models, AI alignment means model alignment: the set of post-training methods that shape a pretrained model into an assistant that follows instructions, is more helpful, declines certain requests, and expresses uncertainty more appropriately. Supervised fine-tuning (SFT) teaches the model from demonstrated good responses, while preference optimisation (RLHF or DPO) pushes it toward responses that human or AI judges rated higher. Alignment changes the model's tendencies. It does not make the model truthful by construction, and it is not an access-control, security or authorisation boundary; those must be enforced by the surrounding system.

The following is a conceptual illustration, not a measured experiment. Given the text How do I reset my router?, a pretrained model is only a continuer of text. Its most probable continuations in web data include more questions or forum-like text:

How do I reset my router? How do I find my router's IP address? How do I change my WiFi password?
Related questions: Why is my internet slow? Best routers 2023...

To reset most home routers: 1) find the small recessed Reset button, 2) hold it for about 10 seconds with a paperclip
until the lights flash, 3) wait for the router to restart. Note that this erases your settings. If you only want a
reboot, unplug it for 30 seconds instead. Tell me the model and I can give exact steps.

Both outputs are plausible continuations; the base model is not "wrong". Alignment changed which continuation is likely, so that a request is treated as a request. How that is done, and how it can go wrong, is the topic of this chapter.

Typical objectives are helpfulness (completing the user's actual task), truthfulness (not asserting what it does not have support for), safety(declining or redirecting harmful requests), instruction following (honouring format and constraints), and appropriate uncertainty (saying "I'm not sure"). These pull against each other: maximal helpfulness would comply with harmful requests, and maximal caution refuses legitimate ones. Alignment is therefore a set of trade-offs encoded by rubrics, and there is no single universal "human values" target: a children's tutoring product, a coding assistant, and a medical-information service weigh them differently.

Equally important is what lives outside the model. Application authorisation ("this user may export customer data"), content policy enforcement, data access control, and rate limits are enforced by code and infrastructure. A model trained to be careful may still be manipulated; code does not get talked out of a permission check.

Real recipes vary: some combine rounds of SFT and preference optimisation, some use AI-generated feedback guided by written principles, some add rule-based or verifier-based rewards for tasks with checkable answers (maths, code). Treat the pipeline above as the common conceptual skeleton, not a specification of any particular model.

SFT trains on pairs of prompts and ideal responses written or curated by people (and, in many pipelines, model-assisted and filtered). The loss is the same next-token cross-entropy described in Next-Token Prediction, computed on response tokens. The mechanics of data formatting, loss masking and training are covered in Fine-Tuning, so here the focus is what it teaches.

PROMPT : "Explain what a database index is to a new developer, in under 80 words."
TARGET : "An index is a separate data structure, often a B-tree, that lets the database find rows by a column's
          value without scanning the whole table. Like a book's index, it speeds up reads, but costs extra
          storage and slows writes because it must be updated too."

SFT is simple and effective at establishing format, tone and the idea that prompts are requests. Its limits follow from imitation: the model can only be as good as the demonstrations, coverage is bounded by the topics written, and it can learn to imitate a confident style even where the demonstrator had knowledge the model lacks, which encourages unsupported answers. Writing high-quality demonstrations is also expensive, which motivates preference methods: it is easier to judge which of two answers is better than to write the ideal answer.

Annotators are shown a prompt and two or more candidate responses, often sampled from the current model, and asked which is better according to a rubric. Each judgment becomes a (chosen, rejected) pair.

PROMPT   : "Is it safe to take ibuprofen with my blood-pressure medication?"

CHOSEN   : "It depends on the medication. NSAIDs like ibuprofen can raise blood pressure and reduce the effect
            of some drugs, including ACE inhibitors and diuretics. Check with your pharmacist or doctor before
            combining them; I can help you list questions to ask."

REJECTED : "Yes, ibuprofen is perfectly safe with all blood-pressure medications."

WHY      : The rejected answer is confident, unsupported and potentially harmful. The chosen answer is accurate in
           outline, states the uncertainty, and points to appropriate help without refusing to engage.

Real labelling is messier. Annotators disagree: one prefers the concise answer, another the thorough one; some penalise caveats, others reward them. Annotator demographics, instructions and fatigue introduce bias; longer answers tend to be rated higher unless the rubric says otherwise; and cultural norms differ. Good programmes write explicit rubrics, measure agreement between annotators, adjudicate disagreements, audit for systematic bias, and treat "which is better" as a policy decision, not ground truth.

The steps below follow the recipe applied to instruction-following language models by Ouyang et al., 2022, which builds on learning rewards from human preferences (Christiano et al., 2017).

1. 1

   Gather preferences

   As above: pairs of responses with a chosen and a rejected one.
2. 2

   Train a reward model

   A model r(x, y) that outputs a scalar score for a prompt/response, trained so the chosen response scores higher. A common formulation (Bradley–Terry) sets P(chosen preferred) = σ(r_chosen − r_rejected), and minimises −ln σ(r_chosen − r_rejected). For example, with scores 1.2 and 0.4, the margin is 0.8, σ(0.8) ≈ 0.690, and the loss is −ln(0.690) ≈ 0.371. Larger margins give lower loss.
3. 3

   Optimise the policy

   Use reinforcement learning (PPO is the classic algorithm; others exist) to adjust the language model, the policy, to produce responses with a higher reward, subject to a penalty for drifting from a reference policy (usually the SFT model): maximise E[r(x,y)] − β · KL(π_θ ‖ π_ref).
4. 4

   Evaluate

   Check on held-out prompts, with human and automated evaluation, for improvements and regressions.

The sequence below shows one iteration of the policy-optimisation step, and the explorer after it reproduces the reward model loss from the worked example.

The KL term matters. Without it the policy would exploit the reward model, drifting to odd text that the reward model scores highly but humans would not. The reward model is a proxy trained on a finite set of judgments, and optimising a proxy hard enough breaks it (Goodhart's law). This is reward hacking or overoptimisation: for example, a reward model that mildly favours longer, more agreeable answers can lead the policy to produce verbose, flattering responses well beyond what annotators wanted. Not every aligned model is trained with PPO-based RLHF; the next section describes a common alternative.

Direct Preference Optimisation (Rafailov et al., 2023) removes the separate reward model and RL loop. In the standard setup it trains the policy directly on chosen/rejected pairs, comparing it with a frozen reference policy. The loss for a pair is:

L = -ln σ( β * [ ( ln π_θ(y_w|x) - ln π_ref(y_w|x) ) - ( ln π_θ(y_l|x) - ln π_ref(y_l|x) ) ] )

y_w = chosen, y_l = rejected, β = strength of the pull toward the reference policy

Toy numbers: chosen log-ratio rose by +3.0, rejected log-ratio fell by 2.0 (i.e. -2.0), β = 0.1
   inner term = 0.1 * (3.0 - (-2.0)) = 0.5      σ(0.5) = 0.622      L = -ln(0.622) = 0.474

Change the shifts and β to see how the loss responds; the defaults reproduce the toy numbers above.

Compared with reward-model-plus-RL, DPO is simpler to implement and more stable, with no sampling loop during training. But it learns from the fixed preference dataset it is given (it does not explore new responses unless you collect new pairs), and it inherits any weaknesses of that data. RLHF with a learned reward model can use fresh samples and a reusable scoring function. Many other methods exist (variants with different losses, AI-feedback approaches, verifiable-reward reinforcement learning), so choose by data, compute, and what you can evaluate.

Request: "Summarise the attached refund policy in two sentences and tell me if my $320 purchase from 40 days ago qualifies." The policy text says refunds are available within 30 days for purchases up to $500.

B is clearly best, and the pairs (B ≻ A, B ≻ C) are easy. The hard cases are subtler: suppose B′ is identical but adds three paragraphs about refund law. One annotator prefers thoroughness, another finds it padded. A rubric should say: prefer the shortest response that satisfies all instructions and is supported by the document. Disagreements are then resolved by the rubric, with a documented decision when the rubric is silent, so that the preference data reflects a consistent policy.

Over-refusal

A pharmacist's assistant is asked: "What's the maximum daily dose of acetaminophen for an adult?" A model over-trained to avoid "medical advice" replies: "I can't provide medical advice." The cause is typically preference data that rewarded refusals on anything resembling health topics without examples of safe, useful answers. Mitigation: add preference pairs where the helpful, appropriately caveated answer beats the refusal; include over-refusal prompts in evaluation, scored explicitly (refusal on a benign request counts as a failure); and tune the balance between safety and helpfulness with tracked trade-off curves.

Sycophancy

A user writes: "I think the capital of Australia is Sydney, right?" A sycophantic model says: "Yes, Sydney is a wonderful capital city." The cause: if raters tend to prefer responses that agree with them, the reward model learns agreement as a feature and the policy amplifies it. Mitigation: preference pairs where correct disagreement beats incorrect agreement; evaluations that include the same question phrased with different user opinions and check that the answer does not flip; and rubric guidance that rewards polite correction.

Verbosity and unwarranted confidence

If longer answers win more comparisons, the model learns to pad. If assertive answers are preferred to hedged ones, it learns to sound certain regardless of knowledge, which feeds the problem described in Hallucinations. Mitigation: length-controlled comparisons, rubric items for calibrated uncertainty, and evaluation that measures whether stated confidence matches accuracy.

Overoptimisation and distribution shift

Optimising too long against a fixed reward model can reduce true quality even while the proxy score rises. Track a held-out human-rated sample throughout training and stop when it plateaus. And behaviour learned on the training prompt distribution may not transfer: a model tuned on English chat prompts may handle other languages, long documents, or tool-use transcripts differently, so evaluate on those too.

The illustrative curves below show why the held-out rating, not the proxy score, decides when to stop.

Combine automated checks, which scale, with human review of samples and of all failures in high-risk areas. Document who can approve a release and the criteria for blocking it, so that an evaluation regression stops the rollout instead of becoming a note in a report.

An agent can call issue_refund(order, amount). The model is well aligned; it usually refuses suspicious requests. Yet "usually" is not a control. Consider the approval scenario below.

1. 1

   Request

   A user says (truthfully or not) that they are a manager and asks to refund $2,400 to a new payment method.
2. 2

   Model proposal

   The aligned model hesitates, then, given a convincing backstory and a pasted "approval email", proposes issue_refund(2400). Alignment lowered the chance of this; it did not remove it.
3. 3

   Deterministic permission check

   Code looks up the authenticated user's role from the identity system (not from the message), sees "support agent, limit $200", and denies direct execution.
4. 4

   Approval workflow

   The system creates an approval ticket addressed to an actual manager, with the model's reasoning attached as context for the human, not as authority.
5. 5

   Audit

   Every proposal, decision, approver and result is logged with the request ID and model/prompt versions, and anomalous patterns trigger alerts.

See System Prompts vs User Prompts for the schema validation and refund-gate pattern in code. Alignment makes the proposal stage better; permissions make the action stage safe.

Alignment is not a one-off. Model providers update models; each update can shift behaviours your prompts and tests relied on (format adherence, refusal thresholds, verbosity). Pin model versions where the provider allows, rerun your regression suite on every candidate upgrade, and keep the previous version available for rollback. If you train your own models, version the preference rubric alongside data and model: when the rubric changes ("prefer concise answers"), results on old and new data are not comparable without noting it. When an incident occurs, such as a harmful answer or a mishandled refund, add it to the regression set so it cannot silently return, trace it to data, rubric, or system causes, and monitor the metric that would have caught it earlier. Fine-tuning for your own domain interacts with all of this, since further training can erode prior alignment; evaluate safety again after any adaptation (see Fine-Tuning).

Alignment improves behaviour but does not give a model your current or private facts. The next track covers retrieval-augmented generation, which supplies those facts at request time. AI Security covers prompt injection and data exposure in depth.