Distillation, explained step by step
How we turn our big 4B OCR model into a small 0.8B one. This page is for a reader who is new to distillation.
Every example below is a real crop from our RL validation set, and every probability on this page was measured
on 2026-11-12 with our real models: the teacher is the final 4B after RL (rl_best_ckpt) and the student is the
0.8B after SFT (sft_best_ckpt_0p8b). Training numbers come from the logs of our runs.
The plan on one picture
We have two branches. Branch A is the product we want. Branch B is the baseline with no teacher, so the paper can show how much the teacher helps.
- Words you need first
- The teacher and the student
- Step 1 — warm-up of the new token rows
- Step 2 — KD: learn from the teacher's probabilities
- Step 2 during training — what the logs show
- Step 3 — KD anneal
- Step 4 — on-policy distillation (OPD)
- Step 5 — RL (GRPO + DAPO)
- Branch B and the decision points
- All steps in one table
Words you need first
Token. The model does not write letters. It writes tokens: pieces of text from a fixed list of 281,920 entries (our extended tokenizer). The English label 15) Draw the human ear. is 8 tokens plus an end token:
Indic text is cut into pieces too: ഓഗസ്റ്റ് (Malayalam "August") is the tokens ഓ ഗ സ്റ്റ്.
Next-token probabilities. At every position, the model reads the image, the prompt and all earlier tokens, and gives a probability to each of the 281,920 tokens. The probabilities add up to 1. Then one token is picked (the most probable one when we benchmark; a random draw when we sample), and the model goes to the next position.
Loss. One number that says how wrong the model is. Training changes the weights a little in the direction that makes the loss smaller. The size of one change is set by the learning rate (LR).
Cross-entropy (CE). The normal loss of SFT: CE = −log p(correct token). If the model gives the correct token
p = 0.9, CE = 0.105; at p = 0.5, CE = 0.69; at p = 0.01, CE = 4.6. Only the probability of the one correct token counts.
KL divergence. A measure of how different two probability lists are. KL = 0 means identical lists. It uses all tokens, not only the correct one. This is the key tool of distillation.
The teacher and the student
Teacher: 4B, final after RL
Qwen3.5-4B → our SFT (stages 1–3, merge r10b_819_r9f) → RL (GRPO + DAPO, step 160 anneal merged with step 130).
Strong but slow and expensive to serve.
Student: 0.8B
Qwen3.5-0.8B with the same tokenizer. Five times fewer weights: fast and cheap, but it learns less from the same data on its own.
OmniDoc = OmniDocBench overall (parse with headers), olmOCR = olmOCR-Bench average, WER = Indic word error rate (house scorer, clean, macro), cat. = catastrophic crops (printed / handwritten). ↓ = lower is better.
Why distil? Trained alone on the same data (Branch B's start), the 0.8B gets 89.67 on OmniDoc. Our first distilled 0.8B (teacher = the 4B before RL) got 90.84, with better Indic WER too. The teacher gives the small model more information per training example than the label alone. The rest of this page shows how.
1Warm-up of the new token rows done
The stock 0.8B only knows the stock Qwen tokens (IDs 0–248,076). Our tokenizer adds 33,784 rows: table tags, the continued-BPE Indic pieces,
<|loc_0..1000|> coordinate tokens and block tokens. A new row starts with no meaning.
Initialisation ("mapinit"). We copied the 4B teacher's trained rows for these tokens into the 0.8B (mapped to its smaller width). In the 2B test this start was clearly better than the "mean" start (the average of the old pieces of each token): validation loss 0.975 vs 1.087 at step 500.
Warm-up training. Then only these new rows train (all other weights frozen), with the same loss as step 2 (half teacher, half label),
LR 2e-4, until the validation loss is flat. The 0.8B stopped at step 1,300: Qwen3.5-0.8B-kdwarm-mapinit-s1300. This is the start
of the KD run.
2KD: learn from the teacher's probabilities running now
Teacher forcing. We give both models the crop, the prompt and the correct answer. At every answer position, each model gives its probability list for the next token. Nothing is generated; it is one forward pass per model.
Both halves of the loss below (KL and CE) use this same input; only the target differs. So KD does correct the student's wrong guesses for the next token: on the Malayalam crop the student gives the label token ഗ only 15%, so CE = −log 0.153 = 1.88 and the KL both push it up. What KD never shows the student is a prefix that contains its own earlier mistake: the text before each position is always the correct label. This is why it is called "off-policy" (the input does not come from the student), and it is the gap that OPD and RL close (step 4).
The loss (per answer token, then averaged):
The CE half says: "the correct token is this one". The KL half says: "make your whole list look like the teacher's list". The KL here is
forward KL: Σ pT(x) · log(pT(x) / pS(x)) over all 281,920 tokens x. It is large when the
student gives a small probability to a token that the teacher thinks is likely, so the student must cover every option the teacher
considers.
Try it on real crops
Pick a crop, then click a token. Bars show the real probabilities of the teacher (blue) and the student (orange) at that position. The small number under each token is the teacher's probability of the label token.
What the examples teach
Temperature T. Some setups divide the scores by T > 1 to flatten the teacher's list and show more small probabilities. We use T = 1: most OCR tokens are near-certain (see the English printed crop: almost all 1.0), and a flatter teacher would teach the student to doubt clear text.
Step 2 during training — what the logs show
Run kd8_0p8b_rlteach: 8 nodes (64 H100), 256 packs per step, each pack up to 24,576 tokens. One step is about
5.8 M tokens of which about 1.2 M are answer tokens (the loss is only on answer tokens; the image and prompt tokens are input).
About 2,800 samples per step, about 12 s per step. LR: warm-up for 25 steps to 2e-4, then constant. A checkpoint and a validation every 50 steps;
benchmarks every 100 steps from step 500.
kd = the KL part, ce = the label part, loss = 0.5 × kd + 0.5 × ce. Values of single steps (noisy).
How to read it
- Steps 0–25 (LR warm-up): the loss can first go up (1.75 → 1.87 at step 10) while the LR grows; the grad norm is high (22 → 7.6). This is normal: the first steps move the weights to the new data and teacher.
- Steps 25–300: fast drop (kd 1.14 → 0.30). The student learns the easy part of the teacher.
- After that: slow drop. The grad norm stays below 1. A spike above 10 later would be a warning sign.
- kd vs ce: kd is lower than ce. The teacher's list is easier to copy than the one-hot label, because the label is "100% certain", even where the image is not.
- Do not compare train loss between runs on different data. The new KD uses the v6 data mix, the old KD used an older mix, so the train loss numbers differ for that reason alone. Compare runs on the same validation set and, above all, on the benchmarks.
Validation loss: new KD vs old KD
Page validation set, every 50 steps (the old KD up to 5,000; the last 1,000 steps were its anneal). The new KD is 0.03–0.12 higher at the same steps (0.09 at step 500). Possible reasons: the RL teacher has sharper lists that are harder to copy, or the change in data. This is still open; the benchmarks decide. Live scores: the "0.8B distillation" page.
3KD anneal next — separate run
During the constant-LR phase, each step moves the weights by a size set by the LR. With a constant LR, the model keeps "jumping around" the best point. Annealing brings the LR down to 0 so the steps get small and the model settles. We use the 1−√t shape (fast at the start, gentle at the end), the same as in our SFT.
The old KD shows the effect: validation loss 0.400 at step 4,000 (end of constant LR) → 0.372 at step 5,000 (end of anneal), and the train kd part 0.080 → 0.055. We start the anneal from a checkpoint of the running KD; the step is chosen when the benchmarks flatten.
4On-policy distillation (OPD) next — smoke test passed
The problem KD cannot see
KD fixes wrong next-token guesses (step 2), but in KD the student always reads the correct earlier tokens. At inference it reads its own earlier tokens. After one wrong token it is in a situation it never trained on, and errors grow. Here are 4 real answers that the 0.8B wrote itself (sampled at temperature 1) for the Malayalam crop (label 1982ൽ ഓഗസ്റ്റ് 6 ന് ഇദ്ദേഹം മരിച്ചു.):
The first error is the same each time (ഓക്സ് for ഓഗസ്റ്റ്). After it, the answers go different ways and invent words. KD trained on the correct prefix only, so it never taught "what to do after ഓക്സ്".
The fix: the student writes, the teacher grades every token
- The student writes a full answer for a crop or page (its own "policy" — hence "on-policy").
- The teacher reads the same image and the student's answer, and gives its probability for each token the student wrote.
- For each token:
score = log pS(token) − log pT(token). A large positive score means "the student was sure, the teacher thinks it is wrong". This is a one-sample estimate of the reverse KL, KL(student ‖ teacher). - The student is trained to make these scores smaller, like a reward per token (−score) in RL.
Pick a crop and one of the student's 4 answers. The colour shows the score per token: darker = the teacher disagrees more. Click a token to see the teacher's top choices there.
Why KD first and not OPD from the start
OPD needs a student whose answers are already close. If the answers are bad, the teacher's grades are uninformative (everything is wrong), and the generation of long answers is slow. KD is cheap and gets the student close; OPD then fixes the student's own errors. The same order is used by Qwen3-VL (off-policy, then on-policy distillation) and studied in "Rethinking OPD".
How OPD looks in training (our smoke test)
Framework verl. Student on 4 GPUs (rollouts with vLLM + FSDP training), teacher on 4 more GPUs (vLLM, which returns the log-probabilities of
the given tokens: prompt_logprobs). 10 steps, 16 prompts per step, LR 1e-6. It runs end to end.
- abs_loss: the mean size of the per-token score. It should go down over many steps. In 10 steps it is noise (0.37–1.03), as expected.
- loss_min / loss_max: single tokens reach +28 to +39. One such token can dominate a step, so the real run clips each token at
loss_max_clamp = 5. - grad norm 6–12 (RL on the 4B: about 2–3): another reason for the clip and for a bigger batch.
- reward is logged (the same OCR reward as RL) but not used, so we can see whether OPD also improves the OCR score.
- 40–97 s per step on 8 GPUs; the real run uses 8 nodes and more prompts per step, for 50–100 steps.
5RL: GRPO + DAPO after OPD
RL does not need a teacher. The student writes several answers per prompt, each answer gets a reward from a program that compares it with the label (for crops: 1 − character error rate; for pages: text, layout, tables and formulas), and the student is pushed towards its better-than-average answers. Same setup as our 4B RL.
GRPO in one example
For each prompt the student writes a group of answers (16 in training, 4 here). The advantage of an answer is its reward minus the group mean, divided by the group's standard deviation. Positive advantage → make this answer more likely; negative → less likely.
Reward here = max(0, 1 − CER) of the real student samples. The real crop reward also gives 0 to answers without an end token and subtracts a repetition penalty. The two run-on answers on the English crop were cut at 200 tokens in this probe, so they would also get 0.
RL vs OPD. OPD gives a signal at every token from the teacher. RL gives one number per answer, from the label. RL can go beyond the teacher (it optimises the benchmark-like reward directly); OPD cannot, but its signal is much denser. That is why A does OPD first, then RL.
How RL looks in training (our 4B run, the template for the 0.8B)
- reward mean rises slowly (0.71 → 0.80 in 170 steps). Small numbers; that is normal, the prompts were chosen to be hard but solvable.
- entropy (how spread the student's lists are) falls: the model gets more sure. If it collapses to near 0, exploration stops.
- clip fraction (share of tokens where the update was clipped) stays below 1%: the updates are small and safe.
- Every 10 steps a checkpoint is benchmarked; it must not be worse than the start on OmniDoc, olmOCR, Indic WER and catastrophic crops. After the best constant-LR step, an anneal (1−√t, 40 steps) and a small merge.
- For the 0.8B: first an LR test (1e-6 vs 2e-6, about one hour each on 8 nodes), then constant LR, then anneal.
Branch B and the decision points
Branch B: 0.8B SFT → RL. The same RL recipe, started from the 0.8B trained without a teacher (sft_best_ckpt_0p8b).
It answers the reviewer's question "what does the teacher add?".
Decision points in A:
- After the KD anneal: the KD 0.8B must beat the old distilled 0.8B (90.84 / 83.8 / 0.1214 / 0.2650) and the 0.8B SFT.
- After OPD: if OPD fails or adds nothing on the benchmarks, A becomes KD → RL (OPD is dropped).
- Dropped unless needed: SFT → OPD → RL and KD → RL as extra arms.
All steps in one table
| Step | Student input | Signal | Loss | Teacher at train time | Status |
|---|---|---|---|---|---|
| 1 Warm-up | correct text | teacher lists + label | 0.5 fwd KL + 0.5 CE, new rows only | 4B SFT, forward pass | done (s1300) |
| 2 KD | correct text | teacher lists + label | 0.5 fwd KL + 0.5 CE, all weights | 4B RL, forward pass | running |
| 3 KD anneal | correct text | same | same, LR → 0 | same | next |
| 4 OPD | its own answers | teacher grade per token | reverse KL (log pS − log pT), clip 5 | 4B RL, vLLM scoring | smoke passed |
| 5 RL | its own answers | reward per answer | GRPO + DAPO | none | after OPD |
| B SFT → RL | correct text, then own answers | label, then reward | CE, then GRPO + DAPO | none | SFT done |