arXiv preprint · 2026
ErrOPD
When Does On‑Policy Distillation Transfer Teacher Errors?
Nanyang Technological University, Singapore
TL;DRA teacher that ends a JSON list one item early in ~20% of its outputs passes the mistake to its student under forward-KL on-policy distillation within 42 steps (+13–23 points), but not under reverse-KL OPD (under +0.05). The gap sits at a single decision the student rarely takes, and reverse KL delays such errors rather than blocking them.
Same teacher, same rollouts, different inheritance.
Both objectives train on the student's own samples and see the same teacher signal. Only the direction of the token-level KL differs, and that decides whether a rare teacher error is copied.
transmits the injected error within 42 steps, across four model families, exceeding reverse KL in every seed pair.
raises the students' error rate by under 0.05 points at a learning rate of 10−6, in every family and size.
The contrast holds for teachers that make the error on 2% to 78% of samples; reverse KL stays at or near zero in every seed.
Giving reverse KL forward KL's gradient magnitude at the error decision alone, with its own direction, makes it transmit on two model pairs.
A distilled student can copy its teacher's mistakes along with its skills. On-policy distillation (OPD), used in recent post-training recipes, has the teacher grade every token of the student's own outputs. Which teacher errors does it pass on?
We build ErrOPD, a testbed whose teachers are fine-tuned to close a JSON list one item early, an error their students almost never make. The answer depends on the KL direction: across four model families, forward-KL OPD transmits the error within the training budget (+13–23 points), while reverse-KL OPD at our reference learning rate adds under 0.05 points. A duplicated ID and, where the decision arises, two SQL errors show the same split.
The reverse-KL update scales with the student's own probability of a token, and an intervention tests this: giving reverse KL forward KL's gradient magnitude at the error decision alone makes it transmit on two model pairs. Reverse-KL OPD can delay such errors rather than block them: some runs adopt the error with longer training, and Qwen3 students that already make it, even rarely, adopt it within the budget. Because greedy evaluation on Qwen3 misses the inherited error, distilled students should also be audited on sampled outputs.
Inject one known error, then watch who inherits it.
Teachers are fine-tuned to close a JSON list one item early in about 20% of outputs, an error their students make on under 0.13% of samples. Each student is distilled with matched reverse-KL and forward-KL OPD, with everything else fixed.
- Four model families. Qwen3-1.7B ← Qwen3-4B, Llama-3.2-3B ← Llama-3.1-8B, Qwen3.5 students of 0.8B / 2B / 4B ← Qwen3.5-9B, and Olmo-3-7B ← Olmo-3.1-32B.
- Four error types. Closing the list early, a duplicated item ID, and, where the decision arises, two SQL errors.
- Matched arms. Same starting checkpoint, teacher, prompts, rollouts, optimiser and budget: 42 steps of 128 prompts at a constant learning rate of 10−6, 3–5 seeds per cell.
- Reverse-KL OPD grades each sampled token by how much more likely the teacher finds it than the student does, the form used by recent post-training recipes.
- Forward-KL OPD trains the student towards one teacher-sampled token at each position of the student's own rollout.
- Controls. Full-vocabulary reverse KL (no sampling), sequence-level distillation, and a clean teacher trained on unmodified lists.
A student-probability factor at the error decision.
The usual story is that reverse KL is mode-seeking. But here both objectives are minimised at the teacher; what differs is the speed. In expectation, the local reverse-KL logit update changes a token's score in proportion to the probability the student already gives it, so a decision the student rarely takes receives little gradient however often the teacher takes it. Forward KL has no such factor.
- The intervention transmits. Forward KL's gradient magnitude at the error decision alone, with reverse KL's direction, transmits the error on two model pairs, and on one pair it is also necessary for forward KL to transmit.
- Not step size, not sampling. Neither a larger whole-model step nor a full-vocabulary gradient, which samples nothing, accounts for the difference.
- A local account. It explains the direction and the end point of the results, not their speed.
What the experiments show.
Forward KL takes on the error in every setting; reverse KL does not, within 42 steps.
It holds at every teacher error rate.
From teachers that err on 2% of samples to 78%, and reverse KL's updates land away from the error decision.
Greedy evaluation can miss it.
On Qwen3, standard greedy decoding misses the transmitted error: it shows up in temperature-1 samples, which is where we measure (four samples per held-out prompt at steps 14, 28 and 42).
Absence after a short run means slow transfer, not none. A clean-looking reverse-KL student at 42 steps is not evidence that the teacher's error was blocked.
Reverse KL can delay the error rather than block it.
When reverse KL does transmit.
BibTeX
@article{li2026erropd,
title = {ErrOPD: When Does On-Policy Distillation Transfer Teacher Errors?},
author = {Li, Xin and Jiang, Hao and Wang, Annan and Zhang, Yichi and Yuen, Chau},
journal = {arXiv preprint arXiv:2605.08737},
year = {2026}
}
- Codegithub.com/LiXin97/ErrOPD: training and evaluation code for the matched reverse-KL and forward-KL OPD arms, the injection protocol and the error-rate metrics.
- ModelsHugging Face collection: 34 checkpoints, namely the error-injected teachers, the students' starting points, and one reverse-KL and one forward-KL student after 42 steps for each main setting in the paper.
- DataErrOPD-ESCI, ErrOPD-Spider and ErrOPD-Fashion: the ranking and text-to-SQL prompts, and the clean and error-injected teacher fine-tuning lists.
- PaperarXiv 2605.08737: full paper with the appendix: implementation details, per-seed results and the additional error types.