From SFT and RL to On-Policy Distillation: A Third Paradigm for Post-Training

Tue, 11 Aug 2026 00:00:00 +0000

Since the release of DeepSeek-R1, GRPO has become the default choice for post-training, and the “SFT + RL” recipe has found its way into virtually every technical report. Yet beginning with Qwen3 in 2025, one flagship model after another has independently placed the same method on its critical path: on-policy distillation (OPD).

The first flagship report to adopt it was Qwen3, in the form of strong-to-weak distillation. The flagship models still went through the full four-stage post-training procedure, but the lightweight models from 0.6B to 14B no longer repeated that procedure individually; they were distilled instead from Qwen3-32B or Qwen3-235B-A22B. The report notes that this yielded higher performance while consuming roughly one tenth of the GPU hours required by the four-stage pipeline.

Six months later, Xiaomi’s MiMo-V2-Flash named the method MOPD (Multi-Teacher On-Policy Distillation) and framed it explicitly as a post-training paradigm. Its post-training is organised into three stages: general SFT, then per-domain teachers trained with RL or SFT, and finally MOPD to merge those teachers back into a single student. Its teachers are the domain specialists produced by the independent RL runs of Stage 2.

Moonshot’s Kimi K3 pushes the same idea one step further. It partitions specialists not only by domain but also by reasoning effort, training three levels — low, high and max — within each of three domains, for nine specialists in total, and again merges them with MOPD.

Zhipu’s GLM-5 takes a different route. Its teacher is neither a larger model nor a domain specialist, but the model’s own checkpoints from earlier training stages; OPD sits at the very end of the post-training pipeline and serves to repair the capability degradation introduced by sequential RL.

The most aggressive adoption is DeepSeek-V4, whose report states outright that the mix-RL stage of the V3.2 era has been replaced in its entirety by OPD: a specialist is first trained for each of more than ten domains — mathematics, code, agentic tasks, instruction following and so on — and multi-teacher OPD then merges them back into a single unified model.

The five labs differ in where they place the method, where their teachers come from, and at what granularity they compute the KL, but the underlying recipe is identical: the student samples its own trajectories, and the teacher scores every token along them. A method that once appeared only in the distillation literature has, within two years, moved onto the critical path of five flagship reports. This article addresses three questions: where exactly SFT and GRPO break down, what the mathematical form of OPD is, and what problem each of these reports uses it to solve.

1. SFT: Dense Supervision, but Exposure Bias

2023 was the breakout year for large language models, and SFT broke out along with them; for many practitioners it was the first thing they ever trained. SFT is the step that turns a pretrained model, capable only of continuation, into a usable assistant. Its rise can be traced along two lines of work:

First, in 2021, FLAN and related work showed that fine-tuning on multi-task instruction data enables zero-shot instruction following, establishing instruction tuning as a standard step;
Second, in 2023, open-source efforts such as Alpaca and Vicuna replaced human annotation with the outputs of a stronger model — using Self-Instruct to generate tens of thousands of responses from GPT-3.5 and training small models directly on them, producing usable assistants at very low cost. It is this second line, the distillation setting, that concerns us here.

Applied to distillation, SFT proceeds as follows:

the teacher generates a complete high-quality response, yielding a full token sequence $y$;
at every position $t$, the teacher-generated prefix is fed to the student, which outputs logits for the next token;
the teacher’s actual token $y_t$ at that position serves as the target, against which the student’s distribution is scored by cross-entropy.

The loss is therefore

$$ \mathcal{L}_{\mathrm{SFT}}(\theta) = -\,\mathbb{E}_{(x, y) \sim \mathcal{D}_T} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{\lt t}) $$

Two details are worth committing to memory: the prefix comes from the teacher, and the supervision target is a single hard token produced by the teacher.

This form makes SFT’s central advantage immediate — the supervision is dense. A sequence of length $T$ supplies $T$ gradient signals, and the further the student deviates at a given position, the larger the loss and the update at that position. Combined with the fact that the objective is standard maximum likelihood, that no sampling is required during training, and that results are stable and reproducible, SFT remains the first step of the distillation pipeline nearly everywhere, from DeepSeek-R1-Distill to the lightweight Qwen3 variants.

At scale, however, several problems have surfaced.

The first is distribution shift. Every prefix presented to the student during training comes from a flawless teacher trajectory, whereas at inference the prefix it continues is whatever it has just produced itself. The moment the student makes an error the teacher would never make, it enters a state absent from the training data; the remaining tokens are generated without guidance, and the error compounds along the sequence. In sequence generation this is known as exposure bias; in imitation learning it is called compounding error. DAGGER argued as early as 2010 that the only remedy is to let the apprentice act on its own and then obtain expert labels at the states the apprentice actually reaches. The gap is quantitative: Tencent’s A Survey of On-Policy Distillation formalises it by showing that the compounding error of off-policy imitation grows roughly with the square of the sequence length, whereas taking feedback on the student’s own distribution reduces it to linear.

The second is the superficiality of imitation. Gudibande et al. provide empirical evidence in The False Promise of Imitating Proprietary LLMs: imitation training readily transfers the teacher’s style and confidence rather than its factual accuracy. The reason is not hard to see — the teacher can produce that response because an entire body of knowledge and reasoning ability stands behind it, while the student observes only the token sequence and can imitate only the part that “looks right”.

The third is mode covering, in which the student is forced to cover every mode of the teacher’s distribution and ends up mastering none of them.

Concretely, suppose a problem admits two entirely different solution paths — setting up an equation and solving it, or enumerating and verifying candidates directly — and the teacher assigns them 20% and 80% probability respectively. The SFT loss is an expectation under the teacher’s distribution, which means every solution the teacher samples is used to examine the student. Consequently, if the student were to abandon path A and drive its probability to zero, then at the positions where the teacher generates A we would have $\log \pi_\theta(\text{A}) = -\infty$ and the loss would diverge.

The student therefore has no choice but to reserve probability mass for every mode:

Forward KL — mode covering: the student is forced to assign probability to both modes, leaving mass even in the region between them.

Given ample capacity this need not be harmful, but the student’s capacity is precisely what is limited: unable to master both paths at once, it converges to an interpolation of the two. What it samples at inference is neither a complete equation-based solution nor a complete enumeration, but an internally inconsistent hybrid of the two.

The root of this behaviour is the zero-avoiding property of forward KL: wherever $\pi_T(x) > 0$, the student is not permitted to let $\pi_\theta(x) \to 0$, or the loss diverges. In other words, the behaviour is dictated by the direction of the divergence itself — the SFT cross-entropy is, in form, a forward KL, a point developed in Section 4. It was on exactly this basis that MiniLLM argued in 2023 that distilling small models calls for the zero-forcing reverse KL instead, so that the student converges onto a single mode rather than covering all of them.

2. GRPO: No Distribution Shift, but Sparse Reward

After DeepSeek-R1, the default on the RL side became GRPO. Where PPO requires an additional value network of comparable size to the policy in order to estimate the advantage, GRPO removes the critic altogether: it samples a group of $G$ responses for the same prompt and uses the group-normalised return as the advantage.

$$ \hat{A}_{i} = \frac{R_i - \mathrm{mean}(\{R_1, \dots, R_G\})}{\mathrm{std}(\{R_1, \dots, R_G\})} $$

This simplification suits the RLVR setting, where each trajectory carries a single verifiable scalar reward at the end. A mathematics task requires only checking whether the answer is correct, a coding task only whether the predefined unit tests pass, so the reward function can be kept extremely simple.

More importantly, RL resolves SFT’s distribution shift at the root: trajectories are generated by the model itself, the state distribution during training matches the one at inference exactly, and exposure bias does not arise.

The cost of this route is equally clear, and it is concentrated in one place: it provides a single scalar at the end of an entire trajectory.

Concretely, the student writes a 500-token chain of reasoning and the verifier replies with one sentence: wrong answer, reward = 0. Thinking Machines Lab offers an information-theoretic comparison (from LoRA Without Regret): reinforcement learning conveys $O(1)$ bits to the model per episode, whereas distillation conveys $O(N)$ bits per episode, where $N$ is the number of tokens.

The first consequence is that the student does not know where it went wrong. Since that single bit must be distributed across 500 tokens for credit assignment, the student learns only that it lost, and cannot localise whether step 17 miscalculated, step 200 skipped too far in its reasoning, or the final step inverted a sign. The only thing it can do is suppress the probability of the entire trajectory — including the intermediate steps that were in fact correct.

The second consequence is time and compute. Because the information content of a single feedback signal is fixed, developing a capability requires accumulating volume through repeated sampling: dozens of rollouts per problem, hundreds of thousands of prompts, until the statistical differences emerge. Notably, the bulk of this compute is not spent on gradient updates but on sampling and exploration. The Qwen3 report, for instance, records 17,920 GPU hours of mathematics and code RL for the 8B model.

One direct line of attack is to make the reward itself dense. OpenAI’s Let’s Verify Step by Step demonstrates that process supervision outperforms outcome supervision by training a process reward model to score each step, which amounts to densifying a sparse reward by hand; the price is a large volume of step-level annotation, and the PRM is itself susceptible to being hacked.

Placing the two routes side by side:

Method	Sampling source	Feedback density
SFT / off-policy distillation	Off-policy	Dense (every token)
Reinforcement learning	On-policy	Sparse (one scalar per trajectory)
On-policy distillation	On-policy	Dense

3. OPD: Dense Reward on On-Policy Trajectories

The construction of on-policy distillation follows directly from the empty cell in that table: the student samples the trajectory, and the teacher scores it token by token.

Relative to SFT, OPD proceeds as follows:

the student first generates a complete response under its current policy, yielding a trajectory $\hat{y}$;
at every position $t$, the student’s own prefix is fed to both the student and the teacher, each of which outputs a distribution over the next token;
the supervision target is no longer a single hard token but the teacher’s full conditional distribution given that prefix, with the KL divergence measuring the gap between the two distributions.

In other words, OPD changes exactly two things relative to SFT: the prefix comes from the student rather than the teacher, and the supervision target becomes the teacher’s full distribution rather than one hard token. The first change eliminates distribution shift; the second raises the information carried at each position from a single token to an entire distribution over the vocabulary.

The reward at each position is correspondingly defined as the negative reverse KL:

$$ R_t = -\,\mathrm{KL}\big(\pi_\theta(\cdot \mid x, y_{\lt t}) \,\|\, \pi_T(\cdot \mid x, y_{\lt t})\big) $$

There is no reward model here and no additional regularisation term: the KL is the entire optimisation signal, and it is defined at every token.

This per-token scoring can be visualised directly. The figure below is a real trajectory from the TML blog; darker tokens carry higher reverse KL, meaning stronger disagreement from the teacher:

One counter-intuitive detail stands out: the final, incorrect answer token is not heavily penalised. The reason is that once the earlier reasoning has gone astray, that answer is in fact “predictable” conditional on the flawed premise; what gets penalised are the intermediate tokens at which the reasoning diverged. RL, by contrast, can only assign one scalar to the whole trajectory and has no way to distinguish between these two kinds of position.

4. Why It Has to Be Reverse KL

Classical distillation uses forward KL; OPD uses reverse KL. The two merely swap the positions of $P$ and $Q$, yet they behave very differently.

Forward KL: mode covering

$$ \mathrm{KL}(\pi_{T} \| \pi_{\theta}) = \mathbb{E}_{x \sim \pi_{T}}\left[\log \frac{\pi_{T}(x)}{\pi_{\theta}(x)}\right] $$

When taking gradients with respect to $\theta$ the first term is constant, and what remains is the expectation of $-\log \pi_\theta$ under the teacher’s distribution — that is, a cross-entropy. Comparing this with the SFT loss above shows that the two are formally the same objective: SFT minimises a forward KL, the only difference being that its teacher distribution degenerates to a one-hot label, whereas in distillation it is the teacher’s full soft distribution.

This also explains the mode covering of Section 1: because the expectation is taken under the teacher’s distribution, every mode the teacher would generate must be covered, and omitting any one of them causes the loss to diverge.

Beyond mode covering, this direction has a complementary blind spot: since the expectation is taken under the teacher, the loss is entirely insensitive to how the student allocates probability in regions where the teacher’s probability is zero. The student is free to pile up probability where the teacher never ventures without incurring any penalty — which is the other half of the explanation for “right style, wrong answer”.

Reverse KL: mode seeking

$$ \mathrm{KL}(\pi_{\theta} \| \pi_{T}) = \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \frac{\pi_{\theta}(x)}{\pi_{T}(x)}\right] $$

Splitting the log-ratio into two terms reveals a further reading:

$$ \mathrm{KL}(\pi_{\theta} \| \pi_{T}) = \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \pi_{\theta}(x)\right] - \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \pi_{T}(x)\right] $$

By definition the first term is the negative entropy of the student, $-H(\pi_\theta)$, and the second, with its sign flipped, is the cross-entropy of the student against the teacher, $H(\pi_\theta, \pi_T)$. Hence

$$ \mathrm{KL}(\pi_\theta \| \pi_T) = \underbrace{H(\pi_\theta, \pi_T)}_{\text{cross-entropy to the teacher}} - \underbrace{H(\pi_\theta)}_{\text{student's own entropy}} $$

Minimising the reverse KL therefore does two things at once: it lowers the student’s cross-entropy against the teacher and raises the student’s own entropy. Were the cross-entropy minimised alone, the objective would be linear in $\pi_\theta$ and its optimum would lie at a vertex of the simplex, placing all probability on the single token the teacher deems most likely and collapsing to a point mass. The entropy term counteracts precisely this concentration: when the two terms carry equal weight, the optimum is exactly $\pi_\theta = \pi_T$. Even when the student lacks the capacity to cover more than a subset of the teacher’s support, the optimum is the teacher’s conditional distribution restricted to that subset, preserving the relative proportions within it. Mode seeking therefore means deciding which modes to abandon, not sharpening the distribution within the modes that are retained.

Returning to behaviour: because the expectation is now taken under the student’s distribution, the teacher only evaluates what the student actually writes. Take the same two solution paths: if the student concentrates all its probability on path B, the teacher assigns B 80% and the KL is small; path A is ignored entirely, but since the student never samples A, that region contributes nothing to the loss.

Reverse KL — mode seeking: the student concentrates all probability mass on the one mode the teacher endorses and abandons the other outright.

Mode seeking is better aligned with the actual objective of distillation. The student’s capacity is far smaller than the teacher’s, so requiring it to cover every teacher mode is unrealistic to begin with; under forward KL, insufficient capacity leads directly to the mode covering of Section 1, in which neither solution path is learned. Reverse KL relaxes the requirement to “the one mode the student selects must be endorsed by the teacher”, allowing the student to concentrate its probability mass on a single solution path and learn it to a usable standard.

The second property of reverse KL is that it is unhackable. Because the expectation is taken under the student’s distribution, a low KL necessarily means that every token the student actually generates is assigned high probability by the teacher; there is no solution that scores well on the metric while producing poor output. Two contrasts convey the weight of this property: when a learned reward model provides the signal, the policy can find regions the reward model does not cover and inflate its score there — reward hacking; forward KL, meanwhile, imposes no penalty at all in regions where the teacher’s probability is zero. Reverse KL admits neither loophole, for the simple reason that the object being evaluated is exactly what the student itself produces.

The choice of direction is not a new conclusion. Besides MiniLLM, mentioned in Section 1, Google DeepMind’s Generalized Knowledge Distillation (GKD) wrote down the full recipe of “on-policy sampling plus token-level KL” in the same year, and today’s industrial OPD variants can largely be viewed as special cases of the GKD framework.

5. The Further Degeneration of Advantage Estimation

OPD can be dropped into an existing RL framework at almost no engineering cost by setting the per-token advantage directly to the log-probability difference between teacher and student. The GLM-5 report gives the cleanest formulation, stating explicitly that this replaces the advantage term in the GRPO loss:

$$ \hat{A}_{i,t} = \mathrm{sg}\left[\log \frac{\pi_{\theta_{\mathrm{teacher}}}(y_{i,t} \mid x, y_{i,\lt t})}{\pi_{\theta}(y_{i,t} \mid x, y_{i,\lt t})}\right] $$

Placing this alongside the GRPO advantage from Section 2 reveals a clear line of development. PPO requires a value network of comparable size to the policy in order to estimate the advantage; GRPO removes the value network in favour of group normalisation; with OPD, even the group normalisation becomes unnecessary.

The GLM-5 report is direct on this point: during training it sets the GRPO group size to 1 and the batch size to 1024 to increase throughput, and this is feasible precisely because the advantage can now be computed directly from the gap to the teacher, without maintaining a group of samples per prompt to estimate it. Reducing the group size from the usual 8–16 down to 1 multiplies throughput several times over. PPO → GRPO → OPD is essentially a trajectory along which advantage estimation degenerates until no estimation is required at all: credit assignment, the hardest problem in RL, is circumvented by the teacher’s per-token scoring.

A second property that lowers cost is that the teacher performs prefill only and never decodes. The student’s complete response is already available, so when the teacher computes $\log \pi_T(y_t \mid y_{\lt t})$ all positions can be processed in parallel in a single forward pass.

In addition, OPD typically takes a discount factor of zero, optimising only the current token at each position without multi-step rollout. This means reward computation need not wait for a trajectory to finish sampling: training can use shorter or even partial rollouts, and there is no need to extend the context merely to await a terminal reward.

Vocabulary granularity: sampled-token, top-k, or full vocabulary

The KL is defined over the entire vocabulary, but an implementation need not evaluate all of it. Three options exist, and they are identical in teacher forward cost — a single prefill in every case — differing only at the loss computation.

In the table below, $B$ is the batch size, $T$ the sequence length, $V$ the vocabulary size, and $k$ the number of tokens retained by top-k:

Variant	Loss complexity	Description
Sampled-token	$O(BT)$	Uses only the log-probability of the single token the student actually sampled
Top-k	$O(BTk)$	Sums over the top-k tokens
Full-vocabulary	$O(BTV)$	Sums over the entire vocabulary at every position

The gap between the two extremes is the vocabulary size itself:

$$ \frac{O(BTV)}{O(BT)} = V $$

With Qwen3’s vocabulary of $V = 151{,}669$, this ratio is roughly $1.5 \times 10^5$, or five orders of magnitude. In concrete terms, at $B = 8$ and $T = 8192$ the full-vocabulary variant must materialise $8 \times 8192 \times 151{,}669 \approx 9.9 \times 10^9$ logit entries, about 20 GB in bf16, and one copy is needed for the teacher and one for the student; the sampled-token variant keeps a single log-probability per position, $8 \times 8192 = 65{,}536$ numbers in total, under 1 MB.

Given a cost gap this large, is a single token sufficient? That depends on two things: whether it is statistically sound, and whether it discards useful signal.

From a statistical standpoint. Monte Carlo estimation requires that samples be drawn from the distribution under which the expectation is taken. The reverse KL expectation is taken under $\pi_\theta$, and the trajectory was sampled by the student in the first place, so estimating it with the token the student actually generated is unbiased, requiring only one gathered log-probability per position. This also settles the engineering question left open in the previous section: the forward KL expectation is taken under $\pi_T$, so an unbiased estimate would require teacher sampling, yet in OPD the teacher only prefills and never decodes, so no teacher-sampled tokens exist. Using sampled-token therefore forces the choice of reverse KL; using forward KL forces one to abandon sampling and evaluate the sum explicitly over the top-k or the full vocabulary.

From a signal standpoint. Rethinking On-Policy Distillation, from Tsinghua’s thunlp group, provides mechanistic evidence: successful OPD manifests as a progressive alignment of high-probability tokens between student and teacher at student-visited states (the overlap rises from 72% to 91%), and this shared token set concentrates 97%–99% of the probability mass of both distributions. They further verify that supervising only the overlapping tokens matches the full top-k. Since the gradient signal is already concentrated on a handful of tokens, the additional vocabulary contributes little.

Taken together, sampled-token has become the industrial default in single-teacher settings. The one exception arises with multiple teachers, discussed in Section 6.2 on DeepSeek-V4.

6. How the Five Technical Reports Use It

The differences across the five labs reduce to two dimensions: where the teacher comes from, and where OPD sits in the post-training pipeline. The following sections proceed along these two axes.

6.1 Qwen3: strong-to-weak distillation for lightweight models

The post-training of the Qwen3 flagship models comprises four stages: long-CoT cold-start SFT, reasoning RL, thinking mode fusion (further SFT that merges the think and no-think modes into a single model), and general RL.

The position of OPD here is unusual: rather than being inserted at one of the four stages, it replaces the pipeline in its entirety. The lightweight models from 0.6B to 14B no longer repeat the four-stage procedure individually, and are instead distilled from Qwen3-32B or Qwen3-235B-A22B, which have already completed it — that is, strong-to-weak distillation. The report’s stated rationale is that distilling the teacher’s output logits directly into lightweight students both improves performance and preserves fine-grained control over their reasoning process.

The distillation itself proceeds in two phases, off-policy first and on-policy second.

The first phase performs response distillation on teacher outputs collected in both think and no-think modes, so that the student first acquires basic reasoning ability and learns to switch between the two modes;
Only in the second phase does training become on-policy: prompts are sampled, the student generates its own responses, and its logits are then aligned with those of the teacher (Qwen3-32B or Qwen3-235B-A22B) by minimising the KL divergence.

The reason for this design is that if the output styles of student and teacher differ too much, most of the token-level KL will be stylistic noise rather than knowledge. An off-policy round is therefore needed first to move the student’s distribution near the teacher’s, before the on-policy phase begins.

Table 21 of the report gives the resulting comparison. All three rows start from the same off-policy-distilled 8B checkpoint, differing only in whether RL or on-policy distillation follows; pass@64 is given in parentheses:

Method	AIME'24	AIME'25	MATH500	LiveCodeBench	GPQA-Diamond	GPU Hours
Off-policy distillation	55.0 (90.0)	42.8 (83.3)	92.4	42.0	55.6	–
+ Reinforcement learning	67.6 (90.0)	55.5 (83.3)	94.8	52.9	61.3	17,920
+ On-policy distillation	74.4 (93.3)	65.5 (86.7)	97.0	60.3	63.3	1,800

On AIME'24 the gain is 6.8 points at one tenth of the GPU hours. More telling are the pass@64 figures in parentheses: RL leaves the 64-sample ceiling entirely unchanged (90.0 → 90.0), whereas on-policy distillation pushes it to 93.3. The report attributes this to the fact that distilling from the teacher’s logits expands the student’s exploration space and raises its reasoning potential, with the pass@64 improvement as the evidence; reinforcement learning, by contrast, produced no pass@64 gain at all.

6.2 DeepSeek-V4: replacing mix RL with multi-teacher OPD

Compared with Qwen3, DeepSeek-V4 has more than ten teachers, each strong in a single domain, and the problem OPD solves is capability merging. Accordingly it is no longer a side branch for lightweight models but sits on the critical path: the report states that post-training retains the overall structure of V3.2 and that the one key methodological substitution occurs at this step, with the mixed reinforcement learning stage replaced entirely by on-policy distillation.

The resulting procedure has two stages. In the first, a specialist is trained independently for each domain — mathematics, code, agentic tasks, instruction following and so on — each obtained by a separate SFT-plus-GRPO run. In the second, multi-teacher OPD merges the ten-plus specialists back into a single unified model.

OPD here therefore takes on the final merging step of the pipeline; what it replaces is not the algorithm inside any single RL stage but the entire approach to optimising several objectives at once. Its objective is

$$ \mathcal{L}_{\mathrm{OPD}}(\theta) = \sum_{i=1}^{N} w_i \cdot D_{\mathrm{KL}}\big(\pi_\theta \,\|\, \pi_{E_i}\big) $$

where $w_i$ is the weight of each expert. The report’s rationale is that this mechanism consolidates knowledge dispersed across expert weights into a unified parameter space through alignment at the logits level, thereby avoiding the performance degradation commonly seen with parameter merging and mix RL.

Another notable engineering decision is that DeepSeek-V4 adopts full-vocabulary KL rather than the industry-standard token-level estimate. The report is explicitly critical of the latter: existing work typically reduces the full-vocabulary KL to a token-level estimate at each position and reuses the RL framework with $\mathrm{sg}[\log \pi_E / \pi_\theta]$ as a per-token advantage; while economical, this incurs high variance in the gradient estimate and frequently destabilises training.

The formulation being criticised is exactly the one given in Section 5. The trade-off is determined by the difference noted above: with a single teacher the variance of the Monte Carlo estimate is tolerable, whereas V4 must align more than ten teachers at the same token position, which amplifies the variance substantially.

The cost falls on memory. With a vocabulary above 100,000, materialising logits for all teachers is infeasible, so V4 caches only the teachers’ final-layer hidden states and reconstructs the logits through the prediction head during training; samples are sorted and dispatched by teacher index so that each teacher head is loaded once per mini-batch and at most one resides in memory at a time; the exact KL is computed by a dedicated TileLang kernel.

6.3 MiMo-V2-Flash: MOPD combined with outcome reward

Xiaomi’s MiMo-V2-Flash names the method MOPD (Multi-Teacher On-Policy Distillation) and positions it as a post-training paradigm rather than a training trick. Its pipeline has three stages: Stage 1 establishes instruction-following ability through general SFT; Stage 2 trains per-domain teachers with RL and SFT (including both non-agentic and agentic RL); Stage 3 uses MOPD to merge those teachers back into one student.

The motivation the report gives for this design is capability imbalance, which it calls the see-saw effect: when several capabilities are improved in sequence on the same model, gains in one are often accompanied by regressions in another. Training each domain separately and merging at the end is precisely how this effect is circumvented, and MOPD, at the third stage, performs the merge. Its reverse KL loss is written as

$$ \mathcal{L}_{\text{reverse-KL}}(\theta) = -\,\mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi(\cdot \mid x)} \log \frac{\pi_{\mathrm{domain}}(y_t \mid x, y_{\lt t})}{\pi_\theta(y_t \mid x, y_{\lt t})} $$

MiMo’s overall structure closely resembles V4’s — domain-wise teachers first, multi-teacher OPD to merge afterwards — but differs in two substantive respects. First, V4’s objective contains the KL alone, whereas MiMo adds the distillation advantage to an outcome reward, $\hat{A} = \hat{A}^{\mathrm{MOPD}} + \hat{A}^{\mathrm{ORM}}$; OPD does not displace RL, and both signals coexist. Second, MiMo permits the student itself to enter the teacher set, so in domains lacking a stronger teacher MOPD effectively degenerates into self-distillation — visible directly in Table 7 below.

Table 7 of the report reports the student before distillation, the best teacher, and the student after distillation, from which the boundary of MOPD’s benefit can be read off directly:

Benchmark	Student Before MOPD	Best Teacher	Student After MOPD	Δ(Student − Teacher)
AIME 2025	89.3	93.9 (RL)	94.1	+0.2
HMMT Feb. 2025	76.9	82.6 (RL)	84.4	+1.8
LiveCodeBench	77.5	82.6 (RL)	83.2	+0.6
GPQA-Diamond	84.9	84.9 (Self)	84.3	−0.6
Arena-Hard (Hard Prompt)	50.0	50.0 (Self)	54.1	+4.1
Arena-Hard (Creative Writing)	90.1	90.1 (Self)	86.2	−3.9
SWE-Bench Verified	67.8	74.2 (RL)	73.4	−0.8
Tau2-Bench	75.9	79.6 (RL)	80.3	+0.7
BrowseComp	42.5	51.7 (SFT)	45.4	−6.3

Three observations follow from this table.

First, in domains with a strong teacher (mathematics, code, agentic tasks) the student essentially matches or slightly exceeds the best teacher, with Δ within ±1.
Second, in the rows marked Self, the best-teacher figure is identical to the student before distillation, meaning no teacher on that benchmark is stronger than the student itself; any gain must then come from the added ORM term, and MOPD serves only to preserve performance, or may even lose points.
Third, BrowseComp drops by 6.3 points, and its teacher happens to be an SFT model — suggesting that when the behavioural patterns of teacher and student diverge too far, token-level alignment may fail to transfer the capability. This table in effect provides direct evidence that the teacher constitutes a capability ceiling.

6.4 Kimi K3: nine specialists partitioned by domain × reasoning effort

Moonshot’s Kimi K3 follows the same route as the previous two but adds one more dimension to the partition. Its post-training is likewise three-staged: SFT cold start to establish basic agentic ability, RL to train domain specialists, and MOPD to merge.

The difference lies in how the second stage is partitioned. K3 divides RL into three broad domains — general, general agents and coding agents — and trains low, high and max reasoning-effort variants within each, giving nine specialists in total. The effort levels are obtained by controlling the token budget during RL: an initial budget is estimated for each problem, trajectories exceeding a threshold of $\tau$ times that budget receive a reward of $-1$, and annealing $\tau$ yields the max, high and low levels in turn.

K3’s MOPD therefore merges not only capabilities but also thinking lengths: the behaviour of nine specialists is compressed into a single model that can switch effort levels on demand at inference. This effectively brings budget distillation of the long-to-short variety under the same mechanism.

In implementation, K3 does not go full-vocabulary as V4 does, but retains the token-level formulation with clipping applied:

$$ r^{d}_{\mathrm{opd}}(y_t \mid e, x, y_{\lt t}) = \mathrm{clip}\left(\mathrm{sg}\left(\log \frac{\pi^{(d,e)}_{\mathrm{teacher}}(y_t \mid x, y_{\lt t})}{\pi_\theta(y_t \mid e, x, y_{\lt t})}\right), -R_{\max}, R_{\max}\right) $$

where $R_{\max}$ is the clipping threshold constraining extreme advantage signals. The report explains that, defined this way, OPD is simply an ordinary dense reward that plugs into an existing RL framework, allowing infrastructure optimisations for long-horizon tasks such as partial rollout to be reused. Notably, they also tried a finer-grained top-$k$ distillation objective but observed no clear advantage in either convergence speed or final performance — consistent with the conclusion of Section 5.

For context, on K2 and K2.5 Moonshot still generated candidate responses with K1.5 and a number of internal domain specialists, filtered them, and trained with SFT — sequence-level off-policy distillation. Only with K3 did the recipe shift to student sampling with per-token scoring by nine specialists.

6.5 GLM-5: cross-stage distillation with historical checkpoints as teachers

GLM-5’s post-training is a long sequential pipeline: multi-task SFT (introducing the interleaved thinking mode) → reasoning RL → agentic RL → general RL. The three RL stages optimise different objectives, which brings a known problem: later stages erode capabilities acquired in earlier ones.

OPD is therefore placed at the very end of the chain as a dedicated repair stage. The report describes it as follows: optimising different objectives in sequence within a multi-stage RL pipeline causes cumulative degradation of existing capabilities; to mitigate this, the final stage performs on-policy cross-stage distillation, taking the final checkpoint of each preceding training stage as a teacher, with training prompts sampled from the corresponding teacher’s RL training set and mixed in proportion.

This is the most fundamental difference from the other four. Elsewhere the teacher is an external model stronger than the student, or at least stronger in some domain; GLM-5’s teacher is a snapshot of the student itself from an earlier stage and is no stronger than it. OPD here therefore does not transfer new capability; it realigns a distribution that has drifted back toward that checkpoint, recovering the capabilities overwritten by subsequent stages.

Two findings support why this works.

First, the gains from RL are highly localised: it fine-tunes only a small subnetwork of the model (Reinforcement Learning Finetunes Small Subnetworks in Large Language Models), which makes strengthening one domain liable to damage another — exactly the cumulative degradation GLM-5 describes.
Second, the source of the data matters decisively when recovering capability: Retaining by Doing shows that training on the model’s own on-policy data substantially mitigates forgetting, and RL’s Razor explains from a KL perspective why online methods forget less than offline SFT.

Together these imply that generating data with the old checkpoint and running another round of SFT would perform markedly worse — capability recovery must itself be on-policy.

6.6 Thinking Machines Lab: self-distillation for continual learning

Thinking Machines Lab generalises this usage to the continual-learning setting in On-Policy Distillation (the section on distillation for personalization). The setting is a Qwen3-8B that has already completed post-training and into which internal enterprise document knowledge is then injected; OPD follows the domain injection and performs capability recovery.

The problem arises in mid-training: internal QA rises from 18% to 43%, but IF-eval falls from 85% to 45%. The blog notes that this degradation is unrelated to the data mixture — IF-eval declines regardless of how the ratio is tuned — and that constraining parameter updates with LoRA merely “learns less while still forgetting”.

The recovery procedure is a round of OPD with the pre-mid-training Qwen3-8B itself as the teacher, using prompts taken from Tulu3 that are entirely unrelated to the internal documents, for the sole purpose of restoring instruction following:

Model	Internal QA (knowledge)	IF-eval (conversation)
Qwen3-8B	18%	85%
+ midtrain (100% documents)	43%	45%
+ midtrain (70% documents)	36%	79%
+ midtrain (70%) + distill	41%	83%

The last row is the notable one: while IF-eval recovers from 79% to 83%, internal QA rises from 36% to 41% — what the blog calls positive transfer between conversational ability and knowledge. This is the same mechanism as GLM-5’s cross-stage distillation: in both cases the teacher is a historical snapshot of the model itself, and OPD repairs the capability loss caused by subsequent training.

The same blog also reports a direct controlled experiment (dense supervision greatly improves compute efficiency): starting from Qwen3-8B-Base, RL on DeepMath yields a policy, and OPD then distils that policy back into the same base model — teacher and student share an architecture and a parameter count, and the only difference is the mode of learning.

The result is that OPD requires roughly one seventh to one tenth of the gradient steps to acquire that RL policy (the AIME score is recovered within 10 steps, against RL’s 70), which translates into a 50–100× improvement in compute efficiency.

An even more extreme setting is training on a single prompt: a limit problem is drawn at random from the dataset, 256 rollouts are sampled per step over 20 consecutive steps for 5,120 scored sequences in total, and the teacher’s AIME'24 performance is approximately reproduced. Under RL such training would almost certainly degenerate into memorising the answer; OPD, by contrast, fits the teacher’s full conditional distribution rather than any particular answer, and can therefore train repeatedly on the same set of prompts.

The blog makes the reason the title of a section — RL searches in the space of semantic strategies — which corroborates the judgement of Section 2: the cost of RL belongs mostly to search rather than learning. What is expensive is discovering, by trial in the space of strategies, which mode of reasoning earns the reward; once that strategy has been found, transferring it to another model by distribution matching is cheap.

7. Where the Efficiency of OPD Comes From

All five reports observe an efficiency gain, and none of them explains its origin. The most immediate explanation, that “dense supervision carries more information”, does not hold: SFT’s supervision is equally dense, yet it is not ten times more efficient than RL. Two works from 2026 answer the question, one at the level of phenomenology and one at the level of parameter dynamics.

Tsinghua’s Rethinking On-Policy Distillation answers “when does it fail”. It identifies two conditions for OPD to hold: the thinking patterns of student and teacher must be mutually compatible; and even with matching patterns and a higher score, the teacher must possess genuinely new capability the student has not seen. The weight of the latter is shown by a reverse control — running OPD between a 1.5B and a 7B model of the same family yields nearly identical final scores whether the direction is 7B → 1.5B or the reverse. Same-family models of different sizes are, in other words, nearly indistinguishable under the metric of token-level KL: superiority on benchmarks does not imply there is anything new to teach at the level of distributions. Two remedies follow: when the thinking patterns are incompatible, run a round of off-policy cold start first (as Qwen3 does); when the teacher has nothing new to teach, select only the prompts the teacher answers well and the student answers poorly.
USTC and Tencent’s Learning to Foresee answers “why is it fast”. Their answer is foresight — OPD locks onto the direction toward the final solution very early in training, after which training mainly accumulates magnitude along that established direction. The evidence sits at two levels:
- At the module level, updates concentrate on the mid-layer MLPs that actually carry reasoning ability, whereas RL distributes a large share of the update norm over the embeddings and the first and last layers, which contribute little;
- At the direction level, an SVD of the parameter increments shows that OPD’s energy is markedly more low-rank and that its dominant direction aligns closely with the final one very early, while RL stabilises much later. The most direct ablation takes a checkpoint at 10% of training progress and rescales the update norm per module to its final magnitude while leaving the direction untouched, recovering roughly 80% of the final reasoning performance.

Together these yield a sharper judgement: OPD is fast not because the signal is dense, but because the destination — the teacher’s distribution — is given in advance, so the direction need not be found by trial. And this is precisely what foreshadows its limits.

8. The Limits of OPD

The teacher constitutes a capability ceiling. The global optimum of OPD is a student distribution identical to the teacher’s, at which point both loss and gradient vanish. As long as the objective contains only the distillation term, the student cannot systematically surpass the teacher along any meaningful dimension. The Δ column in MiMo’s table, clustered near zero with several negative entries, is a direct manifestation of this property. Surpassing the teacher still requires RL to acquire new information from the environment — the ground-truth answer to a maths problem, the result of running code, the formal verification of a theorem — signals that do not pass through the teacher.
A genuine capability gap must exist. If the teacher is merely a larger sibling with a slightly higher score, the two may differ almost not at all at the scale of token-level distributions. Superiority on benchmarks is not necessarily reflected in per-token conditional probabilities. This is also why Qwen3 runs a round of off-policy cold start first: only once the two behavioural patterns are aligned does the KL carry knowledge rather than stylistic noise.
The dense signal is not free. The early directional lock-in of the previous section cuts both ways: the flip side of locking onto a direction is constrained exploration (though this is also, as discussed earlier, one of its advantages over SFT). Rethinking OPD accordingly raises the question of whether OPD can scale to long-horizon distillation. Following a teacher token by token along a long reasoning chain tends to transfer its local token habits rather than its reasoning strategy.

Looking back over the technical reports — Qwen3 uses it to train lightweight models, MiMo stacks it with an ORM reward, GLM-5 places it after three rounds of RL, DeepSeek-V4 and Kimi K3 place it after specialist RL — not one of them replaces RL with OPD outright. OPD always performs capability transfer; capability creation is still done by RL.

What OPD saves is samples, not GPU compute. Every rollout the student runs requires an accompanying teacher forward pass, so training must sustain inference resources for both models simultaneously. Qwen3’s figure of 1,800 GPU hours presupposes that a 32B or 235B teacher is already in hand; for teams with limited compute, the result is not necessarily reproducible.

9. Conclusion

OPD is not a new algorithm: on-policy sampling descends from the DAGGER line of imitation learning, training on student-sampled sequences was consolidated into a framework by Generalized Knowledge Distillation, reverse KL was established by MiniLLM, which also used the teacher-to-student log-probability ratio as a per-step reward, and the engineering scaffold is borrowed wholesale from PPO/GRPO with the advantage term as the only modification. Its real contribution is to have assembled these existing components into a form that is stable and cheap at industrial scale, making capability transfer, for the first time, a step that can be planned independently within post-training. The five labs had different motivations — lowering the training cost of lightweight models, merging domain specialists, repairing the degradation caused by sequential RL — but faced the same problem: as the cost of RL keeps rising and capability ends up dispersed across several specialist models, how does one move capability already paid for in compute into a single model at lower cost?

References

Thinking Machines Lab, On-Policy Distillation, 2025
Qwen Team, Qwen3 Technical Report, 2025
DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, 2026
LLM-Core Xiaomi, MiMo-V2-Flash Technical Report, 2026
Moonshot AI, Kimi K3: Open Frontier Intelligence, 2026
Zhipu AI, GLM-5: from Vibe Coding to Agentic Engineering, 2026
Li et al., Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, 2026
Cai et al., Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation, 2026 (EffOPD)
Song & Zheng, A Survey of On-Policy Distillation for Large Language Models, 2026
Ye et al., Black-Box On-Policy Distillation of Large Language Models (GAD), 2025
Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, 2026
Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024
Gu et al., MiniLLM: Knowledge Distillation of Large Language Models, 2023
Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024
Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010
Gudibande et al., The False Promise of Imitating Proprietary LLMs, 2023
Lightman et al., Let’s Verify Step by Step, 2023
旷野, Zhihu answer, 2026
燕雄飞的一天, Zhihu article, 2026
JMXGODLZ, Zhihu article, 2026
吴宇斌, Zhihu article, 2026
JustBeClaw, Zhihu article, 2026

Post-training | YuyaoGe's Website