Deep Dive | YuyaoGe's Website

Humanity's Language Space Is Converging

Thu, 13 Aug 2026 00:00:00 +0000

In July 2025, Cloudflare published an analysis of data collected from June 19 to 26: the balance between how much content AI platforms crawled from websites and how much referral traffic they returned was breaking down. For Anthropic, based on the web referrals visible to Cloudflare, the platform crawled roughly 70,000 pages for every visit it sent back to a content site.¹ One year later, Cloudflare observed that non-human requests accounted for more than half of all requests on its network for the first time, and that 52% of crawler requests were used for AI training.²

Meanwhile, the rate at which humans actively contribute knowledge is falling. According to the public Stack Exchange API,³ Stack Overflow received 109,294 new questions in the month ChatGPT was released. By July 2026, that number had fallen to 1,437—a decline of 98.7%. The knowledge community that taught language models how to answer programming questions is being replaced by the models it taught.

These two trends lead to the question at the heart of this article: what will happen to humanity’s language space in the age of AI? I begin with the proposition that “language is intelligence,” define the stock and flow of language space, and then offer my view on the question.

I. Language Is Intelligence

In 1950, Alan Turing tried to answer the question “Can machines think?” in Computing Machinery and Intelligence.⁴ He quickly recognized, however, that “thinking” itself could not be defined rigorously. He therefore set the question aside and replaced it with an operational test: place a human interrogator in text-only conversations with both a machine and another human; if the interrogator cannot tell which is which, then the machine has exhibited intelligence.

Alan Turing as portrayed in The Imitation Game

The key move was this: Turing replaced the difficult question “Does it possess intelligence?” with the operationally testable question “Can it express intelligence through language?” This article adopts the same operational definition. It does not ask what intelligence ultimately is at the level of consciousness; it asks only whether intelligence can be expressed, transmitted, and observed.

For computers, this substitution is particularly elegant. Everything in a computer—text, images, audio, and video—can be represented in binary, and binary is itself a language in the broad sense. You cannot, of course, expect me to watch short videos by staring at a string of zeros and ones. Yet with a fixed symbol set and rules of composition, that string can indeed be reconstructed into any of these modalities. Any multimodal information in a computer can therefore be represented as a form of language.

As for memory, consciousness, and the other parts that resist clear description: they have either been expressed in some form, or they have never entered public language. As Wittgenstein wrote in the Tractatus Logico-Philosophicus, “Whereof one cannot speak, thereof one must be silent.” What cannot be expressed does not participate in transmission, accumulation, or training, and therefore lies outside the scope of this article.

Within this framework, then, language is intelligence that can be expressed and transmitted.

II. Language Space

Language Has Boundaries

The Tractatus contains another equally famous line: “The limits of my language mean the limits of my world.”

Our framework considers only intelligence that can be expressed and transmitted. If language has boundaries, then this part of intelligence has boundaries as well. We can therefore substitute the boundary of language for the boundary of intelligence and discuss its extent and size.

To make that discussion possible, I will introduce a model of language space that is not rigorous, but is useful for discussion.

Everything that can be expressed forms a high-dimensional space, which I call the total language space. The content that any individual entity—a human or a language model, for example—can express forms a language subspace within it. Under the definition above, the boundary of that subspace is also the boundary of the intelligence that the entity can manifest.

To describe how much humanity has already understood and expressed, define the subspace size $C(t)$:

$$ C(t)=\left|\Omega(t)\right|,\qquad \Omega(t)=\bigcup_{i=1}^{N(t)}R(i) $$

Here, $\Omega(t)$ denotes the human language subspace that has entered public expression by time $t$; $R(i)$ is the language subspace occupied by the $i$-th human entity; and $N(t)$ is the number of such entities. $C(t)$ is therefore the size of the human-known language subspace. In the two-dimensional projections below, it corresponds to the projected area jointly covered by all these subspaces.

Different entities have different language boundaries. To use a simple example, a senior professor generally knows more than a young child and therefore occupies a larger language subspace. Yet they still share a native language, everyday experience, and much else, so their subspaces overlap substantially. Likewise, a stronger language model will generally occupy a larger subspace than a weaker one.

With these concepts in place, we can project the model into two dimensions:

Language-subspace projections of humans and language models

The circles above represent two-dimensional projections of language subspaces. On the left are hundreds of human subspaces of varying sizes, many of them overlapping; isolated points near the edge correspond to obscure knowledge held by only a few people. On the right are only four language-model subspaces, but because each model has been trained at scale, each individual projection is large.

The contrast is striking: humans form a long tail through a large number of small subspaces, whereas language models cover common knowledge through a small number of large subspaces.

Language models also share large amounts of the same internet training data—sometimes even the same data providers—and are often distilled from one another. Their subspaces therefore overlap heavily. Adding more models does not expand language space in the same proportion.

The other side of the coin is that a language model can attain a very high degree of mastery over parts of its own subspace. Within a familiar region, its answers may be more accurate than those of most humans who occupy the same region.

Add a $z$-axis to the two-dimensional projection, using height to represent an entity’s degree of mastery at each location, and the result is the following three-dimensional terrain:

Projected area and degree of mastery

The dashed lines above mark the boundaries of the subspaces, while height represents mastery. To make the comparison clear, the two projected areas are drawn at the same size. On the left is a human: several domains contain modest peaks of accumulated expertise, separated by flat ground and depressions. On the right is a language model: one extremely high and steep peak rises within its familiar subspace, then drops rapidly beyond it.

Two concepts must therefore remain distinct: the boundary encloses the projected area, while terrain height represents mastery at a particular location. A language model may achieve very high mastery within an existing subspace without increasing the size of that subspace.

The Stock and Flow of Language Space

With subspace size defined, we can return to the original question: is internet content converging or diverging?

The amount of content on the internet can continue to grow while the human-known language subspace stops growing. The former counts the number of points; the latter measures the range jointly occupied by those points.

Because $C(t)$ changes over time, we can distinguish two quantities:

The stock is $C(t)$ itself: the size of the human-known language subspace.
The flow is the rate at which subspace size grows, $F(t)=C^{\prime}(t)$: the amount of language space newly added per unit time that no entity had previously reached.

Whether language space converges should therefore be determined by whether $C(t)$ has a finite limit:

$$ \text{convergence} \iff \exists\, C_{\mathrm{max}} \lt \infty,\quad \lim_{t \to \infty} C(t) = C_{\mathrm{max}} $$

In other words, if $C(t)$ eventually approaches a finite value, the human-known language space converges. In the smooth model used here, its flow $F(t)$ then gradually approaches zero. The converse does not automatically hold: flow approaching zero is not by itself sufficient for convergence. This article makes the additional assumption that the sum of all future additions to language space is finite.

One point is especially important: convergence does not mean collapse to zero. $C(t)$ is cumulative. It can stop growing, but it does not return to the origin.

Total flow has three sources—human entities, language models, and Agents—so I will examine how each affects the human-known language space:

$$ F(t)=F_{\mathrm{human}}(t)+F_{\mathrm{LM}}(t)+F_{\mathrm{Agent}}(t) $$

These terms classify new knowledge by its source. An output from a language model or Agent counts toward $\Omega(t)$ only after it has been confirmed and entered public expression.

III. Language Models and Agents Contribute Differently to Flow

A Model Sees a Projection of the Real World

By language model, I mean a model that does not call tools or receive feedback from an external environment, and instead predicts subsequent tokens solely from the context it has already received. It is trained by fitting human knowledge products. What it encounters is not the real world itself, but a projection of the real world onto human language.

A language model can combine and interpolate within an existing subspace, producing sentences that never appeared in its training set. But being novel at the surface level does not mean that an utterance has left the human-known language space at the semantic level.

Under the central premise of this article, the relation among the three spaces can be written as:

$$ \mathcal{L}_{\mathrm{LM}}(t) \subseteq \mathcal{L}_{\mathrm{human}}(t) \subseteq \mathcal{L}_{\mathrm{all}} $$

These three terms denote the language-model subspace, the human-known language space, and the total language space. A language model can add content within the existing range while leaving the boundary of the human-known language space approximately unchanged.

This requires distinguishing two concepts:

A language model can achieve greater mastery within an existing subspace: understanding the same class of problems more fully and answering them more accurately.
The human-known language space can be expanded: content beyond the old boundary can be understood and expressed for the first time.

Present-day language models mainly do the former.

Adding content versus expanding the boundary

The dark regions above represent new content. In the left panel, the boundary remains fixed and all new content falls within the existing range, so only the quantity of content increases. In the right panel, new content appears beyond the old boundary, expanding the human-known language space. Both cases may be described as “generating new content,” but only the latter increases subspace size.

The Trend in Subspace Size

Once stock and flow have been separated, and language models have been shown to add content primarily within an existing subspace, we can draw the trend projected by this article:

Projected evolution of the human-known language subspace

The vertical axis above is the human-known language subspace size $C(t)$, and the horizontal axis is time. The dot marks the transition from accelerating to decelerating growth; the dashed line shows what would happen if that transition never occurred.

This is a conceptual trend, not a statistical fit to historical data. In prehistory, durable written media were scarce, so expression could not readily accumulate across generations. Ancient societies began to accumulate knowledge, but the rate was limited by literacy and the cost of copying. Printing, the Industrial Revolution, and the information age then increased both the number of people able to express themselves and the speed of transmission, pushing subspace size into a period of accelerating growth.

In the future, once language models enter the picture—and again, I mean language models rather than tool-using Agents—if human-contributed flow continues to decline and Agents fail to replace the missing flow, a fourth stage begins: subspace size shifts from accelerating growth to decelerating growth, and the human-known language space shifts from accelerating expansion toward gradual convergence.

How Humans and Agents Contribute Flow

The human-known language space has expanded because humans interact with the real world.

Reinforcement learning models precisely this process. Humans act in an environment; the physical world and other humans provide feedback; that feedback changes what they know; and the resulting knowledge is externalized in language. A materials scientist, for example, may learn from a series of failed experiments and, in one experiment, finally discover an entirely new material and receive a reward. The boundary of human language—that is, human knowledge—thereby expands.

Human language has evolved through this process. By contrast, if a language model only statistically fits existing human knowledge products, it will struggle to move beyond the boundary already drawn by its training data. This is the generalization boundary at issue here.

The same argument reveals the fundamental difference between a language model and an Agent. A language model can process only what has entered its context; an Agent can call tools, take actions, and receive feedback from an environment, giving it the opportunity to obtain observations absent from existing human knowledge products.

Return to the laboratory. Suppose an Agent controls experimental equipment, varies the conditions automatically, and observes a previously unknown phenomenon. If the phenomenon can be reproduced and validated, then expressed as a new concept by either the Agent or a human, it has the same structure as a scientific discovery: the real world supplies new feedback, the feedback becomes new knowledge, and the knowledge expands the human-known language space. The fact that language models struggle to contribute flow does not mean that Agents cannot contribute it.

Real-world interaction creates new flow into language space

The image above renders this feedback loop as a laboratory: existing literature supplies the starting point, the environment provides new feedback, and a validated result is ultimately expressed as new knowledge.

Even when an Agent acts only inside a digital sandbox, the sandbox is not an absolute constraint. The rules of Go were designed by humans, yet AlphaGo produced moves that no human had played before. Conway designed the rules of the Game of Life, yet he did not foresee either the glider or the system’s Turing completeness. A sandbox constrains the axioms, not every conclusion that follows from them.

Whether an Agent Can Expand the Boundary Depends on Environment and Reward

Whether an Agent genuinely contributes flow depends on whether its environment supplies new information, and whether its reward can distinguish a discovery from an error. An Agent that merely searches existing webpages still primarily recombines human knowledge products. An Agent that obtains new observations, and receives rewards grounded in experimental replication, program verification, or physical outcomes, may expand the human-known language space.

For language models and Agents rewarded only by human preference, the original limitation remains. What counts as a better answer must ultimately be judged either by humans or by a reward model fitted to human preferences. To an existing evaluation system, an output genuinely outside the human-known language space is often indistinguishable from an error: both are unfamiliar, low-probability, and difficult to reward positively.

In other words:

If reward comes only from human preference, then asking a system to leave the language space already occupied by humans is in inherent tension with asking it to align strictly with human preference.

This leads to a further conclusion. Under the current training paradigm, a language model can attain very high mastery of knowledge within an existing subspace, but it struggles to expand the human-known language space reliably, so its language-space flow is approximately zero. An Agent that can engage the real world, obtain new observations, and submit them to objective validation contributes nonzero flow.

IV. Human-Contributed Flow Is Approaching Zero

Although Agents may become a new source of flow, interaction between humans and the real world remains the primary way language space expands today. Yet human-contributed flow is declining for two reasons: strong language models are changing people’s willingness to create, and they are weakening the need to document. At the same time, the volume of language-model output is reducing the share of original human content.

Willingness: Knowledge Is Becoming Cheap

By a strong language model, I mean one whose output quality already exceeds that of most humans. Before such models appeared, writing a long article with genuine insight required research, problem solving, and independent judgment.

Yet strong language models lower the cost of writing for everyone at once. A well-structured article with correct terminology and coherent logic no longer proves whether its author spent years mastering the field or encountered it only yesterday. The same ambiguity applies to this article—although it was in fact written by the author himself.

In my own field, many programmers once wrote blogs partly to demonstrate technical depth and exchange ideas with peers, and partly to summarize their work. When an article can no longer prove that its content came from the author, the incentive to use writing as evidence of one’s individuality, distinctive views, and problem-solving ability naturally declines.

Article-quality distributions before and after strong language models

The image above is conceptual. The dark curve represents authors who genuinely understand the field; the light curve represents authors without the same background. In the past, the quality distributions were far apart, so an article served as a signal of its author’s ability. With a strong language model, the two distributions overlap heavily. The writing remains fluent, but the signal loses its power to discriminate.

Creators respond in two opposite ways:

Some rush in. When production costs have already fallen but evaluation systems have not yet adjusted, the time lag creates an arbitrage opportunity. Academic papers, for example, have historically brought economic and reputational rewards. If language models sharply lower the writing barrier, creators are incentivized to increase output before the evaluation system catches up.
Others leave. For those who once used papers or blogs to demonstrate accumulated expertise, the return on public writing falls once an article can no longer establish that the ideas came from its nominal author. Luo Fuli, for example, has publicly said that she now rarely reads academic papers.

The second mechanism is that the need to document is weakening.

Many people once published articles because they had solved a problem only after reading a great deal and trying repeatedly, and wanted both to preserve the process and summarize the result. Writing was a byproduct of understanding: the author first had to understand the problem before explaining it. Now, more and more problems can be answered with a single prompt. When solving a problem no longer requires the user to understand it deeply, the need to record and share the process also declines.

Someone who solves a century-old mathematical problem will naturally publish a paper announcing the solution. Someone who completes an elementary arithmetic exercise has no reason to write an article about it. One role of language models and retrieval Agents is to turn many problems that once required dedicated documentation into questions that can be answered directly through autonomous search.

Capacity: Human Output Is Being Drowned Out

Whatever happens to incentives, some people will always choose to create. We must therefore consider a factor independent of psychology: the enormous gap between the content generation rates of humans and language models. The gap is not limited to text. Model-generated images, audio, and video also add content within existing subspaces.

This content-generation capacity is also moving from data centers onto every endpoint.

Some 7B–32B models already outperform early GPT-3.5 Turbo on several public benchmarks. As compression and training efficiency improve, language models will continue moving onto phones, personal computers, and embedded devices. Once those devices continually generate content and feed it back onto the internet, the aggregate content generation rate will far exceed human production.

There is an easily overlooked detail here. Model miniaturization often relies on distillation: a smaller model learns the output distribution of a larger teacher. The smaller model is therefore not a wholly independent linguistic entity; its subspace remains substantially constrained by its teacher.

In the language of our two-dimensional projection, miniaturization does not naturally create an independent new subspace. It copies an existing subspace onto billions of devices. The human-known language subspace changes little in size, while the number of content points grows rapidly.

This produces two further consequences.

First, the last throughput constraint on content generation disappears. Large-scale generation once depended on data centers, which imposed compute costs and left only a few organizations that could be held accountable. Once generation happens on every endpoint, total output is no longer constrained by any single provider.

Second, content provenance becomes harder to trace. Watermarking AI content requires the generator to cooperate. With open weights and local deployment, anyone can remove a marker or use a model that never adds one. Model-side watermarking alone is therefore unlikely to produce an enforceable universal standard.

Original Human Content Is Becoming the Long Tail

Here, the long tail no longer means minority languages, peripheral disciplines, or niche technology stacks. It describes the source distribution of future internet content. Because language models generate content much faster than humans, model-generated content will occupy the head by volume, while original human content gradually recedes into the tail.

Plot the content added during a given period by source and quantity, and the distribution looks roughly like this:

The source distribution of newly added internet content

Each point above represents one new piece of content. The enormous quantity of model-generated content forms a dense head, while the smaller amount of original human content forms a sparse, elongated tail. The long tail here describes only the relative quantities of the two sources.

Encountering original human content on the internet often feels like meeting an old friend in a foreign land.

V. The Inflection Point in Acceleration

Agents can contribute flow, but at least for the foreseeable future, they remain insufficient to offset the decline in human flow and have not formed a stable, sustainable source of new additions. There are two reasons:

Very few Agents interact with real environments rather than merely retrieving webpages, and each run produces only a limited amount of new knowledge that can be validated.
Real-world interaction requires tools, environments, time, and capital, making it far more expensive than an ordinary generation call.

Under this assumption, I still expect the total flow of humanity’s language space to approach zero, the sum of future additions to remain finite, and the stock $C(t)$ to approach a horizontal asymptote. Agents can slow this process for now, but cannot reverse it.

Plotting the derivative of the earlier subspace-size curve makes this change easier to see:

The changing flow of the human-known language space

The flow $F(t)=C^{\prime}(t)$ above is the amount of new language subspace added per unit time—the derivative of the subspace-size curve $C(t)$. Before the Industrial Revolution, flow remained low. It then rose rapidly and reached a peak, before declining in the future. Flow approaching zero does not mean that the existing subspace shrinks; it means only that the human-known language space ceases to expand.

Declining flow does not mean less internet content. Quite the opposite: there will be more content than ever, but most of it will consist of extracting, recombining, and republishing existing human knowledge products. Quantity keeps growing, while the boundary fails to expand at the same rate.

Put differently, most current language models and retrieval Agents consume the knowledge products accumulated by humans without expanding the boundary of language space at a comparable rate. Webpages, papers, and images do not disappear, but existing knowledge is processed repeatedly while knowledge beyond the boundary is not produced at the same pace.

The physical concept of heat death offers an analogy for this endpoint. Energy still exists, but without a temperature gradient it can no longer do work. Language space is similar: the stock remains, and content becomes more abundant, but the boundary no longer expands. Agents offer a theoretical exit, yet their present scale, knowledge output, and operating cost remain insufficient to change the outcome.

The heat death of language space—and a possible exit

In the image above, the shelves and terminals filling the room represent an abundant stock that still remains. The door to the real world represents observations that humans and Agents may still bring back.

VI. Three Implications for the Future

Implication One: Individual Gains and Collective Stagnation

Strong language models are plainly valuable to individuals. A knowledge framework that once required years of study can now be acquired through a few hours of conversation; writing, research, and organizing ideas all become faster.

Yet widespread individual benefit is compatible with a language space that has stopped expanding. Raising the average level does not expand the boundary; subspace size increases only when content lands outside the existing range. Strong language models raise everyone’s floor, but also pull more people toward similar answers. The mean rises; the boundary may not move.

Implication Two: Scarcity Shifts Back to People

When one factor of production becomes cheap, value shifts to complementary factors that remain scarce. Once access to knowledge is cheap, at least four things remain scarce: trust, distribution and attention, taste, and responsibility. A language model can provide an answer, but it cannot decide on a person’s behalf which questions are worth answering, nor can it assume responsibility for the consequences. Human connections therefore become more important.

Implication Three: Efficiency Becomes Stratified Again

Within the same group, people who use Agents proficiently will rapidly converge in productivity. At the same time, the efficiency gap between those who use Agents and those who do not will widen sharply.

Efficiency stratification created by Agent use

The vertical axis above represents work efficiency. In the left panel, five people occupy three tiers, with roughly equal gaps between adjacent tiers. In the right panel, the three Agent users rise together and move closer to one another, while the two non-users remain where they were. The result is a smaller gap within the first group and a larger gap between groups.

VII. Conclusion: From Explanation to Bearing Witness

If the flow of humanity’s language space comes from interaction with the real world, then the most important value of personal writing is no longer to repeat explanations of existing knowledge, but to record information that only you could obtain by being there.

A strong language model can write a clearer tutorial and assemble a more comprehensive survey. But it does not know the complete sequence of events from the first alert to the final diagnosis in that production incident; it does not know the bug that reproduces only on your machine; and it does not know why one set of experiments failed seventeen times in a row. Those facts have not yet been expressed, and therefore do not exist in the model’s training world.

Agents offer another possibility. An Agent that only searches webpages remains inside existing knowledge. An Agent connected to sensors, experimental equipment, and the physical environment may, like a human, bring new facts back into language space.

For now, however, a single Agent run still incurs substantial costs in time, capital, and access to a suitable environment, so it produces little verifiable new knowledge. Meanwhile, language models can generate content in bulk at almost no cost, further decoupling content quantity from boundary expansion. Agents can expand the boundary, but they still cannot change the trend of language space toward convergence.

In sum, the author believes that humanity’s language space is converging.

Cloudflare, The crawl before the fall… of referrals: understanding AI’s impact on content providers, 2025. ↩︎
Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet, 2026. ↩︎
Stack Exchange Questions API: November 2022 query; July 2026 query. The totals may change slightly as questions are deleted or restored. ↩︎
Turing, A. M. Computing Machinery and Intelligence. Mind, 1950, 59(236): 433–460. ↩︎

From SFT and RL to On-Policy Distillation: A Third Paradigm for Post-Training

Tue, 11 Aug 2026 00:00:00 +0000

Since the release of DeepSeek-R1, GRPO has become the default choice for post-training, and the “SFT + RL” recipe has found its way into virtually every technical report. Yet beginning with Qwen3 in 2025, one flagship model after another has independently placed the same method on its critical path: on-policy distillation (OPD).

The first flagship report to adopt it was Qwen3, in the form of strong-to-weak distillation. The flagship models still went through the full four-stage post-training procedure, but the lightweight models from 0.6B to 14B no longer repeated that procedure individually; they were distilled instead from Qwen3-32B or Qwen3-235B-A22B. The report notes that this yielded higher performance while consuming roughly one tenth of the GPU hours required by the four-stage pipeline.

Six months later, Xiaomi’s MiMo-V2-Flash named the method MOPD (Multi-Teacher On-Policy Distillation) and framed it explicitly as a post-training paradigm. Its post-training is organised into three stages: general SFT, then per-domain teachers trained with RL or SFT, and finally MOPD to merge those teachers back into a single student. Its teachers are the domain specialists produced by the independent RL runs of Stage 2.

Moonshot’s Kimi K3 pushes the same idea one step further. It partitions specialists not only by domain but also by reasoning effort, training three levels — low, high and max — within each of three domains, for nine specialists in total, and again merges them with MOPD.

Zhipu’s GLM-5 takes a different route. Its teacher is neither a larger model nor a domain specialist, but the model’s own checkpoints from earlier training stages; OPD sits at the very end of the post-training pipeline and serves to repair the capability degradation introduced by sequential RL.

The most aggressive adoption is DeepSeek-V4, whose report states outright that the mix-RL stage of the V3.2 era has been replaced in its entirety by OPD: a specialist is first trained for each of more than ten domains — mathematics, code, agentic tasks, instruction following and so on — and multi-teacher OPD then merges them back into a single unified model.

The five labs differ in where they place the method, where their teachers come from, and at what granularity they compute the KL, but the underlying recipe is identical: the student samples its own trajectories, and the teacher scores every token along them. A method that once appeared only in the distillation literature has, within two years, moved onto the critical path of five flagship reports. This article addresses three questions: where exactly SFT and GRPO break down, what the mathematical form of OPD is, and what problem each of these reports uses it to solve.

1. SFT: Dense Supervision, but Exposure Bias

2023 was the breakout year for large language models, and SFT broke out along with them; for many practitioners it was the first thing they ever trained. SFT is the step that turns a pretrained model, capable only of continuation, into a usable assistant. Its rise can be traced along two lines of work:

First, in 2021, FLAN and related work showed that fine-tuning on multi-task instruction data enables zero-shot instruction following, establishing instruction tuning as a standard step;
Second, in 2023, open-source efforts such as Alpaca and Vicuna replaced human annotation with the outputs of a stronger model — using Self-Instruct to generate tens of thousands of responses from GPT-3.5 and training small models directly on them, producing usable assistants at very low cost. It is this second line, the distillation setting, that concerns us here.

Applied to distillation, SFT proceeds as follows:

the teacher generates a complete high-quality response, yielding a full token sequence $y$;
at every position $t$, the teacher-generated prefix is fed to the student, which outputs logits for the next token;
the teacher’s actual token $y_t$ at that position serves as the target, against which the student’s distribution is scored by cross-entropy.

The loss is therefore

$$ \mathcal{L}_{\mathrm{SFT}}(\theta) = -\,\mathbb{E}_{(x, y) \sim \mathcal{D}_T} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{\lt t}) $$

Two details are worth committing to memory: the prefix comes from the teacher, and the supervision target is a single hard token produced by the teacher.

This form makes SFT’s central advantage immediate — the supervision is dense. A sequence of length $T$ supplies $T$ gradient signals, and the further the student deviates at a given position, the larger the loss and the update at that position. Combined with the fact that the objective is standard maximum likelihood, that no sampling is required during training, and that results are stable and reproducible, SFT remains the first step of the distillation pipeline nearly everywhere, from DeepSeek-R1-Distill to the lightweight Qwen3 variants.

At scale, however, several problems have surfaced.

The first is distribution shift. Every prefix presented to the student during training comes from a flawless teacher trajectory, whereas at inference the prefix it continues is whatever it has just produced itself. The moment the student makes an error the teacher would never make, it enters a state absent from the training data; the remaining tokens are generated without guidance, and the error compounds along the sequence. In sequence generation this is known as exposure bias; in imitation learning it is called compounding error. DAGGER argued as early as 2010 that the only remedy is to let the apprentice act on its own and then obtain expert labels at the states the apprentice actually reaches. The gap is quantitative: Tencent’s A Survey of On-Policy Distillation formalises it by showing that the compounding error of off-policy imitation grows roughly with the square of the sequence length, whereas taking feedback on the student’s own distribution reduces it to linear.

The second is the superficiality of imitation. Gudibande et al. provide empirical evidence in The False Promise of Imitating Proprietary LLMs: imitation training readily transfers the teacher’s style and confidence rather than its factual accuracy. The reason is not hard to see — the teacher can produce that response because an entire body of knowledge and reasoning ability stands behind it, while the student observes only the token sequence and can imitate only the part that “looks right”.

The third is mode covering, in which the student is forced to cover every mode of the teacher’s distribution and ends up mastering none of them.

Concretely, suppose a problem admits two entirely different solution paths — setting up an equation and solving it, or enumerating and verifying candidates directly — and the teacher assigns them 20% and 80% probability respectively. The SFT loss is an expectation under the teacher’s distribution, which means every solution the teacher samples is used to examine the student. Consequently, if the student were to abandon path A and drive its probability to zero, then at the positions where the teacher generates A we would have $\log \pi_\theta(\text{A}) = -\infty$ and the loss would diverge.

The student therefore has no choice but to reserve probability mass for every mode:

Forward KL — mode covering: the student is forced to assign probability to both modes, leaving mass even in the region between them.

Given ample capacity this need not be harmful, but the student’s capacity is precisely what is limited: unable to master both paths at once, it converges to an interpolation of the two. What it samples at inference is neither a complete equation-based solution nor a complete enumeration, but an internally inconsistent hybrid of the two.

The root of this behaviour is the zero-avoiding property of forward KL: wherever $\pi_T(x) > 0$, the student is not permitted to let $\pi_\theta(x) \to 0$, or the loss diverges. In other words, the behaviour is dictated by the direction of the divergence itself — the SFT cross-entropy is, in form, a forward KL, a point developed in Section 4. It was on exactly this basis that MiniLLM argued in 2023 that distilling small models calls for the zero-forcing reverse KL instead, so that the student converges onto a single mode rather than covering all of them.

2. GRPO: No Distribution Shift, but Sparse Reward

After DeepSeek-R1, the default on the RL side became GRPO. Where PPO requires an additional value network of comparable size to the policy in order to estimate the advantage, GRPO removes the critic altogether: it samples a group of $G$ responses for the same prompt and uses the group-normalised return as the advantage.

$$ \hat{A}_{i} = \frac{R_i - \mathrm{mean}(\{R_1, \dots, R_G\})}{\mathrm{std}(\{R_1, \dots, R_G\})} $$

This simplification suits the RLVR setting, where each trajectory carries a single verifiable scalar reward at the end. A mathematics task requires only checking whether the answer is correct, a coding task only whether the predefined unit tests pass, so the reward function can be kept extremely simple.

More importantly, RL resolves SFT’s distribution shift at the root: trajectories are generated by the model itself, the state distribution during training matches the one at inference exactly, and exposure bias does not arise.

The cost of this route is equally clear, and it is concentrated in one place: it provides a single scalar at the end of an entire trajectory.

Concretely, the student writes a 500-token chain of reasoning and the verifier replies with one sentence: wrong answer, reward = 0. Thinking Machines Lab offers an information-theoretic comparison (from LoRA Without Regret): reinforcement learning conveys $O(1)$ bits to the model per episode, whereas distillation conveys $O(N)$ bits per episode, where $N$ is the number of tokens.

The first consequence is that the student does not know where it went wrong. Since that single bit must be distributed across 500 tokens for credit assignment, the student learns only that it lost, and cannot localise whether step 17 miscalculated, step 200 skipped too far in its reasoning, or the final step inverted a sign. The only thing it can do is suppress the probability of the entire trajectory — including the intermediate steps that were in fact correct.

The second consequence is time and compute. Because the information content of a single feedback signal is fixed, developing a capability requires accumulating volume through repeated sampling: dozens of rollouts per problem, hundreds of thousands of prompts, until the statistical differences emerge. Notably, the bulk of this compute is not spent on gradient updates but on sampling and exploration. The Qwen3 report, for instance, records 17,920 GPU hours of mathematics and code RL for the 8B model.

One direct line of attack is to make the reward itself dense. OpenAI’s Let’s Verify Step by Step demonstrates that process supervision outperforms outcome supervision by training a process reward model to score each step, which amounts to densifying a sparse reward by hand; the price is a large volume of step-level annotation, and the PRM is itself susceptible to being hacked.

Placing the two routes side by side:

Method	Sampling source	Feedback density
SFT / off-policy distillation	Off-policy	Dense (every token)
Reinforcement learning	On-policy	Sparse (one scalar per trajectory)
On-policy distillation	On-policy	Dense

3. OPD: Dense Reward on On-Policy Trajectories

The construction of on-policy distillation follows directly from the empty cell in that table: the student samples the trajectory, and the teacher scores it token by token.

Relative to SFT, OPD proceeds as follows:

the student first generates a complete response under its current policy, yielding a trajectory $\hat{y}$;
at every position $t$, the student’s own prefix is fed to both the student and the teacher, each of which outputs a distribution over the next token;
the supervision target is no longer a single hard token but the teacher’s full conditional distribution given that prefix, with the KL divergence measuring the gap between the two distributions.

In other words, OPD changes exactly two things relative to SFT: the prefix comes from the student rather than the teacher, and the supervision target becomes the teacher’s full distribution rather than one hard token. The first change eliminates distribution shift; the second raises the information carried at each position from a single token to an entire distribution over the vocabulary.

The reward at each position is correspondingly defined as the negative reverse KL:

$$ R_t = -\,\mathrm{KL}\big(\pi_\theta(\cdot \mid x, y_{\lt t}) \,\|\, \pi_T(\cdot \mid x, y_{\lt t})\big) $$

There is no reward model here and no additional regularisation term: the KL is the entire optimisation signal, and it is defined at every token.

This per-token scoring can be visualised directly. The figure below is a real trajectory from the TML blog; darker tokens carry higher reverse KL, meaning stronger disagreement from the teacher:

One counter-intuitive detail stands out: the final, incorrect answer token is not heavily penalised. The reason is that once the earlier reasoning has gone astray, that answer is in fact “predictable” conditional on the flawed premise; what gets penalised are the intermediate tokens at which the reasoning diverged. RL, by contrast, can only assign one scalar to the whole trajectory and has no way to distinguish between these two kinds of position.

4. Why It Has to Be Reverse KL

Classical distillation uses forward KL; OPD uses reverse KL. The two merely swap the positions of $P$ and $Q$, yet they behave very differently.

Forward KL: mode covering

$$ \mathrm{KL}(\pi_{T} \| \pi_{\theta}) = \mathbb{E}_{x \sim \pi_{T}}\left[\log \frac{\pi_{T}(x)}{\pi_{\theta}(x)}\right] $$

When taking gradients with respect to $\theta$ the first term is constant, and what remains is the expectation of $-\log \pi_\theta$ under the teacher’s distribution — that is, a cross-entropy. Comparing this with the SFT loss above shows that the two are formally the same objective: SFT minimises a forward KL, the only difference being that its teacher distribution degenerates to a one-hot label, whereas in distillation it is the teacher’s full soft distribution.

This also explains the mode covering of Section 1: because the expectation is taken under the teacher’s distribution, every mode the teacher would generate must be covered, and omitting any one of them causes the loss to diverge.

Beyond mode covering, this direction has a complementary blind spot: since the expectation is taken under the teacher, the loss is entirely insensitive to how the student allocates probability in regions where the teacher’s probability is zero. The student is free to pile up probability where the teacher never ventures without incurring any penalty — which is the other half of the explanation for “right style, wrong answer”.

Reverse KL: mode seeking

$$ \mathrm{KL}(\pi_{\theta} \| \pi_{T}) = \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \frac{\pi_{\theta}(x)}{\pi_{T}(x)}\right] $$

Splitting the log-ratio into two terms reveals a further reading:

$$ \mathrm{KL}(\pi_{\theta} \| \pi_{T}) = \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \pi_{\theta}(x)\right] - \mathbb{E}_{x \sim \pi_{\theta}}\left[\log \pi_{T}(x)\right] $$

By definition the first term is the negative entropy of the student, $-H(\pi_\theta)$, and the second, with its sign flipped, is the cross-entropy of the student against the teacher, $H(\pi_\theta, \pi_T)$. Hence

$$ \mathrm{KL}(\pi_\theta \| \pi_T) = \underbrace{H(\pi_\theta, \pi_T)}_{\text{cross-entropy to the teacher}} - \underbrace{H(\pi_\theta)}_{\text{student's own entropy}} $$

Minimising the reverse KL therefore does two things at once: it lowers the student’s cross-entropy against the teacher and raises the student’s own entropy. Were the cross-entropy minimised alone, the objective would be linear in $\pi_\theta$ and its optimum would lie at a vertex of the simplex, placing all probability on the single token the teacher deems most likely and collapsing to a point mass. The entropy term counteracts precisely this concentration: when the two terms carry equal weight, the optimum is exactly $\pi_\theta = \pi_T$. Even when the student lacks the capacity to cover more than a subset of the teacher’s support, the optimum is the teacher’s conditional distribution restricted to that subset, preserving the relative proportions within it. Mode seeking therefore means deciding which modes to abandon, not sharpening the distribution within the modes that are retained.

Returning to behaviour: because the expectation is now taken under the student’s distribution, the teacher only evaluates what the student actually writes. Take the same two solution paths: if the student concentrates all its probability on path B, the teacher assigns B 80% and the KL is small; path A is ignored entirely, but since the student never samples A, that region contributes nothing to the loss.

Reverse KL — mode seeking: the student concentrates all probability mass on the one mode the teacher endorses and abandons the other outright.

Mode seeking is better aligned with the actual objective of distillation. The student’s capacity is far smaller than the teacher’s, so requiring it to cover every teacher mode is unrealistic to begin with; under forward KL, insufficient capacity leads directly to the mode covering of Section 1, in which neither solution path is learned. Reverse KL relaxes the requirement to “the one mode the student selects must be endorsed by the teacher”, allowing the student to concentrate its probability mass on a single solution path and learn it to a usable standard.

The second property of reverse KL is that it is unhackable. Because the expectation is taken under the student’s distribution, a low KL necessarily means that every token the student actually generates is assigned high probability by the teacher; there is no solution that scores well on the metric while producing poor output. Two contrasts convey the weight of this property: when a learned reward model provides the signal, the policy can find regions the reward model does not cover and inflate its score there — reward hacking; forward KL, meanwhile, imposes no penalty at all in regions where the teacher’s probability is zero. Reverse KL admits neither loophole, for the simple reason that the object being evaluated is exactly what the student itself produces.

The choice of direction is not a new conclusion. Besides MiniLLM, mentioned in Section 1, Google DeepMind’s Generalized Knowledge Distillation (GKD) wrote down the full recipe of “on-policy sampling plus token-level KL” in the same year, and today’s industrial OPD variants can largely be viewed as special cases of the GKD framework.

5. The Further Degeneration of Advantage Estimation

OPD can be dropped into an existing RL framework at almost no engineering cost by setting the per-token advantage directly to the log-probability difference between teacher and student. The GLM-5 report gives the cleanest formulation, stating explicitly that this replaces the advantage term in the GRPO loss:

$$ \hat{A}_{i,t} = \mathrm{sg}\left[\log \frac{\pi_{\theta_{\mathrm{teacher}}}(y_{i,t} \mid x, y_{i,\lt t})}{\pi_{\theta}(y_{i,t} \mid x, y_{i,\lt t})}\right] $$

Placing this alongside the GRPO advantage from Section 2 reveals a clear line of development. PPO requires a value network of comparable size to the policy in order to estimate the advantage; GRPO removes the value network in favour of group normalisation; with OPD, even the group normalisation becomes unnecessary.

The GLM-5 report is direct on this point: during training it sets the GRPO group size to 1 and the batch size to 1024 to increase throughput, and this is feasible precisely because the advantage can now be computed directly from the gap to the teacher, without maintaining a group of samples per prompt to estimate it. Reducing the group size from the usual 8–16 down to 1 multiplies throughput several times over. PPO → GRPO → OPD is essentially a trajectory along which advantage estimation degenerates until no estimation is required at all: credit assignment, the hardest problem in RL, is circumvented by the teacher’s per-token scoring.

A second property that lowers cost is that the teacher performs prefill only and never decodes. The student’s complete response is already available, so when the teacher computes $\log \pi_T(y_t \mid y_{\lt t})$ all positions can be processed in parallel in a single forward pass.

In addition, OPD typically takes a discount factor of zero, optimising only the current token at each position without multi-step rollout. This means reward computation need not wait for a trajectory to finish sampling: training can use shorter or even partial rollouts, and there is no need to extend the context merely to await a terminal reward.

Vocabulary granularity: sampled-token, top-k, or full vocabulary

The KL is defined over the entire vocabulary, but an implementation need not evaluate all of it. Three options exist, and they are identical in teacher forward cost — a single prefill in every case — differing only at the loss computation.

In the table below, $B$ is the batch size, $T$ the sequence length, $V$ the vocabulary size, and $k$ the number of tokens retained by top-k:

Variant	Loss complexity	Description
Sampled-token	$O(BT)$	Uses only the log-probability of the single token the student actually sampled
Top-k	$O(BTk)$	Sums over the top-k tokens
Full-vocabulary	$O(BTV)$	Sums over the entire vocabulary at every position

The gap between the two extremes is the vocabulary size itself:

$$ \frac{O(BTV)}{O(BT)} = V $$

With Qwen3’s vocabulary of $V = 151{,}669$, this ratio is roughly $1.5 \times 10^5$, or five orders of magnitude. In concrete terms, at $B = 8$ and $T = 8192$ the full-vocabulary variant must materialise $8 \times 8192 \times 151{,}669 \approx 9.9 \times 10^9$ logit entries, about 20 GB in bf16, and one copy is needed for the teacher and one for the student; the sampled-token variant keeps a single log-probability per position, $8 \times 8192 = 65{,}536$ numbers in total, under 1 MB.

Given a cost gap this large, is a single token sufficient? That depends on two things: whether it is statistically sound, and whether it discards useful signal.

From a statistical standpoint. Monte Carlo estimation requires that samples be drawn from the distribution under which the expectation is taken. The reverse KL expectation is taken under $\pi_\theta$, and the trajectory was sampled by the student in the first place, so estimating it with the token the student actually generated is unbiased, requiring only one gathered log-probability per position. This also settles the engineering question left open in the previous section: the forward KL expectation is taken under $\pi_T$, so an unbiased estimate would require teacher sampling, yet in OPD the teacher only prefills and never decodes, so no teacher-sampled tokens exist. Using sampled-token therefore forces the choice of reverse KL; using forward KL forces one to abandon sampling and evaluate the sum explicitly over the top-k or the full vocabulary.

From a signal standpoint. Rethinking On-Policy Distillation, from Tsinghua’s thunlp group, provides mechanistic evidence: successful OPD manifests as a progressive alignment of high-probability tokens between student and teacher at student-visited states (the overlap rises from 72% to 91%), and this shared token set concentrates 97%–99% of the probability mass of both distributions. They further verify that supervising only the overlapping tokens matches the full top-k. Since the gradient signal is already concentrated on a handful of tokens, the additional vocabulary contributes little.

Taken together, sampled-token has become the industrial default in single-teacher settings. The one exception arises with multiple teachers, discussed in Section 6.2 on DeepSeek-V4.

6. How the Five Technical Reports Use It

The differences across the five labs reduce to two dimensions: where the teacher comes from, and where OPD sits in the post-training pipeline. The following sections proceed along these two axes.

6.1 Qwen3: strong-to-weak distillation for lightweight models

The post-training of the Qwen3 flagship models comprises four stages: long-CoT cold-start SFT, reasoning RL, thinking mode fusion (further SFT that merges the think and no-think modes into a single model), and general RL.

The position of OPD here is unusual: rather than being inserted at one of the four stages, it replaces the pipeline in its entirety. The lightweight models from 0.6B to 14B no longer repeat the four-stage procedure individually, and are instead distilled from Qwen3-32B or Qwen3-235B-A22B, which have already completed it — that is, strong-to-weak distillation. The report’s stated rationale is that distilling the teacher’s output logits directly into lightweight students both improves performance and preserves fine-grained control over their reasoning process.

The distillation itself proceeds in two phases, off-policy first and on-policy second.

The first phase performs response distillation on teacher outputs collected in both think and no-think modes, so that the student first acquires basic reasoning ability and learns to switch between the two modes;
Only in the second phase does training become on-policy: prompts are sampled, the student generates its own responses, and its logits are then aligned with those of the teacher (Qwen3-32B or Qwen3-235B-A22B) by minimising the KL divergence.

The reason for this design is that if the output styles of student and teacher differ too much, most of the token-level KL will be stylistic noise rather than knowledge. An off-policy round is therefore needed first to move the student’s distribution near the teacher’s, before the on-policy phase begins.

Table 21 of the report gives the resulting comparison. All three rows start from the same off-policy-distilled 8B checkpoint, differing only in whether RL or on-policy distillation follows; pass@64 is given in parentheses:

Method	AIME'24	AIME'25	MATH500	LiveCodeBench	GPQA-Diamond	GPU Hours
Off-policy distillation	55.0 (90.0)	42.8 (83.3)	92.4	42.0	55.6	–
+ Reinforcement learning	67.6 (90.0)	55.5 (83.3)	94.8	52.9	61.3	17,920
+ On-policy distillation	74.4 (93.3)	65.5 (86.7)	97.0	60.3	63.3	1,800

On AIME'24 the gain is 6.8 points at one tenth of the GPU hours. More telling are the pass@64 figures in parentheses: RL leaves the 64-sample ceiling entirely unchanged (90.0 → 90.0), whereas on-policy distillation pushes it to 93.3. The report attributes this to the fact that distilling from the teacher’s logits expands the student’s exploration space and raises its reasoning potential, with the pass@64 improvement as the evidence; reinforcement learning, by contrast, produced no pass@64 gain at all.

6.2 DeepSeek-V4: replacing mix RL with multi-teacher OPD

Compared with Qwen3, DeepSeek-V4 has more than ten teachers, each strong in a single domain, and the problem OPD solves is capability merging. Accordingly it is no longer a side branch for lightweight models but sits on the critical path: the report states that post-training retains the overall structure of V3.2 and that the one key methodological substitution occurs at this step, with the mixed reinforcement learning stage replaced entirely by on-policy distillation.

The resulting procedure has two stages. In the first, a specialist is trained independently for each domain — mathematics, code, agentic tasks, instruction following and so on — each obtained by a separate SFT-plus-GRPO run. In the second, multi-teacher OPD merges the ten-plus specialists back into a single unified model.

OPD here therefore takes on the final merging step of the pipeline; what it replaces is not the algorithm inside any single RL stage but the entire approach to optimising several objectives at once. Its objective is

$$ \mathcal{L}_{\mathrm{OPD}}(\theta) = \sum_{i=1}^{N} w_i \cdot D_{\mathrm{KL}}\big(\pi_\theta \,\|\, \pi_{E_i}\big) $$

where $w_i$ is the weight of each expert. The report’s rationale is that this mechanism consolidates knowledge dispersed across expert weights into a unified parameter space through alignment at the logits level, thereby avoiding the performance degradation commonly seen with parameter merging and mix RL.

Another notable engineering decision is that DeepSeek-V4 adopts full-vocabulary KL rather than the industry-standard token-level estimate. The report is explicitly critical of the latter: existing work typically reduces the full-vocabulary KL to a token-level estimate at each position and reuses the RL framework with $\mathrm{sg}[\log \pi_E / \pi_\theta]$ as a per-token advantage; while economical, this incurs high variance in the gradient estimate and frequently destabilises training.

The formulation being criticised is exactly the one given in Section 5. The trade-off is determined by the difference noted above: with a single teacher the variance of the Monte Carlo estimate is tolerable, whereas V4 must align more than ten teachers at the same token position, which amplifies the variance substantially.

The cost falls on memory. With a vocabulary above 100,000, materialising logits for all teachers is infeasible, so V4 caches only the teachers’ final-layer hidden states and reconstructs the logits through the prediction head during training; samples are sorted and dispatched by teacher index so that each teacher head is loaded once per mini-batch and at most one resides in memory at a time; the exact KL is computed by a dedicated TileLang kernel.

6.3 MiMo-V2-Flash: MOPD combined with outcome reward

Xiaomi’s MiMo-V2-Flash names the method MOPD (Multi-Teacher On-Policy Distillation) and positions it as a post-training paradigm rather than a training trick. Its pipeline has three stages: Stage 1 establishes instruction-following ability through general SFT; Stage 2 trains per-domain teachers with RL and SFT (including both non-agentic and agentic RL); Stage 3 uses MOPD to merge those teachers back into one student.

The motivation the report gives for this design is capability imbalance, which it calls the see-saw effect: when several capabilities are improved in sequence on the same model, gains in one are often accompanied by regressions in another. Training each domain separately and merging at the end is precisely how this effect is circumvented, and MOPD, at the third stage, performs the merge. Its reverse KL loss is written as

$$ \mathcal{L}_{\text{reverse-KL}}(\theta) = -\,\mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi(\cdot \mid x)} \log \frac{\pi_{\mathrm{domain}}(y_t \mid x, y_{\lt t})}{\pi_\theta(y_t \mid x, y_{\lt t})} $$

MiMo’s overall structure closely resembles V4’s — domain-wise teachers first, multi-teacher OPD to merge afterwards — but differs in two substantive respects. First, V4’s objective contains the KL alone, whereas MiMo adds the distillation advantage to an outcome reward, $\hat{A} = \hat{A}^{\mathrm{MOPD}} + \hat{A}^{\mathrm{ORM}}$; OPD does not displace RL, and both signals coexist. Second, MiMo permits the student itself to enter the teacher set, so in domains lacking a stronger teacher MOPD effectively degenerates into self-distillation — visible directly in Table 7 below.

Table 7 of the report reports the student before distillation, the best teacher, and the student after distillation, from which the boundary of MOPD’s benefit can be read off directly:

Benchmark	Student Before MOPD	Best Teacher	Student After MOPD	Δ(Student − Teacher)
AIME 2025	89.3	93.9 (RL)	94.1	+0.2
HMMT Feb. 2025	76.9	82.6 (RL)	84.4	+1.8
LiveCodeBench	77.5	82.6 (RL)	83.2	+0.6
GPQA-Diamond	84.9	84.9 (Self)	84.3	−0.6
Arena-Hard (Hard Prompt)	50.0	50.0 (Self)	54.1	+4.1
Arena-Hard (Creative Writing)	90.1	90.1 (Self)	86.2	−3.9
SWE-Bench Verified	67.8	74.2 (RL)	73.4	−0.8
Tau2-Bench	75.9	79.6 (RL)	80.3	+0.7
BrowseComp	42.5	51.7 (SFT)	45.4	−6.3

Three observations follow from this table.

First, in domains with a strong teacher (mathematics, code, agentic tasks) the student essentially matches or slightly exceeds the best teacher, with Δ within ±1.
Second, in the rows marked Self, the best-teacher figure is identical to the student before distillation, meaning no teacher on that benchmark is stronger than the student itself; any gain must then come from the added ORM term, and MOPD serves only to preserve performance, or may even lose points.
Third, BrowseComp drops by 6.3 points, and its teacher happens to be an SFT model — suggesting that when the behavioural patterns of teacher and student diverge too far, token-level alignment may fail to transfer the capability. This table in effect provides direct evidence that the teacher constitutes a capability ceiling.

6.4 Kimi K3: nine specialists partitioned by domain × reasoning effort

Moonshot’s Kimi K3 follows the same route as the previous two but adds one more dimension to the partition. Its post-training is likewise three-staged: SFT cold start to establish basic agentic ability, RL to train domain specialists, and MOPD to merge.

The difference lies in how the second stage is partitioned. K3 divides RL into three broad domains — general, general agents and coding agents — and trains low, high and max reasoning-effort variants within each, giving nine specialists in total. The effort levels are obtained by controlling the token budget during RL: an initial budget is estimated for each problem, trajectories exceeding a threshold of $\tau$ times that budget receive a reward of $-1$, and annealing $\tau$ yields the max, high and low levels in turn.

K3’s MOPD therefore merges not only capabilities but also thinking lengths: the behaviour of nine specialists is compressed into a single model that can switch effort levels on demand at inference. This effectively brings budget distillation of the long-to-short variety under the same mechanism.

In implementation, K3 does not go full-vocabulary as V4 does, but retains the token-level formulation with clipping applied:

$$ r^{d}_{\mathrm{opd}}(y_t \mid e, x, y_{\lt t}) = \mathrm{clip}\left(\mathrm{sg}\left(\log \frac{\pi^{(d,e)}_{\mathrm{teacher}}(y_t \mid x, y_{\lt t})}{\pi_\theta(y_t \mid e, x, y_{\lt t})}\right), -R_{\max}, R_{\max}\right) $$

where $R_{\max}$ is the clipping threshold constraining extreme advantage signals. The report explains that, defined this way, OPD is simply an ordinary dense reward that plugs into an existing RL framework, allowing infrastructure optimisations for long-horizon tasks such as partial rollout to be reused. Notably, they also tried a finer-grained top-$k$ distillation objective but observed no clear advantage in either convergence speed or final performance — consistent with the conclusion of Section 5.

For context, on K2 and K2.5 Moonshot still generated candidate responses with K1.5 and a number of internal domain specialists, filtered them, and trained with SFT — sequence-level off-policy distillation. Only with K3 did the recipe shift to student sampling with per-token scoring by nine specialists.

6.5 GLM-5: cross-stage distillation with historical checkpoints as teachers

GLM-5’s post-training is a long sequential pipeline: multi-task SFT (introducing the interleaved thinking mode) → reasoning RL → agentic RL → general RL. The three RL stages optimise different objectives, which brings a known problem: later stages erode capabilities acquired in earlier ones.

OPD is therefore placed at the very end of the chain as a dedicated repair stage. The report describes it as follows: optimising different objectives in sequence within a multi-stage RL pipeline causes cumulative degradation of existing capabilities; to mitigate this, the final stage performs on-policy cross-stage distillation, taking the final checkpoint of each preceding training stage as a teacher, with training prompts sampled from the corresponding teacher’s RL training set and mixed in proportion.

This is the most fundamental difference from the other four. Elsewhere the teacher is an external model stronger than the student, or at least stronger in some domain; GLM-5’s teacher is a snapshot of the student itself from an earlier stage and is no stronger than it. OPD here therefore does not transfer new capability; it realigns a distribution that has drifted back toward that checkpoint, recovering the capabilities overwritten by subsequent stages.

Two findings support why this works.

First, the gains from RL are highly localised: it fine-tunes only a small subnetwork of the model (Reinforcement Learning Finetunes Small Subnetworks in Large Language Models), which makes strengthening one domain liable to damage another — exactly the cumulative degradation GLM-5 describes.
Second, the source of the data matters decisively when recovering capability: Retaining by Doing shows that training on the model’s own on-policy data substantially mitigates forgetting, and RL’s Razor explains from a KL perspective why online methods forget less than offline SFT.

Together these imply that generating data with the old checkpoint and running another round of SFT would perform markedly worse — capability recovery must itself be on-policy.

6.6 Thinking Machines Lab: self-distillation for continual learning

Thinking Machines Lab generalises this usage to the continual-learning setting in On-Policy Distillation (the section on distillation for personalization). The setting is a Qwen3-8B that has already completed post-training and into which internal enterprise document knowledge is then injected; OPD follows the domain injection and performs capability recovery.

The problem arises in mid-training: internal QA rises from 18% to 43%, but IF-eval falls from 85% to 45%. The blog notes that this degradation is unrelated to the data mixture — IF-eval declines regardless of how the ratio is tuned — and that constraining parameter updates with LoRA merely “learns less while still forgetting”.

The recovery procedure is a round of OPD with the pre-mid-training Qwen3-8B itself as the teacher, using prompts taken from Tulu3 that are entirely unrelated to the internal documents, for the sole purpose of restoring instruction following:

Model	Internal QA (knowledge)	IF-eval (conversation)
Qwen3-8B	18%	85%
+ midtrain (100% documents)	43%	45%
+ midtrain (70% documents)	36%	79%
+ midtrain (70%) + distill	41%	83%

The last row is the notable one: while IF-eval recovers from 79% to 83%, internal QA rises from 36% to 41% — what the blog calls positive transfer between conversational ability and knowledge. This is the same mechanism as GLM-5’s cross-stage distillation: in both cases the teacher is a historical snapshot of the model itself, and OPD repairs the capability loss caused by subsequent training.

The same blog also reports a direct controlled experiment (dense supervision greatly improves compute efficiency): starting from Qwen3-8B-Base, RL on DeepMath yields a policy, and OPD then distils that policy back into the same base model — teacher and student share an architecture and a parameter count, and the only difference is the mode of learning.

The result is that OPD requires roughly one seventh to one tenth of the gradient steps to acquire that RL policy (the AIME score is recovered within 10 steps, against RL’s 70), which translates into a 50–100× improvement in compute efficiency.

An even more extreme setting is training on a single prompt: a limit problem is drawn at random from the dataset, 256 rollouts are sampled per step over 20 consecutive steps for 5,120 scored sequences in total, and the teacher’s AIME'24 performance is approximately reproduced. Under RL such training would almost certainly degenerate into memorising the answer; OPD, by contrast, fits the teacher’s full conditional distribution rather than any particular answer, and can therefore train repeatedly on the same set of prompts.

The blog makes the reason the title of a section — RL searches in the space of semantic strategies — which corroborates the judgement of Section 2: the cost of RL belongs mostly to search rather than learning. What is expensive is discovering, by trial in the space of strategies, which mode of reasoning earns the reward; once that strategy has been found, transferring it to another model by distribution matching is cheap.

7. Where the Efficiency of OPD Comes From

All five reports observe an efficiency gain, and none of them explains its origin. The most immediate explanation, that “dense supervision carries more information”, does not hold: SFT’s supervision is equally dense, yet it is not ten times more efficient than RL. Two works from 2026 answer the question, one at the level of phenomenology and one at the level of parameter dynamics.

Tsinghua’s Rethinking On-Policy Distillation answers “when does it fail”. It identifies two conditions for OPD to hold: the thinking patterns of student and teacher must be mutually compatible; and even with matching patterns and a higher score, the teacher must possess genuinely new capability the student has not seen. The weight of the latter is shown by a reverse control — running OPD between a 1.5B and a 7B model of the same family yields nearly identical final scores whether the direction is 7B → 1.5B or the reverse. Same-family models of different sizes are, in other words, nearly indistinguishable under the metric of token-level KL: superiority on benchmarks does not imply there is anything new to teach at the level of distributions. Two remedies follow: when the thinking patterns are incompatible, run a round of off-policy cold start first (as Qwen3 does); when the teacher has nothing new to teach, select only the prompts the teacher answers well and the student answers poorly.
USTC and Tencent’s Learning to Foresee answers “why is it fast”. Their answer is foresight — OPD locks onto the direction toward the final solution very early in training, after which training mainly accumulates magnitude along that established direction. The evidence sits at two levels:
- At the module level, updates concentrate on the mid-layer MLPs that actually carry reasoning ability, whereas RL distributes a large share of the update norm over the embeddings and the first and last layers, which contribute little;
- At the direction level, an SVD of the parameter increments shows that OPD’s energy is markedly more low-rank and that its dominant direction aligns closely with the final one very early, while RL stabilises much later. The most direct ablation takes a checkpoint at 10% of training progress and rescales the update norm per module to its final magnitude while leaving the direction untouched, recovering roughly 80% of the final reasoning performance.

Together these yield a sharper judgement: OPD is fast not because the signal is dense, but because the destination — the teacher’s distribution — is given in advance, so the direction need not be found by trial. And this is precisely what foreshadows its limits.

8. The Limits of OPD

The teacher constitutes a capability ceiling. The global optimum of OPD is a student distribution identical to the teacher’s, at which point both loss and gradient vanish. As long as the objective contains only the distillation term, the student cannot systematically surpass the teacher along any meaningful dimension. The Δ column in MiMo’s table, clustered near zero with several negative entries, is a direct manifestation of this property. Surpassing the teacher still requires RL to acquire new information from the environment — the ground-truth answer to a maths problem, the result of running code, the formal verification of a theorem — signals that do not pass through the teacher.
A genuine capability gap must exist. If the teacher is merely a larger sibling with a slightly higher score, the two may differ almost not at all at the scale of token-level distributions. Superiority on benchmarks is not necessarily reflected in per-token conditional probabilities. This is also why Qwen3 runs a round of off-policy cold start first: only once the two behavioural patterns are aligned does the KL carry knowledge rather than stylistic noise.
The dense signal is not free. The early directional lock-in of the previous section cuts both ways: the flip side of locking onto a direction is constrained exploration (though this is also, as discussed earlier, one of its advantages over SFT). Rethinking OPD accordingly raises the question of whether OPD can scale to long-horizon distillation. Following a teacher token by token along a long reasoning chain tends to transfer its local token habits rather than its reasoning strategy.

Looking back over the technical reports — Qwen3 uses it to train lightweight models, MiMo stacks it with an ORM reward, GLM-5 places it after three rounds of RL, DeepSeek-V4 and Kimi K3 place it after specialist RL — not one of them replaces RL with OPD outright. OPD always performs capability transfer; capability creation is still done by RL.

What OPD saves is samples, not GPU compute. Every rollout the student runs requires an accompanying teacher forward pass, so training must sustain inference resources for both models simultaneously. Qwen3’s figure of 1,800 GPU hours presupposes that a 32B or 235B teacher is already in hand; for teams with limited compute, the result is not necessarily reproducible.

9. Conclusion

OPD is not a new algorithm: on-policy sampling descends from the DAGGER line of imitation learning, training on student-sampled sequences was consolidated into a framework by Generalized Knowledge Distillation, reverse KL was established by MiniLLM, which also used the teacher-to-student log-probability ratio as a per-step reward, and the engineering scaffold is borrowed wholesale from PPO/GRPO with the advantage term as the only modification. Its real contribution is to have assembled these existing components into a form that is stable and cheap at industrial scale, making capability transfer, for the first time, a step that can be planned independently within post-training. The five labs had different motivations — lowering the training cost of lightweight models, merging domain specialists, repairing the degradation caused by sequential RL — but faced the same problem: as the cost of RL keeps rising and capability ends up dispersed across several specialist models, how does one move capability already paid for in compute into a single model at lower cost?

References

Thinking Machines Lab, On-Policy Distillation, 2025
Qwen Team, Qwen3 Technical Report, 2025
DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, 2026
LLM-Core Xiaomi, MiMo-V2-Flash Technical Report, 2026
Moonshot AI, Kimi K3: Open Frontier Intelligence, 2026
Zhipu AI, GLM-5: from Vibe Coding to Agentic Engineering, 2026
Li et al., Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, 2026
Cai et al., Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation, 2026 (EffOPD)
Song & Zheng, A Survey of On-Policy Distillation for Large Language Models, 2026
Ye et al., Black-Box On-Policy Distillation of Large Language Models (GAD), 2025
Zhao et al., Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, 2026
Agarwal et al., On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024
Gu et al., MiniLLM: Knowledge Distillation of Large Language Models, 2023
Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024
Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010
Gudibande et al., The False Promise of Imitating Proprietary LLMs, 2023
Lightman et al., Let’s Verify Step by Step, 2023
旷野, Zhihu answer, 2026
燕雄飞的一天, Zhihu article, 2026
JMXGODLZ, Zhihu article, 2026
吴宇斌, Zhihu article, 2026
JustBeClaw, Zhihu article, 2026