Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

Siyan Zhao (et. al) · Paper · June 19, 2026

Core Idea

This paper is really good. The core contribution is an approach to train LLMs to ‘think’ correctly (beyond stuff like SFT on CoT) using the same LLM, not a separate teacher. The teacher here is the same model but with a provided reference solution. Going through the example in Figure 2 of the paper is helpful (the table is horizontally scrollable):

Student context	Teacher context	Student next-token dist.	Teacher next-token dist.
Problem: find derivative of f(x) = 3x^2 + 2x - 5 at x = 2. Answer: f’(x) =	Problem: find derivative of f(x) = 3x^2 + 2x - 5 at x = 2. Reference solution: f'(x)=6x+2, so f'(2)=14. After understanding the reference solution, solve the problem: Answer: f’(x) =	p_s(next \| problem, student_prefix)	p_t(next \| problem, reference_solution, student_prefix)

Importantly, even though the same model is being used, since the teacher is also conditioned on the reference solution, its next-token distribution logits are different from the student.

There is some subtlety in how this is actually implemented—it’s pretty much

the student generates a complete answer, \(\hat{y}\), by itself.
the model is re-run on the student context + \(\hat{y}\) with grad on to get the logits.
the model is re-run on the teacher context + \(\hat{y}\) with grad off to get the teacher logits.
compare the two logit distributions at each point in the \(\hat{y}\) token generation, use some KL loss.
backprop through student only.

Strengths

“on policy” in that it directly steers based on the path the student took, not some concrete “expert” path.
only one model needed.
for problems where generating privileged information is not super hard (maybe math? since you can generate and verify easily with LLM + Lean, for example), this approach feels really good.

Miscellaneous

Apparently clipping KL-divergence per token is practically very important because

stylistic tokens can exhibit higher KL divergence than math-related tokens, causing them to dominate the training signal.

I’d suggest just reading this part of the paper (see sec 4.3.3, figure 4, and table 5).