The Sequence Knowledge #890: A Brief History of Model Distillation

The papers and techniques that laid out the ground work to evolve distillation to this level.

The story most people tell about knowledge distillation starts in 2015, with Geoffrey Hinton, Oriol Vinyals, and Jeff Dean introducing a clever softmax temperature trick and a phrase — “dark knowledge” — that immediately lodged itself in the field’s vocabulary. It is a good story. It is also incomplete by almost a decade.

The real history is quieter, more pragmatic, and worth recovering, because the conceptual moves the field made between 2006 and 2015 still define how we think about distillation today. The vocabulary changed. The diagrams changed. The underlying question — what exactly is being transferred from a teacher to a student? — has not. To understand the modern landscape of on-policy distillation, reasoning distillation, and cross-architecture transfer, it helps to walk back through the three papers that built the foundation, and to notice that each one solved a different problem on its way to inventing the same idea.

2006: Compression as Mimicry