|
[Webinar] Can you prove AI is working? (Sponsored)
 |
AI is in your engineering workflow. While the token spend shows it, the throughput doesn’t. The human is very much still in the loop, and that’s a context problem.
Join live on Aug 19 (FREE) to learn:
The 4 metrics to measure where AI gains leak out before production.
The 8 stages of context maturity, the specific walls capping your metrics, and a free tool to pinpoint where your team is
Why more MCPs and bigger context windows aren’t enough, and what it takes to get real value from your agents.
The most capable AI models are also the most expensive to run. They need specialized hardware, they consume large amounts of memory, and they add cost and delay to every request they handle.
These traits make them hard to deploy in places where resources are limited, such as a mobile device or a service that handles heavy traffic and needs fast and low-cost responses.
There is also a second fact that sounds backward at first. A small model can sometimes match or beat a much larger model on a specific task, even when the small model learned everything it knows from the larger one. On an intuitive level, a model trained on another model’s output would seem to inherit a ceiling rather than break through it, yet the results are different. The method that makes this work is called knowledge distillation, and it has become a standard part of how production AI systems get built.
In this article, we will walk through the idea from the ground up. The main points we will cover are as follows:
What distillation is, and how it differs from compression.
Why learning from a model’s output can beat learning from raw labels.
The three main methods, and which one dominates?
What distilled models achieve in practice.
Where the method breaks down, and where it is heading next.
 |
Distillation
Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models:
The first is a large, capable model called the teacher.
The second is a smaller model called the student, which is trained to reproduce the teacher’s outputs.
Once training finishes, the student runs on its own, and the teacher steps out of the picture.
A common assumption is that the student is the teacher in compressed form. The reality, however, is different.
Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter.
Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher.
One operation shrinks an existing model. The other trains a fresh one. The payoff is practical, since a small student can run inside a single service or on a phone, respond in less time, and cost far less for each request, and in some cases, it can run on the device itself without sending data elsewhere.
 |
This method is now standard practice. For example, Google’s Gemma models are built using distillation during training, drawing on a larger model in the Gemini family. The two ideas also work together in sequence. A model is often distilled first to produce a smaller capable model, then quantized to shrink that model further for a specific device.
Keeping the distinction clear matters because it affects how we understand further concepts. A compressed model carries a copy of the original inside it. A distilled model is a separate thing that was trained to act like the original, which is exactly why it can sometimes behave in ways the original would not.
If the student only copies the teacher, why does copying work so well?
The answer is in what the teacher hands over.