In late January 2025, Nvidia shed roughly $589 billion in market value in a single trading day. It was reportedly the largest one-day loss any company had ever posted. The trigger wasn’t a bad earnings call or a chip recall. It was a small Chinese lab called DeepSeek, which had just released a reasoning model that matched the big American systems at a reported training cost of about $5.6 million. That number should have been impossible. Frontier labs were spending that much on a rounding error.
So how did they do it? The leading suspect was a technique with a quiet, almost academic name: distillation.
If you build, buy, or just use AI, distillation in AI is one of the few concepts worth understanding right now. It sits underneath the chatbot on your phone, the voice assistant that answers in half a second, and a legal fight that has pulled in OpenAI, Anthropic, and the U.S. Congress. Here’s what it actually is, how it works, where it shines, where it breaks, and why the whole industry is arguing about it.
What Is Distillation in AI?
Knowledge distillation is a technique where a small “student” model learns to copy the behavior of a large “teacher” model. Instead of training only on right-or-wrong answers, the student learns from the teacher’s full set of probability outputs, which carry far richer information than a simple label. The payoff is a smaller, faster, cheaper model that keeps most of the teacher’s accuracy. Geoffrey Hinton and colleagues formalized the method in a 2015 paper, though the core idea traces back to work by Cristian Bucila and Rich Caruana in 2006.
That’s the whole trick in one paragraph. Everything else is detail. But the detail is where it gets interesting, because it explains why a smaller model trained this way can outperform the same small model trained the normal way.
Why Distillation Suddenly Matters
Big models are expensive in a way that’s easy to underestimate. Training a frontier language model can burn thousands of GPU hours and millions of dollars. Running one is the part nobody warns you about. Every single query hits a system with hundreds of billions of parameters, which means real money and real electricity for every response. Multiply that by millions of users, and inference costs can dwarf training costs over a model’s lifetime.
This is the wall every company hits eventually. You’ve got a model that’s brilliant and completely impractical to deploy. It won’t fit on a phone. It costs too much to serve at scale. It’s too slow for anything real time.
Distillation is one of the cleanest ways around that wall. And the market has noticed. The knowledge distillation platform market reached about $1.42 billion in 2024 and is projected to grow at a compound annual rate near 27.6% through 2033, according to Dataintelo, though estimates vary widely across research firms. The broader model compression market, which folds in pruning and quantization alongside distillation, is on a similar steep curve. McKinsey’s research found that 72% of organizations had adopted AI in at least one business function by early 2024, and most of them now face the same deployment math: keep the capability, kill the cost.
Then came the DeepSeek moment. Suddenly distillation wasn’t a niche engineering choice. It was a geopolitical flashpoint, a stock-market event, and a question about who gets to own the intelligence baked into a model. More on that fight later, because it changes how you should think about the whole technique.
How Distillation Actually Works (and Why It’s Clever)
Here’s the part most guides skip, and it’s the part that makes distillation feel a little bit like magic the first time it clicks.
Imagine you’re training a model to recognize animals. When it looks at a photo of a wolf, a normal training label says exactly one thing: “wolf,” full stop. That’s a hard label. It’s a single correct answer with zero nuance.
Now run that same photo through a well-trained teacher model. The teacher doesn’t just say “wolf.” It says something like: 90% wolf, 9% dog, 1% cat. Look at what just happened. The teacher quietly revealed that a wolf looks a lot like a dog and almost nothing like a cat. That relationship between the classes is information the hard label completely threw away.
Hinton called this hidden signal “dark knowledge,” which is a slightly creepy name for a genuinely useful idea. The teacher’s soft probabilities encode how it thinks, not just what it concludes. A student trained on those soft targets learns the shape of the problem, not just the answer key.
To make that dark knowledge easier to see, distillation uses a knob called temperature. Cranking the temperature up in the softmax function smooths the probabilities out, so the smaller relationships (the 9% dog, the 1% cat) become more visible instead of getting crushed to near zero. The student trains on these softened outputs, usually by minimizing something called Kullback-Leibler divergence, which is a fancy way of saying “make the student’s guesses match the teacher’s guesses as closely as possible.” After training, you turn the temperature back to normal and deploy.
Does it actually help? The original 2015 experiments say yes, and the numbers are tidy. On the classic MNIST digit dataset, a large teacher network made 67 test errors. A small network trained on its own made 146 errors. The same small network, trained instead to match the teacher’s soft targets, dropped to 74 errors. Same size. Same data. Roughly half the mistakes, purely from learning how the teacher thought. For a deeper walk through the original math, Hinton, Vinyals, and Dean’s paper Distilling the Knowledge in a Neural Network is still the clearest source.
That’s the counterintuitive heart of it. Distillation isn’t really “compression” in the way zipping a file is. You’re not squeezing the same bits into a smaller container. You’re transferring a way of seeing the world.
The Main Types of Knowledge Distillation
Not all distillation looks the same. Researchers generally sort it into three families based on what knowledge gets transferred, a taxonomy popularized by Jianping Gou and colleagues in a widely cited 2021 survey and echoed in IBM’s technical overview.
Response-based distillation is the classic version, the one from the wolf example. The student mimics the teacher’s final output layer, its logits or probabilities. It’s simple, it works across almost any task, and it’s still the most common approach in production.
Feature-based distillation goes deeper. Instead of copying only the final answer, the student learns to match the teacher’s internal representations, the activations in its hidden layers. Think of a bird classifier: early layers spot “there’s an animal,” middle layers narrow to “it’s a bird,” and late layers catch the fine differences between two similar species. Feature-based methods teach the student to build those same internal stepping stones.
Relation-based distillation is the most abstract. It captures the relationships between different samples or different layers, the structural patterns rather than any single output. Useful when the connections between data points carry as much meaning as the points themselves.
There’s a second axis too, based on when distillation happens. Offline distillation uses a fixed, pre-trained teacher (the usual setup). Online distillation trains a group of models together so they learn from each other. Self-distillation is the weird and elegant one, where a model teaches itself, transferring knowledge from its deeper layers back to its shallower ones with no separate teacher at all.
And then there’s the technique that defined 2025.
Reasoning distillation. Instead of just copying answers, newer methods copy the teacher’s step-by-step reasoning traces, the chain of thought it uses to work through a problem. This is exactly what DeepSeek did with its R1 model. According to its public model card, DeepSeek fine-tuned smaller Qwen and Llama models on roughly 800,000 reasoning samples generated by R1. The result: compact models that inherited genuine reasoning ability, with the 32-billion-parameter version reportedly outbenchmarking a comparable OpenAI mini model on several tests. Reasoning that used to require a giant model now runs on hardware you can rent cheaply. That shift is a big part of why the whole field got so loud so fast.
How to Distill a Model: A Practical Framework
Say you actually want to do this. The workflow is more approachable than the theory suggests. Here’s the basic path.
Step 1: Pick your teacher and confirm you’re allowed to use it. This is not a throwaway step in 2025 and 2026. If the teacher is an open-weight model with a permissive license (many Llama, Qwen, and Mistral variants qualify), you’re generally fine. If it’s a proprietary API model, read the terms of service first. We’ll get to why that matters shortly.
Step 2: Choose your student architecture. Smaller than the teacher, obviously, but sized to your real constraint. Deploying to phones? Target something in the tens-of-millions to low-billions of parameters. The classic example is BERT-Tiny learning from a full-size BERT teacher.
Step 3: Build a transfer set. Feed data through the teacher and collect its soft outputs. For language models, this often means generating a large batch of teacher responses (DeepSeek used hundreds of thousands). Quality here beats raw quantity. Garbage prompts produce a garbage student.
Step 4: Train with a blended loss. The student learns from two signals at once: the teacher’s soft targets (the dark knowledge) and the true hard labels where you have them. You weight the two, tune the temperature, and let it run. Getting this balance right is more art than formula, and it’s where most of the tinkering happens.
Step 5: Evaluate honestly on your actual task. Not just aggregate benchmarks. Test the edge cases and the failure modes that matter for your use. A student that scores 97% on average can still fall apart on the exact 3% you care about.
Frameworks like Hugging Face Transformers, PyTorch, and various open toolkits handle most of the plumbing. The hard parts are strategic: which teacher, which data, which trade-offs.
Distillation vs Pruning vs Quantization: Which Should You Use?
Distillation isn’t the only way to shrink a model, and picking the wrong tool wastes time. Here’s the honest comparison.
| Technique | What it does | Best for | Main trade-off |
|---|---|---|---|
| Distillation | Trains a new small model to mimic a big one | Keeping accuracy while changing architecture; transferring reasoning | Needs a good teacher and training compute up front |
| Quantization | Lowers numerical precision (e.g. 32-bit to 8-bit) | Fast memory and speed wins with minimal effort | Precision loss can hurt sensitive tasks |
| Pruning | Removes redundant weights or neurons | Slimming an existing trained network | Aggressive pruning degrades performance |
The key mental model: quantization and pruning modify the model you already have. Distillation builds a fresh, smaller model from scratch using the big one as a guide. That’s why distillation can jump to a completely different architecture while the others are stuck with the original shape. In practice, teams often stack them, distilling first, then quantizing the student for a final squeeze.
Now the part almost no glossary article will tell you: when not to distill.
Skip distillation if you don’t have a strong teacher. The student can only inherit what the teacher knows, and it usually caps out somewhere below the teacher’s ceiling. Skip it if your model is already small and simple, since there’s nothing meaningful to compress. And watch out for a real failure mode with generative models: a student can overfit to its teacher’s outputs and start producing repetitive or oddly narrow responses, a kind of mode collapse. Distillation also faithfully copies the teacher’s flaws. If the teacher hallucinates or carries a bias, congratulations, so will your student. It learns the bad habits along with the good ones.
The research is genuinely mixed on how far you can push generative distillation before quality frays, and anyone selling you a no-tradeoff version is overselling.
The Legal Gray Zone Nobody Warns You About
This is the section competitors leave out, and it might be the most important one.
Distillation only works cleanly when the teacher and the student are the same company’s models, or when the teacher’s license permits it. The moment you distill from someone else’s proprietary model, you walk into contested territory.
That’s the core of the OpenAI-DeepSeek dispute. OpenAI has alleged that DeepSeek queried its models at scale and used the outputs as training data, which OpenAI says violates its terms of service. According to reporting cited by Berkeley Law, OpenAI pointed to abnormal API traffic routed through obfuscated third-party services, and one analysis reportedly found DeepSeek’s responses were about 74% similar in writing style to ChatGPT’s. By February 2026, Anthropic had joined in, accusing three Chinese firms (DeepSeek, Moonshot AI, and MiniMax) of coordinated distillation campaigns against its models.
Here’s the twist that makes the whole thing so slippery. Distillation itself is completely normal. Anthropic openly acknowledged that AI companies routinely distill their own models to make cheaper versions. OpenAI, Anthropic, Mistral, and xAI all use the technique internally. What they object to is other people distilling their models without permission. The fight isn’t really about the method. It’s about consent, terms of service, and access.
And the legal ground is genuinely unsettled. Traditional copyright law protects creative works, but whether an AI model’s raw outputs even qualify for that protection is an open question, as a University of Michigan expert panel pointed out. It’s also worth noting a subtlety that got lost in the noise: DeepSeek’s published distilled models were transparently distilled from its own R1 into open Qwen and Llama models, which is entirely above board. The disputed part is the separate allegation about proprietary American models, which is far harder to prove when only the final model is public and the training data isn’t.
As one AI professor put it, the boundary between legitimate use and adversarial exploitation is often blurry. That blurriness is exactly why this matters to you. If you’re distilling for your own product, know your teacher’s license cold. “Everybody does it” is not a legal defense.
Real-World Distillation You’ve Already Used
Enough theory. Distillation is quietly running under a huge slice of the AI you touch daily.
DistilBERT is the textbook success story. Released by Hugging Face researchers led by Victor Sanh in 2019, it keeps roughly 97% of the original BERT model’s language understanding while being about 40% smaller and 60% faster. That combination is why distilled BERT variants ended up powering search features, chatbots, and text classification in production for years.
On-device voice and vision. When your phone transcribes speech instantly or blurs your background on a video call without a cloud round trip, you’re often watching a distilled model at work. Massive vision networks get compressed into mobile-friendly students (the nano and small variants of models like YOLO are a good example) so they run in real time without draining your battery.
Speculative decoding. Some modern LLM serving systems use a small distilled model to rapidly guess the next few words, then let the big model verify those guesses in parallel. It’s a clever speed hack, and the small guesser is typically a distilled model.
Edge and privacy-sensitive AI. In healthcare, finance, and manufacturing, distilled models let organizations run capable AI locally, on their own hardware, without shipping sensitive data to a remote server. That on-device angle is a major reason the technique keeps growing.
The through line: distillation is the bridge between AI research and AI you can actually ship. The giant models make the headlines. The distilled ones do the work.
Expert Perspective
The tension in this field is best captured by the people watching it up close. Erik Cambria, a professor of artificial intelligence at Nanyang Technological University in Singapore, has noted that because distillation is such a widely accepted practice, the line between legitimate use and adversarial exploitation is often blurry. That single observation explains why the OpenAI and Anthropic complaints are less about the technique and more about access and permission.
On the research side, recent peer-reviewed surveys reinforce that distillation has moved well beyond simple compression. A 2025 review in Frontiers in Robotics and AI places distillation alongside pruning, quantization, and low-rank decomposition as one of the four pillars of modern model compression, and points to knowledge transfer, not just size reduction, as its defining strength. The academic consensus and the industry reality have converged on the same point: the value is in moving intelligence around efficiently.
Frequently Asked Questions
What is distillation in AI in simple terms? It’s a way to make a small AI model smart by having it learn from a big one. The large “teacher” model shows the small “student” model not just the right answers but how confident it is across all options, and that extra nuance helps the student learn faster and better than it could alone.
Is knowledge distillation the same as model compression? Not exactly. Distillation is one type of model compression, alongside pruning and quantization. The difference is that distillation trains a brand-new smaller model from scratch, while pruning and quantization modify the model you already have. Distillation can switch to a totally different architecture, which the others can’t.
Did DeepSeek use distillation, and was it legal? DeepSeek openly distilled its own R1 model into smaller open models like Qwen and Llama, which is standard and permitted. Separately, OpenAI and Anthropic allege DeepSeek distilled their proprietary models against terms of service, which is disputed and legally unsettled. Distillation itself is legal; distilling someone else’s protected model without permission may not be.
Can a distilled student model beat its teacher? Sometimes, but rarely by much. A student generally caps out below the teacher’s performance because it can only learn what the teacher demonstrates. In specific narrow tasks a well-distilled student can match or slightly edge the teacher, especially after extra fine-tuning, but expecting a small model to broadly outperform a much larger one is unrealistic.
How much smaller can a distilled model get? It depends on the task, but meaningful compression is common. DistilBERT is about 40% smaller than BERT while keeping roughly 97% of its performance. Language model distillation can shrink billions of parameters down to a fraction of the original, though the accuracy trade-off grows as you get more aggressive.
What’s the difference between distillation and fine-tuning? Fine-tuning adjusts an existing model to specialize it on new data. Distillation trains a smaller model to imitate a larger one. They’re often combined: you distill to get a compact model, then fine-tune it for your specific task. Different goals, complementary steps.
Do I need a huge dataset to distill a model? Not necessarily a labeled one. Because the teacher generates the training signal (its soft outputs), you can use unlabeled data and let the teacher label it. Quality and coverage of the prompts matter more than sheer volume. A focused, representative transfer set often beats a massive noisy one.
The Bottom Line
Distillation in AI is deceptively simple and quietly enormous. Here’s what actually matters if you remember nothing else.
First, it’s knowledge transfer, not just shrinking. The student learns how the teacher thinks through soft targets and dark knowledge, which is why a distilled model can outperform the same model trained the ordinary way.
Second, it’s a strategic choice, not a default. Weigh it against quantization and pruning, and skip it entirely if you lack a strong teacher or you’re working with generative models where mode collapse and inherited flaws can bite.
Third, the legal side is real. The technique is legitimate and everyone uses it, but distilling from someone else’s proprietary model has become one of the defining disputes in AI. Know your teacher’s license before you start.
The models that grab headlines are the giant ones. But the AI actually running on your phone, answering your questions in half a second, and quietly saving companies a fortune in compute? A lot of that is distilled. It’s the unglamorous engineering that turns research into something real.
If you’re planning to distill a model for your own work, start small: pick an openly licensed teacher, build a clean transfer set, and measure results on your real task before you scale. And if you’ve distilled a model yourself, the trade-offs you hit in practice are worth sharing, because this field is still figuring out where the limits actually are.
