Quantization vs distillation solves the same problem in different ways. Quantization keeps the same model and stores each weight in fewer bits. The parameter count never changes; only the bytes per weight do. Distillation, instead, trains a new, smaller model, called a student, to copy a larger teacher’s behavior. That student has fewer parameters, often with a different architecture. Post-training quantization needs no retraining at all. Distillation, though, always needs a teacher, training data, and compute. A 7 billion parameter model quantized to 4-bit is still a 7 billion parameter model. So it just costs less to store. Distilling that same model into a 1.5 billion parameter student produces a different, smaller model that had to be trained. Most production systems now use both: distill first, then quantize the resulting student.
Quantization vs distillation gets confused constantly. That confusion carries a real price, though. Both terms promise a smaller, cheaper large language model, or LLM. Many pages online use the two terms almost interchangeably. They are not the same thing. One shrinks the file on disk; the other builds an entirely new model.
This guide draws that line precisely, with real numbers throughout. You will see exactly how quantization changes bits per weight, and how distillation trains a smaller student model instead. A worked example then shows what a 7 billion parameter model actually costs in memory, at four different precisions. Check the numbers against your own as you read.
Quantization and distillation are not the only road to a smaller footprint. Retrieval-augmented generation and fine-tuning solve a related but different problem, covered in our guide to Retrieval-Augmented Generation vs Fine-Tuning. Once a shrinking strategy is chosen, though, the model still has to run somewhere.
Why Models Need Shrinking At All
An LLM starts life with billions of parameters. Each one takes real memory just to hold, before the model ever answers a question. More parameters mean more memory, and memory is never free.
Cost follows directly from that memory. So cloud GPU time is billed by the hour, for every request the model handles. A model that needs less memory fits on cheaper hardware, and serves more requests per machine.
Latency matters just as much as cost does. A chat product that takes too long to respond breaks the experience. Fewer bytes to move through memory generally means faster inference.
Then there is the hardware you actually own. A laptop GPU, or a phone, has a fixed memory budget. A model that will not fit simply cannot run there, no matter how good its answers are. Picking hardware for the job is its own question, covered in our GPU vs TPU vs NPU guide. These shrinking techniques apply to neural networks generally, a distinction our Machine Learning vs Deep Learning guide unpacks in more depth.
Quantization and distillation both shrink a model so it fits inside these limits. They do it in very different ways, and that difference is the whole point of this guide.
What Quantization Does
Quantization reduces the numerical precision used to store each weight, and sometimes the activations too. Weights start out at 32-bit floating point, or FP32, when training finishes. Quantization stores them at lower precision instead: 16-bit, 8-bit integer, called INT8, or 4-bit.
Still, the architecture stays exactly the same. So does the parameter count. Only the number of bits per weight changes, so the file gets smaller and each operation gets cheaper.
Quantization comes in two main forms. Post-training quantization, or PTQ, happens after training finishes, and needs no retraining at all. Quantization-aware training, or QAT, simulates the lower precision during training itself. QAT keeps more accuracy, but it costs training time that PTQ skips entirely.
Several established methods handle this work: GPTQ, AWQ, GGUF, and bitsandbytes NF4 are names worth knowing. Each targets a slightly different balance of speed, memory, and accuracy.
The table below shows what quantization actually saves, for a 7 billion parameter model at four common precisions. Figures use decimal gigabytes throughout, where 1 GB equals 1,000,000,000 bytes.
| Precision | Bits per weight | Memory for weights |
|---|---|---|
| FP32 | 32 | 28.00 GB |
| FP16 / BF16 | 16 | 14.00 GB |
| INT8 | 8 | 7.00 GB |
| INT4 | 4 | 3.50 GB |
Precision is not free to reduce, though. Some accuracy loss follows almost every drop in bit width, and that loss grows as precision falls further. Reasoning-heavy tasks tend to suffer more from aggressive quantization than simple classification does.
What Distillation Does
Distillation takes a different route entirely. It trains a new, smaller “student” model to reproduce the behavior of a larger “teacher” model. The student is not a compressed copy; it is a separate model, trained from scratch.
The student does not learn only from correct answers. It learns from the teacher’s soft probability distribution over outputs instead. Those soft targets carry more information than a hard label, since they rank the wrong answers too.
The result is a genuinely smaller model, with fewer parameters, often built on a different architecture. A distilled student does not have to copy its teacher’s design. Our guide to CNN vs RNN Fundamental Differences shows how differently two architecture families can behave.
Distillation is not free, though. In fact, it needs the teacher model, a training dataset, and enough compute for a full training run. That cost sits upfront, before the smaller model even exists.
For example, DistilBERT is the clearest worked case. Hugging Face released it in 2019, distilled from BERT-base during pre-training itself. BERT-base has 110 million parameters; DistilBERT has 66 million, a cut of 40 percent. It runs 60 percent faster, while keeping 97 percent of BERT’s language understanding capability. Training combined three losses: language modeling, distillation, and a cosine-distance term.
Distillation is not the same as pruning. Pruning removes weights or whole structures from an existing model, rather than training a fresh one from a teacher.
Quantization vs Distillation: Comparison Table
| Aspect | Quantization | Distillation |
|---|---|---|
| What changes | Bits used to store each weight | The entire model, retrained |
| Parameter count after | Same as the original | Fewer, set at design time |
| Architecture after | Identical to the original | Often different, usually simpler |
| Training required | None for PTQ; some for QAT | Always, a full training run |
| Data required | None for PTQ; a small calibration set for QAT | Teacher model, training data, and compute |
| Output artefact | The same model, lower-precision weights | A new, separate student model |
| Typical size-reduction lever | Fewer bits per weight | Fewer weights overall |
| Effect on accuracy | Some loss, grows as precision drops | Some loss, depends on the teacher-student gap |
| Hardware support needed | Kernels for the target bit width | Ordinary training hardware only |
| Time to apply | Minutes to hours, for PTQ | Days, for a full training run |
| Reversibility | Reversible; re-quantize at another bit width | Not reversible; a new model must be trained |
| Runs on the original inference stack | Yes, same architecture | No, needs its own deployment |
| Relationship to pruning | Distinct; pruning removes weights instead | Distinct; pruning trims the original net instead |
| Can be combined with the other | Yes, quantize a distilled student | Yes, distill then quantize the student |
Worked Example: What a 7B Model Actually Costs
Take a 7 billion parameter model as the working example. Every figure below follows the same decimal-GB convention used earlier, where 1 GB equals 1,000,000,000 bytes.
At FP16, each weight takes 2 bytes. Multiply it out: 7,000,000,000 weights times 2 bytes equals 14,000,000,000 bytes, or 14 GB. That matches the FP16 row in the table above.
Now drop to INT4, at half a byte per weight. The same 7 billion weights need only 3.5 GB instead. Going from FP16 to INT4 is a 4x reduction: 14 GB down to 3.5 GB.
The contrast that counts is this one. Quantizing that 7B model to INT4 still leaves 7 billion parameters. It is the same model, stored more cheaply. Distilling it into a 1.5 billion parameter student leaves a different, smaller model instead. That model had to be trained from scratch.
One caveat is worth stating plainly. These figures cover weight memory only. Running the model in production also needs memory for activations and the KV cache, on top of the weights themselves.
Using Both Together
Quantization and distillation are not rivals fighting for the same job. They solve different problems, so they compose cleanly.
The common pattern distills first, then quantizes the result. Train a smaller student model from the original teacher. Then apply post-training quantization to that student, shrinking its weights further still.
Doing it in this order keeps each step focused. Distillation handles the architecture and the parameter count. Quantization then handles the bits per weight, on top of whatever distillation already achieved.
A distilled, quantized student can end up dramatically smaller than the original teacher. It shrinks on both axes at once: fewer parameters, and fewer bits per parameter.
When to Use Which
Reach for quantization when the architecture already works, and you only need it to fit smaller hardware. So PTQ is fast, needs no training data, and can run in minutes on a single machine.
Reach for distillation instead when you need a genuinely different model: fewer parameters, or a narrow-task shape. Expect to invest in a teacher, a dataset, and a real training run.
Sometimes the real bottleneck is not the model at all. Retrieval speed or storage can be the constraint instead. Our Vector Database vs Relational Database guide addresses that question directly.
Most teams shipping a production LLM eventually use both. They distill down to a manageable student, then quantize that student for the target hardware.
Interview Questions
Frequently Asked Questions
Wrapping Up
Quantization vs distillation comes down to one clean line. Quantization keeps the same model and shrinks the bytes per weight. Distillation trains a new, smaller model from scratch instead.
Neither one beats the other outright. Production systems increasingly use both. They distill first, then quantize, to reach the smallest workable model for the hardware at hand.
Keep the core numbers in mind. FP16 to INT4 is a 4x memory reduction for the weights alone. DistilBERT trades 40 percent of BERT’s parameters for 60 percent faster inference, while keeping 97 percent of its capability.
Related reading on DiffStudy:
- Retrieval-Augmented Generation vs Fine-Tuning
- GPU vs TPU vs NPU for AI Workloads
- Machine Learning vs Deep Learning
- CNN vs RNN Fundamental Differences
- CS Fundamentals hub
