A dense model runs every parameter on every token. A mixture of experts instead routes each token through a few chosen experts. Mixtral’s router selects 2 of 8 experts. So it touches only 13B parameters per token, though its total reaches 47B. Memory, though, has to hold every expert, whether or not that expert fires. Eight experts of 7B do not sum to 56B, since only the feed-forward sub-block gets replicated. Attention and the embeddings stay shared instead. The biggest myth about this design fails under testing. The Mixtral paper checked whether experts specialise by topic. It found no such pattern, only a syntactic one instead.
Two designs answer one question differently: how many parameters should a token touch? A dense model answers with all of them. Every transformer block pairs attention with a feed-forward sub-block. So both parts run in full for every token, a pairing our BERT vs GPT guide introduces in detail.
This guide leans on two papers throughout. One is Jiang, Sablayrolles, Roux, Mensch and colleagues’ “Mixtral of Experts,” from Mistral AI. The other is Fedus, Zoph and Shazeer’s “Switch Transformers,” from the Journal of Machine Learning Research. Indeed, both tested how sparse routing changes a transformer. So every number below traces back to one of them directly.
How a Dense Model Spends Its Parameters
A dense transformer treats every token the same way. Each block runs attention, then a feed-forward sub-block. Every weight in that sub-block takes part, so parameter count and compute cost move together.
Nothing in a dense model chooses which parameters to skip. So growing the model bigger means every token pays the full compute bill. Later sections show how a mixture of experts breaks that link.
What a Mixture of Experts Changes
A mixture-of-experts, or MoE, layer replaces the feed-forward sub-block with several parallel experts. Instead, a router decides which experts handle each token. So the output becomes a weighted sum over just the chosen experts.
y = Σ over i of G(x)_i · E_i(x)
Here E_0 through E_(n−1) are the n experts. G(x) is the gating vector the router produces. When that vector turns sparse, experts with a zero gate contribute nothing. So their output never needs computing at all. That skipped computation is the entire source of the saving.
Mixtral applies this swap to every feed-forward sub-block in the network. GShard, in contrast, replaces only every other block, leaving the rest dense.
Mixture of Experts vs Dense: Comparison Table
The table below lines up a dense model against a mixture of experts, field by field. So every value traces back to the Mixtral or Switch Transformer paper directly.
| Aspect | Dense Model | Mixture of Experts |
|---|---|---|
| Parameters active per token | All parameters activate every time | Only the routed top-K experts activate |
| Total parameter count | Fixed by depth and width | Grows with the number of experts, n |
| Active parameter count | Equals the total parameter count | Grows with K, up to n (Mixtral: 13B of 47B) |
| Feed-forward sub-block | One shared FFN per block | Replaced by n expert FFNs plus a router |
| Attention mechanism | Standard, unchanged | Not replicated; identical to the dense case |
| Routing mechanism | None; every path is always active | Softmax over the top-K router logits |
| K as a hyperparameter | Not applicable | Set per model; Mixtral uses 2, Switch Transformer uses 1 |
| Memory requirement | Matches the parameter count exactly | Must hold every expert, active or not |
| Compute per token | Full network computed every time | Only the selected experts are computed |
| Load balancing | Not a concern | A real problem under Expert Parallelism |
| Parallelism strategy | No expert routing to distribute | Expert Parallelism routes tokens to the GPU holding their expert |
| Kernel requirements | Standard dense matrix multiplication | Specialised sparse kernels, such as Megablocks |
| Training complexity | One network trained end-to-end | Router adds a load-balancing problem to solve |
| Serving complexity | One set of weights to serve | Every expert served, with variable tokens per expert |
| Where the design fits | Compute-constrained deployments | Capacity gains, when memory can hold more than compute must touch |
| Example architectures | A standard transformer stack | Mixtral (top-2), Switch Transformer (top-1), GShard (every other block) |
One row rewards a second look, the parallelism strategy. A dense model, though, has no expert to route toward. A mixture of experts needs GPUs coordinated instead, as the router section explains next.
The Router and Top-K Selection
The router scores every expert for a token, then keeps only the top few. The papers define it this way.
G(x) := Softmax(TopK(x · W_g))
TopK keeps a logit only if it ranks among the top K; otherwise it becomes −∞. Since softmax sends −∞ to zero weight, every non-selected expert drops out entirely. So the router really works in two steps: rank, then mask.
K itself is a hyperparameter. In the paper’s own words, K “modulates the amount of compute used to process each token.” So Mixtral sets K to 2, while Switch Transformer sets it to 1, a choice covered later in this guide.
Total Parameters Against Active Parameters
The papers separate two counts a dense model never needs. Total parameters, sometimes called the sparse parameter count, grow with the number of experts, n. Active parameters grow instead with K, up to n.
Mixtral’s own numbers make the gap concrete. Total parameters reach 47B, since the router picks 2 of 8 experts. Yet only 13B activate for any single token. Memory has to hold the full 47B regardless of routing. Compute per token, though, only touches the 13B that fire.
That gap is also where the efficiency argument lives. The paper reports Mixtral running with 5x fewer active parameters than Llama 2 70B. Even so, Mixtral’s own 47B total stays smaller than that model’s parameter count.
Why Eight Experts of 7B Is Not 56B
Eight experts at seven billion parameters each suggests simple multiplication: 56B. Mixtral’s real total is 47B, though. So the gap is not a rounding artefact.
Only the feed-forward sub-block gets replicated per expert. Attention, the mechanism our self-attention vs cross-attention guide covers, is not replicated at all. Embedding parameters are shared the same way. So every token passes through one shared attention stack and one shared embedding table, regardless of which experts its router selects.
The “8x7B” label therefore names the expert count and expert size, not the whole model. Instead, shared components pull the true total below what naive multiplication predicts.
What the Experts Actually Specialise In
A common claim holds that each expert learns a subject, one for maths, one for code, one for biology. The Mixtral paper, though, tested this directly. It found the opposite instead.
In the paper’s own words: “Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.” So expert assignment stayed nearly identical across arXiv papers, biology abstracts and philosophy text. Only DM Mathematics showed a distribution that differed, and only by a small margin.
What the router does show is syntactic structure, not topic structure. The paper points to the word ‘self’ in Python and the word ‘Question’ in English. Both often route through the same expert. So routing tracks surface patterns in the token stream, not the subject a passage happens to cover.
Choosing K: One Expert or Two
Mixtral routes each token to 2 experts, using SwiGLU as the expert function itself.
y = Σ over i of Softmax(Top2(x · W_g))_i · SwiGLU_i(x)
Switch Transformer takes the other end of that dial. Instead, it routes each token to only a single expert, k = 1. The paper reports that this simplification “preserves model quality, reduces routing computation and performs better.” It also notes that expert capacity can be at least halved once each token goes to one expert instead of two.
Shazeer and colleagues proposed the natural-language mixture-of-experts layer back in 2017. So Switch Transformers builds directly on that earlier work, just with a smaller K.
So K is a genuine design choice, not a fixed constant every mixture-of-experts model shares.
What a Mixture of Experts Costs You
Routing is not free. Distributing MoE layers across GPUs commonly uses Expert Parallelism, or EP, an approach our GPU vs TPU vs NPU guide covers further. So each token travels to the GPU holding its assigned expert, then the result returns.
EP “introduces challenges in load balancing,” in the paper’s own phrasing. So work must spread evenly across GPUs, or some sit idle while others queue behind a busier expert.
Specialised kernels handle part of that problem. The paper names Megablocks, which casts the feed-forward operations as large sparse matrix multiplications. That same approach also handles experts that receive a variable number of tokens each step.
Cheaper inference has other routes too, like the techniques our quantization vs distillation guide covers. Training cost, likewise, differs by method, an angle our LoRA vs QLoRA vs full fine-tuning guide explores in depth.
When Each Design Wins
Neither design wins outright; the trade-off follows the numbers above. A dense model keeps every parameter active. So its compute cost per token is easy to predict, and its memory footprint matches its parameter count exactly.
A mixture of experts instead trades memory for compute. Total parameters go up, since every expert must sit in memory whether or not a given token routes to it. Active parameters go down, though. So compute per token can stay far below the total.
So the choice comes down to which resource is scarce. When memory is cheap relative to compute, a mixture of experts can raise capacity without raising the per-token bill. When memory itself is the constraint, though, a dense model avoids paying for experts that sit unused.
Interview Questions
Frequently Asked Questions
Wrapping Up
A dense model spends every parameter on every token; a mixture of experts spends only a routed few. Mixtral shows the gap concretely: 47B total parameters, 13B active, chosen by a router selecting 2 of 8 experts.
Remember the two traps this page exists for. Eight experts of 7B never sums to 56B, since only the feed-forward sub-block is replicated while attention and embeddings stay shared. And routing does not track topic, whatever the folk explanation claims. Instead, the Mixtral paper found only syntactic structure in its expert assignment.
Related reading on DiffStudy:
- BERT vs GPT
- Self-Attention vs Cross-Attention
- Quantization vs Distillation
- LoRA vs QLoRA vs Full Fine-Tuning
- GPU vs TPU vs NPU
