A Transformer attends to every token pair, so cost grows quadratically with window length. It also cannot model anything past that window. Mamba instead runs a selective state space model. Its SSM parameters become functions of the input token. That selection, then, lets it remember or forget information along the sequence. Still, the paper is exact here. Mamba-3B “outperforms Transformers of the same size” and “matches Transformers twice its size.” That is not the looser “outperforms twice its size” claim you may have seen. So decoding cost is the real centerpiece here. Meanwhile, a Transformer’s KV cache grows with every generated token. Mamba carries a fixed-size recurrent state instead. So each decoding step costs the same constant time.
Two designs answer one question differently: what happens to cost as a sequence grows longer? Generally, a Transformer leans on dense attention across the whole window. Mamba instead runs a linear-time recurrent state.
This guide leans on one paper: Gu and Dao’s “Mamba: Linear-Time Sequence Modeling with Selective State Spaces.” Indeed, every number below traces back to that paper directly. So every fact and quote comes from its text, not from outside claims.
The Cost Attention Pays for Long Sequences
Self-attention, covered in our self-attention vs cross-attention guide, actually has one real strength. Certainly, in the paper’s words, it can “route information densely within a context window.” That strength comes with a cost, though. The paper names two drawbacks together, not one.
Firstly, a hard limit exists: attention cannot model anything outside a finite window. Secondly, quadratic scaling follows: cost grows with the square of window length. Most articles, though, mention only the second point. Few mention the finite-window limit instead.
What a State Space Model Does Instead
Mamba is, in fact, a selective state space model. That is not the same thing as a plain recurrent network, the kind our CNN vs RNN guide covers. A structured SSM maps each input channel to an output through a latent state h.
Prior SSMs could be computed efficiently as either a recurrence or a convolution. That efficiency, though, had one condition: linear time invariance, or LTI. Under LTI, the Δ, A, B, and C parameters stay constant across time. So that constancy is what made the efficient convolutional path possible.
The Selection Mechanism That Made Mamba Work
Prior models shared one limitation, the paper argues. In fact, they could not “efficiently select data in an input-dependent manner.” In other words, they could not focus on or ignore particular inputs.
Mamba’s fix is a selection mechanism. So it works by “parameterizing the SSM parameters based on the input.” That lets the model filter out noise and remember relevant information indefinitely. So the state is not empty; it holds what matters, on purpose.
The abstract frames the same idea differently. There, the paper lets the SSM parameters “be functions of the input.” So the model can “selectively propagate or forget information along the sequence length dimension.” So it does that “depending on the current token.”
Selectivity is not free. Input-dependent parameters break time invariance. The paper says so plainly: the change “prevents the use of efficient convolutions.” So Mamba trades convolution for a different computation path, covered next.
Mamba vs Transformer: Comparison Table
The table below lines up a Transformer against Mamba, field by field. Every value traces back to the paper directly.
| Aspect | Transformer | Mamba |
|---|---|---|
| Core mechanism | Self-attention across every token pair | Selective state space model with a recurrent state |
| Scaling in sequence length | Quadratic with window length | Linear in sequence length |
| Cost per decoding step | Grows as the KV cache grows | Constant time per step |
| Cache of past tokens | Required; keys and values for every past token | Not required |
| State carried forward | No fixed state; recomputes over the window | A fixed-size recurrent latent state h |
| Context-window limit | Cannot model anything outside the window | None reported; selection remembers indefinitely |
| Time invariance | Not applicable | Deliberately broken; parameters vary with the input |
| How parameters relate to input | Attention weights computed per token pair | SSM parameters (Δ, A, B, C) are functions of the input |
| Efficient computation path | Parallel attention matrix multiplication | Hardware-aware parallel associative scan |
| Training parallelism | Fully parallel across the sequence | Computation and memory scale linearly in sequence length |
| What the architecture contains | Attention and MLP blocks | Neither attention nor MLP blocks |
| Longest sequence reported | Not stated in the paper | Performance gains reported up to 1M tokens |
| Reported throughput figure | Reference point in the paper’s comparison | 5x higher inference throughput than Transformers |
| Reported quality claim (3B scale) | Baseline for the comparison | Matches Transformers twice its size; beats same-size Transformers |
| Best suited to | Dense, in-window routing | Long context, constant-time decoding |
One row deserves a second look: the cost per decoding step. A Transformer’s per-step cost climbs as its cache grows. Mamba’s stays flat instead, the subject of the next section.
Why Mamba Needs No Cache of Past Tokens
Property two in the paper’s introduction is, indeed, the centerpiece of this whole comparison. Training “scales linearly in sequence length,” the paper states. Inference is the more surprising claim.
Unrolling the model autoregressively “requires only constant time per step.” Why: it “does not require a cache of previous elements.”
A Transformer decoder, the autoregressive style our BERT vs GPT guide covers, works differently. It generates one token, then attends back over every prior token to generate the next. So it must keep a cache of past keys and values. That cache, then, grows with every new token the model produces.
Mamba carries a fixed-size recurrent state instead of a cache. Accordingly, the state’s size never changes as the sequence grows longer. So each decoding step costs the same constant time, no matter how long the sequence already is. A Transformer’s per-step cost keeps climbing; Mamba’s still does not.
The Hardware-Aware Scan
Selectivity closed off the convolutional path, so the authors needed a new algorithm. Their answer computes the model “recurrently with a scan instead of convolution.” It also avoids materializing the expanded state in memory.
The stated goal is “to avoid IO access between different levels of the GPU memory hierarchy.” Our GPU vs TPU vs NPU guide covers that memory hierarchy in more depth.
The mechanism itself is, essentially, a parallel associative scan. Scans cost O(B L D N) FLOPs, which is linear in sequence length L. So that linear cost is the whole point of the redesign.
The paper reports two separate speed numbers, and, notably, they measure different things. One is theoretical: scaling linearly in sequence length, “compared to pseudo-linear for all convolution-based SSMs.” The other is measured instead: “up to 3x faster on A100 GPUs.”
That 3x figure compares the scan against earlier SSM implementations, not against Transformers. A separate 5x figure, covered next, compares Mamba’s inference throughput against Transformers instead.
What the Paper Actually Claims
The introduction claims three things for Mamba. Firstly, high quality: selectivity “brings strong performance on dense modalities such as language and genomics.” Secondly, fast training and constant-time inference, detailed above. Thirdly, long context: real gains reported “up to sequence length 1M.”
Architecturally, Mamba integrates selective SSMs into one simplified network. Notably, that network runs “without attention or even MLP blocks.” So no feed-forward sub-block survives either, the piece our mixture of experts vs dense models guide examines closely.
Now the numbers that get misquoted the most often. Mamba reports “5x higher throughput than Transformers” during inference. At 3B parameters, though, the claim is exact and easy to overstate.
Mamba-3B “outperforms Transformers of the same size” — that part is simple. It also “matches Transformers twice its size,” not outperforms them. The widely repeated version, “outperforms Transformers twice its size,” accordingly overstates the paper’s own claim.
On copying and induction-head tasks, Mamba “not only solves them easily” but extrapolates further. So it extends solutions “indefinitely long (>1M tokens).”
The paper also reports state-of-the-art results “across several modalities such as language, audio, and genomics.” Even so, that is not a claim that attention is obsolete. Instead, Mamba reads as an alternative backbone, not a final replacement verdict.
What Mamba Gives Up
Mamba’s selectivity, though, is not without cost inside the model itself. Once parameters depend on the input, time invariance breaks. That, accordingly, “prevents the use of efficient convolutions,” the paper states plainly. So Mamba needs the custom scan kernel covered above, not a standard convolution.
The architecture also gives up two familiar blocks. There is no attention layer anywhere in the network. Likewise, there is no separate MLP block either. Both are simply absent from the design, per the paper’s own description.
Where Each One Fits
Neither design wins outright; the fit depends on the sequence. Attention still routes information densely within a window, when that window is enough. Mamba fits better once the sequence outgrows that window, or once decoding cost is the deciding factor.
The paper reports state-of-the-art results in language, audio, and genomics. Long-context gains, likewise, hold up to the reported 1M-token mark. None of that makes attention obsolete, though. Instead, Mamba reads as an alternative backbone for those workloads, not a verdict against attention.
Interview Questions
Frequently Asked Questions
Wrapping Up
A Transformer’s cost climbs with the window; Mamba’s still does not. That gap comes from one design choice: no cache, just a fixed-size state. Each Mamba decoding step costs the same constant time, however long the sequence already is.
Remember the paper’s exact headline claim, then. Mamba-3B outperforms Transformers of the same size, and matches Transformers twice its size. The looser, widely repeated version overstates that. Get this one right, and the rest of the comparison follows.
Related reading on DiffStudy:
- Self-Attention vs Cross-Attention
- BERT vs GPT
- Mixture of Experts vs Dense Models
- CNN vs RNN: Fundamental Differences
- GPU vs TPU vs NPU
