An autoregressive model writes left to right, one token at a time. A diffusion LLM, by contrast, starts from a fully masked answer and unmasks tokens over repeated passes. LLaDA is the model that proved the second approach scales. Its authors set out to “challenge this notion” that capability depends on autoregression. Two things get overstated, though. Its parallel decoding is a quality-versus-speed dial, not free speed, and the benchmark runs used one token per forward pass. It also did not beat GPT-4o outright. Instead it won the reversal direction, while losing the forward one badly.
Every mainstream LLM you have used generates text the same way: one token, then the next. Diffusion LLMs question whether that order is necessary at all.
This guide reads one paper directly. Nie and colleagues introduced LLaDA, an 8B diffusion model trained from scratch. So far it remains the clearest evidence that the approach scales.
Why Anyone Questioned Left-to-Right
Autoregression is so standard that it looks like a law rather than a choice. The LLaDA authors say as much. Capabilities “are widely regarded as relying on autoregressive models (ARMs)”, and their stated aim was to “challenge this notion”.
One weakness motivated the work. Left-to-right generation “restricts their ability to handle reversal reasoning”, the paper notes. So a model that only ever predicts forward struggles to answer backward.
The question, then, was never whether diffusion could write a sentence. It was whether the approach could scale to a real model, trained from scratch, and hold up on benchmarks.
How an Autoregressive LLM Writes
An autoregressive model factorises text into a chain of next-token predictions. Each token conditions on everything before it and nothing after it. That is the design our BERT vs GPT guide covers, where GPT is the decoder-only side.
The consequence is structural. Autoregressive models, in the paper’s words, “optimize only left-to-right conditional probabilities”. Position and order are therefore baked into training itself.
Generation is therefore strictly sequential. One forward pass yields one token. Then a KV cache carries prior keys and values forward, so each pass stays cheap. Our Mamba vs Transformer guide covers how that cache grows.
How a Diffusion LLM Writes
Instead, LLaDA replaces that chain with masking. The paper describes it as employing “a forward data masking process and a reverse generation process”. Then a Transformer does the work, “parameterized” to “predict masked tokens”.
Generation starts from nothing readable. Given a prompt, sampling begins “from a fully masked response”, and the model works backwards from noise toward text.
Each step then does two things. The model predicts “all masked tokens simultaneously”, and afterwards it remasks a portion of what it just predicted. Low-confidence positions get another attempt on the next pass.
The answer is therefore refined in place rather than extended at the end. Attention runs over the whole sequence, a mechanism our self-attention vs cross-attention guide covers. No position is therefore privileged by being earlier.
Finally, two hyperparameters control the run: the generation length, and the number of sampling steps. Both are set by the user rather than learned.
Diffusion LLM vs Autoregressive: Comparison Table
The table below lines up the two approaches, field by field. Again, every value traces back to the LLaDA paper.
| Aspect | Autoregressive | Diffusion (LLaDA) |
|---|---|---|
| Generation order | Strictly left to right | Any position, refined over passes |
| Starting point | An empty continuation | A fully masked response |
| What one pass produces | One token | Predictions for every masked position |
| Conditioning directions | Left context only | Multiple conditioning directions |
| Training objective | Next-token likelihood | A lower bound on the likelihood |
| Output length | Decided by the model, ends at EOS | A user-set hyperparameter |
| Number of forward passes | One per token | A tunable sampling-step count |
| Quality and speed | Fixed relationship | An explicit trade-off dial |
| KV cache | Standard | Not applied in the LLaDA work |
| Reversal reasoning | A large forward-reversal gap | Consistent across both directions |
| Forward poem completion | GPT-4o 82.7, Qwen2.5-7B 75.9 | LLaDA-8B 51.8 |
| Reversal poem completion | GPT-4o 34.3, Qwen2.5-7B 38.0 | LLaDA-8B 45.6 |
| Largest trained from scratch | Routine at far larger scale | 8B parameters |
| RL alignment | Standard practice | Not yet done for LLaDA |
| Maturity | The industry default | Demonstrated, not yet scaled |
| Best sampling strategy found | Not applicable | Pure diffusion sampling |
Two rows deserve a second look: the poem-completion scores. They are the source of the most-repeated claim about this model, and they do not say what the headlines say.
The Parallel Speed Claim, Examined
The popular summary runs like this: a diffusion LLM predicts all tokens at once, therefore it is faster. The first half is accurate, and the conclusion does not follow.
Sampling steps are a dial. The paper calls the step count “a hyperparameter, which naturally provides LLaDA with a trade-off between efficiency and sample quality”. So fewer steps means faster and worse.
The arithmetic is explicit in the efficiency analysis. Take a generation length of 256. Step counts of 32, 64, 128 and 256 then correspond “to decoding 8, 4, 2, and 1 token(s) per forward pass, respectively”. However, only the aggressive settings decode several tokens at once.
Now the detail that rarely travels. For the benchmark comparisons, the authors are explicit. They “set the number of sampling steps equal to the generated length”. That ensures “just one tokens are transferred from the mask to the text” per step.
Yet one token per forward pass is exactly the autoregressive rate. Those headline results carry no parallel-decoding advantage at all, then. That makes them a quality comparison rather than a speed one.
The Reversal Curse
This is where diffusion earns its keep. The reversal curse describes a model that learns A is B and then fails at B is A. Again, left-to-right training is the suspected cause.
LLaDA shows “consistent zero-shot performance across both forward and reversal tasks”, the paper reports. Its rivals do not: “both Qwen 2.5 and GPT-4o exhibit a significant gap between the two”.
Notably, the authors did not engineer this. They state it plainly: “We did not design anything special for reversal tasks”. Instead they attribute the balance to LLaDA treating “tokens uniformly without inductive bias”.
Their explanation stays a hypothesis, and they label it as one. Autoregressive models “optimize only left-to-right conditional probabilities”. LLaDA, by contrast, “is trained to consider multiple conditioning directions”. So the authors “hypothesize” that this is the source of the gain.
What LLaDA Actually Beat GPT-4o At
The abstract says LLaDA “addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task”. However, coverage tends to shorten that to beats GPT-4o, which is a different claim.
The test used 496 famous Chinese poem sentence pairs. Models generated either the next line or the preceding one. Here are the scores as printed:
| Model | Forward | Reversal |
|---|---|---|
| GPT-4o (2024-08-06) | 82.7 | 34.3 |
| Qwen2.5-7B Instruct | 75.9 | 38.0 |
| LLaDA-8B Instruct | 51.8 | 45.6 |
Read the forward column. LLaDA scores 51.8 against GPT-4o’s 82.7, so it loses that direction by a wide margin. However, it wins reversal, 45.6 against 34.3.
The paper is candid about why the forward gap exists. Both autoregressive models “are strong, benefiting from significantly larger datasets and greater computational resources than LLaDA”.
The finding is about balance, then, not supremacy. LLaDA performs about the same in either direction, whereas GPT-4o falls by nearly 50 points when asked to run backwards.
What Diffusion Costs
Efficiency claims cut both ways, and the paper reports the unflattering side too. Earlier work “suggests that MDM requires 16 times more computation than ARM to achieve the same likelihood”.
The authors push back on that metric rather than the number. Likelihood is “a relatively indirect metric for downstream task performance”. Diffusion also optimises a bound on it, “making it not directly comparable to ARM”.
There is also a missing optimisation that matters for any speed comparison. The paper is blunt here. No specialised attention or position embeddings were designed for LLaDA, “nor were any system-level architectural optimizations such as KV cache applied”.
Yet every production autoregressive model uses a KV cache. Comparing an uncached diffusion model against cached autoregressive ones is not like-for-like. The paper does not pretend otherwise.
What the Paper Says It Has Not Shown
The limitations section is unusually direct, and it is the part worth reading before forming an opinion. The comparison was compute-bound. Direct comparisons “were restricted to a computational budget” of under 1023 FLOPs. Because resources went to the diffusion model, the authors “were unable to scale the ARM baseline to the same extent”.
Scale, though, is the bigger caveat. Both “the model scale and the amount of training data for LLaDA remain smaller” than leading counterparts. That is the paper’s own framing, not a critic’s.
Finally, two more gaps stand out. Sampling algorithms “remain preliminary”. LLaDA also “has yet to undergo alignment with reinforcement learning”, the stage our GRPO vs PPO guide covers. Both are stages every frontier autoregressive model has already been through.
Where Each One Stands
Autoregressive models remain the default, and the reason is maturity rather than theory. After all, they have the scale, the data, the RL alignment stages and the inference optimisations behind them.
Diffusion LLMs have proved the harder point: the approach scales to 8B and holds up. LLaDA is “competitive with strong LLMs like LLaMA3 8B in in-context learning”, trained from scratch, which nobody had shown before.
Still, the genuine advantage today is directional balance. Tasks that reason backwards from an answer suit the masked formulation. So does filling a gap with context on both sides.
Sampling settings also become your problem rather than the model’s. Step count and generation length both need tuning. Our temperature vs top-p guide describes the same problem for autoregressive decoding.
Interview Questions
Frequently Asked Questions
Wrapping Up
The real result is narrower than the headlines and more interesting than them. A diffusion model trained from scratch reached 8B parameters. It then stayed competitive with LLaMA3 8B, which nobody had demonstrated before.
Hold two corrections in mind. Parallel decoding is a dial you pay for in quality. Also, the GPT-4o win was the reversal direction only, alongside a heavy loss on the forward one. Both facts come from the paper itself.
Related reading on DiffStudy:
- BERT vs GPT
- Mamba vs Transformer
- Self-Attention vs Cross-Attention
- GRPO vs PPO
- Temperature vs Top-p Sampling
