Muon vs AdamW: Key Differences

Adam has been the default optimizer for so long that swapping it out counts as news. Muon, however, is one of the few challengers with a published billion-parameter training run.

This guide reads three sources directly. Keller Jordan defined Muon in December 2024. Moonshot AI then scaled it, and Loshchilov and Hutter defined AdamW years earlier.

Why Anyone Looked Past AdamW

Adam and AdamW have been the standard choice for large-scale training. Muon arrived as a practical challenger rather than a theoretical one. It holds “the current training speed records for both NanoGPT and CIFAR-10 speedrunning”.

Those are small benchmarks, though. The open question was scale, and the Moonshot team named it directly. Muon’s results were promising in small runs, yet “the scalability to larger models has not been proven”.

That paper set out to close the gap. So its answer arrived with a 16B-parameter mixture-of-experts model called Moonlight, roughly 3B of which activate per token. Our mixture of experts vs dense models guide covers that architecture.

What AdamW Actually Does

AdamW is Adam plus one specific correction. Loshchilov and Hutter observed that L2 regularisation and weight decay are equivalent for plain gradient descent. For adaptive optimizers, however, “this is not the case”.

Their fix names the method. They recover proper weight decay by “decoupling the weight decay from the optimization steps taken w.r.t. the loss function”. Therefore the W in AdamW is that decoupled weight decay.

Mechanically, AdamW is elementwise. It keeps two running statistics per parameter, then rescales every weight individually. Shape does not matter to it, since each number is treated on its own.

What Muon Does Instead

Muon treats a weight matrix as a matrix, not as a bag of numbers. The name encodes the method: “MomentUm Orthogonalized by Newton-Schulz”. In fact, two stages produce each update.

First comes ordinary momentum. Muon “optimizes 2D neural network parameters by taking the updates generated by SGD-momentum”. Then it applies “a Newton-Schulz (NS) iteration” to each of those updates.

That second stage is the whole idea. Jordan states it in one line: “The function of the NS iteration is to approximately orthogonalize the update matrix”. The Moonshot paper describes the same operation as updating “matrix parameters with orthogonalized gradient momentum using Newton-Schulz iteration”.

Memory follows from stage one. Jordan puts it plainly: “Before the NS iteration is applied, Muon is just standard SGD-momentum, so it has the same memory requirement”. One buffer, then, rather than the two AdamW keeps.

The orthogonalisation is not free, though. Each NS step costs extra matrix multiplications, and five iterations is the typical setting. So you trade a little compute per step for a better-conditioned update.

Muon vs AdamW: Comparison Table

The table below lines up Muon against AdamW, field by field. Every value traces back to the three sources named above.

One row matters more than the rest: whether the optimizer can run a model alone. AdamW can. Muon cannot, which the next section covers.

Why a Muon Run Still Contains AdamW

Headlines describe Muon as Adam’s replacement. The sources, by contrast, describe something narrower. Muon is “an optimizer for 2D parameters of neural network hidden layers”, and that scope leaves parameters uncovered.

Jordan is explicit about what happens to the rest. When training with Muon, “scalar and vector parameters of the network” go elsewhere. So do “the input and output layers”, which “should be optimized by a standard method such as AdamW”.

Two of those exceptions surprise people. Embeddings and the output head are 2D matrices, so Muon could take them in principle. Jordan advises otherwise. It is “important to optimize input and output parameters using AdamW”, he writes, “even though these are typically 2D”.

Likewise, the scaling paper reports the same arrangement. “In practice, AdamW is used in couple with Muon to handle non-matrix based parameters, like RMSNorm, LM head, and embedding parameters”. Normalisation layers appear there too, a topic our batch vs layer normalization guide covers.

The authors call the arrangement what it is. Muon “is utilized in conjunction with the Adam optimizer, where certain parameters remain under the purview of Adam optimization”. They then list that “hybrid approach” as an opportunity for improvement. That placement matters, because it puts full replacement in future work rather than in the results.

Their broader claim still stands alongside that. The paper does argue Muon “can effectively replace AdamW as the de facto optimizer” for large-scale training. Read carefully, though, that is a claim about which optimizer leads the run. It is not a claim that AdamW has left the building.

Muon Had to Borrow Weight Decay

Here is a detail that rarely travels with the headline. Muon did not originally include weight decay at all. The scaling paper states it flatly: “The original implementation of Muon omits weight decay”.

That omission held up at small scale and broke at large scale. Muon beat AdamW on small runs, yet those gains faded on bigger models trained with more tokens. Weights and layer outputs kept growing, eventually exceeding the useful range of bf16.

The fix came from AdamW. Moonshot added the standard decoupled weight-decay mechanism into Muon, and the result held up. Muon with weight decay “outperforms both vanilla Muon and AdamW” in the over-trained regime.

Note what that means for the rivalry. The scaling paper identifies weight decay as playing “a crucial role in Muon’s scalability”. Yet weight decay is precisely the W in AdamW. Therefore Muon reached scale partly by adopting the idea that named its competitor.

The Update-Size Problem

AdamW has a convenient property: its update size stays roughly constant, near a theoretical value of one. Muon, however, does not share it. Update size instead depends on the shape of the matrix.

The paper states this as a lemma. Take a full-rank matrix of shape A by B. Its theoretical update size scales with one over the square root of the larger dimension. Shape therefore drives magnitude.

Both extremes cause trouble. When the larger dimension is big, as in a dense feed-forward matrix, updates shrink. That in turn limits what the model can represent. When it is small, updates grow too large and destabilise training.

Moonshot’s remedy is a rescaling that cancels the effect. They scale Muon’s update to match AdamW’s observed range, which sits around 0.2 to 0.4. Those adjustments, they report, “allow Muon to work out-of-the-box without hyper-parameter tuning”.

What the Efficiency Figure Measures

Two numbers circulate, and they are the same finding stated twice. The abstract reports roughly double the “computational efficiency compared to AdamW with compute optimal training”.

The contributions section gives the same result as a fraction. Muon reaches “comparable performance to AdamW trained counterparts while requiring only approximately 52% of the training FLOPs”. Roughly half the FLOPs is roughly double the efficiency.

Read the qualifiers, because they carry weight. The comparison is against a tuned AdamW baseline, under compute-optimal training, measured in training FLOPs to reach comparable performance. It is not a claim that any given run finishes in half the wall-clock time.

The NS iterations cost real compute as well. Five matrix-multiplication passes per step are not free, so a FLOP-efficiency figure and an end-to-end speed figure are different quantities. Hardware decides how that overhead lands, a topic our GPU vs TPU vs NPU guide covers.

The Checkpoint Problem

The paper closes with an open problem that practitioners hit immediately. Optimizers do not swap cleanly mid-project.

Moonshot observed “the suboptimal performance of models pretrained with AdamW when fine-tuned with Muon, and vice versa”. The failure runs in both directions, so neither optimizer is simply the safer choice.

The consequence is practical rather than theoretical. That mismatch “presents a significant barrier to effectively leveraging the extensive repository of AdamW-pretrained checkpoints”. Almost every open-weight model you might fine-tune was pretrained with AdamW.

So the decision arrives early or not at all. Fine-tuning an existing checkpoint, as our LoRA vs QLoRA guide describes, inherits whatever optimizer produced it. The paper flags the mechanism as still unexplained.

Which One Fits

AdamW remains the safe default, and not merely from inertia. It covers every parameter, and it needs no companion optimizer. Besides, the checkpoints you want to build on were trained with it.

Muon suits a pretraining run you control end to end. Its reported gain is a FLOP-efficiency gain at compute-optimal training, which matters most when compute sets your ceiling. Reinforcement learning stages sit downstream of that choice, as our GRPO vs PPO and RLHF vs DPO guides describe.

Either way, you are configuring two optimizers rather than one. Muon takes the 2D hidden layers, while AdamW keeps the embeddings, the head, the norms and every scalar. Budget for tuning both.

Interview Questions

Frequently Asked Questions

Wrapping Up

Muon changes what an update looks like. Momentum comes first, then an orthogonalisation step, and the result is a better-conditioned update for a 2D hidden layer. The reported gain is roughly half the training FLOPs at compute-optimal training.

Remember the scope, because that is where the headlines slip. Muon covers 2D hidden layers only, so AdamW keeps the embeddings, the head, the norms and every scalar. Muon also needed AdamW’s weight decay before it would scale at all.

Related reading on DiffStudy:

• GRPO vs PPO
• RLHF vs DPO
• LoRA vs QLoRA vs Full Fine-Tuning
• Batch vs Layer Normalization
• Mixture of Experts vs Dense Models
• GPU vs TPU vs NPU