The short answer Muon does not simply replace AdamW, because a Muon run still contains AdamW. Muon is “an optimizer for 2D parameters of neural network hidden layers”. Everything else keeps the old optimizer, and the scaling paper says so…
The short answer GRPO is not PPO’s replacement. Its own authors call it “a variant of Proximal Policy Optimization (PPO)”. The change is narrow but consequential. PPO trains a separate value model to estimate the baseline. That model is “typically…
The short answer Willison and OWASP disagree about where jailbreaking belongs. Willison, who coined the term, treats prompt injection and jailbreaking as separate attacks. His test is simple: if trusted and untrusted strings were never concatenated, it is not injection.…
The short answer A Transformer attends to every token pair, so cost grows quadratically with window length. It also cannot model anything past that window. Mamba instead runs a selective state space model. Its SSM parameters become functions of the…
The short answer Batch normalization and layer normalization do the same job on a different axis. Batch norm averages down a column: one feature, across every sample in the mini-batch. Layer norm averages across a row instead: one sample, across…
The short answer A dense model runs every parameter on every token. A mixture of experts instead routes each token through a few chosen experts. Mixtral’s router selects 2 of 8 experts. So it touches only 13B parameters per token,…
The short answer RLHF and DPO both start from the same preference pairs, one answer marked over another. RLHF trains a separate reward model first, then runs reinforcement learning against it, usually PPO. So DPO skips that separate network and…
The short answer A VANET is not a rival to a MANET. Instead, it is a specialised MANET, one where every node is a vehicle or a roadside unit. A general MANET, by comparison, covers phones, laptops, sensors, and even…
The short answer Self-attention and cross-attention run the same formula. Only the source of Q, K and V differs, though. In self-attention, one sequence supplies all three matrices, unlike cross-attention. The paper puts it plainly: queries, keys and values all…
You cannot copy content of this page
