The short answer Multi-head attention gives every query head its own key head and value head. Multi-query attention keeps all those query heads, though one key head and one value head now serve every one of them. Grouped-query attention sits…
The short answer Autoregressive decoding calls the large model once per output token, serially. Speculative decoding drafts several tokens with a small model. It then checks every draft in one parallel pass of the large model. Leviathan and colleagues (Google,…
The short answer Absolute positional encoding adds a fixed vector to every token embedding, once, at the bottom of the stack. RoPE rotates the query and the key instead, inside every attention layer. The 2017 Transformer paper picked sinusoids partly…
The short answer An encoder collapses many input lines into a short binary code. A decoder expands that code back into many output lines. TI’s SN74HC148 devices “encode eight data lines to 3-line (4-2-1) binary (octal)”, and its SN74HC138 reverses…
The short answer Flow control protects the receiver, while congestion control protects the network path. Both of them run inside the TCP sender, though. RFC 5681 settles the relationship in one line: “The minimum of cwnd and rwnd governs data…
The short answer Merge sort splits, sorts and merges, always in O(n log n), and it needs extra memory. Quick sort partitions around a pivot and sorts in place, averaging O(n log n) with an O(n squared) worst case. The…
The short answer An autoregressive model writes left to right, one token at a time. A diffusion LLM, by contrast, starts from a fully masked answer and unmasks tokens over repeated passes. LLaDA is the model that proved the second…
The short answer Muon does not simply replace AdamW, because a Muon run still contains AdamW. Muon is “an optimizer for 2D parameters of neural network hidden layers”. Everything else keeps the old optimizer, and the scaling paper says so…
The short answer GRPO is not PPO’s replacement. Its own authors call it “a variant of Proximal Policy Optimization (PPO)”. The change is narrow but consequential. PPO trains a separate value model to estimate the baseline. That model is “typically…
