The short answer A Transformer attends to every token pair, so cost grows quadratically with window length. It also cannot model anything past that window. Mamba instead runs a selective state space model. Its SSM parameters become functions of the…
The short answer Batch normalization and layer normalization do the same job on a different axis. Batch norm averages down a column: one feature, across every sample in the mini-batch. Layer norm averages across a row instead: one sample, across…
The short answer A dense model runs every parameter on every token. A mixture of experts instead routes each token through a few chosen experts. Mixtral’s router selects 2 of 8 experts. So it touches only 13B parameters per token,…
The short answer RLHF and DPO both start from the same preference pairs, one answer marked over another. RLHF trains a separate reward model first, then runs reinforcement learning against it, usually PPO. So DPO skips that separate network and…
The short answer Self-attention and cross-attention run the same formula. Only the source of Q, K and V differs, though. In self-attention, one sequence supplies all three matrices, unlike cross-attention. The paper puts it plainly: queries, keys and values all…
The short answer BERT (Bidirectional Encoder Representations from Transformers) vs GPT (Generative Pre-trained Transformer) is not a rivalry between two general-purpose models. Both come from the same 2017 Transformer paper, which described an encoder and a decoder together. BERT kept…
The short answer MCP (Model Context Protocol) vs function calling is not a choice between two rivals. Function calling is a capability of the model’s application programming interface (API). The application sends tool definitions with the request. Then the model…
The short answer Temperature vs top-p differ in what they touch inside a model’s output. Temperature rescales every logit before softmax runs, so it reshapes the whole probability distribution. Below T = 1 it sharpens the leading token’s share. Above…
The short answer LoRA vs QLoRA vs full fine-tuning comes down to what gets frozen and what gets trained. Full fine-tuning updates every parameter, so memory has to hold the weights, the gradients, and the optimizer state together. For a…
You cannot copy content of this page
