MHA vs MQA vs GQA: Key Differences in Attention
The short answer Multi-head attention gives every query head its own key head and value head. Multi-query attention keeps all those query heads, though one key head and one value head now serve every one of them. Grouped-query attention sits…
