Self-attention and cross-attention run the same formula. Only the source of Q, K and V differs, though. In self-attention, one sequence supplies all three matrices, unlike cross-attention. The paper puts it plainly: queries, keys and values all come from one place, the output of the previous layer. In cross-attention, two sequences take part instead. The queries come from the decoder, while the keys and values come from the encoder output. As a result, cross-attention only exists in an encoder-decoder model. Since GPT is decoder-only, it has no encoder to draw K and V from. So it runs masked self-attention only, never cross-attention.
Every attention layer in a Transformer runs the same formula. Only one thing changes between layers: where the query, key and value come from. So that single detail splits attention into two types, self-attention and cross-attention. Since a reader now knows the source of Q, K and V, the rest of the architecture falls into place.
This guide leans on one source throughout: the 2017 paper by Vaswani and colleagues, “Attention Is All You Need.” Readers new to the field should start with our machine learning and deep learning guide instead. Our BERT vs GPT guide already covers the encoder-only versus decoder-only split. That split decides whether a model even has cross-attention.
What Attention Computes
Attention takes three inputs. Specifically, a query, a set of keys, and a set of values. The paper defines it with one formula.
Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V
Q, K and V start as projections of some input sequence. Since the dot product QKᵀ scores how well each query matches each key, that score drives everything after it. So dividing by √d_k stops those scores from growing too large as dimension increases. A softmax then turns the scores into weights, and those weights combine the values.
That scaling step separates scaled dot-product attention from plain dot-product attention. Otherwise, large values of d_k would push the softmax toward tiny gradients. Every attention layer in the Transformer runs this exact formula. So self-attention and cross-attention never differ in how the formula runs, only in where Q, K and V come from.
The base model sets d_k from two other numbers. Model dimension d_model is 512, split across h equal to 8 heads, so 512 divided by 8 gives d_k equal to 64. Since that division is cheap, running 8 heads in parallel costs little more than running one.
Self-Attention
One sequence supplies every Q, K and V in self-attention. The paper states the encoder case directly. “In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder.”
The decoder also runs self-attention, over the output sequence generated so far. That version stays masked, though, so a position can only see the tokens before it. The section below explains why that restriction matters.
Self-attention therefore appears twice in the original Transformer, once in the encoder, unmasked, and once in the decoder, masked. Both pull Q, K and V from the same sequence. So only the masking differs between them.
Cross-Attention
Cross-attention works differently, because two sequences take part instead of one, the sequence being generated and the sequence being read.
The paper calls this layer “encoder-decoder attention.” It describes the source plainly. “The queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence.”
So the rule holds in one line. Self-attention draws Q, K and V from one sequence. Cross-attention, instead, takes Q from the decoder and K and V from the encoder output. Even so, the formula, the softmax, and the scaling stay identical between them.
Self-Attention vs Cross-Attention: Comparison Table
The table below lines up self-attention against cross-attention, field by field. Since every value traces back to the paper’s own description, nothing here is guesswork.
| Aspect | Self-Attention | Cross-Attention |
|---|---|---|
| Source of Q | Own sequence, previous layer | Previous decoder layer |
| Source of K and V | Same sequence as Q | Encoder output |
| Sequences involved | One | Two |
| Attention matrix shape | n × n | n × m |
| Core formula | softmax(QKᵀ/√d_k)·V | softmax(QKᵀ/√d_k)·V |
| Masking | None in encoder, applied in decoder | None; defined by source, not masking |
| Present in the encoder | Yes | No |
| Present in the decoder | Yes, masked | Yes, second sub-layer |
| Occurrences per decoder layer | Once | Once |
| Present in BERT (encoder-only) | Yes, unmasked | No |
| Present in GPT (decoder-only) | Yes, masked | No |
| Purpose | Build context within one sequence | Let the decoder read the source sequence |
| What it enables | Every position sees every other position | Every decoder position attends over all source positions |
| Complexity per layer | O(n² · d) | Scales with n × m, not n² |
| What breaks without it | No context mixing within the sequence | The decoder loses access to the source sequence entirely |
One row is worth repeating on its own. Even so, the formula never changes between the two layers. Only the source of Q, K and V does. That single row, in fact, explains every other row in the table.
Where Each One Sits in the Transformer
The original Transformer places attention in exactly three spots. Encoder layers run self-attention, while decoder layers run masked self-attention first, then cross-attention over the encoder output.
- Encoder: self-attention.
- Decoder: masked self-attention.
- Decoder: cross-attention over the encoder output.
Cross-attention therefore appears once per decoder layer, right after the masked self-attention sub-layer. It never appears in the encoder, because the encoder has no second sequence to attend over. Since six encoder layers and six decoder layers make up the base model, cross-attention runs six times in total.
Why the Decoder Masks Its Own Attention
The paper explains the reason directly. “We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.”
The mechanism itself stays blunt. “We implement this inside of scaled dot-product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.” Once a score hits −∞, the softmax collapses it to zero, so an illegal position contributes nothing.
Masking is easy to misread as the feature that defines cross-attention. It does not. Masking separates decoder self-attention from encoder self-attention, and both of those draw Q, K and V from one sequence. Instead, cross-attention is defined by where K and V come from, not by whether a mask applies.
Encoder-Only, Decoder-Only, and Why GPT Has No Cross-Attention
The original Transformer pairs an encoder with a decoder, so it needs cross-attention to connect the two. Most later models keep only one half of that pair. Because of that choice, cross-attention disappears entirely from those models.
GPT is a decoder-only model. Since it has no encoder, a decoder layer has nothing to read K and V from. GPT therefore runs masked self-attention only, stacked layer after layer, and never runs cross-attention. That single gap trips up more explanations of this topic than any other detail on this page.
In contrast, BERT sits on the other side of the same split. Because it is encoder-only, it runs unmasked self-attention only, the same layer type the original encoder uses. BERT has no decoder either, so cross-attention has no role to play there.
Our BERT vs GPT guide already covers that encoder-only versus decoder-only split, so this section will not repeat it. One point is worth adding here. Cross-attention is not a property of attention in general. Instead, it exists only where an encoder and a decoder sit side by side. One feeds its output into the other.
So the common claim “cross-attention lives in the decoder” is only half right. It lives in the decoder of an encoder-decoder model specifically. A decoder-only model has a decoder stack too. Yet it runs zero cross-attention layers, because nothing plays the encoder’s part.
What Each One Costs
The paper’s own comparison table lines up self-attention against a recurrent layer. That layer, specifically, is the building block behind an RNN. Our CNN vs RNN guide covers recurrence in more depth.
| Layer type | Complexity per layer | Sequential operations | Maximum path length |
|---|---|---|---|
| Self-attention | O(n² · d) | O(1) | O(1) |
| Recurrent | O(n · d²) | O(n) | O(n) |
Self-attention runs in O(1) sequential steps, because every position can look at every other position in one operation. A recurrent layer instead needs O(n) sequential steps, one per position. So its maximum path length grows with n as well.
Cross-attention changes one more thing, since the two sequences it connects rarely match in length. Source length m and target length n instead give an n × m matrix, not n × n. In fact, most explanations of this topic skip that difference and treat every attention matrix as square.
Interview Questions
Frequently Asked Questions
Wrapping Up
Self-attention and cross-attention run the exact same formula. Instead, only the source of Q, K and V ever changes between them. In self-attention, one sequence supplies all three. In cross-attention, Q comes from the decoder, while K and V come from the encoder output.
Remember the trap this page exists for. Cross-attention only lives in an encoder-decoder model. A decoder-only model such as GPT has no encoder to read from. So it never runs cross-attention at all, just masked self-attention, layer after layer. Likewise, BERT sits on the encoder-only side of that same split, running unmasked self-attention with no cross-attention either.
Related reading on DiffStudy:
