RoPE vs Absolute Positional Encoding: Key Differences

A self-attention layer looks at every token at once. Order is simply not part of that view. Positional encoding is how order gets back in.

Two schemes dominate both the syllabus and the model zoo. The 2017 Transformer used absolute sinusoidal encodings. Most current large language models use RoPE.

This guide reads four papers directly. Where the popular explanation disagrees with them, the papers win here.

Why Attention Needs Position At All

Self-attention computes a weighted sum over every token in the input. Shuffle those tokens and each weight simply follows its own token. The layer itself has no sense of order.

A recurrent network never meets that problem, since it reads strictly left to right. Position arrives free with the loop. Our CNN vs RNN guide covers how differently those two families see a sequence.

Attention gave up the loop to win parallelism. So position has to be supplied by hand. The paper says: “we must inject some information about the relative or absolute position of the tokens in the sequence”.

That requirement follows attention wherever it goes. Cross-attention needs position too, as our self-attention vs cross-attention guide explains. Neither variant knows where a token sat.

How Absolute Positional Encoding Works

The original recipe comes from “Attention Is All You Need”, published in 2017 as arXiv 1706.03762. It is short, though its consequences run deep.

The authors add “positional encodings” to the input embeddings “at the bottoms of the encoder and decoder stacks”. Both stacks get them, yet only once each.

Shape makes the addition possible. Encodings carry “the same dimension dmodel as the embeddings, so that the two can be summed”. Addition, rather than concatenation, is the entire operation.

For the values themselves, “we use sine and cosine functions of different frequencies”. Every dimension therefore gets its own wavelength. Each position lands on a pattern no other position shares.

The motivation was already partly relative. Its authors note that “we hypothesized it would allow the model to easily learn to attend by relative positions”. So the tidy absolute-versus-relative split was blurred from the very beginning.

Learned embeddings went into the same table of experiments. The paper reports that “the two versions produced nearly identical results”. Neither choice bought a real quality gap, then.

Encoder-only and decoder-only models inherited the same trick afterwards. Our BERT vs GPT guide covers how those two stacks diverge otherwise.

How RoPE Works

RoPE arrives in 2021, in “RoFormer: Enhanced Transformer with Rotary Position Embedding”, arXiv 2104.09864. Su and colleagues change the operation itself, not merely the vector.

Nothing gets added to the embedding. The layer rotates the query vector and the key vector instead. Its rotation angle comes from the token’s own absolute index.

Different dimension pairs turn at different rates. Low frequencies rotate slowly, while high frequencies rotate quickly. That spread is what makes one angle readable across a long span.

  q_m  ->  R(m) q_m            rotate the query by absolute position m
  k_n  ->  R(n) k_n            rotate the key   by absolute position n
  v_n  ->  v_n                 the value is never rotated at all

  (R(m) q_m) . (R(n) k_n)  =  q_m . R(n - m) k_n

  absolute angles go in  ->  only the offset (n - m) survives the dot

Read the last line closely, because it carries the whole argument. Two rotated vectors give an inner product that depends on the offset alone. One operation therefore delivers an absolute rotation plus a relative score.

Figure 1 of the RoFormer paper labels its inputs Query / Key. The value vector never turns at all. Additive encodings, conversely, shift everything downstream, including the value.

Rotation also preserves length. The paper notes that “RoPE injects position information by rotation, which keeps the norm of hidden representations unchanged”. Our batch vs layer normalization guide covers why norms matter so much inside these stacks.

Placement differs just as sharply. Absolute encodings land once, before any layer runs. RoPE acts inside attention, so every layer applies it again.

RoPE vs Absolute Positional Encoding: Comparison Table

The table sets the two schemes against each other, row by row. Every value traces back to the four papers named above.

One row carries more weight than the rest: the row for tensors touched. Absolute encoding moves the value vector, whereas RoPE leaves it alone.

RoPE Is Absolute and Relative at Once

A popular line says RoPE is relative while the 2017 scheme is absolute. RoFormer’s own abstract breaks that dichotomy.

RoPE “encodes the absolute position with a rotation matrix”. It also “incorporates the explicit relative position dependency in self-attention formulation”. Both properties, then, fall out of one operation.

The mechanism explains why cleanly. Each vector turns by an angle set by its own absolute index. Rotating a query and a key subtracts those angles inside the dot product.

So the resulting score depends on the gap alone. Absolute goes in, while relative comes out.

The 2017 side was never purely absolute either. Its authors hoped the model would “easily learn to attend by relative positions”. Sinusoids were picked partly for that reason.

Treat the two labels as a matter of emphasis, rather than a clean partition. Both schemes were reaching toward relative behaviour.

The Extrapolation Claim, Measured

Here is the claim worth testing: RoPE extrapolates to longer sequences because rotations are periodic. Measurement disagrees, and the story runs across four papers.

Step one lands in 2017, as a hope. Vaswani and colleagues chose sinusoids because that version “may allow the model to extrapolate”. The target was “sequence lengths longer than the ones encountered during training”. Note the word may, since it flags a hypothesis rather than a result.

Step two lands in 2021, with RoFormer. Among its stated properties sits “the flexibility of sequence length”. The claim is a design property, though nobody had yet measured the ceiling.

Step three lands in 2022, when someone finally measured. ALiBi tested the sinusoidal method, “which technically should be able to extrapolate”. Its verdict: that method “in practice has very limited extrapolation capabilities”.

Rotary came out ahead of sinusoids in the same experiments. ALiBi says the rotary method “improves over the sinusoidal one”. Yet it “still does not achieve satisfying results”.

Numbers make the ceiling concrete. At a training length of 512, rotary kept improving perplexity for up to 200 extra tokens. At 1024, that margin fell to roughly 100 tokens. ALiBi adds that “this comes at the cost of slower training and inference”.

So the honest reading is narrow rather than damning. Rotary does extend a little past the trained length. A little, however, is not the unlimited extrapolation the folklore promises.

None of this makes RoPE a poor choice. It dominates current stacks for good reasons, including the decay property below. Only the free-extrapolation story fails, and the RoFormer authors never made that story.

Equally, the 2017 authors were not wrong. They wrote may, and the hedge was correct.

Why Position Interpolation Had to Exist

Step four lands in 2023, as a patch. Chen and colleagues published “Extending Context Window of Large Language Models via Position Interpolation”, arXiv 2306.15595.

Its motivation names the problem outright. Many pre-trained models “use positional encodings that have weak extrapolation properties”, and the paper cites RoPE as the example. LLaMA sits in that same sentence.

The fix avoids extrapolation completely. Position Interpolation “linearly down-scales the input position indices to match the original context window size”. Indices get squeezed into the trained range, rather than pushed beyond it.

Why not simply push beyond it? Because naive extrapolation “may lead to catastrophically high attention scores that completely ruin the self-attention mechanism”. The paper’s own bound for interpolation is far “smaller than that of extrapolation”.

Keep the scope of the claim tight, though. Position Interpolation stretches an already-trained window by rescaling indices. Training data and architecture still set what a model can genuinely use.

A finite window is the underlying constraint here. Alternative architectures attack it from another direction, as our Mamba vs Transformer guide describes. The encoding alone never sets a context length.

The Long-Term Decay Property

RoPE brings one property the 2017 scheme never claimed. Su and colleagues write that “One can prove that this setting provides a long-term decay property”.

The paper then spells out the effect, “which means the inner-product will decay when the relative position increase”. Distant token pairs therefore score lower by construction.

That behaviour matches an obvious intuition about language. Words far apart usually matter less to each other. RoFormer lists it as “decaying inter-token dependency with increasing relative distances”.

Absolute encodings offer no such guarantee. Their dot products depend on two sinusoidal patterns added into the embeddings. Any decay there is learned, rather than proven.

Watch the boundary on this one. Decay is a property of the score, not a hard cutoff. Attention can still reach far, if the content justifies it.

Which One You Meet in Practice

The choice rarely comes down to taste. It usually comes down to which stack you inherited.

Older encoder models and most textbook diagrams use additive absolute encodings. They are easy to draw and easy to examine. Exams still lean on them heavily.

Newer large language models use RoPE almost by default. Its relative behaviour, decay property and untouched value vector all help. Tooling around it is mature too.

Neither one solves long context by itself. If you need a longer window, expect an extra step. Position Interpolation, retraining or a different architecture will do that work.

For an exam answer, lead with the mechanism. Say where each scheme acts, and say which tensors it touches. Those two facts separate them faster than any slogan.

Interview Questions

Frequently Asked Questions

Wrapping Up

The two schemes differ in operation and in placement. Absolute encoding adds a vector once, at the input. RoPE rotates the query and the key in every layer.

Keep three facts from the papers. RoPE is absolute and relative together, since one rotation gives both. It leaves the value vector untouched.

Extrapolation, meanwhile, turned out modest under measurement. That is why Position Interpolation exists at all.

Related reading on DiffStudy:

• Self-Attention vs Cross-Attention
• CNN vs RNN
• BERT vs GPT
• Mamba vs Transformer
• Batch vs Layer Normalization