BERT (Bidirectional Encoder Representations from Transformers) vs GPT (Generative Pre-trained Transformer) is not a rivalry between two general-purpose models. Both come from the same 2017 Transformer paper, which described an encoder and a decoder together. BERT kept only the encoder. GPT kept only the decoder, so neither model is the whole original Transformer. That one choice sets the attention mask. BERT lets every token attend to every other token, so its attention is bidirectional. GPT lets a token attend only to itself and earlier tokens, so its attention is causal instead. Because BERT already sees both directions, next-token prediction is trivial for it. So BERT trains with masked language modelling (MLM) instead: hide some tokens, then predict them from context on both sides. Because GPT can never see the future, it trains by predicting the next token, one at a time. BERT outputs representations: embeddings, labels, or spans. GPT outputs more text, generated one token at a time. Encoder-only models still run production classification and embedding systems. Decoder-only models still power chat and generation.
BERT and GPT trace back to the same 2017 paper, Attention Is All You Need. Students hear that and assume the two models work alike. They do not.
One architectural choice explains the split: which entries of the attention matrix each model is allowed to fill in. That single choice decides the training objective, the output type, and the job each model ends up doing.
Transformers themselves replaced the recurrent and convolutional architectures compared in CNN vs RNN. That earlier shift happened before BERT and GPT split into separate lineages. This guide picks up after the split and stays narrow on one masking decision.
Both Come From the Same Transformer
The original Transformer paper, published by Vaswani and colleagues in 2017, described one architecture with two halves. An encoder sat on one side. A decoder sat on the other side, and the two halves worked together to translate one language into another.
BERT and GPT each kept only one of those halves, not both. BERT is encoder-only. GPT is decoder-only. So neither model is the complete original Transformer; each one specialised on half of it instead.
That single choice, which half to keep, decides almost everything that follows. It sets the attention pattern, the training objective, and the shape of what each model can output.
BERT, the Encoder-Only Model
BERT, introduced by Devlin and colleagues at Google in 2018, kept the Transformer’s encoder and dropped the decoder.
Its attention is bidirectional. Every token can attend to every other token, both to its left and to its right. The model reads a whole sentence at once, not left to right.
Because every token can already see the whole sentence, next-word prediction is not a useful training signal here. So Devlin’s team trained BERT with masked language modelling (MLM) instead. Around 15% of input tokens get hidden, and the model learns to predict them from context on both sides.
The original paper also added a second objective, Next Sentence Prediction (NSP). It trained the model to judge whether one sentence followed another. Later work, most notably RoBERTa, dropped NSP and trained on MLM alone.
BERT outputs representations, not sentences. Feed it text, and it returns embeddings, token labels, or answer spans. It is not designed for autoregressive text generation, though a masked model can still fill in a hidden position.
BERT-base has 110 million parameters across 12 layers. BERT-large has 340 million parameters across 24 layers. Both counts come from the original paper.
Typical uses include classification, named entity recognition (NER), extractive question answering, and sentence embeddings. Teams usually fine-tune BERT: they attach a small task-specific head, then retrain on labelled data. The main strength is representation quality, since a bidirectional read builds a richer sense of a whole sentence than a one-directional read can. The main limitation is the one already mentioned: BERT cannot generate free-form text on its own.
GPT, the Decoder-Only Model
GPT, introduced by Radford and colleagues at OpenAI in 2018, kept the Transformer’s decoder and dropped the encoder.
Its attention is causal, also called masked or unidirectional. A token may attend only to itself and to tokens that came before it. It can never look ahead.
That restriction is deliberate. Because GPT can never see the future, predicting the next token becomes a fair training task. So Radford’s team trained it with causal language modelling instead: given everything before a token, predict that token.
GPT outputs more text. It generates one token at a time, autoregressively, feeding each new token back in as input for the next step. Sampling settings, compared in temperature vs top-p, control exactly how that next token gets picked.
GPT-1 had 117 million parameters across 12 layers, per the original paper.
Typical uses include text generation, chat, completion, and summarisation. Teams increasingly prompt GPT-style models zero-shot or few-shot, rather than fine-tuning for each task. The main strength is generation, since causal attention is exactly what autoregressive decoding needs. The main limitation is representation quality in the other direction; a GPT-style read of a sentence only ever sees what came before, never what comes after.
BERT vs GPT: Comparison Table
| Aspect | BERT | GPT |
|---|---|---|
| Half of Transformer kept | Encoder only | Decoder only |
| Attention direction | Bidirectional | Causal, left to right |
| Attention mask shape | Full square, every position allowed | Lower triangle only |
| Pre-training objective | Masked language modelling (MLM) | Causal language modelling |
| What a training example looks like | Sentence with about 15% of tokens hidden | Sequence where each token predicts the next |
| What the model outputs | Representations: embeddings, labels, spans | More text, one token at a time |
| Text generation ability | Not designed for autoregressive generation | Built for autoregressive generation |
| Typical downstream use | Classification, NER, extractive QA, embeddings | Generation, chat, completion, summarisation |
| How it is usually adapted | Fine-tuned, with a task-specific head added | Prompted zero-shot or few-shot, or fine-tuned |
| Original paper and year | Devlin et al., 2018 | Radford et al., 2018 |
| Original parameter counts | 110 million (base), 340 million (large) | 117 million (GPT-1) |
| Where it sits in a modern stack | Retrieval, embeddings, classifiers | Generation, chat interfaces, agents |
| Main strength | Rich bidirectional representations | Fluent, coherent text generation |
| Main limitation | Cannot generate free-form text on its own | Only ever sees tokens before the current one |
The One Line That Separates Them
Every difference above still traces back to one line: the attention mask. It decides which entries of the attention matrix a token is allowed to use.
Take a four-token sequence. BERT’s mask allows every position to attend to every other position:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
That is 16 allowed attention pairs, the full square, nothing withheld.
GPT applies a causal mask instead. Token i may attend only to tokens up to and including itself:
1 . . . 1 1 . . 1 1 1 . 1 1 1 1
That is 10 allowed pairs, exactly n(n+1)/2 for four tokens. The upper triangle stays empty.
Follow that triangle of zeros through to its consequences. Because GPT can never see the future, it can be trained to predict the next token, and it can generate. Because BERT can see both directions, it builds a richer representation of a whole sentence, but next-token prediction is trivial for it, so it trains by masking instead.
One triangle of zeros decides everything downstream: the training objective, the output type, and what each model ends up used for. Some implementations also apply a separate padding mask to ignore filler tokens, a different concern from the one described here.
How Each One Is Pre-Trained
BERT and GPT both learn from raw, unlabelled text. They learn different things from it, though, because their objectives point in different directions.
BERT’s objective is masked language modelling. Roughly 15% of tokens in a sentence get replaced with a mask token. The model predicts the original token using every other token in the sentence, before and after the mask.
GPT’s objective is causal language modelling instead. Given a sequence of tokens, the model predicts the next one. Then that next token joins the input, and the process repeats.
The two objectives are mirror images of each other. MLM removes information from the middle and asks the model to restore it. Causal language modelling removes nothing, but hides everything after the current position.
What Each One Is Good At
BERT is good at understanding a fixed piece of text. Classification, NER, extractive question answering, and sentence embeddings all lean on its bidirectional read.
Encoder-only models are not obsolete, despite what some articles imply. They remain the standard choice for classification and embeddings, which is why they still ship in production retrieval systems. Smaller distilled versions, such as DistilBERT, covered in quantization vs distillation, keep that same bidirectional design at a fraction of the size.
GPT is good at producing new text instead. Generation, chat, completion, and summarisation all lean on its ability to extend a sequence one token at a time.
The split is not about which model is stronger. It is about which output shape a task actually needs: a label or embedding, versus more text.
Which One to Reach For
Reach for BERT, or an encoder-only model like it, whenever a task ends in a label, a score, or an embedding. Search ranking, spam filters, and semantic search all fit that shape.
Reach for GPT, or a decoder-only model like it, whenever a task ends in more text instead. Drafting, chatting, and summarising all fit that shape.
Adapting either model does not have to mean a full retrain. LoRA vs QLoRA vs full fine-tuning covers cheaper ways to adapt both encoder-only and decoder-only models to a new task.
Plenty of production systems use both at once instead of choosing. One encoder-only model retrieves or classifies. One decoder-only model writes the final response. Picking one over the other is rarely the real decision; picking the right one for each piece of the pipeline is.
Interview Questions
Frequently Asked Questions
Wrapping Up
BERT vs GPT is not a contest with one winner. Both kept one half of the same 2017 Transformer, and each half does a different job.
Keep the one masking decision close. BERT’s mask fills the whole square, so it reads bidirectionally and trains by masking. GPT’s mask fills only the lower triangle, so it reads causally and trains by predicting the next token.
So the honest framing is specialisation, not competition. Reach for BERT when a task ends in a label or an embedding. Reach for GPT when a task ends in more text.
Related reading on DiffStudy:
- CNN vs RNN: Fundamental Differences
- Temperature vs Top-p
- Quantization vs Distillation
- LoRA vs QLoRA vs Full Fine-Tuning
- CS Fundamentals hub
