GRPO is not PPO’s replacement. Its own authors call it “a variant of Proximal Policy Optimization (PPO)”. The change is narrow but consequential. PPO trains a separate value model to estimate the baseline. That model is “typically another model of comparable size as the policy model”. GRPO drops it. Instead, it samples a group of outputs for the same question, then uses their average reward as the baseline. The KL penalty moves too. PPO folds a per-token KL penalty into the reward, whereas GRPO adds the KL divergence directly to the loss. So one model disappears, the advantage becomes group-relative, and the KL term changes address.
Reinforcement learning fine-tuning has a memory problem, and GRPO is a direct answer to it. PPO holds four models in play at once. GRPO holds three, because one of them was doing a job a group of samples can do instead.
This guide leans on three papers, read directly. Shao and colleagues introduced GRPO in the DeepSeekMath paper. DeepSeek-R1 then used it at scale, and that paper now appears in Nature. Schulman and colleagues defined PPO back in 2017.
Why PPO Needed a Cheaper Variant
PPO has been the default for RL fine-tuning of language models for years. The DeepSeekMath paper calls it “an actor-critic RL algorithm”. PPO is, in its words, “widely used in the RL fine-tuning stage of LLMs”.
Actor-critic is the operative phrase there. The critic is a second network, and therefore it has to be trained.
That critic is expensive. The paper is blunt about the reason. That value function “is typically another model of comparable size as the policy model”.
As a result, “it brings a substantial memory and computational burden”. You are effectively training two large models at once.
There is a second, subtler complaint. In language-model training, “usually only the last token is assigned a reward score by the reward model”. That sparse signal “may complicate the training of a value function that is accurate at each token”. The critic is therefore both costly and hard to fit well.
How PPO Trains a Language Model
Schulman and colleagues introduced PPO in 2017 as “a new family of policy gradient methods for reinforcement learning”. Its appeal was practical. PPO keeps some benefits of trust region methods. Yet it is “much simpler to implement, more general, and have better sample complexity (empirically)”.
In the language-model setting, PPO runs with four models. The policy is the model being trained, a decoder-style generator of the kind our BERT vs GPT guide introduces. The reference model is usually the frozen starting point. Our RLHF vs DPO guide covers how that stage is set up.
A reward model scores outputs. Finally, the value model estimates the baseline.
The advantage in PPO comes from Generalized Advantage Estimation, computed from the rewards and that learned value function. Then a clipping hyper-parameter keeps each update close to the old policy. As a result, stability comes from clipping, and variance reduction comes from the critic.
How GRPO Changes the Recipe
GRPO removes one of those four models. The DeepSeekMath figure caption states it directly. GRPO “foregoes the value model, instead estimating the baseline from group scores”. So training resources drop.
The replacement is sampling. For each question, GRPO “samples a group of outputs” from the old policy. A reward model then scores every output in that group. Because you now have several scores for one question, you can compute a baseline from them.
The paper frames the swap plainly. GRPO “obviates the need for additional value function approximation as in PPO”. Instead it “uses the average reward of multiple sampled outputs” as the baseline. So the group does the critic’s job.
That trade is not free, though. You pay in rollouts, since every question now needs several generations rather than one. DeepSeek-R1-Zero sampled 16 outputs per question. Sampling temperature matters here too, a topic our temperature vs top-p guide covers.
GRPO vs PPO: Comparison Table
The table below lines up GRPO against PPO, field by field. Every value traces back to the three papers named above.
| Aspect | PPO | GRPO |
|---|---|---|
| Relationship | The original algorithm, from 2017 | Called a variant of PPO by its own authors |
| Algorithm family | Actor-critic | Actor-critic minus the critic |
| Value model | Required, and trained alongside the policy | Dropped entirely |
| Size of that value model | Typically comparable to the policy model | Not applicable |
| Models held during training | Policy, reference, reward, value | Policy, reference, reward |
| Where the baseline comes from | A learned value function | The average reward across a sampled group |
| Advantage estimation | Generalized Advantage Estimation | Group rewards, normalised within the group |
| Generations per question | One rollout is enough | A group of outputs, so several rollouts |
| Where the KL term sits | A per-token penalty inside the reward | Added directly to the loss |
| KL estimator | The log-ratio penalty at each token | An unbiased estimator, guaranteed positive |
| Reward model | Required | Still required; only the value model went |
| Clipping | Yes, with an epsilon hyper-parameter | Yes, the same clipped ratio is kept |
| Reference model | Kept, usually the initial SFT model | Kept |
| Main stated benefit | Stability, and simpler than trust region methods | Lower memory use, and fewer training resources |
| Main cost | Memory and compute for the critic | Extra generations for every question |
| Fit with reward models | Scores one output at a time | Aligns with the comparative nature of reward models |
One row deserves a second look: the reward model. It survives in GRPO, and only the value model was removed. That single distinction is the one most summaries get wrong.
Where the Advantage Comes From
PPO estimates the advantage with a learned value function, one that must predict a return at every token. GRPO, by contrast, never learns such a function. It compares outputs against each other instead.
The arithmetic is simple. A group of outputs is sampled for one question, and a reward model scores each of them. Then “these rewards are normalized by subtracting the group average and dividing by the group standard deviation”.
An output’s advantage therefore answers a relative question: was this response better or worse than its siblings? That framing has a natural fit, the paper argues. After all, reward models are usually trained on comparisons between outputs for the same question.
The KL Penalty Changes Address
Both algorithms keep a reference model, and both penalise drift away from it. They apply that penalty in different places, though.
PPO puts it in the reward. The aim is to limit over-optimisation of the reward model. So the standard approach adds “a per-token KL penalty from a reference model in the reward at each token”. Then the reward itself is adjusted, before any advantage is computed.
GRPO moves it. The paper is explicit. GRPO works “instead of adding KL penalty in the reward”. It “regularizes by directly adding the KL divergence” to the loss.
The stated motive is to avoid complicating the advantage calculation. The estimator differs as well. GRPO uses an unbiased estimator of the KL divergence, one “which is guaranteed to be positive”. So the claim that GRPO abandons KL regularisation is simply wrong; the term merely lives somewhere else.
The Reward Model Does Not Disappear
This is the most common misreading of GRPO, so it deserves its own section. GRPO drops the value model. It does not drop the reward model.
Those two networks do unrelated jobs. A reward model scores how good an output is. A value model predicts expected future return, and PPO uses it as a baseline for variance reduction. Removing the second one therefore says nothing about the first.
In DeepSeekMath, a reward model still scores every sampled output. The paper even describes retraining it with a replay mechanism. Because of that, GRPO training runs with three models rather than four. Memory drops, yet the reward signal stays exactly where it was.
What DeepSeek-R1 Did Differently
DeepSeek-R1 made GRPO widely known, and it also introduced a separate change that often gets merged with it. The R1 paper describes GRPO as originally proposed “to simplify the training process”. It also aimed to “reduce the resource consumption” of PPO.
Then R1 went further on rewards. For reasoning tasks, the authors write, “we abstain from applying neural reward models” of either kind. Rule-based rewards took over, combining an accuracy reward with a format reward.
The reason was not memory. It was behavioural: “neural reward models are susceptible to reward hacking during large-scale reinforcement learning”. Instead, dropping the neural reward model was a robustness decision, quite separate from dropping the value model.
Note the scope carefully. That substitution applied to reasoning tasks, and for general data the team still used reward models to capture human preferences. DeepSeek-R1-Zero is also the variant that “relies exclusively on reinforcement learning without supervised fine-tuning”, not the shipped R1 model.
What the Paper Says RL Actually Improves
The DeepSeekMath authors ran an analysis that rarely survives into summaries. They measured Pass@K and Maj@K before and after RL, and the results were asymmetric.
Their finding: “RL enhances Maj@K’s performance but not Pass@K”. In plain terms, the model became more reliable at surfacing an answer it could already reach. It did not start reaching answers that were previously out of range.
The paper draws the conclusion itself. The gain is “attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities”. So RL sharpened the output distribution, and that is a narrower claim than the one usually made for it.
Which One Fits Your Setup
Pick GRPO when memory is the binding constraint. Removing a policy-sized critic frees real capacity, which matters on the hardware our GPU vs TPU vs NPU guide covers. The same logic drives parameter-efficient methods in our LoRA vs QLoRA comparison.
GRPO also fits naturally when a question can be answered several ways and scored reliably. Verifiable domains suit it well, since sampling a group is only useful if the scores mean something.
PPO remains the reference point, though. It is the algorithm GRPO is a variant of. Its per-token value estimates also carry information a single group-level score does not. Generation cost is the other consideration, because a group of rollouts per question is not cheap either.
Interview Questions
Frequently Asked Questions
Wrapping Up
One sentence in the DeepSeekMath abstract settles the framing: GRPO is a variant of PPO. It keeps the clipped objective, the reference model and the reward model, then changes three things. The critic goes, the baseline becomes group-relative, and the KL term moves into the loss.
Remember which model actually disappeared, because that is where most explanations slip. The value model went, and the reward model stayed. DeepSeek-R1 later replaced its neural reward model for reasoning tasks too, yet that was a separate decision about reward hacking.
Related reading on DiffStudy:
- RLHF vs DPO
- LoRA vs QLoRA vs Full Fine-Tuning
- Temperature vs Top-p Sampling
- GPU vs TPU vs NPU
- BERT vs GPT
