Group-Relative Advantages

The Group Sampling Procedure

For each training prompt $x$:

1. Sample $G$ complete outputs: $y_1, \ldots, y_G \sim \pi_{\theta_\text{old}}(\cdot \mid x)$
2. Score each output: $r_i = R(x, y_i)$
3. Normalize rewards within the group:

$$\hat{A}_i = \frac{r_i - \mu_G}{\sigma_G}$$

where $\mu_G = \frac{1}{G}\sum_{j=1}^G r_j$ and $\sigma_G = \sqrt{\frac{1}{G}\sum_{j=1}^G (r_j - \mu_G)^2}$.

Interactive: Group vs. Value Baseline

Compare how GRPO’s group-relative advantages differ from PPO’s value-function baseline. The bars show raw rewards; the lines show the resulting advantages:

What to notice

• GRPO advantages (blue) are always centered around zero — some outputs are “above average” and some “below average” within the group
• PPO advantages (orange) depend on a fixed baseline $V(s)$, which may be inaccurate
• Changing the group size $G$ affects the quality of the normalization — larger groups give more stable estimates
• Click “New group” to see different reward distributions

Worked Example: Computing Group Advantages

Suppose we sample $G = 4$ outputs for the same prompt and a reward model scores them:

Output	$r_i$
$y_1$	0.9
$y_2$	0.4
$y_3$	0.7
$y_4$	0.2

Mean: $\mu = (0.9 + 0.4 + 0.7 + 0.2)/4 = 0.55$

Variance: $\tfrac{1}{4}[(0.35)^2 + (-0.15)^2 + (0.15)^2 + (-0.35)^2] = 0.0725$; std $\sigma = 0.269$.

Output	$r_i - \mu$	$\hat A_i = (r_i - \mu) / \sigma$
$y_1$	+0.35	+1.30
$y_2$	−0.15	−0.56
$y_3$	+0.15	+0.56
$y_4$	−0.35	−1.30

Two outputs are reinforced ($\hat A > 0$), two are pushed down — without ever consulting a learned value function. Now suppose a different prompt where all 4 outputs are excellent and score $\{0.91, 0.92, 0.90, 0.93\}$. The mean is 0.915, the spread is tiny, but after normalization the advantages still come out near $\pm 1$ — normalized to peer comparison, not to absolute reward. This is exactly why hard prompts (where every sample looks bad) and easy prompts (where every sample looks good) contribute comparable learning signal.

Why Group Normalization Works

The group mean $\mu_G$ serves the same role as the critic baseline $V(s)$ in PPO:

$$\underbrace{\hat{A}_i^{GRPO} = \frac{r_i - \mu_G}{\sigma_G}}_{\text{group relative}} \quad \leftrightarrow \quad \underbrace{\hat{A}_t^{PPO} = G_t - V_\phi(s_t)}_{\text{critic baseline}}$$

But the group mean is:

• Free to compute — no extra neural network needed
• Adaptive — automatically adjusts per prompt (hard prompts have lower mean, easy prompts have higher mean)
• Unbiased — no function approximation error

The standard deviation normalization $\sigma_G$ additionally stabilizes training by keeping advantage magnitudes consistent across prompts.

The KL Penalty

GRPO adds an explicit KL divergence penalty to the objective:

$$D_{KL}(\pi_\theta \| \pi_\text{ref}) = \mathbb{E}_{y \sim \pi_\theta} \left[ \log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} \right]$$

This prevents the policy from deviating too far from the reference (usually the SFT model). In practice, DeepSeek uses a per-token KL approximation:

$$D_{KL}^{(t)} \approx \frac{\pi_\text{ref}(a_t \mid s_t)}{\pi_\theta(a_t \mid s_t)} - \log \frac{\pi_\text{ref}(a_t \mid s_t)}{\pi_\theta(a_t \mid s_t)} - 1$$

GRPO Algorithm Summary

Initialize policy π_θ from SFT model, set π_ref = π_θ
for each iteration:
    Sample batch of prompts {x₁, ..., x_B}
    for each prompt x:
        Sample G outputs: y₁, ..., y_G ~ π_θ_old(·|x)
        Score: rᵢ = R(x, yᵢ)
        Normalize: Âᵢ = (rᵢ - mean) / std
    for each mini-batch epoch:
        Compute clipped surrogate with Â
        Add KL penalty: L = L_clip - β·D_KL
        Update θ to maximize L
    θ_old ← θ

Connection to the Learning Path

Looking back at our progression:

1. Action chains gave us the MDP framework and the concept of return
2. Policy gradients showed how to optimize a policy directly
3. PPO stabilized updates with clipping and used GAE for advantage estimation
4. GRPO simplified the pipeline by replacing the learned critic with group-relative normalization

Each step builds on the last — GRPO is best understood as “PPO without the critic, made possible by sampling multiple outputs per prompt.”

References

• Shao et al. DeepSeekMath. 2024, §4 — the group-relative advantage and per-token KL approximation used above. arXiv:2402.03300
• Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025 — dynamic sampling: filtering the all-correct/all-wrong groups whose σ = 0 signal vanishes (the “wasted groups” you can watch in the live demo on the previous page). arXiv:2503.14476