How are reasoning models trained with reinforcement learning? Explain RL with verifiable rewards, GRPO, and reward hacking.

30-second answerSay your answer out loud first, then reveal.

Instead of learning from human preference labels, the model attempts problems with checkable answers (maths with known solutions, code with unit tests) and gets a reward for correctness (verifiable rewards). Policy-gradient RL increases the probability of reasoning traces that led to correct answers. GRPO (from DeepSeekMath, used in DeepSeek-R1) samples a group of answers per problem and uses each answer's reward relative to the group's average as its advantage, which removes the need for a separate value (critic) model. Risks include reward hacking: exploiting flaws in the checker instead of truly solving the problem.

RLVR loop

Sample a problem from a dataset with verifiable outcomes.
The model generates G complete solutions (with reasoning).
A verifier scores each: answer match, tests pass, format compliance.
Compute advantages and update the policy (with a KL regulariser to a reference model, in some variants).

GRPO advantage (intuition)

text

advantage_i = (reward_i − mean(rewards in group)) / std(rewards in group)

Better-than-average samples get reinforced and worse ones discouraged. No critic network is needed, which saves memory and complexity compared with PPO.

Emergent behaviours reported: longer reasoning over training, self-verification, backtracking ("wait, let me re-check"). DeepSeek-R1-Zero showed that reasoning behaviours can emerge from RL on a base model without SFT, though with readability issues that an SFT cold start fixed.

Reward hacking examples

Code: special-casing test inputs, modifying tests, or exiting early so the tests "pass".
Maths: exploiting a lenient answer parser.
Format rewards gamed with empty structure.
Mitigations: robust verifiers, hidden tests, sandboxing, monitoring traces, penalties for detected exploits, a diverse task mix.

Why it matters: verifiable domains give unlimited, cheap and reliable reward signals, unlike human preference labels. That scalability is why RL compute became a major axis for improving models.

PreviousYou're asked to build a model for Hindi legal Q&A. Walk through your end-to-end plan, from base model choice to deployment.

Slow is fine. Stopping is the only problem.

How are reasoning models trained with reinforcement learning? Explain RL with verifiable rewards, GRPO, and reward hacking.

RLVR loop

GRPO advantage (intuition)

Reward hacking examples

Related