TL;DR
- The idea
- Push verifiable-reward RL past math and code — into medicine, chemistry, economics and more, where no clean reference answer exists.
- Why it works
- Given an expert reference answer, different LLMs agree on whether a response is correct — so verification is tractable in open domains.
- The tool
- A compact 7B reward model distilled from a 72B teacher grades free-form answers a rule checker cannot parse.
- The payoff
- 7B policies gain up to +8.0% on diverse free-form reasoning — beating far larger aligned models.
Rule checkers run out of runway
Verifiable-reward RL only works where an exact-match rule can grade the answer — and most real reasoning isn't like that.
Reinforcement learning with verifiable rewards trains a policy against a cheap, automatic correctness check. That check is easy in math and code, where answers are single terms you can string-match. But that clean slice is smaller than it looks:
Share of questions a rule-based reward can verify
rule-verifiable · the other 39.7% needs a soft, model-based reward
rule-verifiable · the other 54.6% needs a soft, model-based reward
Even in math, nearly 40% of real questions have answers too unstructured for exact-match checking — and for general subjects the verifiable share falls below half.
The rule-verifiable ratio drops to 45.4% for complex multi-domain queries — a challenge for binary reward schemes.
A generative verifier crosses the bridge
If a rule can't grade the answer, ask a model that can — and it turns out models mostly agree.
The paper extends RLVR into diverse real-world domains including medicine, chemistry, psychology, economics, and education, where structured reference answers are unavailable. The enabling observation is that binary verification judgments exhibit high consistency across various LLMs given an expert reference answer — so a model-based reward model can stand in for a rule.
The method at a glance: a teacher grader labels exploration data, those judgments distill a compact reward model, and that model drives RLVR to the final policy.
Distill a 72B judge into a 7B verifier
An off-the-shelf small judge isn't enough — model size drives the downstream policy, so the authors distill a purpose-built one.
Bigger judge, better policy (math)
The 72B reward model reaches 62.7% while the 7B gets 58.8% — the 3.9-point gap the authors close by distillation, teaching a purpose-built RM-7B from the larger judge.
§3.3 · Reward Model Training
Soft rewards, and RL over imitation
Two design choices carry the gains: a probability-valued soft reward, and RL rather than supervised fine-tuning.
Soft reward vs hard 0/1 (multi-subject)
Where a hard {0,1} snap throws away information, the verifier’s probability-valued soft reward wins by +3.9 points; on clean math the two tie.
Table 1 · Reward score type
Imitation barely moves; RL unlocks it (math)
SFT significantly underperforms RL on both math and multi-subject tasks — fine-tuning on reference labels stalls near 45.7%, while model-based-reward RL reaches 63.0%.
§4.5 · Main Results
A 48-subject exam as the test bed
ExamQA spans college-level subjects across STEM, social sciences, humanities and applied fields.
To trust the automatic verifier as an evaluator, the authors measure agreement between Qwen-based majority voting and GPT-4o using Cohen’s Kappa and find almost-perfect agreement — letting a single evaluation sample stand in with no quality loss.
Model-based rewards scale; rule-based ones decay
Grow the training data and the two reward types diverge — this is the paper's sharpest result. Hover a point to read it off.
Reward scaling: 20k → 100k examples
Data
100k
Model-based
35.0
Gap
+18.1
The learned reward model shows consistent improvement throughout training while the rule-based signal destabilizes and collapses — the paper’s core evidence that model-based rewards have an inherent scalability edge.
Table 2 · Scaling experiments§4.6 · Scaling
Out of distribution — guess it first
The real test of a reward model is whether it holds on benchmarks it never trained on.
On benchmarks it never saw in training — where does RM-7B land?
RM-7B accuracy on held-out benchmarks
RM-7B remains significantly superior to the rule-based reward on datasets from other domains — evidence the verifier is a general-purpose reward signal, not one overfit to its training subjects.
What the verifier doesn't do
The authors are explicit about the edges of the claim.
The verifier emits a bare 0/1 with no chain-of-thought rationale, and the authors do not consider format-based rewards in this work. How much in-depth reasoning or formatting supervision a verifier really needs is left open.
Five things to remember
Rule checkers run out of runway.
Only 60.3% of math and 45.4% of multi-domain queries have answers an exact-match rule can verify — most free-form reasoning is invisible to binary rewards.
Only 60.3% of math is rule-verifiableCoverage drops to 45.4% multi-domain
A generative verifier crosses the bridge.
Because LLMs agree on correctness once given an expert reference answer, a small model-based verifier can score messy answers in medicine, chemistry, law and beyond.
Extending RLVR beyond math and codeBinary judgments agree across LLMsThe three-step RLVR pipeline
Soft rewards beat hard 0/1 snaps.
On ambiguous multi-subject cases the verifier's probability-valued soft reward (31.2%) tops the binary reward (27.3%); on clean math they tie.
SFT trails RL sharplyUp to +8.0% over SOTA open models
Model-based rewards scale; rule-based ones decay.
Grow the data 20k → 100k and the rule reward collapses (26.2 → 16.9) while the distilled reward model climbs monotonically (30.8 → 35.0).
Rule-based rewards fail to scaleModel-based reward keeps improving
A 7B verifier that generalizes.
Distilled from 72B judgments, RM-7B drives up to +8.0% gains and beats rule-based rewards by ~10 points on benchmarks it never saw in training.
Reward-model size drives the gapUp to +8.0% over SOTA open modelsGeneralizes to unseen benchmarks