RL ablations
About this page
4B RL from sft_best_ckpt (verl, GRPO + DAPO fixes / GDPO): training curves, per-component rewards and validation per run (registry rl/runs/registry.json; metrics from the verl file logger and rollout dumps), plus the algorithms explained with real probe groups. Validation reward is weighted by the number of validation prompts per kind. Refreshes every minute.
loading…
Runs
Training curves
One line per run of the selected group; x = step. Hover a point for its value.
Reward components
Mean score of each component per step (verl: from the rollout dumps; TRL: its per-reward logs). Choose a run.
Benchmarks per checkpoint
Scored on node052 with the SFT scripts (OmniDocBench parse_headers prompt, olmOCR-Bench, Indic printed / handwritten WER, lower is better). The dashed line is sft_best_ckpt.
The loss is not cross-entropy
For each token of each sampled answer, PPO's clipped policy-gradient loss:
A is the answer's advantage (below). ρ is how much more likely the token is now than when the answer was sampled. When ρ ≈ 1 the gradient is −A·∇log π(token): cross-entropy on the model's own answer, multiplied by A. A positive A trains the answer like an SFT target; a negative A pushes it away. The reference transcript is never a target: it is used only inside the reward.