RL plan for the unified 4B
How we plan to run RL after SFT, what each part is, which published work already did it, and real samples scored by
our prototype reward (rl/reward/page_reward.py). Design: rl/REWARD_DESIGN.md; research:
rl/EXISTING_WORKS.md and rl/existing_works/.
The whole flow
What the published works tell us to expect
RL after a good SFT is a polish stage: +0.4 to +2 OmniDoc overall, mostly from tables (+0.5 to +4 TEDS), formulas (about +0.8 CDM), reading order and loops; plain text often stays flat. End-to-end page models like ours gained the most (HunyuanOCR +1.6 and +1.8, FD-RL +3.3). The data filter matters more than the optimiser (Hunyuan gained nothing until it filtered by rollout spread). Clean references are a must: teacher pseudo-labels as RL references cost −4.9 TEDS (TRivia).
Each gate and reward part, on a small example
Every number on this tab is computed by the reward code (rl/viz/make_explain.py →
page_reward.py). A gate sets the whole reward to 0; a component is a score in [0, 1].
Each idea, and who already did it
Real pages, scored by the prototype reward
Each page is a real training reference. The variants are typical model failures built from it, and every
number is the real output of page_reward.py. Green boxes: reference. Orange boxes: the selected answer.
Reward = gates × weighted mean of the active components × length ramp − repeat penalty. In training, the advantage uses
each component separately (GDPO); the single number here is for reading.
What these samples already found
Round-1 probe: what the SFT 4B does on each kind of prompt
Every candidate prompt got 16 answers from the SFT 4B (temperature 1.0, full vocabulary), scored with reward v1.1. A prompt is selected when its 16 rewards spread (std ≥ 0.05), the mean is below 0.95 and the best answer scores at least 0.05: only then does GRPO get a signal. flat: solved = all 16 near 1; flat: middle = all 16 get about the same partial score; flat: zero = all 16 fail; bad reference = the reference does not score 1 against itself.
Which reward parts each mode uses
| Mode | text | box | label | order | table | formula | gates |
|---|---|---|---|---|---|---|---|
| parse | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | all |
| parse_headers | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | all; containers checked |
| parse_md_tables | ✓ | ✓ | ✓ | ✓ | ✓ (MD → HTML) | ✓ | all |
| parse_figures | ✓ (+ figure description) | ✓ | ✓ | ✓ | ✓ | ✓ | all |
| layout | – | ✓ | ✓ | ✓ | – | – | all |
| text | ✓ (order inside) | – | – | – | ✓ | ✓ | EOS, length, loop, zlib |
| empty page (any mode) | 1 for an immediate EOS, else exp(−n/20), n = text characters + 10 per box | ||||||
| crops (Stage A) | 0.8B v1.1 rewards: (1 − CER) × script-purity gate − repeat fraction; TEDS; CDM; length ramp | ||||||
Translate mode is not part of RL for now (user, 2026-11-09).
Decisions so far (user, 2026-11-09)
- Start from the best SFT 4B (sft_best_ckpt).
- All 6 modes, no translate. Empty pages in every mode; no faint-text guard set.
- Jaigyan v1, v2 and v3 are treated like the other unified page sets: page level, all blocks, sampled by the allocation percentages.
- Newspapers: manually verified boxes and reading order; one valid order per page.
- Formulas: CDM in the page reward too (compute: 8 nodes for RL, 1 for evaluation).
- Gates for loops, truncation and broken structure; the answer must end with </div>. Loop penalty: repeat fraction and a zlib check.
- 16 rollouts per candidate prompt; prompts without reward spread are dropped.