1 Training data← Back to the data viewer2 TokenizerCompareAuditExplainer slidesCoordinate tokens3 Image budget4 MP vs 8 MP4 NotesData processingAssamese ra checkSynthetic checkSmall held-out blocksData EDAJaigyan latticeJaigyan v4 re-readEmpty pagesManipuri: 4B vs pipeline5 TrainingEmbedding warm-upSFT ablationsSFT merges2B distillation0.8B distillationDistillation explainedRL planRL ablationsRL training

RL plan for the unified 4B

How we plan to run RL after SFT, what each part is, which published work already did it, and real samples scored by our prototype reward (rl/reward/page_reward.py). Design: rl/REWARD_DESIGN.md; research: rl/EXISTING_WORKS.md and rl/existing_works/.

The whole flow

done in a published work from our own 0.8B RL new in this plan
SFT 4B sft_best_ckptstart point FD-RL: RL without SFT fails Data probe 16 rollouts per candidate promptkeep prompts with reward spread Hunyuan 1.5 · Paddle 1.6 · MinerU-Pro RL prompt set pages × 6 modes, crops, emptysampled by allocation % reference must score 1.0 on itself GRPO loop (one step)GRPO + DAPO fixes: clip-higher 0.2 / 0.28 · dynamic sampling · token-level loss · small KL · T 1.0, top-p 1.0 Policy4B, trainer (FSDP)vLLM for rollouts G = 16 rolloutsper prompt, up to~16K tokens Parselabel, box (loc),text, containers Gates → 0EOS, </div>, parse,boxes, tables, loop Componentstext · box · label · ordertable · formula · empty AdvantageA: GRPO (one total) vsB: GDPO (each part) — compare policy update (IS mask for vLLM / trainer mismatch, IcePop) Evaluation every N steps OmniDoc, olmOCR, Indic pr / hw,loop rate, content-only metric Surya, LightOn: RL can buy format Seeds → soup average seed runs; merge partback to SFT if a category drops olmOCR 2 · LightOnOCR-2/3 RL'd 4B (new teacher) and RL recipe ready to rerunon a new SFT round OPD into students 2B and 0.8B; reverse KL onstudent top-k, then short RL OvisOCR2/3 · Xiaomi MOPD · Qwen3-VL Stage A first: crop tasks (handwriting, printed, tables, formulas) with the 0.8B v1.1 rewards, then the page modes.

What the published works tell us to expect

RL after a good SFT is a polish stage: +0.4 to +2 OmniDoc overall, mostly from tables (+0.5 to +4 TEDS), formulas (about +0.8 CDM), reading order and loops; plain text often stays flat. End-to-end page models like ours gained the most (HunyuanOCR +1.6 and +1.8, FD-RL +3.3). The data filter matters more than the optimiser (Hunyuan gained nothing until it filtered by rollout spread). Clean references are a must: teacher pseudo-labels as RL references cost −4.9 TEDS (TRivia).

Each gate and reward part, on a small example

Every number on this tab is computed by the reward code (rl/viz/make_explain.py → page_reward.py). A gate sets the whole reward to 0; a component is a score in [0, 1].

Each idea, and who already did it

paper done in a published work · ours from our 0.8B RL · new new in this plan · the sample column says where to see it in the Samples tab

Real pages, scored by the prototype reward

Each page is a real training reference. The variants are typical model failures built from it, and every number is the real output of page_reward.py. Green boxes: reference. Orange boxes: the selected answer. Reward = gates × weighted mean of the active components × length ramp − repeat penalty. In training, the advantage uses each component separately (GDPO); the single number here is for reading.

page

What these samples already found

A loop that later closes with </div> slipped through. The first loop gate looked only at the end of the whole answer, and the repeat penalty was diluted over the page: "correct text, then a loop" scored 0.971. Now the tail check also runs inside every block, and the worst block's repeat fraction counts: 0.000.
A bad reference in our data. An IndicDLP table reference itself loops (the header "Date of resignation" repeated dozens of times) and fails the loop gate. So every RL reference must score 1.0 against itself, or the page is dropped. The same check can count how many such references the SFT data has.
A count penalty punished legitimate splits. min(1, N_ref/N_pred) scored "each block split into 3 line boxes" (0.648) almost like "every box duplicated 3 times" (0.508). The overlap factor (union area / sum of areas) and containment matching fix it: split 0.847, duplicates 0.508.
One broken box zeroed a third of all answers. On the first probe chunk (12 newspaper prompts × 16 answers at temperature 1.0), 36% of the answers that ended normally failed only because one box among ~100 was broken (e.g. x1 < x0) or one block was not closed. Broken blocks are now a local penalty: 86% of answers pass the page-level gates, and the scores are graded.
Tables legitimately repeat. Many identical empty cells at the end of a table tripped the loop gate; the gate now looks at the text without HTML tags.
The reference must score 1 against itself. On the probe, 12% of layout references and 29% of md_tables references failed this check. Causes, all fixed: (1) the tag filter stopped inside the first <|loc_k|> token, so 8 similar blocks at the end of a page looked like a loop (false zeros for correct answers too); (2) the table part used the 0.8B table reward, which has its own repeat penalty, so a table with a row "5 5 5 5 5" lost points against itself; (3) dot leaders and empty Markdown cells counted as loops; (4) some Jaigyan "Table" blocks are plain text, and TEDS gave them 0. The repeat penalty now counts only the repeats above the reference's own. Bad references fell from about 10% to 1.6% of pages; the rest are real loops or non-rectangular tables in the data, and are dropped.

Round-1 probe: what the SFT 4B does on each kind of prompt

Every candidate prompt got 16 answers from the SFT 4B (temperature 1.0, full vocabulary), scored with reward v1.1. A prompt is selected when its 16 rewards spread (std ≥ 0.05), the mean is below 0.95 and the best answer scores at least 0.05: only then does GRPO get a signal. flat: solved = all 16 near 1; flat: middle = all 16 get about the same partial score; flat: zero = all 16 fail; bad reference = the reference does not score 1 against itself.

input image
reference boxesselected answer's boxes

Which reward parts each mode uses

Modetextboxlabelordertableformulagates
parse✓✓✓✓✓✓all
parse_headers✓✓✓✓✓✓all; containers checked
parse_md_tables✓✓✓✓✓ (MD → HTML)✓all
parse_figures✓ (+ figure description)✓✓✓✓✓all
layout–✓✓✓––all
text✓ (order inside)–––✓✓EOS, length, loop, zlib
empty page (any mode)1 for an immediate EOS, else exp(−n/20), n = text characters + 10 per box
crops (Stage A)0.8B v1.1 rewards: (1 − CER) × script-purity gate − repeat fraction; TEDS; CDM; length ramp

Translate mode is not part of RL for now (user, 2026-11-09).

Decisions so far (user, 2026-11-09)

  • Start from the best SFT 4B (sft_best_ckpt).
  • All 6 modes, no translate. Empty pages in every mode; no faint-text guard set.
  • Jaigyan v1, v2 and v3 are treated like the other unified page sets: page level, all blocks, sampled by the allocation percentages.
  • Newspapers: manually verified boxes and reading order; one valid order per page.
  • Formulas: CDM in the page reward too (compute: 8 nodes for RL, 1 for evaluation).
  • Gates for loops, truncation and broken structure; the answer must end with </div>. Loop penalty: repeat fraction and a zlib check.
  • 16 rollouts per candidate prompt; prompts without reward spread are dropped.