SFT ablations
About this page
LR sweep and box-format ablations of the unified 4B (decisions/02-sft.md). Curves from the training logs; CER from the fixed eval set (10 val pages per source, parse mode, loc; text only, capped at 1 per page), scored on node002 for every checkpoint as it appears. Loop = the output reached its token limit. Refreshes every minute.
Loading…
Arms shown drives the whole tab: summary, benchmarks, per-language, charts, tables and run cards; default = final_* runs
English and Indic benchmarks every 100 steps; earlier runs: only checkpoints with a green value + their last one
How these are scored
OmniDocBench v1.6 EN (610 pages) and olmOCR-Bench EN (1,258 pages, 7,741 tests), scored with the unchanged harnesses of the 0.8B eval chain. The 4B parses the whole page end to end (mode 1 "parse", no layout model; vLLM, greedy, 4 MP cap, 12,288 max tokens). Markdown is built with the 0.8B pipeline's own code (idp_reconstruct: header, footer and figure blocks skipped, dehyphenate, repair_math), so the numbers compare with the IndicOCR 0.8B pipeline row (layout model + 0.8B block OCR). Grey rows = the published table of the Indic OCR blog (not re-run; blog values are rounded, e.g. text edit to 2 decimals). OmniDocBench overall = ((1 − text edit) × 100 + formula CDM + table TEDS) / 3. olmOCR overall = mean of the 8 category scores (baseline included), as in the blog. Indic printed (22,043 crops) / handwritten (10,544): WER (clean) and CER (clean) = the house scorer's macro over languages, "clean" variant, as fractions (lower is better). The blog's word accuracy = 100 × (1 − WER clean), verified 2026-11-03 by re-scoring (Gemini 3.1 Pro 80.4, Surya 67.9, Sarvam 86.6 / 55.4, Chandra handwritten 24.7 reproduce the blog), so WER (clean) compares directly with the blog. Reference rows: Indic values are a house re-score of the saved predictions where available (hover a cell for the source); Gemma-4-31B has WER only, derived from the blog word accuracy; PaddleOCR, GPT and Nemotron have no Indic scores ("–"). The 4B gets the 0.8B crop prompt and crop clamp (eval_bench/indic_infer.py). Per-language values are in the next card. Green = best 4B value per column.
Indic per language WER (clean) and CER (clean) per language, 4B runs
How to read
One row per 4B run and step with Indic scores. Each cell = the house scorer's clean WER and CER for that language (fractions, lower is better). Header: language code and n = scored crops; hover a cell for the count of catastrophic crops. Green = best 4B run per language column (and per metric); ★ = best of all rows shown, reference systems included. Reference rows are the blog systems with per-language house re-scores (Gemma-4-31B has no per-language data, so it is absent here). The 4B rows follow the arm selection.
Loss curves
CER vs step eval set; dashed line = start model, the warm-up checkpoint
Among ablations per-checkpoint comparison table
How to read
One row per checkpoint, grouped by step; best value per column within each step in green (val loss: best within loc and within digits only, as the two formats' losses are not comparable). Per-source columns are mean capped CER. Tokens = mean generated tokens vs mean ground-truth tokens per page (in the arm's coordinate format; digits answers are longer). Box F1 = predicted vs ground-truth boxes matched one-to-one at IoU ≥ 0.5 (label-agnostic); box label = label accuracy of the matched boxes.
Try your own page parse an uploaded image with any checkpoint
Upload (or paste, or drop) a page image; the chosen checkpoint parses it with vLLM on node002 (greedy, training prompt of the mode, 4 MP cap). A checkpoint that is not loaded yet takes ~2–3 min (merge + load); the next pages take seconds.