1 Training data← Back to the data viewer2 TokenizerCompareAuditExplainer slidesCoordinate tokens3 Image budget4 MP vs 8 MP4 NotesData processingAssamese ra checkSynthetic checkSmall held-out blocksData EDAJaigyan latticeJaigyan v4 re-readEmpty pagesManipuri: 4B vs pipeline5 TrainingEmbedding warm-upSFT ablationsSFT merges2B distillation0.8B distillationDistillation explainedRL planRL ablationsRL training

OCR Tokenizer Lab

Inspect the same text. Compare the boundaries.

Data quality audit ↗
Loading sample pool…

One block, two tokenizers.

See where Indic text fragments—and where a vocabulary helps.

——Choose a block
Loading…
Source OCR block crop
Original transcription

Token boundaries

Adjacent tokens use different colors
Token inspector
Click a colored piece
Token ID, vocabulary spelling, character offsets, and byte details appear here.

Colors identify token boundaries, not semantic categories. When several byte-level tokens share one Unicode character, its span uses a striped background and a ×N badge; click to inspect every token. Token pieces may decode individually as � even when the full sequence decodes correctly. Exact-match status refers to this original string, including whitespace and Unicode normalization.

This compares tokenization efficiency, not OCR recognition accuracy. A smaller token count does not establish a better trained model. Future experimental tokenizers can be added to the variant registry and selected above.

Full page with the selected block highlighted