One block, two tokenizers.
See where Indic text fragments—and where a vocabulary helps.
Only tokenization runs. No OCR model is called.
——Choose a block
Original transcription
Token inspector
Click a colored piece
Token ID, vocabulary spelling, character offsets, and byte details appear here.
Colors identify token boundaries, not semantic categories. When several byte-level tokens share one Unicode character, its span uses a striped background and a ×N badge; click to inspect every token. Token pieces may decode individually as � even when the full sequence decodes correctly. Exact-match status refers to this original string, including whitespace and Unicode normalization.
This compares tokenization efficiency, not OCR recognition accuracy. A smaller token count does not establish a better trained model. Future experimental tokenizers can be added to the variant registry and selected above.