1 Training data← Back to the data viewer2 TokenizerCompareAuditExplainer slidesCoordinate tokens3 Image budget4 MP vs 8 MP4 NotesData processingAssamese ra checkSynthetic checkSmall held-out blocksData EDAJaigyan latticeJaigyan v4 re-readEmpty pagesManipuri: 4B vs pipeline5 TrainingEmbedding warm-upSFT ablationsSFT merges2B distillation0.8B distillationDistillation explainedRL planRL ablationsRL training

How we learned 32,768 new tokens from data

Continued byte-level BPE on stock Qwen3.5 · corpus 2.0B chars, 22 Indic languages + Latin · run v3 (figure descriptions removed, 12 HTML table tags as single tokens) · 2026-10-24 · model directory Qwen3.5-4B-cbpe32k-v3-loc-div adds 1,001 location tokens and 3 block tokens (2026-10-28)

Problem. Qwen3.5 splits Indic text into many small pieces. On our real OCR dev text it needs 61.1 tokens per 100 characters for Indic languages. Sarvam needs 38.0. For Santali (Ol Chiki) Qwen needs 211, because each character falls back to raw UTF-8 bytes.

What we did. We kept Qwen's tokenizer exactly as it is (248,044 pieces, 247,587 merge rules, all IDs) and continued its own BPE training on our text. Each new step adds one merge rule and one token. After 32,768 steps, plus 12 HTML table tags as single tokens: 36.3 tokens / 100 chars, which is 59% of stock Qwen and 95% of Sarvam.

No neural network was trained. BPE is a counting procedure. It ran on CPUs in about 5 minutes. The model weights are unchanged.

Navigate with ← → or the buttons below. Slides 4 and 5 are interactive.

Same sentence, three tokenizers

Each coloured box is one token. Blue-outlined boxes in the third row are new tokens (ID ≥ 248,077):

248,077–248,08812 HTML table tags: <table> <tr> <td> …
248,089–280,85632,768 continued-BPE tokens
280,857–281,8571,001 location tokens <|loc_0|> … <|loc_1000|>, one per coordinate value (A/B pilot against digits)
281,858–281,8603 block tokens of the output format: <div data-bbox=" · " data-label=" · </div>

The two "Output block" tabs show one target line of the training data. With digit coordinates it is 33 tokens without the block tokens and 24 with them; with location tokens, 22 and 13.

Pieces that show � are partial UTF-8 characters: byte-level BPE can cut inside a character. Decoding a full token sequence still gives the exact text.

How Qwen encodes text (we do not change this pipeline)

1 · NormalizeUnicode NFC. Composed and decomposed forms become one form.
2 · Pre-tokenizeA regex cuts text into chunks ("pretokens"): a word with its leading space, single digits, punctuation runs, newlines. BPE never merges across these cuts.
3 · BytesEach pretoken becomes its UTF-8 bytes. Each byte is shown as one printable symbol (Ġ = space, Ċ = newline).
4 · Apply mergesRepeat: of all adjacent pairs, apply the merge rule with the lowest rank (learned earliest). Stop when no rule applies.

The regex, verbatim from Qwen3.5's tokenizer.json:

(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

[\p{L}\p{M}]+ keeps vowel signs and viramas (category M) inside the word. That is important for Indic text: प्रश्न stays one pretoken. Note [^\r\n\p{L}\p{N}]?: one leading non-letter can join the word. That is why ।अर्थ (no space after a danda) became one pretoken in the synthetic data, and why we added the space fix.

Why \frac sometimes looks like two pieces

\frac, \times, \sqrt, \alpha are already single tokens in Qwen's vocabulary. The split comes from the regex on the previous slide: punctuation directly before the backslash joins it.

textour tokenizer (v3)if LaTeX commands were added tokens
a \times ba · ␣\ · times · ␣b (4)a · ␣ · \times · ␣b (4)
$$\frac{1}{2}$$$$\ · frac · … (7)$$ · \frac · … (7)
x=\sqrt{2}x · =\ · sqrt · … (6)x · = · \sqrt · … (6)
(\alpha+\beta)(\ · alpha · +\ · beta · ) (5)( · \alpha · + · \beta · ) (5)

The punctuation before the command needs a token either way, so only the boundary moves. Measured on real OCR dev text with the 300 most frequent commands added as single tokens: arXiv −0.9% tokens, NCERT and IndicDLP −0.01%. Qwen also learned all its LaTeX in the current segmentation. Decision: no LaTeX tokens (unlike the 12 table tags, which saved 8.5% on table pages).

Try it: steps 1–3 in your browser

Type any text. This is the real Qwen regex and the real byte mapping (step 4 needs the 247K-rule merge table, so it is shown on the next slide).

Pretokens

Bytes of each pretoken (the units BPE starts from)

Continued BPE, live on real words

The 220 most frequent multi-piece Hindi or Santali pretokens from a slice of our corpus, with their real frequencies and their real stock-Qwen segmentation as the start state. Each step: count every adjacent pair, weighted by word frequency; merge the most frequent pair everywhere; append that merge as the next rule.

Top pairs now (frequency-weighted)

Rules learned so far

Words (frequency · current pieces)

This toy run uses 220 words. The real run used 5.13M distinct pretokens (frequency ≥ 2) from 427M pretoken occurrences, so its rules differ. The algorithm is the same: upstream tokenizer-extension (Purason et al., EACL Findings 2026), commit d7fa143.

What the real run learned first

The first rules join byte pairs that start Meetei Mayek (ê ¯ = bytes EA AF) and Ol Chiki (á ± = E1 B1) characters. Qwen has no merges for these scripts, so the most frequent pairs in the whole corpus are there. Then come whole Ol Chiki and Meetei Mayek characters, and frequent Indic syllables. (Table tags no longer appear here: they are added tokens and BPE never sees them.)

Frequency of the chosen pair, by step

Log scale. Step 1 pair occurs 24.5M times; step 32,768 still 2,589 times. Every added token is frequent in the corpus, and 32,715 of the 32,780 new IDs occur on held-out dev text.

Why learn the rules instead of copying Sarvam's list

Copying vocabulary strings

Adding a string to the vocabulary does not make BPE produce it. BPE emits a token only if a chain of merge rules builds it from Qwen's existing pieces in that order. Many foreign tokens are unreachable and stay as dead embedding rows. Sarvam also uses different whitespace (▁) and pretokenization.

Continued BPE (what we did)

Each new rule merges two pieces that Qwen's current rules actually produce on our text, so every new token is reachable by construction. The ordered rule list is nested: the first K rules form a valid smaller tokenizer, so one run gives 1K…32K.

Overlap after the fact. 22,560 of our 32,768 BPE tokens (69%) are also in Sarvam's vocabulary. Data-driven learning recovers most of Sarvam's useful Indic units and adds what our OCR targets need: LaTeX pieces ($$\, {array) and ।\n.

The training text: which data, how much of each language

Language weights. Share ∝ vlα with α = 0.5, where vl = characters of cleaned OCR training text in language l. Each Indic language gets at least 2%. The Latin group (English, arXiv, archives) is fixed at 8%. α < 1 flattens the distribution: Hindi has 134× more OCR text than Manipuri but gets 4.9× its share.

Sources. Real OCR targets (9 cleaned sources) up to half of each language's budget; synthetic100m text fills the rest. Urdu from gemini_urdu, Kashmiri from synth_kashmiri.

Cleaning. Space fix on synthetic text, exact dedup across all text, and removal of any text equal to a val/test page or block. Test pages are never read for training.

Result on held-out real OCR text

stock QwenSarvamQwen + 32,768 (ours)

All Indic, tokens per 100 chars vs. tokens added

Dev = 400 val pages per language (never used to choose the merges). 0 round-trip failures over 54M chars of dev text. Gains differ by script: Santali 211 → 42, Odia 76 → 35, Urdu 44 → 36 (Qwen already handles Arabic script well).

Putting it into the tokenizer, and what comes next

Integration (done)
  • Stock vocab, merges, normalizer, regex: unchanged.
  • New merges appended after the 247,587 stock rules, so stock rules always win first.
  • 12 table tags (<table> <thead> <tbody> <tr> <th> <td> + closing) as non-special added tokens, IDs 248,077–248,088: matched before the regex, so <td>उत्तर</td> is 3 tokens, not 6. Pages with tables: 8.5% fewer tokens.
  • BPE tokens 248,089 … 280,856. IDs 248,044 … 248,076 stay Qwen's: 26 special tokens plus 7 audio/TTS IDs that Qwen reserves in tokenizer_config.json.
  • Trap found: transformers 5.x Qwen2Tokenizer ignores tokenizer.json's regex and uses the old Qwen2 one without \p{M}, which cuts Indic words at every vowel sign, even for stock Qwen3.5. Our model directory sets PreTrainedTokenizerFast; validated: 0 mismatches on 3,450 dev texts.
  • Bug caught: the tokenizers library re-numbered special tokens that are missing from the model vocab, which moved <|video_pad|>. Fixed by pinning them; verify_ids.py checks every ID.
Next
  • Done (v2, v3): figure descriptions removed from the BPE corpus (figure blocks: bbox + label only; descriptions become a separate task). 97.5% of v1 new tokens are unchanged; no description pieces remain.
  • Done: base_models/Qwen3.5-4B-cbpe32k-v3, embeddings 248,320 → 280,896 rows (vocab 280,857, padded to 64). Untrained, it generates exactly like stock Qwen. New row = mean of the stock-Qwen rows of the pieces that stock Qwen produces for that string (3.3 pieces on average).
  • Embeddings are tied (input = output), so a short adaptation stage must train the new rows before or with full fine-tuning.