Continued byte-level BPE on stock Qwen3.5 · corpus 2.0B chars, 22 Indic languages + Latin · run v3 (figure descriptions removed, 12 HTML table tags as single tokens) · 2026-10-24 · model directory Qwen3.5-4B-cbpe32k-v3-loc-div adds 1,001 location tokens and 3 block tokens (2026-10-28)
Problem. Qwen3.5 splits Indic text into many small pieces. On our real OCR dev text it needs 61.1 tokens per 100 characters for Indic languages. Sarvam needs 38.0. For Santali (Ol Chiki) Qwen needs 211, because each character falls back to raw UTF-8 bytes.
What we did. We kept Qwen's tokenizer exactly as it is (248,044 pieces, 247,587 merge rules, all IDs) and continued its own BPE training on our text. Each new step adds one merge rule and one token. After 32,768 steps, plus 12 HTML table tags as single tokens: 36.3 tokens / 100 chars, which is 59% of stock Qwen and 95% of Sarvam.
No neural network was trained. BPE is a counting procedure. It ran on CPUs in about 5 minutes. The model weights are unchanged.
Navigate with ← → or the buttons below. Slides 4 and 5 are interactive.
Each coloured box is one token. Blue-outlined boxes in the third row are new tokens (ID ≥ 248,077):
| 248,077–248,088 | 12 HTML table tags: <table> <tr> <td> … |
| 248,089–280,856 | 32,768 continued-BPE tokens |
| 280,857–281,857 | 1,001 location tokens <|loc_0|> … <|loc_1000|>, one per coordinate value (A/B pilot against digits) |
| 281,858–281,860 | 3 block tokens of the output format: <div data-bbox=" · " data-label=" · </div> |
The two "Output block" tabs show one target line of the training data. With digit coordinates it is 33 tokens without the block tokens and 24 with them; with location tokens, 22 and 13.
Pieces that show � are partial UTF-8 characters: byte-level BPE can cut inside a character. Decoding a full token sequence still gives the exact text.
Ġ = space, Ċ = newline).The regex, verbatim from Qwen3.5's tokenizer.json:
(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+
[\p{L}\p{M}]+ keeps vowel signs and viramas (category M) inside the word. That is important for Indic text: प्रश्न stays one pretoken. Note [^\r\n\p{L}\p{N}]?: one leading non-letter can join the word. That is why ।अर्थ (no space after a danda) became one pretoken in the synthetic data, and why we added the space fix.
\frac sometimes looks like two pieces\frac, \times, \sqrt, \alpha are already single tokens in Qwen's vocabulary. The split comes from the regex on the previous slide: punctuation directly before the backslash joins it.
| text | our tokenizer (v3) | if LaTeX commands were added tokens |
|---|---|---|
a \times b | a · ␣\ · times · ␣b (4) | a · ␣ · \times · ␣b (4) |
$$\frac{1}{2}$$ | $$\ · frac · … (7) | $$ · \frac · … (7) |
x=\sqrt{2} | x · =\ · sqrt · … (6) | x · = · \sqrt · … (6) |
(\alpha+\beta) | (\ · alpha · +\ · beta · ) (5) | ( · \alpha · + · \beta · ) (5) |
The punctuation before the command needs a token either way, so only the boundary moves. Measured on real OCR dev text with the 300 most frequent commands added as single tokens: arXiv −0.9% tokens, NCERT and IndicDLP −0.01%. Qwen also learned all its LaTeX in the current segmentation. Decision: no LaTeX tokens (unlike the 12 table tags, which saved 8.5% on table pages).
Type any text. This is the real Qwen regex and the real byte mapping (step 4 needs the 247K-rule merge table, so it is shown on the next slide).
The 220 most frequent multi-piece Hindi or Santali pretokens from a slice of our corpus, with their real frequencies and their real stock-Qwen segmentation as the start state. Each step: count every adjacent pair, weighted by word frequency; merge the most frequent pair everywhere; append that merge as the next rule.
This toy run uses 220 words. The real run used 5.13M distinct pretokens (frequency ≥ 2) from 427M pretoken occurrences, so its rules differ. The algorithm is the same: upstream tokenizer-extension (Purason et al., EACL Findings 2026), commit d7fa143.
The first rules join byte pairs that start Meetei Mayek (ê ¯ = bytes EA AF) and Ol Chiki (á ± = E1 B1) characters. Qwen has no merges for these scripts, so the most frequent pairs in the whole corpus are there. Then come whole Ol Chiki and Meetei Mayek characters, and frequent Indic syllables. (Table tags no longer appear here: they are added tokens and BPE never sees them.)
Log scale. Step 1 pair occurs 24.5M times; step 32,768 still 2,589 times. Every added token is frequent in the corpus, and 32,715 of the 32,780 new IDs occur on held-out dev text.
Adding a string to the vocabulary does not make BPE produce it. BPE emits a token only if a chain of merge rules builds it from Qwen's existing pieces in that order. Many foreign tokens are unreachable and stay as dead embedding rows. Sarvam also uses different whitespace (▁) and pretokenization.
Each new rule merges two pieces that Qwen's current rules actually produce on our text, so every new token is reachable by construction. The ordered rule list is nested: the first K rules form a valid smaller tokenizer, so one run gives 1K…32K.
Overlap after the fact. 22,560 of our 32,768 BPE tokens (69%) are also in Sarvam's vocabulary. Data-driven learning recovers most of Sarvam's useful Indic units and adds what our OCR targets need: LaTeX pieces ($$\, {array) and ।\n.
Language weights. Share ∝ vlα with α = 0.5, where vl = characters of cleaned OCR training text in language l. Each Indic language gets at least 2%. The Latin group (English, arXiv, archives) is fixed at 8%. α < 1 flattens the distribution: Hindi has 134× more OCR text than Manipuri but gets 4.9× its share.
Sources. Real OCR targets (9 cleaned sources) up to half of each language's budget; synthetic100m text fills the rest. Urdu from gemini_urdu, Kashmiri from synth_kashmiri.
Cleaning. Space fix on synthetic text, exact dedup across all text, and removal of any text equal to a val/test page or block. Test pages are never read for training.
Dev = 400 val pages per language (never used to choose the merges). 0 round-trip failures over 54M chars of dev text. Gains differ by script: Santali 211 → 42, Odia 76 → 35, Urdu 44 → 36 (Qwen already handles Arabic script well).
<table> <thead> <tbody> <tr> <th> <td> + closing) as non-special added tokens, IDs 248,077–248,088: matched before the regex, so <td>उत्तर</td> is 3 tokens, not 6. Pages with tables: 8.5% fewer tokens.tokenizer_config.json.Qwen2Tokenizer ignores tokenizer.json's regex and uses the old Qwen2 one without \p{M}, which cuts Indic words at every vowel sign, even for stock Qwen3.5. Our model directory sets PreTrainedTokenizerFast; validated: 0 mismatches on 3,450 dev texts.tokenizers library re-numbered special tokens that are missing from the model vocab, which moved <|video_pad|>. Fixed by pinning them; verify_ids.py checks every ID.base_models/Qwen3.5-4B-cbpe32k-v3, embeddings 248,320 → 280,896 rows (vocab 280,857, padded to 64). Untrained, it generates exactly like stock Qwen. New row = mean of the stock-Qwen rows of the pieces that stock Qwen produces for that string (3.3 pieces on average).