Unified Qwen3.5-4B OCR · model base_models/Qwen3.5-4B-cbpe32k-v3-loc · 2026-10-24
Every block in the answer carries a box: <div bbox="x1 y1 x2 y2" label="…">…</div>. We will test two ways to write the four numbers:
152 138 957 258, about 16 tokens per box.<|loc_152|><|loc_138|><|loc_957|><|loc_258|>, exactly 4 tokens per box.We added 1,001 tokens (<|loc_0|> … <|loc_1000|>) to the tokenizer. A matched pilot will decide which format the main run uses.
This deck explains: (1) the 0–1000 grid, (2) why new rows need a careful start, (3) the start we chose, (4) how training makes them precise.
The image is not made square. Only the numbers are normalized: 0 is the left/top edge, 1000 the right/bottom edge.
x_grid = round(1000 · x_px / W) y_grid = round(1000 · y_px / H)
x_px = x_grid · W / 1000 y_px = y_grid · H / 1000
The 6.29 MP cap shrinks width and height by the same factor, so the grid numbers are identical for the original scan and for what the model sees. Precision: 0.1% of a side (≈3 px on a 3,000 px page).
Real encodings of one block header by our tokenizer (each box = one token; blue outline = coordinate token).
bbox_2d digits on 0–1000).Each token has one row in the embedding table (2,560 numbers). Qwen3.5 ties its embeddings: the same row is used twice.
<|loc_152|>, it receives this row.So a new row's start value decides both what the model "sees" and what it tends to predict, before any training. Two rows that start identical are always predicted with identical probability.
For BPE tokens we averaged the rows of the pieces (e.g. प्रश्न = mean of its 3 Qwen pieces). For numbers that fails: an average ignores order.
Numbers whose digit-average row would be identical to it:
Over all 1,001 values, only distinct rows would exist. The model could not tell 152 from 521 at the start.
row(k) = m + noise_k
m = (row("0") + row("1") + … + row("9")) / 10 // "this is a number"
noise_k = random direction, length 0.3 × typical row length, fixed seed
Considered and dropped: a smooth sine-wave position signal (built first; it had ripples: values 500 apart started more similar than values 50 apart, and it needed tuning), a straight line from row(0) to row(1000) (with tied rows the model could then only prefer 0 or 1000 at the start), and hidden states from Qwen (most complex, small expected gain).
Plain cross-entropy puts all target probability on the exact token. Predicting 153 for 152 is then as wrong as predicting 900. Gaussian targets spread a little probability over near values, so near misses cost less.
Only coordinate positions use these targets; text tokens keep plain cross-entropy. σ ≈ 2–3 grid units; the curve is cut at 0 and 1000 and renormalized.