[EMNLP 2026 Main]
*Corresponding author: shamanthak@asu.edu
†Partial work done while at Arizona State University
Unified vision-language models (VLMs) that tokenize images through a vector-quantized (VQ) codebook routinely hallucinate objects on grounded yes/no benchmarks, yet existing decoding-time fixes treat this as generic miscalibration without an architectural account. Using activation patching across twenty-five models spanning eight LLM families, we identify an early-layer (L0) attention routing circuit shared across VQ-tokenized VLMs and propose a three-gate diagnostic that distinguishes the models carrying it. The diagnostic isolates ten positive models (five natural unified-VQ VLMs across three LLM families and five induced variants) and rejects the remaining fifteen. A single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) installs the circuit, while a matched-compute MLP control does not, isolating vector quantization as the source of the pathological signal; the routing pathway that carries it is one that the backbone already provides. Against tuned baselines, DoLA wins on binary calibration, but only L0 ablation reduces object hallucination in open-ended generation (CHAIRi -31% relative; tuned DoLA and VCD leave it unchanged or worsen it). These results recast object hallucination in unified VQ VLMs as a property of architecture and pretraining, and yield a targeted intervention that mechanism-agnostic decoding cannot replicate.
We damage the image and then look for where the damage shows up inside the model. For each of 500 paired records we run the model twice, once on the clean image and once on a damaged one, and save what happens at every layer. Then we run the damaged version again, but copy one piece of the clean run back in at a time, and measure how much of the right answer comes back. Records come in pairs, one expecting yes and one expecting no, which is what lets us put error bars on everything.
Models scale their image features very differently, by more than a factor of ten. One fixed amount of noise would barely bother some models and destroy others. So for each model we tune the noise until the damaged features are about half as similar to the clean ones. We do this using only the input features, never the model's answers, so nothing about the outcome leaks into the setup.
How much noise a model needs sorts the group into three kinds: ones whose projector blows up the scale of image features, ones that feed images in as continuous features, and ones that put image tokens in the same table as text tokens at text scale. Gate A2 uses two different thresholds, and which one a model gets comes from that grouping. We fixed the grouping before looking at any results.
We measure how far the model's internal state drifts away from the clean run, layer by layer. Then we ask two things about that curve.
Does it start? The drift has to reach 0.4 somewhere in the first eight layers. That is where we think the routing happens. Does it carry? It has to reach 0.4 again at layer 12 or later. If it does not, the early event faded out and never reached the answer.
The two ranges do not touch. There are four layers between them, so one bump cannot count for both. A model passes A1 only if both are true.
Models fail these in different ways. UniTok, Show-o, SEED-LLaMA, the matched MLP control and FSQ v2 fail the first question, because not much happens early. Emu3-Chat is the only model that fails the second one. It has a real early event, 0.435 at layer 0, but by the second half of the network it is down to 0.215.
VILA-U clears 0.4 in both ranges. The MLP control, trained on the same data for the same 2000 steps, clears neither, at 0.206 and 0.106.
The three dots on each row are its results for A1, A2 and A3. Click a model to see the drift curve behind its A1 numbers and the attention map behind its A2 number. Both come from the same arrays we used for the paper's tables.
Why we need all three gates. Each one on its own can be fooled.
A model can route image tokens through L0 without anything harmful going down that route, so A2 alone is not enough. The matched MLP control does exactly this. Its L0 mass is 25.52, because a healthy Vicuna-7B already sends early image attention there. But it fails A1, and switching L0 off does nothing to it.
A1 alone is not enough either. Any model that looks hard at the image will carry image damage forward. Janus-Pro does, and it does not quantize images at all.
Nor is A3. Qwen-VQ has a badly collapsed codebook and passes both A1 and A3, and it still has no circuit. It fails on attention mass alone. UniTok and Emu3 are the mirror case: both clear the A2 threshold and fail A1, from opposite sides.
How much do the thresholds matter? We moved one gate at a time across the 19 models with measurable values. A3 can sit anywhere from 60% to 95% without changing a single verdict. A2 can sit anywhere from 8 to 15, and first changes at 18, where VILA-U drops out because it has the lowest mass of any passing model at 17.50. A1 is pinned from both sides. Lower it to 0.35 and FSQ v2 gets in at 0.379. Lower it to 0.30 and UniTok gets in at 0.347. Raise it to 0.45 and FSQ v1 drops out, and its value is 0.449997, so that edge is very close.
We start from LLaVA-1.6 with the language model and image encoder frozen, and train two versions of the part that converts the image. One quantizes through a codebook, one does not. Same data, same 2000 steps, same settings. Only the quantizing one installs the circuit.
The middle panel is the one that matters. Both versions end up with about the same attention mass at L0. So the route was already there in the backbone before we changed anything. What quantizing adds is the bad signal that travels down that route, plus the drift that carries it forward.
Going the other way does not work in the same budget. We trained a normal continuous adapter on top of VILA-U, which was already pretrained with quantization. After the same 2000 steps A1 still passes, L0 mass has gone up rather than down, from 17.50 to 20.53, and the POPE yes-rate has gone up from 81% to 99.2%. We record it as a fail on A3, since a continuous adapter leaves no codebook to test, but the gates it does pass are what make the point.
Codebook size does not decide which way the circuit pushes. All four sizes end up using only about 80 to 160 codes no matter what we set, and they still push in different directions. K = 4096 is the one we cannot explain. We checked its codebook entry by entry and it is strongly yes-biased at 0.836, which is essentially the same as K = 16384 at 0.864 and K = 65536 at 0.870. Switching L0 off still moves it the opposite way. The oddity is in what the intervention does, not in the codebook.
A collapsed codebook is not the cause either. FSQ v1 cannot collapse by design and gets the circuit anyway, with the direction flipped. FSQ v2 flips the behaviour the same way, yes-rate up 53.3 points and AMBER down 19.8, but has no early event we can detect: 0.379, with the drift first clearing 0.4 only at layer 14. So the two versions agree on what the model does and disagree on why. Put that next to Qwen-VQ, which collapses without getting the circuit, and what is left is the quantization step itself. What the surviving codes learned during training decides which way it pushes.
Yes/no benchmarks. Tuned DoLA wins.
Full captions. Only switching L0 off gets below the baseline.
We tuned VCD over 16 settings and DoLA over 20, and report the best one for each benchmark. On POPE and AMBER, tuned DoLA reaches 0.794 and 0.795, which beats everything we tried. When the model only has to say yes or no, a tuned decoding fix can match or beat switching a layer off.
When the model writes a full caption, the order flips. Only switching L0 off lowers CHAIRi, from 10.3% to 7.1%. This is not just captions getting shorter. If we cut each baseline caption down to the length of its ablated version, CHAIRi only falls to 9.53, which leaves 2.40 points that come from the intervention itself.
The effect lands where the problem is. Images the baseline hallucinates on more improve more. Images with at least two made-up objects lose 1.59 of them on average, and the rest are unchanged. Counting by object category, made-up mentions fall from 310 to 192: 36 categories get better, 11 get worse, 31 stay the same.
A partial dose does not carry across. Scaling the L0 output down instead of zeroing it gives +8.67, +8.17 and +12.30 points on POPE at 0.25, 0.50 and 0.75, so a partial dose beats a full one on yes/no. On captions, only the full version gets below the baseline. Every partial dose makes per-object hallucination worse and recall lower. It shortens the caption without grounding it.
On VILA-U, switching L0 off gives +4.23 on POPE and +16.9 on AMBER. Switching off L1, the layer that writes downstream, gives +11.60 and +23.37. VTI gives the largest AMBER effect at +30.23. The same L0 ablation does nothing to LLaVA-1.6, at -0.7 on POPE, and wrecks Qwen2.5-VL, at -34.3 on POPE and -20.1 on AMBER. Getting opposite results on different architectures is hard to explain unless the mechanism is real.
All 500 COCO images from the CHAIR run, with VILA-U's caption under each setting. We mark every COCO object a caption names: red if the object is not really in the image, green if it is. Hover a mark to see the category it maps to. The table below is recomputed from these caption files and matches the paper.
| Setting | CHAIRs | CHAIRi | Recall | Avg. length |
|---|
The filters include the cases that do not make the method look good. Switching L0 off makes the model loop on 62 of the 500 images, and captions get shorter, from 158 tokens to 78. Recall drops from 58.5% to 51.6%. Tuned VCD loops on 130. The looping flag is our own check for repeated phrases, not a CHAIR number. This kind of failure is why every intervention in the paper comes with a sanity check, and why Liquid and Anole count as evidence for the gates only: switching L0 off makes them stop answering properly.
Coming soon!