ECCV 2026 · Malmö, Sweden

Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning

A training-free way to make Vision-Language Models see more by looking at less — carving visual noise away from what the question actually needs.

Yuyao Ge1, Shenghua Liu1,†, Yiwei Wang2, Lingrui Mei1, Baolong Bi1, Xuanshan Zhou1, Jiayu Yao1, Jiafeng Guo1, Xueqi Cheng1
1Institute of Computing Technology, Chinese Academy of Sciences  ·  2UC Merced
† Corresponding author
Paper arXiv Code BibTeX Ask KIMI PaperWeekly
Scroll
The one-image story

Same image. Same model.
One carves the noise away.

Same model, three very different scenes — each flips from a wrong answer to the right one once CARVE masks the clutter. Drag any divider to compare.

Original scene CARVE focus
Original
CARVE
"shape through the cup's handle?"circle → star
Original scene CARVE focus
Original
CARVE
"brand of the whiskey on the right?"straight whiskey → George Washington
Original scene CARVE focus
Original
CARVE
"text under the Japanese characters?"highway bus → JR HIGHWAY BUS
See it work

Real cases — wrong answer, then right.

Each case below fooled the vanilla model. CARVE contrasts the attention, masks the clutter, and re-asks the same question — flipping the answer. Pick a case below.

Case 01 / 18—

—

1original scene
focus
Original scene
→
2contrasted attention
Contrasted attention
→
3focused crop
CARVE focus & re-infer
Vanilla model—✗ wrong
becomes
With CARVE—✓ correct
§1 · Failure to focus

From global scan to focused gaze — unless the scene is noisy.

We visualize Qwen2.5-VL-3B attention on TextVQA. Attention should refine layer by layer, narrowing onto the answer. Visual complexity is what breaks that convergence.

Attention maps across layers
Fig. 1 — Attention across shallow → middle → deep layers. Row 1: a simple scene that converges onto the target; rows 2–3: cluttered scenes whose attention stays dispersed even in the deepest layers.
Shallow layers

A wide global scan

Early layers spread attention almost uniformly across every visual token — high entropy, no commitment yet.

Deep · simple scene

It snaps into focus

For a clear target with few distractors, deep-layer attention narrows precisely onto the task-relevant region.

Deep · complex scene

"Confused where to look"

Rich textures and many similar objects keep attention scattered even at the deepest layers — and the answer is wrong.

One sample · every layer, left → right
Layer 1 · global scanshallow → deepLayer 28 · focused
§3 · Method

CARVE: contrast two attention maps, keep the difference.

A generic instruction ("describe the image") makes attention spread out — it mostly captures visual noise. A specific question adds the semantic signal on top. Divide one by the other and the noise cancels.

Interactive · keep ratio p
1.00 p

The full image — every pixel kept. Slide right to tighten the top-p attention threshold and watch CARVE carve away the noise.

1.0 · full0.1 · tight focus
CARVE three-stage workflow
Fig. 3 — Three passes: ① general attention (noise) · ② task attention (signal+noise) · ③ contrast → mask → crop → re-infer.
Algorithm · CARVE
  1. Run inference with question Q → collect task attention A(Q)
  2. Run inference with general instruction G → collect noise attention A(G)
  3. Contrast: Â = A(Q) / (A(G) + λ)
  4. Fuse layers & time steps with later-token weighting → score map S
  5. Threshold top-p, take top-K connected regions → mask M*
  6. Mask · crop · resize the image, then re-infer for the final answer
§4 · Results

Training-free, yet consistently better.

Across seven benchmarks and four VLMs, CARVE lifts perception the most where it matters most — fine-grained, cluttered scenes. The smaller the model, the larger the gain.

0
Benchmarks evaluated
0
Extra training · fully training-free
0
V* points · LLaVA-1.5-7B (38.7 → 66.5)
0
Peak relative gain
Vanilla CARVE

Numbers are accuracy (%) with the tend configuration, layer range [20,25]. From Tables 1 & 3 of the paper.

Citation

Cite CARVE

@inproceedings{ge2025focusingcontrastiveattentionenhancing,
  title   = {Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning},
  author  = {Yuyao Ge and Shenghua Liu and Yiwei Wang and Lingrui Mei and
             Baolong Bi and Xuanshan Zhou and Jiayu Yao and Jiafeng Guo and Xueqi Cheng},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year    = {2026},
  url     = {https://arxiv.org/abs/2509.06461}
}