Prism-Δ: Differential Subspace Steering for Prompt Highlighting in LLMs

The idea

Attention has two channels; prior methods steer only one.

Prompt highlighting asks a model to prioritize a marked subset of tokens. An attention head's output is shaped by two functionally distinct channels, and prior methods act on only the first.

Routing channel · Keys

Where to look

Key vectors decide the attention weights α — which tokens get read. Prior highlighting methods edit only this channel.

Content channel · Values

What is carried

Value vectors decide what information flows through, once a token is attended to. Routing a model to a token does nothing to enrich what that token transmits.

Measuring both channels under contrastive conditions, we find Value shifts are comparable in magnitude to Key across all five models, with roughly half of all heads carrying significant Value signal — signal that Key-only methods leave on the table. The two channels even peak at different depths.

Per-head Key vs Value shifthover a head

L1L36

Each dot is one attention head (Qwen3-4B · 288 heads). Dashed lines = activation threshold δ_min=0.08; K and V are only weakly correlated (r≈0.34), so the channels carry complementary signal.

Layer-wise mean shiftKey Value

Key signal peaks in the middle layers; Value peaks late — a division of labor across depth. Hover a layer for values.

Highlighting in action

The real tasks — and how steering flips them.

These are the four benchmarks we evaluate on, each shown in its real input format: a span the model should prioritize (a new fact, a biography, an instruction, the gold passage). Left alone the model answers from habit or gets lost among distractors; highlighting that span with Prism-Δ steering flips the answer. Full, at-scale numbers are in Results.

Input · exactly as the model reads it

Vanilla—✗ wrong

becomes

With Prism-Δ—✓ correct

The method

Split the light: keep only what discriminates.

Naively contrasting relevant vs. irrelevant contexts mixes true signal with shared structural patterns present in both. Prism-Δ decomposes the difference of cross-covariances, so shared directions cancel — like a prism dropping the white and keeping the spectrum.

Attention · two channels

$$\text{output}_i=\sum_j \underbrace{\operatorname{softmax}\!\big(\mathbf{q}_i^\top\mathbf{k}_j/\sqrt{d}\big)}_{\alpha_{ij}\ \text{routing}}\ \cdot\ \underbrace{\mathbf{v}_j}_{\text{content}}$$

Differential cross-covariance

$$\Omega_\Delta=\mathbf{H}^\top(\mathbf{H}^{+}-\mathbf{H}^{-})/N=\Omega^{+}-\Omega^{-}\ \xrightarrow{\ \text{SVD}\ }\ P,\;w_{\ell,h}$$

Proposition · (a)

Maximum discriminative energy

The top-k left singular vectors of Ω_Δ provably capture the subspace that maximizes the positive–negative separation (Eckart–Young).

Proposition · (b)

Shared directions vanish

Any direction that co-varies equally under both conditions maps to zero in Ω_Δ — structural redundancy is removed automatically, with no distributional assumptions.

Per-head weighting

Continuous, not binary

A softplus weight lets weak-but-useful heads contribute at reduced strength, instead of a hard on/off gate.

Projection structure · layer 20, head 3

Independent projections P⁺ (rank 89) and P⁻ (rank 39) share overlapping column spaces (tr(P⁺P⁻)=1.31) — structural redundancy. The differential projection P_Δ targets the discriminative subspace directly. Toggle to compare.

Direction consistency across heads & layersΩ⁺ Ω_Δ

Cross-head cosine similarity per layer

Cosine similarity with the next layer

Ω⁺ directions are highly similar across heads (dominated by shared structure); Ω_Δ directions are nearly independent — the differential decomposition recovers head-specific, discriminative directions.

Dual-channel steering · at inference, for each highlighted token

$$\mathbf{k}_j'=\mathbf{k}_j+g_K\,w_{\ell,h}^{K}\,P_K\,\mathbf{k}_j,\qquad \mathbf{v}_j'=\mathbf{v}_j+g_V\,w_{\ell,h}^{V}\,P_V\,\mathbf{v}_j$$

Prism-Δ steers the routing channel (g_V=0); Prism-ΔV steers both, recovering content-channel signal that reduces the fluency cost of steering.

Algorithm · offline learning + online steering

Extract Key/Value reps under neutral / positive / negative conditions → H, H⁺, H⁻
Differential cross-covariance Ω_Δ = Hᵀ(H⁺ − H⁻)/N
SVD → keep top-k until cumulative energy ≥ γ → projection P
Per-head discriminability D_ℓ,h → softplus weight w_ℓ,h
At inference, edit Keys (and Values) of highlighted tokens by g·w·P
Proceed with standard attention — fully FlashAttention-compatible

Offline · projection learning

1 · Contrastive extraction

HH⁺H⁻

Key & Value reps at the answer position under neutral / positive / negative conditions.

→

2 · Differential SVD

Ω_Δ = Ω⁺ − Ω⁻ SVD

Shared directions cancel; top-k singular vectors give projection P and per-head weight w_ℓ,h (softplus).

→

3 · Dual-channel steering online

k′ = k + g_K w P_K k

v′ = v + g_V w P_V v

Applied to highlighted tokens at inference — FlashAttention-compatible.

Results

Evaluated across four benchmarks and five base models.

We evaluate on four benchmarks and five base models, against four highlighting baselines (**-marked, PASTA, SPA, and SEKA). Our methods are competitive with or above the strongest baseline on 19 of 20 configurations, with relative gains up to +13.0%, and Prism-ΔV recovers content-channel signal that lowers the fluency cost of steering.

0/20

Configs matching or beating the best prior method

0

Peak relative gain over the best prior method

0

Gain on 30-passage long-context retrieval

0

Inference latency · +0.02 GB · FlashAttention ✓

Vanilla Baselines · **-marked / PASTA / SPA / SEKA Ours · Prism-Δ / Prism-ΔV best in row

Citation

Cite Prism-Δ

@misc{ge2026prism,
  title         = {Prism-$\Delta$: Differential Subspace Steering for Prompt
                   Highlighting in Large Language Models},
  author        = {Yuyao Ge and Shenghua Liu and Yiwei Wang and Baolong Bi and
                   Lingrui Mei and Jiayu Yao and Jiafeng Guo and Xueqi Cheng},
  year          = {2026},
  eprint        = {2603.10705},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2603.10705}
}