Prism-Δ: Differential Subspace Steering for Prompt Highlighting in Large Language Models
Abstract
Prompt highlighting allows users to designate specific input spans that large language models (LLMs) should prioritize during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose Prism-Δ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. Prism-Δ is compatible with FlashAttention and adds negligible memory overhead.
