The AgentKVShift paper introduces a training-free method to reduce inference latency in LLM agents by reusing Key-Value (KV) caches for structured agentic memory. Unlike prior techniques designed for raw RAG passages, this approach uses probe-guided residual correction to handle metadata like summaries and tags. The method selectively recomputes only necessary tokens per retrieved memory unit, addressing the high prefill costs associated with re-encoding structured context.
- Reduces prefill latency by reusing KV states for structured agentic memory units
- Uses probe-guided residual correction to maintain accuracy without retraining
- Outperforms existing RAG-focused KV reuse methods on metadata-heavy contexts
- Eliminates full re-encoding of retrieved summaries, keywords, and tags