LLMs

AdaptiveSpec introduces a training-free speculative decoding method that adapts both token verification and draft tree shape from internal model signals. Unlike prior work that relaxes these decisions in isolation, this approach uses a per-step margin rule to potentially accept mismatched drafts when probability ratios favor the target. This method aims to improve inference speed without requiring additional training or fixed budget constraints.

• Eliminates fixed token-match verification by using dynamic per-step margins.
• Adapts draft tree shape dynamically based on internal decoding signals.
• Remains training-free, leveraging existing model outputs for optimization.
• Combines lossy verification and adaptive shaping for better throughput.

OpenAI published an engineering deep-dive into GPT-Live, detailing how the system sustains continuous voice interactions by decoupling latency-sensitive media processing from broader application logic. The architecture isolates the media pipeline and inference loop in the live path, while delegating tool use, persistence, and other async tasks across an RPC boundary. This separation allows the core inference cycle to remain responsive without being blocked by heavier backend operations.

• Separates real-time media/inference from async app logic to minimize voice latency
• Uses an asynchronous RPC boundary for delegation, tool use, and persistence
• Keeps the live path lightweight to maintain natural conversation flow
• Provides architectural patterns for building stateful voice agents at scale

Long-output reasoning models frequently bottleneck on KV cache memory, but existing compression methods fix the total capacity per request. GrowPage addresses this by treating KV capacity as a dynamic runtime resource that adjusts to varying demands. The framework uses lightweight dual-timescale query summaries to track both recent and long-term attention behaviors. This allows the system to allocate memory more efficiently based on the evolving needs of each generation.

• Moves beyond fixed per-request KV budgets to dynamic, on-demand allocation.
• Uses dual-timescale summaries to capture recent and long-term attention patterns.
• Reduces memory bottlenecks in long-output reasoning workloads.
• Adapts capacity during decoding as attention demands evolve per request.

This study reveals that exponential-moving-average (EMA) temporal aggregation in decoding-time KV cache eviction renders most scorer modifications functionally equivalent. While value-norm and entropy variants maintain high correlation with attention mechanisms, metrics like KeyDiff and key norm significantly degrade performance by altering the eviction ranking. The research suggests that the aggregation strategy, rather than just the scoring function, is the primary driver of retention set stability.

• EMA aggregation stabilizes eviction sets, making diverse scorers appear similarly effective.
• Value-norm and entropy scorers preserve ranking integrity better than others.
• KeyDiff and learned scorers degrade performance by disrupting the eviction order.
• Layer weighting coupled with temporal retention drives the observed stability.
• Optimizing aggregation rules may yield better gains than tweaking individual scorers.

R2Adapter is a lightweight plug-in designed to optimize Hybrid RAG systems by dynamically routing and rewriting queries based on complexity. It addresses the latency and heuristic overhead of existing text-graph RAG methods by eliminating strong dependence on the underlying LLM for routing decisions. This approach aims to balance the efficiency of vanilla RAG with the reasoning capabilities of graph-based retrieval.

• Replaces heuristic/LLM-based routing with a lightweight adapter to reduce overhead.
• Dynamically handles query complexity, mixing text and graph retrieval efficiently.
• Mitigates latency issues common in pure graph-based RAG approaches.
• Decouples routing logic from the base LLM for more robust performance.