LLMs

Akashic addresses the inefficiency of replaying full conversation histories in LLM agents by organizing context into bounded chunks and modeling semantic relationships between them. This approach preserves cross-chunk evidence without repeatedly rewriting the entire history, reducing prefill costs and context limit exceedances. The system further employs hardware-software co-design to co-locate likely co-retrieved chunks, optimizing memory access patterns for improved serving efficiency.

• MemAttention chunks context to avoid full-history replay, lowering prefill costs.
• Semantic modeling across chunks preserves relevant evidence without bloat.
• Hardware-software co-design co-locates frequently retrieved chunks for speed.
• Mitigates context limit issues and quality degradation from irrelevant content.

This paper introduces a standardized, workload-aware benchmark to compare KV-cache compression techniques that previously suffered from inconsistent evaluation metrics. It tests mechanisms like KIVI, TurboQuant, SnapKV, and CaM on Llama-3.1-8B and Mistral-7B across QA and summarization tasks. The study measures trade-offs between task quality, throughput, time-to-first-token, and compression ratios under long-context constraints.

• Standardized evaluation enables direct comparison of quantization, pruning, and merging methods.
• Tests cover multi-document QA, single-document QA, few-shot learning, and summarization.
• Base models evaluated are Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3.
• Key metrics include task accuracy, mean throughput, TTFT, and realized compression ratio.
• Results highlight trade-offs between serving performance and output quality for long contexts.

Research identifies 'thinking-induced hallucination' where large reasoning models overwrite correct direct answers with incorrect ones during explicit reasoning phases. The study frames this as a 'thinking residual' that can either recover missing knowledge or introduce unsupported associations. The authors propose Mixed-Mode Advantage Regularization to mitigate this specific failure mode in factuality-oriented question answering.

• Explicit thinking traces can actively degrade performance by overturning initially correct answers.
• Thinking-induced hallucination stems from a residual effect that introduces unsupported associations.
• Mixed-Mode Advantage Regularization is proposed to stabilize factual accuracy during reasoning.
• Factuality QA requires balancing knowledge recovery against the risk of reasoning drift.

MemDefrag addresses performance drops in LLM latent memory systems caused by positional encoding misalignment and the inability to isolate specific memory fragments. The authors discovered that middle transformer layers naturally concentrate attention on target fragments, providing a reliable tracing signal. This insight allows the method to refine memory updates without requiring additional training data.

• Latent memory updates suffer from positional encoding misalignment and lack of fragment tracing.
• Middle transformer layers show high attention density on target fragments, serving as a natural tracer.
• MemDefrag leverages this density to improve memory retrieval and update stability.
• No extra training required; exploits inherent model architecture properties.

Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), separating LLM inference phases onto dedicated GPU pools. The system transfers key-value caches between these pools via Elastic Fabric Adapter and GPU-Direct RDMA. This architecture prevents long-context prefill tasks from stalling token generation for concurrent decode requests, addressing latency and throughput issues in mixed traffic scenarios.

• Separate prefill and decode workloads onto distinct GPU pools to eliminate resource contention.
• Use EFA and GPU-Direct RDMA for low-latency KV cache transfer between phases.
• Improve consistent per-token latency for chat, agentic, and RAG pipelines.
• Reduce need to over-provision single-phase capacity to protect against stalls.