LLMs

An audit of a production text-to-SQL pipeline reveals that the deployed gpt-4o-mini judge performs poorly, agreeing with human annotators at only 0.04 Cohen's kappa on a disagreement-enriched dataset. The primary failure mode is identified as GRADE-HALLUCINATION, causing the model to over-flag 77.1% of human-verified faithful cases. Replacing the judge with a self-hosted Qwen3.6-27B model improves agreement to 0.72 kappa while reducing costs by a factor of 300 compared to premium alternatives.

• Deployed LLM judges require empirical validation; assumed accuracy can be dangerously low.
• gpt-4o-mini showed near-random agreement (kappa 0.04) on enriched disagreement sets in this context.
• GRADE-HALLUCINATION is a specific failure mode causing excessive false positives in faithfulness checks.
• Self-hosted Qwen3.6-27B matches Claude Opus 4.7 quality (kappa ~0.71) at 1/300th the cost.
• Ensembling strategies were noted but the excerpt cuts off before detailing their efficacy.

This paper critiques existing KV cache eviction methods like SnapKV for relying too heavily on mean attention scores. It introduces a unified scoring function that incorporates attention dispersion and redundancy penalties similar to maximal marginal relevance, all without requiring additional forward passes. The authors further demonstrate that optimal eviction parameters should vary by network depth, using a reparameterized search to find ideal profiles.

• Adds attention dispersion and redundancy to standard mean attention scores for better token ranking.
• Implements MMR-like penalties for token similarity without incurring extra forward pass costs.
• Shows that eviction coefficients should vary by layer depth rather than using fixed global values.
• Validated on 16 English LongBench datasets using Mistral-7B with constrained compute budgets.

Researchers introduce HARDEN, a method that uses constrained evolutionary search to make existing evaluation benchmarks more difficult without changing the expected answers. By mutating inputs along domain-specific complexity axes while enforcing feasibility constraints like semantic preservation, the technique creates harder test cases. Tests on Qwen3.5 models across FinQA, PubMedQA, and ContractNLI show significant accuracy drops compared to standard baselines, revealing deeper model weaknesses.

• HARDEN adapts existing eval inputs to be harder while keeping ground-truth answers fixed.
• Feasibility constraints ensure new cases remain realistic and semantically valid.
• Qwen3.5 models saw up to 49.9% accuracy drop vs single-pass baselines.
• Current curated benchmarks likely underrepresent real enterprise deployment complexity.

Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native add-on that replaces round-robin balancing with real-time inference-signal routing. It deploys on existing HyperPod infrastructure without application changes, using an Envoy endpoint and model-aware router. This approach reduces first-token latency by up to 82% and drastically lowers p99 TTFT in mixed-hardware scenarios.

• Deploys as an EKS managed add-on with zero client-side changes
• Routes traffic based on model name in the request body
• Reduces first-token latency by up to 82%
• Cuts p99 TTFT by 97-98% in burst traffic
• Serves multiple models from a single private endpoint

Amazon Bedrock Managed Knowledge Base now includes native connectors for Salesforce and Zendesk, eliminating the need for custom ingestion pipelines. Users can sync knowledge articles, community posts, and metadata by simply providing instance credentials. The service handles crawling and incremental synchronization automatically, enabling AI agents to leverage existing support data.

• Native connectors replace custom ETL pipelines for Salesforce and Zendesk data.
• Supports incremental sync of articles and community posts to keep RAG grounded.
• Metadata extraction is automated, simplifying agent context management.
• Reduces engineering overhead for building support-focused AI assistants.