LLMs

Amazon SageMaker HyperPod now supports model caching to accelerate inference autoscaling by pre-loading model weights and container images onto cluster nodes. This optimization addresses cold start bottlenecks in large-scale LLM workloads by storing weights on local NVMe and pre-pulling images, reducing pod startup time from tens of minutes to seconds. The feature targets high-latency scenarios in chat assistants, agentic pipelines, and RAG applications where network downloads previously delayed traffic serving.

• Reduces pod startup from minutes to seconds by bypassing network pulls from S3/FSx
• Stores model weights on local NVMe for faster read access during scale-out events
• Pre-pulls container images to eliminate image download latency during autoscaling
• Critical optimization for large models where cold starts cause significant traffic delays
• Applies to LLM inference workloads including RAG, agentic pipelines, and chat bots

This paper addresses the unreliability of LLMs in arithmetic by introducing a Program-Solve interface for clinical calculators. Rather than performing calculations internally, the model generates case-specific Python code executed by a restricted local solver. Evaluated on MedCalc-Bench Verified, this approach aims to provide deterministic accuracy compared to direct model inference or hardcoded functions.

• Shifts LLM role from calculator to code generator for deterministic execution.
• Mitigates hallucination risks in clinical arithmetic via external Python execution.
• Evaluated on 1,100 cases across 55 calculators using Qwen2.5 models.
• Alternative to maintaining large libraries of hardcoded validated functions.

A new arXiv paper explores how LLM agents can autonomously prepare for unknown tasks by inspecting available tools and data before deployment. Unlike existing methods that rely on task examples or feedback, this approach allows agents to construct reusable resources like indices or scripts without prior knowledge of the downstream workload. The study formalizes this 'task-agnostic environment preprocessing' to enable more flexible adaptation in open-ended settings.

• Agents can build indices and scripts without task-specific training data.
• Reduces dependency on evaluation feedback for initial environment setup.
• Enables autonomous preparation for unknown downstream task distributions.
• Moves beyond fixed preparation strategies tied to specific environment types.

Amazon Bedrock now supports OpenAI's GPT-6 Astra, marking its general availability. The model offers enhanced reasoning, professional content generation, and advanced browser automation capabilities. It features a 1 million input token context window and can align outputs with specific organizational standards. Users can access the model via Bedrock APIs or integrated tools like ChatGPT Work and Codex.

• GPT-6 Astra is now GA on Bedrock for production workloads.
• Supports up to 1M input tokens for large document analysis.
• Enables autonomous agents and advanced computer use tasks.
• Can be invoked via Bedrock APIs or ChatGPT Work integration.

Amazon Bedrock Managed Knowledge Base now exposes CheckIngestedDocumentAcl and GetIngestedDocumentAcl APIs alongside console support for auditing document permissions. These tools allow engineers to self-service diagnose why specific users fail to retrieve expected documents in ACL-enabled sources. By verifying user access or retrieving full ACLs, teams can pinpoint misconfigurations without opening support cases.

• Use CheckIngestedDocumentAcl to verify if a specific user can access an ingested document.
• Use GetIngestedDocumentAcl to retrieve and audit the full ACL attached to a document.
• Self-service debugging reduces reliance on AWS Support for permission troubleshooting.
• Console support complements APIs for easier visual inspection of access policies.
• Critical for maintaining data governance in RAG pipelines with strict access controls.