Agentic AI

AWS has introduced aws-bench as a research preview to objectively measure how accurately and efficiently AI agents perform real-world tasks on AWS. The benchmark provides a public suite of test cases derived from actual AWS usage patterns, covering investigation, troubleshooting, and infrastructure creation. Each test case pairs natural language queries with specific cloud resource states and ground-truth answers, enabling consistent scoring and failure diagnosis for model providers and researchers.

• Use aws-bench to objectively score AI agent performance on AWS-specific tasks like troubleshooting and infrastructure setup.
• Access public test cases derived from real AWS usage to ensure reproducible and verifiable benchmarking results.
• Leverage defined resource states and ground-truth answers to diagnose agent failures and improve foundation models.
• Evaluate model providers and internal agents using a consistent framework tailored for AWS infrastructure operations.

The paper introduces AlphaAgent, a framework that separates evidence retrieval from final report generation using explicit skill contracts. A dedicated retrieval skill translates user queries into material-specific intents, searches a curated index of over 300,000 papers, and reformulates queries if initial evidence is insufficient. This approach addresses the difficulty of handling heterogeneous tasks within single retrieve-then-generate architectures.

• Decouples retrieval and generation via skill contracts for better task isolation
• Retrieval skill reformulates queries to improve evidence quality in metallurgy
• Targets heterogeneous materials science tasks beyond standard RAG pipelines
• Leverages a curated index of 300,000+ papers for domain-specific accuracy

ComposioHQ has released a curated repository of over 1,000 Claude Skills and plugins designed to enhance AI workflows across platforms like Claude.ai, Cursor, and Gemini CLI. The collection emphasizes real-world actions by leveraging their MCP Gateway, which provides secure access to 1,000+ app integrations with built-in authentication and audit logs. This resource aims to simplify connecting AI agents to external tools for production-grade reliability.

• Curated list of 1000+ production-ready Claude Skills for diverse use cases.
• Supports multiple agents including Claude Code, Codex, Cursor, and Gemini CLI.
• MCP Gateway centralizes access with auth, access controls, and audit logs.
• Enables real-world actions by connecting agents to 1000+ external apps.

A retrieval-augmented, multi-agent LLM framework with human oversight was tested for detecting cutaneous immune-related adverse events in clinical notes. The system improved F1 score to 0.88 and Cohen's kappa to 0.82 compared to manual review, while halving the average review time. This approach demonstrates a scalable method for extracting accurate adverse event data across organ systems.

• Human-in-the-loop LLMs boost accuracy (F1 0.88) over manual review (0.77).
• Inter-rater agreement improves significantly (kappa 0.82 vs 0.50) with AI assistance.
• Average review time is reduced by approximately 50% compared to unassisted process.
• Retrieval-augmented multi-agent architecture enables transparent and scalable data extraction.
• Framework is designed for identifying immune-related toxicities across various organ systems.