Agentic AI

Researchers introduce Sentinel, a classifier-based confidence mechanism designed to evaluate the reliability of LLM agents performing multi-step clinical text-to-SQL tasks. The system analyzes reasoning paths, generated code, and database outputs to detect failures at three critical stages: refusing unanswerable queries upfront, halting broken trajectories mid-execution, and suppressing untrustworthy final answers. This approach aims to reduce costly errors and wasted computational resources in high-stakes healthcare applications.

• Sentinel adds a classifier layer to monitor LLM agent trajectories for trustworthiness.
• It prevents wasted compute by halting doomed reasoning paths mid-execution.
• Three intervention points: pre-run refusal, mid-run halt, and post-run withholding.
• Specifically targets high-risk clinical text-to-SQL workloads in healthcare.

Frontier LLM APIs hide token probabilities, leaving practitioners unable to detect silent tool call errors until execution fails. This research proposes using a parallel, low-cost open-weight surrogate to estimate confidence by analyzing the same context and proposed actions. The surrogate employs teacher forcing and request-PMI readouts to score argument likelihoods, effectively flagging mistakes that resampling cannot catch due to model repetitiveness.

• Frontier model APIs hide log-probs, making silent agent errors hard to detect before execution.
• Resampling fails to identify errors because frontier models are highly repetitive in their outputs.
• A parallel open-weight surrogate can score action validity using teacher forcing and PMI metrics.
• This approach provides a practical signal for auditing black-box agent behavior without API access.

This paper argues that autonomous coding agents require specific teachable behaviors rather than relying solely on increased inference-time compute. It identifies three critical failures: narrow location diversity that causes agents to revisit the same code sites, underutilized edit diversity that misses complementary fixes, and verification methods that accept incorrect patches. The authors propose shifting these capabilities into the policy itself to ensure reliable evidence for patch selection.

• Extra compute only helps if it yields useful repairs with reliable selection evidence.
• Agents suffer from narrow location diversity, repeatedly targeting the same code sites.
• Edit diversity is unexploited, causing agents to miss complementary fix methodologies.
• Self-written tests often mislead by accepting incorrect patches as valid.
• Search should be directed by execution to improve reliability over scale.

CUAWright introduces a minimal terminal harness using bash commands as the sole action interface, replacing fixed GUI toolsets. It leverages the file system as an evolvable space for dynamically creating tools and managing context. This approach addresses limitations in current agents by allowing direct programmatic operation on system state as model coding capabilities improve.

• Replaces static GUI harnesses with a minimal 3K line terminal-based interface.
• Uses bash commands as the exclusive action interface for digital agents.
• Leverages the file system for dynamic tool creation and context management.
• Enables agents to directly manipulate system state via programmatic operations.

This research evaluates how language model agents handle evidence changes, specifically when supporting documents are revoked or replaced. Using ICU record tasks, the study compares memory repair strategies against full and source-filtered re-reading across two 7B models. The analysis reveals that while local repair significantly reduces revision tokens, memory pipelines generally incur higher costs than direct re-reading in held-out conditions.

• Local memory repair uses 5-10x fewer revision tokens than rebuilding memory from scratch.
• Despite token efficiency, memory pipelines cost at least twice as much as full re-reading.
• The study uses ICU medication and problem-list tasks to simulate real-world evidence changes.
• Agents must choose between caching, rebuilding, or graph-local repair based on cost constraints.
• Source-filtered re-reading offers an alternative to full re-reading with potential efficiency gains.

Research reveals that multimodal large language models significantly degrade in safety performance when operating in agentic tool-use modes compared to standard inference. Testing across three safety benchmarks shows refusal failure rates can spike by up to 68.7% for top open- and closed-weight models. The study analyzes over 100,000 responses to identify structural reasons behind this vulnerability in tool-calling agents.

• Agentic tool-use paradigms actively degrade model safety refusals across major MLLMs.
• Refusal failure rates increased by up to 68.7% in tool-using settings versus non-tool.
• Both open-weight and closed-weight models exhibit this consistent safety regression.
• Safety benchmarks must account for agentic contexts, not just static inference.
• Tool integration requires new safety guardrails beyond standard alignment techniques.

SkillScriptBench is a new 350-task benchmark evaluating how well LLM agents can self-evolve executable skill packages by fixing injected script faults while preserving correct behavior. Unlike prior work, it systematically separates documentation repair, script repair, and preservation metrics. The dataset draws from over 35,000 GitHub-hosted skill roots, constructing 150 repair tasks with executable checks and a controlled track of 200 tasks for rigorous validation.

• Separates evaluation of doc repair, script repair, and behavior preservation in agent skills.
• Uses 350 tasks derived from 35k+ GitHub repos to test realistic maintenance scenarios.
• Includes executable checks to verify that fixes do not break existing correct behavior.
• Targets LLM self-evolution beyond static markdown, focusing on executable code packages.