Agentic AI

A new paper introduces Harbor Adapters, a unified evaluation infrastructure designed to port over 80 agentic benchmarks for testing arbitrary agents. The authors validate these adapters through rigorous code review and parity experiments, ensuring reliable cross-benchmark comparisons. Additionally, they present a large-scale evaluation of eight models across 54 benchmarks, comparing performance against Terminus-2 and native harnesses to uncover broader capability and failure mode insights.

• Harbor Adapters standardize evaluation across 80+ benchmarks, reducing integration overhead for arbitrary agents.
• Validation includes rigorous code review and parity experiments to ensure adapter reliability.
• Large-scale study of 8 models across 54 benchmarks reveals detailed capability and failure mode patterns.
• Framework enables consistent comparison against Terminus-2 and three native harnesses.

This research demonstrates that improving individual LLM capability can degrade system-level outcomes in financial markets. The authors hypothesize that shared training and architectures cause more capable models to behave similarly, leading to correlated, non-diversified actions. An agent-based simulation of LLM traders confirms that frontier models exhibit significantly higher behavioral correlation, creating a non-diversifiable risk floor.

• Higher model capability does not guarantee better system-level outcomes in multi-agent environments.
• Shared architectures cause correlated behaviors that prevent risk diversification.
• Frontier LLMs exhibit stronger behavioral correlation, increasing systemic risk.
• A non-diversifiable risk floor emerges when agents act in unison.

Researchers introduced tau-tau-bench to evaluate whether coding agents can actually build production-grade LLM agents under realistic constraints. The environment simulates a real client engagement by providing business records, a requirements-holding client, existing codebases, production APIs, and cost limits. This approach shifts the focus from abstract benchmarks to the practical ability of an AI system to deliver a functional agent in a constrained, multi-stakeholder context.

• Benchmarks now test full agent construction rather than isolated coding tasks.
• Evaluation includes real-world constraints like serving cost and model limits.
• Agents must navigate existing codebases and production API integrations.
• Simulates client interactions to assess requirement adherence and delivery.

GRACE addresses the limitations of standard RAG by deconstructing LLM responses into atomic claims and grounding them against trusted knowledge priors using a weighted bipartite graph. Edge weights quantify the closeness of claims to prior knowledge, enabling weighted centrality analysis to classify outputs as Grounded, Refuted, or Boundary. This approach allows for the identification of both hallucinations and potentially novel insights that fall within boundary conditions.

• Moves beyond simple passage retrieval by tracking cross-document evidence relationships.
• Uses weighted centrality analysis to classify claims with higher precision than binary retrieval.
• Distinguishes between outright hallucinations and novel insights in the 'Boundary' category.
• Provides a robust framework for high-stakes LLM deployments requiring expert-in-the-loop validation.

A Hacker News thread with 122 points explores how engineers currently organize, verify, and maintain skills files for AI agents. Participants discuss strategies for ensuring these files function reliably and remain up-to-date as model capabilities evolve. The conversation highlights a gap in standardized tooling for the lifecycle management of agent-specific instructions and capabilities.

• No standard tooling exists yet for versioning and testing skills files
• Community is actively sharing ad-hoc methods for organizing agent capabilities
• Validation of skill functionality remains a key pain point for practitioners