AI / ML

Researchers measured why enterprise AI stalls by testing six regulated financial workflows across 72 model and tool configurations. While 57 of 72 passed a simple demonstration benchmark, only 32 met strict production requirements for sustained accuracy, reproducibility, and verifiable attribution. The study highlights a significant gap between demo viability and operational reliability in high-stakes environments.

• Demonstration success does not predict production readiness; 21 configs failed the production bar.
• Production bars require sustained accuracy, reproducibility, and verifiable attribution, not just single-case correctness.
• Regulated financial services face a 44% failure rate when moving AI from demo to production.
• Confidence signals must carry actual information to meet the new production standards.

This study introduces a benchmarking protocol using frontier LLMs to evaluate autonomous AI research systems across originality, rigor, clarity, and significance. The authors tested four leading frameworks, including Sakana AI and CycleResearcher, against 15 proposals from FARS. The goal is to establish a rigorous method for comparing the quality of AI-generated scientific papers.

• Automated peer review using LLMs can standardize quality assessment for AI-generated research.
• Evaluation focuses on four dimensions: originality, rigor, clarity, and significance.
• Benchmark compares major frameworks like Sakana AI v1/v2, CycleResearcher, and Data-to-Paper.
• Consistent proposal set from FARS enables direct comparison of AI scientist performance.

This paper addresses the inconsistency of LLM-based entity extraction by introducing a production layer that enforces a formal ontology. The system ingests heterogeneous documents via Kafka and uses a two-pass extraction strategy with a local Qwen3.5-9B model to ensure type consistency and deduplication. It specifically targets issues like name variant fragmentation and silent conflation of distinct individuals sharing the same name.

• Enforces formal ontologies to prevent type vocabulary fragmentation across document streams.
• Uses a two-pass extraction approach to detect and remove duplicate relationships.
• Mitigates silent conflation risks for distinct entities sharing identical names.
• Leverages a local Qwen3.5-9B model for consistent, schema-aligned extraction.