AI / ML

An open-source Swift and Metal engine called TurboFieldfare runs 4-bit quantized Gemma 4 26B-A4B-IT on M-series Macs with only 2 GB of RAM. It achieves this by keeping the shared model layers and KV cache in memory while streaming only the necessary routed experts from the SSD for each token. This approach allows large models to run on devices with limited RAM, such as 8 GB or 16 GB Macs, by offloading weight storage to faster SSDs.

• TurboFieldfare is an open-source Swift/Metal engine for on-device AI inference.
• It runs 26B parameter models on 8GB/16GB Macs by streaming experts from SSD.
• Only shared layers and KV cache remain in RAM, reducing memory footprint to ~2GB.
• SSD streaming replaces RAM storage for 4-bit quantized weights to save space.

MoonshotAI has open-sourced FlashKDA, a library implementing high-performance Kimi Delta Attention kernels built on CUTLASS. The project targets hardware with SM90 architecture and requires CUDA 12.9 and PyTorch 2.4 or newer. It supports flexible compilation targets, allowing users to specify specific architectures or build for all supported ones.

• FlashKDA optimizes Kimi Delta Attention via CUTLASS for better inference performance.
• Requires SM90+ GPUs, CUDA 12.9+, and PyTorch 2.4+ to function correctly.
• Installation supports auto-detection or explicit architecture targeting via environment variables.
• A deep-dive blog post details the specific design decisions behind the v1 implementation.

Researchers introduce ClinLens, a benchmark featuring 200 executable tasks across five MIMIC resources including EHRs, notes, and imaging. The evaluation framework uses a 4x5 taxonomy to assess how agents handle longitudinal patient data and complex analysis workflows. It employs program-first reverse synthesis to verify cohort semantics, temporal logic, and final answers against private reference workflows.

• ClinLens moves beyond simple QA to test multi-step coding over longitudinal multimodal clinical data.
• Benchmarks five data types: EHRs, notes, ECGs, chest X-rays, and echocardiograms.
• Uses reverse synthesis to validate agent outputs against executable reference workflows.
• Focuses on temporal and cohort semantics rather than just static table reasoning.

Researchers propose using high-fidelity synthetic customer agents as digital twins to validate LLM-based chatbots in regulated sectors like banking. These agents are grounded in real transactional and conversational data to simulate diverse customer profiles and interaction styles. Evaluations show the synthetic agents achieve high semantic alignment with real users, low hallucination rates, and controllable personality trait reproduction.

• Synthetic Customer Agents (SCAs) enable automated, large-scale chatbot testing without human-in-the-loop bottlenecks.
• Agents are conditioned on real transactional and conversational data for high-fidelity behavioral simulation.
• SCAs demonstrate strong semantic alignment and low hallucination rates compared to real customer interactions.
• Personality traits are reproducible and controllable, allowing targeted stress testing of chatbot responses.

Existing RAG systems struggle with multi-turn conversations because they store raw history or summaries rather than specific reasoning steps. CMT-RAG addresses this by aligning conversational memory with retrieval through sub-question-level reasoning traces. The authors also introduce MuMu-QA, a benchmark featuring explicit cross-turn sub-question dependency annotations to evaluate this approach.

• Stores sub-question reasoning traces instead of raw dialogue history for better retrieval alignment.
• Improves recovery of prior evidence needed for follow-up queries in multi-turn conversations.
• Introduces MuMu-QA benchmark with explicit cross-turn dependency annotations for evaluation.
• Enables multi-hop reasoning by explicitly tracking long-range dependencies across conversation turns.

This project provides a modular pipeline for building low-latency voice agents using open-source models. It chains VAD, STT, LLM, and TTS components, exposing an OpenAI Realtime-compatible WebSocket API. The architecture allows swapping any component, including pointing the LLM slot at local vLLM or llama.cpp servers for a fully private stack.

• Modular pipeline: VAD -> STT -> LLM -> TTS with swappable components.
• OpenAI Realtime-compatible WebSocket API for easy client integration.
• Supports local LLM inference via vLLM or llama.cpp for data privacy.
• Uses Parakeet TDT for local speech-to-text by default.
• Proven in production as the backend for Reachy Mini robots.