AI / ML

This research investigates how simulated 4-bit KV-cache quantization disrupts Top-k Mixture-of-Experts routing by pushing tokens across discontinuous decision boundaries. The authors introduce a causal apparatus to quantify the Route-Mediated Fraction (RMF) of performance degradation, finding that approximately 31% of the total damage in OLMoE-1B-7B stems from these route flips rather than direct numerical precision loss. The study provides empirical evidence and detection limits but proposes no new mitigation strategies for this routing instability.

• 31% of quantization damage in MoE models is caused by expert routing flips, not just precision loss.
• Top-k routing discontinuity makes models sensitive to small numerical disturbances in KV-cache.
• The paper offers a causal detection framework but no solutions to fix the route-mediated damage.
• Findings are validated across three architectures using pre-registered probes and token-level attribution.

Mistral has released version 4.1 of its OCR model, focusing on improved accuracy for complex document layouts and text extraction. The update aims to provide more reliable processing for structured and unstructured data sources in enterprise workflows. This release is available via the Mistral AI documentation and model endpoints.

• New OCR 4.1 version targets better handling of complex document structures.
• Improved text extraction accuracy for enterprise-grade data processing.
• Available through official Mistral AI model endpoints and docs.
• Likely useful for pipelines needing higher fidelity document parsing.
• Monitor benchmarks to assess improvement over previous OCR versions.

The paper introduces VQ-bench, an open-source framework that unifies vector quantization research by decomposing algorithms into seven core primitives. It demonstrates how to compose these primitives arbitrarily, re-expressing 25 common quantizers as standardized pipelines. This approach aims to provide reproducible benchmarks and a unified language for developing new quantization algorithms in AI infrastructure.

• Decomposes VQ into 7 reusable primitives for modular algorithm design.
• Re-expresses 25 existing quantizers as composable pipelines.
• Enables reproducible benchmarking across diverse quantization methods.
• Provides an open-source framework to accelerate VQ research.
• Addresses the fragmented state of current vector quantization engineering.

Amazon Bedrock now supports cost allocation by IAM principal for inference requests sent to the bedrock-mantle endpoint. This feature extends existing capabilities from the bedrock-runtime endpoint, allowing users to tag IAM users and roles with attributes like team or cost center. These tags can be activated in AWS Cost Explorer to analyze and attribute inference costs accurately across different projects and applications.

• Cost allocation by IAM principal is now available for the bedrock-mantle endpoint.
• Previously limited to the bedrock-runtime endpoint, expanding cost tracking coverage.
• Use IAM tags for team, project, or cost center to attribute inference spend.
• Analyze detailed cost breakdowns in AWS Cost Explorer using activation tags.
• Enables precise financial accountability for generative AI model usage.