LLMs

Amazon SageMaker JumpStart now hosts Meta's Muse Spark Open, a coding-optimized foundation model designed for agentic workflows. Built on a Mixture-of-Experts architecture with 816B total parameters, it maintains efficient per-token compute while supporting a 1M-token context window. The model targets long-running coding agents and multimodal engineering tasks on AWS infrastructure.

• SageMaker JumpStart adds Meta's Muse Spark Open for enterprise agentic coding workflows.
• MoE architecture uses 42B active parameters per token out of 816B total.
• 1M-token context window allows ingestion of entire projects and dependencies.
• Optimized for long-running agents and multimodal engineering tasks.

AbridgeKV addresses memory bottlenecks in long chain-of-thought decoding by training models to compress generated text blocks into learned pause tokens. These special tokens hold no output but store the KV cache state of the preceding text, allowing the system to evict the original block's memory. Subsequent generation steps attend to these compressed pause tokens instead of the full history, significantly reducing memory usage without retraining eviction policies.

• Replaces linear KV cache growth with fixed-size learned pause token embeddings
• Enables memory eviction of completed reasoning blocks without quality loss
• Avoids the degradation seen when evicting cache entries from untrained models
• Pause tokens act as compressed attention anchors for subsequent decoding steps

This research compares a non-generative decision model against a frontier LLM for verifying claims in Retrieval-Augmented Generation pipelines. Using the LLM-AggreFact dataset, the study finds both systems are statistically indistinguishable in performance. The non-generative approach offers native probability outputs and typed values without the overhead of text generation.

• Non-generative verifiers match frontier LLM accuracy for binary claim verification.
• Eliminates generation costs by returning typed values and probabilities directly.
• Paired comparison on 495 human-labeled claims shows no statistical difference.
• Simplifies RAG pipelines by removing prose generation for verification steps.

Verdict-MQ addresses the inefficiency of standard cross-encoders that require separate processing passes for each typed question. By using a structured attention mask, the model encodes the input state once while keeping question representations independent and order-invariant. This approach enables simultaneous answers to multiple organizational queries with sufficient probability accuracy for automated action.

• Single encoding pass replaces multiple cross-encoder calls, reducing latency for batched decisions.
• Structured attention masks isolate question processing, preventing cross-question interference.
• Option order invariance is built-in, simplifying prompt engineering and model stability.
• Verified state retention under truncation ensures consistent feature extraction across cases.

TokenSpeed is a new inference engine targeting production agentic workloads, claiming performance comparable to TensorRT-LLM while maintaining usability similar to vLLM. It utilizes a C++ control plane and Python execution plane, encoding request lifecycles and KV cache management via a compile-time finite-state machine. The system features a pluggable kernel architecture with a fast Multi-head Latent Attention implementation optimized for Blackwell GPUs.

• Targets agentic workloads with TensorRT-LLM level performance and vLLM-like usability.
• Uses C++ control plane and Python execution plane for high-performance inference.
• Enforces safe KV resource reuse via compile-time type systems and finite-state machines.
• Features a pluggable kernel system including optimized MLA for Blackwell GPUs.

XERJ is a local AI search tool that automatically indexes folders to enable efficient data retrieval for AI agents. By pre-indexing code, docs, logs, and PDFs, it reduces token consumption compared to traditional text processing commands like grep. The system is compatible with Elasticsearch, allowing existing clients to integrate seamlessly for use cases such as RAG, security audits, and reference coding.

• Autoindexing eliminates manual grep/sed workflows for AI agents
• Claims 40x token reduction versus traditional text search methods
• Elasticsearch compatible interface ensures drop-in integration
• Supports structured retrieval from code, docs, logs, and PDFs
• Optimized for reference coding and agent memory contexts