LLMs

Researchers introduced the Office Comprehension Benchmark (OCB) to assess how well large language models understand Word, Excel, and PowerPoint files in their native formats. The benchmark features two tracks: one testing structural fidelity for elements like tables and charts, and another evaluating expert-level reasoning across 12 professional domains. Responses are graded using atomic claims and an ensemble of LLM judges to ensure precise evaluation.

• First public benchmark targeting native .docx, .xlsx, and .pptx file comprehension.
• Tests visual and structural perception of complex artifacts like embedded charts and formulas.
• Evaluates multi-step reasoning and synthesis across 12 distinct industry domains.
• Uses atomic claim decomposition and LLM judge ensembles for granular scoring.

A new study reveals that BPE tokenization creates exploitable safety gaps by fragmenting critical words into sub-word pieces, a pattern absent from current alignment training data. Character-level perturbations can bypass safety filters while remaining human-readable. Tests across five major model families show this method successfully flips refusal triggers on 80-100% of harmful prompts, with nearly half resulting in genuinely harmful outputs.

• BPE tokenization fragments safety words, exposing a structural blind spot in current alignment.
• Training data lacks intentionally fragmented inputs, creating a predictable attack surface.
• Character-level perturbations bypass filters while keeping prompts human-readable.
• 80-100% of refusals flipped on HarmBench, with 48% yielding harmful outputs.
• Activation patching helps localize the specific mechanism behind the failure.

RuleChef leverages large language models solely during the learning phase to synthesize executable rules for NLP tasks like classification and NER. These rules are iteratively refined using labeled examples and human feedback to patch failures identified on a held-out split. The final output is a fast, deterministic, and fully inspectable rule system that does not rely on runtime LLM inference.

• LLMs are used only at learning time, not for runtime inference, reducing latency and cost.
• Rules are human-editable and inspectable, offering transparency lacking in black-box models.
• System bootstraps from existing model output pairs, easing integration with legacy pipelines.
• Deterministic execution ensures consistent results without stochastic LLM variance.

The Wiola architecture presents a Small Language Model built from scratch, avoiding lineage from GPT, LLaMA, or Mistral families. It features Spiral Rotary Positional Encoding to embed positions on a 3D helical manifold and Gated Cross-Layer Attention for inter-layer coherence. Additionally, Adaptive Token Merging dynamically reduces redundant tokens in middle layers to improve efficiency.

• Wiola is a ground-up SLM design with no structural ties to major existing families like LLaMA or GPT.
• SRPE combines absolute, relative, and hierarchical positional signals on a 3D helical manifold.
• GCLA allows decoder layers to access compressed summaries from preceding layers for coherence.
• ATM dynamically merges semantically redundant adjacent tokens to reduce computational load.

TokenScope is an interactive interpretability tool for decoder-based LLMs that exposes token-level metrics, attention patterns, and structural information during generation. It addresses gaps in existing tools by providing decoding-time signals, fine-grained uncertainty measures, and mechanisms to explore alternative generation paths. The system supports interactive token replacement, counterfactual branching, and code-aware aggregation using abstract syntax trees.

• Provides decoding-time signals for token-level decisions in code generation tasks.
• Enables interactive token replacement and counterfactual branching for path exploration.
• Integrates code-aware aggregation via abstract syntax trees for structural context.
• Offers fine-grained uncertainty measures to improve model interpretability.

RusFinChain introduces the first Russian-language benchmark for verifiable chain-of-thought reasoning in finance, addressing the English-centric limitation of prior tools like FINCHAIN. It contains 5,280 parameterized examples spanning 17 domains, generated from executable Python templates to ensure contamination-free evaluation. Each instance includes a gold-standard reasoning chain with intermediate numeric values, enabling automatic verification of multi-step symbolic logic. The release also features enhanced metrics designed for fuzzy-aligned evaluation of these complex reasoning steps.

• First Russian CoT finance benchmark focusing on verifiable intermediate reasoning steps.
• 5,280 examples derived from executable Python templates to prevent data contamination.
• Covers 17 domains and 172 topics with gold-standard numeric verification chains.
• Replaces multiple-choice formats with step-level supervision for robust evaluation.
• Introduces fuzzy-aligned metrics to better assess nuanced reasoning quality.

Researchers introduce Grounded Optimization, a five-layer engineering framework designed to mitigate hallucinations in LLM-generated resume rewrites. The system addresses specific failure modes like anachronistic technology injection and content fabrication through temporal validation, contamination detection, and structural invariant enforcement. Ablation tests across multiple models and temperatures show that undefended baselines produce significantly more hallucinations per resume than the defended framework.

• Targets resume-specific hallucinations like anachronisms and cross-domain contamination.
• Uses a five-layer structure including temporal validation and evaluator agents.
• Undefended baselines generate 2.48-5.36 hallucinations per resume on average.
• Validated across three LLMs, four temperatures, and six industry sectors.