AI / ML

Standard Sparse Autoencoders assume visual concepts are independent, violating the structure of natural images. The authors introduce the Markov-Field Linear Representation Hypothesis to model these spatial dependencies. Their proposed Spatial-SAE acts as an amortized MAP estimator under this new hypothesis, outperforming standard SAEs in concept recovery and interpretability.

• Standard SAEs fail to capture spatial dependencies in natural images.
• New MFLRH hypothesis integrates spatial structure into linear representation theory.
• Spatial-SAE uses amortized MAP estimation to improve concept recovery.
• Experimental results show consistent gains over baseline SAE variants.

This research paper identifies three distinct constraints that determine the causal influence of localized latent structures in language models: capacity, responsiveness, and alignment. Capacity measures output sensitivity to structural movement, responsiveness captures concept promotability within context, and alignment reflects how well the structure matches context-specific representations. The findings suggest that not all localized structures are equally actionable, offering a more nuanced framework for model interpretability and control.

• Causal influence is not monolithic; it decomposes into capacity, responsiveness, and alignment.
• Capacity quantifies how sensitive model outputs are to interventions along a latent direction.
• Responsiveness measures how easily a concept can be promoted given the current context.
• Alignment assesses the match between a latent structure and context-specific concept representations.
• Framework applies across multiple LM families, suggesting broad utility for interpretability.