SOP-008: Interpretability Gaps

Fresh Updated January 2026

Document Control
SOP ID	SOP-008
Version	1.0
Status	Active
Source	Mechanistic Interpretability Research, Anthropic, EMNLP 2025

Overview

Interpretability gaps refer to our fundamental inability to fully explain why LLMs produce specific outputs. Despite significant research (100+ papers in 2024-2025), we cannot definitively answer "Why did the model cite this source?" or "Why did it choose this phrasing?"

The Interpretability Frontier

Core Problem

The causal pathway from input to output involves non-obvious intermediate steps that we cannot fully characterize.

Key Phenomena

1. Superposition & Polysemanticity

Models compress more features than neurons exist. Individual neurons respond to multiple unrelated concepts (polysemanticity).

Finding	Status
Polysemantic neurons	Common
SAE decomposition	80-90% of variance
Full interpretability	Incomplete

2. Discrete Phase Transitions

Capabilities emerge suddenly at inflection points, not smoothly:

• Training curves show sudden jumps
• Small training variations → large output changes
• Behavior change is unpredictable

3. Layer-Wise Semantic Dynamics

Hallucination correlates with specific hidden-state trajectory changes across layers, but:

• Correlations identified ✓
• Causally grounded ✗
• Prevention strategies limited

4. Attribution Complexity

Question	Answer Status
Why did it cite source X?	Unknown
Why this phrasing vs. that?	Unknown
Why hallucinate here?	Unknown
How to prevent specific behavior?	Limited

Research Status

What We Know

• SAEs can partially decompose representations
• Certain patterns correlate with behaviors
• Attention patterns provide some insight
• Probing can identify some features

What We Don't Know

• Complete causal pathways
• Why specific citations chosen
• How to reliably control behavior
• Predict capability emergence

Implications for GEO/AEO Strategy

1. Current Optimization May Become Obsolete

Future model updates will shift understanding. What works today may not work tomorrow.

2. "Why" Questions Are Often Unanswerable

Question Type	Answerability
What did the model output?	✅ Answerable
How often does X occur?	✅ Answerable
Why did it choose X?	❌ Often unanswerable
How to guarantee Y?	❌ Usually impossible

3. Empirical Testing > Theoretical Prediction

Best Practices Given Interpretability Gaps

1. Test, Don't Assume

• Run multiple tests before deploying
• Verify across different prompts
• Check edge cases empirically

2. Build Robustness

• Don't rely on single behaviors
• Design for variance
• Have fallback strategies

3. Monitor Continuously

• Track output patterns over time
• Watch for behavioral drift
• Update strategies as needed

4. Accept Uncertainty

• Some questions have no answer
• "Why" may remain mysterious
• Focus on "what works" empirically

Verification Checklist

• [ ] Understand that "why" questions often can't be answered
• [ ] Base strategy on empirical testing, not theory
• [ ] Build systems robust to behavioral changes
• [ ] Monitor for drift over time
• [ ] Accept that current approaches may become obsolete
• [ ] Stay updated on interpretability research

Research Resources

Resource	Focus
Anthropic's SAE work	Feature decomposition
EleutherAI interpretability	Open research
EMNLP 2025 proceedings	Latest findings
Mechanistic interpretability papers	Deep analysis

Future-Proofing Strategy

1. Don't over-optimize for current behavior
2. Build adaptive systems that can change
3. Monitor for shifts in model behavior
4. Stay informed on interpretability advances
5. Accept uncertainty as permanent feature

See Also

• SOP-003: Alignment Tax
• SOP-002: Context Rot
• Quick Reference: Data Points

Citations

"Polysemantic neurons common; SAEs partially decompose (80-90% of variance) but incomplete" — Mechanistic Interpretability Research

"Active research (100+ papers 2024-2025); no consensus solutions" — EMNLP 2025 Trends

"Causal pathway from input to output involves non-obvious intermediate steps" — Attribution & Causal Analysis Work