Data Points Reference
Fresh Updated January 2026All key statistics and metrics from LLM technical phenomena research, organized for quick access.
KV Cache Non-Determinism
| Metric | Value | Source |
|---|---|---|
| Unique completions (temp=0) | 11.67 of 30 | Chroma Research 2024 |
| Variation cause | Floating-point batching | KV Cache Implementation |
| Reproducibility | Not guaranteed | Even with identical prompts |
Key Insight
temperature=0 does not guarantee deterministic output.
Context Rot (Hidden State Drift)
| Metric | Value | Source |
|---|---|---|
| Performance degradation | 20-60% | Thinking Machines Lab 2024 |
| Threshold | >100k tokens | Long-context studies |
| Hallucination correlation | r = 0.73 | Hidden state analysis |
| Effect type | Non-monotonic decay | Layer-wise measurement |
RLHF Alignment Tax
| Metric | Value | Source |
|---|---|---|
| NLP performance drop | 5-15% | ACL 2024-2025 |
| Refusal rate increase | Significant | Safety training |
| Knowledge suppression | Observed | Alignment fine-tuning |
Trade-off Spectrum
| More Aligned | Less Aligned |
|---|---|
| Safer responses | Broader coverage |
| More refusals | More direct answers |
| Conservative framing | Complete information |
MoE Routing Variance
| Metric | Value | Source |
|---|---|---|
| Unique completions | 30 of 30 | MoE architecture studies |
| Cause | Load balancing decisions | Expert routing |
| Reproducibility | Very low | Token-level routing varies |
WARNING
MoE models (GPT-4, Mixtral) show higher variance than dense models.
Lost in the Middle Effect
| Metric | Value | Source |
|---|---|---|
| Middle position performance | 30% worse | Liu et al. 2023 |
| Best positions | First 20%, Last 20% | Attention patterns |
| Worst position | 40-60% mark | "Lost in the middle" |
Position Performance Map
Prompt Injection Vulnerability
| Metric | Value | Source |
|---|---|---|
| Architectural separation | None | LLM design |
| Reliable defense | None | Security research |
| Mitigation effectiveness | Partial | Heuristic-based |
Rate Limiting Effects
| Provider | Metrics | Impact |
|---|---|---|
| OpenAI | RPM, TPM, TPD | Batch reshaping |
| Anthropic | RPM, TPM | Variance changes |
| QPM, TPM | Load distribution |
Interpretability Gaps
| Metric | Value | Source |
|---|---|---|
| Papers published (2024-2025) | 100+ | EMNLP 2025 |
| Consensus solutions | 0 | Research status |
| SAE variance explained | 80-90% | Partial decomposition |
| Polysemantic neurons | Common | Superposition |
Platform Citation Statistics
| Platform | Top Source | Diversity | Freshness Priority |
|---|---|---|---|
| ChatGPT | Wikipedia (47.9%) | Moderate | Lower |
| Perplexity | Real-time web | 2-3x higher | Critical |
| Claude | Curated sources | Lower | Moderate |
| Gemini | Google index | Lower (1-2 sources) | Moderate |
Citation Overlap
Content Optimization Targets
Length Recommendations
| Content Type | Optimal | Maximum |
|---|---|---|
| Product page | 2k-5k tokens | 10k |
| Blog post | 3k-8k tokens | 20k |
| Guide/manual | 5k-15k tokens | 30k |
| Documentation | 10k-30k tokens | 50k |
Distractor Impact
| Distractors | Citation Impact |
|---|---|
| 0 | Baseline (100%) |
| 1 | -15% |
| 4 | -40% |
Research Sources
| Source | Focus Area | Year |
|---|---|---|
| Thinking Machines Lab | Context rot, hidden state drift | 2024 |
| Chroma Research | KV cache non-determinism | 2024 |
| ACL Proceedings | RLHF alignment tax | 2024-2025 |
| Liu et al. | Lost in the middle | 2023 |
| EMNLP | Interpretability | 2025 |
| Anthropic | SAE decomposition | 2024-2025 |