SOP-004: MoE Routing Non-Determinism

Fresh Updated January 2026

Document Control
SOP ID	SOP-004
Version	1.0
Status	Active
Source	152334h (2024), Soft MoE Paper

Overview

Mixture of Experts (MoE) routing introduces non-determinism because sparse expert selection operates at the batch level, not per-sequence. This means other users' prompts can affect your output.

How MoE Works

The Non-Determinism Mechanism

Same prompt, different batch composition → Different outputs

Key Evidence

GPT-4 vs GPT-3.5-turbo (152334h Study, 2024)

Model	Unique Completions per 30 runs	Architecture
GPT-4	30 (all different!)	Sparse MoE
GPT-3.5-turbo	3.67 average	Likely denser

DANGER

GPT-4 at temperature=0 produced 30 unique completions in 30 runs - effectively no determinism.

Root Mechanism

1. Fixed-size token groups route to experts
2. When groups mix sequences, tokens from different requests interfere
3. Under capacity constraints, groups compete for expert slots
4. Load balancing creates per-batch variability

Implications

What This Means

Implication	Detail
Other users affect you	Content from concurrent users subtly affects your generation
Time-of-day matters	Peak usage = more batch mixing = more variance
Unprovable but structural	Can't demonstrate for specific outputs, but architecture confirms
Large models most affected	Non-determinism scales with model size

Service Load Impact

Mitigation Options

Option 1: Use Dense Models

Dense models avoid MoE routing entirely, but:

• Slower inference
• Higher cost
• Not available for frontier models

Option 2: Accept Variance

For most GEO/AEO use cases:

• Design systems that accommodate variance
• Use multiple samples and consensus
• Don't rely on single outputs

Option 3: Enterprise Tier

Enterprise API access often provides:

• Dedicated inference capacity
• Lower batch mixing
• More stable outputs

WARNING

There is no MoE mitigation without architectural redesign. This is a fundamental tension between determinism and efficiency.

Verification Checklist

• [ ] Understand that temperature=0 provides no guarantee for MoE models
• [ ] Test critical prompts at different times of day
• [ ] Document variance ranges in your system
• [ ] Implement consensus logic for critical decisions
• [ ] Consider enterprise tier for stability-critical applications

Comparison: KV Cache vs MoE Non-Determinism

Aspect	KV Cache	MoE Routing
Cause	Batched KV computation	Expert selection
Mitigation available?	Yes (2x cost)	No
Affected models	All batched	Sparse MoE only
Variance level	Moderate (38%)	High (100% at temp=0)

See Also

• SOP-001: KV Cache Non-Determinism
• SOP-007: Rate Limiting Effects
• 152334h (2024): "Non-determinism in GPT-4 is caused by Sparse MoE"

Citations

"GPT-4: 30 unique completions / 30 runs at temp=0" — 152334h (2024)

"Fixed-size token groups route to experts; when groups mix sequences, tokens from different requests interfere in expert buffers" — Soft MoE Paper