SOP-007: Rate Limiting Effects

Fresh Updated January 2026

Document Control
SOP ID	SOP-007
Version	1.0
Status	Active
Source	OpenAI, Anthropic, Google API Docs (2024-2025)

Overview

Modern LLM APIs implement multi-level rate limiting that goes beyond simple request counts. These throttling mechanisms can indirectly affect output quality by reshaping batch composition and interacting with MoE routing.

Rate Limiting Evolution

Multi-Level Token Awareness

Provider	Metrics	Documentation
OpenAI	RPM, TPM, TPD	Documented 2024
Anthropic	RPM, TPM	Documented 2024
Google	QPM, TPM	Documented 2025

INFO

All major providers now implement tokens-per-minute (TPM) throttling, not just requests-per-minute.

How Throttling Affects Output

The Cascade Effect

1. High-volume requests trigger throttling
2. Delayed/chunked responses change timing
3. Different batching occurs
4. KV cache state varies from expected
5. Content variance results

Enterprise vs Consumer Tiers

Aspect	Enterprise	Consumer
Batch stability	Higher	Lower
Throughput	Higher	Standard
Cost	Higher	Lower
Output variance	Lower	Higher

Side Effects on MoE Routing

WARNING

Throttling indirectly reshapes batch composition, which can interact with MoE routing for additional variance. Direct measurement is limited.

Practical Implications for GEO/AEO

High-Volume Content Pipelines

Scenario	Impact	Mitigation
Bulk content generation	Higher variance	Space requests over time
Peak usage periods	Queue delays	Avoid peak hours
Bursty workloads	Inconsistent batching	Smooth request patterns
Long-running sessions	Accumulated drift	Fresh sessions

Optimal Request Patterns

Rate Limit Best Practices

1. Monitor Usage

// Example: Track TPM usage
const tokenUsage = {
  current: 0,
  limit: 100000,  // TPM limit
  resetTime: Date.now() + 60000
};

function canMakeRequest(estimatedTokens) {
  if (Date.now() > tokenUsage.resetTime) {
    tokenUsage.current = 0;
    tokenUsage.resetTime = Date.now() + 60000;
  }
  return (tokenUsage.current + estimatedTokens) < tokenUsage.limit;
}

2. Implement Backoff

// Exponential backoff for rate limits
async function requestWithBackoff(prompt, maxRetries = 5) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await makeRequest(prompt);
    } catch (e) {
      if (e.status === 429) {  // Rate limited
        const delay = Math.pow(2, i) * 1000;
        await sleep(delay);
      } else throw e;
    }
  }
}

3. Smooth Request Patterns

• Space requests evenly over time
• Avoid burst patterns
• Pre-schedule non-urgent work for off-peak
• Use request queues with rate limiters

Verification Checklist

• [ ] Understand your tier's TPM/RPM limits
• [ ] Monitor token usage in production
• [ ] Implement exponential backoff
• [ ] Smooth bursty workloads
• [ ] Consider enterprise tier for stability-critical work
• [ ] Test during different load periods

Provider-Specific Notes

OpenAI

• TPM limits vary by model and tier
• Embedding and completion have separate limits
• Rate limit headers provided in responses

Anthropic

• TPM limits documented per tier
• Rate limit info in response headers
• Retry-After header when limited

Google

• QPM (queries per minute) in addition to TPM
• Regional variations in limits
• Quota management in Cloud Console

See Also

• SOP-001: KV Cache Non-Determinism
• SOP-004: MoE Routing Variance
• Provider Documentation

Citations

"Rate limiting now TPM (tokens-per-minute) not just RPM (requests-per-minute)" — OpenAI, Anthropic, Google API Documentation (2024-2025)

"Throttling indirectly reshapes batch composition; can interact with MoE routing" — Rate Limiting Engineering Analysis