Scaling ChatGPT for 1 Billion Users: How OpenAI Achieved 12x Faster Reads with Habitat and Azure Cosmos DB

Key Takeaways

• Massive Performance Leap: OpenAI reduced its overall global read p50 latency from roughly 120 ms to approximately 10 ms—a 12× speed improvement—across mission-critical product workflows.
• Frontier-Scale Workloads: Habitat supports over 1 billion ChatGPT users across 40 geographic regions, managing more than 500 petabytes of aggregate data and processing tens of millions of queries per second.
• Architectural Journey of Multi-Tenancy: OpenAI transitioned through three distinct operational phases: initially relying on native Azure Cosmos DB primitives, moving toward single-tenant container isolation, and finally establishing a managed multi-tenancy default enabled by automated data movement, hot key throttling, and request-priority Quality of Service (QoS).
• Integrated Caching via CDC: To serve read-heavy product traffic without overwhelming core databases, Habitat incorporates localized Valkey caching nodes kept consistent globally within seconds using Change Data Capture (CDC) via Azure Cosmos DB's change feed.

---

In-Depth Analysis

The Multi-Tenancy Conundrum: From Primitives to Managed Co-Tenancy

At the core of OpenAI’s consumer and enterprise offerings—including ChatGPT, Codex, and developer APIs—is Habitat, an internal application storage platform engineered to decouple product logic from backing databases. In the initial phase of deployment, Habitat relied directly on native Azure Cosmos DB primitives, placing multiple workloads within a single shared container per account. While this simplistic layout allowed engineering teams to ship quickly, it introduced severe noisy-neighbor anomalies. Early client implementations allowed direct selection of logical partition keys, triggering severe key collisions across common attributes like user_id or workspace_id. Because Azure Cosmos DB enforces throughput limits at the backing physical partition rather than solely at the logical level, hot traffic from peripheral product features frequently exhausted throughput, throttling unrelated critical paths such as user authentication and login workflows.

To mitigate these disruptions, OpenAI evaluated workload isolation by dedicating separate Cosmos DB containers to distinct teams. While isolated containers eliminated cross-feature noisy-neighbor interference and decoupled physical partitions, single-tenancy introduced prohibitive operational overhead. Aggregate growth patterns across fragmented products proved unpredictable, forcing individual application teams into continual capacity planning and leading to pervasive over-provisioning or acute launch-day scaling emergencies. Recognizing that fragmented infrastructure was unsustainable, the Habitat engineering team executed a deliberate transition to a refined multi-tenant architecture. This third phase established multi-tenancy by default, supported by centralized capacity management, fine-grained hot-key throttling, priority-based QoS policies, and transparent data-movement capabilities to relocate high-demand datasets dynamically.

Overcoming Global Latency: Distributed Caching and Intelligent Data Placement

As Habitat stabilized its multi-tenant foundation, latency emerged as the primary bottleneck for ChatGPT's worldwide user base. Because compute and storage resources are distributed across roughly 40 distinct cloud regions, user interactions frequently required cross-region round trips, inflating global read p50 latencies to approximately 120 ms. Given that Habitat workloads are overwhelmingly read-heavy—with client services repeatedly querying user configuration, conversational state, and metadata—OpenAI implemented a tiered caching and data placement model.

First, OpenAI established geographic affinity, co-locating primary data near users whenever local capacity permitted. Second, for small, strictly latency-critical datasets, OpenAI enabled broad multi-region replication. For all other application traffic, Habitat introduced an integrated lookaside caching tier powered by Valkey nodes co-located with user-facing compute clusters. Unlike conventional caching layers that rely on fixed Time-to-Live (TTL) mechanisms—which force a detrimental tradeoff between data staleness and low hit rates—Habitat adopted a globally synchronized cache invalidation design modeled after large-scale social graph architectures. Invalidation events are emitted through Azure Cosmos DB's change feed via Change Data Capture (CDC), allowing local caches worldwide to evict stale entries within seconds of a write. By intercepting high-frequency reads locally and deferring to the underlying Cosmos DB database primarily on cache misses, the platform drastically reduced cross-region network traversal.

Service Modernization and the Horizon of Exabyte Storage

Performance gains at the database and caching tiers were complemented by internal runtime modernizations. Habitat’s middle-tier proxy service, which sits between consumer products and backing databases to enforce routing, serialization, rate limiting, and residency controls, underwent a complete rewrite. Originally implemented in Python, the proxy was re-architected in Rust. This migration eliminated internal runtime scheduling bottlenecks, reduced memory overhead, and suppressed serialization latency, removing systemic jitter from every read and write cycle.

Together, these cumulative optimizations—managed multi-tenancy controls, regional Valkey caching with CDC invalidations, targeted database replication, and Rust proxy pipelines—shrank Habitat's global read p50 latency from 120 ms to 10 ms. However, with ChatGPT continuing to experience rapid adoption, the sheer physical footprint of conversational transcripts, agent memory logs, and bulky unstructured payloads continues to compound. To prepare for traffic scaling well past the current 500 petabyte boundary, OpenAI confirmed that it has begun developing an in-house, exabyte-scale online storage architecture tailored specifically for bulky data types, ensuring the underlying infrastructure can scale smoothly to support the platform's next billion users.

---

Industry Impact

OpenAI's architectural disclosures provide a rare, highly instructive benchmark for engineering teams constructing infrastructure for hyperscale AI applications. The reality of modern generative AI platforms is that while model inference draws significant attention, traditional stateful OLTP systems remain the backbone of interactive usability, conversation context delivery, and user experience.

Furthermore, the evolution of Habitat underscores a broader industry pattern regarding managed cloud services versus custom infrastructure. Rather than replacing managed cloud components outright during periods of rapid growth, OpenAI demonstrated the value of deep co-engineering with cloud providers—pushing Azure Cosmos DB to tens of millions of requests per second while compensating for platform constraints through specialized upper-layer abstractions like CDC-driven caches and custom proxying. As artificial intelligence systems shift toward persistent memory, multi-agent coordination, and real-time retrieval, Habitat’s lessons in partition isolation, automated data relocation, and caching will serve as a foundational blueprint for distributed systems engineering across the AI industry.

---

Frequently Asked Questions

How did OpenAI achieve a 12x read speedup in ChatGPT?

OpenAI reduced global read p50 latency from roughly 120 ms to 10 ms through three coordinated technical initiatives: deploying a distributed Valkey caching tier synchronized via Cosmos DB Change Data Capture (CDC), broadly replicating small, latency-sensitive datasets across regions while placing user data locally, and eliminating proxy software overhead by rewriting Habitat's core service layer in Rust.

Why did OpenAI revert to multi-tenancy after testing isolated containers?

While dedicating independent Azure Cosmos DB containers to specific workloads solved noisy-neighbor issues, it introduced unsustainable operational costs and management overhead. Product engineering teams were forced to manage complex capacity planning independently, leading to widespread over-provisioning and launch-day capacity bottlenecks. Returning to a shared multi-tenant structure allowed OpenAI to pool capacity efficiently while mitigating interference using hot-key throttling, Quality of Service (QoS) controls, and dynamic data migration.

What role does Change Data Capture (CDC) play in Habitat's cache reliability?

Rather than relying on short Time-to-Live (TTL) expiration policies—which either serve stale user data or overwhelm databases with cache misses—Habitat monitors Azure Cosmos DB’s change feed. When a write occurs, a dedicated pipeline consumes the CDC event and dispatches cache eviction commands to localized Valkey instances worldwide within seconds, ensuring read caches remain fresh without sacrificing high hit rates.