Systems at Scale: Storage, Scheduling, Retrieval and Features

Outside the agent stack, Oct 23–29 brought unusually concrete numbers from large production systems: ChatGPT’s storage layer moving back to multi-tenancy, Uber scheduling by compute capacity instead of cores, Meta fusing ads retrieval into one GPU kernel, and Airbnb cutting embedding staleness from two days to seconds. Google Cloud added managed blue-green upgrades and larger vector indexes to its databases.

Part of This Week in AI & Systems #3: Rules in Code, Controls in IAM.

---

What’s new this week

• OpenAI Habitat, Part II (Oct 23). How ChatGPT’s storage platform went from one shared Azure Cosmos DB container to dedicated containers and back, and cut global read p50 from about 120 ms to about 10 ms.
• Uber’s performance-aware scheduling (InfoQ, Oct 27). Fresh analysis of how Uber’s Odin allocates normalised compute capacity rather than CPU cores to stateful workloads; Uber’s own post dates from Oct 13.
• Meta Andromeda (Oct 28). A new version of Meta’s ads retrieval engine that runs model execution and system logic in one fused GPU kernel.
• Airbnb Chronon (InfoQ, Oct 28). Two additions to the open-source feature platform that keep guest embeddings fresh within seconds.
• Cloud SQL and AlloyDB (Oct 28). Managed blue-green deployments for Cloud SQL for MySQL (preview) and AlloyDB ScaNN indexes for up to 10 billion vector rows (GA).

Why it matters. These are textbook design trade-offs with fresh production numbers: single- versus multi-tenant storage, cache invalidation by change data capture, absolute versus normalised resource requests, and streaming feature updates without putting a model in the request path.

Key ideas

Storage: multi-tenant again, with better controls

• A full circle. Habitat serves more than 500 PB across 40 regions, and its largest Cosmos DB account handles more than 20M QPS. A shared container suffered colliding partition keys, noisy neighbours and “weak partitions”, where splits reduce per-partition throughput. Dedicated containers fixed isolation but pushed capacity planning onto every product team, which led to launch-day incidents.
• What made sharing work. The return to one shared container came with hot-key throttling, QoS tiers that throttle offline traffic first, and a new serverless mode, built with the Azure Cosmos DB team, that splits on request load as well as storage.
• Reads: replicate, cache, rewrite. Reads are over 90% of traffic. Habitat replicates small, latency-critical datasets widely, uses Valkey caches that are invalidated through change data capture within seconds, in the style of Meta’s TAO, and rewrote its proxy from Python to Rust. Together, these cut global read p50 from about 120 ms to about 10 ms. The post also covers a failback “connection stampede” outage and OpenAI’s point that managed services become less “managed” at this scale.

Scheduling: capacity, not cores

• Cores are not equal. Single-core performance across Uber’s roughly 5 million Odin cores varies by almost 2×. Stateful shards can’t shed load to faster hosts the way stateless services can, and a leader can use more than ten times the CPU of its followers.
• Normalise, then convert. Odin gives each server type a performance factor, with an older generation as 100 and the newest x86 at about 190. Factors start from hardware qualification and are refined with production data for each technology. Targets are expressed in normalised compute units and converted to physical cores on whichever host a workload lands.
• The result. Across about 1.4M auto-scaled cores, the spread in Kafka utilisation fell from about 8 to 3 percentage points, and total Kafka CPU allocation fell about 8%. Analysis: Kubernetes CPU requests are absolute millicores, so clusters that mix machine families face the same unevenness for stateful systems unless requests are adjusted per machine type.

Retrieval and features: move the work to where the data is

• One fused kernel. Meta’s new Andromeda scores every eligible ad and returns top candidates at over a billion ads per second, which Meta calls a 100× lift over the 2024 system. At that throughput the bottleneck is memory bandwidth and links between hosts, not compute. Fusing model execution and system logic into one GPU kernel reaches over 10 billion ads per second on a single GH200, and cut mixture-of-experts gating cost by about 300×.
• Fresh embeddings without the model in the request path. Chronon’s Push Mode emits an event when a feature value is committed, and Near Real-Time Model Transform runs the already-deployed Transformer encoder inside the streaming pipeline. Guest embeddings now update within 10 to 30 seconds of activity instead of being nearly two days stale. Airbnb reports a 1.67% gain in offline NDCG and about 0.33% more uncancelled bookings in A/B tests.

Managed data services

• Blue-green for Cloud SQL. Cloud SQL for MySQL now offers managed blue-green deployments in preview: an upgraded green copy stays in sync, can be tested with real traffic, and takes over the IP address, instance name and backups at switchover. Downtime is under a second on Enterprise Plus, and PostgreSQL support is coming.
• Larger vector indexes in AlloyDB. The Oct 28 release notes bring four-level ScaNN indexes for up to 10 billion vector rows, streaming ScaNN scans for better recall on filtered searches, and IAM group authentication to GA.

Analysis: two themes repeat. Isolation has a cost that falls on whoever does capacity planning, and Habitat moved that cost back to a central team with better primitives. And when data movement dominates, as in Andromeda and Chronon, moving the computation next to the data beats scaling the hosts around it. Open question: OpenAI’s next step, a custom exabyte-scale store described at the end of the post, will show whether co-managing a cloud database remains viable at its scale.

Reading list

Grouped loosely: storage (OpenAI), scheduling (InfoQ), retrieval and features (Meta, InfoQ) and managed data services (Google Cloud).

• Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II) — OpenAI, Oct 23, 2026. The three phases of tenancy, the Cosmos DB fixes, the read-latency strategy, CDC at scale and the failback outage.
• Uber Moves beyond CPU Cores with Performance-Aware Scheduling for Stateful Workloads — InfoQ, Oct 27, 2026. Performance factors, normalised compute units and the Kafka results.
• Meta Andromeda: Powering the Next Generation of Ads Retrieval Through Model-System Co-Design — Engineering at Meta, Oct 28, 2026. Why memory bandwidth became the limit and how the fused kernel works.
• Airbnb Cuts Recommendation Staleness from 2 Days to Seconds with Chronon — InfoQ, Oct 28, 2026. Push Mode, Near Real-Time Model Transform and the reported online and offline gains.
• Cloud SQL blue-green deployments — Google Cloud, Oct 28, 2026. How the green copy is kept in sync, tested and switched over.
• Google Cloud release notes, Oct 28 — Google Cloud, Oct 28, 2026. The AlloyDB ScaNN, streaming scan and IAM group authentication GA notes.

Background and prior art

• Ensuring Similar Computational Resources to Stateful Workloads Running on a Diverse Server Fleet — Uber Engineering, Oct 13, 2026 (background, outside this issue’s window). Uber’s original post behind InfoQ’s analysis, with the performance-factor method and production data in Uber’s own words.
• TAO: Facebook’s Distributed Data Store for the Social Graph — Bronson et al., USENIX ATC 2013. The cache design Habitat’s CDC-driven invalidation follows.
• Earlier in this series: The Cost of an Agent Step (Oct 24) on serving-side levers such as KV-cache reuse, another case of keeping computation close to its data.

Enjoyed this? Follow me on X or connect on LinkedIn.