A thousand requests arrive for the same uncached key.
Should Postgres execute a thousand identical queries?
OpenAI’s answer is to let one reader fetch the data while the others wait for the cache to be populated. Its PostgreSQL engineering article describes cache locking and leasing as a defense against sudden bursts of cache misses.
The interesting part is what happens during those few milliseconds between a cache miss and a completed refresh.
That gap can turn a perfectly normal traffic spike into a database incident.
01The dangerous gap in cache-aside
Consider an illustrative endpoint:
GET /profiles/42The application checks Redis. If the profile is missing, it queries Postgres, stores the result in Redis, and returns it.
That works well when requests arrive far enough apart.
But suppose a popular key expires while hundreds of requests are arriving. The first request starts its database query. Before it finishes, the next request checks the cache.
Still empty.
The next hundred requests see the same thing.
Every request follows the correct application logic. Together, they create unnecessary database work.
This is a cache stampede: many callers independently rebuild the same missing value.
For an illustrative hot key receiving 2,000 requests per second, a 100 ms refill window could admit roughly:
2,000 × 0.1 = 200
requests before the cache becomes available.
Without coordination, that can mean around 200 database reads for one value. This is a simplified example, not an OpenAI benchmark.
If those reads slow the database, the refill window gets longer. More requests enter it. The stampede grows.
02How overload feeds itself
OpenAI reports that several PostgreSQL incidents followed a recurring pattern: an upstream problem increased database load, latency rose, requests timed out, and retries amplified the pressure. Cache failures were one possible trigger; expensive queries and write storms were others.
A timeout tells the caller it did not receive a result in time. It does not necessarily mean the database stopped working on the original query.
A retry can therefore add work while the first attempt is still consuming resources.
The system needs a way to prevent redundant work from entering the database in the first place.
03Make one request responsible for the refresh
OpenAI describes coordinating cache misses by key: one request acquires a lock, reads PostgreSQL, and repopulates the cache. Other requests wait for the updated value.
For our profile example, the flow becomes:
- Check the cache.
- On a miss, try to become the refresh owner for
profile:42. - The owner checks the cache again, then queries Postgres if necessary.
- The owner publishes the result.
- Waiting requests read the populated cache.
That second cache check matters. Another request may have completed the refresh between the initial miss and acquiring ownership.
The lock should be scoped to the cache key. Refreshing one profile should not block requests for unrelated profiles.
The goal is simple: callers asking for the same missing value share the refresh work.
04What singleflight gives you in Go
Go’s singleflight package suppresses duplicate calls within a shared Group. Concurrent calls using the same key wait for one execution and receive its result.
For a single application process, that is a useful building block.
But imagine the service has ten replicas. Each replica has its own memory and its own singleflight.Group.
Ten replicas can still perform ten refreshes for the same key.
| Mechanism | Coordination scope |
|---|---|
| Shared singleflight.Group | Callers inside one process |
| Shared Redis lease | Participating application replicas |
A prototype can combine both: singleflight collapses local callers, and a Redis lease coordinates the remaining refresh attempts across replicas.
OpenAI’s article does not specify that it uses Go, Redis, or this particular implementation. These are choices for demonstrating the published idea.
05Why the lock needs an expiry
What happens if the refresh owner crashes?
An indefinite lock could leave every subsequent request waiting for an owner that will never return.
A lease makes ownership temporary.
For a Redis-based prototype, acquisition can use:
SET refresh:profile:42 <unique-token> NX PX <lease-duration-ms>NX acquires the key only if it does not exist. PX sets its expiry. The unique token identifies the owner.
Release must atomically check that token before deleting the lock. Otherwise, an old owner could delete a newer owner’s lease after its own lease expired. Redis documents this ownership-check pattern.
However, expiry creates another edge case.
If a database query takes longer than the lease, a second request can acquire ownership while the first query is still running. Duplicate reads become possible again.
That is why “exactly one query” is a useful illustration of the healthy path, rather than an unconditional production guarantee. Lease expiry, failures, and failover complicate it.
For a cache refresh, occasional duplicate work may be acceptable. Publishing an older result over a newer value may not be. The design needs an explicit policy for ownership loss and cache publication.
06Waiting needs a budget
Suppressing database work does not make waiting free.
A slow refresh can accumulate many blocked requests. They consume memory, occupy connections, and may eventually exceed client deadlines.
For the prototype, followers should:
- Wait only within a bounded deadline.
- Recheck the cache with backoff and jitter.
- Retry lease acquisition within that budget if the owner disappears.
- Return a controlled error, or serve acceptable stale data, when the budget expires.
The dangerous fallback is letting every waiting request query Postgres after a short timeout. That merely delays the stampede.
Similarly, a failed refresh should not trigger unlimited immediate retries. Failure handling is part of the mechanism.
07One hot key is different from an empty cache
Per-key coordination is particularly effective when many requests want the same value.
It offers much less protection when every request wants a different value.
A thousand requests for one missing key can share a refresh. A thousand requests for a thousand missing keys may still require a thousand reads.
A complete cache flush can therefore overwhelm the database even when per-key leases work correctly.
And if the cache and coordination mechanism use the same unavailable Redis service, the lease itself may be unavailable.
These cases need broader controls: a limit on concurrent database fallbacks, load shedding, bounded retries, and enough capacity for recovery. OpenAI also describes rate limiting at multiple layers as part of its database protection strategy.
Cache locking solves duplicate refresh work. Database admission control bounds the total work.
08A prototype that makes the difference visible
Build a small Go service with Redis, Postgres, and several application replicas.
Compare three modes:
| Mode | Cache-miss behavior |
|---|---|
| Cache-aside | Every miss can query Postgres |
| Local singleflight | One refresh per key per process |
| Singleflight plus Redis lease | Refresh attempts coordinated across replicas |
Run a constant offered load of 2,000 requests per second. Warm the cache, then delete a selected hot key.
Graph database query attempts, endpoint p99 latency, errors, and the number of requests waiting for refresh. Include database pool wait time so that a saturated pool does not hide the pressure.
Then repeat with:
- Many distinct missing keys.
- A refresh owner killed mid-query.
- A query that outlasts its lease.
- Redis unavailable.
The expected result for the hot-key test is fewer redundant database reads with coordination. The size of the improvement, and its effect on p99 latency, must come from measurements.
No benchmark results are claimed here.
09Design the cache-miss path as carefully as the hit path
A cache hit is straightforward: retrieve a value and return it.
A cache miss raises harder questions.
Who rebuilds the value? How many callers can do that work? How long do followers wait? What happens when the owner crashes? What happens when every key is missing?
Those decisions determine whether a cache disruption becomes a manageable slowdown or a database overload.
OpenAI’s published mechanism highlights a practical principle: when many requests need the same missing data, coordinate the refresh before they reach Postgres.
The cache protects the database while values are present.
The miss path must protect it while they are being rebuilt.
Source: OpenAI — Scaling PostgreSQL to power 800 million ChatGPT users. The prototype design and numerical examples above are illustrative.
