Your "100 Requests per Second" Rate Limiter Might Actually Allow 1,000

You build an API and decide each customer gets 100 requests per second. You add a rate limiter, test it locally, and it works. Request 101 gets a 429. You ship it.

Production runs ten instances behind a load balancer. That same customer can now push close to 1,000 requests per second, because every instance keeps its own counter and every counter thinks the customer is fine. Nothing is broken on any individual server. The limit lives inside each process while your policy was supposed to apply across the whole system.

I call it the N × L problem:

Effective limit = number of independent limiters × configured limit

It never shows up in local testing. It shows up when you scale out.

The typical in-memory limiter keeps a map of user to request count. One process, one map, and everything behaves. Spread the same code across five instances and each one tracks its own view of the same user. Each instance might see 70 or 80 requests and happily allow more, while the user has already made close to 400 across the fleet.

Switching algorithms does not help. Fixed window, sliding window, token bucket, leaky bucket — it does not matter. The servers cannot enforce a shared policy because they do not share state. This is a coordination problem, not an algorithm problem.

Autoscaling makes it worse in a sneaky way. Traffic spikes, the cluster scales from three pods to ten, and your customer's effective quota just tripled. An infrastructure event silently became a quota increase. Nobody approved it and nobody will notice until the bill or the incident.

Move the bucket out of the process. All instances consume tokens from one logical bucket in Redis, keyed per customer. Stripe has written publicly about running token bucket limiters on Redis this way.

The one thing that actually matters here is atomicity. If the check and the decrement are separate operations, two servers can read the same last token and both allow a request. The read, refill calculation, check, and write have to happen as one indivisible step, which in Redis usually means a Lua script. Get that wrong and your "global" limiter still leaks under concurrency, just less obviously.

I tested this with five local instances of the same server. In local mode, a burst of 500 requests from one user got 500 allowed: each instance let through its own 100. Same test against a shared Redis bucket: 100 allowed, 400 rejected. The instance count stopped mattering, which is what "global" was supposed to mean in the first place.

Shared state puts a network call in your request path. At serious volume you start caring about Redis latency, hot keys, cross-region round trips, and what happens when Redis goes down. Larger systems tend to land on one of three shapes.

The Redis design above. Good when limits need to be accurate, traffic is regional, and quotas attach to customers or API keys.

Key by customer and route, not one giant global key, or you have built yourself a coordination hotspot.

Envoy supports both layers on the same route: a local per-node bucket runs first, then a global service backed by Redis.

The two limiters have different jobs. The local one protects the node and absorbs obvious abuse cheaply. The global one protects the customer quota so that adding proxies does not multiply anyone's allowance. It also degrades sensibly: if the global service is slow, the local layer still blocks the worst traffic.

Cloudflare Durable Objects are the cleanest mental model here. One object per customer, and requests from Singapore, Dubai, and London for that customer all land on the same instance, which owns the bucket.

Just do not create one object for your whole platform. A single object runs on a single thread and has real throughput limits. Scale out across many objects, not up inside one. And accept the latency: exact global state means some requests travel to wherever that object lives. Coordination always costs something.

Cloudflare's "Counting things, a lot of different things" post makes a point I keep coming back to. They could not route every edge request to a central counter, so they scoped coordination to a point of presence, used a sliding-window approximation, and moved counter updates off the hot path.

A rate limiter does not need perfect global accuracy. It needs semantics that match what it protects. A billing quota needs account-wide coordination. DDoS mitigation at the edge is better served by a fast per-location approximation.

Before picking any technology, define what "global" means for you: per process, per cluster, per region, per customer worldwide? Most rate limit bugs I have seen started because a team said "global" without drawing that boundary.

The specific limiter algorithm matters less than the policy boundaries and operational decisions around it.

For authenticated APIs, key on the customer or API key, not the IP. Corporate NAT, mobile carriers, and universities put thousands of legitimate users behind one address.

Redis is down: do you allow the request or reject it?

For a public read endpoint, failing open is reasonable. For an expensive GPU inference endpoint, failing open is a cost incident waiting to happen. For a login brute-force limiter, failing open weakens security.

There is no universal answer. Write the policy down per limiter, and monitor how often it triggers.

A bare 429 turns rate limiting into a retry storm. Return Retry-After and the draft RateLimit headers so clients can back off properly, with jitter, instead of hammering you in sync.

One huge customer at 100k RPS updates one Redis key, and one key lives on one shard no matter how big your cluster is.

The usual escape hatch is batching: each node reserves ten tokens from the global bucket and burns them locally. Fewer Redis calls, at the cost of bounded over-allocation. Five nodes with ten reserved tokens each can exceed the true limit by about 50 requests.

More coordination buys accuracy and costs latency. Less coordination buys speed and costs accuracy. Distributed systems do not remove tradeoffs, they relocate them.

One rollout note from experience: launch new limiters in shadow mode first. Record what would have been rejected, allow everything, watch the metrics, then enforce.

The most dangerous limiter is not the one that misses an attacker. It is the one that blocks your biggest legitimate customer.

A local limiter protects a process. A global limiter protects a policy. They are not the same thing.

Whenever you write "100 requests per second," ask: 100 per second where? Per pod? Per region? Per customer across the platform? If the limit has to survive horizontal scaling, the state has to be shared, coordinated, or deliberately approximated. Otherwise every new server ships with its own fresh bucket, and your limit quietly becomes N × 100.

Simple once you see it. The dangerous part is how long a system runs before anyone does.