Why We Run Hundreds of Copies of the Same Thing on Purpose

The worst outage I was ever part of wasn't caused by a clever attacker or some freak hardware failure. It was a one-line config change. Someone shipped it, it rolled out to every server we had in about ninety seconds, and then every server fell over at the same moment. Every customer. All at once.

There was no healthy region to fail over to, because the broken change was everywhere. We sat there watching the graphs flatline, and the only way out was to roll the change back — which always takes longer than breaking things did. We were down for the better part of an hour. For a few of those customers, that hour cost them real money.

I think about that outage every time a junior engineer looks at AWS's infrastructure and asks the obvious question: why on earth do they run so many copies of the same thing? It looks wasteful. It looks like a maintenance nightmare. And it's the single biggest reason an outage like mine doesn't happen to them.

Let me walk you through how it works.

Instead of running one big system that serves all your customers, you run a bunch of smaller, identical, fully independent systems. Each one is a complete stack — its own app servers, its own database, its own cache, its own everything. We call each of these a cell.

A cell is not a microservice. It's not a shard of your database. It's the whole thing, copied. If your product is "an app server talking to a Postgres database talking to a Redis cache," then one cell is one app server, one Postgres, one Redis, wired up and serving traffic on their own. Cell 2 is another complete copy of that. Cell 7 is another.

Then you put a thin layer in front — a router — whose only job is to look at a request, figure out which customer it belongs to, and send it to that customer's cell. That's the picture at the top of this post: requests arriving, the router handing each one to a single cell. It maps a customer to a cell the same way every single time: if a customer's data lives in cell 3, every request from that customer has to land on cell 3, or it'll be looking at the wrong database. So the router does one job, and does it the same way forever.

Here's the part that trips people up. Everything you've been taught about good engineering pushes you toward consolidation. Don't repeat yourself. One source of truth. Simplify by merging things. Fewer moving parts.

Cell-based architecture says: no, run more parts. Run way more parts. And it's not because the AWS folks forgot the basics. It's because they're optimizing for something different. They're not trying to prevent failure. They've given up on that. Werner Vogels, their CTO, has a line that gets quoted to death precisely because it's correct: "Everything fails, all the time."

Once you actually believe that — not as a slogan, but as a planning assumption — your job changes. You stop asking "how do I keep this from breaking?" and start asking "when this breaks, how few people can I make it hurt?"

That second question is the whole game. And the answer is: don't let any single failure touch your entire customer base. Wall the customers off from each other so a fire in one room doesn't burn down the building.

That's what a cell is. A room with walls.

"Blast radius" is the term for how much damage one failure can do. With cells you can write it down as a simple fraction: the number of customers in one cell divided by your total number of customers.

Say you have 100,000 customers in one big system. A bad deploy takes down all of them — blast radius is 100%. Total loss.

Now split them across 10 cells, 10,000 customers each. That same bad deploy, if it only hit one cell, takes down 10,000 customers. Blast radius: 10%. Still bad! But 90% of your customers never noticed. Your on-call gets paged about one cell, not the apocalypse. And crucially, you've got nine healthy cells to compare against, which makes it obvious what broke and where.

Go to 100 cells and one cell failing is a 1% event. Most of your customers, most of your revenue, and most of your reputation sail through untouched.

You can see the obvious tradeoff already: more cells means a smaller blast radius, but more cells also means more things to operate, deploy, monitor, and pay for. There's no universally correct number. Picking it is a judgment call between "how much damage can I tolerate from one failure" and "how much operational weight can my team carry." Start with a handful and grow. Don't go build a thousand cells on day one to feel clever.

Three failure modes come up over and over, and cells contain all three.

1. The bad deploy. This is my one-line-config story. A broken change can only break the cells it's been deployed to. If you roll your deploys out cell by cell (more on this in a second), a bad change hits one cell, you catch it, and the other 99 are still fine.

2. The poison-pill request. Sometimes a single request is malformed in a way that crashes the thing handling it — a pathological input, a query that triggers a bug, a payload that blows up memory. In one big system, that customer's bad request can take everyone down with it. In a cell, the blast is contained to that cell's customers.

3. The hot key / noisy neighbor. One customer suddenly sends 50x their normal traffic, or has a data pattern that hammers the database. In a shared system they degrade everybody. In a cell, they can only saturate their cell. Everyone in the other cells keeps humming along, blissfully unaware.

The pattern across all three is the same: the damage stops at the cell wall.

This is the part I care about most, because deploys are how we hurt ourselves more than any attacker does.

When you have cells, you stop deploying to "production" as one big bang. You deploy cell by cell, and each cell has to prove itself before the next one gets the new code:

1. Push the new version to one cell.
2. Bake — let it run for a while under real traffic while you watch its health. Error rates, latency, the metrics that actually tell you whether customers are having a good time.
3. If it looks healthy, move to the next cell. If it looks sick, roll that one cell back automatically and stop the whole rollout cold.

Bake time is the unglamorous hero here. A lot of bad changes don't fail in the first ten seconds. They fail after a cache fills up, or after a slow memory leak, or when the first heavy customer wakes up. Letting a cell soak for ten or thirty minutes before you trust it catches the slow-burn failures that a quick smoke test would wave right through.

And because only one cell ever has untrusted code at a time, the absolute worst case for a bad deploy is one cell's worth of customers, for the length of one bake window, caught by an automatic rollback. Compare that to my outage: everyone, instantly, with a manual scramble to recover. Same bug. Wildly different day.

There's one piece every request in this design has to pass through: the router. And the discipline that makes cells work is keeping that router as close to brainless as you can get away with.

Its entire job is to look at a request, figure out which cell owns that customer, and forward it there. That's it — a lookup and a forward. The customer is mapped to a cell by something boring and deterministic, like hashing their ID, so the same customer always lands on the same cell and always reaches the database that holds their data.

The temptation is always to make the router do a little more: a quick auth check here, some request shaping there. Resist it. The router is the one component that touches everyone's traffic, so any logic you bury in it becomes logic that can fail for everyone at once. The moment it starts making real decisions, you've quietly rebuilt the one-big-system problem — a shared brain is a shared point of failure. In serious setups people push the routing even further out, into DNS or into the client SDK itself, so there isn't even a central router to take down. The pattern is always the same: the less the shared layer does, the less it can break for everybody.

Pair that dumb router with the cell-by-cell rollout from the last section and you have the whole machine. A broken version can never get past the first cell it poisons, because the rollout refuses to advance past a sick cell. One cell takes the hit, rolls back on its own, and the damage stops there. None of this is clever — and that's the point. The reliability comes from the boring parts staying boring.

Once you're comfortable with plain cells, there's a genuinely beautiful refinement worth knowing about, because AWS uses it in places like Route 53 (their DNS service).

Plain cells have one weakness against the poison pill. If customer X is assigned to cell 3, and X sends a request that crashes cell 3, then everyone in cell 3 goes down with them. The wall protects the other cells, but the people sharing X's room are still toast.

Shuffle sharding fixes this with a trick. Instead of putting each customer in one cell, you give each customer a small random combination of workers. Picture eight workers. Each customer gets assigned a random pair of them:

• Customer X gets workers {2, 5}
• Customer Y gets workers {2, 7}
• Customer Z gets workers {4, 6}

Now X sends the poison pill and takes down both worker 2 and worker 5. What happens to everyone else?

Customer Y shares worker 2 with X, so Y loses one of their two workers — but Y still has worker 7. With a retry and some failover, Y keeps working at reduced capacity. Customer Z shares nothing with X and doesn't even notice. The only customers fully knocked out are the ones who happen to have X's exact pair, {2, 5}, and that's a tiny slice of everyone.

Here's where it gets fun. With 8 workers and pairs, there are only 28 possible combinations, so the odds another customer has your exact pair are about 1 in 28. But scale it up and the math explodes in your favor. With 100 workers and a shard size of 5, the number of unique combinations — 100 choose 5 — works out to 75,287,520. That's over 75 million distinct shards. So even with millions of customers, the chance that any two of them share the same complete shard is almost nothing. One customer's meltdown can clip a worker here and there for a few unlucky neighbors, but it can't take down a meaningfully large group, because no meaningfully large group shares their exact set of workers. You've turned "blast radius = my whole cell" into "blast radius = a rounding error" without buying a single extra server.

You don't need this on day one. But it's the kind of idea that, once it clicks, changes how you think about isolating tenants from each other.

I'd be doing you a disservice if I made cells sound free. They aren't.

You're now operating N copies of everything. N databases to back up, patch, and migrate. Monitoring has to be cell-aware or you'll drown in noise. Schema changes have to roll out across every cell, which is its own discipline. And anything that needs to span cells — a report across all customers, a feature where customer A interacts with customer B who lives in a different cell — gets harder, because the whole point was to keep cells from knowing about each other. You either design those cross-cell paths very carefully or, better, you design your product so you rarely need them.

So: if you're a three-person startup with a hundred users and one database, you do not need cells. You need to ship. Cells are what you graduate into when an outage stops being "oops" and starts being "we lost a customer and made the news." The complexity is a real cost, and it only pays off once the cost of a full-system outage is high enough to justify it.

If you take one thing from all this, make it the shift in how you think, not the diagram.

The instinct most of us start with is prevent failure. Write better code, add more tests, be more careful. All good, all worth doing — and all guaranteed to be insufficient, because everything fails eventually no matter how careful you are. The senior move is to assume the failure is coming and pour your energy into making sure that when it lands, it lands on as few people as you can manage.

Cells are one of the cleanest ways to do that. Walls between customers. Deploys that prove themselves one room at a time. A failure that ruins someone's afternoon instead of everyone's quarter.

My one-line-config outage took down every customer we had because we'd built one big room with no walls in it. If we'd had cells, that same dumb mistake would've been a footnote in a postmortem instead of an hour of everyone staring at flat graphs. That's the difference. It's worth building for.