Your Architecture Is Cellular. Your Deploy Pipeline Probably Isn't.

Teams spend months carving a big system into isolated pieces. Separate clusters, separate regions, per-tenant shards, whatever the flavor is. The goal is always the same: when one piece breaks, the rest keeps serving traffic.

Then they wire it all up to a single pipeline that pushes every release to every cell at once.

I've watched this happen at three different companies now, and it's the same conversation every time. The runtime is isolated. The release path isn't. So on a normal Tuesday you have twenty independent failure domains, and the moment somebody merges to main you have exactly one. Nobody notices until the bad deploy, because a global pipeline looks great in the happy path. It ships fast, it's simple, one Argo app, one button. The bill comes due later.

A cell is a mostly self-contained copy of your system. In practice it means one of:

• a Kubernetes cluster
• an availability zone
• a region
• a slice of your customers
• a shard with its own compute, cache, queue, and database

The "mostly" is doing a lot of work in that sentence, and we'll come back to it.

Say you have four cells split by customer:

Copy
Cell 1 → Customers A-D
Cell 2 → Customers E-H
Cell 3 → Customers I-L
Cell 4 → Customers M-Z

If Cell 2 falls over, the other three keep working. That's the whole point: failure containment.

But containment only holds if every operational path respects the boundary. Routing, data, config, queues, dashboards, and deploys. Miss one and you've got a decorative architecture diagram.

Here's the pipeline almost everyone starts with:

Copy
Release v2
   ├── Deploy to Cell 1
   ├── Deploy to Cell 2
   ├── Deploy to Cell 3
   └── Deploy to Cell 4

Now suppose v2 has a connection pool leak. Not a crash, not a failed health check, just a slow leak that only shows up under real production concurrency. Your staging environment will never find it.

Four minutes after the rollout completes, every cell is degrading at roughly the same rate. Support tickets don't say "Cell 2 is slow." They say the product is down.

The cells were isolated. The pipeline handed you a global failure domain anyway.

The line I keep repeating to people who are new to this: isolation isn't just about where your code runs, it's about how change enters the system. Your pipeline is a control plane, and a system is only as isolated as its most globally coupled control plane. Deploys, config pushes, feature flag services, and schema migrations all qualify.

Instead of one big fan-out, promote the release through groups of cells with a pause in between.

Copy
Wave 1: Cell 1
Wave 2: Cell 2, Cell 3
Wave 3: Cell 4 through Cell 20

The pause is the important part. People call it the bake period. Deploy, watch, decide, promote. Not deploy everywhere and find out from Twitter.

First wave should be the smallest production blast radius you can construct while still being real production. Twenty cells means you deploy to one and leave nineteen on the last known good version.

Copy
Before:  Cells 1-20  → v1
After:   Cell 1      → v2
         Cells 2-20  → v1

What you watch in Cell 1 depends on the service, but generally: 5xx rate, p99 latency, pod restarts, memory trend, DB query time, queue lag, auth failure rate, and at least one metric a product manager would recognize. Logins per minute. Checkouts completed. Something that breaks when customers are unhappy, not just when pods are unhappy.

Healthy, you continue. Broken, one cell is broken and nineteen are fine. That's the architecture doing the job you paid for.

One green cell is encouraging but not conclusive. Cells differ in traffic shape, tenant behavior, data volume, node sizes, and which upstreams they lean on hardest. A bug can hide in Cell 1 simply because Cell 1 never sends the request pattern that triggers it.

So the second wave expands to a small group that's more representative. Maybe three cells, ideally including one with your noisiest tenant and one in a different region. More load, more variety, still nowhere near everybody.

Then pause again.

Only after the earlier waves clear their gates does the release go fleet-wide. By then it has taken real traffic, several traffic patterns, multiple customer profiles, and an observation window long enough to catch drift.

This doesn't make the release safe. Nothing makes a release safe. It makes the odds much worse that one unknown bug takes out the entire customer base in the same five minutes.

This trips people up. Wave-based cell rollout is not a replacement for canary. They limit different things.

Cell waves control where the version goes:

Copy
Cell 1 → Cells 2-3 → the rest

Canary controls how much traffic sees it inside a given cell:

Copy
1% → 5% → 25% → 100%

So Cell 1's rollout looks like this internally:

Copy
v2 at 1%  traffic
v2 at 5%  traffic
v2 at 25% traffic
v2 at 100% traffic
Cell 1 promoted

Cell 2 doesn't start until Cell 1 finishes its own canary. Two independent limits stacked on top of each other, which is meaningfully safer than either one alone.

Rough model:

Copy
exposure ≈ (traffic % inside a cell) × (number of cells in the wave)

Take ten equal cells.

A global 10% canary means all ten cells are running the new binary and 10% of traffic hits it. Every failure domain has been touched. The number looks small, but there is no cell left holding a clean version, so a rollback is a fleet-wide operation.

A 10% canary in one cell means one failure domain touched, and if each cell carries a tenth of platform traffic, effective exposure is about 1% of total. Nine cells are still untouched and can absorb the traffic if you shift away.

Same percentage on the dashboard. Very different Tuesday.

This is why I push back when someone says "it's only 5%." Five percent of what, across how many failure domains?

A wave pipeline needs promotion gates that a machine can evaluate. Not "Rahul is watching Grafana and it looks fine." Rahul has three other things open and a standup in four minutes.

Pick thresholds like:

Copy
5xx rate            > 1%
p99 latency         > baseline + 20%
pod restart rate    > baseline
DB connection usage > 80%
queue lag           rising for 10 consecutive minutes
login success       < 99%
payment success     < your accepted floor

The exact numbers are yours. What matters is that you argued about them on a calm afternoon instead of during an incident, when everyone's judgment is worse and someone senior is asking for an ETA.

The pipeline should also be able to halt promotion, shift traffic back to stable, and roll back only the wave that failed. That last part is easy to get wrong. If Wave 2 goes bad, you should not be touching cells that were never in Wave 2. If your rollback is "redeploy v1 everywhere," you've rebuilt the global coupling in the recovery path instead of the deploy path.

Some risk isn't about the binary at all. Say v2 ships a new recommendation engine. The code might be perfectly stable while the feature quietly tanks conversion. Deploy safety and business safety are different problems.

Separate them:

Copy
1. Deploy the code with the feature off
2. Turn it on for internal users
3. 1% of users in Cell 1
4. 5%, 25%, 100% in Cell 1
5. Move to the next cells

When something goes sideways you flip a flag in seconds instead of running a rollback and waiting on an image pull.

That gives you three independent dials: which cells have the version, how much traffic sees it, and which behavior is live for that traffic. They're complementary. Teams that treat them as interchangeable end up using whichever one they built first for everything.

Nothing exotic. Usual suspects: Argo CD or Flux for GitOps, Argo Rollouts or Flagger for canary, Prometheus for metrics, and whatever traffic layer you already run (Istio, Linkerd, plain NGINX, Gateway API). Plus a flag service.

Cells get separate deploy targets:

Copy
clusters/
├── cell-01/
├── cell-02/
├── cell-03/
└── cell-04/

Waves live in config, checked into git, reviewed like anything else:

yaml
Copy
1release:
2  version: v2.4.0
3  waves:
4    - name: first-cell
5      cells: [cell-01]
6      bakeTime: 30m
7
8    - name: representative-cells
9      cells: [cell-02, cell-03]
10      bakeTime: 60m
11
12    - name: remaining-fleet
13      cells: [cell-04, cell-05, cell-06]

And inside a cell, Argo Rollouts steps the traffic:

yaml
Copy
1strategy:
2  canary:
3    steps:
4      - setWeight: 1
5      - pause: { duration: 5m }
6      - setWeight: 5
7      - pause: { duration: 10m }
8      - setWeight: 25
9      - pause: { duration: 15m }
10      - setWeight: 100

The tools matter less than the property you're buying: promotion logic that knows cell boundaries exist.

Back to "mostly self-contained." Most cellular systems I've seen still share something:

• one global Postgres
• one auth service
• one Kafka cluster
• one Redis
• one config service
• one third-party API
• one control plane

Change any of those and your careful wave rollout is irrelevant, because the shared thing is global by construction.

Favorite version of this: someone deploys v2 to a single cell, perfectly, textbook, and simultaneously runs a migration that drops a column on the shared database. The app rollout was cell aware. The ALTER TABLE was not. Every cell on v1 starts throwing errors, including the nineteen that were supposed to be your safety net.

Migrations need to be backward compatible. Expand and contract:

Copy
1. Add the new column, keep the old one
2. Deploy code that reads and writes both
3. Backfill in the background
4. Move every cell to the new code
5. Drop the old column, later, on purpose

Step 5 is where discipline goes to die. Write it down as a ticket with a date or it never happens, and you accumulate a schema full of columns nobody dares touch.

Cell-aware delivery is about the whole change, not just the Deployment object.

Treating the first cell as staging. It isn't. It has real customers who will really be annoyed. Wave 1 is a smaller production, not a bigger dev environment, and it doesn't excuse you from having actual pre-prod testing.

Bake times that are too short. A two minute pause catches a crash loop. It will not catch a memory leak, a queue backing up, a connection leak, a cache stampede after TTL expiry, or a cron job that only runs at midnight. If your failure mode is slow, your bake has to be slower.

Only watching infra metrics. CPU and memory can look beautiful while nobody can log in. Ask what the customer-visible symptom would be and put that on the gate.

Auto-promoting when metrics are missing. Absence of signal is not health. If the pipeline can't evaluate the gate, the correct default is to stop and page someone, not to shrug and continue. This one bites people who scrape metrics from the workload they just replaced.

Waves that aren't really waves. Calling the architecture cellular doesn't help if all twenty cells sit in one deploy group with a five minute gap. The structure has to actually reduce exposure.

Early on, most of us think reliability means writing code that doesn't break. That's a fine goal and you should keep trying.

Eventually you flip the assumption. The code will break. Something you didn't think about will reach production on a Friday. The useful question becomes: when this breaks, how much of the system goes with it, and how fast can I put it back?

That's all progressive delivery is. It doesn't prevent bad releases. It makes them smaller, slower, louder, and easier to reverse.

Cell architecture gives you isolation at rest. Cell-aware delivery preserves that isolation during change, which is the only time you actually need it. Without it, one pipeline quietly turns your distributed system back into a monolith every time somebody merges.

So when you review a cellular system, don't stop at "are the cells isolated." Ask whether you can deploy, pause, promote, and roll back one cell without touching the others.

If the answer is no, the isolation is theoretical.