We Pay Software to Break Our Own Servers on Purpose. Here's Why That's Not Insane.

Netflix runs a program in production, during business hours, whose entire job is to walk up to a random server and kill it. No warning. No maintenance window. Middle of the afternoon, peak traffic, someone's halfway through The Witcher — and a machine somewhere just gets its plug pulled.

The first time I explained this to a junior on my team, he looked at me like I'd told him we set fire to the office to test the sprinklers. His exact words were, "Isn't that just… causing outages?"

No. It's the opposite. And when Amazon rebooted roughly a tenth of its EC2 fleet in September 2014 to patch a Xen vulnerability — a thing that took down or degraded a lot of companies — Netflix mostly shrugged and kept streaming. They rode it out precisely because something had been quietly murdering their servers for years.

Let me teach yo¸whole thing, the way I wish someone had taught me. Grab a coffee. This one's worth understanding properly, not just nodding along to.

Your instinct — the one that made you flinch just now — is a good instinct. It's the instinct that keeps prod alive. Don't throw it away. But I need you to see where it quietly lies to you.

Here's the lie: "If nothing is breaking, the system is healthy."

That feels true. It is not true. A system that isn't visibly breaking might be:

• perfectly resilient, or
• one bad afternoon away from a total meltdown that nobody has ever tested.

From the outside, on a calm Tuesday, those two systems look identical. Same green dashboards. Same happy users. The difference only reveals itself the moment something goes wrong for real — which, by definition, is the worst possible moment to be finding out.

So the actual question isn't "is it broken right now?" The question is: "when a piece of this dies — and a piece will die — does the system as a whole survive?"

And here's the uncomfortable truth I want tattooed on your forehead: you do not know the answer to that question until you have watched it happen. Not reasoned about it. Not drawn it on a whiteboard. Watched it. With your own eyes, in an environment that actually matters.

That's the whole idea in one sentence. Everything else is mechanics.

The best way to survive failure is to fail constantly, on purpose, in small controlled doses — so that "failure" stops being a rare catastrophe and becomes a boring, well-rehearsed Tuesday.

Think about how humans handle fire. We don't wait for the building to actually burn to find out whether people know where the exits are. We pull the alarm on a random Wednesday, everyone grumbles, we time how long it takes to clear the floor, and we discover that the door by the kitchen was chained shut. We find the chained door during the drill, not during the fire.

Chaos engineering is fire drills for distributed systems. That's it. That's the mental model. If you remember nothing else, remember: we are running fire drills, and we run them often enough that a real fire feels like just another drill.

The reason this feels insane and the fire drill doesn't is purely familiarity. Nobody thinks fire drills are reckless. In twenty years, nobody will think chaos engineering is reckless either. You're just early.

Around 2010–2011, Netflix was in the middle of a huge migration off their own data centers and onto AWS. That's a big deal, because when you own the metal, you can kind of pretend it's reliable. When you rent thousands of virtual machines from a cloud provider, you cannot pretend anymore. Cloud instances vanish. They get slow. They get rebooted out from under you. Individual failure isn't an edge case in the cloud — it's the ambient weather.

Netflix's engineers realized something clever: if failure is going to be constant anyway, they should stop treating each instance failure as an incident to be mourned, and start treating it as the normal state of the world their code lives in. And the only way to force every engineer to build for that reality — instead of just promising they would in the design doc — was to make failure genuinely constant.

So they wrote Chaos Monkey. A little service that, during business hours, randomly terminates production instances. On purpose. Every day.

Why business hours? This detail is genius, so don't skip it. If your servers are going to die anyway (they are), do you want them dying at 3 PM when your whole engineering team is caffeinated, at their desks, and staring at dashboards? Or at 3 AM on a holiday weekend when the one person who understands the billing service is on a plane with no wifi?

Chaos Monkey forces failure to happen when you're watching. It converts terrifying 3-AM pages into boring 3-PM non-events. Read that twice. That's the entire strategic point of the "business hours" rule, and juniors miss it constantly.

They open-sourced the thing in 2012 under Apache 2.0, and it grew into a whole family they called the Simian Army. Which brings me to the zoo.

The Simian Army is best understood as a ladder of blast radius — how much you're willing to break at once. You climb it slowly, earning trust at each rung. A junior who tries to start at the top rung is a junior who's about to have a very educational quarter.

Chaos Monkey — kills one instance. The smallest unit. "Can we lose a single node and not notice?" If the answer is no, you have no business doing anything bigger. This is your foundation. Master this rung first.

Chaos Gorilla — kills an entire Availability Zone. Now we're asking a bigger question: "If a whole AZ falls over, does traffic fail over cleanly to the others?" AZ-level failures are real and they happen. Testing for them on purpose means you've already seen the failover work.

Chaos Kong — kills an entire AWS region. The nuclear option. "If we lose all of us-east-1, can we shift everything to another region and keep the lights on?" Netflix has actually done this. On purpose. To a service watched by tens of millions of people. And it worked, because they'd practiced.

Here's the ladder in one glance so it sticks:

Notice the progression: node → zone → region. You never skip a rung. You prove resilience at the small scale, build the automation and the confidence, and only then turn up the blast radius. This is a professional habit, not just a chaos-engineering one. Blast radius discipline is how you don't get fired.

Now, "randomly kill stuff" is a fantastic starting philosophy, but if you stopped there you'd be right to be nervous. Real chaos engineering at scale isn't a monkey flailing at a keyboard. It grew up. Netflix built two more sophisticated tools that I want you to understand, because this is where "reckless" turns into "rigorous."

FIT — Failure Injection Testing. Instead of killing a whole instance, FIT lets you inject a specific, surgical failure into a specific slice of traffic. Make calls to the recommendations service time out — but only for 5% of requests, only from a test account, only for the next ten minutes. That's not chaos. That's a scalpel.

ChAP — Chaos Automation Platform. This is the part that turns the whole thing from "stunt" into "science," so pay attention. ChAP runs chaos experiments as controlled A/B experiments with a built-in safety net.

Here's how it thinks. It spins up two tiny copies of the service you want to test: a control group (behaving normally) and an experiment group (with the failure injected). It routes a small, carefully chosen sliver of real traffic to each. Then it watches key business metrics — for Netflix, roughly "are people successfully starting streams?" And here's the crucial bit:

If the experiment group's metrics start diverging badly from control — if the injected failure is actually hurting real users — ChAP automatically aborts the experiment and pulls all traffic back to the healthy path.

That's the safety valve that makes the whole thing sane. You're not blowing up prod and hoping. You're running a measured experiment, on a tiny slice, with an automatic kill switch wired to the metrics that actually matter. The moment reality says "this is going worse than expected," the experiment ends itself.

This is the difference between the cartoon version ("lol we break servers") and the real discipline. The real discipline is:

1. Form a hypothesis. "If the ratings service disappears, the homepage should still render — just without star ratings."
2. Define your steady state. What does "healthy" look like as a number? Streams-started-per-second, checkout success rate, whatever your business actually cares about. If you can't measure "healthy," you can't do chaos engineering. Full stop.
3. Minimize the blast radius. Smallest slice of traffic, one instance, one AZ. Never more than you can afford to lose.
4. Have an abort button, and automate it. Metrics cross a threshold → experiment stops itself. No human required in the critical moment.
5. Run the experiment. Break the thing.
6. Learn. Did steady state hold? If yes, you've proven resilience instead of assuming it. If no, congratulations — you found a real weakness on a Tuesday afternoon instead of during a real outage. That's a win either way. There is no losing outcome to a well-designed experiment.

Internalize that last point. A chaos experiment that reveals a weakness didn't cause the weakness — the weakness was already there, silently waiting for the worst moment. You just dragged it into the daylight where you could fix it calmly.

Enough philosophy. You learn this in your fingers, not your head. We're going to build a mini Chaos Monkey for Kubernetes, in Go, and I'm going to walk you through every design decision — because the decisions are the lesson, not the syntax.

Our controller will:

• watch a single namespace,
• on a schedule (say, every few minutes),
• list the pods there,
• only touch pods that have explicitly opted in (safety first, always),
• pick one at random and delete it,
• and log everything loudly so we can watch the system heal.

Kubernetes makes this almost too easy, and that's a lesson in itself: when a pod dies, its Deployment's ReplicaSet notices the count dropped and immediately schedules a replacement. So a healthy, well-configured service should barely notice us. If it does notice — if killing one pod causes user-facing errors — then we just found a bug, and better now than at 3 AM.

I want to hammer on safety before we write a single line, because this is where the maturity of an engineer shows. The dangerous version of this tool kills anything. The professional version kills only what has raised its hand and volunteered.

We use an opt-in allowlist via a label. A pod only becomes a target if it carries chaos.bhupesh.dev/enabled: "true". No label, no touching. This flips the default to safe: the blast radius is exactly the set of things someone deliberately marked as expendable-for-testing, and nothing else. You should feel the difference in your gut. "Kill anything unless told otherwise" is how you end up writing a postmortem with your name on it. "Kill only what opted in" is how you keep your job and sleep at night.

go
Copy
1package main
2
3import (
4	"context"
5	"flag"
6	"fmt"
7	"math/rand"
8	"path/filepath"
9	"time"
10
11	corev1 "k8s.io/api/core/v1"
12	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
13	"k8s.io/client-go/kubernetes"
14	"k8s.io/client-go/rest"
15	"k8s.io/client-go/tools/clientcmd"
16	"k8s.io/client-go/util/homedir"
17)
18
19const chaosLabel = "chaos.bhupesh.dev/enabled"
20
21func main() {
22	namespace := flag.String("namespace", "default", "namespace to unleash chaos in")
23	interval := flag.Duration("interval", 5*time.Minute, "how often to strike")
24	dryRun := flag.Bool("dry-run", true, "log the victim instead of actually killing it")
25	flag.Parse()
26
27	clientset, err := buildClient()
28	if err != nil {
29		panic(fmt.Sprintf("could not build kubernetes client: %v", err))
30	}
31
32	fmt.Printf("chaos monkey online | namespace=%s | interval=%s | dryRun=%v\n",
33		*namespace, *interval, *dryRun)
34
35	ticker := time.NewTicker(*interval)
36	defer ticker.Stop()
37
38	for range ticker.C {
39		if err := unleash(clientset, *namespace, *dryRun); err != nil {
40			fmt.Printf("chaos round failed: %v\n", err)
41		}
42	}
43}
44
45func unleash(clientset *kubernetes.Clientset, namespace string, dryRun bool) error {
46	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
47	defer cancel()
48
49	pods, err := clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{
50		LabelSelector: fmt.Sprintf("%s=true", chaosLabel),
51	})
52	if err != nil {
53		return fmt.Errorf("listing pods: %w", err)
54	}
55
56	var candidates []corev1.Pod
57	for _, p := range pods.Items {
58		if p.Status.Phase == corev1.PodRunning && p.DeletionTimestamp == nil {
59			candidates = append(candidates, p)
60		}
61	}
62
63	if len(candidates) == 0 {
64		fmt.Println("no eligible victims this round — nothing opted in and running")
65		return nil
66	}
67
68	victim := candidates[rand.Intn(len(candidates))]
69
70	if dryRun {
71		fmt.Printf("[dry-run] would kill pod %q — safety's on, skipping\n", victim.Name)
72		return nil
73	}
74
75	fmt.Printf("killing pod %q — let's see if anyone notices\n", victim.Name)
76	if err := clientset.CoreV1().Pods(namespace).Delete(ctx, victim.Name, metav1.DeleteOptions{}); err != nil {
77		return fmt.Errorf("deleting pod %s: %w", victim.Name, err)
78	}
79
80	fmt.Printf("%q gone. The ReplicaSet should heal it now — go watch.\n", victim.Name)
81	return nil
82}
83
84func buildClient() (*kubernetes.Clientset, error) {
85	if cfg, err := rest.InClusterConfig(); err == nil {
86		return kubernetes.NewForConfig(cfg)
87	}
88
89	kubeconfig := filepath.Join(homedir.HomeDir(), ".kube", "config")
90	cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
91	if err != nil {
92		return nil, err
93	}
94	return kubernetes.NewForConfig(cfg)
95}

Anyone can copy client-go boilerplate. What separates a senior from a junior is why each line is the way it is. So:

dry-run defaults to true. A tool that causes damage the instant you run it, before you've even confirmed it's pointing at the right cluster, is irresponsible. You should have to consciously flip the safety off with --dry-run=false. Notice how the default state of the entire program is "look, don't touch." That's not laziness — that's a design principle. Dangerous defaults are how accidents happen.

Server-side label filtering. We don't list all pods and then filter in Go. We ask the API server to return only pods with our label. Why does that matter? Because it means a pod without the opt-in label is invisible to our tool. It can't be selected by accident, because it was never in the list. We made the unsafe outcome structurally impossible, not just "unlikely if the code is correct." When you can make a mistake unrepresentable instead of merely unlikely, always do that.

We only target Running pods with no DeletionTimestamp. Killing a pod that's already dying is a no-op that pollutes your logs and teaches you nothing. Precision matters. A chaos experiment where you're not sure what you actually broke is a useless experiment.

Randomness is deliberate, not lazy. A junior might think "why not just kill the oldest pod, or a predictable one?" Because then you'd only ever be testing survival of that specific failure. The whole point is robustness to any single pod disappearing. Randomness stops you from accidentally hardening against one scenario while staying fragile to all the others.

The ticker is a stand-in for "business hours." In this toy version it just fires every N minutes. In a real one, you'd wrap it in a check: is it a weekday? Is it between 10 AM and 4 PM in the team's timezone? Is nobody currently mid-incident? Never run chaos during a real outage — that's not brave, that's cruel to your on-call.

I stripped the comments on purpose, by the way. Good Go isn't supposed to be littered with them — if you find yourself explaining what a line does, the line probably isn't clear enough. The names carry it: unleash, candidates, chaosLabel, dryRun. You read it top to bottom and it just says what it means. The why behind the tricky decisions lives in prose, where it belongs.

Running the tool is 10% of the exercise. The other 90% is what you have on the other screen. Here's the setup that makes this a genuine experiment instead of vandalism:

1. Deploy a small service with, say, 3 replicas, and slap the chaos.bhupesh.dev/enabled=true label on its pods.
2. Point a load generator at it — something hammering the endpoint continuously and recording the success rate. That success rate is your steady state. That number is the experiment.
3. Put that success rate on a dashboard — Grafana, whatever you've got. One big number: percentage of requests succeeding.
4. Run the chaos controller with --dry-run=false.
5. Watch the dashboard while a pod dies.

Here's the whole thing in one payoff sentence: if your service is genuinely resilient, you'll watch a pod get deleted, watch the replica count drop to 2, watch Kubernetes schedule a replacement, watch it come back to 3 — and that success-rate number never budges. Users never knew. That flat line, while you are actively killing infrastructure underneath it, is the most reassuring thing you will ever see as an engineer. That flat line is proof. Not a promise. Not a design-doc claim. Proof.

And if the line dips? If the success rate drops when one pod dies? Then you've learned something enormously valuable on a calm afternoon: maybe your readiness probes are wrong and traffic is being sent to the new pod before it's ready. Maybe you had no PodDisruptionBudget. Maybe three replicas wasn't actually enough headroom. Whatever it is — you found it here, in the light, with your whole team watching and a coffee in hand, instead of at 3 AM with the CEO in the incident channel.

That's the trade. That's the entire pitch. Move your failures from the worst possible moment to the best possible moment.

I promised you this wasn't theoretical. On September 25, 2014, Amazon announced it needed to reboot a large chunk of its EC2 fleet — on the order of 10% of instances — to patch a security hole in the Xen hypervisor. This wasn't a graceful, take-your-time thing; it was a "this is happening, get ready" thing across a huge number of machines.

For a lot of companies, that was a genuinely bad few days.

Netflix's engineers later described it as, essentially, a non-event for them. When the reboots hit, instances vanished — and their systems did exactly what they'd been drilled to do a thousand times, because Chaos Monkey had been vanishing their instances a thousand times. Losing nodes wasn't a novel emergency. It was Tuesday. The muscle memory was already built. The failover paths were already worn smooth from use.

That's the payoff of years of deliberate, boring, small-scale failure. When the real, big, unplanned failure finally arrived, it looked familiar, and familiar failures don't cause outages. They cause a slightly-elevated graph and an engineer saying "yep, working as expected."

Don't walk away from this thinking "cool, I'll go kill some prod pods." Please do not do that. Walk away with these, in order:

Resilience you haven't tested is just a hope. Every "it should fail over" in a design doc is an untested claim until you've watched it fail over. Hope is not an SRE strategy.

Blast radius discipline is everything. Node before zone before region. Dry-run before live. Opt-in before opt-out. One slice of traffic before all of it. You earn each rung of the ladder; you never jump to the top.

"Healthy" has to be a number. If you can't state your steady state as a metric you're watching in real time, you're not ready to run an experiment — you're just breaking things and vibing. Define the number first.

Always have an automated abort. The maturity of ChAP over plain Chaos Monkey is the kill switch wired to the metrics. In anything beyond a toy, the experiment must be able to stop itself the moment it starts hurting real users. Don't rely on a human noticing in time. Humans are slow and it'll be 3 AM eventually.

A failed experiment is a successful experiment. Finding the weakness is the entire point. The chaos didn't create the fragility; it revealed fragility that was already lying in wait. You want it revealed on your terms.

Chaos engineering isn't recklessness dressed up as bravery. It's the most rigorous kind of humility there is: admitting you don't actually know if your system survives failure, and then having the discipline to find out on purpose — small, measured, watched, and with the safety on — instead of finding out by accident at the worst possible time.

Now go build the toy version, point it at a scratch namespace, and watch that success-rate line stay flat while pods die under it. Once you've felt that flat line, you'll never again trust a "should work" without wanting to test it. That instinct is what I'm actually trying to give you.