Summary
Modern infrastructure makes every component disposable. When engineering teams codify their systems as repeatable recipes, destroying and rebuilding becomes an automated first response to incidents rather than a costly last resort. That shift reshapes how organizations plan, test, scale, experiment, and design their systems for recovery.
The cloud is really just other people’s computers. When virtualization and containerization define computing resources by code, you can create, destroy, and recreate imaginary computers on demand. Every component restores its own connections and behaves as if each restart is a new deployment. That approach to infrastructure makes physical machines nearly interchangeable, but beyond that, it also changes the way engineers plan and execute projects.
Destroying and rebuilding infrastructure as part of normal operations lets you codify your infrastructure as recipes, rehearse failure before it matters, and experiment knowing that anything you break can recover easily.
Once your computers are truly fungible, you start encoding every part of the infrastructure as a set of repeatable instructions, recipes that rebuild any component the same way every time. It’s the next level of abstraction like human-written code is turned into bytecode or machine instructions. At the next level, Dockerfiles define the machines that run the code, Kubernetes orchestrates how many of those machines run and how they scale, and Terraform provisions the infrastructure itself. Everything is just recipes from top to bottom.
The first time you roll something out, everything is manual. You’re poking a button here, restarting a process there, discovering you forgot to configure something. Once you get used to thinking of it as a recipe you can create and redeploy at will, the real investment is in turning that process into something you can run again. You stop thinking of your first deployment as precious and shift to moving faster.
Once the recipes work, testing is the first thing that gets easier. Teams replicate production at a smaller scale, spin up an environment, play around, and destroy it when they’re done. For example, when trading volume spikes, teams scale horizontally by running the same recipes to spin up more instances, then scale back down by removing instances they no longer need. When a data center goes down, teams rebuild elsewhere from a recipe stored in a separate location.
Netflix pioneered this principle early in its own cloud journey with the Chaos Monkey concept. The premise is that the best way to avoid catastrophic failure is to build for resiliency. The best way to get resiliency is to fail constantly in conditions that you can control. They used it to test resilience in the highly demanding, high-uptime context of streaming video to consumers. The analogy to high-stakes asset management infrastructure is clear. You get good at whatever you practice often. If you practice destruction routinely, resilience becomes a property of the system. If you only rebuild when something has rarely gone wrong, then you’ll be rusty when it’s time to respond.
Chaos Monkey…randomly disables our production instances to make sure we can survive this common type of failure without any customer impact. The name comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables — all the while we continue serving our customers without interruption. By running Chaos Monkey in the middle of a business day, in a carefully monitored environment with engineers standing by to address any problems, we can still learn the lessons about the weaknesses of our system, and build automatic recovery mechanisms to deal with them. – Netflix TechBlogi
The reusability of recipes also plays out at the team level. High-performing organizations come to an agreement about the way recipes should work, codify it in a system that checks and corrects recipes to fit an agreed standard, and stop talking about it.
One example is endless developer debate over style. Commas in lists. Tabs versus spaces. Teams can get genuinely angry over whether to use two spaces or four for indentation. This is an area where creativity isn’t called for. The evolved answer is that before you commit any code, a formatter rewrites it in the team’s common format. If the output is consistent no matter what someone typed, the argument ends.
The same principle is more consequential with infrastructure. You might be able to tolerate the mild friction of inconsistency across four engineers. But with a team of 50, if every person who touches the infrastructure carries multiple styles in the context window of their own brain, that cognitive load costs real time.
If you’re using AI to assist with infrastructure work, inconsistency is worse. The model looks at a couple of samples, decides that’s the pattern, and ignores your secret knowledge hidden in some other file.
The way out is to set a policy and enforce it as a kind of recipe for recipes. When you optimize for consistency, humans and their tools have the best chance of getting things right. People can expend creativity and personality in solving more meaningful business problems.
Modern infrastructure changes your relationship with risk as well. When a recipe defines every component, it creates a greenfield context where experimentation has more room to grow. If you try something and it breaks, you can rebuild from the instructions. But a legacy system with dozens of dependencies and no repeatable build process makes experimentation too risky. If people are running business-critical work on such a system, you can’t take it down and rebuild it by rerunning your recipe. Failure is just too expensive.
Architecture plays a huge part in whether you can shrink the blast radius of any experiment to the cost of running its recipe again. Healthy teams do something new every week because they can afford to throw the results away. The trick is making sure those risk-averse legacy assumptions don’t linger in your culture.
In the bare-metal era, when teams ran software on physical computers they could touch, the instinct was to investigate and troubleshoot when something went wrong. You would do whatever you could to preserve the build because the resources were costly and the effort was high. That sequence puts the human at the front of the triage line.
The destroy-and-rebuild mentality reverses the order. When creating a new instance is cheap, and the replacement won’t interrupt upstream or downstream systems, the first solution to fixing problems is to fix and rerun the build recipe. You take the unhealthy instance down and spin up a new one to see whether the problem persists. It can be as simple as turning it off and on again.
Automated health monitoring can run the same logic without waiting for a human to intervene. When an instance starts dropping connections or slowing down, the system takes it out of the pool and spins up a fresh one from the recipe. The other instances keep serving traffic, so nothing goes fully offline. If the new instance behaves normally, the problem was specific to the old one — maybe a memory leak or a misconfiguration — and you avoided a debugging session entirely. If the new instance has the same problem, it survives a clean rebuild, which means it’s in the code or the environment, and now a human investigates with that information in hand. Automated replacement still needs humans, too, to catch loops where the system keeps cycling instances without resolving the underlying cause.
Destroy-and-rebuild as an operational default raises the bar for application code, too. If your infrastructure automatically replaces instances, your code has to survive the disruption. Junior developers might assume a database will always be there when their code runs. Senior engineers write retry logic that assumes any resource might be temporarily unavailable. Experienced teams let everything come up in any order, retry on failure, and heal automatically once the dependency appears.
That’s what makes destroy-and-rebuild a first principle rather than a last resort. It’s the starting assumption that shapes how you design every layer of the stack and write code. The recipes and the tooling already exist. The question is whether your infrastructure treats rebuilding as the cheapest move available or still treats it as the option you reach for after everything else fails.
If you’re running infrastructure that treats rebuilding as a last resort, start by asking how long it takes to recreate any given component from scratch. If the answer is days or involves tribal knowledge that lives in one person’s head, that’s your gap. The second question is whether your applications assume stability or volatility. Code that expects every resource to be available at all times is code that can’t survive a destroy-and-rebuild environment. The third is when you last broke something on purpose.
If your organization depends on something — if something is critical for your team — plan on it disappearing. Make it resilient. Call it succession planning, call it resiliency, call it business continuity planning. Make rebuilding normal, not a crisis. It costs something; nothing comes for free. But you pay an overhead to know that when things go wrong, you’re ready. If rebuilding is normal because destruction is normal, the bar for what becomes a bad day gets much higher and the cost to try new things gets much lower.
If you’ve never broken something on purpose, your team’s first real failure will also be its first rehearsal.
Matt Katz
As Arcesium's Field CTO, Matt leads Arcesium's Forward Deployed Software Engineering and Client Success teams. His work to empower clients and simplify technical challenges stems from a 25-year career in financial technology working with clients and software. Outside of work, he enjoys books, bikes, and boards.
Sources:
[i] Netflix TechBlog, 2011. https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116
No spam. Just the latest releases and tips, interesting articles, and exclusive interviews in your inbox every week.
