I found a data-loss bug in my own plan before I ran it
My home Kubernetes cluster started as one machine. That's a single point of failure: if that box dies, everything dies with it. I wanted to fix that, the way you're supposed to, by adding more machines so the cluster survives losing any one of them. It's real work with a lot of moving parts, so before touching anything I sat down and wrote a plan. Phases, ordering, the commands, the gotchas.
I was pretty proud of it. It was thorough. Then I made myself do the one thing that's actually worth doing with a plan, which is not admire it. Go back through it line by line and check each step against what the system is really like right now, not what I assumed it was like when I wrote the step. And on about the third pass I found a step that would have quietly destroyed a stack of data.
#The step that looked harmless
One of the machines was set up in a junior role. To promote it to a full member of the cluster, my plan said, roughly, "reinstall the cluster software on it." Clean, standard, the documented way to do it. I'd written it as a disruption, the kind where some apps shuffle around for a minute and then settle back down. No big deal.
Except. That reinstall script, as part of cleaning up, wipes a particular directory on the machine. And that exact directory is where the cluster stores the disks for any app that's using local storage on that node. I went and looked at what was actually sitting in local storage on that machine, and the list was not short. A metrics database. A virtual machine's disk, sixty-odd gigabytes. A couple of smaller databases. And, the one that made my stomach drop, the brand-new secrets vault I'd stood up that same morning, holding every password in the place.
So the real shape of that "harmless reinstall" step was: permanently delete all of that, with no undo. My plan had a line in it that read like routine maintenance and behaved like a controlled demolition. If I'd just executed the plan top to bottom the way I wrote it, trusting Past Me, I'd have found out the hard way, after the data was gone.
#Why the plan was wrong and the cluster wasn't
The thing I want to be honest about: the cluster wasn't broken. The commands weren't wrong. Each one does exactly what it says. The bug lived entirely in my model of the situation. When I wrote the step, I was thinking about that machine as "a compute node that runs some apps," and for a compute node, wiping it and rebuilding is fine, the apps just move. I had stopped thinking about it as "a machine that is also physically holding the only copy of some data on its local disk." Both things were true. I only had one of them loaded in my head when I wrote the step.
That's the whole genre of mistake, by the way. Not "I don't know how the tool works." It's "I know how the tool works, but my picture of the current state drifted from reality, and I built a step on the stale picture." The commands are downstream of the picture. A correct command on a wrong picture is still a wrong move.
#What the pressure-test actually caught
So I rewrote the plan, and the fix wasn't a different command. It was a reordering and a hard rule. Before that machine gets touched, every bit of data living on its local disk has to be moved off it first, onto storage that isn't tied to one machine. That migration got promoted from "something to do eventually" to "a gate you are not allowed to walk past." And the vault got its own loud warning, in red, with a snapshot step in front of it, because the vault is the one piece where losing it doesn't just cost you data, it costs you the keys to get the rest of your data.
None of that is clever. The clever part, if there is one, was the decision to pressure-test the plan at all instead of trusting it. I write a thing, it reads well, and there's a strong pull to just go execute it, because re-reading your own work feels like procrastinating. It isn't. Re-reading a plan against the real system is where the data-loss bugs live, and they only show up if you go looking for the gap between what you assumed and what's actually there.
#The part worth keeping
The most valuable hour of that whole project was the hour I spent trying to break my own plan before it could break me. No data moved. No node changed. Nothing "got done" in the visible sense. But I found a step that would have deleted a vault full of secrets and a 64-gig VM, and I found it on a quiet afternoon at a keyboard instead of at 1am with everything on fire.
A plan you haven't tried to falsify isn't a plan, it's a wish with bullet points. The work isn't writing it. The work is sitting back down with it, cold, and asking every step "okay, but what is actually true right now, and what does this really do to it." Past Me wrote a confident, dangerous line. Present Me caught it. That handoff only works if Present Me refuses to trust Past Me's confidence.