Making my one-machine cluster survive losing a machine
My home Kubernetes cluster has a couple of machines in it, so for a long time I told myself it was redundant. It wasn't, and the gap between "has more than one machine" and "survives losing a machine" is where this whole post lives.
The setup was: one machine was the brain, the part that makes all the decisions about what runs where. The other was a worker that just does what the brain says. If the worker died, fine, the brain reschedules its jobs onto itself. But if the brain died, the entire cluster died with it, because there was only one brain and it also happened to be holding the only copy of the cluster's own memory on its local disk. More than one machine, sure. One single point of failure, also sure.
I wanted any one machine to be able to die without taking the whole thing down. That turns out to require three changes, and the thing nobody tells you up front is that they only work as a set.
#The surprise underneath: the wrong database
The first thing I checked was what the cluster uses to store its own state, the running record of everything that exists. The lightweight Kubernetes distribution I run defaults, on a single server, to a simple file-based database. SQLite, basically. It's a fine default. It's also fundamentally a one-writer thing. You cannot have three machines all being the brain if they're each scribbling in their own separate notebook with no way to agree.
To make the brain survivable, the state has to live in something built for a group: a distributed store where multiple machines hold copies and vote on the truth. So before I could add brains, I had to convert the underlying database from the single-writer file into the clustered kind. That's a one-time, slightly scary conversion, the sort of step where you take a backup first and mean it, because if it goes sideways the thing you're converting is the cluster's entire memory.
The lesson there, which I didn't expect: "add more control-plane nodes" is not step one. It's step three. You can't just bolt brains onto a single-writer database. The foundation has to change first, and if you skip straight to adding machines you'll get a mess that looks like it's working until the first failure.
#Three legs, and you need all three
Once the database can handle a group, the actual survivability needs three things, and missing any one of them means it doesn't work:
A group that can lose a member. The distributed store agrees on truth by majority vote. Two voters is worse than one, because with two, losing either one means you no longer have a majority and everything freezes. You need an odd number, and three is the smallest that survives a loss: two out of three is still a majority, the cluster keeps deciding. So "survive a node failure" has a hard floor of three brains, not two.
A door that doesn't depend on one machine. Everything talks to the cluster through a single address, and that address was just the brain machine's address. So even if the other brains survive, every client is still knocking on a dead door. The fix is a floating address, one that isn't tied to a physical machine and hops to a survivor automatically. The clients knock on the floating address, the address is always answered by whoever's alive.
Storage that isn't trapped on one disk. This is the one I'd half-forgotten and it's the most likely to ruin your day. If an app's data is sitting on the local disk of one machine, then that app cannot move, ever, no matter how healthy the rest of the cluster is. Making the brain survivable does nothing for the database that's physically glued to one box. The data either has to live on shared storage every machine can reach, or, for the data that really matters, be actively copied across machines so a dead disk isn't a dead database.
The reason these only work as a set: a floating door is useless if the brains behind it can't form a majority. A majority of brains is useless if the data they're protecting is stranded on one dead disk. Survivability isn't a feature you add, it's a property that emerges only when all three are true at once, and that's exactly why it's easy to do two of them, feel finished, and still have a cluster that dies on the first real failure.
#The part I respect now that I didn't before
When I started, "high availability" sounded like a checkbox. Turn it on. It is not a checkbox. It's a small system of interlocking requirements where the interesting failures all live in the gaps between the pieces, and where the honest test is not "does it run" but "does it survive me walking over and pulling the power cord out of any one machine."
I haven't pulled the cord yet, that's the next chapter. But the planning alone changed how I see every "redundant" setup I've ever nodded at. Redundancy you haven't tested by actually killing something is just hope with a nicer diagram. The whole point is the day a machine dies, and you don't get to find out whether you built it right until that day comes, so the least you can do is be the one who chooses the day, on purpose, with a coffee, instead of finding out at 3am.