Don't put the keys inside the house you're locking
I decided I wanted a real secrets manager for my homelab. Somewhere proper to keep the passwords and API keys instead of having them scattered through config files like loose change in a couch. HashiCorp Vault is the obvious tool, and running it on my own Kubernetes cluster, on hardware I already own, was the obvious place to put it.
It was about thirty seconds into being pleased with myself that the problem showed up.
A secrets manager exists so that everything else can get its passwords from it. So now everything on the cluster depends on the vault being up and reachable. Fine, that's the point. But the vault is also running on the cluster. So the cluster depends on the vault, and the vault depends on the cluster, and the day you most need to get a secret out, in the middle of fixing a broken cluster, is exactly the day the vault might be down too, because it's part of the thing that's broken.
That's a circular dependency. They're nasty because they're invisible on a good day. Everything's up, everything talks to everything, looks great. The circle only closes when something fails, and by then you're standing in it.
#The unseal problem makes it real
Here's the concrete version. Vault, when it starts, comes up sealed. Its data is encrypted and it can't read it until you give it the unseal key. This is a good security property: if someone steals the disk, they've got a locked box. But it means every time the vault process restarts, somebody has to come along and unseal it before it'll hand out a single secret.
In a homelab, "the vault process restarts" happens constantly. A node reboots. Kubernetes reschedules the pod to a different machine. You apply an update. Every one of those reseals the vault. And if the vault is sealed when the rest of the cluster is trying to come back to life after an outage, nothing can get its secrets, and you've got a chicken-and-egg standoff while you're already having a bad day.
The fix for the unseal problem is auto-unseal: you let the vault unseal itself on startup using a key held somewhere outside itself, usually a cloud key service. Which is the right answer, and it's also the tell. The clean version of running your secrets manager inside your cluster requires anchoring its trust outside your cluster. The circle wants to be broken.
#So why not just keep it outside?
That was the honest question I had to sit with. There's a separate Vault already running on a different box in my house, doing a different job. The boring, correct move would be to use a thing like that as the secrets root: a separate machine, separate failure, not tangled up in the cluster it serves.
I went with the in-cluster one anyway, on purpose, but only after I made myself say out loud what it costs and what makes it safe. It's safe if, and only if, three things are true. It auto-unseals without a human. It runs more than one copy so a single dead machine doesn't take it down. And, the one people forget, the things that depend on it have to keep working for a bit even when it blinks. If a brief vault outage instantly breaks every running app, you've built a fragile cluster with extra steps.
That last one is the quiet hero. The way the secrets actually reach my apps, an operator pulls each one out of the vault and writes it into the cluster's own store, where the apps read it. So a running app already has its password in hand. If the vault goes down, no new secrets sync, but nothing that's already running falls over. The dependency on the vault is real but it's not a live wire under every pod. That's the difference between an outage you notice next time you deploy and an outage that pages you at 2am.
#The line I wrote down
I keep coming back to one rule, and it's the whole post in a sentence: a secret the cluster needs in order to recover cannot live only inside that cluster. The key to the house can't be locked in the house. Everything else is plumbing, and the plumbing is genuinely nice once it's in. But that one line is the thing that turns "I run Vault on Kubernetes" from a resume bullet into something that won't strand you on the worst day.
The trap with self-hosting everything is that it feels like control, and most of the time it is. The circular dependency is the case where it quietly becomes the opposite, and the only way to catch it is to stop admiring the diagram and ask what breaks when the diagram is on fire.