Three things I got wrong about Kubernetes
I held off on Kubernetes for years. The case against it sounded right: overkill for small scale, more overhead than it's worth, "wait until you need it." Then I finally stood up a two-node k3s cluster, and most of what I'd believed turned out to be wrong.
Three things I got wrong.
#"k8s is overkill for two nodes"
Two nodes is exactly when it starts earning its keep.
The case I'd been making to myself was that Docker Compose on a single host is simpler than k8s on a single host. That's true. What I missed is that the meaningful unit of comparison isn't k8s vs Compose at one host. It's k8s vs whatever-I-was-actually-doing once the second host showed up.
What I was actually doing with two hosts: SSHing into each one, keeping their docker-compose.yml files in sync by hand, hoping I'd remembered to update the .env on both, and resolving the inevitable "wait, which host is running this?" question by running docker ps on both. The scheduler does that bookkeeping work without me thinking about it.
The breakeven point isn't "production scale." It's "second machine." Once the second node exists, you stop caring which one the workload lands on. The cluster makes the cost of "I forgot which host" zero. That's the real story.
#"GitOps is just kubectl apply with extra steps"
It isn't.
kubectl apply is imperative. You stand at the terminal, you change state. Whether the state matches what's in git is whatever you remember to make it. In practice it diverges. Someone runs a quick fix at 2am, the fix doesn't make it into git, and now the git repo and the cluster are two different sources of truth that disagree.
GitOps inverts the loop. A controller watches a git repo and reconciles the cluster against it on its own cadence. The cluster has no other way to change state. git push becomes the only path that mutates infrastructure.
What this buys you in practice:
- Drift becomes impossible. Any manual change gets reverted by the next reconcile loop. Force-quitting a job, hand-editing a deployment, running
kubectl edit. All of it gets undone unless it's also in git. - Half-applied changes can't survive a restart. If a deployment fails mid-apply, the controller picks up where it left off, not where the human stopped typing.
- The review process and the production change are the same thing. Code review on a PR is also the change-management review. No separate "do you have approval to apply this?" step.
The mistake I was making was reading "GitOps" as "the workflow of pushing manifests to git, then running kubectl apply." That's not GitOps. That's still imperative apply with a git intermediary. The thing that makes it GitOps is the controller's reconciliation loop.
#"Operators are pattern over substance"
CloudNativePG sets up Postgres with streaming replication, automated failover, and point-in-time recovery primitives, in about 30 lines of YAML. That's not pattern over substance. That's "someone wrote the cluster-admin runbook into a controller."
The pattern critique of operators says: you're paying the abstraction tax of "you have to learn the operator's API on top of the underlying tool's API." Which is true. The implicit claim, though, is that the underlying tool's API is simpler.
That claim breaks the moment the underlying tool has a non-trivial operational surface. Postgres in HA mode isn't simple to operate. There's a long checklist: configure streaming replication, set up the standby's recovery, deal with the primary's WAL retention, configure automated failover (which probably means running Patroni or repmgr, each with their own learning curves), wire up PITR backups, monitor the replication lag.
A well-written operator collapses all of that into a Kubernetes-shaped declaration. The 30-line YAML doesn't replace knowing how Postgres replication works. You still need to know that. It replaces the operational ceremony of setting it up by hand.
That's the inversion. Operators aren't pattern over substance. They're pattern as substance, for systems where the substance is the operational checklist, not the algorithmic logic underneath.
#The reframe
I was wrong about all three. Not because the case against k8s is wrong, exactly. It was aimed at the wrong question.
I kept asking "is the complexity worth it?" The question that actually mattered was "is it less complexity than what I was already eating without it?" Flip it that way and it stops being a close call.
What I was already eating without k8s:
- Two-host docker-compose synchronization
- "Which
.envon which host" cognitive overhead - Imperative operations against bare hosts (systemctl, docker compose, etc.) with no reconciliation
- Manual Postgres HA setup if I ever wanted that
- Hand-rolled secret distribution
- A bespoke "where does this service run?" mental model
None of those are free. Each one is a small operational tax I was paying every time I touched the homelab. The complexity comparison was never k8s vs nothing. It was k8s-complexity vs cumulative-small-complexities-I'd-stopped-noticing.
The k8s complexity is concentrated and visible. You stand it up, you learn the abstractions, and the cost lives in one place. The pre-k8s complexity is diffuse and invisible. Paid in small amounts on every operation, each one too small to write down, all of them adding up to something larger than anyone admits.
When I finally added up what I'd been doing without it, the answer flipped.
What's the abstraction you held off on the longest before realizing it was already cheaper than what you were doing?