← all writing

The DNS quirk that broke seven of my Kubernetes pods

2026-07-24 · ~~9 min read · kubernetes · homelab · networking

Apply a cert-manager pod's spec to your cluster, then watch it try to verify a Let's Encrypt domain by talking to an IP that has nothing to do with Let's Encrypt. Then watch ArgoCD do the same thing trying to clone a GitHub repo. Then watch a Windows VM in a pod fail to talk to the internet on first boot. By the fourth pod with this exact shape of failure, you stop trusting yourself.

I hit it seven times across three weeks. Same bug, different surface every time. The first six fixes were point fixes. The seventh time I sat down and read all four lines of the part of resolv.conf I'd been ignoring, and the whole thing collapsed into one stupid little number.

This is that number.

#What it looked like the first time

cert-manager Certificate stuck in pending. Its DNS01 challenge can't reach Route 53. Pod logs show a timeout to the AWS API. I exec into the pod, run nslookup route53.amazonaws.com, and the resolver hands me back 10.66.0.20.

10.66.0.20 is my reverse proxy. It is not AWS.

The TLS handshake to "Route 53" then fails because my reverse proxy is presenting a *.dillinghamhouse.com certificate, not an amazonaws.com one. cert-manager logs an obscure handshake error. I spend half an hour reading cert-manager source code before I think to check what route53.amazonaws.com was even resolving to in the first place.

The cause, once I found it, took me about ten minutes to fix and three weeks to actually understand.

#The four-line trap

Every pod in Kubernetes gets an /etc/resolv.conf written for it by the kubelet. By default it has roughly this shape:

nameserver 10.43.0.10
search default.svc.cluster.local svc.cluster.local cluster.local dillinghamhouse.com
options ndots:5

Three fields matter. The nameserver, which is CoreDNS inside the cluster. The search list, which is a set of suffixes the resolver will try to append. And ndots, which is just a number that controls when the appending happens.

The ndots:5 value is where the bug lives.

What ndots actually does: when you try to resolve a hostname, the resolver counts the dots in it. If the count is less than the ndots value, the resolver tries every search-list suffix on the hostname first, before trying the bare hostname. Only after all the search suffixes fail does it fall through to the actual name you asked for.

So when cert-manager asks for route53.amazonaws.com (two dots), the resolver counts two, sees that two is less than five, and starts marching down the search list:

  1. route53.amazonaws.com.default.svc.cluster.local (fails, no such record)
  2. route53.amazonaws.com.svc.cluster.local (fails)
  3. route53.amazonaws.com.cluster.local (fails)
  4. route53.amazonaws.com.dillinghamhouse.com (succeeds)

That last one succeeds because my Route 53 zone for dillinghamhouse.com has a wildcard A record pointing at 10.66.0.20. Anything-dot-dillinghamhouse-dot-com resolves to my reverse proxy. Including, evidently, route53.amazonaws.com.dillinghamhouse.com.

The pod has now resolved Amazon's hostname to my reverse proxy. It tries to start a TLS connection. My reverse proxy answers with a different certificate. Everything downstream looks broken in a way that has nothing visible to do with DNS.

#Why every pod inherited the bug

The pod didn't ask for dillinghamhouse.com to be in its search list. The kubelet put it there because the host node had it in its /etc/resolv.conf. The node had it because my home network's DHCP server (AdGuard, in this case) hands out that search domain to everything on the LAN. The cluster nodes are on the LAN. The kubelet inherits the host's DNS, then layers cluster-specific suffixes on top, and the pod inherits the union.

This is a sensible default for almost every cluster on earth. It only breaks if your environment's DNS search domain also has a wildcard A record pointing at something you don't intend to hit. Most production clusters don't have that combination. Mine did.

#The trail of breadcrumbs

Same root cause, in order I hit them:

  1. cert-manager trying to reach Route 53 for DNS01
  2. argocd-repo-server trying to clone from github.com
  3. external-secrets-operator's bitwarden-sdk-server trying to reach api.bitwarden.com
  4. ARC's runner-listener controller trying to talk to github.com
  5. A Windows 11 VM in a pod, failing first-boot setup
  6. A throwaway Ubuntu sandbox trying to apt-get update against deb.debian.org
  7. The external-secrets controller itself, which I'd already half-broken trying to fix #3

Seven pods. Same bug. Looked completely different every time, because the failure was never "DNS returned the wrong IP," it was whatever the downstream code did once it got that wrong IP. TLS errors, hung connections, HTTP 500s, weird auth failures, pods restarting on a loop, install scripts giving up halfway through.

The reason it took me so long to notice the pattern: I diagnosed each one as if it were the first one. I read the cert-manager logs and thought cert-manager bug. I read the ArgoCD logs and thought GitHub SSH issue. I read the Windows VM logs and thought the VM image is broken. I didn't think DNS until I started seeing the same pod-shape across totally unrelated codebases. That's the part I'd do differently if I could. Five out of six diagnostic sessions could have ended in five minutes if the first move had been nslookup <whatever the pod is trying to reach> from inside the pod.

#The fix

Tell the pod to ignore the host's DNS and use one you trust:

spec:
  template:
    spec:
      dnsPolicy: None
      dnsConfig:
        nameservers:
          - 10.43.0.10
        searches:
          - default.svc.cluster.local
          - svc.cluster.local
          - cluster.local
        options:
          - name: ndots
            value: "2"

Two things change versus the default. First, the search list no longer contains dillinghamhouse.com, so the wildcard A record can't get appended to anything. Second, ndots drops from 5 to 2, so any hostname with two or more dots in it gets tried as-is first, before any search-suffix nonsense.

Either fix would work alone. I do both because by the seventh time you've been bitten by something, you stop being precious about which exact knob is the one that solved it.

#The longer-term shape

You can ship the same fix cluster-wide with a PodDefault admission policy, which auto-injects the dnsConfig into every new pod. I haven't done that yet. At my scale it's been enough to put the snippet in a comment at the top of every workload's Deployment manifest, so future-me sees it before the pod ever starts.

I also stopped handing out the dillinghamhouse.com search domain via DHCP. That was the actual root cause and the one I should have addressed first. I kept putting it off because I had clients on the LAN that liked the short-name convenience, and once I sat with it I couldn't think of one I'd actually miss. So the wildcard A record still exists, the search domain doesn't, and pods no longer inherit anything from the host that could collide with public hostnames.

If I had to pick one takeaway out of this for someone running a homelab cluster: when you're picking a domain name for your homelab and setting up wildcard DNS, you're also signing up for everything that domain interacts with. The wildcard is convenient. It is also a footgun pointed at every container that ever runs on your LAN. Pick one or the other.

I picked both, found the footgun, and now have a small piece of YAML I paste into every new workload. That's a fair trade for the convenience the wildcard buys me back. But I'd think twice before doing it again from scratch, knowing what I now know about how many pods would inherit the consequences.

← all writing