It worked on my laptop but not in the pod
I had a small Go program that fetches some public JSON from a handful of websites. On my laptop it worked perfectly. I built it into a container, ran it as a job on my home Kubernetes cluster, and about half the sites started failing with this:
remote error: tls: unrecognized name
Same code. Same URLs. Worked from my laptop, failed from the pod. That gap is one of the most annoying shapes a bug can take, because all your instincts ("it's a code bug") are wrong from the first second. The code is fine. The code is identical. Something about the environment the code runs in is different, and you have to go find it.
This is the walk-through of finding it, because the answer turned out to be a little stack of perfectly reasonable defaults that gang up on you.
#First, read the error for what it actually says
tls: unrecognized name is a specific thing. It's not a certificate problem, not a "couldn't connect" problem. It means the TLS handshake reached a server, and that server looked at the hostname the client said it wanted, and replied "I don't serve that name here." The name in question is the SNI: when your client opens an HTTPS connection, it announces, in the clear, which hostname it's trying to reach, so a server hosting many sites knows which certificate to present.
So the client is connecting to something, and telling it a hostname, and that something is saying "wrong house." Which means either the client is connecting to the wrong IP, or it's announcing the wrong name. Both of those point at one suspect: DNS.
#Prove the network is fine, then suspect the resolver
Before going down the DNS hole I made sure the boxes could even reach the sites. I logged into the cluster nodes directly and curled the failing URL. It worked. HTTP 200, clean. So the machines can reach the site fine. It's only the pods that can't. That narrows it hard: the difference between a node and a pod, for outbound traffic, is mostly how they resolve names.
Here's the stack of reasonable defaults, each fine alone.
One: Kubernetes gives pods their own DNS setup, and it adds a list of internal search domains so that short names like my-service resolve to things inside the cluster. To make that work it sets a knob called ndots to 5. That means "if the name you're looking up has fewer than 5 dots in it, try gluing the search domains on first before you try the name as-is." Reasonable, for cluster-internal names.
Two: the website I was hitting didn't resolve straight to an address. It was a chain. The name I asked for pointed at another name, which pointed at a content-delivery network, which finally pointed at an address. A couple of hops. Reasonable, that's how big sites work.
Three: my container was built from scratch, a stripped-down image with nothing in it but my program. No system libraries. Which means Go used its own built-in name resolver instead of the operating system's. Reasonable, it's smaller and it usually behaves.
Put the three together and here's what happens. The pod's resolver, told to try search domains first because of ndots: 5, goes hunting for my website's name with a bunch of internal cluster suffixes glued on. Those all fail, as they should. But somewhere in that fumbling through a multi-hop chain of name-points-at-name, the lightweight Go resolver ends up resolving to an address and announcing a hostname that don't agree, and the CDN at the other end says "unrecognized name" and hangs up. My laptop never did any of this because it has none of those three defaults. It just looked the name up the normal way and connected.
#The fix is one line
dnsConfig:
options:
- { name: ndots, value: "1" }
That tells the pod to stop trying the cluster search domains first for any name that has a dot in it. Look external names up directly, the way my laptop does, instead of fumbling them through the internal suffixes. Every one of the failing sites started working immediately.
#What I actually took from it
The fix is one line, but the lesson isn't the line. It's that "works here, fails there" is almost never a single bug. It's an interaction. Each piece in that stack, the ndots default, the CNAME chain, the scratch image's resolver, is a sensible choice that somebody made for a good reason, and not one of them is wrong by itself. The failure lives in the seam between them, which is exactly the place no single person was looking, because each piece belongs to a different layer that's "obviously fine."
The way you catch those isn't by being smarter, it's by being stubborn about cutting the problem in half. Node works, pod doesn't: it's the pod environment, not the code. External names fail, internal would've worked: it's the resolver, not the network. Each split throws away half the haystack. You don't have to understand the whole stack at once. You just have to keep asking "which side of this line is the problem on" until there's only one thing left standing in the corner, looking guilty.