My cluster texts me now, and it cost nothing
I've got a handful of little scheduled jobs running on my home Kubernetes cluster. They wake up on a timer, look at some things, and decide whether anything's worth my attention. The missing piece for a long time was the last hop: how does a job running in a basement actually reach me, on my phone, when I'm at the grocery store?
The cloud answer is to sign up for a notification service, get an API key, and pay a few bucks a month. Which is fine, except I'm self-hosting all of this specifically so I'm not renting things I could own. So the rule was: it has to push to my phone, and it can't cost a dollar or depend on an account I don't control.
The whole answer is a small tool called ntfy and an HTTP POST. That's it. Here's the build, because it's genuinely this short.
#The piece in the middle
ntfy is a tiny notification server. You run it, you publish a message to a named "topic" with a plain HTTP request, and anything subscribed to that topic gets a push notification. There's a phone app that subscribes. No account, no key in the cloud, nothing leaves your control. You can run the whole thing on a box you own.
So I run one copy of it on the same home server that hosts my other little web services. A job that wants to tell me something sends a POST to it. My phone is subscribed to the topic. The job runs, posts, my pocket buzzes. The entire architecture fits in that sentence.
Sending a message is about as simple as it gets:
curl -H "Title: Heads up" -d "something needs a look" \
https://ntfy.myhouse/mytopic
The title becomes the notification headline, the body is the body. Tags can add an icon. Priority can make the urgent ones buzz harder. From inside a program it's just an HTTP request with a couple of headers, which every language already knows how to make.
#The scheduling half
The other half is just Kubernetes' built-in timer, a CronJob. You hand it a schedule in the old cron format and a container to run, and it runs that container on that schedule. My watcher jobs are exactly this: a small program in a container, a line that says "every morning at 7," and the program does its thing and, if warranted, fires off that POST at the end.
The nice thing about leaning on the cluster's own scheduler instead of, say, a cron line on one machine, is that I don't care which physical box it runs on. The cluster picks a healthy one. If a machine is down, the job runs somewhere else. The scheduling is somebody else's problem, which is the correct number of problems for it to be.
#The one part that isn't free: don't ship secrets in the clear
Here's the bit I want to flag, because it's the part that's easy to skip and shouldn't be. The messages my jobs send aren't always boring. Some of them name things I'd rather not have readable by anyone who happens to be on my network and guesses the topic name. A notification topic, by default, is wide open: know the name, read the messages.
So I locked it down. The ntfy server is set to deny-by-default, with a token that the jobs use to publish and a login my phone uses to read. It's five minutes of setup and it's the difference between a private channel and a public bulletin board that happens to have a hard-to-guess address. If your notifications were always going to be "the backup finished," you could skip it. Mine aren't, so I didn't.
There was also a small comedy of errors getting the publishing token into the jobs without me accidentally pasting it somewhere it'd get logged forever, but that's its own story. The short version: a credential should never ride in on a command line where it'll sit in your shell history. Pipe it in, don't type it out.
#Why this beats the fancy version
The thing I like most about this setup is how little of it there is. One small service, an HTTP POST, the scheduler that already came with the cluster. No third party in the path between my basement and my pocket. No monthly bill. No account that can get deprecated or rate-limited or sold. If the company that makes ntfy vanished tomorrow, my copy keeps running, because it's my copy.
That's the whole pitch for self-hosting the boring middle pieces. The notification isn't impressive. Nobody's going to be wowed that my phone buzzed. But the buzz travels a path I own end to end, and on the day something actually matters, I'd rather the alert depend on a box in my basement than on whether I'm current on a subscription I forgot I had.