I built alerts that don't cry wolf
I run a few little background jobs on my home cluster that watch for things I'd otherwise forget to check. Stuff that goes quiet too long. New things worth knowing about. The idea is simple: the computer keeps an eye on it so I don't have to, and it taps me on the shoulder when something's worth a look.
The first version tapped me on the shoulder 180 times in about two seconds.
That's the trap with this kind of thing, and almost everybody walks into it. You write a job that wakes up every so often, looks at the world, and sends a notification for anything it finds. It works great the first time you test it with one item. Then you point it at real data, it finds a hundred and eighty things that match, and your phone sounds like a slot machine paying out. You learn to ignore it inside a day, which means you've built an alerting system whose only function is to be muted.
The job isn't "find the problems." Finding them is easy. The job is "tell me about each problem once."
#The dumb version vs the right version
A naive watcher is stateless. It wakes up, queries, and fires. It has no memory of the last time it ran, so a thing that's been broken for a week gets reported every single run for a week. Multiply that by every item that matches and you get the slot machine.
The fix is to give the watcher a memory. I keep a little table I call the signals ledger. Every thing the watcher cares about gets one row, keyed by something stable, and that row carries a status: open, notified, acked, resolved. The watcher's job changes from "send notifications" to "reconcile what I see right now against what's already in the ledger."
That one shift fixes most of the pain. A condition that's already in the ledger as "notified" doesn't get sent again. A condition that's gone away gets marked resolved. Only a genuinely new row, one that wasn't there last time, earns a buzz. The standing backlog goes quiet because the ledger remembers it's already had its say.
#The two bugs hiding in "just send the new ones"
Sounds easy. It isn't quite, because there are two ways it bites you, and I hit both.
The first is the cold start. The very first time the watcher runs, everything is new, because the ledger is empty. So your anti-spam system spams you with the entire backlog on day one and you're right back to 180 buzzes. The fix is to treat the first run as a seed: write all those rows into the ledger, but mark them as already-handled instead of new. Quietly absorb the backlog. From then on, only the genuine new arrivals get through.
The second is more subtle and it's the kind of bug that passes every test and then betrays you in week two. When a condition resolves and later comes back, you want it to fire again. But if your reconcile logic just bumps the timestamp on the old row without re-opening it, the returning problem stays marked "resolved" and you never hear about it. Silent miss. The reverse of crying wolf, and worse, because now you've trained yourself to trust a system that's quietly dropping things on the floor.
So the rule that makes it actually work: a row that's marked resolved gets re-opened the moment its condition reappears. Reappearance is a real event, not a no-op.
#The thing I almost got wrong
I had it re-nudging anything still unhandled on a weekly cadence. Felt responsible. "If you ignore it, I'll remind you in a week." Then I looked at what was actually in the backlog and most of it was stuff that was never going to resolve on its own and that I'd already seen. A weekly reminder of twenty things I've already decided to let sit isn't a nudge, it's a nag, and a nag is just crying wolf on a slower clock.
So I cut it back to notify-once. It fires when a thing first shows up, then it's quiet. If I want a thing to keep bugging me, I can say so. If I want it to go away, I ack it and it's gone. The default is to trust that I saw it the first time. Turns out the discipline isn't reminding people more. It's reminding them less and meaning it.
The whole thing is maybe two hundred lines of Go and one little table. The interesting part was never the notifications. It was deciding, carefully, what was worth a buzz and what wasn't, and then building something that holds that line even when I'm not looking at it. A computer that respects your attention is worth ten that demand it.