Teaching a model to check itself against the truth
I have a habit that started as paranoia and turned out to be useful. I keep a couple of plain text files: one is things that are true about me and my projects, with the numbers and dates pinned down, and the other is things that are not true, the claims I've caught an AI inventing about me before, written down so I never have to catch them twice.
The problem is that having the files isn't the same as using them. When an AI helps me draft something, it doesn't know those files exist. It writes a confident sentence, the sentence is plausible, and plausible is exactly the failure mode, because a plausible wrong claim is the one that slips through. I wanted a way to take a chunk of draft text and ask, against my own ground truth: does this say anything I've explicitly marked as false, and what do my real facts actually support here?
This is the kind of thing people reach for a big cloud service to do. I wanted it small, on my own hardware, in my house. Here's how it goes together and the one bug that's actually interesting.
#The boring, correct way to do it
The mechanism is embeddings. You take a piece of text and turn it into a long list of numbers that captures, roughly, its meaning, in a way that "I shipped a thing" and "I delivered a thing" land near each other even though the words differ. Two pieces of text that mean similar things end up with similar number-lists. So if I turn every line of my truth files into one of these, and then turn a draft sentence into one, I can find the truth-lines that are closest in meaning to what the draft just claimed.
That's the whole trick. To check a sentence, I find the nearest "do not claim" lines and the nearest real facts. If the draft is sitting right on top of a thing I've marked as false, that's a flag. If it's near a real fact, that's the supporting evidence I can lean on to phrase it honestly.
The pieces are all things I can run myself. A small model on the basement box with the GPU does the meaning-into-numbers part. A database that's good at "find me the nearest number-lists" stores them. A little Go service ties it together and exposes one function: hand it text, get back the nearest forbidden rules and the nearest facts. None of my drafts, none of my truth files, ever leave the house.
The important thing to be clear about: this is a personal sanity check for my own writing, not some grand system. It's a spell-checker for honesty. It doesn't decide anything. It surfaces the nearest relevant truth and lets me, a human, look.
#The bug that was actually interesting
Here's the part worth the post. I'd already had a separate function in the same little service that does a normal search over a different pile of my data. When I added the truth files into the same database, I introduced a quiet hole: that older, general search now had the truth-file lines sitting in the same haystack it searched. Which meant a perfectly innocent general query could come back with one of my private "do not claim" rules as a result, surfacing in a context where it had no business showing up.
Nothing crashed. That's what makes it the dangerous kind. The feature worked, it just also started occasionally returning the wrong category of thing, because I'd dumped two different kinds of data into one bucket and only updated one of the two readers to know the difference. The fix was simple, scope the general search so it only ever looks at the category it's supposed to. But the lesson is the one that keeps mattering: the moment you put a new kind of data into a shared store, every existing thing that reads from that store is now a thing you have to re-check. The new writer is obvious. The old readers are the ones that bite you, because nobody changed them and so nobody's looking at them.
#Why semantic, and where it's honestly weak
The reason this uses meaning-similarity instead of just searching for exact words is that drift doesn't use exact words. If I've got a rule that says "don't claim X," the dangerous draft isn't the one that says X verbatim, it's the one that says the same thing three synonyms over. Exact-match would sail right past it. Meaning-match catches the paraphrase, which is the whole point.
It's also where it's weak, and I'll say so plainly. Things being near each other in meaning is not the same as one contradicting the other. A sentence about a project I'm proud of sits near a rule about that same project, because they share vocabulary, even when the sentence breaks no rule at all. So the tool over-flags. It hands me a pile of "here's a rule that's nearby, you decide." That's fine, because the deciding was always going to be mine. A tool that surfaces the nearby truth and trusts me to read it is doing its job. A tool that pretended to judge would just be a new, confident liar, and I already have plenty of those to keep honest.