← all writing

I fine-tuned an AI on myself and it became a vaguely-me-shaped liar

A couple weeks back I built a tiny language model from scratch, just to stop treating these things as magic. Once I understood how one actually worked, the obvious next thought showed up: could I make one write like me?

So I tried. What I got back was a confident little liar that sort of sounds like me and knows nothing. Best failure I've had in a while. Here's what happened.

#You don't build it, you nudge it

First, a thing I had to swallow: you can't train a useful model from scratch at home. That costs millions of dollars and most of the internet. My from-scratch one was a teaching toy, nothing more.

The real move is different. You take a small open model that already speaks English (I used Llama 3.2, the one-billion-parameter version, small enough to run on my laptop) and you nudge it toward your style. The technique is called LoRA. Instead of retraining the whole model, which is gigantic, you freeze it and train a tiny set of extra knobs bolted onto the side. About a million knobs instead of a billion. It's quick and it fits on a MacBook.

I fed it a few thousand words of my own writing and let it train. Fifty seconds on an M1. Then I asked it to talk about one of my projects.

#What it gave back

I built Leonard. It's a simple thing, really. I wanted it to be able to understand and respond to me when I talk... It's like having a little buddy who knows what I'm talking about.

Read that and you can see both halves of the problem at once.

The cadence is kind of me. "It's a simple thing, really." "A little buddy." Short, plain sentences. The nudge worked, a little.

But it has no earthly idea what Leonard actually is. Leonard isn't a thing you talk to. It's a developer tool that sits next to my code. The model made up every single fact in that paragraph, confidently, in a voice that's faintly mine. A vaguely-me-shaped liar.

#Why it lied

Two reasons, and the second is the real one.

First: fine-tuning teaches style, not facts. I was always going to get the rhythm of my writing before I got anything true, because that is what this technique does. If you want a model to actually know things about your stuff, that's a different tool. You let it look the facts up. You don't bake them in.

Second, and this is the one that matters: I gave it almost nothing to learn from. A few thousand words. And here's the tell, the same one I hit on the from-scratch model:

When a model gets better at the material it studied and worse at everything else, it isn't learning. It's memorizing the answer key. With only a few pages of me to chew on, memorizing was the cheapest way to lower its score, so that's what it did. Same lesson the from-scratch toy taught me, now on a grown-up model. The size of the model barely mattered. The size of the data was the whole game.

#So why isn't this a waste

Couple of reasons.

The thing that actually carries my voice right now isn't a fine-tune at all. It's dumber and it works better. I wrote down how I sound, in plain English, and I hand that description to a big capable model as part of the prompt. No training. It reads the description and writes accordingly. For style, a good prompt on a big model beats a fine-tune on a small one, and it will keep beating it for a long time.

But the pipeline I built isn't garbage. It's just hungry. Every time I write more, it gets a little less of a liar, with zero changes to the code. The bottleneck was never the model or the math. It's me, sitting down and producing enough words to learn from. That's a problem I can actually fix.

And honestly, the lie was the most useful part. I can now feel the difference between a model that knows something and a model that's just good at sounding sure. From the outside they look identical. One is right and one is a vaguely-you-shaped liar, and the only way to tell them apart is to already know the answer. Worth remembering every time one of these things looks you in the eye and tells you something.

← all writing