← all writing

I built a tiny GPT from scratch to stop treating it like magic

I use these AI models every single day. And until last week, if you'd asked me how one actually worked under the hood, I'd have given you some hand-wave about "it predicts the next word" and changed the subject. That bugged me. So I sat down and built one from scratch, smallest piece first, until it stopped being magic.

This is the short version of that. The long version is Andrej Karpathy's "build GPT from scratch" series, which is what I followed. I'm not inventing anything here. I'm just a guy who wanted to actually understand the thing he leans on all day.

Everything below ran on my laptop. An M1 MacBook. No fancy GPU, no cloud bill. The whole point was to keep it small enough to watch.

#Start with the dumbest thing that works

The first model I built has no business being called a language model. It's a lookup table. For every character, it stores one thing: given this letter, what letter usually comes next? That's the whole model. 4,225 numbers.

You train it by showing it a pile of text (I used a megabyte of Shakespeare) and letting it learn which letters tend to follow which. Then you ask it to write. Here's what it gave back:

Somyol-mishif mavenouiaricr bundrolal menkepive

Garbage. But look closer. Those are pronounceable letter-runs, with spaces in sensible spots. It even picked up that Shakespeare puts NAMES IN CAPS followed by a colon. It learned the texture of the writing.

It just can't make words, and the reason is the whole story: it only ever sees one letter at a time. By the time it's choosing the fifth letter of a word, it has completely forgotten the first four. No memory.

#Give it memory: attention

The fix is the famous one. Attention. Strip the math off it and here's all it does: it lets each letter look back at the letters before it and decide which ones matter.

Picture a dinner table. Each letter asks a question ("I'm a q, I'm hoping for a u"). Every earlier letter wears a name tag. The letter listens hardest to whoever's tag matches its question, and takes information from them. That's attention. The model learns what to ask and what to put on the name tag.

I bolted one "attention head" onto the same setup and trained it again. Same Shakespeare, about 30 seconds:

Tous sthrey afr omy hirsours

Still not English. But now there's rhythm, longer word-shaped clumps, the bones of sentences. That jump is the model finally having a little memory to work with.

#Stack it up and you've got a GPT

The last step is mostly more of the same. Run several attention heads side by side so each can track a different thing. Add a small layer that lets each letter "think" about what it just gathered. Then stack that whole unit a few times deep, with some plumbing (residual connections and layernorm) whose only job is to keep a deep stack trainable.

That's it. That's a GPT. Mine is tiny, about 600,000 of those little numbers, which is nothing. I trained it for 87 seconds on the laptop:

thy soss with unto why thou. Thou though im thee of yoviles, not MAREOLANES: Wally, should?

Real words now. "thy," "with," "unto," "thou," "should." Made-up character names with colons, like the actual plays. It's nonsense if you read it for meaning, but it's English-shaped nonsense, and it taught itself that from a megabyte of text in under two minutes.

#The whole thing in one table

Same code, same data, same minute and a half of training. The only thing that changed was how much model I gave it.

That progression is the entire trick. There's no magic step in there. You give a next-letter guesser some memory, then more memory used more cleverly, and somewhere along the way it starts looking like language. The giant models you talk to every day are this exact thing, with billions of numbers instead of my 600,000, and most of the internet instead of one play.

#Why bother

I didn't build anything new. I rebuilt something that already exists, badly, on purpose, so I'd quit treating it as a black box. And it worked. I can't un-see the lookup table now. When one of these models confidently makes something up, I picture that next-letter guesser reaching for the most likely thing whether it's true or not, because underneath all the scale, that is what it is.

There's more I want to dig into. What that "temperature" setting actually does to the output. And what happens when I point this thing at my own writing instead of Shakespeare. (Short version: with only a few pages of me to learn from, it doesn't write like me, it just memorizes me, and that failure taught me more than the success did.) Those are the next posts.

If you want to do this yourself, go watch Karpathy, then close the video and type it out by hand. Watching isn't the same as building, and the building is where it stops being magic.

← all writing