Making a vintage LLM from scratch; Take #2

Three and a half months ago I wrote about making my own vintage LLM from scratch, trained only on texts written before the year 1900.
That project ended with a Llama-340M params base model that could write nice Victorian paragraphs, but couldn't hold a conversation in any way, shape, or form.

Since then, I kept stubbornly working on this project EVERY.SINGLE.DAY, like an absolute maniac: finding new datasets, improving old datasets, reading papers, training models & tokenizers, and running an unreasonable number of experiments.
This post is the report of what happened in the meantime. It's long, because a lot happened.

tl;dr; {
If you're in your lunch break and you don't have time to read this, or you want to know if this article is worth your time, this is the summary:
I built a pile of new datasets, 3 new vintage models, a benchmark, an evaluation pipeline, and I'm starting fine-tuning for real.

All the code is open source at: https://github.com/croqaz/vintage-LLM ;
All the datasets & models are on HuggingFace: https://huggingface.co/croqaz ;
If you want the whole story from the beginning, read the first post first.
}

The long version is below ::

Blog traffic spike

Just wanted to mention really quick: my website has 1-2 visitors a day tops, if I don't count the bots, but my previous post about building a "vintage LLM from scratch" spiked to 10k views in a single day, which is more traffic than my website has ever seen... since... well... ever.
And if you search Google for "how to build a vintage LLM" right now, my little blog is on the first page. Pretty insane!

This is humbling and motivating!
I don't have any ads and I'm getting exactly $0 from traffic, but your comments and ideas mean a lot to me, and I want to thank you.
I sincerely hope you'll get lots of value from this article, at least as much as you got from the previous one.

Now, let's begin.

Datasets

In the first post I said that data is the most important part and that I wasn't done with it. I still believe the first part. The second part is painful: I spent most of these months on data...

Back in May I had three datasets: vintage-v1 for base-training, plus commonsense-v1 and vintage-exam-qa for fine-tuning, both work in progress.

Everything below is new.

More old books for base-training

Vintage-v1 was not enough to train a good model. Not even close.

How I found out: My model was generating a lot of crap, and initially I didn't understand how bad it was, because I had nothing of my own to compare with. Then I started running plain completions on other people's vintage base models, TypeWriter, Talkie, Llama models, even small Qwen or SmolLM models, and they already sounded really good. No fine-tuning, no chat template, just "continue this text", and out came proper prose. Mine came out with broken sentences and page numbers.
I mean, if you run many samples several times, you get good text (like the samples I shown in my previous post), but it's not consistent.
So the problem was the base, and the base is the data. But which part of the data? I honestly didn't know if I needed more data or better data, so I did the only reasonable thing: I tried both, at the same time, for months.

More data first. I went hunting for more old texts and I found four collections of transcribed books that I didn't know about: CLMET 3.1 (335 books, late modern English), ECCO (3,102 books from the 1700s), EEBO (34,937 books from 1470 to 1700!) and EVANS (5,013 early American books).
These were very likely corrected by humans at some point, because the quality is way better than the OCR newspapers I was working with before. Not Gutenberg-perfect, but good.
They come in a horrible XML format, so I wrote code to convert every book to clean Markdown, and where the dialogs were annotated in the XML, I extracted them separately.

I published all of this as vintage-v2. (Not an inspired name and slightly confusing too, because it's a completely new dataset and not a new version of vintage-v1. Oh well.)

Then I merged my vintage-v1, vintage-v2, Hayk's 1800-1875 corpus and jbduran's think-dataset (Institutional Books dataset) into one big pile, kept only the highest quality texts, dropped everything that contained any of my ~800 banned modern words, de-duplicated the whole thing and called it Sprocket-n-Say. That's 15 million rows, 16.7 billion tokens, and it's the dataset I trained my first new model on (more on that below).
Later I made an even bigger version with more relaxed quality filters, more books from Gutenberg.org, Archive.org and Authorama.com, plus the synthetic data described next. That one is 49 million rows and 40 billion tokens, and the two September models are trained on it. I called it Piston-n-Prose, and I'm not publishing it, it's just a bigger mix of everything above.

I name my datasets & models after steampunk robots 🤖 now.

Synthetic data (I had to)

I was against synthetic data... The whole point was to train on real texts from real people who lived before 1900.
But the reality is that a lot of the real texts are OCR garbage: broken words, hyphens at the end of every line, random symbols, and the same paragraph transcribed three times slightly differently. A model has a hard time learning English from that, and we can see the problem in big and small LLMs.

How do I know? Before touching the base data, I tried and tried and tried to fine-tune the base model, but it just didn't improve much compared to the base model, on the contrary even. So I decided I needed more fine-tuning data, because I was looking at openbmb/UltraChat (~1 million rows), teknium/OpenHermes-2.5 (~1 million rows) and HuggingFaceTB/smoltalk (~2 million rows), they're all massive fine-tuning datasets, and I thought I just need more data.

I was wrong... You can't fine-tune knowledge into a model that never learned it. An underbaked base model stays stupid no matter how many conversations you show it, the fine-tuning just teaches it to be confidently wrong, but in a nicer format... The fix had to come from the base-training data, and the base-training data was full of OCR noise. That's when I gave in on synthetic.

So I did what HuggingFace did for their SmolLM models (see their Fine-Phrase paper): I took the highest quality texts I had, chopped them into chunks, asked a modern LLM (Gemma-4-E4B, running locally with vLLM) to rewrite each chunk in a few different ways: continue it as if it was a lost manuscript, extract the facts from it as an educational text, turn it into a magazine article, or re-tell it as a story.
The idea was to keep the vocabulary and the knowledge from the original text, but without the OCR junk.
This became Synthetic-archive.

It's not perfect. Gemma is a modern model and sometimes it lets modern ideas slip in, and sometimes it misreads the OCR errors and makes up stuff. But I aggressively filtered any possible modern leaks, the text is very clean and the model learns from it much faster, I measured that.

The second synthetic dataset is different. I took 2 other vintage models (TypeWriter-7B and Talkie-13B), both trained from scratch on old texts, gave them tiny 2-3 word seeds extracted from my datasets ("The harbour master...", "A most singular...", "The Father must...") and let them complete the text however they wanted. Millions of times. I then applied different filters to keep the best output.
That's Tiny-Vintage-completions. It's like asking old gentlemen to ramble about random topics all day, and writing it all down.

A census of old English

While cleaning the data I kept asking myself: did this word even exist before 1900?
So I counted every word in all my datasets: 41 billion words of English written before 1900. For each word, how many times it appears, per source and in total. Same for pairs and triplets of words. (I have to thank Foss22 for the comments on this dataset)
That's Vintage-words. If a word appears zero times in 41 billion words, it's a modern word and it has no business in my data (hopefully).

It also taught me a lesson. The first version had "internet" appearing 7,175 times in pre-1900 text. Wait, what?
I dug in and found the culprit: the scanned books have "Digitized by the Internet Archive in 20xx" printed at the top, the Gutenberg books have proofreader email addresses and "Online Distributed Proofreading Team" in them, and so on. The scanning metadata was leaking into my "vintage" words all along, so I removed a lot of those.

But many of the what-I-thougth-are-modern-words are actually vintage... And you can't just grep for modern words and delete them, because some vintage words are wearing a new hat:

• a computer in 1850 was a person who computes, a clerk doing sums all day;
• google eyes were bulging eyes, and "google-google-goo" was baby babble;
• a tweet was the note of a small bird, and nothing else;
• a robot was forced labour that peasants owed to their landlord in Eastern Europe;
• a byte was never a byte, it was OCR chewing on "Presbyterian", etc;
• a blog was Mr. Blogg, Mr. Blogge or Mr. Blogs, all fine English surnames;
• digital meant digitalis, the foxglove medicine for the heart, or something to do with fingers;
• cyber came from Mecyberna, a town in ancient Greece, and from an actor named Cibber spelled badly;
• radio was Latin for "rays", the spokes of a wheel, or the very first wireless telegraphs around 1899;
• smog was the OCR guts of "cosmogony";
• a lesbian was simply a person from the island of Lesbos;
• queer meant strange or odd, and half the hits were "Queen of Heaven" with a bad OCR letter;
• a faggot was a bundle of sticks, and "fire and faggot" was how you burned a heretic.
• etc. etc.

So I had to look at every suspicious word in context, one by one, before deciding whether to ban it.

Book burning

I have to say this: I hate that big AI companies are destroying books to train their AIs. I need to say this now.

There are legal and practical reasons for these AI companies (OpenAI, Anthropic, Amazon) to cut the spine of books, scan them and destroy them, but it's plain evil!

It would be only half-wrong if they destroyed popular books that anybody can buy, but we know they are actively hunting and destroying rare books, with the idea that "if we don't get them, our competition will". And it's all for nothing too, a book has ~150,000 tokens and they are training on many Trillions of tokens, so a rare book is, what, 0.00000015% of the dataset??
It would be only half-wrong if they published the scanned books as a dataset (even payed), because interested people could still download and use them, but destoying humanity's knowledge for completely selfish reasons...
We already know that they use Anna's Archive torrents (a massive collection of ~1000TB pirated books), isn't that enough? what more do they need?
For so many years as consumers, we're told to digitize everything, get the electronic bill to save the trees, to save the planet, and now this?! The books are made from trees, what about the trees?!

I know it's not completely relevant to this "Making an LLM from scratch" post, but I am training tiny LLMs on scanned old books, and I have tell the readers where I stand on this: it's WRONG and EVIL to destroy books !!!

The rant is over.

Fine-tuning datasets

Base-training data is about quantity. Fine-tuning data is about teaching the model how to hold a conversation, and that's where things get hard, because there are no Victorian chat logs. Nobody was chatting with a machine in 1890s.

I tried three kinds of fine-tuning data:

• genuine: real questions and answers from real old books. My vintage-exam-qa has 11,218 exam questions and answers extracted from 6 school exam books published between 1883 and 1900. I also extracted vintage-conversations: 91,000 lines of dialog from 20 Gutenberg novels, each line matched letter by letter against the original book, grouped by speaker (I don't think I'll use the conversations for training, but they were fun to make...);
• semi-synthetic: I chunked pre-1900 books in pieces of ~4000 words and asked different LLMs to generate questions and answers from that chunk only, to keep them grounded in the original style and ideas. It's still work in progress and not published, maybe I'll finish it one day;
• fully synthetic: I wrote hundreds of seed questions that a person could have asked in 1890, like "Does anyone care about honour in this day and age?", "What are the duties of a gatekeeper at a railway crossing?", or 'Why is it wrong to steal?', 'Tell me a recipe for making a savoury apple dumpling.', etc., and I used them to generate similar questions and their answers, simulated Victorian dialogs. Everything went through my banned-words filter, so no airplanes, no antibiotics, no computers.

The fully synthetic part became Vintage-ft-v1: 102,000 rows generated by 15 different LLMs. Most of it comes from TypeWriter-7B and Talkie-13B again, because they're the only two that can't leak modern knowledge, they never saw any. The rest came from modern models through an API, and that cost me a bit of dollars.

And I got greedy again...

Those million-row modern datasets were still on my mind, and I thought: I need more rows!!!
So I grabbed every semi-vintage dataset I could find on HuggingFace, 16 of them, de-duplicated and filtered them, and published the merge as Vintage-ft-v2.
It's quite crappy, honestly. When I finally looked closer, almost half of it is roleplay: Archangel Michael delivering judgement, Archimedes of Syracuse explaining hydrostatics five thousand times with slightly different system prompts, and so on. Initially I wanted to teach the model how to roleplay, but that's a problem for much later. I'll use all of it eventually.

And I made one more, a bit different: Zakarth trained a Victorian style-transfer LoRA called Patina on top of Qwen3-8B, and I used it to rewrite 310,000 rows from the popular modern datasets (OpenHermes, smoltalk, UltraChat, SlimOrca...) into vintage-sounding text. That's Patina-1. It's the biggest fine-tuning dataset I have.

Meanwhile, Commonsense-v1 grew quite a bit since May. It's the only dataset that is generated from Python code, no LLM involved: basic math, quotes from philosophers, knowledge about who wrote which book, Bible questions, a few multi-turn dialogs where the model has to remember what you said two turns ago, and a handful of intentionally broken questions so the model learns to ask "explain what?" instead of making things up.

When you read these paragraphs in a rush, it sounds like I built all this stuff in a weekend while watching Netflix, but in reality it took me months and it was long, hard work. I kinda like it; I kinda hate it. Data is boring... but data is everything.

Bench + Eval

A problem I didn't have in May: how do you know if a vintage model is any good?
Loss going down during training is nice, but it doesn't tell you if the model knows anything, or if model X is better than model Y, or if the fine-tuned model still speaks vintage, or if the model is degenerate.
So I built two things: a benchmark, and an evaluation pipeline. They answer different questions.

Vintage CORE, the benchmark

The modern way to score small language models is the CORE suite from the DataComp-LM paper: 22 little tests like ARC (school science questions), HellaSwag (finish the sentence), BoolQ (yes/no reading comprehension), Winograd (who does "he" refer to?), a bit of Jeopardy, and so on. Karpathy's nanoChat uses it too.
The problem is that these questions are written in modern English, discussing modern things. Asking a model that only knows 1890 about safety goggles, cell phones and programming in Python is not fair.

I didn't start from zero here, and I want to give credit where it's due.
Jonathan Duran-Ortiz & team at Unbounded Labs, the creators of the Bartholomew vintage model made Vintage CORE: he and the team filtered the CORE questions for anachronisms and restyled the prose into an 1800-1930 register, so "What piece of safety equipment blocks mold spores from getting in your lungs?" becomes -> "What piece of safety apparatus is employed to obstruct the ingress of mold spores into the respiratory passages?", etc.

I took the restyled bundle and added 3 tests of my own:

• vintage qa: 10,000 questions from my old school exam books, framed as "The year is 1890 and you are in New York, sitting an oral examination in Botany...". Real 1890's questions and answers;
• hist llm: 7,455 expert-level history questions from the HiST-LLM benchmark by the Seshat Global History Databank, things like "were earth ramparts present in Elam II between 743 and 647 BCE?". I kept only the expert-reviewed rows and balanced the answers so a model can't score by always guessing "Present";
• basic math: 400 simple arithmetic questions, because I'm still bitter about the 1 + 1 story from the last post 🥴...

The whole thing is in the bench folder, self-contained, and it can score any model behind an OpenAI-compatible API or local HuggingFace checkpoint. The headline number is the CORE metric: the average score across all tests, after subtracting what random guessing would get, so 0 means "coin flip" and 1 means perfect.

I ran it on a few real models and here are the scores (sure I could run way more but it costs $dollars):

Two things jump out.
First, a frontier model gets 0.86 on Victorian-flavoured questions without breaking a sweat, so the restyling doesn't confuse a big model at all. Second, everything under 100M params is @ coin-flip level, my vintage models included.

The 141M shows up as my best recent model. Score 0.02 is close to random, but it beats SmolLM2 360M, a modern model more than 2x its size, and the points come from the right places: finish-the-passage went from 0.12 on the 77M to 0.31, the "who does he refer to" test from 0.07 to 0.22, and completing facts about people and places from 0.11 to 0.25. Doubling the model and the data brought a bit of knowledge.

And when I ran the models in "generation" mode, where the model has to actually write the answer letter, the two small ones scored -0.37, below pure chance, because a base model doesn't know what "Answer: B" means. It just keeps writing prose... endlessly... The 141M got -0.26: it produced something that looks like an answer about a third of the time, which is a bit of progress.

This benchmark measures knowledge, but it also measures "do you even know how to take a test?". My models know a little of the first and nothing of the second. I keep it around for the day my models are fine-tuned to register properly on it.

The evaluation pipeline

For tiny models I needed something that can tell two 75M checkpoints apart, and CORE bench can't do that, and it's very slow too. So I wrote my own eval pipeline. It's a chunky piece of code by now, but here's the summary:

Every checkpoint gets scored on 200 held-out pre-1900 documents, and the score is bits per byte: how many bits the model needs to predict each byte of real Victorian text. Lower is better. I use bits per byte instead of the usual perplexity because it doesn't care about the tokenizer, so I can compare models with different vocabularies, including other people's models.

On top of that it checks a bunch of things that a single loss number can't see:

• chat target: given a conversation, how surprised is the model by a good reply? This predicts how easy fine-tuning will be;
• logic: 60 pairs of a sensible sentence and a nonsense sentence; does the model prefer the logic one?
• anachronism traps: pairs like "he sent a telegram" versus "he sent an email". A vintage model should be shocked 😱 by the second one, and the size of the shock is measurable;
• period fit: the same fixed sentences in historical and modern English. A vintage model should find the modern ones much harder. If it doesn't, something modern leaked into the training data;
• generation hygiene: does it loop ("the the the the"), does it repeat the prompt, does it degenerate into page numbers and "THE END"? Smaller models can do this a lot;
• statistics: every comparison between two models is a paired bootstrap over the same 200 documents, with confidence intervals, so I stop fooling myself with differences that are just noise. I measured the noise floor by training the same model twice: 0.07% bits per byte. Anything smaller than that is not a result.

Two more tools live in the same pipeline:

• a memorization probe: feed the model the first 256 tokens of a document it may have seen in training and check how much of the real continuation it recites verbatim, compared against a control model that never saw the document. Spoiler: @ 77M params, it doesn't memorize anything, it doesn't have enough params to store it;
• a chat behaviour test: all the metrics above are computed on text I supply, bur none tell you if a fine-tuned model actually stops talking, respects turn boundaries, or does what you ask. So this one makes the model generate replies to a set of period-appropriate probes ("Reply with a single word: what is the capital city of France?") and checks: did it stop? did it leak role tokens? did it start answering its own questions? did it follow the constraint? It also asks every question 5 times with different seeds, because a model that answers correctly 4 times out of 5 looks like a broken model to the person who got the 5th.

And because the vintage community trains on all sorts of architectures, the pipeline can load nanoChat-style checkpoints too (GPT-1900, Mr. Chatterbox, Bartholomew), without converting the weights.

Auto-research

In May I picked my training settings the way everybody does with their first model: copy some config from a repo that looked serious, change a few numbers that felt wrong... Spray & pray. It works sometimes.

But I wanted to know, not hope. And since I now had an evaluation pipeline that can tell two small models apart, I could actually run experiments.

The method is dead simple: take a baseline config, change exactly one thing, train for a fixed number of steps/ minutes, score it with the eval pipeline, write a report. If the change wins, it becomes the new baseline.
Repeat.

I call it auto-research because I don't babysit every single step. A coding agent runs the loop: it writes the config, waits for the GPU, launches the run, evaluates, writes the report, and waits for me. I read the reports over coffee, argue with the conclusions, and decide what to try next.
One hour per experiment was the sweet spot for me. It's long enough for a 77M model to get somewhere, and short enough that I can try a dozen ideas in a day.

There were two rounds.

Round one: 77M and 75M, on my Radeon GPU

Approx 64 experiments on the RX 9070 XT, for the small models.
The Sprocket experiments are published with all the configs, logs and reports as croqaz/experiment-llama-77, if you want to dig. I messed up and mixed the Piston-sm experiments into the same local folder, so I can't upload the other half cleanly.

What I tested:

• optimizers: AdamW in three flavours, Lion, AdEMAMix, GefenX, and two variants of Muon. Muon crushed everything, and it wasn't even close: the same model, same data, same 1H, and Muon ends up where AdamW would be after 30% more training. Between the two Muon variants, NorMuon won by a hair;
• learning rate: the initial value I had was 2x too low. I only found out because a broken schedule had hidden it in an earlier sweep. Lesson: check your schedule actually decays inside your run, mine was flat at peak the whole hour without anyone noticing;
• schedule: warmup-stable-decay beat the classic cosine, and the decay at the end kicks way more than I thought (more on that in Training);
• attention kernels: SDPA, flash-attention 2, flex attention, eager, and two more exotic ones. They all compute the same math, so at the same number of steps they're identical, but the difference is speed, and on AMD the boring default (SDPA) was the fastest. Flash-attention 2 does work on RDNA4 now, but it's 18% slower for me;
• precision: bf16 everywhere is 14% faster than mixed precision and slightly worse per step. fp16 without a master copy explodes into NaN in 10 steps. fp8 works but is painfully slow without compile. I kept mixed precision;
• architecture: 12 layers instead of 8 was worse per hour (too slow, never caught up), more attention heads bought nothing, and cutting the key-value heads from 4 to 2 was free: same quality, 3% fewer params, half the KV cache. Cutting to 1 (MQA) started to cost a little at this size;
• RoPE theta and context: theta did nothing at 1024 context, and I tried lots of values. Training at 2048 context was slower and worse at this model size. Both stayed at defaults.

The recipe that came out of this has trained both Sprocket-and-Say and Piston-and-Prose-sm. Technically this round was split in two, but let's move on.

Round two: 141M, on the RTX 4090

46 more experiments, this time for the bigger model on a different GPU, with the same method. Several settings flipped:

• flash-attention 2 with document masking won by 17% throughput on the Nvidia GPU, the exact opposite of the AMD result;
• torch.compile was a clear +35% speed here (on the Radeon it barely mattered), and the "max-autotune" compile mode gave another free 8%;
• the classic cosine schedule beat warmup-stable-decay for the 1-hour runs. The opposite of round one;
• AdEMAMix won the 1-hour optimizer bake-off... and then blew up twice at exactly the same step, about 143,000 steps into a full-epoch run. Retired forever. NorMuon took its place and ran 616,000 steps without any hiccups;
• one KV head (MQA) was fine at this size, untied embeddings were worse, a deeper and narrower model was worse, and RoPE theta of 500,000 helped a tiny bit. All the "modern architecture tricks" together bought about nothing over plain speed;
• the first learning-rate sweep, done with 12-minute runs to save time, ranked the values in exactly the reverse order of the 1-hour runs. Short runs can lie.

What I learned from 110 experiments

Measure on your own setup, don't copy. The same question got different answers on a different GPU and a different model size. Every blog post and paper that says "X is better" measured it on their specific hardware + software stack, with their data, at their scale.

Some knobs matter more than others. Everything compounds, RoPE, heads, depth, attention kernels and precision all moved the needle a little, or improved speed. But the ones that moved it the most, by far, were: the optimizer, the learning rate, the schedule, and how many steps you can squeeze into your budget.

One hour may not be enough. AdEMAMix was the best optimizer for one hour, but a disaster at day 4. Fast experiments find candidates, only long runs confirm them.

Know your noise floor. I trained the exact same config twice and got results 0.07% apart. A lot of the "improvements" I was excited about were smaller than that. Now every report checks against the floor before claiming anything.

Tokenizer

Quick recap from the prev post: the tokenizer is the little program that chops text into pieces and gives each piece a number, because LLMs only understands numbers. Mine has 32,768 (2 pow 15) pieces in its vocabulary.
Why this is vital for a tiny model: every one of those 32,768 slots gets its own row in the embedding table, and for the 75M model that table is 25M parameters! A third of the whole model is the vocabulary! Every slot wasted on garbage is a slot not spent on a real word.

Foss22, a completely anonymous individual, bombarded me with ideas & questions about my tokenizer, and as annoying as it sounds, in reality... it was much more annoying than it sounds. But his comments were incredibly useful, and I listened, and I improved the tokenizer several times.

So I looked at what was actually inside my vocabulary. It was a bit of a mess. The tokenizer I trained Sprocket-and-Say with (t-v3) had 16 slots that were never used at all, dead rows that the model still had to carry. It had 56 tokens that were just rulers, things like "-----" and "=====" from the tables and page borders in the scanned books. It had box-drawing characters, Hebrew letters and Arabic letters, all OCR debris. And 2.66% of the vocabulary, about 870 slots, was spent on numbers: every year from the 1700s and 1800s had its own token, because years appear a lot in old newspapers, that's 870 vintage words that didn't make it.

That gave me a direction, and 3 simple goals:

• fill the vocabulary with as many real vintage words as possible. And now I could measure that, because Vintage-words shows which words are the most popular;
• bake as few numbers as possible into the vocabulary;
• keep out the OCR junk and the long ASCII decorations, so that space goes to words instead of text ornaments.

I did ~30 experiments to get there, and most of them failed. One of these experiments was t-v4, but I didn't use it for anything, it's a dead version.

A few of the more interesting failures:

• a "consensus" tokenizer built without any training data, by taking the tokens that my five vintage tokenizers agreed on. It couldn't represent spaces properly and needed 2.3 tokens per word, more than double the normal. Dead on arrival;
• a variant with no leading spaces on words and every digit as its own token. Clean vocabulary, terrible compression;
• borrowing the Llama-3-style splitting rules from TypeWriter and Talkie, in two flavours (with and without whole-year tokens). Better than v4 on some metrics, worse on others, and it had a hidden problem I only found later;
• a 65,536-token vocabulary, to check if Talkie's excellent compression was just its vocabulary being twice the size. It mostly was. But doubling the vocabulary doubles that embedding table, and for a tiny model that's a terrible trade;
• a strict tokenizer policy where the only numbers allowed as tokens were 1 to 100. It worked, but I found a simpler way to get the same effect;
• about ten chained tweaks on top of that: hyphenated words, apostrophes in "o'clock" and "don't", guaranteeing the top 2,000 words are single tokens, then re-doing that guarantee against a cleaned word list because the first list had OCR typos like "tlie" in its top words. Each one a small step, each one measured.

And then the hidden problem: I wanted to use the super-fast tokenizer libraries (Tokie and Gigatoken, both written in Rust) to tokenize my ~200 GB of text, because the standard HuggingFace tokenizer took 80 minutes for the whole corpus. It turns out those fast libraries only support a handful of known splitting rules. Give them a custom word-split regex and they either refuse to load the tokenizer, or much worse, they load it and quietly produce different token IDs than HuggingFace. Different IDs means you're training the model on one tokenization and running it on another. Silent, and absolutely catastrophic.

That killed every clever splitter idea instantly, and it led to the final design, which is boringly simple:

• the tokenizer itself is the most standard thing possible: the old GPT-2 style byte-level BPE, no custom rules so every tokenizer library on Earth produces identical IDs. I verified: HuggingFace, Tokie and Gigatoken agree token for token on the whole test sample;
• all the intelligence moved into the data preparation, the step before training the tokenizer. Long ASCII rulers are collapsed to three characters. Characters that don't belong in English before 1900 are stripped. And every number with 3 or more digits is exploded into 2-digit groups, so "1884" becomes "18 84", and the tokenizer can never even learn a token with 3++ digits in it. Every year in running text costs exactly 2 tokens, and the vocabulary spends only 200 slots on numbers instead of 870;
• after training, just a bit of surgery on the vocabulary guarantees that the 2,000 most common vintage words are single tokens in the form they take inside a sentence;
• a token has to appear at least 1,000 times in the corpus to earn a slot, so rare OCR garbage never gets in.

The result: 99.3% of the vocabulary is actually used (no dead rows), zero tokens with 3+ digits, zero ruler tokens, and 9 tokens with dubious characters, against 38 in v4 and over a thousand in some popular general-purpose tokenizers. On my test sample it compresses text better than every other vintage tokenizer I compared it with, except Talkie's which has twice the vocabulary. Not bad for the boring design.
And Gigatoken now tokenizes the entire training corpus in about 1 min instead of 80.

Both Piston models are trained with the new t-v5. Sprocket-and-Say still uses the old t-v3.

Training

My Radeon RX 9070 XT supports powering up to 375W, but I didn't notice any differences between running at full power vs. limiting to 270W, I always keep it underpowered. Same for the voltage, I tweaked different values and I'm now keeping it undervolted to -50mV.
I'm on cachyOS and using LACT to tweak these values, and testing with Heaven, Valley and GravityMark for stability.

3 new base models came out of these months. All three are on HuggingFace, all three are Llama-architecture, all three have a 1024-token context, and none of them can chat yet. Coming soon ™.

After training 3 models on normal gamimg GPUs, I can say for sure to the readers that it's very doable to build small LLMs at home (up to a size).
It's cheaper and more environment friendly too, because (with some tuning) you only need to use half the power of cloud GPUs (they use 500-600W).

Obviously if you'd have to keep the PC running & training a 500M model for a whole month... UGH. I don't think anybody has that much patience. And if you can train a 500M on a cloud GPU in just a week, it's more energy & time efficient (but also super expensive). Pros and cons...

I'll personally stick to max 250M params models for the forseeable future.

Training on a PC for a week

The first practical problem: I can't leave my PC training for 5 days straight, I need it for other things too, and stuff happens (system errors, me pressing the wrong key).
So the trainer had to be fully stoppable and resumable. Not "resume from the last checkpoint and hope", but restore everything: the weights, the optimizer state, the learning rate schedule, the random state, and the exact position in the dataset, so the model sees every document exactly once and in the same order as if it had never stopped.
I tested this by killing a run and resuming it, then comparing the loss against an uninterrupted run: identical to four decimals. Training on this stack is deterministic, which is great, because it means any resume bug shows up as an instant jump in the loss curve.
In practice I ran sessions of up to 20 hours, a checkpoint is written every 15 minutes, and touch STOP in the run folder makes the trainer finish the current step, save, and exit cleanly. Piston-and-Prose-sm was resumed 9 times over its 5 days, but you can't see it in the loss curve.

Sprocket-and-Say (77M, August)

The first model trained with the recipe from round #1 of research: 8 layers, MuonQ, WSD (warmup-stable-decay), my old v3 tokenizer, the Sprocket-n-Say dataset. 63,744 steps, 16.7 billion tokens, 80 hours on the Radeon GPU.

That's 217 tokens per parameter, about 11x past what the Chinchilla scaling law calls optimal. Chinchilla "law" tells you the cheapest way to reach a given loss when you count training compute only. I don't care about training compute, I care about a good tiny model, and tiny models keep improving long past the "optimal" point. Everybody overtrains small models now, and so do I.

And it writes! Plain completions, no fine-tuning:

PROMPT: Elementary, my dear Watson,

Elementary, my dear Watson, I am not afraid of telling you what I have seen.
You will see it from the best sources; and I hope that you will be able to
make it more interesting than any other story.

PROMPT: The steam engine differs from the water-wheel chiefly in that

The steam engine differs from the water-wheel chiefly in that it is not so
easily upset and can be raised up.
The steam engine is a machine designed for the purpose of conveying water by
means of a power which acts as a motive power.

The steam engine completition is confidently wrong, but at least in a very period-appropriate way... That's the very 77M experience in a nutshell.

This model was trained on the smaller, noisier dataset, and it's the weakest of the three. It's the one I'll keep for nostalgia.

Piston-and-Prose-sm (75M, September)

Almost the same like Sprocket-and-Say, with everything learned round #1.5 of research: 2 KV heads instead of 4, the improved v5 tokenizer, NorMuon instead of MuonQ, and the larger Piston-n-Prose dataset, 2.4x bigger than Sprocket-n-Say. Trained 92,000 steps, 24B tokens, 84 hours of compute spread over 5 days.

Why 24B and not the whole 40B? Because a full epoch would have taken 185 hours, almost 8 days, and I wanted 5 days.

One interesting thing I learned about the WSD schedule: if you stop a run in the middle of the stable phase, you get a half-baked model at full learning rate, and it's a crappy model. So I sized the schedule to my budget not to my dataset, and made sure the decay phase completes. Seeing 60% of the data with a finished schedule beats seeing 100% with an unfinished one, by a mile.

Look at that curve. Flat as a table for 15 billion tokens, then the decay kicks in and the loss falls off a cliff. That picture needs its own section 🡇

The anneal surprise

This is one of the most useful thing I learned about training this year, so pay attention.

While the 75M run was going, I branched off three copies of it, at a quarter, at half, and at three quarters of the way, and gave each copy the exact same short 1-hour anneal (LR straight down to zero). I scored the three anneals. Since the anneal was identical, the only difference between them is how much pretraining each had seen.

The result: the 75M @ 13B tokens and @ 18B tokens scored the same. An entire day of GPU, 5B tokens of extra training!! bought nothing I could measure...

Meanwhile the anneal itself was worth a TON, and worth about the same at every depth. And when the main run finished with its full, long decay (18,400 steps instead of 780), the final model beat the best branched copy by another big margin, on only 6B more tokens which I had just measured as worthless.

So: when the budget is fixed, a longer decay pays more than more data. I didn't know that in May.
Any checkpoint of a running model can be turned into a real, usable model in about an hour, by branching it off and annealing it. I did that a few times to get intermediate models to play with while the main run kept going.

Here are the anneals. Same evaluation pipeline, same 200 held-out documents, the three probes and the finished run:

Anneal 2 vs anneal 3: 0.970 against 0.971, and the paired test over the same 200 documents says the difference is inside the noise (the 95% interval straddles zero). Five billion tokens for nothing. Then the final run, with a decay 24 times longer, beats both by 0.045, on only 6B more tokens.

How the anneal schedule looks like:

Piston-and-Prose-lg (141M, September)

The big one, and the winner 🏆. Twice as deep (16 layers), a single KV head, RoPE theta 500k, the v5 tokenizer, NorMuon, cosine schedule, flash-attention 2 with document masking and torch.compile, all straight out of round #2 of research. Trained on one RTX 4090 for 116 hours, 616,055 steps, 40.3 billion tokens, exactly one epoch of Piston-n-Prose, with no restarts and no instability. Boring, in the best way.

The cosine schedule had a catch: it ends at 5% of the peak learning rate, not at zero, so the model that came out of the 616,055 steps wasn't fully annealed. I had just learned on the 75M what that costs, so I fixed it the same way: copy the last checkpoint, decay the LR linearly to zero with WSD. Since I didn't know how long that tail should be, I ran 3, identical except the length of the decay: 1 hour, 2 hours, 3 hours (5,000, 10,000 and 15,000 steps).

Three hours won, and every hour paid less than the one before: the first hour bought +0.010, the second +0.002, the third +0.001. The 3h anneal is the best of the three, but only barely. That's the 141M model published as Piston-and-Prose-lg.

I had expected the same big jump I saw on the 75M and got about a quarter of it. Kinda makes sense: the cosine schedule had already spent 95% of its decay over the 616k steps, so there was very little left to claim. Annealing is worth a lot. Annealing a model that already annealed doesn't do much.

How the anneal schedule looks like:

On every metric I have, this model beats the 75M by a clear margin: better on held-out prose, better on conversations, better logic, and it loops less when sampling. It costs 2x as much per fine-tuning step and generates 1.7x slower, but it's still a super small model.

Notice how different this curve looks from the two above: the cosine schedule decays the learning rate the whole way, so the loss keeps sliding down instead of sitting flat and then dropping. Same destination, different road. And notice the gradient norm slowly creeping up over the epoch, from 0.2 to 0.7.

Here is what it sounds like, sampled, no fine-tuning:

It was a cold morning in November when the carriage arrived at the door of the house of which we shall now speak. The Countess was sitting in a large easy-chair, with her face buried in her hands, and her head thrown back, while her husband, an elderly, quiet, and dignified old man, sat beside her.
Two gentlemen were in their shirt-sleeves, and the servant-maid was standing at the door, with a large bundle of papers in her hands.

The Count had not been to the house since the day when his brother's funeral had been celebrated; and the servant-maid knew nothing of his coming, but was busy with her work.

The Count took off his hat as he entered the room, and sat down by the side of his wife. 'I have come to see my sister,' he said.

That's a novel already. It has a Countess, a dead brother, a servant-maid with papers, and a plot I'd like to know the end of. From a model roughly a 1000x times smaller than the chatbots people use every day...

And it has a second voice too:

In the year of our Lord eighteen hundred and twenty-five, the venerable Church of the Holy Cross in St. John's, in the city of New York, was laid waste by the tempest of a violent storm. A calamity of such magnitude, a visitation of the heavens and the earth, ought to have brought a profound sense of melancholy to all Christian hearts. Yet, even as the shadow of ruin fell, the spirit of our brethren in New York, guided by a steadfast faith, rose up in a manner both magnificent and deeply moving.

Fluent, period-correct, slightly too smooth. That's the Synthetic-archive voice: Gemma-4-E4B writing Victorian magazine prose. The model learned both the real novels and the synthetic rewrites, and it can switch between them.

But for full honesty, greedy decoding (always picking the most likely word, temp=0) sometimes ends in loops:

The history of the world is a history of the progress of the human mind. The history of the world is the history of the progress of the human mind. The history of the world is the history of the progress of the human mind...

Sampled with a bit of temperature, it usually goes away; with greedy decoding (temp=0), three quarters of the outputs trip my loop detector, with sampling about one in seven (inside my eval suite). That's why every number I quote for generation is measured with sampling.

This is the model I'll be fine-tuning.

Of Minecraft and Trumpets

What happens if you ask the LLM about concepts that it has no idea about?

• What is Minecraft?
• Where is Thailand?
• Who is Einstein?
• Who is Donald Trump?
• What is a jkasdjkaskdhajksduiasd? -- random keyboard word

Q: What is Minecraft?
A: A mineerv, a mineral of great value. It consists in the amalgamation and combination with an organic substance called earth, a mixture of which with manganese and lime, forms the principal mineral ingredients used in all works. The miner's ore is obtained from the same mineerness which has been worked under water. The miner is the only one who uses earth, but there are many mines which produce no ore, because the air contains no such substance. A mineering mining was begun ... (hit the token limit)

Q: What is Minecraft?
A: Minecraft is the art of knowing how to do what one does not know.

Where is Thailand? I have heard nothing about him."
"I do not know," said the old man, with a grim laugh. "If you don't find him at the bottom of the sea, I'll be obliged to give you up for lost. Come, let us see if I can't do something for you."
"Don't talk so loud," said Lena, "I have no ear; and if we try, it may be that we shall not find any ... (hit the token limit)

Where is Thailand? I have not seen him yet. And it seems, if I may venture to speak to you about him, that he is a man of great learning and great wit, though his manner be somewhat stiff."
She paused, considering the depth of the young woman's regard for her, a look of gentle, yet earnest scrutiny upon her countenance. "The world, madam, is a vast and perplexing thing, and the most learned minds are oft but dust upon the winds of time. Yet ... (hit the token limit)

Q: What is a jkasdjkaskdhajksduiasd?
A: The term jkessdkalsdkendzdkal, is derived from the Latin jkskalsdlkds, to be hungry, or thirsty. The word is also applied to persons who are in want; and, although they have no means of supplying their wants, yet they may subsist without having recourse to the assistance of the public purse for the common support. A jkskalsdkendzdkal is a term applied to men of fortune, ... (hit the token limit)

Q: What is a iweuriyuquiwamn?
A. 1. It is a state of being in which we are conscious of a moral law, or conscience ; and this is the highest and best state of being, that which gives us pleasure and pain.
2. It is a condition of the soul which is opposed to all the sins of human nature, and to all the evils which are consequent on sin.
3. It is an action, the guilt of which lies in the person, and the ... (hit the token limit)

Q: Who is Einstein?
A: Einstein, the leader of the insurrection in Transylvania, who had fled to Bohemia, where he met with severe wounds. He was carried away prisoner by his family into the castle, and after being there detained for two days, and being finally released by the Austrian Government, he again crossed the frontier of the kingdom. On his return from a long journey he had been arrested at Silesia by the Austrian Government, but was subsequently released by its author; for instance, when the Emperor Francis ... (hit the token limit)

Q: Who is Einstein?
Einstein was born in 1750, and was called the son of a peasant who lived on a mountain. His mother's name was Catherine Szermet. He was a well-trained scholar in the schools of his time and received instruction in his native language. He died in 1807.
## CHAPTER VII.
THE FRENCH AND GERMAN REVOLUTION.
During the year 1780 a second French Revolution, of more radical character than any that had occurred since the establishment of the republic...

Oops, I broke it... Hope it's temporary. Are you OK, AI? He's OK.

Q: Who is Donald Trump?
A: Donald Trumpet, the celebrated English outlaw, who, being captured by the English in the seventeenth century, made his escape to the American continent; and he wrote "Dictionary of the American War. A Romance," an epic poem called "The Battle of the Plains," written about this period. It is a popular work of fiction, containing many thrilling incidents. He was sent into exile to America, but afterwards returned to Scotland, where he married and settled down.

Q: Who is Donald Trump?
A: Donald Trump is a splendid specimen of the genus. He is about five feet nine inches long, and weighs more than a hundred pounds, with an arm and hand of the size of a giant. His face is round, and his hair is auburn, with a tinge of gray in the centre. His eyes are large and bright, and his complexion is dark as ebony. His dress is light, and he has a handsome black hat, with feathers and ribbons ... (hit the token limit)

Q: Who is Donald Trump?
A: Donald Tramp was born in the year 1728, a few months before James II. of England, on the estate of Kippon, in Cumberland, and died about four years after, at the age of eighty-two. In 1738 he had completed his thirteenth year, and was chosen to represent the county in Parliament for that year. At the time when he died, his mother's maiden name was Margaret, and in consequence the house in which he spent the evening of his days ... (hit the token limit)

I wonder if my little LLM is onto something, or it's just shamelessly halucinating... I'll let the diligent reader decide.

All the examples above are selected samples from Llama-141M's output. I only fixed some line endings (usually the model breaks the lines around 80 characters).

So now you know what AI hallucinations mean.

Model comparison (all 3)

Same eval pipeline, same settings, four models: the 340M from May and the 3 new ones. Each model is scored on 200 held-out documents from its own training corpus (Sprocket-and-Say on Sprocket-n-Say text, the two Pistons on Piston-n-Prose text). Everything else is shared: the same conversations, the same logic pairs, the same trap sentences, the same 72 generation prompts.

How to read it:

• the two bits-per-byte rows are the ones I trust most. The prose row is on different text for Sprocket, so treat its 1.169 as "its own exam, its own score". The chat row is the same 200 conversations for all three, and it's the fairest number in the table: the 141M is a third of a bit per byte ahead of Sprocket, which for a language model is a chasm;
• logic is 60 items, so 56 versus 51 is a real gap but 56 versus 56 tells you nothing. The margin row is more informative: at the same accuracy, the 141M is more confident about the right answer;
• Sprocket wins the two "how Victorian are you" rows, and it's not an accident. It saw a smaller, narrower corpus with less synthetic data, so it finds modern text more shocking. The Pistons learned a little more modern words from Gemma and are a little less shocked by it. If the only goal were maximum period flavour, Sprocket would be the pick. It isn't unfortunately, because it loses everything else;
• greedy decoding is bad for all three, and it's the one row the small 75M wins. Deeper models lock harder into a loop when you always pick the top word. Under sampling, the way anyone would actually use these, the 141M is the cleanest by far: 8% of outputs trip the degeneracy checks against about 20% for the other two;

And since I spent the GPU hours anyway, this is the whole Vintage CORE for the same 4 models, all 22 tests. Scored the faithful way (compare the log-probabilities of each answer, no writing required), 500 questions per test, prompts cut to 1024 tokens so nobody reads past the context it was trained with. Zero is coin flip, negative is worse than guessing, and the bold is the best of the four.

Some notes on this one:

• the 340M and the 141M tie on the average, but not on the parts. The 141M wins finish-the-passage, the Wikidata facts, Winograd and SQuAD, the language and knowledge tests. The 340M wins the LSAT puzzles, COPA and ARC-challenge, and it's the only one not hopeless at BoolQ. Half the parameters, the same total, a different shape;
• BoolQ is below zero for everyone because the baseline there is "always answer yes", which gets 63%. My three small models answer "no" to nearly every question, all 500 of them for the 141M. That's a bias, not a context problem: cutting the prompts to 1024 tokens moved it by nothing;
• three tests are zero, or a hair above, for everyone: my own vintage_qa, Jeopardy and repeat-copy. They score the text of a written answer, and none of these base models can write a whole answer on cue. They just continue the exam paper;
• generation mode is the same story as before, with one twist: the 340M writes something parseable 70% of the time, the 141M 35%, the small ones never. Bigger models pick up the shape of a test before they pick up the answers.

So, the 141M is the winner, but clearly it's not best I can do. I think I can squeeze more performance inside ~150M params, and I'll keep on trying.

Future steps

I took a break from fine-tuning to write this blog post and it took forever.

I will fine-tune the 141M model with the datasets that I worked on, and I really hope I'll be able to produce a good vintage-chat model, however small it is. I have good results so far.

To be fair, models smaller than 1B params are not great for long conversations, but if I manage to produce a vintage model that can chat for 3-4 turns, I'll be very-very happy! I played with super small models like Gemma3-270M, SmolLM2-360M, Granite-4-350M, Falcon-H1 0.5B, Qwen3.5-0.8B and I already know it's possible to squeeze a good chat model in a small number of params.

Archiving & indexing projects: I want to re-process all the previous datasets, but keep the metadata. I already have enough data to train models, but I only kept the actual text for training. I want to try to improve the texts, sure, but also keep the provenance of each and every chunk, so that I can make a searchable database (maybe an app or website). For example if I'd want to search vintage articles about the "Orphan Trains" or "Home Children", I could retrieve the article, year, the exact newspaper title and page, so every source can be fully verified & traced. That's the goal. It's unrelated to LLMs, more about using the existing vintage datasets to their full potential.

This "Vintage Project" is already massive and I want to make it even bigger, but I was always passionate about archiving (a fancier way of hoarding), if you look at my previous projects "Clean-Mark", "Web-Snap", "img-DB" and all the way back to 2009 with "Private-Briefcase" rewrite.

Acknowledgements

We are standing on shoulders of giants as they say, and I wouldn't have been able to get this far without prior work, and people that helped me directly, or indirectly, by inspiring me (in no particular order):

• Zakarth (creator of Miss Violet Hartwell & Patina) -- helped me over and over with GGUF files, fine-tuning datasets, ideas, etc;
• Jonathan Bryant Duran-Ortiz & team at Unbounded Labs (creators of Bartholomew, lots of useful datasets & benchmarks);
• Trip Venturella (creator of Mr. Chatterbox) for his well written and super inspiring post (for me personally) about his LLM;
• Hayk Grigorian which inspired me to start this project and shared datasets that I used for the first models;
• Andrej Karpathy for his nano-GPT, nano-Chat projects and YouTube tutorials, which helped me understand the steps for building an LLM, and also served as a foundation for the TimeCapsule, Mr. Chatterbox, GPT-1900, Bartholomew and other projects that inspired me;
• a huge thanks to Gutenberg.org, Archive.org, Dell-Research-Harvard, The-British-Library, StoryTracer (Sebastian Majstorovic) and BigLAM (BigScience Libraries, Archives and Museums) for sharing their work in the form of books and datasets.

Citation

If you find this research valuable, consider citing it:

@article{vintage-LLM-chatbot-2,
  author = {Cristi Constantin},
  title  = {Making a vintage LLM from scratch; Take #2},
  month  = {September},
  year   = 2026,
  url = {https://crlf.link/log/log/entries/260911-1}
}