TX 04.2 · TRANSMISSION · 2026-08-20 · SIGNAL CLEAR

I trained a language model from scratch on my gaming PC

29.9M parameters · 2.0B tokens · 9.0 hours · one RTX 3070 Ti (8 GB) · ~8 min read

At noon the weights were random noise. Nine hours later the model wrote this, unedited, from a single end-of-text token:

Once upon a time there was a little girl named Jane. She was three years old and liked to play in the garden. One day, Jane and her daddy went to the park. Jane saw a tall tree with lots of leaves and she was excited. She ran up to the tree and started to climb it. She went higher and higher until she reached the top. As she looked around, she noticed something shiny on the ground.

That is plot, sequence, object permanence, and a hook — from 29.9 million parameters trained on one consumer graphics card in the same machine I play games on. The code is public: github.com/insomniac-asif/able-origin.

What "from scratch" means here, precisely

The phrase gets stretched, so here is the exact scope. Three things are genuinely original:

• The code. model.py and train.py were written from an empty file. Not forked from nanoGPT, not adapted from a training repo.
• The tokenizer. A byte-level BPE, vocabulary 8192, trained on the raw corpus. Its merge table exists nowhere else.
• The weights. Initialized random, trained end-to-end. No base model, no checkpoint inheritance, no distillation from a larger teacher.

And one thing is not original, which matters just as much: the architectural building blocks. RMSNorm, rotary position embeddings, SwiGLU feed-forward, tied embeddings — these are published techniques and I implemented them, I did not invent them. No model on earth invents its own mathematics from scratch; every frontier model is assembled from the same public literature. Claiming otherwise would be the kind of overstatement this lab exists to avoid.

The shape of it

Eight layers, model dimension 512, eight attention heads, context length 512. Pre-norm residual blocks with the output projections scaled by depth at initialization, which keeps the residual stream from exploding as the layers stack. Input and output embeddings tied, which at this vocabulary size is a meaningful fraction of the parameter budget.

The corpus is TinyStories (Eldan & Li, 2023) — synthetic children's stories written with a small vocabulary. It was chosen deliberately: it is the best-known demonstration that a tiny model can learn fluent, coherent English rather than word salad, which makes it the honest first benchmark for a from-scratch run on 8 GB. 458.3 million tokens after encoding with my own tokenizer.

The bug that made the first attempt fifty times too slow

The first launch looked correct and was catastrophically wrong. GPU utilization read 100%, the fans spun up, and the loop produced one thousand tokens per second. Step 0 logged, then nothing for minutes.

Two causes, stacked. First, a model server was still holding roughly 5.7 GB of VRAM from earlier in the day, so my training allocation had nowhere to live and Windows silently spilled it into system RAM — the same sysmem-fallback failure I wrote about for inference, which apparently I needed to learn twice. Second, the rotary embedding produced non-contiguous tensors, so PyTorch's scaled dot-product attention quietly fell back from its fast kernel to the math implementation.

Unloading the resident model and adding .contiguous() before attention took it from 1k to 86,000 tokens per second. A factor of eighty-six, entirely from two things that produced no error message. The lesson is the one that keeps recurring on constrained hardware: 100% GPU utilization does not mean the GPU is doing your work.

The run

bf16 autocast, fused AdamW, cosine schedule with 500 steps of warmup, batch 32 × sequence 512 × gradient accumulation 2, gradients clipped at 1.0. Checkpoint and validation every 500 steps, a text sample every 2,000, and a hard --max-hours stop so the run would end cleanly rather than being killed mid-write.

The step that mattered emotionally was 2,000 — sixteen minutes in — when the first sample came back as English:

Once upon a time, there was a little girl named Lily. She loved to play outside in the snow. One day, she went to the snow and made a snowman with a big sled.

It is not good writing. The sled arrives from nowhere. But sixteen minutes earlier this thing could not spell, and nobody told it what a word was — it inferred that from the merge table up.

Loss fell fast and then crawled, which is what the curve is supposed to do: 9.12 at initialization (roughly the natural log of the vocabulary size, i.e. uniform guessing), 2.38 by step 500, 1.84 by 1,500, and then six more hours to grind out the last 0.5. Around 2 billion tokens for a 30M model is well past compute-optimal, so most of that tail is memorizing the corpus rather than getting smarter about language. A larger model would have used those hours better — but a larger model would not have fit.

What it cost

Nine hours of one RTX 3070 Ti and about a dollar of electricity. That is the entire budget. The received wisdom is that training a language model requires a datacenter, and for frontier scale that is true. At this scale it requires a gaming PC and an afternoon, and the difference between those two facts is where a lot of people stop before they start.

What it is not

It writes children's stories. It cannot answer a question, follow an instruction, or tell you a fact about the world, because none of those things are in TinyStories. It is a research artifact, not an assistant, and the gap between 30M and a useful model is several orders of magnitude of compute rather than a few more epochs.

What it is is a foundation I own end to end — every line of code, every merge in the tokenizer, every weight. That matters for what comes next, because you cannot run controlled architecture experiments on someone else's checkpoint and know what you are measuring.

What's next

1. able-bench — a seeded benchmark harness for this scale, with the multi-seed noise floor published up front so a small improvement can be honestly separated from luck. Most architecture claims at small scale die on exactly this.
2. honest-drift — a controlled study of how delta-rule linear attention degrades under quantization. It is a documented pain point with, as far as I can find, no published measurement at any scale, and answering it needs many cheap small runs — which makes an 8 GB card an advantage rather than a handicap.
3. In the browser — exporting this model to ONNX so it can run client-side, living on this site, asleep whenever the home GPU is training.

The model is small. The point is that it exists, that every part of it is accounted for, and that the barrier to entry was a card that also runs Valorant.

■ END OF TRANSMISSION · all transmissions · return to descent