I trained a 1B-parameter LLM from scratch on 20B tokens for about $200

A few months ago, I had the idea of making a LLM from scratch as a personal project (for learning and partly for improving my resume).

Since I learned a lot from other posts on here over the past year, I wanted to share the results.

TLDR: I trained a 1.1B param model on 20B tokens from fineweb-edu, then finetuned it on openhermes with LoRA to get a chat model. Total cost was about $200 (in February/March though, so it would probably be more expensive now).

- gguf for base and it model (requires my llama.cpp fork so probably not that useful to you): https://huggingface.co/ni-co-la-s/gemmeh-GGUF and https://huggingface.co/ni-co-la-s/gemmeh-it-GGUF

The architecture is based on Gemma3 since it was my most used model when I started. There are a few differences: - I have a smaller context length (4096) and because of that I didn't use sliding window attention. - I have a smaller vocabulary (32k, trained the tokenizer with sentencepiece) - I also tweaked some hyperparameters to reach my target parameter count.

For the data, I used fineweb-edu for training the tokenizer and pretraining the model. Then LoRA finetuned the model on openhermes. I purposely tried to find data from 2023 and earlier because I saw this post back then and thought it would be cool to test the model by asking it questions about the "future" (like I did in the gallery images).

For pretraining, I first did training runs on 2B tokens to test the architecture at 3 sizes: 185M, 500M and 1.1B. Then I did a final run of the 1.1B model on 20B training tokens.

185M 500M 1B (on 2B tokens) 1B (on 20B tokens) Total params 185M 527M 1.1B 1.1B GPU 3090 5090 H100 H100 Duration 19h 17h 13h 130h Final val perplexity 19.2 16.0 15.1 10.93 Also I logged in wandb generations from a few fixed prompts every 30M training tokens or so (was probably the most fun part of the project to check the new samples every couple hours to see the improvements) Here are a few examples for the final 1B model.

At 30M tokens seen text Let me tell you a story: a person, you should your child, and the other person who can take the time and the person with its own. If you do not want to give them a bit, you can learn from a student

At 20B tokens seen text Let me tell you a story: I lived in a large city and we were having a little get-together. We all knew each other for years – so much so that I was surprised to learn that we met. It was around this time that one of us decided to become a vegetarian.

At 30M tokens seen text The capital of France is by the other of the Western Europe. The U.S. and the church are the first of Christ in 1937, the other three times of the world.

At 20B tokens seen text The capital of France is Paris and its currency is the Euro. A French person is called a Francais. After the Second World War, the French government decided to introduce a new currency that was pegged to the dollar.

To get a chat model, I ran some Lora finetuning on the best 1B model, using Openhermes as a dataset. I also did it on vast.ai, but on a 3060 and over 52 hours. Reached a final validation perplexity of 2.71 (not that it means anything since it is not on the same dataset as the previous values)

text The answer is: Gravity is the force that causes objects on Earth to stay together. At 250M tokens seen

```text A frog's heart beating In the dark and damp wood A frog's voice, so soft No one can hear.

At 3M tokens seen text heatwaves and water. mangan What is the chemical formula of oxygen? mangan At 250M tokens seen text H2O.

Overall, over training that the model became more and more concise, especially compared to the base that was very yappy. Still, the quality is not very good for the total price. (when comparing to nanochat for example). When I have some more time, I will probably experiment with some full sft instead of LoRA, and maybe some extended datasets.

The post is already pretty long so I will just list quickly some of the other things I tried out: - Because my version had some differences with the original Gemma3 and also because I wanted to understand a bit better how it works, I added the architecture in a fork of llama.cpp. - To test it out, I vibecoded a WearOS app I used to run a Q2_K GGUF version of the 1B model (runs at about 2tok/s on my watch) - I ran a few benchmarks with lm-eval, nothing really interesting to note, it is weaker than Gemma3 1B across the board. - I deployed a demo website on GCP (deploying the model on CPU with the GGUFs) to analyze logprobs of the base model (and compare it with a few other small models) and chat with the instruction-tuned model. I don't know much about frontend so the React was completely vibecoded.

Even if the model is not that good, I learned a lot while doing it and I can only recommend to anyone who wants to better understand LLMs. It has also helped me in my job search process over the past 4 months (whether for getting more interviews or for doing better in ML technical interviews)