[Experiment 4] I built a game where I learn how an AI lab allocates resources in order to improve their models

This experiment is a direct follow up of experiment 3. In that one I built a data center simulator to learn about the hardware side, the GPUs, cooling, power, bandwidth and so on. But while making it I felt the hardware is only half of the story. The other half is what the AI labs actually do with all that compute. How does text from the internet actually become a model like GPT or Claude? What do words like pre-training, fine-tuning and RLHF?

So same logic as last time. I learn best by playing games about the subject (I learned coding through Codingame.com and I built PlaySupplyChain.com for supply chain), so I built a game about running an AI lab.

The game is called Learn LLM and it is free to play in the browser, no signup and no email needed: https://www.buildlooplabs.com/learnllm/

The Game

Same setup as always. Django backend, HTML/JS/CSS frontend, built with Cursor AI and Fable 5. Since I already own the website, I could just add this one as well. And same disclaimer as last time, the game logic is all in the frontend with almost no validation in the backend, so it is easy to cheat on the high score ranking. The game is about learning, not about competition.

So how is the game played - you start your AI lab with a few dollars, a used gaming GPU and a web scraper script. From there the (game) loop is basically the real life of an AI lab: Collect data, pick a model to train, pick the dataset size, train, sell the model.

Obviously, in real life, these AI labs rarely sell the model, instead they create services around it like chat or coding services.

So let’s look in the details of those steps

1. Collect data. Your scrapers and crawlers gather tokens into a a big dataset (corpus in the game). You start scraping the web, later you get data in various ways until the end where you generate synthetic data (because the public web literally doesn’t have enough data, which is a real thing happening right now)
2. Pick a model to train. The blueprints go from a tiny 10M parameters model up to a 400B frontier model. Bigger models sell for way more but need more amounts of data and compute
3. Pick the dataset size. Here you choose tokens per parameter: Rushed (5:1), Chinchilla-optimal (20:1) or Data-rich (40:1). The more tokens per parameter, the better the model.
4. Train. Your GPU fleet goes through the run (you can also manually click like those stupid click forever games if you don’t have enough electricity). The compute needed follows the real rule of thumb, FLOPs = 6 × parameters × tokens. And your GPUs draw electricity every second, so if your money hits zero the training pauses and you have to sell something.
5. Sell the model and reinvest. Finished models land in your Model Hub with a quality number and an eval score, and you sell them on the market

There is also a simple research tree where the game teaches the most. Instruction tuning (SFT) adds a fine-tuning stage to your pipeline and makes models sell for more, because a raw pretrained model just predicts the next token, it doesn’t follow instructions. RLHF (Reinforcement Learning with Human Feedback) adds an alignment stage on top of that. Then there is mixed precision, FlashAttention, data deduplication, mixture of experts, quantization, distillation. Every purchase pops up a small explanation card of what the thing actually is in the real world.

The Key Learnings

Like last time, this is what I want to remember:

Training compute is just 6 × parameters × tokens. When a lab says they trained a 70B model on 2 trillion tokens, you can calculate the FLOPs yourself on a napkin

Chinchilla, about 20 tokens per parameter, is the compute-optimal sweet spot. But I generally used the 40 tokens per parameter in the game so I could sell the model more expensive

Data is the underrated resource. Deduplication, curation and the mix of code/books/web is where a lot of quality comes from, not just size. And the data is reusable and every future model benefits from it.

A pretrained model is not an assistant. It only predicts next tokens. SFT (instruction tuning) teaches it to follow instructions and RLHF makes it helpful and polite. This is why the post-training research in the game multiplies the sale price so much, it is relatively cheap compute compared to pretraining but it is what makes the product these days.

Electricity is a real cost. In the game you can literally go bankrupt mid training run from the power bill. Frontier runs consume massive amount of energy

Scaling laws are why labs dare to spend billions. Quality improves smoothly and predictably when you scale parameters, data and compute together. The curve tells them in advance roughly how good the next model will be.

Suggestions for next version

The obvious missing piece is inference. In the game you sell the model file and that’s it, but in reality labs earn most from serving the models via API (my experiment 2 was exactly about paying for those tokens). Selling access instead of selling the model would be a fun addition.

I feel I could dig much deeper in one of the subjects. I am pretty sure that Claude Fable knows more about this than what was added in the game so I could probably have deepdives on each of those research subjects.

It would also be nice with rival labs that release models on their own schedule and crash your prices. Maybe even make this a multiplayer competing game would be fun.

And as usual, graphics are basic and it is not really mobile compatible. Same story as the data center game.

Conclusion

Between experiment 3 and this one I feel I now have a better understanding of the AI stack. The data center game taught me about the physical layer, and this one taught me how the models are actually made (or at least the resource allocation) on top of it. Pretraining, Chinchilla, SFT, RLHF, MoE, distillation, these were words I had heard about but playing with them as game mechanics made them stick in a better way.

And again, building a learning game like this takes about a morning with Cursor. The knowledge is already inside the model, the game is just a more fun way to pull it out than reading the papers.

Now let’s see if anyone beats my high score.

Thanks for reading

Andreas