#GuppyLM: A Repo That Shows You How to Train an LLM from Scratch

If you want to understand how large language models actually work, GuppyLM lets you train one yourself in about five minutes on a single Colab notebook.

#What it is

GuppyLM is an educational repository that walks you through the entire process of training a small but functional language model from scratch. It is not a production model. It is a learning tool designed to demystify what happens inside an LLM, from data preparation through tokenizer training, model architecture, the training loop, and inference.

The model itself is tiny, around 8.7 to 9 million parameters. It trains on a T4 GPU in roughly five minutes. After training, it can hold a conversation, though the conversation is that of a small, naive fish named Guppy who is mostly interested in rocks, bubbles, and food. The point is not that the model is powerful. The point is that you can see, read, and modify every line of code that makes it work.

#Why it matters

• LLMs are black boxes for most developers. The vast majority of people who use LLMs have no idea what is happening inside them. GuppyLM makes the entire pipeline transparent and small enough to hold in your head.
• The full pipeline in one notebook. Data, tokenizer, model, training loop, and inference are all there, in order, in readable code. You do not need to jump between five repos and a textbook to understand how the pieces connect.
• Vanilla transformer architecture. GuppyLM uses a standard transformer, not a heavily modified or optimized variant. This means what you learn here transfers directly to understanding larger models.
• Personality is baked into the weights. The fish personality of Guppy is not a system prompt layered on top. It is embedded in the model through training. This is a useful demonstration of how fine-tuning shapes model behavior at the weight level, not just the prompt level.
• Accessible infrastructure. A Colab notebook and a T4 GPU are free. There is no barrier to entry. You can also find the trained weights on Hugging Face and chat with the model directly.

#How it works

GuppyLM is built on a vanilla transformer architecture implemented in PyTorch. The code is intentionally short, readable, and easy to modify. The pipeline runs through five stages: preparing the training data, training a tokenizer, constructing the model, running the training loop, and performing inference.

The training data is curated to give the model its fish personality. After training on a T4 GPU for about five minutes, the model produces responses that are grammatically simple and topically narrow, but coherent. Ask Guppy about the meaning of life and it will tell you the answer is food. Ask if it is hungry and it will promise to swim to the top and eat everything. The personality is consistent because it was learned during training, not injected via a prompt.

The repo includes a ready-to-run Colab notebook, so you can train the model yourself without any local setup. The trained weights are also available on Hugging Face, so you can try the model in inference mode without training at all.

#Caveats

This is explicitly an educational project. The model is tiny, its capabilities are extremely limited, and its outputs are intentionally narrow. If you are looking for a model to use in a real application, GuppyLM is not it.

The simplicity that makes it a great learning tool also means it skips many of the optimizations and techniques used in production-scale models. You will understand the core transformer architecture, but you will not learn about distributed training, mixture-of-experts, RLHF, or any of the other techniques that make modern LLMs performant.

The fish personality is charming but also a limitation. The training data is narrow, which means the model cannot generalize beyond its domain. This is instructive, but it also means you should not expect the model to be useful for anything practical.

#Who it's for

GuppyLM is for developers who want to understand LLMs from first principles rather than treating them as API endpoints. If you have been calling GPT or Claude through an API and want to know what is actually happening under the hood, this repo is one of the most accessible entry points available. It is also useful for educators who need a concrete, runnable example to teach from.

The takeaway: if you want to understand LLMs from the ground up instead of just calling APIs, GuppyLM is one of the best small-scale educational repos available. It will not make you an expert, but it will make the architecture click in a way that reading papers alone rarely does.