nanoGPT
A small, readable codebase for training and finetuning GPT-style language models.
- Stars
- 64.4k
- Forks
- 11.1k
- Licence
- MIT
- Language
- Python
- Last update
- 12 months ago
- Type
- Library
What it is
nanoGPT is a short Python codebase for training GPT-style language models from scratch or finetuning existing GPT-2 checkpoints. The README says its training loop and model definition are each about 300 lines, so people use it to study and modify how these models are built and trained. The README notes the project is now old and deprecated, and points to a newer related project called nanochat.
- Train a small character-level model on Shakespeare text using the included quick start
- Finetune a pretrained GPT-2 checkpoint on a custom dataset
- Reproduce GPT-2 (124M) training on OpenWebText with a multi-GPU setup
- Run training on Apple Silicon by adding the --device=mps flag
Great for
- Developers who want to read a compact GPT training implementation
- Students learning how GPT models are trained
- Researchers who want to hack on a small model and training loop
- People who want to finetune a GPT-2 checkpoint on their own text
Not ideal for
- Anyone looking for the newest project: the README says nanoGPT is deprecated and suggests nanochat instead
- People without a GPU who want to train large models, since the README's full GPT-2 reproduction needs an 8x A100 node running for about 4 days
- Non-programmers, since it requires running Python scripts and editing config files
How to get it
- 1
Clone and build from source
You'll need: Python · PyTorch · pip
Clone the repository
git clone https://github.com/karpathy/nanoGPT.gitInstall dependencies
pip install torch numpy transformers datasets tiktoken wandb tqdm - 2
Run it
- Prepare the Shakespeare character dataset with python data/shakespeare_char/prepare.py
- Train a small model with python train.py config/train_shakespeare_char.py
- Sample text from the trained model with python sample.py --out_dir=out-shakespeare-char
- 3
Get the most out of it
- Use the --device=cpu and smaller-size flags shown in the README to run on a laptop without a GPU
- Add --device=mps on Apple Silicon with a recent PyTorch version for GPU-style acceleration
- Finetune a pretrained GPT-2 checkpoint, which the README suggests may give better results than training from scratch
- Use the optional wandb dependency for logging training runs
Downloads
Strengths
- The code is short and readable, with train.py and model.py each around 300 lines according to the README
- Includes runnable examples, from a few-minute CPU run to multi-GPU GPT-2 reproduction
- Can load OpenAI's GPT-2 weights for finetuning and baseline evaluation
Trade-offs
- The README states the project is deprecated and no longer actively developed in its newer form, with nanochat recommended instead
- Full reproduction requires a large multi-GPU setup and several days of training
- The README's baseline table compares against OpenWebText, which the README says is only an approximate open reproduction of GPT-2's original training data
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| nanoGPT (this) | 64.4k | Library | intermediate | 12 months ago | Developers who want to read a compact GPT training implementation |
| minGPT | 25k | Library | intermediate | 2.2 years ago | Students who want to read a working GPT implementation end to end |
| nanochat | 59.6k | AI model / agent | advanced | 2 months ago | Students who want to see how a full LLM training pipeline fits together |
| llm.c | 31.4k | Learning resource | advanced | 1.4 years ago | Developers who want to see how a GPT-2 style model is trained at the lowest level |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.