Guide · Open source
How to use nanoGPT: train, sample and finetune GPT models
Updated · written with AI from the project's own documentation and checked against our catalogue data
nanoGPT is a short Python codebase for training GPT-style language models from scratch or finetuning pretrained GPT-2 checkpoints. This guide walks through installing it, training a small character-level model on Shakespeare, sampling text from it, and what the larger GPT-2 reproduction involves. Before you start, note that the README says nanoGPT is now old and deprecated, and suggests its newer relative nanochat instead.
nanoGPTkarpathy/nanoGPT · ★ 64.4k · MIT Install options & downloads Quick start
- 1Clone the repository. Run git clone on the nanoGPT GitHub repository to get the code onto your machine.
- 2Install dependencies. Run pip install with torch, numpy, transformers, datasets, tiktoken, wandb and tqdm.
- 3Prepare the Shakespeare data. Run data/shakespeare_char/prepare.py to download the text and create train.bin and val.bin.
- 4Train the model. Run train.py with config/train_shakespeare_char.py. Use the CPU flags from the README if you have no GPU.
- 5Sample from the model. Run sample.py with --out_dir pointing to your checkpoint directory to generate text.
- 6Adjust settings to your hardware. Change the config or command-line flags, such as block size, batch size and layer count, to fit your computer.
Know what nanoGPT is and whether it's the right project
The README describes nanoGPT as the simplest, fastest repository for training and finetuning medium-sized GPTs, and a rewrite of minGPT. The core logic is small: train.py is a roughly 300-line training loop and model.py is a roughly 300-line GPT model definition, which can optionally load GPT-2 weights from OpenAI.
The README includes an update stating that nanoGPT is now very old and deprecated, and that nanochat is a newer, improved cousin. The author notes it is very likely you meant to use nanochat. Read that first if you are choosing a project to learn from today.
Install nanoGPT and its dependencies
Clone the repository from GitHub, then install the Python dependencies with pip. You need Python and PyTorch available on your system.
The README lists these dependencies: PyTorch, numpy, transformers (to load GPT-2 checkpoints), datasets (to download and preprocess OpenWebText), tiktoken (OpenAI's BPE tokenizer code), wandb (optional logging) and tqdm (progress bars).
- pytorch: the deep learning framework
- numpy: array handling
- transformers: loads GPT-2 checkpoints
- datasets: downloads and preprocesses OpenWebText
- tiktoken: BPE tokenization
- wandb: optional logging
- tqdm: progress bars
git clone https://github.com/karpathy/nanoGPT.git
pip install torch numpy transformers datasets tiktoken wandb tqdmTrain a small character-level model on Shakespeare
The README's suggested first run is a character-level GPT trained on the works of Shakespeare. First, run the data preparation script. It downloads the text as a single file of about 1MB and converts it into a stream of integers, saved as train.bin and val.bin in that data directory.
If you have a GPU, train with the provided config file. The config sets a context size of up to 256 characters, 384 feature channels and a 6-layer Transformer with 6 heads per layer. The README reports that one A100 GPU takes about 3 minutes and reaches a best validation loss of 1.4697. Checkpoints are written to the --out_dir directory, which the config sets to out-shakespeare-char.
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.pySample text from your trained model
Once training finishes, point the sampling script at the output directory to generate text from the best model. The README shows samples in a Shakespeare style. It also notes these results are a rough character-level demonstration, not a production model.
If you trained on CPU, add --device=cpu so sampling runs on the same device.
python sample.py --out_dir=out-shakespeare-charRun on a laptop or CPU-only machine
The README gives a smaller command for a Mac or other modest computer. It sets --device=cpu, turns off PyTorch 2.0 compile with --compile=False, and reduces the model and run size: block size 64, batch size 12, 4 layers, 4 heads, embedding size 128, and 2000 iterations. The README reports this takes about 3 minutes and reaches a loss of 1.88, with weaker samples.
On Apple Silicon Macs with a recent PyTorch version, the README suggests adding --device=mps to use the on-chip GPU, which it says can significantly accelerate training (2-3X).
- Use --device=cpu on machines without a supported GPU
- Use --compile=False when running on CPU
- Use --device=mps on Apple Silicon with a recent PyTorch version
python train.py config/train_shakespeare_char.py --device=cpu --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0Finetune a GPT-2 checkpoint or reproduce GPT-2
The README says finetuning a pretrained GPT-2 checkpoint is likely to give better results than training from scratch, and the code can load GPT-2 weights from OpenAI. The README's example of the largest starting point is the GPT-2 1.3B model.
Reproducing GPT-2 (124M) on OpenWebText is a much larger job. The README first runs a data preparation script that creates train.bin and val.bin, then launches training with torchrun across 8 GPUs. It says you will want at least an 8X A100 40GB node, and that the run takes about 4 days to reach a loss of about 2.85. The README also covers multi-node training, and warns that without InfiniBand you should set NCCL_IB_DISABLE=1 and benchmark your interconnect, since multinode training may run much slower.
- For a single GPU, run python train.py with your chosen arguments
- Checkpoints are written periodically to --out_dir
- Sample from a trained model with python sample.py
python data/openwebtext/prepare.py
torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.pyWhat people use it for
- Reading a compact GPT training loop to understand how models are trained
- Training a small character-level model on Shakespeare as a first experiment
- Finetuning a GPT-2 checkpoint on your own text
- Modifying the model definition or training loop for research
- Comparing training speed on CPU, Apple Silicon MPS and CUDA GPUs
Frequently asked questions
Is nanoGPT still maintained?
The README says nanoGPT is now very old and deprecated, though the author will leave it online. It points to nanochat as the newer related project. If you are starting a new project, the README suggests checking nanochat first.
Is nanoGPT free to use?
nanoGPT is released under the MIT licence, as listed on its repository. The README does not discuss costs beyond the hardware you need to run it.
What do I need to run nanoGPT?
You need Python, PyTorch and pip, plus the dependencies listed in the README. A GPU makes training much faster, and the README says it can run on a CPU or an Apple Silicon Mac for small models.
Can I run nanoGPT without a GPU?
Yes, the README gives a CPU command using smaller model settings, which it says takes about 3 minutes to run. Larger models such as the GPT-2 reproduction are not practical without a multi-GPU setup.
How long does it take to reproduce GPT-2 with nanoGPT?
The README says reproducing GPT-2 (124M) on OpenWebText takes about 4 days on an 8x A100 40GB node. It notes this hardware is needed for that configuration.
What are alternatives to nanoGPT?
The README mentions minGPT, nanochat, GPT-NeoX and llm.c as comparable projects. nanochat is the newer project the README recommends instead.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
nanoGPT suits developers, students and researchers who want a compact, readable GPT training codebase and can run Python and edit config files. Because the README calls it deprecated and recommends nanochat, people looking for the newest project should start there, and non-programmers or anyone wanting to train large models without serious GPUs should pick a different tool.
Install options & downloads