Skip to content
Get packs

Guide · Open source

How to use minGPT: install, build a GPT model and train it

Updated · written with AI from the project's own documentation and checked against our catalogue data

minGPT is a small PyTorch re-implementation of the GPT transformer that covers both training and inference. It is meant to be short enough to read end to end, and it works as a library you import into your own Python project. This guide shows you how to install it, create a GPT-2 model, train it with the included Trainer, and find the demos to learn from.

minGPTkarpathy/minGPT · ★ 25k · MIT

Quick start

  1. 1Check prerequisites. Make sure Python and PyTorch are installed, since the README lists both as requirements.
  2. 2Clone the repository. Run git clone on the minGPT repository to get the code locally.
  3. 3Install in editable mode. Enter the minGPT directory and run pip install -e . so you can import mingpt in your project.
  4. 4Build a model. Get the default config, set model_type to gpt2, then create a GPT model from it.
  5. 5Train with your dataset. Write a Dataset subclass that yields LongTensors, then configure and run the Trainer.
  6. 6Study the demos. Open demo.ipynb or generate.ipynb, or run projects/adder or projects/chargpt, to see the library in action.

What minGPT is and what it contains

minGPT aims to be small, clean, interpretable and educational. The model code is about 300 lines, and the README describes the whole library as three files: a model file, a byte pair encoder and a training loop.

The model takes a sequence of indices and outputs a probability distribution over the next index in the sequence.

  • mingpt/model.py: the Transformer model definition
  • mingpt/bpe.py: a Byte Pair Encoder that converts between text and integer sequences, as OpenAI did for GPT
  • mingpt/trainer.py: GPT-independent PyTorch code that trains the model

Install minGPT as a library

To import mingpt into your own project, clone the repository and install it in editable mode. The README lists Python and PyTorch as the prerequisites.

git clone https://github.com/karpathy/minGPT.git cd minGPT pip install -e .

Create a GPT-2 model with a config

Models are built from a config object. The README example starts from the default config, sets the model type to gpt2, and sets the OpenAI vocabulary size and block size before creating the GPT model.

  • model_type: set to 'gpt2' for the 124M parameter version
  • vocab_size: 50257, the OpenAI model vocabulary
  • block_size: 1024, the input context length
from mingpt.model import GPT model_config = GPT.get_default_config() model_config.model_type = 'gpt2' model_config.vocab_size = 50257 # openai's model vocabulary model_config.block_size = 1024 # openai's model block_size (i.e. input context length) model = GPT(model_config)

Train the model with the Trainer

You provide your own subclass of torch.utils.data.Dataset that emits LongTensor examples of length up to 1024, with integers in the range [0, 50257). The Trainer takes a config, the model and the dataset, and run() starts training. The README example sets a learning rate of 5e-4, 1000 iterations and a batch size of 32; the file itself lists the other options.

from mingpt.trainer import Trainer train_config = Trainer.get_default_config() train_config.learning_rate = 5e-4 # many possible options, see the file train_config.max_iters = 1000 train_config.batch_size = 32 trainer = Trainer(train_config, model, train_dataset) trainer.run()

Learn from the demos and projects

The projects folder and two notebooks show the library in use. Start with the notebooks if you want a guided walk-through, or the projects if you want to see training on a specific task.

  • projects/adder: trains a GPT from scratch to add numbers, inspired by the addition section of the GPT-3 paper
  • projects/chargpt: trains a character-level language model on an input text file
  • demo.ipynb: minimal usage of GPT and Trainer on a simple sorting example
  • generate.ipynb: loads a pretrained GPT-2 and generates text from a prompt

Run the unit tests

The README notes that test coverage is not yet extensive. You can still run the existing tests from the repository root.

python -m unittest discover tests

What people use it for

  • Reading a working GPT implementation end to end as a learning exercise
  • Loading GPT-2 weights and generating text with PyTorch, as shown in generate.ipynb
  • Training a small character-level language model on your own text file with projects/chargpt
  • Training a GPT to learn addition with projects/adder
  • Experimenting with a small, hackable transformer base in your own code

Frequently asked questions

Is minGPT still maintained?

The README says minGPT is in a semi-archived state. The author may still accept some changes, and points to nanoGPT for more recent developments.

What are the requirements for minGPT?

The README lists Python and PyTorch as prerequisites. It does not list a requirements.txt file; the README's todo list says one should be added.

Is minGPT free to use?

minGPT is released under the MIT licence, as stated on its repository page.

Does minGPT support distributed training or mixed precision?

No. The README lists mixed precision and distributed training support as todos, so they are not yet available.

Can minGPT load GPT-2 weights?

Yes. The README shows how to create a GPT-2 (124M) model, and the generate.ipynb notebook shows how to load a pretrained GPT-2 and generate text from a prompt. The README also notes that loading weights other than gpt2-* is a todo.

What are the alternatives to minGPT?

The README names nanoGPT as the author's rewrite for newer work. Other comparable projects listed are huggingface/transformers and karpathy/llm.c.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Choose minGPT if you want to read and modify a short, working GPT implementation in PyTorch, or to load GPT-2 weights for simple text generation. Pick nanoGPT for more recent work, huggingface/transformers for a full-featured and maintained ecosystem, or another tool if you need distributed training, mixed precision or a ready-made chatbot.

Install options & downloads