Skip to content
Get packs

llm.c

A small C/CUDA codebase for training GPT-2-style language models without PyTorch.

GitHub Download source
Stars
31.4k
Forks
3.8k
Licence
MIT
Language
Cuda
Last update
1.4 years ago
Type
Learning resource

No commits in over a year — check for forks or alternatives before adopting.

What it is

llm.c is a project by Andrej Karpathy that trains GPT-2 style language models using plain C and CUDA instead of a large framework like PyTorch. Its stated focus is pretraining, including reproducing the GPT-2 and GPT-3 model series. The code is meant to be readable, and the README says a CPU reference version fits in about 1,000 lines.

  • Run the fp32 CUDA training path on the tinyshakespeare dataset
  • Train a small GPT-2 (124M) finetune on CPU as a demo, as described in the README
  • Run the unit tests that compare the C code against the PyTorch reference
  • Follow the layernorm tutorial in doc/layernorm to learn how one layer is implemented

Great for

  • Developers who want to see how a GPT-2 style model is trained at the lowest level
  • Students learning how neural network layers are implemented in C or CUDA
  • People with an NVIDIA GPU who want to reproduce GPT-2 (124M) training
  • Readers who want a small reference implementation to compare against PyTorch

Not ideal for

  • People who want a ready-to-use chatbot or inference app, since the README focuses on training
  • Users without a GPU who want practical training speed, because the README says CPU training will not go far
  • Teams that need Python-based ecosystem tooling, since the README says the repo only maintains C and CUDA code
Step-by-step guide: how to use llm.cInstall, first run, examples, tips and FAQs
llm.c vs nanochatllama.cpp vs llm.cllm.c vs minGPT

How to get it

  1. 1

    Clone and build from source

    You'll need: Python (for dev/data scripts and train_gpt2.py) · C compiler and make · NVIDIA GPU with CUDA for GPU training (optional for CPU demo)

    Clone the repository

    git clone https://github.com/karpathy/llm.c

    Download the starter pack (fp32 path)

    chmod u+x ./dev/download_starter_pack.sh && ./dev/download_starter_pack.sh

    Build and run the 1 GPU fp32 code

    make train_gpt2fp32cu && ./train_gpt2fp32cu
  2. 2

    Run it

    • Clone the repo and enter the directory
    • Run ./dev/download_starter_pack.sh to fetch the .bin files
    • Build with make train_gpt2fp32cu and run ./train_gpt2fp32cu
    • For CPU, build with make train_gpt2 and run OMP_NUM_THREADS=8 ./train_gpt2
  3. 3

    Get the most out of it

    • Use make test_gpt2 and ./test_gpt2 to check the C code against the PyTorch reference
    • Enable cuDNN Flash Attention with make train_gpt2cu USE_CUDNN=1, after installing cuDNN
    • Change -O3 to -g in the Makefile to step through the code in an IDE
    • Read doc/layernorm/layernorm.md for a guided walk through one layer

Downloads

Source code (.zip)master branch · straight from GitHub

Strengths

  • Code is small and direct, making it useful for studying training internals
  • The README reports llm.c running a bit faster than PyTorch Nightly (about 7%) in its own benchmark
  • Includes a PyTorch reference implementation and unit tests to check correctness

Trade-offs

  • Requires NVIDIA CUDA hardware for practical training; CPU use is described as a demo
  • Setup can be involved: multi-GPU and cuDNN paths need MPI, NCCL, and cuDNN installed
  • The README is truncated and the repo has no releases or topics, so some details are unclear

How it compares

ProjectStarsTypeLevelUpdatedBest for
llm.c (this)31.4kLearning resourceadvanced1.4 years agoDevelopers who want to see how a GPT-2 style model is trained at the lowest level
nanoGPT64.4kLibraryintermediate12 months agoDevelopers who want to read a compact GPT training implementation
llama.cpp133kLibraryintermediate4 days agoDevelopers who want to run open-weight models locally without a cloud API
minGPT25kLibraryintermediate2.2 years agoStudents who want to read a working GPT implementation end to end

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.