Skip to content
Get packs

train-llm-from-scratch

A from-scratch PyTorch walkthrough for training a small language model, from raw text to text generation.

GitHub Download source
Stars
13.1k
Forks
1.8k
Licence
MIT
Language
Python
Last update
1 months ago
Type
Learning resource

What it is

This is a tutorial repository that builds a transformer language model in plain PyTorch and walks through the steps to train it: preparing text data, pretraining a base model, generating text, and then post-training steps such as fine-tuning and reinforcement learning. The README says it can run a small model on a laptop CPU, and larger ones on a GPU. It is aimed at people who want to see how each piece works rather than calling a ready-made library.

  • Train a small model (the README mentions a 13 million parameter example) on a laptop CPU or a free Colab or Kaggle T4 GPU
  • Step through the data preparation, tokenization and transformer code in the repository's step-by-step sections
  • Experiment with post-training stages such as SFT, DPO and GRPO using the provided JSON configs
  • Generate text from a trained model using the included generation step

Great for

  • Students who want to learn how a transformer is built and trained step by step
  • Developers with Python and PyTorch experience who want readable reference code
  • Researchers interested in how SFT, reward models, DPO and GRPO are implemented from scratch
  • People with no GPU who want to run a small CPU training example

Not ideal for

  • Anyone who needs a production-ready LLM or chat product, since the README presents this as a learning path
  • People who need to fine-tune an existing pretrained model with standard libraries, because the README says it avoids trl, peft and transformers on purpose
  • Readers without basic Python, neural network or PyTorch knowledge, since the README lists these as prerequisites
Step-by-step guide: how to use train-llm-from-scratchInstall, first run, examples, tips and FAQs
minGPT vs train-llm-from-scratch

How to get it

  1. 1

    Clone and build from source

    You'll need: Python 3.10+ · PyTorch · Git · uv (optional)

    Clone the repository

    git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git

    Enter the folder

    cd train-llm-from-scratch

    Install in editable mode

    pip install -e .
  2. 2

    Run it

    • Clone the repository and install it with pip install -e .
    • On a machine without an NVIDIA GPU, install the CPU build of PyTorch first as the README describes
    • Follow the laptop track in the README to train the small CPU model
    • Check the README's Step 3 for pretraining and Step 4 for generating text
  3. 3

    Get the most out of it

    • Install optional extras such as [train] for data downloads and logging, or [ui] for the Streamlit control panel
    • Use the configs/smoke/ variants to test each stage quickly before running full training
    • Override config fields on the command line, for example --lr 2e-5 --batch_size 16
    • Use the opt-in memory flags --amp, --grad-checkpointing and --grad-accum if a larger config runs out of memory

Downloads

Source code (.zip)main branch · straight from GitHubOfficial website & docsfareedkhan-dev.github.io/train-llm-from-scratch/

Strengths

  • Every algorithm is written out by hand in plain PyTorch, which makes the code easier to read and study
  • The README is organized as a sequence of steps with explanations and expected outputs
  • Provides small smoke-test configs and a CPU path so beginners can try it without a large GPU

Trade-offs

  • The README says it needs a GPU for larger models, and a billion-parameter model will not fit on a T4
  • Written from scratch rather than with standard libraries, so it is not a drop-in tool for production work
  • The README is truncated here, so details of the full setup and later steps are not confirmed in this entry

How it compares

ProjectStarsTypeLevelUpdatedBest for
train-llm-from-scratch (this)13.1kLearning resourceintermediate1 months agoStudents who want to learn how a transformer is built and trained step by step
nanoGPT64.4kLibraryintermediate12 months agoDevelopers who want to read a compact GPT training implementation
minGPT25kLibraryintermediate2.2 years agoStudents who want to read a working GPT implementation end to end
langchain149kFrameworkintermediate4 days agoPython developers building LLM-powered applications
CLIProxyAPI57.3kAppintermediate4 days agoDevelopers who already use OpenAI- or Claude-compatible tools and want to route them to other providers
LibreChat46.6kAppintermediate4 days agoTeams that want one shared chat interface across several AI providers

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.