minbpe
A small, readable Python implementation of Byte Pair Encoding, the text-splitting method used in LLM tokenizer
- Stars
- 11k
- Forks
- 1.1k
- Licence
- MIT
- Language
- Python
- Last update
- 2.4 years ago
- Type
- Learning resource
No commits in over a year — check for forks or alternatives before adopting.
What it is
minbpe is a short Python codebase that implements Byte Pair Encoding (BPE), the algorithm that many large language models use to break text into tokens. It lets you train a tokenizer on your own text, encode text into token IDs, and decode those IDs back into text. The code is kept deliberately minimal and commented so people can read how the algorithm works.
- Train a toy BasicTokenizer on a short string and inspect the merges it learns
- Reproduce GPT-4 tokenization output and compare it with tiktoken's cl100k_base encoding
- Train a RegexTokenizer with a chosen vocabulary size on your own dataset and save it to disk
- Register special tokens such as <|endoftext|> and encode text with them
- Follow the author's step-by-step exercise to build your own version
Great for
- Students and engineers who want to understand how LLM tokenizers work by reading code
- Developers who want to train a small custom tokenizer on their own text
- Anyone following the author's BPE lecture or exercise
- Researchers who want a hackable baseline before using heavier tooling
Not ideal for
- Production systems that need fast tokenization on large corpora, since the README lists a faster optimized version only as a future todo
- People who just need to tokenize text for an existing model, where a library such as tiktoken is the more direct choice
- Users who need a polished packaged tool with documented releases, since the repo has no listed releases
How to get it
- 1
Clone and build from source
You'll need: Python · pip
Clone the repository
git clone https://github.com/karpathy/minbpe - 2
Run it
- Clone the repository and change into its directory
- Run the toy example from the quick start in a Python session using from minbpe import BasicTokenizer
- Call tokenizer.train with a short string and a vocab size of 256 + 3
- Check that encode returns the expected token list and that decode returns the original text
- 3
Get the most out of it
- Read the commented source of basic.py, regex.py, and gpt4.py in that order to see how each layer builds on the last
- Install tiktoken and compare outputs with GPT4Tokenizer as shown in the README
- Always pass allowed_special explicitly when encoding user input, as the README warns about special token handling
- Run the test suite with pytest -v . to see more usage examples
- Work through exercise.md to rebuild the implementation yourself
Downloads
Strengths
- Very short, clean code with comments, designed to be read and modified
- Includes comparisons against tiktoken to verify behavior
- Covers the basic, regex-split, and GPT-4-compatible variants of the algorithm
Trade-offs
- Not optimized for speed; the README lists a more optimized version as an unfinished todo
- Documentation is brief and assumes some familiarity with tokenization concepts
- Last push was 2024-07-01, so recent maintenance activity is unclear
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| minbpe (this) | 11k | Learning resource | intermediate | 2.4 years ago | Students and engineers who want to understand how LLM tokenizers work by reading code |
| tiktoken | 19.6k | Library | beginner | 3 months ago | Python developers who need to count tokens before sending text to OpenAI models |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.