tiktoken
A fast library that splits text into the tokens OpenAI's models read.
- Stars
- 19.6k
- Forks
- 1.7k
- Licence
- MIT
- Language
- Python
- Last update
- 3 months ago
- Type
- Library
What it is
tiktoken is a Python library that converts text into tokens, the numbers language models work with, using byte pair encoding (BPE). It can also turn tokens back into the original text. Developers use it to count tokens or prepare text for OpenAI models.
- Count the tokens in a prompt to estimate cost or check it fits a model's context limit
- Encode text with get_encoding("o200k_base") and decode it back to confirm a lossless round trip
- Look up the tokenizer for a named OpenAI model with encoding_for_model
- Train a small BPE tokenizer or visualize encoding with the educational module
- Register custom encodings through the tiktoken_ext plugin mechanism
Great for
- Python developers who need to count tokens before sending text to OpenAI models
- Developers who want the same tokenizer that matches a specific OpenAI model, such as gpt-4o
- People who want to learn how BPE tokenization works, using the included educational submodule
- Developers building custom encodings who want to extend tiktoken
Not ideal for
- People who need a tokenizer for non-OpenAI models, since the README only describes OpenAI's models
- Non-Python projects, because the README only shows Python installation and usage
- Non-developers looking for a ready-made app, since this is a code library with no interface described
How to get it
- 1
Install with a package manager
You'll need: Python
Install from PyPI
pip install tiktoken - 2
Run it
- Install the package with pip install tiktoken
- Run import tiktoken in a Python session
- Call tiktoken.get_encoding("o200k_base") to load an encoding
- Call enc.encode("hello world") to get tokens and enc.decode(...) to get text back
- 3
Get the most out of it
- Use encoding_for_model("gpt-4o") to match the tokenizer to a specific OpenAI model
- Read tiktoken/core.py for the full tokenizer API
- Use the tiktoken._educational module to see how BPE merges work step by step
- Check the OpenAI Cookbook example on counting tokens with tiktoken
- Follow the README's tiktoken_ext layout if you need get_encoding to find your own encoding
Downloads
Strengths
- The README reports it is 3-6x faster than a comparable open source tokenizer, measured on 1GB of GPT-2 text
- Installs from PyPI with a single pip command
- Released under the MIT license with a large community following
Trade-offs
- The README covers only OpenAI-oriented encodings, so support for other model families is not described
- Performance claims come from a single benchmark setup listed in the README, with older library versions
- Extending it with custom encodings requires care, such as avoiding editable installs for plugin packages
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.