Guide · Open source
How to use doctrine/lexer in PHP: install and basics
Updated · written with AI from the project's own documentation and checked against our catalogue data
doctrine/lexer is an open-source PHP library that breaks a string of text into a sequence of tokens, which is the first step in reading a structured language. This guide is for PHP developers who want to build a parser for a small custom syntax. After reading it you will know what the library does, how to install it with Composer, and when it fits your project.
Doctrine Lexerdoctrine/lexer · ★ 11.1k · MIT Install options & downloads Quick start
- 1Confirm PHP and Composer. Make sure PHP and Composer are installed on your machine. The library is PHP-only.
- 2Install the package. Run composer require doctrine/lexer in your project directory.
- 3Define your token rules. Decide which token types your small language needs, such as identifiers, numbers and symbols.
- 4Write a recursive descent parser. Use the lexer as the base layer and write parsing functions that consume its tokens top-down.
- 5Check the project page for details. The README is short, so consult the official Doctrine lexer page for usage details.
What doctrine/lexer does
Doctrine Lexer is a base library for a lexer that can be used in top-down, recursive descent parsers. A lexer (also called a tokenizer) scans text and produces tokens, the small labelled pieces a parser then works with.
According to the README, the lexer is used inside Doctrine Annotations and Doctrine ORM's query language (DQL). That means the same code that Doctrine relies on internally is available for your own projects.
Check the prerequisites before you start
The library is written in PHP and is installed with Composer, so you need both PHP and Composer on your machine. The library has no user interface; you use it only from PHP code.
- PHP
- Composer
Install doctrine/lexer with Composer
Run the following command in your project directory to add the library as a dependency.
Composer will download the package and add it to your project's dependencies.
composer require doctrine/lexerBuild your parser on top of the lexer
The README describes the library as a base layer for top-down, recursive descent parsers. You write the rules that turn your language's text into tokens, then write a recursive descent parser that consumes those tokens.
The README is brief and does not show a code example, so the official project page linked from the README is the place to look for usage details.
Know when this library is the right tool
doctrine/lexer covers only the lexing layer. The README does not describe a ready-made parser, so if you need a complete parser you will need to write the parsing logic yourself or use a different tool.
If you need a full parser rather than a lexer, compare it with projects such as nikic/php-parser, hoa/compiler and PHP-Parser.
- Good fit: a small custom language or syntax you parse yourself.
- Good fit: you maintain a PHP project and want the same lexer Doctrine uses.
- Poor fit: you need a ready-made parser or an end-user application.
What people use it for
- Parsing a small custom configuration or query syntax inside a PHP application.
- Reusing the same tokenizer that Doctrine Annotations uses in a PHP project.
- Building a recursive descent parser for a domain-specific language in PHP.
Frequently asked questions
Is doctrine/lexer free to use?
Yes. The project is released under the MIT licence, which allows you to use it in your own projects. Check the licence file in the repository for the full terms.
What do I need to install doctrine/lexer?
You need PHP and Composer. Install the package by running composer require doctrine/lexer in your project directory.
Is doctrine/lexer a parser?
No. The README describes it as a base library for a lexer, which turns text into tokens. You build the parser yourself on top of it.
What is doctrine/lexer used for?
The README says it is used inside Doctrine Annotations and Doctrine ORM's query language (DQL). You can use it in your own PHP projects for building top-down, recursive descent parsers.
What are alternatives to doctrine/lexer?
The comparable projects listed are nikic/php-parser, hoa/compiler and PHP-Parser. These are relevant if you need more than a lexer, such as a full parser.
Can I use doctrine/lexer outside PHP?
No. The library is PHP-only, so it cannot be used directly from other languages.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
doctrine/lexer suits PHP developers who are writing a parser for a small custom language and want a tokenizer that Doctrine itself uses. It is a poor choice if you need a ready-made parser, a non-PHP tool, or an end-user application, since it is only a code dependency with no interface.
Install options & downloads