lm-evaluation-harness vs promptfoo
Which should you use? A side-by-side look at what each one does, who it suits, what it costs, how hard it is to set up, and where each falls short.
Open source · MIT
A Python framework that runs standard benchmark tests on language models and reports scores.
Open source · MIT
CLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs.
The short answer
Choose lm-evaluation-harness if you need…
- Researchers who need reproducible scores on standard academic benchmarks
- Engineers comparing several language models on the same tasks
- Teams evaluating a locally hosted or Hugging Face model
- People who want to add their own custom tasks or metrics
Choose promptfoo if you need…
- Developers building apps on top of LLM APIs who want repeatable prompt tests
- Teams comparing several models side by side on the same prompts
- People who want to run automated LLM checks in CI/CD pipelines
- Teams looking for a local, command-line way to scan LLM apps for security issues
Side-by-side
| lm-evaluation-harness | promptfoo | |
|---|---|---|
| Type | Dev tool | Dev tool |
| Price | Free and open source | Free and open source |
| Licence | MIT | MIT |
| GitHub stars | 14.4k | 26.7k |
| Setup difficulty | Intermediate | Intermediate |
| How you get it | Clone and build from source | Install with a package manager |
| Platforms | — | CLI, Web |
| Works with | — | GitHub, OpenAI, Anthropic |
| Main language | Python | TypeScript |
| Last code update | 53 days ago | 4 days ago |
What each one is
lm-evaluation-harness
This is a command-line framework from EleutherAI for testing large language models on a wide set of standard benchmarks. You point it at a model (from Hugging Face, vLLM, or a commercial API) and choose tasks, and it reports how the model scores. Researchers use it to compare models with the same prompts so results are reproducible.
promptfoo
Promptfoo is a command-line tool and library for evaluating LLM-based apps. You write a declarative config listing prompts, models, and test cases, then run it to compare outputs across providers such as OpenAI, Anthropic, and Ollama. It also includes red-teaming features that scan AI apps for security vulnerabilities.
Best for
lm-evaluation-harness
- Researchers who need reproducible scores on standard academic benchmarks
- Engineers comparing several language models on the same tasks
- Teams evaluating a locally hosted or Hugging Face model
- People who want to add their own custom tasks or metrics
promptfoo
- Developers building apps on top of LLM APIs who want repeatable prompt tests
- Teams comparing several models side by side on the same prompts
- People who want to run automated LLM checks in CI/CD pipelines
- Teams looking for a local, command-line way to scan LLM apps for security issues
Strengths
lm-evaluation-harness
- Covers a large number of standard benchmarks (the README says over 60) with many subtasks
- Supports many backends, including Hugging Face transformers, vLLM, SGLang, and commercial APIs
- Permissive MIT license and an active project with a DOI for citation
promptfoo
- The README states that evals run locally and prompts never leave your machine
- Works with many providers and any LLM API, according to the README
- Has a large community with about 26,500 stars and an MIT license
Not ideal for
lm-evaluation-harness
- People who just want to chat with a model; this only runs evaluations
- Anyone without Python experience, since setup and configuration assume it
- Users needing a GUI; the README describes a command-line interface
- Multimodal evaluation at scale, since the README describes multimodal support as a prototype and points to lmms-eval for broader coverage
promptfoo
- People who want a no-code, point-and-click tool, since the workflow centers on the command line and config files
- Projects that do not call any LLM APIs, because there is nothing for it to evaluate
- Anyone who needs a guaranteed security audit, since the README describes scanning as a tool rather than a complete assurance
Trade-offs
lm-evaluation-harness
- The base install omits model backends, so you must add extras such as lm_eval[hf] before running
- Some benchmarks and backends need a GPU and significant setup, and the README's examples assume CUDA
- The README is truncated here, so details on the full set of options and extras are not confirmed
promptfoo
- The README says the project is now part of OpenAI, which may matter to some users evaluating independence
- Requires API keys for most providers, which means usage costs from those providers
- Some claims in the README, such as usage by OpenAI and Anthropic and serving 10M+ users, are not independently verified here
Getting started
lm-evaluation-harness
Clone the repository
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harnessEnter the directory
cd lm-evaluation-harnesspromptfoo
Install globally with npm
npm install -g promptfooCreate a starter example
promptfoo init --example getting-startedFrequently asked questions
What is the difference between lm-evaluation-harness and promptfoo?
lm-evaluation-harness: A Python framework that runs standard benchmark tests on language models and reports scores. promptfoo: CLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs. lm-evaluation-harness suits researchers who need reproducible scores on standard academic benchmarks, while promptfoo suits developers building apps on top of llm apis who want repeatable prompt tests.
Are lm-evaluation-harness and promptfoo free?
lm-evaluation-harness: Free and open source. promptfoo: Free and open source.
Is lm-evaluation-harness or promptfoo more popular?
promptfoo has more GitHub stars (26.7k) than lm-evaluation-harness (14.4k). Popularity isn't the same as fit — check which one matches your use case above.
What are other alternatives to lm-evaluation-harness and promptfoo?
Other options people compare them with include Composio, Prompt flow, Portkey AI Gateway, book-to-skill, Braintrust.
Keep comparing
Built from our catalogue entries for both products: open-source details come from each project's README (AI-assisted, refreshed weekly) and GitHub data; AI-tool details are editorial. Spotted something wrong? Let us know from either product's page.