Skip to content
Get packs

promptfoo

CLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs.

GitHub Download 0.125.1
Stars
26.7k
Forks
2.6k
Licence
MIT
Language
TypeScript
Last update
4 days ago
Type
Dev tool

What it is

Promptfoo is a command-line tool and library for evaluating LLM-based apps. You write a declarative config listing prompts, models, and test cases, then run it to compare outputs across providers such as OpenAI, Anthropic, and Ollama. It also includes red-teaming features that scan AI apps for security vulnerabilities.

  • Run the same prompt against GPT and Claude and view the results in a web viewer
  • Add prompt evaluations to a CI/CD pipeline so changes are checked automatically
  • Scan an LLM app for vulnerabilities with its red-teaming features and generate a report
  • Review pull requests for LLM-related security and compliance issues using its code scanning feature

Great for

  • Developers building apps on top of LLM APIs who want repeatable prompt tests
  • Teams comparing several models side by side on the same prompts
  • People who want to run automated LLM checks in CI/CD pipelines
  • Teams looking for a local, command-line way to scan LLM apps for security issues

Not ideal for

  • People who want a no-code, point-and-click tool, since the workflow centers on the command line and config files
  • Projects that do not call any LLM APIs, because there is nothing for it to evaluate
  • Anyone who needs a guaranteed security audit, since the README describes scanning as a tool rather than a complete assurance
Step-by-step guide: how to use promptfooInstall, first run, examples, tips and FAQs
promptfoo alternativespromptfoo vs PromptLayerpromptfoo vs PromptHubLangfuse vs promptfooLangSmith vs promptfooBraintrust vs promptfooArize Phoenix vs promptfoopromptfoo vs W&B WeaveHelicone vs promptfoo

How to get it

  1. 1

    Install with a package manager

    You'll need: Node.js (npm is used for the install command) · An API key for the LLM provider you want to test

    Install globally with npm

    npm install -g promptfoo

    Create a starter example

    promptfoo init --example getting-started
  2. 2

    Run it

    • Set your provider API key, for example export OPENAI_API_KEY=sk-abc123
    • Run cd getting-started
    • Run promptfoo eval to execute the example
    • Run promptfoo view to see the results in a web viewer
  3. 3

    Get the most out of it

    • Use the red-teaming guide to scan your LLM app for vulnerabilities
    • Connect evals to CI/CD so prompt changes are tested automatically
    • Compare multiple providers in one config to see differences side by side
    • Use the Node.js package if you want to call promptfoo from code instead of the CLI
    • Use the code scanning feature to review pull requests for LLM-related issues

Downloads

Source code (.zip)main branch · straight from GitHubRelease 0.125.1Release notes and sourceOfficial website & docspromptfoo.dev

Strengths

  • The README states that evals run locally and prompts never leave your machine
  • Works with many providers and any LLM API, according to the README
  • Has a large community with about 26,500 stars and an MIT license

Trade-offs

  • The README says the project is now part of OpenAI, which may matter to some users evaluating independence
  • Requires API keys for most providers, which means usage costs from those providers
  • Some claims in the README, such as usage by OpenAI and Anthropic and serving 10M+ users, are not independently verified here

How it compares

ProjectStarsTypeLevelUpdatedBest for
promptfoo (this)26.7kDev toolintermediate4 days agoDevelopers building apps on top of LLM APIs who want repeatable prompt tests
Prompt-Engineering-Guide79.8kLearning resourcebeginner8 months agoPeople who are new to prompting and want structured lessons with examples
awesome-nanobanana-pro10.4kCurated listbeginner4 days agoPeople who want ready-made prompts to copy and adapt for image generation
composio30.9kDev toolintermediate4 days agoDevelopers building AI agents that need to read or act in third-party apps
lm-evaluation-harness14.4kDev toolintermediate2 months agoResearchers who need reproducible scores on standard academic benchmarks
claude-context12.7kDev toolbeginner4 months agoDevelopers working in large codebases where loading everything into an agent's context is impractical

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.