The best promptfoo alternatives
13 tools that do a similar job to promptfoo, 8 of them open source. Each entry says what it does and who it suits, so you can pick the one that fits instead of the most famous.
You're replacing
promptfooCLI and library for testing LLM prompts and red-teaming AI apps, with declarative configs.
- 1
Composio
Open source · MIT · ★ 30.9k
SDK for building AI agents that can take actions in 1000+ apps via pre-authenticated tools.
Best for: Developers building AI agents that need to read or act in third-party apps; Teams already using a supported agent framework such as OpenAI Agents, LangChain, or Vercel AI SDK
- 2
Prompt flow
Open source · MIT · ★ 11.3k
Open-source toolkit for building, testing, evaluating and deploying LLM apps, from prototype to production.
Best for: Teams with existing Prompt flow applications that need to keep running until migration; Developers who want to trace and evaluate LLM app behavior on datasets
- 3
Portkey AI Gateway
Open source · MIT · ★ 13.3k
Open-source gateway that routes requests to many AI models through one API, with guardrails.
Best for: Developers who call several LLM providers and want one OpenAI-style interface; Teams that want automatic retries or fallbacks when a model provider fails
- 4
lm-evaluation-harness
Open source · MIT · ★ 14.4k
A Python framework that runs standard benchmark tests on language models and reports scores.
Best for: Researchers who need reproducible scores on standard academic benchmarks; Engineers comparing several language models on the same tasks
- 5
book-to-skill
Open source · MIT · ★ 36.7k
Turns a technical book or document folder into a structured skill your AI coding agent can query.
Best for: Developers who want to query a technical book from inside their coding agent; People who re-read reference material often and want it searchable by an agent
- 6
Braintrust
AI tool · Freemium
An evaluation and observability platform for scoring prompts and models against datasets and monitoring production.
Best for: Systematic evals before shipping; Comparing prompt versions with scores
- 7
Helicone
AI tool · Freemium
An open-source LLM observability proxy that logs requests, costs and latency with a one-line integration.
Best for: Quick cost and usage visibility; Caching and rate limiting LLM calls
- 8
W&B Weave
AI tool · Freemium
Weights & Biases' toolkit for tracing, evaluating and comparing LLM application versions.
Best for: Teams already on Weights & Biases; Evaluation leaderboards across versions
- 9
Langfuse
AI tool · Open source
An open-source LLM engineering platform for tracing, prompt management, evaluations and cost tracking.
Best for: Tracing LLM apps in production; Self-hosted prompt management
- 10
Arize Phoenix
AI tool · Open source
An open-source tool for tracing, evaluating and experimenting with LLM and agent applications, built on OpenTelemetry.
Best for: Local debugging of LLM traces; OpenTelemetry-based stacks
- 11
LangSmith
AI tool · Freemium
LangChain's platform for tracing, evaluating and monitoring LLM apps, with a prompt hub and playground.
Best for: LangChain and LangGraph users; Dataset-based evaluation
- 12
Opik
AI tool · Open source
Comet's open-source platform for logging, evaluating and monitoring LLM apps, including prompt optimisation.
Best for: Open-source tracing and evals; Automated prompt optimisation
- 13
Confident AI (DeepEval)
AI tool · Freemium
The cloud platform behind DeepEval, an open-source framework for unit-testing LLM outputs with ready-made metrics.
Best for: Pytest-style LLM tests; RAG evaluation metrics
How we pick: tools in the same category that solve the same problem, projects that name each other as alternatives, and open-source projects that describe themselves as an alternative to promptfoo. We don't take payment for placement.