Guide · Agent skill
How to use the eval-driven-dev agent skill for LLM apps
Updated · written with AI from the project's own documentation and checked against our catalogue data
The eval-driven-dev skill is an agent skill, a folder of instructions and reference files that an AI coding agent loads when a task matches it. For a Python application that calls an LLM, it tells the agent to build an automated test pipeline: run the real app with test inputs, score the outputs with evaluators, and report pass/fail through a pixie test run. This guide explains what the skill does, how to install it, how to trigger it and what it needs before you run it.
Eval-Driven Development for LLM Appsgithub/awesome-copilot · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ for a personal install or into your project's .claude/skills/ folder for a single project.
- 2Or upload in the Claude apps. In the Claude apps, go to Settings, then Capabilities, then Skills, and upload the ZIP.
- 3Confirm requirements. Check that you have Python 3.10 or newer, a real LLM API key for the app and evaluators, and permission to install packages and run a background web server.
- 4Ask for evals in plain language. Describe the Python LLM app and what you want tested, such as regressions in answers or prompt changes.
- 5Let setup.sh run. The agent activates the project's virtual environment and runs setup.sh, which updates the skill and pixie-qa and starts a web server.
- 6Review the pixie test results. Open the local web server to view pass/fail results and scores from the pixie test run.
- 7Act on the proposed fixes. Read the action plan the agent produces and decide which changes to apply to your app.
What the skill makes the agent do
The skill is about doing the work, not describing it. The agent reads your code, edits files, runs commands and aims to finish with a working pixie test run that has real scores.
The app's own code runs for real during evaluation, including routing, prompt assembly, LLM calls and response formatting. Nothing in the app's LLM path is mocked or stubbed. Data the app reads from external sources such as databases, caches, third-party APIs or voice streams is replaced with test-specified values through instrumentations, so each test case controls what the app sees.
- Analyses the project's purpose, users and failure modes before choosing eval criteria.
- Identifies the entry point a real user or client would call, such as a web endpoint or CLI.
- Builds a golden dataset of test inputs and writes evaluators such as LLM-as-judge or similarity checks.
- Runs the tests with pixie test and analyses the outcomes, then proposes concrete fixes.
- Starts a local web server so you can view results.
Install the skill (ZIP or Claude apps)
Download the ZIP from this page and unzip it into one of these locations. Use ~/.claude/skills/ for a personal install available across projects, or a project's .claude/skills/ folder to limit it to one project.
Alternatively, in the Claude apps, open Settings, then Capabilities, then Skills, and upload the ZIP there.
- Personal: unzip into ~/.claude/skills/
- Single project: unzip into your project's .claude/skills/ folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Check the requirements first
The skill targets Python LLM applications. You need Python 3.10 or newer, the pixie-qa Python package within the range >=0.8.4,<0.9.0, and access to a real LLM API. You must supply the API keys for the app and its evaluators yourself.
The skill also runs setup.sh, which updates the skill and pixie-qa and starts a web server. Make sure your environment allows installing packages, running scripts and starting a background server.
- Python 3.10+
- pixie-qa Python package, version >=0.8.4,<0.9.0
- Real LLM API access with your own API keys
- Permission to run setup.sh, which updates packages and starts a web server
Trigger the skill with a natural request
The skill's description says to use it when you ask to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviours, improve quality or do quality assurance for a Python project that calls an LLM. You do not need to name the skill, though naming it is clear.
Good requests say what you want measured and point at the app. Mention any spec file, dataset or quality dimension you already have, because the skill treats named items as requirements.
- "Add evals to this Python chatbot so I can catch regressions in its answers."
- "Set up QA for my FastAPI app that calls an LLM and show me the pass/fail results."
- "Use the questions in eval_inputs/support_questions.json as the dataset and check factuality and tone."
- "Benchmark how my prompt changes affect the context assembly in this app."
What you get at the end
A successful run leaves you with an evaluation pipeline, a dataset, evaluators, a pixie test run with scores, and a short action plan for improvements. The web server started by setup.sh lets you inspect results in a browser.
If setup fails, the skill tells the agent which failures are safe to skip. For example, a failed skill update can be skipped, but a failed pixie-qa install or a failed web server start is a stop condition and the agent should ask you for help.
- A dataset built from realistic inputs
- Evaluators that score outputs, including LLM-as-judge or similarity checks
- pixie test results with pass/fail and scores
- A prioritised list of fixes
Safety notes: scripts and network access
This skill runs scripts and uses the network. The setup step installs and upgrades packages, initialises a pixie working directory and starts a background web server. The evaluators call a real LLM, so your API keys and usage will be involved.
Review setup.sh and the skill files before you run them, and run the workflow in a project where these actions are acceptable. The skill also forbids mocking the LLM in evals, so do not expect the eval to run offline with fake responses.
- setup.sh updates the eval-driven-dev skill and pixie-qa, initialises pixie and starts a web server.
- Evaluators make real LLM calls, which use your API keys.
- Keep API keys in your own environment; the skill expects you to provide them.
Tips for better results
Point the agent at the entry point that matters most. The skill asks you to choose when several entry points are equally plausible. Provide any environment variables the app needs up front, because the skill stops and asks you rather than faking missing configuration.
Use the named dataset or spec files you already have. If you only want unit tests with mocked LLM responses, this skill is the wrong tool.
- Name the entry point, such as a specific endpoint or CLI command.
- Set required environment variables before starting, so the app can run.
- Give a real dataset file if you have one.
- Read the references folder (for example evaluators.md and the runnable examples) if you want to see how evaluators are chosen.
What people use it for
- Catching regressions in a customer support chatbot after a prompt change.
- Testing context assembly, such as how an app gathers data and builds prompts, without mocking the LLM.
- Building a golden dataset of realistic questions from a spec or data file you already have.
- Adding QA to an existing FastAPI, Flask-style web server or CLI app that calls an LLM.
Frequently asked questions
Is the eval-driven-dev skill free?
The skill is published under the MIT licence on github/awesome-copilot. The source does not state pricing for the pixie-qa package or for the LLM APIs you will call, so check those separately.
What do I need to install eval-driven-dev?
Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it in the Claude apps under Settings, then Capabilities, then Skills. You also need Python 3.10 or newer and access to a real LLM API.
Does the skill work with languages other than Python?
No. The skill targets Python LLM applications. For projects that are not Python or do not call an LLM, it is not the right fit.
Can I use this skill with mocked LLM responses?
No. The skill explicitly forbids mocking the LLM in evals because the eval would then control both inputs and outputs. If you only want unit tests with mocked responses, use your project's normal test suite instead.
Why does the skill start a web server?
The setup step starts a local web server in the background so you can view eval results and progress. If the server fails to start, the skill tells the agent to stop and ask for help.
How does this compare with writing my own pytest tests?
pytest tests that mock the LLM check code paths in isolation. This skill runs the real app with real LLM calls and scores outputs with evaluators such as LLM-as-judge or similarity checks, using the pixie-qa package and pixie test.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use eval-driven-dev if you maintain a Python app that calls an LLM and want regression tests and scored evals that run the real code path, and you can allow package installs, scripts and a local web server. Pick something else if your project is not Python, does not call an LLM, or you only want unit tests with mocked model responses.
Download the skill