Guide · Agent skill
How to use the Deflake a Flaky Test agent skill (deflake)
Updated · written with AI from the project's own documentation and checked against our catalogue data
The deflake skill is a set of instructions that tells an AI coding agent how to stabilize one flaky test: a test that passed and failed on reruns of the same commit. It is written for maintainers of the qwen-code repository and developers using Vitest who want a small, targeted fix without changing what the test checks. After this guide you will know what the skill does, how to install it, how to trigger it, and what it needs to run.
Deflake a Flaky TestQwenLM/qwen-code · Apache-2.0 Download the skill Quick start
- 1Download and unzip. Download the deflake ZIP from this page and unzip it so the deflake folder contains SKILL.md.
- 2Place or upload the skill. Put the folder in ~/.claude/skills/ for personal use or in .claude/skills/ inside your project, or upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Check the prerequisites. Confirm Node.js, npm and Vitest are available and that your qwen-code checkout has build, typecheck and lint scripts.
- 4Prepare the issue. Make sure the issue has a deflake: title and names the test file and test name, plus the observed failure.
- 5Ask the agent. Request the fix by naming the issue or test, for example asking the agent to use the deflake skill on that issue.
- 6Review the diff and evidence. Check that the change is a single minimal fix and that the repeated test runs passed, and read any failure.md the agent wrote.
What the deflake skill makes the agent do
The skill starts from a GitHub-style issue whose title begins with deflake: and which names one test (file and test name) plus the observed failure, such as a timeout or an order-dependent assertion. The agent reads the test, works out the source of the nondeterminism, and applies the smallest fix from a fixed set of four.
The four allowed fixes are: raise a timeout or poll budget; replace fixed waits with an explicit wait for the real condition; make randomness or time deterministic; and isolate or serialize tests that collide on a shared resource.
- Raise a timeout or poll budget (for example the testTimeout third argument to it)
- Wait for the real condition (for example vi.waitFor) instead of a bare setTimeout or sleep
- Seed randomness or use vi.useFakeTimers() / mock Date.now
- Give colliding tests unique tempdirs, ports or resources, or run them serially
What the agent is not allowed to do
The skill forbids any change that hides the flake. The agent must not delete the test, skip or todo it, loosen an assertion, widen an expected range, add a blanket try/catch, or wrap the assertion in a retry.
If none of the four fixes fits, or the failure looks like a real intermittent product bug, the agent writes a failure.md file explaining what it found and stops so a human can take over.
Install the deflake skill
Download the ZIP from this page and unzip it. For a personal install, place the skill folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.
You can also upload the skill in the Claude apps under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/deflake/
- Project: .claude/skills/deflake/ inside your repository
Trigger the skill with a natural request
Agent skills are usually picked up when your request matches the skill's description. The skill's description is: stabilize a flaky test with a minimal, assertion-preserving fix, never by weakening or deleting the check. Name the test and the failure so the agent can find it.
Because the skill expects a deflake: issue, point the agent at that issue or paste its contents into the conversation.
- Deflake the flaky test in packages/core/src/example.test.ts named 'retries on timeout', which failed with 'Test timed out in 5000ms' on a rerun of the same commit.
- Use the deflake skill on this issue: deflake: flaky timeout in the session tests.
What you need before you start
The skill assumes a specific toolchain. It runs npx vitest run on the named file, so Node.js and npm with Vitest must be available. It also expects a qwen-code repository checkout that has build, typecheck and lint scripts, and it refers to the prepare-pr skill at .qwen/skills/prepare-pr/SKILL.md for the pull request body.
Projects that do not use Vitest or the qwen-code prepare-pr workflow are outside the skill's scope, because the instructions assume those tools and paths exist.
- Node.js and npm, with Vitest
- A qwen-code checkout with build, typecheck and lint scripts
- A GitHub-style issue with a deflake: title
- The prepare-pr skill it refers to
npx vitest run <file>Safety notes: scripts and network
The skill runs scripts, because it executes Vitest and the repository's build, typecheck and lint commands. It does not use the network. Review the diff the agent produces before committing, and run the tests yourself on a branch first.
The skill's own verification step requires the test file to pass repeatedly, so a single lucky pass is not accepted as evidence.
- Runs scripts: yes (tests, build, typecheck, lint)
- Uses network: no
Tips for good results
Give the agent one test per run. The skill is built around a single named test and asks for a minimal, local diff without refactoring unrelated code.
Include the observed failure signature in the issue. The skill uses it to choose the right fix, for example a timeout message points toward a timeout bump, while an order-dependent assertion points toward isolation.
- Keep the change scoped to one test and its file's helpers
- Prefer a per-test or per-file change over a global config change
- Check that the reported failure is intermittent; a consistent failure is a bug, not a flake
What people use it for
- Stabilizing a test that times out only under CI load by raising its per-test timeout
- Replacing a fixed sleep with an explicit wait for a real condition
- Seeding randomness or using fake timers for a test whose value drifts with the clock
- Giving two tests that share a temporary directory or port their own resources
Frequently asked questions
How do I install the deflake agent skill?
Download the ZIP from this page and unzip it. Place the deflake folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder. You can also upload it in the Claude apps under Settings → Capabilities → Skills.
What do I need to run the deflake skill?
You need Node.js and npm with Vitest, a qwen-code repository checkout with build, typecheck and lint scripts, and a GitHub-style issue with a deflake: title. The skill also refers to the prepare-pr skill for the pull request body.
Is the deflake skill free to use?
The source material states the licence is Apache-2.0. It does not list any price, so check the repository for its terms.
Does the deflake skill delete or skip flaky tests?
No. The skill explicitly forbids deleting, skipping, todo-ing or weakening assertions, adding blanket try/catch blocks, or wrapping assertions in retries.
What should the agent do if the failure is a real bug?
It should write a failure.md file in the working directory explaining what it found and stop. A human then fixes the underlying problem.
Can I use deflake with a test framework other than Vitest?
The skill is not meant for that. Its instructions assume Vitest and the qwen-code prepare-pr workflow, so projects that use other tools fall outside what it was written for.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
The deflake skill suits maintainers of the qwen-code repository and developers who want an agent to make one intermittently failing Vitest test deterministic without changing its checks. It is a poor fit for consistently failing tests, which point to real bugs, or for projects that do not use Vitest and the prepare-pr workflow.
Download the skill