Guide · Agent skill
How to use the training-check agent skill for WandB runs
Updated · written with AI from the project's own documentation and checked against our catalogue data
training-check is an agent skill that makes an AI agent check a machine-learning training run on a schedule, looking for warning signs such as NaN values, loss that keeps rising, or evaluation scores well below baseline. It is aimed at ML researchers and engineers who log training to Weights and Biases (WandB) and want to avoid spending GPU hours on broken runs. After this guide you will know what the skill does, how to install it, how to ask the agent to use it, and what it needs and where its risks are.
Training Health Checkwanshuiyin/Auto-claude-code-research-in-sleep · MIT Download the skill Quick start
- 1Download the ZIP. Download the training-check ZIP from this page. It contains SKILL.md.
- 2Install the skill. Unzip it into ~/.claude/skills/ for all projects, or into .claude/skills/ in one project. In the Claude apps, upload it under Settings, then Capabilities, then Skills.
- 3Prepare WandB access. Make sure you have a WandB account with API access, the wandb Python package installed, and your run path in entity/project/run_id format.
- 4Confirm training is running. Start the check only after the session is alive and loss has begun to decrease for the first few steps.
- 5Ask for a check or schedule it. Name the run in your request, or ask the agent to set up recurring checks with CronCreate so it runs about every 10 minutes.
What the training-check skill makes the agent do
An agent skill is a folder containing a SKILL.md file with instructions that an AI agent (such as Claude Code) reads and follows when you ask for that task. This skill tells the agent to periodically read metrics from a training run, judge whether the run looks healthy, and act on the result.
The skill's main data source is WandB. If WandB cannot be reached, it falls back to reading the training log file over SSH. It checks loss trend, eval metrics, NaN or Inf values, sudden loss spikes, the learning rate schedule and gradient norm.
- Clearly bad signals (NaN or Inf in loss, loss diverging, eval metrics significantly worse than baseline): the agent stops training and investigates.
- Clearly fine signals: the agent continues and lengthens the interval between checks.
- Ambiguous signals (flat loss, noisy metrics, slightly worse but still early): the agent asks a second AI reviewer, Codex, for a one-off opinion, which replies STOP, CONTINUE or WAIT.
Install the training-check skill
Download the training-check ZIP from this page. The skill is a single file, SKILL.md, inside the skill folder.
You can install it in one of two ways. For Claude Code, unzip it into your user skills folder at ~/.claude/skills/, or into a specific project's .claude/skills/ folder if you want it available only in that project. In the Claude apps, upload the ZIP under Settings, then Capabilities, then Skills.
- User-wide: ~/.claude/skills/
- Single project: .claude/skills/ inside the project folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
~/.claude/skills/Set up WandB access and the run path
The skill needs a WandB account with API access and the path of your run in entity/project/run_id format. The skill reads WANDB_ENTITY and WANDB_PROJECT from CLAUDE.md or from the argument you pass, so you can put them in your project notes once instead of typing them each time.
The skill reads the run using the Python wandb package, so that package must be installed in the environment where the agent runs. If you rely on the log-file fallback, the agent also needs SSH access to the training server.
- WandB account and API access
- Run path in entity/project/run_id format
- Python with the wandb package installed
- SSH access to the training server, for the log-file fallback only
import wandb
api = wandb.Api()
run = api.run("<entity>/<project>/<run_id>")
history = run.history()Trigger the skill with a natural request
Once the skill is installed, you can ask for it in plain language and include the run path. The skill is designed to start after training is confirmed running, so ask for checks once the run is underway, not before it starts.
To have checks run automatically, the skill's instructions say to set up a recurring Claude Code CronCreate job. The skill starts at a 10-minute interval and lengthens it to 20, 30 and then 60 minutes while the run stays healthy, resetting to 10 minutes after any anomaly.
- Example request: Check the health of wandb run myteam/llm-pretrain/abc123 and tell me if it looks broken.
- Example request: Set up training health checks for run myteam/vision/run42 every 10 minutes while it trains.
- Example request: The loss looks flat on myteam/llm-pretrain/abc123. Do a training check and tell me whether to keep going.
CronCreate (recurring, every 10 minutes initially):
"Run /training-check for wandb run <entity>/<project>/<run_id>"Understand what happens when a run looks broken
When the agent decides to stop, it kills the training session, then saves the WandB run URL, key metrics and the reason for stopping so you can debug later. The skill's rules say not to stop training on the first sign of noise and to look at trends across multiple checkpoints instead.
If both WandB and the log files are unreachable, the agent reports the connectivity problem and tries again at the next interval. The skill is explicit that it should not assume training is broken in that case.
- Stop: kill the training session, save the run URL and key metrics, and log the reason to project notes.
- Continue: do nothing and check again at the next interval.
- Wait: keep the current short interval without lengthening it.
- Connectivity failure: report it and retry at the next interval.
Know the safety limits before you rely on it
This skill runs scripts and uses the network. Its allowed tools include a broad shell permission, and it can stop a training session on its own judgment. The metric thresholds are heuristics, so a false positive could end a run that was actually valid.
It also does not check process health. Crashes and idle GPUs are the job of watchdog.py, which the skill's documentation says checks every 60 seconds. Run both if you want full coverage.
- Review the reviewer and shell permissions in SKILL.md before enabling it on a machine with important runs.
- Keep watchdog.py running for crashes and idle GPUs.
- Expect that some stops may need manual review of the saved evidence.
- The optional reviewer servers (antigravity, grok, claude-review) are used only depending on the configured reviewer backend.
What people use it for
- Catching a run whose loss turns to NaN early, so the GPU hours are not wasted for the rest of the job.
- Spotting a loss that keeps rising across several checkpoints and stopping the session with the run link saved as evidence.
- Getting a second opinion from a reviewer on a plateaued loss curve before deciding whether to let training continue.
- Running periodic checks overnight on a long training job while watchdog.py watches for crashes and idle GPUs.
Frequently asked questions
Is the training-check skill free to use?
The skill is published under the MIT licence. You still need your own WandB account and access to any services it uses, such as Codex for ambiguous-case judgments, which the source does not price.
What do I need to run training-check?
You need a WandB account with API access, the run path in entity/project/run_id format, and Python with the wandb package. For the fallback you also need SSH access to the training server, and for ambiguous cases a Codex MCP server with access to the gpt-6-astra model.
Can training-check work without WandB?
Only partly. It can fall back to reading the training log over SSH, but the skill has no other data source, so without WandB or SSH access it can only report connectivity issues.
Where do I install the training-check skill?
Unzip it into ~/.claude/skills/ for all projects or into a project's .claude/skills/ folder. In the Claude apps you can upload the ZIP under Settings, then Capabilities, then Skills.
Does training-check stop training automatically?
It can. If the agent judges a run clearly broken, it kills the training session and saves the run URL and metrics. Because the thresholds are heuristic, you should review the saved evidence, since a false positive could end a valid run.
How is training-check different from watchdog.py?
training-check checks training quality, such as loss trend and metric degradation, every 10 to 60 minutes. watchdog.py checks process health, such as whether the session is alive and the GPU is active, every 60 seconds. The skill's documentation recommends using both together.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
training-check suits ML researchers and engineers who already log training to WandB and want periodic, automated checks for NaN values, divergence and metric degradation. It is a poor fit if you have no WandB account or SSH access to your logs, or if you want training stopped with no human review of the evidence.
Download the skill