Training Health Check
Periodically checks WandB training metrics to catch NaN, divergence, or stalled runs early
- Stars
- 18,482
- Licence
- MIT
- Files
- 1 file
What it does
When installed, the agent can monitor a machine-learning training run on a schedule. It reads metrics from Weights & Biases (WandB), a popular experiment-tracking service, or falls back to reading the training log over SSH. It looks for warning signs such as NaN values, loss that keeps rising, or eval scores that fall well below baseline. If a run looks clearly broken it can stop the training session and save the run link and metrics as evidence. Unclear cases can be sent to a second AI reviewer (Codex) for a one-off opinion. Process-level checks such as crashes or idle GPUs are handled by a separate tool, watchdog.py.
Great for
- ML researchers running long GPU training jobs who want to avoid wasting compute on broken runs
- Engineers who already log training to WandB and want automated periodic health checks
Skip it if
- Users without a WandB account or SSH access to their training log, since the skill has no other data source (the process would only report connectivity issues)
- Anyone who wants the agent to stop training automatically without review: the skill can kill a session on its own judgment, and metric thresholds are heuristic, so a false positive could end a valid run
Try asking
- “Check the WandB run myteam/llm-pretrain/abc123 for NaN loss or divergence and tell me if it should be stopped”
- “Set up periodic health checks on my current training run and back off the interval while it stays healthy”
- “Read the last 100 lines of the training log on my server and summarise whether the loss is still decreasing”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
training-check.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip training-check.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip training-check.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the full WandB run path as the argument so the agent does not have to guess the entity and project
- Keep the default cautious rules: judge trends over several checkpoints rather than single spikes, and check that the reviewer model name is valid for your account before relying on it
- Pair it with watchdog.py so crashes and idle GPUs are caught immediately while this skill watches training quality
What's inside
- SKILL.md
Redistributed unmodified from wanshuiyin/Auto-claude-code-research-in-sleep under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.