Skip to content
Get packs

GKE JobSet Interruption Troubleshooting

Lets an agent diagnose JobSet restarts and preemptions on GKE AI/ML training clusters.

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
4 files

What it does

Once installed, the agent follows a structured playbook to find out why a large AI/ML training job on Google Kubernetes Engine (GKE) keeps restarting or getting interrupted. It walks through metrics and logs step by step: restart counts, spot VM preemptions, maintenance events, node readiness failures and faulty host machines, then suggests fixes. It is designed to work largely on its own, without stopping to fix credential or permission problems.

Great for

  • Platform or ML engineers running multi-host AI training jobs on GKE who see repeated JobSet restarts
  • Teams using spot VMs who want to confirm whether preemptions caused a failed training run
  • SRE staff who want a repeatable first-pass investigation of node and host failures

Skip it if

  • General GKE cluster setup or basic app deployment, which the skill explicitly excludes (the agent may skip it or do it poorly)
  • Production clusters where the agent should not act without review: it is told to proceed automatically and to keep diagnosing even when access is denied, so it may produce conclusions from incomplete or mock data
  • Non-JobSet application bugs, which the skill is not designed to diagnose

Try asking

  • “My JobSet 'llm-train' in cluster 'ml-prod' keeps restarting. Find out why over the last hour.”
  • “Check whether spot VM preemptions caused the coordinator pod crashes in my training run.”
  • “Investigate node Ready=False events in my GKE AI cluster and tell me if one faulty host is the cause.”
Step-by-step guide: how to use GKE JobSet Interruption TroubleshootingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the Apache-2.0 licence and an attribution note.

    gke-ai-troubleshooting-jobset-interruption.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-jobset-interruption.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-ai-troubleshooting-jobset-interruption.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent the project ID, cluster name, JobSet name, namespace and issue time up front so it does not have to guess, and check that the time window it chooses matches your incident.
  • Use a project with read-only monitoring and logging permissions, and review any commands the agent runs in the terminal before approving them.
  • Treat the agent's conclusions as a starting point: if it reports 403 errors or missing data, verify the access yourself rather than trusting results built on mock or partial data.

What's inside

  • references/attribution.md
  • references/failure_signatures.md
  • scripts/validate_queries.sh
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.