Guide · Agent skill
How to use the GKE JobSet Interruption Troubleshooting agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
This guide explains the GKE JobSet Interruption Troubleshooting skill, an agent skill that guides an AI agent through diagnosing why a large AI/ML training job on Google Kubernetes Engine keeps restarting or being interrupted. It is written for platform engineers, ML engineers and SRE staff who may never have used agent skills before. By the end you will know how to install the skill, how to ask for it in plain language, and what it will check and what it needs to run.
GKE JobSet Interruption Troubleshootinggoogle/skills · Apache-2.0 Download the skill Quick start
- 1Download and unzip the skill. Download the ZIP from this page and unzip it so the folder is named gke-ai-troubleshooting-jobset-interruption and contains SKILL.md.
- 2Place it where the agent looks. Move the folder into ~/.claude/skills/ for personal use or into your project's .claude/skills/ folder. Alternatively, upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
- 3Confirm monitoring is ready. Check that JobSet metrics are enabled in kube-state-metrics and that Cloud Logging and Cloud Monitoring are enabled for your project.
- 4Describe the symptom. Ask in plain language, naming the JobSet, namespace, cluster, project and approximate time of the interruption.
- 5Review the findings. Read the step-by-step results for restarts, nodepool events, node health and pod failures, and check them against your own dashboards.
What is an agent skill and what does this one do?
An agent skill is a folder containing a SKILL.md file and optional scripts and reference files. When a request matches the skill's description, the agent loads the instructions and follows them. This skill is a playbook for JobSet interruptions on GKE clusters that run large AI/ML workloads.
The playbook works through metrics and logs in order: restart counts, spot VM preemptions and other nodepool events, node readiness failures and underlying host VMs, then pod and worker crashes. It is designed to suggest fixes at the end. It is built to run largely on its own and does not stop to fix credential or permission problems.
- Step 1: check whether the JobSet is in a restart loop using the kube_jobset_restarts metric.
- Step 2: check nodepool interruptions (PreemptionEvent, MaintenanceEvent, TerminationEvent) using metrics and Cloud Logging.
- Step 3: check node Ready status and correlate unhealthy nodes to physical host IDs.
- Step 4: inspect pod status and coordinator worker logs for application-level crashes.
Install the skill from the ZIP download
Download the skill as a ZIP file from this page and unzip it. The folder must be named gke-ai-troubleshooting-jobset-interruption and contain SKILL.md, so the agent can find it.
For a personal install, place the folder in ~/.claude/skills/. To share it with one project, place it in that project's .claude/skills/ folder. Start a new agent session afterwards so the skill is picked up.
If you use the Claude apps instead, upload the ZIP under Settings → Capabilities → Skills.
- Personal: ~/.claude/skills/gke-ai-troubleshooting-jobset-interruption/
- Project only: <your-project>/.claude/skills/gke-ai-troubleshooting-jobset-interruption/
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
~/.claude/skills/gke-ai-troubleshooting-jobset-interruption/SKILL.mdCheck the requirements before you start
The skill assumes your cluster and project are already set up for monitoring. Confirm these before asking for an investigation, because the queries return nothing without them.
- Google Cloud SDK (gcloud) installed, for running log queries.
- Access to Cloud Logging and Cloud Monitoring in the target Google Cloud project.
- JobSet metrics package enabled in kube-state-metrics on the cluster.
- Optional: a PROJECT_ID environment variable, used by the validation script.
How to trigger the skill with natural requests
You do not need to name the skill. Describe the symptom and include what you know: project, cluster, JobSet name, namespace and roughly when it happened. If you give a relative time such as 'about 20 minutes ago', the agent works out the query window from the current time or the log timestamps.
When an exact issue time is known, the skill sets the query window to 30 minutes before and 30 minutes after it.
- My JobSet llama-train in namespace ml keeps restarting on cluster tpu-cluster. Find out why.
- Our spot VM training run failed about 20 minutes ago. Check whether preemptions caused it.
- Some nodes in the training pool went NotReady at 14:05 UTC. Did a faulty host cause coordinator pod failures?
- Run a first-pass investigation of JobSet restarts in project my-project for cluster prod-ai.
What the agent will run and what it needs to access
The skill queries Cloud Monitoring with MQL and PromQL and Cloud Logging with LQL filters. Terminal gcloud commands are tagged with a usage attribution environment variable that identifies the skill and its version, as described in references/attribution.md. The skill's validation script is scripts/validate_queries.sh, and the package includes references/failure_signatures.md with example log patterns.
The skill runs scripts and uses the network, so it needs the same access you would give any tool that queries your Google Cloud project. It is told not to loop on authentication errors. On 403 or network isolation errors it fills in the query templates and works from any local telemetry or mock data files it finds, so its conclusions may rest on incomplete data. Check its findings against your own dashboards.
Safety notes before you run it
Use the skill for investigation, not for changes to production. The skill is told to proceed automatically and to keep diagnosing when access is denied. Treat its output as a first-pass hypothesis that a person reviews, especially on production clusters.
Review any script before running it, including scripts/validate_queries.sh. Do not expect the skill to diagnose general GKE cluster setup, basic app deployment, or bugs in non-JobSet applications.
- Read scripts/validate_queries.sh before executing it.
- Run it on a non-production cluster first if you can.
- Verify any recommended fix yourself before applying it.
What people use it for
- Finding out whether repeated restarts of a multi-host training JobSet come from spot VM preemptions.
- Checking whether one physical host keeps causing NotReady nodes and coordinator pod failures.
- Building a repeatable first-pass checklist for node and host failures across training runs.
- Collecting the restart count, interruption counts and related log events for an incident write-up.
Frequently asked questions
How do I install the GKE JobSet troubleshooting agent skill?
Download the ZIP from this page and unzip it so the folder is named gke-ai-troubleshooting-jobset-interruption. Put it in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.
What do I need to run this skill?
You need the Google Cloud SDK (gcloud), access to Cloud Logging and Cloud Monitoring in the target project, and JobSet metrics enabled in kube-state-metrics on the cluster. A PROJECT_ID environment variable is optional and is used by the validation script.
Is the GKE JobSet interruption skill free?
The skill is licensed under Apache-2.0, as listed on this page. Google Cloud usage for the queries it runs is billed under your own Google Cloud account, and this page does not state those costs.
Does the skill work with spot VMs?
Yes, it checks for spot VM preemptions as part of the nodepool step. It looks for PreemptionEvent entries in nodepool logs and the node pool interruption count metric, so you can see whether preemptions caused a failed run.
Why does the agent keep going when I get permission errors?
The skill is designed to avoid credential troubleshooting loops. On 403 or authentication errors, it fills in the query templates and uses any local telemetry or mock data files it finds. That means its conclusions can be based on incomplete data, so verify them.
Can this skill replace a GKE troubleshooting expert or other tools?
No. It covers only JobSet interruptions, restarts and preemptions on AI/ML workloads. For general cluster creation, basic deployments or non-JobSet application bugs, use other tools or documentation.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use this skill if you run multi-host AI training JobSets on GKE and want a repeatable first pass at finding out whether restarts come from preemptions, maintenance or faulty hosts. Pick something else for general GKE setup, ordinary app deployment, or production changes that need careful human control, since the skill is designed to run autonomously and may work from incomplete data.
Download the skill