Skip to content
Get packs

Guide · Agent skill

How to use the GKE GPU/TPU Disruption Troubleshooting agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

This is a GKE skill for AI agents that walks through why GPU or TPU nodes on Google Kubernetes Engine were disrupted, and checks whether Compute Engine host maintenance was the cause. It is aimed at platform and ML engineers running accelerator workloads, and at SRE teams chasing unexpected restarts on GPU or TPU nodepools. After following this guide you will know how to install the skill, ask for help with natural requests, and what the agent will check and recommend.

GKE GPU/TPU Disruption Troubleshootinggoogle/skills · Apache-2.0

Quick start

  1. 1Download and unzip the skill. Download the ZIP from this page and unzip it so the skill folder contains SKILL.md and references/attribution.md.
  2. 2Place it in your skills folder. Move the folder to ~/.claude/skills/ for personal use or to .claude/skills/ inside a project. Alternatively, upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
  3. 3Confirm access. Make sure you have kubectl access to the cluster, the gcloud CLI, and Cloud Monitoring and Cloud Logging access in the project.
  4. 4Describe the problem. Ask in plain language, for example about GPU or TPU node disruptions, and include project_id, location, cluster_name and the incident timestamp.
  5. 5Run the suggested checks. Review each kubectl, PromQL or log-filter step the agent proposes and run the ones you are comfortable with.
  6. 6Review the conclusion. Check whether the agent found host maintenance evidence or ruled it out and pointed to application causes.
  7. 7Apply protections deliberately. If host maintenance is confirmed, review graceful termination, opportunistic maintenance and PodDisruptionBudget settings, then apply them yourself to your cluster.

What is an agent skill?

An agent skill is a folder containing a SKILL.md file with instructions that an AI agent loads when a request matches the skill's description. This skill's SKILL.md describes a structured runbook for GPU and TPU disruptions on GKE. The folder also includes references/attribution.md, which covers how gcloud commands are tagged for usage attribution.

What the skill makes the agent do

The agent follows a fixed diagnostic workflow. It first asks for cluster details when you are debugging a real incident, then checks for scheduled maintenance, queries interruption metrics, reviews logs and node taints, and finally concludes with a summary and recommended mitigations.

  • Step 0: collect project_id, location, cluster_name and the incident timestamp before giving theories, unless you ask for a generic runbook.
  • Step 1: check nodes for the cloud.google.com/scheduled-maintenance-time label, which shows the Unix epoch time of a scheduled VM maintenance.
  • Step 2: provide PromQL on kubernetes_io:node_interruption_count filtered to interruption_reason="HW/SW Maintenance".
  • Step 3: look in Cloud Logging for cloud.google.com/active-node-maintenance set to ONGOING, and check for the cloud.google.com/impending-node-termination:NoSchedule taint.
  • Step 4: report findings and, if host maintenance is confirmed, recommend graceful termination, opportunistic maintenance and PodDisruptionBudgets together.

How to install the skill

Download the ZIP for this skill from this page and unzip it. For a personal install, place the unzipped skill folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

You can also upload the ZIP in the Claude apps. Go to Settings, then Capabilities, then Skills, and upload it there.

  • Personal: ~/.claude/skills/
  • Per project: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills

How to trigger the skill with natural requests

You do not need a special command. Describe the problem in plain language and mention GKE, GPU or TPU nodes and disruptions. The skill's description covers requests about diagnosing node disruptions, predicting host maintenance on GPU or TPU nodepools, inspecting interruption metrics, auditing node taints and configuring workload protection.

If you are debugging a real incident, include the project ID, location, cluster name and the time it happened. The agent will ask for anything you leave out.

  • "My training pods on our A100 nodepool restarted at 03:10 UTC. Help me find out if GKE host maintenance caused it."
  • "Check whether any GPU nodes in cluster ml-prod have a scheduled maintenance time."
  • "Give me a PromQL query to count HW/SW maintenance interruptions per node pool."
  • "How should I configure PodDisruptionBudgets and graceful termination for my Orbax checkpointing job on TPU nodes?"

Example: checking for scheduled maintenance

The agent can propose this kubectl command to list GPU or TPU nodes with a scheduled maintenance label. The SCHEDULED-MAINTENANCE-TIME column shows the Unix epoch time when the VM is scheduled for maintenance. If the label exists, the source says a disruption is guaranteed to occur.

kubectl get nodes -l cloud.google.com/scheduled-maintenance-time -L cloud.google.com/scheduled-maintenance-time

What the skill needs and safety notes

You need kubectl access to the target GKE cluster, the gcloud CLI for terminal commands, and Cloud Monitoring and Cloud Logging access in the Google Cloud project, or an MCP tool such as query_logs. The skill does not run scripts, but it does use the network.

The skill only recommends commands and configurations. You must review and apply any change to a production cluster yourself. For terminal gcloud commands, the skill sets CLOUDSDK_METRICS_REQUEST_ATTRIBUTION inline on each command, and it says not to use gcloud config set for this, so the setting does not persist. The skill also says never to recommend tolerating the impending-node-termination taint.

  • Requires kubectl access, gcloud CLI, and Cloud Monitoring and Logging access
  • Runs scripts: no
  • Uses network: yes
  • Only recommends changes; nothing is applied automatically

What people use it for

  • Finding out whether overnight training pods on a GPU nodepool were stopped by Compute Engine host maintenance.
  • Checking a TPU nodepool for scheduled maintenance labels before a long inference batch run.
  • Building a PromQL view of HW/SW maintenance interruptions per node pool for an SRE dashboard.
  • Planning graceful termination and PodDisruptionBudgets for an ML job that checkpoints with Orbax.
  • Auditing whether GKE has cordoned terminating GPU nodes using the impending-node-termination taint.

Frequently asked questions

How do I install the gke-ai-troubleshooting-handle-disruption-gpu-tpu skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

What do I need to use this GKE skill?

You need kubectl access to the target GKE cluster, the gcloud CLI for terminal commands, and Cloud Monitoring and Cloud Logging access in the Google Cloud project, or an MCP tool such as query_logs. You also need cluster details: project_id, location, cluster_name and the incident timestamp.

Is this skill free and what licence does it use?

The skill is listed under the Apache-2.0 licence. The source does not state any price, so check the licence terms on the listing before you rely on it.

Does the skill fix GKE disruptions automatically?

No. The skill only recommends commands and configurations. You must review and apply any change to production clusters yourself.

Can this skill help with general GKE cluster creation or networking?

No. The skill's description excludes general GKE cluster creation, network policy configuration and non-disruption workload deployment. It also does not apply to non-GKE or non-accelerator environments.

What if the agent finds no host maintenance evidence?

If the scheduled-maintenance labels, PromQL interruption counts and active maintenance logs are all negative or empty, the skill concludes that Compute Engine host maintenance did not cause the disruption. It then points you toward application causes such as OOMKill events, CUDA runtime errors or resource limits.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits platform and ML engineers running GPU or TPU workloads on GKE who need a disciplined way to test whether host maintenance caused a disruption. It is a poor fit for general cluster setup, networking, non-GKE environments, or anyone who wants changes applied automatically.

Download the skill