Skip to content
Get packs

GKE GPU/TPU Disruption Troubleshooting

Diagnose and prevent GKE GPU/TPU node interruptions caused by Compute Engine host maintenance.

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
2 files

What it does

When installed, the agent follows a structured runbook for investigating why GPU or TPU nodes on Google Kubernetes Engine (GKE) were disrupted. It checks for scheduled maintenance labels, suggests Cloud Monitoring PromQL queries for interruption counts, and looks through logs and node taints to see whether Google's host maintenance caused the problem. It then recommends protections such as graceful termination, opportunistic maintenance, and PodDisruptionBudgets, and it asks for cluster details before giving theories when a real incident is being debugged.

Great for

  • Platform or ML engineers running GPU or TPU training and inference workloads on GKE
  • SRE teams investigating unexpected pod restarts or node disruptions on accelerator nodepools
  • Teams planning how to protect long-running ML jobs from host maintenance

Skip it if

  • General GKE cluster creation, networking, or non-disruption deployment tasks, which the skill's own description excludes
  • Non-GKE or non-accelerator environments, where the GKE-specific labels and metrics will not apply
  • Anyone expecting an automatic fix: the skill only recommends commands and configurations, and the user must review and apply changes to production clusters

Try asking

  • “My GPU training job on GKE restarted overnight. Project my-project, cluster ml-cluster, us-central1, around 03:00 UTC. Help me find out whether host maintenance caused it.”
  • “Check my TPU nodepool for upcoming scheduled maintenance and show me the PromQL to track interruption counts.”
  • “Audit the node taints on my GPU nodes and explain whether the impending-node-termination taint is present.”
  • “Help me set up graceful termination, opportunistic maintenance, and PodDisruptionBudgets for my inference deployment on GKE.”
Step-by-step guide: how to use GKE GPU/TPU Disruption TroubleshootingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the Apache-2.0 licence and an attribution note.

    gke-ai-troubleshooting-handle-disruption-gpu-tpu.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-handle-disruption-gpu-tpu.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-ai-troubleshooting-handle-disruption-gpu-tpu.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent the project ID, location, cluster name and timestamp up front so it does not stop to ask for them.
  • Review every kubectl or gcloud command before running it, especially against production clusters.
  • Keep the PromQL queries for ongoing monitoring so you can track interruption trends after the incident.

What's inside

  • references/attribution.md
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.