Skip to content
Get packs

Troubleshoot unresponsive GKE TPU/GPU nodes

Diagnose GKE TPU or GPU nodes stuck NotReady with NodeStatusUnknown, using read-only checks

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
2 files

What it does

When a Google Kubernetes Engine (GKE) node that runs AI accelerator hardware (TPUs or GPUs) stops responding, this skill teaches the agent a structured way to find out why. The agent checks the node's status and Google Cloud logs and metrics to confirm the node has timed out, and then points you to likely causes such as kernel panics, hardware lockups, or auto-repair being turned off. It is designed to only read data and never changes your cluster; any fix is handed back to you to apply yourself.

Great for

  • Platform or ML engineers running multi-host TPU or GPU training jobs on GKE who see nodes stuck NotReady
  • On-call operators investigating pods stuck in Terminating on accelerator node pools
  • Teams checking whether node auto-repair is enabled on a Standard node pool

Skip it if

  • Healthy nodes or pods that crash because of application bugs; the skill tells the agent to rule out node timeouts and move on to other troubleshooting
  • Routine GKE upgrades, which the skill explicitly excludes
  • Anyone expecting the agent to automatically repair or drain nodes; the skill forbids this and leaves fixes to you

Try asking

  • “My TPU node in GKE cluster ml-prod shows NotReady since 14:05 UTC. Help me find out why.”
  • “Pods on GPU node pool a100-pool are stuck in Terminating. Check whether the node stopped heartbeating.”
  • “Is auto-repair enabled on my TPU slice node pool, and did it trigger any repairs recently?”
Step-by-step guide: how to use Troubleshoot unresponsive GKE TPU/GPU nodesInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the Apache-2.0 licence and an attribution note.

    gke-ai-troubleshooting-node-unresponsive-timeout.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-node-unresponsive-timeout.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-ai-troubleshooting-node-unresponsive-timeout.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent the exact node name, cluster, location, and an incident timestamp in UTC so it can search the right 60-minute window.
  • Use a account with only viewer roles for the investigation; cluster admin rights are only needed if you choose to apply a fix yourself.
  • Ask the agent to link the relevant Google Cloud documentation section for any fix it suggests, then review it before running anything.

What's inside

  • references/attribution.md
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.