Guide · Agent skill
How to use the GKE TPU multi-slice hang troubleshooting skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
This page explains how to install and use the gke-ai-troubleshooting-tpu-mxla-hang agent skill, which guides an AI agent through a structured investigation when a JAX training job spread across several Cloud TPU slices on GKE stops making progress. The skill is licensed under Apache-2.0. After following this guide you will know what the skill does, how to add it to Claude, how to ask for it in plain language, and what it needs before it runs.
GKE TPU Multi-Slice Hang Troubleshootinggoogle/skills · Apache-2.0 Download the skill Quick start
- 1Download the ZIP. Download the skill ZIP file from this page.
- 2Unzip into a skills folder. Unzip the file into ~/.claude/skills/ for personal use, or into .claude/skills/ in a project. Alternatively, upload it in the Claude apps under Settings → Capabilities → Skills.
- 3Prepare gcloud and kubectl. Install the Google Cloud SDK with the alpha component and kubectl, then run gcloud auth login and set your project.
- 4Check APIs and IAM roles. Confirm billing is linked and the required APIs are enabled, and that your account has the listed IAM roles.
- 5Describe the hang. Ask the agent in plain language, mentioning HANG_DETECTED or a frozen multi-slice job if you can.
- 6Review the routed path. Read the category the agent picks and the evidence it cites before acting on any suggested fix.
What the skill makes the agent do
A multi-slice TPU hang happens when a worker has waited on a Megascale communication operation for a set timeout. The TPU logs then show a Megascale HANG_DETECTED message. The skill tells the agent to read the Megascale HANG_DETECTED logs, the report from the ML Diagnostics Megascale XLA (MXLA) Hang Analyzer, and 1-minute multi-slice latency metrics, together with GKE node topology labels.
The agent then sorts the likely cause into one of four routes: compiler or HLO divergence, host program queueing or data input stall, hardware or network faults on specific instances, or a case where the signal exists but the analyzer reports NOT_DETECTED or UNKNOWN.
- Category A: compiler or HLO divergence, such as FINGERPRINT_MISMATCH
- Category B: program queueing or data input stall, such as DATA_INPUT_STALL
- Category C: TPU chip, SparseCore, ICI, or DCN networking faults, or unrecoverable errors on specific instances
- Category D: HANG_DETECTED logs or a hang event fired, but the analyzer reports NOT_DETECTED or UNKNOWN
Check the requirements before you install
You need the Google Cloud SDK with the alpha component (for the gcloud alpha mldiagnostics commands) and kubectl. The Google Cloud project must have an active billing account and must have the container, logging, monitoring, and hypercomputecluster APIs enabled. You also need an authenticated gcloud session.
The skill only supports JAX on TPUs. Its ML Diagnostics commands will not list runs or events for PyTorch or other frameworks. Check the GKE and LibTPU minimum versions listed on the skill page, because the hang analyzer and multi-slice metrics depend on them.
- Google Cloud SDK (gcloud) with the alpha component, and kubectl
- Active project with billing enabled and the container, logging, monitoring, and hypercomputecluster APIs enabled
- IAM roles: Cluster Director Editor, Monitoring Viewer, Logs Viewer, and Kubernetes Engine Viewer
- Kubernetes Engine Cluster Admin, only for high-risk remediation steps the user chooses to run
- An authenticated gcloud session
Install the skill in Claude
Download the ZIP file for the skill from this page. Then either unzip it into your personal skills folder at ~/.claude/skills/, or into a project folder at .claude/skills/ inside the project you are working in.
If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.
- Personal install: unzip into ~/.claude/skills/
- Project install: unzip into .claude/skills/ in your project
- Claude apps: Settings → Capabilities → Skills, then upload
Trigger the skill with natural requests
You do not need to type the skill name. The skill is meant to activate when you describe a hang on a multi-slice TPU job. Describe the symptom in plain words and mention the signal you see if you have one, such as HANG_DETECTED in the logs.
The skill is not meant for slow training with no hang, or for pods that were restarted by preemption or eviction. Those belong to the other skills named in its description.
- My multi-slice TPU training job on GKE stopped making progress. Can you find out why?
- I see HANG_DETECTED in the Megascale logs for my JAX job. What is causing it?
- Our jobset on TPU slices is stuck in a collective operation. Triage it read-only.
- Check the MXLA hang analyzer report for our training run and tell me which path applies.
Understand the safety rules and what the skill runs
The skill runs scripts and uses the network, because it calls gcloud, kubectl, and Google Cloud APIs. Its main rule is read-only: the agent runs only read-only diagnostic commands and never drains, deletes, or re-creates nodes, or runs any other command that changes the cluster. Any fix is given to you to apply yourself.
For each category, the skill also tells the agent not to cordon, drain, or replace TPU nodes. Terminal gcloud calls are tagged with an attribution prefix that identifies the skill and its version, and that tag carries no user data. The tag is set inline on each command rather than saved with gcloud config.
- Read-only: diagnostic commands only, with remediation handed back to you
- Categories A and B explicitly forbid cordoning, draining, or replacing nodes
- Terminal gcloud commands carry the attribution prefix shown below
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-mxla-hang)" gcloud <command> [flags]Read the result and act on it
The skill ends each investigation by routing to one path document, and each path lists the doc sections for its recommended fixes. For example, a FINGERPRINT_MISMATCH points to dumping the HLO following the OpenXLA HLO dump guide and sharing the output with Google through a support case. A DATA_INPUT_STALL points to the ML Diagnostics profiler-target and on-demand profiler session commands.
Ask the agent to show you the evidence it used, such as the log lines and the analyzer report fields, so you can check its reasoning before you change anything.
- Path A: compiler or HLO divergence
- Path B: host program queueing or data input stall
- Path C: hardware or network faults
- Path D: no detection or unknown result
- Path E: healthy telemetry
What people use it for
- Triaging a JAX training job on multi-slice Cloud TPUs in GKE that has stopped advancing its steps
- Platform or ML infrastructure teams sorting recurring HANG_DETECTED events into a likely cause
- Producing a documented, read-only first pass before opening a Google Cloud support case
Frequently asked questions
Is the gke-ai-troubleshooting-tpu-mxla-hang skill free?
The skill is licensed under Apache-2.0. Running it requires a Google Cloud project with billing enabled, and this source does not list any prices, so check Google Cloud billing for the costs of the resources you use.
What do I need to install before using this skill?
You need the Google Cloud SDK with the alpha component, kubectl, an authenticated gcloud session, and a project with billing and the required APIs enabled. The skill also needs specific IAM roles and a GKE and LibTPU version at or above the minimums on the skill page.
Does this skill work with PyTorch?
No. The ML Diagnostics platform supports only JAX on TPUs, so its gcloud alpha mldiagnostics commands will not list runs or monitored events for PyTorch or other frameworks.
Will the skill change my cluster or restart my nodes?
No, by design. The skill is read-only and never drains, deletes, or re-creates nodes. It gives remediation steps for you to apply yourself, and it explicitly forbids cordoning, draining, or replacing TPU nodes for compiler and data input causes.
What should I use instead for a slow job with no hang?
Use the gke-ai-troubleshooting-tpu-performance-degradation skill for gradual step-time drops without hangs. Use the gke-ai-troubleshooting-jobset-interruption skill for pod preemption or eviction restarts.
Can the skill be wrong about the cause?
Yes. The skill's own description says the agent may misroute or miss the cause. Check the evidence it cites and consider a Google Cloud support case for confirmation.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits engineers running JAX training on multi-slice Cloud TPUs in GKE who need a documented, read-only first pass on a hang. It is a poor fit for PyTorch workloads, slow training without a hang, or preemption restarts, which the other named skills handle.
Download the skill