Skip to content
Get packs

Guide · Agent skill

How to use the GKE TPU Stockout Troubleshooter agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The GKE TPU Stockout Troubleshooter is an agent skill, a packaged set of instructions that an AI agent such as Claude can load when it matches your request. It helps when a Google Kubernetes Engine (GKE) cluster cannot create or scale up a TPU node pool with errors like GCE_STOCKOUT or ZONE_RESOURCE_POOL_EXHAUSTED, even though your Compute Engine reservation should have capacity. After this guide you will know how to install the skill, how to ask for it in plain language, what it checks, and what it will and will not do on your cluster.

GKE TPU Stockout Troubleshootergoogle/skills · Apache-2.0

Quick start

  1. 1Confirm your cluster is Standard. Check that the cluster uses Standard mode, because Autopilot always auto-repairs nodes and this cause does not apply.
  2. 2Install the tools and roles. Make sure gcloud and kubectl are installed, the project has an active billing account, and you hold the four viewer roles listed in the requirements.
  3. 3Download and unzip the skill. Download the ZIP from the page, then unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ inside a project. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
  4. 4Describe the failure. Ask the agent in plain language, including the error code, the cluster, the node pool, the reservation and the approximate time.
  5. 5Review the diagnosis. Let the agent check audit logs, node states and reservation utilization, then read its explanation of whether unhealthy nodes are holding reserved slots.
  6. 6Apply any fix yourself. Review the recommended fixes and their linked documentation, then carry out the ones you choose. The agent does not change the cluster.

What the skill makes the agent do

When a TPU node pool fails to create or scale up with a stockout error, the agent follows a structured checklist. Its main question is whether broken, NotReady TPU nodes with node auto-repair turned off are still occupying slots in the reserved capacity. If VMs from those nodes are still running, they keep counting against the reservation, so the free slots may be fewer than the new node pool needs.

The agent starts by collecting context such as your project, cluster, location, failing node pool, reservation name, reservation project, zone and the time the issue happened. It then checks Cloud Audit Logs and GKE operations for stockout errors and compares reservation utilization, using the reservation's in-use count against its total count. Once it has found the cause, it explains it and recommends fixes.

  • The agent runs only read-only diagnostic commands.
  • It never drains, deletes or re-creates nodes, and it does not change the cluster.
  • Any fix is given to you to apply yourself, and high-risk steps are labelled as such.

Install the skill from a ZIP file

Download the ZIP file for the skill from the page where you found this guide. Then unzip it into one of two locations, depending on how you want to use it. For a personal setup that works across your projects, unzip it into ~/.claude/skills/. To limit the skill to a single project, unzip it into the .claude/skills/ folder inside that project.

If you use the Claude apps instead of a local setup, you can upload the skill there. Open Settings, go to Capabilities, then choose Skills and upload the ZIP. The skill's folder should contain SKILL.md and a references folder with attribution.md.

  • Personal use: ~/.claude/skills/
  • Single project: .claude/skills/ inside the project folder
  • Claude apps: Settings → Capabilities → Skills

Check the requirements before you start

The skill is built for GKE Standard clusters that use a Compute Engine reservation for TPUs. Autopilot clusters always auto-repair nodes and you cannot turn that off, so the cause this skill looks for does not apply to them. Check your cluster mode first.

You need the Google Cloud SDK (gcloud) and kubectl installed. The target project needs an active Google Cloud billing account. You also need the Logs Viewer, Monitoring Viewer, Kubernetes Engine Viewer and Compute Viewer IAM roles, and the container, compute, logging and monitoring APIs enabled. The Kubernetes Engine Cluster Admin role is needed only for the high-risk remediation steps the skill may suggest.

To confirm billing is attached to your project, you can run this command, replacing the placeholder with your project ID:

  • Google Cloud SDK (gcloud) and kubectl
  • Active billing account on the target project
  • IAM roles: Logs Viewer, Monitoring Viewer, Kubernetes Engine Viewer, Compute Viewer
  • APIs: container, compute, logging and monitoring googleapis.com
  • Kubernetes Engine Cluster Admin only for high-risk remediation
gcloud billing projects describe {project_id}

Trigger the skill with a natural request

You do not need a special command. Describe the problem in plain language and include the error text if you have it. The skill is meant to activate when TPU node pool creation or scale-up fails with GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS while you are targeting a specific Compute Engine TPU reservation.

Give the agent as much of the context as you can: project, cluster name, location, the node pool that failed, the reservation name and the approximate time of the failure. The skill uses a 60-minute window centred on that time by default.

  • My GKE TPU node pool scale-up failed with GCE_STOCKOUT in zone us-central2-b, even though the reservation should have capacity. Can you check why?
  • Our TPU node pool creation returns ZONE_RESOURCE_POOL_EXHAUSTED. The reservation is shared from another project. Please investigate.
  • Help me find out whether NotReady TPU nodes with auto-repair off are using up our reservation.

What the skill does not cover

The skill is scoped to TPU reservation stockouts. The description says not to use it for general pod scheduling problems without a TPU reservation, or for container crashes inside a workload. For those, ask for help with a different issue, since this skill's checklist will not address them.

  • General pod scheduling problems without a TPU reservation
  • Workload-level container crashes
  • Autopilot clusters, where auto-repair is always on

Safety notes: scripts and network use

According to the skill's own description, it runs no scripts. It does use the network, because it queries Google Cloud services such as Cloud Audit Logs, Cloud Monitoring and the GKE and Compute Engine APIs through gcloud and kubectl. Those commands run with the credentials and IAM roles you have granted.

The skill's attribution reference asks that gcloud commands run in the terminal carry a metrics tag identifying the skill. That tag does not contain user data, and the reference says not to set it with gcloud config set, so it does not persist beyond the task. Review the commands the agent proposes before you approve them, as you would with any agent acting in your environment.

  • No scripts are run, according to the skill description.
  • Network access is used for Google Cloud and Kubernetes queries.
  • Read the proposed commands before you approve them.

What people use it for

  • Finding out why a TPU node pool scale-up fails with GCE_STOCKOUT when the reservation looks like it should have free capacity.
  • Checking whether NotReady TPU nodes with auto-repair disabled are still consuming reserved slots after a failure.
  • Investigating a stockout on a shared reservation where the owner project differs from the cluster's project.
  • Getting a documented explanation of the cause and linked fixes to share with a platform team before any change is made.

Frequently asked questions

What is the GKE TPU Stockout Troubleshooter agent skill?

It is an agent skill, a set of instructions an AI agent can load, that diagnoses GKE TPU node pool creation or scale-up failures caused by Compute Engine reservation stockouts. It focuses on whether unhealthy TPU nodes with auto-repair disabled are still holding reserved capacity.

How do I install an agent skill from a ZIP file?

Download the ZIP from the page, then unzip it into ~/.claude/skills/ for personal use or into .claude/skills/ inside a project. In the Claude apps you can instead upload the ZIP under Settings → Capabilities → Skills.

What do I need to use this skill?

You need gcloud and kubectl, an active billing account on the project, the Logs Viewer, Monitoring Viewer, Kubernetes Engine Viewer and Compute Viewer roles, and the container, compute, logging and monitoring APIs enabled. Kubernetes Engine Cluster Admin is needed only for high-risk remediation.

Does this skill work with GKE Autopilot?

No. The skill applies to Standard clusters. Autopilot clusters always auto-repair nodes and you cannot turn that off, so the cause this skill investigates does not occur there.

Will the skill change or delete anything in my cluster?

The skill is instructed to run only read-only diagnostic commands and never to drain, delete or re-create nodes. It gives you any fix to apply yourself.

Is the skill free and what licence does it use?

The source lists the licence as Apache-2.0. The source does not state pricing, but using it requires a Google Cloud billing account on the target project, and Google Cloud usage is billed under that account.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits platform or ML engineers running TPU workloads on GKE Standard clusters with a Compute Engine reservation, especially when scale-ups fail although capacity should be available. If your problem is general pod scheduling, container crashes or an Autopilot cluster, pick a different troubleshooting approach.

Download the skill