Skip to content
Get packs

GKE TPU vBAR OOM Troubleshooter

Diagnoses vbar_control_agent crashes and OOMs on GKE TPU v6e nodes using Cloud Logging queries.

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
4 files

What it does

When installed, the agent follows a fixed troubleshooting procedure for a specific GKE problem: crashes and out-of-memory errors in the vbar_control_agent component on TPU v6e nodes. It works out a time window around an incident, builds Cloud Logging search filters to find matching serial-console and tpu-device-plugin errors, checks whether custom TPU metrics collectors are running, and then recommends either disabling those collectors or waiting for a GKE fix. It can also output the filters without running them if the user only asks how to check.

Great for

  • Platform or SRE engineers running TPU v6e workloads on GKE who see vbar_control_agent crashes or OOM messages
  • Teams that run custom TPU monitoring code using libtpu.sdk.tpumonitoring and suspect it conflicts with the agent
  • Engineers who want ready-made Cloud Logging filters for these specific error signatures

Skip it if

  • General container OOM debugging unrelated to TPUs; the skill's own description excludes this, and its checks are specific to vbar_control_agent
  • Non-TPU clusters or standard node lifecycle tasks, which the description also excludes
  • Production changes made without review: the high-risk step recommends disabling monitoring, which can reduce visibility into the cluster

Try asking

  • “My TPU v6e node in cluster tpu-cluster-prod had a vbar_control_agent crash at 2026-04-14T20:00:00Z. Help me find the cause.”
  • “Show me the Cloud Logging filter to look for memory cgroup OOM messages from vbar_control_agent on my node.”
  • “Check whether any custom libtpu.sdk.tpumonitoring collectors are running in my GKE cluster and could be causing this.”
Step-by-step guide: how to use GKE TPU vBAR OOM TroubleshooterInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the Apache-2.0 licence and an attribution note.

    gke-ai-troubleshooting-tpu-vbar-oom.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-ai-troubleshooting-tpu-vbar-oom.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-ai-troubleshooting-tpu-vbar-oom.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent an incident timestamp so the 30-minute window before and after it is computed correctly.
  • Ask for filters only (informational mode) if you do not want the agent to run live queries against your project.
  • Review any recommendation to disable custom metrics collection with your team before applying it, since the skill itself marks it high risk.

What's inside

  • references/attribution.md
  • references/failure_signatures.md
  • scripts/validate_queries.sh
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.