Skip to content
Get packs

GKE AI Inference Deployment

Deploy and tune LLM and other AI model inference servers on Google Kubernetes Engine with GPUs or TPUs.

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
2 files

What it does

Once installed, the agent knows how to put an AI model (such as Gemma or Llama) into service on Google Kubernetes Engine (GKE), a managed cloud platform for running containers. It guides the agent to look up which model server and accelerator combinations are valid, generate a Kubernetes deployment file, pick a GPU or TPU chip, and set up autoscaling. It also tells the agent to read an attribution file before running Google Cloud commands.

Great for

  • Engineers deploying open-weight LLMs to GKE for online serving
  • Platform teams choosing between GPU and TPU hardware for inference cost and latency
  • Teams already using gcloud and kubectl who want guided manifest generation

Skip it if

  • Migrating existing AI workloads to GKE: the skill itself routes this to a different skill (google-cloud-solution-guided-gke-ai-migration)
  • RAG pipelines backed by Cloud SQL or AlloyDB, or batch/HPC jobs, which are handled by other skills named in the description
  • Anyone without a GCP project, quota, and cluster: the steps will fail and GPU or TPU capacity costs money while running

Try asking

  • “Deploy Gemma 2 9B on GKE with vLLM on an NVIDIA L4 GPU and generate the manifest”
  • “Which accelerators can run Llama 3 8B on GKE, and what are the cost tradeoffs?”
  • “Set up a GPU ComputeClass and an autoscaler for my LLM inference server on Autopilot”
  • “Review this inference.yaml for missing Hugging Face token secrets before I apply it”
Step-by-step guide: how to use GKE AI Inference DeploymentInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the Apache-2.0 licence and an attribution note.

    gke-inference.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gke-inference.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gke-inference.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Review the generated inference.yaml before running kubectl apply, especially placeholders for tokens and volumes, since it deploys billable GPU or TPU resources.
  • Confirm quota and accelerator availability in your region first; the skill's profile lookup helps pick a valid model and server combination.
  • Keep minReplicas at 0 only for batch or on-demand work, since cold starts are slow while models load.

What's inside

  • references/attribution.md
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.