Skip to content
Get packs

Guide · Agent skill

How to use the GKE AI Inference Deployment agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The gke-inference skill is a set of instructions that teaches an AI agent how to put an open-weight model, such as Gemma or Llama, into service on Google Kubernetes Engine (GKE), a managed platform for running containers. It is for engineers and platform teams who already use gcloud and kubectl and want guided help with choosing a model server, accelerator and manifest. After this guide you will know how to install the skill, ask for a deployment in plain language, and check what the agent does before it runs anything.

GKE AI Inference Deploymentgoogle/skills · Apache-2.0

Quick start

  1. 1Download and unzip the skill. Download the gke-inference ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder.
  2. 2Meet the prerequisites. Confirm gcloud is authenticated, kubectl is installed, you have a GKE Autopilot cluster and enough GPU or TPU quota in your region.
  3. 3Ask for a deployment. Describe the model and goal in plain language, such as deploying Gemma 2 9B for online serving.
  4. 4List valid model and accelerator pairs. Let the agent run the profiles list command for your model to find combinations that work.
  5. 5Generate the manifest. Have the agent create inference.yaml with your chosen model server, accelerator and optional latency target.
  6. 6Review and deploy. Check the file for Hugging Face token or PVC placeholders, then apply it with kubectl and watch the pods start.

What is an agent skill, and what does gke-inference make the agent do?

An agent skill is a folder containing a SKILL.md file of instructions that an AI agent loads when a task matches the skill's description. This one, named gke-inference, is for deploying AI and ML inference workloads on GKE, using GPUs, TPUs and model servers.

Once installed, the agent knows how to look up which model server and accelerator combinations are valid for a model, generate a Kubernetes deployment file, pick a GPU or TPU chip, and set up autoscaling. It also tells the agent to read an attribution file before running Google Cloud commands.

Install the gke-inference skill

Download the skill ZIP from this page and unzip it. For a personal setup, place the gke-inference folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/gke-inference/
  • Project: .claude/skills/gke-inference/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Check the requirements before you start

The skill assumes a working Google Cloud setup. The steps will fail without these, and GPU or TPU capacity costs money while it runs.

  • gcloud CLI, authenticated to a Google Cloud account
  • kubectl
  • A GKE Autopilot cluster, with GPU workloads supported via ComputeClasses and NAP
  • Sufficient GPU or TPU quota in the target region
  • A Hugging Face token for some gated models, stored as a Kubernetes Secret
  • Optional MCP Kubernetes tools such as apply_k8s_manifest and get_k8s_logs

Trigger the skill with a natural request

You do not need special syntax. Describe the outcome you want and name the model or the serving goal. The skill's description matches requests about deploying inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE.

Examples of requests that should match: 'Deploy Gemma 2 9B on GKE for online serving', 'Which accelerators can run Llama 3 8B on GKE?', 'Generate a Kubernetes manifest for vLLM with an L4 GPU'.

  • Deploy an AI model such as Llama, Gemma or Mistral to GKE
  • Generate optimized Kubernetes manifests for inference
  • Select GPU or TPU accelerators for model serving
  • Configure autoscaling for LLM inference

Follow the discovery, manifest and deploy workflow

The skill follows three stages. First it lists supported models and finds valid accelerator and server pairs for your model. Then it generates a manifest with the model, server, accelerator and an optional latency target. Last, you review the file for placeholders such as Hugging Face tokens or PVCs, and deploy it.

The model server options named in the skill are vllm, tgi, triton and tensorrt-llm. The accelerator examples include nvidia-l4, nvidia-tesla-a100 and nvidia-h100-80gb.

gcloud container ai profiles list --model=gemma-2-9b-it --quiet gcloud container ai profiles manifests create \ --model=gemma-2-9b-it \ --model-server=vllm \ --accelerator-type=nvidia-l4 \ --target-ntpot-milliseconds=50 --quiet > inference.yaml kubectl apply -f inference.yaml kubectl get pods -w

Configure autoscaling for inference

The skill includes a HorizontalPodAutoscaler example that scales on a GPU duty cycle metric. It advises using DCGM metrics for GPU utilisation, setting minReplicas to at least 1 for always-on serving, and using longer scale-down windows because model loading is slow. For latency-sensitive workloads, it suggests considering queue depth.

Safety notes: scripts and network access

The skill is marked as running scripts and using the network. Before the agent runs any gcloud, REST API or MCP command, the skill requires it to read references/attribution.md. For commands typed directly in a terminal, the agent prefixes gcloud with a metrics environment variable that tags the skill and its version. The file says this tag carries no user data, and it asks that config settings not be changed to persist the tag.

Read the generated inference.yaml and the commands before you approve them. Deployments create billable GPU or TPU capacity.

  • Review the manifest for placeholders and secrets before kubectl apply
  • Confirm the target project, region and cluster
  • Watch quota and remember that accelerators bill while running

Tips for a smoother first deployment

Start with a small model and the cheapest accelerator that fits it. The skill's selection guide ranks options from lowest to highest relative cost, with the NVIDIA L4 described for small to medium model inference. If a model and accelerator combination is rejected, re-run the profiles list command for that model. If you get a GPU quota error, request an increase or try another region.

  • Use quantized models such as GPTQ or AWQ to reduce GPU memory use
  • Tune batch size for your throughput versus latency goal
  • If the agent reports OOM on GPU, check model size against accelerator memory

What people use it for

  • Deploying Gemma 2 9B with vLLM on an NVIDIA L4 GPU for online serving
  • Comparing which accelerators can run a given model before committing to hardware
  • Generating a manifest that includes a Hugging Face token Secret for a gated model
  • Adding a GPU-based HorizontalPodAutoscaler to an existing LLM serving deployment

Frequently asked questions

How do I install the gke-inference skill?

Download the skill ZIP from this page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps you can upload the ZIP under Settings → Capabilities → Skills instead.

What do I need to use the gke-inference skill?

You need the gcloud CLI authenticated to a Google Cloud account, kubectl, a GKE Autopilot cluster, and sufficient GPU or TPU quota in the target region. Some gated models also need a Hugging Face token stored as a Kubernetes Secret.

Is the gke-inference skill free?

The skill is licensed under Apache-2.0. The skill itself is free to use, but the GPU or TPU capacity and cluster resources it deploys are billed by Google Cloud while they run.

Which model servers and accelerators does the skill support?

The skill names vllm, tgi, triton and tensorrt-llm as model servers. Accelerator examples include nvidia-l4, nvidia-tesla-a100 and nvidia-h100-80gb, and the selection guide also covers several Cloud TPU generations. Use the profiles list command to confirm which combinations are valid for your model.

What should I do if I get an invalid model and accelerator combination error?

Re-run the gcloud container ai profiles list command with your model ID to see which accelerator and server pairs are valid. Then regenerate the manifest using one of those combinations.

How does gke-inference compare with deploying to GKE by hand?

The skill guides the agent through model lookup, manifest generation, accelerator choice and autoscaling, so you do not have to write each step from the documentation. It does not replace knowing your cluster, quota and costs. It is not for migrating existing AI workloads, RAG with Cloud SQL or AlloyDB, or batch and HPC jobs, which the skill routes to other skills.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use gke-inference if you already run gcloud and kubectl on a GKE Autopilot cluster and want an agent to generate and check inference manifests for open-weight models. Pick something else if you have no GCP project or quota yet, if you are migrating existing AI workloads, or if you need RAG or batch and HPC pipelines, since the skill sends those to other skills.

Download the skill