How to use the AI SRE Incident Response agent skill

What does the AI SRE Incident Response skill make the agent do?

The skill applies standard site reliability engineering practice to AI systems. It groups AI incidents into four classes: availability (model or provider unavailable, timeout storms), quality (answer accuracy or tool success drops below SLO), safety (harmful or policy-violating outputs increase), and cost (unexpected token or provider spend spikes).

Once installed, the agent can suggest a severity level for an incident, propose Prometheus and Alertmanager alert rules, and walk a responder through step-by-step runbooks, including fallback routing and rollbacks. The file itself contains no executable scripts; the agent is giving guidance, and people run the commands.

• Severity levels SEV1 to SEV4, each with a response time and notification method
• Example Prometheus alert rules for endpoint down, hallucination rate, token cost, p95 latency, guardrail violations, groundedness and provider errors
• Runbooks for model outages, quality regressions and token cost explosions

How do you install the skill?

Agent skills are folders that contain a SKILL.md file with instructions the agent loads when a task matches. This skill is distributed as a ZIP file from the page where you found this guide. Its folder contains a single file, SKILL.md.

You can install it in one of two ways. For a personal setup, unzip the download into ~/.claude/skills/. For a single project, unzip it into the project's .claude/skills/ folder so that it applies only to that repository. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

• Personal: unzip into ~/.claude/skills/
• Project only: unzip into .claude/skills/ inside the project
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP

What do you need before you use it?

The skill assumes you already run the monitoring and delivery tools it refers to. It does not install them. The compatibility note lists the platform CLIs the runbooks use, and the prerequisites list the monitoring and on-call systems the alert rules depend on.

Most importantly, you need authorized access to the target environment. Several runbook steps run kubectl commands that can change live deployments.

• Prometheus and Alertmanager, with scrape targets for your AI services
• Grafana dashboards for latency, error rate, cost and quality
• An on-call tool such as PagerDuty, Opsgenie or equivalent
• A runbook repository that responders can access
• A rollback mechanism for model and prompt versions, such as GitOps or feature flags
• CLIs referenced by the runbooks: kubectl, helm, terraform, git and CI runners
• Authorized access to the target environment

How do you trigger the skill with a natural request?

You do not need special syntax. Describe the incident or the artifact you need in plain words, and mention the kind of problem so the agent can match it to the right class (availability, quality, safety or cost). Include the service, model or tenant if you know them, because the runbooks ask responders to identify scope first.

Good requests name the problem and the output you want. Examples are below. Treat the output as a draft to review with your team, not as a finished plan.

• "Our LLM gateway is returning 503s from the provider. Give me a SEV-level assessment and the model outage runbook steps."
• "Write Prometheus alert rules for token cost per tenant and p95 latency on our inference service."
• "Hallucination rate jumped after last night's model promotion. Walk me through the quality regression runbook."
• "Draft a severity framework for guardrail violation spikes on our customer support bot."

Example alert rules and runbook steps

The skill includes a Prometheus rule file. The rule below fires when the endpoint job reports down for two minutes. Its thresholds are illustrative: the skill says to calibrate them to your declared SLO, metric units, tenant budgets and workload before use.

The model outage runbook starts by checking the provider status page and network connectivity. If the provider is down, it enables a fallback route. If a self-hosted model is down, it checks pod status and restarts the deployment when out-of-memory errors appear.

Safety notes: scripts, network and production changes

The skill file does not run scripts, and it does not use the network itself. However, the runbooks contain commands for your team to run, including curl health checks and kubectl commands that set environment variables, restart deployments and roll back releases. Those commands act on real systems, so only run them when you are authorized and have confirmed the target.

The skill also states that its groundedness proxy is not a verified quality measurement and that its thresholds must be calibrated. A Kubernetes annotation alone does not enforce a deployment freeze, so verify that your controller or pipeline honors it. Treat production traffic as production: do not describe a live call as an offline test.

• Run kubectl and rollback commands only with authorized access to the target environment
• Verify that a deployment freeze is enforced by your controller or pipeline, not just by an annotation
• Use synthetic or redacted cases for evaluation runs
• Check personal-data exposure before increasing trace sampling