How to use the Agent Observability agent skill

What the skill makes the agent do

When you ask for help with observability, the agent uses this skill to propose and adapt Prometheus metric definitions and OpenTelemetry tracing patterns to your codebase. The goal is to see how long each step of an agent turn takes, how many tokens it uses, what it costs, and where errors happen.

The skill's core metrics include an end-to-end agent request duration histogram, an LLM call latency histogram, a tool call latency histogram, prompt and completion token counters, a cost counter, tool call counters with success, error and timeout statuses, and LLM error and retry counters. It also shows how to wrap an LLM call in an OpenTelemetry span that records the model, provider, token counts and finish reason.

• Latency for each full agent turn, each LLM API call and each tool call
• Token usage per model and agent, including cached prompt tokens where the provider reports them
• Estimated cost in USD per model, split into prompt and completion
• Errors, retries and in-flight requests
• Tracing spans for the top-level request, LLM calls, tool execution and retrieval

Install the agent-observability skill

Download the skill ZIP from the skill page on this site. Then choose one of the two install routes below.

For a personal install across projects, unzip the file into ~/.claude/skills/. For a single project, unzip it into the .claude/skills/ folder inside that project. In the Claude apps, you can instead upload the ZIP under Settings, then Capabilities, then Skills.

• Personal install: ~/.claude/skills/
• Project install: .claude/skills/ inside your project folder
• Claude apps: Settings, then Capabilities, then Skills, then upload the ZIP

Check what you need before you start

The skill assumes you can wire up metrics and exporters. You need a Python environment with the prometheus_client package and the OpenTelemetry SDK, including the OTLP exporter. You also need access to the target application and its deployment environment.

An OTLP-compatible collector or backend is optional and depends on your setup. The source names Jaeger and Grafana as examples, plus vendor endpoints. The skill's metadata also lists general platform CLIs such as kubectl, helm and terraform as prerequisites, but the skill itself does not bundle scripts, so you only need those if your own deployment uses them.

• Python with prometheus_client and opentelemetry-sdk, plus the OTLP exporter
• Access to the application and its deployment environment
• Optional: an OTLP collector or backend such as Jaeger, Grafana or a vendor endpoint

Trigger the skill with natural requests

You do not need a special command. Describe the problem in plain language and mention what you want to see. The agent will load the skill when your request matches its description of instrumenting AI agents with tracing, token metrics, latency and cost visibility.

Good requests name the symptom or the goal. The more specific you are about the model, the agent framework and the question you want answered, the more closely the examples can be adapted to your code.

• Add tracing and token metrics to my agent so I can see cost per model.
• Our agent requests are slow. Help me add latency histograms for LLM calls and tool calls.
• Show me how to record prompt and completion tokens with Prometheus counters.
• Set up OpenTelemetry spans around each LLM call and tool call in this codebase.

Review the data safety rules before exporting telemetry

Sending telemetry out of your system is a data disclosure. The skill's guidance says to approve the destination and retention first, and to omit prompts, credentials, direct user IDs and raw exception bodies. Raw exception text may contain prompts, URLs or credentials, so the examples record only the exception type.

Use TLS for any collector that is not on loopback. The example tracing setup only uses an insecure connection when the endpoint starts with localhost, so treat any remote endpoint as requiring encryption. The skill's metadata also marks its risk level as critical, which is a reason to review the code before running it against production.

Do not use the skill on systems you are not authorised to instrument, because it involves connecting to production collectors and environments. The skill runs no scripts itself, but the code it helps you write will make network connections when you run it.

• Approve the telemetry destination and retention period before export
• Never send prompts, credentials, direct user IDs or raw exception bodies
• Use TLS for non-loopback collectors
• Only instrument systems you are authorised to change

Tips for getting useful results

Keep label values bounded. The skill recommends importing shared metric definitions once and using a fixed set of configured label values, so that metric cardinality stays under control. The examples use custom llm.* span attributes and do not claim full OpenTelemetry semantic-convention compliance.

If the provider does not return usage or choices, the example raises an error instead of recording zero. Ask the agent to keep unknown values as unknown rather than inventing numbers.

• Import the metric definitions once and reuse them
• Use the custom llm.* attribute names shown in the examples
• Log only fixed event names, never raw request content, in structured logs
• Ask the agent to adapt the examples to your own agent_name and tool_name values