How to use the Phoenix AI Observability agent skill

What the skill makes the agent do

Agent skills are folders of instructions and reference files that an AI agent loads when a task matches them. This one covers Phoenix: recording traces of an application's calls, inputs, outputs and timing; running LLM-as-judge evaluations on saved data; comparing prompts or models in experiments; and monitoring systems in production.

• Tracing: collect OpenTelemetry-based traces for any LLM framework
• Evaluation: run quality checks such as hallucination and relevance evaluators
• Datasets and experiments: keep versioned test sets and compare prompts, models and configurations
• Playground: test prompts interactively against multiple models

How to install the phoenix-observability skill

Download the ZIP for this skill from this page. Then either unzip it into your personal skills folder at ~/.claude/skills/, or into a project's .claude/skills/ folder so it applies only to that project.

If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.

• Personal: unzip into ~/.claude/skills/
• Per project: unzip into .claude/skills/ inside the project folder
• Claude apps: Settings → Capabilities → Skills, then upload

How to trigger the skill with natural requests

You do not need a special command. Describe the problem in plain language and mention LLM tracing, evaluation or monitoring. The skill's description says it applies when you are debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems.

• Set up Phoenix tracing for my OpenAI chatbot and explain what each span records
• Why did this response go wrong? Help me trace the retrieval step in my RAG app
• Write an evaluator that checks whether answers are helpful and run it on my saved spans
• Compare two prompt versions on my QA dataset using a Phoenix experiment

Set up Phoenix and send your first trace

Install the packages, start a Phoenix server, then register a tracer and instrument your SDK. The skill's examples show Phoenix running locally on port 6006 and using the OpenAI instrumentor. Once instrumented, every OpenAI call your app makes appears in the Phoenix UI.

What you need before you start

You need Python with pip and a Python environment you can install into. The skill lists arize-phoenix>=12.0.0 and related packages: openinference instrumentors, phoenix-otel and phoenix-evals. Evaluations that use an LLM as judge, and any instrumented calls to a model, need an API key from an LLM provider such as OpenAI. A PostgreSQL database is optional and is only needed for a production deployment; the skill notes Phoenix can also run on SQLite.

• Python with pip
• arize-phoenix>=12.0.0 and related packages
• An LLM provider API key, such as OpenAI
• Optional: PostgreSQL for production

Safety notes: scripts and network access

This skill runs scripts and uses the network. Traces can contain the full text of prompts and responses, so check where your traces are stored before you send sensitive prompts or user data to any hosted service. Phoenix is self-hostable, which keeps traces on infrastructure you control.

• Review any script the agent writes before running it
• Do not send personal or confidential data to a hosted service without checking storage
• Keep API keys in environment variables rather than in code

Tips for getting better results

Give the agent your framework and model names, such as LangChain with gpt-4o, so it picks the right instrumentor. If setup fails, ask the agent to check the troubleshooting reference, which covers installation conflicts and port-in-use errors.

• Name your framework and provider up front
• Ask for a single working example first, then extend it
• Use the troubleshooting guide for import errors and port conflicts