Skip to content
Get packs

Guide · Agent skill

How to use the Arize Prompt Optimization agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The arize-prompt-optimization skill is an agent skill from github/awesome-copilot that helps an AI agent improve the prompts behind an LLM application using real production data from Arize. It is aimed at engineers and prompt engineers who already log traces to Arize and want evidence from failures rather than trial and error. After this guide you will know what the skill does, how to install it, how to phrase a request that triggers it, and what you need configured first.

Arize Prompt Optimizationgithub/awesome-copilot · MIT

Quick start

  1. 1Get the ZIP. Download the arize-prompt-optimization skill ZIP from the github/awesome-copilot page for skills/arize-prompt-optimization.
  2. 2Install the skill. Unzip it into ~/.claude/skills/ for all projects or into a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Install the ax CLI. Install arize-ax-cli with uv, pipx or pip, then confirm the version is 0.14.0 or higher.
  4. 4Configure your Arize profile. Create an API key at app.arize.com under Admin > API Keys, then check your setup with ax profiles show.
  5. 5Find your space. Run ax spaces list to get the space name or ID you will use with the skill.
  6. 6Ask for an optimization. Describe the project and the failure in plain language, such as incorrect answers in a named project.

What is an agent skill and what does this one do?

An agent skill is a folder containing a SKILL.md file and optional reference files and scripts. When your AI agent sees a request that matches the skill's description, it loads those instructions and follows them. This skill is licensed MIT.

Once installed, the agent can pull real LLM call records (spans) from an Arize project, extract the prompts that were actually sent to a model, and look at performance signals such as human annotations and LLM-as-judge eval explanations. It then runs a data-driven optimization loop to suggest and test prompt improvements instead of guessing. It relies on the ax command-line tool to talk to Arize.

  • Extracts the system and user messages from LLM spans, or the prompt template and its variables if the app uses templates.
  • Reads error spans, low-scoring spans, and high-latency spans as evidence of failures.
  • Reads human grades (annotation columns) and eval explanations (eval columns) to see why outputs were judged poorly.
  • Can export datasets and experiment results and join them by example_id for analysis.

Install the skill with a ZIP download or the Claude apps

Download the skill as a ZIP from the github/awesome-copilot page for skills/arize-prompt-optimization. Unzip it so the skill folder ends up in ~/.claude/skills/ for all your projects, or in a project's .claude/skills/ folder for one project only.

If you use the Claude apps instead, open Settings, go to Capabilities, then Skills, and upload the skill there.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into the project's .claude/skills/ folder
  • Claude apps: Settings → Capabilities → Skills, then upload

Set up the ax CLI and your Arize profile

The skill does not work on its own. It needs the ax CLI installed and a configured Arize profile that holds an API key from app.arize.com. You also need your Arize space name or ID. The skill's troubleshooting notes say the ax version must be 0.14.0 or higher, and that the CLI can be installed with uv tool install arize-ax-cli, pipx, or pip.

Check your profile with ax profiles show. If the key is missing or wrong, the reference guide walks through fixing it. The skill is written to reference the ARIZE_API_KEY environment variable rather than pasting the key into a command.

  • Install ax: uv tool install arize-ax-cli (or pipx install arize-ax-cli, or pip install arize-ax-cli)
  • Find your space with ax spaces list
  • Get an API key from https://app.arize.com/admin > API Keys
  • Install jq, which the examples use to parse exported JSON
uv tool install arize-ax-cli

Trigger the skill with a natural request

You do not need to name the skill. The description lists trigger phrases, so ask in plain language. Mention your project and what is going wrong, because the agent will need a project name to pull spans.

Good triggers include phrases such as optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.

  • "Improve the system prompt for my support bot in the Arize project support-bot, answers are often incorrect."
  • "Find the LLM spans in my rag-app project where the correctness annotation is incorrect and suggest prompt changes."
  • "Make the AI respond better to billing questions; look at the eval explanations in project billing-assistant first."

What the agent does during a run

The skill follows phases. First it extracts the current prompt from LLM spans. It checks input messages first, then input value as a fallback, and checks the prompt template when the app uses one. Next it gathers performance data from traces, datasets and experiments. Then it runs the optimization loop using that evidence.

Because the skill runs ax commands and may call an LLM provider, expect the agent to ask for information it cannot find, such as the project name or a provider key.

  • Phase 1: extract the current prompt and rebuild it as a messages array.
  • Phase 2: gather failures, low grades and eval explanations.
  • Phase 3: propose and test changes against that evidence.
ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout

Safety notes: scripts and network access

The skill runs scripts and uses the network. It sends requests to Arize through ax and may call an LLM provider using OPENAI_API_KEY or ANTHROPIC_API_KEY, or a platform-managed ai-integration. Review the commands the agent proposes before approving them, especially any that send data outside your machine.

The skill instructs the agent never to read .env files or search the filesystem for credentials. It should use ax profiles for Arize credentials and ax ai-integrations for provider keys, and ask you if neither is available.

  • Credentials come from ax profiles and ax ai-integrations, not from .env files.
  • Your trace data is sent through the Arize API, so check it is appropriate to share with the agent session.
  • Check ax ai-integrations list --space SPACE if a provider call fails before adding a key.

Tips for better results

Give the agent a specific project and a specific failure. A named annotation such as correctness with the label incorrect gives it a clear signal to work from. Eval explanations are the richest source, according to the skill's own notes, so use them when they exist.

Keep the dataset and experiment steps in mind if you already have ground-truth examples. Joining them by example_id shows inputs next to outputs and evaluations in one record.

  • Name the project, space and the failing behaviour in one request.
  • Use annotation or eval columns in your request if your team already grades outputs.
  • Ask the agent to show the current prompt before it changes anything, so you can compare.

What people use it for

  • Finding production LLM calls where a human grade of incorrect appears, then proposing system prompt edits for them.
  • Reading eval explanations from an LLM-as-judge to see why a RAG app's answers were scored low.
  • Spotting high-latency LLM spans that may come from an overly long prompt.
  • Comparing a dataset of ground-truth examples with experiment outputs, joined by example_id.

Frequently asked questions

How do I install the arize prompt optimization skill?

Download the skill ZIP from the github/awesome-copilot page for skills/arize-prompt-optimization and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, upload it under Settings → Capabilities → Skills instead.

What do I need before the skill works?

You need the ax CLI, a configured Arize profile with an API key from app.arize.com, and your Arize space name or ID. jq is needed for the JSON examples. An LLM provider key is optional and only needed for running optimization calls.

Is the arize prompt optimization skill free?

The skill is licensed under MIT, as stated on its page. The source does not mention pricing for Arize itself, so check Arize's own terms for account costs.

Can I use it without an Arize account?

No. The skill analyses trace data stored in Arize, so you need an account, an API key and existing traces before it has anything to work with.

Why does the agent say ax command not found?

The ax CLI is not installed or not on your PATH. Install arize-ax-cli with uv, pipx or pip, make sure the version is 0.14.0 or higher, and add the install folder to PATH if needed.

Does the skill read my .env file for API keys?

No. The skill instructs the agent never to read .env files or search the filesystem for credentials. It uses ax profiles for Arize keys and ax ai-integrations for LLM provider keys.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits engineers and prompt engineers who already log LLM traces to Arize and want prompt changes based on production failures and grades. It is a poor fit if you have no Arize account or trace data, or if you only want a quick wording tweak, where a plain chat is simpler.

Download the skill