How to use the Arize Annotation Manager agent skill

What an agent skill is and what this one does

An agent skill is a folder with a SKILL.md file and optional reference files. When the skill is installed, an AI agent can read these instructions and use them when a request matches the skill's description.

This skill covers three jobs. It creates and manages annotation configs, which are the label definitions reviewers use, such as correct/incorrect, a 0-10 score, or free-text notes. It sets up annotation queues that route records to human reviewers and lists, updates or deletes them. It also applies annotations to project spans through the Python SDK.

• Annotation configs: categorical, continuous or freeform label schemas
• Annotation queues: human review workflows with linked configs and reviewers
• Bulk span annotation with ArizeClient.spans.update_annotations

Install the arize-annotation skill

Download the ZIP for the skill from this page and unzip it. For a personal install, place the unzipped folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

In the Claude apps you can instead upload the skill under Settings → Capabilities → Skills.

• Personal: ~/.claude/skills/
• Project: .claude/skills/ inside your project folder
• Claude apps: Settings → Capabilities → Skills

Set up the ax CLI and an Arize profile

The skill acts on your Arize account through the ax command-line tool, so ax must be installed and on your PATH. The skill's troubleshooting notes say the version must be 0.14.0 or higher. You also need a configured Arize profile that holds an API key, managed with ax profiles, and the name or ID of your Arize space.

You get an API key from your Arize account at app.arize.com, under the admin area's API Keys section. The skill asks the agent to use ax profiles rather than pasting keys into chat, but the credentials still pass through the agent's session, so treat that as part of your setup decision.

• ax CLI installed and on PATH (version 0.14.0 or higher)
• A configured Arize profile with an API key
• An Arize space name or base64 space ID
• Python SDK (ArizeClient) if you want bulk span annotation

Trigger the skill with natural requests

You do not need to type a command name. Ask for the task in plain words and include one of the skill's trigger phrases where you can, such as annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record or human review.

The agent will run the matching ax command. If a command fails with a credential error, the skill directs the agent to check the profile before going further.

• Create a categorical annotation config called Correctness with correct and incorrect values in my space.
• Set up an annotation queue for human review of responses, assigned to reviewer@example.com.
• Show me the records in the Correctness Review queue.
• Bulk annotate these spans with update_annotations using the labels from our review file.

Example: create a config and a queue

A common first task is to create a label and then route records to reviewers. Create the config first, because a queue needs at least one annotation config ID. The commands below are the ones the skill uses for these steps.

Annotations are saved by config name, and you call the annotate command once per config. Supply at least one of score, label or text for each record.

Safety notes: scripts and network access

This skill runs scripts and uses the network. It sends requests to your Arize account, so the agent acts with the permissions of the credentials in your profile. Deleting a config is irreversible, and deleting a config also removes its queue associations in the product, so confirm before running delete commands.

The skill says the agent must never read .env files or search the filesystem for credentials. Keep your API key in a profile or environment variable, and do not paste it into a chat.

• Deletion of annotation configs cannot be undone
• List flags on queue updates fully replace existing values, so pass every config and reviewer you want to keep
• Review commands before the agent runs them against a production space

Tips for reliable results

Check that the config exists in the space before you expect labels to persist. When you update a queue, repeat every --annotation-config-id and --annotator-email you want to keep, because provided list values replace the old ones.

If you have no profile yet, ask the agent to inspect it first with ax profiles show. The skill's profile notes cover 401 Unauthorized errors and wrong regions.

• Find your space name or ID with ax spaces list
• Use -o json when you need output you can parse
• Use --force only when you have already confirmed the delete