Arize Evaluator Skill
Lets an agent design, create, and run LLM-as-judge evaluators and tasks on Arize using the ax CLI.
- Repository
- github/awesome-copilot
- Stars
- 40,703
- Licence
- MIT
- Files
- 3 files
What it does
This skill teaches an AI agent how to set up automated quality checks on AI applications hosted on the Arize observability platform. An 'evaluator' is a judge prompt that asks another AI model to grade outputs (for example, whether an answer is hallucinated), and a 'task' runs those judges against real traces, spans, or experiment results, including ongoing monitoring of new data. The agent uses the ax command-line tool to do this and is instructed to report failed evaluations honestly rather than inventing scores.
Great for
- Engineers who run LLM applications on Arize and want automated hallucination or correctness scoring
- Teams setting up continuous monitoring of production traces with sampled evaluation
- ML or data staff who need to map span fields to evaluator template variables
Skip it if
- People without an Arize account or the ax CLI, since the skill cannot do anything useful without them
- Anyone expecting a general-purpose evaluation method; it only drives Arize's own evaluation system and judge models that the user pays for
- Situations involving sensitive data sent to third-party LLM providers, because span content is passed to the judge model and may leave your environment
Try asking
- “Create an LLM-as-judge evaluator that checks whether answers are faithful to the retrieved context”
- “Run a hallucination evaluator over the last week of spans in my production project”
- “Set up continuous monitoring with a 10% sampling rate on new LLM spans”
- “Score an experiment's runs for correctness and map the input and output columns”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 3 original files, the MIT licence and an attribution note.
arize-evaluator.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip arize-evaluator.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip arize-evaluator.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Provide the Arize space name or ID up front so the agent does not have to search for it
- Keep the judge temperature at 0 and use clear binary labels for more reproducible scores
- Check the query filter on tasks so evaluations run only on the spans you intend, to control cost and data exposure
What's inside
- references/ax-profiles.md
- references/ax-setup.md
- SKILL.md
Redistributed unmodified from github/awesome-copilot under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.