Skip to content
Get packs

Agent Platform Eval Flywheel

Lets an agent plan, run and iterate on quality evaluations of AI models and agents on Google Cloud

Source Download skill
Repository
google/skills
Stars
21,782
Licence
Apache-2.0
Files
13 files

What it does

Once installed, the agent follows a structured method for measuring how good an AI model or agent is on Google Cloud: building test scenarios and datasets, choosing scoring metrics, running LLM-as-judge evaluations, reading the failures, and comparing results before and after a fix. It also ships helper scripts for validating datasets, inspecting and comparing results, parsing agent traces, rendering HTML reports, and evaluating models served on an endpoint or as Model-as-a-Service. It is built around the Agent Platform GenAI Evaluation SDK and requires explicit user confirmation before any step that consumes paid compute.

Great for

  • Developers building or maintaining a GenAI agent on Google Cloud who want a repeatable quality loop
  • Data or ML engineers comparing model versions before and after a prompt or code change
  • Teams evaluating a model served on their own endpoint or a MaaS model by ID

Skip it if

  • Fine-tuning models (the skill itself points to a separate tuning skill for that)
  • Non-Google Cloud stacks, since it is tied to the Agent Platform SDK and project settings
  • Running evaluations without budget review, because remote evaluation and synthetic scenario generation incur compute costs and the skill requires confirmation for each run

Try asking

  • “Generate 50 synthetic user scenarios for my support agent and build an eval dataset from them”
  • “Run an evaluation of my agent in project my-project, location us-central1, and show me the failing cases”
  • “Compare my baseline and candidate result JSON files and tell me whether the fix regressed any metric”
  • “Evaluate the MaaS model with this model ID against my dataset and render an HTML report”
Step-by-step guide: how to use Agent Platform Eval FlywheelInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 13 original files, the Apache-2.0 licence and an attribution note.

    agent-platform-eval-flywheel.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip agent-platform-eval-flywheel.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip agent-platform-eval-flywheel.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Confirm the project and location up front; the skill will ask for them rather than defaulting to global, so give them explicitly.
  • Read the dry-run preview card carefully before answering Yes, since each evaluation or scenario generation run is billed separately.
  • Save result files to disk with model_dump_json so you can compare runs later with compare_results.py.

What's inside

  • references/dataset_schema.md
  • references/deployment.md
  • references/failure_patterns.md
  • references/metric_registry.md
  • references/sdk_patterns.md
  • scripts/compare_results.py
  • scripts/endpoint_evaluation.py
  • scripts/inspect_results.py
  • scripts/maas_evaluation.py
  • scripts/parse_adk_traces.py
  • scripts/render_html_report.py
  • scripts/validate_dataset.py
  • SKILL.md

Redistributed unmodified from google/skills under the Apache-2.0 licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.