Agent Eval (コーディングエージェント比較)

What it does

This skill teaches the agent to set up and run agent-eval, a small command-line tool that runs several AI coding assistants on the same task and scores them. You define a task in a YAML file with a starting commit, a prompt and judge checks such as running pytest or grep, then the tool records whether each agent passed, what it cost, how long it took, and how consistent it was across repeated runs. Each run happens in its own git worktree so runs do not interfere with each other or the base repository.

Try asking

• “Compare Claude Code and Aider on a task that adds retry logic to my HTTP client, running each three times”Copy
• “Write a YAML task definition for a bug fix in my Python project, pinned to a specific commit”Copy
• “Generate a comparison table of pass rate, cost and time for my last three agent runs”Copy
• “Help me design a set of five realistic tasks to evaluate coding agents on my team's codebase”Copy

Get better results

• Start with three to five tasks drawn from real work rather than toy examples, and run each agent at least three times because results vary
• Include at least one deterministic judge such as tests or a build in every task, and treat LLM judges as added noise rather than ground truth
• Track cost alongside pass rate, since a much more expensive agent may not be the right choice, and pin commits in task files for reproducibility

What's inside

• SKILL.md