Skip to content
Get packs

BigCode Code Model Benchmarking

Benchmark code generation models on HumanEval, MBPP, MultiPL-E and more using pass@k metrics

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

Teaches the agent how to run the BigCode Evaluation Harness, a widely used open-source test suite that measures how well AI code models write working code. Once installed, the agent can set up the harness, pick benchmarks such as HumanEval (Python) or MultiPL-E (18 languages), launch the evaluation with the right flags, and read the pass@k scores it produces. The skill also includes reference notes on benchmarks, custom tasks and known issues.

Great for

  • ML engineers comparing several code models on the same benchmarks
  • Researchers who need reproducible pass@k numbers for a paper or report
  • Developers checking whether a fine-tuned code model regressed on coding tasks

Skip it if

  • Non-technical users who only want to chat with a code model; this needs a GPU, Python setup and a working model download
  • Running untrusted generated code on a personal machine without a sandbox, since the harness executes model output when code execution is enabled

Try asking

  • “Evaluate bigcode/starcoder2-7b on HumanEval with pass@1, pass@10 and pass@100”
  • “Compare three code models on HumanEval and MBPP and build a results table”
  • “Test a model's code generation across Python, JavaScript, Java and C++ using MultiPL-E”
  • “Set up a Docker-based safe run for MultiPL-E results”
Step-by-step guide: how to use BigCode Code Model BenchmarkingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    bigcode-evaluation-harness.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip bigcode-evaluation-harness.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip bigcode-evaluation-harness.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run generation on the host and execution inside Docker, as the skill recommends, to limit the risk of running model-written code
  • Use --trust_remote_code only with model sources you trust, since it runs code shipped with the model repository
  • Check the harness's own issues and documentation for version-specific flags before large runs

What's inside

  • references/benchmarks.md
  • references/custom-tasks.md
  • references/issues.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.