Agent skill · research science · davila7
evaluating-code-models
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
Why this skill is useful
Provides domain-specific evaluation metrics and workflows for benchmarking code generation models that the AI wouldn't reliably generate on its own.
What it needs
Requires accelerate, bigcode-evaluation-harness, datasets, transformers installed locally. About 6k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
BigCode Evaluation Harness - Code Model Benchmarking Quick Start BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages). Installation: Evaluate on HumanEval: View available tasks: Common Workflows Workflow 1: Standard Code Benchmark Evaluation Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+). Checklist: Step 1: Choose benchmark suite Python code generation (most common): HumanEval: 164 handwritten problems, function completion HumanEval+: Same 164 problems with 80× more tests (stricter) MBPP: 500 crowd-sourced problems, entry-level difficulty MBPP+: 399 curated problems with 35× more tests Multi-language (18 languages): MultiPL-E: HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc. Advanced: APPS: 10,000 problems (introductory/interview/competition) DS-1000: 1,000 data science problems across 7 libraries Step 2: Configure model and generation Step 3: Run evaluation Step 4: Analyze results Results in results/starcoder2-humaneval.json: Workflow 2: Multi-Language Evaluation (MultiPL-E) Evaluate code generation across 18 programming languages. Checklist: Step 1: Generate solutions on host Step 2: Evaluate in Docker container Supported languages: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket Workflow 3: Instruction-Tuned Model Evaluation Evaluate chat/instruction models with proper formatting. Checklist: Step 1: Choose instruction tasks instruct-humaneval: HumanEval with instruction prompts humanevalsynthesize-{lang}: HumanEvalPack synthesis tasks Step 2: Configure instruction tokens Step 3: HumanEvalPack for instruction models Workflow 4: Compare Multiple Models Benchmark suite for model comparison. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/evaluation-bigcode-evaluation-harness