evaluating-code-models
Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill evaluating-code-modelsuvx research-agent-skills install evaluating-code-modelsDocumentation & Instructions
BigCode Evaluation Harness - Code Model Benchmarking
Quick Start
BigCode Evaluation Harness evaluates code generation models across 15+ benchmarks including HumanEval, MBPP, and MultiPL-E (18 languages).
Installation:
git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git
cd bigcode-evaluation-harness
pip install -e .
accelerate config
Evaluate on HumanEval:
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--n_samples 20 \
--batch_size 10 \
--allow_code_execution \
--save_generations
View available tasks:
python -c "from bigcode_eval.tasks import ALL_TASKS; print(ALL_TASKS)"
Common Workflows
Workflow 1: Standard Code Benchmark Evaluation
Evaluate model on core code benchmarks (HumanEval, MBPP, HumanEval+).
Checklist:
Code Benchmark Evaluation:
- [ ] Step 1: Choose benchmark suite
- [ ] Step 2: Configure model and generation
- [ ] Step 3: Run evaluation with code execution
- [ ] Step 4: Analyze pass@k results
Step 1: Choose benchmark suite
Python code generation (most common):
- HumanEval: 164 handwritten problems, function completion
- HumanEval+: Same 164 problems with 80× more tests (stricter)
- MBPP: 500 crowd-sourced problems, entry-level difficulty
- MBPP+: 399 curated problems with 35× more tests
Multi-language (18 languages):
- MultiPL-E: HumanEval/MBPP translated to C++, Java, JavaScript, Go, Rust, etc.
Advanced:
- APPS: 10,000 problems (introductory/interview/competition)
- DS-1000: 1,000 data science problems across 7 libraries
Step 2: Configure model and generation
# Standard HuggingFace model
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--max_length_generation 512 \
--temperature 0.2 \
--do_sample True \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution
# Quantized model (4-bit)
accelerate launch main.py \
--model codellama/CodeLlama-34b-hf \
--tasks humaneval \
--load_in_4bit \
--max_length_generation 512 \
--allow_code_execution
# Custom/private model
accelerate launch main.py \
--model /path/to/my-code-model \
--tasks humaneval \
--trust_remote_code \
--use_auth_token \
--allow_code_execution
Step 3: Run evaluation
# Full evaluation with pass@k estimation (k=1,10,100)
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks humaneval \
--temperature 0.8 \
--n_samples 200 \
--batch_size 50 \
--allow_code_execution \
--save_generations \
--metric_output_path results/starcoder2-humaneval.json
Step 4: Analyze results
Results in results/starcoder2-humaneval.json:
{
"humaneval": {
"pass@1": 0.354,
"pass@10": 0.521,
"pass@100": 0.689
},
"config": {
"model": "bigcode/starcoder2-7b",
"temperature": 0.8,
"n_samples": 200
}
}
Workflow 2: Multi-Language Evaluation (MultiPL-E)
Evaluate code generation across 18 programming languages.
Checklist:
Multi-Language Evaluation:
- [ ] Step 1: Generate solutions (host machine)
- [ ] Step 2: Run evaluation in Docker (safe execution)
- [ ] Step 3: Compare across languages
Step 1: Generate solutions on host
# Generate without execution (safe)
accelerate launch main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--max_length_generation 650 \
--temperature 0.8 \
--n_samples 50 \
--batch_size 50 \
--generation_only \
--save_generations \
--save_generations_path generations_multi.json
Step 2: Evaluate in Docker container
# Pull the MultiPL-E Docker image
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
# Run evaluation inside container
docker run -v $(pwd)/generations_multi.json:/app/generations.json:ro \
-it evaluation-harness-multiple python3 main.py \
--model bigcode/starcoder2-7b \
--tasks multiple-py,multiple-js,multiple-java,multiple-cpp \
--load_generations_path /app/generations.json \
--allow_code_execution \
--n_samples 50
Supported languages: Python, JavaScript, Java, C++, Go, Rust, TypeScript, C#, PHP, Ruby, Swift, Kotlin, Scala, Perl, Julia, Lua, R, Racket
Workflow 3: Instruction-Tuned Model Evaluation
Evaluate chat/instruction models with proper formatting.
Checklist:
Instruction Model Evaluation:
- [ ] Step 1: Use instruction-tuned tasks
- [ ] Step 2: Configure instruction tokens
- [ ] Step 3: Run evaluation
Step 1: Choose instruction tasks
- instruct-humaneval: HumanEval with instruction prompts
- humanevalsynthesize-{lang}: HumanEvalPack synthesis tasks
Step 2: Configure instruction tokens
# For models with chat templates (e.g., CodeLlama-Instruct)
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks instruct-humaneval \
--instruction_tokens "<s>[INST],</s>,[/INST]" \
--max_length_generation 512 \
--allow_code_execution
Step 3: HumanEvalPack for instruction models
# Test code synthesis across 6 languages
accelerate launch main.py \
--model codellama/CodeLlama-7b-Instruct-hf \
--tasks humanevalsynthesize-python,humanevalsynthesize-js \
--prompt instruct \
--max_length_generation 512 \
--allow_code_execution
Workflow 4: Compare Multiple Models
Benchmark suite for model comparison.
Step 1: Create evaluation script
#!/bin/bash
# eval_models.sh
MODELS=(
"bigcode/starcoder2-7b"
"codellama/CodeLlama-7b-hf"
"deepseek-ai/deepseek-coder-6.7b-base"
)
TASKS="humaneval,mbpp"
for model in "${MODELS[@]}"; do
model_name=$(echo $model | tr '/' '-')
echo "Evaluating $model"
accelerate launch main.py \
--model $model \
--tasks $TASKS \
--temperature 0.2 \
--n_samples 20 \
--batch_size 20 \
--allow_code_execution \
--metric_output_path results/${model_name}.json
done
Step 2: Generate comparison table
import json
import pandas as pd
models = ["bigcode-starcoder2-7b", "codellama-CodeLlama-7b-hf", "deepseek-ai-deepseek-coder-6.7b-base"]
results = []
for model in models:
with open(f"results/{model}.json") as f:
data = json.load(f)
results.append({
"Model": model,
"HumanEval pass@1": f"{data['humaneval']['pass@1']:.3f}",
"MBPP pass@1": f"{data['mbpp']['pass@1']:.3f}"
})
df = pd.DataFrame(results)
print(df.to_markdown(index=False))
When to Use vs Alternatives
Use BigCode Evaluation Harness when:
- Evaluating code generation models specifically
- Need multi-language evaluation (18 languages via MultiPL-E)
- Testing functional correctness with unit tests (pass@k)
- Benchmarking for BigCode/HuggingFace leaderboards
- Evaluating fill-in-the-middle (FIM) capabilities
Use alternatives instead:
- lm-evaluation-harness: General LLM benchmarks (MMLU, GSM8K, HellaSwag)
- EvalPlus: Stricter HumanEval+/MBPP+ with more test cases
- SWE-bench: Real-world GitHub issue resolution
- LiveCodeBench: Contamination-free, continuously updated problems
- CodeXGLUE: Code understanding tasks (clone detection, defect prediction)
Supported Benchmarks
| Benchmark | Problems | Languages | Metric | Use Case |
|---|---|---|---|---|
| HumanEval | 164 | Python | pass@k | Standard code completion |
| HumanEval+ | 164 | Python | pass@k | Stricter evaluation (80× tests) |
| MBPP | 500 | Python | pass@k | Entry-level problems |
| MBPP+ | 399 | Python | pass@k | Stricter evaluation (35× tests) |
| MultiPL-E | 164×18 | 18 languages | pass@k | Multi-language evaluation |
| APPS | 10,000 | Python | pass@k | Competition-level |
| DS-1000 | 1,000 | Python | pass@k | Data science (pandas, numpy, etc.) |
| HumanEvalPack | 164×3×6 | 6 languages | pass@k | Synthesis/fix/explain |
| Mercury | 1,889 | Python | Efficiency | Computational efficiency |
Common Issues
Issue: Different results than reported in papers
Check these factors:
# 1. Verify n_samples (need 200 for accurate pass@k)
--n_samples 200
# 2. Check temperature (0.2 for greedy-ish, 0.8 for sampling)
--temperature 0.8
# 3. Verify task name matches exactly
--tasks humaneval # Not "human_eval" or "HumanEval"
# 4. Check max_length_generation
--max_length_generation 512 # Increase for longer problems
Issue: CUDA out of memory
# Use quantization
--load_in_8bit
# OR
--load_in_4bit
# Reduce batch size
--batch_size 1
# Set memory limit
--max_memory_per_gpu "20GiB"
Issue: Code execution hangs or times out
Use Docker for safe execution:
# Generate on host (no execution)
--generation_only --save_generations
# Evaluate in Docker
docker run ... --allow_code_execution --load_generations_path ...
Issue: Low scores on instruction models
Ensure proper instruction formatting:
# Use instruction-specific tasks
--tasks instruct-humaneval
# Set instruction tokens for your model
--instruction_tokens "<s>[INST],</s>,[/INST]"
Issue: MultiPL-E language failures
Use the dedicated Docker image:
docker pull ghcr.io/bigcode-project/evaluation-harness-multiple
Command Reference
| Argument | Default | Description |
|---|---|---|
--model |
- | HuggingFace model ID or local path |
--tasks |
- | Comma-separated task names |
--n_samples |
1 | Samples per problem (200 for pass@k) |
--temperature |
0.2 | Sampling temperature |
--max_length_generation |
512 | Max tokens (prompt + generation) |
--batch_size |
1 | Batch size per GPU |
--allow_code_execution |
False | Enable code execution (required) |
--generation_only |
False | Generate without evaluation |
--load_generations_path |
- | Load pre-generated solutions |
--save_generations |
False | Save generated code |
--metric_output_path |
results.json | Output file for metrics |
--load_in_8bit |
False | 8-bit quantization |
--load_in_4bit |
False | 4-bit quantization |
--trust_remote_code |
False | Allow custom model code |
--precision |
fp32 | Model precision (fp32/fp16/bf16) |
Hardware Requirements
| Model Size | VRAM (fp16) | VRAM (4-bit) | Time (HumanEval, n=200) |
|---|---|---|---|
| 7B | 14GB | 6GB | ~30 min (A100) |
| 13B | 26GB | 10GB | ~1 hour (A100) |
| 34B | 68GB | 20GB | ~2 hours (A100) |
Resources
- GitHub: https://github.com/bigcode-project/bigcode-evaluation-harness
- Documentation: https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs
- BigCode Leaderboard: https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard
- HumanEval Dataset: https://huggingface.co/datasets/openai/openai_humaneval
- MultiPL-E: https://github.com/nuprl/MultiPL-E
Agent operating procedure
- Check the environment. Confirm the model, checkpoint, benchmark version and evaluation harness version.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Evaluate on a small subset to verify prompts, parsing and scoring.
- Execute the full task using the instructions and references above.
- Validate the result. Use the benchmark's official splits and metrics; check for contamination; report variance across seeds or prompts.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| Scores differ from published numbers | Check prompt format, few-shot settings and harness version before concluding. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Report evaluation settings in full; never cherry-pick prompts, seeds or subsets.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
evaluating-llms-harness: Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag).nemo-evaluator-sdk: Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects evaluating-code-models for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“Run HumanEval on my StarCoder2-7B checkpoint and give me pass@1.”
expects Uses the BigCode evaluation harness with the humaneval task and the --allow_code_execution flag. Runs a small subset first to check prompts and scoring. Reports the harness version, model, temperature, n_samples and the pass@1 result, without making up numbers.
- should trigger
“I've got a fine-tuned model at ./ckpt/mycoder-13b and I need pass@10 on HumanEval and MBPP for a paper table. I'm on a single A100 40GB and the last run hit CUDA out of memory with batch_size 8. Published numbers for the base model are about 5 points higher than what I get. How do I set this up properly?”
expects Recommends n_samples of about 200 with temperature around 0.2 to 0.8 for pass@10. Fixes the OOM by lowering batch size or using fp16/bf16 or 4-bit loading. Explains the gap to published numbers by pointing to prompt format, max_length_generation, stop tokens, precision and harness version. Advises sandboxing code execution (e.g. Docker), reporting full settings, and confirming flags against the installed version.
- should trigger
“We want to know whether our code LLM is just as good at Rust, Go and Java as it is at Python. What's the best way to measure that and compare the languages fairly?”
expects Suggests MultiPL-E, which translates HumanEval/MBPP into many languages, with generation on the host and execution in the dedicated Docker image. Compares per-language pass@k with the same sampling settings and notes that translated problems are not perfectly equivalent across languages. Reports the settings used.
Near misses (2)
- near miss
“Evaluate my Llama-3-8B-Instruct on MMLU, GSM8K and HellaSwag with 5-shot prompts and give me a results table.”
expects Recognizes these are general LLM benchmarks, not code generation. Should use the evaluating-llms-harness skill (lm-evaluation-harness) and not the code-model harness.
- near miss
“I need to run our model on 100+ benchmarks across Slurm on our HPC cluster using containers, including safety and VLM suites, with reproducible configs.”
expects Recognizes this as large-scale multi-harness, multi-backend evaluation. Should use nemo-evaluator-sdk and not the BigCode harness, which targets code generation benchmarks.