Benchmark code generation models

Benchmark code generation models across HumanEval, MBPP, and 15+ benchmarks with pass@k metrics

Comparing code model performance across standard benchmarks when new architecture or training method is evaluated

• · model identifier (HF model or API)
• · benchmark suite (HumanEval, MBPP, MultiPL-E, etc.)
• · number of samples k

• · pass@k score
• · pass@1/10/100
• · per-problem failure analysis

• · evalplus
• · humaneval
• · transformers

Model weights accessible or API token valid; benchmark data downloaded; Python environment set up

API rate limits if k too high; timeout if model inference slow; benchmark data corrupted

• · HumanEval and MBPP are standard code benchmarks
• · pass@k metric prevents inference cost gaming
• · MultiPL-E enables cross-language comparison