BigCodeBench
BigCodeBench evaluates code generation on practical and instruction-rich programming tasks, reporting pass@1 in complete and instruct settings.
126rows
instruct_pass_at_1primary metric
2026-05-06sampled
Metadata
Metrics
Instruct pass@1, Complete pass@1, Average pass@1
| Rank | Subject | Instruct pass@1 | Model Match | Provenance | Sampled |
|---|---|---|---|---|---|
| 1 | GPT-4o-2024-05-13 | 51.10 |  GPT-4o (2024-05-13) openai-gpt-4o-2024-05-13 | Imported | 2026-05-06 |
| 2 | DeepSeek-V3 | 50 |  DeepSeek V3 deepseek-deepseek-chat | Imported | 2026-05-06 |
| 3 | Llama-4-Maverick | 49.70 |  Llama 4 Maverick meta-llama-4-maverick | Imported | 2026-05-06 |
| 4 | Quasar-Alpha | 49.60 | — | Imported | 2026-05-06 |
| 5 | Gemini-Exp-1114 | 49.20 | — | Imported | 2026-05-06 |
| 6 | Qwen2.5-Coder-32B-Instruct | 49 |  Qwen2.5 Coder 32B Instruct qwen-qwen-2.5-coder-32b-instruct | Imported | 2026-05-06 |
| 7 | DeepSeek-V2-Chat (2024-06-28) | 48.90 | — | Imported | 2026-05-06 |
| 8 | GPT-4.1-Mini-2025-04-14 | 48.90 | GPT-4.1 Mini openai-gpt-4.1-mini | Imported | 2026-05-06 |
| 9 | DeepSeek-V2.5-1210 | 48.60 | — | Imported | 2026-05-06 |
| 10 | DeepSeek-Coder-V2-Instruct | 48.20 | — | Imported | 2026-05-06 |
| 11 | GPT-4-Turbo-2024-04-09 | 48.20 | GPT-4 Turbo openai-gpt-4-turbo | Imported | 2026-05-06 |
| 12 | Qwen2.5-Coder-14B-Instruct | 48.20 | — | Imported | 2026-05-06 |
| 13 | GPT-4o-2024-11-20 | 48 | GPT-4o (2024-11-20) openai-gpt-4o-2024-11-20 | Imported | 2026-05-06 |
| 14 | Athene-V2-Chat | 47.20 | — | Imported | 2026-05-06 |
| 15 | Gemini-Exp-1206 | 47 | — | Imported | 2026-05-06 |
| 16 | Llama-3.3-70B-Instruct | 46.90 | Llama 3.3 70B Instruct meta-llama-llama-3.3-70b-instruct | Imported | 2026-05-06 |
| 17 | Claude-3.5-Sonnet-20240620 | 46.80 |  Claude 3.5 Sonnet anthropic-claude-3.5-sonnet | Imported | 2026-05-06 |
| 18 | Athene-V2-Agent | 46.20 | — | Imported | 2026-05-06 |
| 19 | Claude-3.5-Haiku-20241022 | 46.10 | — | Imported | 2026-05-06 |
| 20 | GPT-4o-mini-2024-07-18 | 46.10 | GPT-4o-mini (2024-07-18) openai-gpt-4o-mini-2024-07-18 | Imported | 2026-05-06 |
| 21 | Llama-3.1-70B-Instruct | 46.10 | Llama 3.1 70B Instruct meta-llama-llama-3.1-70b-instruct | Imported | 2026-05-06 |
| 22 | GPT-4-0613 | 46 | GPT-4 openai-gpt-4 | Imported | 2026-05-06 |
| 23 | Gemini-2.0-Flash-Exp | 45.90 | — | Imported | 2026-05-06 |
| 24 | Qwen2.5-72B-Instruct | 45.80 | Qwen2.5 72B Instruct qwen-qwen-2.5-72b-instruct | Imported | 2026-05-06 |
| 25 | Hermes-2-Theta-Llama-3-70B | 45.60 | — | Imported | 2026-05-06 |
| 26 | Claude-3-Opus-20240229 | 45.50 | — | Imported | 2026-05-06 |
| 27 | Phi-4 | 45.50 |  Phi 4 microsoft-phi-4 | Imported | 2026-05-06 |
| 28 | Gemini-Exp-1121 | 45.40 | — | Imported | 2026-05-06 |
| 29 | Mistral-Small-24B-Instruct-2501 | 45.30 |  Mistral: Mistral Small 3 mistralai-mistral-small-24b-instruct-2501 | Imported | 2026-05-06 |
| 30 | Sky-T1-32B-Flash | 45.10 | — | Imported | 2026-05-06 |
| 31 | Qwen2.5-32B-Instruct | 45 | — | Imported | 2026-05-06 |
| 32 | Sky-T1-32B-Preview | 44.90 | — | Imported | 2026-05-06 |
| 33 | Claude-3.5-Sonnet-20241022 | 44.60 | Claude 3.5 Sonnet anthropic-claude-3.5-sonnet | Imported | 2026-05-06 |
| 34 | QwQ-32B-Preview | 44.60 | — | Imported | 2026-05-06 |
| 35 | DeepSeek-R1-Distill-Qwen-32B | 43.90 | R1 Distill Qwen 32B deepseek-deepseek-r1-distill-qwen-32b | Imported | 2026-05-06 |
| 36 | Gemini-1.5-Pro-API-0514 | 43.80 | — | Imported | 2026-05-06 |
| 37 | Llama-3-70B-Instruct | 43.60 | Llama 3 70B Instruct meta-llama-llama-3-70b-instruct | Imported | 2026-05-06 |
| 38 | Gemini-1.5-Flash-API-0514 | 43.50 | — | Imported | 2026-05-06 |
| 39 | OpenCoder-8B-Instruct | 43.20 | — | Imported | 2026-05-06 |
| 40 | Gemma-2-27B-Instruct | 42.80 | — | Imported | 2026-05-06 |
| 41 | Llama-3-70B-Synthia-v3.5 | 42.80 | — | Imported | 2026-05-06 |
| 42 | Claude-3-Sonnet-20240229 | 42.70 | — | Imported | 2026-05-06 |
| 43 | ReflectionCoder-DS-33B | 42.40 | — | Imported | 2026-05-06 |
| 44 | DeepSeek-Coder-33B-Instruct | 42 | — | Imported | 2026-05-06 |
| 45 | Codestral-22B-v0.1 | 41.80 | — | Imported | 2026-05-06 |
| 46 | WhiteRabbitNeo-33B-v1.5 | 41.70 | — | Imported | 2026-05-06 |
| 47 | AutoCoder | 40.70 | — | Imported | 2026-05-06 |
| 48 | CodeLlama-70B-Instruct | 40.70 | — | Imported | 2026-05-06 |
| 49 | Mixtral-8x22B-Instruct | 40.60 | Mistral: Mixtral 8x22B Instruct mistralai-mixtral-8x22b-instruct | Imported | 2026-05-06 |
| 50 | DeepSeek-V2-Chat | 40.40 | — | Imported | 2026-05-06 |
| 51 | Qwen2.5-Coder-7B-Instruct | 40.40 | — | Imported | 2026-05-06 |
| 52 | CodeGeex4-All-9B | 40 | — | Imported | 2026-05-06 |
| 53 | WarriorCoder-6.7B (Reproduced) | 39.90 | — | Imported | 2026-05-06 |
| 54 | Qwen2.5-14B-Instruct | 39.80 | — | Imported | 2026-05-06 |
| 55 | CodeQwen1.5-7B-Chat | 39.60 | — | Imported | 2026-05-06 |
| 56 | Nxcode-CQ-7B-Orpo | 39.60 | — | Imported | 2026-05-06 |
| 57 | Claude-3-Haiku-20240307 | 39.40 | Claude 3 Haiku anthropic-claude-3-haiku | Imported | 2026-05-06 |
| 58 | GPT-3.5-Turbo-0125 | 39.10 | GPT-3.5 Turbo openai-gpt-3.5-turbo | Imported | 2026-05-06 |
| 59 | Llama-3.1-Nemotron-70B-Instruct | 38.70 |  Llama 3.1 Nemotron 70B Instruct nvidia-llama-3.1-nemotron-70b-instruct | Imported | 2026-05-06 |
| 60 | Qwen2-72B-Chat | 38.50 | — | Imported | 2026-05-06 |
| 61 | DeepCoder-14B-Preview | 38.20 | — | Imported | 2026-05-06 |
| 62 | Phind-CodeLlama-34B-v2 | 38.20 | — | Imported | 2026-05-06 |
| 63 | DeepSeek-R1-Distill-Qwen-14B | 38.10 | — | Imported | 2026-05-06 |
| 64 | Yi-Coder-9B-Chat | 38.10 | — | Imported | 2026-05-06 |
| 65 | Artigenz-Coder-DS-6.7B | 38 | — | Imported | 2026-05-06 |
| 66 | ReflectionCoder-CL-34B | 37.70 | — | Imported | 2026-05-06 |
| 67 | Yi-Large | 37.70 | — | Imported | 2026-05-06 |
| 68 | Phi-3-Medium-128K-Instruct | 37.60 | — | Imported | 2026-05-06 |
| 69 | Qwen2.5-7B-Instruct | 37.60 | Qwen2.5 7B Instruct qwen-qwen-2.5-7b-instruct | Imported | 2026-05-06 |
| 70 | StarCoder2-15B-Instruct-v0.1 | 37.60 | — | Imported | 2026-05-06 |
| 71 | Hermes-2-Pro-Llama-3-70B | 37.20 | — | Imported | 2026-05-06 |
| 72 | C4AI-Command-R-08-2024 | 37.10 | — | Imported | 2026-05-06 |
| 73 | OpenCodeInterpreter-DS-6.7B | 37.10 | — | Imported | 2026-05-06 |
| 74 | Tess-v2.5.2-Qwen2-72B | 37 | — | Imported | 2026-05-06 |
| 75 | Athene-70B | 36.80 | — | Imported | 2026-05-06 |
| 76 | DeepSeek-Coder-V2-Lite-Instruct | 36.80 | — | Imported | 2026-05-06 |
| 77 | Phi-3.1-Mini-128K-Instruct | 36.80 | — | Imported | 2026-05-06 |
| 78 | ReflectionCoder-DS-6.7B | 36.80 | — | Imported | 2026-05-06 |
| 79 | Magicoder-S-DS-6.7B | 36.20 | — | Imported | 2026-05-06 |
| 80 | AutoCoder-S-6.7B | 36.10 | — | Imported | 2026-05-06 |
| 81 | Granite-Code-34B-Instruct | 36.10 | — | Imported | 2026-05-06 |
| 82 | Mistral-Small-Instruct-2409 | 36.10 | — | Imported | 2026-05-06 |
| 83 | Qwen2-57B-A14B | 36.10 | — | Imported | 2026-05-06 |
| 84 | Codestral-Mamba | 35.90 | — | Imported | 2026-05-06 |
| 85 | DeepSeek-Coder-6.7B-Instruct | 35.50 | — | Imported | 2026-05-06 |
| 86 | DeepSeek-R1-Distill-Llama-70B | 35.30 | R1 Distill Llama 70B deepseek-deepseek-r1-distill-llama-70b | Imported | 2026-05-06 |
| 87 | Qwen1.5-110B-Chat | 35 | — | Imported | 2026-05-06 |
| 88 | OpenCoder-1.5B-Instruct | 34.90 | — | Imported | 2026-05-06 |
| 89 | Gemma-2-9B-Instruct | 34.70 | — | Imported | 2026-05-06 |
| 90 | Yi-1.5-9B-Chat | 34.50 | — | Imported | 2026-05-06 |
| 91 | Granite-Code-20B-Instruct | 34 | — | Imported | 2026-05-06 |
| 92 | WaveCoder-Ultra-6.7B | 33.90 | — | Imported | 2026-05-06 |
| 93 | Yi-1.5-34B-Chat | 33.90 | — | Imported | 2026-05-06 |
| 94 | Command R+ | 33.80 | Command R (08-2024) cohere-command-r-08-2024 | Imported | 2026-05-06 |
| 95 | Qwen1.5-72B-Chat | 33.20 | — | Imported | 2026-05-06 |
| 96 | Llama-3.1-8B-Instruct | 32.80 | Llama 3.1 8B Instruct meta-llama-llama-3.1-8b-instruct | Imported | 2026-05-06 |
| 97 | Phi-3.5-Mini-Instruct | 32.80 | — | Imported | 2026-05-06 |
| 98 | CodeGemma-7B-Instruct | 32.30 | — | Imported | 2026-05-06 |
| 99 | Qwen1.5-32B-Chat | 32.30 | — | Imported | 2026-05-06 |
| 100 | AutoCoder-QW-7B | 32.20 | — | Imported | 2026-05-06 |
| 101 | Mistral-Small-2402 | 32.10 | — | Imported | 2026-05-06 |
| 102 | Llama-3-8B-Instruct | 31.90 | Llama 3 8B Instruct meta-llama-llama-3-8b-instruct | Imported | 2026-05-06 |
| 103 | Phi-3-Small-128K-Instruct | 31.10 | — | Imported | 2026-05-06 |
| 104 | Mistral-Large-2402 | 30 | — | Imported | 2026-05-06 |
| 105 | Phi-3-Mini-128K-Instruct | 29.60 | — | Imported | 2026-05-06 |
| 106 | Granite-3.0-8B-Instruct | 29.30 | — | Imported | 2026-05-06 |
| 107 | Qwen2-7B-Instruct | 29.10 | — | Imported | 2026-05-06 |
| 108 | CodeLlama-34B-Instruct | 29 | — | Imported | 2026-05-06 |
| 109 | CodeLlama-13B-Instruct | 28.50 | — | Imported | 2026-05-06 |
| 110 | ReflectionCoder-CL-7B | 28.40 | — | Imported | 2026-05-06 |
| 111 | OpenChat-3.6-8B-20240522 | 28.10 | — | Imported | 2026-05-06 |
| 112 | Qwen2.5-Coder-1.5B-Instruct | 27 | — | Imported | 2026-05-06 |
| 113 | InternLM2.5-7B-Chat | 25.80 | — | Imported | 2026-05-06 |
| 114 | Yi-1.5-6B-Chat | 25.60 | — | Imported | 2026-05-06 |
| 115 | OpenCodeInterpreter-DS-1.3B | 25.30 | — | Imported | 2026-05-06 |
| 116 | Llama-3.2-3B-Instruct | 23.40 | Llama 3.2 3B Instruct meta-llama-llama-3.2-3b-instruct | Imported | 2026-05-06 |
| 117 | DeepSeek-Coder-1.3B-Instruct | 22.80 | — | Imported | 2026-05-06 |
| 118 | CodeLlama-7B-Instruct | 21.90 | — | Imported | 2026-05-06 |
| 119 | Granite-3.0-2B-Instruct | 20.50 | — | Imported | 2026-05-06 |
| 120 | Qwen2.5-1.5B-Instruct | 20.30 | — | Imported | 2026-05-06 |
| 121 | Mistral-7B-Instruct-v0.3 | 19.50 | — | Imported | 2026-05-06 |
| 122 | DeepSeek-R1-Distill-Qwen-7B | 17.50 | — | Imported | 2026-05-06 |
| 123 | DeepSeek-R1-Distill-Llama-8B | 10.60 | — | Imported | 2026-05-06 |
| 124 | Qwen2.5-0.5B-Instruct | 8.80 | — | Imported | 2026-05-06 |
| 125 | Llama-3.2-1B-Instruct | 8.20 | Llama 3.2 1B Instruct meta-llama-llama-3.2-1b-instruct | Imported | 2026-05-06 |
| 126 | DeepSeek-R1-Distill-Qwen-1.5B | 7 | — | Imported | 2026-05-06 |
No matching rows.