HumanEval Leaderboard

Python function synthesis from docstrings. · Prices checked November 17, 2026

HumanEval asks a model to write a self-contained Python function from its docstring, then runs the function against hidden unit tests. It measures code generation in isolation, no repository, no dependencies, no existing tests to satisfy.

Leaderboard

60 of 132 scored

LeaderboardScores as published by each benchmark. Prices checked November 17, 2026.

o4-mini leads on HumanEval at 97.3%. The median across the 132 models we hold a score for is 79%. The cheapest model on this board is Nova Micro at $0.0665/M blended, scoring 81.1%.

#ModelProviderScoreInput /1MOutput /1MBlended 70/30Context
1o4-miniOpenAI97.3%$1.10$4.40$2.09200,000
2o3-miniOpenAI96.3%$1.10$4.40$2.09200,000
3GPT-4.1OpenAI94.5%$2.00$8.00$3.801,000,000
4GPT-4.1 miniOpenAI93.8%$0.4000$1.60$0.76001,000,000
5Claude Sonnet 3.5Anthropic93.7%$3.00$15.00$6.60200,000
6Qwen2.5-Coder-32B (Groq)Groq92.7%$0.7900$0.7900$0.7900128,000
7Qwen2.5-Coder-32BQwen92.7%$0.7000$0.7000$0.7000128,000
8Mistral Large 2Mistral92%$2.00$6.00$3.20128,000
9Mistral-Large-2 (NIM)Nvidia92%$2.00$6.00$3.20128,000
10GPT-4oOpenAI90.2%$2.50$10.00$4.75128,000
11DeepSeek-Coder-V2DeepSeek90.2%$0.1400$0.2800$0.1820128,000
12Qwen2.5-MaxQwen90%$1.60$6.40$3.0432,000
13Llama 3.1 405BMeta89%$0.8000$0.8000$0.8000128,000
14Nova ProAmazon89%$0.8000$3.20$1.52300,000
15Llama 3.1 405B (Groq)Groq89%$2.99$2.99$2.99128,000
16Llama 3.1 405B (Together)Together AI89%$3.50$3.50$3.50128,000
17Llama 3.1 405B (fw)Fireworks AI89%$3.00$3.00$3.00131,000
18Llama 3.1 405B (Rep)Replicate89%$9.50$9.50$9.50128,000
19Llama 3.1 405B (DI)DeepInfra89%$0.8000$0.8000$0.8000128,000
20Llama-3.1-405B (Cerebras)Cerebras89%$0.9900$0.9900$0.9900128,000
21Llama 3.1 405B (Hyp)Hyperbolic89%$4.00$4.00$4.00128,000
22Qwen2.5-Coder-14BQwen89%$0.3500$0.3500$0.3500128,000
23Grok 2xAI88.4%$2.00$10.00$4.40131,000
24Grok 2 VisionxAI88.4%$2.00$10.00$4.4032,000
25Llama 3.3 70BMeta88.4%$0.2300$0.4000$0.2810128,000
26Llama 3.3 70B (Groq)Groq88.4%$0.5900$0.7900$0.6500128,000
27Llama 3.3 70B (Together)Together AI88.4%$0.8800$0.8800$0.8800128,000
28Llama 3.3 70B (DI)DeepInfra88.4%$0.2300$0.4000$0.2810128,000
29Llama-3.3-70B (Cerebras)Cerebras88.4%$0.5900$0.9900$0.7100128,000
30Qwen2.5-32B-InstructQwen88.4%$0.7000$0.7000$0.7000128,000
31GPT-4 TurboOpenAI88.2%$10.00$30.00$16.00128,000
32Claude Haiku 3.5Anthropic88.1%$0.8000$4.00$1.76200,000
33o3OpenAI87.4%$2.00$8.00$3.80200,000
34GPT-4o miniOpenAI87.2%$0.1500$0.6000$0.2850128,000
35GPT-4.1 nanoOpenAI87%$0.1000$0.4000$0.19001,000,000
36MiniMax-Text-01MiniMax86.9%$0.7000$2.80$1.331,000,000
37Qwen2.5-72B (Groq)Groq86.6%$0.7900$0.7900$0.7900128,000
38Qwen2.5-72B (Together)Together AI86.6%$1.20$1.20$1.2032,000
39Qwen2.5-72B (fw)Fireworks AI86.6%$0.9000$0.9000$0.900032,000
40Qwen2.5-72B (DI)DeepInfra86.6%$0.3500$0.4000$0.3650128,000
41Qwen2.5-72B (Hyp)Hyperbolic86.6%$0.4000$0.4000$0.4000128,000
42Qwen2.5-72B-InstructQwen86.6%$0.3600$0.4000$0.3720128,000
43Qwen2-72B-InstructQwen86%$0.9000$0.9000$0.9000128,000
44Nova LiteAmazon85.4%$0.0600$0.2400$0.1140300,000
45Command ACohere85.4%$2.50$10.00$4.75256,000
46CodestralMistral85%$0.2000$0.6000$0.3200256,000
47Qwen2.5-Coder-7BQwen85%$0.1000$0.1000$0.1000128,000
48Qwen2.5-7B-InstructQwen84.8%$0.1000$0.1000$0.1000128,000
49Gemini 1.5 ProGoogle84%$1.25$5.00$2.382,000,000
50Qwen2.5-14B-InstructQwen83.5%$0.3500$0.3500$0.3500128,000
51Yi-Lightning01.AI83.5%$0.1400$0.1400$0.140016,000
52Phi-4Microsoft82.6%$0.0700$0.1400$0.091016,000
53Yi-Large01.AI82.3%$3.00$3.00$3.0032,000
54Nova MicroAmazon81.1%$0.0350$0.1400$0.0665128,000
55Llama 3 70BMeta81%$0.5900$0.7900$0.65008,000
56Llama 3.1 70BMeta80.5%$0.3500$0.4000$0.3650128,000
57Llama 3.1 70B (Groq)Groq80.5%$0.5900$0.7900$0.6500128,000
58Llama 3.1 70B (DI)DeepInfra80.5%$0.3500$0.4000$0.3650128,000
59Llama 3.1 70B (Hyp)Hyperbolic80.5%$0.4000$0.4000$0.4000128,000
60Hermes-3-70B (Hyp)Hyperbolic80.5%$0.4000$0.4000$0.4000128,000

54 of the 60 scores shown link to the page they were read on (underlined; hover for publisher, variant and date read). The rest have no recorded source yet and are being worked through.

Most capable models now score very highly here, which is why SWE-bench Verified has largely replaced it as the benchmark that separates coding models.

About these scores

  • Python function synthesis from docstrings.
  • Scores are as published by the model’s provider or a public leaderboard. Provider-reported figures are self-reported and are not independently re-run by us.
  • Where a hosted variant of an open-weights model has no score of its own, it inherits the base model’s, because the weights are the same.
  • Prices beside each score are the provider’s published token rates. How the blended rate is calculated.

Showing the top 60 of 132 models we hold a HumanEval score for.

Frequently asked questions

What is the difference between HumanEval and SWE-bench?

HumanEval gives a model an isolated function to write from a docstring. SWE-bench Verified gives it a whole repository and a real issue, and requires a patch that passes the project’s existing tests. HumanEval measures code generation; SWE-bench measures software engineering, and scores on it are far lower across the board.

Other leaderboards

Rankings that weigh price too

Leaderboards are recomputed on every deploy from the catalogue. Prices checked November 17, 2026.