HealthBench AIOverviewSubmit a resultPublished results on the English MedQA (USMLE-style) test set. The USMLE-style view shows the identical cohorts; they are one evaluation, not two.
Checked Oct 24, 2026. One entry per model version from reviewed 2026 reports or undated current sources; historical cohorts are excluded. Sources are selected by publication recency. Entries are shown highest reported score first within each metric type. No model is pinned or promoted.
Latest available evidence does not mean the latest model release. Publication dates, inference settings and metrics remain visible. Scores from different runs are not directly comparable. No automatic live refresh.
arXiv 2607.10849v1 independent preprint (Jul 12, 2026)
1,273 MedQA questions, zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Independent preprint, not a company endorsement.
Claude Fable 5 refused 17.4% of questions; its 96.6 scored-subset accuracy excludes refusals and is not used for ranking.
arXiv 2607.10849v1 independent preprint (Jul 12, 2026)
1,273 MedQA questions, zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Independent preprint, not a company endorsement.
Claude Fable 5 refused 17.4% of questions; its 96.6 scored-subset accuracy excludes refusals and is not used for ranking.
arXiv 2607.10849v1 independent preprint (Jul 12, 2026)
1,273 MedQA questions, zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Independent preprint, not a company endorsement.
Claude Fable 5 refused 17.4% of questions; its 96.6 scored-subset accuracy excludes refusals and is not used for ranking.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
arXiv 2607.10849v1 independent preprint (Jul 12, 2026)
17.4% refusal rate; scored-subset 96.6 not ranked
1,273 MedQA questions, zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Independent preprint, not a company endorsement.
Claude Fable 5 refused 17.4% of questions; its 96.6 scored-subset accuracy excludes refusals and is not used for ranking.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 (Apr 6, 2026)
MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
No public MedQA-English score verified in our reviewed sources for these exact versions. This is a research gap, not a zero or proof that no result exists. Older model scores are never substituted.
GPT-6.1 Sol · Not verified GPT-6 Astra · Not verified GPT-6 Sol · Not verified GPT-6 Luna · Not verified Claude Sonnet 5.5 · Not verified Claude Opus 5.5 · Not verified Claude Haiku 5.5 · Not verified Claude Fable 5.1 · Not verified Gemini 4 Argon · Not verified Gemini 3.8 Flash · Not verified
Grok 4.7 · Not verified
GLM-5.3 · Not verified DeepSeek V4.1 Flash · Not verified
Kimi K3.1 Pro · Not verified
Muse Wonder Full · Not verifiedModel-release references: Artificial Analysis (Oct 19) ↗Artificial Analysis (Oct 23) ↗
Official · official vendor report9 rows
Source: MedGemma technical report v4, Table 3 · Source date: Apr 6, 2026 · Verified:
Metric: Accuracy (%). Protocol: MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
In Google’s MedGemma technical report v4 (Apr 6, 2026), o3 has the highest MedQA four-option accuracy at 93.3%, followed by Gemini 2.5 Pro at 92.6%.
Source-backed snapshots, last checked Oct 24, 2026; not an exhaustive census. Counts include reported configurations and repeat appearances across sources, not unique base models. A missing model is not a zero. Cohorts are never combined; vendor-reported does not mean independently validated.
| Rank | Model | Organization | Score | Metric | Source date | Notes | Source |
|---|---|---|---|---|---|---|---|
| 01 | o3 | OpenAI | 93.3% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 02 | Gemini 2.5 Pro | 92.6% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 03 | Gemini 2.5 Flash | 92.0% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 04 | DeepSeek R1 | DeepSeek | 90.1% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 05 | MedGemma 27B (test-time scaling) | 87.7% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 06 | GPT-4o | OpenAI | 86.5% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 07 | Gemma 3 27B | 74.9% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 08 | MedGemma 4B | 64.4% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 09 | Gemma 3 4B | 50.7% | Accuracy (%) | Apr 6, 2026 | — | Source |
6 cohorts, 47 result rows, 47 distinct model names. Many models appear in more than one cohort; rows are not unique models. Source-backed snapshots, last checked Oct 24, 2026; not an exhaustive census. Counts include reported configurations and repeat appearances across sources, not unique base models. A missing model is not a zero. Cohorts are never combined; vendor-reported does not mean independently validated.
Official · official vendor report9 rows
Source: MedGemma technical report v4, Table 3 · Source date: Apr 6, 2026 · Verified:
Metric: Accuracy (%). Protocol: MedQA four-option, 1,273 items. Google-run evaluations as reported in the MedGemma report; MedGemma 27B uses test-time scaling.
In Google’s MedGemma technical report v4 (Apr 6, 2026), o3 has the highest MedQA four-option accuracy at 93.3%, followed by Gemini 2.5 Pro at 92.6%.
o3 (OpenAI): 93.3% Gemini 2.5 Pro (Google): 92.6% Gemini 2.5 Flash (Google): 92.0% DeepSeek R1 (DeepSeek): 90.1% MedGemma 27B (test-time scaling) (Google): 87.7% GPT-4o (OpenAI): 86.5% Gemma 3 27B (Google): 74.9% MedGemma 4B (Google): 64.4% Gemma 3 4B (Google): 50.7%independent preprint4 rows
Source: arXiv 2607.10849v1, Table 2 · Source date: Jul 12, 2026 · Verified:
Metric: Raw accuracy (refusals counted wrong, %). Protocol: 1,273 MedQA questions, zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
In the independent preprint arXiv 2607.10849v1 (Jul 12, 2026), GPT-5 has the highest raw MedQA accuracy at 96.0%, followed by Claude Opus 4.8 at 95.0%. Raw accuracy counts refusals as incorrect.
GPT-5 (OpenAI): 96.0% Claude Opus 4.8 (Anthropic): 95.0% Claude Opus 4.6 (Anthropic): 94.3% Claude Fable 5 (Anthropic): 79.7% — 17.4% refusal rate; scored-subset 96.6 not rankedOfficial · official vendor report4 rows
Source: Microsoft Medprompt paper (arXiv 2311.16452), Table 1 · Source date: Nov 2023 · Verified:
Metric: Accuracy (%). Protocol: MedQA four-option. Standard Medprompt: 5-shot, 5-way choice-shuffle ensemble (not the enhanced 20-shot, 11-ensemble 90.6 variant).
In Microsoft’s Medprompt paper (Nov 2023), GPT-4 with Medprompt reported 90.2% accuracy on MedQA four-option, ahead of Med-PaLM 2 (choose best) at 86.5%.
GPT-4 (Medprompt) (Microsoft / OpenAI): 90.2% Med-PaLM 2 (choose best) (Google Research): 86.5% — Cited from prior paper GPT-4 (5-shot) (Microsoft / OpenAI): 81.4% Flan-PaLM 540B (choose best) (Google Research): 67.6% — Cited from prior paperOfficial · official vendor report4 rows
Source: Med-PaLM 2 research paper · Source date: 2023 · Verified:
Metric: MedQA (USMLE) accuracy (%). Protocol: Scores reported in the Med-PaLM 2 paper for MedQA USMLE-style questions.
In the 2023 Med-PaLM 2 paper, Med-PaLM 2 reported 86.5% accuracy on MedQA (USMLE-style), with GPT-4 base at 86.1%.
Med-PaLM 2 (Google Research): 86.5% GPT-4 base (Microsoft / OpenAI): 86.1% Med-PaLM (Google Research): 67.2% GPT-3.5 (OpenAI): 60.2%Official · official vendor report11 rows
Source: BioMistral authors’ repository, Supervised Fine-tuning Benchmark · Source date: 2024 report; repository snapshot checked Oct 24, 2026 · Verified:
Metric: Accuracy (%). Protocol: MedQA four-option, author-reported supervised fine-tuning evaluation, 3-shot averaged across 3 seeds. GPT-3.5 Turbo is a no-SFT baseline.
The BioMistral author table reports GPT-3.5 Turbo 1106 (3-shot, no SFT) at 57.71% and BioMistral 7B DARE (SFT) at 51.1% on MedQA four-option. Training and inference configurations differ.
GPT-3.5 Turbo 1106 (3-shot, no SFT) (OpenAI): 57.71% — BioMistral author-run comparison; GPT baseline is not fine-tuned.
Mistral 7B Instruct (SFT) (Mistral AI): 42.0% — BioMistral author-run comparison; GPT baseline is not fine-tuned.
MediTron 7B (SFT) (EPFL / collaborators): 41.6% — BioMistral author-run comparison; GPT baseline is not fine-tuned.Official · official vendor report15 rows
Source: OpenBioLLM 70B author model card, evaluation table · Source date: Undated evaluation table; checked Oct 24, 2026 · Verified:
Metric: Accuracy (%). Protocol: MedQA four-option column transcribed from the original model card. Comparator prompting/training protocols vary; this table is not a controlled head-to-head evaluation.
The OpenBioLLM author card reports OpenBioLLM 70B at 78.162% and OpenBioLLM 8B at 58.993% on MedQA four-option. Its comparator table uses mixed protocols and should not be read as a controlled ranking.
Med-PaLM 2 (5-shot) (Google Research): 79.7% — Author-card table; comparator protocols vary and are not independently reproduced here. GPT-4 (OpenBioLLM table) (OpenAI): 78.87% — Author-card table; comparator protocols vary and are not independently reproduced here. Med-PaLM 1 (Flan-PaLM, 5-shot) (Google Research): 67.6% — Author-card table; comparator protocols vary and are not independently reproduced here. Gemini 1.0 (Google): 58.0% — Author-card table; comparator protocols vary and are not independently reproduced here. GPT-3.5 Turbo 1106 (OpenAI): 57.71% — Author-card table; comparator protocols vary and are not independently reproduced here. Meditron 70B (EPFL / collaborators): 57.1% — Author-card table; comparator protocols vary and are not independently reproduced here. Mistral 7B v0.1 (Mistral AI): 50.82% — Author-card table; comparator protocols vary and are not independently reproduced here. Gemma 7B (Google): 47.21% — Author-card table; comparator protocols vary and are not independently reproduced here.MedQA is a multiple-choice medical question answering dataset collected from professional medical board exams. The English portion is drawn from United States Medical Licensing Examination (USMLE) style questions.
Each cohort keeps its own prompting protocol. The USMLE-style page on this index shows the identical cohorts because it is the same dataset.
Default cohort: MedGemma technical report v4, Table 3 (Apr 6, 2026; last checked Oct 24, 2026). Cohorts are never combined and nothing updates automatically. MedQA-English and model names belong to their respective owners; REAGEN, Inc. is not affiliated with them.
Related: USMLE-style definition
Have a sourced MedQA-English result not listed here? Apply to the index.