HealthBench AIOverviewSubmit a resultModels answer yes, no or maybe to biomedical research questions using PubMed abstracts. Includes an October 2026 independent report on the standard 500-question test; nonstandard splits are labelled separately.
Checked Oct 24, 2026. One entry per model version from reviewed 2026 reports or undated current sources; historical cohorts are excluded. Sources are selected by publication recency. Entries are shown highest reported score first within each metric type. No model is pinned or promoted.
Latest available evidence does not mean the latest model release. Publication dates, inference settings and metrics remain visible. Scores from different runs are not directly comparable. No automatic live refresh.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Jev in Medicine v4 — official 500-question test (Oct 10, 2026)
95% Wilson CI 74.6–81.8; non-generative choice model
Expert-annotated 500-question PubMedQA test split; abstract conclusions removed. One run per condition, zero-shot; GPT-6 Sol uses constrained JSON, no system prompt.
Independent preprint; preliminary results, no clinical validation or run-to-run replication.
Jev 1.13 is a non-generative choice model, not a conversational LLM.
The paper’s MetaMedQA results are NOT MedQA-English and are not imported into the MedQA/USMLE pages.
Jev in Medicine v4 — official 500-question test (Oct 10, 2026)
95% Wilson CI 74.4–81.6
Expert-annotated 500-question PubMedQA test split; abstract conclusions removed. One run per condition, zero-shot; GPT-6 Sol uses constrained JSON, no system prompt.
Independent preprint; preliminary results, no clinical validation or run-to-run replication.
Jev 1.13 is a non-generative choice model, not a conversational LLM.
The paper’s MetaMedQA results are NOT MedQA-English and are not imported into the MedQA/USMLE pages.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
arXiv 2607.10849v1 — 1,000-question nonstandard set (Jul 12, 2026)
1,000 PubMedQA questions (NOT the standard 500-question test set), zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Nonstandard split: do not compare with official 500-question results.
Claude Fable 5 refused about 20% of questions; its 81.3 scored-subset accuracy excludes refusals and is not ranked.
Independent preprint, not a company endorsement.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
arXiv 2607.10849v1 — 1,000-question nonstandard set (Jul 12, 2026)
1,000 PubMedQA questions (NOT the standard 500-question test set), zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Nonstandard split: do not compare with official 500-question results.
Claude Fable 5 refused about 20% of questions; its 81.3 scored-subset accuracy excludes refusals and is not ranked.
Independent preprint, not a company endorsement.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
arXiv 2607.10849v1 — 1,000-question nonstandard set (Jul 12, 2026)
1,000 PubMedQA questions (NOT the standard 500-question test set), zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Nonstandard split: do not compare with official 500-question results.
Claude Fable 5 refused about 20% of questions; its 81.3 scored-subset accuracy excludes refusals and is not ranked.
Independent preprint, not a company endorsement.
Google MedGemma technical report v4 — standard 500 (Apr 6, 2026)
Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
Vendor report; setups follow Google’s own runs.
arXiv 2607.10849v1 — 1,000-question nonstandard set (Jul 12, 2026)
~20% refusals; scored-subset 81.3 not ranked
1,000 PubMedQA questions (NOT the standard 500-question test set), zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
Nonstandard split: do not compare with official 500-question results.
Claude Fable 5 refused about 20% of questions; its 81.3 scored-subset accuracy excludes refusals and is not ranked.
Independent preprint, not a company endorsement.
No public PubMedQA score verified in our reviewed sources for these exact versions. This is a research gap, not a zero or proof that no result exists. Older model scores are never substituted.
GPT-6.1 Sol · Not verified GPT-6 Astra · Not verified GPT-6 Luna · Not verified Claude Sonnet 5.5 · Not verified Claude Opus 5.5 · Not verified Claude Haiku 5.5 · Not verified Claude Fable 5.1 · Not verified Gemini 4 Argon · Not verified Gemini 3.8 Flash · Not verified
Grok 4.7 · Not verified
GLM-5.3 · Not verified DeepSeek V4.1 Flash · Not verified
Kimi K3.1 Pro · Not verified
Muse Wonder Full · Not verifiedModel-release references: Artificial Analysis (Oct 19) ↗Artificial Analysis (Oct 23) ↗
Official · official vendor report9 rows
Source: MedGemma technical report v4, Table 3 · Source date: Apr 6, 2026 · Verified:
Metric: Accuracy (%). Protocol: Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
In Google’s MedGemma technical report v4 (Apr 6, 2026), on the standard 500-question PubMedQA test set, o3 has the highest accuracy at 80.0%, followed by GPT-4o at 78.4%.
Source-backed snapshots, last checked Oct 24, 2026; not an exhaustive census. Counts include reported configurations and repeat appearances across sources, not unique base models. A missing model is not a zero. Cohorts are never combined; vendor-reported does not mean independently validated.
| Rank | Model | Organization | Score | Metric | Source date | Notes | Source |
|---|---|---|---|---|---|---|---|
| 01 | o3 | OpenAI | 80.0% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 02 | GPT-4o | OpenAI | 78.4% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 03 | DeepSeek R1 | DeepSeek | 77.2% | Accuracy (%) | Apr 6, 2026 | — | Source |
| 04 | MedGemma 27B (test-time scaling) | 76.8% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 05 | Gemini 2.5 Flash | 76.2% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 06 | Gemini 2.5 Pro | 75.8% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 07 | MedGemma 4B | 73.4% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 08 | Gemma 3 27B | 73.4% | Accuracy (%) | Apr 6, 2026 | — | Source | |
| 09 | Gemma 3 4B | 68.4% | Accuracy (%) | Apr 6, 2026 | — | Source |
6 cohorts, 53 result rows, 53 distinct model names. Many models appear in more than one cohort; rows are not unique models. Source-backed snapshots, last checked Oct 24, 2026; not an exhaustive census. Counts include reported configurations and repeat appearances across sources, not unique base models. A missing model is not a zero. Cohorts are never combined; vendor-reported does not mean independently validated.
Official · official vendor report9 rows
Source: MedGemma technical report v4, Table 3 · Source date: Apr 6, 2026 · Verified:
Metric: Accuracy (%). Protocol: Standard 500-question PubMedQA test set. Google-run evaluations; MedGemma 27B uses test-time scaling.
In Google’s MedGemma technical report v4 (Apr 6, 2026), on the standard 500-question PubMedQA test set, o3 has the highest accuracy at 80.0%, followed by GPT-4o at 78.4%.
o3 (OpenAI): 80.0% GPT-4o (OpenAI): 78.4% DeepSeek R1 (DeepSeek): 77.2% MedGemma 27B (test-time scaling) (Google): 76.8% Gemini 2.5 Flash (Google): 76.2% Gemini 2.5 Pro (Google): 75.8% MedGemma 4B (Google): 73.4% Gemma 3 27B (Google): 73.4% Gemma 3 4B (Google): 68.4%Official · official benchmark leaderboard11 rows
Source: PubMedQA official results page · Source date: Nov 2022–Mar 2024 entries; live page checked Oct 24, 2026 · Verified:
Metric: Accuracy (%). Protocol: Reasoning-required setting, as listed on the official leaderboard; per-row entry dates copied from the page.
On the official PubMedQA results page, GPT-4 (Medprompt), dated Nov 28, 2023, is listed at 82.0% accuracy, followed by Med-PaLM 2 at 81.8%, across 11 entries dated Nov 2022 to Mar 2024.
GPT-4 (Medprompt) (Microsoft / OpenAI): 82.0% — Entry date Nov 28, 2023 Med-PaLM 2 (Google Research): 81.8% — Entry date May 16, 2023
MEDITRON (EPFL / collaborators): 81.6% — Entry date Nov 27, 2023
Palmyra-Med (Writer): 81.1% — Entry date Jul 6, 2023
AntGLM-Med (Ant Group): 80.6% — Entry date Dec 1, 2023 GPT-4-base (Microsoft / OpenAI): 80.4% — Entry date Apr 12, 2023 Claude 3 (Anthropic): 79.7% — Entry date Mar 4, 2024 GPT-3.5 + Z-Code++ (Microsoft Azure AI): 79.6% — Entry date Jan 11, 2023 Flan-PaLM (3-shot) (Google): 79.0% — Entry date Dec 26, 2022 Galactica (Meta AI): 77.6% — Entry date Nov 16, 2022 GPT-4 (Microsoft / OpenAI): 75.2% — Entry date Mar 20, 2023independent preprint4 rows
Source: arXiv 2607.10849v1, Table 2 · Source date: Jul 12, 2026 · Verified:
Metric: Raw accuracy (refusals counted wrong, %). Protocol: 1,000 PubMedQA questions (NOT the standard 500-question test set), zero-shot direct answer, no extended thinking, no chain-of-thought, no retrieval.
In arXiv 2607.10849v1 (Jul 12, 2026), on a nonstandard 1,000-question PubMedQA set, Claude Opus 4.6 has the highest raw accuracy at 76.6%, followed by Claude Opus 4.8 at 75.2%. This is not the official 500-question test set.
Claude Opus 4.6 (Anthropic): 76.6% Claude Opus 4.8 (Anthropic): 75.2% GPT-5 (OpenAI): 71.7% Claude Fable 5 (Anthropic): 65.0% — ~20% refusals; scored-subset 81.3 not rankedOfficial · official vendor report11 rows
Source: BioMistral authors’ repository, Supervised Fine-tuning Benchmark · Source date: 2024 report; repository snapshot checked Oct 24, 2026 · Verified:
Metric: Accuracy (%). Protocol: PubMedQA, author-reported supervised fine-tuning evaluation, 3-shot averaged across 3 seeds. GPT-3.5 Turbo is a no-SFT baseline.
The BioMistral SFT table reports BioMistral 7B SLERP at 77.8% on PubMedQA, followed by DARE at 77.7%. These are author-reported SFT configurations.
Mistral 7B Instruct (SFT) (Mistral AI): 75.7% — BioMistral author-run comparison; GPT baseline is not fine-tuned. MediTron 7B (SFT) (EPFL / collaborators): 74.9% — BioMistral author-run comparison; GPT baseline is not fine-tuned. GPT-3.5 Turbo 1106 (3-shot, no SFT) (OpenAI): 72.66% — BioMistral author-run comparison; GPT baseline is not fine-tuned.Official · official vendor report15 rows
Source: OpenBioLLM 70B author model card, evaluation table · Source date: Undated evaluation table; checked Oct 24, 2026 · Verified:
Metric: Accuracy (%). Protocol: PubMedQA column transcribed from the original model card. Comparator prompting/training protocols vary; this table is not a controlled head-to-head evaluation.
The OpenBioLLM author card reports OpenBioLLM 70B at 78.97% and OpenBioLLM 8B at 74.12% on PubMedQA. Comparator protocols and split details are not uniform, so these results remain separate.
Med-PaLM 2 (5-shot) (Google Research): 79.2% — Author-card table; comparator protocols vary and are not independently reproduced here. Med-PaLM 1 (Flan-PaLM, 5-shot) (Google Research): 79.0% — Author-card table; comparator protocols vary and are not independently reproduced here. Meditron 70B (EPFL / collaborators): 76.6% — Author-card table; comparator protocols vary and are not independently reproduced here. Gemma 7B (Google): 76.2% — Author-card table; comparator protocols vary and are not independently reproduced here. Mistral 7B v0.1 (Mistral AI): 75.4% — Author-card table; comparator protocols vary and are not independently reproduced here. GPT-4 (OpenBioLLM table) (OpenAI): 75.2% — Author-card table; comparator protocols vary and are not independently reproduced here. GPT-3.5 Turbo 1106 (OpenAI): 72.66% — Author-card table; comparator protocols vary and are not independently reproduced here. Gemini 1.0 (Google): 70.7% — Author-card table; comparator protocols vary and are not independently reproduced here.independent preprint3 rows
Source: Jev in Medicine v4, Table 2 · Source date: Oct 10, 2026 (v4); runs Sep 26–27, 2026 · Verified:
Metric: Accuracy (%). Protocol: Expert-annotated 500-question PubMedQA test split; abstract conclusions removed. One run per condition, zero-shot; GPT-6 Sol uses constrained JSON, no system prompt.
In Jev in Medicine v4 (Oct 10, 2026), Jev 1.13 scores 78.4%, GPT-6 Sol (medium) 78.2%, and GPT-6 Sol (none) 76.6% on the official 500-question PubMedQA test split.
GPT-6 Sol (medium) (OpenAI): 78.2% — 95% Wilson CI 74.4–81.6 GPT-6 Sol (none) (OpenAI): 76.6% — 95% Wilson CI 72.7–80.1; no reasoningPubMedQA is a biomedical question answering dataset in which each question is derived from a PubMed article title and must be answered yes, no or maybe using the accompanying abstract.
The official test set has 500 questions. Any cohort using a different question set is labelled nonstandard and must not be compared with official 500-question results.
Default cohort: MedGemma technical report v4, Table 3 (Apr 6, 2026; last checked Oct 24, 2026). Cohorts are never combined and nothing updates automatically. PubMedQA and model names belong to their respective owners; REAGEN, Inc. is not affiliated with them.
Have a sourced PubMedQA result not listed here? Apply to the index.