# HealthBench AI

> A source-first index of published health AI benchmark results, published by REAGEN, Inc. Scores are grouped into separate evaluation cohorts (one source, metric, protocol and split each), never combined. Latest verified sources, last checked Oct 24, 2026; no automatic updating. Not medical advice. Not affiliated with benchmark owners.

## Key pages
- [Leaderboard](https://healthbenchai.com/leaderboard): ranked results across five benchmarks
- [Benchmarks](https://healthbenchai.com/benchmarks): definitions and how they relate
- [Methodology](https://healthbenchai.com/methodology): sources, scoring limits, MedQA and USMLE overlap, FAQ
- [About](https://healthbenchai.com/about): REAGEN, Inc.
- [Research and evaluation articles](https://healthbenchai.com/research): individual model results, comparisons, methods and downloadable evidence
- [Full public text](https://healthbenchai.com/llms-full.txt): public research text, answers, score tables and citations; no private prompts or model transcripts

## Benchmarks and direct answers
- [HealthBench Hard](https://healthbenchai.com/benchmarks/hard)
  - Meta Muse Spark evaluation methodology (raw, cross-vendor) [official vendor report; Raw HealthBench Hard score (0–100); source date Undated; checked Oct 24, 2026]: In Meta’s Muse Spark evaluation methodology (undated; checked Oct 24, 2026), Muse Spark Thinking has the highest raw HealthBench Hard score at 42.8, followed by GPT-5.4 Xhigh at 40.1; Gemini 3.1 Pro High scored 20.6 and Claude Opus 4.6 Max 14.8. Source: https://ai.meta.com/static-resource/muse-spark-eval-methodology
  - OpenAI GPT-6.1 Sol addendum, Table 7 (latest checked) [official vendor report; Length-adjusted rubric score (0–100); source date Undated page; checked Oct 24, 2026]: In the latest verified OpenAI GPT-6.1 Sol addendum, Table 7, GPT-6 Astra scores 36.6 and GPT-6.1 Sol scores 36.2 on length-adjusted HealthBench Hard. This is a vendor-run comparison, not a global ranking. Source: https://deploymentsafety.openai.com/gpt-6-1-sol
  - OpenAI GPT-6 Astra system card, Table 6 (Sep 2026) [official vendor report; Length-adjusted rubric score (0–100); source date Sep 2026 (card amendment Sep 9, 2026)]: In the OpenAI GPT-6 Astra system card (Table 6, Sep 2026), GPT-6 Astra has the highest length-adjusted HealthBench Hard score at 36.6, followed by GPT-5.6 Sol at 33.1. This cohort uses a length penalty and is not comparable with raw Hard scores. Source: https://deploymentsafety.openai.com/gpt-6-astra
  - OpenAI GPT-OSS evaluation chart (Aug 2025, historical) [official vendor report; Raw HealthBench Hard score (0–100); source date Aug 2025]: In the August 2025 OpenAI GPT-OSS evaluation chart, o3 had the highest listed HealthBench Hard score at 31.6, followed by gpt-oss-120b at 30.0. Source: https://deploymentsafety.openai.com/data/eval-sets/gpt-oss/assets/capabilities_plots/gptoss_healthbench.pdf
  - Baichuan M3 original report, §4.2.1 (Feb 2026) [official vendor report; Reported HealthBench Hard score (0–100); source date Feb 2026 (arXiv v1)]: Baichuan’s M3 report lists Baichuan M3 at 44.4, GPT-5.2 High at 42.0 and AntAngelMed at 39.6 on HealthBench Hard. These are Baichuan-reported results, not part of Meta’s Muse Spark comparison. Source: https://arxiv.org/html/2602.06570v1
  - Nox Orion 1.4 — post-hoc length-adjusted self-test (Sep 2026) [team-run · score calculation verified; Post-hoc length-adjusted score (0–100); source date Sep 5, 2026 run; checked Oct 24]: Applying the stated length adjustment to submitted Nox Orion 1.4 Ultra outputs gives 30.792 (95% task-bootstrap interval 28.965–32.577). This is a post-hoc calculation, not an independently evaluated score. Source: https://healthbenchai.com/research/nox-orion-evidence-audit
  - Nox Orion 1.4 — raw self-test (Sep 2026) [team-run · score calculation verified; Raw HealthBench Hard score (0–100); source date Sep 5, 2026 run; checked Oct 24]: The supplied Nox Orion 1.4 Ultra grades reproduce a raw score of 46.381 across 1,000 tasks. Arithmetic was checked against official rubric weights; answer quality was not independently regraded. Source: https://healthbenchai.com/research/nox-orion-evidence-audit
- [HealthBench Professional](https://healthbenchai.com/benchmarks/professional)
  - HealthBench AI matched-protocol run (Oct 25, 2026) [Evaluated by HealthBenchAI; Length-adjusted score (0–100); source date Oct 25, 2026]: In HealthBench AI's self-operated run (Oct 25, 2026), GPT-6.1 Sol (max) scored 64.6 length-adjusted on 525/525 tasks; Claude Fable 5.1 (high) scored 58.0 length-adjusted on 525/525 tasks. Both models used one harness. Not third-party certification. Source: https://healthbenchai.com/research/understanding-healthbench-professional
  - OpenAI GPT-6.1 Sol system card, Table 7 (official) [official vendor report; Length-adjusted score (0–100); source date Undated page; checked Oct 24, 2026]: In the OpenAI GPT-6.1 Sol system card (Table 7, checked Oct 24, 2026), GPT-6 Astra has the highest length-adjusted HealthBench Professional score at 64.7, followed by GPT-6.1 Sol at 64.2. Source: https://deploymentsafety.openai.com/gpt-6-1-sol
  - Anthropic Claude Sonnet 5.5 system card, §8.15 (Oct 2026) [official vendor report; Length-adjusted score (0–100); source date Oct 2026]: In Anthropic’s Claude Sonnet 5.5 system card (§8.15, Oct 2026), GPT-6 Astra has the highest length-adjusted HealthBench Professional score at 70.3, followed by Claude Sonnet 5.5 (max) at 69.2, using a Claude Opus 4.8 judge. Source: https://www.anthropic.com/claude-sonnet-5-5-system-card
  - Artificial Analysis original 14-model chart (Oct 19, 2026) [independent evaluation; Length-adjusted score (0–100); source date Oct 19, 2026]: In Artificial Analysis’s independent HealthBench Professional evaluation (Oct 19, 2026), GPT-6 Astra (Max) has the highest length-adjusted score at 66.0, followed by GPT-6.1 Sol (Max) at 65.1 and Claude Sonnet 5.5 (Max) at 63.0. All 14 scores are verified against the evaluator’s original chart. Source: https://artificialanalysis.ai/articles/healthbench-professional-health-conversations
  - OpenAI HealthBench Professional paper (Apr 2026) [official vendor report; Length-adjusted score (0–100); source date Apr 2026]: In the April 2026 OpenAI HealthBench Professional paper, GPT-5.4 in ChatGPT for Clinicians reported the highest listed length-adjusted score at 59.0; the GPT-5.4 base model scored 48.1. Source: https://cdn.openai.com/dd128428-0184-4e25-b155-3a7686c7d744/HealthBench-Professional.pdf
  - Nox Orion 2.1 — HealthBenchAI team evaluation (Oct 23, 2026) [Evaluated by HealthBenchAI; Length-adjusted score (0–100); source date Oct 23, 2026 run; arithmetic checked Oct 24]: Nox Orion 2.1 reports 69.528 length-adjusted (raw 72.833), with a recomputed 95% task-bootstrap interval of 65.814–73.033. This is a submitted self-test whose arithmetic was checked, not independent verification or an established first-place ranking. Source: https://healthbenchai.com/research/nox-orion-evidence-audit
  - Anthropic system-card comparison — includes restricted-access Mythos models [official vendor report; Length-adjusted score (0–100); source date Jun 9, 2026]: Anthropic’s June 9, 2026 system card reports Claude Mythos 5 at 66.0, Mythos Preview at 64.7 and Opus 4.8 at 56.9 length-adjusted on HealthBench Professional. Mythos access is restricted. These are vendor-reported scores, not HealthBench AI tests. The newer Mythos 5.1 is distinct; the reviewed 5.1 card’s Professional score of 62.1 is explicitly for Fable 5.1 and is not attributed to Mythos 5.1. Source: https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620/Claude%20Fable%205%20%26%20Claude%20Mythos%205%20System%20Card.pdf#page=300
- [MedQA-English](https://healthbenchai.com/benchmarks/medqa)
  - Google MedGemma technical report v4 (Apr 6, 2026) [official vendor report; Accuracy (%); source date Apr 6, 2026]: In Google’s MedGemma technical report v4 (Apr 6, 2026), o3 has the highest MedQA four-option accuracy at 93.3%, followed by Gemini 2.5 Pro at 92.6%. Source: https://arxiv.org/pdf/2507.05201v4
  - arXiv 2607.10849v1 independent preprint (Jul 12, 2026) [independent preprint; Raw accuracy (refusals counted wrong, %); source date Jul 12, 2026]: In the independent preprint arXiv 2607.10849v1 (Jul 12, 2026), GPT-5 has the highest raw MedQA accuracy at 96.0%, followed by Claude Opus 4.8 at 95.0%. Raw accuracy counts refusals as incorrect. Source: https://arxiv.org/html/2607.10849
  - Microsoft Medprompt paper, Table 1 (Nov 2023) [official vendor report; Accuracy (%); source date Nov 2023]: In Microsoft’s Medprompt paper (Nov 2023), GPT-4 with Medprompt reported 90.2% accuracy on MedQA four-option, ahead of Med-PaLM 2 (choose best) at 86.5%. Source: https://arxiv.org/pdf/2311.16452
  - Med-PaLM 2 paper (2023, historical) [official vendor report; MedQA (USMLE) accuracy (%); source date 2023]: In the 2023 Med-PaLM 2 paper, Med-PaLM 2 reported 86.5% accuracy on MedQA (USMLE-style), with GPT-4 base at 86.1%. Source: https://arxiv.org/pdf/2305.09617
  - BioMistral author report — SFT / 3-shot (2024) [official vendor report; Accuracy (%); source date 2024 report; repository snapshot checked Oct 24, 2026]: The BioMistral author table reports GPT-3.5 Turbo 1106 (3-shot, no SFT) at 57.71% and BioMistral 7B DARE (SFT) at 51.1% on MedQA four-option. Training and inference configurations differ. Source: https://github.com/BioMistral/BioMistral
  - OpenBioLLM author model card — mixed reported protocols [official vendor report; Accuracy (%); source date Undated evaluation table; checked Oct 24, 2026]: The OpenBioLLM author card reports OpenBioLLM 70B at 78.162% and OpenBioLLM 8B at 58.993% on MedQA four-option. Its comparator table uses mixed protocols and should not be read as a controlled ranking. Source: https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B
- [USMLE-style](https://healthbenchai.com/benchmarks/usmle)
  - Google MedGemma technical report v4 (Apr 6, 2026) [official vendor report; Accuracy (%); source date Apr 6, 2026]: In Google’s MedGemma technical report v4 (Apr 6, 2026), o3 has the highest MedQA four-option accuracy at 93.3%, followed by Gemini 2.5 Pro at 92.6%. Source: https://arxiv.org/pdf/2507.05201v4
  - arXiv 2607.10849v1 independent preprint (Jul 12, 2026) [independent preprint; Raw accuracy (refusals counted wrong, %); source date Jul 12, 2026]: In the independent preprint arXiv 2607.10849v1 (Jul 12, 2026), GPT-5 has the highest raw MedQA accuracy at 96.0%, followed by Claude Opus 4.8 at 95.0%. Raw accuracy counts refusals as incorrect. Source: https://arxiv.org/html/2607.10849
  - Microsoft Medprompt paper, Table 1 (Nov 2023) [official vendor report; Accuracy (%); source date Nov 2023]: In Microsoft’s Medprompt paper (Nov 2023), GPT-4 with Medprompt reported 90.2% accuracy on MedQA four-option, ahead of Med-PaLM 2 (choose best) at 86.5%. Source: https://arxiv.org/pdf/2311.16452
  - Med-PaLM 2 paper (2023, historical) [official vendor report; MedQA (USMLE) accuracy (%); source date 2023]: In the 2023 Med-PaLM 2 paper, Med-PaLM 2 reported 86.5% accuracy on MedQA (USMLE-style), with GPT-4 base at 86.1%. Source: https://arxiv.org/pdf/2305.09617
  - BioMistral author report — SFT / 3-shot (2024) [official vendor report; Accuracy (%); source date 2024 report; repository snapshot checked Oct 24, 2026]: The BioMistral author table reports GPT-3.5 Turbo 1106 (3-shot, no SFT) at 57.71% and BioMistral 7B DARE (SFT) at 51.1% on MedQA four-option. Training and inference configurations differ. Source: https://github.com/BioMistral/BioMistral
  - OpenBioLLM author model card — mixed reported protocols [official vendor report; Accuracy (%); source date Undated evaluation table; checked Oct 24, 2026]: The OpenBioLLM author card reports OpenBioLLM 70B at 78.162% and OpenBioLLM 8B at 58.993% on MedQA four-option. Its comparator table uses mixed protocols and should not be read as a controlled ranking. Source: https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B
- [PubMedQA](https://healthbenchai.com/benchmarks/pubmedqa)
  - Google MedGemma technical report v4 — standard 500 (Apr 6, 2026) [official vendor report; Accuracy (%); source date Apr 6, 2026]: In Google’s MedGemma technical report v4 (Apr 6, 2026), on the standard 500-question PubMedQA test set, o3 has the highest accuracy at 80.0%, followed by GPT-4o at 78.4%. Source: https://arxiv.org/pdf/2507.05201v4
  - Official PubMedQA leaderboard (2022–2024 entries) [official benchmark leaderboard; Accuracy (%); source date Nov 2022–Mar 2024 entries; live page checked Oct 24, 2026]: On the official PubMedQA results page, GPT-4 (Medprompt), dated Nov 28, 2023, is listed at 82.0% accuracy, followed by Med-PaLM 2 at 81.8%, across 11 entries dated Nov 2022 to Mar 2024. Source: https://pubmedqa.github.io/
  - arXiv 2607.10849v1 — 1,000-question nonstandard set (Jul 12, 2026) [independent preprint; Raw accuracy (refusals counted wrong, %); source date Jul 12, 2026]: In arXiv 2607.10849v1 (Jul 12, 2026), on a nonstandard 1,000-question PubMedQA set, Claude Opus 4.6 has the highest raw accuracy at 76.6%, followed by Claude Opus 4.8 at 75.2%. This is not the official 500-question test set. Source: https://arxiv.org/html/2607.10849
  - BioMistral author report — SFT / 3-shot (2024) [official vendor report; Accuracy (%); source date 2024 report; repository snapshot checked Oct 24, 2026]: The BioMistral SFT table reports BioMistral 7B SLERP at 77.8% on PubMedQA, followed by DARE at 77.7%. These are author-reported SFT configurations. Source: https://github.com/BioMistral/BioMistral
  - OpenBioLLM author model card — mixed reported protocols [official vendor report; Accuracy (%); source date Undated evaluation table; checked Oct 24, 2026]: The OpenBioLLM author card reports OpenBioLLM 70B at 78.97% and OpenBioLLM 8B at 74.12% on PubMedQA. Comparator protocols and split details are not uniform, so these results remain separate. Source: https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B
  - Jev in Medicine v4 — official 500-question test (Oct 10, 2026) [independent preprint; Accuracy (%); source date Oct 10, 2026 (v4); runs Sep 26–27, 2026]: In Jev in Medicine v4 (Oct 10, 2026), Jev 1.13 scores 78.4%, GPT-6 Sol (medium) 78.2%, and GPT-6 Sol (none) 76.6% on the official 500-question PubMedQA test split. Source: https://arxiv.org/html/2609.34024v4

## Research and source notes by HealthBench AI
- [Claude Fable 5.1: our Professional test report](https://healthbenchai.com/research/claude-fable-5-1-professional): Our completed 525-task HealthBench Professional test: scores, confidence interval, model settings and downloadable per-task evidence.
- [GPT-6.1 Sol Max: our Professional test report](https://healthbenchai.com/research/gpt-6-1-sol-professional): Our completed 525-task GPT-6.1 Sol Max test: 64.63 length-adjusted, 67.56 raw, confidence interval, protocol and downloadable per-task evidence.
- [Reading HealthBench Professional results](https://healthbenchai.com/research/understanding-healthbench-professional): Our guide to clinical conversation scores, length adjustment and why evaluation protocols matter.
- [Nox Orion: our test results](https://healthbenchai.com/research/nox-orion-evidence-audit): We tested Nox Orion and obtained these scores: 69.53 on HealthBench Professional and 30.79 on HealthBench Hard, both length-adjusted.
- [HealthBench and Hard: a guide to the distinction](https://healthbenchai.com/research/healthbench-and-healthbench-hard): How we interpret the 5,000-conversation HealthBench dataset and its 1,000-example difficult subset.
- [PubMedQA: interpreting yes, no and maybe](https://healthbenchai.com/research/understanding-pubmedqa): Our guide to abstract-based biomedical question answering, test splits and comparable PubMedQA scores.

## Notes
- Scores are not comparable across benchmarks.
- MedQA-English and USMLE-style share the same cohorts and are counted once.
- A missing model is not a zero score; latest scores are never guessed.
- Corrections: https://healthbenchai.com/contact