HealthBench AIOverviewSubmit a resultTraceable before comprehensive
How results enter the index, what the numbers mean, and where they stop meaning anything.
Inclusion rule
A score is listed only when a named, publicly accessible source reports that specific number for that specific model on that specific benchmark. We do not run evaluations, estimate missing values, or average across sources. If a source is ambiguous, the result is left out.
Sources
- HealthBench Hard: Meta Muse Spark eval methodology (Undated; checked Oct 24, 2026); OpenAI GPT-6.1 Sol system card, Table 7 (Undated page; checked Oct 24, 2026); OpenAI GPT-6 Astra system card (Sep 2026 (card amendment Sep 9, 2026)); OpenAI GPT-OSS evaluation chart (Aug 2025); Baichuan M3 technical report, section 4.2.1 / Figure 7 (Feb 2026 (arXiv v1)); HealthBenchAI: Nox Orion test results and evidence (Sep 5, 2026 run; checked Oct 24); HealthBenchAI: Nox Orion test results and evidence (Sep 5, 2026 run; checked Oct 24).
- HealthBench Professional: HealthBenchAI Professional evaluation methodology (Oct 25, 2026); OpenAI GPT-6.1 Sol system card, Table 7 (Undated page; checked Oct 24, 2026); Anthropic Claude Sonnet 5.5 system card, section 8.15 (Oct 2026); Artificial Analysis original announcement and 14-model chart (Oct 19, 2026); OpenAI HealthBench Professional paper (Apr 2026); HealthBenchAI: Nox Orion test results and evidence (Oct 23, 2026 run; arithmetic checked Oct 24); Anthropic Fable 5 / Mythos 5 system card, Table 8.1.A and Figure 8.18.2.A (p. 300) (Jun 9, 2026).
- MedQA-English: MedGemma technical report v4, Table 3 (Apr 6, 2026); arXiv 2607.10849v1, Table 2 (Jul 12, 2026); Microsoft Medprompt paper (arXiv 2311.16452), Table 1 (Nov 2023); Med-PaLM 2 research paper (2023); BioMistral authors’ repository, Supervised Fine-tuning Benchmark (2024 report; repository snapshot checked Oct 24, 2026); OpenBioLLM 70B author model card, evaluation table (Undated evaluation table; checked Oct 24, 2026).
- USMLE-style: MedGemma technical report v4, Table 3 (Apr 6, 2026); arXiv 2607.10849v1, Table 2 (Jul 12, 2026); Microsoft Medprompt paper (arXiv 2311.16452), Table 1 (Nov 2023); Med-PaLM 2 research paper (2023); BioMistral authors’ repository, Supervised Fine-tuning Benchmark (2024 report; repository snapshot checked Oct 24, 2026); OpenBioLLM 70B author model card, evaluation table (Undated evaluation table; checked Oct 24, 2026).
- PubMedQA: MedGemma technical report v4, Table 3 (Apr 6, 2026); PubMedQA official results page (Nov 2022–Mar 2024 entries; live page checked Oct 24, 2026); arXiv 2607.10849v1, Table 2 (Jul 12, 2026); BioMistral authors’ repository, Supervised Fine-tuning Benchmark (2024 report; repository snapshot checked Oct 24, 2026); OpenBioLLM 70B author model card, evaluation table (Undated evaluation table; checked Oct 24, 2026); Jev in Medicine v4, Table 2 (Oct 10, 2026 (v4); runs Sep 26–27, 2026).
Transcription and dates
Scores are copied to one decimal place. Results are grouped into explicit evaluation cohorts: one source, one metric, one protocol, one dataset split. Cohorts are never combined or ranked against each other; raw and length-adjusted scores, and nonstandard splits such as a 1,000-question PubMedQA set, stay separate. Each cohort shows its exact source publication date and a separate verification date (last checked Oct 24, 2026). There is no automatic updating. A newer model may have no published score on a test; a missing model is not a zero, and latest scores are never guessed. Scored-subset accuracies that exclude refusals are noted but not ranked.
Scoring limits
- Metrics differ: rubric scores, length-adjusted scores and multiple-choice accuracy are not on a shared scale.
- Setups differ: prompting, tool use, browsing and system wrappers change results, and sources do not always report them in the same way.
- Rankings are within one benchmark view only. Model ranks across views must not be combined.
- Benchmark performance is not clinical validation and says nothing about safety in real care settings.
MedQA and USMLE overlap
MedQA-English is built from USMLE-style board exam questions. The Med-PaLM 2 paper reports these as MedQA (USMLE). This index exposes both labels because researchers search for both, but both views share the exact same cohorts. Totals count them once. Treat them as one evaluation, never two.
Source types
Every cohort carries an explicit source type: official vendor report, official benchmark leaderboard, independent evaluation, independent preprint or secondary transcription. Original model-author research reports are vendor reports, not independent replications. Each benchmark opens on a multi-source model directory with at least ten scored model versions, not a combined ranking. Known dated 2026 reports are selected newest first; undated current sources follow, without inferring publication dates. Historical cohorts remain in the archive. Separate source-level comparisons remain available, including Meta, Baichuan and OpenAI on Hard. Artificial Analysis’s 14 Professional scores are checked against its original chart. Secondary transcriptions never receive an official badge.
Coverage is not exhaustive. Records include historical runs, fine-tuned models, ensembles and repeated appearances across reports. Some author comparison tables use mixed protocols or omit exact split details; those limitations are stated alongside the results. A source-backed number is not proof of reproducibility, clinical safety or a fair head-to-head comparison.
Independent testing status
Dates and time zones. Machine-readable run timestamps use UTC, with their offset retained. For example, the completed Professional run at 2026-10-25 01:48 UTC was completed on October 24, 2026 at 6:48 p.m. in Los Angeles (PDT). Publication dates from outside sources remain as printed; a UTC date one day ahead of a reader’s local date does not indicate a future evaluation.
Source directory plus explicitly identified team-run evaluations. Most scores on this site come from published sources. HealthBench AI also operates its own evaluations, labelled “Evaluated by HealthBenchAI”. Our individual model evaluation articles contain results, comparisons, costs and supporting evidence. Each completed 525-task model replaces its old current-coverage score; original reports remain in source history. Team runs follow this protocol:
- Pin the official dataset commit and split (for example openai/simple-evals for HealthBench and pubmedqa/pubmedqa for PubMedQA).
- Record the exact model snapshot, API version, reasoning effort, temperature and system prompt.
- Name the grader model and the number of repetitions.
- Report raw and length-adjusted scores separately.
- Count API errors and refusals as failures and publish their rates.
- Set a cost cap before any run.
- Publish reproducible logs and configs. No patient data is used or stored.
Corrections
If a score, date or link is wrong, use the contact form. New sourced results can be proposed through the application form; nothing is published automatically.
Frequently asked questions
Which model has the highest HealthBench Hard score?
There is no single comparable global ranking. Baichuan’s M3 report lists M3 at 44.4 in its own evaluation. Muse Spark Thinking leads the default Meta cross-provider comparison at 42.8, ahead of GPT-5.4 Xhigh at 40.1. OpenAI’s separate length-adjusted cohort is led by GPT-6 Astra at 36.6. Evaluators and protocols differ; raw and length-adjusted scores are not comparable. Sources checked Oct 24, 2026.
Which model leads HealthBench Professional?
It depends on the cohort; they are never combined. In OpenAI’s GPT-6.1 Sol system card, GPT-6 Astra leads at 64.7. Anthropic’s Claude Sonnet 5.5 system card (Claude Opus 4.8 judge) lists GPT-6 Astra 70.3 and Claude Sonnet 5.5 69.2. Artificial Analysis’s original Oct 19, 2026 chart lists GPT-6 Astra (Max) 66.0. Its 14 scores have been checked against the primary chart. Judges differ, so these remain separate.
Which model leads MedQA and PubMedQA?
The default is Google’s official MedGemma technical report v4 (Apr 6, 2026): o3 leads MedQA at 93.3% and the standard 500-question PubMedQA at 80.0%. Independent preprint arXiv 2607.10849v1 reports GPT-5 at 96.0% on MedQA, and its PubMedQA uses a nonstandard 1,000-question set. The official PubMedQA leaderboard lists GPT-4 (Medprompt) at 82.0%.
Has HealthBench AI run its own tests?
Yes. HealthBenchAI completed 525-task Professional evaluations of GPT-6.1 Sol Max (64.63 length-adjusted points) and Claude Fable 5.1 high (57.97), using a no-web API harness and GPT-5.4 low grader. Our separately configured, tool-assisted Nox Orion 2.1 evaluation scored 69.53. Individual articles provide protocol details, comparisons and evidence. Different tools and settings prevent a controlled head-to-head interpretation. These are team-run evaluations, not third-party certification.
Is HealthBench Professional the same as HealthBench?
No. HealthBench has 5,000 conversations, and HealthBench Hard is a 1,000-example subset of it. HealthBench Professional is a separate benchmark of 525 physician-authored tasks scored with a length-adjusted metric.
Are MedQA-English and USMLE-style different results?
No. MedQA-English is the English USMLE-style exam set. Both pages share the exact same cohorts (default: Google MedGemma report v4, Apr 6, 2026), so totals count them once.
Can scores be compared across benchmarks?
No. Each benchmark uses a different task format, metric and evaluation setup. An 80% PubMedQA accuracy and a 30% HealthBench Hard score measure different things and cannot be ranked against each other.
Does a high benchmark score mean a model is safe for clinical use?
No. Benchmark results describe performance on specific datasets under specific conditions. They are not clinical validation, regulatory clearance or medical advice.
How often is this index updated?
Results are added only after manual review of a public source. There is no automatic refresh. Each cohort shows its exact source date and a separate verification date; sources were last checked Oct 24, 2026. A newer model with no published score is simply absent, not zero.