HealthBench AIOverviewSubmit a resultBenchmark definitions
Five views, four distinct datasets. Each definition explains what is tested, which metric is reported and where the numbers come from.
HealthBench Hard
A 1,000-example challenging subset of HealthBench (5,000 conversations). Cohorts differ in scoring: some are length-adjusted, some raw, so each is shown separately.
HealthBench Professional
A separate benchmark of 525 physician-authored clinical conversations scored with a length-adjusted metric. Independent and vendor cohorts are shown separately.
MedQA-English
Published results on the English MedQA (USMLE-style) test set. The USMLE-style view shows the identical cohorts; they are one evaluation, not two.
USMLE-style
USMLE-style questions as reported in MedQA evaluations. This view shares the exact MedQA-English cohorts and is not counted separately.
PubMedQA
Models answer yes, no or maybe to biomedical research questions using PubMed abstracts. Includes an October 2026 independent report on the standard 500-question test; nonstandard splits are labelled separately.
How they relate
HealthBench contains 5,000 conversations; HealthBench Hard is its 1,000-example difficult subset. HealthBench Professional is a separate set of 525 physician-authored tasks. MedQA-English and USMLE-style share the same cohorts and are counted once. Each benchmark lists separate evaluation cohorts that are never combined. PubMedQA tests yes / no / maybe answers over PubMed abstracts. See the methodology for scoring limits.