HealthBench AIHealthBench and Hard: a guide to the distinction
How we interpret the 5,000-conversation HealthBench dataset and its 1,000-example difficult subset.
By HealthBench AI · REAGEN, Inc. ·
Our editorial note. This page is written by HealthBench AI. Research findings and submitted results belong to their cited sources; this is our explanation, not a republication of their articles or an endorsement by their authors.
Beyond exam-style accuracy
OpenAI introduced HealthBench as an evaluation of realistic health conversations. Its 5,000 conversations are assessed using physician-created rubrics, allowing an evaluation to consider several aspects of a response instead of marking only one selected answer right or wrong.
HealthBench Hard is the 1,000-example difficult subset. HealthBench Professional is a separate dataset, not another name for Hard. A score must identify the exact benchmark before it can be interpreted.
What a rubric score represents
Weighted rubric items can reward useful content and penalize unsafe or incorrect content. The resulting score summarizes performance under that evaluator’s rubric and grading setup. It is not an estimate of diagnostic accuracy in an actual hospital.
Length-adjusted results must be separated from raw results. A Professional length-adjustment coefficient should not be reused for Hard. Response extraction, the grader and the handling of failures all affect reproducibility.
How we present the evidence
Our Hard page keeps published evaluations in named source cohorts and identifies the model versions actually tested. An older model in a recent comparison is still that older model; we do not silently relabel it as its provider’s newest release.
The current-coverage view is a directory, not a controlled cross-provider experiment. Use source-level comparisons for within-protocol rankings and read the limitations before interpreting a high score. Benchmark performance alone is not clinical validation or medical advice.
Sources and supporting evidence
Explore the benchmark results · Read our methodology · Suggest a correction