HealthBench AIPubMedQA: interpreting yes, no and maybe
Our guide to abstract-based biomedical question answering, test splits and comparable PubMedQA scores.
By HealthBench AI · REAGEN, Inc. ·
Our editorial note. This page is written by HealthBench AI. Research findings and submitted results belong to their cited sources; this is our explanation, not a republication of their articles or an endorsement by their authors.
A different kind of health AI task
PubMedQA asks a model to answer a biomedical research question using information from a PubMed abstract. The target answer is yes, no or maybe. The task evaluates evidence-based question answering, rather than a free-form clinical consultation or an exam-style diagnosis.
The maybe label matters: an abstract may not support a definite affirmative or negative answer. Interpreting the evidence and its uncertainty is part of the task, not a reason to collapse it into a two-choice test.
Check the split before comparing scores
The expert-labeled dataset has 1,000 questions, with a standard 500-question held-out test set. The full collection also includes additional subsets used for other training and research purposes. A result on all 1,000 expert-labeled items is not automatically equivalent to a result on the standard test split.
Prompting, fine-tuning, extra training data and answer extraction can change accuracy. A model name and a percentage are not enough to establish comparability; the dataset split and evaluation protocol should be reported alongside them.
How to use our PubMedQA page
HealthBench AI links each score to its original paper, benchmark entry or model report. Nonstandard splits remain labelled, and scores that exclude refusals are not silently substituted for all-question accuracy.
PubMedQA is distinct from MedQA-English. MedQA-English and our USMLE-style view share the same underlying exam dataset; PubMedQA does not. We keep those results separate and do not average them into a universal clinical score.
Sources and supporting evidence
Explore the benchmark results · Read our methodology · Suggest a correction