HealthBench AIClaude Fable 5.1: our Professional test report
Our completed 525-task HealthBench Professional test: scores, confidence interval, model settings and downloadable per-task evidence.
By HealthBench AI · REAGEN, Inc. ·
Tested by HealthBench AI. Our model evaluation, with run-specific results and supporting evidence below.
Our test result
Complete · 525/525 assessed57.97 length-adjusted · 64.57 raw
95% task-bootstrap interval: 53.38–62.48.
519 rubric-graded completions, including 2 token-limited completions; 6 explicit no-answer API refusals assessed as fixed zeros under our disclosed local failure policy.
Model ID: claude-fable-5-1 · reasoning effort: high.
Same no-search, no-external-tools API harness; 16,384 output-token cap. Grader: GPT-5.4 (low), one request per rubric item. One run per task, with network recovery attempts disclosed in the run report.
Estimated cost including grading: $87.77; unresolved-request reserve: $0.00.
Professional methodology and evaluation settings · Download retained protocol and cost records (JSON)
Claude Fable 5.1 (high): our score vs other evaluations
All scores below are length-adjusted Professional points. Our run: high effort, GPT-5.4 low grader, one run per task, shared 16,384-token cap. Source settings may differ.
| Other evaluator / source | HealthBenchAI | Their score | Ours − theirs (pp) |
|---|---|---|---|
| Anthropic (vendor report) ↗ Claude Fable 5.1 (max) Source & evaluation settingsAnthropic Claude Sonnet 5.5 system card, section 8.15 · Oct 2026 Anthropic-run cross-model comparison. Judge: Claude Opus 4.8. All models at max effort; Claude safety classifiers enabled; OpenAI model sampled via public API with no system prompt. 1.47 points per 500 characters above or below 2,000. | 57.97 | 62.10 | −4.13 |
| Artificial Analysis ↗ Claude Fable 5.1 (Max, Default Fallback) Source & evaluation settingsArtificial Analysis original announcement and 14-model chart · Oct 19, 2026 525 clinician conversations, GPT-5.4 (low) judge, 4 repeats; subtract 1.47 points per 500 characters above 2,000; add the same amount for shorter responses. | 57.97 | 57.80 | +0.17 |
Different reasoning effort, safety fallback, judge, sampling and trial count can affect these scores. Publisher results remain attributed to their original evaluators; HealthBenchAI did not run those external evaluations.
What we tested
HealthBench AI tested Claude Fable 5.1 through the official Anthropic API, with adaptive thinking at high effort. We assessed every one of the 525 public HealthBench Professional tasks. This is our own model evaluation, not a transcription of a publisher score.
We used no browsing, search, retrieval or external tools for the tested model. The model received the dataset conversation, not physician reference answers or grading rubrics. Our GPT-5.4 low grader assessed each rubric item separately using the public HealthBench reference prompt.
How to interpret this result
The result above uses the documented Professional length adjustment: raw percentage points − 1.47 × (visible response characters − 2,000) / 500. The 95% interval is a 10,000-resample task bootstrap, not an estimate of model-to-model or grader uncertainty.
Six explicit no-answer API refusals received fixed zeros under our disclosed local failure policy, not an asserted official automatic-zero rule. Two visible answers reached the common output-token cap; we graded those actual partial answers without extending them or grading hidden reasoning. These outcomes are individually marked in the evidence.
The completed result replaces the older Fable 5.1 score in our current Professional coverage. Original publisher results remain in their source cohorts. This high-effort result is not a claim to reproduce a vendor’s max-effort run, unpublished internal environment or clinical validation.
Evidence you can check
Download the model-specific JSON above for all 525 task IDs, rubric weights and verdict booleans, raw and adjusted task scores, answer lengths and response hashes, together with the dataset revision, grader-prompt hash, protocol, costs and aggregate result.
The arithmetic is reproducible from these records. Raw prompts, physician answers, rubric text and model answer transcripts remain private to respect the benchmark publisher’s anti-contamination request. Hashes support integrity checks; they do not constitute third-party certification.
Sources and supporting evidence
Explore the benchmark results · Read our methodology · Suggest a correction