GPT-6.1 Sol Max: our Professional test report

GPT-6.1 Sol (max): our score vs other evaluations

All scores below are length-adjusted Professional points. Our run: max effort, GPT-5.4 low grader, one run per task, shared 16,384-token cap. Source settings may differ.

Different reasoning effort, safety fallback, judge, sampling and trial count can affect these scores. Publisher results remain attributed to their original evaluators; HealthBenchAI did not run those external evaluations.

What we tested

HealthBench AI completed its GPT-6.1 Sol evaluation through the official OpenAI API at max reasoning effort. We assessed all 525 public HealthBench Professional tasks under the same no-search, no-external-tools API harness and output-token cap used for our Claude Fable 5.1 evaluation. Our result is 64.63 length-adjusted and 67.56 raw, with a 95% task-bootstrap interval of 60.78–68.35.

Each task uses the official dataset conversation. The tested model is not given physician reference answers or rubrics. GPT-5.4 low grades one rubric item per request using the public HealthBench reference prompt; the weighted result is then length-adjusted.

Final result and the two empty answers

All 525 tasks are now rubric-graded. Our 64.63 replaces the previous Artificial Analysis score of 65.1 in current Professional coverage; the original result remains in its source cohort. The approximately 0.47-point numerical difference is not a controlled regression claim: evaluation configurations and graders may differ.

Eight completions hit the common 16,384-token output cap. Two of them had no visible final answer. We reused those exact cached empty outputs, without regeneration or a higher cap, and sent the actual empty assistant messages to GPT-5.4 low for five independent rubric decisions. All five verdicts were false, giving both tasks raw 0.

The standard length-adjustment formula adds 5.88 percentage points for a zero-character final answer, so each of those two tasks contributes adjusted 5.88, not a manually forced adjusted zero. Their explicit outcome is rubric_graded_empty_truncated. This formula edge case is disclosed, not interpreted as a useful clinical answer.

Network timeouts do not silently become wrong answers or disappear from the accounting. Failed attempts retain their uncertain cost reservations; bounded recovery uses longer network waiting time without raising reasoning effort or the model output-token cap.

Visible answers that reach the token cap are graded as generated and labelled in the evidence. Explicit API refusals with no answer receive fixed zeros under the disclosed local failure policy; visible refusal text is rubric-graded normally. Empty token-budget finals are a separate outcome and are never relabelled as safety refusals.

Evidence and limitations

The completed model-specific JSON contains all task IDs, actual rubric verdicts, weights, scores, response lengths, hashes, the pinned dataset revision, grader-prompt hash, costs and protocol. It supports recomputing the reported arithmetic without exposing benchmark task text.

The documented Professional length adjustment is raw percentage points − 1.47 × (visible response characters − 2,000) / 500. Confidence intervals use 10,000 task-bootstrap resamples. One run per task does not measure run-to-run variability, and an OpenAI-family grader may introduce evaluator-family bias.

Sources and supporting evidence

• HealthBench AI: retained protocol and costs (JSON)
• HealthBench AI: per-task evidence and hashes
• OpenAI: Professional dataset and evaluation caveats