A Better Benchmark Is Not a Better Scientist

At BenchSci, I worked across several layers of scientific AI: extracting evidence from papers and figures, representing it in knowledge graphs, retrieving it for scientists, and building agents for literature triage, scientific question answering, hypothesis generation, and experiment recommendation. That experience changed how I think about evaluating scientific agents.

The obvious evaluation is rarely the one that matters most.

A retrieval model can improve mean average precision. A scientist can prefer one agent’s answer over another. A judge can give a mechanistic hypothesis a higher score. Each result is useful. None establishes that the system will make research more reliable or direct experimental capacity toward better questions.

A scientific agent is not merely producing text. It is influencing what evidence a scientist reads, which hypothesis looks promising, and which experiment gets run next. The evaluation therefore has to follow the decision, not stop at the answer.

Start with the failure, not the metric

Suppose a production agent searches literature and a disease-biology knowledge graph, synthesizes a mechanistic hypothesis, and recommends an experiment.

A new checkpoint looks clearly better than the baseline:

scientists prefer its responses by 14 percent on an internal comparison;
it retrieves more relevant evidence;
it reasons over longer biological paths;
its experiment recommendations are more specific and actionable.

During red-teaming, however, it confidently presents several causal hypotheses whose evidence is incomplete, indirect, or contradictory. In one case it treats co-expression as causal support. In another it extrapolates from one cell line to a disease context without qualification. In a third it recommends an experiment without a control that would distinguish the proposed mechanism from the most likely alternative.

The wrong question is:

Which model has the higher average score?

The useful question is:

Does the new model improve scientific decisions without increasing confident, consequential errors?

The failure to prevent is:

$$ \boxed{ \text{weak or incomplete evidence} \land \text{consequential scientific claim} \land \text{unjustified confidence} \land \text{no attempt to resolve the uncertainty} } $$

That definition comes before ECE, Brier score, preference rate, or any other metric. It identifies the behavior that could waste experimental capacity or redirect a research program.

A scientific answer is a sequence of decisions

Token probabilities are not the right abstraction. Neither is a single scalar called agent_confidence.

Given a scientific question, the agent has several legitimate actions:

$$ x \rightarrow \begin{cases} \text{answer from established evidence},\\ \text{retrieve more evidence},\\ \text{surface a contradiction},\\ \text{ask for experimental context},\\ \text{propose a discriminating experiment},\\ \text{defer to a domain expert}. \end{cases} $$

Calibration is partly about selecting the right action. If evidence is sufficient, another search wastes time. If it is not, producing a more fluent answer makes the system worse. If two mechanisms explain the observations, the right output may be an experiment that separates them rather than a stronger claim about either one.

The behavioral contract should therefore distinguish at least four questions:

Evidence: Did the system find the relevant observations, including contradictory results?
Reasoning: Does the evidence support the stated biological relationship?
Communication: Does the response separate observation, inference, and speculation?
Action: Is the proposed next experiment informative, feasible, and appropriately controlled?

These are related, but they are not interchangeable. A biologically plausible hypothesis can be poorly supported. A well-supported association can still fail to establish causality. A reasonable experiment can be uninformative if both the main hypothesis and its alternative predict the same outcome.

Evaluate the system in layers

Scientific agents often contain a chain of learned and deterministic components:

$$ \text{query} \rightarrow \text{retrieval} \rightarrow \text{entity resolution} \rightarrow \text{graph traversal} \rightarrow \text{evidence synthesis} \rightarrow \text{hypothesis} \rightarrow \text{experiment} $$

An end-to-end score tells us whether the response failed. It rarely tells us why.

I would annotate a failure against the earliest broken contract:

Layer	Question	Example failure
Retrieval	Was the necessary evidence available to the model?	A contradictory paper was never retrieved.
Extraction	Was the evidence represented correctly?	A figure’s control and treatment labels were reversed.
Graph	Was the biological path valid?	Two entities were connected through an ambiguous synonym.
Synthesis	Did the conclusion follow from the evidence?	Correlation was rewritten as causation.
Recommendation	Would the experiment resolve the uncertainty?	The proposed readout cannot distinguish two mechanisms.
Communication	Was confidence appropriate?	A tentative hypothesis was stated as established fact.

This decomposition is operationally important. Post-training the reasoning model will not repair a missing graph edge. Improving retrieval will not fix an agent that consistently overstates causal evidence. Adding generic caution to the prompt can make every answer sound uncertain without improving a single decision.

Build the evaluation distribution around risk

A generic scientific question-answering set is not enough. I would build five complementary evaluation sets, each answering a different question.

Representative workflow traffic

Sample the intended mix of literature triage, target-disease association, mechanism-of-action questions, biomarker exploration, experimental troubleshooting, and experiment recommendation.

Stratify by therapeutic area, target class, evidence density, question type, and user expertise. This set estimates average product utility. It should not be expected to provide enough examples of rare, expensive failures.

Risk-enriched cases

Oversample the situations in which a persuasive answer is most likely to be misleading:

sparse or conflicting literature;
novel targets and long knowledge-graph paths;
evidence from only one organism or cell line;
observational evidence without a perturbational result;
claims that depend on a scientific figure or supplementary material;
ambiguous gene, protein, or disease entities;
experiments with missing controls;
disagreement between retrieval, graph, and model-based signals.

This set does not estimate the production base rate directly. It supplies the statistical power needed to characterize important failure modes.

Counterfactual pairs

Counterfactual cases test whether the model responds to evidence rather than continuing a familiar scientific narrative.

Consider this pair:

Knocking out gene X reduces proliferation in a disease-relevant cell line.

Now add one fact:

The same knockout also reduces proliferation in healthy control cells.

The second result weakens a disease-specific mechanistic claim and changes the next experiment. The agent’s confidence and recommendation should change with it.

Other useful interventions include removing a key paper, changing the model organism, replacing a perturbational study with a correlational one, or adding a contradictory result. These tests reveal whether confidence is actually sensitive to evidence quality.

Temporal holdouts

Provide only evidence available before a historical cutoff, then evaluate the agent’s hypotheses against experiments or publications that appeared later.

This is not equivalent to a prospective study, but it is stronger than asking an expert whether a hypothesis sounds plausible while both the model and evaluator can see the eventual answer. Time-based splits also reduce leakage between closely related papers and experimental programs.

A hidden discovery set

Scientists and red-teamers should continuously search for cases where the new model invents a causal interpretation, misses decisive contradictory evidence, or proposes a non-discriminating experiment.

This set should rotate and remain outside the training loop. Once a team repeatedly optimizes against an evaluation, it becomes part of the training environment and loses much of its value as a scientific instrument.

Turn expert judgment into a measurement process

“Have scientists rate the answers” is not yet an evaluation design.

A useful rubric forces the evaluator to inspect claims and decisions separately:

Does every material claim have supporting evidence?
Does each citation entail the claim attached to it?
Is the evidence direct, indirect, or merely consistent with the hypothesis?
Were credible conflicting findings surfaced?
Are the disease, organism, tissue, and experimental contexts aligned?
What observation would falsify the hypothesis?
Does the experiment discriminate among plausible mechanisms?
Are the essential positive, negative, and orthogonal controls present?
What is the cost of acting on this recommendation if it is wrong?

Multiple experts should label the consequential subset independently. Their disagreement should be measured, not silently averaged away. Scientific ambiguity is part of the target: a model should not be rewarded for sounding certain where qualified experts disagree.

Expert time is most valuable when allocated asymmetrically. Use an initial block to define the rubric and build a high-quality calibration set. Spend the remaining time on:

cases where the models disagree;
high-confidence recommendations;
novel or sparse-evidence questions;
high-severity failures;
automated-grader uncertainty;
newly discovered failure clusters.

This produces far more information per expert-hour than uniformly grading thousands of routine answers.

An LLM judge is useful only where it has been validated

At the scale required for model development, expert review needs a scalable complement. An LLM-as-a-judge can apply a scientist-authored rubric across a much larger evaluation set, but aggregate agreement is a weak validation target.

The relevant quantity is closer to:

$$ P( G_{\text{judge}}=G_{\text{expert}} \mid \text{severity},\text{task},\text{model disagreement} ). $$

A judge can reach 95 percent agreement overall while failing on the five percent of examples that determine whether the system is safe to release. I would report agreement separately for unsupported causal claims, missing controls, contradictory evidence, novel hypotheses, and high-confidence model outputs.

The judge should also see the retrieved evidence and score atomic claims rather than the fluency of the complete response. Otherwise it can reward the same persuasive style that created the failure.

Calibration should control the next action

For a classifier, calibration often asks whether predictions assigned 80 percent confidence are correct about 80 percent of the time. A scientific agent requires several named events:

$$ \begin{aligned} c_{\text{retrieval}} &=P(\text{the necessary evidence was retrieved}),\\ c_{\text{support}} &=P(\text{the evidence supports the stated claim}),\\ c_{\text{experiment}} &=P(\text{the proposed experiment resolves the stated uncertainty}),\\ c_{\text{outcome}} &=P(\text{the experiment meets its prespecified success criterion}). \end{aligned} $$

The probabilities have different labels, denominators, and consequences. They should not be collapsed into one confidence score.

I would also evaluate selective risk. Let coverage $c$ be the fraction of cases in which the agent makes a substantive mechanistic claim without searching again, requesting context, or deferring to a scientist. Then measure:

$$ R(c)=P( \text{materially unsupported claim} \mid \text{agent answers at coverage }c ). $$

The risk-coverage curve asks whether unsupported claims actually become rarer as the agent becomes more selective. It is more informative than counting how often the model uses phrases such as “the evidence is limited.”

For experiment recommendation, uncertainty alone is not enough. The agent should prefer an additional experiment when the expected value of information exceeds its cost:

$$ \begin{aligned} \operatorname{EVI}(e) &=\mathbb{E}\left[\max_a U(a\mid D,e)\right] \\ &\quad-\max_a U(a\mid D)-C(e). \end{aligned} $$

Here $D$ is the current evidence, $e$ is a proposed experiment, $a$ is a subsequent action, $U$ is scientific utility, and $C(e)$ is the experiment’s cost. The expression is conceptual rather than directly observable in most workflows, but it clarifies the desired behavior: request evidence that can change the decision, not evidence that merely makes the report longer.

The model calibration guide covers the underlying reliability diagrams, proper scoring rules, and calibration methods. The important extension here is that calibrated signals should change the agent’s policy.

Compare models with paired decisions

Both models should answer the same cases with access to the same evidence. This supports paired estimates of the difference:

$$ \Delta=M(B)-M(A). $$

Report bootstrap confidence intervals for scientist preference, claim support, contradiction recall, experiment quality, and severity-weighted failures.

Rare-event safety needs special care. If neither model produces a catastrophic error in a small sample, that does not prove equivalence. The release decision should consider the upper confidence bound on the regression that remains compatible with the data. Risk-enriched sampling and high-volume automated evaluation help, but the final high-severity cases still require expert review.

The evaluation should also separate two costly error types:

false advance: confidently pursuing a weak hypothesis or uninformative experiment;
false rejection: dismissing a valuable hypothesis because the evidence is novel, sparse, or outside the model’s familiar distribution.

An agent that avoids every false advance by recommending more review on every case is safe-looking but not useful.

Release with gates, not a blended score

I would not trade one severe scientific regression against many small writing or retrieval improvements through a weighted average.

The release rule should look like:

$$ \boxed{ \begin{aligned} \text{SHIP} &=\text{UtilityImproves} \land \text{GroundingHolds} \land \text{TailRiskHolds} \land \text{CalibrationImproves} \land \text{JudgeValidated} \end{aligned} } $$

Gate	Example release requirement
Scientific utility	A statistically credible gain in blinded expert preference or task completion.
Evidence grounding	No regression in claim support, citation entailment, or contradictory-evidence recall.
Experiment quality	Better discriminating power and control completeness at comparable cost.
Calibration	Lower unsupported-claim risk at the same answer coverage.
Tail risk	No meaningful increase in confident, high-severity scientific errors.
Robustness	Gains persist across therapeutic areas, novel targets, and sparse-evidence cases.
Evaluator validity	The scalable judge remains reliable on release-determining slices.

If the new model improves average utility but fails a tail-risk gate, the answer need not be either global deployment or complete rejection. It can be released behind a narrower policy: show the evidence path, require scientist review for novel targets, or use it for retrieval and experiment ideation without allowing it to state a final mechanistic conclusion.

The restricted workflow must itself be evaluated. Human review is not a magic safety layer if the model’s fluency anchors the reviewer or hides missing evidence.

Let the failure shape the next training run

Suppose the new model is more useful and more specific but repeatedly overstates causal claims. A plausible training diagnosis is:

The objective rewarded persuasive, actionable answers without adequately rewarding evidence completeness, falsifiability, and calibrated deferral.

The next training environment should include cases where the optimal action is to retrieve again, surface a contradiction, ask for context, or propose a test rather than answer directly.

A structured reward might include:

$$ \begin{aligned} r={}&r_{\text{utility}} +r_{\text{evidence support}} +r_{\text{falsifiability}} +r_{\text{appropriate uncertainty}}\\ &-r_{\text{unsupported causality}} -r_{\text{missing controls}}. \end{aligned} $$

The terms still need adversarial validation. A model can game an uncertainty reward by attaching generic caveats to every claim. It can game an experiment reward by recommending exhaustive panels. It can game a citation reward by adding many irrelevant sources.

The real loop is therefore:

$$ \text{observed failure} \rightarrow \text{component diagnosis} \rightarrow \text{evaluation} \rightarrow \text{training intervention} \rightarrow \text{hidden validation}. $$

Evaluation is not the report at the end of training. It is the mechanism that turns a production failure into the next research hypothesis.

Bottom line

A better scientific agent does not merely retrieve more relevant papers, write a more plausible mechanism, or win a preference comparison. It helps a scientist take a better next step.

That requires an evaluation spanning the entire workflow:

$$ \text{evidence} \rightarrow \text{reasoning} \rightarrow \text{uncertainty} \rightarrow \text{experiment} \rightarrow \text{observed result}. $$

Start by naming the consequential failure. Build representative and risk-enriched evaluations. Use counterfactuals to test sensitivity to evidence. Validate automated judges on the cases that matter most. Measure whether uncertainty changes the next action. Release only when average utility improves without hiding a worse tail.

The benchmark is evidence. The scientific decision is the product.