Accuracy vs calibration

Model calibration tests whether predicted probabilities match observed event frequencies:

When a model says “70% probability,” the event should actually happen about 70% of the time.

Accuracy counts how often the final decision is correct. Calibration asks whether the probabilities attached to those decisions have the right numerical meaning. A model can therefore be accurate while still assigning unjustified confidence.

Calibration matters whenever an LLM’s confidence affects a decision:

The percentages in these examples are hypothetical. A confidence value becomes meaningful only after the event, reference standard, population, and operating conditions are specified. In high-stakes settings such as medicine, calibrated confidence should support triage and verification rather than replace clinical review.

A binary classifier predicts whether an object belongs in a room. Consider 1,000 predictions for which it outputs approximately:

If the model is perfectly calibrated, the object should truly belong in the room in roughly 800 of those 1,000 cases.

A model can make every decision correctly while still being poorly calibrated.

Two models make exactly the same classification decisions:

Both get 100% accuracy at threshold 0.5.

Model B assigns 0.98-0.99 probability to its chosen label on every example. If it makes even occasional mistakes, those implied confidence levels are unjustified.

Perfect calibration requires:

rather than simply asking whether $\arg\max \hat P = Y$.

Reliability diagrams

A reliability diagram compares predicted confidence with observed event frequency. For a classifier or LLM confidence score, define the event as “this prediction is correct.” Then $\hat p_i$ is the confidence assigned to the chosen label and $y_i$ records whether that choice was correct.

Take predictions and place them into confidence bins:

• 0-10%
• 10-20%
• …
• 90-100%

For every bin $B_m$, calculate its mean predicted confidence:

and its observed accuracy:

Then plot mean predicted confidence on the x-axis against observed accuracy on the y-axis. A perfectly calibrated model lies on the diagonal:

The following five bins describe an overconfident model:

The model is increasingly overconfident at high probabilities. In the last bin, “95% sure” actually means “correct about 72% of the time.” That 23-point gap is a serious problem if a downstream system treats 0.95 as near-certainty.

Expected calibration error

Expected calibration error (ECE) summarizes the gaps between confidence and accuracy in a reliability diagram.

For bins $B_1,\ldots,B_M$ containing $n$ predictions in total:

Each bin’s absolute calibration gap is weighted by the fraction of predictions that landed in that bin. Perfect calibration gives $\operatorname{ECE}=0$; larger values indicate a greater mismatch between stated confidence and observed accuracy.

Example

Three confidence bins contain:

Then:

Therefore:

Informally, the model’s confidence is misaligned with empirical correctness by about 8 percentage points under this binning.

Limitations of ECE

ECE depends on the binning strategy and sample size, so it is not a definitive calibration score by itself.

Number of bins

With 10 bins:

a model might have $\operatorname{ECE}=0.03$. With 50 bins, the same predictions might produce $\operatorname{ECE}=0.07$ because the finer partition exposes more local calibration errors.

Sample count

Small bins have noisy estimates. A bin containing 12 predictions might have mean confidence:

but only 7 successes:

Sampling noise can explain much of that gap because the bin contains so few observations.

Aggregation effects

Coarse grouping can hide local errors. Consider predictions with probabilities:

that fall into the same broad bin with a mean of $0.60$. The bin as a whole might appear calibrated even when the probability function is not.

ECE is useful for comparing model versions, but it is not a complete calibration report. Its value changes with the number of bins and the binning strategy, and large errors in sparse bins can disappear inside the weighted average. A useful ECE report should therefore include:

• a reliability diagram
• the number of predictions in each bin
• equal-mass bins, which place similar numbers of predictions in each bin
• uncertainty intervals

Brier score

The Brier score avoids binning entirely. For $N$ binary predictions:

where $y_i\in\{0,1\}$. It is simply mean squared error for probabilities.

Example

The model predicts:

and the outcomes are:

The squared errors are:

Therefore:

Lower is better, and $\operatorname{BS}=0$ means perfect probabilistic prediction.

Calibration and sharpness

A useful probabilistic model needs both calibration and sharpness. Calibration means that predictions reported at 80% are correct about 80% of the time. Sharpness means that the model moves probabilities away from the base rate, the overall event frequency, when the evidence supports doing so. Predictions such as $0.02,0.97,0.93$ are sharper than $0.49,0.53,0.51$.

ECE focuses on calibration. Brier score measures overall probabilistic quality, so it rewards both calibration and sharpness.

Two calibrated models show why sharpness matters. Model A always predicts:

The dataset is 50% positive, so Model A is calibrated but cannot distinguish positives from negatives.

Model B predicts 0.9 for many positives and 0.1 for many negatives, and those probabilities match the observed frequencies. It is also calibrated, but much more useful.

ECE could say:

while Brier score strongly prefers Model B. Calibration alone is not enough.

The goal is:

not merely calibrated. Brier score captures both requirements.

Brier score decomposition

The Brier score decomposition separates overall probabilistic error into calibration, discrimination, and dataset uncertainty.

First define the event. For a correctness forecast, let $y_i=1$ when the answer is correct and $y_i=0$ when it is wrong. For a drug-discovery forecast, the event might instead be whether a candidate crosses a prespecified assay threshold.

Divide the predictions into forecast groups $B_m$. Each group has:

• $n_m$: the number of predictions in the group;
• $\bar p_m$: the probability assigned to that group;
• $\bar y_m$: the observed event rate in that group;
• $\bar y$: the event rate across the entire dataset.

The group event rate is:

The group event rate $\bar y_m$ is an observed frequency, not another model prediction. If three of four answers in a group are correct, then $\bar y_m=3/4=0.75$. When the event is “the answer is correct,” the event rate is the accuracy within that group.

The Brier decomposition is:

or, written in terms of the forecast groups:

Each term answers a different question:

Reliability is a calibration penalty. Resolution is subtracted because separating low-event-rate groups from high-event-rate groups improves the forecast. Uncertainty is the baseline difficulty of the dataset and does not depend on the model.

The decomposition is exact when every prediction in a group has the same forecast value. If an interval bin contains different probabilities and $\bar p_m$ replaces them all, the result is a binned approximation unless within-bin variation is included.

Worked example

Eight predictions fall into two groups, and the event is a positive outcome:

The low group has a 25% event rate because the event occurred once in four cases. The high group has a 75% event rate because it occurred three times in four cases. Across the full dataset, it occurred four times in eight cases, so:

Both groups contain half of the samples, so each has weight $n_m/N=0.5$. The reliability penalty compares each forecast with what happened in its group:

The resolution reward compares each group’s event rate with the overall 50% event rate:

The uncertainty is determined only by the overall event rate:

Putting the three pieces together gives:

Predicting the 50% base rate for every case would produce the uncertainty baseline of $0.25$. Separating the cases into 25% and 75% event-rate groups improves the score by $0.0625$. Slightly missing those group rates with forecasts of 20% and 80% adds back a calibration penalty of $0.0025$, producing the final Brier score of $0.19$.

The decomposition shows why Brier score captures more than calibration alone: it rewards probabilities that match observed frequencies and predictions that separate cases with meaningfully different outcomes.

Brier score vs log loss

Negative log-likelihood (NLL), also called log loss or binary cross-entropy, is another proper scoring rule:

Both Brier score and NLL are proper scoring rules: in expectation, the best strategy is to report the probability you actually believe. But they punish errors differently.

For $y=0$ and a prediction of $p=0.999999$, the Brier penalty is:

while the log-loss penalty is:

Log loss therefore punishes a confidently wrong prediction much more severely. The stronger penalty is useful when unjustified certainty is especially costly.

Multiclass calibration gets trickier

Top-label calibration reduces a multiclass probability vector to its highest predicted probability:

Top-label calibration then checks whether predictions at that confidence are correct at the same rate. A model outputs:

The top-label confidence is 0.6, and the other two probabilities are discarded. Discarding them can hide differences between distributions.

For example:

and:

look identical to top-label ECE even though their uncertainty structures are different.

Classwise calibration instead checks every class probability:

for every class $k$. Multiclass Brier score naturally uses the whole probability vector:

where $y_{ik}$ is 1 for the true class and 0 for every other class, a representation called one-hot encoding. Some libraries also divide this quantity by the number of classes, so the convention should be recorded when results are compared.

Calibration can be conditional

Conditional calibration tests whether probabilities remain reliable within important subgroups. An aggregate result can hide a failure. The full dataset might show:

The same predictions split by context might show:

while:

The subgroup errors cancel in aggregate, but the 0.8 score has different meanings in bedrooms and kitchens. Operationally, the model is not calibrated where it matters.

Calibration should therefore be inspected across relevant slices such as:

• classes
• spatial regions
• input complexity
• rare vs frequent objects
• in-distribution vs shifted data
• document types
• model versions
• confidence ranges

A single global ECE can hide all of these failures.

Why neural networks tend to be overconfident

Cross-entropy training does not guarantee finite-sample or out-of-distribution calibration. In an overparameterized network, training can continue increasing the margin between the raw class scores, called logits, even after the classification decision is already correct.

The following logits illustrate the problem:

Softmax converts those scores to probabilities by exponentiating and normalizing them. Here it produces a distribution close to:

But perhaps the evidence only justifies:

The predicted class is identical; only the claimed confidence is wrong. The probabilities are too extreme even though the class selection need not change.

Post-hoc calibration methods

Post-hoc calibration fits a mapping from a model’s existing scores to probabilities using a held-out calibration set. Three common methods form a rough spectrum of flexibility:

Temperature scaling

Given logits $z_1,\ldots,z_K$, ordinary softmax uses:

Temperature scaling instead uses:

A single positive scalar $T$ is fitted on a held-out calibration set, usually by minimizing NLL. Because it learns only one scalar for all logits, temperature scaling has low capacity and a correspondingly low risk of overfitting. When $T\gt1$, the distribution becomes softer.

For example, logits $[10,5,1]$ produce approximately:

With $T=2$, they become approximately:

Dividing every logit by the same positive temperature does not change their ordering, so the highest-probability class stays the same. Classification accuracy therefore stays the same. The reported probabilities can nevertheless become much better calibrated.

Platt scaling

Platt scaling learns a logistic transformation such as:

where $s$ is the model’s uncalibrated scalar score. The parameters $a$ and $b$ are fitted on a held-out calibration set, typically by minimizing binary NLL. The slope $a$ changes the scale, while the intercept $b$ corrects a systematic bias.

A fitted calibrator might produce:

A raw score of $s=2$ would give $\sigma(2)\approx0.881$ if treated directly as a logit. Platt scaling instead gives:

The same mapping transforms several scores as follows:

The fitted mapping says the original scores were both too large in scale and too optimistic around zero. Because $a\gt0$, the transformation preserves the ranking of examples even though their reported probabilities change.

Isotonic regression

Isotonic regression learns a flexible monotonic mapping $f(p)$ so that higher original scores still produce higher calibrated probabilities. It can fit more complex distortions, but it needs more calibration data and can overfit small samples. The standard pool-adjacent-violators algorithm enforces monotonicity by merging neighboring buckets whenever their observed rates are out of order.

Four equally sized score buckets have these observed success rates:

The middle two buckets violate monotonicity: the higher score, 0.60, succeeds less often than 0.40. The algorithm therefore merges those buckets. Because they contain the same number of samples, their pooled rate is:

The fitted mapping becomes:

A new score on the pooled plateau, such as 0.50, maps to 0.45. The method learns this shape directly from the calibration data rather than assuming a sigmoid. With unequal bucket sizes, the pooled value would be the sample-count-weighted average instead.

The right choice depends on the score being calibrated, the number of classes, and how much held-out data is available.

LLM calibration starts with choosing the event

Calibration is harder for LLMs than for ordinary classifiers because the model produces a sequence of tokens while the event we care about is usually semantic:

What is the probability that this answer is correct?

An autoregressive LLM generates one token at a time. The probability of a complete sequence is the product of its conditional next-token probabilities:

For the generated answer:

Ottawa is the capital of Canada.

the model assigns probabilities to Ottawa, is, the, capital, and every subsequent token. Answer calibration requires a different quantity:

Sequence likelihood does not equal proposition correctness. LLM calibration must therefore define the semantic event before assigning confidence to it.

LLM systems can calibrate several distinct targets:

Answer-, claim-, and action-level probabilities are usually the ones that matter in production.

Multiple choice is the easy LLM case

When an LLM must choose from a fixed set of answers, the choice probabilities directly match the event being scored. The task can therefore be evaluated like an ordinary multiclass classifier.

The following multiple-choice question defines a fixed output space:

What is the capital of Canada?

A. Toronto

B. Ottawa

C. Vancouver

D. Montreal

Restrict the output to the four choices and renormalize their scores so the probabilities sum to one. The LLM then gives:

We can apply ECE, Brier score, NLL, and reliability diagrams directly. If questions chosen with confidence near 0.80 are correct only 60% of the time, the model is overconfident by roughly 20 percentage points.

Kadavath et al. found that sufficiently large language models could show useful calibration on multiple-choice and true/false questions when the probability was elicited in the right format. The reported calibration depends on the model, task, and elicitation method.

Why sequence probability fails for free-form answers

Free-form sequence probability measures the likelihood of one exact token string, while answer correctness is a property of its meaning. Paraphrases can therefore express the same fact while receiving very different probabilities.

Two paraphrases show the problem:

Paris.

and:

The capital of France is Paris.

They express the same proposition, but their sequence probabilities can differ dramatically. Every extra token introduces another factor:

so longer strings tend to have smaller probabilities. Length-normalized log probability reduces that bias but does not solve the deeper problem: a model can be uncertain about phrasing while being certain about meaning.

For free-form generation, raw sequence probability and entropy over token strings are therefore weak substitutes for answer-level confidence. Early generative QA experiments likewise found model probabilities to be poorly calibrated (Jiang et al.).

Where answer-level confidence can come from

Answer-level confidence is not a built-in LLM probability. A system must estimate $P(\text{answer correct})$ from observable signals.

Constrained token or choice probability

For an answer constrained to yes or no, $P(\text{yes})$ is a natural score. The same applies to normalized multiple-choice probabilities. This is the cleanest case because the output space already matches the event being scored.

Verbalized confidence

Verbalized confidence asks the model to report a probability directly:

Answer: Ottawa

Confidence: 97%

The reported confidence is:

which can be evaluated with ECE and Brier score. Another elicitation method, $P(\mathrm{True})$, asks the model to judge a proposed answer and uses the normalized probability it assigns to True. Experiments in Language Models (Mostly) Know What They Know show that models can contain genuine self-evaluation signal.

“I am 95% confident” is itself generated text. It does not necessarily expose an internal posterior of 0.95; it is a learned linguistic behavior influenced by pretraining, prompting, instruction tuning, and preference optimization. In one study, reinforcement learning from human feedback (RLHF) increased verbalized overconfidence under the evaluated setups (Leng et al.).

The distinction is:

Semantic uncertainty

Semantic uncertainty groups sampled answers by meaning before measuring how spread out the answers are. Paraphrases count as the same outcome, so wording variation does not masquerade as uncertainty about the underlying answer.

Five samples answer “What caused the failure?” with:

1. “A memory leak in the worker.”
2. “The worker exhausted memory.”
3. “OOM in the worker process.”
4. “A networking timeout.”
5. “Worker memory exhaustion.”

The five strings contain only two meanings. Their sample frequencies define the cluster probabilities:

Rather than compute entropy over strings, cluster semantically equivalent answers and compute entropy over meanings:

In this equation, $k$ indexes semantic answer clusters and $q_k$ is the probability assigned to cluster $k$.

Farquhar et al. use semantic entropy to detect confabulations: generations that vary in meaning across samples. Nine paraphrases plus one alternative signal something very different from three answers naming Alice, three naming Bob, two naming Carol, and two naming Dave. The semantic entropy guide covers the method and its calibration in detail.

Semantic agreement is still not proof of correctness. Every sample can reproduce the same systematic misconception.

Evaluate uncertainty ranking and calibration separately

Any answer-level confidence score should be tested on two distinct properties:

• Uncertainty discrimination: can the score rank likely-correct answers above likely-wrong ones? Measure area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and risk-coverage, which tracks error as the system answers a larger fraction of requests.
• Calibration: does $c=0.8$ actually mean an 80% correctness rate? Measure reliability diagrams, ECE, Brier score, and NLL.

An LLM produces the following relationship:

The confidence is miscalibrated everywhere, yet it ranks answers well. A monotonic calibrator could learn $f(0.9)=0.75$, $f(0.8)=0.65$, and so on.

A useful system needs both discrimination and calibration.

ECE and Brier at the answer level

For each answer, assign a confidence $c_i$ and define its correctness label $y_i$. The general calibration metrics then apply without modification:

For example:

Q2 receives a large penalty because the model was 90% confident and wrong. The mean Brier score for the four answers is:

ConfidenceBench uses this evaluation pattern: it elicits verbalized probabilities and scores them with Brier score. Its reported results show that accuracy and calibration need not rank models in the same order.

At larger scale, an answer-level reliability table might look like:

The rising accuracy shows useful discrimination, while every upper bin remains overconfident. The score ranks answers by likely correctness but does not yet have valid probability semantics.

Risk-coverage connects confidence to abstention

Coverage is the fraction of requests the model answers, and risk is the error rate on those answered requests. Sorting requests by confidence and abstaining on the lowest scores traces a risk-coverage curve.

An example model answers every request at 82% accuracy. Abstaining on its least-confident requests produces these operating points:

The curve answers a production question more directly than ECE:

What fraction of requests can the LLM handle while staying below a specified error rate?

Risk-coverage directly supports an abstention policy. ECE remains complementary: risk-coverage tests ranking and selective prediction, while ECE tests probability semantics.

Long answers need claim-level calibration

For long-form generation, define correctness at the level of individual factual claims rather than assigning one probability to an entire paragraph. Decompose a response into claims $C_1,\ldots,C_j$ and estimate:

for each claim. This is claim-level calibration.

A response contains four factual statements:

Toronto is Canada’s capital, Canada has ten provinces, Ottawa is in Ontario, and the prime minister is X.

A single $P(\text{response correct})=0.8$ hides which parts are trustworthy. Claim-level probabilities can distinguish the supported statements from the incorrect or time-sensitive ones.

Claim-level evaluation also separates hallucination detection from calibration. An uncertainty model might rank a claim with score 8.7 above one with score 4.2, yet neither number is a probability. A calibrator can turn the first score into a statement such as:

The detector asks whether one claim looks more trustworthy than another. The calibrator asks whether claims reported at 0.93 are actually true about 93% of the time.

Reasoning models need trajectory-level signals

A reasoning model can produce hundreds of intermediate tokens before a final answer. Token-level probabilities do not directly estimate:

Useful estimator features can include:

• self-evaluation
• verifier or reward-model scores
• process-reward signals
• multiple reasoning samples
• agreement or self-consistency
• hidden-state probes
• execution results
• tool-based verification

Hidden-state stability is one useful internal signal. Khanmohammadi et al. perturb hidden representations, measure their stability, and train a lightweight correctness classifier; their reported experiments improve both ECE and Brier score relative to the evaluated baselines.

Agreement is not calibration

Agreement measures how often multiple reasoning paths reach the same answer. It can be a useful uncertainty score, but it becomes a probability only after its relationship to correctness is measured on labeled data.

One hundred reasoning samples produce:

The agreement score $c(A)=0.82$ is useful, but it does not imply:

All 82 trajectories could share the same misconception. The mapping has to be measured empirically. If answers with 82% agreement have historically been correct 94% of the time, then the calibrator should learn:

RAG needs several calibrated events

Retrieval-augmented generation (RAG) calibration requires separate probabilities for retrieval, support, correctness, and citation quality:

An event-specific confidence output might be:

1
2
3
4

document relevance         0.93
evidence sufficiency       0.71
answer correctness         0.76
citation faithfulness      0.98

The low evidence-sufficiency probability suggests searching again rather than simply generating harder. Event-specific calibration can therefore drive agent policy, not just annotate an answer after the fact.

Drug-discovery agents turn calibration into a chain of gates

Scientific-agent calibration requires a separate event at each decision gate. These agents search literature, form hypotheses, rank candidates, call chemistry or biology tools, propose experiments, and interpret results. A single “agent confidence” number collapses failure modes with different meanings and costs.

A one-step probability that a candidate becomes a safe, effective drug is not a useful calibration target. Attrition occurs at a sequence of gates: evidence, candidate selection, synthesis, assay, and replication. Each probability should name the gate and the population reaching it. Assay success, for example, is measured among synthesized candidates sent to that assay, while replication is measured among results selected for confirmation.

A calibrated confidence interface should expose measurable events such as:

The five probabilities are not interchangeable. A candidate can be well supported by literature and easy to synthesize yet fail the assay. A strong assay result can also fail to replicate or translate to a different model system. The outcome definition must include the assay, threshold, experimental population, and time horizon rather than simply whether the candidate works.

Examples from scientific agents

Recent scientific agents automate parts of the discovery loop, but their ranks, model scores, and agent agreement remain proxies until calibrated against named outcomes:

ChemCrow and Coscientist are broad chemistry systems. Virtual Lab, Robin, and Co-Scientist close parts of a drug-discovery loop with experimental validation.

Proxy scores are not probabilities

Drug-discovery agents combine signals on incompatible scales: an LLM judge’s rank, agreement among agents, language-model likelihood, structure-model confidence, molecular-simulation energy, and an experimental measurement. A high value can mean “preferred relative to this batch” without saying how often similarly scored candidates succeed.

In the Virtual Lab workflow, an uncertainty estimator could combine protein language-model scores from ESM, structure predictions from AlphaFold-Multimer, Rosetta energy scores, and agent agreement into a raw candidate score $s_i$. A held-out set of physically tested designs would then fit a mapping:

The reliability question is whether designs assigned $\hat c_i\approx0.7$ cross that assay’s binding threshold about 70% of the time. Even perfect calibration for this event would say nothing directly about toxicity, in vivo efficacy, or clinical success.

Robin ranked candidates with an LLM-judged tournament, and its data-analysis agent ran eight trajectories before forming a consensus. Those mechanisms can improve ranking and robustness, but neither tournament position nor eight-of-eight agreement is automatically a calibrated probability. The mapping must be learned from subsequent assay outcomes and checked again when the disease area, assay, model, prompts, or tools change.

Testing only the top-ranked molecules creates selection bias. The resulting reliability diagram estimates calibration among shortlisted candidates, not across everything the agent generated. Randomly testing a small exploration set helps reveal performance outside the shortlist. Propensity-aware evaluation can also reweight tested candidates by their selection probabilities to estimate the broader score-to-success relationship.

Calibration should control the next action

Event-level probabilities should control the next gate in the workflow:

• low evidence sufficiency triggers another literature search;
• uncertain synthesis feasibility triggers chemist review or route generation;
• high predicted assay value but high uncertainty can justify an informative experiment rather than automatic rejection;
• a positive result with low replication confidence triggers confirmation before the agent updates its hypothesis.

Scientific-agent policies extend selective prediction beyond answering or abstaining. An agent can search, simulate, ask for review, run a cheap assay, or stop. Calibration makes those choices auditable because every threshold refers to a frequency that can be measured.

Structured actions need event-specific confidence

Structured-action calibration assigns confidence to each operational event, not to the generated syntax. An LLM proposes this element placement:

1
2
3
4
5
6
7

{
  "element": "desk",
  "room": "office_31",
  "x": 4.23,
  "y": 2.71,
  "rotation": 90
}

The sequence probability of the entire JSON document has little operational meaning. The system should define the events it cares about:

The system can then expose where uncertainty comes from:

1
2
3
4
5

element selection       0.99
room selection          0.96
geometric validity      0.72
constraint validity     0.61
overall acceptance      0.65

Continuous outputs need coverage, not class confidence

Coordinates such as $x,y,\theta$ are not categorical events. They can instead be evaluated with prediction intervals or with threshold events. A prediction interval gives a range intended to contain the true value at a stated frequency. For example, a model might report:

A calibrated 90% prediction interval means:

over the deployment population. For spatial placement, the event could instead be task-specific:

or use an intersection-over-union (IoU) threshold:

Quantile regression estimates interval endpoints directly. Conformal methods use errors on a calibration set to construct intervals or prediction sets with a target coverage rate, assuming the calibration and deployment examples are exchangeable, or comparable draws from the same distribution. SConU is one recent LLM-specific example of selective conformal uncertainty that checks for samples that may violate this assumption.

Treat LLM confidence as a subsystem

A robust confidence architecture separates generation, uncertainty estimation, calibration, and policy. It should not equate confidence with average_token_probability:

1
2
3
4
5
6
7
8

flowchart LR
    A[LLM] --> B[Candidate answer or action]
    B --> C[Uncertainty estimator]
    C --> D[Calibrator]
    D --> E{Decision policy}
    E -->|high confidence| F[Execute]
    E -->|medium confidence| G[Suggest or verify]
    E -->|low confidence| H[Retrieve, regenerate, or defer]

The uncertainty estimator can combine token log-probabilities, entropy, self-consistency, hidden representations, constraint violations, geometric margins, verifier scores, and tool results. It emits a raw score. The calibrator maps that score to an event probability such as:

Only then should a decision policy apply thresholds. For example:

1
2
3

confidence > 0.98        auto-place
0.80-0.98                present suggestion
confidence < 0.80        retrieve, regenerate, or defer

The exact thresholds should come from error costs and risk-coverage analysis, not from round numbers chosen in advance.

A calibrated LLM can still be bad

Calibration is only one part of model quality. Three properties have to be evaluated separately:

Accuracy, task metrics, and execution success measure capability. AUROC, AUPRC, and risk-coverage measure uncertainty ranking. Reliability diagrams, ECE, Brier score, and NLL measure probability quality.

Capability, uncertainty discrimination, and calibration can fail independently. An LLM that always reports $P(\text{correct})=0.30$ and is correct 30% of the time is perfectly calibrated but still performs poorly.

Conversely, a model could be 95% accurate while reporting $P(\text{correct})=0.999$ for every answer. It is capable but miscalibrated.

Define the event before calibrating it

A classic classifier naturally exposes $P(y=k\mid x)$, so calibration starts by repairing that probability. An LLM requires an earlier decision: which random variable deserves a probability?

A production system should rarely calibrate next-token probability:

A production system should instead calibrate an operational event such as:

After the event is explicit, reliability diagrams, ECE, Brier score, post-hoc calibration, selective prediction, and conformal methods become applicable. The goal is to make confidence correspond to an event that a system can measure and act on.

References

• Kadavath et al., Language Models (Mostly) Know What They Know (2022).
• ffrench-Constant et al., ConfidenceBench: Evaluating Confidence Calibration in Large Language Models (2026).
• Farquhar et al., Detecting Hallucinations in Large Language Models Using Semantic Entropy (Nature, 2024).
• Leng et al., Taming Overconfidence in LLMs: Reward Calibration in RLHF (2024, revised 2025).
• Khanmohammadi et al., Calibrating LLM Confidence by Probing Perturbed Representation Stability (EMNLP 2025).
• Wang et al., SConU: Selective Conformal Uncertainty in Large Language Models (ACL 2025).
• Jiang et al., How Can We Know When Language Models Know? (TACL 2021).
• Bran et al., Augmenting Large Language Models with Chemistry Tools (Nature Machine Intelligence, 2024).
• Boiko et al., Autonomous Chemical Research with Large Language Models (Nature, 2023).
• Swanson et al., The Virtual Lab of AI Agents Designs New SARS-CoV-2 Nanobodies (Nature, 2025).
• Ghareeb et al., A Multi-Agent System for Automating Scientific Discovery (Nature, 2026).
• Gottweis et al., Accelerating Scientific Discovery with Co-Scientist (Nature, 2026).