Accuracy vs calibration

The core idea behind model calibration is simple:

When a model says “70% probability,” the event should actually happen about 70% of the time.

It answers one question:

Do predicted probabilities correspond to observed frequencies?

Accuracy counts how often the final decision is correct. Calibration asks whether the probabilities attached to those decisions have the right numerical meaning. A model can therefore be accurate while still assigning unjustified confidence.

That distinction becomes consequential when an LLM output drives a decision. Here are a few illustrative cases:

These percentages are hypothetical. Each becomes meaningful only after the event, reference standard, population, and operating conditions are specified. In high-stakes settings such as medicine, calibrated confidence should support triage and verification rather than replace clinical review.

Imagine a binary classifier predicts whether an object belongs in a room.

For 1,000 predictions where the model outputs approximately:

If the model is perfectly calibrated, the object should truly belong in the room in roughly 800 of those 1,000 cases.

A model can make every decision correctly while still being poorly calibrated.

Suppose two models make exactly the same classification decisions:

Both get 100% accuracy at threshold 0.5.

Model B assigns 0.98-0.99 probability to its chosen label on every example. If it makes even occasional mistakes, those implied confidence levels are unjustified.

Calibration asks:

rather than simply asking whether $\arg\max \hat P = Y$.

Reliability diagrams

The most intuitive calibration diagnostic is a reliability diagram.

A diagram can target any binary event. For a classifier or LLM confidence score, let the event be “this prediction is correct.” Then $\hat p_i$ is the confidence assigned to the chosen label and $y_i$ records whether that choice was correct.

Take predictions and place them into confidence bins:

• 0-10%
• 10-20%
• …
• 90-100%

For every bin $B_m$, calculate its mean predicted confidence:

and its observed accuracy:

Then plot mean predicted confidence on the x-axis against observed accuracy on the y-axis. A perfectly calibrated model lies on the diagonal:

Consider these five bins:

The model is increasingly overconfident at high probabilities. In the last bin, “95% sure” actually means “correct about 72% of the time.” That 23-point gap is a serious problem if a downstream system treats 0.95 as near-certainty.

Expected calibration error

A reliability diagram shows where confidence and accuracy diverge. Expected calibration error (ECE) compresses those gaps into one number.

For bins $B_1,\ldots,B_M$ containing $n$ predictions in total:

Each bin’s absolute calibration gap is weighted by the fraction of predictions that landed in that bin. Perfect calibration gives $\operatorname{ECE}=0$; larger values indicate a greater mismatch between stated confidence and observed accuracy.

Example

Suppose three bins contain:

Then:

So:

Informally, the model’s confidence is misaligned with empirical correctness by about 8 percentage points under this binning.

Limitations of ECE

ECE is useful, but it should not be treated as a definitive calibration score. Its value depends substantially on how it is calculated.

Number of bins

With 10 bins:

a model might have $\operatorname{ECE}=0.03$. With 50 bins, the same predictions might produce $\operatorname{ECE}=0.07$ because the finer partition exposes more local calibration errors.

Sample count

Small bins have noisy estimates. A bin containing 12 predictions might have mean confidence:

but only 7 successes:

That gap does not necessarily mean the model is dramatically miscalibrated. With so few observations, sampling noise is large.

Aggregation effects

Coarse grouping can hide local errors. Consider predictions with probabilities:

that fall into the same broad bin with a mean of $0.60$. The bin as a whole might appear calibrated even when the probability function is not.

ECE is useful for comparing model versions, but it is not a complete calibration report. Its value changes with the number of bins and the binning strategy, and large errors in sparse bins can disappear inside the weighted average. A useful ECE report should therefore include:

• a reliability diagram
• the number of predictions in each bin
• equal-mass bins, which place similar numbers of predictions in each bin
• uncertainty intervals

Brier score

The Brier score avoids binning entirely. For $N$ binary predictions:

where $y_i\in\{0,1\}$. It is simply mean squared error for probabilities.

Example

Suppose the model predicts:

and the outcomes are:

The squared errors are:

Therefore:

Lower is better, and $\operatorname{BS}=0$ means perfect probabilistic prediction.

Calibration and sharpness

A useful probabilistic model needs both calibration and sharpness. Calibration means that predictions reported at 80% are correct about 80% of the time. Sharpness means that the model moves probabilities away from the base rate, the overall event frequency, when the evidence supports doing so. Predictions such as $0.02,0.97,0.93$ are sharper than $0.49,0.53,0.51$.

ECE focuses on calibration. Brier score measures overall probabilistic quality, so it rewards both calibration and sharpness.

Consider two calibrated models. Model A always predicts:

Suppose the dataset really is 50% positive. Model A is calibrated, but it is not useful for distinguishing positives from negatives.

Model B predicts 0.9 for many positives and 0.1 for many negatives, and those probabilities match the observed frequencies. It is also calibrated, but much more useful.

ECE could say:

while Brier score strongly prefers Model B. Calibration alone is not enough.

The goal is:

not merely calibrated. Brier score captures both requirements.

Brier score decomposition

A single Brier score tells us whether the probabilities were good overall. Its decomposition explains why the score is good or bad.

First define the event. For a correctness forecast, let $y_i=1$ when the answer is correct and $y_i=0$ when it is wrong. For a drug-discovery forecast, the event might instead be whether a candidate crosses a prespecified assay threshold.

Now divide the predictions into forecast groups $B_m$. Each group has:

• $n_m$: the number of predictions in the group;
• $\bar p_m$: the probability assigned to that group;
• $\bar y_m$: the observed event rate in that group;
• $\bar y$: the event rate across the entire dataset.

The group event rate is:

It is an observed frequency, not another model prediction. If three of four answers in a group are correct, then $\bar y_m=3/4=0.75$. When the event is “the answer is correct,” the event rate is simply the accuracy within that group.

The Brier decomposition is:

or, written in terms of the forecast groups:

Each term answers a different question:

Reliability is a calibration penalty. Resolution is subtracted because separating low-event-rate groups from high-event-rate groups improves the forecast. Uncertainty is the baseline difficulty of the dataset and does not depend on the model.

This form is exact when every prediction in a group has the same forecast value. If an interval bin contains different probabilities and $\bar p_m$ replaces them all, the result is a binned approximation unless within-bin variation is included.

Worked example

Suppose eight predictions fall into two groups, and the event is a positive outcome:

The low group has a 25% event rate because the event occurred once in four cases. The high group has a 75% event rate because it occurred three times in four cases. Across the full dataset, it occurred four times in eight cases, so:

Both groups contain half of the samples, so each has weight $n_m/N=0.5$. The reliability penalty compares each forecast with what happened in its group:

The resolution reward compares each group’s event rate with the overall 50% event rate:

The uncertainty is determined only by the overall event rate:

Putting the three pieces together gives:

The interpretation is now visible. Predicting the 50% base rate for every case would produce the uncertainty baseline of $0.25$. Separating the cases into 25% and 75% event-rate groups improves the score by $0.0625$. Slightly missing those group rates with forecasts of 20% and 80% adds back a small calibration penalty of $0.0025$. The final Brier score is therefore $0.19$.

This is why Brier score captures more than calibration alone. It rewards both probabilities that match observed frequencies and predictions that separate cases with meaningfully different outcomes.

Brier score vs log loss

Another important metric is negative log-likelihood, also called log loss or binary cross-entropy:

Both Brier score and NLL are proper scoring rules: in expectation, the best strategy is to report the probability you actually believe. But they punish errors differently.

Suppose $y=0$ and the model predicts $p=0.999999$. The Brier penalty is:

while the log-loss penalty is:

Log loss therefore punishes a confidently wrong prediction much more severely. That is useful when unjustified certainty is especially costly.

Multiclass calibration gets trickier

For a multiclass model, top-label calibration reduces the full probability vector to the highest predicted probability:

It then checks whether predictions at that confidence are correct at the same rate. Suppose a model outputs:

The top-label confidence is 0.6, and the other two probabilities are discarded. That loss of information can hide differences between distributions.

For example:

and:

look identical to top-label ECE even though their uncertainty structures are different.

One alternative is classwise calibration:

for every class $k$. Multiclass Brier score naturally uses the whole probability vector:

where $y_{ik}$ is 1 for the true class and 0 for every other class, a representation called one-hot encoding. Some libraries also divide this quantity by the number of classes, so the convention should be recorded when results are compared.

Calibration can be conditional

An aggregate score can look calibrated even when it fails for important subgroups. Suppose the full dataset shows:

Now split the same predictions by context:

while:

The subgroup errors cancel in aggregate, but the 0.8 score has different meanings in bedrooms and kitchens. Operationally, the model is not calibrated where it matters.

Calibration should therefore be inspected across relevant slices such as:

• classes
• spatial regions
• input complexity
• rare vs frequent objects
• in-distribution vs shifted data
• document types
• model versions
• confidence ranges

A single global ECE can hide all of these failures.

Why neural networks tend to be overconfident

Cross-entropy training does not guarantee finite-sample or out-of-distribution calibration. In an overparameterized network, training can continue increasing the margin between the raw class scores, called logits, even after the classification decision is already correct.

Suppose the logits are:

Softmax converts those scores to probabilities by exponentiating and normalizing them. Here it produces a distribution close to:

But perhaps the evidence only justifies:

The predicted class is identical; only the claimed confidence is wrong. The probabilities are too extreme even though the class selection need not change.

Post-hoc calibration methods

Post-hoc calibration fits a mapping from a model’s existing scores to probabilities using a held-out calibration set. Three common methods form a rough spectrum of flexibility:

Temperature scaling

Given logits $z_1,\ldots,z_K$, ordinary softmax uses:

Temperature scaling instead uses:

A single positive scalar $T$ is fitted on a held-out calibration set, usually by minimizing NLL. Because it learns only one scalar for all logits, temperature scaling has low capacity and a correspondingly low risk of overfitting. When $T\gt1$, the distribution becomes softer.

For example, logits $[10,5,1]$ produce approximately:

With $T=2$, they become approximately:

Dividing every logit by the same positive temperature does not change their ordering, so the highest-probability class stays the same. Classification accuracy therefore stays the same. The reported probabilities can nevertheless become much better calibrated.

Platt scaling

Platt scaling learns a logistic transformation such as:

where $s$ is the model’s uncalibrated scalar score. The parameters $a$ and $b$ are fitted on a held-out calibration set, typically by minimizing binary NLL. The slope $a$ changes the scale, while the intercept $b$ corrects a systematic bias.

For example, suppose calibration fitting gives:

A raw score of $s=2$ would give $\sigma(2)\approx0.881$ if treated directly as a logit. Platt scaling instead gives:

The same mapping transforms several scores as follows:

Here, the fitted mapping says the original scores were both too large in scale and too optimistic around zero. Because $a\gt0$, the transformation preserves the ranking of examples even though their reported probabilities change.

Isotonic regression

Isotonic regression learns a flexible monotonic mapping $f(p)$ so that higher original scores still produce higher calibrated probabilities. It can fit more complex distortions, but it needs more calibration data and can overfit small samples. The standard pool-adjacent-violators algorithm enforces monotonicity by merging neighboring buckets whenever their observed rates are out of order.

Suppose four equally sized score buckets have these observed success rates:

The middle two buckets violate monotonicity: the higher score, 0.60, succeeds less often than 0.40. The algorithm therefore merges those buckets. Because they contain the same number of samples, their pooled rate is:

The fitted mapping becomes:

A new score on the pooled plateau, such as 0.50, maps to 0.45. The method learns this shape directly from the calibration data rather than assuming a sigmoid. With unequal bucket sizes, the pooled value would be the sample-count-weighted average instead.

The right choice depends on the score being calibrated, the number of classes, and how much held-out data is available.

LLM calibration starts with choosing the event

Calibration is harder for LLMs than for ordinary classifiers because the model produces a sequence of tokens while the event we care about is usually semantic:

What is the probability that this answer is correct?

An autoregressive LLM generates one token at a time. The probability of a complete sequence is the product of its conditional next-token probabilities:

When it generates:

Ottawa is the capital of Canada.

the model assigns probabilities to Ottawa, is, the, capital, and every subsequent token. The operational quantity is different:

There is no reason for sequence likelihood and proposition correctness to be equal. That distinction is the foundation of LLM calibration.

Several calibration targets are possible:

Answer-, claim-, and action-level probabilities are usually the ones that matter in production.

Multiple choice is the easy LLM case

When an LLM must choose from a fixed set of answers, the choice probabilities directly match the event being scored. The task can therefore be evaluated like an ordinary multiclass classifier.

Suppose we ask:

What is the capital of Canada?

A. Toronto

B. Ottawa

C. Vancouver

D. Montreal

Restrict the output to the four choices and renormalize their scores so the probabilities sum to one. The LLM then gives:

We can apply ECE, Brier score, NLL, and reliability diagrams directly. If questions chosen with confidence near 0.80 are correct only 60% of the time, the model is overconfident by roughly 20 percentage points.

Kadavath et al. found that sufficiently large language models could show useful calibration on multiple-choice and true/false questions when the probability was elicited in the right format. The result is encouraging, but it is specific to the model, task, and elicitation method.

Why sequence probability fails for free-form answers

Free-form sequence probability measures the likelihood of one exact token string, while answer correctness is a property of its meaning. Paraphrases can therefore express the same fact while receiving very different probabilities.

Consider two answers:

Paris.

and:

The capital of France is Paris.

They express the same proposition, but their sequence probabilities can differ dramatically. Every extra token introduces another factor:

so longer strings tend to have smaller probabilities. Length-normalized log probability reduces that bias but does not solve the deeper problem: a model can be uncertain about phrasing while being certain about meaning.

For free-form generation, raw sequence probability and entropy over token strings are therefore weak substitutes for answer-level confidence. Early generative QA experiments likewise found model probabilities to be poorly calibrated (Jiang et al.).

Where answer-level confidence can come from

There is no single built-in $P(\text{answer correct})$. A system has to construct an estimator from observable signals.

Constrained token or choice probability

For an answer constrained to yes or no, $P(\text{yes})$ is a natural score. The same applies to normalized multiple-choice probabilities. This is the cleanest case because the output space already matches the event being scored.

Verbalized confidence

Another option is to ask the model directly:

Answer: Ottawa

Confidence: 97%

This produces:

which can be evaluated with ECE and Brier score. Another elicitation method, $P(\mathrm{True})$, asks the model to judge a proposed answer and uses the normalized probability it assigns to True. Experiments in Language Models (Mostly) Know What They Know show that models can contain genuine self-evaluation signal.

But “I am 95% confident” is itself generated text. It does not necessarily expose an internal posterior of 0.95; it is a learned linguistic behavior influenced by pretraining, prompting, instruction tuning, and preference optimization. In one study, reinforcement learning from human feedback (RLHF) increased verbalized overconfidence under the evaluated setups (Leng et al.).

The distinction is:

Semantic uncertainty

Semantic uncertainty groups sampled answers by meaning before measuring how spread out the answers are. Paraphrases count as the same outcome, so wording variation does not masquerade as uncertainty about the underlying answer.

Suppose five samples answer “What caused the failure?” with:

1. “A memory leak in the worker.”
2. “The worker exhausted memory.”
3. “OOM in the worker process.”
4. “A networking timeout.”
5. “Worker memory exhaustion.”

There are five strings but only two meanings:

Rather than compute entropy over strings, cluster semantically equivalent answers and compute entropy over meanings:

Here, $s$ indexes the distinct semantic answer clusters rather than the original token strings.

Farquhar et al. use this idea to detect confabulations: generations that vary in meaning across samples. Nine paraphrases plus one alternative signal something very different from three answers naming Alice, three naming Bob, two naming Carol, and two naming Dave.

Semantic agreement is still not proof of correctness. Every sample can reproduce the same systematic misconception.

Evaluate uncertainty ranking and calibration separately

Any answer-level confidence score should be tested on two distinct properties:

• Uncertainty discrimination: can the score rank likely-correct answers above likely-wrong ones? Measure AUROC, AUPRC, and risk-coverage, which tracks error as the system answers a larger fraction of requests.
• Calibration: does $c=0.8$ actually mean an 80% correctness rate? Measure reliability diagrams, ECE, Brier score, and NLL.

Suppose an LLM produces this relationship:

It is miscalibrated everywhere, yet its ranking is excellent. A monotonic calibrator could learn $f(0.9)=0.75$, $f(0.8)=0.65$, and so on.

A useful system needs both discrimination and calibration.

ECE and Brier at the answer level

Once each answer has a confidence $c_i$ and a correctness label $y_i$, the general metrics apply without modification:

For example:

Q2 receives a large penalty because the model was 90% confident and wrong. The mean Brier score for the four answers is:

This is also the evaluation pattern used by ConfidenceBench, which elicits verbalized probabilities and scores them with Brier score. Its reported results reinforce that accuracy and calibration need not rank models in the same order.

At larger scale, an answer-level reliability table might look like:

The rising accuracy shows useful discrimination, while every upper bin remains overconfident. The score knows something, but it is not yet a probability.

Risk-coverage connects confidence to abstention

Coverage is the fraction of requests the model answers, and risk is the error rate on those answered requests. Sorting requests by confidence and abstaining on the lowest scores traces a risk-coverage curve.

Suppose the model answers every request at 82% accuracy. If it abstains on its least-confident requests, the operating points might be:

The curve answers a production question more directly than ECE:

What fraction of requests can the LLM handle while staying below a specified error rate?

For deployed systems, this can matter more than whether confidence values are perfectly calibrated. The two evaluations remain complementary: risk-coverage tests ranking and selective prediction, while ECE tests probability semantics.

Long answers need claim-level calibration

For long-form generation, define correctness at the level of individual factual claims rather than assigning one probability to an entire paragraph. Decompose a response into claims $C_1,\ldots,C_j$ and estimate:

for each claim. This is claim-level calibration.

Consider a response containing four factual statements:

Toronto is Canada’s capital, Canada has ten provinces, Ottawa is in Ontario, and the prime minister is X.

A single $P(\text{response correct})=0.8$ hides which parts are trustworthy. Claim-level probabilities can distinguish the supported statements from the incorrect or time-sensitive ones.

It also separates hallucination detection from calibration. An uncertainty model might rank a claim with score 8.7 above one with score 4.2, yet neither number is a probability. A calibrator can turn the first score into a statement such as:

The detector asks whether one claim looks more trustworthy than another. The calibrator asks whether claims reported at 0.93 are actually true about 93% of the time.

Reasoning models need trajectory-level signals

A reasoning model can produce hundreds of intermediate tokens before a final answer. Token-level probabilities do not directly estimate:

Useful estimator features can include:

• self-evaluation
• verifier or reward-model scores
• process-reward signals
• multiple reasoning samples
• agreement or self-consistency
• hidden-state probes
• execution results
• tool-based verification

Internal signals can help. For example, Khanmohammadi et al. perturb hidden representations, measure their stability, and train a lightweight correctness classifier; their reported experiments improve both ECE and Brier score relative to the evaluated baselines.

Agreement is not calibration

Agreement measures how often multiple reasoning paths reach the same answer. It can be a useful uncertainty score, but it becomes a probability only after its relationship to correctness is measured on labeled data.

Suppose 100 reasoning samples produce:

The agreement score $c(A)=0.82$ is useful, but it does not imply:

All 82 trajectories could share the same misconception. The mapping has to be measured empirically. If answers with 82% agreement have historically been correct 94% of the time, then the calibrator should learn:

RAG needs several calibrated events

A retrieval-augmented system has more than one place to be uncertain. Instead of one generic confidence score, it can estimate:

For example:

1
2
3
4

document relevance         0.93
evidence sufficiency       0.71
answer correctness         0.76
citation faithfulness      0.98

The low evidence-sufficiency probability suggests searching again rather than simply generating harder. Event-specific calibration can therefore drive agent policy, not just annotate an answer after the fact.

Drug-discovery agents turn calibration into a chain of gates

Scientific agents make the event-definition problem concrete. They do not make one prediction; they search literature, form a hypothesis, rank candidates, call chemistry or biology tools, propose an experiment, and interpret the result. A single “agent confidence” number collapses failure modes with very different meanings and costs.

The probability that a candidate becomes a safe, effective drug is not a sensible one-step target. Attrition occurs at a sequence of gates: evidence, candidate selection, synthesis, assay, and replication. Each probability should name the gate and the population reaching it. For example, assay success is measured among synthesized candidates sent to that assay, while replication is measured among results selected for confirmation.

A useful confidence interface would expose measurable events such as:

These are not interchangeable. A candidate can be well supported by literature and easy to synthesize yet fail the assay. A strong assay result can also fail to replicate or translate to a different model system. The outcome definition must therefore include the assay, threshold, experimental population, and time horizon, rather than simply whether the candidate works.

Examples from scientific agents

Recent systems show how much of this loop can already be automated. They also show why a rank, model score, or agent agreement is only a proxy until it is calibrated against a named outcome:

ChemCrow and Coscientist are broad chemistry systems. Virtual Lab, Robin, and Co-Scientist close parts of a drug-discovery loop with experimental validation.

Proxy scores are not probabilities

Drug-discovery agents combine signals on incompatible scales: an LLM judge’s rank, agreement among agents, language-model likelihood, structure-model confidence, molecular-simulation energy, and an experimental measurement. A high value can mean “preferred relative to this batch” without saying how often similarly scored candidates succeed.

For the Virtual Lab workflow, for example, an uncertainty estimator could combine signals from ESM, AlphaFold-Multimer, Rosetta, and agent agreement into a raw candidate score $s_i$. A held-out set of physically tested designs would then fit a mapping:

The reliability question is whether designs assigned $\hat c_i\approx0.7$ cross that assay’s binding threshold about 70% of the time. Even perfect calibration for this event would say nothing directly about toxicity, in vivo efficacy, or clinical success.

Robin provides another instructive case. Its candidates were ranked by an LLM-judged tournament, and its data-analysis agent ran eight trajectories before forming a consensus. Those mechanisms can improve ranking and robustness, but neither the tournament position nor an eight-of-eight agreement is automatically a calibrated probability. The mapping has to be learned from subsequent assay outcomes and checked again when the disease area, assay, model, prompts, or tools change.

There is also a selection-bias trap. If only the top-ranked molecules are tested, the resulting reliability diagram estimates calibration among shortlisted candidates, not across everything the agent generated. Randomly testing a small exploration set helps reveal performance outside the shortlist. Propensity-aware evaluation can also reweight tested candidates by their selection probabilities to estimate the broader score-to-success relationship.

Calibration should control the next action

Once each probability names an observable event, it can drive a gate in the workflow:

• low evidence sufficiency triggers another literature search;
• uncertain synthesis feasibility triggers chemist review or route generation;
• high predicted assay value but high uncertainty can justify an informative experiment rather than automatic rejection;
• a positive result with low replication confidence triggers confirmation before the agent updates its hypothesis.

This is the same selective-prediction idea as abstention, but with a richer action space. The agent can search, simulate, ask for review, run a cheap assay, or stop. Calibration makes those choices auditable because every threshold refers to a frequency that can be measured.

Structured actions need event-specific confidence

Event-specific calibration also applies outside scientific discovery. Suppose an LLM proposes an element placement:

1
2
3
4
5
6
7

{
  "element": "desk",
  "room": "office_31",
  "x": 4.23,
  "y": 2.71,
  "rotation": 90
}

The sequence probability of the entire JSON document has little operational meaning. The system should define the events it cares about:

Then it can expose where uncertainty comes from:

1
2
3
4
5

element selection       0.99
room selection          0.96
geometric validity      0.72
constraint validity     0.61
overall acceptance      0.65

Continuous outputs need coverage, not class confidence

Coordinates such as $x,y,\theta$ are not categorical events. They can instead be evaluated with prediction intervals or with threshold events. A prediction interval gives a range intended to contain the true value at a stated frequency. For example, a model might report:

A calibrated 90% prediction interval means:

over the deployment population. For spatial placement, the event could instead be task-specific:

or:

Quantile regression estimates interval endpoints directly. Conformal methods use errors on a calibration set to construct intervals or prediction sets with a target coverage rate, assuming the calibration and deployment examples are exchangeable, or comparable draws from the same distribution. SConU is one recent LLM-specific example of selective conformal uncertainty that checks for samples that may violate this assumption.

Treat LLM confidence as a subsystem

A robust architecture should not equate confidence with average_token_probability. It separates generation, uncertainty estimation, calibration, and policy:

1
2
3
4
5
6
7
8

flowchart LR
    A[LLM] --> B[Candidate answer or action]
    B --> C[Uncertainty estimator]
    C --> D[Calibrator]
    D --> E{Decision policy}
    E -->|high confidence| F[Execute]
    E -->|medium confidence| G[Suggest or verify]
    E -->|low confidence| H[Retrieve, regenerate, or defer]

The uncertainty estimator can combine token log-probabilities, entropy, self-consistency, hidden representations, constraint violations, geometric margins, verifier scores, and tool results. It emits a raw score. The calibrator maps that score to an event probability such as:

Only then should a decision policy apply thresholds. For example:

1
2
3

confidence > 0.98        auto-place
0.80-0.98                present suggestion
confidence < 0.80        retrieve, regenerate, or defer

The exact thresholds should come from error costs and risk-coverage analysis, not from round numbers chosen in advance.

A calibrated LLM can still be bad

Calibration is only one part of model quality. Three properties have to be evaluated separately:

Accuracy, task metrics, and execution success measure capability. AUROC, AUPRC, and risk-coverage measure uncertainty ranking. Reliability diagrams, ECE, Brier score, and NLL measure probability quality.

These properties can fail independently. Suppose an LLM knows almost nothing, always reports $P(\text{correct})=0.30$, and is correct 30% of the time. It is perfectly calibrated and still a terrible model.

Conversely, a model could be 95% accurate while reporting $P(\text{correct})=0.999$ for every answer. It is capable but miscalibrated.

The deepest conceptual shift

For a classic classifier, the model naturally exposes $P(y=k\mid x)$, so calibration starts by repairing that probability. For an LLM, the first problem is deciding which random variable deserves a probability at all.

In a production system, I would rarely choose:

and would usually choose something closer to:

Once that event is explicit, reliability diagrams, ECE, Brier score, post-hoc calibration, selective prediction, and conformal methods become useful again. The operational goal is not merely to make an LLM sound less certain. It is to make confidence correspond to an event that a system can measure and act on.

References

• Kadavath et al., Language Models (Mostly) Know What They Know (2022).
• ffrench-Constant et al., ConfidenceBench: Evaluating Confidence Calibration in Large Language Models (2026).
• Farquhar et al., Detecting Hallucinations in Large Language Models Using Semantic Entropy (Nature, 2024).
• Leng et al., Taming Overconfidence in LLMs: Reward Calibration in RLHF (2024, revised 2025).
• Khanmohammadi et al., Calibrating LLM Confidence by Probing Perturbed Representation Stability (EMNLP 2025).
• Wang et al., SConU: Selective Conformal Uncertainty in Large Language Models (ACL 2025).
• Jiang et al., How Can We Know When Language Models Know? (TACL 2021).
• Bran et al., Augmenting Large Language Models with Chemistry Tools (Nature Machine Intelligence, 2024).
• Boiko et al., Autonomous Chemical Research with Large Language Models (Nature, 2023).
• Swanson et al., The Virtual Lab of AI Agents Designs New SARS-CoV-2 Nanobodies (Nature, 2025).
• Ghareeb et al., A Multi-Agent System for Automating Scientific Discovery (Nature, 2026).
• Gottweis et al., Accelerating Scientific Discovery with Co-Scientist (Nature, 2026).