Why an LLM confidence score is not a measurement

If you ask an LLM "how confident are you?", it will answer with something like 0.85. It looks precise. Amrut Joshi's post from October 20, 2026 makes a simple point about that number: there is no good reason to trust it. I have seen the pattern he describes in agent designs too, where a score from the model decides whether a reply is sent or sent to a person. Here is the argument, and what to do instead.

What a confidence score has to mean

Joshi uses a weather forecaster. "70% chance of rain tomorrow" is a statement you cannot check on one day, because it either rains or it does not. You can only check it over many days: take every day the forecaster said 70%, and see how often it rained. If it rained on about 70% of them, the number means something. This is called calibration.

His example record for one year looks like this:

Two details from the post matter. First, calibration alone is not enough: a forecaster who always says 30%, the usual rate for the season, is also calibrated, and tells you nothing. A useful score is calibrated and moves with the information it has. Second, none of this works unless someone writes down every forecast and what happened. No record, no way to know.

Why the LLM's number fails both tests

An LLM writes text one token at a time. When it answers "0.85", it is not reading a gauge; it is writing a plausible continuation of the conversation. Pretraining did score the model against outcomes, but the outcome was the next token, not whether its stated confidence matched how often it was correct. In Joshi's words, it learned what confidence scores look like in text.

So the number does not reflect the evidence, and it was never checked against results. You can see the symptoms yourself, and they are the ones he lists:

• Ask the same question again and the number changes, 0.85 one time and 0.7 the next.
• Ask "are you sure?" and the score usually moves, though nothing about the problem changed.
• If the model has already written "this is clearly the right answer", a high score is simply the natural next words.

A wrong number looks exactly like a right one. The only way to find out is to grade many of them against an answer key, which is the step most people skip.

Build the score from your own records

Joshi's alternative keeps the LLM doing the task and takes the confidence from a rule built on records. His example is a support agent that searches a help center and drafts replies, where uncertain replies should go to a person first:

1. Pick observable signals. Did search find a matching article? Has the agent seen this kind of question? Do two runs of the same question agree?
2. Start from the base rate. If a person checked 200 past replies and 150 were correct, every reply starts at 75%.
3. Record everything. Signals, the score the rule gave, and whether the reply turned out correct.
4. Update from the record. If replies with a matching article and two agreeing answers were right 95% of the time, the rule says 95% for the next one like it.
5. Keep checking. Every week or so, rebuild the forecaster's table. If the 90% replies were right only 70% of the time, the rule is overconfident and the record shows where.

The weekly check is a few lines. This sketch is illustrative, not from the source: it groups recorded replies by the score they were given and returns how often each group was right.

// records: [{ said: 0.9, correct: true }, ...]
function calibrationTable(records) {
  const groups = new Map();
  for (const r of records) {
    const key = Math.round(r.said * 10) / 10;
    const g = groups.get(key) ?? { n: 0, right: 0 };
    g.n += 1;
    if (r.correct) g.right += 1;
    groups.set(key, g);
  }
  return [...groups]
    .sort((a, b) => a[0] - b[0])
    .map(([said, g]) => ({ said, n: g.n, rate: g.right / g.n }));
}

If rate sits well below said in a group with enough replies, that group is overconfident.

Next step: if anything in your system routes on a model's self-reported score, start logging the score and the real outcome today. A month of records is enough to build the first table and see whether the number ever meant anything. Source: Amrut Joshi, Asking LLM for confidence.