Quy-Anh Dang

Quy-Anh Dang

Researcher, Knovel Engineering


← All work

Conformal Prediction, Step by Step — Part 6: Conformal Prediction for LLMs, and a Checklist

November 7, 2026 · 9 min read · Conformal Prediction, Uncertainty Quantification, Large Language Models, Machine Learning

Large language models answer everything in the same fluent, confident tone, whether they know the answer or are making it up. Ask one how sure it is, and it will happily say "95%." The question this post asks is practical: can we decide, with a guarantee, when an LLM's answer should be trusted?

Conformal prediction fits this job well. It never looks inside the model, so it doesn't matter that an LLM is a closed API with billions of parameters. It only needs a score and a calibration set. This time the recipe changes the target: instead of covering the right answer, we control how often the model is allowed to be wrong.

StepParts 1–5This post
1. Scorehow wrong the model is on a pointthe model's confidence, and whether its answer was right
2. Quantilepick so that 90% of scores are below itpick a threshold so that the expected error stays within a budget
3. Keepanswers with score answer if confident enough, otherwise abstain

A note on the data. To keep everything runnable without an API key, this post uses a simulated question-answering benchmark. It has the property that matters: like real LLMs asked for a verbal confidence, the simulated model is overconfident. The methods work the same way with a real model: replace the simulated confidences and correctness labels with your own.

1. Stated confidence is not a probability#

For each question, the model gives an answer and a stated confidence between 0 and 1. On calibration questions we also know whether the answer was right. Plotting how often the model is right against how sure it claims to be gives this:

Stated confidence against actual accuracy: the model's curve lies well below the diagonal, so it is right much less often than it claims

An honest model would sit on the dashed diagonal: right 70% of the time when it says 70%. This one sits well below it. When it says 85%, it's right about half the time. Overall it answers 65% of questions correctly, but it claims at least 90% confidence on 61% of them, and on those it's right only 82% of the time.

So the obvious rule, "answer only when the model says it's at least 95% sure," rests on numbers that don't mean what they say. We need a threshold chosen from what actually happened on calibration questions.

2. Answer or abstain: conformal risk control#

Here is the goal in plain words: of all the questions we're asked, at most 5% should get a wrong answer. For the rest, the system either answers correctly or says "I'm not sure", and a person takes over.

Let be a confidence threshold: answer if the stated confidence is at least , otherwise abstain. For each calibration question, define its loss at threshold :

Raising can only turn losses from 1 to 0 (we answer less), so the loss never increases as goes up. That is the one property we need. Conformal risk control (Angelopoulos, Bates, Fisch, Lei and Schuster, 2024) picks the smallest threshold whose corrected calibration risk is within budget:

The guarantee is that, for a new question,

In words: the expected share of questions that get a wrong answer is at most .

The "+1" in the numerator is the same idea as the in Part 1. It accounts for the unseen new question, assuming it suffers the worst possible loss (1). In fact, split conformal prediction is a special case: with the loss "1 if the true answer is outside the set", conformal risk control gives back exactly the method from Part 1. The difference is that the loss can now be anything bounded and monotone: wrong answers, missed tumours in a segmentation, false negatives in a search.

ELI5: a hospital tells a new trainee, "You may make the call yourself only when you're very sure. Otherwise, ask a senior doctor." How sure is "very sure"? The hospital looks at the trainee's record on past cases and picks the bar so that, at most, 5 cases in 100 end with a wrong call.

Here is what the choice looks like on 500 calibration questions:

As the confidence threshold rises, the share of questions answered falls from 100%, and the share answered wrongly falls from about 35% to 0; the chosen threshold is where wrong answers drop below the 5% budget

With no threshold, the model answers everything and gets about 35% wrong. Raising the bar removes wrong answers, but also right ones. The chosen threshold, 0.96 on this calibration set, keeps the wrong answers within 5% and still answers 45% of questions.

Repeated 200 times, each with a fresh calibration set of 500 questions and 2,000 new test questions:

RuleQuestions answeredQuestions answered wrongly
Naive: answer if stated confidence 48.6%6.7% (over budget)
Conformal risk control, 41.7%4.9%

The naive rule looks reasonable but breaks the budget, because the model's "95%" is really closer to 80%. Conformal risk control answers a bit less and stays within budget.

import numpy as np

def crc_threshold(conf, right, alpha=0.05):
    """Smallest threshold with (n * risk + 1) / (n + 1) <= alpha; the loss is bounded by 1."""
    n = len(conf)
    for lam in np.linspace(0, 1, 1001):                  # risk can only fall as lam rises
        risk = np.mean((conf >= lam) & ~right)           # answered and wrong
        if (n * risk + 1) / (n + 1) <= alpha:
            return lam
    return np.inf                                        # budget impossible: never answer

# conf: stated confidences, right: True/False, on 500 calibration questions
lam = crc_threshold(conf_cal, right_cal)
answer = conf_test >= lam                                # otherwise say "I'm not sure"

Two things to know about this guarantee.

  1. It is on average over calibration sets, like every guarantee in this series (Part 1, section 4). In our 200 runs, the average was 4.9%, but individual runs went up to 6.9% (95th percentile). If you need "at most 5% with 95% confidence for this calibration set," use a stricter method called Learn then Test (Angelopoulos et al., 2021), or calibrate on more questions.
  2. It controls wrong answers among all questions, not among answered ones. Here 4.9% of all questions are answered wrongly, but the system answers only 41.7%, so about 12% of its answers are wrong. If your promise is about answered questions ("when it answers, it's right 95% of the time"), that is a different target. Learn then Test can handle it too.

3. Sets of answers from repeated sampling#

Stated confidence is one signal. A stronger one, available even from a closed API, is consistency: ask the same question several times with sampling turned on and see how often the answers agree. If 9 out of 10 samples say "Canberra," the model probably knows. If the answers are scattered, it's guessing.

This gives a natural conformal score. Sample answers per question and group the ones that mean the same thing. For a calibration question with known correct answer :

If the right answer appeared in 9 of 10 samples, the score is 0.1. If it never appeared, the score is 1. Steps 2 and 3 are standard split conformal: compute , then for a new question keep every distinct answer that appeared in at least a share of the samples.

With on our simulated benchmark, the threshold usually works out to "keep every answer that appeared at least twice in 10 samples." The sets contain the right answer 92.6% of the time (a bit above 90%, because counts out of 10 come in coarse steps). Their sizes:

Distribution of answer-set sizes: 45% of questions get one answer, 33% get two, 19% get three, 3% four or more

A single answer means the model is consistent, and the answer can be shown directly. Two or three answers are a useful signal in themselves: show the alternatives, ask a follow-up question, or pass it to a person.

One limit is specific to LLMs. If the right answer never shows up in the samples (2.9% of questions here), no set built from the samples can contain it. Coverage above is impossible with 10 samples, so must be larger than the "never sampled" rate. More samples, or allowing the system to abstain, helps. Research methods built on these ideas include conformal language modelling (Quach et al., 2024), which decides when to stop sampling, and conformal factuality (Mohri and Hashimoto, 2024), which removes individual claims from a long answer until what remains is reliable.

Where the correctness labels come from matters. Everything above needs to know whether each calibration answer was right. For short factual answers, exact matching works. For long answers, the labels come from people or from an LLM judge. Then the guarantee is only as good as those labels, so check how much your judges agree before you trust them (see agreement metrics). And the calibration questions must look like the questions users will really ask. That is Part 5's lesson, and it's the easiest one to forget with LLMs.

4. A practical checklist#

Everything from the series, in the order you'll need it:

  1. Hold out calibration data the model never trained on. A few hundred points works. About 1,000 keeps each deployment close to the target (Part 1). With only tens of points, use CV+ or jackknife+ instead of splitting (Part 4).
  2. Choose a coverage level that means something. For classifiers, pick it above the model's accuracy, or the method will return empty sets (Part 3).
  3. Choose the score. It decides how useful the sets are, never whether the guarantee holds.
    • Regression: CQR is a strong default; scaled residuals are simpler (Part 2).
    • Classification: LAC for the smallest sets, APS/RAPS if hard inputs matter (Part 3).
    • Abstention and other error budgets: conformal risk control (this post).
  4. Check coverage, then check it by group. Overall 90% can hide 77% on the group you care most about. If a group must get its own guarantee, calibrate it separately (Mondrian; Parts 2 and 3).
  5. Report the cost, not just the coverage. Interval width or set size is what users actually feel.
  6. Ask whether tomorrow looks like the calibration data. If the inputs shift, reweight. If data arrives over time, use ACI and recalibrate regularly (Part 5).
  7. Remember what is promised: coverage on average over new points and calibration sets, not for every individual input.

5. In code: libraries#

You rarely need to write the methods by hand. MAPIE is the most complete Python library and works with any scikit-learn-style model. Here it reproduces Part 1's house-price intervals and Part 3's digit sets exactly (same models and splits as in those posts):

from mapie.regression import SplitConformalRegressor
from mapie.classification import SplitConformalClassifier

# Part 1: model already trained on X_tr; calibrate on X_cal
reg = SplitConformalRegressor(model, confidence_level=0.9, prefit=True)
reg.conformalize(X_cal, y_cal)
pred, bounds = reg.predict_interval(X_te)                # bounds[:, 0, 0] = lower, bounds[:, 1, 0] = upper
# coverage 0.895, half-width 0.718: same as Part 1

# Part 3: LAC prediction sets for the digit classifier
cls = SplitConformalClassifier(clf, confidence_level=0.95, conformity_score="lac", prefit=True)
cls.conformalize(X_cal_digits, y_cal_digits)
_, sets = cls.predict_set(X_test_digits)                 # sets[:, j, 0] = is digit j in the set?
# coverage 0.948, average set size 1.06: same as Part 3

MAPIE also has CQR (ConformalizedQuantileRegressor), jackknife+ and CV+ (CrossConformalRegressor), APS and RAPS (conformity_score="aps" or "raps"), and a risk_control module for error budgets like the one in section 2. crepes is a lighter alternative with good support for Mondrian (group-wise) calibration.

6. Where to go next#

  • A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification by Angelopoulos and Bates: the best next read, with many worked examples.
  • Algorithmic Learning in a Random World by Vovk, Gammerman and Shafer: the original book, with full conformal prediction and its theory.
  • Theoretical Foundations of Conformal Prediction by Angelopoulos, Barber and Bates: a rigorous treatment of the guarantees used throughout this series. The papers cited in each part give the details behind each method.

Recap#

  • LLMs' stated confidence is usually overconfident, so thresholds on it must be calibrated on real outcomes.
  • Conformal risk control picks a threshold so that the expected share of wrong answers stays within a budget , for any bounded loss that falls as the threshold rises. Split conformal is the special case where the loss is "missed the true answer."
  • With an LLM, sampling several answers gives a strong score: keep the answers that appear often enough. The set size tells you when to involve a person.
  • Guarantees depend on good correctness labels and on calibration questions that look like real ones.
  • Across the whole series, the recipe never changed: score, quantile, keep. Each part changed one step, and the guarantee came along for free.

That's the end of Conformal Prediction, Step by Step. The most useful habit to take away: whenever a model gives you a number, ask what range it should come with, and check how often that range is right on data the model never saw.


© 2026 Quy-Anh Dang. All rights reserved.