Calibrating Verbalized Confidence with Self-Generated Distractors
Extracted source text · arXiv 2509.25532
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{xspace}
\usepackage[capitalise]{cleveref}
\crefformat{equation}{Eq.~#2#1#3}
\usepackage{svg}
\usepackage{tikz}
\usepackage[utf8]{inputenc}
\DeclareUnicodeCharacter{03BC}{$\mu$}
\usepackage{wrapfig}
\usepackage[skip=2pt]{caption}
\usepackage[dvipsnames,table]{xcolor}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage[most]{tcolorbox}
\usepackage{verbatim}
\usepackage{listings}
\usepackage[frozencache,cachedir=.]{minted}
\usepackage{fancyvrb}
\fvset{fontfamily=\rmdefault}
\NewDocumentCommand{\VarField}{m}{\texttt{\{#1\}}}
\DefineVerbatimEnvironment{PromptVerbatim}
{Verbatim}
{commandchars=\\\{\}, breaklines=true}
\lstnewenvironment{myverbatim}[1][]{%
\lstset{
basicstyle=\ttfamily,
frame=tb,
#1
}%
}{}
\BeforeBeginEnvironment{myverbatim}{%
\refstepcounter{myverb}%
}
\newenvironment{customlist}{
\begin{list}{\textbullet}{
\setlength{\leftmargin}{1em}
\setlength{\itemindent}{0em}
\setlength{\itemsep}{-0.1em}
\setlength{\topsep}{-0.3em} % space above/below the list
\setlength{\partopsep}{0em} % extra space if starting a new paragraph
}
}{
\end{list}
}
\setlength{\abovedisplayskip}{1pt}
\setlength{\belowdisplayskip}{1pt}
\title{Calibrating Verbalized Confidence with Self-Generated Distractors}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Victor Wang \qquad \quad Elias Stengel-Eskin \\[2mm] The University of Texas at Austin
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\newcommand{\method}{\textsc{DiNCo}\xspace}
\newcommand{\methodlong}{\underline{Di}stractor-\underline{N}ormalized \underline{Co}herence\xspace}
\newcommand{\myparagraph}[1]{\noindent\textbf{#1\hspace{0.5em}}}
% \newcommand{\nocomment}{}
\definecolor{forestgreen}{RGB}{34, 139, 34}
\definecolor{denim}{rgb}{0.08, 0.38, 0.74}
\hypersetup{
colorlinks = true, %Colours links instead of ugly boxes
urlcolor = denim, %Colour for external hyperlinks
linkcolor = denim, %Colour of internal links
citecolor = denim,%Colour of citations, could be ``red''
}
\newcommand{\factscore}{{FActScore}\xspace}
\newcommand{\ptrue}{${\rm P(True)}$\xspace}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
Calibrated confidence estimates are necessary for large language model (LLM) outputs to be trusted by human users.
While LLMs can express their confidence in human-interpretable ways, verbalized LLM-generated confidence scores have empirically been found to be miscalibrated, reporting high confidence on instances with low accuracy and thereby harming trust and safety.
We hypothesize that this overconfidence often stems from a given LLM's heightened \textit{suggestibility} when faced with claims that it encodes little information about; we empirically validate this hypothesis, finding more suggestibility on lower-accuracy claims.
Building on this finding, we introduce \methodlong (\method), which estimates and accounts for an LLM's suggestibility bias by having the model verbalize its confidence independently across several self-generated distractors (i.e. alternative claims), and normalizes by the total verbalized confidence.
To further improve calibration, we leverage generator-validator disagreement, augmenting normalized validator confidence with a consistency-based estimate of generator confidence.
Here, we frame the popular approach of self-consistency as leveraging coherence across sampled generations, and normalized verbalized confidence as leveraging coherence across validations on incompatible claims, allowing us to integrate these complementary dimensions of coherence into \method.
Moreover, our analysis shows that \method provides less saturated -- and therefore more usable -- confidence estimates, and that further sampling alone cannot close the gap between \method and baselines, with \method at 10 inference calls outperforming self-consistency at 100.\footnote{Code: \url{https://github.com/dubai03nsr/dinco}}
\end{abstract}
\section{Introduction}
\label{sec:intro}
LLMs encode a vast amount of knowledge in their parameters, demonstrating superhuman performance on knowledge-intensive benchmarks \citep{comanici2025gemini, openai2023gpt4}.
Users often rely on information obtained from these models to make important decisions, but the information is not always accurate.
Thus, we seek to qualify LLM responses with confidence estimates that are \textit{calibrated}, i.e. match the probability of correctness.
Users and agentic frameworks often use LLMs in a zero-shot manner without task-specific tuning \citep{manakul2023selfcheckgptzeroresourceblackboxhallucination, geng2024surveyconfidenceestimationcalibration, feng2024donthallucinateabstainidentifying, shorinwa2025surveyuncertaintyquantificationlarge}, motivating the development of confidence estimation methods that work in off-the-shelf settings -- both gray-box settings with logit access, and black-box settings with only textual input and output.
\begin{figure}[t]
\centering
\begin{subfigure}{0.33\textwidth}
% \includesvg[width=\linewidth]{figs/ece_vc.svg}
\includegraphics[width=\linewidth]{figs/ece_vc.pdf}
\end{subfigure}\hfill
\begin{subfigure}{0.33\textwidth}
% \includesvg[width=\linewidth]{figs/ece_nvc.svg}
\includegraphics[width=\linewidth]{figs/ece_nvc.pdf}
\end{subfigure}\hfill
\begin{subfigure}{0.33\textwidth}
\includegraphics[width=\linewidth]{figs/roc_fit.pdf}
\end{subfigure}
\caption{Calibration metrics (Expected Calibration Error $\downarrow$, Brier score $\downarrow$, area under the ROC curve $\uparrow$; see Appendix~\ref{app:evaluation-metrics}) with Qwen3-8B on TriviaQA using \ptrue as verbalized confidence.
\textbf{(Left)} Verbalized confidence is saturated at high confidence.
For each bar, we label the number of instances whose confidence falls in the interval and we darken larger bins.
\textbf{(Center)} \method normalizes by the total confidence over candidate answers, relieving saturation and improving calibration.
\textbf{(Right)}
Since verbalized confidence is saturated at high confidence, it is unable to achieve an acceptable true positive rate (TPR) without incurring a significant false positive rate (FPR) of 0.24.
In other words, no rejection threshold can be chosen to reject a high proportion of false claims.
Meanwhile, \method enjoys better granularity, ranking positives above negatives even among instances with a verbalized confidence of 1.
}
\label{fig:vc-versus-dinco-calibration}
\end{figure}
In these settings, verbalized confidence is a simple and commonly-used approach that prompts the model to report its confidence in an answer \citep{lin2022teachingmodelsexpressuncertainty, xiong2024llmsexpressuncertaintyempirical, wei2024measuringshortformfactualitylarge}.
For brevity, we use \textit{verbalized confidence} as a blanket term for (1) asking the model to decode a numerical confidence like \textit{``80\%''} \citep{tian2023justaskcalibrationstrategies} and (2) asking the model whether an answer is correct and taking \ptrue \citep{kadavath2022languagemodelsmostlyknow}.
Verbalized confidence is appealing for several reasons, including that it resembles one way humans express confidence, making it easy to interpret and integrate into decision-theoretic frameworks \citep{sun2025largelanguagemodelsoverconfident, Steyvers_2025}.
However, verbalized confidence has several drawbacks.
First, it empirically tends to exhibit overconfidence \citep{tian2023justaskcalibrationstrategies, xiong2024llmsexpressuncertaintyempirical, wei2024measuringshortformfactualitylarge, xu2025languagemodelsmirrorhuman}; \cref{fig:vc-versus-dinco-calibration} (left) shows that verbalized confidence scores generally outstrip average accuracy within a confidence bin.
We highlight a second underexplored factor that makes verbalized confidence suboptimal: \textbf{confidence saturation}, wherein the model's reported scores tend to fall into a few bins, making them uninformative.
While this might still lead to an acceptable calibration error, it results in ``jumpy'' curves, as in \cref{fig:vc-versus-dinco-calibration} (right),
where no confidence threshold that accepts at least one claim can avoid accepting a substantial proportion of false claims.
To address these shortcomings, we introduce \method{}, which leverages incoherence in verbalized confidence across related claims to detect overconfidence.
\method is motivated by the intuition that incoherent confidence scores, e.g., a high verbalized confidence in an answer to a question when other distinct answers also have high verbalized confidence, should not be taken at face value. In other words, we should discount high confidence if it does not follow rational coherence norms \citep{hofweber2024language}.
To explain and correct for this kind of incoherence, we first define the notion of suggestibility.
Some studies indicate that when LLMs are epistemically uncertain, they tend to rely on their context to resolve the uncertainty \citep{yadkori2024believebelievellm, ahdritz2024distinguishingknowableunknowablelanguage}, i.e., the confidence on a claim increases \emph{because} it is in the context.
We refer to this phenomenon as \emph{suggestibility}
and hypothesize that it contributes to the model's assignment of high confidence to claims it can neither support nor refute.
To account for this bias, we propose a method for calibrating verbalized confidence that normalizes by the total confidence over self-generated distractors (i.e. alternative claims).
We generate \textit{minimal pair} distractors using beam search when available, or by directly prompting the model for distractors in the black-box setting.
Crucially, we use an off-the-shelf NLI model to downweight distractors that are similar to other distractors or that do not contradict the main claim.
The approach above for normalizing verbalized confidence with distractors aims to leverage coherence within claim validation, but overlooks another relevant facet of coherence in LLMs.
In particular, coherence among sampled generations is correlated with correctness, an observation leveraged by the popular approach of self-consistency \citep{xiong2024llmsexpressuncertaintyempirical}.
Thus, inspired by prior findings on generator-validator disagreement \citep{li2023benchmarkingimprovinggeneratorvalidatorconsistency}, we integrate these complementary dimensions of coherence into \method.
Specifically, we use distractor generation and NLI reweighting to estimate and enforce coherence across validations of related claims (e.g. not accepting contradictory claims),
while using self-consistency to quantify coherence across sampled generations, upweighting more commonly generated claims.
We test our method on open-source and closed-source models, applied to short-form (TriviaQA and SimpleQA)~\citep{joshi2017triviaqalargescaledistantly, wei2024measuringshortformfactualitylarge} and long-form \citep[\factscore;][]{min2023factscorefinegrainedatomicevaluation} generation domains.
\method improves ECE over the next best method by an average of 0.099, 0.092, and 0.055, respectively.
\method effectively extends to long-form biography generation, where it improves Pearson and Spearman correlation with passage-level \factscore over the best baseline by an average of 0.072 and 0.074, respectively.
Further analysis shows that \method relieves confidence saturation, and that simply scaling up self-consistency (the strongest baseline overall) does not suffice to match the calibration of \method.
\section{Distractor-Normalized Coherence (\method)} \label{sec:bigmethod}
We begin with a motivating hypothesis, supported with preliminary evidence. Then we present the details of our method, illustrated in \cref{fig:method}.
\subsection{Background and Motivation}
\label{sec:background-and-motivation}
Let $\mathcal{C}$ be the set of claims with a binary truth value. We denote the truth value of a claim $c \in \mathcal{C}$ as $v(c) \in \{0, 1\}$.
A confidence estimation method is a function $f: \mathcal{C} \to [0, 1]$, which is \textit{calibrated} if it correctly predicts the probability of truth.
Verbalized confidence is an approach that prompts an LLM to output its confidence $f^{\rm VC}(c)$ in a claim $c$.\footnote{We talk about what the model knows, believes, has confidence in, etc. as short-hand notation for the latent ability to produce language similar to that which a human would to demonstrate such knowledge, etc. \citep{piantadosi2022meaningreferencelargelanguage, west2023generativeaiparadoxwhat, hofweber2024language}.}
For a topic that the model knows little about,
it may be willing to adopt the information presented in its context as its prior \citep{yadkori2024believebelievellm, ahdritz2024distinguishingknowableunknowablelanguage}, a phenomenon we refer to as \textit{suggestibility}.
In other words, the very act of presenting a claim for the model to report its confidence on can bias the reported confidence.
For example, if the model does not know who Kang Ji-hwan is, it may assign 60\% confidence to both the claim \textit{``Kang Ji-hwan was born in 1980.''} and the claim \textit{``Kang Ji-hwan was born in 1990.''}, even though this seemingly violates coherence norms (since the claims are mutually exclusive).
Nonetheless, this behavior may not be strictly irrational, as each confidence estimate is conditioned on different information, namely the fact that the respective claim was verbalized in the user prompt.
In other words, the very fact that the claim has been provided in the input might lend credence to the claim.
We further discuss the connection between this notion, LLM sycophancy, and human suggestibility in Appendix~\ref{app:related-work-related-behavior}.
We seek to correct for this bias caused by the model's suggestibility when presented with claims it knows little about.
Let $f^{\rm VC}$ be the verbalized confidence, and let $f^{\rm lat}$ be the latent, inaccessible model confidence.
We model the bias as a multiplicative scalar $\beta(c)$, which depends on the claim $c$ because the model has varying degrees of uncertainty for different topics: $f^{\rm VC}(c) = \beta(c) f^{\rm lat}(c)$.
To approximate $f^{\rm lat}$, we make the assumption that the biases for logically related (e.g., equivalent or contradictory) claims are approximately equal, since they rely on a shared, localized set of knowledge.
Let $C \subset \mathcal{C}$ be a set of mutually exclusive and exhaustive claims, e.g. claims for the year a person was born in.
Since the claims in $C$ are logically related, we assume that $\beta(c)$ is roughly the same for all $c \in C$ and so there is a scalar $\beta(C)$ with $\beta(C) \approx \beta(c)$ for all $c \in C$.
Assuming the latent confidence $f^{\rm lat}$ is probabilistically coherent,
\begin{align}
1 = \sum_{c \in C} f^{\rm lat}(c) = \sum_{c \in C} \frac{f^{\rm VC}(c)}{\beta(c)} \approx \sum_{c \in C} \frac{f^{\rm VC}(c)}{\beta(C)}.
\end{align}
Thus, we can approximate $\beta(C)$ and then $f^{\rm lat}$:
\begin{align}
\beta(C) \approx \sum_{c \in C} f^{\rm VC}(c), && f^{\rm NVC}(c) = \frac{f^{\rm VC}(c)}{\beta(C)} \approx f^{\rm lat}(c)
\label{eq:mutually-exclusive-nvc}
\end{align}
In practice, we set $\beta(C) \gets \max(1, \beta(C))$ to account for the case where $C$ fails to contain a true claim, or more precisely, a claim that the model believes to be true.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figs/method.pdf}
\caption{Normalizing verbalized confidence with \method. \textbf{(Left)} The LLM generates a claim along with several distractors
and reports its confidences on them independently. To calibrate the main claim's confidence, we divide it by $\beta$, the sum over each distractor's confidence, weighted by
uniqueness \textbf{(center)} and counterfactuality \textbf{(right)}.
Details in \cref{sec:method}.}
\label{fig:method}
\end{figure}
\subsection{Preliminary Study}
\label{sec:preliminary-study}
To empirically support our motivation above, we plot the distributions of the total confidence ($\beta(C)$ in \cref{eq:mutually-exclusive-nvc}) over correctly and incorrectly answered questions.
In this preliminary study, we take correctness as a proxy for epistemic certainty, although in reality LLMs may still be uncertain about questions they answered correctly.
In other words, we treat correct and incorrect instances as epistemically certain and uncertain instances, respectively.
Thus, our hypothesis predicts that on incorrect instances, the model would be \emph{more suggestible} and assign high confidence to more answers, leading to \textit{higher} total confidences.
On the other hand, if the model is calibrated (in particular, on uncertain instances it exhibits epistemic humility, i.e. recognizing its lack of knowledge), then the total confidence would tend to be around 1 and 0 for correct and incorrect instances, respectively, so the total confidence would be \textit{lower} on incorrect instances.
Even if the model is confident in its incorrect answers (e.g. due to misconceptions), we would expect \textit{similar} behavior between correct and incorrect instances.
\myparagraph{Experimental Setup.} We use TriviaQA \citep{joshi2017triviaqalargescaledistantly}, a dataset evaluating real-world knowledge with short-form answers.
We sample 1000 questions from the validation split of the {\tt{rc.nocontext}} subset.
We use Qwen3-8B \citep{yang2025qwen3technicalreport}\footnote{Throughout, we use the instruction-tuned versions of Qwen3 models.}
and take verbalized confidence using \ptrue \citep{kadavath2022languagemodelsmostlyknow}, computed as $P({\rm Yes})/(P({\rm Yes}) + P({\rm No}))$ when asking the model whether a given answer is correct (see Appendix~\ref{app:prompts} for prompts).
%
\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\scriptsize
% \includesvg[width=0.5\textwidth]{figs/denoms.svg}
\includegraphics[width=0.5\textwidth]{figs/denoms.pdf}
\caption{Distributions of total confidence over correct and incorrect answers.}
\vspace{-4em}
\label{fig:distr-normalization-factor}
\end{wrapfigure}
In \cref{sec:method}, we specify how we generate distractors and account for answer redundancy.
\myparagraph{Results.} In \cref{fig:distr-normalization-factor}, we first observe that many incorrect answers are assigned a confidence near 0, suggesting epistemic humility.
Even so, the incorrect distribution is heavy-tailed, resulting in a higher mean and median than the correct distribution.
These results are consistent with our hypothesis that LLMs are more prone to accepting claims that they are epistemically uncertain about.
\subsection{Method}
\label{sec:method}
Our preliminary result (\cref{fig:distr-normalization-factor}) shows that LLMs can produce incoherent probability judgments (i.e. the total confidence $\beta(C)$ exceeds 1), especially when epistemically uncertain, suggesting a need to estimate and correct for this bias.
As proposed in \cref{sec:background-and-motivation}, we normalize verbalized confidence by the total confidence $\beta(C)$ over a distractor set $C$.
We now describe in detail how we generate these distractors, and how we account for redundancy among distractors, a process illustrated in \cref{fig:method}.
Finally, we discuss how the phenomenon of generator-validator disagreement motivates incorporating self-consistency into \method.
\paragraph{Distractor generation.}
The distractor set should contain enough plausible distractors to avoid underestimating the normalization factor ($\beta(C)$ in \cref{eq:mutually-exclusive-nvc}), while remaining small enough to be tractably computed.
Thus, we frame the problem of choosing the optimal distractor set containing an original claim $c_0$ as maximizing the total acceptance probability $\sum_{c \in C} f^{\rm VC}(c)$ subject to a size constraint $|C| \le K$; shortly we address relaxing the requirement of mutual exclusivity.
Unfortunately, the validation probability $f^{\rm VC}(c)$ can only be elicited on a per-claim basis, leading to intractable sample complexity.
Motivated by the intuition that an LLM tends to generate claims that it would find plausible during validation, as a proxy for the set of claims with high verbalized confidence, we use the set of claims with high generation probability (under an appropriate prompt, explained below).
To encourage mutual exclusivity among claims, we set up the claims to be minimal pairs (\cref{fig:method} left).
For a given short-form question, we can simply sample many answers,
but independent sampling is inefficient as it overrepresents high probability generations \citep{gekhman2025insideouthiddenfactualknowledge}, limiting the number of unique distractors. Instead, we use beam search \citep{sutskever2014sequencesequencelearningneural} when available to efficiently identify unique sequences that have high probability mass coverage.
For API-access models, we use top token probabilities if available to implement a pseudo-beam search (see Appendix~\ref{app:pseudo-beam-search} for details). Otherwise in the black-box setting, we directly prompt the model to generate a list of candidate answers (see Appendix~\ref{app:prompts} for prompts).
For long-form QA, we follow \cite{min2023factscorefinegrainedatomicevaluation} in decomposing a long generation into claims.
We separately prompt the model to generate one distractor for a given claim, and again use beam search to create multiple distractors.
\paragraph{Addressing Claim Redundancy.}
%
Although the heuristic of generating minimal pairs encourages mutually exclusive claims, we have little guarantee of this mutual exclusivity (1) within the distractor set $C$ and (2) between distractors and the original claim. Assuming such mutual exclusivity when there are actually redundant claims can lead to overcounting in the normalization factor $\beta(C)$.
Thus, we use an NLI model to quantify entailment and contradiction relationships between claims.\footnote{Access to an NLI model poses only a minimal departure from the zero-resource setting, since NLI is a generic task for which there are off-the-shelf models. NLI is a subset of the tasks that LLMs are capable of, so the usage of a separate NLI model is motivated merely by efficiency \citep{kuhn2023semanticuncertaintylinguisticinvariances, lin2024generatingconfidenceuncertaintyquantification}.}
We address (1) and (2) with $w_{\rm unique}$ and $w_{\rm contra}$, respectively (\cref{fig:method}):
\begin{align}
w_{\rm unique}(c) = \frac{1}{\sum_{c' \in C} P({\rm entail} \mid c', c)}, && w_{\rm contra}(c) = \frac{P({\rm contra} \mid c_0, c) + P({\rm contra} \mid c, c_0)}{2}
\end{align}
Intuitively, $w_{\rm unique}$ downweights a claim if it is entailed by other claims, and $w_{\rm contra}$ downweights a claim if it is not contradictory with the main claim.
We now normalize the verbalized confidence of the original claim as
\begin{align}
f^{\rm NVC}(c_0) = \frac{f^{\rm VC}(c_0)}{\beta(C)}, && \beta(C) = \max\left(1, f^{\rm VC}(c_0) + \sum_{c \in C} f^{\rm VC}(c) \cdot w_{\rm unique}(c) \cdot w_{\rm contra}(c)\right),
\end{align}
generalizing the mutually exclusive case in \cref{eq:mutually-exclusive-nvc}. The maximization with 1 allows for defaulting back to the vanilla verbalized confidence in the case where $C$ fails to contain claims that the model considers plausible.
\paragraph{Combining Coherence within Generation and Validation.}
Our approach so far (summarized in \cref{fig:method}) for normalizing verbalized confidence across distractors focuses on coherence within claim \textit{validation}.
Previous studies disagree on the question of whether models are better at the discriminative or generative counterparts of a given task \citep{west2023generativeaiparadoxwhat, gekhman2025insideouthiddenfactualknowledge}, but generally agree on the presence of generator-validator inconsistency \citep{li2023benchmarkingimprovinggeneratorvalidatorconsistency}, wherein a model may produce inconsistent results between the generation and validation stages.
Indeed, we find that in the preliminary study setting in \cref{sec:preliminary-study}, the answer with the highest generation probability and the answer with the highest validation probability (over 10 answers obtained from beam search) agree\footnote{\label{footnote:semantic-match}We consider $c$ and $c'$ equivalent answers to question $q$ if $\frac12 P({\rm entail} \mid c, c'; q) + \frac12 P({\rm entail} \mid c', c; q) > 0.9$.} on only 592 out of 1000 questions.
We integrate these complementary aspects of generation and validation into \method.
In particular, we draw on a distributional view of the generator-validator gap \citep{rodriguez2025rankalignrankingviewgeneratorvalidator}, in which the generation probability distribution over candidate answers (which we approximate with self-consistency sampling, $f^{\rm SC}$) is distinct from the validation probability distribution over them (which we approximate with normalized verbalized confidence, $f^{\rm NVC}$).
\method thus incorporates confidence (i.e. probability mass) in both the generator and validator distributions: $f^{\method}(c) = \frac12 f^{\rm SC}(c) + \frac12 f^{\rm NVC}(c)$.
We leave a full description of the self-consistency component to Appendix~\ref{app:self-consistency}.
\section{Experiments}
\label{sec:expts}
\subsection{Experimental Setup}
\myparagraph{Short-form Datasets.} Short-form QA serves as a testbed for evaluating factuality as well as calibration because of its tractable evaluation and adjustable difficulty.
The task is relevant in practice because it assesses models' ability to respond to information-seeking users.
TriviaQA contains trivia questions requiring diverse world knowledge \citep{joshi2017triviaqalargescaledistantly}.
SimpleQA similarly contains short, fact-seeking questions, curated with the criterion of challenging frontier models \citep{wei2024measuringshortformfactualitylarge}.
We sample 1000 questions from each dataset.\footnote{We use the validation split of the {\tt{rc.nocontext}} subset for TriviaQA. On SimpleQA, Gemini-2.5-Flash produced a refusal error with no output on 86 questions, so we exclude them from all experiments.}
We use LLM-as-a-judge to evaluate binary correctness, following best practices for robust evaluation \citep{wei2024measuringshortformfactualitylarge}; in Appendix~\ref{app:llm-as-a-judge} we confirm high human agreement.
\myparagraph{Long-form Datasets.} While short-form settings are appealing for their simple evaluation, many real-world tasks require longer generations, for which calibrated confidence estimation remains critical.
The long-form setting comes with the evaluation challenge that responses generally contain both correct and incorrect parts, complicating the assignment of a single correctness score.
In our experiments, we evaluate long-form calibration on biography generation using \factscore \citep{min2023factscorefinegrainedatomicevaluation}.
\factscore decomposes a generated biography into atomic claims and verifies each claim against Wikipedia (see Appendix~\ref{app:factscore-example} for an example), thus enabling us to evaluate calibration at the claim level.
We use the labeled subset containing 183 entities from \cite{min2023factscorefinegrainedatomicevaluation}.
\myparagraph{Models.} Since TriviaQA and SimpleQA are adequately challenging for smaller and larger models, respectively, we focus their evaluation accordingly.
TriviaQA is largely solved by frontier models \citep{wei2024measuringshortformfactualitylarge}, and SimpleQA is too difficult for smaller models.\footnote{\url{https://www.kaggle.com/benchmarks/openai/simpleqa}}
On TriviaQA, we use popular open-source models: Qwen3-8B and Qwen3-1.7B \citep{yang2025qwen3technicalreport}, Llama-3.2-3B-Instruct \citep{grattafiori2024llama3herdmodels}, and Gemma-3-4B-IT \citep{gemmateam2025gemma3technicalreport}.
On SimpleQA, we use popular frontier models: GPT-4.1 \citep[2025-04-14;][]{openai2025gpt41} and Gemini-2.5-Flash \citep{comanici2025gemini}.
For SimpleQA evaluated on frontier models, we also consider the black-box setting where no logit access is assumed; here, rather than using pseudo-beam search to generate distractors, we prompt the model directly to generate diverse distractors.
Moreover, we replace \ptrue with verbalized numerical confidence to forgo logit access.
For the long-form task of biography generation, we limit our evaluation to Qwen3-8B and Gemma-3-4B-IT, due to the cost of \factscore evaluation with GPT-4.1.
We use the NLI model {\tt{DeBERTa-v3-base-mnli-fever-anli}} \citep{he2021debertadecodingenhancedbertdisentangled} for our methods and self-consistency.
\myparagraph{Evaluation Metrics.} We evaluate Expected Calibration Error \citep[\textbf{ECE} $\downarrow$;][]{Naeini2015ObtainingWC} with 10 bins, Brier score \citep[\textbf{BS} $\downarrow$;][]{Brier1950VERIFICATIONOF}, and area under the ROC curve \citep[\textbf{AUC} $\uparrow$;][]{hanley1982auroc}.
See Appendix~\ref{app:evaluation-metrics} for descriptions.
For biography generation, we also evaluate Pearson and Spearman correlation between average claim-level confidence and passage-level \factscore (without length penalty), where the latter measures the proportion of claims that are correct.
\myparagraph{Baselines.} Following past work in zero-resource calibration, we compare against training-free methods that produce probabilities without post-hoc calibration \citep{tian2023justaskcalibrationstrategies, xiong2024llmsexpressuncertaintyempirical, Steyvers_2025}.
We provide prompts for our methods and baselines in Appendix~\ref{app:prompts}.
Verbalized confidence \citep[\textbf{VC}; \ptrue from][]{kadavath2022languagemodelsmostlyknow} asks the model whether its answer is correct and computes $P({\rm Yes})/(P({\rm Yes}) + P({\rm No}))$.
It is straightforward to replace \ptrue with verbalized numerical confidence; in Appendix~\ref{sec:verbalized-numerical-confidence} we show that the latter similarly benefits from our methods, showing robustness to the format of verbalized confidence.
Top-$K$ prompting \citep[\textbf{$K$-VC}; Verb. 1S from][]{tian2023finetuninglanguagemodelsfactuality} prompts the model to provide its top $K$ guesses along with verbalized numerical confidences.
For our black-box setting, we use the candidate answers generated using the $K$-VC prompt as distractors, but we discard the verbalized confidences contained in the same generation, and instead separately collect verbalized numerical confidence on each distractor independently.
Maximum sequence probability \citep[\textbf{MSP};][]{fadeeva2023lmpolygraphuncertaintyestimationlanguage} is the LLM's probability of generating its answer.
Self-consistency \citep[\textbf{SC};][]{xiong2024llmsexpressuncertaintyempirical} samples several answers (we use temperature 1) and computes the proportion that match the main answer.
Following \cite{kuhn2023semanticuncertaintylinguisticinvariances}, we use an NLI model to determine semantic equivalence.\footref{footnote:semantic-match}
For biography generation, we modify self-consistency to sample several biographies and measure entailment of each claim \citep{zhang2024luqlongtextuncertaintyquantification}.
\textbf{SC-VC} \citep{xiong2024llmsexpressuncertaintyempirical} is SC weighted by verbalized confidence \citep[we use \ptrue following][]{Taubenfeld_2025}.
\myparagraph{Inference Budget.} As our methods and several baseline methods ($K$-VC, SC, SC-VC) operate on a variable inference-time budget, we control this budget at $K=10$. For \method, we use 5 samples for self-consistency and 5 distractors for normalized verbalized confidence.
\subsection{Results}
\begin{table}[t]
\caption{TriviaQA results. We evaluate Expected Calibration Error (ECE), Brier score (BS), and area under the ROC curve (AUC).
In each column, we bold the best result and underline results not significantly worse under a paired test ($\alpha = 0.05$; see Appendix~\ref{app:significance-testing} for tests).
}
\label{tab:triviaqa}
\begin{center}
\setlength{\tabcolsep}{4pt}
% \small
\resizebox{\linewidth}{!}{%
\renewcommand{\arraystretch}{1.3}
\begin{tabular}{lcccccccccccc}
\toprule
& \multicolumn{3}{c}{\textit{Qwen3-8B}} & \multicolumn{3}{c}{\textit{Qwen3-1.7B}} & \multicolumn{3}{c}{\textit{Llama-3.2-3B-Instruct}} & \multicolumn{3}{c}{\textit{Gemma-3-4B-IT}} \\ \cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ \\
\midrule
VC \citep{kadavath2022languagemodelsmostlyknow} & 0.240 & 0.242 & 0.817 & 0.387 & 0.383 & 0.720 & 0.189 & 0.208 & 0.826 & 0.300 & 0.299 & 0.702 \\
$K$-VC \citep{tian2023finetuninglanguagemodelsfactuality} & 0.341 & 0.348 & 0.604 & 0.538 & 0.524 & 0.596 & 0.146 & 0.228 & 0.678 & 0.254 & 0.262 & 0.786 \\
MSP \citep{fadeeva2023lmpolygraphuncertaintyestimationlanguage} & 0.149 & 0.203 & 0.819 & 0.104 & 0.186 & 0.774 & 0.243 & 0.253 & 0.764 & 0.252 & 0.268 & 0.790 \\
SC-VC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.299 & 0.325 & 0.704 & 0.451 & 0.474 & 0.559 & 0.122 & 0.211 & 0.761 & 0.362 & 0.378 & 0.653 \\
SC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.236 & 0.244 & 0.785 & 0.233 & 0.229 & 0.780 & 0.065 & 0.177 & 0.808 & 0.303 & 0.304 & 0.713 \\
NVC & 0.171 & 0.190 & 0.853 & \textbf{0.084} & \textbf{0.164} & 0.806 & 0.168 & 0.192 & 0.845 & 0.218 & 0.236 & 0.791 \\
\method & \textbf{0.097} & \textbf{0.155} & \textbf{0.879} & 0.177 & 0.179 & \textbf{0.835} & \textbf{0.044} & \textbf{0.148} & \textbf{0.864} & \textbf{0.121} & \textbf{0.191} & \textbf{0.817} \\
\bottomrule
\end{tabular}
}
\end{center}
\end{table}
\myparagraph{Short-form QA.} \cref{tab:triviaqa,tab:simpleqa} report results for TriviaQA and SimpleQA.
On TriviaQA, \method outperforms the best baseline, SC, by an average ECE of 0.099.
On SimpleQA, \method outperforms the best baseline, MSP, by an average ECE of 0.092.
While the better baseline varies by dataset, \method consistently achieves the strongest calibration.
MSP is sometimes a competitive baseline (e.g. AUC 0.800 surpasses \method at 0.786 on SimpleQA with GPT-4.1), but this often does not hold across metrics (e.g. MSP has an ECE of 0.263 in the same setting, heavily underperforming \method at 0.089) or across settings (e.g. MSP underperforms \method on AUC in TriviaQA by an average of 0.062).
Moreover, we note that the effectiveness of MSP relies on answers having a canonical form, preventing its generalization to long-form settings \citep{Farquhar2024DetectingHI}.
We highlight that NVC outperforms SC (e.g. by an ECE of 0.049 and 0.060 on TriviaQA and SimpleQA, respectively) despite only leveraging coherence in validation and not in generation (\cref{sec:method}).
Nonetheless, \method is more consistently calibrated than NVC (in particular on AUC, e.g. 0.786 \method vs. 0.729 NVC with GPT-4.1 on SimpleQA), empirically supporting our motivation in \cref{sec:method} for integrating coherence in generation (SC) and validation (NVC) into \method.
In the black-box setting on SimpleQA, \method continues to do well (e.g. outperforming the baselines on ECE), but it tends to fall behind \method with logit access, underscoring the benefit of leveraging token probabilities for calibration.
\begin{table}[t]
\caption{SimpleQA results.
The black-box variants of our methods assume no logit access.
Metrics and text styling follow \cref{tab:triviaqa}.
}
\label{tab:simpleqa}
\begin{center}
\small
\begin{tabular}{lcccccc}
\toprule
& \multicolumn{3}{c}{\textit{GPT-4.1}} & \multicolumn{3}{c}{\textit{Gemini-2.5-Flash}} \\ \cmidrule(lr){2-4}\cmidrule(lr){5-7}
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ \\
\midrule
VC \citep{kadavath2022languagemodelsmostlyknow} & 0.547 & 0.549 & 0.644 & 0.409 & 0.393 & 0.617 \\
$K$-VC \citep{tian2023finetuninglanguagemodelsfactuality} & 0.338 & 0.337 & 0.632 & 0.535 & 0.511 & 0.566 \\
MSP \citep{fadeeva2023lmpolygraphuncertaintyestimationlanguage} & 0.263 & 0.255 & \textbf{0.800} & 0.098 & \underline{0.177} & \textbf{0.773} \\
SC-VC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.223 & 0.252 & 0.761 & 0.186 & 0.221 & \underline{0.755} \\
SC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.220 & 0.252 & 0.750 & 0.170 & 0.212 & \underline{0.748} \\
NVC$_{\text{black-box}}$ & 0.213 & 0.270 & 0.607 & 0.208 & 0.262 & 0.595 \\
\method{}$_{\text{black-box}}$ & 0.161 & 0.251 & 0.605 & \textbf{0.079} & 0.199 & 0.697 \\
NVC & 0.164 & 0.222 & 0.729 & 0.105 & 0.199 & 0.662 \\
\method & \textbf{0.089} & \textbf{0.183} & \underline{0.786} & 0.088 & \textbf{0.174} & \underline{0.762} \\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\begin{table}[t]
\caption{\factscore results.
In addition to the claim-level metrics, we report Pearson ($r$) and Spearman ($\rho$) correlation with passage-level \factscore.
Text styling follows \cref{tab:triviaqa}, and we bold the best $r$ and $\rho$.
}
\label{tab:factscore}
\begin{center}
\resizebox{\linewidth}{!}{%
\begin{tabular}{lcccccccccc}
\toprule
& \multicolumn{5}{c}{\textit{Qwen3-8B}} & \multicolumn{5}{c}{\textit{Gemma-3-4B-IT}} \\ \cmidrule(lr){2-6}\cmidrule(lr){7-11}
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & $r$ $\scriptstyle\uparrow$ & $\rho$ $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & $r$ $\scriptstyle\uparrow$ & $\rho$ $\scriptstyle\uparrow$ \\
\midrule
VC \citep{kadavath2022languagemodelsmostlyknow} & 0.433 & 0.431 & 0.625 & 0.073 & 0.122 & 0.527 & 0.527 & 0.683 & -0.081 & -0.129 \\
SC \citep{zhang2024luqlongtextuncertaintyquantification} & 0.162 & \underline{0.226} & \textbf{0.771} & 0.468 & 0.494 & 0.197 & \underline{0.233} & \underline{0.787} & 0.629 & 0.607 \\
NVC & 0.191 & 0.263 & 0.681 & 0.444 & 0.443 & \textbf{0.123} & \underline{0.230} & 0.726 & 0.695 & 0.704 \\
\method & \textbf{0.076} & \textbf{0.202} & \underline{0.767} & \textbf{0.518} & \textbf{0.538} & 0.172 & \textbf{0.210} & \textbf{0.793} & \textbf{0.724} & \textbf{0.712} \\
\bottomrule
\end{tabular}
}
\end{center}
\end{table}
\myparagraph{Long-form QA.} \cref{tab:factscore} reports results on \factscore.
While VC is extremely miscalibrated (e.g. ECE of 0.433 with Qwen3-8B), \method is able to leverage incoherence across related claims to normalize verbalized confidence and achieve strong calibration.
Whether SC or NVC performs better varies by the model Qwen3-8B or Gemma-3-4B-IT, but \method continues to outperform SC (0.076 vs. 0.162 ECE with Qwen3-8B, and 0.172 vs. 0.197 ECE with Gemma-3-4B-IT).
Furthermore, \method is the method most strongly correlated with passage-level \factscore (e.g. improving Pearson and Spearman correlation over SC by an average of 0.072 and 0.074, respectively), demonstrating that the effectiveness of \method extends to the long-form setting.
Taken together with the short-form results in \cref{tab:triviaqa,tab:simpleqa}, these results indicate that \method{} is applicable to open- and closed-source models, and crucially can transfer seamlessly between short-form QA and long-form generation settings.
\section{Discussion and Analysis}
We conduct further analysis using Qwen3-8B on TriviaQA with \ptrue, which appeared in \cref{tab:triviaqa} in the main experiments.
\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-1em}
\centering
% \scriptsize
% \includesvg[width=\linewidth]{figs/ece_bs_vs_samples.svg}
\includegraphics[width=\linewidth]{figs/ece_bs_vs_samples.pdf}
\captionof{figure}{Scaling self-consistency does not close the gap with \method.}
\label{fig:scaling-analysis}
\vspace{-1em}
\end{wrapfigure}
\paragraph{Scaling Self-Consistency.} While our main experiments in \cref{sec:expts} were controlled for the inference budget of each method, here we examine whether simply scaling the inference budget of self-consistency allows it to recover the calibration of \method. \cref{fig:scaling-analysis} shows that self-consistency alone is unable to reach the performance of \method (using 5 distractors and 5 self-consistency samples as in \cref{sec:expts}) even when scaling up to 100 samples, demonstrating that the effectiveness of \method comes from leveraging coherence in both generation and validation, which is not matched by scaling the generation axis alone.
\begin{figure}[btp]
% First figure
\begin{minipage}[t]{0.47\textwidth}
\vspace{0pt}
\centering
\small
\captionof{table}{Saturation analysis (higher $\Delta$ = lower saturation). \method alleviates saturation.}
\label{tab:saturation}
\begin{tabular}{lcc}
\toprule
Method & $\Delta_0$ & $\Delta_{0.001}$ \\
\midrule
VC & 0.670 & 0.605 \\
SC & 0.734 & 0.734 \\
SC@100 & 0.832 & 0.832 \\
\method & 0.998 & 0.984 \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
% Second figure
\begin{minipage}[t]{0.47\textwidth}
\vspace{0pt}
\centering
\small
\captionof{table}{Ablation of NLI-based weighting shows it is crucial to performance.}
\label{tab:gate-ablation}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{lccc}
\toprule
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ \\
\midrule
NVC & \textbf{0.171} & \textbf{0.190} & \textbf{0.853} \\
\quad w/o NLI & 0.358 & 0.335 & 0.778 \\\hline
\method & \textbf{0.097} & \textbf{0.155} & \textbf{0.879} \\
\quad w/o NLI & 0.130 & 0.185 & 0.810 \\
\bottomrule
\end{tabular}
\end{minipage}
\end{figure}
\paragraph{Quantifying Saturation.} A core motivation for our method is the notion (shown in \cref{fig:vc-versus-dinco-calibration}) that verbalized confidence exhibits saturation at high confidence.
To better quantify this notion, we introduce a metric $\Delta_\epsilon$ that measures the absence of saturation.
We define $\Delta_\epsilon$ as the proportion of pairs of distinct instances that have a confidence difference exceeding $\epsilon$.
For example, if all confidence scores are the same, then $\Delta_0 = 0$, and if all confidence scores are distinct, then $\Delta_0 = 1$.
We consider $\epsilon \in \{0, 0.001\}$.
\cref{tab:saturation} shows that \method leads to substantially higher rates of distinct confidence, indicating lower saturation.
In particular, self-consistency scaled to 100 samples (as above) continues to be more saturated than \method.
While the absence of saturation alone means little without calibration (as evaluated in \cref{sec:expts}), this analysis helps explain \method's calibration improvement, as hinted at by \cref{fig:vc-versus-dinco-calibration}.
Moreover, we argue that lower saturation leads to more usable confidence estimates: a saturated distribution is inherently less controllable, with large jumps in error between thresholds.
\paragraph{Ablating NLI.} Our main experiments in \cref{tab:triviaqa,tab:simpleqa,tab:factscore} showed comparisons with SC and NVC, which ablate NVC and SC, respectively, from \method.
Here, to understand how necessary access to an NLI model is, we ablate the NLI component, which is used to downweight distractors that overlap with the main claim or other distractors (\cref{sec:method}, \cref{fig:method}).
In \cref{tab:gate-ablation}, we see that performance substantially decreases without NLI-based weighting, emphasizing the utility of an off-the-shelf NLI model to account for claim redundancy.
\section{Related Work}
\label{sec:related-work}
\myparagraph{Considering Multiple Answers for Verbalized Confidence.} The approach most related to \method is to have the model consider several candidates within a single prompt and assign confidences to them \citep{tian2023justaskcalibrationstrategies, kadavath2022languagemodelsmostlyknow, wang2024calibratingverbalizedprobabilitieslarge}.
Similarly, \cite{chhikara2025mindconfidencegapoverconfidence} samples answers and presents them in the context as distractors.
A subtle but crucial distinction between these methods and \method is that if we present all the candidates together, we become unable to gauge the probabilistic coherence of the confidence estimates, i.e. whether they form a valid probability distribution, since LLMs can satisfy probabilistic coherence via simple arithmetic.
In \cref{sec:preliminary-study}, we show that the probabilistic coherence of confidence estimates is correlated with answer correctness.
Instruction-tuned LLMs have a tendency to assert confidence even when it is undue \citep{openai2023gpt4, leng2025tamingoverconfidencellmsreward, sun2025largelanguagemodelsoverconfident, xu2025languagemodelsmirrorhuman}, leading joint prompting to suffer similar issues of overconfidence as vanilla prompting.
With independent prompting, we can expose and account for inconsistencies in self-declared knowledge.
In \cref{sec:expts}, we empirically verify that our method leads to better calibration than joint prompting for verbalized confidence.
\myparagraph{Confidence Estimation in Long-form Generation.} While historically confidence estimation has mostly been applied to classification \citep{houlsby2011bayesianactivelearningclassification}, multiple-choice \citep{jiang2021knowlanguagemodelsknow}, and short-form QA \citep{xiong2024llmsexpressuncertaintyempirical}, with the advent of LLMs it has increasingly been considered for long-form generation.
Since token-level uncertainty is often ill-suited for representing claim-level uncertainty, the primary approaches have been self-consistency and verbalized confidence \citep{manakul2023selfcheckgptzeroresourceblackboxhallucination, zhang2024luqlongtextuncertaintyquantification, zhang2025reinforcementlearningbetterverbalized}.
We propose a method to normalize verbalized confidence, enabling \method to combine the complementary confidence signals in these two prior approaches.
\myparagraph{Reconciling Inconsistent LLM Probability Judgments.} In Appendix~\ref{app:reconciling-inconsistent-llm}, we discuss a related line of work demonstrating the benefits of reconciling inconsistent LLM probability judgments.
In this work, we propose a zero-resource confidence estimator that normalizes verbalized confidence over self-generated distractors, motivated by the suggestibility of LLMs in unfamiliar topics.
\section{Conclusion}
We present \method, which estimates LLM confidence by leveraging coherence in generation as well as validation (through verbalized confidence).
Verbalized confidence tends to be saturated at high confidence.
We show evidence that this behavior is correlated with \textit{suggestibility}, where the LLM is more likely to accept claims that it knows less about.
Motivated by this finding, \method has the LLM verbalize its confidence independently on several self-generated distractors to estimate and correct for the bias caused by suggestibility.
\method outperforms existing methods in the zero-resource setting on short-form QA (TriviaQA, SimpleQA) and long-form QA (\factscore) and mitigates saturation.
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\appendix
\section{Related Work}
\subsection{Related Behavior in LLMs and Humans}
\label{app:related-work-related-behavior}
Humans are also known to be susceptible to suggestibility. They can alter their memories to match the suggestions of other people, especially at a young age \citep{bruck1999thesuggestibilityofchildrensmemory}.
Sycophancy is a similar behavior observed in LLMs, where an epistemically vacuous prompt such as \textit{``Are you sure?''} often leads the model to change its answer \citep{laban2024surechallengingllmsleads}.
The user expressing doubt suggests to the model that its answer may be incorrect, since the user has some assumed level of credibility and would be unlikely to ask again if they agreed.
Since instruction-tuned models aim to adhere to user preferences, it is plausible that they would employ an \textit{argument from ignorance} \citep{walton1996arguments} to accept a user claim that they cannot refute.
\cite{zhu2025incoherentprobabilityjudgmentslarge} provide evidence of probabilistic incoherence in LLMs and attribute this finding to the prior from the Bayesian Sampler model, which has been used to explain incoherence in human probability judgments \citep{zhu2018bayesianinferencecausesincoherence}.
In particular, if the same prior is used for every probability judgment, the sum of the probability judgments for mutually exclusive events can exceed 1 \citep{zhu2025incoherentprobabilityjudgmentslarge}.
This failure to satisfy the axioms of probability is consistent with our empirical evidence in \cref{sec:preliminary-study}.
\subsection{Reconciling Inconsistent LLM Probability Judgments.}
\label{app:reconciling-inconsistent-llm}
Prior work has demonstrated the benefits of reconciling inconsistent LLM probability judgments instead of taking them at face value.
\cite{jung2022maieuticpromptinglogicallyconsistent} improve factuality by selecting claims to which the LLM assigns coherent truth values upon negation.
\cite{hou2025probabilisticframeworkllmhallucination} use belief tree propagation with logically related claims to detect hallucinations.
\cite{nafar2025extractingprobabilisticknowledgelarge} find that independent prompting followed by normalization outperforms joint prompting for Bayesian network parameter estimation.
\cite{feng2025birdtrustworthybayesianinference, xia2024letsthinkvarbyvarlarge} optimize a probability distribution to approximately satisfy LLM-generated probability constraints.
In this work, we propose a zero-resource confidence estimator that normalizes verbalized confidence over self-generated distractors, motivated by the suggestibility of LLMs in unfamiliar topics.
\section{Experimental Setup} \label{app:setup}
\subsection{Evaluation Metrics}
\label{app:evaluation-metrics}
We adopt the notation from \cref{sec:background-and-motivation}. For a claim $c$, the truth value is $v(c) \in \{0, 1\}$ and the assigned confidence is $f(c) \in [0, 1]$.
\paragraph{Expected Calibration Error \citep[ECE;][]{Naeini2015ObtainingWC}.}
The confidence space $[0, 1]$ is partitioned into $K$ intervals of equal length. Out of the $N$ claims in the dataset, let $B_k$ be the list of claims assigned a confidence in the interval $I_k = (\frac{k-1}{K}, \frac{k}{K}]$ (with $I_1$ including 0). We compute bin-level truthfulness and confidence as
\begin{align}
\bar{v}_k = \frac{1}{|B_k|} \sum_{c \in B_k} v(c), && \bar{f}_k = \frac{1}{|B_k|} \sum_{c \in B_k} f(c).
\end{align}
ECE is the bin size-weighted average of the absolute differences:
\begin{align}
{\rm ECE} = \sum_{k=1}^K \frac{|B_k|}{N} |\bar{v}_k - \bar{f}_k|
\end{align}
\paragraph{Brier score \citep[BS;][]{Brier1950VERIFICATIONOF}.}
For a dataset of claims $c_1, \ldots, c_N$, the Brier score is the mean squared error between the truth values and the confidence estimates:
\begin{align}
{\rm BS} = \frac1N \sum_{i=1}^N (v(c_i) - f(c_i))^2
\end{align}
\paragraph{Area under ROC curve \citep[AUC;][]{hanley1982auroc}}
The ROC curve (depicted in \cref{fig:vc-versus-dinco-calibration} right) captures the tradeoffs between true and false positive rate (TPR, FPR) that we can obtain with selective prediction, i.e. setting a confidence threshold above which to accept claims.
We take correct and incorrect claims to be labeled positive and negative, respectively.
The TPR is the proportion of positive instances that are accepted, and the FPR is the proportion of negative instances that are accepted.
We want the TPR to be high but the FPR to be low.
By setting a lower confidence threshold, TPR will be higher, but FPR may also be higher.
By setting a higher confidence threshold, FPR will be lower, but TPR may also be lower.
As not every TPR (or FPR) in the interval $[0, 1]$ may be achievable, the ROC fills in the gaps between achievable (TPR, FPR) tradeoffs with linear interpolations, as seen with verbalized confidence in \cref{fig:vc-versus-dinco-calibration}.
As a summary statistic for the selective predictive power that a confidence estimator grants us, we compute the area under the ROC curve (AUC).
For example, if all positive instances are assigned a higher confidence than all negative instances, the AUC is 1. Meanwhile, if confidences are sampled independently at random from the same distribution, the expected AUC is 0.5.
The AUC can also be characterized as the probability that a random positive instance is assigned higher confidence than a random negative instance, with ties randomly broken. Denoting $C_+$ and $C_-$ as the list of correct and incorrect claims in the dataset, respectively,
\begin{align}
{\rm AUC} = \frac{1}{|C_+||C_-|} \sum_{c_+ \in C_+} \sum_{c_- \in C_-} \frac{\displaystyle\bm{1}\{f(c_+) \ge f(c_-)\} + \displaystyle\bm{1}\{f(c_+) > f(c_-)\}}{2}.
\end{align}
\subsection{Prompts}
\label{app:prompts}
\subsubsection{Short-form QA}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Prompt to generate main answer. Also used for beam search in \method and sampling for self-consistency.}]
\begin{PromptVerbatim}
Here are 2 sets of example prompt and answer.
Example Prompt: Which American-born Sinclair won the Nobel Prize for Literature in 1930?
Example Answer: Sinclair Lewis
Example Prompt: Where in England was Dame Judi Dench born?
Example Answer: York
---
Now, here is a new prompt to answer. Answer with a concise phrase, as in the examples.
Prompt: \VarField{question}
Answer:
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={\ptrue}]
\begin{PromptVerbatim}
Below is a question and a candidate answer. Your task is to determine whether the answer is correct or not. Only output "Yes" (correct) or "No" (incorrect).
Question: \VarField{question}
Candidate answer: \VarField{candidate_answer}
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Verbalized numerical confidence}]
\begin{PromptVerbatim}
Below is a question and a candidate answer. State your confidence that the candidate answer is correct. Only output an integer followed by "\%".
Question: \VarField{question}
Candidate answer: \VarField{candidate_answer}
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={$K$-VC \citep[Verb. 1S from][]{tian2023justaskcalibrationstrategies}. We use $K=10$.}]
\begin{PromptVerbatim}
Provide your \VarField{K} best guesses and the probability that each is correct (0.0 to 1.0) for the following question. Give ONLY the guesses and probabilities, no other words or explanation. For example:
G1: \textless{}first most likely guess, as short as possible; not a complete sentence, just the guess!\textgreater
P1: \textless{}the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!\textgreater
...
G\VarField{K}: \textless\VarField{K}th most likely guess, as short as possible; not a complete sentence, just the guess!\textgreater
P\VarField{K}: \textless{}the probability between 0.0 and 1.0 that G\VarField{K} is correct, without any extra commentary whatsoever; just the probability!\textgreater
The question is: \VarField{question}
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Follow-up after main answer for SC-VC}]
\begin{PromptVerbatim}
Is your answer correct? Only output "Yes" or "No".
\end{PromptVerbatim}
\end{tcolorbox}
\subsubsection{Biography Generation}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Prompt to generate main biography. Also used to sample biographies for self-consistency.}]
\begin{PromptVerbatim}
Write me a paragraph biography on \VarField{entity}.
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Prompt to generate one distractor. We use beam search to extract multiple.}]
\begin{PromptVerbatim}
You will be given a fact about a person. Assuming the fact is accurate, your task is to generate a plausible but inaccurate statement of a similar nature. The distractor statement should form a minimal pair with the original statement, i.e. the distractor should be as similar to the original as possible while ensuring that the distractor is not factual. The distractor should be crafted so that someone with only superficial knowledge about the topic is likely to be fooled.
Let's see some examples before the real task.
Topic: Barack Obama
Fact: Barack Obama was born in Hawaii.
Distractor: Barack Obama was born in Kenya.
Topic: Wright brothers
Fact: Wright airplanes were involved in fatal crashes.
Distractor: Wright airplanes were praised for their safety.
Topic: John Clempert
Fact: John Clempert was inspired by Houdini when developing acts.
Distractor: John Clempert was inspired by Penn and Teller when developing acts.
Now for the real task. Only output a distractor as in the examples.
Topic: \VarField{entity}
Fact: \VarField{claim}
Distractor:
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={\ptrue}]
\begin{PromptVerbatim}
Your task is to determine whether the following claim related to \VarField{entity} is correct. Only output "Yes" (correct) or "No" (incorrect).
Claim: \VarField{claim}
Yes or No:
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Verbalized numerical confidence}]
\begin{PromptVerbatim}
The claim below was found in a passage about \VarField{entity}. State your confidence that the claim is correct. Only output an integer followed by "\%".
Claim: \VarField{claim}
\end{PromptVerbatim}
\end{tcolorbox}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Prompt for the LLM to determine whether a sampled biography entails a claim for self-consistency}]
\begin{PromptVerbatim}
You will be given a passage and a claim. Your task is to determine whether the passage supports, refutes, or does not mention the claim. Output only "Support", "Refute", or "No Mention".
Let's see some examples before the real task.
Passage: Barack Obama was the 44th President of the United States, serving from 2009 to 2017. Born on August 4, 1961, in Honolulu, Hawaii, he was the first African American to hold the office. Before his presidency, Obama served as a state senator in Illinois and later as the 47th Governor of Illinois. A former constitutional law professor, he was known for his eloquence, bipartisan approach, and focus on issues such as healthcare reform, climate change, and foreign policy. His presidency was marked by significant legislative achievements, including the Affordable Care Act, and a commitment to diplomacy and international cooperation. After leaving office, he authored memoirs and remained active in public life, advocating for social justice and community engagement.
Claim: Barack Obama was born in Hawaii.
Relationship: Support
Passage: Tiger Woods is one of the most iconic and accomplished golfers in history, known for his extraordinary talent, dominance on the course, and global influence on the sport. Born on December 30, 1975, in Cypress, Florida, Woods rose to fame in the mid-1990s and quickly became a household name, winning his first major championship at the 1997 Masters at just 21 years old. Over his career, he has claimed 15 major titles, the most in PGA Tour history, and has consistently ranked among the world's top golfers for over two decades. His aggressive playing style, precision, and mental toughness set him apart, making him a symbol of excellence in golf. Despite personal challenges and setbacks, Woods has remained a dominant force in the sport, inspiring millions of fans around the world.
Claim: Tiger Woods won a major championship at 19 years old.
Relationship: Refute
Passage: Albert Einstein was a theoretical physicist renowned for developing the theory of relativity, which revolutionized the understanding of space, time, and gravity. Born in 1879 in Ulm, Germany, he later moved to Switzerland and eventually to the United States. Einstein's work, including the famous equation E=mc², laid the foundation for modern physics and contributed to the development of nuclear energy. Despite his scientific achievements, he was also a passionate advocate for peace, civil rights, and education. His legacy endures as one of the most influential scientists in history.
Claim: Albert Einstein became a US citizen.
Relationship: No Mention
Now for the real task.
Passage: \VarField{sampled_biography}
Claim: \VarField{claim}
Relationship:
\end{PromptVerbatim}
\end{tcolorbox}
\subsubsection{Pseudo-Beam Search}
\label{app:pseudo-beam-search-prompt}
\begin{tcolorbox}[colback=white!5!white, colframe=black!75!black, width=\columnwidth, title={Prompt for the LLM to complete the prefix of an answer. Used for pseudo-beam search (Appendix~\ref{app:pseudo-beam-search}).}]
\begin{PromptVerbatim}
You will be given a prompt along with a prefix to begin your answer with. Your answer should start with the given prefix. If the prefix itself is your final answer, you can simply output just the prefix.
Let's look at 2 examples before the real task.
Example Prompt: Which American-born Sinclair won the Nobel Prize for Literature in 1930?
Example Answer Prefix: Sin
Example Answer: Sinclair Lewis
Example Prompt: Where in England was Dame Judi Dench born?
Example Answer Prefix: York
Example Answer: York
---
Now, here is a new prompt to answer. Answer with a concise phrase starting with the given prefix, as in the examples.
Prompt: \VarField{question}
Prefix: \VarField{prefix}
Answer:
\end{PromptVerbatim}
\end{tcolorbox}
\subsection{LLM-as-a-Judge}
\label{app:llm-as-a-judge}
We score model responses for short-form QA (TriviaQA and SimpleQA) using an LLM judge rather than lexical matching for robust evaluation \citep{wei2024measuringshortformfactualitylarge}.
Our biography generation task is evaluated with \factscore \citep{min2023factscorefinegrainedatomicevaluation}, which uses a strong LLM for atomic claim decomposition and verification; we use GPT-4.1.
We use Llama-3.1-8B-Instruct \citep{grattafiori2024llama3herdmodels} as the judge on TriviaQA, and GPT-4.1 as the judge on SimpleQA.
On a sample of 100 questions from each dataset, we compared LLM judgments with human judgments.
We used responses from Qwen3-8B on TriviaQA, and responses from GPT-4.1 on SimpleQA.
The rate of agreement was 96/100 and 99/100, respectively.
Upon reviewing disagreements, we found that in most cases there was genuine room for interpretation.
For example, for the question \textit{``In The Living Daylights what did Carla keep in her cello case?''}, the target answer is \textit{``A machine gun''} while the model answer was \textit{``A gun''}. It is unclear whether the model answer has the desired specificity.
As another example, for the question \textit{``In Charles Dickens' ``Great Expectations'', who or what was Abel Magwitch?''}, the target answer is \textit{``Convict''} while the model answer was \textit{``A convict and the main benefactor of Pip''}. The model answer contains the answer but contains more information, and without external information, it is impossible to determine whether the model's answer is correct.
Overall, given the high agreement on unambiguously gradable questions, we deem it safe to adopt LLM-as-a-judge as a reliable evaluator in our experiments.
\subsection{\factscore Example}
\label{app:factscore-example}
\cref{tab:factscore-example} presents an example of atomic claim decomposition and verification with \factscore.
\begin{table}[t]
\caption{Example of \factscore atomic claim decomposition and verification.
}
\label{tab:factscore-example}
\begin{center}
\rowcolors{5}{gray!20}{white}
\begin{tabular}{p{0.8\linewidth}c}
\toprule
\textbf{Generation} & \\\midrule
\multicolumn{2}{p{0.9\linewidth}}{Kang Ji-hwan is a renowned South Korean actor and singer, best known for his role as the lead vocalist of the popular K-pop group BE:FIRST. Born on April 15, 2001, in Seoul, South Korea, Kang began his career in the entertainment industry at a young age, showcasing his talent through various music projects and performances. His distinctive voice and charismatic stage presence quickly earned him a loyal fanbase. Beyond his work in music, Kang has also ventured into acting, appearing in television dramas and variety shows, further solidifying his status as a multifaceted entertainer. With his dedication and natural talent, Kang Ji-hwan continues to make a significant impact in the K-pop and entertainment world.} \\
\hline\hline
\textbf{Extracted claim} & \textbf{Correct?} \\\midrule
Kang Ji-hwan is a South Korean actor. & Yes \\
Kang Ji-hwan is a South Korean singer. & No \\
Kang Ji-hwan is renowned. & Yes \\
Kang Ji-hwan is best known for his role as the lead vocalist of BE:FIRST. & No \\
BE:FIRST is a K-pop group. & Yes \\
BE:FIRST is a popular group. & Yes \\
Kang was born on April 15, 2001. & No \\
Kang was born in Seoul, South Korea. & Yes \\
Kang began his career in the entertainment industry at a young age. & No \\
Kang has showcased his talent through various music projects. & No \\
Kang has showcased his talent through various performances. & Yes \\
He has a distinctive voice. & No \\
He has a charismatic stage presence. & No \\
His distinctive voice quickly earned him a loyal fanbase. & No \\
His charismatic stage presence quickly earned him a loyal fanbase. & No \\
He quickly earned a loyal fanbase. & Yes \\
Kang has worked in music. & No \\
Kang has ventured into acting. & Yes \\
Kang has appeared in television dramas. & Yes \\
Kang has appeared in variety shows. & Yes \\
Kang is a multifaceted entertainer. & Yes \\
Kang's status as a multifaceted entertainer has been further solidified. & Yes \\
Kang Ji-hwan is dedicated. & Yes \\
Kang Ji-hwan has natural talent. & No \\
Kang Ji-hwan continues to make a significant impact in the K-pop world. & No \\
Kang Ji-hwan continues to make a significant impact in the entertainment world. & No \\
\bottomrule
\end{tabular}
% }
\end{center}
\end{table}
\subsection{NLI Model}
We use the NLI model DeBERTa-v3-base-mnli-fever-anli \citep{he2021debertadecodingenhancedbertdisentangled}.\footnote{\url{https://huggingface.co/MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli}}
\subsection{Significance Testing}
\label{app:significance-testing}
\paragraph{ECE.}
We subsample 10k subsets of size 0.9 times the original dataset, where sampling is done without replacement.
We construct an upper one-sided confidence interval with confidence level 0.95 for the tested method's ECE minus the best method's ECE and check whether the interval contains 0.
\paragraph{BS.}
As the Brier score is simply the mean squared error between confidences and truth values, it is well behaved and amenable to bootstrapping.
We sample 10k subsets with the same size as the original dataset, where sampling is done with replacement.
We construct an upper one-sided confidence interval with confidence level 0.95 for the tested method's BS minus the best method's BS and check whether the interval contains 0.
\paragraph{AUC.}
As AUC is a U-statistic, we use a one-sided DeLong test \citep{delong1988,xu2011delong} with confidence level 0.95.
\section{Methods}
\subsection{Pseudo-Beam Search}
\label{app:pseudo-beam-search}
\method uses beam search to generate distinct distractors.
However, beam search may not be available for API-access models.
Here, we describe an approximation of beam search with similar inference cost that can be implemented if top token probabilities are provided, as they are in many API-access models such as GPT and Gemini.
We first run one inference pass to generate the main answer, which also gives us the highest probability tokens (and their probabilities) at each token position in the main answer.
We consider the set of sequences that are a prefix of the main answer followed by a top token that is not the token in the main answer.
We sort the sequences by their probabilities (computed with the chain rule).
For each of the highest probability sequences, we present it to the LLM as a prefix to complete as an answer to the question (see Appendix~\ref{app:pseudo-beam-search-prompt} for the prompt).
This procedure chooses distinct prefixes with relatively high probability, shaping their completions to be distinct as well as plausible answers.
\subsection{Self-consistency}
\label{app:self-consistency}
Self-consistency estimates confidence in a main answer by sampling many answers and counting the proportion of answers that match the main answer \citep{xiong2024llmsexpressuncertaintyempirical, wang2023selfconsistencyimproveschainthought}.
Given a short-form question, a main answer $c_0$, and sampled answers $c_1, \ldots, c_K$, we compute the confidence of $c_0$ as
\begin{align}
f^{\rm SC}(c_0) = \frac{1}{K+1} \sum_{k=0}^K \displaystyle \bm{1}\{c_0 = c_k\}.
\label{eq:self-consistency}
\end{align}
Following \cite{kuhn2023semanticuncertaintylinguisticinvariances}, we use an NLI model to determine semantic equivalence.\footref{footnote:semantic-match}
For the long-form setting, we follow \cite{zhang2024luqlongtextuncertaintyquantification} and decompose a long-form response $r_0$ into claims $c_1, \ldots, c_n$ and measure entailment from other sampled responses $r_1, \ldots, r_K$. We compute the confidence $f^{\rm SC}(c_i)$ of a claim $c_i$ by \cref{eq:self-consistency} with the summand replaced with $P({\rm entail} \mid r_k, c_i)$.
Since the NLI task here involves a long-form text, we use the original LLM instead of an NLI model (see Appendix~\ref{app:prompts} for prompts).
\section{Results}
\subsection{\method with Verbalized Numerical Confidence}
\label{sec:verbalized-numerical-confidence}
\begin{table}[t]
\caption{TriviaQA results. Building on \cref{tab:triviaqa}, we report results when replacing \ptrue with verbalized numerical confidence. Text styling follows \cref{tab:triviaqa}.}
\label{tab:triviaqa-with-verbnumconf}
\begin{center}
\setlength{\tabcolsep}{4pt}
% \small
\resizebox{\linewidth}{!}{%
\renewcommand{\arraystretch}{1.3}
\begin{tabular}{lcccccccccccc}
\toprule
& \multicolumn{3}{c}{\textit{Qwen3-8B}} & \multicolumn{3}{c}{\textit{Qwen3-1.7B}} & \multicolumn{3}{c}{\textit{Llama-3.2-3B-Instruct}} & \multicolumn{3}{c}{\textit{Gemma-3-4B-IT}} \\ \cmidrule(lr){2-4}\cmidrule(lr){5-7}\cmidrule(lr){8-10}\cmidrule(lr){11-13}
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ \\
\midrule
VC$_{\text{numerical}}$ \citep{tian2023justaskcalibrationstrategies} & 0.310 & 0.304 & 0.791 & 0.364 & 0.342 & 0.697 & 0.226 & 0.226 & 0.773 & 0.311 & 0.321 & 0.731 \\
VC$_{\rm P(True)}$ \citep{kadavath2022languagemodelsmostlyknow} & 0.240 & 0.242 & 0.817 & 0.387 & 0.383 & 0.720 & 0.189 & 0.208 & 0.826 & 0.300 & 0.299 & 0.702 \\
$K$-VC \citep{tian2023finetuninglanguagemodelsfactuality} & 0.341 & 0.348 & 0.604 & 0.538 & 0.524 & 0.596 & 0.146 & 0.228 & 0.678 & 0.254 & 0.262 & 0.786 \\
MSP \citep{fadeeva2023lmpolygraphuncertaintyestimationlanguage} & 0.149 & 0.203 & 0.819 & 0.104 & 0.186 & 0.774 & 0.243 & 0.253 & 0.764 & 0.252 & 0.268 & 0.790 \\
SC-VC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.299 & 0.325 & 0.704 & 0.451 & 0.474 & 0.559 & 0.122 & 0.211 & 0.761 & 0.362 & 0.378 & 0.653 \\
SC \citep{xiong2024llmsexpressuncertaintyempirical} & 0.236 & 0.244 & 0.785 & 0.233 & 0.229 & 0.780 & 0.065 & 0.177 & 0.808 & 0.303 & 0.304 & 0.713 \\
NVC$_{\text{numerical}}$ & 0.232 & 0.225 & \underline{0.870} & \textbf{0.069} & \textbf{0.162} & 0.812 & 0.183 & 0.202 & 0.825 & 0.267 & 0.272 & 0.787 \\
\method{}$_{\text{numerical}}$ & 0.117 & 0.171 & 0.857 & 0.170 & 0.183 & \underline{0.825} & \textbf{0.037} & \underline{0.150} & \underline{0.857} & \textbf{0.101} & 0.202 & 0.791 \\
NVC$_{\rm P(True)}$ & 0.171 & 0.190 & 0.853 & 0.084 & \underline{0.164} & 0.806 & 0.168 & 0.192 & 0.845 & 0.218 & 0.236 & 0.791 \\
\method{}$_{\rm P(True)}$ & \textbf{0.097} & \textbf{0.155} & \textbf{0.879} & 0.177 & 0.179 & \textbf{0.835} & 0.044 & \textbf{0.148} & \textbf{0.864} & 0.121 & \textbf{0.191} & \textbf{0.817} \\
\bottomrule
\end{tabular}
}
\end{center}
\end{table}
\begin{table}[t]
\caption{\factscore results.
Building on \cref{tab:factscore}, we report results when replacing \ptrue with verbalized numerical confidence. Text styling follows \cref{tab:factscore}.
}
\label{tab:factscore-with-verbnumconf}
\begin{center}
\resizebox{\linewidth}{!}{%
\begin{tabular}{lcccccccccc}
\toprule
& \multicolumn{5}{c}{\textit{Qwen3-8B}} & \multicolumn{5}{c}{\textit{Gemma-3-4B-IT}} \\ \cmidrule(lr){2-6}\cmidrule(lr){7-11}
Method & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & $r$ $\scriptstyle\uparrow$ & $\rho$ $\scriptstyle\uparrow$ & ECE $\scriptstyle\downarrow$ & BS $\scriptstyle\downarrow$ & AUC $\scriptstyle\uparrow$ & $r$ $\scriptstyle\uparrow$ & $\rho$ $\scriptstyle\uparrow$ \\
\midrule
VC$_{\text{numerical}}$ \citep{tian2023justaskcalibrationstrategies} & 0.327 & 0.328 & 0.749 & 0.508 & 0.565 & 0.482 & 0.465 & 0.648 & 0.186 & 0.169 \\
VC$_{\rm P(True)}$ \citep{kadavath2022languagemodelsmostlyknow} & 0.433 & 0.431 & 0.625 & 0.073 & 0.122 & 0.527 & 0.527 & 0.683 & -0.081 & -0.129 \\
SC \citep{zhang2024luqlongtextuncertaintyquantification} & 0.162 & 0.226 & 0.771 & 0.468 & 0.494 & 0.197 & 0.233 & 0.787 & 0.629 & 0.607 \\
NVC$_{\text{numerical}}$ & 0.181 & 0.259 & 0.693 & 0.555 & 0.566 & \textbf{0.090} & 0.208 & 0.744 & 0.683 & 0.698 \\
\method{}$_{\text{numerical}}$ & \textbf{0.045} & \textbf{0.193} & \textbf{0.779} & \textbf{0.559} & \textbf{0.574} & 0.120 & \textbf{0.188} & \textbf{0.808} & 0.713 & 0.696 \\
NVC$_{\rm P(True)}$ & 0.191 & 0.263 & 0.681 & 0.444 & 0.443 & 0.123 & 0.230 & 0.726 & 0.695 & 0.704 \\
\method{}$_{\rm P(True)}$ & 0.076 & 0.202 & 0.767 & 0.518 & 0.538 & 0.172 & \underline{0.210} & 0.793 & \textbf{0.724} & \textbf{0.712} \\
\bottomrule
\end{tabular}
}
\end{center}
\end{table}
In \cref{tab:triviaqa-with-verbnumconf,tab:factscore-with-verbnumconf}, we expand on \cref{tab:triviaqa,tab:factscore} and report results on TriviaQA and biography generation when replacing \ptrue with verbalized numerical confidence (see Appendix~\ref{app:prompts} for prompts). For SimpleQA, the black-box variants of our methods in \cref{tab:simpleqa} use verbalized numerical confidence rather than \ptrue.
We find that the calibration benefits of \method generalize to the format of verbalized numerical confidence.
For example, \method{}$_{\text{numerical}}$ compared to VC$_{\text{numerical}}$ lowers ECE by 0.196 on average on TriviaQA.
\method{}$_{\text{numerical}}$ outperforms the stronger baselines as well, e.g. a lower ECE than SC by 0.103 on average on TriviaQA.
In the long-form setting of biography generation (\cref{tab:factscore-with-verbnumconf}), \method{}$_{\text{numerical}}$ continues to achieve strong calibration, even outperforming \method{}$_{\rm P(True)}$ in many cases, e.g. 0.045 vs. 0.076 ECE with Qwen3-8B.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
% \def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{59}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ahdritz et~al.(2024)Ahdritz, Qin, Vyas, Barak, and Edelman]{ahdritz2024distinguishingknowableunknowablelanguage}
Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin~L. Edelman.
\newblock Distinguishing the knowable from the unknowable with language models.
\newblock In \emph{Proceedings of the 41st International Conference on Machine Learning}, ICML'24. JMLR.org, 2024.
\bibitem[Brier(1950)]{Brier1950VERIFICATIONOF}
Glenn~W. Brier.
\newblock Verification of forecasts expressed in terms of probability.
\newblock \emph{Monthly Weather Review}, 78:\penalty0 1--3, 1950.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:122906757}.
\bibitem[Bruck \& Ceci(1999)Bruck and Ceci]{bruck1999thesuggestibilityofchildrensmemory}
Maggie Bruck and Stephen Ceci.
\newblock The suggestibility of children's memory.
\newblock \emph{Annual review of psychology}, 50:\penalty0 419--39, 02 1999.
\newblock \doi{10.1146/annurev.psych.50.1.419}.
\bibitem[Chhikara(2025)]{chhikara2025mindconfidencegapoverconfidence}
Prateek Chhikara.
\newblock Mind the confidence gap: Overconfidence, calibration, and distractor effects in large language models, 2025.
\newblock URL \url{https://arxiv.org/abs/2502.11028}.
\bibitem[Comanici et~al.(2025)Comanici, Bieber, Schaekermann, Pasupat, Sachdeva, Dhillon, Blistein, Ram, Zhang, Rosen, et~al.]{comanici2025gemini}
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et~al.
\newblock Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025.
\newblock URL \url{https://arxiv.org/abs/2507.06261}.
\bibitem[DeLong et~al.(1988)DeLong, DeLong, and Clarke-Pearson]{delong1988}
Elizabeth~R. DeLong, David~M. DeLong, and Daniel~L. Clarke-Pearson.
\newblock Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach.
\newblock \emph{Biometrics}, 44\penalty0 (3):\penalty0 837--845, 1988.
\newblock ISSN 0006341X, 15410420.
\newblock URL \url{http://www.jstor.org/stable/2531595}.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, et~al.]{grattafiori2024llama3herdmodels}
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et~al.
\newblock The llama 3 herd of models.
\newblock \emph{CoRR}, 2024.
\bibitem[Fadeeva et~al.(2023)Fadeeva, Vashurin, Tsvigun, Vazhentsev, Petrakov, Fedyanin, Vasilev, Goncharova, Panchenko, Panov, et~al.]{fadeeva2023lmpolygraphuncertaintyestimationlanguage}
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et~al.
\newblock Lm-polygraph: Uncertainty estimation for language models.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, pp.\ 446--461, 2023.
\bibitem[Farquhar et~al.(2024)Farquhar, Kossen, Kuhn, and Gal]{Farquhar2024DetectingHI}
Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal.
\newblock Detecting hallucinations in large language models using semantic entropy.
\newblock \emph{Nature}, 630:\penalty0 625 -- 630, 2024.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:270615909}.
\bibitem[Feng et~al.(2024)Feng, Shi, Wang, Ding, Balachandran, and Tsvetkov]{feng2024donthallucinateabstainidentifying}
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov.
\newblock Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 14664--14690, 2024.
\bibitem[Feng et~al.(2025)Feng, Zhou, Lin, and Roth]{feng2025birdtrustworthybayesianinference}
Yu~Feng, Ben Zhou, Weidong Lin, and Dan Roth.
\newblock Bird: A trustworthy bayesian inference framework for large language models.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\bibitem[Gekhman et~al.(2025)Gekhman, David, Orgad, Ofek, Belinkov, Szpektor, Herzig, and Reichart]{gekhman2025insideouthiddenfactualknowledge}
Zorik Gekhman, Eyal~Ben David, Hadas Orgad, Eran Ofek, Yonatan Belinkov, Idan Szpektor, Jonathan Herzig, and Roi Reichart.
\newblock Inside-out: Hidden factual knowledge in llms, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.15299}.
\bibitem[Geng et~al.(2024)Geng, Cai, Wang, Koeppl, Nakov, and Gurevych]{geng2024surveyconfidenceestimationcalibration}
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych.
\newblock A survey of confidence estimation and calibration in large language models.
\newblock In \emph{Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, pp.\ 6577--6595, 2024.
\bibitem[Hanley \& McNeil(1982)Hanley and McNeil]{hanley1982auroc}
James~A. Hanley and Barbara~J. McNeil.
\newblock The meaning and use of the area under a receiver operating characteristic (roc) curve.
\newblock \emph{Radiology}, 143:\penalty0 29--36, 05 1982.
\newblock \doi{10.1148/radiology.143.1.7063747}.
\bibitem[He et~al.(2021)He, Liu, Gao, and Chen]{he2021debertadecodingenhancedbertdisentangled}
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
\newblock Deberta: Decoding-enhanced bert with disentangled attention.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Hofweber et~al.(2024)Hofweber, Hase, Stengel-Eskin, and Bansal]{hofweber2024language}
Thomas Hofweber, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal.
\newblock Are language models rational? the case of coherence norms and belief revision.
\newblock \emph{arXiv preprint arXiv:2406.03442}, 2024.
\bibitem[Hou et~al.(2025)Hou, Zhang, Andreas, and Chang]{hou2025probabilisticframeworkllmhallucination}
Bairu Hou, Yang Zhang, Jacob Andreas, and Shiyu Chang.
\newblock A probabilistic framework for llm hallucination detection via belief tree propagation.
\newblock In \emph{Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, pp.\ 3076--3099, 2025.
\bibitem[Houlsby et~al.(2011)Houlsby, Huszár, Ghahramani, and Lengyel]{houlsby2011bayesianactivelearningclassification}
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.
\newblock Bayesian active learning for classification and preference learning, 2011.
\newblock URL \url{https://arxiv.org/abs/1112.5745}.
\bibitem[Jiang et~al.(2021)Jiang, Araki, Ding, and Neubig]{jiang2021knowlanguagemodelsknow}
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig.
\newblock How can we know when language models know? on the calibration of language models for question answering.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 9:\penalty0 962--977, 2021.
\bibitem[Joshi et~al.(2017)Joshi, Choi, Weld, and Zettlemoyer]{joshi2017triviaqalargescaledistantly}
Mandar Joshi, Eunsol Choi, Daniel~S Weld, and Luke Zettlemoyer.
\newblock Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1601--1611, 2017.
\bibitem[Jung et~al.(2022)Jung, Qin, Welleck, Brahman, Bhagavatula, Le~Bras, and Choi]{jung2022maieuticpromptinglogicallyconsistent}
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le~Bras, and Yejin Choi.
\newblock Maieutic prompting: Logically consistent reasoning with recursive explanations.
\newblock \emph{CoRR}, 2022.
\bibitem[Kadavath et~al.(2022)Kadavath, Conerly, Askell, Henighan, Drain, Perez, Schiefer, Hatfield-Dodds, DasSarma, Tran-Johnson, et~al.]{kadavath2022languagemodelsmostlyknow}
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et~al.
\newblock Language models (mostly) know what they know.
\newblock \emph{CoRR}, 2022.
\bibitem[Kuhn et~al.(2023)Kuhn, Gal, and Farquhar]{kuhn2023semanticuncertaintylinguisticinvariances}
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.
\newblock Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\bibitem[Laban et~al.(2024)Laban, Murakhovs'ka, Xiong, and Wu]{laban2024surechallengingllmsleads}
Philippe Laban, Lidiya Murakhovs'ka, Caiming Xiong, and Chien-Sheng Wu.
\newblock Are you sure? challenging llms leads to performance drops in the flipflop experiment, 2024.
\newblock URL \url{https://arxiv.org/abs/2311.08596}.
\bibitem[Leng et~al.(2025)Leng, Huang, Zhu, and Huang]{leng2025tamingoverconfidencellmsreward}
Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang.
\newblock Taming overconfidence in llms: Reward calibration in rlhf.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\bibitem[Li et~al.(2024)Li, Shrivastava, Li, Hashimoto, and Liang]{li2023benchmarkingimprovinggeneratorvalidatorconsistency}
Xiang~Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang.
\newblock Benchmarking and improving generator-validator consistency of language models.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Lin et~al.(2022)Lin, Hilton, and Evans]{lin2022teachingmodelsexpressuncertainty}
Stephanie Lin, Jacob Hilton, and Owain Evans.
\newblock Teaching models to express their uncertainty in words.
\newblock \emph{Transactions on Machine Learning Research}, 2022.
\bibitem[Lin et~al.(2024)Lin, Trivedi, and Sun]{lin2024generatingconfidenceuncertaintyquantification}
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun.
\newblock Generating with confidence: Uncertainty quantification for black-box large language models.
\newblock \emph{Transactions on Machine Learning Research}, 2024.
\bibitem[Manakul et~al.(2023)Manakul, Liusie, and Gales]{manakul2023selfcheckgptzeroresourceblackboxhallucination}
Potsawee Manakul, Adian Liusie, and Mark Gales.
\newblock {S}elf{C}heck{GPT}: Zero-resource black-box hallucination detection for generative large language models.
\newblock In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 9004--9017, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.emnlp-main.557}.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.557/}.
\bibitem[Min et~al.(2023)Min, Krishna, Lyu, Lewis, Yih, Koh, Iyyer, Zettlemoyer, and Hajishirzi]{min2023factscorefinegrainedatomicevaluation}
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.
\newblock Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 12076--12100, 2023.
\bibitem[Naeini et~al.(2015)Naeini, Cooper, and Hauskrecht]{Naeini2015ObtainingWC}
Mahdi~Pakdaman Naeini, Gregory~F. Cooper, and Milos Hauskrecht.
\newblock Obtaining well calibrated probabilities using bayesian binning.
\newblock \emph{Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence}, 2015:\penalty0 2901--2907, 2015.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:6292807}.
\bibitem[Nafar et~al.(2025)Nafar, Venable, Cui, and Kordjamshidi]{nafar2025extractingprobabilisticknowledgelarge}
Aliakbar Nafar, Kristen~Brent Venable, Zijun Cui, and Parisa Kordjamshidi.
\newblock Extracting probabilistic knowledge from large language models for bayesian network parameterization, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.15918}.
\bibitem[OpenAI(2023)]{openai2023gpt4}
OpenAI.
\newblock Gpt-4 technical report, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.08774}.
\bibitem[OpenAI(2025)]{openai2025gpt41}
OpenAI.
\newblock Introducing gpt-4.1 in the api.
\newblock \url{https://openai.com/index/gpt-4-1/}, April 2025.
\newblock Accessed: 2025-09-23.
\bibitem[Piantadosi \& Hill(2022)Piantadosi and Hill]{piantadosi2022meaningreferencelargelanguage}
Steven~T. Piantadosi and Felix Hill.
\newblock Meaning without reference in large language models, 2022.
\newblock URL \url{https://arxiv.org/abs/2208.02957}.
\bibitem[Rodriguez et~al.(2025)Rodriguez, Ding, Erk, and Durrett]{rodriguez2025rankalignrankingviewgeneratorvalidator}
Juan~Diego Rodriguez, Wenxuan Ding, Katrin Erk, and Greg Durrett.
\newblock Rankalign: A ranking view of the generator-validator gap in large language models, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.11381}.
\bibitem[Shorinwa et~al.(2025)Shorinwa, Mei, Lidard, Ren, and Majumdar]{shorinwa2025surveyuncertaintyquantificationlarge}
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen~Z Ren, and Anirudha Majumdar.
\newblock A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions.
\newblock \emph{ACM Computing Surveys}, 2025.
\bibitem[Steyvers et~al.(2025)Steyvers, Tejeda, Kumar, Belem, Karny, Hu, Mayer, and Smyth]{Steyvers_2025}
Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas~W. Mayer, and Padhraic Smyth.
\newblock What large language models know and what people think they know.
\newblock \emph{Nature Machine Intelligence}, 7\penalty0 (2):\penalty0 221–231, January 2025.
\newblock ISSN 2522-5839.
\newblock \doi{10.1038/s42256-024-00976-7}.
\newblock URL \url{http://dx.doi.org/10.1038/s42256-024-00976-7}.
\bibitem[Sun et~al.(2025)Sun, Li, Wang, and Goette]{sun2025largelanguagemodelsoverconfident}
Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette.
\newblock Large language models are overconfident and amplify human bias, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.02151}.
\bibitem[Sun \& Xu(2014)Sun and Xu]{xu2011delong}
Xu~Sun and Weichao Xu.
\newblock Fast implementation of delong’s algorithm for comparing the areas under correlated receiver operating characteristic curves.
\newblock \emph{IEEE Signal Processing Letters}, 21\penalty0 (11):\penalty0 1389--1393, 2014.
\newblock \doi{10.1109/LSP.2014.2337313}.
\bibitem[Sutskever et~al.(2014)Sutskever, Vinyals, and Le]{sutskever2014sequencesequencelearningneural}
Ilya Sutskever, Oriol Vinyals, and Quoc~V Le.
\newblock Sequence to sequence learning with neural networks.
\newblock \emph{Advances in neural information processing systems}, 27, 2014.
\bibitem[Taubenfeld et~al.(2025)Taubenfeld, Sheffer, Ofek, Feder, Goldstein, Gekhman, and Yona]{Taubenfeld_2025}
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona.
\newblock Confidence improves self-consistency in llms.
\newblock In \emph{Findings of the Association for Computational Linguistics: ACL 2025}, pp.\ 20090–20111. Association for Computational Linguistics, 2025.
\newblock \doi{10.18653/v1/2025.findings-acl.1030}.
\newblock URL \url{http://dx.doi.org/10.18653/v1/2025.findings-acl.1030}.
\bibitem[Team et~al.(2025)Team, Kamath, Ferret, Pathak, Vieillard, Merhej, Perrin, Matejovicova, Ramé, Rivière, et~al.]{gemmateam2025gemma3technicalreport}
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et~al.
\newblock Gemma 3 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2503.19786}.
\bibitem[Tian et~al.(2023)Tian, Mitchell, Zhou, Sharma, Rafailov, Yao, Finn, and Manning]{tian2023justaskcalibrationstrategies}
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher~D Manning.
\newblock Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
\newblock In \emph{The 2023 Conference on Empirical Methods in Natural Language Processing}, 2023.
\bibitem[Tian et~al.(2024)Tian, Mitchell, Yao, Manning, and Finn]{tian2023finetuninglanguagemodelsfactuality}
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher~D Manning, and Chelsea Finn.
\newblock Fine-tuning language models for factuality.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Walton(1996)]{walton1996arguments}
D.N. Walton.
\newblock \emph{Arguments from Ignorance}.
\newblock Arguments from Ignorance. Pennsylvania State University Press, 1996.
\newblock ISBN 9780271014746.
\newblock URL \url{https://books.google.com/books?id=peTWAAAAMAAJ}.
\bibitem[Wang et~al.(2023)Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery, and Zhou]{wang2023selfconsistencyimproveschainthought}
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc~V Le, Ed~H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
\newblock Self-consistency improves chain of thought reasoning in language models.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\bibitem[Wei et~al.(2024)Wei, Karina, Chung, Jiao, Papay, Glaese, Schulman, and Fedus]{wei2024measuringshortformfactualitylarge}
Jason Wei, Nguyen Karina, Hyung~Won Chung, Yunxin~Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus.
\newblock Measuring short-form factuality in large language models, 2024.
\newblock URL \url{https://arxiv.org/abs/2411.04368}.
\bibitem[West et~al.(2024)West, Lu, Dziri, Brahman, Li, Hwang, Jiang, Fisher, Ravichander, Chandu, et~al.]{west2023generativeaiparadoxwhat}
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena~D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et~al.
\newblock The generative ai paradox:“what it can create, it may not understand”.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Xia et~al.(2024)Xia, Lu, and Eisner]{xia2024letsthinkvarbyvarlarge}
Shepard Xia, Brian Lu, and Jason Eisner.
\newblock Let's think var-by-var: Large language models enable ad hoc probabilistic reasoning, 2024.
\newblock URL \url{https://arxiv.org/abs/2412.02081}.
\bibitem[Xiong et~al.(2024)Xiong, Hu, Lu, LI, Fu, He, and Hooi]{xiong2024llmsexpressuncertaintyempirical}
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi.
\newblock Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Xu et~al.(2025)Xu, Wen, Han, Wolfe, Wang, and Howe]{xu2025languagemodelsmirrorhuman}
Chenjun Xu, Bingbing Wen, Bin Han, Robert Wolfe, Lucy~Lu Wang, and Bill Howe.
\newblock Do language models mirror human confidence? exploring psychological insights to address overconfidence in {LLM}s.
\newblock In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad~Taher Pilehvar (eds.), \emph{Findings of the Association for Computational Linguistics: ACL 2025}, pp.\ 25655--25672, Vienna, Austria, July 2025. Association for Computational Linguistics.
\newblock ISBN 979-8-89176-256-5.
\newblock \doi{10.18653/v1/2025.findings-acl.1316}.
\newblock URL \url{https://aclanthology.org/2025.findings-acl.1316/}.
\bibitem[Yadkori et~al.(2024)Yadkori, Kuzborskij, Gy{\"o}rgy, and Szepesvari]{yadkori2024believebelievellm}
Yasin~Abbasi Yadkori, Ilja Kuzborskij, Andr{\'a}s Gy{\"o}rgy, and Csaba Szepesvari.
\newblock To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 58077--58117, 2024.
\bibitem[Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, et~al.]{yang2025qwen3technicalreport}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et~al.
\newblock Qwen3 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.09388}.
\bibitem[Zhang et~al.(2024{\natexlab{a}})Zhang, Liu, Basaldella, and Collier]{zhang2024luqlongtextuncertaintyquantification}
Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier.
\newblock {LUQ}: Long-text uncertainty quantification for {LLM}s.
\newblock In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 5244--5262, Miami, Florida, USA, November 2024{\natexlab{a}}. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.299}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.299/}.
\bibitem[Zhang et~al.(2025)Zhang, Zhu, Li, Collier, and Vlachos]{zhang2025reinforcementlearningbetterverbalized}
Caiqi Zhang, Xiaochen Zhu, Chengzu Li, Nigel Collier, and Andreas Vlachos.
\newblock Reinforcement learning for better verbalized confidence in long-form generation, 2025.
\newblock URL \url{https://arxiv.org/abs/2505.23912}.
\bibitem[Zhang et~al.(2024{\natexlab{b}})Zhang, Huang, Shi, Guo, Peng, Yan, Zhou, and Qiu]{wang2024calibratingverbalizedprobabilitieslarge}
Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu.
\newblock Calibrating the confidence of large language models by eliciting fidelity.
\newblock In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 2959--2979, Miami, Florida, USA, November 2024{\natexlab{b}}. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.173}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.173/}.
\bibitem[Zhu \& Griffiths(2024)Zhu and Griffiths]{zhu2025incoherentprobabilityjudgmentslarge}
Jian-Qiao Zhu and Tom Griffiths.
\newblock Incoherent probability judgments in large language models.
\newblock In \emph{Proceedings of the Annual Meeting of the Cognitive Science Society}, volume~46, 2024.
\bibitem[Zhu et~al.(2019)Zhu, Sanborn, and Chater]{zhu2018bayesianinferencecausesincoherence}
Jianqiao Zhu, Adam Sanborn, and Nicholas Chater.
\newblock Bayesian inference causes incoherence in human probability judgments.
\newblock In \emph{Proceedings of the Annual Meeting of the Cognitive Science Society}, volume~41, 2019.
\end{thebibliography}