Title: Think Twice Before Assure: Confidence Estimation for Large Language Models through Reflection on Multiple Answers

Institute: National University of Singapore, University of Science and Technology of China, Meta AI

Authors: Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, Tat-Seng Chua

Arxiv Link: https://arxiv.org/abs/2403.09972

Date: 2024.3.15

Abstract

Assessing the credibility of the output is a critical task for large language models (LLMs), especially black-box ones. Because LLMs are overconfident in the incorrect answers they generate, confidence estimation for LLMs is often inaccurate. Existing approaches to the overconfidence problem share a clear limitation: they consider the confidence of only a single answer generated by the LLM. The authors address this by proposing a new method that assesses the credibility of multiple candidate answers together, thereby mitigating overconfidence in incorrect answers.

Introduction

The core of confidence estimation is calibration — ensuring that the confidence level matches the actual accuracy of the answer.

The key to calibrating a black-box LLM is solving the overconfidence problem. LLMs are inherently biased toward trusting the answers they themselves generate, which makes it hard for a model to judge honestly whether its own answer is correct; it tends to assign those answers very high confidence scores.

Two research paradigms exist today:

1. The first (self-consistency) attributes overconfidence in part to the contextual bias between the question, or prompt, and the answer. Researchers therefore try prompt ensembling, constructing a variety of prompt templates and rephrasings of the question.

2. The second (verbalized methods) focuses on the LLM’s self-assessment, with techniques such as instructing the LLM to judge the truthfulness of an answer or applying Chain-of-Thought (CoT) reasoning.

The limitation of this line of work is that these methods typically consider only a single target answer generated by the LLM. Since the model is overconfident in its own answers, they may still be biased toward incorrect ones.

The paper gets around this with a new multi-answer assessment paradigm in which the LLM weighs several candidate answers at once, which improves confidence calibration.

Problem Definition

Given an input consisting of a question $q$ and a prompt $p$ — the latter comprising an instruction and optional in-context examples — the LLM generates an answer $a$. Confidence estimation then aims to recover the LLM’s confidence in $a$, expressed as a score $c \in R$. Writing the confidence estimation strategy as $CE(·)$, the process is: $$ a = LLM(p(q)), \ c = CE(LLM(·), p(q), a). $$

Prior Approaches

The question thus becomes how to design the confidence estimation function $CE(·)$.

The obvious approach is to estimate the confidence score from the LLM’s output probabilities, written as $c = Pr(LLM(·), p(q), a)$. The paper assumes a black-box setting, though, where the model’s internal states are not visible, which rules this out.

Two approaches currently exist for the black-box setting:

• self-consistency, shown as subfigure 1 above
• verbalized methods, shown as subfigure 2 above

Self-consistency essentially asks the same question several times with different phrasings and looks at the proportion of correct answers. Formally: $$ \begin{aligned}&c=\frac{\sum_{i=1}^D \mathbb1(a_i=a)}D,\&\mathrm{where~}a_i=LLM(p(q)).\end{aligned} $$ where $D>1$.

Verbalized methods simply ask the LLM to state several candidate answers along with their corresponding probabilities. $$ [{a_1,c_1},…{a_K,c_K}]=LLM(p^b(q)). $$

Both, however, consider only a single target answer generated by the LLM, and since the model is overconfident in its own answers, both may still be biased toward incorrect ones.

The Proposed Method

The authors argue that if the LLM thoroughly compares the credibility of more candidate answers, a high confidence in $a$ may be tempered when the LLM has a strong sense that another answer is correct, which lowers the risk of overconfidence.

They therefore propose TTA, a multi-answer assessment paradigm, in two steps.

Step 1: Reflect and justify

The LLM first reflects on the credibility of each answer $a^q_i$ and is asked to find evidence $e_i$ justifying $a^q_i$ as a correct answer to $q$. $$ e_i = LLM(p^e(q, a^q_i)), i \in {1, …, N} $$ The rationale for this step is that $p^e$ guides the LLM to derive a justification from $q$ and $a^q_i$, avoiding the bias the LLM carries in generating $a$ from $p(q)$.

Step 2: Joint confidence estimation

Once evidence $e_i$ has been gathered for every $a^q_i$, a Top-$K$ verbalized method integrates the $e_i$ and yields the probability of answer $a$. $$ c = Pr(LLM(\cdot), p^v(q, [e_1, …, e_N]), a) $$ The Top-$K$ verbalized method is chosen here because it produces $K$ candidate answers together with their probabilities in a single response.

Here $K$ is set to the number of answers $N$.

The whole process can be formalized as: $$ c = Pr(LLM(\cdot), p^v(q, [e_1, …, e_N]), a), \ \notag \text{where } e_i = LLM(p^e(q, a^q_i)), i \in {1, …, N}. $$ The pipeline is shown as subfigure 3 above.

Experiments

Datasets & Models

The paper runs experiments on three tasks across six datasets.

Sentiment analysis (SA) uses IMDB and Flipkart; natural language inference (NLI) uses SNLI and HANS; commonsense question answering (CQA) uses CommonsenseQA and PIQA.

The models are OpenAI’s GPT-3.5 and GPT-4 and ZhipuAI’s GLM-4.

Baseline

Self-cons, CoT-cons, Top-K Verb, Hybrid, Self-detect, CAPE, P(True).

Evaluation

AUROC and PRAUC, both ranging from 0 to 1.

Result

Results of the compared methods on GPT-3.5

1. Answer accuracy is better separated by the confidence scores, yielding better calibration.
2. TTA has strong application potential in selective prediction scenarios.
3. LLMs’ parameter sensitivity varies across different target answers.
4. TTA achieves better calibration performance than the other methods.