Reflection-Window Decoding: Text Generation with Selective Refinement
Extracted source text ยท arXiv 2502.03678
\documentclass[hidelinks]{article}
% Use the following line for the initial blind version submitted for review:
\usepackage[accepted]{icml2025}
% If accepted, instead use the following line for the camera-ready submission:
% \usepackage[accepted]{icml2025}
% --> prefix and global option
\input{file/prefix.tex}
% \interfootnotelinepenalty=10000
\newcommand{\cgy}[1]{{\textcolor{red}{\it cgy: #1}}}
\newcommand{\xs}[1]{{\textcolor{brown}{\it xs: #1}}}
\setlist[itemize]{topsep=0pt}
% \usepackage{algorithm} % in conflict with iclr style
\usepackage{tabularx}
\usepackage{graphicx}
\usepackage{soul}
\usepackage[normalem]{ulem}
\usepackage{xcolor}
\usepackage{colortbl}
\usepackage[most]{tcolorbox}
\definecolor{highlight}{HTML}{81CE6D}
\definecolor{mycolor}{HTML}{2650CC}
\definecolor{second}{HTML}{666666}
\definecolor{rightcolor}{HTML}{28a745}
\definecolor{wrongcolor}{HTML}{dc3545}
\definecolor{Gray}{gray}{0.95}
\definecolor{Cyan}{rgb}{0.88,1,1}
\definecolor{LightCyan}{rgb}{0.92,1,1}
\definecolor{DarkCyan}{rgb}{0.82,1,1}
\definecolor{tabhighlight}{HTML}{e5e5e5}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{stfloats} % Improved behavior for double-column floats
\definecolor{questiongray}{RGB}{211, 211, 211}
\definecolor{reflectiongreen}{RGB}{144, 238, 144}
\definecolor{greedyblue}{RGB}{173, 216, 230}
\definecolor{beamyellow}{RGB}{255, 255, 180}
\tcbset{
decoding/.style={
enhanced,
rounded corners=all,
arc=8pt,
colback=white,
colframe=black,
fonttitle=\bfseries\large,
left=8pt,
right=8pt,
top=8pt,
bottom=8pt,
width=0.85\textwidth,
center
}
}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Reflection-Window Decoding: Text Generation with Selective Refinement}
\begin{document}
% --> title
\twocolumn[
\icmltitle{Reflection-Window Decoding: Text Generation with Selective Refinement}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2025
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Zeyu Tang}{equal,cmu}
\icmlauthor{Zhenhao Chen}{equal,mbzuai}
\icmlauthor{Xiangchen Song}{cmu}
\icmlauthor{Loka Li}{mbzuai}
\icmlauthor{Yunlong Deng}{mbzuai}
\icmlauthor{Yifan Shen}{mbzuai}
\icmlauthor{Guangyi Chen}{cmu,mbzuai}
\icmlauthor{Peter Spirtes}{cmu}
\icmlauthor{Kun Zhang}{cmu,mbzuai}
\end{icmlauthorlist}
\icmlaffiliation{cmu}{Carnegie Mellon University}
\icmlaffiliation{mbzuai}{Mohamed bin Zayed University of Artificial Intelligence}
\icmlcorrespondingauthor{Zeyu Tang}{\href{mailto:zeyutang@cmu.edu}{\texttt{zeyutang@cmu.edu}}}
\icmlcorrespondingauthor{Zhenhao Chen}{\href{mailto:zhenhao.chen@mbzuai.ac.ae}{\texttt{zhenhao.chen@mbzuai.ac.ae}}}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Reflection Window, Selective Refinement, Autoregressive, Decoding, Large Language Model}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
% \printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
% \vspace{-4ex}
\begin{abstract}
\looseness=-1
The autoregressive decoding for text generation in large language models (LLMs), while widely used, is inherently suboptimal due to the lack of a built-in mechanism to perform refinement and/or correction of the generated content. In this paper, we consider optimality in terms of the joint probability over the generated response, when jointly considering all tokens at the same time. We theoretically characterize the potential deviation of the autoregressively generated response from its globally optimal counterpart that is of the same length. Our analysis suggests that we need to be cautious when noticeable uncertainty arises during text generation, which may signal the sub-optimality of the generation history. To address the pitfall of autoregressive decoding for text generation, we propose an approach that incorporates a sliding reflection window and a pausing criterion, such that refinement and generation can be carried out interchangeably as the decoding proceeds. Our selective refinement framework strikes a balance between efficiency and optimality, and our extensive experimental results demonstrate the effectiveness of our approach.
\end{abstract}
\input{file/main/introduction}
% Motivation and intuition
\input{file/main/motivation.tex}
% Theory
\input{file/main/theory.tex}
% Semi synthetic
\input{file/main/synthetic.tex}
% Experiment section
\input{file/main/experiments.tex}
\input{file/main/conclusion.tex}
% \clearpage
\input{file/main/broader_impact.tex}
{\bibliography{references}}
\bibliographystyle{icml2025}
\input{supp}
\end{document}
\usepackage{graphicx}
\usepackage[normalem]{ulem} % normalem prevents \emph from being underlined
\usepackage{float}
\usepackage{microtype}
% \usepackage{subfigure} % outdated, use subcaption instead
\usepackage{subcaption}
\usepackage{stackengine}
\usepackage{verbatim} % comment things
\usepackage{amsmath, amssymb, bm, bbm, mathtools}
\usepackage{booktabs} % for professional tables
\usepackage{color}
\usepackage[english]{babel}
\usepackage{times}
\usepackage{nicefrac}
\usepackage{authblk} % author affiliation
\usepackage{xurl} % long url
% Change subfigure numbering style to (a), (b), ...
\renewcommand\thesubfigure{(\alph{subfigure})}
% Apply the format to the sub-figure caption
\captionsetup[subfigure]{labelformat=simple}
% Change equation numbering style to (1), (2), ...
\renewcommand{\theequation}{\arabic{equation}}
% Define new command for equation references
\newcommand{\eqnref}[1]{%
\ifnum\pdfstrcmp{(}{\unexpanded\expandafter{\@car#1\@nil}}=0
Equation~\ref{#1}%
\else
Equation~(\ref{#1})%
\fi
}
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\usepackage[linesnumbered,lined,ruled]{algorithm2e}
\SetKwComment{Comment}{$\triangleright$ }{}
% example https://www.overleaf.com/learn/latex/Algorithms#The_algorithm2e_package
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
% \SetKwFor{ForEach}{ForEach}{Do}{End}
\SetKwFor{ForEach}{ForEach}{Do}{End}
\SetKwFor{While}{While}{Do}{}
% \SetKwIF{If}{ElseIf}{Else}{If}{Then}{Else If}{Else}{End}
\SetKwIF{If}{ElseIf}{Else}{If}{Then}{Else If}{Else}{}
\SetKwFunction{Return}{\textbf{Return}}
\SetInd{1.5ex}{1.5ex} % Set the indentation width
% --> toc for appendix only
\usepackage[header,page,toc]{appendix}
\usepackage{titletoc}
\definecolor{darkblue}{rgb}{0,0.08,0.45}
\definecolor{DarkBlue}{HTML}{2A4398}
\usepackage[
colorlinks=true,
citecolor=DarkBlue, % citation links
linkcolor=black, % section headings, ToC, equations
urlcolor=DarkBlue, % hyperlinks
filecolor=black % file links, if any
]{hyperref} % should only appear after titletoc
\usepackage{amsthm}
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
% unnumbered in appendix
\newtheorem*{thm}{Theorem}
\newtheorem*{prop}{Proposition}
\newtheorem*{lma}{Lemma}
\newtheorem*{coro}{Corollary}
\theoremstyle{definition}
% \newtheorem{strategy}{Strategy}[section]
% \renewcommand{\thestrategy}{\Roman{strategy}}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{question}[theorem]{Question}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% unnumbered in appendix
\newtheorem*{asmp}{Assumption}
\usepackage{enumitem}
\usepackage[globalcitecopy]{bibunits}
\usepackage{chngcntr}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=normalsize]{todonotes}
\usepackage[disable,textwidth=1.3in,textsize=footnotesize]{todonotes}
% % if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
\usepackage{xcolor}
\usepackage{soul} % after xcolor
% Define multiple highlight colors
\definecolor{yellowhighlight}{RGB}{255, 255, 0}
\definecolor{pinkhighlight}{RGB}{255, 192, 203}
\definecolor{greenhighlight}{RGB}{144, 238, 144}
% Create custom highlight commands
\newcommand{\hlyellow}[1]{{\sethlcolor{yellowhighlight}\hl{#1}}}
\newcommand{\hlpink}[1]{{\sethlcolor{pinkhighlight}\hl{#1}}}
\newcommand{\hlgreen}[1]{{\sethlcolor{greenhighlight}\hl{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%% ---------- New Commands ---------- %%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{multirow}
\usepackage{makecell} % use \multirowcell to handle auto line break in a cell
\usepackage{array}
% Define new column types for left and center alignment with column width as fraction of \textwidth
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1\textwidth-2\tabcolsep}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1\textwidth-2\tabcolsep}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1\textwidth-2\tabcolsep}}
% \newcolumntype{L}[1]{>{\raggedright\arraybackslash}m{#1}}
% \newcolumntype{C}[1]{>{\centering\arraybackslash}m{#1}}
% \newcommand{\ubar}[1]{\text{\b{$#1$}}}
\newcommand{\ubar}[1]{\underline{#1}}
\newcommand{\rulesep}{\unskip\ \vrule\ }
\makeatletter
\newcommand*{\indep}{
\mathbin{
\mathpalette{\@indep}{}
}
}
\newcommand*{\nindep}{
\mathbin{% The final symbol is a binary math operator
\mathpalette{\@indep}{\not}% \mathpalette helps for the adaptation of the symbol to the different math styles.
}
}
\newcommand*{\@indep}[2]{%
% #1: math style
% #2: empty or \not
\sbox0{$#1\perp\m@th$}% box 0 contains \perp symbol
\sbox2{$#1=$}% box 2 for the height of =
\sbox4{$#1\vcenter{}$}% box 4 for the height of the math axis
\rlap{\copy0}% first \perp
\dimen@=\dimexpr\ht2-\ht4-.2pt\relax
% The equals symbol is centered around the math axis.
% The following equations are used to calculate the
% right shift of the second \perp:
% [1] ht(equals) - ht(math_axis) = line_width + 0.5 gap
% [2] right_shift(second_perp) = line_width + gap
% The line width is approximated by the default line width of 0.4pt
\kern\dimen@
{#2}%
% {\not} in case of \nindep;
% the braces convert the relational symbol \not to an ordinary
% math object without additional horizontal spacing.
\kern\dimen@
\copy0 % second \perp
}
\makeatother
\DeclareMathOperator*{\argmax}{argmax} % thin space, limits underneath in displays
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\fracpartial}[2]{\frac{\partial #1}{\partial #2}}
% for creating titles for both main paper and appendix
\makeatletter
\newcommand{\settitle}{\@maketitle}
\makeatother
%% \begin{}\label{bunch of symbols}
% mathbb
\newcommand{\Ebb}{\mathbb{E}}
\newcommand{\Nbb}{\mathbb{N}}
\newcommand{\Rbb}{\mathbb{R}}
% mathbf
\newcommand{\Fbf}{\mathbf{F}}
\newcommand{\Ibf}{\mathbf{I}}
\newcommand{\Kbf}{\mathbf{K}}
\newcommand{\Lbf}{\mathbf{L}}
\newcommand{\Vbf}{\mathbf{V}}
\newcommand{\Zbf}{\mathbf{Z}}
\newcommand{\abf}{\mathbf{a}}
\newcommand{\cbf}{\mathbf{c}}
\newcommand{\fbf}{\mathbf{f}}
\newcommand{\sbf}{\mathbf{s}}
\newcommand{\tbf}{\mathbf{t}}
\newcommand{\ubf}{\mathbf{u}}
\newcommand{\vbf}{\mathbf{v}}
\newcommand{\wbf}{\mathbf{w}}
\newcommand{\xbf}{\mathbf{x}}
\newcommand{\ybf}{\mathbf{y}}
\newcommand{\zbf}{\mathbf{z}}
% boldsymbol
\newcommand{\Kbs}{\boldsymbol{K}}
\newcommand{\abss}{\boldsymbol{a}}
\newcommand{\fbs}{\boldsymbol{f}}
\newcommand{\kbs}{\boldsymbol{k}}
\newcommand{\wbs}{\boldsymbol{w}}
\newcommand{\xbs}{\boldsymbol{x}}
\newcommand{\ybs}{\boldsymbol{y}}
\newcommand{\zbs}{\boldsymbol{z}}
\newcommand{\Phibs}{\boldsymbol{\Phi}}
\newcommand{\Psibs}{\boldsymbol{\Psi}}
\newcommand{\alphabs}{\boldsymbol{\alpha}}
\newcommand{\betabs}{\boldsymbol{\beta}}
\newcommand{\phibs}{\boldsymbol{\phi}}
\newcommand{\xibs}{\boldsymbol{\xi}}
\newcommand{\psibs}{\boldsymbol{\psi}}
% mathcal
\newcommand{\Acal}{\mathcal{A}}
\newcommand{\Dcal}{\mathcal{D}}
\newcommand{\Ecal}{\mathcal{E}}
\newcommand{\Fcal}{\mathcal{F}}
\newcommand{\Gcal}{\mathcal{G}}
\newcommand{\Hcal}{\mathcal{H}}
\newcommand{\Ical}{\mathcal{I}}
\newcommand{\Kcal}{\mathcal{K}}
\newcommand{\Lcal}{\mathcal{L}}
\newcommand{\Ncal}{\mathcal{N}}
\newcommand{\Qcal}{\mathcal{Q}}
\newcommand{\Rcal}{\mathcal{R}}
\newcommand{\Scal}{\mathcal{S}}
\newcommand{\Vcal}{\mathcal{V}}
\newcommand{\Wcal}{\mathcal{W}}
\newcommand{\Xcal}{\mathcal{X}}
\newcommand{\Ycal}{\mathcal{Y}}
\newcommand{\Zcal}{\mathcal{Z}}
\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\indicator}{\mathbbm{1}}
\usepackage{pifont}
\newcommand{\cmark}{\color{StrokeColorCBGreen} \small \ding{51}}
\newcommand{\xmark}{\color{FontColorCBRed} \scriptsize \ding{56}}
%\end{}
\errorcontextlines=1000 % easier to debug
% For double column
\setlist[enumerate]{leftmargin=*}
\section{Introduction}\label{main:introduction}
\looseness=-1
It is a well-known insight in optimization theory that coordinate-wise optimization, conditioned on the previously optimized arguments along each axis sequentially, generally does not guarantee finding the global optimum \citep{torn1989global,nocedal1999numerical}.
Similarly, in the context of decoding in large language models (LLMs), expecting to achieve a globally optimal response by sequentially accumulating per-token optimal decisions, as done in purely autoregressive decoding, may be overly optimistic.
Despite significant recent progress \citep{vaswani2017attention,radford2019language,brown2020language,openai2023gpt4,touvron2023llama,google2023gemini,meta2024llama,deepmind2024gemma,abdin2024phi4,deepseek2025deepseek}, how to approach the optimal text that one can possibly sample from a model still remains an open question.
\looseness=-1
Previous works have demonstrated challenges faced by autoregressive decoding in terms of the handling of long sequences \citep{wu2021autoformer} and the inefficient inference \citep{lin2021limitations,li2022elmer}.
At the level of decoding, other than the traditional greedy search, \citet{holtzman2020curious} proposed Top-$p$ sampling (also known as Nucleus Sampling), a stochastic method that adjusts the next-token set based on the shape of conditional probability mass function.
Alternatively, different from the cumulated probability mass, Top-$k$ sampling limits the number of available options when sampling for the next token \citep{fan2018hierarchical,holtzman2018learning,radford2019language}.
Another empirical technique involves adjusting the shape of probability distribution with the temperature hyperparameter \citep{fan2018hierarchical,holtzman2018learning,shih2023long,peeperkorn2024temperature}, or decoding as optimization with more than one models \citep{li2023contrastive}.
% \looseness=-1
In order to improve the sampling efficiency, speculative decoding approaches have been proposed, where multiple tokens are predicted in parallel as if one were sampling from the model (or its lighter counterpart) repetitively \citep{leviathan2023fast,chen2023accelerating,xia2023speculative,kim2024speculative,sun2024spectr}.
Efficient inference with beam search \citep{xie2024self,zhu2024deductive,wei2024confidence,yang2024language} and probabilistic programs \citep{lew2023smcp3} have been explored in the recent literature.
Improving generated content through high-level behaviors, e.g., through instructing self-correction or conducting self-improvement with external or internal feedback \citep{yao2022react,bai2022constitutional,pan2023automatically,shinn2023reflexion,ganguli2023capacity,chen2023teaching,kim2023language,tyen2023llms,madaan2024self}, has also been studied.
\looseness=-1
While previous literature has explored various methods to enhance generated content, the fundamental limitation of autoregressive decoding for text generation remains under-explored.
This gap represents a distinct perspective, different from high-level model behaviors (e.g., self-refinement studied in \citealt{madaan2024self}) or inference efficiency (e.g., various speculative decoding methods surveyed in \citealt{xia2024unlocking}).
In this paper, we theoretically characterize the inherent shortcoming of autoregressive decoding and propose an empirical method to mitigate this issue.
Our contributions can be summarized as follows:
\begin{itemize}[topsep=0pt,leftmargin=*]%,noitemsep]
\item We theoretically characterize the sub-optimality of autoregressive decoding for text generation, demonstrating its lack of a built-in mechanism to perform refinement/correction of generated content at the decoding level.
\item We propose a framework involving a sliding reflection window and a corresponding pause criterion, enabling an interchangeable process of refinement and generation.
\item Through extensive empirical evaluations, our approach demonstrates significant improvement over existing decoding approaches, and maintains performance comparable or superior to beam search while being more efficient.
\end{itemize}
\section{Motivation and High-Level Illustration}\label{main:motivation}
In this section, we first present our motivations behind addressing the inherent shortcoming of purely autoregressive decoding for text generation (Section~\ref{main:motivation:issue}).
Then in Section~\ref{main:motivation:reflection_window}, we present a high-level summary of our proposed approach involving interchangeably switching between refinement (upon reflection on previously generated content) and generation (of the additional new content).
\subsection{Inherent Shortcoming of Autoregressive Decoding}\label{main:motivation:issue}
\looseness=-1
Insight from the optimization theory highlights the gap between coordinate-wise accumulation of optimum and the global optimum \citep{torn1989global,nocedal1999numerical}.
Recent research advances in cognitive linguistics have also argued that language is primarily a tool for communication (for humans) rather than thought \citep{fedorenko2024language}.
While language naturally unfolds in a one-dimensional sequence, its underlying dependence pattern extends beyond a purely autoregressive structure.
% (where the current content is conditioned solely on what has been generated so far).
% Before presenting the detailed theoretical analysis (Section~\ref{main:formulation}), we
Let us consider an example of writing a novel.
For a long-format writing like novels, outlining (also referred to as plotting) is essential for structuring ideas, planning narratives, and crafting engaging drafts \citep{king2000writing,serravallo2017writing}.
Sub-goals refer to relatively small and achievable tasks that guide the author through each stage of the story, for instance, the setting of the circumstance, the element of tension and emotion, the sensory imagination of the scene.
\looseness=-1
As we illustrate in Figure~\ref{fig:comparison:selection}, $X_i^*$'s represent words or phrases in the novel, and $S_i$'s represent sub-goals, which may be related in a hierarchical way.
For instance, sub-goals within a single scene altogether serve the purpose of furthering the development of the story.
We model sub-goals in terms of selection variables $S_i$ since they represent constraints or objectives to achieve, which involve certain criteria to be satisfied over the variables that they operate upon.\footnote{
This modeling choice is consistent with the modeling of causal relations among variables of interest through selection variables in a directed acyclic graph (DAG) \citep{spirtes1993causation,pearl2009causality}.
}
As we can see from Figure~\ref{fig:comparison:selection}, the variables in optimal sequence (the novel in this example) $X_i^*$'s \emph{jointly} satisfy criteria, or optimize objectives, specified by sub-goals $S_i$'s.
This indicates that the best $X_i^*$ in the optimal sequence depends on best values of all other $X_j^*$'s.
However, with an autoregressive way of text generation, as illustrated in Figure~\ref{fig:comparison:autoregressive}, we only allow $X_i$ to depend on $X_j$'s if $j < i$, which is clearly sub-optimal.
\begin{figure}[t]
\centering
% \captionsetup{format=hang}
\begin{subfigure}{.48\columnwidth}
\centering
\includegraphics[height=7.5ex]{file/figures/selection_diagram.pdf}
\caption{\small
(Sub-)goals jointly satisfied by the optimal sequence}
\label{fig:comparison:selection}
\end{subfigure}
\hfill
\begin{subfigure}[b]{.48\columnwidth}
\centering
\includegraphics[height=7.5ex]{file/figures/autoregressive_diagram.pdf}
\caption{\small
Purely autoregressive way of decoding for text generation}
\label{fig:comparison:autoregressive}
\end{subfigure}
\caption{%\small
\looseness=-1
Illustrative diagrams of different dependence patterns among variables representing tokens or phrases in text generation.
Panel (a): the dependence pattern among variables in the optimal sequence where there are (sub-)goals $S_i$'s to achieve, specifying conditions or constraints that should \emph{jointly} be satisfied by $X_i^*$'s.
Panel (b): the autoregressive way of text generation, where $X_i$ is only allowed to depend on $X_j$ if $j < i$.
}
\label{fig:comparison}
\vspace{-2ex}
\end{figure}
\subsection{Selective Refinement Through Reflection Window}\label{main:motivation:reflection_window}
As illustrated in Section~\ref{main:motivation:issue}, one outcome of limiting the dependence pattern to the autoregressive structure is the lack of a built-in mechanism to correct or refine previously generated content at the decoding level, since what was generated is not subject to further edit (if without further processing).
In this subsection, we present a high-level summary of our approach that attempts to address this issue.\footnote{We present in detail our technical approach in Section~\ref{main:experiments:technical_approach}.}
Specifically, we propose a selective refinement framework that facilitates an interchangeable process of refinement and generation, so that the overall response satisfies requirements or objectives that operate jointly over all involved tokens.
\begin{figure*}[t]
\centering
% \captionsetup{format=hang}
\includegraphics[height=12.5ex]{file/figures/pipeline_diagram.pdf}
\caption{%\small
\looseness=-1
Overview of our approach to address the inherent shortcoming of autoregressive decoding for text generation, where the LLM interchangeably switches between refinement and generation.
The fast pointer pauses if the \emph{pausing criterion} is triggered, indicating the existence of a potential issue in the generated history.
Then, the model \emph{refines/corrects} the content between the fast and slow pointers before continuing generation, so that the slow pointer catches up with the fast pointer before the latter can move on.
Reflection window refers to the content between the fast and slow pointers.
}
\label{fig:pipeline}
\end{figure*}
\looseness=-1
As illustrated in Figures~\ref{fig:pipeline} and \ref{fig:example}, we can use fast and slow pointers on the generated content to form segments of a certain length, namely, the sliding reflection window, and perform potential refinements within this sliding window as the text generation proceeds.\footnote{
The naming of pointers is motivated by \citet{kahneman2011thinking}.
}
Our reflection-window decoding framework allows for revision before the entire output is completed, which offers several advantages.
\looseness=-1
First, we can improve the generated content in a timely manner.
If there are multiple potential issues in the generation history, the revision after finishing the generation can be inefficient since we allow errors to accumulate.
In other words, without a built-in mechanism for refinement or correction at the decoding level, we are forced to rely on high-level model behaviors and operate at a coarser granularity.
This often involves regenerating entire sentences (rather than refining words or phrases), and/or editing through multiple iterations (rather than interchangeably generate and refine in a single run), as in self-correction and self-improvement approaches \citep{yao2022react,bai2022constitutional,pan2023automatically,shinn2023reflexion,ganguli2023capacity,chen2023teaching,kim2023language,tyen2023llms,madaan2024self}.
% and we might have to perform multiple iterations of editing.
Second, our focus on selective refinement during decoding is not solely driven by inference efficiency considerations.
The primary goal of previous approaches, e.g., speculative decoding \citep{leviathan2023fast,chen2023accelerating,xia2023speculative,kim2024speculative,sun2024spectr,xia2024unlocking}, is to accelerate sampling from (a lighter version of) the original model, while the underlying decoding mechanism remains purely autoregressive.
Third, due to the one-dimensional progression of text generation, our sliding reflection window mechanism, given a pausing criterion, enables timely and assured detection of issues in the generated text.
Our framework complements previous approaches, and furthermore, offers versatility.
One can incorporate \emph{pausing criteria} and \emph{refinement/correction methods} at the decoding level, while preserving the ability to further leverage strategies that rely on high-level behaviors.
\looseness=-1
The empirical pausing criteria we use (detailed in Section~\ref{main:experiments}) are guided by our theoretical characterization of the sub-optimality of autoregressive text generation, and to this theoretical analysis we now turn.
\begin{figure*}[t]
\centering
% \captionsetup{format=hang}
\includegraphics[height=22.5ex]{file/figures/example.pdf}
\caption{%\small
\looseness=-1
A concrete example demonstrating our reflection-window decoding.
}
\label{fig:example}
\end{figure*}
\begin{comment}
Let us denote the text between \texttt{<START>} and the slow index as $A$, the content between slow and fast indices as $B$, and the immediate token after the fast indices as $C$.
Let us denote the goal (currently, use the prompt/instruction itself as the surrogate) as $G_{\mathrm{raw}}$, and its paraphrased mission-complete version as $G_{\mathrm{success}}$.
For instance, if $G_{\mathrm{raw}}$ is ``tell a joke,'' then $G_{\mathrm{success}}$ can be ``the generated text tells a joke.''
The reason of having these two versions is to have a way to evaluate the generated content and see if we are approaching the goal.
When slow index does not forward, and only the fast index is moving forward (generating mode)
\begin{equation}
\begin{split}
B & \leftarrow (B, C) \\
C & \leftarrow C'
\end{split}
\end{equation}
When fast index is paused, and only the slow index is trying to locate the problematic part in generated text (reflecting mode)
\begin{equation}
\begin{split}
A & \leftarrow (A, B[0]) \\
B & \leftarrow B[1:]
\end{split}
\end{equation}
\end{comment}
\begin{comment}[High-level picture]
\subsection{High-Level Picture}
The goal of this project is to incorporate following characteristics during text generating:
\begin{enumerate}
\item going beyond autoregressive way of text generation, explicitly introduce the selection mechanism among generated text, where the conditions under which the (final) objective/target/expectation/goal is achieve are specified by the prompt
\item enable the ability to reflect on generated text as the generation goes on, always with the target in mind (analogous to human writing, where we constantly reflect on what was written, and whether we are \emph{approaching} the goal as we proceed)
\item a mechanism that is not too heavy to be practically operationalizable
\end{enumerate}
\end{comment}
\section{Theoretical Characterization of the Sub-Optimality of Autoregressive Decoding}\label{main:theory}
\looseness=-1
We theoretically characterize the sub-optimality of autoregressive text generation.
We show that even if an LLM is sufficiently trained and can perfectly capture any autoregressive decomposition of the joint density, the autoregressive way of text generation can still deviate from the globally optimal response, even for the well-defined objective of maximizing output probability given a fixed length (setting aside whether this objective fully aligns with the ultimate goal).
Let us denote a token from the vocabulary $\Vcal$ as $w_v \in \Vcal$, whose index in the vocabulary is $v \in \lvert \Vcal \rvert$.
We use ``$i:j$'' to denote the increasing integer sequence from $i$ to $j$ if $i \leq j$, e.g., $1:t \coloneqq 1, 2, \ldots, t$ if $t > 1$, otherwise, $i:j \coloneqq \varnothing$.
\begin{definition}[Globally Optimal Length-$T$ Response]\label{def:optimal_sequence}
% \looseness=-1
We say a sequence $w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[T]}$ is globally optimal among all possible length-$T$ responses following the prompt $X_{\leq 0}$, if it has the highest ground-truth conditional probability, denoted by $f(X_{1:t} \mid X_{\leq 0})$ where $t \in [1, T]$:
\begin{equation}
\small
\begin{split}
& ~ \vbf_T^* = (\vbf_T^*[1], \vbf_T^*[2], \ldots, \vbf_T^*[T]) \\
& \coloneqq \argmax_{v_i \in \lvert \Vcal \rvert, i=1,2,\ldots,T} f(X_{1:T} = w_{v_1} w_{v_2} \ldots w_{v_T} \mid X_{\leq 0}).
\end{split}
\end{equation}
\end{definition}
\begin{definition}[Stepwise Optimal Length-$T$ Response]\label{def:stepwise_sequence}
We say a sequence $w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[T]}$ is stepwise optimal for prompt $X_{\leq 0}$, if the sequence consists of tokens that correspond to highest token-by-token conditional probabilities, denoted by $g(X_{t} \mid X_{1:t-1}, X_{\leq 0})$ where $t \in [1, T]$:\footnote{
For notional clarity, instead of $g(X_{t} \mid X_{1:t-1}=\xbf_{1:t-1}, X_{\leq 0}=\xbf_{\leq 0})$, we use shorthand notations in the conditioning set, i.e., $g(X_{t} \mid X_{1:t-1}, X_{\leq 0})$.
We will adopt this simplification throughout the paper, as long as it remains unambiguous.}
\begin{equation}
\small
\begin{split}
\widehat{\vbf}_T[1] & \coloneqq \argmax_{v_1 \in \lvert \Vcal \rvert} g(X_1 = w_{v_1} \mid X_{\leq 0}), \\
\widehat{\vbf}_T[2] & \coloneqq \argmax_{v_2 \in \lvert \Vcal \rvert} g(X_2 = w_{v_2} \mid X_1 = w_{\widehat{\vbf}_T[1]}, X_{\leq 0}), \\
& \cdots \\
\widehat{\vbf}_T[T] & \coloneqq \argmax_{v_T \in \lvert \Vcal \rvert}
\begin{array}{l}
g\left(X_T = w_{v_T} \mid X_{\leq 0}, \text{ and } \right. \\
~~ \left. X_{1:T - 1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[T - 1]}\right)
\end{array}, \\
\text{and } \widehat{\vbf}_T & = (\widehat{\vbf}_T[1], \widehat{\vbf}_T[2], \ldots, \widehat{\vbf}_T[T]).
\end{split}
\end{equation}
\end{definition}
In general, the longer the sequence, the lower the joint probability tends to be.
The fair comparison of optimality should be length-specific, and the optimal response of a shorter length is not necessarily identical to the prefix of the optimal response that is longer in length.
For instance, if we were to use 10 words to distinguish between joint and conditional densities, one might say ``\emph{joint density combines all variables; conditional adjusts for known variables}.''
However, if we can use 15 words, one might say ``\emph{joint density reflects combined probabilities of all variables; conditional density adjusts probabilities given known variables}.''
Therefore, we explicitly keep the length $T$ in the notation of vocabulary indices of tokens that constitute the length-$T$ responses.
\begin{assumption}[Oracle LLM]\label{aspt:oracle_LLM}
We say an autoregressive LLM is an oracle LLM, if the following relation holds for any response of a length $T \geq 1$:
\begin{equation}
\small
f(X_{1:T} \mid X_{\leq 0}) = \Pi_{t = 1}^{T} g(X_{t} \mid X_{1:t-1}, X_{\leq 0}).
\end{equation}
% where $g(\cdot)$ denotes the conditional probability of autoregressive way of token-by-token generation.
\end{assumption}
Assumption~\ref{aspt:oracle_LLM} specifies that, after given the prompt or generated text history $X_{\leq 0}$, an oracle (or very well-trained) LLM can recover the ground-truth probability of $X_{1:T}$ as a whole follows $X_{\leq 0}$, by multiplying token-by-token generating probabilities in an autoregressive way.\footnote{
Here, we implicitly assume that the context length of LLM is sufficiently large to allow for a meaningful discussion.
}
We would like to note that Assumption~\ref{aspt:oracle_LLM} only states that an oracle LLM can perfectly capture the autoregressive way of probability partitioning of text sequences, and this itself does \emph{not} guarantee the equivalence between stepwise optimal response and the same-length globally optimal response for $T > 1$.\footnote{
When $T = 1$, i.e., if the response is of a length $1$, the stepwise optimal is just the globally optimal for an oracle LLM, since there is only one step in total, and $f(X_1 \mid X_{\leq 0}) = g(X_1 \mid X_{\leq 0})$.
}
\begin{assumption}[Strict Preference Among Same-Length Sequences]\label{aspt:strict_preference}
For any two length-$T$ different sequences following the prompt $X_{\leq 0}$, there is a strict preference between them in terms of the ground-truth conditional probability $f(X_{1:T} \mid X_{\leq 0})$.
In other words, the ground-truth conditional probabilities of two length-$T$ sequences equal to each other if and only if the sequences are identical.
\end{assumption}
Assumption~\ref{aspt:strict_preference} specifies that from the ground-truth conditional probability perspective, there is a strict preference between how well two different same-length responses follow the prompt $X_{\leq 0}$, i.e., the ground-truth probability mass function $f(X_{1:T} \mid X_{\leq 0})$ is injective for any given $T > 0$.
\begin{assumption}[Irreversible Advantage Once Manifested]\label{aspt:no_return_after_detour}
When a stepwise optimal length-$T$ response from an oracle (Assumption~\ref{aspt:oracle_LLM}) autoregressive LLM $w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[T]}$ is not the globally optimal length-$T$ response $w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[T]}$, then if the deviation manifests itself at the length-$L$ ($1 < L \leq T$) prefix-sequences, the advantage of the globally optimal length-$T$ response will not be reversed afterwards:
\begin{equation*}
\small
\begin{split}
& \text{if }\exists L \in (1, T], f(X_{1:L} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[L]} \mid X_{\leq 0}) \\
& \quad\quad\quad\quad\quad\quad < f(X_{1:L} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[L]} \mid X_{\leq 0}), \\
& \text{then }\forall M \in [L, T], f(X_{1:M} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[M]} \mid X_{\leq 0}) \\
& \quad\quad\quad\quad\quad\quad < f(X_{1:M} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[M]} \mid X_{\leq 0}).
\end{split}
\end{equation*}
\end{assumption}
Assumption~\ref{aspt:no_return_after_detour} specifies that if the advantage (in terms of a higher ground-truth conditional probability) of the globally optimal length-$T$ sequence can be observed at the length-$L$ prefix-sequence, such advantage will not be reversed when considering longer prefix-sequences.
\begin{theorem}[Indication of Deviation from the Globally Optimal Length-$T$ Response]\label{thm:small_conditional_prob}
% Suppose that the context length $N$ of LLM is sufficiently large.
Given the prompt $X_{\leq 0}$, when an oracle LLM (Assumption~\ref{aspt:oracle_LLM}) generates a stepwise optimal length-$T$ response which is not the globally optimal response with the same length, let $L \leq T$ denote the minimum length of prefix-sequence needed in order for such deviation to manifest itself (Assumptions \ref{aspt:strict_preference} and \ref{aspt:no_return_after_detour}).
Then, the deviation from the globally optimal response happens at some step $K < L$.
Furthermore, the conditional probability when generating the token $w_{v_L} \in \Vcal$ is strictly smaller than a positive number, which itself is strictly smaller than $1$, i.e.,
% In other words, the LLM will be relatively uncertain about
\begin{equation*}
\small
\begin{split}
& \max_{w \in \Vcal} g(X_L = w \mid X_{1:L-1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]}, X_{\leq 0}) < \epsilon_L, \\
& \text{and } \epsilon_L = \frac{f(X_{1:L} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0})}{f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})} < 1.
\end{split}
\end{equation*}
\end{theorem}
Theorem~\ref{thm:small_conditional_prob} provides a necessary (but not sufficient) condition for the deviation of the stepwise optimal length-$T$ response from the same-length globally optimal response.
The uncertainty (i.e., low conditional probabilities) in generating the next token can result from different factors.
For instance, a previous mistake or detour makes it challenging to continue in any way that could possibly satisfy the goal specified by the prompt.
Such uncertainty can also result from multiple valid ways to proceed in order to achieve the goal.
Although we do not have access to the ground-truth conditional probability $f(X_{1:T} \mid X_{\leq 0})$, Theorem~\ref{thm:small_conditional_prob} suggests that when noticeable uncertainty arises, one should to be cautious of a potential deviation from the globally optimal response in the generated text.
\paragraph{Remark: Implication of Our Theoretical Analysis}
\looseness=-1
Our Theorem~\ref{thm:small_conditional_prob} indicate that, \emph{even if} with an oracle LLM (Assumption~\ref{aspt:oracle_LLM}), there is still suboptimality in the purely autogressive way of decoding.
In other words, even if the LLM itself perfectly decomposes the (conditional) probabilities (which is a far-fetched benefit to assume in practice), there is still no guarantee in obtaining the globally optimal sequence with purely autoregressive decoding.
Our empirical evaluations (Sections~\ref{main:synthetic}--\ref{main:experiments}) do not rely on or employ this assumption.
\begin{figure*}[t]
\centering
% \captionsetup{format=hang}
\begin{subfigure}[b]{0.22\textwidth}
\centering
\includegraphics[height=18ex]{file/figures/synthetic_0.pdf}
\caption{0'th token}
\label{fig:subfig1}
\end{subfigure}%
\begin{subfigure}[b]{0.22\textwidth}
\centering
\includegraphics[height=18ex]{file/figures/synthetic_20.pdf}
\caption{20'th token}
\label{fig:subfig2}
\end{subfigure}%
\begin{subfigure}[b]{0.22\textwidth}
\centering
\includegraphics[height=18ex]{file/figures/synthetic_50.pdf}
\caption{50'th token}
\label{fig:subfig3}
\end{subfigure}%
\begin{subfigure}[b]{0.22\textwidth}
\centering
\includegraphics[height=18ex]{file/figures/synthetic_200.pdf}
\caption{200'th token}
\label{fig:subfig4}
\end{subfigure}%
\begin{subfigure}[b]{0.12\textwidth}
\centering
\raisebox{3ex}{\includegraphics[height=18ex]{file/figures/synthetic_legend.pdf}}
% \caption{Legend}
\label{fig:subfig5}
\end{subfigure}
\caption{%\small
\looseness=-1
The probability that greedy decoding can attain globally optimal response, with respect to the number of newly generated tokens, and with different starting positions in the generation history.
The legend is shared across sub-figures.
}
\label{fig:synthetic_setting}
% \vspace{-1em}
\end{figure*}
\section{Sanity Check: Semi-Synthetic Settings}\label{main:synthetic}
The implication of our theoretical analysis is straightforward.
However, it is natural to ask whether the phenomenon actually occurs in real-world LLM decoding scenarios.
To provide clear empirical evidence accompanying our theoretical analysis, in this section, we present semi-synthetic experiments that serves as a sanity check.
In particular, in moderately realistic settings, we show that greedy decoding for text generation with stepwise optimization results in suboptimal responses.
We first outline the semi-synthetic setting, and then present the empirical findings.
% \vspace{-2ex}
\paragraph{Illustrative Approximation}
\looseness=-1
For any modern LLM with a vocabulary size $|\Vcal|$ (typically on the order of $10^4$ to $10^5$), identifying the globally optimal sequence across multiple steps becomes computationally intractable, even for relatively short sequence lengths ($<100$).
To ensure the validity of our claim while providing a clear and accessible illustration, we adopt beam search as an approximation strategy of obtaining globally optimal sequence.
Since we measure the chance that greedy decoding can attain the global optimum with the stepwise optimal response, this approximation serves as an upper bound on achievable performance, indicating the discrepancy between greedy decoding and the true globally optimal response.
% \vspace{-2ex}
\paragraph{Approximating Natural Language Scenarios}
Since the prompt or context of the generation influence model behavior, we align our experimental setting with common human-LLM interactions.
Specifically, we utilize MT-Bench \citep{zheng2023judging} questions as curated prompts, which are designed to evaluate conversational chat models.
These samples serve as an approximation of real-world natural language context distributions, ensuring that our findings are grounded in practical scenarios.
% \vspace{-2ex}
\paragraph{Findings}
We use Llama-3.1-8B-Instruct \citep{meta2024llama}, and for each prompt, together with a certain length of generation history ($0$ means only the prompt is given), we evaluate whether the joint probability of the sequence generated with greedy decoding is greater than or equal to that produced by beam search (with beam width set to $10$, as a proxy of the global optimum).
This comparison indicates the extent to which greedy decoding deviates from the globally optimal response.
As illustrated in Figure~\ref{fig:subfig1}, greedy decoding consistently results in suboptimal sequences, and the phenomenon can be observed with a small number of newly generated tokens.
In addition, the potential deviation may behave differently across various positions in the generated text.
For instance, when openings of response diverge, it is hard for greedy decoding to achieve optimal results afterwards.
To reduce potential inductive bias resulting from the diversity at early stages of generation, we evaluate generations starting/continuing from various positions throughout generation history, as presented in Figures~\ref{fig:subfig2}--\ref{fig:subfig4}.
We can observe that the deviation persists across different positions, which empirically demonstrate the common existence of sub-optimality in autoregressive decoding for text generation.
% \section{(Semi)-Synthetic Experiments}
% The principles behind our approach are straightforward; however, one might question whether the observed phenomenon persists in real-world LLM decoding scenarios. To provide a clear evidence of the previous theoretical analysis and also address the aforementioned concern, we introduce a (semi)-synthetic experiment designed to demonstrate that, in moderately realistic settings, greedy decoding during text generation with step-wise optimization results in suboptimal response. Before presenting the experimental results, we first outline the semi-synthetic setup.
% \paragraph{Illustrative Approximation}\looseness=-1
% Although there is no doubt on the existence of optimal response, for any modern large language model with a vocabulary size \(|\Vcal|\) ( typically on the order of \(O(10^4)\) to \(O(10^5)\)), identifying the joint optimal sequence across multiple steps becomes computationally intractable, even for relatively short sequence lengths (\(<100\)). To uphold the validity of our claim while offering an accessible illustration, we adopt beam search as an approximation of the globally optimal sequence. Since we measure the probability that greedy decoding achieves optimal response, this approximation serves as an upper bound on achievable performance, highlighting the discrepancy between greedy decoding and the true global optimal response.
% \paragraph{Approximating Natural Language Scenarios}
% One may also curious about the prompt or context in the generation since those factors will influence the generation behaviour as well. It is crucial to demonstrate in realistic scenarios. Specifically, we align our experiment setting with common human interactions with large language models. To achieve this, we utilize MT-Bench questions as curated prompts and contexts, which are designed for evaluating conversational, open-ended chat models. These samples serve as an approximation of real-world natural language context distributions, ensuring that our findings are grounded in practical, human-like use cases.
% \paragraph{Results}
% For each prompt or context, we evaluate whether the joint probability of the sequence generated through greedy decoding is greater than or equal to that of the sequence produced by the approximate global optimum method, i.e., beam search. This comparison highlights the extent to which greedy decoding diverges from the optimal sequence. As demonstrated in Figure \ref{fig: synthetic exp}, greedy decoding consistently results in suboptimal sequences, even when generating a small number ($<10$) of new tokens. Additionally, the behavior varies across positions in the generated text. For example, the beginning of the answer are usually diverge and it can be harder for greedy decoding to achieve optimal. To reduce potential inductive biases arising from greater diversity at the initial stages of generation, we evaluate sequences at various positions throughout the generated text (Figure \ref{fig:subfig1}-\ref{fig:subfig4} capture different starting positions for the evaluation). Our findings indicate that this behavior persists across all evaluated positions, underscoring the importance of the issue. This observation naturally suggests a solution: a reflective decoding strategy using a sliding window mechanism.
\input{file/main/alg_pipeline.tex}
\section{Empirical Approach and Experiments}\label{main:experiments}
In Sections \ref{main:experiments:technical_approach}--\ref{main:experiments:setting}, we provide technical details about our empirical approach and settings of our experiments.
Then in Sections \ref{main:experiments:experiment_results}--\ref{main:experiments:discussions}, we present experimental results to demonstrate both the effectiveness and efficiency of our method.
% @TODO say sth more about across what models,
\subsection{Reflection-Window Decoding: Technical Details}\label{main:experiments:technical_approach}
Our findings through both the theoretical characterization of sub-optimality in autoregressive decoding for text generation (Section~\ref{main:theory}), and the sanity check with empirical verifications in semi-synthetic settings (Section~\ref{main:synthetic}), suggest the necessity of a built-in reflection-and-refine mechanism at the decoding level.
To empirically address this issue, we propose a selective refinement framework that interchangeably refine and generate as the response unfolds.
Text typically unfolds in a single direction, i.e., from the start to the end, with words, phrases, and sentences.
This differentiates text from other forms of objects that occupy multiple dimensional spaces, e.g., images or videos.
Taking advantage of this one-dimensional nature, our decoding framework integrates a sliding reflection window along with two additional modules: (1) a \emph{pausing criterion} that specifies whether we should pause the generation upon reflecting on generated content, and (2) a \emph{refinement/correction method} that facilitates revision at the decoding level (if the pausing criterion is triggered).
We present the pseudocode of our reflection-window decoding approach in Algorithm~\ref{alg:overall_pipeline}.
% \vspace{-2ex}
\paragraph{Pausing Criterion}
\looseness=-1
Guided by our theoretical characterization (Theorem~\ref{thm:small_conditional_prob}), the reflection at the decoding level needs to capture the (increasing trend of) uncertainty as text generation proceeds.
For an empirical pausing criterion, we use the conditional entropy $H(\cdot)$ based on the next-token logits across the vocabulary.
Specifically, given an LLM which models the conditional distribution $g(X_{t} \mid X_{1:t-1})$ of the token at $t$-th step given all the observed history $X_{1:t-1}= \xbf_{1:t - 1}$, we use the pausing criterion $h(t; \sigma, d)$:
\begin{equation}
\small
h(t; \sigma, d) =
\begin{cases}
\text{True} & \text{if }~H\big( X_{t - i} \mid X_{1:(t - i - 1)} \big) > \sigma~, \\
& \forall i \in [0,d-1], \\
\text{False} & \text{Else},
\end{cases}
\end{equation}
where $\sigma$ denotes the hyperparameter for the threshold of conditional entropy, and $d$ denotes that for the window size (how far we look back in history, in terms of the token counts).
\looseness=-1
When $h(t^{\mathrm{(fast)}}; \sigma, d)$ is True, the pausing criterion (denoted by $\mathrm{IfPause(\cdot)}$ in Algorithm~\ref{alg:overall_pipeline}, with hyperparameter enclosed) is triggered upon reflecting on the most recent $d$ generated tokens, i.e., the length-$d$ reflection window when the fast pointer is at $t^{\mathrm{(fast)}}$.
The two parameters, $\sigma$ and $d$, jointly decide the sensitivity and effective region of the pausing criterion, and we present more discussions in Section~\ref{main:experiments:entropy_window_size}.
% \vspace{-2ex}
\paragraph{Refinement/Correction Method}
When the pausing criterion is triggered, tokens within the current sliding reflection window need to be refined or corrected.
Since beam search can approximate the global optimum relatively well, empirically we introduce a fixed-length beam search to generate a new segment with a length $d$ (denoted by the hyperparameter-enclosed function $\mathrm{ReGenerate(\cdot)}$ in Algorithm~\ref{alg:overall_pipeline}), to replace the content within the reflection window.
After the refinement, the slow pointer $t^{\mathrm{(slow)}}$ catches up with the fast one $t^{\mathrm{(fast)}}$ and the model proceeds with generation while maintaining the sliding reflection window.
% \vspace{-2ex}
\paragraph{Remark: Versatile Decoding Framework}
We would like to note that our reflection-window decoding approach is highly versatile.
While our empirical approach employs a specific pausing criterion and refinement/correction method, practitioners can customize these components by incorporating different functions, namely, $\mathrm{IfPause(\cdot)}$ and $\mathrm{ReGenerate(\cdot)}$ in Algorithm~\ref{alg:overall_pipeline}, to meet their needs.
Our selective refinement framework integrates the sliding reflection window mechanism with these components, enabling simultaneous refinement and generation at the decoding level while retaining the flexibility to incorporate additional strategies, such as those based on high-level model behaviors and/or speculative decoding (Section~\ref{main:motivation:reflection_window}).
\subsection{Experiment Settings}\label{main:experiments:setting}
We provide technical details about settings of our experiments, including models, benchmarks, evaluation metrics, and baseline methods.
\paragraph{LLM Models}
We conduct experiments using multiple models across different families/herds.
Specifically, we use Llama-3.1-8B-Instruct (denoted as Llama3.1-8B), which belongs to the Llama 3.1 herds \citep{meta2024llama}, Phi-3-Medium-128K-Instruct \citep{abdin2024phi} (denoted as Phi3-Medium) with 14 billion parameters, Qwen2.5-14B-Instruct \citep{qwen2} (denoted as Qwen2.5-14B) with 14 billion parameters, Qwen2.5-7B-Instruct (denoted as Qwen2.5-7B) with 7 billion parameters, and Mistral-Nemo-Instruct-2407 \citep{mistralNemo} (denoted as Mistral-Nemo) with 12 billion parameters.
\paragraph{Benchmarks and Evaluation Metrics}
Our experiments are conducted on MMLU \citep{hendrycks2020measuring} and MT-Bench \citep{zheng2023judging}.
MMLU tests model's general knowledge across 57 diverse subjects, e.g., humanities, STEM (Science, Technology, Engineering, and Mathematics), and social sciences, at varying difficulty levels, making it a comprehensive evaluation of model's reasoning performance and factual knowledge.
MT-Bench, on the other hand, provides a fine-grained evaluation through multi-turn conversational tasks, evaluating not just correctness, but also coherence and fluency.
For MMLU, we adopt the macro averaged accuracy metric because the number of questions varies across subjects.
For MT-Bench, for each pair of responses, we prompt the LLM judge (for which we use GPT-4o, \citealt{openai2024gpt4o}) with two responses following the prompting method outlined in \citet{zheng2023judging}.
The LLM judge should return a decision from three options: \texttt{win}, \texttt{lose} or \texttt{tie}.
To avoid the influence from the preference bias, for each pair we prompt the LLM judge twice with responses placed in different sequences.
When a response gets two \texttt{win}'s (\texttt{lose}'s) or one \texttt{win} (\texttt{lose}) plus one \texttt{tie}, we record the response as (not) prevailing.
The rest situations are treated as \texttt{tie}.
We use win rate as the measurement of performance on MT-Bench, calculated by $\text{win rate} \coloneqq \frac{\text{number of wins}}{\text{number of wins}+\text{number of losses}}$.
% We employ GPT-4o to score the responses, selecting the one with the higher score as the winner.
\paragraph{Baseline Methods}
\looseness=-1
We compare reflection-window decoding with three baseline methods: greedy decoding, vanilla beam search (with a finite beam width), and Top-$k$/Top-$p$ sampling \citep{fan2018hierarchical,holtzman2020curious}.\footnote{
If there is no limit on the computation and storage, beam search with an unconstrained beam width could yield the globally optimal output through brute force.
However, in practice, maintaining a full frontier quickly becomes intractable, and a finite beam width is often introduced as a hyperparameter.
}
We consider greedy search as one baseline method since it corresponds closely to our theoretical analysis.
However, since the widely usage of Top-$p$ and Top-$k$ sampling (and often used in combination), we also include it as a baseline approach.
Reflection-window decoding only leverages beam search when necessary, i.e., when $\mathrm{IfPause(\cdot)}$ in Algorithm~\ref{alg:overall_pipeline} is triggered.
For fair comparison, we set the beam size to $4$ in both reflection-window decoding and the vanilla beam search.
\input{file/tables/MMLU_acc_phi3}
\begin{figure}[t]
\centering
\includegraphics[height=21ex]{file/figures/win_rate_comparison_squeeze.pdf}
\caption{%\small
Comparison of win rates between beam search and our reflection-window decoding (both against greedy decoding) on MT-Bench across categories}
\label{fig:winrate_comparison}
\end{figure}
\subsection{Experiment Results}\label{main:experiments:experiment_results}
We compare the performance among greedy decoding, beam search, Top-$k$/Top-$p$ sampling, and our reflection-window decoding.
We use an entropy threshold of $\sigma=0.5$ and a window size of $d=4$ in reflection-window decoding.
\paragraph{MMLU}
\looseness=-1
Table~\ref{tab:mmlu_category_phi3} presents the comparison among greedy decoding, beam search, and reflection-window decoding (with greedy search as the ``regular decoding'' in Algorithm~\ref{alg:overall_pipeline}) on four question categories in MMLU (STEM, humanities, social sciences, and others), using Phi3-Medium as the base LLM.
Our approach demonstrates comparable or superior performance across all categories, achieving the highest average accuracy of $78.15\%$.
The results containing all MMLU subjects can be found in Table~\ref{tab:mmlu_subject_phi3} in Appendix.
\input{file/tables/topktopp}
\paragraph{MT-Bench}
\looseness=-1
We choose Llama3.1-8B as the base model in this experiment.
On MT-Bench, the Reflection Window method clearly outperformed both greedy decoding and beam search.
In comparisons with greedy decoding, according to assessments by GPT-4o, reflection window prevails in $66.67\%$ of cases, a win rate significantly higher than that of beam search, which only outperformed greedy decoding in $46.3\%$ of cases.
Figure~\ref{fig:winrate_comparison} shows that our reflection-window decoding significantly outperforms beam search in Roleplay, STEM, Math, and Reasoning tasks.
These tasks require strong logical consistency, and greedy or fixed search strategies often lead to early sub-optimal choices that degrade output quality.
Our approach mitigates this by enabling refinement during generation, making generated contents more coherent.
\input{file/tables/regeneration}
\input{file/tables/entropy_threshold}
\input{file/tables/window_size}
\paragraph{Compatibility with Top-$k$/Top-$p$ Sampling}
\looseness=-1
Reflection-window decoding generates tokens autogressively when the pausing criterion is not triggered and only performs selective refinements.
The framework is compatible with Top-$k$/Top-$p$ sampling except in the refinement/correction mode.
In these experiments, we set $k=10$, $p=0.9$, and temperature as 1.0 for both our approach and the baseline Top-$k$/Top-$p$ sampling.
As shown in Table~\ref{tab:summary_mmlu}, our decoding approach consistently outperforms the standard Top-$k$/Top-$p$ approach across all four models.
In particular, the average accuracy improvements range from $0.88$ percentage points (Mistral-Nemo) to $2.76$ percentage points (Phi3-Medium), highlighting the effectiveness of our approach even when stochastic sampling is introduced.
Notably, the largest performance gains are observed in STEM and humanities categories, suggesting that reflection-window decoding is particularly beneficial for reasoning-heavy tasks.
This aligns with observations from the MT-Bench experiments, which also demonstrate that our approach excels in tasks demanding structured logical thinking and complex problem-solving.
We present further results from Table~\ref{tab:decoding_comparison_llama} to Table~\ref{tab:decoding_comparison_qwen} in Appendix.
\subsection{Further Discussions and Analyses}\label{main:experiments:discussions}
\paragraph{Efficiency of Reflection-Window Decoding}
% \input{file/tables/table5}
In Table~\ref{tab:regeneration_metric_mmlu}, we aggregate the regeneration statistics on MMLU with Llama3.1-8B.
We record two metrics: (1) the regeneration ratio, which calculates the overall ratio of refined/corrected tokens in the completed response, and (2) the regeneration call, which counts the number of times the pausing criterion is triggered and refinement/correction is needed before finishing any particular response.
We find that regeneration ratio ranges from $3.5\%$ to $5.5\%$ across all categories with a relatively mild pausing criterion.
This suggests that the refinements are usually needed during decoding.
While beam search always maintains a complete frontier of candidate sequences, our reflection-window decoding approach only activates beam search when necessary, and at the sub-sequence level.
\paragraph{Entropy Threshold $\sigma$ and Window Size $d$}\label{main:experiments:entropy_window_size}
We investigate the impact of threshold $\sigma$ on MMLU Social Sciences with Qwen2.5-7B with fixed window size $d=4$.
The results in Table~\ref{tab:acc_mmlu_social_science_entropy_threthold} demonstrate that our method performs robustly across $\sigma$ values ranging from $0.25$ to $0.75$, with $\sigma=0.25$ achieving the highest macro average of $80.31\%$.
While our default setting of $\sigma=0.5$ is not the best in this specific experiment, it maintains strong performance and shows consistent improvements on most subjects, suggesting that it serves as a reliable default configuration for general usage.
In terms of window size $d$, we also choose social science subjects using Qwen2.5-7B.
The results in Table~\ref{tab:acc_mmlu_social_science_window_size} show that our method maintains strong performance across various window sizes ($d=2$ to $d=6$), with overall macro averages consistently around $79.7\%$.
While $d=6$ achieves the highest macro average, $d=4$ demonstrates comparable performance and maintains better computational efficiency.
These results further support our choice of $d=4$ as a robust default setting, offering a good balance between performance and efficiency across different models and tasks.
We provide additional results on MT-Bench in Table~\ref{tab:mistral_mtbench_per_d} and Table~\ref{tab:llama31_mtbench_per_d}, and on MMLU in Table~\ref{tab:qwen_mmlu_social_science_per_d} and Table~\ref{tab:qwen_mmlu_social_science_per_sigma} in Appendix.
\begin{comment}[Removed for ICML 2025 submission]
% \begin{table*}[h]
% \centering
% \caption{Regeneration Measurement by Category on MMLU.}
% \label{tab:regeneration_metric_mmlu}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Category} & \textbf{Regeneration Rate (\%)} & \textbf{Num of Regeneration} & \textbf{Num of All Tokens} \\
% \midrule
% Humanities & 5.04 & 4.27 & 336.69 \\
% Other & 5.54 & 4.31 & 305.93 \\
% STEM & 3.50 & 3.15 & 398.55 \\
% Social Sciences & 4.82 & 3.84 & 316.29 \\
% \bottomrule
% \end{tabular}
% % \setlength{\abovecaptionskip}{-0.1cm}
% \end{table*}
% \begin{figure}[h!]
% \centering
% % Single row of four subfigures
% \begin{subfigure}[b]{0.32\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/anatomy.pdf}
% \caption{anatomy}
% \label{fig:anatomy}
% \end{subfigure}
% % \hfill
% \begin{subfigure}[b]{0.32\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/computer_security.pdf}
% \caption{computer\_security}
% \label{fig:computer_security}
% \end{subfigure}
% % \hfill
% \begin{subfigure}[b]{0.32\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/global_facts.pdf}
% \caption{global\_facts}
% \label{fig:global_facts}
% \end{subfigure}
% % \hfill
% % \begin{subfigure}[b]{0.23\textwidth}
% % \centering
% % \includegraphics[width=\textwidth]{file/figures/nutrition.pdf}
% % \caption{nutrition}
% % \label{fig:subfig4}
% % \end{subfigure}
% \caption{Accuracy on the Selected Subsets with Entropy Threshold $\sigma$}
% \label{fig:sigma}
% \end{figure}
% \begin{figure*}[t]
% \centering
% \begin{subfigure}[b]{0.25\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/anatomy.pdf}
% \caption{anatomy}
% \label{fig:anatomy}
% \end{subfigure}
% \hspace{0.5em}
% \begin{subfigure}[b]{0.25\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/computer_security.pdf}
% \caption{computer\_security}
% \label{fig:computer_security}
% \end{subfigure}
% \hspace{0.5em}
% \begin{subfigure}[b]{0.25\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/global_facts.pdf}
% \caption{global\_facts}
% \label{fig:global_facts}
% \end{subfigure}
% \caption{Accuracy on the Selected Subsets with Entropy Threshold $\sigma$}
% \label{fig:sigma}
% \end{figure*}
% In Figure~\ref{fig:sigma}, we examined the impact of the hyperparameter: entropy threshold $\sigma$. We choose five distinctive values between 0 and 1: 0.1, 0.25, 0.5, 0.75, and 1. We choose three subsets from MMLU: "anatomy", "computer\_security" from STEM and "global\_facts" from Others section.When the threshold is set to 0.5, the Reflection Window method achieves its best performance. Conversely, when the threshold is set to extreme values such as 0.1 or 1.0, the Reflection Window method may degrade to perform similarly to Greedy decoding. It is noteworthy that the performance trends on these three subjects are different. The reason could probably be the domain shift between different tasks, subjects, etc. It indicates that, although $\sigma = 0.5$ is a relatively safe and stable option for pausing criterion, a more dynamic and adaptive strategy could help to alleviate the effect caused by text domain gap. We also provide more experiment on the selection strategy of $\sigma$ and $d$ on in \ref{tab:qwen-d-social-science} and \ref{tab:qwen-sigma-social-science} in Appendix.
% It could be also an indication that, the overall characteristics of different categories and tasks are also different.
% \begin{figure}[h!]
% \centering
% \includegraphics[width=0.85\textwidth]{file/figures/win_rate_comparison.pdf} % Adjust the width as needed
% \caption{Win rate comparison by category on MT-Bench between Beam Search and Reflection Window} % Caption for the figure
% \label{fig:winrate_comparison} % Label for referencing the figure in the text
% \end{figure}
% \begin{figure}[t]
% \centering
% % First row of two subfigures
% \begin{subfigure}[b]{0.36\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/anatomy.pdf}
% \caption{anatomy}
% \label{fig:subfig1}
% \end{subfigure}
% % Second subfigure in the same row
% \begin{subfigure}[b]{0.36\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/computer_security.pdf}
% \caption{computer\_security}
% \label{fig:subfig2}
% \end{subfigure}
% % Second row of two subfigures
% \begin{subfigure}[b]{0.36\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/global_facts.pdf}
% \caption{global\_facts}
% \label{fig:subfig3}
% \end{subfigure}
% % Fourth subfigure in the same row
% \begin{subfigure}[b]{0.36\textwidth}
% \centering
% \includegraphics[width=\textwidth]{file/figures/nutrition.pdf}
% \caption{nutrition}
% \label{fig:subfig4}
% \end{subfigure}
% \caption{todo}
% \label{fig:subfigures}
% \end{figure}
\end{comment}
\begin{algorithm}[t]
% \DontPrintSemicolon
\caption{Reflection-Window Decoding with Selective Refinement for Text Generation}
\label{alg:overall_pipeline}
% --> The input output relation
\Input{%
The prompt $X_{\leq 0}$, the reflection window size $d$, as well as hyperparameter-enclosed functions for the pause criterion $\mathrm{IfPause}(\cdot)$ and the refine/correct method $\mathrm{ReGenerate}(\cdot)$.
}
\Output{%
Text generated with selective refinement $\xbf$.
}
$t^{\mathrm{(slow)}} \leftarrow 0$, $t^{\mathrm{(fast)}} \leftarrow 1$\tcp*[r]{slow/fast pointer}
response $\xbf \leftarrow$ empty string\;
\While{not stopped, or response not completed}{
regular decoding for the next token $w_{t^{\mathrm{(fast)}}}$\;
\uIf{$t^{\mathrm{(fast)}} - t^{\mathrm{(slow)}} < d$}{
$\xbf \leftarrow \big( \xbf_{1 : t^{\mathrm{(fast)}} - 1}, w_{t^{\mathrm{(fast)}}} \big)$\;
}
\Else{
concatenate and get temporary sequence $\widehat{\xbf} \leftarrow \big( \xbf_{1 : t^{\mathrm{(fast)}} - 1}, w_{t^{\mathrm{(fast)}}} \big)$\;
\uIf{$\texttt{True} = \mathrm{IfPause}\big( \widehat{\xbf}_{t^{\mathrm{(fast)}} - d + 1 : t^{\mathrm{(fast)}}} \big)$}{
\Comment{Refine upon reflection}
obtain a length-$d$ replacement text
$\xbf^*_{t^{\mathrm{(fast)}} - d + 1 : t^{\mathrm{(fast)}}} \leftarrow \mathrm{ReGenerate} \big( \xbf_{1 : t^{\mathrm{(fast)}} - d} \big)$\;
$\xbf \leftarrow \big( \xbf_{1 : t^{\mathrm{(fast)}} - d}, \xbf^*_{t^{\mathrm{(fast)}} - d + 1 : t^{\mathrm{(fast)}}} \big)$\;
$t^{\mathrm{(slow)}} \leftarrow t^{\mathrm{(fast)}}$\tcp*[r]{update slow ptr}
}
\Else{
\Comment{Continue to generate}
$\xbf \leftarrow \widehat{\xbf}$\;
}
}
$t^{\mathrm{(fast)}} \leftarrow t^{\mathrm{(fast)}} + 1$\tcp*[r]{update fast ptr}
}
\Return{$\xbf$}.
\end{algorithm}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table*}[t]
\caption{%\small
Comparison of macro averaged accuracy on MMLU across subject categories with Phi3-Medium}
\label{tab:mmlu_category_phi3}
\centering
\footnotesize
\newcolumntype{g}{>{\columncolor[rgb]{0.9,0.9,1.0}}C{0.15}}
\begin{tabular}{L{0.24}C{0.13}C{0.15}C{0.15}C{0.13}g}
\toprule
\textbf{Decoding Method} & \textbf{STEM} & \textbf{Humanities} & \textbf{Social Sciences} & \textbf{Others} & \textbf{Macro Average} \\
\midrule
Greedy Decoding & 78.40\% & 71.92\% & \textbf{83.91}\% & \textbf{81.59}\% & 78.14\% \\
Beam Search & 78.62\% & 68.65\% & 82.93\% & 79.59\% & 76.44\% \\
Reflection-Window (Greedy) & \textbf{79.06}\% & \textbf{71.98}\% & 83.65\% & 81.10\% & \textbf{78.15}\% \\
\bottomrule
\end{tabular}
\end{table*}
\endgroup
% % ... Example table format
% \begingroup
% \renewcommand{\arraystretch}{1.3}
% \begin{table}[t]
% \caption{Example table template}
% \label{tbl:name_of_table}
% \centering
% \footnotesize % for font size across this tabular environment
% \begin{tabular}{L{0.09}C{0.09}C{0.09}C{0.09}C{0.09}} % number add up to 0.45 for single column, or 0.9 for double column (use table* environment)
% \toprule
% 1 & \scriptsize{1} & \tiny{1} & 1 & 1 \\
% \midrule
% \\
% \end{tabular}
% \end{table}
% \endgroup
% \begin{table*}[t]
% \centering
% \caption{Accuracy on MMLU by greedy decoding, beam search and reflection window}
% \label{tab:table1}
% \begin{tabular}{lcccccc}
% \toprule
% \textbf{Method} & \textbf{STEM} & \textbf{Humanities} & \textbf{Social Science} & \textbf{Other} & \textbf{Average} \\
% \midrule
% Greedy Decoding & 68.31\% & 64.38\% & 73.29\% & 73.37\% & 69.60\% \\
% Beam Search & \textbf{70.22\%} & \textbf{64.87\%} & \textbf{75.30\%} & \textbf{73.97\%} & \textbf{70.83\%} \\
% Reflection Window & 69.39\% & 64.40\% & 73.45\% & \textbf{73.97\%} & 70.05\% \\
% \bottomrule
% \end{tabular}
% % \setlength{\abovecaptionskip}{-0.1cm}
% \end{table*}
\begingroup
\renewcommand{\arraystretch}{1.2}
\begin{table*}[t]
\caption{Summary of comparison between Top-$k$/Top-$p$ and reflection-window decoding on MMLU (multiple models)}
\label{tab:summary_mmlu}
\centering
\footnotesize
\newcolumntype{g}{>{\columncolor[rgb]{0.9,0.9,1.0}}C{0.12}}
\begin{tabular}{L{0.14}L{0.18}C{0.12}C{0.12}C{0.14}C{0.12}g}
\toprule
\textbf{Model} & \ \ \textbf{Decoding Method} & \textbf{STEM} & \textbf{Humanities} & \textbf{Social Sciences} & \textbf{Others} & \textbf{Average} \\\midrule
\multirow{2}{*}{Llama3.1-8B} & \ \ Top-$k$ Top-$p$ & 65.94\% & 58.43\% & 72.60\% & 73.06\% & 66.46\% \\
& \ \ Reflection-Window & \textbf{67.21\%} & \textbf{59.43\%} & \textbf{72.86\%} & \textbf{73.41\%} & \textbf{67.21\%} \\\midrule
\multirow{2}{*}{Phi3-Medium} & \ \ Top-$k$ Top-$p$ & 71.36\% & 65.76\% & 79.10\% & \textbf{74.93\%} & 71.97\% \\
& \ \ Reflection-Window & \textbf{73.07\%} & \textbf{71.31\%} & \textbf{79.85\%} & 74.86\% & \textbf{74.73\%} \\\midrule
\multirow{2}{*}{Qwen2.5-14B} & \ \ Top-$k$ Top-$p$ & 83.00\% & 69.86\% & 82.74\% & 79.82\% & 77.84\% \\
& \ \ Reflection-Window & \textbf{83.30\%} & \textbf{70.33\%} & \textbf{83.85\%} & \textbf{81.41\%} & \textbf{78.48\%} \\\midrule
\multirow{2}{*}{Mistral-Nemo} & \ \ Top-$k$ Top-$p$ & 60.74\% & 50.16\% & 67.79\% & \textbf{65.66\%} & 59.83\% \\
& \ \ Reflection-Window & \textbf{61.78\%} & \textbf{51.41\%} & \textbf{69.22\%} & 65.27\% & \textbf{60.71\%} \\
\bottomrule
\end{tabular}
\end{table*}
\endgroup
\begin{comment}[Previous versions]
% \begin{table*}[t]
% \small
% \centering
% % \vspace{0.2cm}
% \newcolumntype{g}{>{\columncolor{Gray}}c}
% \renewcommand{\arraystretch}{1.0}
% \setlength{\tabcolsep}{2.6pt}
% \begin{tabular}{llccccg}
% \toprule
% & & \textbf{anatomy} & \textbf{computer\_security} & \textbf{global\_facts} & \textbf{nutrition} & \textbf{Average}\\\midrule
% \multirow{3}{*}{Llama3.1-8B} & \ \ Greedy Decoding & 67.40\% & 73.00\% & 43.00\% & 73.86\% & 67.55\% \\
% & \ \ Beam Search & \textbf{74.07\%} & \textbf{78.00\%} & 45.00\% & 75.82\% & 70.98\% \\
% & \ \ Reflection Window & 72.59\% & 77.00\% & \textbf{50.00\%} & \textbf{76.47\%} & \textbf{71.61\%} \\\midrule
% \multirow{3}{*}{Phi-3-medium} & \ \ Greedy Decoding & 73.33\% & 70.00\% & \textbf{56.00\%} & 82.68\% & 70.50\%\\
% & \ \ Beam Search & 70.37\% & \textbf{78.00\%} & 58.00\% & 81.37\% & 71.94\%\\
% & \ \ Reflection Window & \textbf{74.07\%} & 76.00\% & 62.00\% & \textbf{82.35\%} & \textbf{73.61\%}\\\midrule
% % \multirow{3}{*}{Phi-3.5-mini} & \ \ Greedy & & & & & \\
% % & \ \ Beam Search & & & & & \\
% % & \ \ Reflection Window & & & & & \\\midrule
% \multirow{3}{*}{Qwen2.5-14B} & \ \ Greedy Decoding & 74.81\% & 88.00\% & 57.00\% & \% & 73.27\%\\
% & \ \ Beam Search & 72.59\% & 57.00\% & 85.00\% & \% & \%\\
% & \ \ Reflection Window & \% & \% & \% & \% & \%\\\midrule
% % \multirow{3}{*}{Gemma-2-9b} & \ \ Greedy Decoding & & & & & \\
% % & \ \ Beam Search & & & & & \\
% % & \ \ Reflection Window & & & & & \\\midrule
% \multirow{3}{*}{Mistral-Nemo} & \ \ Greedy Decoding & 62.96\% & \textbf{74.00\%} & 45.00\% & & \\
% & \ \ Beam Search & 60.74\% & 65.00\% & \textbf{53.00\%} & & \\
% & \ \ Reflection Window & \textbf{64.44\%} & \textbf{74.00\%} & 48.00\% & & \\
% \bottomrule
% \end{tabular}
% \setlength{\abovecaptionskip}{0.1cm}
% \caption{ee}
% \label{tab:main_results}
% \vspace{-0.4cm}
% \end{table*}
% \begin{table*}[t]
% \small
% \centering
% \vspace{0.2cm}
% \caption{Accuracy of Reflection Window on Selected Subsets by Several LLMs}
% % \newcolumntype{g}{>{\columncolor{Gray}}c}
% \newcolumntype{g}{>{\columncolor[rgb]{0.9,0.9,1.0}}c}
% \renewcommand{\arraystretch}{1.0}
% \setlength{\tabcolsep}{2.6pt}
% \begin{tabular}{llcccg}
% \toprule
% \textbf{Model} & \ \ \textbf{Method} & \textbf{anatomy} & \textbf{computer\_security} & \textbf{global\_facts} & \textbf{Average} \\\midrule
% \multirow{3}{*}{Llama3.1-8B} & \ \ Greedy Decoding & 67.40\% & 73.00\% & 43.00\% & 61.79\% \\
% & \ \ Beam Search & \textbf{74.07\%} & \textbf{78.00\%} & 45.00\% & 66.57\% \\
% & \ \ Reflection Window & 72.59\% & 77.00\% & \textbf{50.00\%} & \textbf{67.16\%} \\\midrule
% \multirow{3}{*}{Phi-3-medium} & \ \ Greedy Decoding & 73.33\% & 70.00\% & 56.00\% & 66.44\% \\
% & \ \ Beam Search & 70.37\% & \textbf{78.00\%} & 58.00\% & 68.79\% \\
% & \ \ Reflection Window & \textbf{74.07\%} & 76.00\% & \textbf{62.00}\% & \textbf{70.69\%} \\\midrule
% % \multirow{3}{*}{Phi-3.5-mini} & \ \ Greedy & & & & & \\
% % & \ \ Beam Search & & & & & \\
% % & \ \ Reflection Window & & & & & \\\midrule
% \multirow{3}{*}{Qwen2.5-14B} & \ \ Greedy Decoding & \textbf{74.81\%} & \textbf{88.00}\% & 57.00\% & 73.27\% \\
% & \ \ Beam Search & 72.59\% & 85.00\% & 57.00\% & 71.53\% \\
% & \ \ Reflection Window & 74.07\% & 85.00\% & \textbf{63.00\%} & \textbf{74.02\%} \\\midrule
% \multirow{3}{*}{Mistral-Nemo} & \ \ Greedy Decoding & 62.96\% & \textbf{74.00\%} & 45.00\% & 60.90\% \\
% & \ \ Beam Search & 60.74\% & 65.00\% & \textbf{53.00\%} & 59.70\% \\
% & \ \ Reflection Window & \textbf{64.44\%} & \textbf{74.00\%} & 48.00\% & \textbf{62.39\%} \\
% \bottomrule
% \end{tabular}
% \setlength{\abovecaptionskip}{0.1cm}
% \label{tab:summary_mmlu}
% % \vspace{-0.4cm}
% \end{table*}
\end{comment}
\begingroup
\renewcommand{\arraystretch}{1.1}
\begin{table}[t]
\caption{Regeneration metrics on MMLU with Llama3.1-8B}
\label{tab:regeneration_metric_mmlu}
\centering
\footnotesize
\begin{tabular}{L{0.14}C{0.18}C{0.16}}
\toprule
\textbf{Category} & \textbf{ReGen. Ratio (\%)} & \textbf{ReGen. Call} \\
\midrule
STEM & 3.50 & 3.15 \\
Humanities & 5.04 & 4.27 \\
Social Sciences & 4.82 & 3.84 \\
Others & 5.54 & 4.31 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
% \begin{table*}[h]
% \centering
% \caption{Regeneration Measurement by Category on MMLU.}
% \label{tab:regeneration_metric_mmlu}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Category} & \textbf{Regeneration Rate (\%)} & \textbf{Num of Regeneration} & \textbf{Num of All Tokens} \\
% \midrule
% Humanities & 5.04 & 4.27 & 336.69 \\
% Other & 5.54 & 4.31 & 305.93 \\
% STEM & 3.50 & 3.15 & 398.55 \\
% Social Sciences & 4.82 & 3.84 & 316.29 \\
% \bottomrule
% \end{tabular}
% % \setlength{\abovecaptionskip}{-0.1cm}
% \end{table*}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{
Acc. by $\sigma$ (MMLU Social Sciences, Qwen2.5-7B)}
\label{tab:acc_mmlu_social_science_entropy_threthold}
\centering
\footnotesize
\begin{tabular}{C{0.095}C{0.07}C{0.07}C{0.07}C{0.07}C{0.07}}
\toprule
$\sigma$ & \textbf{0.1} & \textbf{0.25} & \textbf{0.5} & \textbf{0.75} & \textbf{1.0} \\
\midrule
Acc. ($\%$) & 79.40 & \textbf{80.31} & 79.82 & 79.88 & 79.69 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Acc. by $d$ (MMLU Social Sciences, Qwen2.5-7B)}
\label{tab:acc_mmlu_social_science_window_size}
\centering
\footnotesize
\begin{tabular}{C{0.095}C{0.07}C{0.07}C{0.07}C{0.07}C{0.07}}
\toprule
$d$ & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6} \\
\midrule
Acc. ($\%$) & 79.66 & 79.75 & 79.82 & 79.62 & \textbf{79.88} \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
\section{Conclusion}
\looseness=-1
In this paper, we theoretically characterize one inherent shortcoming, among others, of the autoregressive decoding for text generation in LLMs.
In particular, we show that even when the optimality is defined in terms of the joint probability over all generated tokens, an oracle LLM can still potentially deviate from the globally optimal response of the same length.
To mitigate the sub-optimality of the autoregressive way of text generation, we propose an empirical approach guided by our theoretical characterization.
We incorporate a sliding reflection window and a pausing criterion so that refinement and generation can be performed interchangeably.
Our experimental results demonstrate that our reflection-window decoding strategy achieves significant improvement over regular decoding strategies in inference-intensive settings and maintains performance that is comparable, or even superior to, beam search while being more efficient.
\section*{Acknowledgments} % intentionally subsection
\looseness=-1
We would also like to acknowledge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program.
ZT is supported by the National Institute of Justice (NIJ) Graduate Research Fellowship, Award No. 15PNIJ-24-GG-01565-RESS.
\section*{Impact Statement}\label{sec:broader_impact}
\looseness=-1
In this paper, we theoretically characterize and empirically address the sub-optimality of the autoregressive decoding for text generation.
In particular, we propose a selective refinement framework and implement it with a sliding reflection window mechanism, enabling interchangeable refinement and generation as the decoding proceeds.
Our approach strikes a balance between efficiency and optimality.
There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
\newpage
\appendix
% --> supp
\begin{bibunit}
\onecolumn
\icmltitle{Supplement to ``Reflection-Window Decoding: \\
Text Generation with Selective Refinement''}
\startcontents[supp]
\renewcommand\contentsname{Table of Contents: Appendix}
\printcontents[supp]{l}{1}{\section*{\contentsname}\setcounter{tocdepth}{2}}
\listoftables
\newpage
\input{file/supp/proof.tex}
\input{file/supp/further_analyses.tex}
\input{file/supp/full_result_on_mmlu.tex}
\stopcontents[supp]
% \putbib
\end{bibunit}
\section{The Proof of Theoretical Result}
\begin{theorem}[Indication of Deviation from the Globally Optimal Length-$T$ Response]\label{supp:thm:small_conditional_prob}
% Suppose that the context length $N$ of LLM is sufficiently large.
Given the prompt $X_{\leq 0}$, when an oracle LLM (Assumption~\ref{aspt:oracle_LLM}) generates a stepwise optimal length-$T$ response which is not the globally optimal response with the same length, let $L \leq T$ denote the minimum length of prefix-sequence needed in order for such deviation to manifest itself (Assumptions \ref{aspt:strict_preference} and \ref{aspt:no_return_after_detour}).
Then, the deviation from the globally optimal response happens at some step $K < L$.
Furthermore, the conditional probability when generating the token $w_{v_L} \in \Vcal$ is strictly smaller than a positive number, which itself is strictly smaller than $1$, i.e.,
% In other words, the LLM will be relatively uncertain about
\begin{equation}
\begin{split}
& 1 > \epsilon_L > \max_{w \in \Vcal} g(X_L = w \mid X_{1:L-1} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[L - 1]}, X_{\leq 0}), \\
& \text{ where } \epsilon_L = \frac{f(X_{1:L} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0})}{f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})}.
\end{split}
\end{equation}
\end{theorem}
\begin{proof}
We first show that the deviation from the globally optimal response happens before step $L$.
Then, we show that the conditional probability when generating the token $w_{v_L}$ is bounded away from $1$.
By definition of oracle LLM (Assumption~\ref{aspt:oracle_LLM}), the advantage of the globally optimal response cannot manifest itself at $L = 1$ (even if the deviation happens at step $1$), i.e., $L > 1$.
Since the minimum length of prefix-sequence needed in order for the deviation of stepwise optimal response from the same-length globally optimal response to manifest is $L$, then the advantage of the globally optimal response is not manifested until step $L$.
Until step $L - 1$, in terms of the ground-truth conditional probability following the prompt $X_{\leq 0}$, prefix-sequences of the globally optimal response is \emph{not} strictly preferred compared to their same-length counterparts of the stepwise optimal response:
\begin{equation}\label{equ:before_manifest}
\small
\begin{split}
f(X_1 = w_{\widehat{\vbf}_T[1]} \mid X_{\leq 0}) & \geq f(X_1 = w_{\vbf_T^*[1]} \mid X_{\leq 0}), \\
f(X_{1:2} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \mid X_{\leq 0}) & \geq f(X_{1:2} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \mid X_{\leq 0}), \\
& \cdots \\
f(X_{1:L-1} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})
& \geq f(X_{1:L-1} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[L - 1]} \mid X_{\leq 0}).
\end{split}
\end{equation}
Starting from step $L$ and onwards (Assumption~\ref{aspt:no_return_after_detour}), prefix-sequences of the globally optimal response are strictly preferred compared to their counterparts of the stepwise optimal response:
\begin{equation}\label{equ:after_manifest}
\small
\begin{split}
f(X_{1:L} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} w_{\widehat{\vbf}_T[L]} \mid X_{\leq 0})
& < f(X_{1:L} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0}), \\
& \cdots \\
f(X_{1:T} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[T]} \mid X_{\leq 0})
& < f(X_{1:T} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[T]} \mid X_{\leq 0}).
\end{split}
\end{equation}
Assumption~\ref{aspt:strict_preference} specifies that for any two same-length but different sequences following the prompt $X_{\leq 0}$, there is a strict ordering between them.
Then, in order for the advantage of the globally optimal length-$T$ response to manifest, in terms of strict preferences staring from the length-$L$ prefix-sequence (\eqnref{equ:after_manifest}), there is \emph{at least one} strict preference of the prefix-sequence of stepwise optimal response over its globally optimal counterpart before step $L$.
In other words, there is at least one step $K \in [1, L-1]$ such that a strict preference (``$>$'' instead of ``$\geq$'') is present in \eqnref{equ:before_manifest}:
\begin{equation}\label{equ:step_of_strict_preference}
\small
\begin{split}
% \exists K \in [1, L-1], ~
f(X_{1:K} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[K]} \mid X_{\leq 0})
& > f(X_{1:K} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[K]} \mid X_{\leq 0}).
\end{split}
\end{equation}
In order to see why this is the case, consider the opposite scenario where there is \emph{no} strict preference in \eqnref{equ:before_manifest}.
Under Assumption~\ref{aspt:strict_preference}, the comparison between prefix-sequences is either strict preference (they are different) or exactly the same (identical sequences).
If there is no strict preference in \eqnref{equ:before_manifest}, then for all $t \in [1, L-1]$, $w_{\widehat{\vbf}_T[t]} = w_{\vbf_T^*[t]}$, i.e., the first $L-1$ tokens in the stepwise optimal response are the length-$(L - 1)$ prefix of the globally optimal response.
If this is the case, the token generated at step $L$ has to deviate from the globally optimal response (since $L$ is the minimum length for the deviation to manifest) $w_{\widehat{\vbf}_T[L]} \neq w_{\vbf_T^*[L]}$:
\begin{equation}\label{equ:opposite_preference}
\small
\begin{split}
& f(X_{1:L} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} w_{\widehat{\vbf}_T[L]} \mid X_{\leq 0}) \\
\overset{(i)}{=} & g(X_L = w_{\widehat{\vbf}_T[L]} \mid X_{1:L-1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]}, X_{\leq 0})
\cdot f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0}) \\
\overset{(ii)}{=} & g(X_L = w_{\widehat{\vbf}_T[L]} \mid X_{1:L-1} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]}, X_{\leq 0})
\cdot f(X_{1:L - 1} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} \mid X_{\leq 0}) \\
\overset{(iii)}{>} & g(X_L = w_{\vbf_T^*[L]} \mid X_{1:L-1} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]}, X_{\leq 0})
\cdot f(X_{1:L - 1} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} \mid X_{\leq 0}) \\
\overset{(iv)}{=} & f(X_{1:L} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0}),
\end{split}
\end{equation}
where $(i)$ and $(iv)$ follow Assumption~\ref{aspt:oracle_LLM}, $(ii)$ corresponds to the setting in this opposite scenario, and $(iii)$ follows Definition~\ref{def:stepwise_sequence} and that $w_{\widehat{\vbf}_T[L]} \neq w_{\vbf_T^*[L]}$.
This preference relation in \eqnref{equ:opposite_preference} contradicts with that in \eqnref{equ:after_manifest}, and therefore, \eqnref{equ:step_of_strict_preference} has to hold true.
Therefore, when the advantage of the globally optimal response does not manifest itself until step $L$, the stepwise optimal response deviates from the globally optimal counterpart at some step $K < L$, and that under Assumption~\ref{aspt:strict_preference}, the following strict preference relations hold true:
\begin{equation}\label{equ:before_manifest_strict}
\small
\begin{split}
f(X_{1:K} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[K]} \mid X_{\leq 0})
& > f(X_{1:K} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[K]} \mid X_{\leq 0}), \\
& \cdots \\
f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} w_{\widehat{\vbf}_T[2]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})
& > f(X_{1:L - 1} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[L - 1]} \mid X_{\leq 0}).
\end{split}
\end{equation}
This, together with \eqnref{equ:after_manifest} and Assumption~\ref{aspt:oracle_LLM}, indicates that:
\begin{equation}
\small
\begin{split}
& g(X_L = w_{\widehat{\vbf}_T[L]} \mid X_{1:L-1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]}, X_{\leq 0}) \\
\overset{(i)}{=} & \frac{f(X_{1:L} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} w_{\widehat{\vbf}_T[L]} \mid X_{\leq 0})}{f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})} \\
\overset{(ii)}{<} & \frac{f(X_{1:L} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0})}{f(X_{1:L - 1} = w_{\widehat{\vbf}_T[1]} \ldots w_{\widehat{\vbf}_T[L - 1]} \mid X_{\leq 0})} = \epsilon_L \\
\overset{(iii)}{<} & \frac{f(X_{1:L} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]} w_{\vbf_T^*[L]} \mid X_{\leq 0})}{f(X_{1:L - 1} = w_{\vbf_T^*[1]} w_{\vbf_T^*[2]} \ldots w_{\vbf_T^*[L - 1]} \mid X_{\leq 0})} \\
\overset{(iv)}{=} & g(X_L = w_{\vbf_T^*[L]} \mid X_{1:L-1} = w_{\vbf_T^*[1]} \ldots w_{\vbf_T^*[L - 1]}, X_{\leq 0}) \leq 1,
\end{split}
\end{equation}
where $(i)$ and $(iv)$ follow Assumption~\ref{aspt:oracle_LLM}, $(ii)$ follows \eqnref{equ:after_manifest}, and $(iii)$ follows \eqnref{equ:before_manifest_strict}.
Therefore, the conditional probability of generating any $w_{v_L}$ is strictly smaller than a positive number $\epsilon_L$, which is further strictly smaller than a positive number upper-bounded by $1$.
\end{proof}
% \section{Performance across all subsets of MMLU}
% % \input{file/tables/table6}
% % \appendix
% \renewcommand{\thetable}{A1}
% \begin{table*}[t]
% \centering
% \caption{Model Accuracy by Subject Category on MMLU Using Different Methods}
% \label{tab:fullresult}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Category} & \textbf{Greedy (\%)} & \textbf{Beam Search (\%)} & \textbf{Ours (\%)} \\
% \midrule
% abstract\_algebra & 47.00 & 46.00 & 47.00 \\
% anatomy & 67.40 & 74.07 & 72.59 \\
% astronomy & 80.92 & 73.02 & 80.26 \\
% business\_ethics & 66.00 & 68.00 & 64.00 \\
% clinical\_knowledge & 76.22 & 76.60 & 78.11 \\
% college\_biology & 83.33 & 80.55 & 79.86 \\
% college\_chemistry & 56.00 & 53.00 & 57.00 \\
% college\_computer\_science & 55.00 & 63.00 & 58.00 \\
% college\_mathematics & 35.00 & 47.00 & 36.00 \\
% college\_medicine & 69.36 & 65.89 & 68.78 \\
% college\_physics & 64.70 & 69.61 & 61.76 \\
% computer\_security & 73.00 & 78.00 & 77.00 \\
% conceptual\_physics & 68.93 & 69.78 & 69.78 \\
% econometrics & 56.14 & 58.77 & 57.89 \\
% electrical\_engineering & 58.62 & 64.82 & 61.37 \\
% elementary\_mathematics & 88.09 & 88.62 & 89.41 \\
% formal\_logic & 50.00 & 50.79 & 51.58 \\
% global\_facts & 43.00 & 45.00 & 50.00 \\
% high\_school\_biology & 79.03 & 80.96 & 80.32 \\
% high\_school\_chemistry & 62.07 & 67.98 & 63.05 \\
% high\_school\_computer\_science & 81.00 & 79.00 & 79.00 \\
% high\_school\_european\_history & 76.36 & 75.15 & 75.15 \\
% high\_school\_geography & 78.79 & 80.81 & 78.79 \\
% high\_school\_government\_and\_politics & 88.60 & 86.53 & 88.08 \\
% high\_school\_macroeconomics & 69.74 & 74.62 & 68.97 \\
% high\_school\_mathematics & 64.44 & 68.89 & 68.52 \\
% high\_school\_microeconomics & 75.63 & 76.05 & 77.33 \\
% high\_school\_physics & 54.97 & 54.97 & 57.62 \\
% high\_school\_psychology & 85.32 & 88.07 & 86.24 \\
% high\_school\_statistics & 62.96 & 60.19 & 62.96 \\
% high\_school\_us\_history & 83.33 & 80.88 & 80.39 \\
% high\_school\_world\_history & 78.90 & 80.59 & 76.79 \\
% human\_aging & 63.68 & 66.82 & 62.78 \\
% human\_sexuality & 52.67 & 60.31 & 53.44 \\
% international\_law & 80.17 & 80.17 & 77.69 \\
% jurisprudence & 72.22 & 74.07 & 76.85 \\
% logical\_fallacies & 78.53 & 78.53 & 79.14 \\
% machine\_learning & 56.25 & 61.61 & 53.57 \\
% nutrition & 73.86 & 75.82 & 76.47 \\
% us\_foreign\_policy & 84.00 & 80.00 & 81.00 \\
% public\_relations & 64.55 & 66.36 & 65.45 \\
% world\_religions & 83.04 & 79.53 & 82.46 \\
% professional\_accounting & 51.42 & 53.98 & 51.06 \\
% security\_studies & 62.45 & 66.12 & 63.27 \\
% professional\_law & 49.86 & 51.54 & 53.50 \\
% professional\_psychology & 68.63 & 68.95 & 68.46 \\
% prehistory & 70.37 & 72.22 & 70.37 \\
% professional\_medicine & 79.41 & 79.78 & 79.78 \\
% sociology & 74.63 & 77.11 & 73.63 \\
% philosophy & 73.06 & 68.49 & 71.40 \\
% virology & 48.19 & 47.59 & 49.40 \\
% marketing & 88.03 & 84.18 & 87.17 \\
% medical\_genetics & 77.00 & 79.00 & 75.00 \\
% miscellaneous & 86.97 & 87.73 & 87.61 \\
% moral\_disputes & 67.34 & 66.76 & 64.16 \\
% moral\_scenarios & 45.47 & 48.72 & 47.03 \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \color{blue} % begin add-on
% \section{Parameter Analysis \& Qualitative Results}
% \begin{table*}[h]
% \centering
% \caption{Performance Comparison of Different Methods by Mistral-Nemo on MMLU Social Science}
% \label{tab:mistral-social-science}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Category} & \textbf{Greedy} & \textbf{Beam Search} & \textbf{Ours} \\
% \midrule
% Econometrics & 53.51 & 56.14 & \textbf{59.65} \\
% High School Geography & 78.79 & 77.78 & \textbf{79.80} \\
% High School Government and Politics & 87.56 & 87.05 & \textbf{88.08} \\
% High School Macroeconomics & \textbf{74.61} & 74.09 & 73.83 \\
% High School Microeconomics & 71.24 & \textbf{73.39} & \textbf{73.39} \\
% High School Psychology & \textbf{67.16} & 66.01 & 66.50 \\
% Human Sexuality & 66.41 & 64.89 & \textbf{70.99} \\
% Professional Psychology & \textbf{67.16} & 66.01 & 66.50 \\
% Public Relations & 59.26 & \textbf{63.89} & 56.48 \\
% Sociology & 76.62 & \textbf{82.09} & 74.63 \\
% Security Studies & 57.38 & 54.10 & \textbf{59.02} \\
% US Foreign Policy & \textbf{77.00} & 76.00 & \textbf{77.00} \\
% \midrule
% Macro Average (\%) & 71.93 & 72.10 & \textbf{72.20} \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \subsection{Analysis on Window Size}
% \begin{table*}[h]
% \centering
% \caption{Performance Comparison of Different Window Sizes with Llama3.1-8B on MT-Bench}
% \label{tab:llama3-performance}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Method} & \textbf{Rating 1} & \textbf{Rating 2} & \textbf{Overall Mean} \\
% \midrule
% $d=2$ & 8.29 & 7.09 & 7.69 \\
% $d=3$ & 8.35 & 7.51 & 7.93 \\
% $d=4$ & \textbf{8.36} & 7.42 & 7.89 \\
% $d=5$ & 8.31 & \textbf{7.62} &\textbf{7.97} \\
% Greedy Decoding & 8.28 & 7.49 & 7.88 \\
% Beam Search & 8.07 & 7.19 & 7.63 \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \begin{table*}[h]
% \centering
% \caption{Performance Comparison of Different Window Sizes with Mistral-Nemo on MT-Bench}
% \label{tab:mistral-performance}
% \begin{tabular}{lccc}
% \toprule
% \textbf{Method} & \textbf{Rating 1} & \textbf{Rating 2} & \textbf{Overall Mean} \\
% \midrule
% $d=2$ & 8.38 & 7.28 & 7.82 \\
% $d=3$ & \textbf{8.44} & 7.42 & \textbf{7.93} \\
% $d=4$ & 8.28 & 7.41 & 7.84 \\
% Greedy Decoding & 8.38 & 7.29 & 7.83 \\
% Beam Search & 8.32 & \textbf{7.49} & 7.91 \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \begin{table*}[h]
% \centering
% \caption{Performance Comparison of Different Window Sizes with Qwen on MMLU Social Science Tasks}
% \label{tab:qwen-social-science}
% \begin{tabular}{lccccc}
% \toprule
% \textbf{Task} & \textbf{d=2} & \textbf{d=3} & \textbf{d=4} & \textbf{d=5} & \textbf{d=6} \\
% \midrule
% Econometrics & 62.28 & 62.28 & \textbf{64.91} & 64.04 & \textbf{64.91} \\
% High School Geography & 85.86 & 84.34 & 86.36 & \textbf{87.37} & 85.86 \\
% High School Government and Politics & \textbf{93.26} & \textbf{93.26} & 92.23 & 91.19 & 92.23 \\
% High School Macroeconomics & 75.90 & \textbf{76.15} & 75.13 & \textbf{76.15} & 75.64 \\
% High School Microeconomics & 83.61 & \textbf{84.03} & 83.61 & 82.77 & 83.19 \\
% High School Psychology & 87.89 & 88.07 & 88.07 & 88.07 & \textbf{88.26} \\
% Human Sexuality & 77.86 & 75.57 & 78.63 & \textbf{79.39} & 77.86 \\
% Professional Psychology & \textbf{73.86} & 73.37 & 73.20 & 72.88 & 73.37 \\
% Public Relations & 68.18 & \textbf{70.00} & \textbf{70.00} & 65.45 & 68.18 \\
% Sociology & 71.02 & 73.06 & \textbf{73.47} & 72.24 & 73.06 \\
% Security Studies & 83.08 & 83.58 & 83.08 & 84.08 & \textbf{84.58} \\
% US Foreign Policy & 86.00 & 86.00 & 86.00 & 86.00 & \textbf{88.00} \\
% \midrule
% Macro Average & 79.66 & 79.75 & 79.82 & 79.62 & \textbf{79.88} \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \subsection{Analysis on Threshold}
% \begin{table*}[h]
% \centering
% \caption{Performance Comparison of Different $\sigma$ Values on MMLU Social Science Tasks}
% \label{tab:sigma-social-science}
% \begin{tabular}{lccccc}
% \toprule
% \textbf{Task} & \textbf{$\sigma$=0.1} & \textbf{$\sigma$=0.25} & \textbf{$\sigma$=0.5} & \textbf{$\sigma$=0.75} & \textbf{$\sigma$=1.0} \\
% \midrule
% Econometrics & 62.28 & \textbf{64.91} & \textbf{64.91} & \textbf{64.91} & \textbf{64.91} \\
% High School Geography & \textbf{92.23} & 91.71 & \textbf{92.23} & \textbf{92.23} & 91.19 \\
% High School Government and Politics & \textbf{92.23} & 91.71 & \textbf{92.23} & \textbf{92.23} & 91.19 \\
% High School Macroeconomics & 75.13 & \textbf{76.67} & 75.13 & 75.90 & 75.90 \\
% High School Microeconomics & \textbf{84.45} & \textbf{84.45} & 83.61 & 83.19 & 83.61 \\
% High School Psychology & 87.52 & \textbf{88.44} & 88.07 & 88.26 & 88.07 \\
% Human Sexuality & 74.05 & 77.86 & \textbf{78.63} & 77.10 & 77.10 \\
% Professional Psychology & 73.20 & \textbf{74.35} & 73.20 & 73.20 & 73.04 \\
% Public Relations & 69.09 & \textbf{70.00} & \textbf{70.00} & 67.27 & 67.27 \\
% Sociology & 84.58 & \textbf{85.07} & 83.08 & 84.58 & 84.58 \\
% Security Studies & 72.24 & 71.43 & \textbf{73.47} & 72.24 & 72.24 \\
% US Foreign Policy & \textbf{86.00} & 85.00 & 85.00 & 85.00 & \textbf{87.00} \\
% \midrule
% Macro Average & 79.40 & \textbf{80.31} & 79.82 & 79.88 & 79.69 \\
% \bottomrule
% \end{tabular}
% \end{table*}
% \subsection{Concrete Example of MT-Bench}
% \color{black}
\section{Additional Results and Analyses}
In this section, we present additional results and further discussions on influences from hyperparameters.
We also provide concrete examples demonstrating the process and overall performance of our reflection-window decoding.
\subsection{Analysis on Window Size $d$}
We conduct extensive experiments on MT-Bench to analyze the impact of window size using both Mistral-Nemo (Table~\ref{tab:mistral_mtbench_per_d}) and Llama3.1-8B (Table~\ref{tab:llama31_mtbench_per_d}), with a fixed pausing criterion with $\sigma=0.5$.
These GPT-4o evaluator scores on MT-Bench provide additional evidence that our approach consistently outperforms traditional decoding methods.
For Mistral-Nemo, the optimal performance is achieved at $d=3$ with an overall score of 7.93, surpassing both greedy decoding and beam search.
For Llama3.1-8B, our method consistently outperforms both greedy decoding and beam search across different window sizes, with $d=5$ achieving the best overall performance.
While $d=4$ may not always yield the best result, it demonstrates robust performance across both models and serves as a reliable default setting.
We further evaluate how different window sizes affect the performance on MMLU social science tasks using Qwen2.5-7B.
The results are presented in Table~\ref{tab:qwen_mmlu_social_science_per_d}.
%, confirming its effectiveness beyond the human evaluations presented in the main text.
\begin{table*}[t]
\centering
\caption{Performance across different window sizes $d$ on MT-Bench with Mistral-Nemo}
\label{tab:mistral_mtbench_per_d}
\begin{tabular}{lccc}
\toprule
\textbf{Decoding Method} & \textbf{Rating 1} & \textbf{Rating 2} & \textbf{Overall Mean} \\
\midrule
Reflection-Window ($d=2$) & 8.38 & 7.28 & 7.82 \\
Reflection-Window ($d=3$) & \textbf{8.44} & 7.42 & \textbf{7.93} \\
Reflection-Window ($d=4$) & 8.28 & 7.41 & 7.84 \\
Greedy Decoding & 8.38 & 7.29 & 7.83 \\
Beam Search & 8.32 & \textbf{7.49} & 7.91 \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[t]
\centering
\caption{Performance across different window sizes $d$ on MT-Bench with Llama3.1-8B}
\label{tab:llama31_mtbench_per_d}
\begin{tabular}{lccc}
\toprule
\textbf{Decoding Method} & \textbf{Rating 1} & \textbf{Rating 2} & \textbf{Overall Mean} \\
\midrule
Reflection-Window ($d=2$) & 8.29 & 7.09 & 7.69 \\
Reflection-Window ($d=3$) & 8.35 & 7.51 & 7.93 \\
Reflection-Window ($d=4$) & \textbf{8.36} & 7.42 & 7.89 \\
Reflection-Window ($d=5$) & 8.31 & \textbf{7.62} & \textbf{7.97} \\
Greedy Decoding & 8.28 & 7.49 & 7.88 \\
Beam Search & 8.07 & 7.19 & 7.63 \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table*}[h]
\centering
\caption{Accuracy ($\%$) across different window sizes $d$ on MMLU Social Sciences with Qwen2.5-7B}
\label{tab:qwen_mmlu_social_science_per_d}
\begin{tabular}{lccccc}
\toprule
\textbf{Subject} & $d=2$ & $d=3$ & $d=4$ & $d=5$ & $d=6$ \\
\midrule
Econometrics & 62.28 & 62.28 & \textbf{64.91} & 64.04 & \textbf{64.91} \\
High School Geography & 85.86 & 84.34 & 86.36 & \textbf{87.37} & 85.86 \\
High School Government and Politics & \textbf{93.26} & \textbf{93.26} & 92.23 & 91.19 & 92.23 \\
High School Macroeconomics & 75.90 & \textbf{76.15} & 75.13 & \textbf{76.15} & 75.64 \\
High School Microeconomics & 83.61 & \textbf{84.03} & 83.61 & 82.77 & 83.19 \\
High School Psychology & 87.89 & 88.07 & 88.07 & 88.07 & \textbf{88.26} \\
Human Sexuality & 77.86 & 75.57 & 78.63 & \textbf{79.39} & 77.86 \\
Professional Psychology & \textbf{73.86} & 73.37 & 73.20 & 72.88 & 73.37 \\
Public Relations & 68.18 & \textbf{70.00} & \textbf{70.00} & 65.45 & 68.18 \\
Sociology & 71.02 & 73.06 & \textbf{73.47} & 72.24 & 73.06 \\
Security Studies & 83.08 & 83.58 & 83.08 & 84.08 & \textbf{84.58} \\
US Foreign Policy & 86.00 & 86.00 & 86.00 & 86.00 & \textbf{88.00} \\
\midrule
Macro Average & 79.66 & 79.75 & 79.82 & 79.62 & \textbf{79.88} \\
\bottomrule
\end{tabular}
\end{table*}
\clearpage
\subsection{Analysis on Threshold $\sigma$}
We investigate the impact of threshold $\sigma$ on MMLU social science subjects using Qwen2.5-7B with a fixed window size $d=4$.
The detailed results are presented in Table~\ref{tab:qwen_mmlu_social_science_per_sigma}.
% demonstrate that our method performs robustly across $\sigma$ values ranging from 0.25 to 0.75, with $\sigma=0.25$ achieving the highest macro average of 80.31\%. While our default setting of $\sigma=0.5$ is not the absolute best in this specific experiment, it maintains strong performance and shows consistent improvements across most tasks, suggesting it serves as a reliable default configuration for general use.
% \renewcommand{\thetable}{A5}
\begin{table*}[h]
\centering
\caption{Accuracy ($\%$) across different entropy thresholds $\sigma$ on MMLU Social Sciences with Qwen2.5-7B}
\label{tab:qwen_mmlu_social_science_per_sigma}
\begin{tabular}{lccccc}
\toprule
\textbf{Subject} & $\sigma=0.1$ & $\sigma=0.25$ & $\sigma=0.5$ & $\sigma=0.75$ & $\sigma=1.0$ \\
\midrule
Econometrics & 62.28 & \textbf{64.91} & \textbf{64.91} & \textbf{64.91} & \textbf{64.91} \\
High School Geography & \textbf{92.23} & 91.71 & \textbf{92.23} & \textbf{92.23} & 91.19 \\
High School Government and Politics & \textbf{92.23} & 91.71 & \textbf{92.23} & \textbf{92.23} & 91.19 \\
High School Macroeconomics & 75.13 & \textbf{76.67} & 75.13 & 75.90 & 75.90 \\
High School Microeconomics & \textbf{84.45} & \textbf{84.45} & 83.61 & 83.19 & 83.61 \\
High School Psychology & 87.52 & \textbf{88.44} & 88.07 & 88.26 & 88.07 \\
Human Sexuality & 74.05 & 77.86 & \textbf{78.63} & 77.10 & 77.10 \\
Professional Psychology & 73.20 & \textbf{74.35} & 73.20 & 73.20 & 73.04 \\
Public Relations & 69.09 & \textbf{70.00} & \textbf{70.00} & 67.27 & 67.27 \\
Sociology & 84.58 & \textbf{85.07} & 83.08 & 84.58 & 84.58 \\
Security Studies & 72.24 & 71.43 & \textbf{73.47} & 72.24 & 72.24 \\
US Foreign Policy & \textbf{86.00} & 85.00 & 85.00 & 85.00 & \textbf{87.00} \\
\midrule
Macro Average & 79.40 & \textbf{80.31} & 79.82 & 79.88 & 79.69 \\
\bottomrule
\end{tabular}
\end{table*}
\subsection{Analysis on Regeneration Ratio}
To further understand the computational efficiency of our method, we analyze the regeneration ratio under different window size settings.
We select six college-level subject categories from the MMLU test set (including biology, chemistry, computer science, mathematics, medicine, and physics) for analysis, and conduct experiments with the Llama3.1-8B model with a threshold of $\sigma=0.5$.
We consider the window size $d$ as the key hyperparameter, because it directly influences the regeneration ratio, which is calculate by the product of the times criterion get triggered (the regeneration call) and the window size ($d$), divided by the total length of final response.
As shown in Table \ref{tab:regeneration_ratio} and Figure~\ref{fig:regeneration_ratio}, as the window size increases from 2 to 4, the average regeneration ratio shows a clear downward trend, decreasing from $9.60\%$ to $3.70\%$.
The trend indicates that larger window sizes lead to a faster decrease in the number of modifications needed.
Notably, across all settings, the regeneration ratio remains below $15\%$, suggesting that our method maintains comparable computational workload as greedy decoding for the majority of the time.
These results demonstrate the efficiency of our approach, since it only invoke beam search to find optimal approximations for sub-sequences when necessary, while maintaining the overall efficiency comparable to greedy decoding.
\begin{table}[h]
\centering
\caption{Average regeneration ratio by window sizes $d$ on MMLU college-level subjects with Llama3.1-8B}
\label{tab:regeneration_ratio}
\begin{tabular}{lccc}
\toprule
Window Size $d$ & 2 & 3 & 4 \\
\midrule
Average Regeneration Ratio ($\%$) & 9.60 & 6.02 & \textbf{3.70} \\
\bottomrule
\end{tabular}
\end{table}
% \renewcommand{\thefigure}{A6}
\begin{figure}[h]
\centering
\includegraphics[height=35ex]{file/figures/regeneration_ratio.pdf}
\caption{Regeneration ratio on MMLU college-level subjects with Llama3.1-8B}
\label{fig:regeneration_ratio}
\end{figure}
\newpage
\subsection{Demonstrative Examples on MT-Bench}
In this subsection, we present three examples selected from MT-Bench to illustrate the generation process in our reflection-window decoding.
The first example is the 28th test sample from MT-Bench.
As is shown in Figure~\ref{fig:demo1}, the pausing criterion gets triggered when \textit{``part of the car"} was generated.
The part gets refined and replaced by \textit{``specific part, whereas"} which leads to a more flexible and natural answer.
Figure~\ref{fig:demo2} and Figure~\ref{fig:demo3} are selected from the 29th and 7th test sample of MT-Bench, respectively.
% % \renewcommand{\thefigure}{A7}
% \begin{figure}[h]
% \centering
% \includegraphics[width=1.0\textwidth]{file/figures/Demo_1.pdf} % Adjust the width as needed
% \caption{Test Sample \#28} % Caption for the figure
% \label{fig:demo1} % Label for referencing the figure in the text
% \end{figure}
% % \renewcommand{\thefigure}{A8}
% \begin{figure}[t]
% \centering
% \includegraphics[width=1.0\textwidth]{file/figures/demo_2.pdf} % Adjust the width as needed
% \caption{Test Sample \#29} % Caption for the figure
% \label{fig:demo2} % Label for referencing the figure in the text
% \end{figure}
% % \renewcommand{\thefigure}{A9}
% \begin{figure}[t]
% \centering
% \includegraphics[width=1.0\textwidth]{file/figures/demo_3.pdf} % Adjust the width as needed
% \caption{Test Sample \#7} % Caption for the figure
% \label{fig:demo3} % Label for referencing the figure in the text
% \end{figure}
% \color{black}
\begin{figure}[h]
\centering
\begin{tcolorbox}[decoding, colback=questiongray, title=Question]
\textbf{Which word does not belong with the others?}
tyre, steering wheel, car, engine
\end{tcolorbox}
\vspace{2ex}
% Reflection Window
\begin{tcolorbox}[decoding, colback=reflectiongreen, title=Reflection Window]
The word that does not belong with the others is ``tyre''.
The other three words--``steering wheel'', ``car'', and ``engine''--are all related to a vehicle, specifically a car.
A ``tyre'', on the other hand, is a component of a vehicle, but it's a \textbf{\sout{part of the car}} \textcolor{red}{\textbf{specific part, whereas}} the other three words are more general terms related to the vehicle as a whole.
However, if I had to choose one word that stands out as not belonging, I would say ``steering wheel''. The other three words--``car'', ``engine'', and ``tyre''--are all nouns that refer to parts or aspects of a vehicle, whereas ``steering wheel'' is a specific component of a vehicle.
But if I had to choose one word that is the least related to the others, I would say ``car''. The other three words--``steering wheel'', ``engine'', and ``tyre''--are all components or parts of a vehicle, whereas ``car'' is the vehicle itself.
So, it really depends on how you look at it!
\end{tcolorbox}
\vspace{2ex}
% Greedy Decoding
\begin{tcolorbox}[decoding, colback=greedyblue, title=Greedy Decoding]
The word that does not belong with the others is ``tyre''.
The other three words--``steering wheel'', ``car'', and ``engine''--are all related to a vehicle, specifically a car.
A ``tyre'', on the other hand, is a component of a vehicle, but it's a \hlyellow{part of the car}, rather than the car itself.
\end{tcolorbox}
\vspace{2ex}
% Beam Search
\begin{tcolorbox}[decoding, colback=beamyellow, title=Beam Search]
The word that does not belong with the others is ``tyre''.
The other three words--``steering wheel'', ``car'', and ``engine''--are all related to a vehicle, specifically a car.
A ``tyre'', on the other hand, is a component of a vehicle, but it's not a vehicle itself.
\end{tcolorbox}
\caption{Test Sample \#28}
\label{fig:demo1}
\end{figure}
\begin{figure}[h]
\centering
\begin{tcolorbox}[decoding, colback=questiongray, title=Question]
\textbf{Could you write a captivating short story beginning with the sentence:} \\ \textit{The old abandoned house at the end of the street held a secret that no one had ever discovered.}
\end{tcolorbox}
\vspace{2ex}
% Reflection Window
\begin{tcolorbox}[decoding, colback=reflectiongreen, title=Reflection Window]
Here's a captivating short story for you:
The old abandoned house at the end of the street held a secret that no one had ever discovered\ldots
\ldots(omitted)
Suddenly, Emily heard a faint whispering in her ear. It was a soft, raspy voice that seemed to be \textbf{\sout{speaking an ancient}} \textcolor{red}{\textbf{coming from all around}} her. ``Welcome, Emily,'' it said. ``I've been waiting for you.''
\ldots(omitted)
From that day on, Emily returned to the old house again and again, learning the secrets of the nexus point and the magic that lay within. And as she did, she discovered that the house was not just a place -- it was a doorway to a whole new world of wonder and possibility.
\end{tcolorbox}
\vspace{2ex}
% Greedy Decoding
\begin{tcolorbox}[decoding, colback=greedyblue, title=Greedy Decoding]
Here's a captivating short story for you:
The old abandoned house at the end of the street held a secret that no one had ever discovered\ldots
\ldots
Suddenly, Emily heard a faint whispering in her ear. It was a soft, raspy voice that seemed to be \hlyellow{speaking an ancient} language she couldn't quite understand. But as she listened more closely, she began to pick out words -- words that seemed to be calling to her, drawing her deeper into the heart of the house.
\ldots(omitted)
But as she looked up, she saw that she was no longer alone. A figure stood in the doorway, watching her with eyes that seemed to hold a deep sadness and a hint of longing. And as Emily looked into those eyes, she knew that she had stumbled upon a secret that was not just about the house, but about herself -- a secret that would change her life forever.
\end{tcolorbox}
\vspace{2ex}
% Beam Search
\begin{tcolorbox}[decoding, colback=beamyellow, title=Beam Search]
Here's a captivating short story for you:
The old abandoned house at the end of the street held a secret that no one had ever discovered. For as long as anyone could remember, the house had stood vacant, its grandeur and beauty slowly being consumed by the passing of time.
\ldots (omitted)
It was then that Emily realized that she was not just a curious adventurer, but a key player in a much larger game. The old abandoned house at the end of the street was not just a secret, but a doorway to a world of wonder and discovery, and Emily was about to embark on a journey that would change her life forever.
\end{tcolorbox}
\caption{Test Sample \#29}
\label{fig:demo2}
\end{figure}
\begin{figure}[t]
\centering
\begin{tcolorbox}[decoding, colback=questiongray, title=Question]
One morning after sunrise, Suresh was standing facing a pole. The shadow of the pole fell exactly to his right. Can you tell me the direction towards which the shadow was pointing--east, south, west, or north? \ldots
\textbf{To which direction was Suresh facing? How do you solve this?}
\end{tcolorbox}
\vspace{2ex}
% Reflection Window
\begin{tcolorbox}[decoding, colback=reflectiongreen, title=Reflection Window]
To determine the direction Suresh was facing, we need to consider the position of the sun and the direction of the shadow.
Since the \textbf{\sout{shadow of the pole}} \textcolor{red}{\textbf{sun is in the east}} and its rays cast shadows to the west, and the shadow of the pole fell to Suresh's right, we can infer that Suresh was facing \textbf{south}. This is because, when the sun is in the east, the shadows fall to the west, and if the shadow falls to the right, it means Suresh was facing south.
\end{tcolorbox}
\vspace{2ex}
% Greedy Decoding
\begin{tcolorbox}[decoding, colback=greedyblue, title=Greedy Decoding]
To determine the direction in which Suresh was facing, we need to consider the position of the sun and the direction of the shadow.
Since the \hlyellow{shadow of the pole} fell to Suresh's right, we can infer that the sun is in the east. This is because the sun's rays are coming from the east, and the shadows are cast in the opposite direction.
Now, since the shadow fell to Suresh's right, we can conclude that Suresh was facing the \textbf{west}. This is because the shadow fell to his right, which means he was facing the opposite direction, i.e., the west.
Therefore, the direction in which Suresh was facing is \textbf{west}.
\end{tcolorbox}
\vspace{2ex}
% Beam Search
\begin{tcolorbox}[decoding, colback=beamyellow, title=Beam Search]
Since Suresh was standing facing a pole and the shadow of the pole fell exactly to his right, we can deduce the direction of the shadow based on the position of the sun at sunrise.
1. At sunrise, the sun rises in the east.
2. Since Suresh was facing the pole and the shadow of the pole fell to his right, it means the shadow was pointing in a direction perpendicular to his line of sight.
3. Given that the sun rises in the east, the shadow of the pole would fall to the right of Suresh's line of sight, which means the shadow would be pointing \textbf{west}.
Therefore, the direction towards which the shadow of the pole was pointing is \textbf{west}.
\end{tcolorbox}
\caption{Test Sample \#7}
\label{fig:demo3}
\end{figure}
\clearpage
\section{Performance Across All MMLU Subjects}
Due to the size of the table, the material is arranged in the one-table-per-page manner (starting from the next page).
\input{file/tables/compare_gd_bs_mmlu_phi3.tex}
\input{file/tables/compare_top_kp_mmlu_llama31.tex}
\input{file/tables/compare_top_kp_mmlu_mistral.tex}
\input{file/tables/compare_top_kp_mmlu_phi3.tex}
\input{file/tables/compare_top_kp_mmlu_qwen.tex}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Comparison among greedy decoding, beam search, and reflection-window decoding on MMLU with Phi3-Medium}
\label{tab:mmlu_subject_phi3}
\centering
\footnotesize
\begin{tabular}{L{0.35}C{0.15}C{0.15}C{0.15}}
\toprule
\textbf{Subject} & \textbf{Greedy Decoding} & \textbf{Beam Search} & \textbf{Reflection-Window} \\
\midrule
abstract\_algebra & 58.00 & 56.00 & 59.00 \\
anatomy & 73.33 & 74.81 & 72.59 \\
astronomy & 87.50 & 84.21 & 88.82 \\
business\_ethics & 80.00 & 74.00 & 79.00 \\
clinical\_knowledge & 84.91 & 83.40 & 84.53 \\
college\_biology & 86.81 & 88.19 & 88.19 \\
college\_chemistry & 55.00 & 59.00 & 58.00 \\
college\_computer\_science & 68.00 & 68.00 & 70.00 \\
college\_mathematics & 62.00 & 58.00 & 57.00 \\
college\_medicine & 78.61 & 75.14 & 75.72 \\
college\_physics & 77.45 & 73.53 & 80.39 \\
computer\_security & 80.00 & 75.00 & 79.00 \\
conceptual\_physics & 80.85 & 82.98 & 82.55 \\
econometrics & 59.65 & 57.89 & 61.40 \\
electrical\_engineering & 70.34 & 68.97 & 72.41 \\
elementary\_mathematics & 94.44 & 93.92 & 93.92 \\
formal\_logic & 61.11 & 53.17 & 61.11 \\
global\_facts & 62.00 & 64.00 & 63.00 \\
high\_school\_biology & 88.71 & 91.29 & 90.32 \\
high\_school\_chemistry & 78.33 & 75.37 & 77.34 \\
high\_school\_computer\_science & 87.00 & 87.00 & 88.00 \\
high\_school\_european\_history & 80.00 & 71.52 & 81.21 \\
high\_school\_geography & 88.89 & 88.89 & 88.38 \\
high\_school\_government\_and\_politics & 95.34 & 93.78 & 94.82 \\
high\_school\_macroeconomics & 84.10 & 82.56 & 86.15 \\
high\_school\_mathematics & 74.07 & 75.93 & 74.07 \\
high\_school\_microeconomics & 89.92 & 89.92 & 89.08 \\
high\_school\_physics & 66.89 & 73.51 & 69.54 \\
high\_school\_psychology & 91.56 & 91.01 & 91.74 \\
high\_school\_statistics & 78.24 & 77.78 & 78.70 \\
high\_school\_us\_history & 85.78 & 77.94 & 85.78 \\
high\_school\_world\_history & 85.23 & 75.95 & 83.12 \\
human\_aging & 74.44 & 73.54 & 71.75 \\
human\_sexuality & 83.21 & 81.68 & 79.39 \\
international\_law & 85.12 & 85.95 & 81.82 \\
jurisprudence & 85.19 & 84.26 & 85.19 \\
logical\_fallacies & 84.66 & 85.89 & 85.89 \\
machine\_learning & 65.18 & 66.96 & 66.07 \\
management & 85.44 & 82.52 & 83.50 \\
marketing & 89.32 & 87.18 & 88.46 \\
medical\_genetics & 86.00 & 90.00 & 87.00 \\
miscellaneous & 91.57 & 91.95 & 91.19 \\
moral\_disputes & 78.03 & 75.72 & 78.32 \\
moral\_scenarios & 74.53 & 75.53 & 73.85 \\
nutrition & 82.68 & 81.70 & 83.01 \\
philosophy & 75.88 & 76.21 & 77.17 \\
prehistory & 84.57 & 84.88 & 83.95 \\
professional\_accounting & 75.89 & 74.11 & 75.18 \\
professional\_law & 56.98 & 50.72 & 57.43 \\
professional\_medicine & 77.94 & 63.60 & 79.04 \\
professional\_psychology & 80.07 & 78.27 & 79.08 \\
public\_relations & 68.18 & 67.27 & 66.36 \\
security\_studies & 73.06 & 73.47 & 73.47 \\
sociology & 87.06 & 85.57 & 86.07 \\
us\_foreign\_policy & 85.00 & 85.00 & 84.00 \\
virology & 52.41 & 53.61 & 53.01 \\
world\_religions & 84.21 & 83.63 & 86.55 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
\newpage
% \section{Performance across all subsets of MMLU}
% \renewcommand{\thetable}{A6}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Comparison between Top-$k$/Top-$p$ and reflection-window decoding on MMLU with Llama3.1-8B}
\label{tab:decoding_comparison_llama}
\centering
\footnotesize
\begin{tabular}{L{0.35}C{0.20}C{0.20}}
\toprule
\textbf{Subject} & \textbf{Top-$k$/Top-$p$} & \textbf{Reflection-Window} \\
\midrule
abstract\_algebra & 43.00 & 50.00 \\
anatomy & 70.37 & 68.15 \\
astronomy & 71.05 & 75.00 \\
business\_ethics & 69.00 & 71.00 \\
clinical\_knowledge & 75.09 & 74.72 \\
college\_biology & 80.56 & 81.94 \\
college\_chemistry & 54.00 & 50.00 \\
college\_computer\_science & 58.00 & 47.00 \\
college\_mathematics & 38.00 & 45.00 \\
college\_medicine & 68.79 & 70.52 \\
college\_physics & 49.02 & 65.69 \\
computer\_security & 69.00 & 73.00 \\
conceptual\_physics & 64.68 & 66.81 \\
econometrics & 47.37 & 48.25 \\
electrical\_engineering & 57.24 & 60.00 \\
elementary\_mathematics & 88.89 & 85.71 \\
formal\_logic & 53.17 & 46.03 \\
global\_facts & 48.00 & 41.00 \\
high\_school\_biology & 78.71 & 79.03 \\
high\_school\_chemistry & 61.08 & 62.07 \\
high\_school\_computer\_science & 71.00 & 72.00 \\
high\_school\_european\_history & 72.73 & 73.94 \\
high\_school\_geography & 78.79 & 80.81 \\
high\_school\_government\_and\_politics & 84.97 & 83.42 \\
high\_school\_macroeconomics & 68.97 & 73.33 \\
high\_school\_mathematics & 60.74 & 62.59 \\
high\_school\_microeconomics & 70.17 & 69.33 \\
high\_school\_physics & 53.64 & 53.64 \\
high\_school\_psychology & 85.50 & 85.50 \\
high\_school\_statistics & 63.89 & 65.74 \\
high\_school\_us\_history & 77.94 & 78.92 \\
high\_school\_world\_history & 75.11 & 79.32 \\
human\_aging & 69.51 & 64.13 \\
human\_sexuality & 53.44 & 54.96 \\
international\_law & 78.51 & 74.38 \\
jurisprudence & 73.15 & 76.85 \\
logical\_fallacies & 74.85 & 79.14 \\
machine\_learning & 49.11 & 53.57 \\
management & 80.58 & 82.52 \\
marketing & 86.32 & 85.47 \\
medical\_genetics & 75.00 & 76.00 \\
miscellaneous & 86.21 & 86.21 \\
moral\_disputes & 63.29 & 59.25 \\
moral\_scenarios & 44.47 & 49.72 \\
nutrition & 72.22 & 72.88 \\
philosophy & 69.13 & 68.81 \\
prehistory & 71.30 & 70.06 \\
professional\_accounting & 47.87 & 53.19 \\
professional\_law & 47.72 & 48.24 \\
professional\_medicine & 76.47 & 77.94 \\
professional\_psychology & 68.14 & 68.30 \\
public\_relations & 69.09 & 63.64 \\
security\_studies & 64.49 & 61.22 \\
sociology & 77.61 & 78.61 \\
us\_foreign\_policy & 81.00 & 81.00 \\
virology & 48.80 & 51.20 \\
world\_religions & 78.36 & 78.36 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
\newpage
% \renewcommand{\thetable}{A7}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Comparison between Top-$k$/Top-$p$ and reflection-window decoding on MMLU with Mistral-Nemo}
\label{tab:decoding_comparison_mistral}
\centering
\footnotesize
\begin{tabular}{L{0.35}C{0.20}C{0.20}}
\toprule
\textbf{Subject} & \textbf{Top-$k$/Top-$p$} & \textbf{Reflection-Window} \\
\midrule
abstract\_algebra & 34.00 & 41.00 \\
anatomy & 60.74 & 59.26 \\
astronomy & 67.76 & 66.45 \\
business\_ethics & 61.00 & 56.00 \\
clinical\_knowledge & 69.81 & 70.94 \\
college\_biology & 74.31 & 77.08 \\
college\_chemistry & 41.00 & 48.00 \\
college\_computer\_science & 56.00 & 54.00 \\
college\_mathematics & 45.00 & 39.00 \\
college\_medicine & 58.96 & 67.63 \\
college\_physics & 51.96 & 52.94 \\
computer\_security & 64.00 & 69.00 \\
conceptual\_physics & 63.40 & 61.70 \\
econometrics & 51.75 & 57.89 \\
electrical\_engineering & 55.17 & 53.10 \\
elementary\_mathematics & 73.81 & 74.87 \\
formal\_logic & 42.06 & 42.86 \\
global\_facts & 45.00 & 41.00 \\
high\_school\_biology & 76.77 & 75.16 \\
high\_school\_chemistry & 51.72 & 57.14 \\
high\_school\_computer\_science & 70.00 & 73.00 \\
high\_school\_european\_history & 66.67 & 66.06 \\
high\_school\_geography & 75.76 & 70.20 \\
high\_school\_government\_and\_politics & 80.83 & 84.46 \\
high\_school\_macroeconomics & 66.92 & 69.49 \\
high\_school\_mathematics & 59.63 & 60.00 \\
high\_school\_microeconomics & 65.97 & 69.33 \\
high\_school\_physics & 43.71 & 45.70 \\
high\_school\_psychology & 78.72 & 77.80 \\
high\_school\_statistics & 58.80 & 62.50 \\
high\_school\_us\_history & 64.71 & 73.04 \\
high\_school\_world\_history & 70.46 & 73.42 \\
human\_aging & 61.43 & 58.30 \\
human\_sexuality & 61.83 & 64.89 \\
international\_law & 68.60 & 66.94 \\
jurisprudence & 67.59 & 64.81 \\
logical\_fallacies & 69.94 & 65.64 \\
machine\_learning & 49.11 & 51.79 \\
management & 69.90 & 69.90 \\
marketing & 76.50 & 71.79 \\
medical\_genetics & 77.00 & 72.00 \\
miscellaneous & 79.69 & 79.57 \\
moral\_disputes & 58.38 & 60.12 \\
moral\_scenarios & 26.15 & 27.49 \\
nutrition & 67.32 & 65.03 \\
philosophy & 61.41 & 67.85 \\
prehistory & 61.11 & 61.11 \\
professional\_accounting & 46.10 & 47.16 \\
professional\_law & 44.39 & 44.46 \\
professional\_medicine & 58.46 & 58.46 \\
professional\_psychology & 61.93 & 65.69 \\
public\_relations & 65.45 & 59.09 \\
security\_studies & 52.65 & 55.92 \\
sociology & 68.66 & 72.64 \\
us\_foreign\_policy & 75.00 & 67.00 \\
virology & 37.95 & 42.17 \\
world\_religions & 71.35 & 76.02 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
% \renewcommand{\thetable}{A8}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Comparison between Top-$k$/Top-$p$ and reflection-window decoding on MMLU with Phi3-Medium}
\label{tab:decoding_comparison_phi3}
\centering
\footnotesize
\begin{tabular}{L{0.35}C{0.20}C{0.20}}
\toprule
\textbf{Subject} & \textbf{Top-$k$/Top-$p$} & \textbf{Reflection-Window} \\
\midrule
abstract\_algebra & 52.00 & 62.00 \\
anatomy & 70.37 & 72.59 \\
astronomy & 82.89 & 82.24 \\
business\_ethics & 72.00 & 77.00 \\
clinical\_knowledge & 78.87 & 79.25 \\
college\_biology & 82.64 & 81.25 \\
college\_chemistry & 51.00 & 54.00 \\
college\_computer\_science & 61.00 & 63.00 \\
college\_mathematics & 50.00 & 48.00 \\
college\_medicine & 69.94 & 72.25 \\
college\_physics & 71.57 & 63.73 \\
computer\_security & 74.00 & 74.00 \\
conceptual\_physics & 75.74 & 74.47 \\
econometrics & 57.89 & 59.65 \\
electrical\_engineering & 60.00 & 63.45 \\
elementary\_mathematics & 85.98 & 85.71 \\
formal\_logic & 56.35 & 57.14 \\
global\_facts & 45.00 & 46.00 \\
high\_school\_biology & 84.52 & 86.45 \\
high\_school\_chemistry & 74.38 & 73.40 \\
high\_school\_computer\_science & 81.00 & 83.00 \\
high\_school\_european\_history & 73.33 & 75.76 \\
high\_school\_geography & 81.82 & 82.83 \\
high\_school\_government\_and\_politics & 88.60 & 91.19 \\
high\_school\_macroeconomics & 80.00 & 79.74 \\
high\_school\_mathematics & 60.37 & 63.70 \\
high\_school\_microeconomics & 84.03 & 87.39 \\
high\_school\_physics & 58.28 & 70.20 \\
high\_school\_psychology & 85.50 & 85.50 \\
high\_school\_statistics & 70.83 & 72.69 \\
high\_school\_us\_history & 82.84 & 78.92 \\
high\_school\_world\_history & 81.01 & 81.86 \\
human\_aging & 71.30 & 68.16 \\
human\_sexuality & 73.28 & 77.10 \\
international\_law & 77.69 & 83.47 \\
jurisprudence & 83.33 & 75.00 \\
logical\_fallacies & 79.14 & 82.82 \\
machine\_learning & 54.46 & 64.29 \\
management & 76.70 & 80.58 \\
marketing & 83.76 & 81.62 \\
medical\_genetics & 85.00 & 87.00 \\
miscellaneous & 85.31 & 84.55 \\
moral\_disputes & 71.97 & 67.92 \\
moral\_scenarios & 61.34 & 63.24 \\
nutrition & 75.82 & 78.10 \\
philosophy & 74.92 & 71.06 \\
prehistory & 76.85 & 78.09 \\
professional\_accounting & 68.09 & 66.67 \\
professional\_law & 53.13 & 50.00 \\
professional\_medicine & 69.12 & 67.65 \\
professional\_psychology & 76.31 & 76.96 \\
public\_relations & 64.55 & 69.09 \\
security\_studies & 71.43 & 66.12 \\
sociology & 83.08 & 84.58 \\
us\_foreign\_policy & 81.00 & 84.00 \\
virology & 49.40 & 49.40 \\
world\_religions & 77.78 & 74.85 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
% \renewcommand{\thetable}{A9}
\begingroup
\renewcommand{\arraystretch}{1}
\begin{table}[t]
\caption{Comparison between Top-$k$/Top-$p$ and reflection-window decoding on MMLU with Qwen2.5-14B}
\label{tab:decoding_comparison_qwen}
\centering
\footnotesize
\begin{tabular}{L{0.35}C{0.20}C{0.20}}
\toprule
\textbf{Subject} & \textbf{Top-$k$/Top-$p$} & \textbf{Reflection-Window} \\
\midrule
abstract\_algebra & 78.00 & 81.00 \\
anatomy & 71.85 & 76.30 \\
astronomy & 88.16 & 91.45 \\
business\_ethics & 76.00 & 75.00 \\
clinical\_knowledge & 81.51 & 82.64 \\
college\_biology & 90.97 & 86.11 \\
college\_chemistry & 64.00 & 60.00 \\
college\_computer\_science & 73.00 & 73.00 \\
college\_mathematics & 72.00 & 80.95 \\
college\_medicine & 73.41 & 90.91 \\
college\_physics & 82.35 & 84.31 \\
computer\_security & 82.00 & 84.00 \\
conceptual\_physics & 82.55 & 84.68 \\
econometrics & 61.40 & 65.79 \\
electrical\_engineering & 73.10 & 73.79 \\
elementary\_mathematics & 95.24 & 94.97 \\
formal\_logic & 67.46 & 61.11 \\
global\_facts & 51.00 & 52.00 \\
high\_school\_biology & 88.39 & 90.97 \\
high\_school\_chemistry & 80.30 & 75.37 \\
high\_school\_computer\_science & 91.00 & 89.00 \\
high\_school\_european\_history & 76.97 & 81.82 \\
high\_school\_geography & 89.39 & 88.89 \\
high\_school\_government\_and\_politics & 91.19 & 92.75 \\
high\_school\_macroeconomics & 85.64 & 86.41 \\
high\_school\_mathematics & 84.44 & 84.27 \\
high\_school\_microeconomics & 89.08 & 88.60 \\
high\_school\_physics & 82.78 & 79.47 \\
high\_school\_psychology & 90.09 & 91.38 \\
high\_school\_statistics & 83.33 & 81.48 \\
high\_school\_us\_history & 87.25 & 88.73 \\
high\_school\_world\_history & 88.61 & 86.92 \\
human\_aging & 73.54 & 73.99 \\
human\_sexuality & 74.05 & 81.68 \\
international\_law & 80.17 & 77.69 \\
jurisprudence & 78.70 & 84.26 \\
logical\_fallacies & 83.44 & 85.28 \\
machine\_learning & 72.32 & 74.11 \\
management & 82.52 & 85.44 \\
marketing & 88.46 & 90.60 \\
medical\_genetics & 89.00 & 91.00 \\
miscellaneous & 89.40 & 91.42 \\
moral\_disputes & 71.97 & 72.54 \\
moral\_scenarios & 69.61 & 68.49 \\
nutrition & 82.35 & 81.70 \\
philosophy & 78.78 & 80.06 \\
prehistory & 86.11 & 83.02 \\
professional\_accounting & 73.76 & 71.99 \\
professional\_law & 54.04 & 56.00 \\
professional\_medicine & 81.62 & 86.40 \\
professional\_psychology & 77.94 & 79.08 \\
public\_relations & 71.82 & 71.82 \\
security\_studies & 71.43 & 73.06 \\
sociology & 85.07 & 88.06 \\
us\_foreign\_policy & 87.00 & 84.00 \\
virology & 50.00 & 51.81 \\
world\_religions & 84.21 & 84.80 \\
\bottomrule
\end{tabular}
\end{table}
\endgroup
\begin{thebibliography}{53}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abdin et~al.(2024{\natexlab{a}})Abdin, Aneja, Behl, Bubeck, Eldan, Gunasekar, Harrison, Hewett, Javaheripi, Kauffmann, et~al.]{abdin2024phi4}
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R.~J., Javaheripi, M., Kauffmann, P., et~al.
\newblock Phi-4 technical report.
\newblock \emph{arXiv preprint arXiv:2412.08905}, 2024{\natexlab{a}}.
\bibitem[Abdin et~al.(2024{\natexlab{b}})Abdin, Jacobs, Awan, Aneja, Awadallah, Awadalla, Bach, Bahree, Bakhtiari, Behl, et~al.]{abdin2024phi}
Abdin, M., Jacobs, S.~A., Awan, A.~A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et~al.
\newblock Phi-3 technical report: A highly capable language model locally on your phone.
\newblock \emph{arXiv preprint arXiv:2404.14219}, 2024{\natexlab{b}}.
\bibitem[Bai et~al.(2022)Bai, Kadavath, Kundu, Askell, Kernion, Jones, Chen, Goldie, Mirhoseini, McKinnon, et~al.]{bai2022constitutional}
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et~al.
\newblock Constitutional ai: Harmlessness from ai feedback.
\newblock \emph{arXiv preprint arXiv:2212.08073}, 2022.
\bibitem[Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, et~al.]{brown2020language}
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.~D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et~al.
\newblock Language models are few-shot learners.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 1877--1901, 2020.
\bibitem[Chen et~al.(2023{\natexlab{a}})Chen, Borgeaud, Irving, Lespiau, Sifre, and Jumper]{chen2023accelerating}
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J.
\newblock Accelerating large language model decoding with speculative sampling.
\newblock \emph{arXiv preprint arXiv:2302.01318}, 2023{\natexlab{a}}.
\bibitem[Chen et~al.(2023{\natexlab{b}})Chen, Lin, Sch{\"a}rli, and Zhou]{chen2023teaching}
Chen, X., Lin, M., Sch{\"a}rli, N., and Zhou, D.
\newblock Teaching large language models to self-debug.
\newblock \emph{arXiv preprint arXiv:2304.05128}, 2023{\natexlab{b}}.
\bibitem[DeepSeek-AI(2025)]{deepseek2025deepseek}
DeepSeek-AI.
\newblock {DeepSeek-R1}: Incentivizing reasoning capability in {LLMs} via reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2501.12948}, 2025.
\bibitem[Fan et~al.(2018)Fan, Lewis, and Dauphin]{fan2018hierarchical}
Fan, A., Lewis, M., and Dauphin, Y.
\newblock Hierarchical neural story generation.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 889--898, 2018.
\bibitem[Fedorenko et~al.(2024)Fedorenko, Piantadosi, and Gibson]{fedorenko2024language}
Fedorenko, E., Piantadosi, S.~T., and Gibson, E.~A.
\newblock Language is primarily a tool for communication rather than thought.
\newblock \emph{Nature}, 630\penalty0 (8017):\penalty0 575--586, 2024.
\bibitem[Ganguli et~al.(2023)Ganguli, Askell, Schiefer, Liao, Luko{\v{s}}i{\=u}t{\.e}, Chen, Goldie, Mirhoseini, Olsson, Hernandez, et~al.]{ganguli2023capacity}
Ganguli, D., Askell, A., Schiefer, N., Liao, T., Luko{\v{s}}i{\=u}t{\.e}, K., Chen, A., Goldie, A., Mirhoseini, A., Olsson, C., Hernandez, D., et~al.
\newblock The capacity for moral self-correction in large language models.
\newblock \emph{arXiv preprint arXiv:2302.07459}, 2023.
\bibitem[{Gemini Team}(2023)]{google2023gemini}
{Gemini Team}.
\newblock Gemini: A family of highly capable multimodal models.
\newblock \emph{arXiv preprint arXiv:2312.11805}, 2023.
\bibitem[{Gemma Team}(2024)]{deepmind2024gemma}
{Gemma Team}.
\newblock Gemma: Open models based on gemini research and technology.
\newblock \emph{arXiv preprint arXiv:2403.08295}, 2024.
\bibitem[Hendrycks et~al.(2020)Hendrycks, Burns, Basart, Zou, Mazeika, Song, and Steinhardt]{hendrycks2020measuring}
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.
\newblock Measuring massive multitask language understanding.
\newblock \emph{arXiv preprint arXiv:2009.03300}, 2020.
\bibitem[Holtzman et~al.(2018)Holtzman, Buys, Forbes, Bosselut, Golub, and Choi]{holtzman2018learning}
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., and Choi, Y.
\newblock Learning to write with cooperative discriminators.
\newblock \emph{arXiv preprint arXiv:1805.06087}, 2018.
\bibitem[Holtzman et~al.(2020)Holtzman, Buys, Du, Forbes, and Choi]{holtzman2020curious}
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y.
\newblock The curious case of neural text degeneration.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Kahneman(2011)]{kahneman2011thinking}
Kahneman, D.
\newblock \emph{Thinking, Fast and Slow}.
\newblock Farrar, Straus and Giroux, 2011.
\bibitem[Kim et~al.(2023)Kim, Baldi, and McAleer]{kim2023language}
Kim, G., Baldi, P., and McAleer, S.
\newblock Language models can solve computer tasks.
\newblock \emph{arXiv preprint arXiv:2303.17491}, 2023.
\bibitem[Kim et~al.(2024)Kim, Mangalam, Moon, Malik, Mahoney, Gholami, and Keutzer]{kim2024speculative}
Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M.~W., Gholami, A., and Keutzer, K.
\newblock Speculative decoding with big little decoder.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[King(2000)]{king2000writing}
King, S.
\newblock \emph{On writing: A memoir of the craft}.
\newblock Pocket Books, 2000.
\bibitem[Leviathan et~al.(2023)Leviathan, Kalman, and Matias]{leviathan2023fast}
Leviathan, Y., Kalman, M., and Matias, Y.
\newblock Fast inference from transformers via speculative decoding.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 19274--19286. PMLR, 2023.
\bibitem[Lew et~al.(2023)Lew, Matheos, Zhi-Xuan, Ghavamizadeh, Gothoskar, Russell, and Mansinghka]{lew2023smcp3}
Lew, A.~K., Matheos, G., Zhi-Xuan, T., Ghavamizadeh, M., Gothoskar, N., Russell, S., and Mansinghka, V.~K.
\newblock {SMCP}3: Sequential monte carlo with probabilistic program proposals.
\newblock In \emph{International conference on artificial intelligence and statistics}, pp.\ 7061--7088. PMLR, 2023.
\bibitem[Li et~al.(2022)Li, Tang, Zhao, Nie, and Wen]{li2022elmer}
Li, J., Tang, T., Zhao, W.~X., Nie, J.-Y., and Wen, J.-R.
\newblock Elmer: A non-autoregressive pre-trained language model for efficient and effective text generation.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing}, 2022.
\bibitem[Li et~al.(2023)Li, Holtzman, Fried, Liang, Eisner, Hashimoto, Zettlemoyer, and Lewis]{li2023contrastive}
Li, X.~L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M.
\newblock Contrastive decoding: Open-ended text generation as optimization.
\newblock In \emph{The 61st Annual Meeting of the Association for Computational Linguistics}, 2023.
\bibitem[Lin et~al.(2021)Lin, Jaech, Li, Gormley, and Eisner]{lin2021limitations}
Lin, C.-C., Jaech, A., Li, X., Gormley, M.~R., and Eisner, J.
\newblock Limitations of autoregressive models and their alternatives.
\newblock In \emph{Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)}, 2021.
\bibitem[{Llama Team}(2024)]{meta2024llama}
{Llama Team}.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint arXiv:2407.21783}, 2024.
\bibitem[Madaan et~al.(2024)Madaan, Tandon, Gupta, Hallinan, Gao, Wiegreffe, Alon, Dziri, Prabhumoye, Yang, et~al.]{madaan2024self}
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et~al.
\newblock Self-refine: Iterative refinement with self-feedback.
\newblock In \emph{Advances in Neural Information Processing Systems}, volume~36, 2024.
\bibitem[MistralAI(2024)]{mistralNemo}
MistralAI.
\newblock Mistral nemo, 2024.
\newblock URL \url{https://mistral.ai/news/mistral-nemo/}.
\bibitem[Nocedal \& Wright(1999)Nocedal and Wright]{nocedal1999numerical}
Nocedal, J. and Wright, S.~J.
\newblock \emph{Numerical Optimization}.
\newblock Springer, 1999.
\bibitem[OpenAI(2023)]{openai2023gpt4}
OpenAI.
\newblock {GPT}-4 technical report.
\newblock \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem[OpenAI(2024)]{openai2024gpt4o}
OpenAI.
\newblock {GPT}-4o system card.
\newblock \emph{arXiv preprint arXiv:2410.21276}, 2024.
\bibitem[Pan et~al.(2023)Pan, Saxon, Xu, Nathani, Wang, and Wang]{pan2023automatically}
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W.~Y.
\newblock Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.
\newblock \emph{arXiv preprint arXiv:2308.03188}, 2023.
\bibitem[Pearl(2009)]{pearl2009causality}
Pearl, J.
\newblock \emph{Causality}.
\newblock Cambridge University Press, 2009.
\bibitem[Peeperkorn et~al.(2024)Peeperkorn, Kouwenhoven, Brown, and Jordanous]{peeperkorn2024temperature}
Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jordanous, A.
\newblock Is temperature the creativity parameter of large language models?
\newblock In \emph{Proceedings of the 15th International Conference on Computational Creativity}, 2024.
\bibitem[Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, and Sutskever]{radford2019language}
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.
\newblock Language models are unsupervised multitask learners.
\newblock \emph{OpenAI Blog}, 1\penalty0 (8):\penalty0 9, 2019.
\bibitem[Serravallo(2017)]{serravallo2017writing}
Serravallo, J.
\newblock \emph{The writing strategies book: Your everything guide to developing skilled writers}.
\newblock Heinemann Portsmouth, NH, 2017.
\bibitem[Shih et~al.(2023)Shih, Sadigh, and Ermon]{shih2023long}
Shih, A., Sadigh, D., and Ermon, S.
\newblock Long horizon temperature scaling.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 31422--31434. PMLR, 2023.
\bibitem[Shinn et~al.(2023)Shinn, Cassano, Gopinath, Narasimhan, and Yao]{shinn2023reflexion}
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K.~R., and Yao, S.
\newblock Reflexion: Language agents with verbal reinforcement learning.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\bibitem[Spirtes et~al.(1993)Spirtes, Glymour, and Scheines]{spirtes1993causation}
Spirtes, P., Glymour, C., and Scheines, R.
\newblock \emph{Causation, Prediction, and Search}.
\newblock Springer New York, 1993.
\bibitem[Sun et~al.(2024)Sun, Suresh, Ro, Beirami, Jain, and Yu]{sun2024spectr}
Sun, Z., Suresh, A.~T., Ro, J.~H., Beirami, A., Jain, H., and Yu, F.
\newblock Spectr: Fast speculative decoding via optimal transport.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[T{\"o}rn \& {\v{Z}}ilinskas(1989)T{\"o}rn and {\v{Z}}ilinskas]{torn1989global}
T{\"o}rn, A. and {\v{Z}}ilinskas, A.
\newblock \emph{Global Optimization}, volume 350.
\newblock Springer, 1989.
\bibitem[Touvron et~al.(2023)Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale, et~al.]{touvron2023llama}
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et~al.
\newblock Llama 2: Open foundation and fine-tuned chat models.
\newblock \emph{arXiv preprint arXiv:2307.09288}, 2023.
\bibitem[Tyen et~al.(2023)Tyen, Mansoor, Chen, Mak, and C{\u{a}}rbune]{tyen2023llms}
Tyen, G., Mansoor, H., Chen, P., Mak, T., and C{\u{a}}rbune, V.
\newblock Llms cannot find reasoning errors, but can correct them!
\newblock \emph{arXiv preprint arXiv:2311.08516}, 2023.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{vaswani2017attention}
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.~N., Kaiser, L., and Polosukhin, I.
\newblock Attention is all you need.
\newblock \emph{Advances in Neural Information Processing Systems}, 2017.
\bibitem[Wei et~al.(2024)Wei, Koo, Tavanaei, and Bouyarmane]{wei2024confidence}
Wei, C., Koo, K.~K., Tavanaei, A., and Bouyarmane, K.
\newblock Confidence-aware sub-structure beam search (cabs): Mitigating hallucination in structured data generation with large language models.
\newblock \emph{arXiv preprint arXiv:2406.00069}, 2024.
\bibitem[Wu et~al.(2021)Wu, Xu, Wang, and Long]{wu2021autoformer}
Wu, H., Xu, J., Wang, J., and Long, M.
\newblock Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.
\newblock \emph{Advances in neural information processing systems}, 34:\penalty0 22419--22430, 2021.
\bibitem[Xia et~al.(2023)Xia, Ge, Wang, Chen, Wei, and Sui]{xia2023speculative}
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z.
\newblock Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2023}, pp.\ 3909--3925, 2023.
\bibitem[Xia et~al.(2024)Xia, Yang, Dong, Wang, Li, Ge, Liu, Li, and Sui]{xia2024unlocking}
Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z.
\newblock Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding.
\newblock \emph{arXiv preprint arXiv:2401.07851}, 2024.
\bibitem[Xie et~al.(2024)Xie, Kawaguchi, Zhao, Zhao, Kan, He, and Xie]{xie2024self}
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J.~X., Kan, M.-Y., He, J., and Xie, M.
\newblock Self-evaluation guided beam search for reasoning.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Yang et~al.(2024{\natexlab{a}})Yang, Yang, Hui, Zheng, Yu, Zhou, Li, Li, Liu, Huang, et~al.]{qwen2}
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et~al.
\newblock Qwen2 technical report.
\newblock \emph{arXiv preprint arXiv:2407.10671}, 2024{\natexlab{a}}.
\bibitem[Yang et~al.(2024{\natexlab{b}})Yang, Lee, and Tadepalli]{yang2024language}
Yang, Y., Lee, S., and Tadepalli, P.
\newblock Language-informed beam search decoding for multilingual machine translation.
\newblock \emph{arXiv preprint arXiv:2408.05738}, 2024{\natexlab{b}}.
\bibitem[Yao et~al.(2022)Yao, Zhao, Yu, Du, Shafran, Narasimhan, and Cao]{yao2022react}
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y.
\newblock React: Synergizing reasoning and acting in language models.
\newblock \emph{arXiv preprint arXiv:2210.03629}, 2022.
\bibitem[Zheng et~al.(2023)Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, et~al.]{zheng2023judging}
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et~al.
\newblock Judging {LLM}-as-a-judge with {MT}-bench and chatbot arena.
\newblock In \emph{Advances in Neural Information Processing Systems}, volume~36, pp.\ 46595--46623, 2023.
\bibitem[Zhu et~al.(2024)Zhu, Zhang, Xie, and Su]{zhu2024deductive}
Zhu, T., Zhang, K., Xie, J., and Su, Y.
\newblock Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning.
\newblock \emph{arXiv preprint arXiv:2401.17686}, 2024.
\end{thebibliography}