Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs
Extracted source text · arXiv 2502.02362
\documentclass{article}
\PassOptionsToPackage{hyphens}{url}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage[submission]{icml2024} with \usepackage[nohyperref]{icml2024} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
\usepackage[accepted]{icml2025}
\usepackage{xurl}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{multirow}
\usepackage{enumitem}
\usepackage{tcolorbox}
\usepackage{pifont}
\usepackage{newtxmath}
\usepackage{colortbl}
\usepackage{xcolor}
\definecolor{lightblue}{RGB}{220, 220, 220}
\definecolor{lightgreen}{RGB}{144, 238, 144}
\definecolor{lightred}{RGB}{255, 182, 193}
\definecolor{darkerblue}{RGB}{150, 150, 150}
% \renewcommand{\mathcal}{\mathscr}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% TERMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{xspace}
\newcommand{\ie}{\textit{i.e.}}
\newcommand{\eg}{\textit{e.g.}}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Premise-Augmented Reasoning Chains Improve Error Identification in Math Reasoning with LLMs}
\begin{document}
\twocolumn[
\icmltitle{Premise-Augmented Reasoning Chains Improve Error Identification\\ in Math Reasoning with LLMs}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2025
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Sagnik Mukherjee}{equal,uiuc}
\icmlauthor{Abhinav Chinta}{equal,uiuc}
\icmlauthor{Takyoung Kim}{uiuc}
\icmlauthor{Tarun Sharma}{uiuc}
\icmlauthor{Dilek Hakkani-Tür}{uiuc}
\end{icmlauthorlist}
\icmlaffiliation{uiuc}{University of Illinois at Urbana Champaign}
\icmlcorrespondingauthor{Sagnik Mukherjee}{sagnikm3@illinois.edu}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
%\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
% Chain-of-Thought (CoT) prompting enhances mathematical reasoning in large language models (LLMs) by enabling detailed step-by-step solutions. However, due to the verbosity of LLMs, this prompting approach introduces significant challenges in both traceability (logical flow) and verifiability (correctness). Importantly, mathematical reasoning allows each step to be derived from a small set of premises. For traceable and verifiable evaluation of reasoning chains in mathematical reasoning, we present a framework that identifies these premises to improve the evaluation of reasoning. We restructure conventional linear reasoning chains into \textbf{Premise Augmented Reasoning Chains (PARC)} by establishing premise links, thereby improving transparency and traceability. Through experiments with a PARC-based dataset that we built, namely \textbf{PERL (Premises and ERrors identification in LMs)}, we demonstrate that LLMs can reliably identify premises within complex reasoning chains. In particular, even open-source LLMs achieve 90\% recall in premise identification. We also show that PARC helps to identify errors in reasoning chains more reliably. The accuracy of error identification improves by 6\% to 16\% when step-by-step verification is carried out in PARC under the premises.
% Our findings highlight the utility of premise-centric representations in addressing complex problem-solving tasks and open new avenues for improving the reliability of LLM-based reasoning evaluations.
Chain-of-Thought (CoT) prompting enhances mathematical reasoning in large language models (LLMs) by enabling detailed step-by-step solutions. However, due to the verbosity of LLMs, the resulting reasoning chains can be long, making it harder to verify the reasoning steps and trace issues resulting from dependencies between the steps that may be farther away in the sequence of steps. Importantly, mathematical reasoning allows each step to be derived from a small set of premises, which are a subset of the preceding steps in the reasoning chain. In this paper, we present a framework that identifies the premises for each step, to improve the evaluation of reasoning. We restructure conventional linear reasoning chains into \textbf{Premise Augmented Reasoning Chains (PARC)} by introducing premise links, resulting in a directed acyclic graph where the nodes are the steps and the edges are the premise links. Through experiments with a PARC-based dataset that we built, namely \textbf{PERL (Premises and ERrors identification in LLMs)}, we demonstrate that LLMs can reliably identify premises within complex reasoning chains. In particular, even open-source LLMs achieve 90\% recall in premise identification. We also show that PARC helps to identify errors in reasoning chains more reliably. The accuracy of error identification improves by 6\% to 16\% absolute when step-by-step verification is carried out in PARC under the premises.
Our findings highlight the utility of premise-centric representations in addressing complex problem-solving tasks and open new avenues for improving the reliability of LLM-based reasoning evaluations.
\end{abstract}
\input{intro}
\input{related_work}
\input{method}
\input{experiments}
\input{conclusion}
\bibliography{custom}
\bibliographystyle{icml2025}
\appendix
\input{appendix}
\end{document}
% \section{Introduction}
% \begin{figure*}[t!]
% \centering
% \includegraphics[width=0.9\textwidth]{sections/overview.pdf}
% \caption{Comparison between a general-purpose Chain of Thought (CoT) and a fully verifiable CoT. In the general-purpose CoT (left), errors can propagate through the reasoning steps without verification. In the fully verifiable CoT (right), premise links are explicitly established, enabling identification of correct and incorrect steps. Accumulation errors can be traced back to faulty premises, improving the interpretability and reliability of the reasoning process.}
% \label{fig:overview}
% \end{figure*}
% There has been a well-documented success of the reasoning capabilities of large language models (LLMs) with Chain-of-Thought (CoT; \citet{wei2023chainofthoughtpromptingelicitsreasoning}). Across a plethora of applications such as embodied reasoning \cite{NEURIPS2023_3d0758f0, philipov2024simulatinguseragentsembodied}, code generation \cite{jiang2024surveylargelanguagemodels}, mathematical and scientific reasoning \cite{imani-etal-2023-mathprompter, ahn2024largelanguagemodelsmathematical}, CoT-based LLMs have been shown to produce exceptional performance.
% However, the corresponding evaluation methods and metrics have a significant limitation: they focus solely on the correctness of the \textit{final} answer, neglecting intermediate reasoning processes and rationales that contribute to it. Although the final answer serves as a proxy for the reasoning capability, it is not sufficient to judge the model's reasoning performance. Hence, evaluating the final answer provides a narrower view of the reasoning capabilities.
% % since LLMs can produce correct answers while the rationales could be flawed.
% % The example in Figure \ref{simplify_expression} illustrates a representative situation where the generated reasoning chain incurs multiple errors while getting the final answer right.
% Since the intermediate reasoning process is equally important as getting the correct final answer \cite{huang2023reasoninglargelanguagemodels, golovneva2023roscoesuitemetricsscoring, prasad2023recevalevaluatingreasoningchains}, a comprehensive evaluation of the reasoning chains is crucial to holistically understanding the reasoning capabilities of LLMs.
% Evaluation of a reasoning chain has been studied in literature in the context of self-verification \cite{weng-etal-2023-large}. Previous work has shown that LLMs struggle to find reasoning errors in chain-of-thought traces without the help of external verifiers \cite{stechly2024selfverificationlimitationslargelanguage, wu-etal-2024-large, tyen-etal-2024-llms}, casting doubt on the overoptimism of LLMs' self-critique abilities.
% Existing research focusing on the evaluation of reasoning chains in LLMs can be broadly categorized into \textit{reference-based} and \textit{reference-free} methods.
% Reference-based methods, which rely on the availability of a ground truth reasoning chain \cite{welleck2021naturalproofs, han2024folionaturallanguagereasoning, tian-etal-2021-diagnosing}, are reliable but are constrained by the significant cost of human annotations, restricting their application.
% In contrast, reference-free methods \cite{prasad2023recevalevaluatingreasoningchains, golovneva2023roscoesuitemetricsscoring, zhu2024deductivebeamsearchdecoding} bypass the need for annotations, but suffer from two major drawbacks. Firstly, most of such existing works assign a chain-level score and not target error localization. Further, they often need fine-tuning of task-specific models, restricting their generalizability.
% One workaround could be to do reasoning with formal proof assistants like Lean \cite{10.1007/978-3-030-79876-5_37}, which natively support the verifiability of generated proofs \cite{yang2024formalmathematicalreasoningnew, yang2023leandojotheoremprovingretrievalaugmented, murphy2024autoformalizingeuclideangeometry}. However, this requires the challenging task of auto-formalizing natural language text to such proof assistants, adding an additional layer of difficulty \cite{wu2022autoformalizationlargelanguagemodels}. Another challenge for such an approach is that even for math word problems, such proof assistants need the answer to be known beforehand, thereby restructuring the problem to a proof.
% % The challenges in evaluating LLM reasoning chains have significant implications for transparency and reliability of these models. Without systematic evaluation of intermediate reasoning steps, there is an increased risk of hallucinations or flawed deductions, which can lead to correct answers reached through faulty logic. Additionally, while reference-based methods are reliable, the high cost of human annotations restricts their use to simpler tasks, limiting their scalability. Reference-free methods, though promising, still struggle with the complex logical and scientific reasoning and fail to provide step by step evaluation.
% In our work, we focus on reference-free verification of LLM reasoning chains in the context of mathematical reasoning. Mathematical problem solving requires a series of deductive reasoning steps, where each step is performed under some premise steps. We hypothesize that a step in a reasoning chain should be verified only under its premises. Previous work also shown that having unnecessary context hurts performance of LLMs \cite{10.5555/3618408.3619699}. Thus, by excluding unnecessary context from the verifier, we make the verification of a reasoning step only conditioned on its premises, addressing the known susceptibility of LLMs to errors under irrelevant contexts.
% We propose a framework, namely \textbf{PARC (Premise-Augmented Reasoning Chains)}, to induce a structure on a typical linear reasoning chain (LRC) to convert it into a premise-augmented format by identifying the premises of each step. Identifying those premises improves the tracability of a reasoning chain and the logical flow. By constructing corresponding dataset, called \textbf{PERL (Premises and ERrors identification in Language models)}, we verify the effectiveness of premise augmentation in annotating both premises and errors.
% Additionally, we refine the error taxonomy for mathematical reasoning by introducing a new category: ``accumulation error" (see Fig. \ref{fig:overview} for example). This type of error occurs when a reasoning step is correct in isolation but is derived from flawed premise steps, resulting in error propagation throughout the reasoning chain. Although this type of errors are quite common in practice, existing work has not addressed this error type to the best of our knowledge. Previous work focuses on the first mistake in a reasoning chain \cite{lightman2023letsverifystepstep, zheng2024processbenchidentifyingprocesserrors, daheim2024stepwiseverificationremediationstudent}, discarding the rest of the steps, due to the ambiguity of their correctness. Identifying accumulation errors is critical for improving the reliability and interpretability of reasoning traces in LLMs. By distinguishing accumulation errors from reasoning errors, we enable a more precise understanding of error types and facilitate targeted interventions. In addition, establishing explicit premise links between reasoning steps improves both the efficiency of verification and the transparency of the reasoning process. This, in turn, strengthens the robustness and reliability of chain-of-thought reasoning, making it a more trustworthy tool for complex tasks.
% % ds\textcolor{red}{We can mention partial credit and tie it to error classification, accumulation errors...etc}
% To this end, our contributions are as follows:
% \begin{itemize}
% \item We demonstrate that off-the-shelf LLMs can detect premises for a given step with high accuracy for mathematical reasoning to successfully convert a Linear Reasoning Chain (LRC) to a Premise Augmented Reasoning Chain (PARC).
% \item We establish that the use of these premises improves the accuracy in detecting errors and their types in reasoning chains.
% \item We propose an error taxonomy for mathematical reasoning in LLMs, notably, we introduce the use of \textit{Accumulation Errors} to classify erroneous steps beyond the first wrong step in the chain.
% \item We create PERL, a dataset of reasoning chains, annotated with premises and error types that we will make publicly available to foster future research.
% \end{itemize}
\section{Introduction}
\begin{figure*}[t!]
\centering
\includegraphics[width=0.95\textwidth]{overview.pdf}
\vspace{-3ex}
\caption{Comparison between a Linear Reasoning Chain (LRC) and our proposed PARC (Premise-Augmented Reasoning Chain). The LRC (left), is linear and there no explicit premise link between steps. In PARC (right), premise links are explicitly established, enabling better identification of correct and incorrect steps. Accumulation errors can be traced back to faulty premises. Establishing these premises helps improve error detection with LLMs.}
\label{fig:overview}
\vspace{-3ex}
\end{figure*}
Chain-of-thought reasoning enhances problem solving by breaking down complex tasks into a series of logical steps, improving the accuracy and clarity of decision-making processes.
There has been a well-documented success of the reasoning capabilities of LLMs with chain of thought reasoning (CoT; \cite{wei2023chainofthoughtpromptingelicitsreasoning}) across applications such as embodied reasoning~\cite{NEURIPS2023_3d0758f0, philipov2024simulatinguseragentsembodied}, code generation~\cite{jiang2024surveylargelanguagemodels}, and mathematical and scientific reasoning~\cite{imani-etal-2023-mathprompter, ahn2024largelanguagemodelsmathematical}. However, the corresponding evaluation methods and metrics have a significant limitation: they focus solely on the correctness of the \textit{final} answer, neglecting intermediate reasoning processes and rationales that contribute to it. Although the final answer serves as a proxy for the reasoning capability, it is not sufficient to judge the reasoning performance of the model. Hence, evaluating the final answer provides a narrower view of the reasoning capabilities. Since the intermediate reasoning process is equally important as getting the correct final answer~\cite{huang2023reasoninglargelanguagemodels, golovneva2023roscoesuitemetricsscoring, prasad2023recevalevaluatingreasoningchains}, a comprehensive evaluation of the reasoning chains is crucial to holistically understanding the reasoning capabilities of LLMs.
Evaluation of reasoning chains has been studied in literature in the context of self-verification~\cite{weng-etal-2023-large}. Previous work has shown that LLMs struggle to find reasoning errors in chain-of-thought traces without the help of external verifiers~\cite{stechly2024selfverificationlimitationslargelanguage, wu-etal-2024-large, tyen-etal-2024-llms}, casting doubt on the overoptimism of LLMs’ self-critique abilities. Existing research focusing on the evaluation of reasoning chains in LLMs can be broadly categorized into \textit{reference-based} and \textit{reference-free} methods. Reference-based methods, which rely on the availability of a ground truth reasoning chain~\cite{welleck2021naturalproofs, han2024folionaturallanguagereasoning, tian-etal-2021-diagnosing}, are reliable but are constrained by the significant cost of human annotations, restricting their application. In contrast, reference-free methods~\cite{prasad2023recevalevaluatingreasoningchains, golovneva2023roscoesuitemetricsscoring, zhu2024deductivebeamsearchdecoding} bypass the need for annotations, but suffer from two major drawbacks: (1) most such works assign only a chain-level score rather than localizing specific errors, and (2) they often need fine-tuning of task-specific models, restricting their generalizability. One workaround could be using formal proof assistants like Lean~\cite{10.1007/978-3-030-79876-5_37}, which natively support the verifiability of generated proofs~\cite{yang2024formalmathematicalreasoningnew, yang2023leandojotheoremprovingretrievalaugmented, murphy2024autoformalizingeuclideangeometry}, but this requires a challenging auto-formalization of natural language text~\cite{wu2022autoformalizationlargelanguagemodels} and often assumes the solution is already known for proof construction.
In our work, we focus on reference-free verification of LLM reasoning chains in the context of mathematical reasoning. Mathematical problem solving requires a series of deductive reasoning steps, where each step is performed under a small set of premises. We hypothesize that a step in a reasoning chain should be verified only under its premises. Previous work has shown that having unnecessary context hurts the performance of LLMs for solving math word problems~\cite{10.5555/3618408.3619699}. Thus, by excluding irrelevant context from the verifier, we make the verification of a reasoning step only conditioned on its premises, addressing the known susceptibility of LLMs to errors under distractors.
We restructure conventional linear reasoning chains (LRCs) into \textbf{PARC (Premise-Augmented Reasoning Chains)}, by identifying the premises of each step. Identifying those premises improves the traceability of a reasoning chain. We create a corresponding dataset, called \textbf{PERL (Premises and ERrors identification in Language models)} to test LLMs' capability in identifying premises as well as the effectiveness of premise augmentation in annotating both premises and errors. Additionally, we refine the error taxonomy (as proposed in \citet{golovneva2023roscoesuitemetricsscoring}) for mathematical reasoning by introducing a new category: “accumulation error.” This error arises when a reasoning step is correct in isolation but is derived from flawed premises (refer to step 5 in Fig.\ref{fig:overview}), resulting in error propagation throughout the chain; while such errors are common, existing work~\cite{lightman2023letsverifystepstep, zheng2024processbenchidentifyingprocesserrors, daheim2024stepwiseverificationremediationstudent} has not addressed them beyond discarding all subsequent steps once the first mistake appears. Distinguishing accumulation errors from inherently flawed steps (mathematical errors or logical inconsistencies) is crucial for a holistic evaluation of CoT traces. Moreover, establishing explicit premise links between steps allows each step to be verified only under its premises which reduces irrelevant information for each verification step, thus improving the error detection for mathematical problem solving.
Our main contributions are as follows:
\begin{itemize}
\item We demonstrate that off-the-shelf LLMs can detect premises for a given step with high accuracy for mathematical reasoning, enabling the conversion of a Linear Reasoning Chain (LRC) to a Premise-Augmented Reasoning Chain (PARC).
\item We establish that verifying each step under its corresponding premises increases the accuracy of identifying errors and their types.
\item We propose a refined error taxonomy for mathematical reasoning, introducing the notion of \textit{accumulation errors} for steps that are locally correct but inherit upstream errors.
\item We introduce and will release \textbf{PERL}, a dataset of reasoning chains annotated with premises and error types, to facilitate broader research on premise-centered reasoning verification.
\end{itemize}
The rest of the paper is structured as follows: in Section 2, we discuss relevant prior work in context of mathematical reasoning and verification of reasoning chains with LLMs. Section 3 introduces the research questions, alongside the necessary background as well as mathematical definition of premises, conversion of LRC to PARC, as well as how we identify errors with premises. Section 4 details PERL, our dataset to show effectiveness of the framework, and lastly, in section 5 we discuss the results and insights from our experiments. \footnote{Our code and data is available on \url{https://github.com/SagnikMukherjee/PARC}}
\section{Related work}
\label{sec:related_work}
% \paragraph{Math reasoning with LLMs:} The reasoning capabilities of large language models (LLMs) are often tested on mathematical and scientific reasoning benchmarks such as \cite{hendrycks2021measuringmassivemultitasklanguage,arora-etal-2023-llms,wang2024scibench}. Deriving a solution to a mathematical problem requires complex planning, recalling formulae, and grounding the plan to the given problem \cite{arora-etal-2023-llms} to derive the final answers. A popular approach to improve such reasoning is to first generate a series of rationales and then generate the answer \cite{wei2023chainofthoughtpromptingelicitsreasoning} instead of directly generating the final answer. Previous work also investigated reasoning in a structured fashion, rather than step-by-step, such as Tree of thoughts \cite{yao2023treethoughtsdeliberateproblem}, Graph of thoughts \cite{Besta_2024}, etc.
% \paragraph{Evaluation of Chain-of-thoughts:} While rationale generation helps reasoning, it is important to verify whether the reasoning process was correct to have a holistic view of the reasoning capabilities of the model.
% Although earlier work has focused on word overlap or embedding similarity to evaluate text generation, they are not equipped well enough to judge deductive reasoning capabilities \cite{celikyilmaz2021evaluationtextgenerationsurvey, reiter-2019-natural}. Other existing efforts include Roscoe \cite{golovneva2023roscoesuitemetricsscoring}, which uses SimCSE \cite{gao2022simcsesimplecontrastivelearning} embeddings to find similarity between the context and the generated solution to identify multiple types of errors from a taxonomy. Receval \cite{prasad2023recevalevaluatingreasoningchains} extends this further with metrics based on informativeness and correctness. Socreval \cite{he2024socrevallargelanguagemodels} further extends these frameworks using GPT4 \cite{openai2024gpt4technicalreport} in the loop with in-context learning to alleviate the need for costly human annotations and proposes a completely training-free approach.
% While such methods perform well in a reference-free setting, they fail to judge correctness at a step level, and instead assign a chain level score. Furthermore, the metrics used in these methods offer limited explainability, making it difficult to clearly identify the specific error type associated with each reasoning step. In contrast, our framework provides a natural language description of both the reasoning step’s correctness and its corresponding error type.
% \paragraph{Verifiers for math reasoning:} Verifiers of the reasoning chain have multiple applications in the reasoning of LLMs. \citet{uesato2022solvingmathwordproblems} and \citet{lightman2023letsverifystepstep} showed that step-by-step process rewards generated by a verifier lead to a stable training of LLMs and improve accuracy on end tasks. Deductive Beam Search \cite{zhu2024deductivebeamsearchdecoding} trained a verifier on manually crafted datasets and used it in beam search to improve the accuracy of the end task. Although existing work has shown that a verifier in the loop can improve LLM reasoning in multiple stages of the pipeline (training/inference), a clear approach on how to train a reliable verifier is missing from prior work. We address this issue in our work. The closest to our work is \citet{ling2023deductiveverificationchainofthoughtreasoning}, however, their approach requires the models need to generate the reasoning alongside the premises in their natural program style. This has a drawback in that it requires the model to generate the reasoning chain in a certain way, making it impossible to verify a general purpose chain of thought, hurting its scalability. In contrast, our approach to identify premises as a separate step and proposing it as a task ensures generalizability across reasoning styles and is not under any assumptions.
\paragraph{Math Reasoning with LLMs.}
Mathematical and scientific reasoning tasks have become a primary testbed to assess the capabilities of large language models (LLMs) \cite{hendrycks2021measuringmassivemultitasklanguage,arora-etal-2023-llms,wang2024scibench}. Solving these tasks requires complex planning, recall of relevant formulae, and grounding the solution to the given problem \cite{arora-etal-2023-llms}. A widely adopted approach to improve the reasoning of LLMs is to generate intermediate rationales, commonly known as chain-of-thought (CoT), before predicting a final answer \cite{wei2023chainofthoughtpromptingelicitsreasoning}. Recent work explores alternative structures for organizing these solutions, including Tree-of-Thoughts \cite{yao2023treethoughtsdeliberateproblem} and Graph-of-Thoughts \cite{Besta_2024}.
\paragraph{Evaluation of Chain-of-Thoughts.}
Although generating rationales can significantly improve model reasoning, systematically evaluating these explanations remains a challenge. Traditional metrics focused on word overlap or embedding similarity \cite{celikyilmaz2021evaluationtextgenerationsurvey, reiter-2019-natural} fail to capture logical soundness, especially for step-by-step deductions. To address this gap, methods such as ROSCOE \cite{golovneva2023roscoesuitemetricsscoring}, ReCEval \cite{prasad2023recevalevaluatingreasoningchains}, and Socreval \cite{he2024socrevallargelanguagemodels} offer reference-free evaluation frameworks that assess correctness and identify various categories of errors. However, these methods generally provide a single chain-level score and offer limited explainability, making it difficult to pinpoint specific error types in individual reasoning steps. In contrast, our proposed framework provides a natural language characterization of the correctness of each step, providing fine-grained insights into the precise nature of errors.
\paragraph{Verifiers for Math Reasoning.}
Verifiers have proven effective in enhancing LLM-based reasoning. \citet{uesato2022solvingmathwordproblems} and \citet{lightman2023letsverifystepstep} demonstrate that providing reward signals for intermediate steps can stabilize training and improve final performance. Deductive Beam Search \cite{zhu2024deductivebeamsearchdecoding} similarly incorporates a trained verifier to refine model outputs during inference, leading to higher accuracy for the final task. While these studies highlight the benefits of verifiers, a clear strategy for training a robust verifier, one that generalizes across diverse problem formats, remains elusive. The closest existing effort, \citet{ling2023deductiveverificationchainofthoughtreasoning}, requires models to encode premises in a particular “natural program” style, which constrains applicability to general chains-of-thought. In contrast, our method treats premise extraction as a standalone step and refrains from making strong assumptions about how the reasoning chain is structured.
%\zhenhailong{add our distinctive contribution here compared with AssistantBench/MindSearch}
% \begin{tcolorbox}[colback=red!5!white,colframe=red!75!black,label=simplify_expression]
% \textbf{Q: Simplify the following expression:
% \[
% \frac{\sec x}{\sin x} - \frac{\sin x}{\cos x}
% \]}
% \textbf{Step 1: (Correct \textcolor{green}{\ding{51}})} expressing the given expression in terms of sine and cosine,
% \[
% \frac{\sec x}{\sin x} - \frac{\sin x}{\cos x} = \frac{1}{\sin x \cos x} - \frac{\sin x}{\cos x}.
% \]
% \textbf{Step 2: (Error \textcolor{red}{\ding{55}})} Rewriting, we get
% \[
% \frac{1}{\sin x \cos x} - \frac{\sin x}{\cos x} = \frac{1}{\sin x \cos x} - \frac{\sin x}{\cos x} \cdot \frac{1}{1}.
% \]
% \[
% = \frac{1}{\sin x \cos x} - \frac{\sin x}{\cos x \cdot \sin x}.
% \]
% \textbf{Step 3: (Accumulation Error \textcolor{red}{!!})} Simplifying further
% \[
% \frac{1}{\sin x \cos x} - \frac{\sin x}{\sin x \cos x} = \frac{1 - \sin x}{\sin x \cos x}.
% \]
% \textbf{Step 4: (Error \textcolor{red}{\ding{55}})} expressing the numerator in terms of a cosine
% \[
% \frac{1 - \sin x}{\sin x \cos x} = \frac{\cos^2 x}{\sin x \cos x}
% = \frac{\cos x}{\sin x}
% = \cot x.
% \]
% \textbf{Feedback:} There are two wrong steps in this step-by-step process. First, Step 2 incorrectly simplifies $\frac{\sin x}{\cos x} \cdot \frac{1}{1}$ to $\frac{\sin x}{\cos x \cdot \sin x}$. Further Step 4 incorrectly replaces $1 - \sin x$ with $\cos^2 x$, making the final answer correct.
% \end{tcolorbox}
% \begin{comment}
% \begin{itemize}
% \item WebLinx \citepp{lu2024weblinx}
% \item WebArena \citepp{zhou2023webarena}
% \item WebGPT \citepp{nakano2021webgpt}
% \item Mind2Web \citepp{deng2024mind2web}
% \item AutoGPT \citepp{}
% \end{itemize}
% \end{comment}
% \begin{table*}
% \centering
% \begin{tabular}{lll}
% \hline
% \textbf{Error Type} & \textbf{Description} \\
% \hline
% Logical Inconsistency & Violates logic or common sense. \\
% Mathematical Error & Incorrect calculations or quantitative reasoning in \(S_t\). \\
% Accumulation Error & Errors in \(S_t\) due to incorrect premises. \\
% \hline
% \end{tabular}
% \caption{\label{Step-Level error}
% Description of step level errors
% % \textcolor{red} {You can add examples to this table and maybe their corrections.}
% }
% \end{table*}
% \begin{table*}
% \centering
% \begin{tabular}{p{0.25\textwidth} p{0.65\textwidth}}
% \hline
% \textbf{Error Type} & \textbf{Description} \\
% \hline
% Planning Error & Reasoning steps lacks logical coherence or fails to form a valid strategy to solve the given problem. \\
% Missing Step Error & Reasoning steps which are necessary for reaching the correct solution are omitted. \\
% \hline
% \end{tabular}
% \caption{\label{ReasoningError}
% Description of reasoning chain level errors
% }
% \end{table*}
\section{Method}
In this work, we explore how the establishment of premise links improves the identification of errors in reasoning chains with LLMs. Our approach works in two steps. First, we identify premises for each step and augment a linear reasoning chain (LRC) to convert it into a Premise Augmented Reasoning Chain (PARC). And then verify each step in the PARC under its premises only. Lastly, we do a graph traversal to identify steps that are logically correct but have faulty premises to identify accumulation errors. In particular, we are interested in the following research questions.
\textbf{RQ 1} Given a sequential step-by-step answer to a math word problem, can LLMs identify premises for each step?
\textbf{RQ 2} Given premise annotations, can LLMs identify errors in reasoning more faithfully?
\textbf{RQ 3} Can LLMs perform the entire process \textit{end-to-end}, i.e., given a reasoning chain, can they identify premises for each step and detect errors?
\subsection{Premise Augmented Reasoning Chains (PARC)}
Let $\mathbf{q}$ represent the question, $\hat{a}$ the predicted answer, $a$ the ground truth answer, and $\mathbf{r_{\leq t}} = [s_1, s_2, \dots, s_t]$ the generated reasoning chain composed of $t$ intermediate steps $s_i$, leading to $\hat{a}$. Using CoT reasoning, the predicted answer $\hat{a}$ is derived by first generating a reasoning chain $\mathbf{r_{\leq t}}$. The probability distribution for $\hat{a}$ can be expressed as:
\[
\mathbb{P}_{\mathrm{LM}}(\hat{a} \mid \mathbf{q}) = \mathbb{P}_{\mathrm{LM}}(\hat{a} \mid \mathbf{r_{\leq t}}) \times \mathbb{P}_{\mathrm{LM}}(\mathbf{r_{\leq t}} \mid \mathbf{q}),
\]
where the reasoning chain $\mathbf{r_{\leq t}}$ can be decomposed into intermediate steps as:
\[
\mathbb{P}_{\text{LM}} (\mathbf{r_{\leq t}} \mid \mathbf{q}) = \mathbb{P}_{\text{LM}} (s_1 \mid \mathbf{q}) \times \prod_{i=1}^{t-1} \mathbb{P}_{\text{LM}} (s_{i+1} \mid \mathbf{q}, s_i).
\]
Our objective is to identify errors in the intermediate steps $s_i$ of the reasoning chain $\mathbf{r_{\leq t}}$.
The \textit{premises} for a step $\mathbf{s}_i$ are defined as the necessary and sufficient subset of prior steps, denoted as
\[
\mathcal{P}_i \subseteq \{s_j \forall j < i\},
\]
satisfying the following properties:
1. \textbf{Verifiability}: The correctness of $s_i$ is verifiable based on $\mathcal{P}_i$ alone:
\[
\mathcal{F}(s_i \mid \mathcal{P}_i) = 1.
\]
2. \textbf{Minimality}: The set $\mathcal{P}_i$ is minimal such that removing any element $\mathbf{s}_j \in \mathcal{P}_i$ results in $s_i$ becoming unverifiable:
\begin{align}
\mathcal{F}(s_i \mid \mathcal{P}_i \setminus \{s_j\}) &= 0,
\quad \forall s_j \in \mathcal{P}_i.
\end{align}
Given premises $\mathcal{P}_i$ for a step $\mathbf{s}_i$, we convert a linear reasoning chain into a PARC, where each step $\mathbf{s}_i$ is augmented with its premises $\mathcal{P}_i$. $\mathcal{F}$ is a function that estimates whether a step is verifiable or not.
\[
\mathbf{r_{\leq t}}' = [(s_1, \mathcal{P}_1), (s_2, \mathcal{P}_2), \dots, (s_t, \mathcal{P}_t)],
\]
where $\mathbf{r_{\leq t}}'$ represents the transformed reasoning chain.
\subsection{Progressive Premise Mapping}
To transform a reasoning chain \(\mathbf{r_{\leq t}} = [s_1, s_2, \dots, s_t]\) into a premise-augmented reasoning chain (PARC), each step \(s_i\) must be explicitly linked to its premises \(\mathcal{P}_i\). We explore two approaches to identify these premises: \textit{Aggregative Premise Mapping} and \textit{Dyadic Premise Mapping}.
\subsubsection{Aggregative Premise Mapping}
In the \textit{Aggregative} approach, the premises for each step \(\mathbf{s}_k\) are collectively identified by querying an LLM with the complete reasoning context up to the step. For a given question \(\mathbf{q}\), the reasoning chain so far $\mathbf{r_{< k}}$, and the next step \(s_k\), the LLM is prompted to output \(\mathcal{P}_k\).
\subsubsection{Dyadic Premise Mapping}
In the \textit{Dyadic} approach, the task of identifying premises is reformulated as a pairwise evaluation. Instead of querying the LLM to identify all premises \(\mathcal{P}_k\) for \(s_k\) at once, we evaluate whether each individual step \(s_i\) (where \(i < k\)) serves as a valid premise for \(s_k\). For each pair \((s_i, s_k)\), the LLM is queried to compute:
\[
\mathcal{I}(s_k \mid s_i) =
\begin{cases}
1, & \text{if } s_i \text{ is a valid premise for } s_k, \\
0, & \text{otherwise}.
\end{cases}
\]
The premises for \(s_k\) are then given by:
\[
\mathcal{P}_k = \{s_i \mid \mathcal{F}(s_k \mid s_i) = 1, \, \forall i < k\}.
\]
\subsection{Error Identification}
Next, we present our setup for error identification and how premises play a significant role in this task. We introduce a taxonomy of error types that occur in math reasoning with LLMs and then describe our approach to identify them.
\subsubsection{Types of Error}
Traditionally, prior research has predominantly focused on identifying \textit{native errors}, which refer to inaccuracies inherent to individual reasoning steps. These errors often arise from issues such as incorrect mathematical calculations (defined as \textit{Mathematical Error}), logical irregularities (defined as \textit{logical inconsistencies}) and are evaluated independently of the broader reasoning context. Although significant, this focus on native errors tends to overlook another critical category of errors, called \textit{accumulation errors}.
An \textit{accumulation error} occurs when a reasoning step is valid in isolation but is built upon one or more erroneous premises from earlier steps. Unlike native errors, accumulation errors emerge from the compounding effects of prior inaccuracies, propagating through sequential steps. Formally, in the context of the reasoning chain, a step \(s_i\) is classified as:
1. \textit{Correct}, if it contains no logical or mathematical errors, and is performed under premises that are also \textit{correct}
2. A \textit{native error} if it contains an inherent discrepancy (e.g., a miscalculation or logical inconsistency)
3. An \textit{accumulation error} if \(s_i\) is logically valid, but at least one of its premises is incorrect.
By addressing both native and accumulation errors, we can achieve a more comprehensive understanding of error dynamics in sequential reasoning processes.
\begin{algorithm}[tb]
\small
\caption{Constructing and Evaluating PARC}
\label{alg:pipeline}
\begin{algorithmic}
\STATE {\bfseries Input:} $R = [s_1, s_2, \dots, s_t]$
{\small \texttt{// Step 1: Premise Extraction}}
\FOR{$k=1$ {\bfseries to} $t$}
\STATE $\mathcal{P}_k \gets \textsc{ExtractPremise}(s_k, \{s_1,\dots,s_{k-1}\})$
\ENDFOR
\STATE $R' \gets [(s_1, \mathcal{P}_1), \dots, (s_t, \mathcal{P}_t)]$
{\small \texttt{// Step 2: Error Detection}}
\FOR{$k=1$ {\bfseries to} $t$}
\STATE $\mathcal{E}_k^{(1)} \gets \textsc{IsMathematicalError}(s_k)$
\STATE $\mathcal{E}_k^{(2)} \gets \textsc{IsLogicallyInconsistent}(s_k, \mathcal{P}_k)$
\STATE $\mathcal{E}_k \gets \mathcal{E}_k^{(1)} \lor \mathcal{E}_k^{(2)}$
\ENDFOR
{\small \texttt{// Step 3: Accumulation Error Detection}}
\FOR{$k=1$ {\bfseries to} $t$}
\IF{$s_k$ is correct}
\FOR{$s_j \in \mathcal{P}_k$}
\IF{$s_j$ is incorrect}
\STATE $\mathcal{E}_k \gets \textsc{AccumulationError}$
\STATE \textbf{break}
\ENDIF
\ENDFOR
\ENDIF
\ENDFOR
\STATE $R' \gets [(s_1, \mathcal{P}_1, \mathcal{E}_1), \dots, (s_t, \mathcal{P}_t, \mathcal{E}_t)]$
\end{algorithmic}
\end{algorithm}
\subsubsection{Error Identification}
\begin{figure*}[t!]
\centering
\includegraphics[width=\linewidth]{example.pdf}
\vspace{-2.5em}
\caption{An example where the baseline method fails to detect errors, while our verification method with established premise links successfully identifies the mathematical error in step 6, and the accumulation error in step 7.}
% \vspace{-1.5em}
\label{fig:example}
\end{figure*}
\paragraph{Baseline.} In the baseline approach, a large language model (LLM) is used in a zero-shot setting to classify the reasoning steps. For a given question \(\mathbf{q}\), the reasoning chain so far \(r_{<k}\), and the next step proposed \(s_k\), the model is queried to assign an error type \(\mathcal{E}_k\) for \(s_k\) according to the predefined error taxonomy (\textit{Correct} / \textit{Mathematical Error} / \textit{Logical Error} / \textit{Accumulation Error}):
% \[
% \mathcal{E}_k \in \{\texttt{Mathematical Error}, \texttt{Logical Inconsistency}, \texttt{Accumulation Error}, \texttt{None}\}.
% \]
The model processes the entire context \((\mathbf{q}, r_{< k}, s_k)\) and generates a classification directly. The zero-shot prompt explicitly defines all error types to guide the classification.
\paragraph{Proposed Approach.} Our proposed approach refines the classification process by situating each reasoning step \(\mathbf{s}_k\) explicitly within the context of its premises \(\mathcal{P}_k\). Algorithm \ref{alg:pipeline} illustrates the overall pipeline of conversion of LRC to PARC and step-by-step error classification. The prompts for the following are reported in Appendix \ref{tab:ours_math_error} and \ref{tab:ours_logical_error}.
\subparagraph{1. Mathematical Error:} To detect mathematical errors, we prompt an LLM to assess whether \(s_k\) contains a mathematical error by analyzing the step in isolation, excluding the broader reasoning chain.
\subparagraph{2. Logical Inconsistency:} For the detection of logical inconsistencies, we restrict the evaluation to the premises \(\mathcal{P}_k\) of the reasoning step \(s_k\). The LLM is prompted to determine whether \(s_k\) is logically consistent with its premises. This ensures that the step's validity is evaluated in the context of the dependencies defined by \(\mathcal{P}_k\), without considering unrelated parts of the reasoning chain.
\subparagraph{3. Accumulation Error:} After identifying native errors, accumulation errors are identified by analyzing the dependency graph of the reasoning chain. The premises \(\mathcal{P}_k\) form directed edges in the graph, where each step depends on its premises. A Depth-First Search (DFS) traversal is performed, and a step \(s_k\) is classified as an Accumulation Error if it satisfies the following criteria: 1. \(s_k\) itself is classified as correct, and 2. At least one premise \(s_j \in \mathcal{P}_k\) (where \(j < k\)) is classified as incorrect.
% \subsection{Reasoning-chain level errors}
% For reasoning chain level errors we consider the following two error types: Planning error and Missing Step Error. Each of these errors are detailed in Table~\ref{ReasoningError}.
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lccc|ccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{GSM8K}} & \multicolumn{3}{c|}{\textbf{MATH}} & \multicolumn{3}{c|}{\textbf{Orca Math}} & \multicolumn{3}{c}{\textbf{MetaMathQA}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
Llama 3.1 8b & 65.64 & 89.33 & 75.66 & 51.00 & 82.40 & 62.94 & 54.65 & 79.77 & 64.85 & 53.92 & 81.42 & 64.75 \\
Llama 3.1 70b & 81.26 & 97.55 & 88.65 & 69.80 & 96.82 & 81.11 & 71.05 & 96.73 & 81.91 & 73.10 & 96.46 & 83.05 \\
\midrule
Qwen 2.5 7b & 67.42 & 63.53 & 65.36 & 55.87 & 59.73 & 57.70 & 63.08 & 72.64 & 67.05 & 55.54 & 61.66 & 58.14 \\
Qwen 2.5 32b & 84.07 & 95.35 & 89.34 & 72.56 & 88.55 & 79.74 & 74.27 & 90.63 & 81.62 & 75.57 & 90.89 & 82.49 \\
\midrule
GPT4o-mini & 72.54 & 86.18 & 78.75 & 57.72 & 69.29 & 62.96 & 60.12 & 73.96 & 66.31 & 60.77 & 76.70 & 67.71 \\
GPT-4o & 85.93 & 94.42 & 89.96 & 69.40 & 89.78 & 78.28 & 71.12 & 92.17 & 80.79 & 75.49 & 90.37 & 81.85 \\
\bottomrule
\end{tabular}
\caption{Precision, Recall and F1 scores for premise identification under the Aggregative approach. Note that ideally this needs to be a high recall system, because missing even a single premise step could hurt the verifiability of the reasoning chain.}
\label{tab:premise_identification}
\end{table*}
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
\textbf{GPT4o-mini} \\
Aggregative & 74.93 & 84.81 & 79.56 & 70.85 & 87.01 & 78.10 & 71.84 & 86.73 & 78.59 \\
Dyadic & 75.04 & 82.82 & 78.74 & 64.57 & 71.12 & 67.68 & 55.18 & 53.98 & 54.57 \\
\midrule
\textbf{GPT4o} \\
Aggregative & 89.38 & 93.66 & 91.47 & 84.97 & 94.95 & 89.69 & 83.43 & 94.65 & 88.72 \\
Dyadic & 75.32 & 99.01 & 85.55 & 67.70 & 92.68 & 78.25 & 67.27 & 79.42 & 72.84 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Precision, Recall and F1 scores for the premise mapping task for the GSM8K dataset under the aggregative and dyadic approaches. We saw a consistent drop in F1 score for the dyadic approach, in spite of it being computationally more expensive.}
\label{tab:premise_identification_per_candidate_score}
\end{table*}
\section{PERL: Premises and ERrors identification in Language models}
In order to test the ability of LLMs to identify premises and error categories, we designed a testbed. We used existing datasets of math word problems to create PERL, our testbed for step-level premise and error annotations.
\paragraph{Generating Reasoning Chains:}
In order to generate the reasoning chains, we used two popular benchmarks (i) GSM8K \cite{cobbe2021trainingverifierssolvemath}, a collection of 8,500 grade school math word problems, and (ii) MATH \cite{hendrycksmath2021}, a dataset of 12,500 challenging competition level math word problems. In addition, we use the (iii) Orca-Math dataset by \citet{mitra2024orcamathunlockingpotentialslms}, a synthetic dataset of 200K math problems alongside solutions written by GPT4Turbo, and the (iv) MetaMathQA dataset by \cite{yu2023metamath}. We first randomly sampled 1000 examples from the GSM8k and MATH test split and the Orca-Math and MetaMathQA training split (since these are training datasets). We began by using the open weight Large Language Model Llama-3.1-8B-Instruct \cite{grattafiori2024llama3herdmodels} to generate step-by-step reasoning chains. Next, we categorized the reasoning chains as positive or negative, depending on whether the final answer matches the ground-truth answer. Next, we randomly sampled 50 positive (correct) and 50 negative (incorrect) reasoning chains. To expand our dataset, we employed GPT-4o to systematically introduce mathematical or logical errors into the correct reasoning chains, creating additional synthetic negative examples (since prior work also focused on such synthetic negatives). When introducing errors, we ensured that the subsequent steps were appropriately modified to reflect the impact of the initial error. In contrast with existing works such as \citet{golovneva2023roscoesuitemetricsscoring}, where the perturbation in a step is not followed up in subsequent steps, our synthetic negatives are more realistic.
\paragraph{Premise and Error Annotation:} Next, for each step in the reasoning chain, we identified the ground truth premises for each step using the OpenAI o1-Preview model \cite{openai2024openaio1card}. Then, we use the same o1-Preview model to map each step to the predefined taxonomy of errors or mark it as correct, to create ground-truth error annotations. Upon obtaining the annotated data sets, 2 authors manually went through 10\% of the generated data points to identify the discrepancy between the human annotations and the annotations done by the o1 model. Manual inspection of 71 samples showed that, in the case of premise annotations, only 4 out of 71 had a discrepancy and 5 out of 71 had issues in error annotations. For premise identification, the discrepancy typically indicates additional steps as premises. However, for error annotation, the annotation errors do not have a clear pattern and manifest themselves in forms of misclassification of the correctness of the step. The prompts used to generate the annotations of the premises and errors are in Appendix \ref{sec:appendix_prompt}.
% This results in a total of 607 reasoning chains with 203 positives, 214 negatives, and 190 synthetic negatives. In total, we have 2,134 steps annotated as \textit{Correct}, 321 steps as \textit{Mathematical Error}, 443 steps as \textit{Logical Inconsistency}, and 741 steps as \textit{Accumulation Error}, indicating that native and accumulation errors appear at almost equal rates.
A detailed summary of the dataset statistics can be found in Appendix \ref{appx:data-stats}
% Additionally, we model each chain-of-thought as a directed acyclic graph (PARC) based on the step-level premise annotations. Across 1,370 PARCs, each chain contains on average 7.30 steps, representing the length of the chain-of-thought; 11.27 premises, corresponding to the total premise references across steps; and 10.42 edges linking premises to conclusions. The average depth of the PARC is 6.02, measuring the longest path of dependencies, while the maximum width is 1.90, quantifying how many steps can appear at the same layer in the PARC. Finally, the branching factor is 1.37, which is the ratio of edges to nodes, indicating that each step typically spawns fewer than two subsequent steps on average.
% \paragraph{Data statistics:}
% This results in a total of 607 reasoning chains with 203 positives, 214 negatives, and 190 synthetic negatives. In total, we have 2,134 steps annotated as \textit{Correct}, 321 steps as \textit{Mathematical Error}, 443 steps as \textit{Logical Inconsistency}, and 741 steps as \textit{Accumulation Error}, indicating that native and accumulation errors appear at almost equal rates. Additionally, we model each chain-of-thought as a directed acyclic graph (PARC) based on the step-level premise annotations. Across 1,370 PARCs, each chain contains on average 7.30 steps, representing the length of the chain-of-thought; 11.27 premises, corresponding to the total premise references across steps; and 10.42 edges linking premises to conclusions. The average depth of the PARC is 6.02, measuring the longest path of dependencies, while the maximum width is 1.90, quantifying how many steps can appear at the same layer in the PARC. Finally, the branching factor is 1.37, which is the ratio of edges to nodes, indicating that each step typically spawns fewer than two subsequent steps on average.
% \begin{table*}[!t]
% \centering
% \setlength\tabcolsep{4pt}
% \fontsize{8pt}{9pt}\selectfont
% \begin{tabular}{lccc|ccc|ccc|ccc}
% \toprule
% \multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{GSM8K}} & \multicolumn{3}{c|}{\textbf{MATH}} & \multicolumn{3}{c|}{\textbf{Orca Math}} & \multicolumn{3}{c}{\textbf{MetaMathQA}} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
% & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
% \midrule
% Llama 3.1 8b & 65.64 & 89.33 & 75.66 & 51.00 & 82.40 & 62.94 & 54.65 & 79.77 & 64.85 & 53.92 & 81.42 & 64.75 \\
% Llama 3.1 70b & 81.26 & 97.55 & 88.65 & 69.80 & 96.82 & 81.11 & 71.05 & 96.73 & 81.91 & 73.10 & 96.46 & 83.05 \\
% \midrule
% Qwen 2.5 7b & 67.42 & 63.53 & 65.36 & 55.87 & 59.73 & 57.70 & 63.08 & 72.64 & 67.05 & 55.54 & 61.66 & 58.14 \\
% Qwen 2.5 32b & 84.07 & 95.35 & 89.34 & 72.56 & 88.55 & 79.74 & 74.27 & 90.63 & 81.62 & 75.57 & 90.89 & 82.49 \\
% \midrule
% GPT4o-mini & 72.54 & 86.18 & 78.75 & 57.72 & 69.29 & 62.96 & 60.12 & 73.96 & 66.31 & 60.77 & 76.70 & 67.71 \\
% GPT-4o & 85.93 & 94.42 & 89.96 & 69.40 & 89.78 & 78.28 & 71.12 & 92.17 & 80.79 & 75.49 & 90.37 & 81.85 \\
% \bottomrule
% \end{tabular}
% \caption{Precision, Recall and F1 scores for premise identification under the Aggregative approach. Note that ideally this needs to be a high recall system, cause missing even a single premise step could hurt the verifiability of the reasoning chain.}
% \label{tab:premise_identification}
% \end{table*}
% \section{Experimental Details}
% \paragraph{Model}
% For the Llama model \cite{grattafiori2024llama3herdmodels}, we used vLLM \cite{kwon2023efficientmemorymanagementlarge} for model serving and AzureOpenAI for the GPT4o and GPT4-o1 \cite{openai2024o1} models. To ensure reproducibility, all generations were performed with a temperature=0. To annotate the premise and error types, we used the GPT4o1 model. For all models, we used their instruct variant.
% \begin{table*}[t!]
% \centering
% \resizebox{\textwidth}{!}{
% \begin{tabular}{cc|cccc|cccc}
% \hline
% \textbf{Model} & \textbf{Con-} & \multicolumn{4}{c|}{\textbf{GSM Negatives}} & \multicolumn{4}{c}{\textbf{GSM Positives Perturbed}} \\
% \textbf{Name}& \textbf{text} & \textbf{COR ($\uparrow$)} & \textbf{ERR ($\uparrow$)} & \textbf{AE ($\uparrow$)} & \textbf{Avg.} & \textbf{COR ($\uparrow$)} & \textbf{ERR ($\uparrow$)} & \textbf{AE ($\uparrow$)} & \textbf{Avg.} \\
% \hline
% Llama 3.1 & Full & 97.3 & 44.15 & 4.8 & 55.34 & 87.5 & 66.7 & 9.7 & 58.58 \\
% (8B) & Partial & 80.77 & 65.28 & 50 & 67.35 & 83.75 & 93.94 & 50.65 & 77.52 \\
% Llama 3.1 & Full & 95.5 & 61.03 & 10.57 & \textbf{60.48} & 97.5 & 86.3 & 35.36 & \textbf{76.13} \\
% (70B) & Partial & 93.08 & 76.39 & 78.26 & \textbf{84.36} & 95 & 90.91 & 72.73 & 87.32 \\
% \hline
% Qwen 2.5 & Full & 89.62 & 19.48 & 2.8 & - & 87.5 & 46.96 & 3.65 & - \\
% (7B) & Partial & - & - & - & - & - & - & - & - \\
% Qwen2.5 & Full & 78.15 & 29.87 & 4.8 & - & 78.75 & 65.15 & 12.19 & - \\
% (72B-Instruct) & Partial & - & - & - & - & - & - & - & - \\
% \hline
% \multirow{2}{*}{GPT-4o-mini} & Full & 93.3 & 50.6 & 10.57 & 56.95 & 95 & 87.8 & 25.6 & 72.82 \\
% & Partial & 91.54 & 54.17 & 72.83 & 76.53 & 91.25 & 90.91 & 68,83 & 84.72 \\
% \multirow{2}{*}{GPT-4o} & Full & 97 & 46.75 & 9.6 & 57.34 & 100 & 83.3 & 30.48 & 74.76 \\
% & Partial & 93.85 & 54.17 & 82.61 & 80.62 & 96.25 & 86.36 & 80.52 & \textbf{88.91} \\
% \hline
% \end{tabular}
% }
% \caption{Classification accuracy for the step-level error identification task on GSM Negatives and GSM Positives Perturbed. COR stands for Correct, ERR for Error, and AE for Accumulated Error.}
% \label{tab:gsm_negatives_perturbed}
% \end{table*}
% \begin{table*}[!t]
% \centering
% \setlength\tabcolsep{4pt}
% \fontsize{8pt}{9pt}\selectfont
% \begin{tabular}{cccc|ccc|ccc}
% \toprule
% \multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
% & Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
% \midrule
% \textbf{GPT4o-mini} \\
% Aggregative & 74.93 & 84.81 & 79.56 & 70.85 & 87.01 & 78.10 & 71.84 & 86.73 & 78.59 \\
% Dyadic & 75.04 & 82.82 & 78.74 & 64.57 & 71.12 & 67.68 & 55.18 & 53.98 & 54.57 \\
% \midrule
% \textbf{GPT4o} \\
% Aggregative & 89.38 & 93.66 & 91.47 & 84.97 & 94.95 & 89.69 & 83.43 & 94.65 & 88.72 \\
% Dyadic & 75.32 & 99.01 & 85.55 & 67.70 & 92.68 & 78.25 & 67.27 & 79.42 & 72.84 \\
% \bottomrule
% \end{tabular}
% \caption{Precision, Recall and F1 scores for the premise mapping task for the GSM8K dataset under the aggregative and dyadic approaches. We saw a consistent drop in F1 score for the dyadic approach, in spite of it being computationally more expensive.}
% \label{tab:premise_identification_per_candidate_score}
% \end{table*}
\section{Results and Discussion}
For detailed information on our experimental setup, including model configurations and implementation details, refer to Appendix \ref{appx:model-choice}.
\subsection{Premise Mapping}
In this section we try to answer RQ1 - ``Can LLMs identify premises to convert a Linear Reasoning Chain to a PARC?"
\paragraph{Metrics:} For identification of premises, we use precision, recall, and F1 score as metrics. However, we would want to highlight to the reader that the most important metric here is Recall. Since marking an extra step as a premise will not hurt the verifiability of the step, missing one will hurt its verifiability and all subsequent steps that rely on this step as a premise. Further, we note that some reasoning chains have a higher number of reasoning steps compared to others. In order to ensure robustness of the metrics, we first compute them at the level of each data point and later average the metrics across the dataset. We tried two approaches for the task of premise mapping.
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lcccccccccccccccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model}} & \multicolumn{4}{c}{\textbf{GSM8K}} & \multicolumn{4}{c}{\textbf{MATH}} & \multicolumn{4}{c}{\textbf{Orca Math}} & \multicolumn{4}{c}{\textbf{MetaMathQA}} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
& Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg \\
\textbf{Llama 3.1 8b} \\
Full Context & 48.87 & 57.68 & 91.9 & 58.6 & 46.91 & 60.29 & 90.61 & 60.2 & 43.26 & 51.2 & 96.7 & 55.06 & 50.47 & 55.15 & 95.17 & 59.45 \\
Model Premises & 54.61 & 76.74 & 65.44 & 64.53 & 52.16 & 59.83 & 50.96 & 54.88 & 55.57 & 59.82 & 67.36 & 59.22 & 54.52 & 69.13 & 62.63 & 61.78 \\
\midrule
\textbf{Llama 3.1 70b} \\
Full Context & 58.35 & 77.03 & 98.61 & 71.37 & 55.55 & 72.83 & 96.24 & 69.77 & 50.33 & 64.25 & 96.74 & \cellcolor{lightblue}63.52 & 55.59 & 73.13 & 97.93 & 69.49 \\
Model Premises & 74.51 & 86.22 & 94.69 & 81.92 & 70.94 & 79.29 & 77.91 & 75.45 & 64.95 & 72.28 & 93.37 & \cellcolor{darkerblue}72.52 & 65.69 & 82.74 & 89.71 & 76.52 \\
\midrule
\textbf{Qwen 2.5 7b} \\
Full Context & 40.88 & 54.13 & 100 & \cellcolor{lightblue}54.94 & 40.47 & 53.60 & 98.07 & 56.26 & 41.20 & 53.69 & 100 & \cellcolor{lightblue}55.85 & 43.36 & 53.18 & 99.39 & \cellcolor{lightblue}56.26 \\
Model Premises & 53.03 & 78.40 & 90.37 & \cellcolor{darkerblue}69.73 & 58.47 & 66.87 & 82.72 & 65.96 & 52.07 & 71.12 & 84.28 & \cellcolor{darkerblue}65.38 & 54.98 & 73.87 & 91.02 & \cellcolor{darkerblue}68.43 \\
\midrule
\textbf{Qwen 2.5 32b} \\
Full Context & 46.76 & 69.26 & 99.8 & \cellcolor{lightblue}63.56 & 49.58 & 66.5 & 98 & \cellcolor{lightblue}65.11 & 49.44 & 63.01 & 99.5 & 63.06 & 49.7 & 67.8 & 99.6 & \cellcolor{lightblue}65.02 \\
Model Premises & 65.58 & 86.34 & 96.95 & \cellcolor{darkerblue}79.37 & 68.61 & 76.96 & 88.71 & \cellcolor{darkerblue}75.57 & 61.89 & 69.24 & 95.25 & 70.26 & 62.18 & 85.60 & 97.32 & \cellcolor{darkerblue}77.40 \\
\midrule
\textbf{Qwen 2.5 72b} \\
Full Context & 47.48 & 67.17 & 100 & \cellcolor{lightblue}63.11 & 53.19 & 67.87 & 99.79 & \cellcolor{lightblue}67.52 & 50.41 & 60.33 & 100 & \cellcolor{lightblue}62.50 & 47.31 & 63.59 & 99.05 & \cellcolor{lightblue}62.20 \\
Model Premises & 69.87 & 85.12 & 96.38 & \cellcolor{darkerblue}80.56 & 69.51 & 79.92 & 90.84 & \cellcolor{darkerblue}77.28 & 69.45 & 71.34 & 94.19 & \cellcolor{darkerblue}74.41 & 64.73 & 85.52 & 98.12 & \cellcolor{darkerblue}78.53 \\
\midrule
\textbf{GPT4o-mini} \\
Full Context & 52.68 & 73.14 & 95.02 & 66.68 & 50.41 & 62.39 & 93.4 & 63.03 & 55.22 & 61.18 & 98.6 & 64.59 & 51.52 & 65.42 & 97.35 & 64.47 \\
Model Premises & 58.25 & 80.81 & 86.72 & 72.33 & 65.84 & 72.4 & 71.88 & 69.45 & 61.18 & 72.48 & 85.94 & 69.93 & 57.04 & 81.63 & 91.15 & 72.51 \\
\midrule
\textbf{GPT-4o} \\
Full Context & 53.81 & 75.3 & 98.33 & 68.52 & 50.9 & 68.41 & 98.57 & \cellcolor{lightblue}66.52 & 59.55 & 65.75 & 100 & 68.55 & 56.26 & 71.72 & 99.3 & 69.41 \\
Model Premises & 65.23 & 86.67 & 99.76 & 79.82 & 70.12 & 76.67 & 91.84 & \cellcolor{darkerblue}76.45 & 66.33 & 71.74 & 93.14 & 72.96 & 67.28 & 88.19 & 89.85 & 79.41 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Accuracy for Error identification of various models under Full Context (baseline) and Premises settings across different datasets. Note that Neg, Syn and Pos means Negative, Synthetic Negatives and Positive splits of PERL, as explained earlier. For each dataset, we highlighted the models that benefit the most from the PARC.}
\vspace{-2ex}
\label{tab:error_identification_restructured}
\end{table*}
\paragraph{Aggregative Approach:}
Table \ref{tab:premise_identification} contains the results of the Aggregative approach for a suite of models and datasets. From the table, it is evident that LLMs are efficient in identifying premises for a given step. In particular, Llama 3.1 70b \cite{grattafiori2024llama3herdmodels}, Qwen 2.5 32b \cite{qwen2025qwen25technicalreport} and GPT4o are very efficient in this task, all having a recall greater than 90\%. It also shows that these models can convert an LRC to a PARC with high accuracy.
We note that the scale of the model is an important factor. The increase in recall varies from 8.2\% to 32.2\% when we scale up for the same model class, and the pattern is consistent. Tables \ref{tab:gsm8k_premise_identification_per_candidate_score}, \ref{tab:math_premise_identification_per_candidate_score}, \ref{tab:orca_math_premise_identification_per_candidate_score} and \ref{tab:meta_math_premise_identification_per_candidate_score} in the appendix contain the detailed metrics across the splits Positive, Negative and Synthetic negatives for GSM8K, MATH, OrcaMath and MetaMathQA respectively.
\paragraph{Dyadic Approach: }
Table \ref{tab:premise_identification_per_candidate_score} provides a comparative analysis of precision, recall, and F1 scores for the GSM8K dataset using two models for the aggregative and dyadic approaches.
This approach causes the LLM to be more biased to mark a step as a premise, causing precision to drop significantly. In addition, this approach has a time complexity of $O(n^2)$, where $n$ is the number of reasoning steps. Note that for almost all cases the precision drops, and the change in recall gives a mixed view on how this approach might be helpful.
Our analysis revealed that this methodology often led the model to incorrectly classify second- or higher-order dependencies as premises, even when the current step did not directly rely on them. This misclassification contributed to the observed degradation in precision.
% \begin{table*}[t!]
% \centering
% \resizebox{\textwidth}{!}{
% \begin{tabular}{l|ccc|ccc|ccc}
% \hline
% \textbf{Model Name} & \multicolumn{3}{c|}{\textbf{GSM Positives}} & \multicolumn{3}{c|}{\textbf{GSM Negatives}} & \multicolumn{3}{c}{\textbf{GSM Positives Perturbed}} \\
% & \textbf{Precision} & \textbf{Recall} & \textbf{F1} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} \\
% \hline
% GPT-4o-mini & 74.8 & 91.5 & 82.31 & 65.9 & 84.5 & 79.35 & 59.7 & 74 & 66.09 \\
% GPT-4o & 66.5 & 97 & 78.91 & 60.1 & 92.7 & 72.92 & 56.2 & 80.8 & 66.29 \\
% \hline
% \end{tabular}
% }
% \caption{Premise Identification Performance Metrics for Different Models on GSM Positives, GSM Negatives, and GSM Positives Perturbed for the Pairwise Approach. The pairwise approach causes overprediction of premises, which further causes a drop in precision. }
% \label{tab:premise_identification_per_candidate_score}
% \end{table*}
% \begin{table*}[!t]
% \centering
% \setlength\tabcolsep{4pt}
% \fontsize{8pt}{9pt}\selectfont
% \begin{tabular}{lcccccccccccccccc}
% \toprule
% \multirow{2}[2]{*}{\textbf{Model}} & \multicolumn{4}{c}{\textbf{GSM8K}} & \multicolumn{4}{c}{\textbf{MATH}} & \multicolumn{4}{c}{\textbf{Orca Math}} & \multicolumn{4}{c}{\textbf{MetaMathQA}} \\
% \cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
% & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg \\
% \textbf{Llama 3.1 8b} \\
% Full Context & 48.87 & 57.68 & 91.9 & 58.6 & 46.91 & 60.29 & 90.61 & 60.2 & 43.26 & 51.2 & 96.7 & 55.06 & 50.47 & 55.15 & 95.17 & 59.45 \\
% Model Premises & 54.61 & 76.74 & 65.44 & 64.53 & 52.16 & 59.83 & 50.96 & 54.88 & 55.57 & 59.82 & 67.36 & 59.22 & 54.52 & 69.13 & 62.63 & 61.78 \\
% \midrule
% \textbf{Llama 3.1 70b} \\
% Full Context & 58.35 & 77.03 & 98.61 & 71.37 & 55.55 & 72.83 & 96.24 & 69.77 & 50.33 & 64.25 & 96.74 & \cellcolor{lightblue}63.52 & 55.59 & 73.13 & 97.93 & 69.49 \\
% Model Premises & 74.51 & 86.22 & 94.69 & 81.92 & 70.94 & 79.29 & 77.91 & 75.45 & 64.95 & 72.28 & 93.37 & \cellcolor{darkerblue}72.52 & 65.69 & 82.74 & 89.71 & 76.52 \\
% \midrule
% \textbf{Qwen 2.5 7b} \\
% Full Context & 40.88 & 54.13 & 100 & \cellcolor{lightblue}54.94 & 40.47 & 53.60 & 98.07 & 56.26 & 41.20 & 53.69 & 100 & \cellcolor{lightblue}55.85 & 43.36 & 53.18 & 99.39 & \cellcolor{lightblue}56.26 \\
% Model Premises & 53.03 & 78.40 & 90.37 & \cellcolor{darkerblue}69.73 & 58.47 & 66.87 & 82.72 & 65.96 & 52.07 & 71.12 & 84.28 & \cellcolor{darkerblue}65.38 & 54.98 & 73.87 & 91.02 & \cellcolor{darkerblue}68.43 \\
% \midrule
% \textbf{Qwen 2.5 32b} \\
% Full Context & 46.76 & 69.26 & 99.8 & \cellcolor{lightblue}63.56 & 49.58 & 66.5 & 98 & \cellcolor{lightblue}65.11 & 49.44 & 63.01 & 99.5 & 63.06 & 49.7 & 67.8 & 99.6 & \cellcolor{lightblue}65.02 \\
% Model Premises & 65.58 & 86.34 & 96.95 & \cellcolor{darkerblue}79.37 & 68.61 & 76.96 & 88.71 & \cellcolor{darkerblue}75.57 & 61.89 & 69.24 & 95.25 & 70.26 & 62.18 & 85.60 & 97.32 & \cellcolor{darkerblue}77.40 \\
% \midrule
% \textbf{Qwen 2.5 72b} \\
% Full Context & 47.48 & 67.17 & 100 & \cellcolor{lightblue}63.11 & 53.19 & 67.87 & 99.79 & \cellcolor{lightblue}67.52 & 50.41 & 60.33 & 100 & \cellcolor{lightblue}62.50 & 47.31 & 63.59 & 99.05 & \cellcolor{lightblue}62.20 \\
% Model Premises & 69.87 & 85.12 & 96.38 & \cellcolor{darkerblue}80.56 & 69.51 & 79.92 & 90.84 & \cellcolor{darkerblue}77.28 & 69.45 & 71.34 & 94.19 & \cellcolor{darkerblue}74.41 & 64.73 & 85.52 & 98.12 & \cellcolor{darkerblue}78.53 \\
% \midrule
% \textbf{GPT4o-mini} \\
% Full Context & 52.68 & 73.14 & 95.02 & 66.68 & 50.41 & 62.39 & 93.4 & 63.03 & 55.22 & 61.18 & 98.6 & 64.59 & 51.52 & 65.42 & 97.35 & 64.47 \\
% Model Premises & 58.25 & 80.81 & 86.72 & 72.33 & 65.84 & 72.4 & 71.88 & 69.45 & 61.18 & 72.48 & 85.94 & 69.93 & 57.04 & 81.63 & 91.15 & 72.51 \\
% \midrule
% \textbf{GPT-4o} \\
% Full Context & 53.81 & 75.3 & 98.33 & 68.52 & 50.9 & 68.41 & 98.57 & \cellcolor{lightblue}66.52 & 59.55 & 65.75 & 100 & 68.55 & 56.26 & 71.72 & 99.3 & 69.41 \\
% Model Premises & 65.23 & 86.67 & 99.76 & 79.82 & 70.12 & 76.67 & 91.84 & \cellcolor{darkerblue}76.45 & 66.33 & 71.74 & 93.14 & 72.96 & 67.28 & 88.19 & 89.85 & 79.41 \\
% \bottomrule
% \end{tabular}
% \caption{Accuracy for Error identification of various models under Full Context (baseline) and Premises settings across different datasets. Note that Neg, Syn and Pos means Negative, Synthetic Negatives and Positive splits of our PERL, as explained earlier. For each dataset, we highlighted the models that benefit the most from the PARC.}
% \label{tab:error_identification_restructured}
% \end{table*}
% \begin{table}[!t]
% \centering
% \setlength\tabcolsep{4pt}
% \fontsize{8pt}{9pt}\selectfont
% \begin{tabular}{lcccc}
% \toprule
% \textbf{Model} & \textbf{GSM8K} & \textbf{MATH} & \textbf{Orca-Math} & \textbf{MetaMath} \\
% \midrule
% \textbf{Llama 3.1 70b} \\
% Oracle Premises & 81.46 & 74.68 & 73.45 & 76.05 \\
% Model Premises & 81.92 & 75.45 & 72.52 & 76.52 \\
% \midrule
% \textbf{Qwen 2.5 72b} \\
% Oracle Premises & 80.58 & 79.50 & 74.87 & 79.49 \\
% Model Premises & 80.56 & 77.28 & 74.41 & 78.53 \\
% \midrule
% \textbf{GPT-4o} \\
% Oracle Premises & 78.74 & 76.79 & 75.56 & 82.88 \\
% Model Premises & 79.82 & 76.45 & 72.96 & 79.41 \\
% \bottomrule
% \end{tabular}
% \caption{Comparison of Error identification under oracle premises vs model generated premises. Since these models have high recall in premise identification, we observe that the error identification accuracy is comparable.}
% \label{tab:avg_performance_selected}
% \end{table}
\vspace{-2ex}
\subsection{Error Identification}
In this section, we address RQ2 and RQ3, i.e. is error identification more robust when we only use the premises for a given step as context. We explain the experimental setup for error identification for the baseline, as well as our approach, in the previous sections. For the identification of errors under premises, we consider two settings. For the first one, we use the oracle premise annotation (from the ground truth). For the second, we first identify the premises with the LLMs and then use those premises for the identification of errors. Note that in the previous section we established that LLMs can identify premises with a high recall, and in this experiment, we exploit that. The prompts we used for our experiments are shared in Appendix \ref{sec:baseline_error_prompts}.
\paragraph{Metrics:} For this task, we report the average accuracy for the identification of error types for each step. In our analysis, we observe that the boundaries between the error types \textit{Mathematical Error} and \textit{Logical Inconsistency} are quite thin, and models often classify one as the other, so we merge these two error categories into a single error type \textit{Error}, while keeping \textit{Accumulation Error} separate. Furthermore, since an early erroneous step for a solution containing multiple steps could cause the number of errors to skew toward \textit{accumulation errors}, we first normalize the accuracy by the number of steps in a datapoint, and later normalize across the dataset.
\paragraph{Results:}
\begin{table}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lcccc}
\toprule
\textbf{Model} & \textbf{GSM8K} & \textbf{MATH} & \textbf{Orca-Math} & \textbf{MetaMath} \\
\midrule
\textbf{Llama 3.1 70b} \\
Oracle Premises & 81.46 & 74.68 & 73.45 & 76.05 \\
Model Premises & 81.92 & 75.45 & 72.52 & 76.52 \\
\midrule
\textbf{Qwen 2.5 72b} \\
Oracle Premises & 80.58 & 79.50 & 74.87 & 79.49 \\
Model Premises & 80.56 & 77.28 & 74.41 & 78.53 \\
\midrule
\textbf{GPT-4o} \\
Oracle Premises & 78.74 & 76.79 & 75.56 & 82.88 \\
Model Premises & 79.82 & 76.45 & 72.96 & 79.41 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Comparison of Error identification under oracle premises vs model generated premises. Since these models have high recall in premise identification, we observe that the error identification accuracy is comparable.}
\vspace{-2ex}
\label{tab:premise_ablation}
\end{table}
Table \ref{tab:error_identification_restructured} presents the experimental results for the error identification task. We evaluate performance under two setups: under complete context as input (\textit{Full Context}), on the other hand, the \textit{Model Premises} setup denotes the end-to-end approach, wherein the model is tasked with first identifying the relevant premises and subsequently utilizing them. All models are provided with the question and the solutions generated up to the current step and are tasked with predicting the error type observed in the current step.
\textbf{Premises Help Error Identification :} It is evident from Table \ref{tab:error_identification_restructured}, that providing only the premise steps as context (instead of providing the full context) improves the accuracy of error detection with LLMs. Figure \ref{fig:example} shows one such example, where the baseline approach (using an LRC, and error detection performed in the full context) could not identify errors, but our approach was able to successfully identify the mathematical and accumulation errors. Our results show that the models are not as biased towards marking a step as correct, when given a more precise context (no distractor). Hence, upon providing only the premises, identification of errors becomes much more robust. However, it is important to note that this also comes at the cost of a drop of accuracy for correct steps, this can be observed from comparing the Pos column for Full context and Model Premises. However, this accuracy drop is much smaller for larger models. In addition, our approach impacts larger and more capable models more than smaller models.
\textbf{Models Struggle With Error Detection, Especially Accumulation Errors:} We find that, while capable of identifying correct steps, all models struggle with identifying errors. This can be attributed to the fact that models are inherently biased toward marking a step as correct. This issue was also pointed out by \citet{ling2023deductiveverificationchainofthoughtreasoning}. Identifying errors with directly connected premises as context yields significantly better performance than simply providing the full context. Moreover, models struggle to identify accumulation errors. Table \ref{tab:expanded_view} contains the accuracies across error types in the negative split of GSM8K. Note that, in full context, accuracy is quite high for correct steps, but lower for erroneous steps (mathematical and logical errors), and significantly lower for accumulation errors. However, in our approach, the identification of errors becomes significantly robust, with a minor drop in performance in identifying correct steps.
\textbf{Synthetic Negatives are Easier:} The average performance of all models on synthetically generated negatives is consistently higher compared to the performance on true negatives (evident from comparing the Neg and Syn columns in Table \ref{tab:error_identification_restructured}). This observation highlights an important insight: perturbing reasoning chains does not serve as a reliable proxy for evaluating a model's ability to identify genuine errors in reasoning. While prior work has focused on such synthetic negatives, it might provide an overoptimistic view of LLMs' ability to detect errors.
\begin{table}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lcccc}
\toprule
\textbf{Model} & \textbf{Correct} & \textbf{Error} & \textbf{Acc. Error} & \textbf{Avg} \\
\midrule
\textbf{Llama 3.1 70b} \\
Full Context & 96.79 & 60.87 & 12 & 58.35 \\
Oracle Premises & 90.18 & 75.25 & 55.63 & 74.41 \\
Model Premises & 88.78 & 75.25 & 57.54 & 74.51 \\
\midrule
\textbf{GPT-4o} \\
Full Context & 95.75 & 48.2 & 13.2 & 53.81 \\
Oracle Premises & 96.83 & 60.02 & 41.79 & 67.03 \\
Model Premises & 94.15 & 55.13 & 44.91 & 65.23 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Error identification accuracy for each type of steps in ground truth. Acc. Error means Accumulation Errors}
\label{tab:expanded_view}
\vspace{-2ex}
\end{table}
\textbf{Oracle Premises vs Model Generated Premises:} In Table \ref{tab:premise_ablation}, we present results of an ablation, where instead of providing the model generated premises, we provide the ground truth premises (oracle premises from PERL). As anticipated, the accuracy of error identification improves when oracle premises are provided. However, it is noteworthy that for most models, the performance remains comparable to that achieved with oracle premises. This can be attributed to the fact that all these models can identify premises with a recall of higher than 90.
\subsection{Experiments on ProcessBench}
\label{sec:processbench}
Beyond the PERL dataset, we also evaluate our framework on ProcessBench~\cite{zheng2024processbenchidentifyingprocesserrors}, a benchmark that contains human-annotated stepwise labels for mathematical solutions. Unlike PERL, ProcessBench only includes annotations up to the \emph{first} incorrect step in each solution.
\paragraph{Experimental Setup.} We focus on the step-level error identification task, following the protocol used by \citet{zheng2024processbenchidentifyingprocesserrors}. Specifically, for correct solutions (i.e., ones where the final answer is correct), every step is marked as correct. For incorrect solutions, the task is to identify the \emph{first} incorrect step in the chain; all subsequent steps are automatically deemed irrelevant by design of the benchmark. We adopt the same prompting strategy as in our previous experiments, with LLMs classifying each step as correct or incorrect. The baseline is a standard critic model as proposed by \cite{zheng2024processbenchidentifyingprocesserrors} that identifies the first erroneous step from the solution as a whole.
\paragraph{Results and Observations.} Tables \ref{tab:gsm8k_comparison} and \ref{tab:math_comparison} show that our findings on ProcessBench (on the GSM8K and MATH splits) align with the trends we observe on PERL. \emph{Premise-augmented verification} continues to outperform verification conducted in the full reasoning chain. We see consistent gains from evaluating each step under its extracted premises and classifying its correctness accordingly. This further strenghtens our previous findings.
\begin{table}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lccccc}
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Correct} & \textbf{Wrong} & \textbf{F1} & \textbf{Delta} \\
\midrule
\textbf{Qwen 2.5 7B} & Baseline & 66.3 & 36.7 & 47.2 & \\
& Ours & 60.1 & 38.6 & 47.0 & -0.2 \\
\midrule
\textbf{Qwen 2.5 32B} & Baseline & 97.9 & 43.0 & 59.8 & \\
& Ours & 95.9 & 55.1 & 70.0 & +10.2 \\
\midrule
\textbf{Qwen 2.5 72B} & Baseline & 98.4 & 61.4 & 75.6 & \\
& Ours & 97.8 & 59.7 & 74.1 & -1.5 \\
\midrule
\textbf{Llama 3.1 8B} & Baseline & 17.1 & 36.7 & 23.3 & \\
& Ours & 33.7 & 37.8 & 35.6 & +12.3 \\
\midrule
\textbf{Llama 3.1 70B} & Baseline & 77.7 & 57.5 & 66.1 & \\
& Ours & 89.6 & 70.0 & 78.6 & +12.5 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Performance comparison on the GSM8K split of ProcessBench with Baseline vs. PARC (Ours implies premise augmented verification) for various model.}
\vspace{-2ex}
\label{tab:gsm8k_comparison}
\end{table}
\begin{table}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lccccc}
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Correct} & \textbf{Wrong} & \textbf{F1} & \textbf{Delta} \\
\midrule
\textbf{Qwen 2.5 7B} & Baseline & 46.0 & 25.4 & 32.7 & \\
& Ours & 45.6 & 41.2 & 43.3 & +10.6 \\
\midrule
\textbf{Qwen 2.5 32B} & Baseline & 90.0 & 22.4 & 35.9 & \\
& Ours & 86.9 & 53.9 & 66.5 & +30.7 \\
\midrule
\textbf{Qwen 2.5 72B} & Baseline & 88.5 & 33.7 & 48.8 & \\
& Ours & 86.7 & 53.9 & 66.5 & +17.7 \\
\midrule
\textbf{Llama 3.1 8B} & Baseline & 5.6 & 19.1 & 8.7 & \\
& Ours & 11.0 & 27.5 & 15.7 & +7.1 \\
\midrule
\textbf{Llama 3.1 70B} & Baseline & 32.4 & 32.8 & 32.6 & \\
& Ours & 61.6 & 55.4 & 58.3 & +25.7 \\
\bottomrule
\end{tabular}
\vspace{-1ex}
\caption{Performance comparison on the MATH split of ProcessBench with Baseline vs. PARC (Ours implies premise augmented verification) for various model.}
\vspace{-2ex}
\label{tab:math_comparison}
\end{table}
\label{sec:related_work}
% \heng{add some related work about information aggregation, include your SmartBook paper}\zhenhailong{added}
%\zhenhailong{add our distinctive contribution here compared with AssistantBench/MindSearch}
% \begin{comment}
% \begin{itemize}
% \item WebLinx \citepp{lu2024weblinx}
% \item WebArena \citepp{zhou2023webarena}
% \item WebGPT \citepp{nakano2021webgpt}
% \item Mind2Web \citepp{deng2024mind2web}
% \item AutoGPT \citepp{}
% \end{itemize}
% \end{comment}
\vspace{-2ex}
\section{Conclusion}
In this paper, we introduced a framework to enhance the evaluation of mathematical reasoning chains in large language models by transforming Linear Reasoning Chains into Premise Augmented Reasoning Chains. Through experiments with our annotated dataset, PERL, we empirically show that error identification under premises in PARC is more reliable and has higher accuracy than error identification under full context in LRC. We also show that LLMs can convert an LRC to PARC with no additional guidance, and then can do error identification under the premises in PARC. Further, we show that accumulation errors are particularly challenging to detect, and our method improves their identification by verifying each step under its premises.
\section*{Acknowledgments}
This research project has benefited from the Microsoft Accelerate Foundation Models Research (AFMR) grant program through which leading foundation models hosted by Microsoft Azure along with access to Azure credits were provided to conduct the research.
\section*{Impact Statement}
The broader impact of this research extends to multiple domains where step-by-step reasoning is critical, including automated theorem proving, educational AI tutors, and scientific discovery. By enabling more reliable self-verification mechanisms in LLMs, our approach contributes to the long-term goal of enhancing the transparency and trustworthiness of AI systems in high-stakes applications.
Potential ethical considerations include the risk of overreliance on LLM-generated verification without human oversight, particularly in domains where reasoning errors can have significant real-world consequences.
This paper primarily aims to advance the field of Machine Learning by proposing a structured approach to reasoning evaluation. Although we acknowledge that any advances in LLM-based reasoning may have broader societal implications, we do not foresee any immediate ethical concerns beyond those generally associated with the development of AI reasoning systems.
% HERE
\label{sec:related_work}
% \heng{add some related work about information aggregation, include your SmartBook paper}\zhenhailong{added}
%\zhenhailong{add our distinctive contribution here compared with AssistantBench/MindSearch}
% \begin{comment}
% \begin{itemize}
% \item WebLinx \citepp{lu2024weblinx}
% \item WebArena \citepp{zhou2023webarena}
% \item WebGPT \citepp{nakano2021webgpt}
% \item Mind2Web \citepp{deng2024mind2web}
% \item AutoGPT \citepp{}
% \end{itemize}
% \end{comment}
\section{Appendix}
\subsection{Experimental Details}
\label{appx:model-choice}
\paragraph{Model}
For the Llama model \cite{grattafiori2024llama3herdmodels}, we used vLLM \cite{kwon2023efficientmemorymanagementlarge} for model serving and AzureOpenAI for the GPT4o and GPT4-o1 \cite{openai2024o1} models. To ensure reproducibility, all generations were performed with a temperature=0. For all models, we used their instruct variant.
\subsection{Data statistics}
\label{appx:data-stats}
This results in a total of 607 reasoning chains with 203 positives, 214 negatives, and 190 synthetic negatives. In total, we have 2,134 steps annotated as \textit{Correct}, 321 steps as \textit{Mathematical Error}, 443 steps as \textit{Logical Inconsistency}, and 741 steps as \textit{Accumulation Error}, indicating that native and accumulation errors appear at almost equal rates. Additionally, we model each chain-of-thought as a directed acyclic graph (PARC) based on the step-level premise annotations. Across 607 PARCs, each chain contains on average 7.30 steps, representing the length of the chain-of-thought; 11.27 premises, corresponding to the total premise references across steps; and 10.42 edges linking premises to conclusions. The average depth of the PARC is 6.02, measuring the longest path of dependencies, while the maximum width is 1.90, quantifying how many steps can appear at the same layer in the PARC. Finally, the branching factor is 1.37, which is the ratio of edges to nodes, indicating that each step typically spawns fewer than two subsequent steps on average.
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{lcccc|cccc|cccc|cccc}
\toprule
\multirow{2}[2]{*}{\textbf{Method}} & \multicolumn{4}{c|}{\textbf{GSM8K}} & \multicolumn{4}{c|}{\textbf{MATH}} & \multicolumn{4}{c|}{\textbf{Orca Math}} & \multicolumn{4}{c}{\textbf{MetaMathQA}} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13} \cmidrule(lr){14-17}
& Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg & Neg & Syn & Pos & Avg \\
\midrule
\textit{\textbf{Full Context}} \\
Llama 3.1 8b & 48.87 & 57.68 & 91.9 & 58.6 & 46.91 & 60.29 & 90.61 & 60.2 & 43.26 & 51.2 & 96.7 & 55.06 & 50.47 & 55.15 & 95.17 & 59.45 \\
Llama 3.1 70b & 58.35 & 77.03 & 98.61 & 71.37 & 55.55 & 72.83 & 96.24 & 69.77 & 50.33 & 64.25 & 96.74 & 63.52 & 55.59 & 73.13 & 97.93 & 69.49 \\
Qwen 2.5 7b & 40.88 & 54.13 & 100 & 54.94 & 40.47 & 53.60 & 98.07 & 56.26 & 41.20 & 53.69 & 100 & 55.85 & 43.36 & 53.18 & 99.39 & 56.26 \\
Qwen 2.5 32b & 46.76 & 69.26 & 99.8 & 63.56 & 49.58 & 66.5 & 98 & 65.11 & 49.44 & 63.01 & 99.5 & 63.06 & 49.7 & 67.8 & 99.6 & 65.02 \\
Qwen 2.5 72b & 47.48 & 67.17 & 100 & 63.11 & 53.19 & 67.87 & 99.79 & 67.52 & 50.41 & 60.33 & 100 & 62.50 & 47.31 & 63.59 & 99.05 & 62.20 \\
GPT4o-mini & 52.68 & 73.14 & 95.02 & 66.68 & 50.41 & 62.39 & 93.4 & 63.03 & 55.22 & 61.18 & 98.6 & 64.59 & 51.52 & 65.42 & 97.35 & 64.47 \\
GPT-4o & 53.81 & 75.3 & 98.33 & 68.52 & 50.9 & 68.41 & 98.57 & 66.52 & 59.55 & 65.75 & 100 & 68.55 & 56.26 & 71.72 & 99.3 & 69.41 \\
\midrule
\textit{\textbf{Oracle Premises}} \\
Llama 3.1 8b & 57.02 & 82.64 & 74.61 & 69.26 & 54.58 & 60.94 & 51.65 & 56.47 & 57.43 & 60.61 & 59.48 & 59.09 & 52.65 & 68.48 & 60.51 & 60.37 \\
Llama 3.1 70b & 74.41 & 85.12 & 94.69 & 81.46 & 68.55 & 79.88 & 78.11 & 74.68 & 66.78 & 72.65 & 93.37 & 73.45 & 63.31 & 83.15 & 92.16 & 76.05 \\
Qwen 2.5 7b & 52.40 & 79.37 & 89.56 & 69.74 & 56.86 & 70.23 & 79.85 & 65.96 & 55.01 & 66.66 & 76.65 & 63.39 & 56.32 & 77.06 & 91.94 & 70.45 \\
Qwen 32b & 66.57 & 88.64 & 96.95 & 80.76 & 69.24 & 82.26 & 89.32 & 77.98 & 63.58 & 69.88 & 96.31 & 71.4 & 59.84 & 85.74 & 96.87 & 76.38 \\
Qwen 2.5 72b & 69.48 & 86.34 & 98.98 & 80.58 & 72.98 & 81.10 & 91.03 & 79.50 & 69.76 & 72.77 & 93.24 & 74.87 & 66.23 & 86.21 & 98.06 & 79.50 \\
GPT4o-mini & 63.83 & 85.77 & 91.29 & 76.16 & 65.86 & 74.99 & 78.41 & 71.71 & 64.03 & 72.66 & 83.56 & 70.74 & 58.77 & 85.11 & 91.44 & 74.73 \\
GPT-4o & 67.03 & 87.2 & 94.76 & 78.74 & 70.32 & 77.3 & 90.65 & 76.79 & 71.19 & 73.69 & 92.15 & 75.56 & 72.76 & 88.95 & 94.82 & 82.88 \\
\midrule
\textit{\textbf{Model Premises}} \\
Llama 3.1 8b & 54.61 & 76.74 & 65.44 & 64.53 & 52.16 & 59.83 & 50.96 & 54.88 & 55.57 & 59.82 & 67.36 & 59.22 & 54.52 & 69.13 & 62.63 & 61.78 \\
Llama 3.1 70b & 74.51 & 86.22 & 94.69 & 81.92 & 70.94 & 79.29 & 77.91 & 75.45 & 64.95 & 72.28 & 93.37 & 72.52 & 65.69 & 82.74 & 89.71 & 76.52 \\
Qwen 2.5 7b & 53.03 & 78.40 & 90.37 & 69.73 & 58.47 & 66.87 & 82.72 & 65.96 & 52.07 & 71.12 & 84.28 & 65.38 & 54.98 & 73.87 & 91.02 & 68.43 \\
Qwen 32b & 65.58 & 86.34 & 96.95 & 79.37 & 68.61 & 76.96 & 88.71 & 75.57 & 61.89 & 69.24 & 95.25 & 70.26 & 62.18 & 85.60 & 97.32 & 77.40 \\
Qwen 2.5 72b & 69.87 & 85.12 & 96.38 & 80.56 & 69.51 & 79.92 & 90.84 & 77.28 & 69.45 & 71.34 & 94.19 & 74.41 & 64.73 & 85.52 & 98.12 & 78.53 \\
GPT4o-mini & 58.25 & 80.81 & 86.72 & 72.33 & 65.84 & 72.4 & 71.88 & 69.45 & 61.18 & 72.48 & 85.94 & 69.93 & 57.04 & 81.63 & 91.15 & 72.51 \\
GPT-4o & 65.23 & 86.67 & 99.76 & 79.82 & 70.12 & 76.67 & 91.84 & 76.45 & 66.33 & 71.74 & 93.14 & 72.96 & 67.28 & 88.19 & 89.85 & 79.41 \\
\bottomrule
\end{tabular}
\caption{Detailed results for error identification for all dataset and models}
\label{tab:error_identification}
\end{table*}
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
Llama 3.1 8b & 67.80 & 89.91 & 77.31 & 65.03 & 91.31 & 75.96 & 64.08 & 86.76 & 73.71 \\
Llama 3.1 70b & 83.49 & 97.09 & 89.78 & 80.36 & 98.87 & 88.66 & 79.92 & 96.68 & 87.51 \\
\midrule
Qwen 7b & 69.94 & 62.85 & 66.21 & 66.14 & 67.60 & 66.86 & 66.18 & 60.15 & 63.02 \\
Qwen 32b & 85.98 & 93.49 & 89.58 & 83.78 & 96.95 & 89.88 & 82.45 & 95.61 & 88.54 \\
\midrule
GPT4o-mini & 74.93 & 84.81 & 79.56 & 70.85 & 87.01 & 78.10 & 71.84 & 86.73 & 78.59 \\
GPT-4o & 89.38 & 93.66 & 91.47 & 84.97 & 94.95 & 89.69 & 83.43 & 94.65 & 88.72 \\
\bottomrule
\end{tabular}
\caption{Precision, Recall and F1 score for premise identification with GSM8K}
\label{tab:gsm8k_premise_identification_per_candidate_score}
\end{table*}
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
Llama 3.1 8b & 53.98 & 82.36 & 65.22 & 46.39 & 82.43 & 59.37 & 52.63 & 82.42 & 64.24 \\
Llama 3.1 70b & 71.13 & 95.82 & 81.65 & 67.82 & 97.16 & 79.88 & 70.46 & 97.47 & 81.80 \\
\midrule
Qwen 7b & 59.40 & 60/30 & 59.85 & 48.82 & 56.02 & 52.17 & 59.40 & 62.87 & 61.09 \\
Qwen 32b & 74.57 & 87.30 & 80.43 & 69.98 & 87.31 & 77.69 & 73.13 & 91.04 & 81.11 \\
\midrule
GPT4o-mini & 57.58 & 66.57 & 61.75 & 54.42 & 69.01 & 60.85 & 61.18 & 72.29 & 66.27 \\
GPT-4o & 69.76 & 90.19 & 78.67 & 66.95 & 87.16 & 75.73 & 71.48 & 91.99 & 80.45 \\
\bottomrule
\end{tabular}
\caption{Precision, Recall and F1 score for premise identification in MATH}
\label{tab:math_premise_identification_per_candidate_score}
\end{table*}
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
Llama 3.1 8b & 58.94 & 84.52 & 69.45 & 44.14 & 76.76 & 56.05 & 58.67 & 82.99 & 68.74 \\
Llama 3.1 70b & 77.82 & 97.37 & 86.50 & 65.18 & 96.13 & 77.69 & 76.29 & 95.88 & 84.97 \\
\midrule
Qwen 7b & 58.05 & 59.09 & 58.57 & 48.72 & 66.48 & 56.23 & 59.86 & 59.41 & 59.63 \\
Qwen 32b & 78.73 & 91.60 & 84.68 & 69.89 & 88.99 & 78.29 & 78.09 & 92.07 & 84.50 \\
\midrule
GPT4o-mini & 65.06 & 76.92 & 70.50 & 51.74 & 73.81 & 60.84 & 65.52 & 79.37 & 71.78 \\
GPT-4o & 79.91 & 92.51 & 84.68 & 67.13 & 88.11 & 76.26 & 79.42 & 90.49 & 84.62 \\
\bottomrule
\end{tabular}
\caption{Precision, Recall and F1 score for premise identification with MetaMathQA.}
\label{tab:meta_math_premise_identification_per_candidate_score}
\end{table*}
\begin{table*}[!t]
\centering
\setlength\tabcolsep{4pt}
\fontsize{8pt}{9pt}\selectfont
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}[2]{*}{\textbf{Model Name}} & \multicolumn{3}{c|}{\textbf{Positives}} & \multicolumn{3}{c|}{\textbf{Negatives}} & \multicolumn{3}{c}{\textbf{Synthetic Negatives }} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& Precision & Recall & F1 & Precision & Recall & F1 & Precision & Recall & F1 \\
\midrule
Llama 3.1 8b & 56.02 & 80.12 & 65.94 & 52.60 & 78.92 & 63.13 & 55.31 & 80.26 & 65.49 \\
Llama 3.1 70b & 72.82 & 96.83 & 83.13 & 68.15 & 96.42 & 79.86 & 72.17 & 96.93 & 82.74 \\
\midrule
Qwen 7b & 62.33 & 61.81 & 62.07 & 68.15 & 96.42 & 79.86 & 58.75 & 59.69 & 59.22 \\
Qwen 32b & 77.73 & 91.18 & 83.92 & 71.55 & 89.55 & 79.54 & 73.55 & 91.16 & 81.41 \\
\midrule
GPT4o-mini & 63.24 & 75.36 & 68.77 & 57.36 & 73.38 & 64.39 & 59.75 & 73.13 & 65.77 \\
GPT-4o & 73.90 & 92.63 & 83.92 & 66.97 & 92.87 & 77.92 & 72.50 & 91.02 & 80.53 \\
\bottomrule
\end{tabular}
\caption{Precision, Recall and F1 score for premise identification in OrcaMath}
\label{tab:orca_math_premise_identification_per_candidate_score}
\end{table*}
\subsection{Prompt for Error Annotation with O1}
You are an expert mathematical reasoning analyzer. Your task is to analyze mathematical solutions and generate detailed error annotations in a specific JSON format. For each solution provided, you must carefully examine the reasoning chain and individual steps to identify any errors or issues.
Response Format
Your response must be a valid JSON object following exactly this structure:
\begin{verbatim}
{
"error_annotations": {
"chain_level": {
"has_errors": boolean,
"errors": [
{
"error_type": string,
"error_description": string
}
]
},
"step_level": [
{
"step_number": ,
"has_error": boolean,
"errors": [
{
"error_type": ,
"error_description":
}
]
}
]
}
}
\end{verbatim}
Chain-Level Error Types
1. ``Missing\_Steps" \newline
\textbf{Definition:} Solution lacks crucial concluding steps or final answer derivation \newline
\textbf{Examples:} \newline
- Not showing the final calculated value \newline
- Missing the ultimate conclusion \newline
- Failing to complete the proof \newline
\textbf{Impact:} Makes the solution incomplete or inconclusive
2. ``Planning\_Error" \newline
\textbf{Definition:} The reasoning takes an invalid or fundamentally incorrect approach \newline
\textbf{Examples:} \newline
- Using inapplicable theorems or methods \newline
- Solving for incorrect variables \newline
- Taking an approach that cannot possibly lead to a solution \newline
\textbf{Impact:} Makes the entire solution path invalid \newline
- Note: Valid but longer approaches (e.g., integration by parts instead of a substitution trick) should NOT be marked as errors
Step-Level Error Types
1. ``Logical\_Inconsistency" \newline
\textbf{Definition:} Steps that violate logical principles or make unjustified conclusions \newline
\textbf{Examples:} \newline
- False equivalences \newline
- Invalid deductions \newline
- Unsupported assumptions \newline
- Note that incorrect use of previous information (example the step uses a wrong value of a variable) is a Logical\_Inconsistency \newline
\textbf{Impact:} Breaks the logical flow of the solution
2. ``Mathematical\_Error" \newline
\textbf{Definition:} Incorrect calculations, misuse of formulas, or mathematical operations \newline
\textbf{Examples:} \newline
- Arithmetic mistakes \newline
- Incorrect algebraic manipulations \newline
- Wrong formula application \newline
- Note that Mathematical\_Error can only appear when there is an error in calculation \newline
\textbf{Impact:} Produces incorrect numerical or algebraic results
3. ``Accumulation\_Error" \newline
\textbf{Definition:} Errors that propagate from previous incorrect steps \newline
\textbf{Examples:} \newline
- Using wrong intermediate results \newline
- Building upon previously miscalculated values \newline
\textbf{Impact:} Compounds previous mistakes into larger errors \newline
4. ``Other" \newline
\textbf{Definition:} Any error that doesn't fit into the above categories
Examples: \newline
- Notation mistakes \newline
- Unclear explanations \newline
- Formatting issues \newline
\textbf{Impact:} Varies depending on the specific error
Analysis Requirements \newline
1. Examine each step against mathematical principles and theorems \newline
2. Verify all calculations and mathematical operations \newline
3. Check for proper use of definitions and formulas \newline
4. Ensure logical flow between steps \newline
5. Compare against the provided ground truth answer \newline
6. Consider the completeness of the solution
Important Notes \newline
- Provide ONLY the JSON output, no additional text or explanations \newline
- Every step in the solution must have a corresponding entry in step\_level array \newline
- Keep error descriptions clear, specific, and mathematically precise \newline
- Use empty arrays for errors when no errors exist \newline
- Ensure your response is always valid JSON that matches the exact format specified \newline
- Each error must have both an error\_type and a corresponding detailed error\_description \newline
- Error descriptions should be specific to the mathematical context of the problem \newline
- Do NOT penalize valid but verbose approaches (e.g., breaking down algebra into multiple steps) \newline
- Do NOT mark alternative solution methods as errors unless they are genuinely invalid \newline
- Focus on correctness rather than elegance or brevity \newline
Workflow \newline
1. Read and understand the problem statement \newline
2. Analyze the reasoning chain step-by-step \newline
3. Check for chain-level errors \newline
4. Analyze each step for specific errors \newline
5. Verify all premises and justifications \newline
6. Ensure completeness of the solution \newline
\label{sec:appendix_prompt}
\subsection{Prompt for Premise Annotation with O1}
The system prompt and the instruction for the O1 model for identifying premises for a given step is shared in Table \ref{tab:premise_prompt}
\subsection{Prompt for Error identification}
The prompts used for the baseline approach are shared in Appendix \ref{tab:baseline_error_id_instruction} and \ref{tab:baseline_error_id_sys}
\label{sec:baseline_error_prompts}. The evaluation for our error identification with premises are done with the prompts outlined in Tables \ref{tab:ours_math_error} and \ref{tab:ours_logical_error},
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth, colback=gray!10, colframe=black, sharp corners]
{\bf Instruction:} \newline
Given this math word problem and its solution steps, identify the key premises and their relationships. \newline
{\bf Problem:} \{problem['question']\} \newline
{\bf Solution Steps:} \newline
\{chr(10).join(problem['steps'])\} \newline
{\bf Return your analysis in this exact JSON format:} \newline
\{json\_template\} \newline
{\bf Critical Rules for Premises:} \newline
1. A step can NEVER use itself as a premise. For example, Step 3 cannot use any premise labeled as [3, "..."]. \newline
2. Premises can only come from: \newline
- Step 0 (problem statement and fundamental mathematical principles) \newline
- Previous steps (steps with lower index) \newline
3. Any intermediate calculations or logical steps within a single step should be part of that step's reasoning, not treated as separate premises. \newline
4. Mathematical principles (like properties of operations) should be treated as part of Step 0. \newline
\newline
{\bf Additional Requirements:} \newline
1. Start with Step 0 containing the problem statement. \newline
2. For each step after 0, copy the EXACT text from the student's solution into `original\_step`. \newline
3. Show clear reasoning for how premises lead to conclusions. \newline
4. Return ONLY valid JSON with no additional text. \newline
5. Do not use any special characters like \&, <, >, etc. \newline
6. Do not add any additional text for formatting it (e.g., "json"), just output the raw JSON. \newline
7. Maintain the exact same number of steps as in the original solution. \newline
\newline
{\bf Remember:} \newline
- Each step's premises must strictly come from either the problem statement (Step 0) or previous steps. Never from the current step. \newline
- Keep each step atomic—do not split steps into multiple substeps even if they contain multiple calculations. \newline
- The number of steps in your output (excluding Step 0) must match exactly with the number of steps in the student's solution. \newline
\newline
{\bf System Prompt:} \newline
You are an expert in mathematical reasoning. Your task is to analyze solution steps and output a JSON object containing: \newline
1. The premises used in each step. \newline
2. The conclusion reached. \newline
3. The reasoning that connects premises to conclusions. \newline
\newline
Output MUST be valid JSON with no additional text or explanation.
\end{tcolorbox}
\caption{\label{tab:premise_prompt} System prompt and instruction for the O1 model to identify premises for a given step}
\end{table*}
\clearpage
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth]
{\bf Instruction:}\\
Question: \{question\} \newline
Solution so far: \{solution\} \newline \newline
1. ``Logical\_Inconsistency" \newline
- \textbf{Definition}: Steps that violate logical principles or make unjustified conclusions \newline
- \textbf{Examples}: \newline
- False equivalences \newline
- Invalid deductions \newline
- Unsupported assumptions \newline
- Note that incorrect use of previous information (example the step uses a wrong value of a variable) is a Logical\_Inconsistency \newline
- \textbf{Impact}: Breaks the logical flow of the solution \newline
2. ``Mathematical\_Error" \newline
- \textbf{Definition}: Incorrect calculations, misuse of formulas, or mathematical operations \newline
- \textbf{Examples}: \newline
- Arithmetic mistakes \newline
- Incorrect algebraic manipulations \newline
- Wrong formula application \newline
- Note that Mathematical\_Error can only appear when there is an error in calculation \newline
- \textbf{Impact}: Produces incorrect numerical or algebraic results \newline
3. ``Accumulation\_Error" \newline
- \textbf{Definition}: Errors that propagate from previous incorrect steps \newline
- \textbf{Examples}: \newline
- Using wrong intermediate results \newline
- Building upon previously miscalculated values \newline
- \textbf{Impact}: Compounds previous mistakes into larger errors \newline
4. ``Other" \newline
- \textbf{Definition}: Any error that doesn't fit into the above categories \newline
- \textbf{Examples}: \newline
- Notation mistakes \newline
- Unclear explanations \newline
- Formatting issues \newline
- \textbf{Impact}: Varies depending on the specific error \newline
Statement to analyze:
{step}
Format your response as:
Reasoning: [detailed analysis of the statement's validity]
Verdict: [CORRECT, Mathematical\_Error, Logical\_Inconsistency, or Accumulation\_Error]
\end{tcolorbox}
\caption{\label{tab:baseline_error_id_instruction} Baseline error identification Instruction }
\end{table*}
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth]
{\bf System Prompt:}\\
You are a helpful AI assistant that analyzes mathematical solution steps.
Your task is to determine if each statement is COMPLETELY correct by carefully analyzing its validity.
Focus ONLY on whether the current step is valid - do not consider whether it helps reach the final answer or whether better steps could have been taken.
Mark a statement as CORRECT unless you find a specific error.
\end{tcolorbox}
\caption{\label{tab:baseline_error_id_sys} Baseline error identification System prompt }
\end{table*}
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth]
{\bf System Prompt:}\\
Your task is to determine whether a given sentence contains any mathematical errors.
For mathematical error, check if the sentence contains mathematical calculations (arithmetic or algebraic), and whether they are incorrect. If there are such errors, mark the sentence as "mathematical\_error"
- Mathematical errors can only come from incorrect results of mathematical operations
If no such errors exist, mark it as "correct".
Note: mathematical error can only come from incorrect numerical or algebraic calculations (i.e. wrong multiplication, wrong addition etc.)
if there are no numerical or algebraic calculations done, you can mark it as correct \newline
{\bf Instruction :}\\
Statement to analyze:\newline
{step}\newline
Format your response as:\newline
Reasoning: [detailed analysis of the statement's validity]\newline
Verdict: [correct or mathematical\_error]
\end{tcolorbox}
\caption{\label{tab:ours_math_error} Math error identification instruction }
\end{table*}
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth]
{\bf System Prompt:}\\
You are provided with a math question, a statement which is a step in the solution to the question and the premises to this steps (the question is also a premise). Your task is to identify whether the step follow naturally from the premises or not.
If the current step contradicts the premises, mark is as a logical\_inconsistency
If the step can be directly inferred from the premises, mark it as correct.
You should not check whether the premises are correct, assume they are correct. Only check the sentence given.\newline
{\bf Instruction :}\\
Given Premises: \newline
Question: \{question\} \newline
Previous steps as premise: \{premises\} \newline
Statement to analyze: \{step\} \newline
Guidelines:\newline
1. for logical\_inconsistency check if the step was performed under misinterpretation of the premises, made invalid deductions or had unsupported assumptions \newline
2. Don't check for correctness of the premises, your only task is to check correctness of the given sentence
Format your response as:
Reasoning: [detailed analysis of the statement's validity]
Verdict: [correct, logical\_inconsistency]
\end{tcolorbox}
\caption{\label{tab:ours_logical_error} Logical error identification instruction }
\end{table*}
\begin{table*}[h!]
\centering
\begin{tcolorbox}[width=0.9\linewidth]
{\bf Instruction:}\\
You are provided with a question, a partial solution, and the next step in the solution. \newline
Your task is to identify the steps that serve as premises for the given next step.\newline
A step qualifies as a premise if the next step directly relies on information from that step. Based on the identified premises, the correctness of the next step should be fully verifiable.\newline
Question (Step 0):\newline
\{question\}\newline
Solution so far:\newline
\{solution\}\newline
Next step to analyze:\newline
\{step\}\newline
For the step above, identify which previous steps (including Step 0 - the question) are premises and explain why each one is necessary. Remember:\newline
1. A step cannot be a premise to itself\newline
2. The question (Step 0) can be a premise if used directly\newline
Generate ONLY the premises and nothing else.\newline
Format your response with one premise per line as:\newline
Step X: [explanation of why this step is necessary for the current step]\newline
\end{tcolorbox}
\caption{\label{tab:premise_ideintification_prompt} Prompt for evaluating models in the premise identification task (zero shot) }
\end{table*}
\appendix
\begin{thebibliography}{45}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ahn et~al.(2024)Ahn, Verma, Lou, Liu, Zhang, and Yin]{ahn2024largelanguagemodelsmathematical}
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W.
\newblock Large language models for mathematical reasoning: Progresses and challenges, 2024.
\newblock URL \url{https://arxiv.org/abs/2402.00157}.
\bibitem[Arora et~al.(2023)Arora, Singh, and {Mausam}]{arora-etal-2023-llms}
Arora, D., Singh, H., and {Mausam}.
\newblock Have {LLM}s advanced enough? a challenging problem solving benchmark for large language models.
\newblock In Bouamor, H., Pino, J., and Bali, K. (eds.), \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 7527--7543, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.emnlp-main.468}.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.468}.
\bibitem[Besta et~al.(2024)Besta, Blach, Kubicek, Gerstenberger, Podstawski, Gianinazzi, Gajda, Lehmann, Niewiadomski, Nyczyk, and Hoefler]{Besta_2024}
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., and Hoefler, T.
\newblock Graph of thoughts: Solving elaborate problems with large language models.
\newblock \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, 38\penalty0 (16):\penalty0 17682–17690, March 2024.
\newblock ISSN 2159-5399.
\newblock \doi{10.1609/aaai.v38i16.29720}.
\newblock URL \url{http://dx.doi.org/10.1609/aaai.v38i16.29720}.
\bibitem[Celikyilmaz et~al.(2021)Celikyilmaz, Clark, and Gao]{celikyilmaz2021evaluationtextgenerationsurvey}
Celikyilmaz, A., Clark, E., and Gao, J.
\newblock Evaluation of text generation: A survey, 2021.
\newblock URL \url{https://arxiv.org/abs/2006.14799}.
\bibitem[Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser, Plappert, Tworek, Hilton, Nakano, Hesse, and Schulman]{cobbe2021trainingverifierssolvemath}
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.
\newblock Training verifiers to solve math word problems, 2021.
\newblock URL \url{https://arxiv.org/abs/2110.14168}.
\bibitem[Daheim et~al.(2024)Daheim, Macina, Kapur, Gurevych, and Sachan]{daheim2024stepwiseverificationremediationstudent}
Daheim, N., Macina, J., Kapur, M., Gurevych, I., and Sachan, M.
\newblock Stepwise verification and remediation of student reasoning errors with large language model tutors, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.09136}.
\bibitem[Gao et~al.(2023)Gao, Thattai, Shakiah, Gao, Pansare, Sharma, Sukhatme, Shi, Yang, Zhang, Hu, Arumugam, Hu, Wen, Guthy, Chung, Khanna, Ipek, Ball, Bland, Rocker, Johnston, Ghanadan, Hakkani-Tur, and Natarajan]{NEURIPS2023_3d0758f0}
Gao, Q., Thattai, G., Shakiah, S., Gao, X., Pansare, S., Sharma, V., Sukhatme, G., Shi, H., Yang, B., Zhang, D., Hu, L., Arumugam, K., Hu, S., Wen, M., Guthy, D., Chung, S., Khanna, R., Ipek, O., Ball, L., Bland, K., Rocker, H., Johnston, M., Ghanadan, R., Hakkani-Tur, D., and Natarajan, P.
\newblock Alexa arena: A user-centric interactive platform for embodied ai.
\newblock In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), \emph{Advances in Neural Information Processing Systems}, volume~36, pp.\ 19170--19194. Curran Associates, Inc., 2023.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2023/file/3d0758f0b95e19abc68c1c8070d36510-Paper-Datasets_and_Benchmarks.pdf}.
\bibitem[Golovneva et~al.(2023)Golovneva, Chen, Poff, Corredor, Zettlemoyer, Fazel-Zarandi, and Celikyilmaz]{golovneva2023roscoesuitemetricsscoring}
Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettlemoyer, L., Fazel-Zarandi, M., and Celikyilmaz, A.
\newblock Roscoe: A suite of metrics for scoring step-by-step reasoning, 2023.
\newblock URL \url{https://arxiv.org/abs/2212.07919}.
\bibitem[Grattafiori et~al.(2024)Grattafiori, Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Vaughan, Yang, Fan, Goyal, Hartshorn, Yang, Mitra, Sravankumar, Korenev, Hinsvark, Rao, Zhang, Rodriguez, Gregerson, Spataru, Roziere, Biron, Tang, Chern, Caucheteux, Nayak, Bi, Marra, McConnell, Keller, Touret, Wu, Wong, Ferrer, Nikolaidis, Allonsius, Song, Pintz, Livshits, Wyatt, Esiobu, Choudhary, Mahajan, Garcia-Olano, Perino, Hupkes, Lakomkin, AlBadawy, Lobanova, Dinan, Smith, Radenovic, Guzmán, Zhang, Synnaeve, Lee, Anderson, Thattai, Nail, Mialon, Pang, Cucurell, Nguyen, Korevaar, Xu, Touvron, Zarov, Ibarra, Kloumann, Misra, Evtimov, Zhang, Copet, Lee, Geffert, Vranes, Park, Mahadeokar, Shah, van~der Linde, Billock, Hong, Lee, Fu, Chi, Huang, Liu, Wang, Yu, Bitton, Spisak, Park, Rocca, Johnstun, Saxe, Jia, Alwala, Prasad, Upasani, Plawiak, Li, Heafield, Stone, El-Arini, Iyer, Malik, Chiu, Bhalla, Lakhotia, Rantala-Yeary, van~der Maaten, Chen, Tan, Jenkins, Martin, Madaan, Malo, Blecher,
Landzaat, de~Oliveira, Muzzi, Pasupuleti, Singh, Paluri, Kardas, Tsimpoukelli, Oldham, Rita, Pavlova, Kambadur, Lewis, Si, Singh, Hassan, Goyal, Torabi, Bashlykov, Bogoychev, Chatterji, Zhang, Duchenne, Çelebi, Alrassy, Zhang, Li, Vasic, Weng, Bhargava, Dubal, Krishnan, Koura, Xu, He, Dong, Srinivasan, Ganapathy, Calderer, Cabral, Stojnic, Raileanu, Maheswari, Girdhar, Patel, Sauvestre, Polidoro, Sumbaly, Taylor, Silva, Hou, Wang, Hosseini, Chennabasappa, Singh, Bell, Kim, Edunov, Nie, Narang, Raparthy, Shen, Wan, Bhosale, Zhang, Vandenhende, Batra, Whitman, Sootla, Collot, Gururangan, Borodinsky, Herman, Fowler, Sheasha, Georgiou, Scialom, Speckbacher, Mihaylov, Xiao, Karn, Goswami, Gupta, Ramanathan, Kerkez, Gonguet, Do, Vogeti, Albiero, Petrovic, Chu, Xiong, Fu, Meers, Martinet, Wang, Wang, Tan, Xia, Xie, Jia, Wang, Goldschlag, Gaur, Babaei, Wen, Song, Zhang, Li, Mao, Coudert, Yan, Chen, Papakipos, Singh, Srivastava, Jain, Kelsey, Shajnfeld, Gangidi, Victoria, Goldstand, Menon, Sharma, Boesenberg,
Baevski, Feinstein, Kallet, Sangani, Teo, Yunus, Lupu, Alvarado, Caples, Gu, Ho, Poulton, Ryan, Ramchandani, Dong, Franco, Goyal, Saraf, Chowdhury, Gabriel, Bharambe, Eisenman, Yazdan, James, Maurer, Leonhardi, Huang, Loyd, Paola, Paranjape, Liu, Wu, Ni, Hancock, Wasti, Spence, Stojkovic, Gamido, Montalvo, Parker, Burton, Mejia, Liu, Wang, Kim, Zhou, Hu, Chu, Cai, Tindal, Feichtenhofer, Gao, Civin, Beaty, Kreymer, Li, Adkins, Xu, Testuggine, David, Parikh, Liskovich, Foss, Wang, Le, Holland, Dowling, Jamil, Montgomery, Presani, Hahn, Wood, Le, Brinkman, Arcaute, Dunbar, Smothers, Sun, Kreuk, Tian, Kokkinos, Ozgenel, Caggioni, Kanayet, Seide, Florez, Schwarz, Badeer, Swee, Halpern, Herman, Sizov, Guangyi, Zhang, Lakshminarayanan, Inan, Shojanazeri, Zou, Wang, Zha, Habeeb, Rudolph, Suk, Aspegren, Goldman, Zhan, Damlaj, Molybog, Tufanov, Leontiadis, Veliche, Gat, Weissman, Geboski, Kohli, Lam, Asher, Gaya, Marcus, Tang, Chan, Zhen, Reizenstein, Teboul, Zhong, Jin, Yang, Cummings, Carvill, Shepard, McPhie,
Torres, Ginsburg, Wang, Wu, U, Saxena, Khandelwal, Zand, Matosich, Veeraraghavan, Michelena, Li, Jagadeesh, Huang, Chawla, Huang, Chen, Garg, A, Silva, Bell, Zhang, Guo, Yu, Moshkovich, Wehrstedt, Khabsa, Avalani, Bhatt, Mankus, Hasson, Lennie, Reso, Groshev, Naumov, Lathi, Keneally, Liu, Seltzer, Valko, Restrepo, Patel, Vyatskov, Samvelyan, Clark, Macey, Wang, Hermoso, Metanat, Rastegari, Bansal, Santhanam, Parks, White, Bawa, Singhal, Egebo, Usunier, Mehta, Laptev, Dong, Cheng, Chernoguz, Hart, Salpekar, Kalinli, Kent, Parekh, Saab, Balaji, Rittner, Bontrager, Roux, Dollar, Zvyagina, Ratanchandani, Yuvraj, Liang, Alao, Rodriguez, Ayub, Murthy, Nayani, Mitra, Parthasarathy, Li, Hogan, Battey, Wang, Howes, Rinott, Mehta, Siby, Bondu, Datta, Chugh, Hunt, Dhillon, Sidorov, Pan, Mahajan, Verma, Yamamoto, Ramaswamy, Lindsay, Lindsay, Feng, Lin, Zha, Patil, Shankar, Zhang, Zhang, Wang, Agarwal, Sajuyigbe, Chintala, Max, Chen, Kehoe, Satterfield, Govindaprasad, Gupta, Deng, Cho, Virk, Subramanian, Choudhury,
Goldman, Remez, Glaser, Best, Koehler, Robinson, Li, Zhang, Matthews, Chou, Shaked, Vontimitta, Ajayi, Montanez, Mohan, Kumar, Mangla, Ionescu, Poenaru, Mihailescu, Ivanov, Li, Wang, Jiang, Bouaziz, Constable, Tang, Wu, Wang, Wu, Gao, Kleinman, Chen, Hu, Jia, Qi, Li, Zhang, Zhang, Adi, Nam, Yu, Wang, Zhao, Hao, Qian, Li, He, Rait, DeVito, Rosnbrick, Wen, Yang, Zhao, and Ma]{grattafiori2024llama3herdmodels}
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C.~C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E.~M., Radenovic, F., Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G.~L., Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I.~A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J.,
Shah, J., van~der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K.~V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala-Yeary, L., van~der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de~Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M.~K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P.~S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R.~S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar,
R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S.~S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X.~E., Xia, X., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z.~D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A.,
Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B.~D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Caggioni, F., Kanayet, F., Seide, F., Florez, G.~M., Schwarz, G., Badeer, G., Swee, G., Halpern, G.,
Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj, I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K.~H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M.~L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark,
M., Macey, M., Wang, M., Hermoso, M.~J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N.~P., Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S.~J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S.~C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S.,
Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V.~S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V.~T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang, X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z.
\newblock The llama 3 herd of models, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.21783}.
\bibitem[Han et~al.(2024)Han, Schoelkopf, Zhao, Qi, Riddell, Zhou, Coady, Peng, Qiao, Benson, Sun, Wardle-Solano, Szabo, Zubova, Burtell, Fan, Liu, Wong, Sailor, Ni, Nan, Kasai, Yu, Zhang, Fabbri, Kryscinski, Yavuz, Liu, Lin, Joty, Zhou, Xiong, Ying, Cohan, and Radev]{han2024folionaturallanguagereasoning}
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng, D., Qiao, Y., Benson, L., Sun, L., Wardle-Solano, A., Szabo, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L., Kasai, J., Yu, T., Zhang, R., Fabbri, A.~R., Kryscinski, W., Yavuz, S., Liu, Y., Lin, X.~V., Joty, S., Zhou, Y., Xiong, C., Ying, R., Cohan, A., and Radev, D.
\newblock Folio: Natural language reasoning with first-order logic, 2024.
\newblock URL \url{https://arxiv.org/abs/2209.00840}.
\bibitem[He et~al.(2024)He, Zhang, and Roth]{he2024socrevallargelanguagemodels}
He, H., Zhang, H., and Roth, D.
\newblock Socreval: Large language models with the socratic method for reference-free reasoning evaluation, 2024.
\newblock URL \url{https://arxiv.org/abs/2310.00074}.
\bibitem[Hendrycks et~al.(2021{\natexlab{a}})Hendrycks, Burns, Basart, Zou, Mazeika, Song, and Steinhardt]{hendrycks2021measuringmassivemultitasklanguage}
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.
\newblock Measuring massive multitask language understanding, 2021{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2009.03300}.
\bibitem[Hendrycks et~al.(2021{\natexlab{b}})Hendrycks, Burns, Kadavath, Arora, Basart, Tang, Song, and Steinhardt]{hendrycksmath2021}
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.
\newblock Measuring mathematical problem solving with the math dataset.
\newblock \emph{NeurIPS}, 2021{\natexlab{b}}.
\bibitem[Huang \& Chang(2023)Huang and Chang]{huang2023reasoninglargelanguagemodels}
Huang, J. and Chang, K. C.-C.
\newblock Towards reasoning in large language models: A survey, 2023.
\newblock URL \url{https://arxiv.org/abs/2212.10403}.
\bibitem[Imani et~al.(2023)Imani, Du, and Shrivastava]{imani-etal-2023-mathprompter}
Imani, S., Du, L., and Shrivastava, H.
\newblock {M}ath{P}rompter: Mathematical reasoning using large language models.
\newblock In Sitaram, S., Beigman~Klebanov, B., and Williams, J.~D. (eds.), \emph{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)}, pp.\ 37--42, Toronto, Canada, July 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.acl-industry.4}.
\newblock URL \url{https://aclanthology.org/2023.acl-industry.4/}.
\bibitem[Jiang et~al.(2024)Jiang, Wang, Shen, Kim, and Kim]{jiang2024surveylargelanguagemodels}
Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S.
\newblock A survey on large language models for code generation, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.00515}.
\bibitem[Kwon et~al.(2023)Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica]{kwon2023efficientmemorymanagementlarge}
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.~H., Gonzalez, J.~E., Zhang, H., and Stoica, I.
\newblock Efficient memory management for large language model serving with pagedattention, 2023.
\newblock URL \url{https://arxiv.org/abs/2309.06180}.
\bibitem[Lightman et~al.(2023)Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, and Cobbe]{lightman2023letsverifystepstep}
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K.
\newblock Let's verify step by step, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.20050}.
\bibitem[Ling et~al.(2023)Ling, Fang, Li, Huang, Lee, Memisevic, and Su]{ling2023deductiveverificationchainofthoughtreasoning}
Ling, Z., Fang, Y., Li, X., Huang, Z., Lee, M., Memisevic, R., and Su, H.
\newblock Deductive verification of chain-of-thought reasoning, 2023.
\newblock URL \url{https://arxiv.org/abs/2306.03872}.
\bibitem[Mitra et~al.(2024)Mitra, Khanpour, Rosset, and Awadallah]{mitra2024orcamathunlockingpotentialslms}
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A.
\newblock Orca-math: Unlocking the potential of slms in grade school math, 2024.
\newblock URL \url{https://arxiv.org/abs/2402.14830}.
\bibitem[Moura \& Ullrich(2021)Moura and Ullrich]{10.1007/978-3-030-79876-5_37}
Moura, L.~d. and Ullrich, S.
\newblock The lean 4 theorem prover and programming language.
\newblock In Platzer, A. and Sutcliffe, G. (eds.), \emph{Automated Deduction -- CADE 28}, pp.\ 625--635, Cham, 2021. Springer International Publishing.
\newblock ISBN 978-3-030-79876-5.
\bibitem[Murphy et~al.(2024)Murphy, Yang, Sun, Li, Anandkumar, and Si]{murphy2024autoformalizingeuclideangeometry}
Murphy, L., Yang, K., Sun, J., Li, Z., Anandkumar, A., and Si, X.
\newblock Autoformalizing euclidean geometry, 2024.
\newblock URL \url{https://arxiv.org/abs/2405.17216}.
\bibitem[{OpenAI}(2024)]{openai2024o1}
{OpenAI}.
\newblock Openai o1 system card.
\newblock Technical report, December 2024.
\newblock URL \url{https://cdn.openai.com/o1-system-card-20241205.pdf}.
\bibitem[OpenAI et~al.(2024)OpenAI, :, Jaech, Kalai, Lerer, Richardson, El-Kishky, Low, Helyar, Madry, Beutel, Carney, Iftimie, Karpenko, Passos, Neitz, Prokofiev, Wei, Tam, Bennett, Kumar, Saraiva, Vallone, Duberstein, Kondrich, Mishchenko, Applebaum, Jiang, Nair, Zoph, Ghorbani, Rossen, Sokolowsky, Barak, McGrew, Minaiev, Hao, Baker, Houghton, McKinzie, Eastman, Lugaresi, Bassin, Hudson, Li, de~Bourcy, Voss, Shen, Zhang, Koch, Orsinger, Hesse, Fischer, Chan, Roberts, Kappler, Levy, Selsam, Dohan, Farhi, Mely, Robinson, Tsipras, Li, Oprica, Freeman, Zhang, Wong, Proehl, Cheung, Mitchell, Wallace, Ritter, Mays, Wang, Such, Raso, Leoni, Tsimpourlas, Song, von Lohmann, Sulit, Salmon, Parascandolo, Chabot, Zhao, Brockman, Leclerc, Salman, Bao, Sheng, Andrin, Bagherinezhad, Ren, Lightman, Chung, Kivlichan, O'Connell, Osband, Gilaberte, Akkaya, Kostrikov, Sutskever, Kofman, Pachocki, Lennon, Wei, Harb, Twore, Feng, Yu, Weng, Tang, Yu, Candela, Palermo, Parish, Heidecke, Hallman, Rizzo, Gordon, Uesato, Ward,
Huizinga, Wang, Chen, Xiao, Singhal, Nguyen, Cobbe, Shi, Wood, Rimbach, Gu-Lemberg, Liu, Lu, Stone, Yu, Ahmad, Yang, Liu, Maksin, Ho, Fedus, Weng, Li, McCallum, Held, Kuhn, Kondraciuk, Kaiser, Metz, Boyd, Trebacz, Joglekar, Chen, Tintor, Meyer, Jones, Kaufer, Schwarzer, Shah, Yatbaz, Guan, Xu, Yan, Glaese, Chen, Lampe, Malek, Wang, Fradin, McClay, Pavlov, Wang, Wang, Murati, Bavarian, Rohaninejad, McAleese, Chowdhury, Chowdhury, Ryder, Tezak, Brown, Nachum, Boiko, Murk, Watkins, Chao, Ashbourne, Izmailov, Zhokhov, Dias, Arora, Lin, Lopes, Gaon, Miyara, Leike, Hwang, Garg, Brown, James, Shu, Cheu, Greene, Jain, Altman, Toizer, Toyer, Miserendino, Agarwal, Hernandez, Baker, McKinney, Yan, Zhao, Hu, Santurkar, Chaudhuri, Zhang, Fu, Papay, Lin, Balaji, Sanjeev, Sidor, Broda, Clark, Wang, Gordon, Sanders, Patwardhan, Sottiaux, Degry, Dimson, Zheng, Garipov, Stasi, Bansal, Creech, Peterson, Eloundou, Qi, Kosaraju, Monaco, Pong, Fomenko, Zheng, Zhou, McCabe, Zaremba, Dubois, Lu, Chen, Cha, Bai, He, Zhang, Wang,
Shao, and Li]{openai2024openaio1card}
OpenAI, :, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., Iftimie, A., Karpenko, A., Passos, A.~T., Neitz, A., Prokofiev, A., Wei, A., Tam, A., Bennett, A., Kumar, A., Saraiva, A., Vallone, A., Duberstein, A., Kondrich, A., Mishchenko, A., Applebaum, A., Jiang, A., Nair, A., Zoph, B., Ghorbani, B., Rossen, B., Sokolowsky, B., Barak, B., McGrew, B., Minaiev, B., Hao, B., Baker, B., Houghton, B., McKinzie, B., Eastman, B., Lugaresi, C., Bassin, C., Hudson, C., Li, C.~M., de~Bourcy, C., Voss, C., Shen, C., Zhang, C., Koch, C., Orsinger, C., Hesse, C., Fischer, C., Chan, C., Roberts, D., Kappler, D., Levy, D., Selsam, D., Dohan, D., Farhi, D., Mely, D., Robinson, D., Tsipras, D., Li, D., Oprica, D., Freeman, E., Zhang, E., Wong, E., Proehl, E., Cheung, E., Mitchell, E., Wallace, E., Ritter, E., Mays, E., Wang, F., Such, F.~P., Raso, F., Leoni, F., Tsimpourlas, F., Song, F., von Lohmann, F., Sulit, F., Salmon, G., Parascandolo, G., Chabot,
G., Zhao, G., Brockman, G., Leclerc, G., Salman, H., Bao, H., Sheng, H., Andrin, H., Bagherinezhad, H., Ren, H., Lightman, H., Chung, H.~W., Kivlichan, I., O'Connell, I., Osband, I., Gilaberte, I.~C., Akkaya, I., Kostrikov, I., Sutskever, I., Kofman, I., Pachocki, J., Lennon, J., Wei, J., Harb, J., Twore, J., Feng, J., Yu, J., Weng, J., Tang, J., Yu, J., Candela, J.~Q., Palermo, J., Parish, J., Heidecke, J., Hallman, J., Rizzo, J., Gordon, J., Uesato, J., Ward, J., Huizinga, J., Wang, J., Chen, K., Xiao, K., Singhal, K., Nguyen, K., Cobbe, K., Shi, K., Wood, K., Rimbach, K., Gu-Lemberg, K., Liu, K., Lu, K., Stone, K., Yu, K., Ahmad, L., Yang, L., Liu, L., Maksin, L., Ho, L., Fedus, L., Weng, L., Li, L., McCallum, L., Held, L., Kuhn, L., Kondraciuk, L., Kaiser, L., Metz, L., Boyd, M., Trebacz, M., Joglekar, M., Chen, M., Tintor, M., Meyer, M., Jones, M., Kaufer, M., Schwarzer, M., Shah, M., Yatbaz, M., Guan, M.~Y., Xu, M., Yan, M., Glaese, M., Chen, M., Lampe, M., Malek, M., Wang, M., Fradin, M., McClay, M.,
Pavlov, M., Wang, M., Wang, M., Murati, M., Bavarian, M., Rohaninejad, M., McAleese, N., Chowdhury, N., Chowdhury, N., Ryder, N., Tezak, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Chao, P., Ashbourne, P., Izmailov, P., Zhokhov, P., Dias, R., Arora, R., Lin, R., Lopes, R.~G., Gaon, R., Miyara, R., Leike, R., Hwang, R., Garg, R., Brown, R., James, R., Shu, R., Cheu, R., Greene, R., Jain, S., Altman, S., Toizer, S., Toyer, S., Miserendino, S., Agarwal, S., Hernandez, S., Baker, S., McKinney, S., Yan, S., Zhao, S., Hu, S., Santurkar, S., Chaudhuri, S.~R., Zhang, S., Fu, S., Papay, S., Lin, S., Balaji, S., Sanjeev, S., Sidor, S., Broda, T., Clark, A., Wang, T., Gordon, T., Sanders, T., Patwardhan, T., Sottiaux, T., Degry, T., Dimson, T., Zheng, T., Garipov, T., Stasi, T., Bansal, T., Creech, T., Peterson, T., Eloundou, T., Qi, V., Kosaraju, V., Monaco, V., Pong, V., Fomenko, V., Zheng, W., Zhou, W., McCabe, W., Zaremba, W., Dubois, Y., Lu, Y., Chen, Y., Cha, Y., Bai, Y., He, Y., Zhang, Y.,
Wang, Y., Shao, Z., and Li, Z.
\newblock Openai o1 system card, 2024.
\newblock URL \url{https://arxiv.org/abs/2412.16720}.
\bibitem[Philipov et~al.(2024)Philipov, Dongre, Tur, and Hakkani-Tür]{philipov2024simulatinguseragentsembodied}
Philipov, D., Dongre, V., Tur, G., and Hakkani-Tür, D.
\newblock Simulating user agents for embodied conversational-ai, 2024.
\newblock URL \url{https://arxiv.org/abs/2410.23535}.
\bibitem[Prasad et~al.(2023)Prasad, Saha, Zhou, and Bansal]{prasad2023recevalevaluatingreasoningchains}
Prasad, A., Saha, S., Zhou, X., and Bansal, M.
\newblock Receval: Evaluating reasoning chains via correctness and informativeness, 2023.
\newblock URL \url{https://arxiv.org/abs/2304.10703}.
\bibitem[Qwen et~al.(2025)Qwen, :, Yang, Yang, Zhang, Hui, Zheng, Yu, Li, Liu, Huang, Wei, Lin, Yang, Tu, Zhang, Yang, Yang, Zhou, Lin, Dang, Lu, Bao, Yang, Yu, Li, Xue, Zhang, Zhu, Men, Lin, Li, Tang, Xia, Ren, Ren, Fan, Su, Zhang, Wan, Liu, Cui, Zhang, and Qiu]{qwen2025qwen25technicalreport}
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z.
\newblock Qwen2.5 technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2412.15115}.
\bibitem[Reiter(2019)]{reiter-2019-natural}
Reiter, E.
\newblock Natural language generation challenges for explainable {AI}.
\newblock In Alonso, J.~M. and Catala, A. (eds.), \emph{Proceedings of the 1st Workshop on Interactive Natural Language Technology for Explainable Artificial Intelligence (NL4XAI 2019)}, pp.\ 3--7. Association for Computational Linguistics, 2019.
\newblock \doi{10.18653/v1/W19-8402}.
\newblock URL \url{https://aclanthology.org/W19-8402}.
\bibitem[Shi et~al.(2023)Shi, Chen, Misra, Scales, Dohan, Chi, Sch\"{a}rli, and Zhou]{10.5555/3618408.3619699}
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Sch\"{a}rli, N., and Zhou, D.
\newblock Large language models can be easily distracted by irrelevant context.
\newblock In \emph{Proceedings of the 40th International Conference on Machine Learning}, ICML'23. JMLR.org, 2023.
\bibitem[Stechly et~al.(2024)Stechly, Valmeekam, and Kambhampati]{stechly2024selfverificationlimitationslargelanguage}
Stechly, K., Valmeekam, K., and Kambhampati, S.
\newblock On the self-verification limitations of large language models on reasoning and planning tasks, 2024.
\newblock URL \url{https://arxiv.org/abs/2402.08115}.
\bibitem[Tian et~al.(2021)Tian, Li, Chen, Xiao, He, and Jin]{tian-etal-2021-diagnosing}
Tian, J., Li, Y., Chen, W., Xiao, L., He, H., and Jin, Y.
\newblock Diagnosing the first-order logical reasoning ability through {L}ogic{NLI}.
\newblock In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), \emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pp.\ 3738--3747, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.emnlp-main.303}.
\newblock URL \url{https://aclanthology.org/2021.emnlp-main.303}.
\bibitem[Tyen et~al.(2024)Tyen, Mansoor, Carbune, Chen, and Mak]{tyen-etal-2024-llms}
Tyen, G., Mansoor, H., Carbune, V., Chen, P., and Mak, T.
\newblock {LLM}s cannot find reasoning errors, but can correct them given the error location.
\newblock In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), \emph{Findings of the Association for Computational Linguistics: ACL 2024}, pp.\ 13894--13908, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.findings-acl.826}.
\newblock URL \url{https://aclanthology.org/2024.findings-acl.826/}.
\bibitem[Uesato et~al.(2022)Uesato, Kushman, Kumar, Song, Siegel, Wang, Creswell, Irving, and Higgins]{uesato2022solvingmathwordproblems}
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I.
\newblock Solving math word problems with process- and outcome-based feedback, 2022.
\newblock URL \url{https://arxiv.org/abs/2211.14275}.
\bibitem[Wang et~al.(2024)Wang, Hu, Lu, Zhu, Zhang, Subramaniam, Loomba, Zhang, Sun, and Wang]{wang2024scibench}
Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A.~R., Zhang, S., Sun, Y., and Wang, W.
\newblock Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2024.
\newblock URL \url{https://openreview.net/forum?id=u6jbcaCHqO}.
\bibitem[Wei et~al.(2023)Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, and Zhou]{wei2023chainofthoughtpromptingelicitsreasoning}
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D.
\newblock Chain-of-thought prompting elicits reasoning in large language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2201.11903}.
\bibitem[Welleck et~al.(2021)Welleck, Liu, Bras, Hajishirzi, Choi, and Cho]{welleck2021naturalproofs}
Welleck, S., Liu, J., Bras, R.~L., Hajishirzi, H., Choi, Y., and Cho, K.
\newblock Naturalproofs: Mathematical theorem proving in natural language.
\newblock In \emph{Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)}, 2021.
\newblock URL \url{https://openreview.net/forum?id=Jvxa8adr3iY}.
\bibitem[Weng et~al.(2023)Weng, Zhu, Xia, Li, He, Liu, Sun, Liu, and Zhao]{weng-etal-2023-large}
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J.
\newblock Large language models are better reasoners with self-verification.
\newblock In Bouamor, H., Pino, J., and Bali, K. (eds.), \emph{Findings of the Association for Computational Linguistics: EMNLP 2023}, pp.\ 2550--2575, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.findings-emnlp.167}.
\newblock URL \url{https://aclanthology.org/2023.findings-emnlp.167/}.
\bibitem[Wu et~al.(2022)Wu, Jiang, Li, Rabe, Staats, Jamnik, and Szegedy]{wu2022autoformalizationlargelanguagemodels}
Wu, Y., Jiang, A.~Q., Li, W., Rabe, M.~N., Staats, C., Jamnik, M., and Szegedy, C.
\newblock Autoformalization with large language models, 2022.
\newblock URL \url{https://arxiv.org/abs/2205.12615}.
\bibitem[Wu et~al.(2024)Wu, Zeng, Zhang, Tan, Shen, and Jiang]{wu-etal-2024-large}
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., and Jiang, M.
\newblock Large language models can self-correct with key condition verification.
\newblock In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 12846--12867, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.714}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.714/}.
\bibitem[Yang et~al.(2023)Yang, Swope, Gu, Chalamala, Song, Yu, Godil, Prenger, and Anandkumar]{yang2023leandojotheoremprovingretrievalaugmented}
Yang, K., Swope, A.~M., Gu, A., Chalamala, R., Song, P., Yu, S., Godil, S., Prenger, R., and Anandkumar, A.
\newblock Leandojo: Theorem proving with retrieval-augmented language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2306.15626}.
\bibitem[Yang et~al.(2024)Yang, Poesia, He, Li, Lauter, Chaudhuri, and Song]{yang2024formalmathematicalreasoningnew}
Yang, K., Poesia, G., He, J., Li, W., Lauter, K., Chaudhuri, S., and Song, D.
\newblock Formal mathematical reasoning: A new frontier in ai, 2024.
\newblock URL \url{https://arxiv.org/abs/2412.16075}.
\bibitem[Yao et~al.(2023)Yao, Yu, Zhao, Shafran, Griffiths, Cao, and Narasimhan]{yao2023treethoughtsdeliberateproblem}
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.~L., Cao, Y., and Narasimhan, K.
\newblock Tree of thoughts: Deliberate problem solving with large language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.10601}.
\bibitem[Yu et~al.(2023)Yu, Jiang, Shi, Yu, Liu, Zhang, Kwok, Li, Weller, and Liu]{yu2023metamath}
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J.~T., Li, Z., Weller, A., and Liu, W.
\newblock Metamath: Bootstrap your own mathematical questions for large language models.
\newblock \emph{arXiv preprint arXiv:2309.12284}, 2023.
\bibitem[Zheng et~al.(2024)Zheng, Zhang, Zhang, Lin, Lu, Yu, Liu, Zhou, and Lin]{zheng2024processbenchidentifyingprocesserrors}
Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J.
\newblock Processbench: Identifying process errors in mathematical reasoning, 2024.
\newblock URL \url{https://arxiv.org/abs/2412.06559}.
\bibitem[Zhu et~al.(2024)Zhu, Zhang, Xie, and Su]{zhu2024deductivebeamsearchdecoding}
Zhu, T., Zhang, K., Xie, J., and Su, Y.
\newblock Deductive beam search: Decoding deducible rationale for chain-of-thought reasoning, 2024.
\newblock URL \url{https://arxiv.org/abs/2401.17686}.
\end{thebibliography}