Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
Extracted source text · arXiv 2501.14275
%
\documentclass{article}
%
\usepackage{microtype}
\usepackage{graphicx}
%
\usepackage{booktabs} %
%
%
%
%
\usepackage{hyperref}
\usepackage{url}
\usepackage{amssymb} %
\usepackage{pifont}
\usepackage{multirow}
\usepackage[most]{tcolorbox}
\usepackage{makecell}
\usepackage{caption} %
\usepackage{subcaption} %
\usepackage{array} %
\usepackage{adjustbox} %
\usepackage{xspace}
\usepackage{rotating}
\usepackage{pifont}%
\usepackage{float} %
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
%
\newcommand{\theHalgorithm}{\arabic{algorithm}}
%
%
%
\usepackage[accepted]{icml2025}
%
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
%
\usepackage[capitalize,noabbrev]{cleveref}
%
%
%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
%
%
%
\usepackage[textsize=tiny]{todonotes}
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\iid{\emph{i.i.d}\onedot} \def\IID{\emph{I.I.D}\onedot}
\def\eg{\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot}
\def\ie{\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot}
\def\cf{\emph{c.f}\onedot} \def\Cf{\emph{C.f}\onedot}
\def\etc{\emph{etc}\onedot} \def\vs{\emph{vs}\onedot}
\def\wrt{w.r.t\onedot} \def\dof{d.o.f\onedot}
\def\aka{\emph{a.k.a}\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}[1]{\textcolor{black}{#1}}
\def\ourdataset{AoPS-Instruct\xspace}
\def\ourtrain{AoPS-Instruct\xspace}
\def\oureval{LiveAoPSBench\xspace}
\def\ourevalearly{LiveAoPSBench-0824\xspace}
%
%
%
\icmltitlerunning{Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation}
\begin{document}
\twocolumn[
\icmltitle{Leveraging Online Olympiad-Level Math Problems for LLMs \\
Training and Contamination-Resistant Evaluation}
%
%
%
%
%
%
%
%
%
%
%
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Sadegh Mahdavi}{equal,ubc}
\icmlauthor{Muchen Li}{equal,ubc}
\icmlauthor{Kaiwen Liu}{ubc}
\icmlauthor{Christos Thrampoulidis}{ubc}
\icmlauthor{Leonid Sigal}{ubc,vector,cifar,crc}
\icmlauthor{Renjie Liao}{ubc,vector,cifar}
\end{icmlauthorlist}
\icmlaffiliation{ubc}{University of British Columbia, Vancouver, Canada}
\icmlaffiliation{vector}{Vector Institute for AI, Toronto, Canada}
\icmlaffiliation{cifar}{Canada CIFAR AI Chair}
\icmlaffiliation{crc}{NSERC CRC Chair}
\icmlcorrespondingauthor{Sadegh Mahdavi}{smahdavi@ece.ubc.ca}
\icmlcorrespondingauthor{Muchen Li}{muchenli@cs.ubc.ca}
%
%
%
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
%
%
%
%
%
%
%
%
\printAffiliationsAndNotice{\icmlEqualContribution} %
\begin{abstract}
Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems.
However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts.
In addition, current benchmarks are prone to contamination, leading to unreliable evaluations.
In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions.
Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in \textbf{\ourdataset}, a dataset of more than 600,000 high-quality QA pairs.
Our experiments demonstrate that fine-tuning LLMs on \ourdataset improves their reasoning abilities across various benchmarks.
Moreover, we build an automatic pipeline that introduces \textbf{\oureval}, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance.
Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability.
Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain.
Our benchmark and code is available at \url{https://livemathbench.github.io/leaderboard.html}
\end{abstract}
\input{secs/1_intro}
\input{secs/2_related_work}
\input{secs/3_method}
\input{secs/4_experiments}
\input{secs/5_conclusion}
\section*{Acknowledgments}
This work was funded, in part, by the NSERC DG Grant (No. RGPIN-2022-04636), the Vector Institute for AI, Canada CIFAR AI Chair, NSERC Canada Research Chair (CRC), NSERC Discovery Grants, a Google Gift Fund, and the Government of Canada’s New Frontiers in Research Fund NFRFE-2023-00936.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through the Digital Research Alliance of Canada \url{alliance.can.ca}, and companies sponsoring the Vector Institute \url{www.vectorinstitute.ai/#partners}, and Advanced Research Computing at the University of British Columbia.
Additional hardware support was provided by John R. Evans Leaders Fund CFI grant.
Sadegh Mahdavi and Muchen Li are supported by UBC Four Year Doctoral Fellowships.
%
%
%
\bibliography{icml_references}
\bibliographystyle{icml2025}
%
%
%
%
%
\newpage
\appendix
\onecolumn
\input{secs/appendix}
%
%
\end{document}
%
%
%
%
%
%
%
%
%
%
%
%
%
\section{Introduction}
Large language models (LLMs) have shown tremendous success in solving various tasks %
such as code generation~\citep{alpha-code}, math reasoning~\citep{deepseek-math}, and commonsense reasoning~\citep{Zellers2019HellaSwagCA,gpt4}, suggesting that current models may show signs of artificial general intelligence (AGI)~\citep{bubeck2023sparks}. Math reasoning is perhaps one of the most challenging tasks for the LLMs, since mathematics is inherently structured, requiring not just recall of facts but also rigorous logical inference, abstraction, and understanding of formal symbolic systems. As such, there have been grand challenges~\citep{imo_grand_challenge} and million-dollar prizes~\citet{AIMO2023} established for a model capable of solving Olympiad-level math problems.
On the training side, despite significant progress in certain areas, such as geometry, particularly with the assistance of symbolic methods~\citep{AlphaGeometryTrinh2024}, the performance of LLMs remains limited on Olympiad-level problems~\citep{olympiadbench}. One of the key challenges in advancing competition-level math reasoning, compared to other domains like coding or grade-school math, is the scarcity of large-scale data.
Creating valid and challenging math questions, along with providing correct solutions, is costly.
This is especially true for Olympiad-level problems, which can be time-consuming even for experts.
This highlights the need for scalable and automated methods to collect high-quality data for Olympiad-level problems to facilitate further advancements in this field.
%
On the evaluation side, in contrast to the rapid advancements in LLMs, the evaluation of their math reasoning capabilities remains relatively underdeveloped.
First, as aforementioned, the cost of creating and annotating advanced math problems is high.
%
Second, popular datasets such as MATH~\citep{math_dataset} and GSM8K~\citep{gsm8k_dataset} have been saturated by both open-source and closed-source models \citep{gpt4, qwen25}.
Third, benchmarks whose test sets are publicly available online \citep{math_dataset, gsm8k_dataset, olympiadbench, gaokao_bench} are prone to potential contamination.
Although techniques like n-gram matching and locality-sensitive hashing have been applied as a common practice \citep{gpt4,llama3,qwen2} to reduce contamination, they still suffer low accuracy and would not be able to rule out rephrased questions, as shown by \citet{yang2023rethinking}.
%
Given these limitations, it is crucial to develop an evolving evaluation benchmark that contains abundant and up-to-date test samples, and designed with appropriate difficulty to fairly assess a model’s math reasoning abilities.
The Art of Problem Solving\footnote{\hyperlink{https://artofproblemsolving.com/community}{https://artofproblemsolving.com/community}} (AoPS) forum is a rich resource for Olympiad-level math problems, featuring discussions on topics such as algebra, geometry, combinatorics, and number theory from competitions like AMC~\citep{amc23}, AIME~\citep{aime24}, and the International Mathematical Olympiad (IMO). However, the forum's unstructured nature, including irrelevant comments and incomplete solutions, poses challenges in extracting high-quality, structured question-answer (QA) pairs. Developing an effective automated pipeline to curate these QA pairs is essential to address the scarcity of large-scale, high-quality data for training and evaluating models in Olympiad-level math reasoning. In this paper, we utilize the posts from the AoPS forum to create a large-scale training and a contamination-resistant evaluation set. Our pipeline is designed to run automatically, enabling us to build and maintain evolving train/evaluation datasets. This automated approach is crucial, as it allows for continuously updating the datasets, ensuring they are less likely to suffer from contamination, even as existing data potentially becomes compromised over time. In summary, our key contributions are as follows:
\begin{itemize}
\item We build a pipeline to extract questions and solutions from raw AoPS forum data, constructing the \ourdataset, a novel large-scale dataset with $652$K Olympiad-level math QA pairs.
\item Using the most recent QA pairs, We build an automatic pipeline that introduces \oureval, a contamination-resistant evaluation set for assessing the math reasoning capabilities of LLMs. \oureval stays regularly updated.
\item Our experiments on \oureval show a declining performance trend over time for various LLMs, indicating potential data contamination, and stressing the need for up-to-date evaluation data.
\item Fine-tuning various LLMs on \ourdataset lead to improved performance on standard benchmarks such as OlympiadBench, Omni-Math, and our \oureval dataset, verifying the effectiveness of our dataset in enhancing math reasoning capabilities of LLMs.
\end{itemize}
\section{Related Work}
In this section, we provide an overview of the existing mathematical datasets used for evaluation and training purposes. Additionally, we review the latest methods and LLMs for enhancing and evaluating these math datasets.
%
\input{secs/tabs/eval_result}
\textbf{Evaluation Datasets for Math.}
The evaluation of the mathematical capabilities of LLMs has traditionally relied on well-established and widely-used datasets such as GSM8K and MATH \citep{gsm8k_dataset,math_dataset}, which have served as benchmarks for several years.
These datasets typically contain math problems ranging from middle-school to high-school level, providing broad coverage across various problem categories.
However, they present two significant limitations: 1) being older, their test sets are more susceptible to contamination from current training data of LLMs \citep{yang2023rethinking}, and 2) they have reached a level of saturation, with state-of-the-art (SOTA) models achieving over 90\% accuracy \citep{qwen25}.
To address these shortcomings, \citet{gaokao_bench} introduced the Gaokao dataset, which includes more challenging high school-level problems from the Chinese college entrance exam.
In addition, newer datasets such as OlympiadBench \citep{olympiadbench}, AMC23 \citep{amc23}, AIME24 \citep{aime24}, and Omni-Math \citep{omni-math} represent higher levels of difficulty, collecting from more recent high school competition problems.
While these datasets temporarily mitigate the risk of data contamination, they remain susceptible to this issue as LLMs continue to evolve, particularly with fine-tuning on newer data.
To address this, we introduce \oureval, which utilizes the most recent posts from the AoPS forum and applies substring-matching techniques to exclude any previously used problems from the new posts. More importantly, our pipeline is \emph{fully automated}, allowing the evaluation set to evolve with forum posts, thereby significantly decreasing the likelihood of contamination.
\textbf{Training Datasets for Math.}
Training datasets for mathematical reasoning can be categorized into two types: pretraining and supervised fine-tuning (SFT) datasets.
First, pretraining datasets consist of large-scale math data, \eg, billions of tokens used during the pretraining phase of LLMs.
Notable examples include Open-Web-Math \citep{openwebmath} and Minerva \citep{Minerva}, which contain 38.5B and 14.7B tokens of math data, respectively.
Second, SFT datasets focus on high-quality question-answer pairs. Examples include Open-Math-Instruct \citep{openmathinstruct}, Orca-math \citep{orca-math}, MetaMath~\citep{yu2024metamath}, DART-Math~\cite{tong2024dart}, ScaleQuest~\citep{ding2024unleashing}, $\text{MATH}^2$~\citep{shah2024ai}, and the training sets of widely used benchmarks such as GSM8K \citep{gsm8k_dataset} and MATH \citep{math_dataset}.
However, these datasets are generally limited to grade-school and intermediate high-school level mathematics and do not target more advanced topics like Olympiad-level math. One of the most closely related datasets to ours is Numina \citep{numina_math_datasets}, which combines popular SFT datasets like Orca-math, MATH, and GSM8K, along with approximately 190K new Olympiad-level QA pairs.
Concurrently, \citet{yue2024mammoth2} introduced a large-scale instruction fine-tuning dataset for math and science, which has also shown improvements in mathematical reasoning.
Table \ref{tab:related_dataset_com} presents a detailed comparison of our dataset with these related datasets.
%
\input{secs/tabs/tab_dataset_compare}
\textbf{Contamination-Resistant Evaluation.}
Benchmarks that are publicly accessible are prone to be contaminated due to the potential inadvertent data overlap during training.
%
The typical decontamination method involves using exact substring (\eg, $n$-gram) matching to detect overlaps with the target evaluation sets \citep{zhuo2024bigcodebench}.
However, this approach fails to catch rephrased examples and can not eliminate all overlaps with the test set~\citep{yang2023rethinking}.
While alternative LLM-based methods for decontamination have been proposed, they often lack guarantees and may result in high false-positive rates \citep{yang2023rethinking}.
A reliable way to mitigate contamination is to select data that appeared after LLMs were trained, known as the \emph{knowledge cut-off}.
In the code generation domain, LiveCodeBench \citep{livecodebench} addresses this issue by categorizing data based on timestamps, setting a cutoff date, and designating data beyond this point as unseen.
We adopt a similar strategy in the math domain, partitioning the dataset by timestamps and enabling users to select data based on specific dates.
Although this approach may not fully eliminate rephrased existing questions, it ensures that evaluation data remains unseen and less contaminated, providing a more accurate and fair assessment of LLMs.
\begin{figure*}[ht]
\centering
\includegraphics[width=0.85\textwidth]{figs/pipeline_v0.2.pdf}
\caption{{\bf The overall process of our dataset curation.} \textit{Top:} Processing pipeline of AoPS-Instruct (training data). First, irrelevant topics are detected using a small LLM. We then extract question-answer pairs from relevant discussions, and rewrite each answer into a step-by-step solution. \textit{Bottom:} \oureval Processing pipeline of LiveAoPSBench (evaluation data). We take the most recent posts, use two LLMs to rewrite the solution, filter out the questions without clear answers, and create the final evaluation set.}
\label{fig:dataset_pipeline}
%
\end{figure*}
\textbf{Math-Specific Models.}
%
%
Several specialized models have been developed to improve the mathematical reasoning capabilities of LLMs \citep{deepseek-math,mathstral,numina_math_datasets,qwen25,azerbayev2024llemma}.
These models are typically initialized from pretrained general-purpose models, trained on large math datasets, followed by math-specific SFT, and then refined through reinforcement learning with human feedback (RLHF).
In this paper, we fine-tune both general and math-specific models to demonstrate that \ourdataset brings consistent improvements.
\begin{figure*}[t]
\centering
%
\begin{minipage}{0.38\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/combined_models_accuracy_plot.pdf}
\end{minipage}
\begin{minipage}[c]{0.4\linewidth}
\centering
\resizebox{\textwidth}{!}{
%
\begin{tabular}{rl|ccc}
\toprule
& \multirow{2}{*}{Model} & {2023} & {2024} & {Drop}\\
%
& & ($5.2$K) & ($3.8$K) & (\%) \\
\midrule
\midrule
\includegraphics[width=0.35cm]{figs/avatar/Qwen2.5-Math-7B-Instruct_avatar.png} & 7B-Qwen2.5-Math-Ins & 34.80 & 33.40 & 4.02 \\
\includegraphics[width=0.35cm]{figs/avatar/Qwen2-Math-72B-Instruct_avatar.png} & 72B-Qwen2-Math-Ins & 37.84 & 36.15 & 4.45\\
\includegraphics[width=0.35cm]{figs/avatar/Qwen2.5-Math-72B-Instruct_avatar.png} & 72B-Qwen2.5-Math-Ins & 42.36 & 40.45 & 4.51\\
\includegraphics[width=0.35cm]{figs/avatar/NuminaMath-72B-CoT_avatar.png} & 72B-NuminaMath-CoT & 25.59 & 24.14 & 5.68\\
%
\includegraphics[width=0.35cm]{figs/avatar/Internlm2-Math-plus-20b_avatar.png} & 20B-Internlm2-Math-Plus & 17.78 & 16.03 & 9.83\\
\includegraphics[width=0.35cm]{figs/avatar/DeepSeek-Coder-V2-Lite-Ins_avatar.png} & 16B-DeepSeekCoderV2-Ins & 22.08 & 19.80 & 10.31\\
%
\includegraphics[width=0.35cm]{figs/avatar/Qwen2-Math-7B-Instruct_avatar.png} & 7B-Qwen2-Math-Ins & 33.26 & 29.32 & 11.85\\
\includegraphics[width=0.35cm]{figs/avatar/NuminaMath-7B-CoT_avatar.png} & 7B-NuminaMath-CoT & 16.88 & 14.76 & 12.55\\
\includegraphics[width=0.35cm]{figs/avatar/DeepSeek-Math-7b-rl_avatar.png} & 7B-DeepSeek-Math-RL & 14.35 & 12.44 & 13.35\\
\includegraphics[width=0.35cm]{figs/avatar/Mathstral-7B-v0.1_avatar.png} & 7B-Mathstral-v0.1 & 15.25 & 13.00 & 14.76\\
\includegraphics[width=0.35cm]{figs/avatar/Internlm2-Math-plus-7b_avatar.png} & 7B-Internlm2-Math-Plus & 16.26 & 13.64 & 16.16\\
\midrule
\includegraphics[width=0.35cm]{figs/avatar/Gemma-2-27b-it_avatar.png} & 27B-Gemma2-it & 12.78 & 11.59 & 9.30\\
\includegraphics[width=0.35cm]{figs/avatar/Llama-3.1-70B-Instruct_avatar.png} & 70B-Llama-3.1-Ins& 22.02 & 19.34 & 12.16 \\
\includegraphics[width=0.35cm]{figs/avatar/Llama-3.1-8B-Ins_avatar.png} & 8B-Llama3.1-Ins & 13.01 & 10.85 & 16.55 \\
\includegraphics[width=0.35cm]{figs/avatar/Llama-3.2-3B-Ins_avatar.png} & 3B-Llama3.2-Ins & 12.67 & 10.32 & 18.51\\
\includegraphics[width=0.35cm]{figs/avatar/Gemma-2-9b-it_avatar.png} & 9B-Gemma2-it & 11.63 & 9.30 & 20.01\\
%
\includegraphics[width=0.35cm]{figs/avatar/Llama-3.2-1B-Ins_avatar.png} & 1B-Llama-3.2-Ins & 6.32 & 4.83 & 23.62\\
\bottomrule
\end{tabular}
}
\end{minipage}
\caption{Accuracy trends of various LLMs on \oureval over an 18-month period, highlighting a consistent decline in performance. We separate the math expert model from the general purpose model on the right. The degradation in accuracy varies across models, ranging from $2.4\%$ to $23.6\%$. Please refer to figure \ref{fig:aops_2024_trend} for results on state-of-the-art reasoning models like Qwen-QwQ and DeepSeek-R1.}
\label{fig:acc_wrt_time}
\end{figure*}
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
\begin{table}[t]
\centering
\caption{Comparison of our dataset with other related datasets from the literature. Our dataset uniquely includes timestamp information and leverages open-source large language models (LLMs)s like Qwen 2.5 72B for solution rewrites. $^{\star}$ denotes inclusion of additional training datasets such as GSM8K, Orca-Math, and MATH. Datasets marked with $^\dagger$ have their solutions entirely generated by LLM.}
\vspace{0.15cm}
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{lccccc}
\toprule
\multirow{2}{*}{Dataset} & \multicolumn{2}{c}{Dataset Size} & Time & Olympiad & Solution\\
\cline{2-3}
& Train & Eval & Stamp & Level & Rewrite\\
\midrule
\midrule
Numina & $859$K$^\star$ & $0.1$K & \ding{55} & \ding{51} & GPT4-o \\
%
OpenMathInstruct & $1.8$M & $-$ & \ding{55} & \ding{55} & Mixtral$^\dagger$ \\
%
OlympiadBench & $-$ & $6.1$K & \ding{55} & \ding{51} & Human \\
DART-Math & $585$K & $-$K & \ding{55} & \ding{55} & DeepSeekMath$^\dagger$ \\
%
GSM8K & $7.5$K & $1$K & \ding{55} & \ding{55} & Human \\
%
MATH & $7.5$K & $5$K & \ding{55} & \ding{55} & Human \\
%
Orca-Math & $200$K & - & \ding{55} & \ding{55} & GPT-4$^\dagger$ \\
%
%
MAmmoTH2 & $10$M & - & \ding{55} & \ding{55} & Mixtral \\
\midrule
AoPS (Ours) & $647.2$K & $3.8$K & \ding{51} & \ding{51} & Qwen 2.5 \\
\bottomrule
\end{tabular}
}
\label{tab:related_dataset_com}
\end{table}
\input{secs/tabs/example_rewritten}
\section{AoPS Dataset}
In this section, we first describe the process of extracting and cleaning QA pairs from the AoPS forum to construct our training set.
Then we explain how to utilize the latest forum data to create a reliable, contamination-resistant evaluation dataset for assessing model performance.
\subsection{Math Instruction Fine-tuning Dataset: \ourtrain} \label{sec:train-dataset}
We now describe the five steps of our automated pipeline for constructing the instruction fine-tuning dataset \ourtrain.
\textbf{Step 0: Raw Forum Discussion Collection.}
We begin by collecting raw discussions from the forum website, where each discussion is called a ``topic''.
In these topics, the author presents math problems (\eg, competition-level problems) or general questions, such as seeking advice or resources.
Our raw dataset consists of $1,076,712$ topics.
Topics posted up until December 2023 are used as the training set, while those posted between January and August 2024 are reserved as the evaluation dataset.
\textbf{Step 1: Math Question Detection.}
We then filter out irrelevant topics, specifically those not containing a mathematical question.
To achieve this, we use Qwen 2.5 14B \citep{qwen2} to decide the relevance of each topic.
The first post of each topic determines whether the topic is a mathematical question or not, so we manually design a few-shot prompt, provide the first post of the topic to the model, and prompt the model to output if the topic is a math question or not.
This step reduces the dataset to $478,337$ topics with math questions after pruning $598,375$ irrelevant ones.
\textbf{Step 2: Question-Answer Extraction.}
After filtering, we extract the math question from the first post of each topic and identify potential solutions provided in subsequent posts.
Since this task requires understanding the entire conversation and determining which responses contain valid solutions, we employ the 70B variant of Llama 3.1 for this step, enabling the detection of both the question and all relevant answers from the discussion.
\textbf{Step 3: Solution rewriting.}
Math solutions generated by users on the AOPS forum are often concise, omitting details assumed to be common knowledge among the target audience.
For instance, a user might write ${(x+yz)}/{2} \geq \sqrt{xyz}$ without explicitly mentioning the application of the AM-GM inequality to $(x,yz)$.
While such brevity is typical for expert-level discussions, LLMs trained on these succinct solutions often struggle to maintain their chain-of-thought reasoning capabilities.
Our experiments show that fine-tuning a model on these concise solutions significantly degrades its performance on standard benchmarks (see Section \ref{sec:ablation} and Figure \ref{fig:ablate_rewrite}).
To address this issue, we utilize the Qwen 2.5 72B model~\citep{qwen25} to rewrite all solutions into detailed, step-by-step explanations.
This approach aligns with similar techniques used in prior work, such as the Numina project \citep{numina_math_datasets}, which also employed solution rewriting to improve response quality.
An example of a rewritten solution is provided in Figure \ref{fig:example_rewritten}, and the overall dataset curation process is illustrated in Figure \ref{fig:dataset_pipeline}.
\textbf{Step 4: Data Decontamination.}
After processing all the QA pairs, we apply data decontamination to remove any overlap with the test sets of commonly used math benchmarks.
Following the approach used in DeepSeekMath \citep{deepseek-math}, we employ a 10-gram exact match decontamination~\citep{zhuo2024bigcodebench} method to ensure that our dataset remains distinct from those benchmarks.
%
After following the steps described above, we have a total of $652$K QA pairs, out of which $647,255$ are before Jan 2024 and constitute the \ourtrain. We provide further statistics of our dataset in Section \ref{sec:dataset_stats} and Figure \ref{fig:stats}.
\subsection{Contamination-Resistant Evaluation: \oureval} \label{sec:eval-dataset}
Math LLMs are trained on large instructional corpora. A common issue with current evaluation sets is the risk of contamination, where test samples may inadvertently overlap with training data.
To create contamination-resistant benchmarks, we constructed our evaluation set by sorting the raw data based on the initial posting timestamp and including only the most recent entries.
%
Our evaluation set, denoted as \oureval, is sourced from the AoPS forum, with posts strictly between January 2023 and September 2024.
We utilize the same pre-processing pipeline, depicted in Figure \ref{fig:dataset_pipeline}, to extract QA pairs and have the raw solutions rewritten for consistency.
\textbf{Filtering}.
%
The correctness of the solution is typically verified by comparing the final answer to the human-annotated answer.
%
Note that human-annotated answers may still contain errors, as we do not perform formal proofs or verification.
When constructing an evaluation set, it is essential that each question has a concrete and definite answer, which is enclosed as $\boxed{ans}$ format for ease of parsing, as illustrated in Figure \ref{fig:example_rewritten}.
%
%
We start by applying a series of heuristic filters to exclude proof-based questions and extract only those with explicit, boxed answers.
To ensure that our test set does not contain problems included in widely used training sets, we use an stricter 8-gram matching filter—stricter compared to the 10-gram filter used for training set decontamination. This helps eliminate any potential overlap with common training corpora \citep{math_dataset,gsm8k_dataset,orca-math}.
%
%
\textbf{Cross-Check by LLMs}.
A key challenge in building a fair evaluation set is ensuring the accuracy and validity of QA pairs.
To automate this process, we employed two different models—Llama3.1-70B-Ins \citep{llama3} and Qwen2.5-72B-Ins \citep{qwen2} to perform the rewriting step twice for each question. Consequently, for each question $Q$, we obtain a triplet: $({A}_{\text{qwen}}, {A}_{\text{llama}}, {A}_{\text{original}})$.
%
If a boxed answer is detected in ${A}_{\text{original}}$, it is automatically accepted as a candidate answer for the question.
%
Following this, we performed a cross-check between ${A}_{\text{qwen}}$ and ${A}_{\text{llama}}$, removing all cases with inconsistent answers.
This was done through string matching for text and value matching for numbers, while a SymPy-based \citep{sympy} symbolic equivalence program was used for SymPy-parsable expressions.
The final answers are obtained by deduplicating the candidate answers.
%
%
%
%
%
%
%
Through this process, we constructed \oureval, which contains 3863 \footnote{The version of \oureval we present in this paper corresponds to LiveAoPSBench-0824 which contains problems posted between 01-2024 to 08-2024.}
%
examples, all of which are from posts in the year 2024.
Further details can be found in Appendix \ref{appendix:liveaopsbench_detail}.
\textbf{Quality Verification.}
We assess the quality of our dataset by having a group of 10 graduate students annotate a randomly selected 10\% subset (386 cases) from our evaluation set. Each human annotator verifies whether the final answer is correct based on the raw post, with each question annotated by two different individuals. We report the percentage of cases marked as correct by the human annotators to measure the correlation between human judgment and our method. Additionally, since Olympiad-level questions can be challenging even for humans, we also report the inter-annotator agreement to evaluate consistency between different groups of human annotators.
More details can be found in Section \ref{sec:ablation}.
%
\textbf{Evolving Evaluation with Up-to-date Data.}
%
Since our pipeline does not require human annotators, we are able to continuously update our \oureval in an automated manner.
This makes our benchmark an up-to-date and timestamped evaluation set that is resistant to contamination, thereby providing a more reliable mathematical evaluation resource for the research community.
%
%
%
%
%
\begin{figure*}[t]
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Find all solutions in $\mathbb{N}$: $2^{2^{x}}+1=17y$
\vspace{3mm}
\textbf{User Response:}
The multiplicative order of $2$ modulo $17$ is $8$, since $2^4 = 16 \equiv -1 \pmod{17}$. Therefore, if $x\geq 3$, then $8 \mid 2^x$, and so $2^{2^x} \equiv 1 \pmod{17}$. So $x\leq 2$, and so the only solution is $(x,y) = (2,1)$.
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution:}
1. Determine the multiplicative order of $2$ modulo $17$. We find that $2^4 = 16 \equiv -1 \pmod{17}$.
2. Since $2^4 \equiv -1 \pmod{17}$, it follows that $2^8 \equiv 1 \pmod{17}$. Therefore, the multiplicative order of $2$ modulo $17$ is $8$.
3. Consider the equation $2^{2^x} + 1 = 17y$. If $x \geq 3$, then $2^x$ is a multiple of $8$ (since $2^3 = 8$ and higher powers of $2$ are multiples of $8$).
4. If $2^x$ is a multiple of $8$, then $2^{2^x} \equiv 1 \pmod{17}$, because the order of $2$ modulo $17$ is $8$.
5. Substituting $2^{2^x} \equiv 1 \pmod{17}$ into the equation, we get $1 + 1 \equiv 17y \pmod{17}$, which simplifies to $2 \equiv 0 \pmod{17}$. This is a contradiction.
6. Therefore, $x$ must be less than $3$. The possible values for $x$ are $0$, $1$, and $2$.
7. Check each possible value of $x$:
\begin{itemize}
\item For $x = 0$: $2^{2^0} + 1 = 2^1 + 1 = 3$, which is not divisible by $17$.
\item For $x = 1$: $2^{2^1} + 1 = 2^2 + 1 = 5$, which is not divisible by $17$.
\item For $x = 2$: $2^{2^2} + 1 = 2^4 + 1 = 16 + 1 = 17$, which is divisible by $17$.
\end{itemize}
8. The only solution is $(x, y) = (2, 1)$.
Thus, the solution is \boxed{(2, 1)}.
\end{small}
\end{tcolorbox}
\caption{
Example of a posted problem, a user solution, and a LLM-rewritten solution. The model fills in the trivial intermediate steps, reformats the response, and puts the final answer in a box.
}
\label{fig:example_rewritten}
\end{figure*}
\section{Experiments}
\subsection{Dataset Statistics} \label{sec:dataset_stats}
We provide a better overview of the AoPS dataset in Figure \ref{fig:stats}. As shown in Figure \ref{fig:stats_ans_per_q}, more than $60\%$ of the questions have only one answer, while around $24\%$ and $8\%$ have two and three answers, respectively.
Figure \ref{fig:stats_q_per_year} shows the number of posts across each year, with a cut-off of August 2024.
We observe that each year at least $15$K mathematical questions are posted to the forum. This translates to more than $1,000$ monthly questions, which shows the potential of the AoPS forum to be used as training, and especially evaluation set.
Figure \ref{fig:stats_category} shows a breakdown of the types of questions in our dataset. Proof questions and numerical questions with about $32\%$ and $28\%$ constitute the majority of the questions in our dataset.
Finally, Figure \ref{fig:stats_overlap} shows the pairwise overlap between each pair of popular supervised fine-tuning datasets using substring matching between the two datasets of each pair. Among the two Olympiad-level datasets (\ie, ours and Numina), our dataset has the least overlap with common datasets (with less than $14.1\%$ overlap), which shows the number of new data points.
\subsection{Evaluation Set with Timestamps Mitigate Contamination}
\begin{table}[t]
\centering
\caption{10-gram Overlap Statistics Across Time Periods}
\resizebox{\linewidth}{!}{
\begin{tabular}{lccccc}
\toprule
Time & 23/01-04 & 23/05-08 & 23/09-12 & 24/01-04 & 24/05-08 \\
\midrule
Overlap (\%) & 13.24 & 11.65 & 12.82 & 9.92 & \textbf{6.88} \\
Count & 229 & 208 & 218 & 226 & 109 \\
%
\bottomrule
\end{tabular}
}
\label{tab:contam_rate}
\end{table}
We analyze the relationship between timestamps and contamination using our evaluation set, tested against the Numnia Training Set (released July 2024). As shown in \cref{tab:contam_rate}, we report contamination levels based on 10-gram overlap rates across different timestamps. We observe a consistent decrease in contamination rate as the test data timestamps become more recent.
\subsection{Evaluating Open-Sourced Models}
We evaluate the models' performance as a function of time window. As shown in Fig \ref{fig:acc_wrt_time}, we find that all the models experience a performance drop when evaluating 2024 questions compared to questions in 2023. This decline suggests that performance on earlier examples may not accurately reflect the true capabilities of LLMs, as the initial results could be inflated by inadvertent data overlap.
\begin{figure*}[ht]
\centering
\begin{subfigure}[b]{0.2\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/aops_ans_per_q.pdf}
\caption{Number of answers per question.}
\label{fig:stats_ans_per_q}
\end{subfigure}
\begin{subfigure}[b]{0.29\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/aops_timestamp.pdf}
\caption{Number of questions per year, based on post date.}
\label{fig:stats_q_per_year}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.2\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/aops_categories_pie.pdf}
\caption{Problem category distribution.}
\label{fig:stats_category}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.29\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/overlap_heatmap.pdf}
\caption{Pairwise overlap between various datasets.}
\label{fig:stats_overlap}
\end{subfigure}
\hfill
%
\caption{AoPS Dataset Statistics. The statistics are across all the datapoints in our dataset before split. In (d), the percentage at row $i$ and column $j$ shows the fraction of the training set of $i$-th dataset (based on exact substring match) present in the $j$-th dataset. Our dataset has the least overlap with others with less than $14.1\%$ overlap.}
\label{fig:stats}
%
\end{figure*}
\subsection{Instruction Fine-Tuning}
\input{secs/tabs/sft_result}
We show that the collected training dataset is effective at improving the math reasoning capabilities of LLMs.
To this end, we choose $4$ representative LLMs and fine-tune them on our dataset combined with the Numina~\citep{numina_math_datasets} dataset, and show that such a combination provides superior performance compared to training on either of the datasets alone.
We use the following set of diverse models for fine-tuning evaluation: (1) Mathstral-7B~\citep{mathstral}: a math-specialized model derived from Mistral-7B~\citep{mistral}, (2) DeepSeekMath-7B~\citep{deepseek-math}: a math-specialized model based on the DeepSeek family, and (3) Llama 3.2 3B~\citep{llama3} and (4) Llama 3.2 1B~\citep{llama3}, two recent general state-of-the-art models.
For each QA pair, only the question is used as the instruction, with the rewritten solution serving as the response, formatted within the model’s respective chat template. For instance, with Mathstral, we use the prompt: \texttt{<s>[INST] {question} [/INST]{solution}} for instruction tuning.
Consistent with prior work, we train each model for three epochs~\citep{deepseek-math,qwen25}, as we observe additional epochs provide no further benefit (see Figure \ref{fig:ablation_train_budget} in the Appendix for ablation studies on the number of epochs). We explore three data mixtures for fine-tuning: (1) AoPS alone, (2) Numina alone, and (3) AoPS + Numina. For each dataset, we use the full decontaminated dataset for fine-tuning (\ie., Numina with $824$K QA pairs, and AoPS-Instruct with $647$K QA pairs).
After fine-tuning each model, we evaluate the performance of each model on the following standard competition-level benchmarks: (1) OlympiadBench~\citep{olympiadbench}, which is an Olympiad-level evaluation dataset. Following prior literature~\citep{qwen2}, we take only the math questions which have final answers and do not contain images or figures. This leaves us with $675$ samples from this dataset (2) Omni-MATH~\citep{omni-math}, which is a collection of $4428$ problems from various mathematical olympiad competitions. (3) \oureval set for the year 2024. The results are shown in Table \ref{tab:method_comparison}. As shown by the table, fine-tuning with our dataset consistently boost the performance.
\subsection{Ablation Studies}
\label{sec:ablation}
\textbf{Evaluation Quality Assessment}. We assess the quality of our evaluation set in two ways: by measuring its correlation with a well-established dataset and through manual evaluation over a subset of the data. First, \citet{olympiadbench} compiled an Olympiad-level math evaluation set using manual assessment, which we leverage in our context to verify the quality of our method through the correlation between accuracies. Figure \ref{fig:olympbench_correlation}, demonstrates that the evaluation on \oureval is highly correlated with carefully established benchmarks such as OlympiadBench. This demonstrates that our automatically generated benchmark aligns closely with the quality of those created through extensive human effort.
%
Next, we subsample 10\% of our evaluation set and ask human annotators to verify the correctness of the final parsed answers by referring to the original post. Annotators are given three options: yes, no, and no-answer. ``Yes" and ``no" indicate whether the answer is deemed correct, while ``no-answer" is selected when a concrete answer is not appropriate (\eg, abstract concept questions answered with concrete examples).
As a result, we found that 92\% of the annotations were marked as correct, while 5\% were incorrect and 3\% fell under the no-answer category.
%
%
%
%
\textbf{Rewritting's effect on performance.}
We also ablate the effect of solution rewriting, which is an important part of our pipeline. As shown in Figure \ref{fig:ablate_rewrite}, rewriting solutions into a step-by-step format substantially improves the test accuracy across all benchmarks. The Qwen-2.5 72B based rewriting performs favorably against Llama-3.1 70b based rewriting on competition-level math benchmarks, while being slightly worse on easier grade-school math. Overall, we found Qwen to be a stronger model, providing a higher amount of details and being less verbose compared to Llama in rewriting solutions (see Figure \ref{fig:example_rewritten_qwen_vs_llama} in the Appendix for a qualitative example). This suggests that rewriting solutions with stronger models can significantly improve performance on benchmarks.
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/aops24_olympiad_accuracy_correlation_scatter_plot.pdf}
\caption{Correlation with the OlympiadBench dataset.}
\label{fig:olympbench_correlation}
\end{subfigure}
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=\linewidth]{figs/ablate_rewrite.pdf}
\caption{Ablation on Rewriting.}
\label{fig:ablate_rewrite}
\end{subfigure}
\hfill
\caption{Ablations on \oureval. (a) The performance of models on our benchmark is highly correlated with established datasets. (b) The effect of rewriting user solutions into a step-by-step solution with two different models, on an early version of our dataset. Rewriting solutions always improves accuracy, and using stronger models leads to larger accuracy gains.}
\label{fig:correlation}
\end{figure}
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
\begin{table}[t]
\centering
\caption{Performance comparison of different models fine-tuned on various datasets across multiple benchmarks. Bold values in the columns for No SFT, Numina, and AoPS-Ins represent the highest scores for individual datasets. Additionally, bold values for Numina+AoPS-Ins indicate performance that matches or surpasses all other fine-tuning alternatives.}
\vspace{0.15cm}
\resizebox{\linewidth}{!}{
\begin{tabular}{c|c|cccc}
\toprule
Model & SFT Dataset & AoPS24 & Math & \makecell{Olympiad \\ Bench} & \makecell{Omni \\ Math} \\
\midrule
\midrule
& No SFT & 11.7 & 47.1 & 14.5 & 12.3 \\
Deepseek-Math & Numina & 16.3 & 55.5 & 22.7 & {17.0} \\
7b-Ins & AoPS-Ins & \new{\textbf{19.0}} & \new{\textbf{58.8}} & \new{\textbf{24.3}} & \new{\textbf{17.8}} \\
\cmidrule{2-6}
& Numina+AoPS-Ins & {\textbf{19.7}} & {\textbf{58.8}} & {\textbf{25.6}} & {\textbf{18.0}} \\
\midrule
& No SFT & 15.40 & 56.30 & 21.20 & 15.90 \\
Mathstral & Numina & 16.60 & 54.60 & 23.40 & 17.10 \\
7B & AoPS-Ins & {\textbf{23.60}} & {\textbf{60.80}} & {\textbf{27.10}} & {\textbf{19.90}} \\
\cmidrule{2-6}
& Numina+AoPS-Ins & {\textbf{24.90}} & {{59.60}} & {\textbf{29.60}} & {\textbf{21.10}} \\
\midrule
& No SFT & 12.0 & 47.4 & 16.1 & 12.9 \\
Llama-3.2 & Numina & 12.9 & 49.5 & 19.3 & 14.4 \\
3B-Ins & AoPS-Ins & \textbf{16.7} & \textbf{54.6} & \textbf{19.6} & \textbf{16.4} \\
\cmidrule{2-6}
& Numina+AoPS-Ins & \textbf{17.4} & \textbf{55.6} & \textbf{22.8} & \textbf{17.2} \\
\midrule
& No SFT & 5.30 & 28.80 & 4.70 & 7.00 \\
Llama-3.2 & Numina & 8.00 & 32.70 & 6.40 & 9.70 \\
1B-Ins & AoPS-Ins & {\textbf{10.00}} & {\textbf{34.70}} & {\textbf{11.10}} & {\textbf{11.00}} \\
\cmidrule{2-6}
& Numina+AoPS-Ins & {\textbf{11.20}} & {\textbf{36.60}} & {\textbf{12.00}} & {\textbf{11.70}} \\
\bottomrule
\end{tabular}
}
\label{tab:method_comparison}
\end{table}
\section{Limitations}
\textbf{Absence of Visual Content.} Our dataset currently focuses on text-only problems, which may limit its effectiveness in certain areas, particularly geometry. Many geometry problems rely heavily on diagrams to fully convey the problem statement. Incorporating relevant images and figures could significantly enhance the dataset's comprehensiveness and applicability, especially in visually-dependent mathematical domains.
\textbf{Evaluation of Proof-based Questions.} Our evaluation dataset focuses on QA pairs with clear, final answers, which is well-suited to a broad range of Olympiad-level problems. However, a significant portion of such types of problems involve more complex proof-based questions that require detailed logical reasoning and multiple steps. While we incorporate proof-based questions in our instruction-tuning pipeline, the current evaluation pipeline lacks the ability to evaluate such questions effectively.
\textbf{Quality Variability in Community-Generated Content.} The community-driven content from the AoPS forum provides a rich source of high-quality data. Nevertheless, as with any community-generated content, the quality of answers and solutions can vary. While our filtering and refinement processes have successfully mitigated much of this noise, incorporating more advanced techniques in future iterations could result in better consistency and precision.
\section{Conclusion and Future Work}
In conclusion, this paper introduces the \ourdataset dataset and \oureval, leveraging community-driven content from the Art of Problem-Solving forum to address the challenges of limited training data and unreliable evaluation for LLMs solving Olympiad-level math problems. By developing a scalable and automated pipeline for extracting and refining question-answer pairs, this work presents a dataset containing over $600,000$ QA pairs, along with an up-to-date, contamination-resistant evaluation benchmark. Our experiments demonstrate significant performance improvements across multiple standard benchmarks for models fine-tuned on the \ourdataset, highlighting enhanced mathematical reasoning capabilities. Furthermore, the observed performance decline of various LLMs on \oureval underscores the importance of continuously updating evaluation sets to mitigate the risks of data contamination.
For future work, there are several promising directions to explore. First, while this paper focuses on the AoPS forum, the pipeline developed is not limited to this domain. It is generalizable and can be applied to other online forums or different subject areas, enabling the creation of high-quality datasets for various fields, such as physics, computer science, or even non-technical disciplines. Expanding this pipeline to other knowledge-intensive communities could further improve the training and evaluation of LLM across disciplines.
Additionally, the quality of the dataset can be significantly improved by incorporating more advanced LLMs into the pipeline. Leveraging state-of-the-art models for question extraction, answer detection, and solution rewriting would result in more accurate and detailed data, ultimately enhancing the effectiveness of the fine-tuned models.
Moreover, the problems is AoPS-Instruct could be used as a diverse set to distill larger models into smaller models.
Lastly, the current pipeline focuses on question-answer pairs with clear final answers, but a significant portion of Olympiad-level problems involves proof-based questions that require a deeper evaluation of logical reasoning, argument structure, and intermediate steps.
Future work could include adapting the pipeline to accommodate these proof-based problems, potentially using another advanced LLM as a judge~\new{\citep{alpaca_eval}}, or incorporating formalization methods to better assess these complex solutions.
\section*{Impact Statement}
This paper aims to advance the field of mathematical reasoning. It is important to note that \ourdataset and \oureval contain no personal information. Additionally, we have not identified any societal consequences that require special attention in this context.
\section{More details on \oureval}
\label{appendix:liveaopsbench_detail}
\subsection{Evaluation pipeline statistics}
To begin with, we have 14158 QA pairs with time stamps between Jan-2024 and Aug-2024.
Decontamination with 8-gram matching is performed against Math and GSM8K training set \citep{math_dataset,gsm8k_dataset}, which removes 664 Q-A pairs.
After removing proof questions and non-boxed solutions, we are left with 7173 Q-A pairs over 5416 unique questions.
Lastly, The LLM cross-check filters out 1553 questions with inconsistent solutions and the resulting \oureval contains 3863 questions.
We apply the same pipeline described in Sec \ref{sec:eval-dataset} to data with a time stamp between Jan-2023 and Dec-2023 and get 5216 questions for the 2023 split result.
\subsection{Human Annotation}
As shown in Figure \ref{fig:human_annotation}, we develop a simple web interface for human annotators to verify the answers extracted by our LLMs. Annotators compare the ``Voted Answer'', ``Original Answers'' and all posts in the original topic page identified by LLMs to verify if the ``Voted Answer'' matches the original posts' answers. The verification process provides four results: Positive (``Yes''), negative (``No/No Answer''), and neutral (``Not sure''). The ``Not sure'' option is provided since verifying the answer sometimes requires a certain mathematical foundation and a significant amount of reading time.
%
We also show highlight two examples of disagreement in Figure \ref{fig:example_inconsistent_annotation}.
\subsection{Derivation of Difficulty Levels}
The difficulty levels in this dataset do not reflect the exact difficulty of the problems but rather approximate the general education background of the problem, \eg, this is a ``High School'' level problem. However, a challenging high school problem may be more complex than an easy college-level problem. The classification is derived from the problem tag in the AOPS forum, where the categories correspond to ``Middle School'', ``High School'', ``College'', and ``High School Olympiads''. In addition, some problems originate from special forums, which do not fit into the above categories and are classified as ``Others'' in our dataset.
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{figs/Human_Annotation.jpg}
\caption{Human Annotation Interface.}
\label{fig:human_annotation}
\end{figure}
\input{secs/tabs/example_annotation_disagreement}
\section{Detailed Evaluation Results on \oureval}
\subsection{LiveAoPSBench-2024}
We update the \oureval with the complete 2024 data. This resulted in LiveAoPSBench-2024 with 5328 questions. In figure \ref{fig:aops_2024_trend} we provide the latest result with the state-of-the-art reasoning models like deepseek-r1\cite{deepseek_r1} and QwQ\cite{qwq-32b-preview}.
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{figs/aops_24_performance_over_time.pdf}
\caption{Accuracy trend over time for LiveAoPSBench-2024. For the most up-to-date results, please visit: \url{https://livemathbench.github.io/leaderboard.html.}}
\label{fig:aops_2024_trend}
\vspace{-1em}
\end{figure}
\subsection{Evaluating open-sourced LLMs}
We have selected several mainstream open-source general LLMs and math-specific LLMs that demonstrate high performance on the previous math evaluation datasets.
For math-specific LLMs, we choose DeepSeek-Math-7b-rl~\citep{deepseek-math}, Mathstral-7B-v0.1~\citep{mathstral}, 7b and 20b versions of Internlm2-Math-plus~\citep{internlm2-math}, 7B and 72B versions of NuminaMath-CoT~\citep{numina_math_datasets}, 1.5B,7B,72B version of Qwen2-Math-Instruct~\citep{qwen2} and Qwen2.5-Math-Instruct~\citep{qwen25} as the representative of the math specific LLMs. Additionally, we include DeepSeek-Coder-V2-Lite-Instruct~\citep{deepseek-coder}, which is a code specialist model trained on both math and code corpus.
For general purpose LLMs, We report performance on 1B, 3B and 8B versions of the Llama3 family models~\citep{llama3} as well as 9B and 27B versions of Gemma-2-Instruct~\citep{gemma2} model.
\subsection{Detailed Results}
The accuracy comparison for these mainstream open source LLMs are shown in Tables~\ref{tab:combined_tables},~\ref{tab:accuracy_type2},~\ref{tab:accuracy_type3} split by Month, Difficulty and Answer Type. The Month tables separately include evaluation results for 2023 and 2024. For the Difficulty and Answer Type tables, we use only the most recent evaluation results from 2024. Notably, the difficulty labels represent the general educational background of the problems rather than their exact difficulty. Over half of the problems originate from educational backgrounds associated with High School or High School Olympiads, and only around 7\% are from Middle School, indicating our dataset's focus is more on the complex problems. Similarly, in the Answer Type Table, more than half of the problems are categorized as numeric-int.
\input{secs/tabs/appendix_aops23-24_by_time}
\input{secs/tabs/appendix_aops23-24_by_diff}
\input{secs/tabs/appendix_aops23-24_by_type}
\section{Training set details}
\subsection{Decontamination Details}
We use 10-gram substring matching to decontaminate against test set for a comprehensive list of math evaluation datasets available. \citep{gsm8k_dataset,math_dataset,olympiadbench,amc23,aime24,gaokao_bench,Minerva,omni-math,Asdiv,Mmlu,Mawps,Svamp,Carp}.
In Figure \ref{fig:decon_details}. We show the decontamination statistic for our dataset and Numina.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figs/decon_numina_vs_aops.pdf}
\caption{Decontamination Statistics: We perform decontamination on the raw dataset to produce \ourtrain, with the same method as the Numina-Math-COT. Both datasets show considerable overlap with the MATH dataset. \ourtrain exhibits more contamination within our 2024 split due to repeated questions, while Numina-Math-COT has higher contamination with other external datasets, reflecting its multi-source composition.}
\label{fig:decon_details}
\end{figure}
\section{SFT Experiements}
\subsection{Ablation with controlled computation budget}
As shown in Tab \ref{tab:method_comparison}, Numina + \ourtrain performs favorably against using \ourtrain or Numina alone. To show this gain is not simply achieved by doubling the computation available for fine-tuning. We control the total fine-tune budget the same for \ourtrain only, numina only and \ourtrain + numina. This results in approximately 6 epoch on \ourdataset or Numina or 3 epoch of training on \ourtrain + Numina. We show the curve of ACC on Math,\oureval, OlympiadBench w.r.t. training steps.
\begin{figure}[t]
\centering
\begin{minipage}{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figs/acc_curve/accuracy_aops.pdf}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figs/acc_curve/accuracy_math.pdf}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{figs/acc_curve/accuracy_olympiadbench.pdf}
\end{minipage}
%
\caption{Ablation study on accuracy with respect to training steps. Here, 18,000 steps approximately correspond to 6 epochs for \ourdataset and Numina, and 3 epochs for \ourdataset + Numina. We can see that \oureval + Numina consistenly improve as training goes on.}
\label{fig:ablation_train_budget}
\end{figure}
\new{\subsection{Rewriting model ablation}}
\new{
We use Qwen 2.5 72B to rewrite the solutions, and then we fine-tune smaller models on our dataset. This may raise the question of whether the effectiveness of our dataset would be limited by the capabilities of its rewriting model. To show the effectiveness of our dataset, we use a Qwen 2.5 1.5B to rewrite the solutions and then fine-tune DeepSeek-Math 7B-instruct on the dataset. Table \ref{tab:ablate_qwen_15_rewriting} shows the performance of the original DeepSeek-Math, the performance of Qwen2.5-1.5B, and the performance of fine-tuned DeepSeek on Qwen2.5-1B-rewritten solutions. As shown by the results, the fine-tuned version outperforms both models, which shows that our dataset can improve the reasoning capability beyond its rewriting model solution.
}
\begin{table}[t]
\centering
\caption{\new{Performance comparison of original DeepSeek-Math, Qwen2.5-1.5B, and DeepSeek-Math fine-tuned on solutions rewritten by Qwen2.5-1.5B. The fine-tuned DeepSeek-Math significantly outperforms both the original model and the rewriting model, demonstrating that our dataset enhances reasoning capabilities beyond the limitations of its rewriting model.}}
\resizebox{\textwidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Model & AIME24 & AMC23 & Olympiad Bench & Math & AoPS24 & Omni Math \\
\midrule
\midrule
Deekseek-Math-7b-Ins & \textbf{1/30} & 8/40 & 14.5 & 47.1 & 11.7 & 12.3 \\
Qwen2.5-1.5b-Ins & 0/30 & 9/40 & 21.3 & 55.0 & 16.7 & 16.8 \\
Deepseek-Math-7b-Ins (fine-tuned) & \textbf{1/30} & \textbf{13/40} & \textbf{22.7} & \textbf{61.0} & \textbf{19.4} & \textbf{19.2} \\
\bottomrule
\end{tabular}
}
\label{tab:ablate_qwen_15_rewriting}
\end{table}
\section{AIME and AMC results on the main models}
Results of fine-tuning on AIME 2024 and AMC 2024 are present in table \ref{tab:method_comparison_competitions}.
\begin{table}[t]
\centering
\caption{Performance comparison of different models on AIME24 and AMC23 benchmarks. Bold values indicate the highest scores for individual datasets.}
\vspace{0.15cm}
\begin{tabular}{c|c|cc}
\toprule
Model & SFT Dataset & AIME24 & AMC23 \\
\midrule
\midrule
& No SFT & \textbf{1/30} & 8/40 \\
Deepseek-Math & Numina & 0/30 & 12/40 \\
7b-Ins & AoPS-Ins & \new{\textbf{1/30}} & \new{\textbf{15/40}} \\
\cmidrule{2-4}
& Numina+AoPS-Ins & {\textbf{2/30}} & {11/40} \\
\midrule
& No SFT & 0/30 & \textbf{16/40} \\
Mathstral & Numina & 0/30 & 15/40 \\
7B & AoPS-Ins & {0/30} & {12/40} \\
\cmidrule{2-4}
& Numina+AoPS-Ins & {0/30} & {\textbf{16/40}} \\
\midrule
& No SFT & \textbf{2/30} & \textbf{11/40} \\
Llama-3.2 & Numina & 1/30 & 6/40 \\
3B-Ins & AoPS-Ins & \textbf{2/30} & \textbf{11/40} \\
\cmidrule{2-4}
& Numina+AoPS-Ins & 0/30 & \textbf{12/40} \\
\midrule
& No SFT & 0/30 & 5/40 \\
Llama-3.2 & Numina & 0/30 & \textbf{6/40} \\
1B-Ins & AoPS-Ins & {0/30} & {\textbf{6/40}} \\
\cmidrule{2-4}
& Numina+AoPS-Ins & {0/30} & {\textbf{9/40}} \\
\bottomrule
\end{tabular}
\label{tab:method_comparison_competitions}
\end{table}
\begin{table}[t]
\centering
\caption{Comparison of fine-tuning DeepSeek-Math-7b-Instruct on DART-Math-Hard~\cite{tong2024dart} vs. AoPS-Instruct on various benchmarks.}
\begin{tabular}{l|c|c|c|c|c}
\toprule
Dataset & Size & LiveAoPSBench & Olympiad Bench & Omni Math & MATH \\
\midrule
\midrule
DART-MATH-Hard & $585$K & 14.4 & 21.8 & 15.4 & 52.5 \\
AoPS-Instruct & $647$K & \textbf{19.0} & \textbf{24.3} & \textbf{17.8} & \textbf{58.8} \\
\bottomrule
\end{tabular}
\label{tab:ablate_aops_vs_dartmath}
\end{table}
\section{Use of AoPS as a Data Source}
Concurrent to our work, both Numina~\citep{numina_math_datasets} and Omni-math~\citep{omni-math} also use AoPS as their data source. Different from us, Numina only includes data from the contest page with 30K questions \footnote{\url{https://artofproblemsolving.com/community/c13_contest_collections}}, while we utilize all the $1.07$ available posts on this forum. Furthermore, Omni-math~\citep{omni-math} includes only 4428 evaluation questions from all timestamps, while we include the \emph{most recent} problems posted in 2024, as well as a \emph{large-scale} training set.
\section{Prompts}
We provide the Prompts used in our pipeline in Figures \ref{fig:prompt_classify}, \ref{fig:prompt_parse}, and \ref{fig:prompt_rewrite}.
\input{secs/tabs/prompt_classify}
\input{secs/tabs/prompt_parse}
\input{secs/tabs/prompt_rewrite}
\section{Dataset Examples}
We provide further examples of our dataset and its rewritten solutions in Figures \ref{fig:example_rewritten2}, \ref{fig:example_rewritten3}, \ref{fig:example_rewritten_wrong}, \ref{fig:example_rewritten_wrong_2}. \ref{fig:example_rewritten_qwen_vs_llama}, and \ref{fig:qwen_generated_solution}.
\input{secs/tabs/example_rewritten_2}
\input{secs/tabs/example_rewritten_3}
\input{secs/tabs/example_rewritten_wrong}
\input{secs/tabs/example_rewritten_wrong_2}
\input{secs/tabs/example_rewritten_qwen_vs_llama}
\input{secs/tabs/qwen_entirely_sol_generation_example}
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
\begin{figure}[!t]
\centering
\textbf{Example 1}
\begin{tcolorbox}[colback=red!5!white, colframe=red!75!black, width=\textwidth]
\begin{small}
\textbf{Question:}
In a triangle, $ABC$, Angle $BAC = 90^{\circ}$; $AD$ is the altitude from $A$ onto $BC$. Draw $DE$ perpendicular to $AC$ and $DF$ perpendicular to $AB$. Suppose $AB = 15$ and $BC = 25$. Then the length of $EF$ is?
\vspace{3mm}
\textbf{Raw Post:}
Because the triangle is a right triangle, so by the Pythagorean Theorem, the length of AC is 25² - 15² = 20. The area of ABC is AB * AC / 2 = 15 * 20 / 2. But it can also be represented by 25 * AD / 2. Putting them together we get 15 * 20 / 2 = 25 * AD / 2. So AD = 15 * 20 / 25 = 12. Because DE perpendicular to AC and DF perpendicular to AB, AEDF is a rectangle, which means that EF = AD, EF = 12. The answer is A.
\vspace{3mm}
\textbf{Voted Answer}:
12
\end{small}
\tcblower
\textbf{Is the Voted answer consistent with answer in raw post?}
Human Annotator 1: \cmark
Human Annotator 2: \xmark
\end{tcolorbox}
\vspace{5mm} %
%
%
%
\textbf{Example 2}
\begin{tcolorbox}[colback=red!5!white, colframe=red!75!black, width=\textwidth]
\begin{small}
\textbf{Question:}
For a positive integer k we write $(1 + x) (1 + 2 x) (1 + 3 x )......(1 + k x ) = a_0 + a_1 x + a_2 x^2 +.......+ a_k x^k$ where $ a_0 , ... a_k $ are the coefficients of the polynomial. Find the sum of all the digits of smallest possible value of k if $ a_0 + a_1 + a_2 + ...... a_(k-1) $ is divisible by 2005.
\vspace{3mm}
\textbf{Raw Post:}\\
$f(x)=(1+x)(1+2x)\ldots(1+kx)=a_0+a_1x+\ldots+a_kx^k$\\ $a_0+a_1+\ldots+a_{k-1}=f(1)-a_k$\\ $a_k=1 \cdot 2\cdot 3\ldots k=k!$\\ $f(1)=2 \cdot 3 \cdot 4 \ldots (1+k)=(k+1)!$\\ $2005 \mid (k+1)!-k! \Longrightarrow 2005 \mid k\cdot k!$\\ $2005=5 \cdot 401$\\ $k \ge 401$
\vspace{3mm}
\textbf{Voted Answer}:
5
\end{small}
\tcblower
\textbf{Is the Voted answer consistent with answer in raw post?}
Human Annotator 1: \xmark
Human Annotator 2: \cmark
\end{tcolorbox}
\caption{We highlight two examples of annotation inconsistencies caused by human annotators:
1. Example 1: Annotator 2 failed to recognize that the answer is explicitly stated in the raw post.
2. Example 2: The raw post does not directly provide the final answer. Annotator 1 was unable to reason that 4 + 0 + 1 = 5 constitutes the correct solution.}
\label{fig:example_inconsistent_annotation}
\end{figure}
\begin{table}[tbp]
\centering
%
\small
%
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|rrrrrrrrrrrrr}
\toprule
Model & 2023 & Jan & Feb & Mar & Apr & May & Jun & Jul & Aug & Sep & Oct & Nov & Dec \\
\midrule
Count & 5216 & 483 & 388 & 444 & 415 & 472 & 412 & 396 & 505 & 381 & 409 & 404 & 507 \\
\midrule
DeepSeek-Coder-V2-Lite-it & 22.45 & 23.40 & 19.33 & 22.97 & 22.17 & 22.67 & 21.12 & 23.99 & 25.74 & 22.83 & 23.72 & 22.28 & 18.93 \\
DeepSeek-Math-7b-rl & 15.38 & 18.22 & 15.46 & 17.57 & 13.49 & 14.41 & 12.86 & 15.66 & 14.65 & 15.75 & 15.89 & 17.08 & 13.61 \\
Internlm2-Math-plus-20b & 18.23 & 18.84 & 16.75 & 17.79 & 18.31 & 20.55 & 17.72 & 16.41 & 18.42 & 17.06 & 21.27 & 18.81 & 16.57 \\
Internlm2-Math-plus-7b & 17.10 & 17.81 & 16.49 & 19.82 & 17.11 & 18.01 & 15.05 & 19.95 & 16.24 & 13.91 & 18.58 & 17.08 & 15.19 \\
Mathstral-7B-v0.1 & 15.91 & 17.18 & 14.95 & 15.54 & 14.22 & 16.31 & 16.75 & 16.67 & 18.22 & 13.91 & 14.18 & 18.32 & 14.20 \\
NuminaMath-72B-CoT & 26.15 & 29.40 & 25.77 & 23.87 & 25.30 & 29.87 & 22.57 & 25.51 & 25.54 & 24.41 & 27.87 & 25.50 & 27.02 \\
NuminaMath-7B-CoT & 17.29 & 18.43 & 14.95 & 18.24 & 15.90 & 19.70 & 15.53 & 17.68 & 17.43 & 15.22 & 18.83 & 19.80 & 15.38 \\
Qwen2-Math-1.5B-it & 29.06 & 31.47 & 26.55 & 31.31 & 31.08 & 28.81 & 26.21 & 26.77 & 29.90 & 27.82 & 29.58 & 32.43 & 26.43 \\
Qwen2-Math-72B-it & 37.96 & 41.41 & 36.34 & 37.39 & 38.31 & 38.35 & 37.14 & 37.37 & 38.61 & 34.91 & 40.10 & 39.60 & 35.50 \\
Qwen2-Math-7B-it & 33.07 & 33.13 & 35.31 & 33.33 & 31.33 & 36.86 & 28.64 & 35.10 & 33.27 & 30.45 & 32.52 & 36.39 & 30.57 \\
Qwen2.5-Math-1.5B-it & 34.72 & 36.02 & 32.47 & 35.81 & 34.94 & 37.50 & 32.04 & 31.31 & 36.04 & 33.86 & 36.67 & 35.15 & 33.73 \\
Qwen2.5-Math-72B-it & 42.04 & 44.31 & 38.40 & 41.22 & 41.93 & 45.55 & 44.17 & 40.15 & 42.18 & 35.43 & 46.21 & 43.32 & 40.43 \\
Qwen2.5-Math-7B-it & 34.87 & 35.82 & 31.70 & 36.26 & 35.42 & 37.71 & 29.61 & 33.59 & 37.43 & 32.28 & 34.72 & 37.87 & 34.52 \\
\midrule
Llama-3.2-1B-it & 6.75 & 6.83 & 5.93 & 4.73 & 5.54 & 9.53 & 5.10 & 8.08 & 6.73 & 8.66 & 7.09 & 7.92 & 5.13 \\
Llama-3.2-3B-it & 13.77 & 14.49 & 13.40 & 12.16 & 13.98 & 14.19 & 13.11 & 16.92 & 12.08 & 14.44 & 15.16 & 13.61 & 12.43 \\
Llama-3.1-8B-it & 14.03 & 15.53 & 13.66 & 14.86 & 13.73 & 13.56 & 11.65 & 15.15 & 16.04 & 12.86 & 14.91 & 14.60 & 11.64 \\
Gemma-2-27b-it & 13.80 & 11.39 & 13.92 & 15.77 & 13.01 & 15.04 & 10.68 & 14.90 & 15.64 & 12.86 & 14.91 & 14.36 & 13.02 \\
Gemma-2-9b-it & 12.42 & 10.97 & 10.05 & 13.51 & 12.53 & 13.35 & 11.17 & 13.38 & 15.05 & 10.76 & 13.45 & 13.61 & 10.85 \\
\bottomrule
\end{tabular}
}
\vspace{1em} %
%
\resizebox{0.8\textwidth}{!}{%
\begin{tabular}{l|rrrrrrrrr}
\toprule
Model & 2024 & Jan & Feb & Mar & Apr & May & Jun & Jul & Aug\\
\midrule
Count & 3863 & 634 & 527 & 614 & 503 & 511 & 380 & 363 & 331 \\
\midrule
DeepSeek-Coder-V2-Lite-it & 20.86 & 18.14 & 22.77 & 25.41 & 22.47 & 18.00 & 17.89 & 21.21 & 19.64 \\
DeepSeek-Math-7b-rl & 13.64 & 14.04 & 17.08 & 15.31 & 15.71 & 10.37 & 10.53 & 13.50 & 9.97 \\
Internlm2-Math-plus-20b & 16.93 & 14.51 & 22.01 & 17.59 & 17.50 & 14.48 & 14.74 & 18.73 & 15.71 \\
Internlm2-Math-plus-7b & 14.81 & 11.83 & 18.60 & 17.59 & 16.50 & 11.15 & 17.11 & 13.77 & 10.88 \\
Mathstral-7B-v0.1 & 14.29 & 11.04 & 17.65 & 17.43 & 16.50 & 10.96 & 12.11 & 12.95 & 15.11 \\
NuminaMath-72B-CoT & 24.95 & 22.08 & 28.65 & 28.66 & 23.46 & 23.48 & 22.37 & 27.55 & 22.36 \\
NuminaMath-7B-CoT & 16.13 & 13.56 & 20.87 & 17.10 & 19.68 & 13.89 & 13.42 & 16.25 & 12.69 \\
Qwen2-Math-1.5B-it & 26.84 & 25.24 & 29.79 & 30.46 & 27.24 & 22.90 & 25.79 & 28.37 & 23.56 \\
Qwen2-Math-72B-it & 36.68 & 34.38 & 37.57 & 41.69 & 36.98 & 36.59 & 35.79 & 36.91 & 30.82 \\
Qwen2-Math-7B-it & 30.05 & 28.08 & 31.69 & 34.20 & 31.41 & 27.98 & 29.74 & 30.85 & 24.17 \\
Qwen2.5-Math-1.5B-it & 32.57 & 28.71 & 33.59 & 39.25 & 31.21 & 31.70 & 30.00 & 34.99 & 29.61 \\
Qwen2.5-Math-72B-it & 40.56 & 40.54 & 41.56 & 45.11 & 40.16 & 37.96 & 40.53 & 42.42 & 33.23 \\
Qwen2.5-Math-7B-it & 34.04 & 32.18 & 35.48 & 41.37 & 32.41 & 32.09 & 32.11 & 34.99 & 28.40 \\
\midrule
Llama-3.2-1B-it & 5.80 & 3.63 & 7.40 & 6.84 & 6.16 & 4.70 & 5.53 & 7.16 & 5.44 \\
Llama-3.2-3B-it & 11.75 & 8.68 & 13.85 & 12.87 & 14.12 & 10.18 & 13.42 & 9.92 & 11.18 \\
Llama-3.1-8B-it & 12.71 & 10.57 & 16.13 & 14.17 & 11.33 & 12.33 & 12.63 & 12.67 & 11.48 \\
Gemma-2-27b-it & 12.87 & 10.57 & 16.13 & 13.03 & 13.52 & 11.94 & 11.58 & 14.05 & 12.39 \\
Gemma-2-9b-it & 10.67 & 8.68 & 11.95 & 13.36 & 10.93 & 8.02 & 8.16 & 14.05 & 10.27 \\
\bottomrule
\end{tabular}
}
\caption{Accuracy per Month for Different Models}
\label{tab:combined_tables}
\end{table}
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
\begin{table}[tbp]
\caption{Accuracy per Difficulty for Different Models: The difficulty labels are for general education background of the problem and do not reflect the exact difficulty of the problem.}
\label{tab:accuracy_type2}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|ccccccccccccc}
\toprule
Model & Overall & Middle School & High School & College & High School & Others \\&&&&&Olympiads \\
\midrule
Count & 3863 & 286 & 1349 & 314 & 889 & 1025 \\
\midrule
DeepSeek-Coder-V2-Lite-it & 20.86 & 24.48 & 19.79 & 22.93 & 17.21 & 23.80 \\
DeepSeek-Math-7b-rl & 13.64 & 22.73 & 12.23 & 14.97 & 8.55 & 16.98 \\
Internlm2-Math-plus-20b & 16.93 & 24.83 & 15.64 & 19.11 & 12.71 & 19.41 \\
Internlm2-Math-plus-7b & 14.81 & 20.63 & 13.94 & 17.20 & 9.67 & 18.05 \\
Mathstral-7B-v0.1 & 14.29 & 19.23 & 12.90 & 15.61 & 10.24 & 17.85 \\
NuminaMath-72B-CoT & 24.95 & 33.57 & 25.28 & 26.75 & 18.45 & 27.22 \\
NuminaMath-7B-CoT & 16.13 & 18.18 & 15.12 & 16.24 & 12.60 & 19.90 \\
Qwen2-Math-1.5B-it & 26.84 & 32.17 & 26.17 & 28.98 & 22.61 & 29.27 \\
Qwen2-Math-72B-it & 36.68 & 43.36 & 38.62 & 42.04 & 28.01 & 38.15 \\
Qwen2-Math-7B-it & 30.05 & 37.06 & 30.17 & 31.21 & 24.86 & 32.10 \\
Qwen2.5-Math-1.5B-it & 32.57 & 38.81 & 33.65 & 31.85 & 28.91 & 32.78 \\
Qwen2.5-Math-72B-it & 40.56 & 48.25 & 42.70 & 45.86 & 32.62 & 40.88 \\
Qwen2.5-Math-7B-it & 34.04 & 42.66 & 34.84 & 36.62 & 27.11 & 35.80 \\
\midrule
Llama-3.2-1B-it & 5.80 & 10.14 & 4.30 & 5.41 & 3.71 & 8.49 \\
Llama-3.2-3B-it & 11.75 & 18.53 & 10.23 & 9.55 & 8.10 & 15.71 \\
Llama-3.1-8B-it & 12.71 & 17.48 & 10.60 & 15.29 & 7.54 & 17.85 \\
Gemma-2-27b-it & 12.87 & 20.63 & 11.86 & 13.38 & 7.65 & 16.39 \\
Gemma-2-9b-it & 10.67 & 15.38 & 8.30 & 12.74 & 7.65 & 14.44 \\
\bottomrule
\end{tabular}
}
\end{table}
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
\begin{table}[tbp]
\caption{Accuracy per Answer Type for Different Models: As not all answers can be easily verified, we divide the answers into different types to facilitate more accurate comparison and more convenient observation of the structural distribution of the dataset.}
\label{tab:accuracy_type3}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|rrrrrrrrrrrrr}
\toprule
Model & Overall & equation & expression & list & numeric-dec & numeric-int & numeric-irr & others \\
\midrule
Count & 3863 & 296 & 950 & 195 & 57 & 2114 & 176 & 75 \\
\midrule
DeepSeek-Coder-V2-Lite-it & 20.86 & 18.24 & 16.00 & 20.00 & 15.79 & 24.36 & 11.93 & 21.33 \\
DeepSeek-Math-7b-rl & 13.64 & 11.15 & 10.42 & 10.77 & 8.77 & 16.18 & 6.25 & 21.33 \\
Internlm2-Math-plus-20b & 16.93 & 14.53 & 12.74 & 13.33 & 14.04 & 20.20 & 8.52 & 18.67 \\
Internlm2-Math-plus-7b & 14.81 & 11.49 & 9.26 & 12.31 & 12.28 & 18.31 & 8.52 & 22.67 \\
Mathstral-7B-v0.1 & 14.29 & 14.53 & 10.74 & 6.67 & 12.28 & 16.65 & 10.80 & 21.33 \\
NuminaMath-72B-CoT & 24.95 & 19.59 & 20.21 & 16.92 & 24.56 & 28.71 & 20.45 & 32.00 \\
NuminaMath-7B-CoT & 16.13 & 14.86 & 12.11 & 11.28 & 12.28 & 18.78 & 12.50 & 21.33 \\
Qwen2-Math-1.5B-it & 26.84 & 23.31 & 22.11 & 28.72 & 15.79 & 29.52 & 25.57 & 32.00 \\
Qwen2-Math-72B-it & 36.68 & 27.70 & 30.84 & 36.41 & 26.32 & 41.15 & 31.82 & 40.00 \\
Qwen2-Math-7B-it & 30.05 & 23.99 & 26.21 & 27.69 & 19.30 & 33.30 & 25.00 & 37.33 \\
Qwen2.5-Math-1.5B-it & 32.57 & 25.34 & 28.74 & 32.82 & 22.81 & 35.86 & 22.73 & 46.67 \\
Qwen2.5-Math-72B-it & 40.56 & 31.42 & 38.32 & 41.54 & 31.58 & 43.19 & 36.36 & 45.33 \\
Qwen2.5-Math-7B-it & 34.04 & 28.72 & 30.21 & 33.85 & 24.56 & 36.90 & 30.11 & 40.00 \\
\midrule
Llama-3.2-1B-it & 5.80 & 2.70 & 3.79 & 2.56 & 5.26 & 7.66 & 3.41 & 5.33 \\
Llama-3.2-3B-it & 11.75 & 7.77 & 8.32 & 7.18 & 7.02 & 14.71 & 6.25 & 16.00 \\
Llama-3.1-8B-it & 12.71 & 4.05 & 9.58 & 10.77 & 17.54 & 15.33 & 9.66 & 21.33 \\
Gemma-2-27b-it & 12.87 & 7.77 & 10.11 & 9.74 & 14.04 & 15.28 & 9.09 & 16.00 \\
Gemma-2-9b-it & 10.67 & 7.09 & 7.89 & 8.72 & 14.04 & 12.54 & 7.95 & 16.00 \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{figure}[tbp]
\begin{tcolorbox}[colback=yellow!5!white,colframe=yellow!75!black]
\begin{small}
\textbf{Insturction:}
You are given an online Math QA post. Your task is to identify whether the post asked is a concrete mathematical question, note that this means it shouldn't be an abstract general question related to math, and output the result as \verb|\|boxed\{0\} for no and \verb|\|boxed\{1\} for yes.
A few examples are provided below:
\\
\\
\textbf{\textit{Few shot examples...}}
\\
\\
Now, your task is to provide output for the following post:
\\
\\
Post: \textbf{\textit{Post 1}}
\end{small}
\tcblower
\begin{small}
\textbf{Example Classify result:}
\\
\\
\verb|\|boxed\{\textbf{\textit{0/1}}\}
\end{small}
\end{tcolorbox}
\caption{
Prompt for the Topic Filtering part in Fig \ref{fig:dataset_pipeline}.
}
\label{fig:prompt_classify}
\end{figure}
\begin{figure}[tbp]
\begin{tcolorbox}[colback=yellow!5!white,colframe=yellow!75!black]
\begin{small}
\textbf{Insturction:}
You are given an online Math QA forum where each user post in each topic is in the format ``post i by user j: [post i text]''. Each user may reply to other users by quoting their post. Your task is to find all potential answers within the follow-up discussion, and output them in a structured json format.
Your output json must have one ``answers'' key containing the list of answers. Each answer must have two keys: a ``user'' key to identify the user who posted the solution, a ``post number'' to identify which post number the answer originates from. Do not add any additional information to the question or answers. In case the dialogou does not contain any mathematical question, or there are no valid answers, leave the ``answer'' key as an empty list. Before you output your JSON answer, provide a short summary of the discussion. A few examples are proided below:
\\
\\
\textit{\textbf{Few shot examples...}}
\\
\\
Now, your task is to provide JSON output for the following Topic:
\\
\\
\textit{\textbf{post 1 by user1: ...}}
\\
\textit{\textbf{post 2 by user2: ...}}
\end{small}
\tcblower
\begin{small}
\textbf{Example Parse result:}
\{
\hspace{2em}``question'': ``\textbf{\textit{Question from Post 1}}'',
\hspace{2em}``answers'':
[
\hspace{2em}\hspace{2em}\{
\hspace{2em}\hspace{2em}\hspace{2em}``user'': ``\textbf{\textit{User2}}'',
\hspace{2em}\hspace{2em}\hspace{2em}``post number'': \textbf{\textit{2}}
\hspace{2em}\hspace{2em}\},
\hspace{2em}\hspace{2em}\{
\hspace{2em}\hspace{2em}\hspace{2em}``use'': "\textbf{\textit{User4}}",
\hspace{2em}\hspace{2em}\hspace{2em}``post number'': \textbf{\textit{5}}
\hspace{2em}\hspace{2em}\}
\hspace{2em}]
\}
\end{small}
\end{tcolorbox}
\caption{
Prompt for the QA extraction part in Fig \ref{fig:dataset_pipeline}.
}
\label{fig:prompt_parse}
\end{figure}
\begin{figure}[tbp]
\begin{tcolorbox}[colback=yellow!5!white,colframe=yellow!75!black]
\begin{small}
\textbf{Insturction:}
You are given a solution to a mathematical question. Your task is to re-write the solution into a step-by-step solution with itemized steps(1..., 2...., 3....). You should re-write the solution in a formal and clean way, without missing any details. Make sure to include all the necessary steps and explanations. Do not include any irrelevant information in the answer. Do not add any additional information to the solution. Always surround mathematical questions with \$ symbols for LaTeX formatting. If the question contains a final number, make sure to put the number in a \\\\boxed{} for LaTeX formatting.
\\
\\
Question: \textbf{\textit{Question}}
\\
\\
Solution: \textbf{\textit{Raw Answer}}
\\
\\
Now provide the formalized re-written answer. DO NOT include the question in your answer. Only respond with the re-written solution.
\end{small}
\tcblower
\begin{small}
\textbf{Example Rewrite result:}
\textbf{\textit{Rewritten solutions}}
\end{small}
\end{tcolorbox}
\caption{
Prompt for the Solution Rewriting Part in Fig \ref{fig:dataset_pipeline}.
}
\label{fig:prompt_rewrite}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Find all pairs $(k, l)$ of positive integers such that $\frac{k^{l}}{l^{k}}=\frac{k!}{l!}$
\vspace{3mm}
\textbf{User Response 1:}
W.l.o.g we may assume $k \geq l$.
If $k \geq l+2$, then $k-1$ divides the RHS, thus the LHS, thus $k^{l}$, but $k , k-1$ are coprime, thus this is only possible if $k=2$, which would imply $l=0$, contradiction.
So the cases $k=l , l+1$ are left.
If $k=l$, then we always have equality.
If $k=l+1$, then the we need ${\frac{k^{k-1}}{(k-1)^{k}}=k \iff k^{k-2}=(k-1)^{k}}$.
Again $k-1$ is supposed to divide a power of $k$, thus we remain with $k=2$, so with $l=1$, clearly being a solution.
\vspace{3mm}
\textbf{User Response 2:}
If (k,l) solution, then (l,k) is solution too. Therefore we can find solutions $k\ge l$. If l=1 we have $k=k!\Longrightarrow k=1 \ or \ k=2$.
If l=2 we have $k^{2}=2^{k-1}k!$ solution is k=2.
If $l\ge 3$ then $\frac{k^{l}}{l^{k}}\le 1$ and $\frac{k!}{l!}\ge 1$, therefore solution is l=k.
All solutions $(k,l)=(1,2)(2,1),(n,n),n\in N$.
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution 1:}
1. Without loss of generality, assume $k \geq l$.
2. Consider the case $k \geq l+2$:
\begin{itemize}
\item The right-hand side (RHS) of the equation $\frac{k^{l}}{l^{k}}=\frac{k!}{l!}$ is divisible by $k-1$.
\item Therefore, the left-hand side (LHS) must also be divisible by $k-1$.
\item Since $k$ and $k-1$ are coprime, $k-1$ must divide $k^{l}$.
\item This is only possible if $k=2$, which would imply $l=0$, leading to a contradiction.
\end{itemize}
3. Therefore, the only possible cases are $k=l$ and $k=l+1$.
4. If $k=l$:
\begin{itemize}
\item The equation $\frac{k^{l}}{l^{k}}=\frac{k!}{l!}$ simplifies to $\frac{k^{k}}{k^{k}}=\frac{k!}{k!}$, which is always true.
\end{itemize}
5. If $k=l+1$:
\begin{itemize}
\item The equation becomes $\frac{(l+1)^{l}}{l^{l+1}}=\frac{(l+1)!}{l!}$.
\item Simplifying, we get $\frac{(l+1)^{l}}{l^{l+1}}=l+1$.
\item This further simplifies to $(l+1)^{l-1}=l^{l+1}$.
\item Again, $l$ and $l+1$ are coprime, so $l$ must divide a power of $l+1$.
\item The only solution is $l=1$, which implies $k=2$.
\end{itemize}
6. Therefore, the pairs $(k, l)$ that satisfy the equation are $\boxed{(k, l) = (k, k)}$ and $\boxed{(k, l) = (2, 1)}$.
\vspace{3mm}
\textbf{Qwen Rewritten Solution 2:}
1. If $(k, l)$ is a solution, then $(l, k)$ is also a solution. Therefore, we can assume without loss of generality that $k \ge l$.
2. Consider the case when $l = 1$. The equation becomes $\frac{k^1}{1^k} = \frac{k!}{1!}$, which simplifies to $k = k!$. The only positive integer solutions to this equation are $k = 1$ and $k = 2$.
3. Consider the case when $l = 2$. The equation becomes $\frac{k^2}{2^k} = \frac{k!}{2!}$. This simplifies to $k^2 = 2^{k-1} k!$. The only positive integer solution to this equation is $k = 2$.
4. Consider the case when $l \ge 3$. We have $\frac{k^l}{l^k} \le 1$ and $\frac{k!}{l!} \ge 1$. For these two inequalities to hold simultaneously, it must be that $k = l$.
5. Therefore, the solutions are $(k, l) = (1, 2)$, $(2, 1)$, and $(n, n)$ for any positive integer $n$.
\end{small}
\end{tcolorbox}
\caption{
An example illustrating two distinct rewritten solutions for a single posted problem, where each user has proposed a unique approach to solving it.
}
\label{fig:example_rewritten2}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Let $ABC$ be an equilateral triangle. On the plane $(ABC)$ rise the perpendiculars $AA'$ and $BB'$ on the same side of the plane, so that $AA' = AB$ and $BB' =\frac12 AB$. Determine the measure the angle between the planes $(ABC)$ and $(A'B'C')$.
\vspace{3mm}
\textbf{User Response:}
Denote $A'B'\cap AB=D$.Then $BD=BA=BC$ so $\angle ACD=90^\circ$. Thus the plane $AA'C$ is perpendicular to $CD$ and the angle between $A'B'C$ and $ABC$ is equal to $\angle A'CA=45^\circ$.
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution:}
1. Denote the intersection of $A'B'$ and $AB$ as $D$.
2. Since $BD = BA = BC$, it follows that $\triangle BDA$ and $\triangle BDC$ are isosceles triangles.
3. Therefore, $\angle ACD = 90^\circ$.
4. The plane $AA'C$ is perpendicular to $CD$.
5. The angle between the planes $(A'B'C')$ and $(ABC)$ is equal to $\angle A'CA$.
6. Since $AA' = AB$ and $BB' = \frac{1}{2} AB$, and given the symmetry and properties of the equilateral triangle, $\angle A'CA = 45^\circ$.
7. Thus, the measure of the angle between the planes $(ABC)$ and $(A'B'C')$ is $\boxed{45^\circ}$.
\end{small}
\end{tcolorbox}
\caption{
An example of a rewritten geometry solution for a posted problem.
}
\label{fig:example_rewritten3}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Solve the equation in parameter $p$: $x^3+px^2+(p-1+\frac{1}{p-1})x+1=0$
\vspace{3mm}
\textbf{User Response:}
Note that $p\ne1$. Then $x=1-p$ works, you can get the other roots (when they exist) by quadratic formula.
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution:}
1. First, observe that the parameter $p$ must satisfy $p \ne 1$ to avoid division by zero in the term $\left(\frac{1}{p-1}\right)$.
2. Next, consider the given equation:
\[
x^3 + px^2 + \left(p - 1 + \frac{1}{p-1}\right)x + 1 = 0
\]
3. We test the value $x = 1 - p$ to see if it is a root of the equation. Substitute $x = 1 - p$ into the equation:
\[
(1 - p)^3 + p(1 - p)^2 + \left(p - 1 + \frac{1}{p-1}\right)(1 - p) + 1 = 0
\]
4. Simplify each term:
\[
(1 - p)^3 = 1 - 3p + 3p^2 - p^3
\]
\[
p(1 - p)^2 = p(1 - 2p + p^2) = p - 2p^2 + p^3
\]
\[
\left(p - 1 + \frac{1}{p-1}\right)(1 - p) = (p - 1)(1 - p) + \frac{1 - p}{p-1} = (p - 1)(1 - p) - 1 = -p^2 + 2p - 1 - 1 = -p^2 + 2p - 2
\]
5. Combine all the simplified terms:
\[
1 - 3p + 3p^2 - p^3 + p - 2p^2 + p^3 - p^2 + 2p - 2 + 1 = 0
\]
6. Simplify the expression:
\[
(1 - 3p + 3p^2 - p^3) + (p - 2p^2 + p^3) + (-p^2 + 2p - 2) + 1 = 0
\]
\[
1 - 3p + 3p^2 - p^3 + p - 2p^2 + p^3 - p^2 + 2p - 2 + 1 = 0
\]
\[
1 - 3p + 3p^2 - p^3 + p - 2p^2 + p^3 - p^2 + 2p - 2 + 1 = 0
\]
\[
0 = 0
\]
7. Since the equation holds true, $x = 1 - p$ is indeed a root of the equation.
8. To find the other roots, we can perform polynomial division of the original cubic polynomial by $(x - (1 - p))$ to obtain a quadratic polynomial.
9. The quadratic polynomial can then be solved using the quadratic formula:
\[
x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}
\]
where $a$, $b$, and $c$ are the coefficients of the quadratic polynomial obtained from the division.
10. The final roots of the equation are:
\[
x = 1 - p \quad \text{and the roots of the quadratic polynomial obtained from the division.}
\]
11. The solution to the equation is:
\[
\boxed{x = 1 - p}
\]
\end{small}
\end{tcolorbox}
\caption{
An example of an incomplete solution where the author left finding the other two roots of the equation to the reader. While the language model has adequately and correctly expanded the solution of the author, two more solutions of the equation are not mentioned (neither by the author nor the LLM): $-\frac{p - 1 + \sqrt{5 - 6p + p^2}}{2(p - 1)}$ and $\frac{1 - p + \sqrt{5 - 6p + p^2}}{2(p - 1)}$.
}
\label{fig:example_rewritten_wrong}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Find a 5 × 5 checkerboard with a square removed that cannot be tiled using right triominoes(L shaped). Prove that such a tiling does not exist for this board.
\vspace{3mm}
\textbf{User Response:}
Alright, then colour
01010
11111
01010
11111
01010
The sum of the squares covered by each L is at least $2$, so if we place $8$ L's we get a total of at least $16$, meaning that every $1$ must be covered. Hence removing a $1$ makes the covering impossible.
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution:}
1. Consider a 5 × 5 checkerboard with the following coloring pattern:
\[
\begin{array}{ccccc}
0 & 1 & 0 & 1 & 0 \\
1 & 1 & 1 & 1 & 1 \\
0 & 1 & 0 & 1 & 0 \\
1 & 1 & 1 & 1 & 1 \\
0 & 1 & 0 & 1 & 0 \\
\end{array}
\]
2. Each right triomino (L-shaped) covers exactly 3 squares. When placed on the board, the sum of the numbers on the squares covered by each L-shaped triomino is at least 2. This is because each L-shaped triomino must cover at least one square with a 1 and two squares with 0s or 1s.
3. If we place 8 L-shaped triominoes on the board, the total sum of the numbers on the squares covered by these 8 triominoes is at least \(8 \times 2 = 16\).
4. The total number of squares on the 5 × 5 board is 25. Removing one square leaves 24 squares. Since each L-shaped triomino covers 3 squares, 8 L-shaped triominoes will cover exactly 24 squares.
5. For the 8 L-shaped triominoes to cover the 24 squares, the sum of the numbers on these 24 squares must be at least 16. Given the coloring pattern, the sum of the numbers on all 25 squares is 13 (since there are 12 squares with 0 and 13 squares with 1).
6. If a square with a 1 is removed, the sum of the numbers on the remaining 24 squares is \(13 - 1 = 12\).
7. Since the sum of the numbers on the squares covered by the 8 L-shaped triominoes must be at least 16, and the sum of the numbers on the remaining 24 squares is only 12, it is impossible to cover the board with 8 L-shaped triominoes if a square with a 1 is removed.
8. Therefore, a 5 × 5 checkerboard with a square removed that cannot be tiled using right triominoes (L-shaped) is one where a square with a 1 is removed. The tiling is impossible in this case.
\(\boxed{1}\)
\end{small}
\end{tcolorbox}
\caption{
A failure example for rewriting the solution. While the Qwen model gets the overall logic of the proof correct, it produces mistakes in counting the number of ``ones'' on the checkboard and provides incorrect intermediate steps.
}
\label{fig:example_rewritten_wrong_2}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Let $x$ be a real number such that $\log_2{x}+\log_4{5}+\log_8{7x}=\log_{\sqrt{2}}{x^{5/6}}$. Find $x$.
\textbf{User Response:}
Turn everything into $\log_2$ of something. Here is the algebra:
\begin{align*}
\log_2{x}+\log_{2^2}{5}+\log_{2^3}{7x}=\log_{2^{1/2}}{x^{5/6}}, \\
\log_2 x + \log _2 \sqrt{5} + \log_2 \sqrt[3]{7x} = \log_2 \sqrt[3]{x^5}, \\
\log_2 (x \cdot \sqrt{5} \cdot \sqrt[3]{7x}) = \log _2 (\sqrt[3]{x^5}), \\
x \cdot \sqrt{5} \cdot \sqrt[3]{7x} = \sqrt[3]{x^5} \implies x = \boxed{35\sqrt{5}}.
\end{align*}
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Rewritten Solution:}
1. Convert all logarithms to base 2:
$
\log_2{x} + \log_{2^2}{5} + \log_{2^3}{7x} = \log_{2^{1/2}}{x^{5/6}}
$
2. Simplify the logarithms using the change of base formula:
$
\log_2{x} + \frac{1}{2}\log_2{5} + \frac{1}{3}\log_2{7x} = \frac{1}{\frac{1}{2}}\log_2{x^{5/6}}
$
3. Simplify the right-hand side:
$
\log_2{x} + \frac{1}{2}\log_2{5} + \frac{1}{3}\log_2{7x} = 2\log_2{x^{5/6}}
$
4. Use the property of logarithms to combine the terms on the left-hand side:
$
\log_2{x} + \log_2{5^{1/2}} + \log_2{(7x)^{1/3}} = \log_2{x^{5/3}}
$
5. Combine the logarithms on the left-hand side:
$
\log_2 \left( x \cdot 5^{1/2} \cdot (7x)^{1/3} \right) = \log_2 \left( x^{5/3} \right)
$
6. Since the logarithms are equal, the arguments must be equal:
$
x \cdot 5^{1/2} \cdot (7x)^{1/3} = x^{5/3}
$
7. Simplify the equation:
$
x \cdot \sqrt{5} \cdot \sqrt[3]{7x} = x^{5/3}
$
8. Divide both sides by \(x\) (assuming \(x \neq 0\)):
$
\sqrt{5} \cdot \sqrt[3]{7x} = x^{2/3}
$
9. Raise both sides to the power of 3 to eliminate the cube root:
$
(\sqrt{5} \cdot \sqrt[3]{7x})^3 = (x^{2/3})^3
$
10. Simplify the equation:
$
5^{3/2} \cdot 7x = x^2
$
11. Simplify \(5^{3/2}\):
$
5^{3/2} = 5 \sqrt{5}
$
12. Substitute back into the equation:
$
5 \sqrt{5} \cdot 7x = x^2
$
13. Simplify the equation:
$
35 \sqrt{5} x = x^2
$
14. Rearrange the equation to form a quadratic equation:
$
x^2 - 35 \sqrt{5} x = 0
$
15. Factor out \(x\):
$
x(x - 35 \sqrt{5}) = 0
$
16. Solve for \(x\):
$
x = 0 \quad \text{or} \quad x = 35 \sqrt{5}
$
17. Since \(x = 0\) is not a valid solution (as it would make the original logarithms undefined), the solution is:
$
x = \boxed{35\sqrt{5}}
$
\vspace{3mm}
\textbf{Llama Rewritten Solution:}
To solve the given equation, we first need to express all logarithms in terms of $\log_2$. This can be achieved by applying the change of base formula for logarithms.
Step 1: Express $\log_4{5}$ and $\log_8{7x}$ in terms of $\log_2$.
Using the change of base formula, we have:
$\log_4{5} = \frac{\log_2{5}}{\log_2{4}} = \frac{\log_2{5}}{\log_2{2^2}} = \frac{\log_2{5}}{2} = \log_2{\sqrt{5}}$
and
$\log_8{7x} = \frac{\log_2{7x}}{\log_2{8}} = \frac{\log_2{7x}}{\log_2{2^3}} = \frac{\log_2{7x}}{3} = \log_2{\sqrt[3]{7x}}$.
Step 2: Express $\log_{\sqrt{2}}{x^{5/6}}$ in terms of $\log_2$.
Using the change of base formula, we have:
$\log_{\sqrt{2}}{x^{5/6}} = \frac{\log_2{x^{5/6}}}{\log_2{\sqrt{2}}} = \frac{\log_2{x^{5/6}}}{\log_2{2^{1/2}}} = \frac{\log_2{x^{5/6}}}{\frac{1}{2}} = 2\log_2{x^{5/6}} = \log_2{(x^{5/6})^2} = \log_2{x^{5/3}} = \log_2{\sqrt[3]{x^5}}$.
Step 3: Substitute the expressions from Steps 1 and 2 into the original equation.
$\log_2{x} + \log_2{\sqrt{5}} + \log_2{\sqrt[3]{7x}} = \log_2{\sqrt[3]{x^5}}$
Step 4: Combine the logarithms on the left-hand side using the product rule for logarithms.
$\log_2{(x \cdot \sqrt{5} \cdot \sqrt[3]{7x})} = \log_2{\sqrt[3]{x^5}}$
Step 5: Since the logarithms are equal, the arguments of the logarithms must be equal. Therefore, we have:
$x \cdot \sqrt{5} \cdot \sqrt[3]{7x} = \sqrt[3]{x^5}$.
Step 6: Solve for $x$.
$x \cdot \sqrt{5} \cdot \sqrt[3]{7x} = \sqrt[3]{x^5}$
$\implies x = \boxed{35\sqrt{5}}$.
\end{small}
\end{tcolorbox}
\caption{
Comparison of Qwen 2.5 72B and Llama 3.1 70B on rewriting solutions. Overall, the Qwen model shows a higher level of detail in expanding equations compared to Llama.
}
\label{fig:example_rewritten_qwen_vs_llama}
\end{figure}
%
\begin{figure}
\begin{tcolorbox}[colback=blue!5!white,colframe=blue!75!black]
\begin{small}
\textbf{Question:}
Solve the equation in parameter $p$: $x^3+px^2+(p-1+\frac{1}{p-1})x+1=0$
\end{small}
\tcblower
\begin{small}
\textbf{Qwen Generated Solution:}
To solve the equation \(x^3 + px^2 + \left(p - 1 + \frac{1}{p-1}\right)x + 1 = 0\) in the parameter \(p\), we will follow a systematic approach. Let's denote the polynomial by \(f(x)\):
$ f(x) = x^3 + px^2 + \left(p - 1 + \frac{1}{p-1}\right)x + 1 $
\#\#\# Step 1: Check for Rational Roots
We will use the Rational Root Theorem to check for possible rational roots. The Rational Root Theorem states that any rational root, expressed in its lowest terms \(\frac{p}{q}\), must have \(p\) as a factor of the constant term (1) and \(q\) as a factor of the leading coefficient (1). Therefore, the possible rational roots are \(\pm 1\).
\#\#\# Step 2: Test \(x = 1\)
Substitute \(x = 1\) into the polynomial:
$ f(1) = 1^3 + p \cdot 1^2 + \left(p - 1 + \frac{1}{p-1}\right) \cdot 1 + 1 $
$ f(1) = 1 + p + p - 1 + \frac{1}{p-1} + 1 $
$ f(1) = 2p + 1 + \frac{1}{p-1} $
For \(x = 1\) to be a root, \(f(1) = 0\):
$ 2p + 1 + \frac{1}{p-1} = 0 $
\#\#\# Step 3: Solve the Equation \(2p + 1 + \frac{1}{p-1} = 0\)
Multiply through by \(p-1\) to clear the fraction:
$ (2p + 1)(p-1) + 1 = 0 $
$ 2p^2 - 2p + p - 1 + 1 = 0 $
$ 2p^2 - p = 0 $
Factor out \(p\):
$ p(2p - 1) = 0 $
This gives us two solutions:
$ p = 0 \quad \text{or} \quad p = \frac{1}{2} $
\#\#\# Step 4: Verify the Solutions
We need to verify if these values of \(p\) indeed make \(x = 1\) a root of the original polynomial.
\#\#\#\# For \(p = 0\):
Substitute \(p = 0\) into the polynomial:
$ f(x) = x^3 + 0 \cdot x^2 + \left(0 - 1 + \frac{1}{0-1}\right)x + 1 $
$ f(x) = x^3 - 2x + 1 $
Check \(f(1)\):
$ f(1) = 1^3 - 2 \cdot 1 + 1 = 1 - 2 + 1 = 0 $
So, \(x = 1\) is a root when \(p = 0\).
\#\#\#\# For \(p = \frac{1}{2}\):
Substitute \(p = \frac{1}{2}\) into the polynomial:
$ f(x) = x^3 + \frac{1}{2}x^2 + \left(\frac{1}{2} - 1 + \frac{1}{\frac{1}{2}-1}\right)x + 1 $
$ f(x) = x^3 + \frac{1}{2}x^2 + \left(\frac{1}{2} - 1 - 2\right)x + 1 $
$ f(x) = x^3 + \frac{1}{2}x^2 - \frac{3}{2}x + 1 $
Check \(f(1)\):
$ f(1) = 1^3 + \frac{1}{2} \cdot 1^2 - \frac{3}{2} \cdot 1 + 1 $
$ f(1) = 1 + \frac{1}{2} - \frac{3}{2} + 1 $
$ f(1) = 1 + 1 - 1 = 1 $
So, \(x = 1\) is not a root when \(p = \frac{1}{2}\).
\#\#\# Conclusion
The only value of \(p\) for which \(x = 1\) is a root of the polynomial is \(p = 0\).
Thus, the solution to the equation is:
$
\boxed{0}
$
\end{small}
\end{tcolorbox}
\caption{
An example of letting Qwen generate the entire solution without the user input (from Figure \ref{fig:example_rewritten_wrong}). In this case, the model completely fails to respond correctly, misinterpreting the question and taking an incorrect approach to solving the problem. This shows the cruciality of rewriting solutions (rather than generating entire solutions from stronger models).
}
\label{fig:qwen_generated_solution}
\end{figure}
\begin{thebibliography}{46}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Achiam et~al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman,
Almeida, Altenschmidt, Altman, Anadkat, et~al.]{gpt4}
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.~L.,
Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et~al.
\newblock Gpt-4 technical report.
\newblock \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem[AIMO(2023)]{AIMO2023}
AIMO.
\newblock The aimo prize.
\newblock \url{https://aimoprize.org}, November 2023.
\bibitem[AOPS(2023)]{amc23}
AOPS.
\newblock 2023 amc 12a, and 12b problems.
\newblock
\url{https://artofproblemsolving.com/wiki/index.php/2023_AMC_12A_Problems},
\url{https://artofproblemsolving.com/wiki/index.php/2023_AMC_12B_Problems},
2023.
\bibitem[AOPS(2024)]{aime24}
AOPS.
\newblock 2024 aime community page.
\newblock \url{https://artofproblemsolving.com/community/c3370201_2024_aime},
2024.
\bibitem[Azerbayev et~al.(2024)Azerbayev, Schoelkopf, Paster, Santos, McAleer,
Jiang, Deng, Biderman, and Welleck]{azerbayev2024llemma}
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M.~D., McAleer, S.~M.,
Jiang, A.~Q., Deng, J., Biderman, S., and Welleck, S.
\newblock Llemma: An open language model for mathematics.
\newblock In \emph{The Twelfth International Conference on Learning
Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=4WnqRR915j}.
\bibitem[Bubeck et~al.(2023)Bubeck, Chandrasekaran, Eldan, Gehrke, Horvitz,
Kamar, Lee, Lee, Li, Lundberg, et~al.]{bubeck2023sparks}
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E.,
Lee, P., Lee, Y.~T., Li, Y., Lundberg, S., et~al.
\newblock Sparks of artificial general intelligence: Early experiments with
gpt-4.
\newblock \emph{arXiv preprint arXiv:2303.12712}, 2023.
\bibitem[Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser,
Plappert, Tworek, Hilton, Nakano, Hesse, and Schulman]{gsm8k_dataset}
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert,
M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.
\newblock Training verifiers to solve math word problems.
\newblock \emph{arXiv preprint arXiv:2110.14168}, 2021.
\bibitem[DeepSeek-AI(2025)]{deepseek_r1}
DeepSeek-AI.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via
reinforcement learning, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.12948}.
\bibitem[Ding et~al.(2024)Ding, Shi, Liang, Li, Zhu, and
Zhang]{ding2024unleashing}
Ding, Y., Shi, X., Liang, X., Li, J., Zhu, Q., and Zhang, M.
\newblock Unleashing reasoning capability of llms via scalable question
synthesis from scratch.
\newblock \emph{arXiv preprint arXiv:2410.18693}, 2024.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman,
Mathur, Schelten, Yang, Fan, et~al.]{llama3}
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A.,
Mathur, A., Schelten, A., Yang, A., Fan, A., et~al.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint arXiv:2407.21783x qx q}, 2024.
\bibitem[Gao et~al.(2024)Gao, Song, Yang, Cai, Miao, Dong, Li, Ma, Chen, Xu,
et~al.]{omni-math}
Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen,
L., Xu, R., et~al.
\newblock Omni-math: A universal olympiad level mathematic benchmark for large
language models.
\newblock \emph{arXiv preprint arXiv:2410.07985}, 2024.
\bibitem[He et~al.(2024)He, Luo, Bai, Hu, Thai, Shen, Hu, Han, Huang, Zhang,
Liu, Qi, Liu, and Sun]{olympiadbench}
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z., Shen, J., Hu, J., Han, X., Huang,
Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M.
\newblock {O}lympiad{B}ench: A challenging benchmark for promoting {AGI} with
olympiad-level bilingual multimodal scientific problems.
\newblock In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), \emph{Proceedings
of the 62nd Annual Meeting of the Association for Computational Linguistics
(Volume 1: Long Papers)}, pp.\ 3828--3850, Bangkok, Thailand, August 2024.
Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.acl-long.211}.
\newblock URL \url{https://aclanthology.org/2024.acl-long.211}.
\bibitem[Hendrycks et~al.(2021{\natexlab{a}})Hendrycks, Burns, Basart, Zou,
Mazeika, Song, and Steinhardt]{Mmlu}
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and
Steinhardt, J.
\newblock Measuring massive multitask language understanding.
\newblock \emph{Proceedings of the International Conference on Learning
Representations (ICLR)}, 2021{\natexlab{a}}.
\bibitem[Hendrycks et~al.(2021{\natexlab{b}})Hendrycks, Burns, Kadavath, Arora,
Basart, Tang, Song, and Steinhardt]{math_dataset}
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song,
D., and Steinhardt, J.
\newblock Measuring mathematical problem solving with the math dataset.
\newblock \emph{NeurIPS}, 2021{\natexlab{b}}.
\bibitem[Jain et~al.(2024)Jain, Han, Gu, Li, Yan, Zhang, Wang, Solar-Lezama,
Sen, and Stoica]{livecodebench}
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S.,
Solar-Lezama, A., Sen, K., and Stoica, I.
\newblock Livecodebench: Holistic and contamination free evaluation of large
language models for code.
\newblock \emph{arXiv preprint arXiv:2403.07974}, 2024.
\bibitem[Jiang et~al.(2023)Jiang, Sablayrolles, Mensch, Bamford, Chaplot,
Casas, Bressand, Lengyel, Lample, Saulnier, et~al.]{mistral}
Jiang, A.~Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.~S., Casas,
D. d.~l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et~al.
\newblock Mistral 7b.
\newblock \emph{arXiv preprint arXiv:2310.06825}, 2023.
\bibitem[Koncel-Kedziorski et~al.(2016)Koncel-Kedziorski, Roy, Amini, Kushman,
and Hajishirzi]{Mawps}
Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H.
\newblock Mawps: A math word problem repository.
\newblock In \emph{Proceedings of the 2016 conference of the north american
chapter of the association for computational linguistics: human language
technologies}, pp.\ 1152--1157, 2016.
\bibitem[Lewkowycz et~al.(2022)Lewkowycz, Andreassen, Dohan, Dyer, Michalewski,
Ramasesh, Slone, Anil, Schlag, Gutman-Solo, Wu, Neyshabur, Gur-Ari, and
Misra]{Minerva}
Lewkowycz, A., Andreassen, A.~J., Dohan, D., Dyer, E., Michalewski, H.,
Ramasesh, V.~V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y.,
Neyshabur, B., Gur-Ari, G., and Misra, V.
\newblock Solving quantitative reasoning problems with language models.
\newblock In Oh, A.~H., Agarwal, A., Belgrave, D., and Cho, K. (eds.),
\emph{Advances in Neural Information Processing Systems}, 2022.
\newblock URL \url{https://openreview.net/forum?id=IFXTZERXdM7}.
\bibitem[Li et~al.(2024)Li, Beeching, Tunstall, Lipkin, Soletskyi, Costa~Huang,
Rasul, Yu, Jiang, Shen, Qin, Dong, Zhou, Fleureau, Lample, and
Polu]{numina_math_datasets}
Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Costa~Huang, S.,
Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L.,
Fleureau, Y., Lample, G., and Polu, S.
\newblock Numinamath.
\newblock
\url{https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf},
2024.
\bibitem[Li et~al.(2023)Li, Zhang, Dubois, Taori, Gulrajani, Guestrin, Liang,
and Hashimoto]{alpaca_eval}
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang,
P., and Hashimoto, T.~B.
\newblock Alpacaeval: An automatic evaluator of instruction-following models.
\newblock \url{https://github.com/tatsu-lab/alpaca_eval}, 5 2023.
\bibitem[Li et~al.(2022)Li, Choi, Chung, Kushman, Schrittwieser, Leblond,
Eccles, Keeling, Gimeno, Dal~Lago, et~al.]{alpha-code}
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R.,
Eccles, T., Keeling, J., Gimeno, F., Dal~Lago, A., et~al.
\newblock Competition-level code generation with alphacode.
\newblock \emph{Science}, 378\penalty0 (6624):\penalty0 1092--1097, 2022.
\bibitem[Meurer et~al.(2017)Meurer, Smith, Paprocki, \v{C}ert\'{i}k, Kirpichev,
Rocklin, Kumar, Ivanov, Moore, Singh, Rathnayake, Vig, Granger, Muller,
Bonazzi, Gupta, Vats, Johansson, Pedregosa, Curry, Terrel, Rou\v{c}ka, Saboo,
Fernando, Kulal, Cimrman, and Scopatz]{sympy}
Meurer, A., Smith, C.~P., Paprocki, M., \v{C}ert\'{i}k, O., Kirpichev, S.~B.,
Rocklin, M., Kumar, A., Ivanov, S., Moore, J.~K., Singh, S., Rathnayake, T.,
Vig, S., Granger, B.~E., Muller, R.~P., Bonazzi, F., Gupta, H., Vats, S.,
Johansson, F., Pedregosa, F., Curry, M.~J., Terrel, A.~R., Rou\v{c}ka, v.,
Saboo, A., Fernando, I., Kulal, S., Cimrman, R., and Scopatz, A.
\newblock Sympy: symbolic computing in python.
\newblock \emph{PeerJ Computer Science}, 3:\penalty0 e103, January 2017.
\newblock ISSN 2376-5992.
\newblock \doi{10.7717/peerj-cs.103}.
\newblock URL \url{https://doi.org/10.7717/peerj-cs.103}.
\bibitem[Miao et~al.(2020)Miao, Liang, and Su]{Asdiv}
Miao, S.-Y., Liang, C.-C., and Su, K.-Y.
\newblock A diverse corpus for evaluating and developing english math word
problem solvers.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pp.\ 975--984, 2020.
\bibitem[Mistral(2024)]{mathstral}
Mistral.
\newblock Mathstral blog.
\newblock \url{https://mistral.ai/news/mathstral/}, 2024.
\bibitem[Mitra et~al.(2024)Mitra, Khanpour, Rosset, and Awadallah]{orca-math}
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A.
\newblock Orca-math: Unlocking the potential of slms in grade school math.
\newblock \emph{arXiv preprint arXiv:2402.14830}, 2024.
\bibitem[Paster et~al.(2024)Paster, Santos, Azerbayev, and Ba]{openwebmath}
Paster, K., Santos, M.~D., Azerbayev, Z., and Ba, J.
\newblock Openwebmath: An open dataset of high-quality mathematical web text.
\newblock In \emph{The Twelfth International Conference on Learning
Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=jKHmjlpViu}.
\bibitem[Patel et~al.(2021)Patel, Bhattamishra, and Goyal]{Svamp}
Patel, A., Bhattamishra, S., and Goyal, N.
\newblock Are {NLP} models really able to solve simple math word problems?
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pp.\ 2080--2094, Online, June 2021. Association for
Computational Linguistics.
\newblock \doi{10.18653/v1/2021.naacl-main.168}.
\newblock URL \url{https://aclanthology.org/2021.naacl-main.168}.
\bibitem[Selsam et~al.(2019)Selsam, de~Moura, Buzzard, Barton, Liang, Loos, and
Wiedijk]{imo_grand_challenge}
Selsam, D., de~Moura, L., Buzzard, K., Barton, R., Liang, P., Loos, S., and
Wiedijk, F.
\newblock Imo grand challenge.
\newblock \url{https://imo-grand-challenge.github.io/}, 2019.
\bibitem[Shah et~al.(2024)Shah, Yu, Lyu, Park, Yu, He, Ke, Mozer, Bengio,
Arora, et~al.]{shah2024ai}
Shah, V., Yu, D., Lyu, K., Park, S., Yu, J., He, Y., Ke, N.~R., Mozer, M.,
Bengio, Y., Arora, S., et~al.
\newblock Ai-assisted generation of difficult math questions.
\newblock \emph{arXiv preprint arXiv:2407.21009}, 2024.
\bibitem[Shao et~al.(2024)Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu, and
Guo]{deepseek-math}
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and
Guo, D.
\newblock Deepseekmath: Pushing the limits of mathematical reasoning in open
language models.
\newblock \emph{arXiv preprint arXiv:2402.03300}, 2024.
\bibitem[Team et~al.(2024)Team, Riviere, Pathak, Sessa, Hardin, Bhupatiraju,
Hussenot, Mesnard, Shahriari, Ram{\'e}, et~al.]{gemma2}
Team, G., Riviere, M., Pathak, S., Sessa, P.~G., Hardin, C., Bhupatiraju, S.,
Hussenot, L., Mesnard, T., Shahriari, B., Ram{\'e}, A., et~al.
\newblock Gemma 2: Improving open language models at a practical size.
\newblock \emph{arXiv preprint arXiv:2408.00118}, 2024.
\bibitem[Team(2024)]{qwq-32b-preview}
Team, Q.
\newblock Qwq: Reflect deeply on the boundaries of the unknown, November 2024.
\newblock URL \url{https://qwenlm.github.io/blog/qwq-32b-preview/}.
\bibitem[Tong et~al.(2024)Tong, Zhang, Wang, Wu, and He]{tong2024dart}
Tong, Y., Zhang, X., Wang, R., Wu, R., and He, J.
\newblock Dart-math: Difficulty-aware rejection tuning for mathematical
problem-solving.
\newblock \emph{Advances in Neural Information Processing Systems},
37:\penalty0 7821--7846, 2024.
\bibitem[Toshniwal et~al.(2024)Toshniwal, Moshkov, Narenthiran, Gitman, Jia,
and Gitman]{openmathinstruct}
Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman,
I.
\newblock Openmathinstruct-1: A 1.8 million math instruction tuning dataset.
\newblock \emph{arXiv preprint arXiv:2402.10176}, 2024.
\bibitem[Trinh et~al.(2024)Trinh, Wu, Le, He, and
Luong]{AlphaGeometryTrinh2024}
Trinh, T., Wu, Y., Le, Q., He, H., and Luong, T.
\newblock Solving olympiad geometry without human demonstrations.
\newblock \emph{Nature}, 2024.
\newblock \doi{10.1038/s41586-023-06747-5}.
\bibitem[Yang et~al.(2024{\natexlab{a}})Yang, Yang, Hui, Zheng, Yu, Zhou, Li,
Li, Liu, Huang, et~al.]{qwen2}
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu,
D., Huang, F., et~al.
\newblock Qwen2 technical report.
\newblock \emph{arXiv preprint arXiv:2407.10671}, 2024{\natexlab{a}}.
\bibitem[Yang et~al.(2024{\natexlab{b}})Yang, Zhang, Hui, Gao, Yu, Li, Liu, Tu,
Zhou, Lin, et~al.]{qwen25}
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou,
J., Lin, J., et~al.
\newblock Qwen2. 5-math technical report: Toward mathematical expert model via
self-improvement.
\newblock \emph{arXiv preprint arXiv:2409.12122}, 2024{\natexlab{b}}.
\bibitem[Yang et~al.(2023)Yang, Chiang, Zheng, Gonzalez, and
Stoica]{yang2023rethinking}
Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J.~E., and Stoica, I.
\newblock Rethinking benchmark and contamination for language models with
rephrased samples.
\newblock \emph{arXiv preprint arXiv:2311.04850}, 2023.
\bibitem[Ying et~al.(2024)Ying, Zhang, Li, Zhou, Shao, Fei, Ma, Hong, Liu,
Wang, et~al.]{internlm2-math}
Ying, H., Zhang, S., Li, L., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J.,
Liu, K., Wang, Z., et~al.
\newblock Internlm-math: Open math large language models toward verifiable
reasoning.
\newblock \emph{arXiv preprint arXiv:2402.06332}, 2024.
\bibitem[Yu et~al.(2024)Yu, Jiang, Shi, YU, Liu, Zhang, Kwok, Li, Weller, and
Liu]{yu2024metamath}
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z.,
Weller, A., and Liu, W.
\newblock Metamath: Bootstrap your own mathematical questions for large
language models.
\newblock In \emph{The Twelfth International Conference on Learning
Representations}, 2024.
\newblock URL \url{https://openreview.net/forum?id=N8N0hgNDRt}.
\bibitem[Yue et~al.(2024)Yue, Zheng, Zhang, and Chen]{yue2024mammoth2}
Yue, X., Zheng, T., Zhang, G., and Chen, W.
\newblock Mammoth2: Scaling instructions from the web.
\newblock \emph{arXiv preprint arXiv:2405.03548}, 2024.
\bibitem[Zellers et~al.(2019)Zellers, Holtzman, Bisk, Farhadi, and
Choi]{Zellers2019HellaSwagCA}
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.
\newblock Hellaswag: Can a machine really finish your sentence?
\newblock In \emph{Annual Meeting of the Association for Computational
Linguistics}, 2019.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:159041722}.
\bibitem[Zhang et~al.(2023{\natexlab{a}})Zhang, Zhou, Wei, Zhao, Sha, Wang, and
Wen]{Carp}
Zhang, B., Zhou, K., Wei, X., Zhao, W.~X., Sha, J., Wang, S., and Wen, J.-R.
\newblock Evaluating and improving tool-augmented computation-intensive math
reasoning.
\newblock \emph{arXiv preprint arXiv:2306.02408}, 2023{\natexlab{a}}.
\bibitem[Zhang et~al.(2023{\natexlab{b}})Zhang, Li, Zong, Ying, He, and
Qiu]{gaokao_bench}
Zhang, X., Li, C., Zong, Y., Ying, Z., He, L., and Qiu, X.
\newblock Evaluating the performance of large language models on gaokao
benchmark.
\newblock 2023{\natexlab{b}}.
\bibitem[Zhu et~al.(2024)Zhu, Guo, Shao, Yang, Wang, Xu, Wu, Li, Gao, Ma,
et~al.]{deepseek-coder}
Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao,
H., Ma, S., et~al.
\newblock Deepseek-coder-v2: Breaking the barrier of closed-source models in
code intelligence.
\newblock \emph{arXiv preprint arXiv:2406.11931}, 2024.
\bibitem[Zhuo et~al.(2024)Zhuo, Vu, Chim, Hu, Yu, Widyasari, Yusuf, Zhan, He,
Paul, et~al.]{zhuo2024bigcodebench}
Zhuo, T.~Y., Vu, M.~C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I.
N.~B., Zhan, H., He, J., Paul, I., et~al.
\newblock Bigcodebench: Benchmarking code generation with diverse function
calls and complex instructions.
\newblock \emph{arXiv preprint arXiv:2406.15877}, 2024.
\end{thebibliography}