Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
Extracted source text · arXiv 2111.05364
%
% File emnlp2020.tex
%
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%% taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{mwpsurvey}
\usepackage{times}
\usepackage{latexsym}
\usepackage{graphicx}
\usepackage{amsmath}
\renewcommand{\UrlFont}{\ttfamily\small}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} % Enter the acl Paper ID here
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\newcommand\BibTeX{B\textsc{ib}\TeX}
\title{
Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities
}
\title{
Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
}
\author{Keyur Faldu \thanks{\hspace{0.1cm}Correspondence to k@embibe.com}\\
Embibe \\\And
Amit Sheth \\
University of\\ South Carolina \\\And
Prashant Kikani \\
Embibe \\\And
Manas Gaur \\
University of\\ South Carolina \\\And
Aditi Avasthi \\
Embibe \\
}
\date{}
\begin{document}
\maketitle
\begin{abstract}
\end{abstract}
Mathematical reasoning would be one of the next frontiers for artificial intelligence to make significant progress. The ongoing surge to solve math word problems (MWPs) and hence achieve better mathematical reasoning ability would continue to be a key line of research in the coming time. We inspect non-neural and neural methods to solve math word problems narrated in a natural language. We also highlight the ability of these methods to be generalizable, mathematically reasonable, interpretable, and explainable. Neural approaches dominate the current state of the art, and we survey them highlighting three strategies to MWP solving: (1) direct answer generation, (2) expression tree generation for inferring answers, and (3) template retrieval for answer computation. Moreover, we discuss technological approaches, review the evolution of intuitive design choices to solve MWPs and examine them for mathematical reasoning ability. We finally identify several gaps that warrant the need for external knowledge and knowledge-infused learning, among several other opportunities in solving MWPs.
\begin{figure}[h]
\centering
\includegraphics[scale=0.32]{Mathamatical_Reasoning.png}
\caption{An illustration of the mathematical reasoning process that involves understanding of natural language and visual narratives. Essentially, the process comprises extracting entities with explicit and implicit quantities building a self-consistent symbolic representation invoking mathematical laws, axioms, and symbolic rules.
}
\label{fig:math_reason}
\end{figure}
\section{Introduction}
The recent success of artificial intelligence over a wide range of natural language processing tasks like hypothesis entailment, question answering, summarization, translation, etc., has posed the question that \textit{how intelligent are these systems?}. The General Language Understanding Evaluation (GLUE) \cite{wang2018glue}, and SuperGLUE \cite{wang2019superglue} benchmarks have been created to assess their natural language understanding abilities. Soon enough, deep learning systems have outperformed human-level performance over these benchmarks. However it is not understood if such performance is attributed to underlying reasoning capabilities \cite{sap2020commonsense}\cite{storks2019commonsense}\cite{khashabi2019capabilities}. Discovering and learning salient heuristic patterns to produce a likely outcome is also much better than rule-based systems. When learning is achieved without reasoning, it will reduce to mimicking, which would not be generalizable \cite{bandura2008observational}. Reasoning ability is at the center of performing higher-order complex tasks. Mathematical reasoning is one of the niche abilities humans possess \cite{lithner2000mathematical} \cite{english2013mathematical}, it constitutes linguistic reasoning, visual reasoning, common sense reasoning, logical reasoning, numerical reasoning, and symbolic reasoning. Humans do not learn mathematical reasoning just based on experience and evidence, but it results from learning, inferring, and applying laws, axioms, and symbolic rules. Mathematical reasoning could be one of the most important benchmarks to test the progress made by artificial intelligence.
\textit{Mathematical reasoning ability involves understanding natural language and visual narratives, extracting explicit and implicit quantities using common sense and domain-specific knowledge, inferring a self-consistent symbolic universe, and exploiting mathematical laws, axioms, and symbolic rules} \cite{amini2019mathqa}\cite{saxton2019analysing}. The problems requiring mathematical reasoning ability pose a unique opportunity to build a system that requires collaborative interactions between neural systems and symbolic systems \cite{kahneman2011thinking}. AI models for natural languages understanding and visual understanding could be developed using fast or probabilistic systems, and corresponding mathematical laws, axioms, and rules could be executed using slow and deterministic systems. Math word problems (MWP) are natural language questions that require mathematical reasoning to solve them. \textit{MWP is the natural language narrative that contains a self-consistent universe where entities with quantitative information interact with each other under well-defined mathematical axioms, laws, and theorems. Answering MWP would require solving unknown quantities or sets of unknown quantities from known explicit or implicit quantities using semantic information, commonsense world knowledge, and mathematical domain knowledge.} Thus, MWPs can be of varying difficulty levels based on the complexity of their narratives, implicit world and domain knowledge needed to solve the problem, and the application of mathematical topics and their interactions. Figure \ref{fig:complexmwp} demonstrates couple of examples of relatively complex MWPs.
The ability to solve MWPs in an automated fashion could lead to many applications in the education domain for content creation and delivering learning outcomes \cite{donda2020framework} \cite{faldu2020adaptive}. For example, it could aid students’ learning journey by resolving their doubts on the fly; potentially solve the problem of content deficiency for practice assessments \cite{dhavala2020auto}. Particularly by complementing MWPs recommendation systems with AI-based solution generation system \cite{faldu2020system}.
The task of solving MWPs has a long history with consistent research attention. This article surveys the progress in solving MWPs, including datasets, approaches, and architectures, broadly divided into non-neural and neural techniques. We also analyze the mathematical reasoning ability of such methods.
\begin{figure}[h]
\centering
\includegraphics[scale=0.37]{MWP_Problem_Formulation.png}
\caption{Example of math word problem and its possible problem formulations}
\label{fig:mwp}
\end{figure}
\section{Background and Non-Neural Approaches}
The complexity of solving MWPs has been well understood for a long time, demanding logical systems representation and machine’s ability to apply such systems with natural language understanding and generative capabilities \cite{feigenbaum1963computers}. Before the surge of neural network-based approaches to solving MWPs, several non-neural techniques were explored since the 1960s, classified into rule-based or pattern matching, semantic parsing, statistical, and machine learning approaches. However, most non-neural methods focus on the template retrieval approach or predicting the answer using rules. These approaches are explained in the following sections.
\begin{figure*}[h]
\centering
\includegraphics[scale=0.31]{ComplexMWPs.png}
\caption{Example of relatively complex MWPs from dataset MathQA \cite{amini2019mathqa}, requiring understanding of specific words like umbra and penumbra, applying multiple math concepts, and accessing mathematical knowledge like math formula for area of circle, sphere, value of $\pi.$}
\label{fig:complexmwp}
\end{figure*}
\subsection{Rule-based or Pattern Matching Approaches}
Normalizing and transforming input narratives using heuristics, such as rules, regular expressions, or pattern matching, was the first attempt in solving MWPs \cite{feigenbaum1963computers} \cite{bobrow1964natural} \cite{charniak1968calculus}. Such approaches are very intuitive but could only handle limited scenarios, and hence, they lack the flexibility in handling simple variations of math word problems. Further, efforts have been made to develop schema, frame, or extract entities and quantities associated with it to apply rules deterministically. \cite{fletcher1985understanding} \cite{dellarosa1986computer} \cite{yuhui2010frame}. However, such rule-based and pattern matching algorithms would not be scalable to learn general-purpose mathematical reasoning ability.
\subsection{Semantic Parsing}
A semantic parsing-based system attempts to learn the semantic structure of the input problem and transforms it into mathematical expressions or a set of equations. It represents problems in object-oriented structure, much similar to text-SQL generation, following methods in natural language processing \cite{liguda2012modeling} \cite{koncel2015parsing}. Underlying tree dependencies of mathematical semantics are captured by context-free grammar rules, which is analogous to dependency parse trees from the perspective of mathematical semantics \cite{shi2015automatically}.
\subsection{Statistical and Machine Learning Approaches}
Prior works on Statistical and machine learning approaches developed automated methods that are rule-based and semantic parsing, which processes training data to derive a set of templates or learn the semantic structure from the input. They range from text classification of MWPs with equation templates, extracting and mapping quantities with equations, or entities extraction and classifying their relationships \cite{kushman2014learning}. These methods use machine learning algorithms like support vector machines, probabilistic models, or margin classifiers for concept prediction, equation prediction, or slot filling. They leverage sentence semantics or verb categorization to map mathematical operations with segments of an MWP \cite{hosseini2014learning} \cite{amnueypornsakul2014machine}. A further set of candidate hypotheses are reduced by handling noun slots and number slots separately \cite{zhou2015learn}. Few other techniques process narratives to fill concepts-specific slots first and derive equations from them using domain-specific rules \cite{mitra2016learning}\cite{roy2018mapping}.
\section{Mathematical Reasoning in Non-Neural Approaches}
Mathematical reasoning ability is the key to solving mathematical problems. There were few attempts made in this direction using reasoning-oriented sub-tasks like quantity entailment for semantic parsing \cite{roy2015reasoning}, comprehending numerical information, quantity alignment prediction for slot filling , handling quantity slots and noun slots separately, ignoring extraneous information present in narratives, etc \cite{bakman2007robust} \cite{mitra2016learning} \cite{roy2018mapping} \cite{zhou2015learn}.
Non-neural methods that consider rule-based, pattern matching, semantic parsing, and machine learning-based approaches are limited to specific areas of mathematics like addition-subtractions, arithmetic, linear algebra, calculus, or quadratic equations. The performance of such methods is constrained by the coverage and diversity of MWPs in the training corpus. The performance of such approaches could not scale on out-of-corpus MWPs mainly because of the relatively small-sized training corpus \cite{koncel2016mawps} \cite{kushman2014learning}. \cite{huang2016well} attempted these techniques on a large-scale dataset Dolphin18K, having 18000 annotated linear and non-linear MWPs. Non-neural methods performed much worse on diverse and large-scale datasets than their reported performance on their corresponding small and specific datasets, and their performance improved sub-linearly with more extensive training data \cite{huang2016well}. It made a solid case to build more generic systems with better reasoning ability.
\begin{table*}[hbt!]
\fontsize{10pt}{15pt}
\selectfont
\begin{tabular}{p{0.58\linewidth} | p{0.08\linewidth} | p{0.24\linewidth}}
\hline Addition-Subtraction Math Word Problems & Subtype & Semantic Slots \\
\hline
Sam’s dog had 8 puppies. He gave 2 to his friends. He now has 6 puppies . How many puppies did he have to start with? & Change & (i) start
(ii) end
(iii) gains
(iv) losses \\
\hline
Tom went to 4 hockey games this year, but missed 7 . He went to 9 games last year. How many hockey games did Tom go to in all ? &
Part-Whole &
(i) whole
(ii) parts \\
\hline
Bill has 9 marbles. Jim has 7 more marbles than Bill. How many marbles does Jim have? & Compare & (i) large quantity
(ii) small quantity
(iii) difference \\
\hline
\end{tabular}
\caption{Classifying math word problems into subtypes, and semantic slots required for solving MWPs in each subtypes.
}
\label{tab:expdomain}
\end{table*}
\subsection{Domain Knowledge or External Knowledge}
Classifying MWPs into subtypes and handling them using mathematical domain knowledge is another way to further push such systems' capabilities. It becomes easier to reason about a problem classified to a subtype as it reduces the candidate math laws, axioms, and symbolic rules and contextualizes semantics related to the specific area of math \cite{mitra2016learning} \cite{roy2018mapping}\cite{bakman2007robust} \cite{amnueypornsakul2014machine}. For example, addition and subtraction problems can be grouped in three classes of mathematical concepts, (i) change, (ii) part-whole and (iii) compare as shown in table \ref{tab:expdomain} \cite{mitra2016learning}. Semantic parsing techniques form rules to extract semantic information specific to these sub-types. Table \ref{tab:expdomain} illustrates subtypes of a problem and how each problem subtype would have different semantic slots to apply math rules. The concept of \textit{part-whole} has two slots, one for the whole that accepts a single variable and the other for its parts that accepts a set of variables of size at least two. The \textit{change} concept has four slots, namely start, end, gains, losses which respectively denote the original value of a variable, the final value of that variable, and the set of increments and decrements that happen to the original value of the variable. The \textit{comparison} concept has three slots, namely the large quantity, the small quantity, and their difference \cite{mitra2016learning}. Similarly, other such mutually exclusive groups were identified by researchers to categorize MWPs, like (i) join-separate, (ii) part-whole and (iii) compare \cite{amnueypornsakul2014machine}; (i) change, (ii) combine (iii) compare \cite{bakman2007robust}; or (i) transfer, (ii) dimensional analysis, (iii) part-whole relation, (iv) explicit math, \cite{roy2018mapping} etc.
Commonsense and linguistic knowledge could be useful to derive the semantic structure and represent the meaning of MWPs. \cite{briars1984integrated} investigates how commonsense knowledge could be helpful to solve complex problems narrating real-world situations. The knowledge that an object defined in the MWP is a member of both a set and its superset and whether subsets can be exchanged in the context of deriving an answer for an MWP. For example, the math word problem shown in figure \ref{fig:mwp} illustrates objects “orange” and “apple.” Both belong to a superset “fruits,” and they can be exchanged if the unknown quantity in the question is agnostic to the type of fruit. Dolphin language representation of MWPs requires extracting nodes and their relationships, where nodes could be constants, classes, or functions. Commonsense knowledge is used to classify entities sharing common semantic properties \cite{shi2015automatically}. Linguistic knowledge like verb sense and verb entailment could help efficiently semantic parsing of MWPs narratives to extract semantic slots and fill their values. Researchers have explored such external knowledge from WordNet \cite{hosseini2014learning} or E-HowNet \cite{chen2005extended} \cite{lin2015designing}.
\section{Neural Approaches}
The recent advancement in deep learning approaches has opened up new possibilities. There is a flurry of research that aims to apply neural networks to solve MWPs. Sequence to sequence architectures, transformers, graph neural networks, convolutions, and attention mechanisms are a few such architectures and techniques of neural approaches. To empower neural approaches, larger and diverse datasets have been also created \cite{amini2019mathqa}, \cite{wang2017deep}, \cite{zhang2020graph}, \cite{saxton2019analysing}, \cite{lample2019deep}. Also, the potential of neural approaches solving complex problems on calculus and integration has raised expectations on its future promise. These approaches could be further categorized based on their problem formulation, (i) to predict answers directly, (ii) generate intermediate math expressions, or (iii) retrieve a template. However, it is evident that neural networks are black boxes, and hard to interpret their functioning and explain their decision \cite{gaur2021semantics}. Attempts to interpret its functioning reveal how volatile its reasoning ability is, and they lack generalizability \cite{patel2021nlp}.
In the following sections, we aim to categorize the current state-of-the-art development. Then, we navigate problem formulations, datasets and analysis, approaches and architectural design choices, data augmentation methods, and interpretability and reasoning ability.
\begin{table*}[hbt!]
\fontsize{9pt}{13.5pt}
\selectfont
\begin{tabular}{p{0.25\linewidth} | p{0.23\linewidth} | p{0.07\linewidth} | p{0.06\linewidth}| p{0.18\linewidth} | p{0.06\linewidth}}
\hline Dataset & Mathematics Area & Language & Type & Annotation & Size \\
\hline
AI2 \cite{hosseini2014learning} & Arithmetic & English & Curated & Equation / Answer & 395 \\
IL \cite{roy2015reasoning} & Arithmetic & English & Curated & Equation / Answer & 562 \\
ALGES \cite{koncel2015parsing} & Arithmetic & English & Curated & Equation / Answer & 508 \\
AllArith \cite{roy2017unit} & Arithmetic & English & Derived & Equation / Answer & 831 \\
\hline
Alg514 \cite{kushman2014learning} & Algebraic (linear) & English & Curated & Equation/Answer & 514 \\
Dolphin1878 \cite{shi2015automatically} & Algebraic (linear) & English & Curated & Equation / Answer & 1,878 \\
DRAW \cite{upadhyay2015draw} & Algebraic (linear) & English & Curated & Equation / Answer / Template & 1,000 \\
Dolphin18K \cite{huang2016well} & Algebraic (linear, nonlinear) & English & Curated & Equation / Answer & 18,000 \\
\hline
MAWPS \cite{koncel2016mawps} & Arithmetic, Algebraic & English & Curated & Equation/answer & 3,320 \\
AQuA \cite{ling2017program} & Arithmetic, Algebraic (linear, nonlinear) & English & Curated & Rationale/MCQ-choices/Answer & 100,000 \\
MathQA \cite{amini2019mathqa} & Arithmetic, Algebraic & English & Derived & Equation/MCQ-choices/Answer & 37,000 \\
MATH \cite{hendrycks2021measuring} & Algebra,
Number Theory, Probability, Geometry, Calculus
& English & Curated & Step-by-step solution / Answer & 12500 \\
\hline
ASDiv \cite{miao2021diverse} & Arithmetic, Algebraic & English & Derived & Equation/Answer +
Grade/Problem-type & 2,305 \\
SVAMP \cite{patel2021nlp} & Arithmetic & English & Derived & Equation/Answer & 1,000 \\
\hline
Math23K \cite{wang2017deep} & Algebraic (linear) &
Chinese & Curated & Equation/Answer & 23,161 \\
HMWP \cite{qin2020semantically} & Algebraic (linear + nonlinear) & Chinese & Curated & Equation/Answer & 5,491 \\
Ape210K \cite{zhao2020ape210k} & Algebraic (linear) & Chinese & Curated & Equation/Answer & 210,488 \\
\hline
Deepmind Mathematics \cite{saxton2019analysing} & Algebra, Probability, Calculus & English & Synthetic & Answer & 2,000,000 \\
Integration \& Differentiation Synthetic Dataset \cite{lample2019deep} & Integration, Differentiation & English & Synthetic & Answer & 160,000,000 \\
AMPS \cite{hendrycks2021measuring} & Algebra, Calculus, Geometry, Statistics, Number Theory & English & Synthetic & Step-by-step solution / Answer & 5,000,0000 \\
\hline
\end{tabular}
\caption{Categorization of MWP Datasets available for training AI models based on (a) Mathematics Area, (b) Language, (c) Type, and (d) Annotation.}
\label{tab:MWPDatasets}
\end{table*}
\subsection{Problem Formulation}
We can categorize the problem formulation to solve MWPs in three different ways, (i) predicting the answer directly, (ii) generating a set of equations or mathematical expressions, and inferring answers from them by executing them, and (iii) retrieving the templates from a pool of templates derived from training data and augmenting numerical quantities to compute the answer. An example is shown in Figure \ref{fig:mwp}.
The first approach to predict the answers directly could demonstrate the inherent ability of the neural model to learn complex mathematical transformations \cite{csaji2001approximation}; however, the black-box nature of such models suffers from poor interpretability and explainability \cite{gaur2021semantics}. It has been shown that sequence to sequence (seq-to-seq) neural models could compute free form answers for complex problems with high accuracy when trained on large datasets \cite{saxton2019analysing}\cite{lample2019deep}. It has been analyzed how models behave when just numerical parameters are changed vs. mathematical concepts in the test set. \cite{lample2019deep} achieved near-perfect accuracy of 99\% mainly attributed to the learning ability of neural models over a huge dataset. \cite{ran2019numnet} proposes a numerically aware graph neural network to directly predict the type of answer and the actual answer.
The second approach deals with generating an expression tree and executing them to compute the answer. Such models have relatively better interpretability and explainability and generated expressions, which provides scaffolding to the reasoning ability of the model. Seq-to-seq neural models framing from LSTM, Transformers, GNN and Tree decoders have proven to useful \cite{wang2017deep} \cite{amini2019mathqa} \cite{xie2019goal} \cite{qin2020semantically} \cite{liang2021mwp}. The key challenge for methods in this approach is the need for expert annotated datasets, as they need expression trees as the labels for each problem in addition to its answer \cite{amini2019mathqa}\cite{wang2017deep}\cite{koncel2016mawps}.
The third approach derives template equations from training data, retrieves the most similar template, and substitutes numerical parameters. This approach suffers from generalization ability, as the set of templates are limited to the training set. Such methods would fail to solve diverse and out-of-corpus MWPs as they would not be able to retrieve the template needed to solve them. It was one of the popular statistical machine learning approach \cite{kushman2014learning} \cite{koncel2015parsing}, \cite{mitra2016learning}, but there are also few other neural approaches, which have studied its value for an ensemble setup or standalone model with larger training data \cite{wang2017deep} \cite{robaidek2018data}.
Generally, these neural approaches follow encoder-decoder architecture to predict expression trees or directly compute the answer, specifically when the answer could be an expression in itself. We further explain different variants of encoder-decoder architectures in section 4.3.
\subsection{Analyzing Datasets}
One of the current limitations of achieving performance similar to human-level mathematical intelligence is the availability of labeled datasets. Table \ref{tab:MWPDatasets} categorizes various MWPs datasets along with information like dataset size, language, type of dataset, mathematics area etc. Each dataset focus on specific areas of Mathematics, i.e., \textit{Arithmetic, Algebra, Calculus, etc}. AI2, IL, ALGES, ALLArith, SVAMP are examples of datasets for arithmetic MWPs. Datasets such as Alg514, Dolphin1878, DRAW, Dolphin18K, Math23K, HMWP, and Ape210K focus on Algebraic problems. MAWPS, AQuA, MathQA, and ASDiv are examples of datasets covering MWPs from \textit{Arithmetic and Algebra}. There have been efforts to create MWPs datasets for complex areas including \textit{Probability, Integration and Differentiation} \cite{saxton2019analysing} \cite{lample2019deep}. \ref{tab:MWPDatasets} covers examples of MWP datasets in English and Chinese languages.
MWPs datasets are either curated from educational resources, derived by processing other available datasets or synthetically created. For example., Alg514 is created by curating problems from algerba.com. Similarly, Dolphin1878 comprises of curated problems from both \textit{algebra.com} and \textit{answers.yahoo.com}. Derived datasets have been derived by processing existing datasets to address concerns related to training corpus size \cite{roy2017unit}, equation annotation \cite{amini2019mathqa}, lexical diversity \cite{miao2021diverse}, problem types \cite{roy2017unit} or adversarial reasoning ability \cite{patel2021nlp}. Synthetic datasets could be very large, but they lack generalization. They are ultimately driven from deterministic sets of templates or rules and combinations of them for specific math areas \cite{saxton2019analysing} \cite{lample2019deep} \cite{hendrycks2021measuring}. Synthetic datasets could also be used in pre-training neural models. Curated datasets are generally small in size, e.g., AI2, IL, Alg514, DRAW, MAWPS, etc., or suffer from inconsistencies, e.g., AQuA or MathQA. Neural models generating a set of equations or expression trees would need a dataset, where MWPs are labeled with a set of equations or expression trees. Many of these curated datasets would not have associated annotated expression trees; hence these datasets would require manual effort to annotate. If datasets are created by curating relevant webpages, it would also possibly need answer extraction from free-form explanations \cite{huang2016well}. Equations for MWPs could also be extracted from their solutions on webpages using pattern matching, but which could lead to some inconsistencies \cite{amini2019mathqa}. On the other hand, it is very costly to augment these datasets with expert annotations. There can be multiple possible ways to write a semantically unique equation, which demands equation normalization to make sure labels are normalized \cite{huang2018neural}.
There is an active interest among the research community to analyze and improve the quality of datasets. Datasets like AsDiv and SVAMP are derived to address the quality concerns by processing existing datasets. Lexically similar questions would lead the model to learn the heuristic patterns to predict the answer \cite{miao2021diverse}. Lexicon Diversity Score measures the lexical diversity of questions provided in the dataset, and datasets with better lexicon diversity score would be more reliable to test a model's performance \cite{miao2021diverse}. Also, adversarial approaches further extend such datasets with adversarial variants to test the mathematical reasoning ability \cite{patel2021nlp}.
\subsection{Encoder Decoder Architectures}
\begin{figure*}[h]
\centering
\includegraphics[scale=0.5]{MWP_Neural_Architecutres}
\caption{Popular Neural Architectures for Solving Math Word Problems}
\label{fig:neural_arch}
\end{figure*}
\cite{wang2017deep} proposed the first encoder-decoder architecture to generate equations or expression trees for MWPs. It used GRU based encoder and LSTM based decoder. Since then, researchers have actively sought to explore the optimal architecture to get better performance for predicting equations or expression trees on diverse datasets comprising complex MWPs. As datasets are limited in size, researchers kept iterating to find better architectures to learn semantic structures of math word problems well. These novel neural architectures can be grouped into four categories, \textbf{(1) seq-to-seq, (2) seq-to-tree, (3) transformer-to-tree, (4) graph-to-tree}.
Sequence-to-sequence architecture is the default choice of any natural language generative model to transform a natural language input to output. Output sequence is either free form answer expression, equations, or traversal sequence of expression trees \cite{griffith2021solving} \cite{amini2019mathqa}. \cite{wang2017deep} leverages GRU-based encoder and LSTM based decoder, \cite{amini2019mathqa} uses attention mechanisms over RNNs, and conditions decoder with category of MWPs. For larger synthetic datasets \cite{lample2019deep} \cite{saxton2019analysing}, there isn't a need for semantically rich hybrid architectures as seq-to-seq models could achieve good performance. Transformers outperforms attentional LSTMs on larger datasets \cite{saxton2019analysing}, mainly attributed to more computations with the same numbers of parameters, internal memory of transformers which is pre-disposed to mathematical objects like quantities, and better gradient propagation with shallower architectures. There has been an effort to model situations narrated in MWPs using agents, attributes, and relations and train a seq-to-seq model to convert relation text to equations \cite{hong2021smart}.
Sequence-to-tree, transformer-to-tree, and Graph-to-tree architectures attempt to incorporate the tree structure of the output mathematical equations or expressions into it. The decoder in such models is a tree decoder, where it generates the next token by taking hidden states of its parent and sibling and its subtree (if any) into the account \cite{xie2019goal}\cite{liu2019tree}\cite{qin2020semantically}\cite{liang2021mwp} \cite{wu2021math}. Attention-based architectures leverage attention mechanisms to attend only parents and siblings \cite{qin2020semantically}\cite{liang2021mwp}. Operators would be parent nodes, and operands would be their children nodes, which would recursively expand further. Encoders in such architectures varied from GRU/LSTM based \cite{xie2019goal} or transformer-based \cite{liang2021mwp}. As tree decoders help model learning the tree semantics out output expressions, such models perform better than the seq-to-seq model on datasets with limited size. The expected output could be represented as a computational tree, which is then traversed and evaluated to compute the answer.
Graph-to-tree architecture further exploits the graph semantics present in MWPs \cite{ran2019numnet} \cite{zhang2020graph} \cite{li2019modeling}. Graph semantics captures the relationships between different entities, which can be thought of as relationships among numerical quantities, their description, and unknown quantities. The goal-driven tree-structured model is designed to generate expressions using computational goals inferred from an MWP, which uses graph transformer network as the encoder to incorporate quantity comparison graph and quantity cell graph \cite{zhang2020graph}. It leverages self-attention blocks for incorporating these instances of graphs. The numerical reasoning module attempts to learn comparative relations between numerical quantities by connecting them in a graph. Encoder representations are appended with representations learned by numerical reasoning module to improve the performance \cite{ran2019numnet}. An MWP can be segmented into a quantity spans and a question span. Specific self-attention head blocks which could just attend quantity span, question span, or both could be used to learn graph relationships between quantities and question present in an MWP \cite{li2019modeling}.
\subsection{Design Choices}
In this section, we aim to highlight several design decisions for the above architectures. First, the seq-to-seq model suffers from generating spurious numbers or predicting numbers at the wrong position. Copy and alignment mechanisms can be used to avoid this problem \cite{huang2018neural}. Second, it has become a common practice to keep decoder vocabulary limited to numbers, constants, operators, and other helpful information present in the question \cite{xie2019goal} \cite{zhang2020graph}.
There are multiple ways of generating output expressions, and maximum likelihood expectations would compromise learning if predicted, and expected expressions are different but semantically the same. Reinforcement learning would solve this problem by rewarding the learning strategy based on the final answer \cite{huang2018neural}.
Tree-based decoders have received lots of attention because of their ability to invoke tree relationships present in mathematical expressions. Tree decoders attend both parents and siblings to generate the next token. Bottom-up representation of subtree of a sibling could further help to derive better outcomes \cite{qin2020semantically}. For efficient implementation of tree decoder, the stack data structure can be used to store and retrieve hidden representations of parent and sibling \cite{liu2019tree}. Tree regularization transforms encoder representation and subtree representations and minimizes their L2 distance as a regularization term of the loss function \cite{qin2020semantically}.
Graph-based encoder aims to learn different types of relationships among the constituents of MWPs. There are several attempts to construct different types of graphs. \cite{ran2019numnet} inserts a numerical reasoning module that uses numerically aware graph neural networks, where two types of edge less-than-equal-to and greater-than connect numerical quantities present in the question. A situation model for algebra story problem SMART aims to build a graph using attributed grammar, which connects nodes with its attributes using relationships extracted from the problem \cite{hong2021smart}. Self-attention blocks of transformers could also be used to model the graph relationships \cite{zhang2020graph} \cite{li2019modeling}. The goal-driven tree-structured model incorporated quantity cell graph and quantity comparison graph \cite{zhang2020graph}. Quantity cell graph connects numerical quantity of the input equations with its descriptive words or entities, whereas quantity comparison graph is similar to the concept used by \cite{ran2019numnet}. Segmentation of question into quantity span and question span, and using self-attention blocks to derive global attention, quantity-related attention, quantity pair attention, and question-related attention is another such approach to derive better semantic representation \cite{li2019modeling}.
Multitasking is a prevalent paradigm to train the same model for multiple tasks. It enriches the semantic representations of models and avoids them getting overfitted. Auxiliary tasks could also be part of such a setup. Auxiliary tasks like Common Sense Prediction Task, Number Quantity Prediction, and Number location prediction could be helpful \cite{qin2021neural}. The Commonsense prediction task aims to predict the commonsense required to solve an MWP, like the number of legs of a horse, or a number of days in a week, etc. On the other hand, mask language modeling pre-training on mathematical corpus Ape210k in a self-supervised way to help its downstream application for math word problems \cite{liang2021mwp}.
\subsection{Data Augmentation and Weak Supervision}
Large datasets have empowered neural models to learn complex mathematical concepts like integration, differentiation \cite{lample2019deep} \cite{saxton2019analysing}, etc, but such models often overfit on smaller datasets \cite{miao2021diverse} \cite{patel2021nlp}. Data augmentation is a popular preprocessing technique to increase the size of training data. Generally, data augmentation creates the variant of training records by applying domain-specific argumentation techniques. Several techniques were proposed for the data augmentation of MWP datasets. Reverse operation-based augmentation techniques swap an unknown quantity with a known quantity in an MWP, and their expression trees are modified to reflect the change \cite{liu2020reverse}. Different traversal orders of expression trees and other preprocessing techniques like lemmatization, POS tagging, sentence reordering, stop word removal, etc., and their impact on the performance of models have been studied by \cite{griffith2021solving}. Data processing and augmentation could also help to generate adversarial datasets, which could test the reasoning ability of models \cite{patel2021nlp}\cite{miao2021diverse}.
Weak supervision is another popular technique that generates large datasets with noisy labels. The model training process could cancel noise and learn actual patterns. For example, generating expression trees for an MWP dataset with question and answer could be done using weak supervision, where expression trees may not always be accurate. Still, the answer it evaluates would match the desired answer. Learning by fixing is a technique where expression trees are generated iteratively by fixing operators and operands till it evaluates to the desired answer \cite{hong2021learning}. Such a technique helps model learning the more diverse ways of solving mathematical problems.
\section{Mathematical Reasoning in Neural Approaches}
Mathematical reasoning is core to human intelligence. As explained before, it is a complex phenomenon interfacing natural language understanding and visual understanding to invoke mathematical transformations. Over the past few decades, there has been a flurry of research to develop models with mathematical reasoning ability; however, the progress is limited. Larger datasets definitely help models learning to solve complex and niche problems \cite{lample2019deep} \cite{saxton2019analysing}, but a common purpose model to solve a diverse variety of math problems is still a distinct reality \cite{patel2021nlp} \cite{miao2021diverse}. Smaller datasets overfit the models, and hence reasoning ability is questionable. Non-neural models could not improve performance with larger datasets \cite{huang2016well}.
On the other hand, neural models achieve much better performance on the smaller datasets and attain near-perfect performance over very large datasets; however, its BlackBox structure hinders interpreting its reasoning ability. There is a decent consensus among researchers to solve problems by first generating intermediate expression trees and evaluating them using solvers to predict the answer \cite{wang2017deep} \cite{amini2019mathqa} \cite{xie2019goal}. As opposed to directly predicting the final answer, generating intermediate expression trees helps understand models' reasoning ability better. Currently, the efforts are mainly to solve MWPs of primary or secondary schools as shown in Table \ref{tab:MWPDatasets}. The state of the current art is still very far from building mathematical intelligence that could also infer visual narratives and natural language and use domain-specific knowledge.
Models with desired reasoning ability would not only be generalizable, but it would also help to make systems interpretable and explainable. On the other hand, interpretable and explainable systems are easier to comprehend, which helps to progress further by expanding the scope of its reasoning on more complex mathematical concepts. The field of mathematics is full of axioms, rules, and formulas, and it would also be important to plug explicit and definite knowledge into such systems.
\subsection{Interpretability, and Explainability}
Deep learning models directly predicting answers from an input problem lack interpretability and explainability. As models often end up learning shallow heuristic patterns to arrive at the answer. \cite{huang2016well}, \cite{patel2021nlp}. That’s where it is important to generate intermediate representations or explanations, then infer the answer using them. It not only improves interpretability but also provides scaffolding to streamline reasoning ability and hence generalizability. Such intermediate representations could be of different forms, and they could be interspersed with natural language explanations to aid the explainability of the model further.
\subsubsection{Intermediate Representations}
The semantic parsing approach attempts to derive semantic structure from the problem narratives. Still, they are not generalizable because of the inherent limitations of statistical and classical machine learning techniques to represent the meaning. However, deriving intermediate representations and inferring the solution henceforth would remain one of the niche areas to explore in the context of neural approaches. Transforming problem narratives to expressions or equations and then evaluating them to get an answer is one of the most important approaches suitable for elementary level MWPs \cite{wang2017deep}. There were other attempts in deriving representation language \cite{shi2015automatically}, using logic forms \cite{liang2018meaning} and intermediate meaning representations \cite{huang2018using}, etc. It would require much more effort to derive intermediate representation for comprehensive and complex mathematical forms and expressions in hindsight.
\subsubsection{Interspersed Natural language Explanations}
Natural language explanations to describe the mathematical rationale would reflect the system's reasoning ability and also makes the system easier to comprehend and explainable. For example, \cite{ling2017program} uses LSTM to focus on generating answer rationale, which is a natural language explanation interspersed with algebraic expression to arrive at a decision. AQuA dataset contains 100,000 such algebraic problems with answer rationales. Such answer rationales not only improve interpretability but provide scaffolding which helps the model learn mathematical reasoning better. Language models like WT5, which produce explanations along with prediction, could be an inspiration \cite{narang2020wt5}. It would be essential to extend such approaches for more complex mathematical concepts beyond algebra. Natural language explanations interspersed with mathematical expressions would be critical areas for the researchers to focus on for building more comprehensive systems. Such interspersed explanations would open up the possibility to help students understand how to solve the problem more effectively or would foster users' trust in the system.
\subsection{Infusing Explicit and Definitive Knowledge}
Mathematical problem solving requires identifying relevant external knowledge and applying it to the mathematical rules to arrive at an answer. External knowledge could be commonsense world knowledge, particular domain knowledge, or complex mathematical formulas. There is some progress to infuse commonsense world knowledge \cite{liang2021mwp} or predict it using auxiliary tasks \cite{qin2021neural}, however remaining two dimensions are yet to be addressed. Such external knowledge is incorporated using rule-based, pattern-matching approaches in non-neural models; on the other hand, neural models leverage data augmentation and auxiliary task training. \cite{zhang2020graph} \cite{xie2019goal} developed novel neural architectures to represent semantics captured in the problems using graph and tree relations. These examples are of the shallow knowledge infusion category. It would be intriguing to explore deep knowledge infusion wherein model architecture would transform and leverage external knowledge vector spaces during its training \cite{wang2020k} \cite{faldu2021ki}. Furthermore, it would be promising to build an evaluation benchmark for such knowledge-intensive mathematical problems to encourage and streamline efforts to solve math problems \cite{sheth2021knowledge}.
\subsection{Reinforcement Learning}
Mathematical reasoning ability comprises applying a series of mathematical transformations. Rather than learning mathematical transformations, the system could focus on synthesizing the computational graph. Learning algorithms like gradient descent works on iterative reduction of the error. The choices of these mathematical transformations are not differential, and hence it is unclear if its the optimal strategy. Learning policy to construct such a computation graph using reinforcement learning could be helpful \cite{palermo2021reinforcement} \cite{huang2018neural}. Exponential search space of mathematical concepts is another key problem where reinforcement learning could be useful \cite{wang2018mathdqn}. Reinforcement learning reduces a problem to state transition problem with reward function. It initializes the state from a given problem, and based on the semantic information extracted from the problem, it derives actions from updating the state with a reward or penalty. It learns a policy to maximize the reward as it traverses through several intermediate states. For an MWP, the computation graph of expression trees denotes the state, and action denotes the choices of mathematical transformations and their mapping with operands from the input. The evaluation of the computation graph would give an answer, and a penalty or reward would be given based on the comparison between the computed answer and the expected ground truth. Researchers have also leveraged reinforcement learning on difficult math domains like automated theorem proving \cite{crouse2021deep} \cite{kaliszyk2018reinforcement}. It is still an early stage of the application of reinforcement learning for solving math word problems. It could bring the next set of opportunities to build systems capable of mathematical reasoning ability.
\section{Outlook}
The mathematical reasoning process involves understanding natural language and visual narratives and extracting entities with explicit and implicit quantities building a self-consistent symbolic representation invoking mathematical laws, axioms, and symbolic rules.
The ability to solve MWPs in an automated fashion could lead to many applications in the education domain for content creation and delivering learning outcomes \cite{donda2020framework} \cite{faldu2020adaptive}.
We surveyed sets of different approaches and their claims to have reasonable solved MWPs on specific datasets. However, we have also highlighted studies with “concrete evidence” that existing MWP solvers tend to rely on shallow heuristics to achieve their high performance and questions these models’ capabilities to solve even the simplest of MWPs robustly \cite{huang2016well} \cite{patel2021nlp} \cite{miao2021diverse}.
Non-neural approaches do not improve linearly with larger training data \cite{huang2016well}, on the other hand, neural approaches have shown a promise to solve even complex MWPs trained with very large corpus \cite{lample2019deep} \cite{saxton2019analysing}. However, the availability of extensive training data with diverse MWPs is a crucial challenge. The inherent limitation of neural models in terms of interpretability and explainability could be addressed partly by predicting expression trees instead of directly predicting answers \cite{gaur2021semantics}. Such expression trees could be converted into equations, and which are evaluated to infer the final answer. Further, expression trees also provide scaffolding to assist mathematical reasoning ability and open up design choices like graph-based encoder and tree-based decoder, which helps to learn from the inadequately sized training corpus. Interspersed natural language explanations and infusing explicit knowledge could be of significant value as it not only improves the explainability of the system but also guides the model to traverse through intermediate states anchoring mathematical reasoning \cite{ling2017program} \cite{gaur2021semantics}. Interspersed explanations not only solve the math word problems but also engage users on how to solve them.
We highlight that solving MWPs would require the inherent ability of mathematical reasoning, which comprises natural language understanding, image understanding, and the ability to invoke domain knowledge of mathematical laws, axioms, and theorems. There is a massive scope of complementing neural approaches with external expertise/knowledge and developing design choices as we extend the problem to complex mathematical areas.
\bibliography{mwpsurvey}
\bibliographystyle{acl_natbib}
\end{document}
\begin{thebibliography}{73}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Amini et~al.(2019)Amini, Gabriel, Lin, Koncel-Kedziorski, Choi, and
Hajishirzi}]{amini2019mathqa}
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and
Hannaneh Hajishirzi. 2019.
\newblock Mathqa: Towards interpretable math word problem solving with
operation-based formalisms.
\newblock \emph{arXiv preprint arXiv:1905.13319}.
\bibitem[{Amnueypornsakul and Bhat(2014)}]{amnueypornsakul2014machine}
Bussaba Amnueypornsakul and Suma Bhat. 2014.
\newblock Machine-guided solution to mathematical word problems.
\newblock In \emph{Proceedings of the 28th Pacific Asia Conference on Language,
Information and Computing}, pages 111--119.
\bibitem[{Bakman(2007)}]{bakman2007robust}
Yefim Bakman. 2007.
\newblock Robust understanding of word problems with extraneous information.
\newblock \emph{arXiv preprint math/0701393}.
\bibitem[{Bandura(2008)}]{bandura2008observational}
Albert Bandura. 2008.
\newblock Observational learning.
\newblock \emph{The international encyclopedia of communication}.
\bibitem[{Bobrow(1964)}]{bobrow1964natural}
Daniel~G Bobrow. 1964.
\newblock Natural language input for a computer problem solving system.
\bibitem[{Briars and Larkin(1984)}]{briars1984integrated}
Diane~J Briars and Jill~H Larkin. 1984.
\newblock An integrated model of skill in solving elementary word problems.
\newblock \emph{Cognition and instruction}, 1(3):245--296.
\bibitem[{Charniak(1968)}]{charniak1968calculus}
Eugene Charniak. 1968.
\newblock \emph{Calculus word problems}.
\newblock Ph.D. thesis, Ph. D. thesis, Massachusetts Institute of Technology.
\bibitem[{Chen et~al.(2005)Chen, Huang, Shih, and Chen}]{chen2005extended}
Keh-Jiann Chen, Shu-Ling Huang, Yueh-Yin Shih, and Yi-Jun Chen. 2005.
\newblock Extended-hownet: A representational framework for concepts.
\newblock In \emph{Proceedings of OntoLex 2005-Ontologies and Lexical
Resources}.
\bibitem[{Crouse et~al.(2021)Crouse, Abdelaziz, Makni, Whitehead, Cornelio,
Kapanipathi, Srinivas, Thost, Witbrock, and Fokoue}]{crouse2021deep}
Maxwell Crouse, Ibrahim Abdelaziz, Bassem Makni, Spencer Whitehead, Cristina
Cornelio, Pavan Kapanipathi, Kavitha Srinivas, Veronika Thost, Michael
Witbrock, and Achille Fokoue. 2021.
\newblock A deep reinforcement learning approach to first-order logic theorem
proving.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~35, pages 6279--6287.
\bibitem[{Cs{\'a}ji et~al.(2001)}]{csaji2001approximation}
Bal{\'a}zs~Csan{\'a}d Cs{\'a}ji et~al. 2001.
\newblock Approximation with artificial neural networks.
\newblock \emph{Faculty of Sciences, Etvs Lornd University, Hungary}, 24(48):7.
\bibitem[{Dellarosa(1986)}]{dellarosa1986computer}
Denise Dellarosa. 1986.
\newblock A computer simulation of children’s arithmetic word-problem
solving.
\newblock \emph{Behavior Research Methods, Instruments, \& Computers},
18(2):147--154.
\bibitem[{Dhavala et~al.(2020)Dhavala, Bhatia, Bose, Faldu, and
Avasthi}]{dhavala2020auto}
Soma Dhavala, Chirag Bhatia, Joy Bose, Keyur Faldu, and Aditi Avasthi. 2020.
\newblock Auto generation of diagnostic assessments and their quality
evaluation.
\newblock \emph{International Educational Data Mining Society}.
\bibitem[{Donda et~al.(2020)Donda, Dasgupta, Dhavala, Faldu, and
Avasthi}]{donda2020framework}
Chintan Donda, Sayan Dasgupta, Soma~S Dhavala, Keyur Faldu, and Aditi Avasthi.
2020.
\newblock A framework for predicting, interpreting, and improving learning
outcomes.
\newblock \emph{arXiv preprint arXiv:2010.02629}.
\bibitem[{English(2013)}]{english2013mathematical}
Lyn~D English. 2013.
\newblock \emph{Mathematical reasoning: Analogies, metaphors, and images}.
\newblock Routledge.
\bibitem[{Faldu et~al.(2020{\natexlab{a}})Faldu, Avasthi, and
Thomas}]{faldu2020adaptive}
Keyur Faldu, Aditi Avasthi, and Achint Thomas. 2020{\natexlab{a}}.
\newblock Adaptive learning machine for score improvement and parts thereof.
\newblock US Patent 10,854,099.
\bibitem[{Faldu et~al.(2021)Faldu, Sheth, Kikani, and Akabari}]{faldu2021ki}
Keyur Faldu, Amit Sheth, Prashant Kikani, and Hemang Akabari. 2021.
\newblock Ki-bert: Infusing knowledge context for better language and domain
understanding.
\newblock \emph{arXiv preprint arXiv:2104.08145}.
\bibitem[{Faldu et~al.(2020{\natexlab{b}})Faldu, Thomas, and
Avasthi}]{faldu2020system}
Keyur Faldu, Achint Thomas, and Aditi Avasthi. 2020{\natexlab{b}}.
\newblock System and method for recommending personalized content using
contextualized knowledge base.
\newblock US Patent App. 16/586,512.
\bibitem[{Feigenbaum et~al.(1963)Feigenbaum, Feldman
et~al.}]{feigenbaum1963computers}
Edward~A Feigenbaum, Julian Feldman, et~al. 1963.
\newblock \emph{Computers and thought}.
\newblock New York McGraw-Hill.
\bibitem[{Fletcher(1985)}]{fletcher1985understanding}
Charles~R Fletcher. 1985.
\newblock Understanding and solving arithmetic word problems: A computer
simulation.
\newblock \emph{Behavior Research Methods, Instruments, \& Computers},
17(5):565--571.
\bibitem[{Gaur et~al.(2021)Gaur, Faldu, and Sheth}]{gaur2021semantics}
Manas Gaur, Keyur Faldu, and Amit Sheth. 2021.
\newblock Semantics of the black-box: Can knowledge graphs help make deep
learning systems more interpretable and explainable?
\newblock \emph{IEEE Internet Computing}, 25(1):51--59.
\bibitem[{Griffith and Kalita(2021)}]{griffith2021solving}
Kaden Griffith and Jugal Kalita. 2021.
\newblock Solving arithmetic word problems with transformers and preprocessing
of problem text.
\newblock \emph{arXiv preprint arXiv:2106.00893}.
\bibitem[{Hendrycks et~al.(2021)Hendrycks, Burns, Kadavath, Arora, Basart,
Tang, Song, and Steinhardt}]{hendrycks2021measuring}
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric
Tang, Dawn Song, and Jacob Steinhardt. 2021.
\newblock Measuring mathematical problem solving with the math dataset.
\newblock \emph{arXiv preprint arXiv:2103.03874}.
\bibitem[{Hong et~al.(2021{\natexlab{a}})Hong, Li, Ciao, Huang, and
Zhu}]{hong2021learning}
Yining Hong, Qing Li, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu.
2021{\natexlab{a}}.
\newblock Learning by fixing: Solving math word problems with weak supervision.
\newblock In \emph{AAAI Conference on Artificial Intelligence}.
\bibitem[{Hong et~al.(2021{\natexlab{b}})Hong, Li, Gong, Ciao, Huang, and
Zhu}]{hong2021smart}
Yining Hong, Qing Li, Ran Gong, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu.
2021{\natexlab{b}}.
\newblock Smart: A situation model for algebra story problems via attributed
grammar.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~35, pages 13009--13017.
\bibitem[{Hosseini et~al.(2014)Hosseini, Hajishirzi, Etzioni, and
Kushman}]{hosseini2014learning}
Mohammad~Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman.
2014.
\newblock Learning to solve arithmetic word problems with verb categorization.
\newblock In \emph{Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 523--533.
\bibitem[{Huang et~al.(2018{\natexlab{a}})Huang, Liu, Lin, and
Yin}]{huang2018neural}
Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. 2018{\natexlab{a}}.
\newblock Neural math word problem solver with reinforcement learning.
\newblock In \emph{Proceedings of the 27th International Conference on
Computational Linguistics}, pages 213--223.
\bibitem[{Huang et~al.(2016)Huang, Shi, Lin, Yin, and Ma}]{huang2016well}
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016.
\newblock How well do computers solve math word problems? large-scale dataset
construction and evaluation.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 887--896.
\bibitem[{Huang et~al.(2018{\natexlab{b}})Huang, Yao, Lin, Zhou, and
Yin}]{huang2018using}
Danqing Huang, Jin-Ge Yao, Chin-Yew Lin, Qingyu Zhou, and Jian Yin.
2018{\natexlab{b}}.
\newblock Using intermediate representations to solve math word problems.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 419--428.
\bibitem[{Kahneman(2011)}]{kahneman2011thinking}
Daniel Kahneman. 2011.
\newblock \emph{Thinking, fast and slow}.
\newblock Macmillan.
\bibitem[{Kaliszyk et~al.(2018)Kaliszyk, Urban, Michalewski, and
Ol{\v{s}}{\'a}k}]{kaliszyk2018reinforcement}
Cezary Kaliszyk, Josef Urban, Henryk Michalewski, and Mirek Ol{\v{s}}{\'a}k.
2018.
\newblock Reinforcement learning of theorem proving.
\newblock \emph{arXiv preprint arXiv:1805.07563}.
\bibitem[{Khashabi et~al.(2019)Khashabi, Azer, Khot, Sabharwal, and
Roth}]{khashabi2019capabilities}
Daniel Khashabi, Erfan~Sadeqi Azer, Tushar Khot, Ashish Sabharwal, and Dan
Roth. 2019.
\newblock On the capabilities and limitations of reasoning for natural language
understanding.
\newblock \emph{arXiv preprint arXiv:1901.02522}.
\bibitem[{Koncel-Kedziorski et~al.(2015)Koncel-Kedziorski, Hajishirzi,
Sabharwal, Etzioni, and Ang}]{koncel2015parsing}
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and
Siena~Dumas Ang. 2015.
\newblock Parsing algebraic word problems into equations.
\newblock \emph{Transactions of the Association for Computational Linguistics},
3:585--597.
\bibitem[{Koncel-Kedziorski et~al.(2016)Koncel-Kedziorski, Roy, Amini, Kushman,
and Hajishirzi}]{koncel2016mawps}
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh
Hajishirzi. 2016.
\newblock Mawps: A math word problem repository.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 1152--1157.
\bibitem[{Kushman et~al.(2014)Kushman, Artzi, Zettlemoyer, and
Barzilay}]{kushman2014learning}
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014.
\newblock Learning to automatically solve algebra word problems.
\newblock In \emph{Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 271--281.
\bibitem[{Lample and Charton(2019)}]{lample2019deep}
Guillaume Lample and Fran{\c{c}}ois Charton. 2019.
\newblock Deep learning for symbolic mathematics.
\newblock \emph{arXiv preprint arXiv:1912.01412}.
\bibitem[{Li et~al.(2019)Li, Wang, Zhang, Wang, Dai, and
Zhang}]{li2019modeling}
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing~Tian Dai, and Dongxiang
Zhang. 2019.
\newblock Modeling intra-relation in math word problems with different
functional multi-head attentions.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 6162--6167.
\bibitem[{Liang et~al.(2018)Liang, Wong, Lin, and Su}]{liang2018meaning}
Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, and Keh-Yih Su. 2018.
\newblock A meaning-based statistical english math word problem solver.
\newblock \emph{arXiv preprint arXiv:1803.06064}.
\bibitem[{Liang et~al.(2021)Liang, Zhang, Shao, and Zhang}]{liang2021mwp}
Zhenwen Liang, Jipeng Zhang, Jie Shao, and Xiangliang Zhang. 2021.
\newblock Mwp-bert: A strong baseline for math word problems.
\newblock \emph{arXiv preprint arXiv:2107.13435}.
\bibitem[{Liguda and Pfeiffer(2012)}]{liguda2012modeling}
Christian Liguda and Thies Pfeiffer. 2012.
\newblock Modeling math word problems with augmented semantic networks.
\newblock In \emph{International Conference on Application of Natural Language
to Information Systems}, pages 247--252. Springer.
\bibitem[{Lin et~al.(2015)Lin, Liang, Hsu, Huang, Miao, Ma, Ku, Liau, and
Su}]{lin2015designing}
Yi-Chung Lin, Chao-Chun Liang, Kuang-Yi Hsu, Chien-Tsung Huang, Shen-Yun Miao,
Wei-Yun Ma, Lun-Wei Ku, Churn-Jung Liau, and Keh-Yih Su. 2015.
\newblock Designing a tag-based statistical math word problem solver with
reasoning and explanation.
\newblock In \emph{International Journal of Computational Linguistics \&
Chinese Language Processing, Volume 20, Number 2, December 2015-Special Issue
on Selected Papers from ROCLING XXVII}.
\bibitem[{Ling et~al.(2017)Ling, Yogatama, Dyer, and Blunsom}]{ling2017program}
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.
\newblock Program induction by rationale generation: Learning to solve and
explain algebraic word problems.
\newblock \emph{arXiv preprint arXiv:1705.04146}.
\bibitem[{Lithner(2000)}]{lithner2000mathematical}
Johan Lithner. 2000.
\newblock Mathematical reasoning in task solving.
\newblock \emph{Educational studies in mathematics}, pages 165--190.
\bibitem[{Liu et~al.(2020)Liu, Guan, Li, Cheng, Kawahara, and
Kurohashi}]{liu2020reverse}
Qianying Liu, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, and Sadao
Kurohashi. 2020.
\newblock Reverse operation based data augmentation for solving math word
problems.
\newblock \emph{arXiv preprint arXiv:2010.01556}.
\bibitem[{Liu et~al.(2019)Liu, Guan, Li, and Kawahara}]{liu2019tree}
Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. 2019.
\newblock Tree-structured decoding for solving math word problems.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 2370--2379.
\bibitem[{Miao et~al.(2021)Miao, Liang, and Su}]{miao2021diverse}
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2021.
\newblock A diverse corpus for evaluating and developing english math word
problem solvers.
\newblock \emph{arXiv preprint arXiv:2106.15772}.
\bibitem[{Mitra and Baral(2016)}]{mitra2016learning}
Arindam Mitra and Chitta Baral. 2016.
\newblock Learning to use formulas to solve simple arithmetic problems.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 2144--2153.
\bibitem[{Narang et~al.(2020)Narang, Raffel, Lee, Roberts, Fiedel, and
Malkan}]{narang2020wt5}
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and
Karishma Malkan. 2020.
\newblock Wt5?! training text-to-text models to explain their predictions.
\newblock \emph{arXiv preprint arXiv:2004.14546}.
\bibitem[{Palermo et~al.(2021)Palermo, Ye, and
Singh}]{palermo2021reinforcement}
Joseph Palermo, Johnny Ye, and Alok Singh. 2021.
\newblock A reinforcement learning environment for mathematical reasoning via
program synthesis.
\newblock \emph{arXiv preprint arXiv:2107.07373}.
\bibitem[{Patel et~al.(2021)Patel, Bhattamishra, and Goyal}]{patel2021nlp}
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.
\newblock Are nlp models really able to solve simple math word problems?
\newblock \emph{arXiv preprint arXiv:2103.07191}.
\bibitem[{Qin et~al.(2021)Qin, Liang, Hong, Tang, and Lin}]{qin2021neural}
Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. 2021.
\newblock Neural-symbolic solver for math word problems with auxiliary tasks.
\newblock \emph{arXiv preprint arXiv:2107.01431}.
\bibitem[{Qin et~al.(2020)Qin, Lin, Liang, Zhang, and
Lin}]{qin2020semantically}
Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020.
\newblock Semantically-aligned universal tree-structured solver for math word
problems.
\newblock \emph{arXiv preprint arXiv:2010.06823}.
\bibitem[{Ran et~al.(2019)Ran, Lin, Li, Zhou, and Liu}]{ran2019numnet}
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019.
\newblock Numnet: Machine reading comprehension with numerical reasoning.
\newblock \emph{arXiv preprint arXiv:1910.06701}.
\bibitem[{Robaidek et~al.(2018)Robaidek, Koncel-Kedziorski, and
Hajishirzi}]{robaidek2018data}
Benjamin Robaidek, Rik Koncel-Kedziorski, and Hannaneh Hajishirzi. 2018.
\newblock Data-driven methods for solving algebra word problems.
\newblock \emph{arXiv preprint arXiv:1804.10718}.
\bibitem[{Roy and Roth(2017)}]{roy2017unit}
Subhro Roy and Dan Roth. 2017.
\newblock Unit dependency graph and its application to arithmetic word problem
solving.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~31.
\bibitem[{Roy and Roth(2018)}]{roy2018mapping}
Subhro Roy and Dan Roth. 2018.
\newblock Mapping to declarative knowledge for word problem solving.
\newblock \emph{Transactions of the Association for Computational Linguistics},
6:159--172.
\bibitem[{Roy et~al.(2015)Roy, Vieira, and Roth}]{roy2015reasoning}
Subhro Roy, Tim Vieira, and Dan Roth. 2015.
\newblock Reasoning about quantities in natural language.
\newblock \emph{Transactions of the Association for Computational Linguistics},
3:1--13.
\bibitem[{Sap et~al.(2020)Sap, Shwartz, Bosselut, Choi, and
Roth}]{sap2020commonsense}
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020.
\newblock Commonsense reasoning for natural language processing.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics: Tutorial Abstracts}, pages 27--33.
\bibitem[{Saxton et~al.(2019)Saxton, Grefenstette, Hill, and
Kohli}]{saxton2019analysing}
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019.
\newblock Analysing mathematical reasoning abilities of neural models.
\newblock \emph{arXiv preprint arXiv:1904.01557}.
\bibitem[{Sheth et~al.(2021)Sheth, Gaur, Roy, and Faldu}]{sheth2021knowledge}
Amit Sheth, Manas Gaur, Kaushik Roy, and Keyur Faldu. 2021.
\newblock Knowledge-intensive language understanding for explainable ai.
\newblock \emph{IEEE Internet Computing}, (01):1--1.
\bibitem[{Shi et~al.(2015)Shi, Wang, Lin, Liu, and Rui}]{shi2015automatically}
Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015.
\newblock Automatically solving number word problems by semantic parsing and
reasoning.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 1132--1142.
\bibitem[{Storks et~al.(2019)Storks, Gao, and Chai}]{storks2019commonsense}
Shane Storks, Qiaozi Gao, and Joyce~Y Chai. 2019.
\newblock Commonsense reasoning for natural language understanding: A survey of
benchmarks, resources, and approaches.
\newblock \emph{arXiv preprint arXiv:1904.01172}, pages 1--60.
\bibitem[{Upadhyay and Chang(2015)}]{upadhyay2015draw}
Shyam Upadhyay and Ming-Wei Chang. 2015.
\newblock Draw: A challenging and diverse algebra word problem set.
\newblock Technical report, Citeseer.
\bibitem[{Wang et~al.(2019)Wang, Pruksachatkun, Nangia, Singh, Michael, Hill,
Levy, and Bowman}]{wang2019superglue}
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael,
Felix Hill, Omer Levy, and Samuel~R Bowman. 2019.
\newblock Superglue: A stickier benchmark for general-purpose language
understanding systems.
\newblock \emph{arXiv preprint arXiv:1905.00537}.
\bibitem[{Wang et~al.(2018{\natexlab{a}})Wang, Singh, Michael, Hill, Levy, and
Bowman}]{wang2018glue}
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel~R
Bowman. 2018{\natexlab{a}}.
\newblock Glue: A multi-task benchmark and analysis platform for natural
language understanding.
\newblock \emph{arXiv preprint arXiv:1804.07461}.
\bibitem[{Wang et~al.(2018{\natexlab{b}})Wang, Zhang, Gao, Song, Guo, and
Shen}]{wang2018mathdqn}
Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng~Tao
Shen. 2018{\natexlab{b}}.
\newblock Mathdqn: Solving arithmetic word problems via deep reinforcement
learning.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~32.
\bibitem[{Wang et~al.(2020)Wang, Tang, Duan, Wei, Huang, Cao, Jiang, Zhou
et~al.}]{wang2020k}
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao,
Daxin Jiang, Ming Zhou, et~al. 2020.
\newblock K-adapter: Infusing knowledge into pre-trained models with adapters.
\newblock \emph{arXiv preprint arXiv:2002.01808}.
\bibitem[{Wang et~al.(2017)Wang, Liu, and Shi}]{wang2017deep}
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017.
\newblock Deep neural solver for math word problems.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing}, pages 845--854.
\bibitem[{Wu et~al.(2021)Wu, Zhang, Wei, and Huang}]{wu2021math}
Qinzhuo Wu, Qi~Zhang, Zhongyu Wei, and Xuan-Jing Huang. 2021.
\newblock Math word problem solving with explicit numerical values.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 5859--5869.
\bibitem[{Xie and Sun(2019)}]{xie2019goal}
Zhipeng Xie and Shichao Sun. 2019.
\newblock A goal-driven tree-structured neural model for math word problems.
\newblock In \emph{IJCAI}, pages 5299--5305.
\bibitem[{Yuhui et~al.(2010)Yuhui, Ying, Guangzuo, Yun, and
Ronghuai}]{yuhui2010frame}
Ma~Yuhui, Zhou Ying, Cui Guangzuo, Ren Yun, and Huang Ronghuai. 2010.
\newblock Frame-based calculus of solving arithmetic multi-step addition and
subtraction word problems.
\newblock In \emph{2010 Second International Workshop on Education Technology
and Computer Science}, volume~2, pages 476--479. IEEE.
\bibitem[{Zhang et~al.(2020)Zhang, Wang, Lee, Bin, Wang, Shao, and
Lim}]{zhang2020graph}
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi~Bin, Yan Wang, Jie Shao, and Ee-Peng
Lim. 2020.
\newblock Graph-to-tree learning for solving math word problems.
\newblock Association for Computational Linguistics.
\bibitem[{Zhao et~al.(2020)Zhao, Shang, Liu, Wang, and Liu}]{zhao2020ape210k}
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020.
\newblock Ape210k: A large-scale and template-rich dataset of math word
problems.
\newblock \emph{arXiv preprint arXiv:2009.11506}.
\bibitem[{Zhou et~al.(2015)Zhou, Dai, and Chen}]{zhou2015learn}
Lipu Zhou, Shuaixiang Dai, and Liwei Chen. 2015.
\newblock Learn to solve algebra word problems using quadratic programming.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 817--822.
\end{thebibliography}