MuGER$^2$: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid Question Answering
Extracted source text · arXiv 2210.10350
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage[]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{tablefootnote}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\usepackage{multirow}
% \usepackage{algorithm}
\usepackage{wrapfig}
\usepackage{url}
\usepackage{latexsym}
\usepackage{color}
\usepackage{booktabs}
\usepackage{subfigure}
\usepackage{multirow}
\usepackage{verbatim}
\usepackage{amsfonts}
\usepackage{float}
\usepackage{bm}
\usepackage{comment}
\usepackage{arydshln}
\usepackage{array}
%\setlength\titlebox{5cm}
\usepackage{graphicx} % DO NOT CHANGE THIS
\usepackage{amsmath}
\usepackage{blindtext}
\usepackage{wrapfig}
\usepackage{algorithmic}
\usepackage[lined,boxed,commentsnumbered,ruled,linesnumbered]{algorithm2e}
\definecolor{mygreen}{RGB}{102, 205, 170}
\definecolor{mypink}{RGB}{238, 162, 173}
\definecolor{myyellow}{RGB}{238, 232, 170}
\definecolor{myblue}{RGB}{164, 211, 238}
\DeclareMathOperator*{\argmaxB}{argmax}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{MuGER$^2$: {M}ult{i}-{G}ranularity {E}vidence \textbf{R}etrieval and \textbf{R}easoning for Hybrid Question Answering }
% \title{MuGER$^2$: {M}ult{i}-{G}ranularity {E}vidence \textbf{R}etrieval and \textbf{R}easoning over Hybrid Tabular and Textual Knowledge }
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
\author{Yingyao Wang\textsuperscript{1}, Junwei Bao\textsuperscript{2}\thanks{~~Work was done during the first author’s internship at JD AI mentored by Junwei Bao: baojunwei001@gmail.com.}, Chaoqun Duan\textsuperscript{2}, Youzheng Wu\textsuperscript{2}, \\ \bf Xiaodong He\textsuperscript{2}, Tiejun Zhao\textsuperscript{1}\thanks{~~Corresponding author.}\\
\textsuperscript{1}Harbin Institute of Technology \;\; \textsuperscript{2}JD AI Research \\
yywang@hit-mtlab.net \;\; baojunwei001@gmail.com \;\; tjzhao@hit.edu.cn
}
\begin{document}
\maketitle
% \begin{abstract}
% Hybrid question answering (HQA) requires multi-hop reasoning over heterogeneous knowledge including structured tables and unstructured text.
% It usually faces two challenges: (1) \textit{retrieval}, namely how to
% retrieve semantic-level evidence, e.g., related table cells and passages, from heterogeneous data based on given natural language questions; and (2) \textit{reasoning}, namely how to leverage the retrieved evidence to reason natural language answers.
% Unlike previous methods which treat them in a cascaded manner, we propose MERDR, a unified \textbf{m}ulti-\textbf{e}vidence \textbf{r}etrieval (MER) and \textbf{d}iscriminative \textbf{r}easoning (DR) approach, to address these HQA challenges.
% Specifically, retrieval consists of multiple retrievers, each of which retrieves atomic evidence for the question.
% Moreover, the MER further benefits from joint training over retrievers and contrastive learning among the evidence.
% The multiple atomic retrievers are then utilized for answer reasoning.
% Furthermore, answer reasoning is advanced by an iterative discrimination procedure judging whether to stop retrieval then extract answers or re-retrieve for further reasoning.
% Extensive experiment results on the HybridQA dataset show that MERDR dramatically improves the performance.
% Further ablation and analysis verify the effectiveness of both the MER and DR designs.\footnote{Our code will be released upon the acceptance of the paper.}
% \end{abstract}
% \begin{abstract}
% Hybrid question answering (HQA) aims to answer questions over heterogeneous knowledge including both structured tables and unstructured text linked to tables.
% Conventional HQA methods usually retrieve \textit{mono-granularity} evidence, e.g., column, row, cell, or link, and directly feed it into a reading comprehension (RC) model to reason the answer.
% Through comparison, we find that coarse-granularity methods achieve higher evidence retrieval recall but lower answer reasoning F1 score, while fine-granularity methods are the opposite.
% % A problem of a mono-granularity method is the trade-off between evidence retrieval and answer reasoning, i.e., coarse evidences raise the retrieval recall but lower the reasoning accuracy, while fine evidence is on opposite.
% To avoid the trade-off
% % among different types of mono-granularity evidence,
% between evidence retrieval and answer reasoning,
% we propose \textbf{MuGER$^2$}, a {\textbf{Mu}lti-\textbf{G}ranularity} \textbf{E}vidence \textbf{R}etrieval and \textbf{R}easoning approach, which boosts the end-to-end HQA performance.
% Specifically, multi-granularity evidence is first retrieved based on a unified retriever that is learned by joint and contrastive training on multi-granularity evidence.
% The answer is then derived by an RC-centered reasoner which is advanced by answer type -aware reasoning.
% Extensive experiment results on the HybridQA dataset show that {MuGER$^2$} achieves remarkable improvements.
% Further ablation and analysis verify the effectiveness of both the multi-granularity retrieval and reasoning designs.\footnote{Our code will be publicly available.}
% \end{abstract}
\begin{abstract}
Hybrid question answering (HQA) aims to answer questions over heterogeneous data, including tables and passages linked to table cells. The heterogeneous data can provide different granularity evidence to HQA models, e.t., column, row, cell, and link. Conventional HQA models usually retrieve coarse- or fine-grained evidence to reason the answer.
Through comparison, we find that coarse-grained evidence is easier to retrieve but contributes less to the reasoner, while fine-grained evidence is the opposite.
To preserve the advantage and eliminate the disadvantage of different granularity evidence,
we propose \textbf{MuGER$^2$}, a {\textbf{Mu}lti-\textbf{G}ranularity} \textbf{E}vidence \textbf{R}etrieval and \textbf{R}easoning approach.
In evidence retrieval, a unified retriever is designed to learn the multi-granularity evidence from the heterogeneous data. In answer reasoning, an evidence selector is proposed to navigate the fine-grained evidence for the answer reader based on the learned multi-granularity evidence.
Experiment results on the HybridQA dataset show that {MuGER$^2$} significantly boosts the HQA performance.
Further ablation analysis verifies the effectiveness of both the retrieval and reasoning designs.\footnote{Our code is publicly available in \url{https://github.com/JD-AI-Research-NLP/MuGER2}.}.
\end{abstract}
%===========================================
%===========================================
%===========================================
%===========================================
% \begin{figure*}[t]
% \centering
% \subfigure[Table-G. ]{\includegraphics[width=0.195\textwidth]{fig/table-g.pdf}}
% \subfigure[Row-g. ]{\includegraphics[width=0.195\textwidth]{fig/row-g.pdf}}
% \subfigure[Cell-g. ]{\includegraphics[width=0.195\textwidth]{fig/cell-g.pdf}}
% \subfigure[Passage-g. ]{\includegraphics[width=0.195\textwidth]{fig/passage-g.pdf}}
% \subfigure[Multi-g. ]{\includegraphics[width=0.195\textwidth]{fig/multi-g.pdf}}
% \caption{.}
% \label{evidence}
% \end{figure*}
%===========================================
\section{Introduction}
%Traditional knowledge-based question answering systems derive answers to questions based on homogeneous knowledge, including structured knowledge bases or tables~\cite{yih2015semantic,jauhar2016tables,zhang2020table}, and unstructured text \cite{joshi2017triviaqa,dunn2017searchqa}, which fail to answer questions that require reasoning over heterogeneous data.
Traditional knowledge-based question answering systems derive answers to questions based on homogeneous knowledge, including knowledge graphs~\cite{bao2016constraint,gu2021beyond}, tables~\cite{jauhar2016tables,zhang2020table}, or passages~\cite{zhao-etal-2021-ror-read,zhou-etal-2022-opera} and achieve remarkable performances.
However, they neglect a more general scenario requiring reasoning over heterogeneous data to answer a question.
%To address this limitation, hybrid question answering (HQA) together with the HybridQA dataset is proposed \cite{chen-etal-2020-hybridqa}.
To investigate this challenge, \citet{chen-etal-2020-hybridqa} propose the hybrid question answering (HQA) and publish a corresponding dataset, HybridQA.
Specifically, each question of HybridQA is aligned with a table and passages linked to table cells (abbreviated as links) as knowledge. The answer to the question may come from either table cells or links.
Figure~\ref{task} illustrates two examples of HybridQA. For the first one, the answer `\textit{Philip Mulkey}' comes from the cell, while for the second one, its answer `\textit{Mississippi River}' is derived from the link.
\begin{figure}[t]
\setlength{\belowcaptionskip}{-0.3cm}
\centering
\includegraphics[width=0.48\textwidth]{fig/task.pdf}
\caption{Two example questions of HQA task.}
\label{task}
\end{figure}
% \vspace{0.5cm}
%Conventional HQA methods usually first retrieve a \textit{mono-granularity} table region as evidence, e.g.,column, row, cell, and link shown in Figure~\ref{motivation}, and then directly feed it into a reading comprehension (RC) model to reason the answer.
% \citet{zhang-etal-2020-table} first encode the whole table as evidence, and then answer questions based on its representation.
% \citet{glass2021capturing}
%Hybrider \cite{chen-etal-2020-hybridqa} pinpoints a cell as evidence, and extracts answer from the cell content and its link with an RC model.
%Dochopper \cite{sun2021end} retrieves a text fragment as evidence, and extracts the answer with an RC model.
Existing HQA models usually consist of two components, a \textbf{retriever} to learn evidence and a \textbf{reasoner} to leverage the evidence to derive the answer. Intuitively, the heterogeneous data shown in Figure~\ref{task} can provide different granularity evidence to the HQA models, such as coarse-grained column and row and fine-grained cell and link.
Previous models always retrieve coarse or fine-grained evidence and directly use a span-based reading comprehension model to reason the answer.
For example, \citet{kumar2021multi} choose a coarse-grained region as the evidence, e.g., a table row.
On the contrary, \citet{chen-etal-2020-hybridqa} and \citet{sun2021end} focus on the fine-grained units, table cell and link.
Intuitively, compared with fine-grained evidence, coarse-grained evidence is easier to be accurately retrieved.
However, it contributes less to the reasoner's performance since it contains more noisy information.
The fine-grained evidence is just the opposite for the retriever and the reasoner.
To prove this argument, we conduct experiments on the development set of HybridQA to analyze the retriever and the reasoner performance\footnote{In these experiments, the retriever and reader models are all initialized with BERT-base.} with different granularity evidence. The results of evidence retrieval and answer reasoning are evaluated with the retrieval recall (R@1) and the F1 score, respectively.
As shown in Figure~\ref{motivation}, the coarse-grained evidence achieves better retrieval recalls but lower F1 scores than the fine-grained evidence. With the granularity changing from coarse to fine, the F1 score increases while the R@1 declines.
The results are consistent with our hypothesis.
\begin{figure}[t]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\includegraphics[width=0.45\textwidth]{fig/motivation.pdf}
\caption{The retriever and reasoner performance with different granularity evidence.
% The answer recall $R@1$ is defined to evaluate the performance of the retriever, which denotes the probability that the retrieved evidence contains the answer.
Note that the $F1$ score in this figure is calculated based on the \textbf{oracle} evidence to evaluate the answer reasoning performance.
Intuitively, $R@1 \times F1$ reflects the end-to-end HQA performance. }
\label{motivation}
\end{figure}
%Existing HQA systems mainly consist of two stages, i.e., \textbf{retrieval} and \textbf{reasoning}. In the retrieval stage, a certain granularity region of the heterogeneous data is retrieved as evidence, e.g., column, row \cite{kumar2021multi}, cell \cite{chen-etal-2020-hybridqa}, or link \cite{sun2021end}.
%In the reasoning stage, the retrieved evidence is usually fed into the reading comprehension (RC) process to reason the answer.
%Despite the success of the existing HQA systems, they still face limitation since they only consider mono-granularity evidence in both the retrieval and reasoning stages. Actually, evidence of different granularity has advantages in the two stages respectively.
%For example, coarser-grained evidence like rows is easier to retrieved accurately than finer-grained evidence like cells, but it is more difficult for RC process to reason the answer based on a table row than a table cell.
%To reveal that, we give the retrieval and reasoning performance\footnote{The two stages are both implemented using BERT-base model.} with different granularity evidence in Figure~\ref{motivation}. We
%take the retrieval recall (R@1) and the F1 score (calculated with the oracle evidence) as the evaluation metrics of the two stages. The results demonstrate that coarser granularity evidence like columns and rows is easier to retrieved accurately and brings higher recall in the retrieval stage. By contrast, based on the oracle evidence, reasoning answer from the finer granularity evidence like cells and links is easier for the RC process, which achieves higher F1 score in the reasoning stage.
To preserve the advantage and eliminate the disadvantage of different granularity evidence, we propose \textbf{MuGER$^2$}, a {\textbf{Mu}lti-\textbf{G}ranularity} \textbf{E}vidence \textbf{R}etrieval and \textbf{R}easoning approach for HQA.
In the retrieval stage, a unified retriever is designed to learn the multi-granularity evidence from the heterogeneous data involving columns, rows, cells, and links. Compared with existing methods, the multi-granularity evidence preserves the retrieval recall and provides more aspects of information to reason the answer. In the reasoning stage, to avoid the redundant information in the multi-granularity evidence that lowers the answer reader's performance, an evidence selector (E-SEL) is designed to navigate the fine-grained evidence for the reader by fusing the learned multi-granularity evidence. Together with our multi-granularity evidence retrieval and reasoning designs, our MuGER$^2$ preserves both the retrieval recall and the F1 score and further boosts the end-to-end HQA performance.
% dcq: After that, we design an answer type-aware reasoning (ATR) process to automatically select the suitable evidence to predict the answer.
% dcq: Compared with existing methods, our model employs different granularity evidence and this mechanism preserves the retrieval recall and provides more information to the reasoning stage.
% dcq: Meanwhile, we add the ATR process and it performs coarse-to-fine symbolic reasoning by integrating the multi-granularity information and detecting the answer type (in cell values or in links).
%To combine the advantages of different granularity evidence and simultaneously develop the performance of both the retrieval and reasoning stages, we propose \textbf{MuGER$^2$}, a {\textbf{Mu}lti-\textbf{G}ranularity} \textbf{E}vidence \textbf{R}etrieval and \textbf{R}easoning approach for HQA.
%The multi-granularity involves $\{\texttt{column},\texttt{row}, \texttt{cell}, \texttt{link}\}$ as shown in Figure~\ref{task}.
%Firstly, our retrieval stage retrieves evidence of multiple granularity involving $\{\texttt{column},\texttt{row}, \texttt{cell}, \texttt{link}\}$ shown in Figure~\ref{motivation}. Compared to the mono-granularity evidence retrieval, it increases the retrieval recall and provides more information to the reasoning stage. Afterwards, an answer type-aware reasoning (ATR) process is designed to automatically reason the answer . The ATR performs coarse-to-fine symbolic reasoning by integrating the multi-granularity information and detecting the answer type (in cell values or in links). Our reasoning stage raises the $F1$ compared to simply adopt the RC process. Figure~\ref{motivation} shows that our MuGER$^2$ will boosts the end-to-end HQA performance together with the multi-granularity evidence retrieval and reasoning stages.
%We visualize $R$ and $F1$ of different evidence granularity based HQA systems in Figure~\ref{motivation}.
%It demonstrates that coarse-granularity evidence achieves higher retrieval recall but lower answer reasoning accuracy, while fine-granularity evidence is the opposite.
%Intuitively, integrating evidence with different granularity has the potential to raise the retrieval recall since it provides more information for answer reasoning.
%Together with a more precise answer reasoning design, it will boost the end-to-end HQA performance.
% avoid the trade-off among different mono-granularity evidence and boost the end-to-end HQA performance.
%Intuitively, HQA faces the following challenges from \textit{multi-granularity}:
%(1) whether a column or a row contains the answer;
%(2) whether the answer comes from a specific value or links of a cell;
%and (3) whether a link includes the answer.
%To address the above challenges and avoid the trade-off between evidence retrieval and answering reasoning in mono-granularity methods, we propose \textbf{MuGER$^2$}, a {\textbf{Mu}lti-\textbf{G}ranularity} \textbf{E}vidence \textbf{R}etrieval and \textbf{R}reasoning approach for HQA.
%First, evidence of multi-granularity involving $\{\texttt{column},\texttt{row}, \texttt{cell}, \texttt{link}\}$ as shown in Figure~\ref{task} is retrieved based on a unified retriever.
%The unified retriever is improved by integrating the learning of multi-granularity evidence with joint and contrastive training.
%Afterwards, the answer is derived by an RC-centered reasoner which is further advanced by answer type -aware reasoning.
% Specifically, we propose MER which consists of multiple atomic retrievers including $\langle$anchor, table hop, passage hop, passage, header$\rangle$ retrievers, each of which maps the given question into the corresponding evidence for reasoning.
% Moreover, the MER further benefits from joint training over retrievers and contrastive learning among the evidence.
% Nevertheless, the retrieved results may still contain errors.
% To accurately reason the answer, an HQA model should be capable of judging whether the retrieved evidence is correct or not.
% Furthermore, we propose DR for reasoning which is advanced by an iterative discrimination procedure to determine to extract the answer with current retrieved evidence or re-retrieve for further reasoning.
We conduct extensive experiments on the HybridQA dataset to verify the effectiveness of our proposed MuGER$^2$.
Experiment results show that MuGER$^2$ achieves remarkable improvements, which outperforms a publicly available strong baseline by $10.0\%$ EM and $12.9\%$ F1 scores on the end-to-end HQA performance.
Ablation studies verify the effectiveness of the proposed multi-granularity evidence retrieval and the evidence selector.
Moreover, results compared with different mono-granularity baselines prove the effectiveness of multi-granularity evidence for HQA.
Our contributions are as follows:
(1) We analyze the limitation of HQA systems which solely rely on only one certain granularity evidence in both the retriever and reasoner and propose a {{mu}lti-{g}ranularity} {e}vidence {r}etrieval and {r}easoning approach, which boosts the end-to-end HQA results.
(2) We propose a joint retrieval method that improves the evidence retrieval performance and an evidence selector to accurately navigate the fine-grained evidence from the multi-granularity information to preserve the reader's performance.
(3) We conduct extensive experiments on the HybridQA dataset and show the effectiveness of {MuGER$^2$}.
% \begin{itemize}
% \item We formalize HQA as evidence retrieval and answer reasoning, and find the issue of mono-granularity methods via empirically analyzing the trade-off between the two stages.
% \item We propose {MuGER$^2$},
% a {{mu}lti-{g}ranularity} {e}vidence {r}etrieval and {r}easoning approach,
% which upgrades mono- to multi- granularity evidence as bridge from question to answer.
% \item We conduct extensive experiments on HybridQA dataset and show the effectiveness of {MuGER$^2$}. We will release our code to further the research on HQA for the community.
% \end{itemize}
%===========================================
\begin{figure*}[t]
\setlength{\belowcaptionskip}{-0.3cm}
\centering
\includegraphics[width=0.99\textwidth]{fig/model.pdf}
\caption{The framework of MuGER$^2$ which performs multi-granularity evidence retrieval and answer reasoning. Evidence of four kinds of granularity including $\{ \colorbox{mypink}{\texttt{column}}, \colorbox{mygreen}{\texttt{row}}, \colorbox{myyellow}{\texttt{cell}}, \colorbox{myblue}{\texttt{link}}\}$ are in color. }
\label{minda}
% \vspace{-2em}
\end{figure*}
%===========================================
\section{Methodology}
\subsection{Task: Hybrid Question Answering}
The hybrid question answering (HQA) aims to tackle the answer reasoning over heterogeneous data, as shown in Figure~\ref{task}.
The input of this task includes a question $\mathcal{Q}$, a table $\mathcal{T}$, and a set of links $\mathcal{L}$.
Specifically, the table $\mathcal{T}$ consists of cells $\{c_{i,j}\}_{i=1,j=1}^{i=M,j=N}$, where $M$ and $N$ are the numbers of rows and columns. Each column has a header $h_j$ to describe the field of the cells $\{c_{i,j}\}_{i=1}^{i=M}$.
A table cell $c_{i,j}$ may link to a subset of links $\mathcal{L}_{i,j} \subseteq \mathcal{L}$.
Given the input, an HQA system aims to return the answer text $\mathcal{A}$ of the question $\mathcal{Q}$ by reasoning on the heterogeneous information. In this work, $\mathtt{In}\text{-}\mathtt{Table}$ and $\mathtt{In}\text{-}\mathtt{Passage}$ respectively represent that the answer is in a cell or in a link.
%Notably, the answer is divided into two types according to whether it come from a cell value or a link.
% It is worth noting that the questions are divided into two types, `in-table' and `in-passage', which means the answer is distributed in the table cells or in the linked passages.
% , noted as `in cell link'.
%\subsection{Preliminary: Analysis of HQA Methods on Mono-Granularity Evidence}
%Conventional HQA methods usually first retrieve a \textit{mono-granularity} table region as \textbf{evidence}, e.g., column, row, cell, and link shown in Figure~\ref{motivation}, and then directly feed it into a reading comprehension (RC) model to reason the answer.
% For instance, Hybrider
%Despite the success of the mono-granularity methods, they still face the problem of the trade-off between evidence retrieval and answer reasoning.
%To reveal that, we define the recall of evidence retrieval as $R$ and F1 of answering reasoning as $F1$.
%We design four baseline methods with different evidence granularity with details described in Section \ref{baselines}.
%Results in Figure~\ref{motivation} demonstrate that coarse evidence achieves higher retrieval recall but lower answer reasoning accuracy, while fine evidence is on the opposite.
%Besides, we show that the multi-granularity method achieves better performance than mono-granularity methods.
%This is natural, since integrating evidence with different granularity has the potential to raise the recall of evidence retrieval.
%Moreover, a more precise answer reasoning design will boost the end-to-end HQA performance.
% The HQA model first retrieves a table region as evidence, and then performs answer reasoning feed it into a reading comprehension (RC) model to reason the answer.
\subsection{Model Overview}
\label{overview}
As described above, coarse-grained evidence is easier to be derived by the retriever and it makes less contribution to the reasoner while the fine-grained is just the opposite.
To preserve their advantage and get rid of their disadvantage, we propose {MuGER$^2$}, a {{mu}lti-{g}ranularity} {e}vidence {r}etrieval and reasoning approach.
As shown in Figure~\ref{minda}, in the retrieval stage, MuGER$^2$ adopts a unified retriever to learn the multi-granularity evidence.
In the reasoning stage, an evidence selector is designed to navigate the fine-grained evidence for the reader based on the multi-granularity evidence.
The details of these modules are introduced in the following sections.
% However, the HybridQA does not offer the annotation of the evidence.
% Thereby, we can't directly use them to compute the objective function.
% An intuitive approach is to use the marginal probability to formalize this task as follow:
% % Formally, the model can be formulated as:
% %To preserve the advantage and eliminate the disadvantage of different granularity evidence, we propose {MuGER$^2$}, a {{mu}lti-{g}ranularity} {e}vidence {r}etrieval and reasoning approach. As shown in Figure~\ref{minda}, the framework of MuGER$^2$ is divided into two stages, retrieval and reasoning. Defining the multi-granularity evidence as $\mathcal{E}$, the overall model of MuGER$^2$ is formalized as:
% \begin{equation}
% p(\mathcal{A}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L})=\sum_{\mathcal{E}}p(\mathcal{A}|\mathcal{E}) \cdot p(\mathcal{E}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L})
% \label{mml}
% \end{equation}
% where $p(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})$ denotes the retriever and $\mathcal{E}$ is the retrieved multi-granularity evidence.
% $p(\mathcal{A}|\mathcal{E})$ is the combination of evidence selector and reader.
% %where $p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})$ is to model the retrieval stage which retrieves $\mathcal{E}$ for the question $\mathcal{Q}$ from the heterogeneous data $\mathcal{T}$ and $\mathcal{L}$, and $p_{rea}(\mathcal{A}|\mathcal{E})$ models the reasoning stage which reasons the answer $\mathcal{A}$ based on $\mathcal{E}$.
% %In our method, the retrieval and the reasoning stages are trained independently.
% However, it is intractable to exactly optimizing Eq.~(\ref{mml}) as logarithm is outside the summation when use it to compute the loss.
% As an alternative, we annotate the evidence using distant supervision.
% And then, this task can be formalized as
% \begin{align}
% % \begin{split}
% % p(\mathcal{A}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L})=p(\mathcal{A}|\mathcal{E}^{*})& \\
% % \mathcal{E}^{*}=\argmaxB_{ \mathcal{E}}p(\mathcal{E}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L}&)
% % \end{split}
% p(\mathcal{A}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L})=p(\mathcal{A}|\mathcal{E})\cdotp(\mathcal{E}|\mathcal{Q},\!\mathcal{T},\!\mathcal{L})
% \end{align}
% Specifically, the retriever and the reader can be trained independently.
% %The details of the two stages are introduced in the following sections.
% The details of these modules are introduced in the following sections.
%The details of the two stages are given in following sections.
\subsection{Stage-1: Multi-Granularity Evidence Retrieval }
In this section, we first introduce the definition of the multi-granularity evidence and then introduce the retrieval model of MuGER$^2$.
% Firstly, we setup a separate retrieval (SR) method as the baseline. Then we introduce the joint retrieval (JR) method used in our MuGER$^2$, which further benefit from the contrastive learning (CL).
% The retrievers in this stage are based on the pre-trained BERT~\cite{devlin2018bert} encoder.
% We propose multi-granularity (multi-g) evidence retrieval which decomposes retrieval procedure into multiple retrievers, which respectively retrieve atomic evidence of different granularity.
\subsubsection{ Multi-Granularity Evidence\label{concept_section}}
\label{evi}
% We propose an answer reasoning framework to address this task, and the algorithm is given in Alg~\ref{al1}.
According to the characteristics of the HQA,
we define the multi-granularity evidence as $\mathcal{E}$. $\mathcal{E}$ involves four kinds of evidence with different granularity which is denoted as
$\mathcal{E}$$=$$\langle \mathcal{E}_{\mathtt{col}}$, $\mathcal{E}_{\mathtt{row}}$, $ \mathcal{E}_{\mathtt{cell}}$, $\mathcal{E}_{\mathtt{link}}\rangle$.
% $\mathcal{E}$$=$$\langle \mathcal{E}_{\mathtt{column}}$, $\mathcal{E}_{\mathtt{row}}$, $ \mathcal{E}_{\mathtt{cell(value)}}$, $\mathcal{E}_{\mathtt{cell(link)}}$, $\mathcal{E}_{\mathtt{link}}\rangle$.
%These evidence belong to four levels of granularity.
% The multi-granularity evidence $\mathcal{E}$ is defined as $\mathcal{E}=
% \langle \mathcal{E}_{\mathtt{column}},\mathcal{E}_{\mathtt{row}}, \mathcal{E}_{\mathtt{cell(value)}}, \mathcal{E}_{\mathtt{cell(link)}}, \mathcal{E}_{\mathtt{link}}\rangle$, which contains five types of evidence.
% Note that, the five types of evidence belong to four kinds of granularity as shown in Figure~\ref{minda}.
Specifically, $\mathcal{E}_{\mathtt{col}}$ indicates the {column} which includes the answer;
$\mathcal{E}_{\mathtt{row}}$ represents the {row} that contains the answer;
The evidence $\mathcal{E}_{\mathtt{cell}}$ means the cell that contains the answer in its cell value; and $\mathcal{E}_{\mathtt{link}}$ indicates the {link} containing the answer.
\iffalse
\paragraph{Column.}
For the column evidence retriever, it aims to select a suitable header.
% The column retrieval task is to rank the table headers.
% , to assist in the reasoning of the two types of hop cell, $Thop$ and $Phop$.
To this end, we concatenate the question $\mathcal{Q}$, and a table header $h_j$ into a sequence and feed it into the retriever as the input to learn its representation.
Then we adopt a max pooling to convert the representation into a vector.
This process can be formulated as:
% The input of the retriever for this task is a sequence concatenating the question $\mathcal{Q}$, and a table header $h_j$, and represented as
\begin{equation}
\mathbf{h}_t = \mathtt{MaxP}(\mathtt{BERT}([\text{[CLS]};\mathcal{Q};\text{[SEP]};h_j]))
\end{equation}
where $\mathtt{MaxP}(\cdot)$ is the max-pooling operation.
\paragraph{Row.}
To retrieve the row that contains answer to the question, each row is scored based on rich structural information.
In this paper, we score a row with its cell that has the max similarity with the question.
In detail, each cell $c_{i,j}$ in the $i$-th row is formalized as a sequence $r(c_{i,j})$ concatenating its header $h_j$, neighbors $\{h_j,c_{i,j}\}_{j=1}^{j=n}$, and links $\mathcal{L}_{i,j}$ with a special token $\text{[SEP]}$ as follows:
\begin{equation}
r(c_{i,j})\!=\! [h_j;\!\text{[SEP]};\!\{h_j,c_{i,j}\}_{j=1}^{j=n};\!\text{[SEP]};\!\mathcal{L}_{i,j}]
\end{equation}
To represent the similarity between $c_{i,j}$ and question $\mathcal{Q}$, $r(c_{i,j})$ is concatenated with $\mathcal{Q}$, and then encoded as follows:
\begin{equation}
\mathbf{h}_t\!=\!\mathtt{MaxP}(\mathtt{BERT}([\text{[CLS]};\!\mathcal{Q};\!\text{[SEP]};\!r(c_{i,j})]))
\label{eq5}
\end{equation}
\paragraph{Cell.}
To represent a cell, we formalize the input of the retriever on task $t \in \{\mathtt{cell(value)}, \mathtt{cell(link)}\}$ as the concatenation of $\mathcal{Q}$ and $r(c_{i,j})$, and then encode them as Equation~\ref{eq5}.
Note that, since an answer may come from either the value or links of a cell, we represents a cell from two aspects, i.e., $\mathbf{h}_{\mathtt{cell(value)}}$ and $\mathbf{h}_{\mathtt{cell(link)}}$, which will further be used for answering reasoning introduced in Section~\ref{ATR}.
% \begin{equation}
% \mathbf{h}_t=\mathtt{MaxP}(\mathtt{BERT}([\text{[cls]};t;\mathcal{Q};\text{[sep]};r(c_{i,j})]))
% \end{equation}
\paragraph{Link.}
To retrieve a link $l_k \in \mathcal{L}$ that contains the answer, we represent it as the connection of the question $\mathcal{Q}$ and $l_k$. Formally, it is defined as
\begin{equation}
\mathbf{h}_t = \mathtt{MaxP}(\mathtt{BERT}([\text{[CLS]};\mathcal{Q};\text{[SEP]};l_k]))
\end{equation}
\fi
\subsubsection{Evidence Retriever}
% % Since our model leverages five kinds of evidence, the retrieval stage can be formulated as:
% To integrate the above mentioned five kinds of evidence into our method, we adopt their joint probabilities to model the evidence retriever and formulate it as follows:
% % we adopt the joint probability distribution over them to model the evidence retriever and formulate it as follows:
To integrate the above mentioned four kinds of evidence into our method, we adopt their joint probabilities to model the evidence retriever.
Formally, given a question $\mathcal{Q}$ and the multi-granularity evidence candidate $\mathcal{E}$ derived from $\mathcal{T}$ and $\mathcal{L}$, the evidence retriever can be formulated as follows:
\begin{equation}
p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})=\prod_{t}p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})
\end{equation}
where $t\in\{\mathtt{col}, \mathtt{row}, \mathtt{cell}, \mathtt{link}\}$, $\mathcal{E}_{t}$ denotes $t$-kind element of $\mathcal{E}$, and $p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$ represents the probability that $\mathcal{E}_{t}$ is the evidence.
% of the candidate evidence $\mathcal{E}_{t}$ containing the answer to the question $\mathcal{Q}$.
To compute $p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$, we further introduce a retrieval score $s_{t}$ for $\mathcal{E}_{t}$, which weighs the possibility that $\mathcal{E}_{t}$ contains the answer, and adopt it as follows:
\begin{equation}
p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})=s_{t}/Z_{t}
\end{equation}
where $Z_{t}$ is a normalization factor and equal to the sum of retrieval scores of all $t$-kind evidence.
Given that for different evidence $Z_{t}$ is consistent, we omit it in our implementation, and only use the retrieval score for evidence selection.
% For simplicity, we substitute $p_{e}(\mathcal{E}_{t})$ when referring to $p_{e}(\mathcal{E}_{t}|\mathcal{Q},\mathcal{T},\mathcal{L})$ in the following.
%$\mathbf{h}_t \in \mathbb{R}^{K}$ indicates the representation of the candidate evidence, and $\mathbf{W}_t \in \mathbb{R}^{K}$ are trainable parameters.
To compute all retrieval scores for the multi-granularity evidence $\mathcal{E}$, a straightforward method is to train four separate models and predict the different granularity evidence respectively.
However, this method neglects relations among different granularity evidence.
To tackle this challenge, we propose to adopt a unified framework for all kinds of evidence and train it in a joint way.
Specifically, we first utilize BERT~\cite{devlin2018bert} as the encoder to learn the representation for each evidence.
Formally, given evidence $\mathcal{E}_{t}$, we concatenate $t$, $\mathcal{Q}$, and $\mathcal{E}_{t}$ with ``[SEP]'' into a sequence and prepend ``[CLS]'' at its start.
Subsequently, we feed the sequence into BERT.
Finally, we adopt the max-pooling operation over all tokens of the BERT output to obtain the sequence representation $\mathbf{h}_{t}$.
This process is denoted as follows:
% we first utilize the BERT~\cite{devlin2018bert} encoder followed by the max-pooling operation to learn the representation $\mathbf{h}_{t}$ of the candidate evidence $\mathcal{E}_{t}$:
\begin{equation}
\begin{split}
\mathcal{S}_{t}=[\text{[CLS]}&;t;\text{[SEP]};\mathcal{Q};\text{[SEP]};\mathcal{E}_{t}] \\
\mathbf{h}_{t} =& \mathtt{MaxP}(\mathtt{BERT}(\mathcal{S}_{t}))
\end{split}
% \mathbf{h}_{t} = \mathtt{MaxP}(\mathtt{BERT}([\text{[CLS]};t;\text{[SEP]};\mathcal{Q};\text{[SEP]};\mathcal{E}_{t}]))
\end{equation}
where $t$ is inserted into the input sequence to indicate the kind of the evidence. After obtaining the representation of $\mathcal{E}_{t}$, we feed it into a linear projection followed by the sigmoid function to compute the retrieval score as follows, where $\mathbf{W}\in \mathbb{R}^{K}$ is a $K$-dimensional trainable parameter.
% Each of the retrievers first learns the representation $\mathbf{h}_{t}$ of the candidate evidence $\mathcal{E}_{t}$, and then calculate the binary cross entropy loss as the training objective. Each retriever is formulated as
\begin{equation}
% p_{e}(\mathcal{E}_{t})\propto(\mathbf{h}_{t}^{\top}\mathbf{W}_t)
s_{t}=\mathtt{sigmoid}{(\mathbf{h}_{t}^{\top}\mathbf{W})}
%\mathbf{h}_{t} = \mathtt{MaxP}(\mathtt{BERT}([\text{[CLS]};\mathcal{Q};\text{[SEP]};\mathcal{E}_{t}]))
\label{sr}
\end{equation}
\subsubsection{Content of Different Evidence}
For $\mathcal{E}_{t}$, it refers to different content when $t$ denotes different kinds of evidence as follows. Notably, the granularity of certain evidence is determined by what kinds of candidates are ranked, e.g., cells or rows, but NOT what content a candidate uses.
% The contents of $\mathcal{E}_{t}$ of different granularity are defined as follows:
% \paragraph{Column}
\paragraph{Column}
Column evidence aims to take the meaning of a specific column header in the table into account. The column header usually describes the fields of an answer.
Therefore, we utilize the table header as column evidence.
Namely, $\mathcal{E}_\mathtt{col}\!=\!h_j$.
% The column retrieval aims to select a suitable table header, thus we define the $\mathcal{E}_\mathtt{col} = h_j$.
\paragraph{Link}
Link evidence indicates a passage that a table cell links to.
It is another form of knowledge in the HQA task and we assign each $\mathcal{E}_\mathtt{link}$ candidate a specific passage.
\paragraph{Cell}
We formalize cell evidence $\mathcal{E}_\mathtt{cell}$ as a concatenation of its header $h_j$, its neighbors in the same row $\{h_j,c_{i,j}\}_{j=1}^{j=N}$, and its links $\mathcal{L}_{i,j}$. Namely, $\mathcal{E}_\mathtt{cell}\!=\![h_j;$ $\text{[SEP]};$ $\{h_j,c_{i,j}\}_{j=1}^{j=N};$ $\text{[SEP]};$ $\mathcal{L}_{i,j}]$, where ``[SEP]'' is the separator. In cell encoding, the above neighbor information refers to the values of other cells in the same row without their linked passages. Neighbor information and the corresponding header of a cell are used as its positional information (row, column) which is essential to distinguish cells, especially for those with the same value but in different positions.
This process is also applied to the modeling of row evidence.
\paragraph{Row} Traditional methods usually concatenate all row cells into an input sequence to score a table row evidence. However, in the HQA task, a table row may link to several passages, and directly concatenating all these contents will result in an overlength of input sequences. Therefore, we score a row based on its cells. Therefore, to encode and score a row candidate, we first encode all its cells separately and then aggregate the cell scores by maximization operation to obtain the row score.
Namely, $\mathcal{E}_\mathtt{row}$ includes a set of $\mathcal{E}_\mathtt{cell}$.
% where $\mathbf{W}\in \mathbb{R}^{K}$ means our model use a shared parameter to learn the five kinds of evidence.
%Denote $y_t \in \{0,1\}$ as the label of the candidate evidence, the training loss is calculated as follows:
%Note that, since only the answers are annotated in the HybridQA dataset, the evidence labels are obtained by distant supervision method like \citet{chen-etal-2020-hybridqa} which basically follows the principle that the positive evidence contains the answer.
\subsubsection{Training}
\paragraph{Joint Training for Multi-Granularity}
As mentioned above, we only adopt the retrieval score for the evidence selection.
Therefore, the core of training of evidence retriever is to learn a model to produce the retrieval score.
Since the retrieval score $s_{t}$ means the probability that the evidence $\mathcal{E}_{t}$ contains the answer, we treat the retrieval score prediction as a binary classification and use the binary cross entropy (BCE) loss as the training objective to learn the model as follows:
% To train the unified retriever, an intuitive method is to treat the five retrieval tasks as binary classification and use the binary cross entropy (BCE) loss as the training objective.
% For each kind of evidence, the probability $p_{e}(\mathcal{E}_{t})$ is first obtained from the retriever, then the binary cross entropy loss of $\mathcal{E}_{t}$ is calculate as follows:
\begin{equation}
\mathtt{Loss}_t^{bce}= {\mathtt{BCE}(s_{t}, y_{t})}
\label{all_loss}
\end{equation}
where $y_t \in \{0,1\}$ is the label of $\mathcal{E}_{t}$ indicating whether the evidence $\mathcal{E}_{t}$ contains the answer. Note that, since only the answer texts are annotated in the HybridQA dataset, we obtain the evidence labels by distant supervision method like \citet{chen-etal-2020-hybridqa} which basically follows the principle that the positive evidence contains the answer.
Furthermore, to consider the relative information among different granularity evidence, we propose to leverage a unified retriever to model them and learn it in a joint way:
% Therefore, the final objective function can be formulated as:
% the unified retriever learns the five retrieval tasks in a joint way. Therefore, the training objective of the unified retriever can be formulated as
\begin{equation}
\mathtt{Loss}^{bce}= \sum_{t}{\mathtt{Loss}^{bce}_t}
\label{bin}
\end{equation}
\paragraph{Contrastive Learning within Each Granularity}
The above joint training method is capable of capturing the relation among different granularity evidence.
However, it neglects the relations among different candidate evidence within the same granularity.
Because it depends on the BCE loss, which regards each candidate evidence as an individual instance and does not require any involvement of other instances besides the given evidence.
% Although the above joint training way can capture the relations of cross-granularity evidence, the binary classification loss still regards each item of evidence candidates as individuals, which misses the relations of inner-granularity evidence.
% To alleviate this problem, we propose to leverage contrastive learning (CL) ~\cite{1640964} among inner-granularity evidence candidates to model their relative similarities and further enhance the model's representation and scoring capability.
To alleviate this problem, we propose to leverage contrastive learning (CL) ~\cite{1640964} within candidate evidence with the same granularity to model their relative similarities and further enhance the model's representation and scoring capability.
To construct the training objective, we first define a function $\mathtt{sim}(\cdot)$ to denote the similarity between different evidence as follows:
\begin{align}
\mathtt{sim}({\mathbf{h}_t},{\mathbf{h}_{t}^{'}})= \mathbf{h}_t \mathbf{W} + \mathbf{h}_{t}^{'}\mathbf{W}
\end{align}
% where $\mathbf{W}$ is a trainable parameter.
It also requires building positive and negative instances for a given evidence candidate.
For positive instance, following ~\citet{gao2021simcse}, we set $\mathcal{E}_{t}^{+}\!=\!\mathcal{E}_{t}$.
Since there are dropout masks placed on fully-connected layers as well as attention probabilities during training of BERT,
feeding the same input to the encoder twice we get two different representations $\mathbf{h}_t$ and $\mathbf{h}_t^{+}$ for it.
To obtain the negative samples $\mathcal{E}_{t}^{-}$ for the given evidence, like most existing work with contrastive learning~\cite{gao2021simcse}, we regard other evidence in the same batch as negative ones and the representation of $\mathcal{E}_{t}^{-}$ is denoted as $\mathbf{h}_t^{-}$.
After that, we take a cross-entropy objective with in-batch negatives.
Formally, given a batch, we split it into four groups and each one corresponds to a granularity.
Let $D$ denote all instances in a group, the training objective is:
\begin{align}
\label{loss2}
\mathtt{Loss}_{t}^{cl}=- \mathtt{log}\frac{e^{\mathtt{sim}(\mathbf{h}_t ,\mathbf{h}_t^{+})/ \tau }}{\sum_{\mathbf{h}_t^{-} \in D}e^{\mathtt{sim}(\mathbf{h}_t,\mathbf{h}_t^{-})/ \tau } }
\end{align}
where $\tau$ is a temperature hyperparameter.
Actually, we compute four loss as Eq.~(\ref{loss2}) based on four groups and jointly learn them as follows:
\begin{align}
\mathtt{Loss}^{cl}=\sum_{t}\mathtt{Loss}_{t}^{cl}
\end{align}
Finally, we combine $\mathtt{Loss}^{bce}$ and $\mathtt{Loss}^{cl}$ to produce the final objective function:
\begin{align}
\mathtt{Loss}=\mathtt{Loss}^{bce}+\mathtt{Loss}^{cl}
\end{align}
\iffalse
Denote the training objectives of the contrastive learning of task $t$ as $\mathtt{l}_{t}^{cl}$, the final loss function for training the unified retriever is formalized as
% Denoting the training objectives of the contrastive learning as $loss_{t}^{cl}$, for an instance of task $t$, the final loss function of the joint retrieval is formalized as
% Equation~\ref{lo}:
\begin{equation}
\label{lo}
\mathtt{l}^{ret} = \mathtt{l}^{bce} + \sum_{t}\mathtt{l}_{t}^{cl}
\end{equation}
The details of obtaining $\mathtt{l}_{t}^{cl}$ is introduced in the following.
% The loss function of the JR process is defined as $L=\sum_{t}\mathtt{loss}_{t}^{jr}$ over all the retrieval tasks.
\paragraph{Contrastive Learning}
The training of our unified retriever is further benefit from the contrastive learning (CL). The contrastive learning among inner-granularity evidence candidates modeling their relative similarities may further enhance the model's representation and scoring capability.
Typically, in contrastive learning, a set of evidence pairs containing a semantically related pair as positive and multiple unrelated pairs as negative are required.
For evidence represented as $\mathbf{h}_t$, assuming its positive evidence is represented as $\mathbf{h}_t^{+}$ and negative evidence as $\mathbf{h}_t^{-} \in D$, the training objective is formalized as follows:
\begin{equation}
\label{loss2}
\mathtt{l}_{t}^{cl}=- \mathtt{log}\frac{e^{sim(\mathbf{h}_t ,\mathbf{h}_t^{+})/ \tau }}{\sum_{\mathbf{h}_t^{-} \in D}e^{sim(\mathbf{h}_t,\mathbf{h}_t^{-})/ \tau } }
\end{equation}
where $\tau$ is a temperature hyperparameter and $\mathtt{sim}(\cdot)$ denotes the similarity between the representation pair which is defined as follows:
\begin{equation}
\mathtt{sim}({\mathbf{h}_t},{\mathbf{h}_t^+})= \mathbf{h}_t \mathbf{W} + \mathbf{h}_t^+\mathbf{W}
\end{equation}
To obtain the contrastive loss, we construct the representation pairs as follows.
Specifically, a training batch is subdivided into four groups for the four types of evidence.
Inspired by \cite{gao2021simcse}, each group contains two \textbf{same} instances and multiple different instances.
The two same instances are regarded as the positive pair which will be converted into different representations ${\mathbf{h}_t}$ and ${\mathbf{h}_t^+}$ based on the dropout operation.
% In this way, the Equation~\ref{loss2} is executed inside each mini-batch to obtain the CT loss.
\fi
% In this work, ($\bm h,\bm{\Bar{h}}$) is defined as the representation pair of the question and a heterogeneous data such as a table cell or a passage. The similarity between them is calculated by the multi-layer self-attention mechanism in the BERT model.
% During training, we divide a training batch into five mini-batches respectively for the five reasoning tasks. For each mini-batch, we make it only contain one positive input sequence with label $l=1$, and the other input sequences in the mini-batch are all negative. The loss value of the mini-batch is calculated by the function~\ref{loss2}.
% \end{itemize}
% =\prod_{t}p_{min}(\mathcal{C}_{t}|\mathcal{Q},\mathcal{T},\mathcal{P},t)
% =\langle \mathcal{C}_{anc}, \mathcal{C}_{thop}, \mathcal{C}_{phop}, \mathcal{C}_{p}, \mathcal{C}_{h}\rangle
\subsection{Stage-2: Answer Reasoning with the Multi-Granularity Evidence}
As shown in Figure~\ref{minda}, answer reasoning module $p_{rea}(\mathcal{A}|\mathcal{E})$ is responsible for deriving answer based on the output of the evidence retriever.
This module consists of two essential components, evidence selector and reader.
Algorithm~\ref{al1} shows the whole workflow of this module.
% Algorithm~\ref{al1} shows the answer reasoning procedure $p_{rea}(\mathcal{A}|\mathcal{E})$ which aims to leverage the multi-granularity evidence distribution $p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})$ (line 1) to derive natural language answer $\mathcal{A}$. We introduce the two components of the reasoning stage, E-SEL and RC, in the following sections.
\subsubsection{Evidence Selector\label{ATR}}
% Although the multi-granularity evidence achieves a high retrieval recall, its redundant information causes weak RC performance.
As analysed above, multi-granularity evidence is easier to be accurately scored by the retriever.
However, they contain too much redundant information and brings burden to the reader.
% To alleviate this issue, the evidence selector (E-SEL) is proposed to fuse the multi-granularity evidence.
To alleviate this issue, the evidence selector (E-SEL) is proposed to navigate the fine-grained evidence for the reader based on the multi-granularity evidence.
% Specifically, we assign each cell two scores, $s_{tab}^{i,j}$ and $s_{pass}^{i,j}$, to respectively indicate the probability that the cell value and the cell links contain the answer.
Specifically, we define two scores $s_{tab}^{i,j}$ and $s_{pass}^{i,j}$ to respectively indicate the probabilities that the answer is in the cell $c_{i,j}$ and the links $\mathcal{L}_{i,j}$.
% These two scores are calculated by considering multiple granularity information, including the cell itself and the row and column in which the cell is located in.
When calculating the scores, we take the multi-granularity evidence into account. Suppose that the link set $\mathcal{L}_{i,j}$ has $X$ links denoted as $\{l_x, x\in[1,X]\} $.
The $s_{tab}^{i,j}$ and $s_{pass}^{i,j}$ are calculated as follows:
%Specifically, evidence retriever assigns each evidence candidate a retrieval score.
%Namely,for each table cell $c_{i,j}$, it is attached with several scores, including $s_{col}^{j}$, $s_{row}^{i}$, $s_{cval}^{i,j}$, $s_{clink}^{i,j}$, etc.
%%They can be regarded as weights from different aspects to measure the possibility that $c_{i,j}$ contains the answer.
%We adopt some of them to compute two scores for $c_{i,j}$ as follows:
\begin{align}
\begin{split}
s_{tab}^{i,j} &= s_{\mathtt{col}}^{j}+s_{\mathtt{row}}^{i}+s_{\mathtt{cell}}^{i,j} \\
s_{pass}^{i,j} &= s_{\mathtt{col}}^{j}+s_{\mathtt{row}}^{i}+\mathtt{Max}_{x=1}^{x=X}{(s_{\mathtt{link}}^{x})}
\end{split}
\label{esel}
\end{align}
where $s_{col}^{j}$, $s_{row}^{i}$, $s_{cell}^{i,j}$, and $s_{link}^{x}$ are retrieval scores.
% After that, we further obtain two candidate target cells with the global maximum scores $s_{tab}$ and $s_{pass}$:
% After that, E-SEL is able to judge the answer type based on these scores as follows:
After that, we can obtain two global maximum scores $s_{tab}$ and $s_{pass}$ as follows:
\begin{align}
\begin{split}
s_{tab}&=\mathtt{Max}_{i=1,j=1}^{i=M,j=N}(s_{tab}^{i,j}) \\
s_{pass}&=\mathtt{Max}_{i=1,j=1}^{i=M,j=N}(s_{pass}^{i,j})
\end{split}
\end{align}
%We regards these two scores as probabilities that the answer is within the table and passages.
These two scores, $s_{tab}$ and $s_{pass}$, indicate that the answer is in the table or passages by simply comparing their values.
Meanwhile, the cell and link corresponding to them are considered as the navigated evidence for the reader.
Based on $s_{tab}$ and $s_{pass}$, E-SEL is able to judge the answer type of the current question as follows:
\begin{equation}
\mathtt{E\text{-}SEL}(p_{ret}(\cdot))=
\left\{
\begin{aligned}
& \mathtt{In}\text{-}\mathtt{Table}, & s_{tab} > s_{pass} \\
& \mathtt{In}\text{-}\mathtt{Passage}, & s_{tab} \leq s_{pass}
\end{aligned}
\right.
\end{equation}
where $\mathtt{In}\text{-}\mathtt{Table}$ denotes that the answer to the question is in the cell, while $\mathtt{In}\text{-}\mathtt{Passage}$ indicates that the answer is in the link.
% , i.e., $\mathtt{In}\text{-}\mathtt{Table}$ and $\mathtt{In}\text{-}\mathtt{Passage}$.
% To this end, we first compute two scores for cell $c_{i,j}$ as follows:
% Afterwards, the fine-grained cell or link is selected based on the answer type and the scores. If the answer is in $\mathtt{In}\text{-}\mathtt{Table}$, the cell value of $c^1$ with score $s^1$ is returned as the answer, otherwise (line 6-10) the top-1 link of cell $c^2$ with score $s^2$ is selected to derive answer by the RC model.
After obtaining the answer type, we can navigate the cell or link as the final fine-grained evidence to derive the answer.
Formally, if the type is $\mathtt{In}\text{-}\mathtt{Table}$, the cell with score $s_{tab}$ will be selected and its value will be returned as the answer.
Otherwise, the link with the score $s_{pass}$ will be selected and it will be feed to the reader to extract answer.
\subsubsection{Reader}
% \subsubsection{Reading Comprehension}
% Following previous work~\cite{chen-etal-2020-hybridqa,sun2021end},
% We adopt BERT to train the span-based reading comprehension (RC) models like ~\cite{chen-etal-2020-hybridqa,sun2021end}.
% An RC model $\mathtt{RC}(\mathcal{Q},{p})$ derives answer by extracting a span from the retrieved link $p$ as answer based on the given question $\mathcal{Q}$ (line 9).
For the reader $\mathtt{RC}(\mathcal{Q},{l})$, following existing works~\cite{chen-etal-2020-hybridqa,sun2021end}, we implement it with a span-based reading comprehension model which adopt BERT as backbone.
\begin{algorithm}[t]
\caption{Answer Reasoning}
\label{al1}
\SetKwData{Index}{Index}
% \LinesNumbered
\KwIn{question $\mathcal{Q}$, table $\mathcal{T}$,
links $\mathcal{L}$}
\KwOut{answer text $\mathcal{A}$}
calculate $p_{ret}(\mathcal{E}|\mathcal{Q},\mathcal{T},\mathcal{L})$\;
\If {$\mathtt{E\text{-}SEL}(p_{ret}(\cdot))$==$\mathtt{In}\text{-}\mathtt{Table}$}
{
get the cell $c_{tab}$ with score $s_{tab}$\;
return cell value of $c_{tab}$ as answer $\mathcal{A}$
}
\ElseIf {$\mathtt{E\text{-}SEL}(p_{ret}(\cdot))$==$\mathtt{In}\text{-}\mathtt{Passage}$}
{
get the link $l_{pass}$ with score $s_{pass}$\;
% $l$ top-1 link of $c_{i,j}$ based on $p_{ret}(\cdot)$\;
% $l$ is the passage which $c_{i,j}$ links to\;
return $\mathtt{RC}(\mathcal{Q},l)$ as answer $\mathcal{A}$
}
\end{algorithm}
\begin{table*}[ht]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\small
\linespread{1.5}
% \resizebox{0.99 \textwidth}{!}{
% \setlength{\tabcolsep}{1mm}{
\begin{tabular}{l|ccc|ccc}
\toprule
\toprule
\specialrule{0em}{1pt}{1pt}
\multirow{3}{*}{\bf Model} & \multicolumn{3}{c|}{\bf Dev} & \multicolumn{3}{c}{\bf Test} \\
\specialrule{0em}{1pt}{1pt}
\cline{2-7}
\specialrule{0em}{1pt}{1pt}
& In-Table & In-Passage & Total & In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-7}
\specialrule{0em}{1pt}{1pt}
& EM / F1 & EM / F1 & EM / F1 & EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
MQA-QG (unsupervised) &-- / --& -- / -- & -- / -- &36.2 / 40.6& 19.8 / 25.0 & 25.7 / 30.5 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
Table-Only& 14.7 / 19.1 & 2.4 / 4.5 & 8.4 / 12.1 & 14.2 / 18.8 & 2.6 / 4.7 & 8.3 / 11.7\\
Passage-Only& 9.2 / 13.5 & 26.1 / 32.4 & 19.5 / 25.1 & 8.9 / 13.8 & 25.5 / 32.0 & 19.1 / 25.0 \\
% BM25 + ETC& -- / -- & -- / -- & 24.8 / 29.1 &-- / --& -- / --& 25.5 / 31.1 \\
% Dense + ETC& -- / --& -- / --& 37.0 / 43.5 &-- / --& -- / --& 34.1 / 40.3 \\
% Sequential (ETC)& -- / --& -- / --& 39.4 / 44.8 &-- / --& -- / --& 37.0 / 43.0\\
Hybrider (BERT-base, $\tau$=0.9)& 51.5 / 58.6 & 40.5 / 47.9 & 43.7 / 50.9 &52.1 / 59.3 &38.1 / 46.3 & 42.5 / 50.2\\
Hybrider (BERT-large, $\tau$=0.8)& 54.3 / 61.4 & 39.1 / 45.7 & 44.0 / 50.7 &56.2 / 63.3 &37.5 / 44.4 & 43.8 / 50.6\\
Dochopper & -- / --& -- / -- & 47.7 / 55.0 & -- / -- &-- / --& 46.3 / 53.3\\
% MATE & 68.6 / 74.2 & 62.8 / 71.9 & \bf 63.4 / \bf 71.0 & 66.9 / 72.3& 62.8 / 71.9 &\bf 62.8 / \bf 70.2 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
% Ours (top1hop w/o pred00) & 54.5 / 61.9 & 48.6 / 58.7 & 49.6 / 58.5 &-- / -- &-- / -- & -- / --\\
% Ours (top2hop w/o pred00) & 54.4 / 61.9 & 50.6 / 61.0 & 50.8 / 59.9 &-- / -- &-- / --& -- / --\\
% Ours (top3hop w/o pred00) & 54.7 / 62.2 & 51.1 / 61.7 & 51.2 / 60.3&-- / -- &-- / --& -- / --\\ \hline
% MINDA (RC-base) & 58.3 / 66.3 & 52.3 / 63.0 & \bf 53.3 / \bf 62.7&56.1 / 63.7 &51.6 / 63.0& \bf 52.1 / \bf 61.7 \\
%MERDR (RC-base) & 56.9 / 64.9 & 52.3 / 63.0 & \bf 52.7 / \bf 62.2&54.7 / 62.3 &51.3 / 62.6& \bf 51.4 / \bf 61.0\\
MuGER$^2$-base & 58.2 / 66.1 & 52.9 / 64.6 & 53.7 / 63.6&56.7 / 64.0 &52.3 / 63.9 & 52.8 / 62.5\\
% \ \ \ \ \ w/o constractive learning &56.6 / 64.2 &50.7 / 61.2 &51.7 / 60.8&-- / -- &-- / --& -- / -- \\
% \ \ \ \ \ w/o DA & 38.6 / 47.0 &52.7 / 63.8& 45.9 / 55.7 &-- / -- &-- / --& -- / --\\
%\ \ \ \ \ w/o iterative determination & 54.7 / 62.2 & 51.5 / 62.0 & 51.4 / 60.5 &-- / --&-- / -- &-- / --\\
% MINDA (RC-large)& 58.6 / 66.7 &54.1 / 65.2 &\bf 54.5 / \bf 64.2&56.4 / 64.1 &54.6 / 65.5& \bf 54.0 / \bf 63.3 \\
% MERDR (RC-large)& 57.4 / 65.3 &53.9 / 64.9 &\bf 53.9 / \bf 63.5&54.8 / 62.5&53.8 / 64.6 &\bf 52.8 / \bf 62.2\\
%\textbf{MuGER$^2$-base (RC-Large)}& 58.9 / 66.6 &55.4 / 66.9 &\bf 55.4 / \bf 65.1&56.9 / 64.2 &55.9 / 66.8& \bf 54.8 / \bf 64.1\\
%\textbf{MuGER$^2$-large (RC-base)}& 60.6 / 69.0 &53.2 / 65.5 &\bf 54.8 / \bf 65.3&58.0 / 66.0 &53.6 / 65.5& \textbf{54.0} / \textbf{64.1}\\
MuGER$^2$-large & 60.9 / 69.2 &56.9 / 68.9 &\bf 57.1 / \bf 67.3&58.7 / 66.6 &57.1 / 68.6& \textbf{56.3} / \textbf{66.2}\\
% \ \ \ \ \ w/o MIN& 52.3 / 60.5 &51.5 / 62.5 & 50.5 / 60.2 &-- / -- &-- / --& -- / --\\
% \ \ \ \ \ w/o constractive learning & 57.2 / 64.8 &52.1 / 63.0 & 52.7 / 62.1&-- / -- &-- / --& -- / -- \\
% \ \ \ \ \ w/o DA & 42.0 / 50.3 &54.2 / 65.5 & 48.1 / 58.0 &-- / -- &-- / --& -- / --\\
% \ \ \ \ \ w/o iterative determination & 55.5 / 62.7 &53.1 / 64.2& 52.6 / 62.0 &-- / -- &-- / --& -- / --\\
\specialrule{0em}{1pt}{1pt}
\bottomrule \bottomrule
\end{tabular}
% }}
\caption{The EM and F1 scores on HybridQA of different models.}
\label{main}
\end{table*}
\begin{table}[ht]
\small
\setlength{\belowcaptionskip}{-0.3cm}
\centering
\begin{tabular}{l|cccc}
\toprule
\toprule
\specialrule{0em}{1pt}{1pt}
\textbf{Split}&\textbf{Train}&\textbf{Dev}&\textbf{Test}&\textbf{Total}\\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
In Passage&35,215&2,025&2,045 &39,285\\
In Table&26,803&1,349&1,346& 29,498\\
Compute&664&92&72&864 \\
Total&62,682&3,466&3,463&69,611 \\
\bottomrule
\bottomrule
\end{tabular}
\caption{Statistics of HybridQA dataset.}
\label{tab}
\end{table}
\section{Experiments}
\subsection{Implementation Details}
\subsubsection{Dataset}
To verify our proposed model, we conduct experiments on HybridQA\footnote{\url{https://github.com/wenhuchen/HybridQA}} \cite{chen-etal-2020-hybridqa}, a dataset of multi-hop question answering
over tabular and textual data.
% For each question, HybridQA provides a WiKiTable along with its hyperlinked Wikipedia passages as the knowledge.
% HybridQA provides a WiKiTable along with its hyperlinked Wikipedia passages as the knowledge to answer a question.
The basic statistics of HybridQA are listed in Table~\ref{tab}.
In the data split, `In-Table' means the answer is a table cell value, and `In-Passage' means the answer exists in a linked passage.
`Compute' means the answer is computed by performing numerical operations. In this paper, we mainly focus on the first two types.
\subsubsection{Settings} Two variants of
BERT \cite{devlin2018bert} are utilized in MuGER$^2$, namely MuGER$^2$-base and MuGER$^2$-large. The learning rate is set to $5$e-$5$ and the training batch size is set to $30$.
% In the unified evidence retriever, a training batch is subdivided into $5$ mini-batches for five types of evidence with a mini-batch size of $6$.
% Each mini-batch contains $2$ same positive instances and $4$ sampled negative instances for contrastive learning.
A batch is divided into 4 groups corresponding to the 4 kinds of evidence and each group contains 6 instances.
\subsection{Comparison with Previous Methods}
\subsubsection{Baselines}
\textbf{MQA-QG} \cite{pan2020unsupervised} is an unsupervised framework that generates multi-hop questions from the hybrid evidence and uses the generated questions to train the QA model.
\textbf{Table-Only} \cite{chen-etal-2020-hybridqa} only relies on the tabular information to find the answer by parsing the question into a symbolic form and executes it.
\textbf{Passage-Only} \cite{chen-etal-2020-hybridqa} only uses the hyperlinked passages to find the answer by retrieving related passages to perform the RC process.
\textbf{Hybrider} \cite{chen-etal-2020-hybridqa} solves the HybridQA using a two-stage pipeline framework to retrieve a table cell and extract the answer in its value or linked passages.
\textbf{Dochopper} \cite{sun2021end} builds a retrieval index for each cell and passage sentence, and perform RC on the retrieved evidence.
\subsubsection{Results and Analysis}
% We use the EM and F1 as the evaluation metrics to compare the question-answering performance of our MuGER$^2$ and the baseline models.
We use the EM and F1 as the evaluation metrics to compare the performance of our MuGER$^2$ and that of baselines.
As shown in Table~\ref{main}, MuGER$^2$-base outperforms strong baselines of HybridQA task and achieves significant improvements on both development and test sets.
Specifically, its performance exceeds the Hybrider-larger whose parameter scale is larger than it.
This indicates that our proposed multi-granularity evidence is beneficial for this task.
In addition, compared with the base model, MuGER$^2$-large further obtains gains.
% Both of them indicate that our proposed multi-granularity evidence is beneficial for this task.
%In the ablation studies, we respectively show the effectiveness of each modules in our framework. All the ablation experiments are conducted on the develop set of HybridQA.
\subsection{Ablation Study}
\subsubsection{Stage-1: Retrieval}
\paragraph{Effect of Contrastive Learning}Without the contrastive learning (CL) within each granularity of evidence, the unified retriever is only trained by the binary cross entropy objective in the joint way.
The results in Table~\ref{cl} demonstrate that CL brings $0.8\%$ EM and $1.1\%$ F1 improvements to our model. These improvements are from the ability of CL to enhance the evidence representation learning by pushing away the positive and negative evidence representation in the semantic space.
\paragraph{Effect of Joint Training}
In our method, we adopt the joint training (JT) method to train the unified retriever to capture the relative information among different granularity. To prove the effectiveness of this design, we compare it with a straightforward way which trains four separate retrievers for different granularity evidence. For fairness, we turn off the contrastive learning and only use the binary cross entropy objective to train the four separate retrievers and the unified retriever. The results in Table~\ref{cl} show that JT brings $1.6\%$ EM and $1.3\%$ F1 improvements to the HQA performance.
%The reason is that JR allows the learning of different granularity evidence share the learned knowledge.
Besides the end-to-end HQA performance, we further give the retrieval recall of the four different granularity evidence and the complete multi-granularity evidence in Figure~\ref{recall}. The retrieval recall (R@1) indicates whether the retrieved top-$1$ candidate of the corresponding evidence contains the answer.
The results show that JT improves the retrieval performance on all these retrieval tasks. It is proved that sharing knowledge from different granularity brings benefit to the evidence retrieval.
%\subsubsection{Retrieval Recall of Different Granularity Evidence}
%Moreover, the multi-granularity evidence achieves the highest retrieval recall due to its higher answer coverage which achieves $0.98$.
%For the mono-granularity evidence retrieval, the finer the granularity, the lower the recall.
%In addition, the retrieval recall of the cell(value) is significantly lower than the other types of evidence.
%This is because the cell(value) evidence usually has numeric values and has no linked passages, which makes it difficult to retrieve from huge candidates based on limited semantic information.
%It is observed that the improvement is more significant on the in-table questions because such questions have less retrieval error tolerance than in-passage questions since sometimes the answer to in-passage questions are mentioned in multiple passages.
\begin{table}[t]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\small
\linespread{1.8}
% \resizebox{0.49 \textwidth}{!}{
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{l|ccc}
\toprule
\toprule
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf Model}
&In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-4}
\specialrule{0em}{1pt}{1pt}
&EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
MuGER$^2$-base & 58.2 / 66.1 & 52.9 / 64.6 & \bf 53.7 / \bf 63.6 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
\ \ \ w/o CL &59.1 / 66.9 &50.6 / 62.2 &52.9 / 62.5 \\
\ \ \ \ \ \ \ \ \ w/o JT & 54.2 / 61.9 &51.5 / 63.2& 51.3 / 61.2\\
% % \hline
% % \ \ \ \ \ \ w/o CD in DA & 53.7 / 61.6 & 53.5 / 64.3 & 52.2 / 61.6 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
\ \ \ w/o E-SEL & 38.6 / 47.0 &52.7 / 63.8& 45.9 / 55.7\\
% \ w/o ATR & 42.6 / 49.7 &46.9 / 57.3& 44.0 / 53.0\\
% \specialrule{0em}{1pt}{1pt}
% \hline
% \specialrule{0em}{1pt}{1pt}
% \bf MuGER$^2$ & 58.9 / 66.6 &55.4 / 66.9 &\bf 55.4 / \bf 65.1 \\
% \hline
% \ w/o CT & 59.6 / 66.7 &53.3 / 64.5 & 54.4 / 63.8 \\
% \ w/o CT and JT& 54.6 / 62.2 &53.9 / 65.3 & 52.8 / 62.5\\
% \hline
% \ \ \ \ \ \ w/o CD in DA & 54.2 / 62.1 & 55.4 / 66.6 & 53.5 / 63.2 \\
% \ \ \ \ \ \ w/o DA & 42.0 / 50.3 &54.2 / 65.5 & 48.1 / 58.0\\
% \ w/o ATR & 51.2 / 58.3 &53.7 / 65.1 & 51.4 / 60.9\\
% \ w/o ATR & 47.1 / 54.9 &51.3 / 62.2 & 48.4 / 57.8\\
\specialrule{0em}{1pt}{1pt}
\bottomrule
\bottomrule
\specialrule{0em}{1pt}{1pt}
\end{tabular}
}
\caption{Ablation results based on MuGER$^2$-base.}
\label{cl}
\end{table}
\begin{figure}[t]
\setlength{\belowcaptionskip}{-0.3cm}
\centering
\includegraphics[width=0.48\textwidth]{fig/jt.pdf}
% \caption{Retrieval precision (P) of each granularity of evidence in MuGER$^2$.}
\caption{R@1 of different granularity evidence.}
\label{recall}
\end{figure}
\begin{comment}
\begin{table}[t]
\centering
\small
\linespread{1.8}
\begin{tabular}{c|c|cc}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\multicolumn{2}{c|}{\bf Question Type} &\bf In-T(1349) & \bf In-P (2025) \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf w/ DR}& \bf In-T & \bf 1148 (85.1\%) &173 (8.5\%) \\
& \bf In-P&201 (14.9\%) &\bf 1852 (91.5\%) \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf w/o DR}& \bf In-T & \bf 432 (32.0\%) &12 (0.6\%) \\
& \bf In-P&917 (68.0\%) &\bf 2013(99.4\%) \\
\specialrule{0em}{1pt}{1pt}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\end{tabular}
\caption{Confusion matrix of question type discrimination.}
\label{confuse}
\end{table}
\end{comment}
\begin{comment}
\begin{table}[t]
\centering
\small
\linespread{1.8}
\begin{tabular}{c|ccc}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf Derivation}
&In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-4}
\specialrule{0em}{1pt}{1pt}
&EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
\bf Top1-Thop & 57.7 / 66.0 & 0.0 / 2.1 & 22.5 / 26.9 \\
\bf Top1-Phop & 17.2 / 22.9 & 53.7 / 64.6 & 38.2 / 46.8 \\
\bf Phops w/ CD & 14.6 / 18.1 & 54.6 / 65.5 & 37.6 / 45.4 \\
\bf DA & 57.4 / 65.3 &53.9 / 64.9 &\bf 53.9 / \bf 63.5\\
\specialrule{0em}{1pt}{1pt}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\end{tabular}
\caption{Different answer derivation ways.}
\label{dis}
\end{table}
\end{comment}
\subsubsection{Stage-2: Reasoning}
\paragraph{Effect of Evidence Selector} To show the effectiveness of the evidence selector (E-SEL), we compare it with a straightforward answer reasoning method without any fine-grained evidence navigation. This method simply flattens the retrieved top-1 multi-granularity evidence and feeding the sequence into the RC process. Results in Table~\ref{cl} show that the evidence selector increases the results by $7.8\%$ EM and $7.9\%$ F1 compared with the above reasoning method. This is due to its ability of accurately navigating the cell or link based on the multi-granularity evidence. The evidence selector reduces the burden of the RC model. In contrast, the above straightforward method brings redundant and noisy information which leads to weak RC performance.
%The setting without DR first obtains the top1 hop cell by adding the retrieval score of $thop$ and $phop$ of each cell from MER, and then extracts the answer from the cell content or passages only according to whether the top1 hop cell has hyperlinked passage or not. If the top1 hop cell has hyperlinked passage, its cell content is prepended to the passage for reading comprehension. We compare the results of our MERDR with or without DR in Table~\ref{cl}. It is shown that DR increases $8.1\%$ EM and $7.8\%$ F1 for MERDR (RC-base), and $7.4\%$ EM and F1 for MERDR (RC-large). These improvements are mainly from two aspects. Firstly, the QTD in DR reduces loss in answering in-table questions. That is because in the setting without DR, some in-table questions are also carried out reading comprehension since its hop cell has hyperlinked passages. Secondly, the ED in DR has the ability to judge whether the retrieved results are correct or not and performs a re-retrieve to increase the answer reasoning accuracy.
%\subsection{Case Study}
%In addition to the QA performance, we also give the confusion matrix as Table~\ref{confuse}, to clearly show the question type discrimination performance of DR. The matrix shows that DR correctly discriminates $85.1\%$ in-table questions and $91.5\%$ in-passage questions. By contrast, in the setting without DR, most questions are answered by using reading comprehension since most of the hop cells have hyperlinked passages. This leads to high coverage of in-passage questions, but very few in-table questions are directly answered using the table cell. The confusion matrix further proves the capability and necessity of the question-type discrimination operation in our proposed DR.
%To further explore the capability of our DA, we give three upper bound results based on the grounding information from MIN in Table~\ref{dis}: (1) Answer all questions using the \textbf{Top1-Thop}. (2) Answer all questions using the \textbf{Top1-Phop}. (3) Answer all questions using the \textbf{Phops} with the iterative discrimination (ID).
%Obviously, using the top1 Thop is almost impossible to solve the in-passage questions, but using top1 Phop can solve some of in-table questions because the answers of in-table questions are often mentioned in hyperlinked passages simultaneously. In addition, using the Phops with ID improves the performance on in-passage questions by correcting the Phop grounding error. However, in this case, the discrimination makes the answers of some in-table questions lose the opportunity to be extracted from the passages, which leads to lower total performance. In general, our DA achieves better performances on in-table questions than using the top1-thop because some of their answers are extracted from passages. DA also outperforms only using the top1 Phop on in-passage questions thanks to ID. Compared to using Phops with ID,
%DA performs slightly worse on in-passage questions, these losses are caused by the misjudgment of the question type discrimination.
\begin{table}[t]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\small
\linespread{1.8}
\begin{tabular}{l|ccc}
\toprule
\toprule
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf Granularity}
&In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-4}
\specialrule{0em}{1pt}{1pt}
&EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\midrule
\specialrule{0em}{1pt}{1pt}
% Table & 12.8 / 18.2 & 11.6 / 15.4 & 12.4 / 16.8 \\
Column & 10.4 / 16.9 & 17.1 / 22.9 & 14.1 / 20.1 \\
Row &52.0 / 59.3 &49.6 / 60.1 & 49.3 / 58.3 \\
Cell& 21.6 / 29.6 &0.0 / 1.4 & 8.4 / 12.3\\
Link & 13.0 / 18.3 &40.6 / 50.1& 28.8 / 36.5\\
Multi (ours) & 58.2 / 66.1 & 52.9 / 64.6 & \bf 53.7 / \bf 63.6 \\
% \specialrule{0em}{1pt}{1pt}
% \hline
% \specialrule{0em}{1pt}{1pt}
% Table-g & 14.8 / 20.7 & 12.9 / 17.2 & 14.0 / 18.8 \\
% Row-g &53.3 / 60.6 &52.4 / 62.9 &51.4 / 60.4 \\
% Cell-g & 23.6 / 29.8 &41.4 / 51.4& 33.4 / 41.8\\
% Link-g & 14.0 / 19.1 &41.7 / 51.8& 29.9 / 37.8\\
% \bf MuGER$^2$ & 58.9 / 66.6 &55.4 / 66.9 &\bf 55.4 / \bf 65.1 \\
\specialrule{0em}{1pt}{1pt}
\bottomrule
\bottomrule
\specialrule{0em}{1pt}{1pt}
\end{tabular}
\caption{HQA results of different granularity evidence.}
\label{evidence-g}
\end{table}
\subsection{Different Granularity Evidence\label{baselines}}
\subsubsection{Baselines}
To compare the HQA performance based on different granularity evidence, we conduct baselines for the four kinds of evidence including column, row, cell, and link. Specifically, the baselines of column, row and link granularity first retrieve the top-1 evidence of the selected granularity, and then flatten the retrieved evidence into a sequence to derive the answer by the RC model. While for the cell granularity, the value of the retrieved top-1 cell is directly returned as the answer. Note that, all the retrievers and the RC models of these baselines are initialized by the pre-trained BERT-base model.
\begin{figure*}[t]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\includegraphics[width=0.99\textwidth]{fig/cases.pdf}
\caption{An example of evidence retrieval and answer reasoning results based on different granularity evidence.}
\label{muger-case}
\end{figure*}
\subsubsection{Results and Analysis}
The end-to-end HQA results based on different granularity evidence are shown in Table~\ref{evidence-g} and Figure~\ref{motivation}.
The coarse-grained \textbf{Column} evidence achieves the best retrieval recall. This is because the question domains are easily mapped to a certain table header. However, reasoning the answer in a column is difficult because cells in a column are nearly indistinguishable in the absence of other semantic information, which leads to the lowest HQA performance.
In contrast, the fine-grained \textbf{Cell} and \textbf{Link} evidence reduces the burden of answer reasoning but lowers the retrieval recall due to its huge candidates and limited semantic information. Moreover, only adopting the cell or link evidence cannot solve all questions which further causes low HQA performance.
\textbf{Row} granularity achieves the best performance among these baselines since it balances the retrieval recall and RC performance to a certain extent.
Compared with these baselines, our \textbf{Multi} granularity evidence brings remarkable improvements to the end-to-end HQA performance by preserving the evidence retrieval recall and the RC performance at the same time.
\begin{comment}
\subsubsection{The Evidence Discriminator (ED).}
The evidence discrimination brings the ability to judge the correctness of retrieved evidence to DR. Without ED, the DR directly extracts answers using the retrieved passage, no matter the passage whether indeed contains the answer or not. To show the effectiveness of ED, we turn off it from our DR and report the results in Table~\ref{cl}. It is observed that ED brings $0.6\%$ EM and F1 improvements to MERDR (RC-large), and $0.7\%$ EM and $0.8\%$ F1 to the MERDR (RC-base). The performance increasing of in-table questions is considerable which means most of the ED's judgment of the retrieved evidence is correct.
Though the performance is slightly decreased on the in-passage questions because of the misjudgment of CD, it brings benefits to total performance. Overall, MERDR achieves more balanced improvements on the two types of questions with the evidence discrimination.
\end{comment}
\begin{comment}
\subsubsection{The Number of Candidate Cells.}
The DA derives $N$ hop cells as the candidates to perform answer derivation process. Intuitively, the larger $N$ brings higher coverage of the correct hop cells. To choose the suitable value of $N$, we set it from $1$ to $5$ to perform grid search. We report the performance of our MINDA with different candidate number in Table~\ref{N}. It is shown the larger $N$ brings better total EM and F1 in case of $1\leq N \leq 4$. When we continue to increase the value of $N$, the total performance no longer improves, thence we finally choose the candidate number $N=4$.
\begin{table}[t]
\centering
\small
\linespread{1.8}
\begin{tabular}{c|ccc}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
% \multirow{2}{*}{\bf Discriminator}
\multirow{2}{*}{\bf N}
&In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-4}
\specialrule{0em}{1pt}{1pt}
&EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
%1 & 54.9 / 62.4 & 52.7 / 63.7 &52.3 / 61.7\\
%2& 54.6 / 62.3 & 54.3 / 65.4 & 53.1 / 62.6\\
%3& 54.2 / 62.1 & 55.4 / 66.6 & 53.5 / 63.2 \\
%4& 54.2 / 62.1 & 55.4 / 66.6 & 53.5 / 63.2 \\
%5& 54.7 / 62.2 & 51.5 / 62.0 & 51.4 / 60.5 \\
%\specialrule{0em}{1pt}{1pt}
% \hline
%\specialrule{0em}{1pt}{1pt}
% 1 &59.7 / 67.5 & 51.5 / 62.2 & 53.4 / 62.7 \\
% 2& 58.6 / 66.7 & 53.2 / 64.1 & 54.1 / 63.7 \\
1 &58.6 / 66.8 & 52.0 / 62.5 & 53.3 / 62.7 \\
2& 57.7 / 65.9 & 53.0 / 63.8 & 53.5 / 63.1 \\
%3& 58.6 / 66.7 &54.1 / 65.2 &\bf 54.5 / \bf 64.2\\
3& 57.9 / 65.8 &53.4 / 64.2 &\bf 53.8 / \bf 63.3\\
4& 57.6 / 65.5 &53.8 / 64.7 &\bf 53.9 / \bf 63.5\\
5& 57.4 / 65.3 &53.9 / 64.9 &\bf 53.9 / \bf 63.5\\
%4& 58.6 / 66.7 &54.1 / 65.2 &\bf 54.5 / \bf 64.2\\
\specialrule{0em}{1pt}{1pt}
\hline
\hline
\end{tabular}
\caption{Results of different candidate numbers.}
\label{N}
\end{table}
\end{comment}
%\include{exp}
\begin{comment}
\begin{table}[t]
\centering
\small
\linespread{1.8}
\begin{tabular}{l|ccc}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\bf Hit@1(\%) &\bf anchor &\bf thop & \bf phop \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
\bf MER & 80.1 & 29.1 &77.1 \\
\specialrule{0em}{1pt}{1pt}
\cdashline{1-4}
\specialrule{0em}{1pt}{1pt}
\bf MERDR w/o DR &80.1 & 51.7&75.9 \\
% \bf MINDA w/ DA &80.1 & 68.6&79.4 \\
\bf MERDR &80.1 & 68.2&79.9 \\
\specialrule{0em}{1pt}{1pt}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\end{tabular}
\caption{Hit@1 of the crucial evidence.}
\label{hit1}
\end{table}
\end{comment}
\begin{comment}
\subsection{Discussion}
Since the three crucial types of atomic evidence, i.e., $anchor$, $thop$, and $phop$, directly participate in the answer reasoning process, their retrieval accuracy fundamentally affects the QA performance.
In this section, we adopt Hit@1 score to evaluate the retrieval performance of different models for these three types of evidence.
The results are shown in table~\ref{hit1}.
% It is observed that the performance of MER on the hop cell retrieval especially the $thop$ cell is not ideal, because the MER lacks the ability to reason by combining information from the multiple retrievers.
It is observed that the performance of MER, which only adopt atomic retriever for prediction, performs poor on the $thop$ evidence prediction.
% since the lacks the ability to reason by combining information from the multiple retrievers.
In contrast, MERDR w/o DR which simply adds the anchor score for hop detection achieves dramatic improvements on the $thop$ evidence prediction.
However, the $phop$ retrieval result declines $1.2\%$ because the confusion of the question types brings losses to the retrieval of the $thop$ and $phop$.
Our MERDR which utilizes the discriminative reasoning is capable of judging the correctness of evidence.
It achieves the best performance on $thop$ and $phop$ evidence prediction with only a little sacrifice on $anc$ detection.
% Essentially, an end-to-end neural network cannot directly and accurately learn the multi-hop reasoning process.
% Compared to the MER,
% MERDR w/o DA explicitly performs reasoning by leveraging the retrieved anchors to further guide the hop cell reasoning, which brings improvement on $thop$ retrieval performance.
% However, the $phop$ retrieval result declines $1.2\%$ because the confusion of the question types brings losses to the retrieval of the $thop$ and $phop$.
% Our MERDR with DR is capable of discriminating the question types, which significantly improves the retrieval performance.
\end{comment}
\begin{comment}
\begin{table}[t]
\centering
\small
\linespread{1.8}
\begin{tabular}{l|ccc}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\multirow{2}{*}{\bf Granularity}
&In-Table & In-Passage & Total \\
\specialrule{0em}{1pt}{1pt}
\cline{2-4}
\specialrule{0em}{1pt}{1pt}
&EM / F1 & EM / F1 & EM / F1 \\
\specialrule{0em}{1pt}{1pt}
\hline
\specialrule{0em}{1pt}{1pt}
% Table & 12.8 / 18.2 & 11.6 / 15.4 & 12.4 / 16.8 \\
Column & 10.4 / 16.9 & 17.1 / 22.9 & 14.1 / 20.1 \\
Row &52.0 / 59.3 &49.6 / 60.1 & 49.3 / 58.3 \\
Cell& 22.1 / 28.6 &39.7 / 50.0& 31.9 / 40.5\\
Link & 13.0 / 18.3 &40.6 / 50.1& 28.8 / 36.5\\
\bf Multi (ours) & 58.2 / 66.1 & 52.9 / 64.6 & \bf 53.7 / \bf 63.6 \\
% \specialrule{0em}{1pt}{1pt}
% \hline
% \specialrule{0em}{1pt}{1pt}
% Table-g & 14.8 / 20.7 & 12.9 / 17.2 & 14.0 / 18.8 \\
% Row-g &53.3 / 60.6 &52.4 / 62.9 &51.4 / 60.4 \\
% Cell-g & 23.6 / 29.8 &41.4 / 51.4& 33.4 / 41.8\\
% Link-g & 14.0 / 19.1 &41.7 / 51.8& 29.9 / 37.8\\
% \bf MuGER$^2$ & 58.9 / 66.6 &55.4 / 66.9 &\bf 55.4 / \bf 65.1 \\
\specialrule{0em}{1pt}{1pt}
\hline
\hline
\specialrule{0em}{1pt}{1pt}
\end{tabular}
\caption{Results of MuGER$^2$-base and baselines.}
\label{evidence-g}
\end{table}
\subsection{Case Study}
\begin{figure}[t]
\centering
\subfigure[]{\includegraphics[width=0.47\textwidth]{fig/casestudy1.pdf}
\label{case1}}
\subfigure[]{\includegraphics[width=0.46\textwidth]{fig/casestudy2.pdf}
\label{case2}}
\caption{Reasoning examples of different settings.}
\label{case}
\end{figure}
Figure~\ref{case} gives two answer reasoning examples to intuitively show the effectiveness of DR and ED. To answer the question in Figure~\ref{case1}, MER correctly retrieves the top1 anchor (Pick:6) according to the question. Afterward, DR discriminates that the answer is in the passage and successfully reasons the $phop$ cell (Player:Zack Wheeler). However, the setting without DR confuses the question type and wrongly reasons the $thop$ cell (Team:San Francisco Giants) since its column header is related to the question. For the question in Figure~\ref{case2}, the anchor (Championship:2010 Albufeira) in the fourth line of the table is retrieved at the first iteration, and then the $phop$ cell in its neighbor cells is reasoned i.e., (Name:Amela Terzic). Without ED, DR directly uses the passage of the incorrect top1 $phop$ (Name:Amela Terzic) to extract the answer. By contrast, ED can discriminate that the passage of the does not contain the answer and feeds the information back to MER to perform re-retrieval. Finally, the DR with ED successfully extracts the answer from the $phop$ (Name:Nemanja Cerovac) at the second iteration.
\end{comment}
\section{Case Study}
Through statistics on the development set of HybridQA, we find there are 135 questions whose answers are only recalled by our multi-granularity evidence, but failed to be recalled by the coarse- and fine- grained evidence.
An example of such a question is given in Figure~\ref{muger-case}.
The table along with linked passages on the left is the given knowledge to answer the question.
The answer \textit{facebook watch} is contained in \textit{link1} of \textit{cell(2,0)} where \textit{(2,0)} represents (\textit{row2, column0}).
The table on the right lists the retrieved different granularity evidence. It is observed that the methods based on row, cell and link granularity evidence all retrieve incorrect evidence, including <\textit{row4, cell(4,0), link2}>. The reason is that the wrong evidence has high semantic similarity with the question which causes a higher ranking score. Although the column granularity based method retrieves the evidence <\textit{column0}> correctly, the reading comprehension model fails to derive the answer text accurately from the column.
In contrast, our method retrieves evidence including \textit{<column, row, cell, link>}. The evidence is correctly retrieved except for the \textit{cell} since the question is an in-passage question. The result demonstrates that the multi-granularity evidence improves the retrieval recall which benefits from learning the interactive information. The evidence selector further help to navigate the fine-grained cell or link based on the multi-granularity evidence.
\section{Error Analysis}
\begin{figure}[t]
\setlength{\belowcaptionskip}{-0.4cm}
\centering
\includegraphics[width=0.45\textwidth]{fig/error.pdf}
\caption{Error of the three modules of MuGER$^2$-large.}
\label{error}
\vspace{-0.1cm}
\end{figure}
%\section{Appendix}
%\label{sec:appendix}
%\subsection{Error Analysis}
To further analyze our method, we also make statistics on error cases in the prediction of MuGER$^2$-large. The error statistics are based on the development set of HybridQA.
As shown in Figure~\ref{error}, these cases can be divided into three categories according to the causes.
The first category is caused by evidence retrieval errors, which means the retrieved evidences overlook the answer.
According to our statistics, multi-granularity evidence has an advantage in improving coverage of answers.
Therefore, the proportion of this category is very low, only 3\%.
The second category is caused by the evidence selector, which accounts for about 40\% in the whole results.
There are two reasons for the evidence selector to make the error. Firstly, although multi-granularity evidence improves the coverage, it also introduces noise.
As a result, it may also lead to a mis-navigation of the fine-grained cell or link.
In our results, it accounts for about 33\%.
In addition, for those questions which are offered correct cells or links, they also may be answered incorrectly due to the wrong judgment of the answer type.
It accounts for about 7\% in our results.
The last category is caused by the reading comprehension module.
Limited by the performance of the RC model, even if feed correct fine-grained evidence, the RC model also may produce the wrong answer.
According to our analysis, this category has the maximum proportion, accounting for about 57\% of the whole error cases.
\section{Discussion}
More recently, two novel approaches, MATE \cite{eisenschlos2021mate} and MITQA \cite{kumar2021multi}, are proposed and achieve significant results, respectively $70.0$ and $71.9$ F1 scores on the HybridQA test set. Specifically, MATE mainly focuses on strengthening the ability to encode large tables with multi-view attention. While MITQA designs a multi-instance training method based on distant supervision to filter the noise from multiple answer spans.
In contrast, our MuGER$^2$ puts more effort into integrating multi-granularity evidence for the HQA problem to enhance the reasoning ability. We think that the two novel approaches and our model pay attention to different perspectives for solving the HQA problems, and it’s probably to combine them to achieve better performance. %strengthens the transformer's ability to encode large tables with the multi-view attention, benefiting the tabular knowledge-based question-answering performance.
\section{Related Work}
Recently, many researchers attempt to tackle the question answering task using heterogeneous knowledge \cite{sun2019pullnet,sawant2019neural,zayats2021representations,li2021dual,xiong-etal-2019-improving,unirpg-zhou}. However, traditional researches only combine the information of heterogeneous knowledge, but fail to answer questions that require reasoning over heterogeneous data. To fill in the gap, the hybrid question answering (HQA) task together with the HybridQA dataset is proposed by \cite{chen-etal-2020-hybridqa}. HybridQA provides a WiKiTable \cite{pasupat2015compositional} along with its hyperlinked Wikipedia passages \cite{rajpurkar-etal-2016-squad} as evidence for each question.
%The HQA systems require to aggregate information from different forms to reason answers.
Based on the HybridQA, \citet{chen2020open} further propose the OTT-QA dataset, which requires the system to retrieve relevant tables and text for the given questions. \citet{zhu2021tat} and and \citet{chen-etal-2021-finqa} propose TAT-QA and FinQA requiring numerical reasoning over the heterogeneous data.
%This paper focus on the HybridQA task.
Existing works on HybridQA usually retrieve mono-granularity evidence from the heterogeneous data to derive the answer. Hybrider \cite{chen-etal-2020-hybridqa} proposes a two-phase pipeline framework to retrieve a table cell as the evidence and feeds its value and linked passages into an RC model to extract the answer. Dochopper \cite{sun2021end} propose an end-to-end multi-hop retrieval method to directly retrieve a passage sentence or a cell value as the evidence. In addition, \citet{pan2020unsupervised} explore an unsupervised multi-hop QA model MQA-QG, which can generate human-like multi-hop training data from heterogeneous data resources. %Different from the previous work, this paper proposes the multi-granularity evidence retrieval and reasoning method to trade-off the answer recall and extraction accuracy.
\section{Conclusion}
We propose MuGER$^2$, a multi-granularity evidence retrieval and reasoning approach for hybrid question answering (HQA). In our method, a unified retriever is designed to learn multi-granularity evidence and an evidence selector is proposed to navigate fine-grained evidence for the reader. Experiment results show that the MuGER$^2$ boosts the end-to-end HQA performance and outperforms the strong baselines on the HybridQA benchmark.
\section*{Limitations}
In this paper, we focus on the hybrid question answering task, in which the answers to most questions can be extracted from the cell values or linked passages using the reading comprehension model. Although our MuGER$^2$ performs well on this task, one limitation is that it cannot answer the questions needing numerical operations, such as count, compare and etc. To enable our model to answer more complex questions, in future work we will develop the numerical reasoning capabilities of our model.
\section*{Acknowledgment}
The authors would like to thank the anonymous
reviewers for their thoughtful comments.
The work of this paper is funded by the projects of National key research and development program of China (No.2020YFB1406902) and the National Key Research and Development Program of China (No. 2020AAA0108600).
% Firstly, a unified retriever retrieves multi-granularity evidence with high answer recall from the table and linked passages. Afterwards, an answer type-aware reasoning process and an RC model are designed to derive the answer from the retrieved evidence. Experiment results on the HybridQA dataset show that our method significantly outperforms the strong baselines.
\bibliography{anthology,custom}
% \bibliography{MuGER2.new,custom}
\bibliographystyle{acl_natbib}
%\newpage
%\appendix
\end{document}
\begin{thebibliography}{25}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Bao et~al.(2016)Bao, Duan, Yan, Zhou, and Zhao}]{bao2016constraint}
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016.
\newblock \href {https://aclanthology.org/C16-1236} {Constraint-based question
answering with knowledge graph}.
\newblock In \emph{Proceedings of {COLING} 2016, the 26th International
Conference on Computational Linguistics: Technical Papers}, pages 2503--2514,
Osaka, Japan. The COLING 2016 Organizing Committee.
\bibitem[{Chen et~al.(2021{\natexlab{a}})Chen, Chang, Schlinger, Wang, and
Cohen}]{chen2020open}
Wenhu Chen, Ming{-}Wei Chang, Eva Schlinger, William~Yang Wang, and William~W.
Cohen. 2021{\natexlab{a}}.
\newblock \href {https://openreview.net/forum?id=MmCRswl1UYl} {Open question
answering over tables and text}.
\newblock In \emph{9th International Conference on Learning Representations,
{ICLR} 2021, Virtual Event, Austria, May 3-7, 2021}. OpenReview.net.
\bibitem[{Chen et~al.(2020)Chen, Zha, Chen, Xiong, Wang, and
Wang}]{chen-etal-2020-hybridqa}
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William~Yang
Wang. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.findings-emnlp.91}
{{H}ybrid{QA}: A dataset of multi-hop question answering over tabular and
textual data}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2020}, pages 1026--1036, Online. Association for Computational
Linguistics.
\bibitem[{Chen et~al.(2021{\natexlab{b}})Chen, Chen, Smiley, Shah, Borova,
Langdon, Moussa, Beane, Huang, Routledge, and Wang}]{chen-etal-2021-finqa}
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan
Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and
William~Yang Wang. 2021{\natexlab{b}}.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.300} {{F}in{QA}: A
dataset of numerical reasoning over financial data}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 3697--3711, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin2018bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Eisenschlos et~al.(2021)Eisenschlos, Gor, M{\"u}ller, and
Cohen}]{eisenschlos2021mate}
Julian Eisenschlos, Maharshi Gor, Thomas M{\"u}ller, and William Cohen. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.600} {{MATE}:
Multi-view attention for table transformer efficiency}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 7606--7619, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Gao et~al.(2021)Gao, Yao, and Chen}]{gao2021simcse}
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.552} {{S}im{CSE}:
Simple contrastive learning of sentence embeddings}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 6894--6910, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Gu et~al.(2021)Gu, Kase, Vanni, Sadler, Liang, Yan, and
Su}]{gu2021beyond}
Yu~Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and
Yu~Su. 2021.
\newblock Beyond iid: three levels of generalization for question answering on
knowledge bases.
\newblock In \emph{Proceedings of the Web Conference 2021}, pages 3477--3488.
\bibitem[{Hadsell et~al.(2006)Hadsell, Chopra, and LeCun}]{1640964}
R.~Hadsell, S.~Chopra, and Y.~LeCun. 2006.
\newblock \href {https://doi.org/10.1109/CVPR.2006.100} {Dimensionality
reduction by learning an invariant mapping}.
\newblock In \emph{2006 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR'06)}, volume~2, pages 1735--1742.
\bibitem[{Jauhar et~al.(2016)Jauhar, Turney, and Hovy}]{jauhar2016tables}
Sujay~Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016.
\newblock \href {https://doi.org/10.18653/v1/P16-1045} {Tables as
semi-structured knowledge for question answering}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 474--483,
Berlin, Germany. Association for Computational Linguistics.
\bibitem[{Kumar et~al.(2021)Kumar, Chemmengath, Gupta, Sen, Bharadwaj, and
Chakrabarti}]{kumar2021multi}
Vishwajeet Kumar, Saneem Chemmengath, Yash Gupta, Jaydeep Sen, Samarth
Bharadwaj, and Soumen Chakrabarti. 2021.
\newblock \href {https://arxiv.org/abs/2112.07337} {Multi-instance training for
question answering across table and linked text}.
\newblock \emph{ArXiv preprint}, abs/2112.07337.
\bibitem[{Li et~al.(2021)Li, Ng, Xu, Zhu, Wang, and Xiang}]{li2021dual}
Alexander~Hanbo Li, Patrick Ng, Peng Xu, Henghui Zhu, Zhiguo Wang, and Bing
Xiang. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.acl-long.315} {Dual
reader-parser on hybrid textual and tabular evidence for open domain question
answering}.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 4078--4088,
Online. Association for Computational Linguistics.
\bibitem[{Pan et~al.(2021)Pan, Chen, Xiong, Kan, and
Wang}]{pan2020unsupervised}
Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, and William~Yang Wang.
2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.469} {Unsupervised
multi-hop question answering by question generation}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 5866--5880, Online. Association for Computational
Linguistics.
\bibitem[{Pasupat and Liang(2015)}]{pasupat2015compositional}
Panupong Pasupat and Percy Liang. 2015.
\newblock \href {https://doi.org/10.3115/v1/P15-1142} {Compositional semantic
parsing on semi-structured tables}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 1470--1480,
Beijing, China. Association for Computational Linguistics.
\bibitem[{Rajpurkar et~al.(2016)Rajpurkar, Zhang, Lopyrev, and
Liang}]{rajpurkar-etal-2016-squad}
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.
\newblock \href {https://doi.org/10.18653/v1/D16-1264} {{SQ}u{AD}: 100,000+
questions for machine comprehension of text}.
\newblock In \emph{Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing}, pages 2383--2392, Austin, Texas. Association
for Computational Linguistics.
\bibitem[{Sawant et~al.(2019)Sawant, Garg, Chakrabarti, and
Ramakrishnan}]{sawant2019neural}
Uma Sawant, Saurabh Garg, Soumen Chakrabarti, and Ganesh Ramakrishnan. 2019.
\newblock Neural architecture for question answering using a knowledge graph
and web corpus.
\newblock \emph{Information Retrieval Journal}, 22(3):324--349.
\bibitem[{Sun et~al.(2019)Sun, Bedrax-Weiss, and Cohen}]{sun2019pullnet}
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1242} {{P}ull{N}et: Open
domain question answering with iterative retrieval on knowledge bases and
text}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 2380--2390, Hong Kong,
China. Association for Computational Linguistics.
\bibitem[{Sun et~al.(2021)Sun, Cohen, and Salakhutdinov}]{sun2021end}
Haitian Sun, William~W Cohen, and Ruslan Salakhutdinov. 2021.
\newblock \href {https://arxiv.org/abs/2106.00200} {End-to-end multihop
retrieval for compositional question answering over long documents}.
\newblock \emph{ArXiv preprint}, abs/2106.00200.
\bibitem[{Xiong et~al.(2019)Xiong, Yu, Chang, Guo, and
Wang}]{xiong-etal-2019-improving}
Wenhan Xiong, Mo~Yu, Shiyu Chang, Xiaoxiao Guo, and William~Yang Wang. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1417} {Improving question
answering over incomplete {KB}s with knowledge-aware reader}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 4258--4264, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Zayats et~al.(2021)Zayats, Toutanova, and
Ostendorf}]{zayats2021representations}
Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.eacl-main.253}
{Representations for question answering from documents with tables and text}.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
2895--2906, Online. Association for Computational Linguistics.
\bibitem[{Zhang et~al.(2020)Zhang, Wang, Wang, Cao, Zhang, and
Wang}]{zhang2020table}
Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and
Zhongyuan Wang. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.126} {Table fact
verification with structure-aware transformer}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 1624--1629, Online. Association
for Computational Linguistics.
\bibitem[{Zhao et~al.(2021)Zhao, Bao, Wang, Zhou, Wu, He, and
Zhou}]{zhao-etal-2021-ror-read}
Jing Zhao, Junwei Bao, Yifan Wang, Yongwei Zhou, Youzheng Wu, Xiaodong He, and
Bowen Zhou. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-emnlp.160} {{R}o{R}:
Read-over-read for long document machine reading comprehension}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2021}, pages 1862--1872, Punta Cana, Dominican Republic. Association
for Computational Linguistics.
\bibitem[{Zhou et~al.(2022{\natexlab{a}})Zhou, Bao, Duan, Sun, Liang, Wang,
Zhao, Wu, He, and Zhao}]{zhou-etal-2022-opera}
Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang,
Jing Zhao, Youzheng Wu, Xiaodong He, and Tiejun Zhao. 2022{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/2022.naacl-main.119} {{OPERA}:
Operation-pivoted discrete reasoning over text}.
\newblock In \emph{Proceedings of the 2022 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 1655--1666, Seattle, United States. Association for
Computational Linguistics.
\bibitem[{Zhou et~al.(2022{\natexlab{b}})Zhou, Bao, Duan, Wu, He, and
Zhao}]{unirpg-zhou}
Yongwei Zhou, Junwei Bao, chaoqun Duan, Youzheng Wu, Xiaodong He, and Tiejun
Zhao. 2022{\natexlab{b}}.
\newblock Uni{RPG}: Unified discrete reasoning over table and text as program
generation.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}. Association for Computational
Linguistics.
\bibitem[{Zhu et~al.(2021)Zhu, Lei, Huang, Wang, Zhang, Lv, Feng, and
Chua}]{zhu2021tat}
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv,
Fuli Feng, and Tat-Seng Chua. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.acl-long.254} {{TAT}-{QA}: A
question answering benchmark on a hybrid of tabular and textual content in
finance}.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 3277--3287,
Online. Association for Computational Linguistics.
\end{thebibliography}