BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language?
Extracted source text · arXiv 2411.12235
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{tcolorbox}
\newcommand{\datasetname}{\textsc{BoolQuestions}}
\newcommand{\taskname}{BDR}
\newcommand{\bridge}[1]{{\color{red} \textbf{\textit{#1}}}}
\newcommand{\operator}[1]{{\color{blue} \textbf{#1}}}
\newcommand{\keyinfo}[1]{{\color{green!55!black} \textbf{\textit{#1}}}}
\newcommand{\negkeyinfo}[1]{{\color{red!80!black} \textbf{\textit{#1}}}}
\newcommand{\blue}[1]{{\color{blue}#1}}
\newcommand{\support}[1]{{\color{blue}\textit{#1}}}
\newcommand{\answer}[1]{{\color{green!55!black}\textbf{#1}}}
\newcommand{\sentid}[1]{{\color{orange}#1}}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{\datasetname: Does Dense Retrieval Understand \\ Boolean Logic in Language?}
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
% \author{First Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\\And
% Second Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\}
% \author{Zongmeng Zhang \\
% University of Science and Technology of China \\
% \texttt{zhangzm@mail.ustc.edu.cn} \\\And
% Second Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\}
\author{
\textbf{Zongmeng Zhang\textsuperscript{1}},
\textbf{Jinhua Zhu\textsuperscript{1}},
\textbf{Wengang Zhou\textsuperscript{1,3}},
\\
\textbf{Xiang Qi\textsuperscript{2}},
\textbf{Peng Zhang\textsuperscript{2}},
\textbf{Houqiang Li\textsuperscript{1,3}}
\\
\textsuperscript{1}University of Science and Technology of China
\textsuperscript{2}Ant Group
\\
\textsuperscript{3}Institute of Artificial Intelligence, Hefei Comprehensive Nation Science Center
\\
\texttt{
\{zhangzm, teslazhu\}@mail.ustc.edu.cn, }\\
\texttt{
\{zhwg, lihq\}@ustc.edu.cn, \{qixiang.qx, minghua.zp\}@antgroup.com
}
% \small{
% \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
% }
}
\begin{document}
\maketitle
\begin{abstract}
% Dense retrieval has emerged as a effective and efficient paradigm for text retrieval in various fields. It is believed capable of encoding the semantics of queries and documents into dense vectors for efficient search. We challenge this viewpoint by investigating whether current retrieval systems understand Boolean logic implied in language. Specifically, we formulate the task of Boolean Dense Retrieval and collect a benchmarking dataset \datasetname, which covers complex queries containing basic Boolean logic and corresponding annotated passages. With the experimental results on the proposed task and benchmarking dataset, we draw the conclusion that current dense retrieval systems do not understand Boolean logic in language. In addition, we propose a contrastive continue learning method as a starting point for further research on promoting the understanding of language for language models.
Dense retrieval, which aims to encode the semantic information of arbitrary text into dense vector representations or embeddings, has emerged as an effective and efficient paradigm for text retrieval, consequently becoming an essential component in various natural language processing systems. These systems typically focus on optimizing the embedding space by attending to the \textit{relevance} of text pairs, while overlooking the \textit{Boolean logic} inherent in language, which may not be captured by current training objectives. In this work, we first investigate whether current retrieval systems can comprehend the Boolean logic implied in language. To answer this question, we formulate the task of Boolean Dense Retrieval and collect a benchmark dataset, \datasetname, which covers complex queries containing basic Boolean logic and corresponding annotated passages. Through extensive experimental results on the proposed task and benchmark dataset, we draw the conclusion that current dense retrieval systems do not fully understand Boolean logic in language, and there is a long way to go to improve our dense retrieval systems. Furthermore, to promote further research on enhancing the understanding of Boolean logic for language models, we explore Boolean operation on decomposed query and propose a contrastive continual training method that serves as a strong baseline for the research community.\footnote{Code and dataset are available at \url{https://github.com/zmzhang2000/boolean-dense-retrieval}.}
\end{abstract}
\section{Introduction}
% \begin{figure}[!t]
% \framebox{
% \parbox{0.45\textwidth}{
% \small
% \textbf{Boolean Question:} How can I start a \keyinfo{career in the accounting field} \operator{and} pursue an \keyinfo{online degree program}?\newline
% \smallskip\newline
% \textbf{Positive Paragraph:} Lower gastrointestinal bleeding is typically from the colon, rectum or anus. Common causes of lower gastrointestinal bleeding include hemorrhoids, cancer, angiodysplasia, ulcerative colitis, Crohn's disease, and aortoenteric fistula. It may be indicated by the passage of fresh red blood rectally, especially in the absence of bloody vomiting.
% \newline
% \textbf{Answer:} Common causes of fresh red blood rectally or black stool include hemorrhoids, cancer, angiodysplasia, ulcerative colitis, Crohn's disease, and aortoenteric fistula.\smallskip\newline
% \textbf{Negative Paragraph:} In cirrhosis of the liver, not only does fluid accumulate in the abdominal cavity, but there may be leg swelling,easy bruising, enlargement of the breasts, and confusion due to encephalopathy (encephalo=brain + pathy=disorder).
% \newline
% \textbf{Answer:} Cannot answer.}
% }
% \caption{An example of the NOT questions in \datasetname{}. We also provide answers as the part of the dataset.} \label{fig:example}
% \end{figure}
\begin{figure}[!t]
\framebox{
\parbox{0.45\textwidth}{
\small
\textbf{AND Question:} How can I start a \keyinfo{career in the accounting field} \operator{and} pursue an \keyinfo{online degree program}?
\smallskip\newline
\textbf{OR Question:} What are the impacts of \keyinfo{global warming} \operator{or} \keyinfo{climate change} on nature and humans?
\smallskip\newline
\textbf{NOT Question:} What causes \keyinfo{upper abdomen pain} \operator{but is unrelated to} \negkeyinfo{liver issues}?
\smallskip\newline
\textbf{Question in MS MARCO:} What flower is symbol of endurance?
\smallskip\newline
\textbf{Question in Natural Questions:} Who sings Does He Love me with Reba?
}
}
\caption{Examples of the AND, OR and NOT question in \datasetname and questions from MS MARCO and Natural Questions. Questions in \datasetname{} are more complex to understand than those in MS MARCO and Natural Questions.} \label{fig:example}
\end{figure}
Text retrieval is a fundamental component of various natural language processing systems, including question answering~\cite{chen-etal-2017-reading,karpukhin-etal-2020-dense}, dialogue systems~\cite{Chen2017SurveyDialogue}, web search~\cite{mitra-2017-learning} and so on. In the era of large language models, text retrieval has become increasingly critical as it provides an offline and incrementally updatable knowledge database, directly influencing the reliability and quality of generated responses~\cite{karpukhin-etal-2020-dense,shuster-etal-2021-retrieval-augmentation} in the Retrieval-Augmented Generation paradigm~\cite{Lewis2020rag}.
Traditional text retrieval methods estimate the relevance of query and document based on lexical overlap~\cite{salton1975vector,SALTON1988termweighting}. Utilizing the "bag-of-words" assumption and set theory~\cite{WALLER1979mathematical,bookstein1980fuzzy}, these methods organize text content in the form of inverted indexes~\cite{zobel1998inverted,zobel2006inverted}, which handle Boolean logic efficiently. The Boolean retrieval model was later extended to return ranked lists of documents by leveraging term weights~\cite{Salton1983extended}. To more flexibly consider the weights of different words, probabilistic models such as BM25~\cite{Robertson2009Probabilistic} and statistical language modeling~\cite{zhai-2007-statistical} were introduced. Nevertheless, these probabilistic models still rely on lexical overlap while omitting Boolean operations, making it hard to handle Boolean logic in queries.
With the advent of deep learning, these hand-crafted sparse text features have gradually been replaced by low-dimensional dense vectors learned by neural networks~\cite{reimers-gurevych-2019-sentence,karpukhin-etal-2020-dense,qu-etal-2021-rocketqa}, particularly using LSTM~\cite{hochreiter1997long} and the powerful Transformer architecture~\cite{Vaswani2017Attention}. Unlike sparse vector spaces, dense vectors are believed to capture the semantics implied in texts~\cite{zhao2024densesurvey}. In these frameworks, retrieval systems are expected to assign high scores to ground-truth query-document pairs and relatively lower scores to irrelevant or randomly combined pairs, often framed within modern deep learning paradigms such as contrastive learning frameworks. Benefiting from pre-trained language models, text retrieval has achieved significant performance improvements~\cite{guo2022semantic,fan2022pre,yates-etal-2021-pretrained}.
However, dense retrieval systems primarily focus on encoding the \emph{relevance} of texts, which may be insufficient for handling complex Boolean logic in natural language queries. Specifically, since current retrieval systems do not incorporate Boolean logic in their training paradigms, there is no guarantee of comparability in the output scores for query-document pairs involving Boolean logic. For example, the relevance scores for queries containing logical NOT, which exclude undesired information, have not been thoroughly investigated. In contrast, lexical-based retrieval systems effectively address complex Boolean logic using set theory, a method not directly applicable to dense retrieval. Consequently, it remains unclear whether dense retrieval systems can fully comprehend and process complex Boolean logic.
To answer this question, we formulate the task of Boolean Dense Retrieval~(\taskname) and collect a benchmark dataset, \datasetname, which includes complex queries containing basic Boolean logic and corresponding annotated passages. We evaluate several state-of-the-art dense retrieval systems on the proposed task and benchmark dataset, finding significant performance drops on NOT questions, as illustrated in Figure~\ref{fig:example}. Our findings indicate that current dense retrieval models do not fully understand Boolean logic in language.
Based on these observations, we generate additional training data for NOT questions and propose a contrastive continual learning baseline to enhance the understanding of logical NOT in natural language. While the proposed baseline reduces the negative rate of the passage list returned by retrieval systems, it also slightly sacrifices accuracy. This phenomenon underscores the difficulty of understanding logical NOT in natural language for dense retrieval systems. Nevertheless, the proposed baseline serves as a starting point for further research on dense retrieval in the community.
\section{Related Work}
In this section, we briefly introduce several highly related works about our work.
\subsection{Sparse Retrieval}
Sparse retrieval refers to the lexical-based retrieval model that conceives both the query and documents as a set of terms, known as the bag-of-words assumption. Boolean model is the most classical model developed in the early stage of information retrieval~\cite{Stefan2016}. Queries in Boolean retrieval model are formed as Boolean expressions, which comprise terms joint by Boolean operators including ``AND'', ``OR'' and ``NOT''. The retrieval process is typically based on set theory and Boolean algebra. Inverted index~\cite{zobel1998inverted,zobel2006inverted} is utilized as the data structure to implement the Boolean model.
Due to the special form of queries, logic in queries is precisely delivered to the retrieval system. Despite that, the binary decision of Boolean model lacks the ability of providing the ranking of the documents. TF-IDF and BM25~\cite{Robertson2009Probabilistic} term weighting is then proposed to assign continuous relevance scores to documents, turning retrieval to a ranking task. However, the schema of retrieving documents barely on the ranking scores of documents discards the ability to express the Boolean logic explicitly, and thus struggles to tackle the complex Boolean logic in queries.
\subsection{Dense Retrieval}
With the re-surge of deep learning, the hand-crafted scoring function has been gradually replaced by learnable neural networks. Specifically, texts are encoded to low-dimensional dense vectors and the relevance of texts is measured in latent semantic space. LSTM~\cite{hochreiter1997long} and the powerful Transformer architecture~\cite{Vaswani2017Attention} are typically used as encoders and trained with contrastive learning frameworks that pull together the representations of relevant texts and push apart those of irrelevant texts~\cite{reimers-gurevych-2019-sentence,karpukhin-etal-2020-dense,qu-etal-2021-rocketqa}. The pre-training then fine-tuning paradigm~\cite{devlin-etal-2019-bert,liu2023pretrain} has significantly pushed forward the development of dense retrieval~\cite{guo2022semantic,fan2022pre,yates-etal-2021-pretrained}.
Despite these dedicated efforts, there is no evidence suggesting that dense retrievers truly understand logic in language. In this work, we propose the task of Boolean dense retrieval and collect a benchmark dataset to investigate whether dense retrieval models understand the Boolean logic in natural language.
\subsection{Datasets for Modern Retrieval}
A number of datasets are publicly released to provide large-scale relevance judgments or test beds for retrieval, significantly facilitating the research of modern text retrieval systems. Natural Questions~\cite{kwiatkowski-etal-2019-natural} includes questions from Google search engines, along with related paragraphs and answer spans from top-ranked Wikipedia documents. MS MARCO~\cite{Bajaj2018MS} consists of a large number of queries from Bing search logs and annotated relevant passages collected from Web pages. Variants of MS MARCO like mMARCO~\cite{bonifacio2021mmarco} and MS MARCO Chameleons~\cite{Arabzadeh2021MSMARCOChemeleons} are created to enrich the evaluation characteristics with respect to multilingual and question difficulty. Domain-specific retrieval datasets such as those for financial~\cite{Maia2018financial}, scientific~\cite{wadden-etal-2020-fact} and biomedical~\cite{Tsatsaronis2015AnOO} fields are also released to advance the development of retrieval in other areas. Additionally, BEIR~\cite{thakur2021beir} and KILT~\cite{petroni-etal-2021-kilt} aggregate representative datasets to measure the overall performance of retrieval models.
To the best of our knowledge, none of these efforts probes into the logic implied in queries. BoolQ~\cite{clark-etal-2019-boolq} contains questions that can be answered with barely ``yes'' or ``no'', but does not involve complex Boolean logic in its queries. Although \citet{malaviya-etal-2023-quest} and \citet{zhong-etal-2023-romqa} construct questions with Boolean logic, their atomic questions are entity queries rather than natural language questions. This work is the first to construct a benchmark dataset for Boolean dense retrieval.
\section{Task Definition}
In this section, we initially review the technique of dense retrieval, subsequently give a formal definition of what we term as Boolean dense retrieval.
The objective of text retrieval is to identify documents within a substantial text corpus that are relevant to specific queries. We denote the query text by $q$ and the corpus by $\mathcal{D} = \big\{{d_i}\big\}_{i=1}^{|\mathcal{D}|}$. Typically, retrieval models evaluate each document in the corpus, assigning relevance scores and returning the top $k$ documents based on these scores. The scoring function is typically implemented through either lexical or semantic matching methodologies.
In this work, we focus on dense retrieval models, which are reputed for their capacity to encode the semantics of language into a dense vector representation. In dense retrieval, both query and document texts are represented as dense vectors, and their relevance is assessed within this vector space, expressed as:
\begin{equation}
\text{Rel}(q, d) = f_{\text{sim}}(\phi(q), \psi(d)),
\end{equation}
where $\phi(\cdot)$ and $\psi(\cdot)$ are encoding functions that transform text into dense vectors, and $f_{\text{sim}}$ is a similarity function designed to measure the distance of these vectors as an indication of relevance between the encoded texts.
Building upon the conventional dense retrieval model, we introduce the concept of Boolean dense retrieval. This approach enables the integration of complex Boolean logic within text queries. Formally, a complex Boolean query is semantically equivalent to a Boolean logic expression consisting of several simpler queries, represented as:
\begin{equation}
q \iff q_1 \star q_2 \star \cdots \star q_m,
\end{equation}
where $q_i$ represents individual simple queries and $\star$ denote the Boolean operations chosen from $\{\text{AND}, \text{OR}, \text{NOT}\}$.
Apart from the construction of the query, the retrieval process remains consistent as outlined above. Moreover, because our query incorporates a more intricate construction method, we emphasize the need for more sophisticated evaluation metrics and training techniques for retrieval models.
\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figures/pipeline.pdf}
\caption{Data collection pipeline of \datasetname.}
\label{fig:pipeline}
\end{figure*}
\section{Data Collection}
In our work, we developed a question generation framework that progresses from simple to complex structures to gather questions embedded with Boolean logic. Specifically, the process involves sampling text passages from a corpus, generating basic questions from these passages, and then combining these questions using Boolean operators. The combined questions are subsequently rephrased into natural language. To ensure the relevance between generated questions and corresponding passages, we introduce a cyclic consistency filtering strategy, retaining only those questions that meet this criterion.
Addressing the challenge posed by restricted access to query logs in widely used search engines, we leverage GPT-4\footnote{We use the {gpt-4-0125-preview} version.}, one of the most advanced language generation models currently available. This model is tasked with constructing Boolean questions tailored to the document collections found in prevalent retrieval datasets. Next, we will provide a detailed introduction to our data collection pipeline, which is shown in Figure \ref{fig:pipeline}. All prompts used are detailed in Appendix~\ref{app:prompts}.
\subsection{Passage Clustering}
A complex question containing Boolean logic consists of simple questions with interrelated topics. The passage sampling process must be meticulously designed, as combining questions about disparate objects could result in incoherent and peculiar complex questions. To address this, we propose a hierarchical passage sampling strategy for the subsequent question generation process. Specifically, we conduct clustering on passages of the corpus, assuming that passages from the same cluster discuss similar topics.
\datasetname{} is constructed based on the widely-used passage retrieval datasets MS MARCO~\cite{Bajaj2018MS} and Natural Questions~\cite{kwiatkowski-etal-2019-natural}. For the passage collection in MS MARCO, we first encode the passages using a BERT-style encoder pre-trained for the dense retrieval task\footnote{\url{https://huggingface.co/sentence-transformers/msmarco-distilbert-dot-v5}.}. We then reduce the dimensionality of these vector representations via truncated SVD, with random sampling performed beforehand to lower computation costs. Finally, we apply agglomerative clustering from Sklearn\footnote{\url{https://github.com/scikit-learn/scikit-learn}.} library to the compressed vectors. We also conduct clustering with TF-IDF vectorization but find that clusters formed in this manner focus more on word overlap, which is not suitable for selecting passages within the same topics compared with BERT-style encoder vectorization. The corpus of Natural Questions comprises split English Wikipedia pages, allowing similar topic passages to be directly drawn from successive passages within the same Wikipedia pages.
\subsection{Atomic Question Generation}
For each cluster, we randomly select $n$ passages for atomic question generation. Since a larger number of candidate passages incurs higher computational costs, we randomly set $n$ to 2 or 3. These selected passages are referred to as candidate passages $\{p_i\}_{i=1}^n$.
\paragraph{Simple Question} We prompt GPT-4 to generate simple questions $\{q_i^{\text{simple}}\}_{i=1}^n$ for each candidate passage sampled from the same cluster. Naturally, candidate passages provide answers to their corresponding generated simple questions. Therefore, each simple question is paired with 1 positive passage, from which it is generated.
% \begin{tcolorbox}[title=Single Questioner Prompt]
% \small
% Please propose a question that can be answered by the following paragraph.\\
% \\
% $[$PARAGRAPH$]$
% \end{tcolorbox}
\paragraph{Disjunctive Question} GPT-4 is also required to generate a disjunctive question $q^{\text{disj}}$ that can be answered with any of the candidate passages individually. The disjunctive question is regarded as a more abstract question than a simple question. Any of the candidate passages provide answers to the disjunctive question and are annotated as positive passages for the generated question.
% \begin{tcolorbox}[title=Disjunctive Questioner Prompt]
% \small
% Please propose a question that can be answered by any of the following paragraphs. Please make sure that each paragraph can provide answers to the question individually.\\
% \\
% $[$PARAGRAPHS$]$
% \end{tcolorbox}
% We label source passages as positive passages of generated questions. Each simple question has one positive passage. Meanwhile, each disjunctive question has more than one positive passages as these questions are derived from multiple passages and can be answered by any of these passages.
\subsection{Boolean Question Generation}
Simple questions and disjunctive questions operate at different levels of abstraction. Generally, simple questions focus on more detailed objects than disjunctive questions within the same cluster. We leverage this property to generate complex questions containing Boolean logic.
\paragraph{AND Questions} Disjunctive questions encompass the topics discussed in all of their candidate passages, whereas simple questions within the same cluster address more concrete details of the topic. To simulate the AND operator, we prompt GPT-4 to add extra constraints to the disjunctive question, thus forming an AND question. This can be represented by the Boolean expression:
\begin{equation}
q^{\text{AND}} \iff q^{\text{disj}}\ [\text{AND}] \ \text{constraints}.
\end{equation}
We ensure that the generated question can be answered solely with a randomly selected candidate passage $\{p^{\text{AND}+}\}$. All remaining candidate passages in the same cluster are labeled as negative passages $\{p_i^{\text{AND}-}\}_{i=1}^{n-1} = \{p_i\}_{i=1}^n \setminus \{p^{\text{AND}+}\}$.
\paragraph{OR Questions} Logical disjunction tends to describe the union of two topics at the same level of abstraction. We randomly selected two simple questions $q_a^{\text{OR}}$ and $q_b^{\text{OR}}$ from $\{q_i^{\text{simple}}\}_{i=1}^n$, join them with OR operator and ask GPT-4 to paraphrase the Boolean expression to a natural language question. Under these circumstances, the OR question is constructed by
\begin{equation}
q^{\text{OR}} \iff q_a^{\text{OR}}\ [\text{OR}] \ q_b^{\text{OR}}.
\end{equation}
Candidate passages $p_a^{\text{OR}}$ and $p_b^{\text{OR}}$ corresponding to $q_a^{\text{OR}}$ and $q_b^{\text{OR}}$ are annotated as positive passages $\{p_i^{\text{OR}+}\}_{i=1}^{2} = \{p_a^{\text{OR}},p_b^{\text{OR}}\}$, while other candidate passages in the same cluster are labeled as negative passages $\{p_i^{\text{OR}-}\}_{i=1}^{n-2} = \{p_i\}_{i=1}^n \setminus \{p_i^{\text{OR}+}\}_{i=1}^{2}$.
\paragraph{NOT Questions} In contrast to the construction of AND questions, NOT questions are created by adding an exclusion to the disjunctive question. Here, we adopt the information in simple questions as an exclusion to the disjunctive question. Concretely, we concatenate the disjunctive question and a randomly sampled simple question $q^{\text{NOT}}$ with NOT operator to formulate the Boolean expression of NOT question. Then the expression is fed into GPT-4 to be paraphrased into a NOT question in the form of natural language. The creation of NOT questions is defined as
\begin{equation}
q^{\text{NOT}} \iff q^{\text{disj}}\ [\text{NOT}] \ q_a^{\text{NOT}}.
\end{equation}
Candidate passage $p^{\text{NOT}}$ corresponding to $q^{\text{NOT}}$ is annotated as negative passages $\{p^{\text{NOT}-}\}$ as it contradicts the negation in $q^{\text{NOT}}$, while other candidate passages in the same cluster are labeled as positive passages $\{p_i^{\text{NOT}+}\}_{i=1}^{n-1} = \{p_i\}_{i=1}^n \setminus \{p_i^{\text{NOT}-}\}$.
% \paragraph{Boolean Expression} Built upon aforementioned atomic questions, we connect these questions using Boolean operators to construct Boolean expressions of complex questions. Each Boolean expression involves one type of Boolean operator for simplicity. The disjunctive question and $k_{\text{AND}}$ simple questions are jointed with ``AND'' to form a AND Boolean expression. The corresponding $k_{\text{AND}}$ passages of simple questions are labeled positives of the AND Boolean expression while other passages under the same cluster are labeled negatives. $k_{\text{OR}}$ simple questions are connected with ``OR'' to form a OR Boolean expression. The corresponding $k_{\text{OR}}$ passages of simple questions are labeled positive of the OR Boolean expression. ``NOT'' Boolean expression is constructed by adding $k_{\text{NOT}}$ sampled negative simple questions with the prefix ``AND NOT''. Naturally, passages of negative simple questions are annotated as positives and other passages in the same cluster as negatives.
% \paragraph{Paraphrasing}
% Dense retrievers are typically trained on queries in natural language form, and thus we can not expect the encoder Boolean expressions to be understood by encoders in dense retrieval. Hence, we utilize powerful language models to paraphrase the Boolean expressions to the form of natural language.
% \begin{tcolorbox}[title=Question Paraphraser Prompt]
% \small
% I need to test if the retrieval system can understand the logic of negation (NOT) implied in natural language. Please convert the following expression containing the logic of negation (NOT) into a natural language question.\\
% \\
% $[$BOOLEAN EXPRESSION$]$
% \end{tcolorbox}
\subsection{Cyclic Consistency Filtering}
\label{sec:cyclic_consistency_filtering}
Despite that questions generated by strong language generation models are fluent, coherence is not guaranteed. Inspired by the cyclic consistency in image generation~\cite{CycleGAN2017}, we filter the generated questions by checking the cyclic consistency in question answering. Formally, we ask the language generation model whether the passage contains the answer to questions generated from the passage itself. Only questions that can be answered with their associated passages are deemed to be valid questions. After the cyclic consistency filtering, 1151 and 1258 questions are obtained based on MS MARCO and Natural Questions, respectively.
% \begin{tcolorbox}[title=Answerer Prompt]
% \small
% Please provide a brief answer to the following question according to the given paragraph(s). If the question cannot be answered by the paragraph(s), please respond with "Cannot answer".\\
% \\
% question:\\
% $[$QUESTION$]$\\
% \\
% paragraphs:\\
% $[$PARAGRAPHS$]$
% \end{tcolorbox}
\section{Data Analysis}
In this section, we detail statistics of the proposed \datasetname, analyze the distribution of question types and display examples to provide a more intuitive understanding of our data.
\begin{table}[ht]
\centering
\small
\begin{tabular}{ccc}
\toprule
Statistics & \makecell{BQ-MARCO} & \makecell{BQ-NQ} \\
\midrule
\#questions (ALL) & 1151 & 1258 \\
avg \#pos (ALL) & 1.27 & 1.32 \\
avg \#neg (ALL) & 0.63 & 0.38 \\
\midrule
\#questions (AND) & 354 & 403 \\
avg \#pos (AND) & 1.00 & 1.00 \\
avg \#neg (AND) & 0.94 & 0.59 \\
\midrule
\#questions (OR) & 469 & 485 \\
avg \#pos (OR) & 1.58 & 1.74 \\
avg \#neg (OR) & 0.35 & 0.21 \\
\midrule
\#questions (NOT) & 328 & 370 \\
avg \#pos (NOT) & 1.13 & 1.11 \\
avg \#neg (NOT) & 0.69 & 0.37 \\
\bottomrule
\end{tabular}
\caption{Data statistics of \datasetname{} built upon MS MARCO (BQ-MARCO) and Natural Questions~(BQ-NQ).}
\label{tab:data_statictics}
\end{table}
\begin{table}[ht]
\centering
\small
% \setlength{\extrarowheight}{1pt}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c|cc}
\toprule
& pos (prediction) & neg (prediction) \\
\hline
pos (ground truth) & 62.50\% & 9.38\% \\
neg (ground truth) & 3.13\% & 25.00\% \\
\bottomrule
\end{tabular}
\caption{Confusion matrix of 32 random samples in \datasetname{} under human evaluation.}
\label{tab:confusion_matrix}
\end{table}
\subsection{Data Statistics}
\label{sec:data_statistics}
\begin{table*}[t]
\vspace{-1em}
\centering
\small
\begin{tabular}{cp{13cm}}
\toprule
\makecell[c]{Question Type} & Example(s) \\
\midrule
\operator{AND} &
\textbf{Boolean Question:} How can I start a \keyinfo{career in the accounting field} \operator{and} pursue an \keyinfo{online degree program}?\newline
\textbf{Paragraph A (Positive):} If you want to work toward a \keyinfo{career in the accounting field}, take the first steps with Penn Foster College. Contact us to learn more about our \keyinfo{online Accounting degree program}. ...\newline
\textbf{Simple Question for Paragraph A:} Is Penn Foster College's \keyinfo{online Accounting degree program} designed for those aiming to start a career in accounting?\newline
\textbf{Paragraph B (Negative):} And indeed, if you're speaking primarily to promote your \negkeyinfo{business} ' say, you're offering a seminar on tax preparation ...\newline
\textbf{Simple Question for Paragraph B:} Do you need to charge for a seminar if promoting your \negkeyinfo{accounting} firm?
\\
\midrule
\operator{OR} &
\textbf{Boolean Question:} What are the impacts of \keyinfo{global warming} \operator{or} \keyinfo{climate change} on nature and humans?\newline
\textbf{Paragraph A (Positive):} Global \keyinfo{climate change} will affect people and the environment in many ways. Some of these impacts, like stronger hurricanes and severe heat waves, ...\newline
\textbf{Simple Question for Paragraph A:} What are the potential impacts of global \keyinfo{climate change} on people and the environment?\newline
\textbf{Paragraph B (Positive):} \keyinfo{Global warming} is harming the environment in several ways including: 1 Desertification. 2 Increased melting of snow and ice. ...\newline
\textbf{Simple question for Paragraph B:} What environmental issues are caused by \keyinfo{global warming}?
\\
\midrule
\operator{NOT} &
\textbf{Boolean Questions:} What causes \keyinfo{upper abdomen pain} \operator{but is unrelated to} \negkeyinfo{liver issues}?\newline
\textbf{Paragraph A (Positive):} Pain originating in the stomach or esophagus is often felt in the \keyinfo{upper abdomen} and can be due to heartburn, gastroesophageal reflux disease (GERD), or hiatal ...\newline
\textbf{Simple Question for Paragraph A:} What causes \keyinfo{upper abdomen pain} linked to the stomach or esophagus?\newline
\textbf{Paragraph B (Negative):} This pain is usually felt in the \keyinfo{upper right part of the abdomen}, often under the rib cage, and is almost always associated with \negkeyinfo{a swelling or enlargement of the liver}, acute inflammation or distention of the liver's surface, or any other ...\newline
\textbf{Simple Question for Paragraph B:} What symptoms are linked with \negkeyinfo{liver enlargement} or injury?
\\
\bottomrule
\end{tabular}
\caption{Examples of \datasetname{} built upon MS MARCO corpus. We show in \operator{blue bold} natural language phrases indicating the logical operation, \keyinfo{green bold italics} key information to retrieve the positive passages, and \negkeyinfo{red bold italics} key information to exclude the negative passages.}
\label{tab:detailed_examples}
\end{table*}
We display the number of questions, average number of positives and negatives for each question of \datasetname{} built upon MS MARCO and Natural Questions. Statistics are calculated on whole datasets and subsets of each type of question individually. Notably, 500 questions are initially generated for each type and we only provide statistics for the final datasets which are filtered by cyclic consistency mentioned in Section~\ref{sec:cyclic_consistency_filtering}.
As AND questions are constructed with the logical conjunction of a disjunctive and a simple question, there is only 1 positive passage for each AND question. OR questions are built by disjunction of several simple questions, and thus more than 1 passages are annotated as positives. NOT questions also have more than 1 positive passage on average since they are collected by removing irrelevant passages within a cluster, where left passages are labeled as positives.
\subsection{Data Quality}
This work utilizes generative language models to construct benchmark datasets without human involvement, and thus it is vital to ensure the data quality for more precise evaluation on retrieval models. Several studies~\cite{li-etal-2023-making,zhao-etal-2023-verify,lightman2024lets} have demonstrated that large language models excel at verification tasks compared to generating novel information. Building on this insight, we deem GPT-4 as a robust filter in our data generation process. However, human involvement is critical for the quality assessment of the generated dataset. Therefore, we randomly sample 32 questions from our dataset and manually check the false-negative and false-positive rates of our proposed Cyclic Consistency Filtering. As the confusion matrix in~\ref{tab:confusion_matrix} indicated, the filtering process effectively eliminates most false positives. Meanwhile, discarding false negatives does not significantly compromise the quality of the filtered data.
\subsection{Question Types}
\begin{figure}[ht]
\centering
\includegraphics[width=0.85\columnwidth]{figures/question_sunburst.pdf}
\caption{Question types covered in \datasetname. Questions are firstly grouped by the Boolean logic implied in questions and then heuristically categorized following the method in \citet{yang-etal-2018-hotpotqa}. Colored blocks without labels indicate questions whose types can not be determined.}
\label{fig:question_sunburst}
\end{figure}
Following the approach in~\citet{yang-etal-2018-hotpotqa}, we heuristically analyze the question types of \datasetname, visualized in Figure~\ref{fig:question_sunburst}. \datasetname{} covers a variety of question types including ``what'', ``how'', ``who'', yes/no questions and so on. General questions with leading ``did'' or ``is'' constitute a larger proportion in OR than in other types of questions. One possible explanation is that general questions have a large capacity for questions comprising the disjunction of multiple atomic questions.
\subsection{Data Examples}
We present examples of \datasetname{} in Table~\ref{tab:detailed_examples}. AND and OR questions are easy to create as ``and'' and ``or'' serve as conjunctions in natural language. The main merit of leveraging GPT-4 to generate Boolean questions lies in NOT questions. As the example shown, GPT-4 paraphrases the NOT operator to ``but is unrelated to'' to form a more natural sentence while preserves the Boolean logic. Overall, the quality of generated questions is ensured in our data collection framework.
\section{Experiments}
In this section, we begin by describing the metrics employed in our experiments. We then evaluate the performance of prevalent dense retrievers on the proposed dataset, introduce two baseline methods for approaching the Boolean dense retrieval task, and finally, conduct a detailed analysis of these baseline methods.
\subsection{Metrics for Boolean Dense Retrieval}
Following previous works~\cite{qu-etal-2021-rocketqa,zhang2022adversarial,ren-etal-2021-rocketqav2}, we adopt Mean Reciprocal Rank at top-$k$~(MRR@$k$), which measures the average reciprocal rank of the first retrieved relevant passage, as the main metric of retrieval model performance in our experiments.
In addition, we propose an additional metric named Negative Recall at top-$k$~(NegRecall@$k$) to evaluate the capability of retrieval models for tackling the logic of negation. Formally, NegRecall@$k$ computes the recall of explicit negative passages for the top-$k$ returned retrieval results. Lower NegRecall@$k$ indicates that the retrieval system excludes the related passages in the negation more successfully.
\begin{table*}[!t]
\centering
\small
\begin{tabular}{cp{2.95cm}ccccccccc}
\toprule
\multirow{2.5}{1cm}{Corpus} & \multirow{2.5}{1.2cm}{Model} & MRR@10 $\uparrow$ & \multicolumn{4}{c}{\taskname{} MRR@10 $\uparrow$} & \multicolumn{4}{c}{\taskname{} NegRecall@10 $\downarrow$} \\
\cmidrule(lr){3-3} \cmidrule(lr){4-7} \cmidrule(lr){8-11}
& & ALL & ALL & AND & OR & NOT & ALL & AND & OR & NOT \\
\midrule
\multirow{10}{1cm}{MS MARCO}& distilbert-cos-v5 & 33.78 & 29.11 & 31.46 & 39.44 & 11.80 & \textbf{11.64} & \textbf{3.02} & \textbf{0.00} & 33.08 \\
& MiniLM-L12-cos-v5 & 32.75 & 31.00 & 32.32 & 43.56 & 11.60 & 11.88 & 3.77 & \textbf{0.00} & \textbf{32.82} \\
& MiniLM-L6-cos-v5 & 32.25 & 31.09 & 33.80 & 43.98 & 9.73 & 13.08 & 4.15 & \textbf{0.00} & 36.15 \\
& e5-base & 36.26 & 36.68 & 41.91 & 50.60 & 11.14 & 12.92 & 3.58 & \textbf{0.00} & 36.41 \\
& distilbert-dot-v5 & 37.25 & 37.61 & 39.92 & 53.04 & 13.04 & 19.50 & 5.28 & 0.61 & 54.62 \\
& bert-base-dot-v5 & 38.08 & 39.89 & 44.36 & 55.31 & 13.01 & 17.01 & 4.15 & \textbf{0.00} & 48.72 \\
& distilbert-base-tas-b & 34.43 & 40.00 & 46.60 & 54.48 & 12.18 & 19.98 & 5.28 & \textbf{0.00} & 56.67 \\
& e5-large-v2 & 35.74 & 41.20 & 49.66 & 54.84 & 12.56 & 16.37 & 3.58 & \textbf{0.00} & 47.44 \\
& bge-large-en-v1.5 & 35.73 & 42.47 & 46.50 & \textbf{57.90} & 16.06 & 15.25 & 4.34 & \textbf{0.00} & 42.82 \\
& gte-Qwen2-7B-instruct & \textbf{39.20} & \textbf{43.52} & \textbf{52.01} & 55.54 & \textbf{17.16} & 15.01 & 5.28 & \textbf{0.00} & 40.77 \\
\midrule
\multirow{7}{1cm}{Wikipedia}& distilbert-base-v1 & 52.75 & 28.89 & 28.32 & 37.91 & 17.71 & \textbf{20.39} & \textbf{15.84} & \textbf{14.42} & \textbf{32.42} \\
& dpr-single-nq-base & 56.81 & 35.20 & 36.18 & 43.07 & 23.81 & 35.14 & 34.16 & 23.08 & 46.48 \\
& stella-en-1.5B-v5 & 44.51 & 42.37 & 42.38 & 51.40 & 30.51 & 36.98 & 33.66 & 22.12 & 54.30 \\
& bge-large-en-v1.5 & 57.54 & 64.98 & 67.47 & 78.34 & 44.77 & 61.29 & 55.94 & 39.42 & 87.50 \\
& e5-base-v2 & 64.00 & 67.04 & 72.16 & 80.82 & 43.42 & 61.87 & 55.20 & 45.19 & 85.94 \\
& e5-large-v2 & 65.93 & 68.61 & 74.20 & \textbf{83.48} & 43.04 & 66.01 & 60.64 & 48.08 & 89.06 \\
& gte-Qwen2-7B-instruct & \textbf{66.64} & \textbf{71.55} & \textbf{75.65} & 83.03 & \textbf{52.04} & 70.85 & 66.34 & 56.73 & 89.45 \\
\bottomrule
\end{tabular}
\caption{Performance of strong dense retrieval models on the original dataset and \datasetname. MRR@10 without BDR prefix shows the model performance on the original dataset, while BDR MRR@10 and BDR NegRecall@10 denote the model performance on \datasetname. The performance of the whole dataset and subset for specific type of questions are shown individually.}
\label{tab:main_results}
\end{table*}
\subsection{Performance of Existing Dense Retrieval}
We evaluate several strong dense retrievers on \datasetname, including those with BERT-style architecture and those fine-tuned from large language models that are capable of solving various natural language tasks.
%These models are implemented using Sentence-Transformers~\cite{reimers-gurevych-2019-sentence}.
From the results shown in Table~\ref{tab:main_results} we can observe that higher retrieval performance on the original datasets indicates higher performance on our generated \datasetname. Notably, weaker retrievers like distilbert-base-v1 perform better on the original datasets than \datasetname{}, while stronger retrievers like gte-Qwen2-7B-instruct show better performance on \datasetname{} than the original datasets. However, stronger retrievers also suffer from high NegRecall on \datasetname, indicating that these retrievers return more passages possibly relevant to the queries but do not have the ability to distinguish the true positives.
%From the results shown in Table~\ref{tab:main_results} we can observe that MRR@10 for the whole dataset of \datasetname{} is similar to MRR@10 for the original dataset on MS MARCO's corpus, but significantly lower on Natural Questions' corpus. We attribute this phenomenon to the average number of positive passages. Most question of \datasetname{} has less than 2 annotated positive passages as analyzed in Section~\ref{sec:data_statistics}, while the original datasets of MS MARCO and Natural Questions have less than 2 and much more than 3 positive passages, respectively. Thus MRR@10 on \datasetname{} with Natural Questions' corpus would drop with a reduced number of positives.
Among the performance of AND, OR and NOT subset of \datasetname, OR subset enjoys the best performance, achieving 8.9\% higher MRR@10 than AND subset on average. It can also be explained by the average number of positives. OR Questions own more positive passages than other subsets owing to the nature of logical disjunction in the construction of OR questions.
The most remarkable results lie in the NOT subset of \datasetname. MRR@10 for NOT subset of \datasetname{} is significantly worse than MRR@10 for other subsets. On the corpus of MS MARCO, most negative passages recalled are from the NOT questions in consideration of the limited NegRecall@10 on AND and OR subsets. Furthermore, even those retrievers fine-tuned from large language models that are believed to have strong language skills show the same trends with the traditional retrievers.
In view of these observations mentioned above, we draw the conclusion that logical negation is a challenging problem for current dense retrieval models. This aligns with our intuition that dense retrieval can only model the relevance of texts, but lack the capability of realizing the
explicit irrelevance. We propose two baseline methods to tackle this problem, detailed in the next section.
\begin{table*}[!t]
\vspace{-1em}
\centering
\small
\begin{tabular}{ccccccccc}
\toprule
\multirow{2.6}{*}{Model} & \multicolumn{4}{c}{\taskname{} MRR@10 $\uparrow$} & \multicolumn{4}{c}{\taskname{} NegRecall@10 $\downarrow$} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9}
& ALL & AND & OR & NOT & ALL & AND & OR & NOT \\
\midrule
\multicolumn{1}{l}{distilbert-dot-v5} & \textbf{37.61} & \textbf{39.92} & \textbf{53.04} & 13.04 & 19.50 & 5.28 & 0.61 & 54.62 \\
\multicolumn{1}{l}{Decomposed Query} & 32.34 & 26.73 & 51.55 & 10.93 & \textbf{2.73} & 3.21 & 0.61 & \textbf{3.85} \\
\multicolumn{1}{l}{Boolean Contrastive (AND)} & 35.98 & 37.44 & 51.49 & 12.24 & 10.87 & \textbf{2.41} & \textbf{0.00} & 31.44 \\
\multicolumn{1}{l}{Boolean Contrastive (OR)} & 37.23 & 39.07 & 52.92 & 12.83 & 11.29 & 2.50 & \textbf{0.00} & 32.65 \\
\multicolumn{1}{l}{Boolean Contrastive (NOT)} & 35.96 & 36.89 & 50.81 & \textbf{13.71} & 13.72 & 3.77 & \textbf{0.00} & 38.72 \\
\bottomrule
\end{tabular}
\caption{Performance comparison of baselines on \datasetname. ``Decomposed Query'' and ``Boolean Contrastive'' denote the Boolean operation on decomposed query and Boolean contrastive continuous training on corresponding train set, respectively. The performance of the whole dataset and each subset are shown individually.}
\label{tab:baseline_results}
\end{table*}
\subsection{Baseline Methods}
We propose two baseline methods as a starting point for tackling the Boolean logic implied in language queries:
\begin{itemize}
\item \textbf{Boolean Operation on Decomposed Query} Inverted index is directly incorporated into the dense retrieval procedure in this baseline method. We ask the large language model to decompose complex Boolean questions into a Boolean expression of simple questions. For each simple question, we retrieve top-$2k$ relevant passages as the candidate retrieval results. Then candidate lists of these simple questions are merged based on the Boolean operator in the Boolean expression. Specifically, set intersection, union and difference are performed under AND, OR and NOT operations, respectively. Scores of candidates are recomputed by addition, maxing and subtraction in these situations. Finally, the merged candidate list is ranked according to the recomputed scores.
\item \textbf{Boolean Contrastive Continuous Training} Following the data-driven schema of dense retrieval models, we propose to conduct continued training on pre-trained dense retrievers. We generate 2000 extra training data for AND, OR and NOT questions. These additional data are mixed with original training data to individually fine-tune the pre-trained model using the same objectives in pre-training. For reproducibility, we only conduct continued training on the distilbert-dot-v5 model whose training codes are publicly available.
\end{itemize}
\subsection{Analysis}
We implement the two baselines with distilbert-dot-v5, and show the results in Table~\ref{tab:baseline_results}. It can be observed from the results that both baselines reduce the NegRecall@$k$ on all data samples. Boolean operation on decomposed query reduces the NegRecall@$10$ from 19.50\% to 2.73\%, nearly eliminating the false positives in the Boolean dense retrieval. However, MRR@10 also suffers from significant drops when conducting Boolean operation on decomposed queries. MRR@10 on the AND subset and the whole dataset drops 13.19\% and 5.27\%, respectively, indicating that many true positives are also rejected by this baseline. This is reasonable since there are various ways to decompose a complex question and the logic of decomposed questions and original questions may be not consistent. Na\"ive set subtraction on the retrieval results of decomposed questions exposes the low quality of question decomposition, resulting in low retrieval performance.
Boolean contrastive training on NOT questions impacts the pre-trained model more slightly, reducing MRR@10 from 37.61\% to 35.96\% and NegRecall@10 from 19.50\% to 13.72\% on the whole dataset. Interestingly, Boolean contrastive continuous training improves the MRR@10 and reduces NegRecall@10 simultaneously on the NOT subset of \datasetname, illustrating the improved ability to understand the logic of negation in natural language. Fine-tuning with AND and OR questions does not significantly improve MRR@10 but reduces NegRecall@10. A possible explanation is that AND and OR questions are relatively easier and fine-tuning on extra training set do not lead to further improvement but harm the performance. In contrast, NOT questions are more challenging for current retrievers, thus fine-tuning with them enhances their ability to tackle such questions. We expect that a better-designed learning-based approach would lead to more robust retrieval systems to address the Boolean dense retrieval task.
These results demonstrate that a dilemma exists in retrieving relevant passages for Boolean questions. While NOT questions are the main type of Boolean questions that are hard to tackle, over-emphasizing them could lead to the exclusion of true positive passages in the returned list.
%NOT questions are the main type of Boolean questions hard to tackle, while putting more emphasis on NOT questions would expel true positive passages in the returned list.
\section{Conclusions}
In this work, we formulate the task of Boolean dense retrieval and investigate whether current dense retrieval systems understand Boolean logic implied in natural language queries. By collecting a benchmark dataset \datasetname{} and evaluating the performance of prevalent dense retrievers, we find that NOT questions are challenging for current dense retrieval systems to understand correctly. Further, we explore Boolean operation on decomposed query and propose a contrastive continual training method that serves as a strong baseline for the research community.
\section*{Limitations}
Dataset collected in this work is generated by large language models. We have not conduct human annotation or filtering on the generated dataset. Due to limited budget, the number of samples are not large enough to provide a complete training dataset. The diversity and coverage of proposed dataset are limited by the MS MARCO and Natural Questions dataset since we build our dataset based on their corpus. Besides, we only focus on English, evaluations on other languages is limited.
\section*{Acknowledgements}
This work is supported by National Key R\&D Program of China under Contract 2022ZD0119802, National Natural Science Foundation of China under Contract 623B2097 and the Youth Innovation Promotion Association CAS. It was supported by GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and the Supercomputing Center of the USTC. This work was also supported by Ant Group Research Fund.
% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{anthology,custom}
\appendix
% \section{Example Appendix}
% \label{sec:appendix}
% This is an appendix.
\clearpage
\section{Prompts}
\label{app:prompts}
\subsection{System Message for Questioner:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
You are an experienced questioner and retrieval system tester. You need to generate questions based on the given paragraphs and related instructions, which will be used as queries to test if the retrieval system can understand the Boolean logic contained in natural language. The questions you pose should align as closely as possible with the retrieval system's scenario, meaning the language style of the questions should resemble that of a search engine user. Besides, please vary your expressions more and avoid sticking to just a few ways of saying things. Note, you only need to output one question no longer than 32 words, without any extra content.
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{System Message for Answerer:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
You are an expert answerer who needs to provide answers to the questions based on the given paragraphs. If the question can be answered by the paragraph(s), please provide a brief answer. If the question cannot be answered by the paragraph(s), please respond with "Cannot answer". Note, you only need to output one answer no longer than 64 words or "Cannot answer", without any extra content.
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for Simple Questioner:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
Please propose a question that can be answered by the following paragraph.\\
\\
$[$PARAGRAPH$]$
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for Disjuntive Questioner:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
Please propose a question that can be answered by any of the following paragraphs. Please make sure that each paragraph can provide answers to the question individually.\\
\\
$[$PARAGRAPH$]$
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for AND Question Converter:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
I need to test whether the retrieval system can understand the logical conjunction (AND) implied in natural language. Please generate a new question by adding constraints to the question "[QUESTION]", so that only paragraphs marked with [positive] provide the answer to the new question, while paragraphs marked with [negative] cannot provide the answer.\\
\\
$[$POSITIVE PARAGRAPHS$]$\\
\\
$[$NEGATIVE PARAGRAPHS$]$
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for OR Question Converter:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
I need to test if the retrieval system can understand the logical disjunction (OR) implied in natural language. Please convert the following expression containing the logical disjunction (OR) into a natural language question.\\
\\
$[$LOGICAL EXPRESSION$]$
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for NOT Question Converter:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
I need to test if the retrieval system can understand the logic of negation (NOT) implied in natural language. Please convert the following expression containing the logic of negation (NOT) into a natural language question.\\
\\
$[$LOGICAL EXPRESSION$]$
\end{em}
\end{tcolorbox}
\end{quote}
\subsection{Prompt for Answerer:}
\begin{quote}
\small
\begin{tcolorbox}[size=normal,opacityfill=0.05]
\begin{em}
Please provide a brief answer to the following question according to the given paragraph(s). If the question cannot be answered by the paragraph(s), please respond with "Cannot answer".\\
\\
question:\\
$[$QUESTION$]$\\
\\
paragraphs:\\
$[$PARAGRAPHS$]$
\end{em}
\end{tcolorbox}
\end{quote}
\end{document}
\begin{thebibliography}{45}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Arabzadeh et~al.(2021)Arabzadeh, Mitra, and
Bagheri}]{Arabzadeh2021MSMARCOChemeleons}
Negar Arabzadeh, Bhaskar Mitra, and Ebrahim Bagheri. 2021.
\newblock \href {https://doi.org/10.1145/3459637.3482011} {Ms marco chameleons:
Challenging the ms marco leaderboard with extremely obstinate queries}.
\newblock In \emph{Proceedings of the 30th ACM International Conference on
Information \& Knowledge Management}, CIKM '21, page 4426–4435, New York,
NY, USA. Association for Computing Machinery.
\bibitem[{Bajaj et~al.(2018)Bajaj, Campos, Craswell, Deng, Gao, Liu, Majumder,
McNamara, Mitra, Nguyen, Rosenberg, Song, Stoica, Tiwary, and
Wang}]{Bajaj2018MS}
Payal Bajaj, Daniel Campos, Nick Craswell, Li~Deng, Jianfeng Gao, Xiaodong Liu,
Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg,
Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018.
\newblock \href {https://arxiv.org/abs/1611.09268} {{MS MARCO}: A human
generated machine reading comprehension dataset}.
\newblock \emph{arXiv:1611.09268}.
\bibitem[{Bonifacio et~al.(2021)Bonifacio, Jeronymo, Abonizio, Campiotti,
Fadaee, , Lotufo, and Nogueira}]{bonifacio2021mmarco}
Luiz~Henrique Bonifacio, Vitor Jeronymo, Hugo~Queiroz Abonizio, Israel
Campiotti, Marzieh Fadaee, , Roberto Lotufo, and Rodrigo Nogueira. 2021.
\newblock \href {https://arxiv.org/abs/2108.13897} {mmarco: A multilingual
version of ms marco passage ranking dataset}.
\newblock \emph{Preprint}, arXiv:2108.13897.
\bibitem[{Bookstein(1980)}]{bookstein1980fuzzy}
Abraham Bookstein. 1980.
\newblock \href
{https://www.proquest.com/scholarly-journals/fuzzy-requests-approach-weighted-boolean-searches/docview/216636606/se-2}
{Fuzzy requests: An approach to weighted boolean searches}.
\newblock \emph{Journal of the American Society for Information Science
(pre-1986)}, 31(4):240.
\bibitem[{Büttcher et~al.(2016)Büttcher, Clarke, and Cormack}]{Stefan2016}
Stefan Büttcher, Charles L.~A. Clarke, and Gordon~V. Cormack. 2016.
\newblock \emph{Information Retrieval: Implementing and Evaluating Search
Engines}.
\newblock The MIT Press.
\bibitem[{Chen et~al.(2017{\natexlab{a}})Chen, Fisch, Weston, and
Bordes}]{chen-etal-2017-reading}
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/P17-1171} {Reading {W}ikipedia to
answer open-domain questions}.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1870--1879,
Vancouver, Canada. Association for Computational Linguistics.
\bibitem[{Chen et~al.(2017{\natexlab{b}})Chen, Liu, Yin, and
Tang}]{Chen2017SurveyDialogue}
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017{\natexlab{b}}.
\newblock \href {https://doi.org/10.1145/3166054.3166058} {A survey on dialogue
systems: Recent advances and new frontiers}.
\newblock \emph{ACM SIGKDD Explorations Newsletter}, 19.
\bibitem[{Clark et~al.(2019)Clark, Lee, Chang, Kwiatkowski, Collins, and
Toutanova}]{clark-etal-2019-boolq}
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1300} {{B}ool{Q}: Exploring
the surprising difficulty of natural yes/no questions}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 2924--2936,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Fan et~al.(2022)Fan, Xie, Cai, Chen, Ma, Li, Zhang, Guo
et~al.}]{fan2022pre}
Yixing Fan, Xiaohui Xie, Yinqiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li,
Ruqing Zhang, Jiafeng Guo, et~al. 2022.
\newblock Pre-training methods in information retrieval.
\newblock \emph{Foundations and Trends{\textregistered} in Information
Retrieval}, 16(3):178--317.
\bibitem[{Guo et~al.(2022)Guo, Cai, Fan, Sun, Zhang, and
Cheng}]{guo2022semantic}
Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng.
2022.
\newblock \href {https://doi.org/10.1145/3486250} {Semantic models for the
first-stage retrieval: A comprehensive review}.
\newblock \emph{ACM Transactions on Information Systems}, 40(4).
\bibitem[{Hochreiter and Schmidhuber(1997)}]{hochreiter1997long}
Sepp Hochreiter and J{\"u}rgen Schmidhuber. 1997.
\newblock Long short-term memory.
\newblock \emph{Neural computation}, 9(8):1735--1780.
\bibitem[{Karpukhin et~al.(2020)Karpukhin, Oguz, Min, Lewis, Wu, Edunov, Chen,
and Yih}]{karpukhin-etal-2020-dense}
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey
Edunov, Danqi Chen, and Wen-tau Yih. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.550} {Dense
passage retrieval for open-domain question answering}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 6769--6781, Online. Association
for Computational Linguistics.
\bibitem[{Kwiatkowski et~al.(2019)Kwiatkowski, Palomaki, Redfield, Collins,
Parikh, Alberti, Epstein, Polosukhin, Devlin, Lee, Toutanova, Jones, Kelcey,
Chang, Dai, Uszkoreit, Le, and Petrov}]{kwiatkowski-etal-2019-natural}
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang,
Andrew~M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.
\newblock \href {https://doi.org/10.1162/tacl_a_00276} {Natural questions: A
benchmark for question answering research}.
\newblock \emph{Transactions of the Association for Computational Linguistics},
7:452--466.
\bibitem[{Lewis et~al.(2020)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal,
K\"{u}ttler, Lewis, Yih, Rockt\"{a}schel, Riedel, and Kiela}]{Lewis2020rag}
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich K\"{u}ttler, Mike Lewis, Wen-tau Yih, Tim
Rockt\"{a}schel, Sebastian Riedel, and Douwe Kiela. 2020.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf}
{Retrieval-augmented generation for knowledge-intensive nlp tasks}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~33, pages 9459--9474. Curran Associates, Inc.
\bibitem[{Li et~al.(2023)Li, Lin, Zhang, Fu, Chen, Lou, and
Chen}]{li-etal-2023-making}
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and
Weizhu Chen. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.acl-long.291} {Making
language models better reasoners with step-aware verifier}.
\newblock In \emph{Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 5315--5333,
Toronto, Canada. Association for Computational Linguistics.
\bibitem[{Lightman et~al.(2024)Lightman, Kosaraju, Burda, Edwards, Baker, Lee,
Leike, Schulman, Sutskever, and Cobbe}]{lightman2024lets}
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker,
Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024.
\newblock \href {https://openreview.net/forum?id=v8L0pN6EOi} {Let's verify step
by step}.
\newblock In \emph{The International Conference on Learning Representations}.
\bibitem[{Liu et~al.(2023)Liu, Yuan, Fu, Jiang, Hayashi, and
Neubig}]{liu2023pretrain}
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and
Graham Neubig. 2023.
\newblock \href {https://doi.org/10.1145/3560815} {Pre-train, prompt, and
predict: A systematic survey of prompting methods in natural language
processing}.
\newblock \emph{ACM Computing Surveys}, 55(9).
\bibitem[{Maia et~al.(2018)Maia, Handschuh, Freitas, Davis, McDermott, Zarrouk,
and Balahur}]{Maia2018financial}
Macedo Maia, Siegfried Handschuh, Andr\'{e} Freitas, Brian Davis, Ross
McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.
\newblock \href {https://doi.org/10.1145/3184558.3192301} {Www'18 open
challenge: Financial opinion mining and question answering}.
\newblock In \emph{Companion Proceedings of the The Web Conference 2018}, WWW
'18, page 1941–1942, Republic and Canton of Geneva, CHE. International
World Wide Web Conferences Steering Committee.
\bibitem[{Malaviya et~al.(2023)Malaviya, Shaw, Chang, Lee, and
Toutanova}]{malaviya-etal-2023-quest}
Chaitanya Malaviya, Peter Shaw, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.acl-long.784} {{QUEST}: A
retrieval dataset of entity-seeking queries with implicit set operations}.
\newblock In \emph{Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 14032--14047,
Toronto, Canada. Association for Computational Linguistics.
\bibitem[{Mitra et~al.(2017)Mitra, Diaz, and Craswell}]{mitra-2017-learning}
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017.
\newblock \href {https://doi.org/10.1145/3038912.3052579} {Learning to match
using local and distributed representations of text for web search}.
\newblock In \emph{Proceedings of the 26th International Conference on World
Wide Web, {WWW} 2017, Perth, Australia, April 3-7, 2017}, pages 1291--1299.
{ACM}.
\bibitem[{Petroni et~al.(2021)Petroni, Piktus, Fan, Lewis, Yazdani, De~Cao,
Thorne, Jernite, Karpukhin, Maillard, Plachouras, Rockt{\"a}schel, and
Riedel}]{petroni-etal-2021-kilt}
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani,
Nicola De~Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean
Maillard, Vassilis Plachouras, Tim Rockt{\"a}schel, and Sebastian Riedel.
2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.200} {{KILT}: a
benchmark for knowledge intensive language tasks}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 2523--2544, Online. Association for Computational
Linguistics.
\bibitem[{Qu et~al.(2021)Qu, Ding, Liu, Liu, Ren, Zhao, Dong, Wu, and
Wang}]{qu-etal-2021-rocketqa}
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne~Xin Zhao, Daxiang
Dong, Hua Wu, and Haifeng Wang. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.466}
{{R}ocket{QA}: An optimized training approach to dense passage retrieval for
open-domain question answering}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 5835--5847, Online. Association for Computational
Linguistics.
\bibitem[{Reimers and Gurevych(2019)}]{reimers-gurevych-2019-sentence}
Nils Reimers and Iryna Gurevych. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1410} {Sentence-{BERT}:
Sentence embeddings using {S}iamese {BERT}-networks}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 3982--3992, Hong Kong,
China. Association for Computational Linguistics.
\bibitem[{Ren et~al.(2021)Ren, Qu, Liu, Zhao, She, Wu, Wang, and
Wen}]{ren-etal-2021-rocketqav2}
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne~Xin Zhao, QiaoQiao She, Hua Wu, Haifeng
Wang, and Ji-Rong Wen. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.224}
{{R}ocket{QA}v2: A joint training method for dense passage retrieval and
passage re-ranking}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 2825--2835, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Robertson and Zaragoza(2009)}]{Robertson2009Probabilistic}
Stephen Robertson and Hugo Zaragoza. 2009.
\newblock \href {https://doi.org/10.1561/1500000019} {The probabilistic
relevance framework: {BM25} and {Beyond}}.
\newblock \emph{Foundations and Trends\textregistered{} in Information
Retrieval}, 3(4):333--389.
\bibitem[{Salton and Buckley(1988)}]{SALTON1988termweighting}
Gerard Salton and Christopher Buckley. 1988.
\newblock \href {https://doi.org/10.1016/0306-4573(88)90021-0} {Term-weighting
approaches in automatic text retrieval}.
\newblock \emph{Information Processing \& Management}, 24(5):513--523.
\bibitem[{Salton et~al.(1983)Salton, Fox, and Wu}]{Salton1983extended}
Gerard Salton, Edward~A. Fox, and Harry Wu. 1983.
\newblock \href {https://doi.org/10.1145/182.358466} {Extended boolean
information retrieval}.
\newblock \emph{Communications of the ACM}, 26(11):1022–1036.
\bibitem[{Salton et~al.(1975)Salton, Wong, and Yang}]{salton1975vector}
Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975.
\newblock A vector space model for automatic indexing.
\newblock \emph{Communications of the ACM}, 18(11):613--620.
\bibitem[{Shuster et~al.(2021)Shuster, Poff, Chen, Kiela, and
Weston}]{shuster-etal-2021-retrieval-augmentation}
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-emnlp.320}
{Retrieval augmentation reduces hallucination in conversation}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2021}, pages 3784--3803, Punta Cana, Dominican Republic. Association
for Computational Linguistics.
\bibitem[{Thakur et~al.(2021)Thakur, Reimers, R{\"u}ckl{\'e}, Srivastava, and
Gurevych}]{thakur2021beir}
Nandan Thakur, Nils Reimers, Andreas R{\"u}ckl{\'e}, Abhishek Srivastava, and
Iryna Gurevych. 2021.
\newblock \href {https://openreview.net/forum?id=wCu6T5xFjeJ} {{BEIR}: A
heterogeneous benchmark for zero-shot evaluation of information retrieval
models}.
\newblock In \emph{Thirty-fifth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track (Round 2)}.
\bibitem[{Tsatsaronis et~al.(2015)Tsatsaronis, Balikas, Malakasiotis, Partalas,
Zschunke, Alvers, Weissenborn, Krithara, Petridis, Polychronopoulos,
Almirantis, Pavlopoulos, Baskiotis, Gallinari, Arti{\`e}res, Ngomo, Heino,
Gaussier, Barrio-Alvers, Schroeder, Androutsopoulos, and
Paliouras}]{Tsatsaronis2015AnOO}
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas,
Matthias Zschunke, Michael~R. Alvers, Dirk Weissenborn, Anastasia Krithara,
Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John
Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Arti{\`e}res,
Axel-Cyrille~Ngonga Ngomo, Norman Heino, {\'E}ric Gaussier, Liliana
Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios
Paliouras. 2015.
\newblock \href {https://api.semanticscholar.org/CorpusID:7903613} {An overview
of the bioasq large-scale biomedical semantic indexing and question answering
competition}.
\newblock \emph{BMC Bioinformatics}, 16.
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{Vaswani2017Attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, \L~ukasz Kaiser, and Illia Polosukhin. 2017.
\newblock \href
{https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf}
{Attention is all you need}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~30. Curran Associates, Inc.
\bibitem[{Wadden et~al.(2020)Wadden, Lin, Lo, Wang, van Zuylen, Cohan, and
Hajishirzi}]{wadden-etal-2020-fact}
David Wadden, Shanchuan Lin, Kyle Lo, Lucy~Lu Wang, Madeleine van Zuylen, Arman
Cohan, and Hannaneh Hajishirzi. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.609} {Fact or
fiction: Verifying scientific claims}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 7534--7550, Online. Association
for Computational Linguistics.
\bibitem[{Waller and Kraft(1979)}]{WALLER1979mathematical}
W.G. Waller and Donald~H. Kraft. 1979.
\newblock \href {https://doi.org/10.1016/0306-4573(79)90030-X} {A mathematical
model of a weighted boolean retrieval system}.
\newblock \emph{Information Processing \& Management}, 15(5):235--245.
\bibitem[{Yang et~al.(2018)Yang, Qi, Zhang, Bengio, Cohen, Salakhutdinov, and
Manning}]{yang-etal-2018-hotpotqa}
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan
Salakhutdinov, and Christopher~D. Manning. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-1259} {{H}otpot{QA}: A dataset
for diverse, explainable multi-hop question answering}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 2369--2380, Brussels, Belgium.
Association for Computational Linguistics.
\bibitem[{Yates et~al.(2021)Yates, Nogueira, and
Lin}]{yates-etal-2021-pretrained}
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-tutorials.1}
{Pretrained transformers for text ranking: {BERT} and beyond}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies: Tutorials}, pages 1--4, Online. Association for Computational
Linguistics.
\bibitem[{Zhai(2007)}]{zhai-2007-statistical}
ChengXiang Zhai. 2007.
\newblock \href {https://aclanthology.org/N07-5002} {Statistical language
models for information retrieval}.
\newblock In \emph{Proceedings of the Human Language Technology Conference of
the {NAACL}, Companion Volume: Tutorial Abstracts}, pages 3--4, Rochester,
New York. Association for Computational Linguistics.
\bibitem[{Zhang et~al.(2022)Zhang, Gong, Shen, Lv, Duan, and
Chen}]{zhang2022adversarial}
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen.
2022.
\newblock \href {https://openreview.net/forum?id=MR7XubKUFB} {Adversarial
retriever-ranker for dense text retrieval}.
\newblock In \emph{The International Conference on Learning Representations}.
\bibitem[{Zhao et~al.(2023)Zhao, Li, Joty, Qin, and
Bing}]{zhao-etal-2023-verify}
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.acl-long.320}
{Verify-and-edit: A knowledge-enhanced chain-of-thought framework}.
\newblock In \emph{Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 5823--5840,
Toronto, Canada. Association for Computational Linguistics.
\bibitem[{Zhao et~al.(2024)Zhao, Liu, Ren, and Wen}]{zhao2024densesurvey}
Wayne~Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024.
\newblock \href {https://doi.org/10.1145/3637870} {Dense text retrieval based
on pretrained language models: A survey}.
\newblock \emph{ACM Transactions on Information Systems}, 42(4).
\bibitem[{Zhong et~al.(2023)Zhong, Shi, Yih, and
Zettlemoyer}]{zhong-etal-2023-romqa}
Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.findings-emnlp.470}
{{R}o{MQA}: A benchmark for robust, multi-evidence, multi-answer question
answering}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2023}, pages 7055--7067, Singapore. Association for Computational
Linguistics.
\bibitem[{Zhu et~al.(2017)Zhu, Park, Isola, and Efros}]{CycleGAN2017}
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei~A Efros. 2017.
\newblock Unpaired image-to-image translation using cycle-consistent
adversarial networks.
\newblock In \emph{Computer Vision (ICCV), 2017 IEEE International Conference
on}.
\bibitem[{Zobel and Moffat(2006)}]{zobel2006inverted}
Justin Zobel and Alistair Moffat. 2006.
\newblock \href {https://doi.org/10.1145/1132956.1132959} {Inverted files for
text search engines}.
\newblock \emph{ACM Computing Surveys}, 38(2):6–es.
\bibitem[{Zobel et~al.(1998)Zobel, Moffat, and
Ramamohanarao}]{zobel1998inverted}
Justin Zobel, Alistair Moffat, and Kotagiri Ramamohanarao. 1998.
\newblock \href {https://doi.org/10.1145/296854.277632} {Inverted files versus
signature files for text indexing}.
\newblock \emph{ACM Transactions on Database Systems}, 23(4):453–490.
\end{thebibliography}