Augmenting Passage Representations with Query Generation for Enhanced Cross-Lingual Dense Retrieval
Extracted source text ยท arXiv 2305.03950
%%
%% This is file `sample-sigconf.tex',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% samples.dtx (with options: `sigconf')
%%
%% IMPORTANT NOTICE:
%%
%% For the copyright see the source file.
%%
%% Any modified versions of this file must be renamed
%% with new filenames distinct from sample-sigconf.tex.
%%
%% For distribution of the original source see the terms
%% for copying and modification in the file samples.dtx.
%%
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%%
%% The first command in your LaTeX source must be the \documentclass command.
%\documentclass[sigconf]{acmart}
\documentclass[sigconf]{acmart}
\usepackage{amsmath,nccmath,tabularx} %amssymb
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{color}
\usepackage{graphicx}
\graphicspath{{images/}}
\usepackage{wrapfig,lipsum}
\usepackage{tabularx}
\usepackage{graphicx}
\usepackage{lscape}
\usepackage{subcaption}
\usepackage{latexsym}
%\usepackage{amssymb}
\usepackage{pifont}% http://ctan.org/pkg/pifont
%\usepackage{subfig}
\usepackage{multirow}
\newcommand\todo[1]{\textcolor{red}{#1}}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{enumitem}
\setlist{leftmargin=3mm}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[font=small,labelfont=bf]{caption}
\usepackage{babel}
\newcommand\fh{\babelhyphen{hard}}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{xspace}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{placeins}
%\usepackage{natbib}
\newcommand{\linjun}[1]{[{\em{linjun}: \textcolor{red}{#1}}]}
\hyphenation{ap-proach com-ponent rep-re-sen-ta-tions mis-spelled mech-a-nism re-trieval qu-eries down-stream ef-fect-ive-ness ar-chi-tec-ture re-triev-ers chal-lenge cir-cum-stances learn-ing va-ri-a-tions ex-plici-tly con-tain-ing Hugging-face check-point bench-mark meth-ods non-mis-spelled con-sist-ent-ly im-prove-ment fur-ther-more ex-per-i-men-tal cor-re-spond-ing en-cap-su-la-te mag-ni-tudes check-points con-tri-bu-tion mul-ti-lin-gual ad-di-tion-al in-for-ma-tion con-tributed knowl-edge ag-gre-ga-tion an-no-tat-ed specif-i-cal-ly ques-tions ac-knowl-edge in-ves-ti-ga-tion hy-po-thet-i-cal re-searched em-beddings re-triever out-performs}
%%
%% \BibTeX command to typeset BibTeX logo in the docs
\AtBeginDocument{%
\providecommand\BibTeX{{%
\normalfont B\kern-0.5em{\scshape i\kern-0.25em b}\kern-0.8em\TeX}}}
%% Rights management information. This information is sent to you
%% when you complete the rights form. These commands have SAMPLE
%% values in them; it is your responsibility as an author to replace
%% the commands and values with those provided to you when you
%% complete the rights form.
\copyrightyear{2023}
\acmYear{2023}
\setcopyright{acmlicensed}\acmConference[SIGIR '23]{Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval}{July 23--27, 2023}{Taipei, Taiwan}
\acmBooktitle{Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23), July 23--27, 2023, Taipei, Taiwan}
\acmPrice{15.00}
\acmDOI{10.1145/3539618.3591952}
\acmISBN{978-1-4503-9408-6/23/07}
\settopmatter{printacmref=true}
%%
%% Submission ID.
%% Use this when submitting an article to a sponsored event. You'll
%% receive a unique submission ID from the organizers
%% of the event, and this ID should be used as the parameter to this command.
%%\acmSubmissionID{123-A56-BU3}
%%
%% The majority of ACM publications use numbered citations and
%% references. The command \citestyle{authoryear} switches to the
%% "author year" style.
%%
%% If you are preparing content for an event
%% sponsored by ACM SIGGRAPH, you must use the "author year" style of
%% citations and references.
%% Uncommenting
%% the next command will enable that style.
%%\citestyle{acmauthoryear}
%%
%% end of the preamble, start of the body of the document source.
\begin{document}
%\fancyhead{}
%%
%% The "title" command has an optional parameter,
%% allowing the author to define a "short title" to be used in page headers.
\title[Augmenting Passage Representations with Query Generation]{Augmenting Passage Representations with Query Generation for Enhanced Cross-Lingual Dense Retrieval}
%SIGIR title: A typo-aware pre-training strategy for Robust dense retrievers
%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
%% Of note is the shared affiliation of the first two authors, and the
%% "authornote" and "authornotemark" commands
%% used to denote shared contribution to the research.
\author{Shengyao Zhuang}
\affiliation{%
\institution{The University of Queensland}
\city{Brisbane}
\state{QLD}
\country{Australia}}
\email{s.zhuang@uq.edu.au}
\author{Linjun Shou}
\affiliation{%
\institution{Microsoft STCA}
\city{Beijing}
\country{China}}
\email{lisho@microsoft.com}
\author{Guido Zuccon}
\affiliation{%
\institution{The University of Queensland}
\city{Brisbane}
\state{QLD}
\country{Australia}}
\email{g.zuccon@uq.edu.au}
%%
%% By default, the full list of authors will be used in the page
%% headers. Often, this list is too long, and will overlap
%% other information printed in the page headers. This command allows
%% the author to define a more concise list
%% of authors' names for this purpose.
%%\renewcommand{\shortauthors}{Trovato and Tobin, et al.}
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
%Recent studies have highlighted the limitations of dense retrievers (DRs) in their ability to effectively process misspelled queries, which constitute a significant portion of query traffic in commercial search engines.
Effective cross-lingual dense retrieval methods that rely on multilingual pre-trained language models (PLMs) need to be trained to encompass both the relevance matching task and the cross-language alignment task. However, cross-lingual data for training is often scarcely available.
%, especially for low resource languages.
In this paper, rather than using more cross-lingual data for training, we propose to use cross-lingual query generation to augment passage representations with queries in languages other than the original passage language. These augmented representations are used at inference time so that the representation can encode more information across the different target languages.
Training of a cross-lingual query generator does not require additional training data to that used for the dense retriever. The query generator training is also effective because the pre-training task for the generator (T5 text-to-text training) is very similar to the fine-tuning task (generation of a query). The use of the generator does not increase query latency at inference and can be combined with any cross-lingual dense retrieval method.
Results from experiments on a benchmark cross-lingual information retrieval dataset show that our approach can improve the effectiveness of existing cross-lingual dense retrieval methods. Implementation of our methods, along with all generated query files are made publicly available at \url{https://github.com/ielab/xQG4xDR}.
%Cross-lingual dense retrieval methods that rely on multilingual pre-trained language models (PLMs) face the challenge of aligning query and passage embeddings in different languages. Previous works rely on contrastive learning to directly align query and passage embeddings which requires large among of training data. However, it is often the case that collecting this cross-lingual training data is very expensive. This paper presents a novel approach to address this issue by augmenting passage representations. Instead of encoding query and passage directly with PLMs, we train a cross-lingual query generation model using mT5 to generate queries in different languages for each passage in the corpus. The original passages and the generated queries are then encoded into embeddings and the passage embeddings are augmented with the embeddings of the generated queries. Our approach can be applied to any cross-lingual dense retrieval method without increasing query latency. Results from experiments on a benchmark cross-lingual information retrieval dataset show that our approach can improve the effectiveness of existing cross-lingual dense retrieval methods in both zero-shot and supervised settings.
\end{abstract}
%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10002951.10003317.10003325.10003326</concept_id>
<concept_desc>Information systems~Query representation</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003317.10003338.10003341</concept_id>
<concept_desc>Information systems~Language models</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Information systems~Query representation}
\ccsdesc[500]{Information systems~Language models}
%\ccsdesc[500]{Information systems~Information retrieval query processing}
%%
%%Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
\keywords{Cross-lingual query generation; Cross-lingual retrieval; Dense retriever}
%% A "teaser" image appears between the author and affiliation
%% information and the body of the document, and typically spans the
%% page.
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\input{sections/introduction}
\input{sections/related_work}
\input{sections/method}
\input{sections/setup}
\input{sections/results}
\input{sections/conclusion}
%%
%% The acknowledgments section is defined using the "acks" environment
%% (and NOT an unnumbered section). This ensures the proper
%% identification of the section in the article metadata, and the
%% consistent spelling of the heading.
%\begin{acks}
%The authors need 3 lines of text for acknowledgements.
%\end{acks}
%%
%% The next two lines define the bibliography style to be used, and
%% the bibliography file.
\bibliographystyle{ACM-Reference-Format}
\balance
\bibliography{sigir2023}
%%
%% If your work has an appendix, this is the place to put it.
\appendix
\end{document}
\endinput
%%
%% End of file `sample-sigconf.tex'.
\section{Introduction}
Pre-trained language model-based (PLM) dense retrievers (DRs) have achieved remarkable success in the task of English-only passage retrieval~\cite{karpukhin-etal-2020-dense,xiong2020approximate,zhan2020repbert,Zhan2021OptimizingDR,ren2021pair,ren2021rocketqav2,qu2021rocketqa,lin2021batch,lin2020distilling,hofstatter2020improving,hofstatter2021efficiently}. These models use a dual-encoder architecture that encodes both queries and passages with a PLM encoder into dense embeddings. They then perform approximate nearest neighbor (ANN) searching in the embedding space. Compared to traditional bag-of-words approaches, DRs benefit from semantic soft matching, which helps overcome the problem of word mismatch in passage retrieval~\cite{tonellotto2022lecture,zhao2022dense}.
To leverage the semantic modelling power of DRs, recent research has extended English-only DRs to support cross-lingual settings~\cite{asai2021one,sorokin2022ask,li2022learning,ren2022empowering,xorqa,longpre2021mkqa}, i.e, where queries and passages are in different languages. This is achieved using multi-lingual PLMs, such as multilingual BERT~\cite{DBLP:conf/naacl/DevlinCLT19}, in place of the English-only PLMs. This approach is particularly important in this setting where traditional bag-of-words methods are ineffective due to the limited number of matching terms across languages. In contrast, cross-lingual DRs (xDRs) are able to encode queries and passages in different languages into a shared embedding space, enabling efficient ANN search across languages.
However, such multi-lingual PLM-based xDRs usually are less effective on the cross-lingual passage retrieval task than DRs in the English-only setting~\cite{xorqa}.
The hypothesis to explain this result is that, in the English-only setting, a DR only needs to learn relevance matching between queries and passages. In contrast, a xDR not only needs to learn the relevance matching task, but also needs to learn how to align the embeddings of texts with similar semantic meaning but in different language~\cite{nair2022transfer,yang2022c3}. It is this language gap that makes cross lingual retrieval a relatively harder task for xDRs.
Based on this hypothesis, this paper proposes the use of cross-lingual query generation (xQG) to bridge the language gap for xDRs. Our approach is illustrated in Figure~\ref{illustration}. Specifically, we fine-tune a multilingual T5 (mT5) model using language-specific prompts to generate several queries per passage for each target language. In the indexing phase, we use a given xDR model to encode passages and their associated generated queries in different languages into embeddings. Finally, we augment the original passage embeddings with the embeddings of the generated queries before adding them to the ANN index. By doing so, we move the passage embeddings to a space that is closer to the target user query language. Our approach does not add extra query latency and can be applied to any existing xDRs.
\begin{figure*}
\centering
\includegraphics[width=1\textwidth]{images/illustration.png}
\vspace{-25pt}
\caption{Augmenting passage representations with cross-lingual generated query embeddings. The query examples shown in this figure were generated using our trained xQG model. For each query, we report the corresponding translation obtained using Google's translation service. \vspace{-12pt}}
\label{illustration}
\end{figure*}
\vspace{-10pt}
\section{Related works}
\textbf{\textit{Cross-lingual dense retrievers.}}
The development of cross-lingual pre-trained language models has significantly contributed to the progress of cross-lingual dense retrieval (xDR)~\cite{conneau2020unsupervised,wu2022unsupervised,chi2021infoxlm,feng2022language}. Notable examples of recent xDR methods include CORA~\cite{asai2021one}, which employs a generator to facilitate retrieval training data mining, Sentri~\cite{sorokin2022ask}, which proposes a single encoder and self-training, and DR.DECR~\cite{li2022learning}, which utilizes parallel queries and sentences for cross-lingual knowledge distillation. Among these, our work is most closely related to QuiCK~\cite{ren2022empowering}, which also utilizes a cross-lingual query generator for xDR. However, unlike QuiCK, which uses the xQG as a teacher model in knowledge distillation for xDR in the training phase, our method directly augments passage embeddings with xQG queries without any xDR training involved.
\textbf{\textit{Query generation for information retrieval.}} Query generation is a well-established technique that has been widely used to improve retrieval performance in various retrieval models~\cite{nogueira2019doc2query,gospodinov2023doc2query}. In addition, it has been shown to be effective for domain adaptation in dense passage retrieval tasks~\cite{thakur2021beir,ma2021zero,wang2022gpl}, as well as for enhancing the effectiveness of other PLM-based rankers~\cite{zhuang2022bridging,wangneural}. In our approach, we also rely on a query generation model to generate high-quality queries for downstream passage embedding augmentation tasks.
\textbf{\textit{Augmenting embeddings for dense retrievers.}}
Our method relies on effectively augmenting representations encoded by DR encoders -- a direction recently explored also by other works. For instance, \citet{li2023pseudo,li2022pyserini} use the top retrieved passages as pseudo-relevant feedback to augment query embeddings using the Rocchio aggregation function. Similarly, \citet{zhuang2022implicit} extend this idea by using embeddings of clicked passages to augment query embeddings. Other PRF methods have also been extensively researched to enhance both English-only dense retrieval~\cite{yu2021improving,li2022improving,wang2021pseudo,wang2023colbert,zhu2022lol} and cross-lingual dense retrieval~\cite{chandradevan2022learning}. On the other hand, the HyDE method uses large pre-trained language models to generate hypothetical passages for a given query and directly uses the embedding of the hypothetical passage to perform the search~\cite{gao2022precise}. While all these works focus on augmenting query embeddings for DRs at query time, our work focuses on augmenting passage embeddings \textit{at indexing time}, thereby avoiding the extra overhead in terms of query latency.
\vspace{-8pt}
\section{Method}
\vspace{-2pt}
Our approach consists of two components: (1) a cross-lingual query generation model that, for each passage, generates high-quality queries in different languages, and (2) an embedding aggregation function that augments the original passage embedding with the embeddings of the generated queries.
To obtain a xQG model, we fine-tune a mT5 model with labeled relevant $(q_t, p)$ pairs, where $t$ is the target query language. We use a seq2seq objective akin to docTquery-T5's objective~\cite{nogueira2019doc2query}; in our case the input is the passage text with language-specific prompts:
\begin{equation}
\textit{prompt }(t,p) = \textit{Generate a } [t] \textit{ question for this passage: } [p],
\end{equation}
where $[t]$ and $[p]$ are prompt placeholders for the target language and passage text, respectively. Once we have trained a xQG model, we can generate a set of queries for each target language and passage by replacing the placeholders accordingly:
\begin{equation}\small
Q_p =\bigcup_{t\in T} Q_{p}^t; \textit{ }Q_{p}^t=\textit{xQG }(\textit{prompt }(t,p)) \times n,
\end{equation}
where $T$ is the target language set and $Q_p$ is the set of all generated queries for passage $p$ which includes $n$ generated queries for each target language. In our experiments, we use a top-$k$ sampling scheme~\cite{fan2018hierarchical} to generate queries, and set $k=10$. We find that these simple language-specific prompts can effectively lead the T5 model to generate the desired output in that language.
For the embedding aggregation function, we use a Rocchio-like aggregation function that is similar to previous works that aggregated dense representations~\cite{zhuang2022implicit,li2023pseudo}. We use this aggregation function to obtain the embeddings of all passages in the corpus:
\begin{equation}
emb(p, Q_p, \theta, \alpha) = (1-\alpha)\cdot\theta(p) + \alpha \sum_{\hat{q_t}\in Q_p}\theta(\hat{q_t}),
\end{equation}
where $\hat{q_t}$ is a generated query for target language $t$, $\theta$ is the xDR encoder model and $\theta(.)$ is the text embedding given by the xDR encoder. The hyper-parameter $\alpha$ is the augmentation ratio, $\alpha \in [0,1]$. This ratio is used to control the weights assigned to the original passage embedding and the generated query embeddings.
\begin{table*} [t]
\centering
\caption{xDR models trained with different backbone PLMs. Zero-shot means trained with only the English subset of the NQ training data (thus, zero-shot for the cross-lingual task). Statistical differences against the base model (without xQG) are labelled with $^\star$ ($p<0.05$). \vspace{-12pt}}
\begin{subtable}[c]{1\linewidth}
\centering
\caption{R@2kt}
\vspace{-5pt}
\scalebox{0.94}{
\begin{tabular}{l|lllllll|l}
\hline
Model & Ar & Bn & Fi & Ja & Ko & Ru & Te & Average \\ \hline
XLM-R & 28.5 & 26.3 & 29.3 & 22.4 & 34.0 & 17.3 & 34.9 & 27.5 \\
XLM-R + xQG & \textbf{30.1} & \textbf{29.6} & \textbf{31.2} & \textbf{23.7} & \textbf{36.1} & \textbf{19.4} & \textbf{38.2} & \textbf{ 29.8}$^\star$ \\ \hline
mBERT & 41.1 & 49.0 & 52.2 & \textbf{37.3}$^\star$ & 48.1& 33.3 & 47.9& 44.1 \\
mBERT + xQG & \textbf{42.4}& \textbf{54.9}$^\star$ & \textbf{54.1}& 33.6 & \textbf{52.3}$^\star$ & \textbf{33.8} & \textbf{52.5}$^\star$ & \textbf{46.2}$^\star$ \\ \hline
mBERT (zero-shot) &32.6 & 25.0 & 38.2 & 29.5 & 38.9 & 27.0 & 39.9 & 33.0 \\
mBERT (zero-shot) + XQG & \textbf{33.9} & \textbf{28.9}$^\star$ & \textbf{43.0}$^\star$& \textbf{32.8}$^\star$ & \textbf{41.4} & \textbf{29.5} & \textbf{42.0} & \textbf{36.0}$^\star$ \\ \hline
\end{tabular}
}
\end{subtable}\\
% \quad%
\begin{subtable}[c]{1\linewidth}
\centering
\caption{R@5kt}
\vspace{-5pt}
\scalebox{0.94}{
\begin{tabular}{l|lllllll|l}
\hline
Model & Ar & Bn & Fi & Ja & Ko & Ru & Te & Average \\ \hline
XLM-R & 38.5 & 33.6 & 37.9 & \textbf{32.8} & 42.8 & 28.3 & 47.5 & 37.3 \\
XLM-R + xQG & \textbf{38.8} & \textbf{41.1}$^\star$ & \textbf{39.8} & \textbf{32.8} & \textbf{43.2} & \textbf{30.8} & \textbf{49.6} & \textbf{39.4}$^\star$ \\ \hline
mBERT & 49.2& 57.6& \textbf{58.6}& 42.7 & 57.5 & \textbf{41.4} & 55.9 & 51.8 \\
mBERT + xQG & \textbf{51.5} & \textbf{60.9}$^\star$ & 58.3 & \textbf{43.6}& \textbf{58.6} & \textbf{41.4} & \textbf{60.1}$^\star$ & \textbf{53.5}$^\star$ \\ \hline
mBERT (zero-shot) & 38.5 & 36.5 & 47.5 & 38.2 & 48.1 & 35.0 & 48.7 & 41.8 \\
mBERT (zero-shot) + xQG & \textbf{43.7}$^\star$ & \textbf{40.8}$^\star$ & \textbf{50.0}$^\star$ & \textbf{40.2} & \textbf{49.8} & \textbf{39.7}$^\star$ & \textbf{51.7} & \textbf{45.1}$^\star$ \\ \hline
\end{tabular}
}
\end{subtable}
\label{main}
\vspace{-14pt}
\end{table*}
\section{Experimental Settings}
We design our experiments to answer the research questions:
\begin{itemize}
\setlength{\itemindent}{16pt}
\item[\bf RQ1:] How does the number of generated queries affect xDR effectiveness?
\item[\bf RQ2:] How does the augmentation ratio $\alpha$ impact xDR effectiveness?
\item[\bf RQ3:] How do the queries generated for each language impact xDR effectiveness?
\end{itemize}
\textbf{\textit{Datasets and Evaluation.}} We train and evaluate our approach on XOR-TyDi~\cite{xorqa}, a cross-lingual open retrieval question answering benchmark dataset. The dataset contains approximately 15k annotated relevant passage-query pairs in the training set and 2k passage-answer pairs in the dev set. Queries in both the train and dev sets are in seven typologically diverse languages (Ar, Bn, Fi, Ja, Ko, Ru, and Te), while passages are in English. There are about 18M passages in the corpus. We use the training set to train both the xQG model and the xDRs. We evaluate the effectiveness of our approach and baselines using recall at $m$ kilo-tokens (R@$m$kt), which is the dataset's official evaluation metric. This metric computes the fraction of queries for which the minimal answer is contained in the top $m$ tokens of the retrieved passages. We consider $m = 2k, 5k$ (R@2kt and R@5kt), as per common practice for this dataset. Statistical significant differences are reported with respect to a two-tailed t-test with Bonferroni correction.
\textbf{\textit{Baselines.}} Following common practice on XOR-TyDi~\cite{xorqa}, we adapt DPR~\cite{karpukhin-etal-2020-dense} to the cross-lingual setting by initializing it with different multilingual pre-trained language models (PLMs). Specifically, in our experiments we use the multilingual variants of BERT~\cite{DBLP:conf/naacl/DevlinCLT19} (mBERT) and XLM-RoBERTa~\cite{conneau2020unsupervised} (XLM-R) for this purpose. In addition to the standard supervised baselines, we also explore how our xQG embedding augmentation approach can improve a zero-shot xDR model, where the xDR is initialized with mBERT but it is trained with English only passage-query pairs and is directly applied to the XOR-TyDi cross-lingual retrieval task.
\textbf{\textit{Implementation details.}} We initialize our xQG model with the multilingual T5-base checkpoint provided by Google\footnote{\url{https://huggingface.co/google/mt5-base}} and available in the Huggingface library~\cite{wolf-etal-2020-transformers}. We fine-tune this checkpoint with the passage-query pairs in the XOR-TyDi training set. We train the xQG model for 1600 steps using a batch size of 128 and a learning rate of 1e-4. After training, we generate 5 queries per passage for each target language, resulting in about 7 * 5 * 18M = 630M generated queries in total. We use four A100 GPUs to generate all queries; this process took about 70 hours to complete. We release our generated queries on the Huggingface hub~\footnote{\url{https://huggingface.co/datasets/ielab/xor-tydi-xqg-augmented}} to allow the community to reuse this resource~\cite{scells2022reduce}. For training the xDRs, we use the Tevatron dense retriever training toolkit~\cite{Gao2022TevatronAE}, which uses with BM25 hard negative passages. We train the xDRs with a batch size of 128, initializing them with mBERT base\footnote{\url{https://huggingface.co/bert-base-multilingual-cased}} or XLM-R base\footnote{\url{https://huggingface.co/xlm-roberta-base}} checkpoints and training them on the XOR-TyDi training set for 40 epochs. For each training sample, we set the number of hard negative passages in the contrastive loss to 7 and applied in-batch negatives training. We use a learning rate of 1e-5 for mBERT-based xDRs and of 2e-5 for XLM-R-based xDRs.
For the zero-shot xDR, we use the same training configurations as for the mBERT-based xDR trained on XOR-TyDi but using the Natural Questions (NQ) training data~\cite{47761} which contains English-only query-passage training samples.
\section{Results}
\subsection{Main results}
Table~\ref{main} presents the effectiveness of xDR models initialized with the XLM-R and mBERT backbone PLMs and trained on the XOR-TyDi dataset. Zero-shot denotes the models trained only on the English subset of the NQ dataset. In these experiments, we use all the queries generated by our xQG and set the augmentation ratio to $\alpha=0.01$ for augmenting the passage embeddings of the xDRs. %We evaluate on R@2kt and R@5kt metrics on seven languages of the dev set.
\begin{figure}[t]
\centering
%\vspace{-10pt}
\includegraphics[width=0.85\columnwidth]{images/num_queries_plot_bonferroni.png}
\vspace{-10pt}
\caption{Impact of the amount of generated queries for target language. Scores are averaged across all languages. Statistical significant improvements over no augmentation (number of queries $n=0$) are labelled with stars ($p<0.05$). \vspace{-16pt}}
\label{fig:num_query}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=0.83\columnwidth]{images/recall_plot_bonferroni.png}
\vspace{-10pt}
\caption{Impact of the augmentation ratio $\alpha$. Scores are averaged across all languages. Statistical significant improvements over no augmentation ($\alpha=0$) are labelled with stars ($p<0.05$). \vspace{-18pt}}
\label{fig:ratio}
\end{figure}
For the R@2kt metric, the xDR initialized with mBERT outperforms the xDR initialized with XLM-R, achieving an average R@2kt score of 44.1, while XLM-R achieves an average score of 27.5. %XLM-R improves its effectiveness with
Our xQG passage embedding augmentation approach improves the XLM-R xDR, achieving an average score of 29.8, which is a statistically significant improvement compared to its baseline effectiveness ($p < 0.05$). Similarly, mBERT's effectiveness improves with xQG, achieving an average score of 46.2, which is also a statistically significant improvement compared to its corresponding baseline ($p < 0.05$). The zero-shot mBERT model achieves an average R@2kt of 33.0; this also improves when combined with xQG, achieving an average score of 36.0. This improvement is statistically significant ($p < 0.05$).
Similar trends are found for R@5kt. %mBERT also outperforms XLM-R, achieving an average R@5kt score of 51.8, while XLM-R achieves an average score of 37.3.
Overall, we find that our xQG can significantly improve all investigated xDR models.
In terms of per language effectiveness, xQG improves almost all models across all languages with the exceptions of mBERT's R@2kt for Japanese (Ja) and mBERT's R@5kt for Finnish (Fi).
In summary, mBERT performs better than XLM-R for both R@2kt and R@5kt. The use of xQG embedding augmentation statistically significantly improves the effectiveness of both backbones.
\subsection{RQ1: Impact of number of generated queries}
Figure~\ref{fig:num_query} reports the impact of using different amounts of generated queries to augment passage embeddings when using mBERT xDR. The results suggest that using more generated queries is beneficial for both R@2tk and R@5tk. The improvements become statistically significant when 4 or more generated queries are used for each target language. While the curves do not plateau, indicating that using even more generated queries could further improve the effectiveness, our experiments were limited to up to 5 generated queries per target language due to computational constraints.
%In answering to RQ1, therefore, we find that using more generated queries to augment passage embeddings for xDR is helpful, and we suggest that using even more generated queries than those explored here could potentially lead to further improvements in effectiveness.
\subsection{RQ2: Impact of augmentation ratio}
We report the impact of the augmentation ratio $\alpha$ on the effectiveness of xDR in Figure~\ref{fig:ratio}. Higher values of $\alpha$ correspond to assigning more weight to the generated query embeddings during embedding aggregation, and $\alpha=0$ corresponds to no augmentation. As shown in the figure, even a small value of $\alpha$ (0.01) leads to a significant improvement in both R@2kt and R@5kt. The best effectiveness is achieved when $\alpha=0.02$. However, using higher values of $\alpha$ does not result in further improvements -- rather, it can even hurt the effectiveness when $\alpha>0.05$. Based on these results, we conclude that the augmentation ratio in our embedding aggregation function has a significant impact on the effectiveness of xDR, and using a small value of $\alpha$ can be beneficial for improving effectiveness.
\subsection{RQ3: Impact of each languages}
In the previous experiments we used all the queries generated for a passage to augment the original passage embedding, irrespective of language of the generated query. Next, we investigate the impact on effectiveness of using generated queries from each of the languages separately. %Specifically, we aim to understand how using only the five generated queries from each language affects the performance of the associated passage embedding in retrieving relevant information for target language queries in the dev set.
We analyze this in Figure~\ref{fig:per_lang}, where we plot R@2kt for each target language (rows) against different values of $\alpha$ (x-axis). %The results of our experiment are presented in Figure~\ref{fig:per_lang}.
The plots in the diagonal show that xDR effectiveness improves when passage embeddings are augmented with generated queries for the same language.
%using the same language of generated queries to augment the passage embedding of the corresponding language.
Notably, we observe that the best value of $\alpha$ varies across different languages, and the improvements in effectiveness are not always statistically significant. We also observe that, for some languages, queries generated for other languages can improve the effectiveness of target queries in a different language. For instance, using queries generated for Japanese (Ja) can improve the effectiveness of target queries in Korean (ko), while using Russian (Ru) generated queries can help target queries in Telugu (Te).
%The last row in Figure~\ref{fig:per_lang} reports the average effectiveness: this improves (though not always statistically significantly) by using only the generated queries from a single language.
%Furthermore, from the bottom row of the plots, we observe that the average R@2kt score can be improved by using only the generated queries from a single language.
These results suggest that the embeddings of the queries generated for any single language potentially can also provide useful information for target queries in other languages.
\begin{figure}[t]
\centering
\includegraphics[width=1\columnwidth]{images/recall_per_lang_bonferroni.png}
\vspace{-16pt}
\caption{R@2kt for target languages (rows) augmented by source generated queries (columns). We plot $\alpha$ from 0 to 0.1 with step size of 0.01 (x axis). Statistically significant better results with respect to no augmentation ($\alpha=0$) are labelled with stars ($p<0.05$). \vspace{-14pt}}
\label{fig:per_lang}
\end{figure}
\section{Conclusion and future work}
In this paper we propose a passage embedding augmentation approach to enhance the effectiveness of cross-lingual DRs. Our method can be applied to any cross-lingual DR and it requires no further DR training nor changes to the retrieval pipeline. We empirically showed the method is effective for the cross-lingual DPR method across different backbones. However, a limitation of our empirical investigation is that we did not evaluate the method across other dense retriever architectures. We leave the extension of this investigation to future work. We also note that our approach relies on a xQG model that can generate high quality queries. However, a recent work has shown that T5-based query generation is prone to hallucination and that along with highly effective queries, the generator also produces poor queries that negatively impact retrieval effectiveness~\cite{gospodinov2023doc2query}. They then propose the use of a cross-encoder ranker to filter out some ineffective generated queries; this practice can further improve effectiveness. We leave the adaptation of this approach in our xQG setting to future work.
%\vspace{6pt}
%\textbf{Acknowledgments.} \todo{The authors need 2 lines of text to acknowledge funding sources.}
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{48}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[\protect\citeauthoryear{Asai, Kasai, Clark, Lee, Choi, and
Hajishirzi}{Asai et~al\mbox{.}}{2021a}]%
{xorqa}
\bibfield{author}{\bibinfo{person}{Akari Asai}, \bibinfo{person}{Jungo Kasai},
\bibinfo{person}{Jonathan~H. Clark}, \bibinfo{person}{Kenton Lee},
\bibinfo{person}{Eunsol Choi}, {and} \bibinfo{person}{Hannaneh Hajishirzi}.}
\bibinfo{year}{2021}\natexlab{a}.
\newblock \showarticletitle{{XOR} {QA}: Cross-lingual Open-Retrieval Question
Answering}. In \bibinfo{booktitle}{\emph{NAACL-HLT}}.
\newblock
\bibitem[\protect\citeauthoryear{Asai, Yu, Kasai, and Hajishirzi}{Asai
et~al\mbox{.}}{2021b}]%
{asai2021one}
\bibfield{author}{\bibinfo{person}{Akari Asai}, \bibinfo{person}{Xinyan Yu},
\bibinfo{person}{Jungo Kasai}, {and} \bibinfo{person}{Hanna Hajishirzi}.}
\bibinfo{year}{2021}\natexlab{b}.
\newblock \showarticletitle{One question answering model for many languages
with cross-lingual dense passage retrieval}.
\newblock \bibinfo{journal}{\emph{Advances in Neural Information Processing
Systems}} \bibinfo{volume}{34} (\bibinfo{year}{2021}),
\bibinfo{pages}{7547--7560}.
\newblock
\bibitem[\protect\citeauthoryear{Chandradevan, Yang, Yarmohammadi, and
Agichtein}{Chandradevan et~al\mbox{.}}{2022}]%
{chandradevan2022learning}
\bibfield{author}{\bibinfo{person}{Ramraj Chandradevan},
\bibinfo{person}{Eugene Yang}, \bibinfo{person}{Mahsa Yarmohammadi}, {and}
\bibinfo{person}{Eugene Agichtein}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Learning to Enrich Query Representation with
Pseudo-Relevance Feedback for Cross-lingual Retrieval}. In
\bibinfo{booktitle}{\emph{Proceedings of the 45th International ACM SIGIR
Conference on Research and Development in Information Retrieval}}.
\bibinfo{pages}{1790--1795}.
\newblock
\bibitem[\protect\citeauthoryear{Chi, Dong, Wei, Yang, Singhal, Wang, Song,
Mao, Huang, and Zhou}{Chi et~al\mbox{.}}{2021}]%
{chi2021infoxlm}
\bibfield{author}{\bibinfo{person}{Zewen Chi}, \bibinfo{person}{Li Dong},
\bibinfo{person}{Furu Wei}, \bibinfo{person}{Nan Yang},
\bibinfo{person}{Saksham Singhal}, \bibinfo{person}{Wenhui Wang},
\bibinfo{person}{Xia Song}, \bibinfo{person}{Xian-Ling Mao},
\bibinfo{person}{He-Yan Huang}, {and} \bibinfo{person}{Ming Zhou}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{InfoXLM: An Information-Theoretic Framework for
Cross-Lingual Language Model Pre-Training}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies}}. \bibinfo{pages}{3576--3588}.
\newblock
\bibitem[\protect\citeauthoryear{Conneau, Khandelwal, Goyal, Chaudhary, Wenzek,
Guzm{\'a}n, Grave, Ott, Zettlemoyer, and Stoyanov}{Conneau
et~al\mbox{.}}{2020}]%
{conneau2020unsupervised}
\bibfield{author}{\bibinfo{person}{Alexis Conneau}, \bibinfo{person}{Kartikay
Khandelwal}, \bibinfo{person}{Naman Goyal}, \bibinfo{person}{Vishrav
Chaudhary}, \bibinfo{person}{Guillaume Wenzek}, \bibinfo{person}{Francisco
Guzm{\'a}n}, \bibinfo{person}{{\'E}douard Grave}, \bibinfo{person}{Myle Ott},
\bibinfo{person}{Luke Zettlemoyer}, {and} \bibinfo{person}{Veselin
Stoyanov}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Unsupervised Cross-lingual Representation Learning
at Scale}. In \bibinfo{booktitle}{\emph{Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics}}.
\bibinfo{pages}{8440--8451}.
\newblock
\bibitem[\protect\citeauthoryear{Devlin, Chang, Lee, and Toutanova}{Devlin
et~al\mbox{.}}{2019}]%
{DBLP:conf/naacl/DevlinCLT19}
\bibfield{author}{\bibinfo{person}{Jacob Devlin}, \bibinfo{person}{Ming{-}Wei
Chang}, \bibinfo{person}{Kenton Lee}, {and} \bibinfo{person}{Kristina
Toutanova}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{BERT:} Pre-training of Deep Bidirectional
Transformers for Language Understanding}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies, {NAACL-HLT} 2019, Minneapolis, MN, USA, June 2-7,
2019, Volume 1 (Long and Short Papers)}},
\bibfield{editor}{\bibinfo{person}{Jill Burstein}, \bibinfo{person}{Christy
Doran}, {and} \bibinfo{person}{Thamar Solorio}} (Eds.).
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{pages}{4171--4186}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.18653/v1/n19-1423}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Fan, Lewis, and Dauphin}{Fan
et~al\mbox{.}}{2018}]%
{fan2018hierarchical}
\bibfield{author}{\bibinfo{person}{Angela Fan}, \bibinfo{person}{Mike Lewis},
{and} \bibinfo{person}{Yann Dauphin}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Hierarchical Neural Story Generation}. In
\bibinfo{booktitle}{\emph{Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers)}}.
\bibinfo{pages}{889--898}.
\newblock
\bibitem[\protect\citeauthoryear{Feng, Yang, Cer, Arivazhagan, and Wang}{Feng
et~al\mbox{.}}{2022}]%
{feng2022language}
\bibfield{author}{\bibinfo{person}{Fangxiaoyu Feng}, \bibinfo{person}{Yinfei
Yang}, \bibinfo{person}{Daniel Cer}, \bibinfo{person}{Naveen Arivazhagan},
{and} \bibinfo{person}{Wei Wang}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Language-agnostic BERT Sentence Embedding}. In
\bibinfo{booktitle}{\emph{Proceedings of the 60th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers)}}.
\bibinfo{pages}{878--891}.
\newblock
\bibitem[\protect\citeauthoryear{Gao, Ma, Lin, and Callan}{Gao
et~al\mbox{.}}{2022a}]%
{gao2022precise}
\bibfield{author}{\bibinfo{person}{Luyu Gao}, \bibinfo{person}{Xueguang Ma},
\bibinfo{person}{Jimmy Lin}, {and} \bibinfo{person}{Jamie Callan}.}
\bibinfo{year}{2022}\natexlab{a}.
\newblock \showarticletitle{Precise Zero-Shot Dense Retrieval without Relevance
Labels}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2212.10496}}
(\bibinfo{year}{2022}).
\newblock
\bibitem[\protect\citeauthoryear{Gao, Ma, Lin, and Callan}{Gao
et~al\mbox{.}}{2022b}]%
{Gao2022TevatronAE}
\bibfield{author}{\bibinfo{person}{Luyu Gao}, \bibinfo{person}{Xueguang Ma},
\bibinfo{person}{Jimmy~J. Lin}, {and} \bibinfo{person}{Jamie Callan}.}
\bibinfo{year}{2022}\natexlab{b}.
\newblock \showarticletitle{Tevatron: An Efficient and Flexible Toolkit for
Dense Retrieval}.
\newblock \bibinfo{journal}{\emph{ArXiv}} \bibinfo{volume}{abs/2203.05765}
(\bibinfo{year}{2022}).
\newblock
\bibitem[\protect\citeauthoryear{Gospodinov, MacAvaney, and
Macdonald}{Gospodinov et~al\mbox{.}}{2023}]%
{gospodinov2023doc2query}
\bibfield{author}{\bibinfo{person}{Mitko Gospodinov}, \bibinfo{person}{Sean
MacAvaney}, {and} \bibinfo{person}{Craig Macdonald}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Doc2Query: When Less is More}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2301.03266}}
(\bibinfo{year}{2023}).
\newblock
\bibitem[\protect\citeauthoryear{Hofst{\"a}tter, Althammer, Schr{\"o}der,
Sertkan, and Hanbury}{Hofst{\"a}tter et~al\mbox{.}}{2020}]%
{hofstatter2020improving}
\bibfield{author}{\bibinfo{person}{Sebastian Hofst{\"a}tter},
\bibinfo{person}{Sophia Althammer}, \bibinfo{person}{Michael Schr{\"o}der},
\bibinfo{person}{Mete Sertkan}, {and} \bibinfo{person}{Allan Hanbury}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Improving efficient neural ranking models with
cross-architecture knowledge distillation}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2010.02666}}
(\bibinfo{year}{2020}).
\newblock
\bibitem[\protect\citeauthoryear{Hofst{\"a}tter, Lin, Yang, Lin, and
Hanbury}{Hofst{\"a}tter et~al\mbox{.}}{2021}]%
{hofstatter2021efficiently}
\bibfield{author}{\bibinfo{person}{Sebastian Hofst{\"a}tter},
\bibinfo{person}{Sheng-Chieh Lin}, \bibinfo{person}{Jheng-Hong Yang},
\bibinfo{person}{Jimmy Lin}, {and} \bibinfo{person}{Allan Hanbury}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Efficiently teaching an effective dense retriever
with balanced topic aware sampling}. In \bibinfo{booktitle}{\emph{Proceedings
of the 44th International ACM SIGIR Conference on Research and Development in
Information Retrieval}}. \bibinfo{pages}{113--122}.
\newblock
\bibitem[\protect\citeauthoryear{Karpukhin, Oguz, Min, Lewis, Wu, Edunov, Chen,
and Yih}{Karpukhin et~al\mbox{.}}{2020}]%
{karpukhin-etal-2020-dense}
\bibfield{author}{\bibinfo{person}{Vladimir Karpukhin}, \bibinfo{person}{Barlas
Oguz}, \bibinfo{person}{Sewon Min}, \bibinfo{person}{Patrick Lewis},
\bibinfo{person}{Ledell Wu}, \bibinfo{person}{Sergey Edunov},
\bibinfo{person}{Danqi Chen}, {and} \bibinfo{person}{Wen-tau Yih}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Dense Passage Retrieval for Open-Domain Question
Answering}. In \bibinfo{booktitle}{\emph{Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Processing (EMNLP)}}.
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{address}{Online}, \bibinfo{pages}{6769--6781}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.18653/v1/2020.emnlp-main.550}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Kwiatkowski, Palomaki, Redfield, Collins,
Parikh, Alberti, Epstein, Polosukhin, Kelcey, Devlin, Lee, Toutanova, Jones,
Chang, Dai, Uszkoreit, Le, and Petrov}{Kwiatkowski et~al\mbox{.}}{2019}]%
{47761}
\bibfield{author}{\bibinfo{person}{Tom Kwiatkowski},
\bibinfo{person}{Jennimaria Palomaki}, \bibinfo{person}{Olivia Redfield},
\bibinfo{person}{Michael Collins}, \bibinfo{person}{Ankur Parikh},
\bibinfo{person}{Chris Alberti}, \bibinfo{person}{Danielle Epstein},
\bibinfo{person}{Illia Polosukhin}, \bibinfo{person}{Matthew Kelcey},
\bibinfo{person}{Jacob Devlin}, \bibinfo{person}{Kenton Lee},
\bibinfo{person}{Kristina~N. Toutanova}, \bibinfo{person}{Llion Jones},
\bibinfo{person}{Ming-Wei Chang}, \bibinfo{person}{Andrew Dai},
\bibinfo{person}{Jakob Uszkoreit}, \bibinfo{person}{Quoc Le}, {and}
\bibinfo{person}{Slav Petrov}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Natural Questions: a Benchmark for Question
Answering Research}.
\newblock \bibinfo{journal}{\emph{Transactions of the Association of
Computational Linguistics}} (\bibinfo{year}{2019}).
\newblock
\bibitem[\protect\citeauthoryear{Li, Mourad, Zhuang, Koopman, and Zuccon}{Li
et~al\mbox{.}}{2023a}]%
{li2023pseudo}
\bibfield{author}{\bibinfo{person}{Hang Li}, \bibinfo{person}{Ahmed Mourad},
\bibinfo{person}{Shengyao Zhuang}, \bibinfo{person}{Bevan Koopman}, {and}
\bibinfo{person}{Guido Zuccon}.} \bibinfo{year}{2023}\natexlab{a}.
\newblock \showarticletitle{Pseudo relevance feedback with deep language models
and dense retrievers: Successes and pitfalls}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Information Systems}}
(\bibinfo{year}{2023}).
\newblock
\bibitem[\protect\citeauthoryear{Li, Zhuang, Ma, Lin, and Zuccon}{Li
et~al\mbox{.}}{2023b}]%
{li2022pyserini}
\bibfield{author}{\bibinfo{person}{Hang Li}, \bibinfo{person}{Shengyao Zhuang},
\bibinfo{person}{Xueguang Ma}, \bibinfo{person}{Jimmy Lin}, {and}
\bibinfo{person}{Guido Zuccon}.} \bibinfo{year}{2023}\natexlab{b}.
\newblock \showarticletitle{Pseudo-Relevance Feedback with Dense Retrievers in
Pyserini}. In \bibinfo{booktitle}{\emph{Proceedings of the 26th Australasian
Document Computing Symposium}} \emph{(\bibinfo{series}{ADCS '22})}.
\newblock
\bibitem[\protect\citeauthoryear{Li, Zhuang, Mourad, Ma, Lin, and Zuccon}{Li
et~al\mbox{.}}{2022b}]%
{li2022improving}
\bibfield{author}{\bibinfo{person}{Hang Li}, \bibinfo{person}{Shengyao Zhuang},
\bibinfo{person}{Ahmed Mourad}, \bibinfo{person}{Xueguang Ma},
\bibinfo{person}{Jimmy Lin}, {and} \bibinfo{person}{Guido Zuccon}.}
\bibinfo{year}{2022}\natexlab{b}.
\newblock \showarticletitle{Improving query representations for dense retrieval
with pseudo relevance feedback: A reproducibility study}. In
\bibinfo{booktitle}{\emph{Advances in Information Retrieval: 44th European
Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10--14, 2022,
Proceedings, Part I}}. Springer, \bibinfo{pages}{599--612}.
\newblock
\bibitem[\protect\citeauthoryear{Li, Franz, Sultan, Iyer, Lee, and Sil}{Li
et~al\mbox{.}}{2022a}]%
{li2022learning}
\bibfield{author}{\bibinfo{person}{Yulong Li}, \bibinfo{person}{Martin Franz},
\bibinfo{person}{Md~Arafat Sultan}, \bibinfo{person}{Bhavani Iyer},
\bibinfo{person}{Young-Suk Lee}, {and} \bibinfo{person}{Avirup Sil}.}
\bibinfo{year}{2022}\natexlab{a}.
\newblock \showarticletitle{Learning Cross-Lingual IR from an English
Retriever}. In \bibinfo{booktitle}{\emph{Proceedings of the 2022 Conference
of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies}}. \bibinfo{pages}{4428--4436}.
\newblock
\bibitem[\protect\citeauthoryear{Lin, Yang, and Lin}{Lin et~al\mbox{.}}{2020}]%
{lin2020distilling}
\bibfield{author}{\bibinfo{person}{Sheng-Chieh Lin},
\bibinfo{person}{Jheng-Hong Yang}, {and} \bibinfo{person}{Jimmy Lin}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Distilling dense representations for ranking using
tightly-coupled teachers}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2010.11386}}
(\bibinfo{year}{2020}).
\newblock
\bibitem[\protect\citeauthoryear{Lin, Yang, and Lin}{Lin et~al\mbox{.}}{2021}]%
{lin2021batch}
\bibfield{author}{\bibinfo{person}{Sheng-Chieh Lin},
\bibinfo{person}{Jheng-Hong Yang}, {and} \bibinfo{person}{Jimmy Lin}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{In-batch negatives for knowledge distillation with
tightly-coupled teachers for dense retrieval}. In
\bibinfo{booktitle}{\emph{Proceedings of the 6th Workshop on Representation
Learning for NLP (RepL4NLP-2021)}}. \bibinfo{pages}{163--173}.
\newblock
\bibitem[\protect\citeauthoryear{Longpre, Lu, and Daiber}{Longpre
et~al\mbox{.}}{2021}]%
{longpre2021mkqa}
\bibfield{author}{\bibinfo{person}{Shayne Longpre}, \bibinfo{person}{Yi Lu},
{and} \bibinfo{person}{Joachim Daiber}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{MKQA: A linguistically diverse benchmark for
multilingual open domain question answering}.
\newblock \bibinfo{journal}{\emph{Transactions of the Association for
Computational Linguistics}} \bibinfo{volume}{9} (\bibinfo{year}{2021}),
\bibinfo{pages}{1389--1406}.
\newblock
\bibitem[\protect\citeauthoryear{Ma, Korotkov, Yang, Hall, and McDonald}{Ma
et~al\mbox{.}}{2021}]%
{ma2021zero}
\bibfield{author}{\bibinfo{person}{Ji Ma}, \bibinfo{person}{Ivan Korotkov},
\bibinfo{person}{Yinfei Yang}, \bibinfo{person}{Keith Hall}, {and}
\bibinfo{person}{Ryan McDonald}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Zero-shot Neural Passage Retrieval via
Domain-targeted Synthetic Question Generation}. In
\bibinfo{booktitle}{\emph{Proceedings of the 16th Conference of the European
Chapter of the Association for Computational Linguistics: Main Volume}}.
\bibinfo{pages}{1075--1088}.
\newblock
\bibitem[\protect\citeauthoryear{Nair, Yang, Lawrie, Duh, McNamee, Murray,
Mayfield, and Oard}{Nair et~al\mbox{.}}{2022}]%
{nair2022transfer}
\bibfield{author}{\bibinfo{person}{Suraj Nair}, \bibinfo{person}{Eugene Yang},
\bibinfo{person}{Dawn Lawrie}, \bibinfo{person}{Kevin Duh},
\bibinfo{person}{Paul McNamee}, \bibinfo{person}{Kenton Murray},
\bibinfo{person}{James Mayfield}, {and} \bibinfo{person}{Douglas~W Oard}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Transfer learning approaches for building
cross-language dense retrieval models}. In \bibinfo{booktitle}{\emph{Advances
in Information Retrieval: 44th European Conference on IR Research, ECIR 2022,
Stavanger, Norway, April 10--14, 2022, Proceedings, Part I}}. Springer,
\bibinfo{pages}{382--396}.
\newblock
\bibitem[\protect\citeauthoryear{Nogueira and Lin}{Nogueira and Lin}{2019}]%
{nogueira2019doc2query}
\bibfield{author}{\bibinfo{person}{Rodrigo Nogueira} {and}
\bibinfo{person}{Jimmy Lin}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \bibinfo{title}{From doc2query to {docTTTTTquery}}.
\newblock
\newblock
\bibitem[\protect\citeauthoryear{Qu, Ding, Liu, Liu, Ren, Zhao, Dong, Wu, and
Wang}{Qu et~al\mbox{.}}{2021}]%
{qu2021rocketqa}
\bibfield{author}{\bibinfo{person}{Yingqi Qu}, \bibinfo{person}{Yuchen Ding},
\bibinfo{person}{Jing Liu}, \bibinfo{person}{Kai Liu},
\bibinfo{person}{Ruiyang Ren}, \bibinfo{person}{Wayne~Xin Zhao},
\bibinfo{person}{Daxiang Dong}, \bibinfo{person}{Hua Wu}, {and}
\bibinfo{person}{Haifeng Wang}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{RocketQA: An optimized training approach to dense
passage retrieval for open-domain question answering}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies}}. \bibinfo{pages}{5835--5847}.
\newblock
\bibitem[\protect\citeauthoryear{Ren, Shou, Wu, Gong, and Jiang}{Ren
et~al\mbox{.}}{2022}]%
{ren2022empowering}
\bibfield{author}{\bibinfo{person}{Houxing Ren}, \bibinfo{person}{Linjun Shou},
\bibinfo{person}{Ning Wu}, \bibinfo{person}{Ming Gong}, {and}
\bibinfo{person}{Daxin Jiang}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Empowering Dual-Encoder with Query Generator for
Cross-Lingual Dense Retrieval}. In \bibinfo{booktitle}{\emph{Proceedings of
the 2022 Conference on Empirical Methods in Natural Language Processing}}.
\bibinfo{pages}{3107--3121}.
\newblock
\bibitem[\protect\citeauthoryear{Ren, Lv, Qu, Liu, Zhao, She, Wu, Wang, and
Wen}{Ren et~al\mbox{.}}{2021a}]%
{ren2021pair}
\bibfield{author}{\bibinfo{person}{Ruiyang Ren}, \bibinfo{person}{Shangwen Lv},
\bibinfo{person}{Yingqi Qu}, \bibinfo{person}{Jing Liu},
\bibinfo{person}{Wayne~Xin Zhao}, \bibinfo{person}{Qiaoqiao She},
\bibinfo{person}{Hua Wu}, \bibinfo{person}{Haifeng Wang}, {and}
\bibinfo{person}{Ji-Rong Wen}.} \bibinfo{year}{2021}\natexlab{a}.
\newblock \showarticletitle{PAIR: Leveraging Passage-Centric Similarity
Relation for Improving Dense Passage Retrieval}. In
\bibinfo{booktitle}{\emph{Findings of the Association for Computational
Linguistics: ACL-IJCNLP 2021}}. \bibinfo{pages}{2173--2183}.
\newblock
\bibitem[\protect\citeauthoryear{Ren, Qu, Liu, Zhao, She, Wu, Wang, and
Wen}{Ren et~al\mbox{.}}{2021b}]%
{ren2021rocketqav2}
\bibfield{author}{\bibinfo{person}{Ruiyang Ren}, \bibinfo{person}{Yingqi Qu},
\bibinfo{person}{Jing Liu}, \bibinfo{person}{Wayne~Xin Zhao},
\bibinfo{person}{Qiaoqiao She}, \bibinfo{person}{Hua Wu},
\bibinfo{person}{Haifeng Wang}, {and} \bibinfo{person}{Ji-Rong Wen}.}
\bibinfo{year}{2021}\natexlab{b}.
\newblock \showarticletitle{RocketQAv2: A Joint Training Method for Dense
Passage Retrieval and Passage Re-ranking}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2021 Conference on Empirical
Methods in Natural Language Processing}}. \bibinfo{pages}{2825--2835}.
\newblock
\bibitem[\protect\citeauthoryear{Scells, Zhuang, and Zuccon}{Scells
et~al\mbox{.}}{2022}]%
{scells2022reduce}
\bibfield{author}{\bibinfo{person}{Harrisen Scells}, \bibinfo{person}{Shengyao
Zhuang}, {and} \bibinfo{person}{Guido Zuccon}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Reduce, Reuse, Recycle: Green Information Retrieval
Research}. In \bibinfo{booktitle}{\emph{Proceedings of the 45th International
ACM SIGIR Conference on Research and Development in Information Retrieval}}.
\bibinfo{pages}{2825--2837}.
\newblock
\bibitem[\protect\citeauthoryear{Sorokin, Abulkhanov, Piontkovskaya, and
Malykh}{Sorokin et~al\mbox{.}}{2022}]%
{sorokin2022ask}
\bibfield{author}{\bibinfo{person}{Nikita Sorokin}, \bibinfo{person}{Dmitry
Abulkhanov}, \bibinfo{person}{Irina Piontkovskaya}, {and}
\bibinfo{person}{Valentin Malykh}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Ask me anything in your native language}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2022 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies}}. \bibinfo{pages}{395--406}.
\newblock
\bibitem[\protect\citeauthoryear{Thakur, Reimers, R{\"u}ckl{\'e}, Srivastava,
and Gurevych}{Thakur et~al\mbox{.}}{2021}]%
{thakur2021beir}
\bibfield{author}{\bibinfo{person}{Nandan Thakur}, \bibinfo{person}{Nils
Reimers}, \bibinfo{person}{Andreas R{\"u}ckl{\'e}}, \bibinfo{person}{Abhishek
Srivastava}, {and} \bibinfo{person}{Iryna Gurevych}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{{BEIR}: A Heterogeneous Benchmark for Zero-shot
Evaluation of Information Retrieval Models}. In
\bibinfo{booktitle}{\emph{Thirty-fifth Conference on Neural Information
Processing Systems Datasets and Benchmarks Track (Round 2)}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=wCu6T5xFjeJ}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Tonellotto}{Tonellotto}{2022}]%
{tonellotto2022lecture}
\bibfield{author}{\bibinfo{person}{Nicola Tonellotto}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Lecture Notes on Neural Information Retrieval}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2207.13443}}
(\bibinfo{year}{2022}).
\newblock
\bibitem[\protect\citeauthoryear{Wang, Thakur, Reimers, and Gurevych}{Wang
et~al\mbox{.}}{2022}]%
{wang2022gpl}
\bibfield{author}{\bibinfo{person}{Kexin Wang}, \bibinfo{person}{Nandan
Thakur}, \bibinfo{person}{Nils Reimers}, {and} \bibinfo{person}{Iryna
Gurevych}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{GPL: Generative Pseudo Labeling for Unsupervised
Domain Adaptation of Dense Retrieval}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2022 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies}}. \bibinfo{pages}{2345--2360}.
\newblock
\bibitem[\protect\citeauthoryear{Wang, Macdonald, Tonellotto, and Ounis}{Wang
et~al\mbox{.}}{2021}]%
{wang2021pseudo}
\bibfield{author}{\bibinfo{person}{Xiao Wang}, \bibinfo{person}{Craig
Macdonald}, \bibinfo{person}{Nicola Tonellotto}, {and} \bibinfo{person}{Iadh
Ounis}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Pseudo-relevance feedback for multiple
representation dense retrieval}. In \bibinfo{booktitle}{\emph{Proceedings of
the 2021 ACM SIGIR International Conference on Theory of Information
Retrieval}}. \bibinfo{pages}{297--306}.
\newblock
\bibitem[\protect\citeauthoryear{Wang, Macdonald, Tonellotto, and Ounis}{Wang
et~al\mbox{.}}{2023}]%
{wang2023colbert}
\bibfield{author}{\bibinfo{person}{Xiao Wang}, \bibinfo{person}{Craig
Macdonald}, \bibinfo{person}{Nicola Tonellotto}, {and} \bibinfo{person}{Iadh
Ounis}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{ColBERT-PRF: Semantic pseudo-relevance feedback for
dense passage and document retrieval}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on the Web}}
\bibinfo{volume}{17}, \bibinfo{number}{1} (\bibinfo{year}{2023}),
\bibinfo{pages}{1--39}.
\newblock
\bibitem[\protect\citeauthoryear{Wang, Hou, Wang, Miao, Wu, Chen, Xia, Chi,
Zhao, Liu, et~al\mbox{.}}{Wang et~al\mbox{.}}{[n.d.]}]%
{wangneural}
\bibfield{author}{\bibinfo{person}{Yujing Wang}, \bibinfo{person}{Yingyan Hou},
\bibinfo{person}{Haonan Wang}, \bibinfo{person}{Ziming Miao},
\bibinfo{person}{Shibin Wu}, \bibinfo{person}{Qi Chen},
\bibinfo{person}{Yuqing Xia}, \bibinfo{person}{Chengmin Chi},
\bibinfo{person}{Guoshuai Zhao}, \bibinfo{person}{Zheng Liu},
{et~al\mbox{.}}} \bibinfo{year}{[n.d.]}\natexlab{}.
\newblock \showarticletitle{A Neural Corpus Indexer for Document Retrieval}. In
\bibinfo{booktitle}{\emph{Advances in Neural Information Processing
Systems}}.
\newblock
\bibitem[\protect\citeauthoryear{Wolf, Debut, Sanh, Chaumond, Delangue, Moi,
Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite,
Plu, Xu, Scao, Gugger, Drame, Lhoest, and Rush}{Wolf et~al\mbox{.}}{2020}]%
{wolf-etal-2020-transformers}
\bibfield{author}{\bibinfo{person}{Thomas Wolf}, \bibinfo{person}{Lysandre
Debut}, \bibinfo{person}{Victor Sanh}, \bibinfo{person}{Julien Chaumond},
\bibinfo{person}{Clement Delangue}, \bibinfo{person}{Anthony Moi},
\bibinfo{person}{Pierric Cistac}, \bibinfo{person}{Tim Rault},
\bibinfo{person}{R{\'e}mi Louf}, \bibinfo{person}{Morgan Funtowicz},
\bibinfo{person}{Joe Davison}, \bibinfo{person}{Sam Shleifer},
\bibinfo{person}{Patrick von Platen}, \bibinfo{person}{Clara Ma},
\bibinfo{person}{Yacine Jernite}, \bibinfo{person}{Julien Plu},
\bibinfo{person}{Canwen Xu}, \bibinfo{person}{Teven~Le Scao},
\bibinfo{person}{Sylvain Gugger}, \bibinfo{person}{Mariama Drame},
\bibinfo{person}{Quentin Lhoest}, {and} \bibinfo{person}{Alexander~M. Rush}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Transformers: State-of-the-Art Natural Language
Processing}. In \bibinfo{booktitle}{\emph{Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Processing: System Demonstrations}}.
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{pages}{38--45}.
\newblock
\bibitem[\protect\citeauthoryear{Wu, Liang, Ren, Shou, Duan, Gong, and
Jiang}{Wu et~al\mbox{.}}{2022}]%
{wu2022unsupervised}
\bibfield{author}{\bibinfo{person}{Ning Wu}, \bibinfo{person}{Yaobo Liang},
\bibinfo{person}{Houxing Ren}, \bibinfo{person}{Linjun Shou},
\bibinfo{person}{Nan Duan}, \bibinfo{person}{Ming Gong}, {and}
\bibinfo{person}{Daxin Jiang}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Unsupervised context aware sentence representation
pretraining for multi-lingual dense retrieval}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2206.03281}}
(\bibinfo{year}{2022}).
\newblock
\bibitem[\protect\citeauthoryear{Xiong, Xiong, Li, Tang, Liu, Bennett, Ahmed,
and Overwijk}{Xiong et~al\mbox{.}}{2020}]%
{xiong2020approximate}
\bibfield{author}{\bibinfo{person}{Lee Xiong}, \bibinfo{person}{Chenyan Xiong},
\bibinfo{person}{Ye Li}, \bibinfo{person}{Kwok-Fung Tang},
\bibinfo{person}{Jialin Liu}, \bibinfo{person}{Paul~N Bennett},
\bibinfo{person}{Junaid Ahmed}, {and} \bibinfo{person}{Arnold Overwijk}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Approximate Nearest Neighbor Negative Contrastive
Learning for Dense Text Retrieval}. In
\bibinfo{booktitle}{\emph{International Conference on Learning
Representations}}.
\newblock
\bibitem[\protect\citeauthoryear{Yang, Nair, Chandradevan, Iglesias-Flores, and
Oard}{Yang et~al\mbox{.}}{2022}]%
{yang2022c3}
\bibfield{author}{\bibinfo{person}{Eugene Yang}, \bibinfo{person}{Suraj Nair},
\bibinfo{person}{Ramraj Chandradevan}, \bibinfo{person}{Rebecca
Iglesias-Flores}, {and} \bibinfo{person}{Douglas~W Oard}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{C3: Continued pretraining with contrastive weak
supervision for cross language ad-hoc retrieval}. In
\bibinfo{booktitle}{\emph{Proceedings of the 45th International ACM SIGIR
Conference on Research and Development in Information Retrieval}}.
\bibinfo{pages}{2507--2512}.
\newblock
\bibitem[\protect\citeauthoryear{Yu, Xiong, and Callan}{Yu
et~al\mbox{.}}{2021}]%
{yu2021improving}
\bibfield{author}{\bibinfo{person}{HongChien Yu}, \bibinfo{person}{Chenyan
Xiong}, {and} \bibinfo{person}{Jamie Callan}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Improving Query Representations for Dense Retrieval
with Pseudo Relevance Feedback}. In \bibinfo{booktitle}{\emph{Proceedings of
the 30th ACM International Conference on Information \& Knowledge
Management}}. \bibinfo{pages}{3592--3596}.
\newblock
\bibitem[\protect\citeauthoryear{Zhan, Mao, Liu, Guo, Zhang, and Ma}{Zhan
et~al\mbox{.}}{2021}]%
{Zhan2021OptimizingDR}
\bibfield{author}{\bibinfo{person}{Jingtao Zhan}, \bibinfo{person}{Jiaxin Mao},
\bibinfo{person}{Yiqun Liu}, \bibinfo{person}{Jiafeng Guo},
\bibinfo{person}{Min Zhang}, {and} \bibinfo{person}{Shaoping Ma}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Optimizing Dense Retrieval Model Training with Hard
Negatives}.
\newblock \bibinfo{journal}{\emph{Proceedings of the 44th International ACM
SIGIR Conference on Research and Development in Information Retrieval}}
(\bibinfo{year}{2021}).
\newblock
\bibitem[\protect\citeauthoryear{Zhan, Mao, Liu, Zhang, and Ma}{Zhan
et~al\mbox{.}}{2020}]%
{zhan2020repbert}
\bibfield{author}{\bibinfo{person}{Jingtao Zhan}, \bibinfo{person}{Jiaxin Mao},
\bibinfo{person}{Yiqun Liu}, \bibinfo{person}{Min Zhang}, {and}
\bibinfo{person}{Shaoping Ma}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{RepBERT: Contextualized text embeddings for
first-stage retrieval}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2006.15498}}
(\bibinfo{year}{2020}).
\newblock
\bibitem[\protect\citeauthoryear{Zhao, Liu, Ren, and Wen}{Zhao
et~al\mbox{.}}{2022}]%
{zhao2022dense}
\bibfield{author}{\bibinfo{person}{Wayne~Xin Zhao}, \bibinfo{person}{Jing Liu},
\bibinfo{person}{Ruiyang Ren}, {and} \bibinfo{person}{Ji-Rong Wen}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Dense text retrieval based on pretrained language
models: A survey}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2211.14876}}
(\bibinfo{year}{2022}).
\newblock
\bibitem[\protect\citeauthoryear{Zhu, Pang, Lan, Shen, and Cheng}{Zhu
et~al\mbox{.}}{2022}]%
{zhu2022lol}
\bibfield{author}{\bibinfo{person}{Yunchang Zhu}, \bibinfo{person}{Liang Pang},
\bibinfo{person}{Yanyan Lan}, \bibinfo{person}{Huawei Shen}, {and}
\bibinfo{person}{Xueqi Cheng}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{LoL: A Comparative Regularization Loss over Query
Reformulation Losses for Pseudo-Relevance Feedback}. In
\bibinfo{booktitle}{\emph{Proceedings of the 45th International ACM SIGIR
Conference on Research and Development in Information Retrieval}}.
\bibinfo{pages}{825--836}.
\newblock
\bibitem[\protect\citeauthoryear{Zhuang, Li, and Zuccon}{Zhuang
et~al\mbox{.}}{2022a}]%
{zhuang2022implicit}
\bibfield{author}{\bibinfo{person}{Shengyao Zhuang}, \bibinfo{person}{Hang Li},
{and} \bibinfo{person}{Guido Zuccon}.} \bibinfo{year}{2022}\natexlab{a}.
\newblock \showarticletitle{Implicit feedback for dense passage retrieval: A
counterfactual approach}. In \bibinfo{booktitle}{\emph{Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval}}. \bibinfo{pages}{18--28}.
\newblock
\bibitem[\protect\citeauthoryear{Zhuang, Ren, Shou, Pei, Gong, Zuccon, and
Jiang}{Zhuang et~al\mbox{.}}{2022b}]%
{zhuang2022bridging}
\bibfield{author}{\bibinfo{person}{Shengyao Zhuang}, \bibinfo{person}{Houxing
Ren}, \bibinfo{person}{Linjun Shou}, \bibinfo{person}{Jian Pei},
\bibinfo{person}{Ming Gong}, \bibinfo{person}{Guido Zuccon}, {and}
\bibinfo{person}{Daxin Jiang}.} \bibinfo{year}{2022}\natexlab{b}.
\newblock \showarticletitle{Bridging the gap between indexing and retrieval for
differentiable search index with query generation}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2206.10128}}
(\bibinfo{year}{2022}).
\newblock
\end{thebibliography}