Causal Document-Grounded Dialogue Pre-training
Extracted source text · arXiv 2305.10927
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage{ACL2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
\usepackage{upgreek}
\usepackage{color}
% \usepackage[T1]{fontenminipc}
\usepackage{colortbl}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{multicol}
\usepackage{multirow}
\usepackage{float}
\usepackage{graphics}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{array}
\usepackage{bm}
\usepackage{booktabs}
\usepackage{subfigure}
\usepackage{ bbold }
\usepackage{bbding}
\usepackage{times}
\usepackage{latexsym}
\usepackage{graphicx}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
\usepackage{amsfonts,amssymb}
\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage{cleveref}
\newcommand{\secref}[1]{\S\ref{#1}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\bl}[1]{\textcolor{red}{[bowen: #1]}}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{Causal Document-Grounded Dialogue Pre-training}
\author{
Yingxiu Zhao$^{1}$, \
Bowen Yu$^2$\thanks{\quad \small Corresponding author.}, Haiyang Yu$^2$, Bowen Li$^2$, Jinyang Li$^2$, \\
\textbf{
Chao Wang$^2$,
Fei Huang$^2$,
Yongbin Li$^2$, \
Nevin L. Zhang$^1$} \\
$^1$ The Hong Kong University of Science and Technology,
$^2$ Alibaba Group\\
\small \texttt{\{yzhaocx,lzhang\}@connect.ust.hk}, \texttt{libowen.ne@gmail.com},\\
\small \quad \texttt{\{yubowen.ybw,yifei.yhy,ruiyi.wc,f.huang,shuide.lyb\}@alibaba-inc.com}
}
% Yingxiu Zhao, Bowen Yu, Haiyang Yu, Bowen Li, chao wang, Fei Huang, Yongbin Li and Nevin L. Zhang
\begin{document}
\maketitle
\begin{abstract}
The goal of document-grounded dialogue (DocGD) is to generate a \textit{response} by grounding the \textit{evidence} in a \textit{supporting document} in accordance with the \textit{dialogue context}. This process involves four variables that are causally connected. Recently, task-specific pre-training has greatly boosted performances on many downstream tasks. Existing DocGD methods, however, continue to rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To tackle this issue, we are the first to present a causally-complete dataset construction strategy for building million-level DocGD pre-training corpora. To better capture causality, we further propose a causally-perturbed pre-training strategy, which introduces causal perturbations on the variables and optimizes the overall causal effect. Experiments on three benchmark datasets demonstrate that our causal pre-training achieves considerable and consistent improvements under fully-supervised, low-resource, few-shot, and zero-shot settings\footnote{The data and code will be made available to the public.}.
\end{abstract}
\section{Introduction}
\label{sec:intro}
Goal-oriented dialogue, focusing on assisting users in achieving their goals through natural language interactions, has made significant progress in recent years~\citep{peng2021soloist, he2022galaxy}.
However, these systems often face limitations in their ability to provide information beyond what can be retrieved from specific databases or domains.
To address this issue, researchers have proposed the goal-oriented \underline{Doc}ument-\underline{G}rounded \underline{D}ialogue (DocGD) task~\cite{feng2020doc2dial, feng2021multidoc2dial}, which leverages documents as the external knowledge source to support dialogue systems in meeting the diverse information needs of users.
\begin{figure}
\small
\centering
\includegraphics[width=0.45\textwidth]{figures/causal_example.pdf}
\caption{Causal graph of an example in DocGD, where four variables are causally connected: document $d$, evidence $e$, dialogue context $c$ and response $r$.}
\label{fig:causal-graph}
\vspace{-0.3in}
\end{figure}
Recently, task-specific pre-training has shown an extraordinary ability to boost performances and alleviate overfitting on downstream tasks, by mitigating the gaps between pre-training and fine-tuning due to different data distributions and training objectives~\cite{mengge2020coarse, liu2021tapex, bai2022graph}.
Nonetheless, the study of pre-training for DocGD is hindered due to the difficulty of modeling the causal effects, which is a prominent characteristic of DocGD and the key to the success of pre-training.
As shown in Figure~\ref{fig:causal-graph}, the task requires the model to identify \textbf{\textit{evidence}} in a \textbf{\textit{document}} based on the dialogue \textbf{\textit{context}}, and then utilize the grounding evidence to generate a corresponding \textbf{\textit{response}}.
This process involves the interplay of four variables that are causally connected.
To achieve precise modeling of causal effects during pre-training, two challenges must be overcome:
(1) the scarcity of large-scale and causally-complete DocGD datasets for pre-training, as opposed to dialogue generation tasks that have the advantage of utilizing conversational data from social media sources~\cite{zhang2020dialogpt}.
% (2) currently, popular pre-training techniques like reconstructing missing spans in T5~\cite{raffel2020exploring},
(2) conventional likelihood pre-training techniques (e.g.,~\citet{raffel2020exploring}) are insufficient to capture the causal relationships among variables.
To overcome the first challenge, we propose a novel strategy for building a DocGD pre-training corpus.
We define a dataset as \textbf{\textit{causally-complete}} if the dataset contains all the variables of a task, and all the reasonable causal relationships among the variables exist.
% The strategy involves the creation of a causally-complete dataset.
To build such a causally-complete corpus, we convert the documents in Wikipedia into human-like dialogues by generating pseudo-user utterances using a trained generator and modifying underlying evidence embedded in the documents to serve as agent responses.
However, it should be noted that although Wikipedia contains a wealth of knowledge in various domains, the virtual dialogues generated in this manner may not accurately reflect real-world conversations.
To address this, we supplement the Wikipedia dataset with human dialogues collected from the Reddit website, which are natural and realistic.
Briefly, we select dialogues containing valid URLs and extract the associated documents from the web pages to serve as virtual external documents to support the dialogues, thus being causally complete.
In this way, we have constructed two complementary datasets to support the pre-training for DocGD.
To address the second challenge, we propose a causally-perturbed pre-training strategy to better model the causality within our pre-training corpora.
This approach involves introducing causal interventions to both the document and evidence variables and optimizing the total effect of responses for different causes.
In general, the total effect can be decomposed into the sum of the natural direct effect (NDE) and total indirect effect (TIE) \citep{causal_counter}.
Intuitively, NDE measures the effect of irrelevant sentences in the supporting document, while TIE measures the effect of the evidence (detailed explanations in~\secref{sec:causal_framework}).
Our goal is to improve the robustness of our model against perturbations on irrelevant sentences by minimizing NDE and to encourage reliance on evidence in dialogue responses by maximizing TIE.
To achieve this, we preserve relevant evidence while perturbing the rest of the document to enhance the consistency of responses when using two document versions, thus minimizing NDE.
To maximize TIE, we remove required evidence from the document while keeping other information intact, and then decrease the probability of generating original responses.
Overall, we refer to the two aforementioned strategies jointly as \underline{Causal} \underline{D}ocument-Grounded \underline{D}ialogue (CausalDD).
We conduct adequate experiments and analyses on three DocGD benchmark datasets for both English and Chinese. The results under fully-supervised, few-shot, low-resource, and zero-shot settings demonstrate the effectiveness of our dataset construction and causally-perturbed pre-training strategies.
\section{Related Work}
\subsection{Causal Inference For NLP}
Causal Inference is a statistical modeling tool that has been applied in explanatory analysis to better understand the relationships between variables~\cite{glymour2016causal, kuang2020causal}.
In the context of named entity recognition, \citeauthor{zeng2020counterfactual}\citeyear{zeng2020counterfactual} sought to eliminate spurious correlations between context and entity tokens by replacing entities with counterfactual tokens.
\citeauthor{wu2020biased}\citeyear{wu2020biased} similarly utilized counterfactual samples in sentiment classification, replacing causal terms with their antonyms.
In our study, we take DocGD from a causal perspective, presenting the causal relationships among DocGD variables for the first time.
% Based on this analysis, we propose two strategies for enhancing the pre-training stage of DocGD, with the goal of improving the model's effectiveness in capturing causal relationships between DocGD variables.
\subsection{Document-Grounded Dialogue}
Goal-oriented dialogue generation grounded in documents is a challenging and realistic task~\cite{ma2020survey, yu2022survey}.
Researchers have increasingly utilized documents in a more flexible manner to improve the fluency and informativeness of model-generated responses, including in tasks such as Machine Reading Comprehension, Convention Question Answering, and the focus of this paper, DocGD.
To support the development of models for these tasks, various datasets have been proposed, including CoQA~\cite{reddy2019coqa}, QuAC~\cite{choi2018quac}, DoQA~\cite{campos2020doqa}, Wizard~\cite{dinan2018wizard}, Doc2dial~\cite{feng2020doc2dial}, MultiDoc2Dial~\cite{feng2021multidoc2dial} and Doc2bot~\cite{fu2022doc2bot}.
However, the high annotation requirements for document-grounded dialogues have limited the scale of available annotated data.
To address this issue,
\citeauthor{li2020zero}\citeyear{li2020zero} expressed the document knowledge as latent variables and devised a variational approach to achieve zero-resource knowledge-grounded dialogue generation.
\citeauthor{zhao2019low}\citeyear{zhao2019low} proposed a disentangled response decoder in order to isolate parameters that depend on knowledge-grounded dialogues to investigate low-resource knowledge-grounded dialogue generation.
% \citeauthor{liu2021three}\citeyear{liu2021three} leveraged weakly supervised learning to help reduce requirements for annotated knowledge-grounded dialogue data.
\citeauthor{li2021knowledge}\citeyear{li2021knowledge} homogenized different sources of knowledge (e.g., dictionaries, or knowledge graphs) into a unified representation to alleviate reliance on a single source.
These approaches omit pre-training on DocGD and merely initialize the parameters with general language models such as T5~\citep{raffel2020exploring}.
Thus, how to effectively pre-train the DocGD model is still an open problem.
In this paper, we give the answer from the perspective of causal inference and demonstrate that causal pre-training is effective in various DocGD settings.
\section{Methodology}
% \vspace{-0.07in}
In this section, we first take a casual-effect perspective on the DocGD task in~\secref{sec:causal_look}.
We then propose two strategies to overcome the challenges discussed in~\secref{sec:intro}:
(1) a dataset construction strategy in~\secref{sec:causal_construction} for building a causally-complete pre-training corpus; (2) a causally-perturbed pre-training strategy in~\secref{sec:causal_framework} for better modeling causality of DocGD.
\subsection{A Causal-Effect Perspective on DocGD}
\label{sec:causal_look}
The DocGD task is commonly formulated as a sequential process including two sub-tasks: \textit{knowledge grounding} and \textit{response generation}~\citep{feng2020doc2dial,unigdd}.
Given a dialogue context $c$, and a supporting document $d$, the objective of knowledge grounding is to determine a pertinent segment of text within the document as the supporting evidence, denoted as $e$, for the subsequent response.
Response generation then leverages this evidence $e$, along with the context $c$ of the dialogue, to produce a response $r$.
Hence, four variables $\{c,d,e,r\}$ are interrelated in a causal manner.
Specifically, the direct impact of $d$ and $c$ on $r$ can be observed through the causal paths $d \rightarrow r$ or $c \rightarrow r$, respectively. Additionally, there is also an indirect effect of $d$ and $c$ on $r$ through the intermediary variable $e$.
(See an example causal graph illustrated in Fig.~\ref{fig:causal-graph}).
% Hence, these four variables \{$c$, $d$, $e$, $r$\} are causally connected, where there are direct effects of $d$ and $c$ on $r$ via $d \rightarrow r$ or $c \rightarrow r$, and indirect effect of $d$ and $c$ via the evidence $e$ on $r$.
% , with $c$ and $d$ serving as the inputs for identifying $e$, and $c$ and $e$ used as inputs for generating $r$.
% \subsection{Cause-Effect Look at DocGD}
% Following the formulation in \ref{sec:perliminary}, the causal graph of DocGD is illustrated in Fig.xx.
% We compare two hypothetical situations $a=\{H,D,E\}$ and $a=\{H,D',E'\}$.
% We regard \textit{disturbing the variable by randomly inserting irrelevant sentence or deleting sentence} as a different treatment.
\begin{table}
\small
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccc}
\toprule
Datasets & Dialogues & Documents & Total Turns \cr \midrule
WikiDialog & 1.00M & 0.12M & 3.00M \cr
Reddit & 1.00M & 1.00M & 1.39M \cr
All & 2.00M & 1.12M & 4.39M \cr
\bottomrule
\end{tabular}
}
\caption{Statistics of CausalDD pre-training corpora.}
\label{tab:pretrain_statistics}
\vspace{-0.3in}
\end{table}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{figures/dataset.pdf}
\caption{Two complementary datasets WikiDialog and Reddit built for DocGD, which are causally complete.}
\label{fig:dataset}
\vspace{-0.15in}
\end{figure*}
\subsection{Causally-complete Dataset Construction}
\label{sec:causal_construction}
Although task-specific pre-training has been extensively studied in various fields due to its demonstrated effectiveness, there hasn't been much research on pre-training for DocGD.
This is due to the difficulty of constructing a pre-training DocGD corpus, particularly one that fully captures the causal relationships among relevant variables.
Without meticulously modeling causality, trivially integrating a dialog corpus and unverified documents may bring issues like missing variables or weak causal relationships among them.
Consequently, the model may learn spurious features being trained on the deficient data, such as the response generation solely depending on the dialogue context $c$, ignoring the document $d$.
This can lead to a significant reduction in model performance, as we demonstrate in our experiments (\secref{sec:effect_of_causal_complete}).
To overcome this challenge, we propose a pre-training data construction strategy that utilizes high-quality documents from Wikipedia to create a causally-complete dataset with virtual dialogues.
We further complement the corpus by leveraging real-world dialogues from Reddit to construct another dataset with virtual external knowledge.
Table~\ref{tab:pretrain_statistics} summarizes the basic statistics of our constructed causally-complete datasets.
\subsubsection{Causally-complete WikiDialog}
% DocGD相较于常规的Dialog的最大区别就是引入了额外的D；
% Wikipedia是已知的最高质量的文档集合，因此我们想探究能否基于Wikipedia构建因果完毕的DocGD预训练数据
% 受到[]的启发，我们观察到，wikipedia的每个段落，往往都在描述一个实体（wikipedia的title）相关的连续的信息，因此他们可以顺序地作为一段流畅对话的所grounding的部分G，注意，这部分我描述的不太清楚，可以参考wikidialog那篇文章
% 基于这样一个假设，一段wikipedia文本是D，里面选择的一句话作为当前对话轮次的G，那么如果我们可以根据D和G生成出U，以及R，就可以构建出因果完毕的；
% 为了实现这一想法，我们首先训练了一个question genenration模型，它能够根据之前的对话历史，和G，生成对话连贯且针对于G的用户question；
% 具体来说，这一部分请参考wikidialog，包括怎么训练的xxxx
% 到目前为止，我们通过条件生成保证了D，G和H三个变量之间的因果关系，遗漏了R。为了解决这一问题，我们提出，使用paraphrase模型，对G进行变换，得到R；
% 这一操作的动机在于，生成的U本来就是以G作为答案生成的，所以G本身就可以作为R，但是考虑到在真实的DocGD中R是对G的改写，所以我们对R进行了复述，保留了原始语义，但是在表达上进行了修改，贴合DocGD的任务需求，而paraphrase操作，也保证了R和G之间的因果关系；通过上述操作，我们基于wikipedia构建了因果完毕的xx
Wikipedia is an abundant source of high-quality articles, many of which are written or edited by experts who have invested significant time and effort in making their discourse clear, accurate and addressing potential readers' questions~\cite{chen2017reading}.
One essential characteristic of Wikipedia is that each page focuses on describing a specific entity, and each paragraph within the page is a cohesive piece of information about that entity, which can be viewed as a summary of a conversation discussing the entities.
Thus, Wikipedia can be naturally converted into a two-person DocGD conversation between a user and an imagined agent.
In this study, we consider a paragraph, denoted as $d=(e_1,e_2,\dots,e_m)$, consisting of $m$ sentences. Each sentence in the paragraph is treated as evidence $e$, corresponding to an agent's response in an $m$-round dialogue.
To complete the dialogue, we employ a dialogue inpainter~\cite{dai2022dialog} to generate the missing user utterances, which are interleaved with the agent's responses.
Typically, the dialog inpainter generates a pseudo session $s_{\text{inpainter}} = (u_1, e_1, u_2, e_2, ..., u_m, e_m)$, where $u_i$ is the $i$-th generated utterance for the user and $e_i$ is used as the response for the agent\footnote{We refer readers to the original paper for more details.}.
% More specifically, we initialize a partial dialogue session, $s_p=(\circ, e_1)$, where $\circ$ is a placeholder, and pass it into the dialogue inpainter to generate the first missing user utterance $u_1$.
% We then fill in $u_1$ into the partial dialogue, updating it to $s_p=(u_1, e_1, \circ, e_2)$ and repeat the process to generate the missing utterance $u_2$.
% % and use the dialogue inpainter again to predict the next missing question, $q_2$. This process is repeated,
% The dialogue inpainter will be called $m$ times to generate $m$ context-aware utterances, \{$u_1, \dots, u_m$\}.
For our causally-complete Wiki data, we first copy the first two turns from the inpainter, $s = (u_1, e_1, u_2, e_2)$.
% In order to ensure alignment with the structure of the DocGD training data, we keep the initial two turns of each dialogue as the dialogue context.
For the third turn, we randomly select one turn from the remaining turns to serve as the utterance and the grounding knowledge, respectively.
Notably, to make the target response more natural,
% \footnote{We regard the response in the third round as the target for pre-training.}
we utilize a well-trained paraphrase model to rewrite the evidence $e_3$ as the agent's response $r_3$, where $s = (u_1, e_1, u_2, e_2, u_3, r_3)$.
Finally, we acquire the dialogue context $c=(u_1, e_1, u_2, e_2, u_3)$, and the response $r_3$ paraphrased from the evidence $e_3$.
\textbf{Remark.} Why is the above construction causally complete?
First, the evidence $e$ is an exact sentence in the document $d$, and the dialogue context $c$ is generated by the dialogue inpainter based on $e$, so the evidence $e$ can be uniquely determined in $d$ by $c$, that is, $e$ is the effect of $\{c,d\}$ .
Considering response $r$ is a paraphrase of $e$, so $e$ is the direct cause of $r$, and the causal paths $c\rightarrow r$ and $d\rightarrow r$ can also be implicitly established.
\subsubsection{Causally-complete Reddit}
Despite the causally-complete dataset of WikiDialog, it is still possible that the generated virtual dialogues do not fully align with the distribution of real-world human conversations.
One way to address this issue is to use diverse and realistic conversations to supplement the pre-training corpus.
We consider Reddit as a popular online platform for person-to-person discussions, providing a wealth of dialogue context $c$ and response $r$.
However, some variables are missing: document $d$ and evidence $e$.
To gather more information about these missing variables, we notice that many submissions on Reddit include URLs.
The web pages linked by these URLs, such as related news articles, often provide specific information about the topics discussed in the submission.
The discussions and replies under a submission often revolve around these topics. Therefore, we can crawl submissions with a URL on Reddit and use the contents pointed to by the URL as the document $d$, with the conversations and replies under the submission serving as $c$ and $r$, respectively.
To create this dataset, we extract comment chains from Reddit that were posted between the years 2020 and 2022.
Given the fact that discussions on Reddit frequently take the form of tree-structured reply chains, we follow the methodology outlined in \citeauthor{zhang2020dialogpt}~\citeyear{zhang2020dialogpt} and extract each path from the root node to the leaf node as a training instance containing multiple turns of dialogue.
We selected samples that contained valid URL \footnote{A valid URL is defined as a webpage containing non-offensive English sentences. We ignored links referring to emails, videos, or images.} in either the dialogue context $c$ or response $r$.
We then extracted the documents from the corresponding webpage and treated them as the grounded external knowledge $d$ for the dialogue.
It is worth noting that we can only infer a certain degree of semantic relevance between $d$, $c$, and $r$ based on the co-occurrence of URLs and conversations.
However, we cannot guarantee that (1) the response is generated by grounding on external knowledge (i.e., evidence) or that (2) the evidence can be found in the document to support the response.
To address this issue, we obtain the evidence by paraphrasing the response and then insert this pseudo-evidence to a random position in the original document, which ensures that each document contains underlying evidence entailed in the associated dialogue response.
\textbf{Remark.} Why is the above construction causally complete?
First, the evidence $e$ is a paraphrase of response $r$, so a causal relationship can be established between $r$ and $e$.
Given that $r$ is a natural response to the dialogue context $c$ on Reddit, causal paths $c \rightarrow r$ and $c \rightarrow e$ exist.
Furthermore, through the random insertion of evidence $e$ into the document $d$, $d$ can thus become the cause of $e$, and then the cause path $d \rightarrow r$ could also be established.
% Specifically, we extract comment chains scraped from Reddit spanning from 2019 to 2022 to construct the dataset.
% Considering the discussions of Reddit can be naturally expanded as tree-structured reply chains, following~\citeauthor{zhang2020dialogpt}\citeyear{zhang2020dialogpt}, we extract each path from the root node to the leaf node as a training instance containing multiple turns of dialogue.
% We select samples containing valid URLs \footnote{A valid URL means that the content contained in the webpage is composed of non-offensive English sentences. We ignore some links such as belonging to emails, videos or images.} in dialogue context $c$ or response $r$.
% Then we extract documents from the corresponding webpages and regard the documents as the grounded external knowledge $d$ of dialogues.
% Reddit是目前最大规模的公开对话论坛，里面包含着丰富的人-人对话，可以天然地得到对话历史H和回复R，但是缺失了D和G。
% 为了补充缺失的变量，我们通过大量地观察Reddit论坛，发现很多对话所在的submission中都会附带一个URL，URL指向的page描述了这个submission想讨论的主题的具体信息，比如与之相关的新闻，而一个submission下的对话往往就是围绕这个主题展开的，所以我们爬取了reddit上包含URL的submission，以URL指向的内容作为D，submission下的对话和回复作为H和R；
% 然而，我们的URL相关假设只能说明H和R是与D相关的，但是无法准确定位到底是和D里的哪部分内容相关，也就是无法找到grounding的G；
% 对于这一个挑战，我们再次使用了复述，对R进行改写后作为G随机插入D中。
% 尽管这种方式伪造了G，但是R是真实且流畅的，所以我们称这部分因果完毕的预训练数据为gold-response；
% 我们希望gold-response的数据能够和gold-grounding的组合训练，从而弥补各自的数据质量GAP
% 最终，我们构建了200万的gold-grounding和100万的gold-response数据，数据的统计分析见表xxx；
% \paragraph{Reddit}
% We extract comment chains scraped from Reddit spanning from 2019 to 2022 to construct the dataset.
% Considering the discussions of Reddit can be naturally expanded as tree-structured reply chains, following [dialogpt], we extract each path from the root node to the leaf node as a training instance containing multiple turns of dialogue.
% We select samples containing valid URLs \footnote{A valid URL means that the content contained in the webpage is composed of non-offensive English sentences. We ignore some links such as belonging to emails, videos or images.} in dialogue contexts or responses. Then we extract documents from the corresponding webpages, and regard the documents as the grounded external knowledge of dialogues.
% Following [dialogpt], we further filter the dataset by removing the instances where
% (1) documents have less than 10 words,
% (2) each document has no common spans with dialogue context or response, where \textit{common spans} refer to the overlapped phrases except stop-words between the extracted document and the dialogue,
% (3) dialogue contexts contain offensive languages, identified by phrase matching against a large blocklist,
% (4) all words in the sample are non-English.
% We perform other filtering strategies following [dialogpt], see details in Appendix xx.
% After filtering, the dataset comprises xxx dialogue instances, xxx documents, in total xxx words.
% \paragraph{Reddit Causal Completion}
% The Reddit dataset we construct above is causal-incomplete, where each sample lacks document-based evidence to guide corresponding response. To build a complete causal-effect graph for the Reddit dataset, we propose to enhance the relationship between documents and dialogues through paraphrasing.
% Specifically, we paraphrase the response in a dialogue via xx tool to another sentence as the evidence. Then we inject this pseudo-evidence before or after any sentence in the original document.
% In this way, we can ensure that each document contains a underlying evidence entailed in the associated dialogue response.
% Though the pseudo evidence is
\begin{figure}
\centering
\includegraphics[width=0.45\textwidth]{figures/model.pdf}
\caption{The illustration of causally-perturbed pre-training for CausalDD. The document in the middle on the left represents the unperturbed document $d$, while the upper and bottom documents represent perturbed documents $\Bar{d}$ and $\hat{d}$ for optimizing NDE (Eq.~\ref{eq:NDE}) and TIE (Eq.~\ref{eq:TIE}), respectively.}
\label{fig:model}
\vspace{-0.2in}
\end{figure}
\subsection{Causal Pre-Training Framework}
\label{sec:causal_framework}
\paragraph{DocGD-specific Pre-Training}
We follow the best and simplest fine-tuning method for the DocGD task described by \citet{unigdd} and align our pre-training task with fine-tuning by sequentially generating the grounding evidence $e$ and the agent response $r$ based on the input of dialogue context $c$ and associated document $d$. The basic pre-training objective is:
\begin{equation}
\vspace{-6pt}
\mathcal{L}_{\rm DocGD}=-\sum_{ (d,c,e,r) \in C}\log(p_\theta(e;r|d;c)),
\label{eq:loss_docgd}
\end{equation}
where $C$ is our constructed causally-complete pre-training corpus in~\secref{sec:causal_construction}, $\theta$ is the model parameters to be pre-trained.
\paragraph{Causally-perturbed Pre-Training}
To further facilitate the causal modeling of DocGD, we propose a causally-perturbed pre-training strategy by introducing causal perturbations to variables of DocGD and evaluating the outcomes under different causes.
% in addition to the standard $\mathcal{L}_{\rm DocGD}$ loss.
Here, we utilize a common measurement, \textit{causal effect}, to compare two potential outcomes for the same variable under two distinct treatment conditions~\cite{rubin1978bayesian, robins1986new}.
Supposed that the random variable $X$ assigned with the observed value $x$, $X = x$, represents ``under no-treatment condition'' and $X = x^{*}$ represents ``under treatment condition'' \citep{causal_counter}. The \textit{total effect} (TE) of treatment $X = x$ on the variable $Y$ compares two hypothetical situations $X = x$ and $X = x^{*}$, which is denoted as: ${\rm TE}=Y_{x^*}-Y_{x}$.
% \begin{equation}
% {\rm TE}=Y_{x}-Y_{x^*}.
% \end{equation}
In DocGD, we aim to estimate the effect of the document $d$ on the identification of evidence $e$ and the generation of response $r$. We denote $X = x$ to represent the original document $d$ (under no-treatment condition), and $X = x^*$ to represent applying perturbations to the document (under treatment condition). We use $Y$ to denote the generated sequence of evidence and response.
More precisely, we further divide the document $d$ into two parts, the sentence $e$ where the evidence span lies and the other sentences $\{d \backslash e\}$ outside the evidence scope, i.e., $d=e \cup \{d \backslash e\}$. Hence, the total effect in DocGD can be written as:
\begin{equation}
\vspace{-6pt}
{\rm TE}=Y_{\{d\backslash e\}^*, e^*}-Y_{\{d \backslash e\},e}.
% \vspace{-3pt}
\end{equation}
We adopt the decomposition in ~\citet{causal_counter} and adapt it to our causal DocGD scenario.
Concretely, TE can be decomposed into the sum of the natural direct effect (NDE) and total indirect effect (TIE).
NDE expresses the increase in the outcome $Y$ when $\{d \backslash e\}$ changes to $\{d \backslash e\}^*$:
\begin{equation}
\vspace{-6pt}
{\rm NDE}=Y_{\{d \backslash e\}^*, e}-Y_{\{d \backslash e\}, e}.
\end{equation}
TIE is the difference between TE and NDE:
\begin{equation}
\vspace{-6pt}
{\rm TIE}=Y_{\{d \backslash e\}^*, e^*}-Y_{\{d \backslash e\}^*, e}.
\end{equation}
For an ideal causal-aware DocGD pre-trained model,
variations in ${d\backslash e}$ should not significantly affect the model's output,
while the presence of evidence $e$ within $d$ determines the model's ability to identify evidence and generate responses that are relevant to the dialogue context.
Hence, it is necessary to improve the robustness of our model against perturbations on variables by minimizing NDE, and promote reliance on evidence in the generation of dialogue responses by maximizing TIE. (See the illustration of causally-perturbed pre-training of CausalDD in Figure~\ref{fig:model}.)
To achieve this, we design two causally-perturbed pre-training objectives.
Firstly, we use Kullback-Leibler divergence to measure NDE:
\begin{equation}
\small
\mathcal{L}_{\rm NDE}=\sum_{ (d,c,e,r) \in C} KL(p_\theta(e;r|d;c)||p_\theta(e;r|\overline{d};c)),
\label{eq:NDE}
\end{equation}
where $\overline{d}=e \cup \{d \backslash e\}^*$ refers to disturbing the document $d$ by randomly deleting or inserting some sentences while keeping the evidence sentence $e$ in $d$ retained.
Secondly, to maximize TIE, we introduce the following unlikelihood loss:
\begin{align}
\mathcal{L}_{\rm TIE}=-\sum_{(d,c,e,r) \in C }^{}&\underbrace{\log(1-p_\theta(e;r|\widehat{d};c))}_{\text{unlikelihood}}.
\label{eq:TIE}
\end{align}
The situation $\widehat{d}$ here represents removing of evidence $e$ from the document $\overline{d}$, i.e., $\widehat{d}=\{d \backslash e\}^*$.
After the removal, we aim to decrease the model's probability of generating tokens in the ground truth evidence $e$ and response $r$.
Overall, our pre-training objective is the sum of standard DocGD loss and our newly proposed causally-perturbed losses as follows:
\begin{equation}
\vspace{-6pt}
\mathcal{L}=\mathcal{L}_{\rm DocGD}+\mathcal{L}_{\rm NDE}+\mathcal{L}_{\rm TIE}
\end{equation}
After pre-training, we fine-tune the obtained model on downstream datasets by optimizing $\mathcal{L}_{\rm DocGD}$ in Eq.~\ref{eq:loss_docgd} following \citet{unigdd}.
% the only difference is that UniGDD's fine-tuning is based on T5 initialized parameters, while our model is initialized using the parameters obtained from Causal Pre-training.
% Denote $w_k$ as the $k$-th token in $\hat{Y}$, $P_{\hat{Y}'}$ and $P_{\hat{Y}}$ as the predicted distribution for token $w_k$ under two hypothetical situations $a'$ and $a$, respectively.
% The situation $a'$ refers to disturbing the document $D$ while keeping the evidence $E$ retained.
% Such the disturbance facilitates the model to be more robust and invariant to irrelevant surrounding knowledge.
% \begin{equation}
% NDE=Y_{D',E,H}-Y_{D,E,H} =
% D_{KL}(P_{\hat{Y}'}||P_{\hat{Y}})
% \end{equation}
% To measure TIE, we optimize the following objective,
% \begin{equation}
% TIE=Y_{D',E',H}-Y_{D',E,H}
% \end{equation}
% The situation $a''$ here represents removing of evidence $E$ from the disturbed document $D'$.
% After the removal, we aim to decrease the model's probability of generating tokens in the ground truth response $R$, which we call them \textit{negative candidates}.
% Specifically, given the input $[H;D']$ under the situation $a''$, we define the unlikelihood loss as:
% \begin{equation}
% \mathcal{L}_{\mathrm{UL}}=-\sum_{r_i \in \mathcal{R} }^{}\log(1-p_\theta(r_i|[H;D'])).
% \end{equation}
% Following \citep{unlikelihood}, we incorporate the unlikelihood loss into a token-level unlikelihood objective to augment the maximum likelihood training.
% % \begin{align}
% \begin{equation}
% \mathcal{L}=-\sum_{r_i \in R}^{}\underbrace{\log(1-p_\theta(r_i|[H;D']))}_{\text{unlikelihood}}-\underbrace{\log p_\theta(r_i|[H;D])}_{\text{likelihood}}.
% To achieve this, we divide the ability into three parts:
% \textbf{\textit{Understanding}}, \textbf{\textit{Grounding}} and \textit{\textbf{Responding}}. We then design three tasks to equip the model with these abilities.
% \paragraph{Understanding} is the basis of the other two capabilities, which means that the model needs to understand documents, dialogue histories and current utterances to achieve the alignment between documents and dialogues.
% We adopt the span denoising objective introduced in T5 [cite] to facilitate the understanding ability of the model. Specifically, we randomly corrupt $15\%$ of the concatenated sequences from dialogues and documents by sentinel tokens and reconstruct them given the unmasked tokens.
% \paragraph{Grounding} refers to identifying a text span as the evidence given the dialogue context and supporting document to assist the dialogue response generation.
% \paragraph{Responding} refers to generating fluent responses which are coherent to dialogue contexts and consistent to the grounding evidence.
% \subsection{causally-perturbed Pre-training Objective}
% \label{sec:causal_objective}
% 除了因果完毕的预训练数据，我们还想在预训练目标上进一步增强，强化预训练模型对DocGD的因果性建模；在因果推断中，衡量因果性建模的主要手段是计算Causal effects，这里把preliminary里关于Causal effects的部分移过来；
% 为了充分地将我们构建的预训练数据集中的因果性嵌入到与训练模型中，我们进一步提出strengthen the pre-training objective for modeling the causality of the DocGD task。
% 具体来说，在Causal Inference中，causal effects reflect the comparisons between two potential outcomes of the same individual under two different situations.
% The total effect (TE) is used to measure the differences of the effect $Y$ under two situations $X=x$ and $X=x^*$, which is denoted as:
% \begin{equation}
% \mathcal{L}_{DocGD}=-\sum_{ (d,c,e,r) \in C}\log(p_\theta(e;r|d;c)).
% \end{equation}
% where $C$ is our constructed causally-complete pre-training corpus in Section~\ref{sec:causal_construction}, $\theta$ is the model parameters to be pre-trained.
% In addition to the constructed large-scale pre-training dataset that is causally-complete, we also aim to further strengthen the pre-training objectives for modeling the causality of the DocGD task.
% We propose two ways to measure the the effectiveness of causal modeling in causal inference as illustrated in Sec.xx.
% % 因此，我们希望将Causal effects的度量加入预训练目标中，希望最大化NDE，最小化TIE（天色已晚，是否是最大化/最小化你再check一下。
% \subsubsection{Casual Disturbance}
% We use Kullback-Leibler divergence (KL) to measure NDE.
% Denote $w_k$ as the $k$-th token in $\hat{Y}$, $P_{\hat{Y}'}$ and $P_{\hat{Y}}$ as the predicted distribution for token $w_k$ under two hypothetical situations $a'$ and $a$, respectively.
% The situation $a'$ refers to disturbing the document $D$ while keeping the evidence $E$ retained.
% Such the disturbance facilitates the model to be more robust and invariant to irrelevant surrounding knowledge.
% \begin{equation}
% NDE=Y_{D',E,H}-Y_{D,E,H} =
% D_{KL}(P_{\hat{Y}'}||P_{\hat{Y}})
% \end{equation}
% To measure TIE, we optimize the following objective,
% \begin{equation}
% TIE=Y_{D',E',H}-Y_{D',E,H}
% \end{equation}
% The situation $a''$ here represents removing of evidence $E$ from the disturbed document $D'$.
% After the removal, we aim to decrease the model's probability of generating tokens in the ground truth response $R$, which we call them \textit{negative candidates}.
% Specifically, given the input $[H;D']$ under the situation $a''$, we define the unlikelihood loss as:
% \begin{equation}
% \mathcal{L}_{\mathrm{UL}}=-\sum_{r_i \in \mathcal{R} }^{}\log(1-p_\theta(r_i|[H;D'])).
% \end{equation}
% Following \citep{unlikelihood}, we incorporate the unlikelihood loss into a token-level unlikelihood objective to augment the maximum likelihood training.
% % \begin{align}
% \begin{equation}
% \mathcal{L}=-\sum_{r_i \in R}^{}\underbrace{\log(1-p_\theta(r_i|[H;D']))}_{\text{unlikelihood}}-\underbrace{\log p_\theta(r_i|[H;D])}_{\text{likelihood}}.
% \end{equation}
% \end{align}
% models can be made significantly more robust against inconsistent predictions using our edited data
% make the models consistent to semantic variations
% \subsection{Pre-training Model}
% 我们选择了目前最好也是最简单的backbone UNIGDD，作为我们预训练的基础模型，它将DocGD建模为一个Seq2Seq任务，将D和U拼接输入，顺序地Decoding出G和R，其主要的训练目标为
% 结合NDE和TIE，整体的预训练目标为三个相加，如下
% We choose the best and simplest backbone UniGDD \citep{unigdd} at present as our pre-trained basic model. It models DocGD as a Seq2Seq task, splicing D and U into input, and sequentially Decoding G and R. The main The training target is
% Combining NDE and TIE, the overall pre-training goal is three additions, as follows
\section{Experiments}
\begin{table}
\centering
\small
\begin{tabular}{lcc}
\toprule
Model & EM & F1 \cr \midrule
% VLP (Q-only) & 34.9 &22.2 &13.4 & -- \cr
BERTQA & 42.6 &36.7 \cr
BERT-PR-large & 44.2 & 38.9 \cr
RoBERTa-PR-large & 55.7 & 54.6 \cr
% \midrule
Multi-Sentence & 56.1 & 57.4 \cr
DIALKI & 65.9 & 57.4 \cr
UniGDD & 65.6 & 76.4 \cr
\midrule
CausalDD & \textbf{66.0} & \textbf{77.3} \cr
\bottomrule
\end{tabular}
\caption{Results on Doc2dial knowledge identification.}
\label{tab:doc2dial_grounding}
\end{table}
\begin{table}
\centering
\small
\begin{tabular}{lc}
\toprule
Model & BLEU \cr \midrule
DIALKI+BART-base & 25.8 \cr
RoBERTa-PR-large+BART-base & 39.6 \cr
RoBERTa-large+T5-base & 40.7 \cr
% \midrule
UniGDD & 42.4 \cr
\midrule
CausalDD & \textbf{43.1} \cr
\bottomrule
\end{tabular}
\caption{Results on Doc2dial response generation.}
\label{tab:doc2dial_response}
\vspace{-0.1in}
\end{table}
\begin{table}
\centering
\small
\begin{tabular}{lccc}
\toprule
Model & F1 & EM & BLEU \cr \midrule
% VLP (Q-only) & 34.9 &22.2 &13.4 & -- \cr
D$^{\rm token}$-bm25 & 36.3 & 19.0 & 13.3 \cr
D$^{\rm struct}$-bm25 & 30.5 & 14.5 & 12.8 \cr
D$^{\rm token}$-nq & 40.0 & 22.3 & 15.7 \cr
D$^{\rm struct}$-nq & 39.8 & 22.3 & 16.6 \cr
% \midrule
D$^{\rm token}$-ft & 43.6 & 26.4 & 18.8 \cr
D$^{\rm struct}$-ft & 43.5 & 26.1 & 19.5 \cr
% \midrule
D$^{\rm token}$-rr-cls-ft & 42.1 & 25.0 & 18.4 \cr
D$^{\rm struct}$-rr-cls-ft & 43.5 & 26.2 & 19.8 \cr
% \midrule
UniGDD & 61.5 & 45.8 & 31.8 \cr
\midrule
CausalDD & \textbf{63.7} & \textbf{49.3} & \textbf{33.9} \cr
\bottomrule
\end{tabular}
\caption{Results on MultiDoc2dial.}
\label{tab:multidoc2dial_response}
\end{table}
\begin{table}
\centering
\small
\begin{tabular}{lccc}
\toprule
Model & F1 & EM & BLEU \cr \midrule
% VLP (Q-only) & 34.9 &22.2 &13.4 & -- \cr
UniGDD$_{\rm Randeng}$ & 38.6 & 39.6 & 15.3 \cr
CausalDD$_{\rm Randeng}$ & \textbf{42.9} & \textbf{41.8 }& \textbf{22.3} \cr
\midrule
UniGDD$_{\rm mT5}$ & 40.6 & 41.5 & 17.7 \cr
CausalDD$_{\rm mT5}$ & \textbf{48.2} & \textbf{47.4} & \textbf{22.0} \cr
\midrule
UniGDD$_{\rm Mengzi}$ & 44.2 & 44.9 & 20.7 \cr
CausalDD$_{\rm Mengzi}$ & \textbf{48.5} & \textbf{47.8} & \textbf{25.4} \cr
\bottomrule
\end{tabular}
\caption{Results on Doc2bot.}
\label{tab:doc2bot_results}
\vspace{-0.1in}
\end{table}
\begin{table*}[htbp]
\small
\centering
\resizebox{0.95\textwidth}{!}{
\begin{tabular}{cl|ccc|ccc}
\toprule
\multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c|}{\textbf{Few-Shot}} & \multicolumn{3}{c}{\textbf{Low-Resource}} \\
& & \textbf{5-Shot} & \textbf{50-Shot} & \textbf{100-Shot} & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2dial}}}}
& UniGDD & 13.7 & 14.8 & 17.1 & 17.8 & 36.9 & 39.8 \\
& CausalDD & 25.8 (12.1$\uparrow$) & 27.4 (12.6$\uparrow$) & 29.8 (12.7$\uparrow$) & 34.7 (16.9$\uparrow$) & 43.4 (6.5$\uparrow$) & 46.9 (7.1$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{MultiDoc2dial} }}}
& UniGDD & 13.5 & 20.1 & 19.8 & 23.9 & 33.9 & 34.5 \\
& CausalDD & 29.8 (16.3$\uparrow$) & 29.7 (9.6 $\uparrow$) & 30.6 (10.8$\uparrow$) & 34.4 (10.5$\uparrow$) & 42.0 (8.1$\uparrow$) & 44.3 (9.8$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2bot} }}}
& UniGDD$_{\rm Mengzi}$ & 0.00 & 0.01 & 0.10 & 0.00 & 2.3 & 12.6 \\
& CausalDD$_{\rm Mengzi}$ & 12.3 (12.3$\uparrow$) & 12.7 (12.7$\uparrow$) & 14.9 (14.8$\uparrow$) & 13.7 (13.7$\uparrow$) & 24.8 (22.5$\uparrow$) & 31.0 (18.4$\uparrow$) \\
\bottomrule
\end{tabular}}
\caption{Few-Shot and Low-resource results for knowledge identification (F1 score). }
\label{tab:low_identification}
\vspace{-0.1in}
\end{table*}
\subsection{Datasets}
We evaluate the effectiveness of our CausalDD for DocGD in both English and Chinese. For English, we use the two causally-complete datasets constructed in~\secref{sec:causal_construction} for pre-training and evaluate the performances on two goal-oriented document-grounded dialogue datasets: Doc2dial~\citep{feng2020doc2dial} and MultiDoc2dial~\citep{feng2021multidoc2dial};
For Chinese, we utilize a translation model~\citep{wei2022learning} to translate the English pre-training data into Chinese, and evaluate the performance on a Chinese DocGD dataset Doc2bot~\citep{fu2022doc2bot}. See more details in Appendix~\ref{appendix: dataset}
\subsection{Implementation Details}
In the pre-training stage, we use the pre-trained T5-base~\cite{raffel2020exploring} to initialize the parameters of CausalDD for English DocGD.
For Chinese DocGD, we try three types of initialization: T5-Mengzi~\cite{zhang2021mengzi}, mT5~\cite{xue2021mt5}, and T5-Randeng~\cite{fengshenbang}.
After CausalDD pre-training, the model is then fine-tuned on the Doc2dial, MultiDoc2dial, and Doc2bot datasets following \citet{unigdd}, with batch size of 4 and training epochs of 5. See more training details in Appendix~\ref{appendix:implement}.
\subsection{Baselines}
We compare CausalDD with several strong baselines, including UniGDD~\citep{unigdd} and many commonly measured methods specific to each dataset.
UniGDD utilizes the original pre-trained T5 model\citep{raffel2020exploring} as the initialization and optimizes $\mathcal{L}_{\rm DocGD}$ in Eq.~\ref{eq:loss_docgd} on downstream datasets, which also serves as our most relevant baseline. See more baselines in Appendix~\ref{appendix:baselines}.
\subsection{Metrics}
In concurrence with prevalent measurements~\citep{unigdd,feng2020doc2dial,feng2021multidoc2dial}, we utilize the metrics of Exact Match (EM) and token-level F1 for the identification of evidence and BLEU~\citep{papineni2002bleu} for the generation of responses.
\subsection{Main Results}
\label{sec:main-result}
\paragraph{Fully-Supervised} As shown in Tables~\ref{tab:doc2dial_grounding}, \ref{tab:doc2dial_response}, \ref{tab:multidoc2dial_response}, and \ref{tab:doc2bot_results}, our proposed CausalDD method outperforms the baseline model on all evaluation metrics in both English and Chinese datasets.
The improvements of CausalDD over other baselines are statistically significant with $p\text{-value}<0.05$ under $t$-test. Regardless of the language or different T5 parameter initialization, CausalDD consistently surpasses the strongest baseline UniGDD.
The superiority of CausalDD demonstrates the value of the causal-complete datasets we construct for pre-training by simulating the distributions of downstream DocGD tasks and verifies that modeling causality among DocGD variables leads to better performance.
We also observe that the model's improvements are more significant on the Doc2bot dataset.
For example, with T5-Mengzi initialization, the average improvement is 3.97pt, and with mT5 initialization, the improvement further increases to 5.93pt. We speculate that it's because the training data of Doc2bot is smaller than that of Doc2dial and MultiDoc2dial, and our model obtains better initialization for DocGD through causal pre-training, thus being more data-efficient for fine-tuning.
\paragraph{Few-Shot \& Low-Resource} To verify the above speculation, we further conduct experiments on three datasets under few-shot and low-resource settings. We consider few-shot settings with only 5, 50, and 100 training examples and low-resource settings with 1\%, 5\%, and 10\% of the original training datasets to train the models.
% \footnote{Samples are randomly chosen five times and we report the average performances.}
The results are shown in Table~\ref{tab:low_identification},~\ref{tab:low_em} and \ref{tab:low_response} for knowledge identification and response generation tasks, respectively (see more results in Appendix~\ref{appendix:few-shot}).
We can notice that the improvements of CausalDD are more significant in scenarios with such a small amount of training data.
Specifically, an average improvement of 9.7 points is achieved in these settings, indicating that our method is more effective with limited human-annotated data, which is particularly important for real-world applications.
\begin{figure*}[t]
\small
\centering
\subfigure[Convergence Speed]{
\includegraphics[width=0.31\linewidth]{figures/epoch.png}
\label{fig.epoch}
}
\subfigure[Multiple Turns]{
\includegraphics[width=0.31\linewidth]{figures/multi_turns.png}
\label{fig.turn}
}
\subfigure[Multiple Evidence]{
\includegraphics[width=0.31\linewidth]{figures/evidence.png}
\label{fig.evidence}
}
\caption{Compared to the UniGDD model, CausalDD has faster convergence time, better modeling of dialog history, and a stronger grounding of multiple evidence.}
\label{fig:complicated}
\vspace{-0.1in}
\end{figure*}
\paragraph{Zero-Shot}
We evaluate the performances on three datasets under the zero-shot setting.
Results in Table~\ref{tab:zero-shot} indicate that CausalDD achieves superior performances over UniGDD without any training samples of downstream tasks. This verifies the high quality of our constructed causally-complete corpora, and the good initialization provided by CausalDD for downstream fine-tuning.
\begin{table}
\centering
\small
\resizebox{0.46\textwidth}{!}{
\begin{tabular}{llccc}
\toprule
Dataset & Model & F1 & EM & BLEU \cr \midrule
\multirow{2}{*}{Doc2dial} & UniGDD & 13.9 & 0.00 & 1.98 \cr
& CausalDD & 26.1 & 1.43 & 10.9 \cr
\midrule
\multirow{2}{*}{MultiDoc2dial} & UniGDD & 14.3 & 0.00 & 2.32\cr
& CausalDD & 30.0 & 2.76 & 13.9 \cr
\midrule
\multirow{2}{*}{Doc2Bot} & UniGDD$_{\rm Mengzi}$ & 0.00 & 0.00 & 0.00 \cr
& CausalDD$_{\rm Mengzi}$ & 11.46 & 5.25 & 1.98 \cr
\bottomrule
\end{tabular}}
\caption{Results under the zero-shot setting.}
\label{tab:zero-shot}
\end{table}
\begin{table}
\small
\centering {
\resizebox{0.46\textwidth}{!}{
\begin{tabular}{lcccccc}
\toprule
Model & \multicolumn{3}{c}{Doc2dial} & \multicolumn{3}{c}{Doc2bot}
\\ \cmidrule(r){2-4} \cmidrule(r){5-7}
& F1 & EM & BLEU & F1 & EM & BLEU \\
\midrule
Wikipedia & 76.9 & 65.7 & 42.5 & 46.8 & 46.5 & 23.6 \cr
\quad+ Reddit & 77.2 & 65.8 & 42.7 & 47.6 & 47.0 & 24.8 \cr
\quad+ NDE & 77.2 & 65.9 & 43.0 & 47.5 & 47.2 & 24.0 \cr
\quad+ TIE & 77.0 & 66.0 & 42.8 & 46.8 & 46.6 & 23.9 \cr
\bottomrule
\end{tabular}}
\caption{Ablation study of Causal on the dev set.
}
\vspace{-0.1in}
\label{tab:ablation}
}
\end{table}
\subsection{Ablation Studies}
To evaluate the contribution of each component in CausalDD, we conduct a series of ablation studies on two language datasets: Doc2dial and Dot2Bot. The baseline for these studies is pre-training CausalDD exclusively on the causally-complete WikiDialog dataset. We then assess the impact of adding additional components, including (1) supplementing WikiDialog with our constructed Reddit dataset (+Reddit), (2) minimizing the NDE loss in Eq.~\ref{eq:NDE} during pre-training (+NDE), and (3) maximizing the TIE loss in Eq.~\ref{eq:TIE} (+TIE).
Results in Table~\ref{tab:ablation} indicate that:
(1) introducing a causally-complete Reddit containing real-world dialogues enhance the ability of the model to identify knowledge and generate better responses;
(2) optimizing NDE to enhance the consistency of the model outputs with different support documents can enhance the robustness of the model;
(3) optimizing TIE to prevent the normal output of the model when removing evidence from documents increases the model's reliance on the grounding evidence.
These results validate that each component has a positive effect on CausalDD, leading to its better capability of modeling causal relationships among DocGD variables.
To assess the effectiveness of our created complementary datasets, we also conduct \textbf{a study of the case} provided by \citet{unigdd} that compares the response outputs of CausalDD after pre-training only on WikiDialog, only on Reddit, and on their combined corpora followed by Doc2dial fine-tuning.
\textbf{See details in Appendix~\ref{appendix:case}}.
% \vspace{-0.3in}
\subsection{Effects of Causally-complete Data}
\label{sec:effect_of_causal_complete}
The creation of causally-complete pre-training data is one of the contributions of this paper.
But (1) \textit{is causally-complete data really necessary for DocGD pre-training?}
(2) \textit{what problems would arise if part of the causality in the pre-training data was missing?}
To address these two questions, we build a causally-incomplete pre-training dataset by removing the introduced evidence $e$ from the previously-built Reddit dataset (i.e, the document $d=\{d \backslash e\}$). Then pre-training task is to generate responses $r$ based solely on documents and dialogue context $c$, without identifying knowledge first. We also pre-train a model using a causally-complete Reddit dataset for comparison.
The results of Table~\ref{tab:doc2dial_incomplete} indicate that performance degrades when pre-training data cannot adequately model causal connections. The comparison with UniGDD (i.e., initialized with the original pre-trained T5) demonstrates that causally-incomplete pre-training introduces bias, resulting in a discrepancy between pre-training and fine-tuning.
\begin{table}
\centering
\small
\begin{tabular}{lcc}
\toprule
Model & EM & F1 \cr \midrule
UniGDD & 65.6 & 76.4 \cr
CausalDD$_{incomplete}$ & 65.3 & 76.4 \cr
\midrule
CausalDD$_{complete}$ & 65.8 & 77.1 \cr
\bottomrule
\end{tabular}
\caption{Performance comparison with causal-incomplete and complete pre-training data on Doc2dial}
\label{tab:doc2dial_incomplete}
\vspace{-0.1in}
\end{table}
\subsection{Other Benefits of Causal Pre-training}
In addition to overall performance improvement, we also observe some additional benefits brought by causal pre-training: (1) faster convergence speed (Figure~\ref{fig.epoch}), our model achieves good results in the first epoch due to having more DocDG-specific initialization parameters compared to general pre-training;
(2) better modeling of dialog history (Figure~\ref{fig.turn}), we find better performance across all turns when we divided the Doc2bot test set based on the number of turns in the dialog history;
(3) a better ability to ground complex evidence in Doc2bot, many samples require the model to ground multiple relevant segments in the document, and as the number of relevant evidence increases, CausalDD still shows better performance compared to UniGDD (Figure~\ref{fig.evidence}).
\section{Conclusion}
In this paper, we demonstrate that modeling complete causal relationships among variables (i.e., documents, dialogue contexts, evidence, and responses) is necessary for pre-training for document-grounded dialogue task (DocGD). We propose a strategy for creating causally-complete pre-training datasets and design a causally-perturbed pre-training strategy to model the causality of DocGD. To the best of our knowledge, this is the first work that analyzes DocGD from a causal perspective.
Extensive experiments and analyses verify that our causal pre-training method CausalDD significantly improves performance under fully-supervised, few-shot, and low-resource settings, while also accelerating convergence and enhancing the ability to handle complex cases.
\section{Limitations}
Despite the fact that CausalDD has demonstrated its superior performance on three benchmarks, it still has a few limitations.
Firstly, the pre-training data we construct is generated by models such as the dialogue inpainter and paraphrase model. Despite the large size of our causal-complete datasets, the data quality is slightly inferior to manually annotated data.
Secondly, we only focus on the DocGD task, but in a wider knowledge-grounded Dialogue, there are other tasks such as knowledge graph-grounded dialogue, and our proposed pre-training data construction strategy may not be applicable.
Lastly, the effectiveness of task-specific pre-training will decrease as the amount of labeled data increases, so if a large amount of DocGD labeled data is provided, the performance gains brought from our approach may be marginal.
\section{Ethical Statement}
This paper constructs a new pre-training dataset for DocGD, and we discuss some related ethical considerations here.
%
First, in regards to intellectual property, the Wikipedia corpus and Reddit dump used as data sources are both freely available for research use, with the Wikipedia corpus shared under the CC BY-SA 3.0 license\footnote{\url{https://creativecommons.org/licenses/by-sa/3.0}} and the Reddit dump shared for research purposes~\cite{baumgartner2020pushshift}.
%
Second, we have taken measures to control potential risks by ensuring that the texts in Wikipedia do not contain private information.
Additionally, we have ensured that the conversation data from Reddit does not include any personal information and that the topics discussed are public and harmless.
Lastly, we note that the dataset construction procedure is fully automated and does not involve manual annotation, thereby eliminating any potential risks related to worker treatment.
\newpage
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\clearpage
\appendix
\section{Appendix}
\label{sec:appendix}
\subsection{Datasets}
\label{appendix: dataset}
Doc2dial~\cite{feng2020doc2dial} contains 3,474 dialogues with 44,149 turns for training and 661 dialogues with 8,539 turns for evaluation\footnote{Since we cannot access the test set, we report results on the development set for comparison following previous work~\cite{unigdd}.}.
MultiDoc2dial~\cite{feng2021multidoc2dial} contains 4,796 dialogues with an average of 14 turns grounded in 488 documents, with multiple documents supporting each dialogue in four different domains.
Doc2bot~\cite{fu2022doc2bot} has samples of Chinese conversations that are natural, coherent, and grounded in diverse documents. Considering Doc2bot has not been released yet, we obtained non-sensitive data from the authors, including 7199/1818/1820 turns of dialogue for training, validation, and testing, respectively.
\subsection{Implementation Details}
\label{appendix:implement}
CausalDD is pre-trained on 4 80G NVIDIA A100 GPUs with a maximum learning rate of 1e-5 and a warm-up ratio of 0.1 for one epoch.
The batch size per iteration is set to 8, and we use the AdamW optimizer with parameters beta1 = 0.9, beta2 = 0.98, and epsilon = 1e-6.
The total pre-training time is about 60 hours.
Note that both MultiDoc2dial and Doc2bot require the model to first retrieve relevant documents, identify evidence in those documents, and then generate corresponding responses.
Hence, following~\citet{glass2022re2g}, we first train an additional retrieval model and a ranking model to locate documents relevant to dialogues as the external knowledge for downstream fine-tuning (see details in Appendix~\ref{appendix:retrieval}).
\subsection{Details of Retrieval and ranking}
\label{appendix:retrieval}
Because the Multidoc2dial and Doc2bot datasets do not give the document that the current dialog needs to be grounded but require the model to find the relevant document in the document corpus $\mathbb{Z}$, so we introduce an additional retrieval model and ranking model to find the most relevant document of the current dialogue context $c$
\textbf{Retrieve.}
We use DPR~\cite{karpukhin2020dense} as our retriever, which projects dialog context and documents to a shared space using two BERT encoders~\cite{kenton2019bert}).
During retrieval, we perform a maximum inner-product search with FAISS~\cite{lewis2020retrieval}.
Formally, we retrieve $K$ most relevant document $\mathbb{Z}_K=z_{[1, \cdots, K]}\in \mathbb{Z}$ for dialogue $c$ as:
\begin{equation}
\small
\mathbb{Z}_K = \left\{z_i\in \mathbb{Z}|\mathrm{topK}\, \{\textrm{BERT}(q)^\top \textrm{BERT}(z_i)\} \right\}.
\end{equation}
The goal of retrieval training is to develop an encoder that maps a given dialogue $d$ and all relevant documents into an embedding space such that the dialogue is close in proximity to its corresponding ground-truth document $z^+$.
During training, we would like to maximize
$P_{retr}(z^+|q, \mathbb{Z})$:
\begin{equation}
P_{retr}(z^+|q, \mathbb{Z})=\frac{{\rm exp}({\rm sim}(q,z^+))}{\sum_{z\in \mathbb{Z}} {\rm exp}({\rm sim}(q,z))}
\label{equ:retrieval2}
\end{equation}
where ${\rm sim}(q,z)$ is the cosine similarity between the normalized embeddings of the dialogue and document, generated by the BERT encoder.
In order to perform contrastive learning, a set of negative documents must be sampled as it is not feasible to enumerate all other evidence.
This is done by using the BM25 algorithm to retrieve the most difficult negative clue for each positive clue and then placing them into batches of 128 instances.
The training loss is then calculated as the negative log-likelihood for the positive document.
% where BERT(⋅) means the vectors encoded by BERT.
\textbf{Rank.} The ranker we use is based on the sequence-pair classification.
The dialogue $q$ and each candidate document $z_i \in \mathbb{Z}_K$ are input together to a BERT followed by a projection layer and Sigmoid function to calculate the ranking score of $z_i$:
\begin{equation}
s_i={\rm Sigmoid}({\rm Linear}({\rm BERT} (z_i \oplus q))\}.
\end{equation}
The training of the ranker begins by gathering the initial retrieval results on the training set.
The top 36 samples (excluding the ground-truth evidence $z ^+$) returned by the retrieval module are used as negative examples, and the ranker model is trained to distinguish positive cases from negative cases.
During inference, we first use the retrieval model to obtain the relevant document list and then use the rank model to identify the most relevant one as the supporting document.
Note that we use the same retrieval and ranking model for CausalDD and our baseline UniGDD.
\subsection{Baselines}
\label{appendix:baselines}
In Doc2dial, for the task of knowledge identification, we compare CausalDD with several strong baselines, including UniGDD~\cite{unigdd}, BERTQA~\cite{kenton2019bert}, BERT-PR~\cite{daheim2021cascaded}, RoBERTa-PR~\cite{daheim2021cascaded}, Multi-Sentence~\cite{wu2021dialki}, and DIALKI~\cite{wu2021dialki}.
The other models formulate knowledge identification as a machine reading comprehension task and extract the grounding span from the document.
For the response generation task, we compare CausalDD with UniGDD and several pipeline methods, including DIALKI+BART that uses DIALKI for knowledge identification and BART~\cite{lewis2019bart} for response generation, RoBERTa-PR+BART, and RoBERTa+T5 \citep{unigdd}.
In MultiDoc2dial, we first use the same Retrieval and Ranking module as Re2G~\cite{glass2022re2g} to obtain relevant documents as input for UniGDD and CausalDD.
We also compare a series of baselines set up by \citet{feng2021multidoc2dial}, which use BM25 and multiple DPR variances as retrievers, and use a BART-large pre-trained on the CNN dataset as the generation module.
In Doc2bot, we mainly compare with UniGDD and use three different T5 pre-trained models T5-Mengzi~\cite{zhang2021mengzi}, mT5~\cite{xue2021mt5}, and T5-Randeng~\cite{fengshenbang}. to initialize for a more comprehensive comparison.
\subsection{More Results of Few-Shot \& Low Resource}
We present experimental results of CausalDD and the strongest baseline UniGDD in Table~\ref{tab:low_em} and~\ref{tab:low_response}.
\label{appendix:few-shot}
\begin{table*}[htbp]
\small
\centering
\begin{tabular}{cl|ccc|ccc}
\toprule
\multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c|}{\textbf{Few-Shot}} & \multicolumn{3}{c}{\textbf{Low-Resource}} \\
& & \textbf{5-Shot} & \textbf{50-Shot} & \textbf{100-Shot} & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2dial}}}}
& UniGDD & 0.00 & 0.00 & 0.00 & 0.38 & 6.11 & 3.43 \\
& CausalDD & 1.50 (1.50$\uparrow$) & 1.95 (1.95$\uparrow$) & 2.49 (2.49$\uparrow$) & 6.16 (5.78$\uparrow$) & 15.4 (9.29$\uparrow$) & 20.1 (16.7$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{MultiDoc2dial} }}}
& UniGDD & 0.00 & 0.24 & 0.07 & 0.00 & 9.42 & 3.95 \\
& CausalDD & 2.93 (2.93$\uparrow$) & 3.02 (2.78$\uparrow$) & 3.31 (3.24$\uparrow$) & 7.24 (7.24$\uparrow$) & 14.5 (5.08$\uparrow$) & 17.1 (13.2$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2bot} }}}
& UniGDD$_{\rm Mengzi}$ & 0.00 & 0.00 & 0.00 & 0.00 & 1.18 & 11.7 \\
& CausalDD$_{\rm Mengzi}$ & 5.83 (5.83$\uparrow$) & 5.99 (5.99$\uparrow$) & 6.36 (6.36$\uparrow$) & 5.52 (5.52$\uparrow$) & 21.0 (19.8$\uparrow$) & 28.7 (17.0$\uparrow$) \\
\bottomrule
\end{tabular}%
\caption{Few-Shot and Low-resource results for knowledge identification (EM score). }
\label{tab:low_em}
\end{table*}
\begin{table*}[htbp]
\small
\centering
\begin{tabular}{cl|ccc|ccc}
\toprule
\multirow{2}{*}{\textbf{Dataset}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c|}{\textbf{Few-Shot}} & \multicolumn{3}{c}{\textbf{Low-Resource}} \\
& & \textbf{5-Shot} & \textbf{50-Shot} & \textbf{100-Shot} & \textbf{1\%} & \textbf{5\%} & \textbf{10\%} \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2dial}}}}
& UniGDD & 2.00 & 2.27 & 3.07 & 5.07 & 14.6 & 15.6 \\
& CausalDD & 11.1 (9.1$\uparrow$) & 11.2 (8.9 $\uparrow$) & 11.6 (8.6 $\uparrow$) & 14.9 (9.8$\uparrow$) & 16.1 (1.5$\uparrow$) & 18.7 (3.1$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{MultiDoc2dial} }}}
& UniGDD & 2.20 & 2.31 & 3.37 & 6.98 & 14.1 & 12.7 \\
& CausalDD & 14.0
(11.8$\uparrow$) & 14.1 (11.8$\uparrow$) & 13.9 (10.5$\uparrow$) & 13.5 (6.5$\uparrow$) & 14.7 (0.6$\uparrow$) & 17.0 (4.3$\uparrow$) \\
\midrule
\multicolumn{1}{c}{\multirow{2}[2]{*}{\shortstack{\textbf{Doc2bot} }}}
& UniGDD$_{\rm Mengzi}$ & 0.00 & 0.00 & 0.00 & 0.00 & 2.10 & 2.31 \\
& CausalDD$_{\rm Mengzi}$ & 2.50 (2.50$\uparrow$) & 2.61 (2.61$\uparrow$) & 3.70 (3.70$\uparrow$) & 3.00 (3.00$\uparrow$) & 12.3 (10.2$\uparrow$) & 13.1 (10.8$\uparrow$) \\
\bottomrule
\end{tabular}%
\caption{Few-Shot and Low-resource results for response generation (BLEU score). }
\label{tab:low_response}
\end{table*}
\subsection{Case Study}
From the case in Figure~\ref{fig:case}, we can refer that:
\begin{itemize}
\item UniGDD is able to accurately identify the grounding evidence in the supporting document, however, the generated response is just a simple copy of the evidence.
\item After training solely on WikiDialog, the predicted response is more fluent and more consistent with the dialogue context rather than the copy of the evidence. This verifies the high quality of our constructed causally complete WikiDialog.
\item After training solely on Reddit, the response is more colloquial while retaining high quality.
\item Pre-training the complementary datasets (i.e.,WikiDialog + Reddit) with CausalDD, the generated response is more precise and natural compared with the ground truth. This demonstrates that constructing complementary datasets that are both causally complete yields better performances for downstream tasks' fine-tuning.
\end{itemize}
\label{appendix:case}
\begin{figure*}
\small
\centering
\includegraphics[width=0.8\textwidth]{figures/case.pdf}
\caption{Case study for Doc2dial.}
\label{fig:case}
\end{figure*}
\end{document}
\begin{thebibliography}{42}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Bai et~al.(2022)Bai, Chen, and Zhang}]{bai2022graph}
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022.
\newblock Graph pre-training for amr parsing and generation.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 6001--6015.
\bibitem[{Baumgartner et~al.(2020)Baumgartner, Zannettou, Keegan, Squire, and
Blackburn}]{baumgartner2020pushshift}
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy
Blackburn. 2020.
\newblock The pushshift reddit dataset.
\newblock In \emph{Proceedings of the international AAAI conference on web and
social media}, volume~14, pages 830--839.
\bibitem[{Campos et~al.(2020)Campos, Otegi, Soroa, Deriu, Cieliebak, and
Agirre}]{campos2020doqa}
Jon~Ander Campos, Arantxa Otegi, Aitor Soroa, Jan~Milan Deriu, Mark Cieliebak,
and Eneko Agirre. 2020.
\newblock Doqa-accessing domain-specific faqs via conversational qa.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7302--7314.
\bibitem[{Chen et~al.(2017)Chen, Fisch, Weston, and Bordes}]{chen2017reading}
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.
\newblock Reading wikipedia to answer open-domain questions.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1870--1879.
\bibitem[{Choi et~al.(2018)Choi, He, Iyyer, Yatskar, Yih, Choi, Liang, and
Zettlemoyer}]{choi2018quac}
Eunsol Choi, He~He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy
Liang, and Luke Zettlemoyer. 2018.
\newblock Quac: Question answering in context.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 2174--2184.
\bibitem[{Daheim et~al.(2021)Daheim, Thulke, Dugast, and
Ney}]{daheim2021cascaded}
Nico Daheim, David Thulke, Christian Dugast, and Hermann Ney. 2021.
\newblock Cascaded span extraction and response generation for
document-grounded dialog.
\newblock In \emph{Proceedings of the 1st Workshop on Document-grounded
Dialogue and Conversational Question Answering (DialDoc 2021)}, pages 57--62.
\bibitem[{Dai et~al.(2022)Dai, Chaganty, Zhao, Amini, Rashid, Green, and
Guu}]{dai2022dialog}
Zhuyun Dai, Arun~Tejasvi Chaganty, Vincent~Y Zhao, Aida Amini, Qazi~Mamunur
Rashid, Mike Green, and Kelvin Guu. 2022.
\newblock Dialog inpainting: Turning documents into dialogs.
\newblock In \emph{International Conference on Machine Learning}, pages
4558--4586. PMLR.
\bibitem[{Dinan et~al.(2018)Dinan, Roller, Shuster, Fan, Auli, and
Weston}]{dinan2018wizard}
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason
Weston. 2018.
\newblock Wizard of wikipedia: Knowledge-powered conversational agents.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Feng et~al.(2021)Feng, Patel, Wan, and Joshi}]{feng2021multidoc2dial}
Song Feng, Siva~Sankalp Patel, Hui Wan, and Sachindra Joshi. 2021.
\newblock Multidoc2dial: Modeling dialogues grounded in multiple documents.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 6162--6176.
\bibitem[{Feng et~al.(2020)Feng, Wan, Gunasekara, Patel, Joshi, and
Lastras}]{feng2020doc2dial}
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis
Lastras. 2020.
\newblock doc2dial: A goal-oriented document-grounded dialogue dataset.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 8118--8128.
\bibitem[{Fu et~al.(2022)Fu, Zhang, Yu, Sun, Huang, Si, Li, and
Nguyen}]{fu2022doc2bot}
Haomin Fu, Yeqin Zhang, Haiyang Yu, Jian Sun, Fei Huang, Luo Si, Yongbin Li,
and Cam-Tu Nguyen. 2022.
\newblock Doc2bot: Accessing heterogeneous documents via conversational bots.
\newblock \emph{arXiv preprint arXiv:2210.11060}.
\bibitem[{Gao et~al.(2022)Gao, Zhang, and Lam}]{unigdd}
Chang Gao, Wenxuan Zhang, and Wai Lam. 2022.
\newblock Unigdd: A unified generative framework for goal-oriented
document-grounded dialogue.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers)}, pages 599--605.
\bibitem[{Glass et~al.(2022)Glass, Rossiello, Chowdhury, Naik, Cai, and
Gliozzo}]{glass2022re2g}
Michael Glass, Gaetano Rossiello, Md~Faisal~Mahbub Chowdhury, Ankita~Rajaram
Naik, Pengshan Cai, and Alfio Gliozzo. 2022.
\newblock Re2g: Retrieve, rerank, generate.
\newblock \emph{arXiv preprint arXiv:2207.06300}.
\bibitem[{Glymour et~al.(2016)Glymour, Pearl, and Jewell}]{glymour2016causal}
Madelyn Glymour, Judea Pearl, and Nicholas~P Jewell. 2016.
\newblock \emph{Causal inference in statistics: A primer}.
\newblock John Wiley \& Sons.
\bibitem[{He et~al.(2022)He, Dai, Zheng, Wu, Cao, Liu, Jiang, Yang, Huang, Si
et~al.}]{he2022galaxy}
Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng
Jiang, Min Yang, Fei Huang, Luo Si, et~al. 2022.
\newblock Galaxy: A generative pre-trained model for task-oriented dialog with
semi-supervised learning and explicit policy injection.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~36, pages 10749--10757.
\bibitem[{Karpukhin et~al.(2020)Karpukhin, Oguz, Min, Lewis, Wu, Edunov, Chen,
and Yih}]{karpukhin2020dense}
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey
Edunov, Danqi Chen, and Wen-tau Yih. 2020.
\newblock Dense passage retrieval for open-domain question answering.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 6769--6781.
\bibitem[{Kenton and Toutanova(2019)}]{kenton2019bert}
Jacob Devlin Ming-Wei~Chang Kenton and Lee~Kristina Toutanova. 2019.
\newblock Bert: Pre-training of deep bidirectional transformers for language
understanding.
\newblock In \emph{Proceedings of NAACL-HLT}, pages 4171--4186.
\bibitem[{Kuang et~al.(2020)Kuang, Li, Geng, Xu, Zhang, Liao, Huang, Ding,
Miao, and Jiang}]{kuang2020causal}
Kun Kuang, Lian Li, Zhi Geng, Lei Xu, Kun Zhang, Beishui Liao, Huaxin Huang,
Peng Ding, Wang Miao, and Zhichao Jiang. 2020.
\newblock Causal inference.
\newblock \emph{Engineering}, 6(3):253--263.
\bibitem[{Lewis et~al.(2019)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis2019bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.
\newblock Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
\newblock \emph{arXiv preprint arXiv:1910.13461}.
\bibitem[{Lewis et~al.(2020)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal,
K{\"u}ttler, Lewis, Yih, Rockt{\"a}schel et~al.}]{lewis2020retrieval}
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich K{\"u}ttler, Mike Lewis, Wen-tau Yih, Tim
Rockt{\"a}schel, et~al. 2020.
\newblock Retrieval-augmented generation for knowledge-intensive nlp tasks.
\newblock \emph{Advances in Neural Information Processing Systems},
33:9459--9474.
\bibitem[{Li et~al.(2020)Li, Xu, Wu, Zhao, Zhao, and Tao}]{li2020zero}
Linxiao Li, Can Xu, Wei Wu, Yufan Zhao, Xueliang Zhao, and Chongyang Tao. 2020.
\newblock Zero-resource knowledge-grounded dialogue generation.
\newblock \emph{Advances in Neural Information Processing Systems},
33:8475--8485.
\bibitem[{Li et~al.(2021)Li, Peng, Shen, Mao, Liden, Yu, and
Gao}]{li2021knowledge}
Yu~Li, Baolin Peng, Yelong Shen, Yi~Mao, Lars Liden, Zhou Yu, and Jianfeng Gao.
2021.
\newblock Knowledge-grounded dialogue generation with a unified knowledge
representation.
\newblock \emph{arXiv preprint arXiv:2112.07924}.
\bibitem[{Liu et~al.(2021)Liu, Chen, Guo, Ziyadi, Lin, Chen, and
Lou}]{liu2021tapex}
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and
Jian-Guang Lou. 2021.
\newblock Tapex: Table pre-training via learning a neural sql executor.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Ma et~al.(2020)Ma, Zhang, Li, and Liu}]{ma2020survey}
Longxuan Ma, Wei-Nan Zhang, Mingda Li, and Ting Liu. 2020.
\newblock A survey of document grounded dialogue systems (dgds).
\newblock \emph{arXiv preprint arXiv:2004.13818}.
\bibitem[{Mengge et~al.(2020)Mengge, Yu, Zhang, Liu, Zhang, and
Wang}]{mengge2020coarse}
Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020.
\newblock Coarse-to-fine pre-training for named entity recognition.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 6345--6354.
\bibitem[{Niu et~al.(2021)Niu, Tang, Zhang, Lu, Hua, and Wen}]{causal_counter}
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong
Wen. 2021.
\newblock Counterfactual vqa: A cause-effect look at language bias.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 12700--12710.
\bibitem[{Papineni et~al.(2002)Papineni, Roukos, Ward, and
Zhu}]{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In \emph{Proceedings of the 40th annual meeting of the Association
for Computational Linguistics}, pages 311--318.
\bibitem[{Peng et~al.(2021)Peng, Li, Li, Shayandeh, Liden, and
Gao}]{peng2021soloist}
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and
Jianfeng Gao. 2021.
\newblock Soloist: Buildingtask bots at scale with transfer learning and
machine teaching.
\newblock \emph{Transactions of the Association for Computational Linguistics},
9:807--824.
\bibitem[{Raffel et~al.(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, Liu et~al.}]{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, Peter~J Liu, et~al. 2020.
\newblock Exploring the limits of transfer learning with a unified text-to-text
transformer.
\newblock \emph{J. Mach. Learn. Res.}, 21(140):1--67.
\bibitem[{Reddy et~al.(2019)Reddy, Chen, and Manning}]{reddy2019coqa}
Siva Reddy, Danqi Chen, and Christopher~D Manning. 2019.
\newblock Coqa: A conversational question answering challenge.
\newblock \emph{Transactions of the Association for Computational Linguistics},
7:249--266.
\bibitem[{Robins(1986)}]{robins1986new}
James Robins. 1986.
\newblock A new approach to causal inference in mortality studies with a
sustained exposure period—application to control of the healthy worker
survivor effect.
\newblock \emph{Mathematical modelling}, 7(9-12):1393--1512.
\bibitem[{Rubin(1978)}]{rubin1978bayesian}
Donald~B Rubin. 1978.
\newblock Bayesian inference for causal effects: The role of randomization.
\newblock \emph{The Annals of statistics}, pages 34--58.
\bibitem[{Wang et~al.(2022)Wang, Zhang, Zhang, Yang, Gao, Wu, Dong, He, Zhuo,
Yang, Huang, Li, Wu, Lu, Zhu, Chen, Han, Pan, Wang, Wang, Wu, Zeng, Chen,
Gan, and Zhang}]{fengshenbang}
Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun
Dong, Junqing He, Jianheng Zhuo, Qi~Yang, Yongfeng Huang, Xiayu Li, Yanghan
Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao
Wang, Xiaojun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang.
2022.
\newblock Fengshenbang 1.0: Being the foundation of chinese cognitive
intelligence.
\newblock \emph{CoRR}, abs/2209.02970.
\bibitem[{Wei et~al.(2022)Wei, Yu, Hu, Weng, Luo, and Jin}]{wei2022learning}
Xiangpeng Wei, Heng Yu, Yue Hu, Rongxiang Weng, Weihua Luo, and Rong Jin. 2022.
\newblock Learning to generalize to more: Continuous semantic augmentation for
neural machine translation.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 7930--7944.
\bibitem[{Wu et~al.(2020)Wu, Kuang, Zhang, Liu, Sun, Xiao, Zhuang, Si, and
Wu}]{wu2020biased}
Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao,
Yueting Zhuang, Luo Si, and Fei Wu. 2020.
\newblock De-biased court’s view generation with causality.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 763--780.
\bibitem[{Wu et~al.(2021)Wu, Lu, Hajishirzi, and Ostendorf}]{wu2021dialki}
Zeqiu Wu, Bo-Ru Lu, Hannaneh Hajishirzi, and Mari Ostendorf. 2021.
\newblock Dialki: Knowledge identification in conversational systems through
dialogue-document contextualization.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 1852--1863.
\bibitem[{Xue et~al.(2021)Xue, Constant, Roberts, Kale, Al-Rfou, Siddhant,
Barua, and Raffel}]{xue2021mt5}
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya
Siddhant, Aditya Barua, and Colin Raffel. 2021.
\newblock mt5: A massively multilingual pre-trained text-to-text transformer.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 483--498.
\bibitem[{Yu et~al.(2022)Yu, Zhu, Li, Hu, Wang, Ji, and Jiang}]{yu2022survey}
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and
Meng Jiang. 2022.
\newblock A survey of knowledge-enhanced text generation.
\newblock \emph{ACM Computing Surveys (CSUR)}.
\bibitem[{Zeng et~al.(2020)Zeng, Li, Zhai, and Zhang}]{zeng2020counterfactual}
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020.
\newblock Counterfactual generator: A weakly-supervised method for named entity
recognition.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 7270--7280.
\bibitem[{Zhang et~al.(2020)Zhang, Sun, Galley, Chen, Brockett, Gao, Gao, Liu,
and Dolan}]{zhang2020dialogpt}
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,
Jianfeng Gao, Jingjing Liu, and William~B Dolan. 2020.
\newblock Dialogpt: Large-scale generative pre-training for conversational
response generation.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics: System Demonstrations}, pages 270--278.
\bibitem[{Zhang et~al.(2021)Zhang, Zhang, Chen, Guo, Hua, Wang, and
Zhou}]{zhang2021mengzi}
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong
Wang, and Ming Zhou. 2021.
\newblock \href {http://arxiv.org/abs/2110.06696} {Mengzi: Towards lightweight
yet ingenious pre-trained models for chinese}.
\bibitem[{Zhao et~al.(2019)Zhao, Wu, Tao, Xu, Zhao, and Yan}]{zhao2019low}
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. 2019.
\newblock Low-resource knowledge-grounded dialogue generation.
\newblock In \emph{International Conference on Learning Representations}.
\end{thebibliography}