Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality
Extracted source text · arXiv 2305.14981
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage{ACL2023}
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{fontawesome5}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
\usepackage{inconsolata}
\usepackage{refstyle}
\usepackage{multirow}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage[utf8]{inputenc}
\usepackage{xcolor,colortbl}
\usepackage{microtype}
\usepackage{url}
\usepackage{longtable}
\usepackage{tikz}
\usepackage{tabularx}
\usepackage{lscape}
\usepackage{hhline}
\usepackage{bbm}
\usepackage{array}
\usepackage{xspace}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{makecell}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{multirow}
% \usepackage{xcolor}
\usepackage{caption}
\usepackage{amsmath}
\usepackage{float}
\usepackage{ragged2e}
\usepackage{breqn}
\usepackage{soul}
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
\usepackage{cleveref}
\crefformat{section}{\S#2#1#3}
\crefformat{subsection}{\S#2#1#3}
\crefformat{subsubsection}{\S#2#1#3}
\crefrangeformat{section}{\S#3#1#4 to~\S#5#2#6}
\crefmultiformat{section}{\S#2#1#3}{ and~\S#2#1#3}{, #2#1#3}{ and~#2#1#3}
\Crefformat{figure}{#2Fig.~#1#3}
\Crefmultiformat{figure}{Figs.~#2#1#3}{ and~#2#1#3}{, #2#1#3}{ and~#2#1#3}
\Crefformat{table}{#2Tab.~#1#3}
\Crefmultiformat{table}{Tabs.~#2#1#3}{ and~#2#1#3}{, #2#1#3}{ and~#2#1#3}
\Crefformat{appendix}{#2Appx.~\S#1#3}
\crefformat{algorithm}{Alg.~#2#1#3}
\Crefformat{equation}{#2Eq.~#1#3}
% \newcommand{\stitle}[1]{\vspace{1ex} \noindent{\bf #1.}}
\newcommand{\stitle}[1]{\noindent{\bf #1.}}
\interfootnotelinepenalty=10000
% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
\newcommand{\muhao}[1]{\textcolor{purple}{[\textbf{MC:} #1]}}
\newcommand{\fei}[1]{\textcolor{brown}{[\textbf{FW:} #1]}}
\newcommand{\todo}[1]{\textcolor{red}{[\textbf{}#1]}}
\newcommand{\modelname}{\textsc{EFactSum}\xspace}
\newcommand{\modelnamens}{\textsc{EFactSum}}
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\newcommand{\usc}{\raisebox{5pt}{\includegraphics[height=12pt]{fire_flower}}}
\newcommand{\iitm}{\raisebox{5pt}{\includegraphics[height=12pt]{mario_star.jpg}}}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{Improving Factuality of Abstractive Summarization \\
without Sacrificing Summary Quality }
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
\author{\textbf{Tanay Dixit}\iitm\thanks{\;\;This work was done when the first author was visiting the University of Southern California.} ~~~
\textbf{Fei Wang}\usc ~~~
\textbf{Muhao Chen}\usc \\
\iitm Indian Institute of Technology Madras ~~~
\usc University of Southern California \\
{\tt dixittanay@gmail.com} ~~
{\tt \{fwang598,muhaoche\}@usc.edu}
}
\begin{document}
\maketitle
\begin{abstract}
Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality. We propose \modelname (i.e., \textbf{E}ffective \textbf{Fact}ual \textbf{Sum}marization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics. Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training. Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.\footnote{Code is available at \url{https://github.com/tanay2001/EFactSum}.}
% Summarization is a multifaceted task that requires satisfying multiple desirable criteria, such as reference-based similarity and factual consistency. However, nearly all prior works have reported a negative relationship between reference-based similarity (e.g. ROUGE) and source-based factuality (e.g. FactCC). In this work, we propose a method to improve both types of scores at the same time. Specifically, we generate multiple candidate summaries for each training article with a base model, and then finetune the model to identify summaries that both types of scores are higher. We show that models trained using this process outperform prior works by generating reliable and concise summaries according to both automatic metrics and human evaluation.
% While prior work has proposed models that improve faithfulness, they are unable to preserve the original models’ performance across the other evaluation dimensions, that are equally important.
\end{abstract}
\section{Introduction}
% Three paras
% backround
Although recent methods have made significant improvements in abstractive summarization \cite{lewis-etal-2020-bart, raffel2020exploring,10.5555/3524938.3525989}, they do still lack a very critical component - factual consistency. Recent works \citep{cao-etal-2020-factual, kryscinski-etal-2019-neural, maynez-etal-2020-faithfulness} have shown that a majority of the model-generated summaries are unfaithful and suffer from a wide range of hallucination \citep{tang2022understanding}. Making summarization models factually consistent is critical for its trustworthiness in real-world applications.
Recent studies have made several attempts to improve factuality of abstractive summarization by either modifying the maximum likelihood estimation (MLE) training objective \citep{cao-wang-2021-cliff, goyal-durrett-2021-annotating}, directly optimizing factuality metrics using reinforcement learning \citep{cao-etal-2022-hallucinated} or improving the quality of the training data \citep{goyal-durrett-2021-annotating, nan-etal-2021-entity}. However, most of these works have reported a negative relationship between factual consistency and summary quality\footnote{summary quality as measured by metrics like ROUGE, BERTScore, etc.}. For example, \citet{goyal-durrett-2021-annotating} improve factuality at a cost of a 6-point drop in ROUGE-L, \citet{wan-bansal-2022-factpegasus} also observe a 2-point drop in ROUGE-L. Prior approaches have also optimized factuality at the cost of abstractiveness \cite{ladhak-etal-2022-faithful}. % measured by n-gram based metrics such as ROUGE scores
% \cite{wan-bansal-2022-factpegasus, chen-etal-2021-improving}.
% similar to cliff use contrastive learning just instead of generating synthetic summaries use beam search to generate model summaries that contains several hallucinations.
This leads to a critical question: \emph{Can we improve the factuality of summarization without the cost on the summary quality?}
\begin{figure}[t!]
\centering
\includegraphics[scale=0.37]{teaser_fig_fixed.png}
%\vspace{-0.5em}
\caption{Overview of our approach. For a given article, we generate a number of summaries that can be either factual (blue) or non-factual (yellow). Grey summaries are filtered out. We select a balanced set using ROUGE and then finally train the model to rank them based on the factuality score.}
\label{figure:teaser_fig}
%\vspace{-1em}
\end{figure}
To this end, we propose \modelname (i.e. \textbf{E}ffective \textbf{Fact}ual \textbf{Sum}marization): A candidate summary generation and ranking technique for contrastive summarization training (\Cref{figure:teaser_fig}) that not only achieves significant gains in factuality of abstractive summarization but also improves the summary quality. %as measured by similarity-based metrics. % Unlike prior works which focus on complex modelling techniques in order to make models more faithful, % we take an alternatetive approach focusing on generating high-quality training instances.
% Unlike prior works which focus soley on improving faithfulness of summarization models,
Unlike prior works which often sacrifice summary quality for improving faithfulness,
we take an alternative approach to improve both faithfulness and summary quality. We make use of the fine-tuning strategy by \citet{liu-etal-2022-brio} and make key modifications to the ranking process. As depicted in \Cref{figure:teaser_fig} we start %of
with generating a number of candidate summaries using existing fine-tuned models. Using these summaries, we select a subset by effectively combining two evaluation metrics of the two different criteria (\Cref{sec:ranking_strat}), thus avoiding optimizing one at the cost of the other. This technique helps obtain gains over methods that simply optimize one metric (\Cref{sec:ablation}). %thus emphasizing the need to focus on training on quality training data.
The promising results by \modelname on XSUM and CNN/DM have shown consistent improvements in both aspects over strong baselines, demonstrating effectively enhanced summarization factuality without sacrificing the quality.
% In this paper, we propose \modelname (\todo{source of name}), to break the negative relationship by
% % We present a training technique inspired from \cite{liu-etal-2022-brio}, where we not only
% training the model to generate factually consistent summaries but also favour good quality summaries at the same time. \muhao{missing the description and intuition of the method.}
% \muhao{Perhaps also introduce the name and meaning of the method.}
% We show that models trained using this technique outperform factuality aware summarization models on average by X\% on two popular news datasets, without any decrease in overall summary quality. % Human analysis of our summaries show that the summaries are factually consistent without any trade-off of overall summary quality.
% \muhao{can add more discoveries, and 1-2 contributions.}
% \todo{talk about the fact that we give importance to generating quality training data. How prior works focus solely on modelling ...}
\section{Approach}
\label{sec:approach}
Given a document ($D$), the task of summarization seeks to generate its summary ($S$) that satisfies some conditions like factuality, coherence, etc. The standard fine-tuning process involved the use of Maximum Likelihood Estimation (MLE).
%MLE cannot explicitly model the hard constraints between the document and the summary (e.g., the entities and relations must appear in the document) which are critical for factual consistency.
% \citet{liu-etal-2022-brio} highlight some fundamental issues that are present with models trained using MLE.
Inspired by \citet{liu-etal-2022-brio}, in addition to the cross-entropy loss, we incorporate a contrastive loss that encourages models to provide a higher probability mass to the more factual summaries. Formally, for every training document $D$ and a ranked list of the most probable candidate summaries $[S_{1}, S_{2},\dots S_{n}]$, the model learns to rank the summaries according to the factuality score. To achieve this, we make use of the following loss:
%\vspace{-1em}
\begin{equation}
%\small
\label{eq:contr}
\mathcal{L}_{CL} = \sum_i \sum_{j > i} \max(0, f(S_j) - f(S_i) + \lambda_{ij}) ,
\end{equation}
% %\vspace{-1em}
\noindent
where $S_i$ and $S_j$ are two different candidate summaries and $S_i$ ranks higher than $S_j$, % $ \forall i, j, i < j$.
$\lambda_{ij} = (j - i) * \lambda$ is a rank-based margin, %multiplied by the difference in rank between the candidates, i.e., $\lambda_{ij} $.
and $f(.)$ is the estimated log-probability normalized by length:
% $f(S_i)$ is the length-normalized estimated log-probability
% \footnote{We length-normalize as it is standard in comparing hypotheses in neural sequence generation \citep{cho-etal-2014-properties}.}
% \vspace{-3em}
% \small
\begin{equation}
f(S) = \frac{\sum_{t=1}^{l} \log p_{g_\theta} (s_t |D, S_{<t}; \theta)}{|S|^\alpha} .
\label{eq:score}
\end{equation}
\stitle{Candidate Set Generation} To generate the candidate summarization set $\{S_{i}\}$, we make use of an existing model and sample summaries using beam search \cite{Vijayakumar_Cogswell_Selvaraju_Sun_Lee_Crandall_Batra_2018}. We observe that just using the model trained with cross-entropy leads to generating a number of unfaithful summaries. In order to generate more faithful summaries, we make use of factually improved models.
%This model can either be a fine-tuned model using MLE or other competative factuality-aware models as discussed in \Cref{sec:imp_details}.
\stitle{Ranking Strategy}
\label{sec:ranking_strat}
Since our primary goal is to optimize factuality without adversarially affecting summary quality, we need to consider two metrics while deciding the ideal ranking. In order to measure the factuality of $S_{i}$, we choose FactCC \citep{kryscinski-etal-2020-evaluating} %due to %the following reasons. It
because it correlates well with human judgments of faithfulness \cite{pagnoni-etal-2021-understanding} and %it's runtime is considerably faster than other metrics
it is also computationally more efficient than other question-answering based metrics \cite{scialom-etal-2021-questeval}. To measure the summary quality, we use the popular ROUGE metric \citep{lin-2004-rouge}. Now, amongst the set of candidate summaries %, the summaries
that have been scored to be faithful, we further choose the top $m$ summaries that have the highest ROUGE score.
We select the set of unfaithful summaries in the same way just that we choose the $m$ summaries with the lowest ROUGE scores. This technique of incorporating two evaluation metrics helps overcome the inherent conflict \cite{chaudhury-etal-2022-x}. We highlight the importance of the proposed steps in \cref{sec:ablation}.
%the model to jointly prefer those candidate that are both faithful and of high quality (\Cref{sec:ablation}).
At last, these $2m$ summaries are used in creating the ranked list of candidate summaries for each article in the training set. The intuition behind this approach is that since the FactCC scores are not confidence scores, summaries from only one set can not provide sufficient supervision signals. Instead, training the model with balanced summaries from both sets would be beneficial.
Finally, our training objective combines the cross-entropy loss and our contrastive loss
% \vspace{-1em}
\begin{equation}
% \small
\label{eq:mul}
\mathcal{L}_{total} = \mathcal{L}_{CE} + \gamma \mathcal{L}_{CL} ,
% \vspace{-1em}
\end{equation}
where $\gamma$ is the weight of the contrastive loss.
% We note that the contrastive and the cross-entropy loss can effectively complement each other -- since the contrastive loss is defined on the sequence level, the token-level cross-entropy loss serves as a normalization to ensure that the model could assign balanced probability mass across the whole sequence.
\input{main_results}
\input{examples}
\section{Experiments}
We state the experimental setup in \Cref{sec:exp_setup} and report the results in \Cref{sec:main_results}, followed by an abstractiveness analysis in \Cref{sec:analysis}. In \Cref{sec:ablation}, we analyze the importance of the various components in our approach.
\subsection{Experimental Settings}
\label{sec:exp_setup}
\stitle{Datasets}
\label{sec:datasets}
To understand the effectiveness of \modelname, we make use of two widely-used news summarization datasets, XSUM \citep{narayan-etal-2018-dont} and CNN/DM \citep{HermannKGEKSB15}.
% XSUM is known to be extremely abstractive in nature and several models that perform well on this dataset have shown to be unfaithful
% \citep{maynez-etal-2020-faithfulness}. On the other hand, CNN/DM is more extractive in nature, and models tend to hallucinate less \citep{ladhak-etal-2022-faithful}.
\stitle{Baselines}
\label{sec:baselines}
In addition to models fine-tuned using \textit{cross-entropy} and competitive fine-tuning techniques: \textbf{BRIO} \cite{liu-etal-2022-brio}, we compare \modelname with prior works that have modified the fine-tuning process to improve factuality, including (1) \textbf{CLIFF} \cite{cao-wang-2021-cliff} which uses contrastive learning to train summarization models to differentiate between consistent and hallucinated summaries, (2) \textbf{FASum} \cite{zhu-etal-2021-enhancing} that modifies the Transformer architecture by incorporating knowledge graphs for factual consistency, and (3) \textbf{DAE} \cite{goyal-durrett-2021-annotating} that masks out the nonfactual tokens during training. This comparison is only available for the XSUM dataset.
\stitle{Metrics}
\label{sec:metrics}
To evaluate factuality, we make use of FactCC \citep{kryscinski-etal-2020-evaluating}, a popular metric that uses a BERT-based metric to measure whether the generated output is faithful. We also consider DAE \citep{goyal-durrett-2020-evaluating}, a textual-entailment-based metric that correlates well with human judgment of factuality \citep{tang2022understanding}.
It uses an arc entailment model to evaluate the factuality of a summary. We make use of the token-level score in order to complement the sentence-level scores from FactCC. For quality assessment, we use ROUGE \cite{lin-2004-rouge} and BERTScore \cite{zhang2019bertscore} to evaluate the summary against the reference.
% For quality assessment, we first use
% ROUGE \cite{lin-2004-rouge} to evaluate summary against the reference.
% Since ROUGE is shown not to correlate well with human judgements \cite{fabbri2020summeval, sai-etal-2021-perturbation}, we also use BERTScore \cite{zhang2019bertscore}, another model-based semantic similarity metric.
\stitle{Implementation Details}
\label{sec:imp_details}
We use CLIFF and cross-entropy trained models to generate the candidate set of summaries ($S_{1}, S_{2}, ..., S_{n}$).
%The most important part of our approach (\Cref{sec:analysis}) is selecting the right candidate set of summaries ($S_{1}, S_{2}, ..., S_{n}$) for every training article. We observe that just using the model trained with cross-entropy leads to generating a number of unfaithful summaries.
%In order to generate more faithful summaries, we make use of factually improved models like CLIFF.
We use $n =6$ and only retain those training articles that contain at least 2 factual and non-factual candidate summaries. Using this new subset of training data, we fine-tune BART-Large \cite{lewis-etal-2020-bart} on CNN/DM and PEGASUS \cite{10.5555/3524938.3525989} on XSUM. More details are in
\Cref{sec:app_imp_details}.
% \begin{itemize}
% \item XSUM , pegasus scores on rouge, bertscore, factcc, dae
% \item CNNDM, BART scores ....
% \item just show that choosing dataset is key compared to just simply ranking n summaries based on factuality
% \item comparisons with cliff, dae maybe
% \item abstraction analysis
% \end{itemize}
\subsection{Main Results}
\label{sec:main_results}
We report the results of the model fine-tuned using our approach in \Cref{tab:main_results}. Outputs of models fine-tuned using our strategy are presented in \Cref{tab:example} and \Cref{sec:app_outputs}. Overall we can observe the proposed \modelname leads to improvements on both the factuality metrics %while preserving if not improving on the reference-similarity metrics.
while preserving or improving the performance on reference-based similarity metrics.
%, with an overall gain on FactCC of X\% and Y\% on XSUM and CNN/DM respectively.
\noindent\par For XSUM, \modelname achieves a notable relative gain of 25\% on FactCC and 3\% on DAE (token) in comparison to PEGASUS while simultaneously showing non-trivial gains on both ROUGE and BERTScore. Although \modelname is trained to optimize FactCC, it also does well on the other evaluation metric, thus pointing out that the training process does not exploit any biases related to the evaluation metrics. One should note that although CLIFF does better on DAE, it is sacrificing summary quality. A similar story holds for CNN/DM also where \modelname achieves a relative gain of 20\% and 16\% on FactCC and DAE respectively. Unlike some of the prior works, this gain in factuality has not come at a cost of summary quality or abstractivness (\Cref{sec:analysis}). Although BRIO outperforms our approach on ROUGE and BERTScore, it substantially decreases factuality score, which is not desirable. Our approach aims to strike a balance between factuality and summary quality.
% We can see non-trivial gains on both ROUGE and BERTScore across both the datasets, thus highlighting the effectiveness of our approach.
\begin{figure}[t!]
\centering
\includegraphics[scale=.37]{abs_plot_2.pdf}
%\vspace{-0.5em}
\caption{Extractiveness scores for the various models on both the datasets. The x-axis represents the Extractiveness calculated using the \textit{coverage} score defined by \citet{grusky-etal-2018-newsroom}. Smaller the extractivness score, more the abstraction in the summaries.}
\label{fig:tradeoff}
%\vspace{-1em}
\end{figure}
% \subsection{Analysis}
\subsection{Factuality vs Abstractiveness Tradeoff}
\label{sec:analysis}
\noindent\citet{ladhak-etal-2022-faithful} show that it is naively possible to increase the factuality of generated summaries by increasing extractiveness (decreasing abstractiveness). Hence we analyze the extractiveness level of the generated summaries to understand if our method suffers from this tradeoff. Along with the extractiveness scores \cite{grusky-etal-2018-newsroom}, we compute the MINT (Metric for lexical INdependence of generated Text) scores and the abstractiveness-adjusted metrics scores \cite{dreyer-etal-2023-evaluating}. \Cref{fig:tradeoff} depicts the extractiveness levels for the various summarization systems. Scores are also presented in \Cref{sec:app_extr_results}. We can observe that the extractiveness score for our model (\modelname) is lesser than other models; it also achieves higher MINT scores (\Cref{tab:mint_scores}), which measures the abstractiveness of the summaries. Additionally, \modelname shows higher scores for abstractiveness calibrated FactCC metric ($\mu$FactCC) for both datasets. This clarifies that the additional gains in factuality are not at a cost of absractiveness.
\begin{table}[t]
\centering
\begin{tabular}{llcc}
\toprule
\bf Dataset & \bf Model & \bf MINT & \bf $\mu$FactCC \\
\midrule
\multirow{3}{*}{CNN/DM} & BART & 57.94 & 42.14 \\
& CLIFF & 52.18 & 39.77 \\
& \modelname & \bf 60.70 & \bf 47.47 \\
\midrule
\multirow{3}{*}{XSUM} & PEGASUS & 25.21 & 44.12 \\
& CLIFF & 25.31 & 43.36 \\
& \modelname & \bf 31.24 & \bf 48.61 \\
\bottomrule
\end{tabular}
\caption{Abstractiveness scores as calculated by \citet{dreyer-etal-2023-evaluating} and abstractiveness-adjusted FactCC.}
\label{tab:mint_scores}
\end{table}
\subsection{Ablation Study}
\label{sec:ablation}
In order to justify the modification made in the %training data
candidate ranking process of \modelname, we compute baselines that highlight the importance of each individual component. We perform the following studies using PEGASUS fine-tuned on XSUM.
%Results are presented in \Cref{tab:ablation}.
\stitle{Candidate Selecting Process} As explained in \Cref{sec:approach} we restrict the number of candidates summaries in-order to maintain a class \textit{balanced} set. We relax this constraint by simply scoring \textit{all} the candidate summaries using FactCC. This is represented by \modelname - w/o select. in \Cref{tab:ablation}. We can observe that this process leads to improved model factuality but still falls far short of the main approach by 4 points. Hence highlighting the advantage of focusing on generating quality training data.
\stitle{Dual Scoring Technique} To understand the importance of using ROUGE to select the top candidates from both factual and non-factual sets, we ablate this step by selecting the top factual and non-factual summaries using FactCC itself. %This leads to training data %contain the set of
%of balanced candidate summaries obtained \textit{solely} using FactCC.
This is marked as \modelname - w/o ROUGE in \Cref{tab:ablation}. Although the gains from this model on factuality are almost the same as \modelname, it negatively affects the ROUGE score.
%by more than 1 point.
\begin{table}[t!]
\centering
\small
\begin{tabular}{l|c|c}
\toprule
\bf Model & \bf R-L & \bf FactCC \\
\midrule
PEGASUS & 39.26 & 24.33 \\
\midrule
\modelname - w/o select. & 38.32 & 26.38 \\ % ablates data generation
\modelname - w/o ROUGE & 38.34 & 29.83 \\ % ablates the scoring process
\midrule
\modelname & \bf 39.45 & \bf 30.48 \\
\bottomrule
\end{tabular}
%\vspace{-0.5em}
\caption{Evaluation results for the various baseline models in \cref{sec:ablation}. We can observe that both the components in the ranking strategy is required in order to obtain maximum benefits from the training process.}
\label{tab:ablation}
%\vspace{-1em}
\end{table}
\section{Related Work}
% prior works on improving the factual consistency of abstractive summarization
% line 1: CLIFF ..., gunasekara-etal-2021-using-question, cao2021inspecting
% line 2: EntailRank,
% line 3：FactPegasus
% Our work is closely related to line 1. The difference is ...
Factual consistency in abstractive summarization has garnered much attention recently \cite{goyal-durrett-2020-evaluating,zhu-etal-2021-enhancing}.
Existing works have explored improving factual consistency during fine-tuning, inference, and pre-training stages, respectively.
For factual fine-tuning, works have applied contrastive learning \cite{cao-wang-2021-cliff, nan-etal-2021-improving}, reinforcement learning \cite{gunasekara-etal-2021-using-question} or knowledge integration \cite{zhu-etal-2021-enhancing} to teach the model identify summaries of high factual consistency while \citet{wan-bansal-2022-factpegasus} modify the pretraining process to introduce factuality-awareness. Several works have also improved summary factuality through post-processing in inference, such as correcting errors and re-ranking by factual scores \cite{cao-etal-2020-factual,dong2020multi,balachandran-etal-2022-correcting,chen2021improving,zhu-etal-2021-enhancing}.
% A more recent work has introduced factuality-awareness to pretraining
%Existing works have explored with improving factual consistency at fine-tuning stage through contrastive learning, reinforcement learning or knowledge integration \cite{cao-wang-2021-cliff,gunasekara-etal-2021-using-question,nan2021improving,zhu-etal-2021-enhancing},
%at inference stage through post-processing \cite{cao-etal-2020-factual,dong2020multi,cao-wang-2021-cliff,chen2021improving,zhu-etal-2021-enhancing},
%and at pre-training stage \cite{wan-bansal-2022-factpegasus}, respectively.
Our work differs from the aforementioned works as we improve both factuality and summary quality, unlike other methods, which often sacrifice one for the other.
%More recently \citet{chaudhury-etal-2022-x} show the negative relationship between factuality and summary quality evaluation metrics
% Although these methods have shown improvements on factuality, often at the costs of other evaluation dimensions, unlike our work, where we focus on improving both factuality and summary quality.
% Our work follows the first line of works improving factual consistency through modifying the finetuning objective.
% Different from prior works seeking better factuality solely, we focus on the trade off between summary quality and factuality, and improve both metrics simultaneously.
\section{Conclusion}
We present \modelname (\textbf{E}ffective \textbf{Fact}ual \textbf{Sum}marization), a candidate summary generation and ranking technique for contrastive summarization training, which helps make models more faithful without adversely affecting summary quality. Results show that this simple, yet effective method can achieve consistent gains on both factuality and similarity-based metrics without negatively affecting the degree of abstractiveness. We hope that our findings will encourage future research on factuality-consistent summarization to focus more on the tradeoffs between summary quality and factuality.
\section*{Acknowledgement}
We appreciate the reviewers for their insightful
comments and suggestions. We would also like to thank Raj Dabre and Sumanth Doddapaneni for their feedback on the initial versions of the work.
Tanay Dixit was supported by the NSF REU Site Grant 2051101.
Fei Wang was supported by the Annenberg Fellowship at USC.
Muhao Chen was supported by the NSF Grant IIS 2105329, by Air Force Research Laboratory under
agreement number FA8750-20-2-10002, by an Amazon Research Award and a Cisco Research Award.
Computing of this work was partly supported by a subaward of NSF Cloudbank 1925001 through UCSD.
\section*{Limitations}
While our approach helps train factuality-aware summarization models, it comes at an additional computation cost. It takes 3X time to train compared to the vanilla cross-entropy model. There is also an additional overhead computational cost in generating and scoring the candidate summaries for each article in the training dataset, but we believe that the gains justify the additional computation cost.
Improving faithfulness in summarization models is a challenging task. Although we make improvements over prior work %yet improving numerical consistency is difficult \Cref{sec:app_outputs}.
by achieving improved factuality metrics, like the compared prior works, our work has not focused on numerical consistency. This could be a meaningful research direction for follow-up work.
% \section*{Ethical Consideration}
% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\appendix
\input{appendix}
\end{document}
\definecolor{colboxcolor}{HTML}{edfbd5}
\begin{table}[t!]
\centering
\small
\setlength{\tabcolsep}{5pt}
\begin{tabular}{ l | c c c | c c}
\toprule
& \multicolumn{3}{c|}{\cellcolor[HTML]{D9EAD3}\textbf{Summ. Quality}} & \multicolumn{2}{c}{\cellcolor[HTML]{FCE5CD}\textbf{Factuality}} \\
\multirow{-2}{*}{\textbf{Model}} & \multicolumn{1}{c}{\cellcolor[HTML]{D9EAD3}\textbf{R-1}} & \textbf{\cellcolor[HTML]{D9EAD3}R-L} & \cellcolor[HTML]{D9EAD3}\textbf{BS.} & \multicolumn{1}{c}{\cellcolor[HTML]{FCE5CD}\textbf{FactCC}} & \multicolumn{1}{c}{\cellcolor[HTML]{FCE5CD}\textbf{\begin{tabular}[c|]{@{}c@{}}DAE $\downarrow$ \end{tabular}}} \\
\midrule
\multicolumn{6}{c}{\textit{\texttt{XSUM}}} \\
\midrule
PEGASUS
& 47.07
& 39.26
& 89.19
& 24.33
& 0.426 \\
BRIO
& \cellcolor[HTML]{edfbd5}\underline{48.69}
& \cellcolor[HTML]{edfbd5}\underline{40.13}
& \cellcolor[HTML]{edfbd5}\underline{90.87}
& 21.47
& 0.452
\\
\midrule
FASum
& 29.72
& 23.29
& 88.57
& \cellcolor[HTML]{edfbd5}26.08
& 0.616 \\
DAE
& 38.63
& 30.22
& 88.44
& \cellcolor[HTML]{edfbd5}26.66
& 0.462 \\
CLIFF
& 46.33
& 38.27
& 88.96
& \cellcolor[HTML]{edfbd5}24.54
& \cellcolor[HTML]{edfbd5} \bf \underline{0.386} \\
% & SimCLS
% & \bf 47.61
% & 39.44
% &
% & 23.37
% & \\
%\multirow{-5}{*}{XSUM}
\modelname
& \cellcolor[HTML]{edfbd5}\bf 47.24
& \cellcolor[HTML]{edfbd5}\bf 39.45
& \cellcolor[HTML]{edfbd5}\textbf{89.79}
& \cellcolor[HTML]{edfbd5}\textbf{\underline{30.48}}
& \cellcolor[HTML]{edfbd5} 0.417 \\
\midrule
\midrule
\multicolumn{6}{c}{\textit{\texttt{CNN/DM}}} \\
\midrule
BART
& 43.04
& 39.41
& 87.21
& 49.07
& 0.049 \\
BRIO
& \cellcolor[HTML]{edfbd5}\underline{47.53}
& \cellcolor[HTML]{edfbd5}\underline{44.02}
& \cellcolor[HTML]{edfbd5}\underline{89.12}
& 30.35
& 0.093 \\
\midrule
FASum
& 40.40
& 36.97
& \cellcolor[HTML]{edfbd5}88.23
& \cellcolor[HTML]{edfbd5}51.17
& 0.046 \\
CLIFF
& \cellcolor[HTML]{edfbd5}44.14
& \cellcolor[HTML]{edfbd5}40.72
& \cellcolor[HTML]{edfbd5}\bf 88.82
& \cellcolor[HTML]{edfbd5}51.84
& 0.047
\\
% & SimCLS
% & \bf 46.67
% & \bf 43.54
% &
% & 32.56
% & 0.70 \\
%\multirow{-4}{*}{CNN/DM}
\modelname
& \cellcolor[HTML]{edfbd5}\bf 44.37
& \cellcolor[HTML]{edfbd5}\bf 40.92
& \cellcolor[HTML]{edfbd5}88.36
& \cellcolor[HTML]{edfbd5}\textbf{\underline{60.74}}
& \cellcolor[HTML]{edfbd5}\textbf{\underline{0.041}} \\
\bottomrule
\end{tabular}
%\vspace{-0.5em}
\caption{Results of models fine-tuned on the XSUM and CNN/DM. R-1: Rouge-1 , R-L: Rouge-L , BS: BERTScore. For DAE smaller the better the score. \colorbox{colboxcolor}{Models} perform significantly better than the PEGASUS/BART model ($p<0.05$). The best result for factuality-aware training methods is \textbf{bolded}. Overall best score per metric is \underline{underlined}.}
\label{tab:main_results}
%\vspace{-1.5em}
\end{table}
\begin{table*}[t!]
% \vspace{-1em}
\scriptsize
\centering
\extrarowheight=\aboverulesep
\addtolength{\extrarowheight}{\belowrulesep}
\aboverulesep=3pt
\belowrulesep=3pt
\setlength{\tabcolsep}{2pt}
\begin{tabular}{@{}c p{0.32\textwidth} p{0.62\textwidth}}
\toprule
%\rowcolor{gray!50}
\multicolumn{1}{c}{ \bf System} & \multicolumn{1}{c}{ \bf Summary} & \multicolumn{1}{c}{ \bf Article} \\\midrule
%\rowcolor{gray!25}
\multicolumn{1}{c}{\bf Base.} & The number of migrants and refugees arriving on the Greek island of Lesbos has halved in the past week.
&
\multirow{4}{*}{\parbox[height=1.8\textwidth]{0.62\textwidth}{Lesbos used to get more than 5,000 a day. On Monday there were just four. But with Europe's borders closed, more than 50,000 migrants remain in Greece waiting for a decision about their futures. \dots But here she is in Moria, once a transit camp for migrants, now \textbf{since the EU deal with Turkey}, a detention centre, run by central government. \dots It is another sign of how Greece was simply overwhelmed by the numbers who came, while itself in the middle of an economic crisis. Most of those who arrived before March 20, the start of the EU-Turkey agreement, are free to come and go, but cannot leave the island. Those who came after that date are locked in, waiting for a decision \dots}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} The number of migrants arriving on the Greek island of Lesbos has halved since the EU struck a deal with Turkey to stem the flow.
& \\
\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & Goldman Sachs will no longer conduct face-to-face interviews with students applying for analyst jobs.
& \multirow{2}{*}{\parbox[height=1.5\textwidth]{0.62\textwidth}{The US investment bank will switch to video interviews with first-round undergraduate candidates from next month. \dots Goldman hoped the move will allow it to find students who do not attend top-tier US universities. \dots \textbf{It will still conduct second-round interviews in person}. The shift will not affect business schools or professional hires, but is part of a broader move by Goldman to use technology in the hiring process. The new method will include structured interviews, which the bank said will allow for greater comparisons between candidates \dots}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} Goldman Sachs is changing the way it hires students. \\
\multicolumn{1}{c}{\cellcolor{gray!25}} & \cellcolor{gray!25} &
\\\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & The pilot of a Turkish military jet has died after it crashed in the south-west of the country, state media report.
& \multirow{4}{*}{\parbox[height=1.5\textwidth]{0.62\textwidth}{The plane was flying over the Amanos Mountains in the southern province of Osmaniye on Monday when it lost radio contact, Anatolia news agency said. \dots Rescuers found the pilot's body near to the wreckage of the aircraft. Osmaniye Governor Celalettin Cerrah had earlier announced that a cockpit window and some other pieces of the aircraft had been found in the Caksir area\dots People living around the village of Yarpuz, about 25km (16 miles) north of the \textbf{Syrian border, said that they had heard a loud bang like an explosion}, according to local media A Turkish fighter jet was shot down by Syria over the Mediterranean in June 2012, after Syrian forces said it had entered the country's airspace.}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} A Turkish air force pilot has been killed after his jet crashed near the Syrian border , officials say.
\\
\bottomrule
\end{tabular}
%\vspace{-0.5em}
\caption{Sample summaries from PEGASUS (Base) and \modelname (Ours) on XSUM articles. The information from the article that contradicts the Base summaries is in \textbf{bold}. We can see that the outputs from our fine-tuned model not only generate faithful summaries but also capture the essential information from the article well.
% Additional outputs are in \Cref{sec:app_outputs}
}
\label{tab:example}
%\vspace{-2em}
\end{table*}
\clearpage
\section{Additional Training Details}
All experiments were carried out using 4, 24GB NVIDIA RTX A5000 GPUs. Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kgCO$_2$eq/kWh. Total emissions are estimated to be 4.84 kgCO$_2$eq of which 0 percents were directly offset. Estimations were conducted using the \href{https://mlco2.github.io/impact#compute}{MachineLearning Impact calculator} presented in \cite{lacoste2019quantifying}.
\label{sec:app_imp_details}
\paragraph{XSUM}: For every news article in XSUM, we use diverse beam search \citep{Vijayakumar_Cogswell_Selvaraju_Sun_Lee_Crandall_Batra_2018} to generate 16 summaries using fine-tuned PEGASUS\footnote{google/pegasus-xsum} and 16 summaries using CLIFF (\textit{maskrel, syslowcon, swapent} and \textit{regenrel}). We use the standard ROUGE-L\footnote{https://github.com/summanlp/evaluation/tree/master/ROUGE-RELEASE-1.5.5} implementation and for FactCC, we use the \texttt{FactCC} checkpoint from the official implementation provided by the authors\footnote{https://github.com/salesforce/factCC}. Articles for which we are unable to generate the required number of factual and non-factual summaries are discarded. In the end, our training dataset contains 145,040 data points. Choosing a bigger candidate size (>6) led to a decrease in the training dataset size as mentioned in \S\ref{sec:ranking_strat}.
\begin{table}[h]
\centering
\begin{tabular}{l|l}
\toprule
\bf Hyperparameters & \bf Value \\
\midrule
\midrule
model & google/pegasus-xsum \\
no. of params & 568M \\
max learning rate & 1e-4\\
warmup steps & 500 \\
number of epochs & 5 \\
per device batch size & 1 \\
accumulation step & 16 \\
margin & 0.001 \\
max seq length & 512 \\
mle weight & 1 \\
ranking weight & 10 \\
\bottomrule
\end{tabular}
\caption{Hyperparameters for PEGASUS on XSUM.}
\label{tab:xsum_hyper}
\end{table}
\paragraph{CNN/DM} For CNN/DM we follow the same process as described for XSUM, except here we use BART Large\footnote{facebook/bart-large-cnn}. For CLIFF on CNN/DM we use \textit{syslowcon\_maskrel, syslowcon, syslowcon\_swapent} and \textit{syslowcon\_regenrel} models. In the end our training dataset has 246,796 articles.
\paragraph{Training details} For training we use the Adam optimizer with linear learning rate scheduling for the model training. \Cref{tab:xsum_hyper} and \Cref{tab:cnndm_hyper} contain the best set of hyper-parameters for training PEGASUS on XSUM and BART on CNN/DM. These hyper-parameters were obtained after an extensive grid search. We perform validation after every 1600 steps and save the best model using the validation cross-entropy loss.
\begin{table}[h]
\centering
\begin{tabular}{l|l}
\toprule
\bf Hyperparameters & \bf Value \\
\midrule
\midrule
model & facebook/bart-large-cnn \\
no. of params & 400M \\
max learning rate & 3e-5\\
warmup steps & 500 \\
number of epochs & 5 \\
per device batch size & 1 \\
accumulation step & 16 \\
margin & 0.001 \\
max seq length & 1024 \\
mle weight & 0.1 \\
ranking weight & 10 \\
\bottomrule
\end{tabular}
\caption{Hyperparameters for BART on CNN/DM.}
\label{tab:cnndm_hyper}
\end{table}
\paragraph{Decoding parameters} We follow \citet{cao-wang-2021-cliff} and use the beam search algorithm to decode summaries. For BART, we set the beam sizes as 4 on CNN/DM and a beam size of 8 is used for PEGASUS on XSUM. The additional decoding parameters are in \Cref{tab:decod_hyper}.
\begin{table}[h]
\centering
\begin{tabular}{ll}
\toprule
\bf Hyperparameters & \bf Value \\
\midrule
\multicolumn{2}{c}{\bf BART} \\
\midrule
beam size & 4 \\
length penalty & 2 \\
max-length & 140 \\
min-length & 55 \\
\midrule
\multicolumn{2}{c}{\bf PEGASUS} \\
\midrule
beam size & 8 \\
length penalty & 0.6 \\
max-length & 62 \\
min-length & 11 \\
\bottomrule
\end{tabular}
\caption{Decoding parameters for BART and PEGASUS}
\label{tab:decod_hyper}
\end{table}
\section{Extractiveness Results}
\label{sec:app_extr_results}
The extractivenes scores as calculated using the \textit{coverage} score defined by \citet{grusky-etal-2018-newsroom} are present in \Cref{tab:extractive_analysis_cnndm} and \Cref{tab:extractive_analysis_xsum}. Lower the score the higher the abstraction. We can observe that \modelname achieves a lower abstraction level than CLIFF on both the datasets.
\begin{table}[h]
\centering
\begin{tabular}{l | c}
\toprule
\textbf{Model} & \textbf{Abstractiveness ($\downarrow$)} \\
\midrule
\midrule
Reference & \textbf{0.666} \\
Pegasus & 0.735 \\
CLIFF & 0.759 \\
\modelname & 0.720 \\
\bottomrule
\end{tabular} \\
\caption{Extractivness analysis for XSUM}
\label{tab:extractive_analysis_xsum}
\end{table}
\begin{table}[h]
\centering
\begin{tabular}{l | c}
\toprule
\textbf{Model} & \textbf{Abstractiveness ($\downarrow$)} \\
\midrule
\midrule
Reference & \textbf{0.880} \\
BART & 0.991 \\
CLIFF & 0.989 \\
\modelname & 0.979 \\
\bottomrule
\end{tabular} \\
\caption{Extractivness analysis for CNN/DM}
\label{tab:extractive_analysis_cnndm}
\end{table}
\section{Generated outputs}
More examples generated outputs by \modelname on different backbones and raw documents are in \Cref{tab:app_xsum,tab:app_cnndm}.
\label{sec:app_outputs}
\begin{table*}[t!]
\scriptsize
\centering
\extrarowheight=\aboverulesep
\addtolength{\extrarowheight}{\belowrulesep}
\aboverulesep=2pt
\belowrulesep=2pt
\setlength{\tabcolsep}{2pt}
\begin{tabular}{@{}c p{0.25\textwidth} p{0.6\textwidth}}
\toprule
%\rowcolor{gray!50}
\multicolumn{1}{c}{ \bf System} & \multicolumn{1}{c}{ \bf Summary} & \multicolumn{1}{c}{ \bf Article} \\\midrule
%\rowcolor{gray!25}
\multicolumn{1}{c}{\bf Base.} & The number of migrants and refugees arriving on the Greek island of Lesbos has halved in the past week.
&
\multirow{4}{*}{\parbox[height=1.8\textwidth]{0.6\textwidth}{Lesbos used to get more than 5,000 a day. On Monday there were just four. But with Europe's borders closed, more than 50,000 migrants remain in Greece waiting for a decision about their futures. \dots But here she is in Moria, once a transit camp for migrants, now \textbf{since the EU deal with Turkey}, a detention centre, run by central government. \dots It is another sign of how Greece was simply overwhelmed by the numbers who came, while itself in the middle of an economic crisis. Most of those who arrived before March 20, the start of the EU-Turkey agreement, are free to come and go, but cannot leave the island. Those who came after that date are locked in, waiting for a decision \dots}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} The number of migrants arriving on the Greek island of Lesbos has halved since the EU struck a deal with Turkey to stem the flow .
& \\
\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & Hundreds of eggs from two rare bird species have been stolen.
& \multirow{2}{*}{\parbox[height=1.5\textwidth]{0.6\textwidth}{The Mediterranean gull and black-headed gull eggs were illegally harvested from from islands in Poole Harbour, Dorset.\dots Natural England is urging any restaurants or pubs to ask to see a valid licence before buying eggs to prepare in meals. Birds of Poole Harbour had been surveying a group of islands in the harbour when the theft was discovered. Mediterranean gulls are classified as a Schedule One species, meaning anyone disturbing their nests must have a special licence. Paul Morton, who runs the charity, said Mediterranean gulls' eggs were not approved for human consumption, and could be a "health issue". "I'm distraught, really. To see the taking of hundreds and hundreds of eggs from an important colony is quite sickening," he said. Mr Moreton said there had been previous convictions for egg poaching in the last 10 or 15 years...}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} Hundreds of gull eggs have been stolen from a protected colony.\\
\multicolumn{1}{c}{\cellcolor{gray!25}} & \cellcolor{gray!25} & \\
\multicolumn{1}{c}{\cellcolor{gray!25}} & \cellcolor{gray!25} &
\\\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & A volcano in western Indonesia has erupted for the second time in two years, killing at least 11 people, officials say.
& \multirow{4}{*}{\parbox[height=1.5\textwidth]{0.6\textwidth}{The victims were farming in an area that was declared unsafe because of its close proximity to Mount Sinabung. The volcano was still spewing ash on Sunday, hampering rescue operations. More than a dozen people were killed when it erupted in 2014. It also erupted in 2010, after having been dormant for 400 years. Rescue teams are still scouring the area, looking for more victims who may have been killed or badly burned by the hot gas and ash clouds released in the eruption. Rescue teams were searching homes and farms in the village of Gamber, which was also evacuated in 2014. What causes volcanoes? The 2,460-metre (8,070 foot) tall volcano is among the country's most active. Indonesia, located on the Pacific Ring of Fire, has more than 120 active volcanoes.}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} At least 11 people have been killed after a volcano on the Indonesian island of Sumatra erupted , officials say .
% \multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
% & \cellcolor{gray!25} Hundreds of gull eggs have been stolen from a protected colony.\\
% \multicolumn{1}{c}{\cellcolor{gray!25}} &
\\\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & The SNP and Labour have won seats on Edinburgh Council in two by-elections.
& \multirow{4}{*}{\parbox[height=1.5\textwidth]{0.6\textwidth}{It was the first time the Single Transferable Vote (STV) system had been used to select two members in the same ward in a by-election. The SNP topped the vote in the Leith Walk by-election, while Scottish Labour won the second seat from the Greens. The by-election was called after Deidre Brock of the SNP and Maggie Chapman of the Scottish Greens stood down.\dots The turnout for the by-election was 25.1\%. The SNP also held the Midlothian West seat on Midlothian Council with a swing of 6.3\% from Labour. The party's Kelly Parry secured 1,540 votes, ahead of Labour's Ian Miller on 945 votes. The by-election was called after Owen Thompson was elected as SNP MP for the Midlothian constituency.}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} A by-election has been held in Edinburgh to fill two seats on the city council . & \\
\multicolumn{1}{c}{\cellcolor{gray!25}} & \cellcolor{gray!25} &
\\
\bottomrule
\end{tabular}
\vspace{-0.5em}
\caption{Sample summaries from PEGASUS (Base) and \modelname (Ours) on XSUM articles.}
\vspace{-1.5em}
\label{tab:app_xsum}
\end{table*}
\begin{table*}[t!]
\scriptsize
\centering
\extrarowheight=\aboverulesep
\addtolength{\extrarowheight}{\belowrulesep}
\aboverulesep=1pt
\belowrulesep=1pt
\setlength{\tabcolsep}{2pt}
\begin{tabular}{@{}c p{0.32\textwidth} p{0.62\textwidth}}
\toprule
%\rowcolor{gray!50}
\multicolumn{1}{c}{ \bf System} & \multicolumn{1}{c}{ \bf Summary} & \multicolumn{1}{c}{ \bf Article} \\\midrule
% %\rowcolor{gray!25}
% \multicolumn{1}{c}{\bf Base.} & The number of migrants and refugees arriving on the Greek island of Lesbos has halved in the past week.
% &
% \multirow{4}{*}{\parbox[height=1.8\textwidth]{0.62\textwidth}{Lesbos used to get more than 5,000 a day. On Monday there were just four. But with Europe's borders closed, more than 50,000 migrants remain in Greece waiting for a decision about their futures. \dots But here she is in Moria, once a transit camp for migrants, now \textbf{since the EU deal with Turkey}, a detention centre, run by central government. \dots It is another sign of how Greece was simply overwhelmed by the numbers who came, while itself in the middle of an economic crisis. Most of those who arrived before March 20, the start of the EU-Turkey agreement, are free to come and go, but cannot leave the island. Those who came after that date are locked in, waiting for a decision \dots}}
% \\\cmidrule{1-2}
% \multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
% & \cellcolor{gray!25} The number of migrants arriving on the Greek island of Lesbos has halved since the EU struck a deal with Turkey to stem the flow .
% & \\
\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & Video shows the lions interacting with the visitors who stand inside a metal cage attached to a car. The video was captured by ekant veer, 35, an associate professor at the university of canterbury. A number of the lions are fed directly through the metal bars, while others receive meat dropped from the back of the cage.
& \multirow{2}{*}{\parbox[height=1.5\textwidth]{0.62\textwidth}{visitors to a wildlife park in new zealand got to encounter a pride of lions up-close and personal. filmed at the orana wildlife park the countrys only open-range zoo the video shows the lions interacting with the visitors who stand inside a metal cage attached to a car. the video, which was captured by ekant veer, 35, an associate professor at the university of canterbury, also shows the lions scaling the cage and eating meat through its bars.\dots as the keeper speaks, the lion licks at a piece of meat that is held up against the bars as another lion walks across the roof of the cage. looking down at the people below, the lion wanders around as if deciding who it would like to make its prey before staring down the lens of the camera. set tongues wagging! one of the lions notices meat and begins sticking out its tongue in the hope of being fed. a lion stands next to one of the keepers and its large paw is the same size as the lady's head. the people inside can be seen recording the many lions from their phones, while another with paws the same size as the keepers head holds itself up against the cage and chews on some meat. later in the video people can be seen pointing out the various felines as a keeper moves her hand along the cage, instigating the lion to follow. still frames capture a lion standing up against the side of the cage alongside the keeper its power and size is plain to see.\dots orana wildlife trust. located on the outskirts of christchurch, the wildlife park is unique in that the people are caged in order to view the animals, not the other way around.}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} the video was filmed at the orana wildlife park in new zealand , the country 's only open-range zoo . the video shows the lions interacting with the visitors who stand inside a metal cage attached to a car . a number of the lions are fed directly through the metal bars , while others receive meat dropped from the back of the cage . \\
\multicolumn{1}{c}{\cellcolor{gray!25}} & \cellcolor{gray!25} &
\\\cmidrule{0-2}
\multicolumn{1}{c}{\bf Base} & Taxpayers are having to find 11billion a year to top up the wages of millions of people working in supermarkets and other low paid jobs. Money is paid to some 5.2million workers in the form of tax credits and other benefits. Total amount of benefits paid to staff at some companies exceeds what the firms pay in corporation tax.
& \multirow{4}{*}{\parbox[height=1.5\textwidth]{0.62\textwidth}{taxpayers are having to find 11billion a year to top up the wages of millions of people working in supermarkets and other low paid jobs. the money, which amounts to a massive public subsidy for the companies involved, is paid to some 5.2million workers in the form of tax credits and other benefits. \dots the charity is campaigning for the adoption of the living wage - 9.15 an hour in london and 7.85 for the rest of the uk - across both the public and private sector. it estimates this would reduce the need for in-work benefits by 6.7bn a year, which would make a massive dent in the 12billion reduction in welfare spending which the conservatives say is necessary. the current minimum wage for those over 21 is 6.50 an hour and will rise to 6.70 in october, da and sainsburys posted combined profits of 3.9bn last year, but between them cost the taxpayer more than 750m in benefits paid to their staff. tesco paid 519m in tax but received 364m in public subsidy for its 209,000 low-paid workers. asda spent 150m in tax but its 120,000 low-paid workers received 221m in benefits. \dots thesupermarkets said they paid above the minimum wage of 6.50 an hour for those aged over 21, regularly reviewed pay and gave employees benefits such as staff discounts. asda, which is part of the us retail goliath walmart, said pay and benefits should be considered in the round. in the usa, it is estimated that walmarts low-wage workers cost u.s. taxpayers an estimated \$6.2 billion (4.2bn) in public assistance including food stamps, medicaid and subsidised housing. \dots}}
\\\cmidrule{1-2}
\multicolumn{1}{c}{\cellcolor{gray!25}\bf Ours}
& \cellcolor{gray!25} Taxpayers are having to find 11billion a year to top up the wages of millions of people working in supermarkets and other low paid jobs. Money is paid to some 5.2million workers in the form of tax credits and other benefits. Total amount of benefits paid to staff at some companies exceeds what the firms pay in corporation tax.
\\
\bottomrule
\end{tabular}
\caption{Sample summaries from BART Large (Base) and \modelname (Ours) on CNN/DM articles.}
\label{tab:app_cnndm}
\end{table*}
\begin{thebibliography}{33}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Balachandran et~al.(2022)Balachandran, Hajishirzi, Cohen, and
Tsvetkov}]{balachandran-etal-2022-correcting}
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov.
2022.
\newblock \href {https://aclanthology.org/2022.emnlp-main.667} {Correcting
diverse factual errors in abstractive summarization via post-editing and
language model infilling}.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing}, pages 9818--9830, Abu Dhabi, United Arab
Emirates. Association for Computational Linguistics.
\bibitem[{Cao et~al.(2022)Cao, Dong, and Cheung}]{cao-etal-2022-hallucinated}
Meng Cao, Yue Dong, and Jackie Cheung. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.acl-long.236} {Hallucinated
but factual! inspecting the factuality of hallucinations in abstractive
summarization}.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 3340--3354,
Dublin, Ireland. Association for Computational Linguistics.
\bibitem[{Cao et~al.(2020)Cao, Dong, Wu, and Cheung}]{cao-etal-2020-factual}
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi~Kit Cheung. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.506} {Factual
error correction for abstractive summarization models}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 6251--6258, Online. Association
for Computational Linguistics.
\bibitem[{Cao and Wang(2021)}]{cao-wang-2021-cliff}
Shuyang Cao and Lu~Wang. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.532} {{CLIFF}:
Contrastive learning for improving faithfulness and factuality in abstractive
summarization}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 6633--6649, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Chaudhury et~al.(2022)Chaudhury, Swaminathan, Gunasekara, Crouse,
Ravishankar, Kimura, Murugesan, Fernandez~Astudillo, Naseem, Kapanipathi, and
Gray}]{chaudhury-etal-2022-x}
Subhajit Chaudhury, Sarathkrishna Swaminathan, Chulaka Gunasekara, Maxwell
Crouse, Srinivas Ravishankar, Daiki Kimura, Keerthiram Murugesan, Ram{\'o}n
Fernandez~Astudillo, Tahira Naseem, Pavan Kapanipathi, and Alexander Gray.
2022.
\newblock \href {https://aclanthology.org/2022.emnlp-main.478} {{X}-{FACTOR}: A
cross-metric evaluation of factual correctness in abstractive summarization}.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing}, pages 7100--7110, Abu Dhabi, United Arab
Emirates. Association for Computational Linguistics.
\bibitem[{Chen et~al.(2021)Chen, Zhang, Sone, and Roth}]{chen2021improving}
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021.
\newblock Improving faithfulness in abstractive summarization with contrast
candidate generation and selection.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 5935--5941.
\bibitem[{Dong et~al.(2020)Dong, Wang, Gan, Cheng, Cheung, and
Liu}]{dong2020multi}
Yue Dong, Shuohang Wang, Zhe Gan, Yu~Cheng, Jackie Chi~Kit Cheung, and Jingjing
Liu. 2020.
\newblock Multi-fact correction in abstractive text summarization.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 9320--9331.
\bibitem[{Dreyer et~al.(2023)Dreyer, Liu, Nan, Atluri, and
Ravi}]{dreyer-etal-2023-evaluating}
Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, and Sujith Ravi. 2023.
\newblock \href {https://aclanthology.org/2023.findings-eacl.156} {Evaluating
the tradeoff between abstractiveness and factuality in abstractive
summarization}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EACL 2023}, pages 2089--2105, Dubrovnik, Croatia. Association for
Computational Linguistics.
\bibitem[{Goyal and Durrett(2020)}]{goyal-durrett-2020-evaluating}
Tanya Goyal and Greg Durrett. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.findings-emnlp.322}
{Evaluating factuality in generation with dependency-level entailment}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2020}, pages 3592--3603, Online. Association for Computational
Linguistics.
\bibitem[{Goyal and Durrett(2021)}]{goyal-durrett-2021-annotating}
Tanya Goyal and Greg Durrett. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.114} {Annotating
and modeling fine-grained factuality in summarization}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 1449--1462, Online. Association for Computational
Linguistics.
\bibitem[{Grusky et~al.(2018)Grusky, Naaman, and
Artzi}]{grusky-etal-2018-newsroom}
Max Grusky, Mor Naaman, and Yoav Artzi. 2018.
\newblock \href {https://doi.org/10.18653/v1/N18-1065} {{N}ewsroom: A dataset
of 1.3 million summaries with diverse extractive strategies}.
\newblock In \emph{Proceedings of the 2018 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers)}, pages 708--719, New Orleans,
Louisiana. Association for Computational Linguistics.
\bibitem[{Gunasekara et~al.(2021)Gunasekara, Feigenblat, Sznajder, Aharonov,
and Joshi}]{gunasekara-etal-2021-using-question}
Chulaka Gunasekara, Guy Feigenblat, Benjamin Sznajder, Ranit Aharonov, and
Sachindra Joshi. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-emnlp.47} {Using
question answering rewards to improve abstractive summarization}.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2021}, pages 518--526, Punta Cana, Dominican Republic. Association for
Computational Linguistics.
\bibitem[{Hermann et~al.(2015)Hermann, Kociský, Grefenstette, Espeholt, Kay,
Suleyman, and Blunsom}]{HermannKGEKSB15}
Karl~Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will
Kay, Mustafa Suleyman, and Phil Blunsom. 2015.
\newblock \href
{http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend}
{Teaching machines to read and comprehend}.
\newblock In \emph{NIPS}, pages 1693--1701.
\bibitem[{Kryscinski et~al.(2019)Kryscinski, Keskar, McCann, Xiong, and
Socher}]{kryscinski-etal-2019-neural}
Wojciech Kryscinski, Nitish~Shirish Keskar, Bryan McCann, Caiming Xiong, and
Richard Socher. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1051} {Neural text
summarization: A critical evaluation}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 540--551, Hong Kong,
China. Association for Computational Linguistics.
\bibitem[{Kryscinski et~al.(2020)Kryscinski, McCann, Xiong, and
Socher}]{kryscinski-etal-2020-evaluating}
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.750} {Evaluating
the factual consistency of abstractive text summarization}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 9332--9346, Online. Association
for Computational Linguistics.
\bibitem[{Lacoste et~al.(2019)Lacoste, Luccioni, Schmidt, and
Dandres}]{lacoste2019quantifying}
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres.
2019.
\newblock Quantifying the carbon emissions of machine learning.
\newblock \emph{arXiv preprint arXiv:1910.09700}.
\bibitem[{Ladhak et~al.(2022)Ladhak, Durmus, He, Cardie, and
McKeown}]{ladhak-etal-2022-faithful}
Faisal Ladhak, Esin Durmus, He~He, Claire Cardie, and Kathleen McKeown. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.acl-long.100} {Faithful or
extractive? on mitigating the faithfulness-abstractiveness trade-off in
abstractive summarization}.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1410--1421,
Dublin, Ireland. Association for Computational Linguistics.
\bibitem[{Lewis et~al.(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis-etal-2020-bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.703} {{BART}:
Denoising sequence-to-sequence pre-training for natural language generation,
translation, and comprehension}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7871--7880, Online. Association for
Computational Linguistics.
\bibitem[{Lin(2004)}]{lin-2004-rouge}
Chin-Yew Lin. 2004.
\newblock \href {https://aclanthology.org/W04-1013} {{ROUGE}: A package for
automatic evaluation of summaries}.
\newblock In \emph{Text Summarization Branches Out}, pages 74--81, Barcelona,
Spain. Association for Computational Linguistics.
\bibitem[{Liu et~al.(2022)Liu, Liu, Radev, and Neubig}]{liu-etal-2022-brio}
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.acl-long.207} {{BRIO}:
Bringing order to abstractive summarization}.
\newblock In \emph{Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 2890--2903,
Dublin, Ireland. Association for Computational Linguistics.
\bibitem[{Maynez et~al.(2020)Maynez, Narayan, Bohnet, and
McDonald}]{maynez-etal-2020-faithfulness}
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.173} {On
faithfulness and factuality in abstractive summarization}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 1906--1919, Online. Association for
Computational Linguistics.
\bibitem[{Nan et~al.(2021{\natexlab{a}})Nan, Nallapati, Wang, Nogueira~dos
Santos, Zhu, Zhang, McKeown, and Xiang}]{nan-etal-2021-entity}
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira~dos Santos, Henghui
Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/2021.eacl-main.235} {Entity-level
factual consistency of abstractive text summarization}.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
2727--2733, Online. Association for Computational Linguistics.
\bibitem[{Nan et~al.(2021{\natexlab{b}})Nan, Nogueira~dos Santos, Zhu, Ng,
McKeown, Nallapati, Zhang, Wang, Arnold, and Xiang}]{nan-etal-2021-improving}
Feng Nan, Cicero Nogueira~dos Santos, Henghui Zhu, Patrick Ng, Kathleen
McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew~O. Arnold, and
Bing Xiang. 2021{\natexlab{b}}.
\newblock \href {https://doi.org/10.18653/v1/2021.acl-long.536} {Improving
factual consistency of abstractive summarization via question answering}.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 6881--6894,
Online. Association for Computational Linguistics.
\bibitem[{Narayan et~al.(2018)Narayan, Cohen, and
Lapata}]{narayan-etal-2018-dont}
Shashi Narayan, Shay~B. Cohen, and Mirella Lapata. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-1206} {Don{'}t give me the
details, just the summary! topic-aware convolutional neural networks for
extreme summarization}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 1797--1807, Brussels, Belgium.
Association for Computational Linguistics.
\bibitem[{Pagnoni et~al.(2021)Pagnoni, Balachandran, and
Tsvetkov}]{pagnoni-etal-2021-understanding}
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.383}
{Understanding factuality in abstractive summarization with {FRANK}: A
benchmark for factuality metrics}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 4812--4829, Online. Association for Computational
Linguistics.
\bibitem[{Raffel et~al.(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, Liu et~al.}]{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, Peter~J Liu, et~al. 2020.
\newblock Exploring the limits of transfer learning with a unified text-to-text
transformer.
\newblock \emph{J. Mach. Learn. Res.}, 21(140):1--67.
\bibitem[{Scialom et~al.(2021)Scialom, Dray, Lamprier, Piwowarski, Staiano,
Wang, and Gallinari}]{scialom-etal-2021-questeval}
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo
Staiano, Alex Wang, and Patrick Gallinari. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.emnlp-main.529}
{{Q}uest{E}val: Summarization asks for fact-based evaluation}.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 6594--6604, Online and Punta Cana,
Dominican Republic. Association for Computational Linguistics.
\bibitem[{Tang et~al.(2022)Tang, Goyal, Fabbri, Laban, Xu, Yahvuz,
Kry{\'s}ci{\'n}ski, Rousseau, and Durrett}]{tang2022understanding}
Liyan Tang, Tanya Goyal, Alexander~R Fabbri, Philippe Laban, Jiacheng Xu, Semih
Yahvuz, Wojciech Kry{\'s}ci{\'n}ski, Justin~F Rousseau, and Greg Durrett.
2022.
\newblock Understanding factual errors in summarization: Errors, summarizers,
datasets, error detectors.
\newblock \emph{arXiv preprint arXiv:2205.12854}.
\bibitem[{Vijayakumar et~al.(2018)Vijayakumar, Cogswell, Selvaraju, Sun, Lee,
Crandall, and
Batra}]{Vijayakumar_Cogswell_Selvaraju_Sun_Lee_Crandall_Batra_2018}
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan
Lee, David Crandall, and Dhruv Batra. 2018.
\newblock \href {https://doi.org/10.1609/aaai.v32i1.12340} {Diverse beam search
for improved description of complex scenes}.
\newblock \emph{Proceedings of the AAAI Conference on Artificial Intelligence},
32(1).
\bibitem[{Wan and Bansal(2022)}]{wan-bansal-2022-factpegasus}
David Wan and Mohit Bansal. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.naacl-main.74}
{{F}act{PEGASUS}: Factuality-aware pre-training and fine-tuning for
abstractive summarization}.
\newblock In \emph{Proceedings of the 2022 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 1010--1028, Seattle, United States. Association for
Computational Linguistics.
\bibitem[{Zhang et~al.(2020)Zhang, Zhao, Saleh, and
Liu}]{10.5555/3524938.3525989}
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter~J. Liu. 2020.
\newblock Pegasus: Pre-training with extracted gap-sentences for abstractive
summarization.
\newblock In \emph{Proceedings of the 37th International Conference on Machine
Learning}, ICML'20. JMLR.org.
\bibitem[{Zhang et~al.(2019)Zhang, Kishore, Wu, Weinberger, and
Artzi}]{zhang2019bertscore}
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian~Q Weinberger, and Yoav Artzi.
2019.
\newblock Bertscore: Evaluating text generation with bert.
\newblock \emph{arXiv preprint arXiv:1904.09675}.
\bibitem[{Zhu et~al.(2021)Zhu, Hinthorn, Xu, Zeng, Zeng, Huang, and
Jiang}]{zhu-etal-2021-enhancing}
Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng,
Xuedong Huang, and Meng Jiang. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.naacl-main.58} {Enhancing
factual consistency of abstractive summarization}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 718--733, Online. Association for Computational
Linguistics.
\end{thebibliography}