Where did you get that? Towards Summarization Attribution for Analysts
Extracted source text · arXiv 2511.08589
This evaluation will use the CrisisFACTS data set. The data set consists of content snippets (approximately a sentence in length) from three content streams (Twitter, Reddit, online news sources) related to multiple crises (wildfires, hurricanes, floods). A summary is based on important facts for a day of a crisis event.
\subsection*{Examples}
The upcoming examples are from fictitious content snippets (in the style of news and Twitter) related to crises during \textit{The Wonderful Wizard of Oz}. The examples are based on the original book.
\subsubsection*{Example: Support, Refute=no}
\mysample{
[SUMMARY] A cyclone dropped a house on the Wicked Witch of the East and killed her.\newline
[**EVAL**] Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch
}
The sentence to evaluate for attribution is “Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch”. This sentence provides supporting evidence for a cyclone dropping a house on the Wicked Witch of the East.
The summary statement is not fully supported since no evidence is provided for the Wicked Witch of the East being killed. However, partial support meets the definition of support. You may mark “support” for an answer.
Consider the refute question next. The attribution sentence does not contain contradictory information. The answer is ``refute=no.''
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] A scarecrow was stuck on a pole in the middle of a river after a rafting incident, and was saved by a child, a tin man, a lion, and a small animal of unknown origin.\newline
[**EVAL**] A scarecrow stuck on a pole in the middle of a cornfield was rescued by an unknown good samaritan.
}
The attribution sentence does not provide supporting evidence for a river rescue because it describes a cornfield rescue. This example may be marked “no support”.
Consider the refute question next. While the attribution sentence describes a scarecrow rescue, this example will be marked as ``refute=yes'' because the type of rescue (cornfield versus river) differs.
This example could have also been marked as ``refute=no'' because the rescues differ too much to be related. There will be ambiguous cases like this where it is not clear if an attribution refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another (e.g., Winkies identified as Munchkins), or misinterprets the source document during text generation (e.g., child saves scarecrow misinterpreted as scarecrow saves child). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., cornfield rescue versus river rescue).
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with “refute=no”.
\subsubsection*{Example: No Support, Refute=no}
\mysample{
[SUMMARY] A sleeping lion was rescued from a deadly poppy field by a large group of field mice, aided by a tin man, scarecrow, child, and an animal of unknown origin.\newline
[**EVAL**] WARNING: Large spider approaching. Seek shelter immediately! \#giantspider
}
There will be a topic-based relationship between a statement and an attribution. CrisisFACTS content snippets will be based on the same crises event. The examples in this document are based on crises from a single book, \textit{The Wonderful Wizard of Oz}. However, the connection is too tenuous for this example (lion rescue, spider attack), and it should be marked “no support”.
\subsection*{Examples}
The upcoming examples are from fictitious content snippets (in the style of news and Twitter) related to crises during \textit{The Wonderful Wizard of Oz}. The examples are based on the original book.
\subsubsection*{Example: Full Support, Refute=no}
\mysample{
[SUMMARY] Winged Monkeys are feared due to frightful attacks, and are believed to be aligned with the Wicked Witch of the West.\newline
[**EVAL**] Winged Monkeys drop tin man from great heights to rocks below \#oznews\newline
[**EVAL**] Winged Monkeys tear out straw from scarecrow and leave clothes in tall tree. \#oznews\newline
[**EVAL**] Winged Monkeys kidnap child and small animal of unknown origin, imprison them in castle of Wicked Witch of West. \#oznews
}
You should consider all 3 sentences as part of the attribution for this task. The sentences in combination provide supporting evidence for multiple attacks by Winged Monkeys. The third sentence implies an affiliation between Winged Monkeys and the Wicked Witch of the West. The sentences fully support a statement that Winged Monkeys conduct fearful attacks and are aligned with the Wicked Witch of the West. This example can be marked “full support”.
Consider the refute question next. The three sentences support the summary statement and do not contain contradictory information. The answer is “refute=no”.
\subsubsection*{Example: Full support, Refute=no}
\mysample{
[SUMMARY] A cyclone dropped a house on the Wicked Witch of the East and killed her.\newline
[**EVAL**] The Wicked Witch of the East died due to injuries from a home dropped by a cyclone.\newline
[**EVAL**] Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch\newline
[**EVAL**] WARNING: tin man with sharp axe in area, attacked Tree Border Patrol. Considered very dangerous. Do not approach! \#livingforest
}
The first sentence fully supports the summary statement. The second sentence partially supports the statement. The third sentence is not related to this accident. While the third sentence does not provide supporting evidence, the three sentences when viewed as a group provide full support, and this example can be marked “full support”.
Consider the refute question next. The first two sentences support the summary statement and do not contain contradictory information. The third sentence is irrelevant. An irrelevant sentence is not a refutation. The answer is “refute=no”.
\subsubsection*{Example: Partial support, Refute=no}
\mysample{
[SUMMARY] The Wicked Witch of the West is aligned with Wolves, Crows, and Munchkins, who further her political agenda through acts which induce fear.\newline
[**EVAL**] Oz News Exclusive: Wicked Witch of West pays Wolves, Crows, and Black Bees to terrify populace. \#oznews\newline
[**EVAL**] Wolves attack and kill protesters outside of Wicked Witch of the West’s castle \#oznews\newline
[**EVAL**] WARNING: Crows approach. Seek shelter immediately! \#oznews \#landofthewest
}
The first two sentences provide supporting evidence that Wolves and Crows are aligned with the Wicked Witch of the West and use fear-based methods on her behalf. The third statement provides additional information that crows are viewed as dangerous in the Land of the West. None of the sentences support an alliance with Munchkins. Mark this example as “partial support” since the sentences partially support the summary.
The detail about Munchkins is a mistake in the summary. Munchkins are a peaceful people living in a region controlled by the Wicked Witch of the East, and are not agents of terror. Full support will not be possible due to the mistake in the summary.
Consider the “refute” question next. None of the attribution sentences contain information that refute the summary statement. The answer may be marked “refute=no”.
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] The King of the Winged Monkeys thanked the Good Witch of the South for freeing them from magical cap enslavement, and declared that they would return to the Great Forest.\newline
[**EVAL**] Dorothy thanked the Good Witch of the South for telling her about the magical silver shoes and used them to return to Kansas.
[**EVAL**] China cow breaks leg after startled by giants. \#oznews \#daintychina\newline
[**EVAL**] Insure your home with Oz Insurance. \#ozinsurance
}
The 3 attribution sentences do not provide supporting evidence for the summary statement about Winged Monkeys. This example may be marked “no support”.
Consider the refute question next. The summary statement and first attribution sentence are about two different cases where the Good Witch was thanked for magic-related repatriations (Winged Monkeys/magical cap versus Dororthy/magical shoes). This sentence will be treated as a refutation because it is related but different. The last two sentences are irrelevant and are not viewed as refutations.
Since there is one attribution sentence that refutes the summary statement, the question may be marked “refute=yes”.
This example could have also been marked as “refute=no” if you decided that the first sentence differed too much from the summary statement. There will be ambiguous cases like this where it is not clear if an attribution sentence is a refutation or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the
purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another (e.g., Winkies identified as Munchkins), or misinterprets the source document during text generation (e.g., Dorothy thanks Good Witch misinterpreted as Good Witch thanks Dorothy). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., magical cap versus magical shoes).
Another type of misfire is when the attribution method has low accuracy and
picks an irrelevant sentence. We would like to capture this with “refute=no”
\section*{Questions or Feedback?}
If you have questions about the task, or any feedback about how we could make it better or what your experience was like with it, please contact us on Slack using the channel \texttt{\#inf-attribution-evaluation} and we'll get back to you. Thanks!
\section*{Overview}
We are a team interested in evaluating attribution for summarization (how to link automatically generated summaries back to their source material.)
In this project, you will evaluate the quality of attributions for summaries that have been created by humans or by automatic methods that mimic what humans would write. We would like to evaluate if an attribution method can provide accurate attributions, and if the accuracy is influenced by the type of automatic summarization method used.
At a high level, the project breaks down into two tasks:
\begin{enumerate}
\item Given a statement in a summary, judge the quality of a single attribution.
\item Given a statement in a summary, judge whether the statement is fully supported by three (ranked) attributions.
\end{enumerate}
Examples of the task you have chosen will be provided in this document.
\section*{Preliminaries}
\textcolor{red}{@aaron - THIS SECTION IS FOR YOU TO PROVIDE INFINITYPOOL GUIDANCE}
\section*{Task 1: Judge the quality of a single attribution}
In this part, you will be provided a statement from the summary, and will be asked to judge the quality of a corresponding attribution in the source material. The statement will be a single sentence from the summary. The attribution will be a single (1) sentence from the source document, along with surrounding context sentences (sentence before and sentence after the attribution sentence in the source document, if available).
Carefully read the summary statement and the attribution sentence with the surrounding context. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding attribution in the source document that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Support}: Some or all of the information in the summary statement is supported by the attribution sentence.
\item \textbf{No Support}: The attribution sentence does not support the summary statement.
\item \textbf{Unclear (Can’t Make Judgement)}: The information provided does not make it possible to determine whether the attribution sentence supports or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is ``yes'' or ``no'' to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (``yes'') with ``support'' or ``no support''. Answer ``no'' if the information is unclear.
\end{itemize}
\section*{Task 1: Judging the quality of a single attribution}
In this part, you will be provided a statement from the summary, and will be asked to judge the quality of a corresponding attribution in the source material. The statement will be a single sentence from the summary. The attribution will be a single (1) sentence from the source document.
Carefully read the summary statement and the attribution sentence with the surrounding context. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding attribution in the source document that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Support}: Some or all of the information in the summary statement is supported by the attribution sentence.
\item \textbf{No Support}: The attribution sentence does not support the summary statement.
\item \textbf{Unclear (Can’t Make Judgement)}: The information provided does not make it possible to determine whether the attribution sentence supports or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is ``yes'' or ``no'' to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (“yes”) with “support” or “no support”. Answer “no” if the information is unclear.
\end{itemize}
\section*{Task 2: Judge the quality of a group of attributions}
In this part, you will be provided a statement from the summary, and will be asked to judge the amount of support by the attribution. The statement will be a single sentence from the summary. The attribution will be three (3) sentences from the source material.
Carefully read the summary statement and the 3 attribution sentences. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding 3 attribution sentences from the source data that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Full Support}: All of the information in the summary statement is supported by the attribution sentences.
\item \textbf{Partial Support}: Only some of the information in the summary statement is supported, but other parts of the information are missing.
\item \textbf{No Support}: The attribution sentences do not support the summary statement.
\item \textbf{Unclear (Can't Make Judgment)}: The information provided does not make it possible to determine whether the attribution sentence fully supports, partially supports, or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is “yes” or “no” to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (“yes”) with “full support”, “partial support”, or “no support”. Answer “no” if the information is unclear.
\end{itemize}
This evaluation will use the TAC 2011 (Text Analysis Conference) data set for multi-document summarization. The data set contains news articles grouped by topic (10 documents per topic). A summary is based on a single topic (10 documents).
\subsection*{Examples}
For TAC 2011, a summary is based on 10 documents for a single topic. The upcoming examples are from a single topic on threats to sea turtles.
\subsubsection*{Example: Support, Refute=no}
\mysample{
[SUMMARY] Sea turtles are threatened by increased poaching, habitat destruction for land development, illegal fishing, and pollution.\newline
[CONTEXT] The fishermen do not properly release the turtles and often kill them, leaving them to wash ashore. \newline
[**EVAL**] Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists. \newline
[CONTEXT] Mohammad Aminul Islam, the top administrator for the area, ordered local officials to teach people, from fishermen to tourists, to change their behavior.
}
The sentence to evaluate for attribution is “Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists.” This sentence provides supporting evidence that sea turtles are threatened by pollution. While the first context sentence supports illegal fishing, the decision for the evaluation should only be based on the sentence with the {\fontfamily{lmss}\selectfont[**EVAL**]} (evaluate) prefix.
The summary statement is not fully supported since no supporting evidence is provided for poaching, habitat destruction for land development, and illegal fishing by the attribution sentence. However, partial support meets the definition of support. You may mark “support” for an answer.
Consider the refute question next. The attribution sentence supports the summary statement and also contains supplemental information. It does not contain contradictory information. The answer is ``refute=no''.
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] Malaysia has arrested a number of poachers, many from south China, for poaching turtles and other animals.\newline
[CONTEXT] At that price, a single nest containing about 100 eggs nets a poacher 1,000 pesos. Compare that, he says, to Mexico's minimum daily wage of 45 pesos. \newline
[**EVAL**] The Mexican government takes poaching seriously, he continued, assessing steep fines or jail time--``it's a worse offense than drugs''--but it lacks the manpower to prevent the crime in the first place or prosecute offenders in the second. \newline
[CONTEXT] Rapid development endangers the turtles.
}
The attribution sentence is the second of three consecutive sentences. This sentence does not provide supporting evidence for poaching in Malaysia because the attribution is about poaching in Mexico which is a different country. (The difference is due to the summary being based on a different news article than the attribution.) You may mark ``no support.''
Consider the refute question next. While the attribution sentence supports turtle poaching having legal repercussions, this example will be marked as “refute=yes” because the country differs.
This example could have also been marked as “refute=no” because the statement and attribution sentence are about different countries. There will be ambiguous cases like this where it is not clear if an attribution refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide between “refute=yes” and “refute=no”, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality for another, or misinterprets the source document during text generation (e.g., dog bites man misinterpreted as man bites dog). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., Mexican poaching instead of Malaysian poaching).
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with the “refute=no”.
\subsubsection*{Example: No Support, Refute=no}
\mysample{
[SUMMARY] 90 sea turtles were rescued in Mexico when they became comatose from a cold snap.\newline
[CONTEXT] Some sea turtle species eat jellyfish like they are going out of style.\newline
[**EVAL**] If the turtles weren't around to eat the jellyfish, the jellyfish population would explode.\newline
[CONTEXT] The hungry jellyfish hordes would gorge on zooplankton.
}
The attribution sentence is “If the turtles weren't around to eat the jellyfish, the jellyfish population would explode.” This sentence does not provide supporting evidence for a turtle rescue. You may mark ``no support.''
Consider the refute question. There will be a topic-based relationship between a statement and an attribution with TAC 2011 since the summary is based on news articles from the same topic (e.g., threats to sea turtles). However, the connection is too tenuous for this example, and it should be marked “refute=no”.
\subsection*{Examples}
For TAC 2011, a summary is based on 10 documents for a single topic. The upcoming examples are from a single topic on threats to sea turtles.
\subsubsection*{Example: Full support, Refute=no}
\mysample{
[SUMMARY] Some countries have passed laws to protect the turtles.\newline
[**EVAL**] The Mexican government takes poaching seriously, he continued, assessing steep fines or jail time--``it's a worse offense than drugs''--but it lacks the manpower to prevent the crime in the first place or prosecute offenders in the second.\newline
[**EVAL**] A Malaysian court Friday jailed a group of 13 Chinese fishermen for 18 months each for poaching nearly 80 protected turtles.\newline
[**EVAL**] Ahmed would not give any specific reason for the spike in deaths, but said the use of illegal fishing nets near the shoreline has apparently increased recently.
}
%You should consider all 3 sentences as part of the attribution for Task 2 of the evaluation.
The first sentence describes a poaching law in Mexico. Malaysian law can be inferred by the second sentence. The third sentence describes illegal fishing nets but does not specify the country (Bangladesh). Combined, the sentences are sufficient to fully support a statement that more than one country has laws to protect turtles, and the example can be marked ``full support''.
Consider the refute question next. Each sentence supports the summary statement and does not contain contradictory information. The answer is ``refute=no''.
\subsubsection*{Full support, Refute=no}
\mysample{
[SUMMARY] 90 sea turtles were rescued in Mexico when they became comatose from a cold snap.\newline
[**EVAL**] Nearly 90 sea turtles rescued after a cold snap left them comatose have been returned to the Gulf of Mexico.\newline
[**EVAL**] Dozens of rehabilitated sea turtles, threatened by cold, returned to Gulf of Mexico.\newline
[**EVAL**] Green turtles are born in southern Mexico and spend their early years feeding on turtle grass in shallow areas such as the Laguna Madre.
}
The first sentence fully supports the summary statement. The second sentence partially supports the statement. The third sentence is about turtles in Mexico but not a rescue. %While the third sentence does not provide supporting evidence, the three sentences when viewed as a group provide full support, and this example can be marked “full support.”
If one or two sentences fully support the summary statement, the question may be marked ``full support''. Support is not required from all 3 sentences.
Consider the refute question next. The first two sentences support the summary sentence and do not contain contradictory information. The third sentence is about Mexican turtles but is about a different topic (rescue versus early life). The connection is tenuous and will not be viewed as a refutation. Since none of the sentences contain a refutation, the answer to the question is “refute=no”.
This example could have also been marked as “refute=yes” because the third sentence is about Mexican turtles so there is a connection between that sentence and the summary statement, but the facts are different. There will be ambiguous cases like this where it is not clear if an attribution sentence refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another, or misinterprets the source document during text generation (e.g., dog bites man misinterpreted as man bites dog). Or when an attribution method provides a link which is related to the summary statement but misses the mark.
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with ``refute=no''.
\subsubsection*{Full support, Refute=yes}
\mysample{
[SUMMARY] 17 Chinese nationals found with 274 protected turtles, most of them dead, were recently arrested in Malaysian waters.\newline
[**EVAL**] The poachers were caught by Malaysian marine police on March 28 in waters off the coast of eastern Sabah state with a catch of 274 protected Hawksbill and Green turtles, only 32 of which were still alive.\newline
[**EVAL**] They were the second group of fishermen from China's Hainan island jailed this week for turtle poaching off the waters of Malaysia's eastern Sabah state, with 17 men sentenced on Wednesday for catching 274 of the creatures.\newline
[**EVAL**] A Malaysian court Friday jailed a group of 13 Chinese fishermen for 18 months each for poaching nearly 80 protected turtles.
}
The first sentence provides supporting evidence for poachers being caught with 274 protected turtles, most of them dead. The second sentence provides support for the arrest of 17 Chinese nationals with 274 turtles. The third sentence is about a different arrest.
The first two sentences provide full support for the summary statement, and this example may be marked ``full support''.
Consider the refute question next. The third sentence is on the same subject: a Malaysian arrest of Chinese nationals for poaching protected turtles. However, it is about a different arrest. This is an example of a misfire where a sentence is related but misses the mark, and can be viewed as an instance of ``refute=yes''.
\subsubsection*{Partial support, Refute=no}
\mysample{
[SUMMARY] Sea turtles are threatened by increased poaching, habitat destruction for land development, illegal fishing, and pollution.\newline
[**EVAL**] The turtles--which can be found in other parts of Asia along the Andaman Coast and the South China Sea--have seen their numbers reduced drastically in recent years mostly due to poaching of their eggs, pollution and habitat loss.\newline
[**EVAL**] Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists.\newline
[**EVAL**] Sea turtles face a host of threats around the world.
}
The three sentences provide supporting evidence for a threat to sea turtles for several reasons but do not support illegal fishing. This example may be marked “partial support”.
None of the sentences refute the summary sentence. The answer is “refute=no”.
\subsubsection*{Example: No support, Refute=no}
\mysample{
[SUMMARY] Scientists say that the loss of sea turtles could have unforeseen, even catastrophic, consequences.\newline
[**EVAL**] Efforts in Mexico to protect sea turtles are a patchwork of official and unofficial endeavors, according to Sea Turtle, Inc., a nonprofit based in South Padre Island.\newline
[**EVAL**] ``We are trying to mobilize resources to make a bigger plan to save the sea turtles in the future.''\newline
[**EVAL**] Thai villagers have caught a river terrapin turtle that was thought to be extinct in the country.
}
Sentences will be related to a common topic (threats to sea turtles). However, the 3 attribution sentences do not support the summary statement. This example may be marked “no support”.
Consider the refute question next. Each attribution sentence is irrelevant to the summary statement. An irrelevant sentence is not a refutation. None of the sentences refute the summary statement and the question may be marked “refute=no”.
\newcounter{question}
\setcounter{question}{0}
\newcommand{\question}[1]{\item[Q\refstepcounter{question}\thequestion.] #1}
\newcommand{\answer}[1]{\item[A\thequestion.] #1}
\subsubsection*{FAQ}
\begin{itemize}
\question\textit{How do we mark a group of attribution sentences if one of the sentences fully supports the summary statement, no matter what the rest say? Do we mark this as ``full support'' or ``partial support''?}
\answer If one attribution sentence fully supports the summary statement, then mark the group as ``full support'' for the first question.
\question\textit{In InfinityPool (the annotator platform), choices were not pre-selected for the first question about support. However, an answer was already highlighted for the second refute question before I answered it. Should this happen?}
\answer The refute questions all default to ``no''. Change an answer to ``yes'' (when applicable).
\end{itemize}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{CrisisFACTS}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/FACTS/overview-data}
%\input{Shared/preliminaries}
\input{AnnotationGuidelines/Shared/task1-nocontext}
\input{AnnotationGuidelines/FACTS/task1-examples}
%\input{Shared/task2}
%\input{FACTS/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{CrisisFACTS}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/FACTS/overview-data}
%\input{Shared/preliminaries}
%\input{Shared/task1-nocontext}
%\input{FACTS/task1-examples}
\input{AnnotationGuidelines/Shared/task2}
\input{AnnotationGuidelines/FACTS/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%\title{Annotation Guidelines}
%\author{TAC 2011}
%\date{}
\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/TAC2011/overview-data}
%\input{Shared/preliminaries}
\input{AnnotationGuidelines/Shared/task1-context}
\input{AnnotationGuidelines/TAC2011/task1-examples}
%\input{Shared/task2}
%\input{TAC2011/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{TAC 2011}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/TAC2011/overview-data}
%\input{Shared/preliminaries}
%\input{Shared/task1-context}
%\input{TAC2011/task1-examples}
\input{AnnotationGuidelines/Shared/task2}
\input{AnnotationGuidelines/TAC2011/task2-faq}
\input{AnnotationGuidelines/TAC2011/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%\DeclareOption{}{\gdef\@journal{}\gdef\@journalfull{}\gdef\@journalshort{}\gdef\@doiabbr{}\gdef\@ISSN{}\gdef\@continuouspages{yes}}
\DeclareOption{scads}
{\gdef\@journal{scads}\gdef\@journalfull{Summer Conference on Applied Data Science (SCADS)}\gdef\@journalshort{SCADS}\gdef\@continuouspages{yes}}
% LaTeX support: latex@mdpi.com
% For support, please attach all files needed for compiling as well as the log file, and specify your operating system, LaTeX version, and LaTeX editor.
%=================================================================
\documentclass[scads,technicalreport,accept,pdftex,moreauthors]{Definitions/scads}
%--------------------
% Class Options:
%--------------------
% For posting an early version of this manuscript as a preprint, you may use "preprints" as the journal. Changing "submit" to "accept" before posting will remove line numbers.
%---------
% Manuscript Type
%---------
% The default type of manuscript is "technicalreport", but can be replaced by:
% abstract, addendum, article, book, bookreview, briefreport, casereport, comment, commentary, communication, conferenceproceedings, correction, conferencereport, entry, expressionofconcern, extendedabstract, datadescriptor, editorial, essay, erratum, hypothesis, interestingimage, obituary, opinion, projectreport, reply, retraction, review, perspective, protocol, shortnote, studyprotocol, systematicreview, supfile, technicalnote, technicalreport, viewpoint, guidelines, registeredreport, tutorial
% supfile = supplementary materials
%----------
% submit
%----------
% The class option "submit" will be changed to "accept" by the Editorial Office when the paper is accepted. This will only make changes to the frontpage (e.g., the logo of the journal will get visible), the headings, and the copyright information. Also, line numbering will be removed. Journal info and pagination for accepted papers will also be assigned by the Editorial Office.
%------------------
% moreauthors
%------------------
% If there is only one author the class option oneauthor should be used. Otherwise use the class option moreauthors.
%---------
% pdftex
%---------
% The option pdftex is for use with pdfLaTeX. Remove "pdftex" for (1) compiling with LaTeX & dvi2pdf (if eps figures are used) or for (2) compiling with XeLaTeX.
%=================================================================
% MDPI internal commands - do not modify
\firstpage{1}
\makeatletter
\setcounter{page}{\@firstpage}
\makeatother
\pubvolume{1}
\issuenum{1}
\articlenumber{0}
\pubyear{2023}
\copyrightyear{2023}
%\externaleditor{Academic Editor: Firstname Lastname}
\datereceived{ }
\daterevised{ } % Comment out if no revised date
\dateaccepted{ }
\datepublished{ }
%\datecorrected{} % For corrected papers: "Corrected: XXX" date in the original paper.
%\dateretracted{} % For corrected papers: "Retracted: XXX" date in the original paper.
\hreflink{https://doi.org/} % If needed use \linebreak
%\doinum{}
%\pdfoutput=1 % Uncommented for upload to arXiv.org
%=================================================================
% Add packages and commands here. The following packages are loaded in our class file: fontenc, inputenc, calc, indentfirst, fancyhdr, graphicx, epstopdf, lastpage, ifthen, float, amsmath, amssymb, lineno, setspace, enumitem, mathpazo, booktabs, titlesec, etoolbox, tabto, xcolor, colortbl, soul, multirow, microtype, tikz, totcount, changepage, attrib, upgreek, array, tabularx, pbox, ragged2e, tocloft, marginnote, marginfix, enotez, amsthm, natbib, hyperref, cleveref, scrextend, url, geometry, newfloat, caption, draftwatermark, seqsplit
% cleveref: load \crefname definitions after \begin{document}
\usepackage{ulem}
\usepackage{nopageno}
%\usepackage{draftwatermark}
\usepackage{datetime}
\mmddyydate
%\SetWatermarkLightness{0.9}
%\SetWatermarkText{\parbox{6em}
%{\centering{DRAFT\\\today}}}
%\SetWatermarkScale{0.7}
%=================================================================
% Please use the following mathematics environments: Theorem, Lemma, Corollary, Proposition, Characterization, Property, Problem, Example, ExamplesandDefinitions, Hypothesis, Remark, Definition, Notation, Assumption
%% For proofs, please use the proof environment (the amsthm package is loaded by the MDPI class).
\usepackage[indentfirst=false]{quoting}
\newenvironment{myquotation}{
\begin{quoting}
}
{
\end{quoting}
}
\def\gray#1{\textcolor{gray}{\textit{#1}}}
%=================================================================
% Full title of the paper (Capitalized)
\Title{Where did you get that? \\Towards Summarization Attribution for Analysts\footnote{An earlier version of this report appeared in the \textit{SCADS 2023 Proceedings}.}}
% MDPI internal command: Title for citation in the left column
\TitleCitation{Title}
% Author Orchid ID: enter ID or remove command
\newcommand{\orcidauthorA}{0000-0000-0000-000X} % Add \orcidA{} behind the author's name
%\newcommand{\orcidauthorB}{0000-0000-0000-000X} % Add \orcidB{} behind the author's name
% Authors, for the paper (add full first names)
\Author{Violet B, John M. Conroy$^{1}$, Sean Lynch$^{2}$, Danielle M, Neil P. Molino$^{1}$, Aaron Wiechmann$^{2}$, and Julia S. Yang$^{1}$}
%\longauthorlist{yes}
% MDPI internal command: Authors, for metadata in PDF
\AuthorNames{Violet B, John M. Conroy, Sean Lynch, Danielle M, Neil P. Molino, Aaron Wiechmann, and Julia S. Yang}
% MDPI internal command: Authors, for citation in the left column
\AuthorCitation{Lastname, F.; Lastname, F.; Lastname, F.}
% If this is a Chicago style journal: Lastname, Firstname, Firstname Lastname, and Firstname Lastname.
% Affiliations / Addresses (Add [1] after \address if there is only one affiliation.)
\address{%
$^{1}$ \quad IDA Center for Computing Sciences, Bowie, MD, USA; \{conroy,npmolin,jsyang\}@super.org\\
$^{2}$ \quad Laboratory for Analytic Sciences, Raleigh, NC; \{sclynch,awiechm\}@ncsu.edu
}
% Contact information of the corresponding author
%\corres{Correspondence: e-mail@e-mail.com; Tel.: (optional; include country code; if there are multiple corresponding authors, add author initials) +xx-xxxx-xxx-xxxx (F.L.)}
% Current address and/or shared authorship
%\firstnote{Current address: Affiliation 3.}
%\secondnote{These authors contributed equally to this work.}
% The commands \thirdnote{} till \eighthnote{} are available for further notes
%\simplesumm{} % Simple summary
%\conference{} % An extended version of a conference paper
% Abstract (Do not insert blank lines, i.e. \\)
\abstract{
Analysts require attribution, as nothing can be reported without knowing the source of the information. In this paper, we will focus on automatic methods for attribution, linking each sentence in the summary to a portion of the source text, which may be in one or more documents. We explore using a hybrid summarization, i.e., an automatic paraphrase of an extractive summary, to ease attribution. We also use a custom topology to identify the proportion of different categories of attribution-related errors.}
% Keywords
%\keyword{keyword 1; keyword 2; keyword 3 (List three to ten pertinent keywords specific to the article; yet reasonably common within the subject discipline.)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
Automatic text summarization methods have been studied for over sixty-five years \cite{10.1147/rd.22.0159} and have shown promise as a content selection and filtering tool. Evaluations from the National Institute of Standards (NIST), such as the Document Understanding Conferences (DUC) and Text Analysis Conferences (TAC), help scope the problem and advance techniques in text summarization. The methods developed in the time frame of the conferences 2001-2011 were largely extractive-based methods, i.e., algorithms for finding critical portions of one or more documents, typically sentences, which best served the user's need for a summary.
In the last six years, significant progress in abstractive summarization has been made with the advent of recurrent neural networks for abstractive summarization \cite{journals/corr/SeeLM17}, and then transformer models. The transformer model can encode or decode text to generate summaries \cite{liu-lapata-2019-text} \cite{lewis-etal-2020-bart}. The limits of transfer learning to perform multiple natural language processing (NLP) tasks were extended by \cite{T5:10.5555/3455716.3455856} to tackle dozens of tasks. OpenAI extended such approaches \cite{chatgpt}, and many research laboratories produce fluent text and multi-modal models (including images, speech, etc.)
We focus on automatic methods for attribution, i.e., linking each sentence in the summary to a portion of the source text, which may be in one or more documents.
Given that large language models too often \emph{hallucinate}, i.e., make statements that are either not supported by the text or even contradicted by the text, the problem of automatic attribution methods is particularly daunting.
Key to our approach is a fast extractive summarization method, \texttt{occams}~\cite{analytics2030030}, which produces extracts with strong content coverage. The \texttt{occams} summary, typically less than 1K tokens, is then paraphrased with an abstractive summarizer. In contrast to this \emph{hybrid} approach, the abstractive-only method requires applying the neural abstractive model to the entire set of input documents, increasing the computational cost and complicating the attribution problem.
Automatic methods can link summary sentences with likely attributing sentences in the source documents. To test the accuracy of these methods, we employ human evaluation. For each sentence in the abstract, one or more putative supporting sentences are presented to a human to judge if these sentences support, partially support, contradict, partially contradict, or are neutral.
To test if the hybrid approach to summarization makes the problem of attribution easier, we compare it to an abstractive method on three datasets representing a range of information tasks for an analyst. We describe these in section \ref{sec:data}.
We note that the recent paper ``Attribute First, then Generate'' by \citet{slobodkin-etal-2024-attribute} takes an alternate approach of identifying relevant source segments before conditioning the generation process. This approach like our hybrid improves the citation accuracy. Also, \citet{worledge2024extractive} introduce the extractive-abstractive spectrum, with search engines and large language models as extremes with multiple intermediate approaches. The authors found ``...we find that perceived utility improves by as much as 200\%, while the proportion of properly cited sentences decreases by as much as 50\% and users take up to 3 times as long to verify cited information.''
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Summarization Algorithms}
In this experiment, we evaluate two automatic summary methods: abstractive and hybrid summarization. While both ways utilize GPT-3.5 Turbo (GPT in the rest of this paper), we use different inputs and prompts to change how the summaries are generated. The abstractive method will use GPT on the entire document to generate the final summary, relying on how the model itself was trained to handle summarizing documents. The hybrid approach will use \texttt{occams} to extract the most important text from the document. Then GPT will be used to rewrite the resulting paragraph into a more human-readable final summary. This section will outline how we execute each of these methods.
\subsection{Abstractive Summarization}
To generate our abstractive summaries, we will provide basic instructions for GPT. This experiment compares a hybrid summary with a purely abstractive summary, so we want to allow GPT to behave according to its training. While some recent research has investigated the impact of different prompting styles on the quality and factuality of summaries generated by large language models, we are seeking to evaluate the summaries that the model generates without refinement.
With these goals in mind, we used the following prompt to generate the abstractive summaries for all three datasets:
{\fontfamily{qcr}\selectfont
\noindent``````
\noindent You are an abstractive summarizer that follows the output pattern:
\noindent Text:
\noindent \{text\}
\noindent Summary:
\noindent ''''''
}
We specify an output pattern to ensure no extraneous references to the source text or other words beyond the expected summary. Also, it is essential to note that we do not constrict the length of the abstractive summary, allowing GPT to make it as long as it would by default. With that, we do generate these summaries early in the pipeline because the lengths of these summaries will influence the length we specify to \texttt{occams}.
One consideration we made when generating these summaries is the token length of our input. To prevent over-complicating our pipeline, we sought to abstractively summarize each document all at once in a single API call to GPT. While this was possible for the DUC/TAC and Cyber Threat Intelligence datasets, given the length of their documents, CrisisFACTS had some events with too many documents to process all at once. To reduce the number of documents for each event to a number that can be passed to GPT, we filtered the documents by their provided importance score and removed duplicates. This way, we could ensure that we had enough tokens to process the entire summary without sacrificing the most essential information.
\subsection{Hybrid Summarization}
We generate the hybrid summaries for each document via a two-step process, with the first step being carried out using the \texttt{occams} package while the second is a rewrite using GPT.
\subsubsection{OCCAMS Extractive Summarization}
Unlike GPT, \texttt{occams} requires a word or character budget to generate summaries. We need to use a different character budget for our experiment depending on the dataset we are working with. This is because we want to be able to compare the hybrid and abstractive summaries, and length is a major contributing factor when determining how much of the information from the source text can be represented within the summary.
We will determine this character budget for the DUC/TAC and Cyber Threat Intelligence datasets based on the abstractive summaries from GPT. Because these summaries are of varying lengths, we decided to set the budget at the 75th percentile of the length of all abstractive summaries for each dataset. We decided to go for this instead of the median length because \texttt{occams} is very rigid with its budget and will not exceed it. It will get very close to this target length for the summary, with minimal variation. There is a lot of variation within the abstractive summary lengths, and we must ensure that outliers will not impact the length we choose for our extractive summaries.
For CrisisFACTS, the nature of the dataset does not require as much consideration for the length of the summary. Given that the discrete documents in the dataset tend to be much shorter, we expect that our extractive summary will pull individual documents to construct our fact summary. Additionally, our gold-standard summaries are already extractive and represent an ideal extractive summary. Consequently, we use the median summary from the facts to generate our extractive summaries.
Upon determining the character budget, we can generate the summaries for each dataset using \texttt{occams}. This will be directly used in the next step of our pipeline to determine our final hybrid summary.
\subsubsection{GPT Summarization}
We will use GPT to rewrite our abstractive summaries for the second step in our hybrid summarizer. Given that the advantages of abstractive summarization over extractive summarization are mostly related to readability and human preference, we decided to use a different prompt than that used for abstractive summaries. This is because we seek to rewrite our extractive summaries, generating a more cohesive paragraph rather than a collection of sentences. With that, we want to make sure that no additional information is being added or hallucinated by GPT. With this goal, we used the following prompt:
{\fontfamily{qcr}\selectfont
\noindent``````
\noindent Please rewrite the following into a coherent and readable paragraph. Do not deviate from the facts of these sentences or add any new information. Follow the output pattern:
\noindent Text:
\noindent \{text\}
\noindent Summary:
\noindent ''''''
}
Like the abstractive prompt, we provide an output format to ensure the generated summary has no extraneous words or sentences. We specify that no information should be added. It is also important to note that we are still not concerned with length and do not expect the final summary to be shorter than the extractive summary. The output returned by this prompt will be our final hybrid summary used for our evaluation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Three Exemplar Datasets}
\label{sec:data}
To sample a range of content, we performed our attribution study on three open datasets, which mimic specific analysts' needs. First, the CrisisFACTS data set\footnote{CrisisFACTS (https://crisisfacts.github.io) is a track at the NIST Text Retrieval Conference (TREC) meant to mimic the needs of crisis managers from the Federal Emergency Management Agency as they monitor an ongoing disaster, such as a fire or flood.} contains extracted content snippets (from Twitter, Reddit, and news articles) that are relevant to the current query. Second, a Cyber Threat Intelligence data set prepared by Elemendar for the Laboratory for Analytic Sciences (LAS) represents a specialist requirement, in this case, analysts who keep up with cyber threat activity. Third, the Text Analysis Conference (TAC) 2011 data set is a multi-document summarization data set of newswire documents where the documents are grouped by topic.
The CrisisFACTS dataset from the Text Retrieval Evaluation Conference (TREC) 2022 and 2023 is a good model of a task-based summary where an analyst, in this case, one working for FEMA, has a requirement to write a report on an ongoing disaster such as a fire or flood. For the 2022 dataset, there are eight events. Multiple report requests were made for these events, with the total number of reports (summaries) being 55. As some of the summaries and retrieved relevant documents were small, we filtered the EventID to remove those with too few facts or too many.\footnote{In particular, we filtered the data so that min\_sum\_len=200, max\_sum\_len=5000, min\_doc\_len=100, where length is measured in characters.} An EventID labels each report request. For each EventID, there is a list of facts, loosely an extracted summary for the period of the given EventID. Note there is an abstractive summary for each of the eight events from Wikipedia. We opted not to use these in our study.
The Cyber Threat Intelligence dataset was explicitly created for SCADS by Elemendar. These data contain reports on cyber events. Each document includes a summary. The document set consists of 59 training documents and 15 tests. We will sample the training data for our initial study. In addition to the documents, there are knowledge graphs and some sample questions and answers, which could be used for future content evaluation and fact-checking.
The Text Analysis Conference (TAC) 2011 dataset is a classic multi-document summarization task by the National Institute of Technology and Standards (NIST). The task of interest from 2011 is the guided summary task, which was to perform a multi-document summary of 10 documents on a given topic. The dataset has 45 such topics and an updated summary of each topic. For each topic, there are four human-written summaries.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Automatic Attribution}
SummaC is by no means perfect for attribution, as experimentation indicates the metric has an underlying bias toward extractive summaries. During most of our experiments where we computed scores for abstractive, hybrid, and extractive summaries, SummaC consistently scored extractive summaries higher than any other type. We believe this is due to the Natural Language Inference (NLI) used initially in the metric pipeline to score the entailments of document-summary sentence pairs, as these probabilities are run through the model to output the score.
After performing a document reduction experiment on every summary type where we ordered the document sentences by ascending neutrality, reduced the original document by removing the most neutral sentence, and recomputing the SummaC score, we found that only a few document sentences influenced the SummaC score for extractive summaries. Notably, the exact number of document sentences matching the total summary sentences contributed most heavily to the score, implying that source text facts are not evenly distributed among extractive summary sentences. On the other hand, for abstractive and hybrid summaries, more document sentences were relevant to the score. This indicated that source text facts are more evenly distributed for hybrid and abstractive. Because of this even distribution, no single summary sentence is highly entailed by a document sentence, so the entailment probabilities are typically low. By extension, the SummaC score is very low, but this output doesn’t necessarily mean the abstractive or hybrid summary is bad. Instead, SummaC is bad at capturing summary text's abstract nature. As a result, using solely SummaC as the attribution solution for an entire summary would cause potential unwanted disqualification of summaries from being marked as consistent with the original document.
For this human evaluation experiment, we propose using NLI for sentence-level attribution as opposed to the SummaC metric for the entire summary. Because NLI produces low entailment probabilities for abstractive and hybrid summary sentences, we need to account for any missing context for a single abstracted summary sentence. To do this, we combine a targeted document sentence with a previous and following sentence to provide a range of context. Then we present this three-sentence text to our human evaluators for attribution labeling.
To test the ability of automatic methods for attribution, we use a natural language inference (NLI) model to find good matching sentences for each sentence in the GPT and Hybrid summaries. Following \cite{laban-etal-2022-summac}, we use the NLI \texttt{roberta-large-mnli} model. We also use a T5 sentence embedding cosine similarity for a baseline, with the \texttt{sentence-t5-xxl} model.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Human Labeling Experimental Design}
\label{sec:experiment}
Our evaluation methods are motivated by a recent paper \cite{liu2023evaluating}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}
The three summary types are human-written and two machine-generated summaries.
The machine-generated methods are purely abstractive using OpenAI GPT-3.5 Turbo model
\texttt{gpt-3.5-turbo-16k-0613} (GPT) and hybrid, which is a paraphrase of an extractive summary generated by \texttt{occams}~\cite{analytics2030030} (hybrid). We note that the hybrid method is a paraphrased extract, and the extract's length was chosen to be comparable with the GPT summary to make the comparison easier. The input limit of the GPT model is 16K tokens, which generally allows GPT to summarize the entire document without repeatedly applying it. The only exception is the CrisisFACTS data.
\subsection{Content-Based}
We compare the two summarization approaches, GPT and hybrid, and benchmark their content coverage on the three datasets. Here, we measure the agreement of human-written summaries with the two automatically generated methods. To this end, we use both ROUGE \cite{lin-2004-rouge} and a recently proposed measure SMART \cite{amplayo2023smart} as a check to indicate how closely the automatic summaries match the human-written ones. Both these metrics correlate well with human assessment of content coverage. Table~\ref{tab:RS} gives the mean scores of these automatic evaluations. Broadly, the two summarization methods perform comparably regarding their content similarity with the human summaries for each dataset. Each dataset has about 50 documents (or document sets in the case of TAC 2011). In most instances, the differences are insignificant, meaning that both methods likely have comparable quality content.%\footnote{In follow-on work, we plan to conduct a human evaluation for content to confirm the automatic evaluation results.}
\begin{table}
\begin{center}
\begin{tabular}{l|c|c|c|c}
Dataset &
\multicolumn{2}{c|}{GPT} &
\multicolumn{2}{c}{Hybrid} \\
& {ROUGE-2} & {SMART-2} & {ROUGE-2} & {SMART-2}\\
\hline
CrisisFACTS & 3.37 & 15.3 & 3.90 & 15.3 \\
Cyber Data & 10.2 & 24.9 & 9.3 & 22.9 \\
TAC 2011 & 11.5 & 24.4 & 10.4 & 24.5 \\
\end{tabular}
\end{center}
\caption{Automatic Evaluation Scores.} \label{tab:RS}
\end{table}
\subsection{Attribution}
We now turn to the question of automatic attribution. We evaluated two approaches for attribution, one based on an NLI model and the second using a T5 sentence embedding. We were uncertain how important the context would be for allowing the analysts to judge if the automatic attribution gave any support or contradicted the summary sentence. To this end, we considered two approaches when presenting the sentences in the document that the NLI or embedding model chooses for attribution. We called these two approaches Task 1 and 2 in the study.
In Task 1, the attributing document sentence is presented in context with the sentence preceding and following it. In Task 2, we opted to allow the automatic attribution methods to give three supporting sentences, as some summary sentences may be a fusion of two or more sentences. Figures \ref{Task1:human} and \ref{Task2:human} give pie charts summarizing the results of the three analysts' annotations of 30 summary sentences to attribution text pairings for the NLI and embedding approaches. Here, the summaries are the human-generated summaries from TAC 2011. We note that in both tasks, the sentence embedding method gave attributions that the analysts judged better than the NLI model. Also, it appears that giving three possible supporting sentences gave more accurate attribution.
We next compared summaries generated by GPT on the TAC 2011 data with a hybrid approach where we asked GPT to paraphrase \texttt{occams} extracts. To limit the amount of annotation, we decided only to use the embedding model and the Task 2 approach of providing the top three sentences for the automatic attribution evaluation. In Figure~\ref{Task2:machine}, the hybrid summaries were easier to attribute automatically. Note that the attribution sentences for the hybrid were selected from the extracted summary. As the summaries are approximately 100 words long, the analysts were given a significant portion of the original extract, typically about 5 sentences.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Task1_NLI.png}
\includegraphics[height=5cm]{Figures/Task1_emb.png}
%\end{adjustwidth}
\caption{Using embeddings in Task~1 gives stronger automatic attribution as judged by analysts compared to the NLI model.\label{Task1:human}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Task2_NLI.png}
\includegraphics[height=5cm]{Figures/Task2_emb.png}
%\end{adjustwidth}
\caption{Using embeddings in Task~2 gives stronger automatic attribution as judged by analysts compared to the NLI model.\label{Task2:human}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/TAC2011_GPT.png}
\includegraphics[height=5cm]{Figures/TAC2011_Hybrid.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the TAC 2011 dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{Task2:machine}}
\end{figure}
For the Cyber Threat Intelligence data, we conducted a similar experiment. We generated GPT summaries and hybrid summaries (GPT paraphrases of \texttt{occams} extracts). As the Cyber data might be consumed by both a general audience and experts in cyber security, we separated the annotation into two groups. The first group with general knowledge is labeled as ``analysts,'' and the second group with expertise in cyber security is labeled as ``experts.'' Figures \ref{Cyber:analysts} and \ref{Cyber:experts} give the results. In both cases, the hybrid approach was more likely to provide attribution with full support. Of note is that the cyber experts judged the attributions to be of higher quality. Specifically, the cyber experts labeled the attributions as giving ``full support'' more often than analysts without a background in cyber security.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Cyber_GPT_Analysts.png}
\includegraphics[height=5cm]{Figures/Cyber_Hybrid_Analysts.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the Cyber dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{Cyber:analysts}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Cyber_GPT_Experts.png}
\includegraphics[height=5cm]{Figures/Cyber_Hybrid_Experts.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the Cyber dataset gives rise to stronger automatic attribution as judged by cyber experts compared to using GPT alone.\label{Cyber:experts}}
\end{figure}
For the CrisisFACTS automatic attribution results (Figure~\ref{CrisisFACTS}), we see again the hybrid summary sentences had better automatic attribution than those using GPT without the \texttt{occams} extracts. As this data set was the least fluent of the three, it is not surprising to see that there was a drop in the fraction of attributions that received \emph{full support} relative to the results in Cyber and TAC datasets. For example, with the CrisisFACTS hybrid summaries, 58.9\% of the sentences had full support attributions. In contrast, with the TAC and Cyber datasets, the corresponding summaries had 74\% full support or more.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/CrisisFACTS_GPT.png}
\includegraphics[height=5cm]{Figures/CrisisFACTS_hybrid.png}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the CrisisFACTS dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{CrisisFACTS}}
\end{figure}
\subsection{Refutation}
Finally, we tally the percentage of refutations found in each experiment. Table~\ref{tab:refT12} gives the percentage of refutations for Task~1 and 2. Here, the embedding attribution method is more likely to link to a refuting sentence than the NLI approach. %Further analysis is needed here, but one potential explanation is that the embedding method found information from an earlier time period e.g., the number of fatalities increased in a disaster.
\begin{table}[H]
\centering
\begin{tabular}{c|c|c}
&NLI & Embedding \\
\hline
Task 1 & 0.0 & 10\\
Task 2 & 7.0 & 10\\
\end{tabular}
\caption{Percentage refutations for Task 1 and 2.}
\label{tab:refT12}
\end{table}
Table~\ref{tab:hybridVSgpt} compiles the remaining refutation results for the TAC and Cyber datasets. Here, the hybrid approach had fewer refutations than the GPT purely abstractive method except for the CrisisFACTS data where they both had about 8.9\% refutations.
\begin{table}[H]
\centering
\begin{tabular}{c|c|c}
& Hybrid & GPT\\
\hline
TAC 2011 & 5.6 & 13.3\\
Cyber Analysts & 7.8 & 14.4 \\
Cyber Experts & 8.9 & 10.0 \\
CrisisFACTS & 8.9 & 8.9 \\
\end{tabular}
\caption{Percentages of refutations in all annotations comparing GPT and hybrid approaches.}
\label{tab:hybridVSgpt}
\end{table}
An in-depth analysis of the refutations follows.
\subsection{Refutation: In-Depth}\label{sec:refutation-analysis}
Three data sets -- TAC 2011 (multi-document over time), Cyber Threat Intelligence (single document, specialist) and CrisisFACTS (multi-stream, closest to raw data) -- were analyzed for different types of refutations. The findings were as follows:
\begin{itemize}
\item Allowing 3 attributing sentences instead of 1 improved support significantly, but has the potential to add sentences of lower quality.
\item Refutations often differ from LLM hallucinations in a machine summary.
\item The types of refutations reflect the underlying data. TAC 2011 and CrisisFACTS refutations include timing errors, where attributing sentences refer to different periods of time when facts differed, but Cyber Threat Intelligence did not. Attributing sentences for Cyber Threat Intelligence refutations were more likely to be about related, rather than directly relevant, topics for lower-ranked attributions. CrisisFACTS refutations were of a variety of types and included issues with ambiguity and parsing.
\item Parsing problems due to poorly formatted source text can lead to meaningless attributing sentences.
\item Annotators may differ on what was considered a refutation.
\end{itemize}
Details follow, starting with a new refutation typology based on an existing typology for factual errors in machine summaries from ``Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics'' by Artidoro Pangoni, Vidhisha Balachandran, and Yulia Tsetkov at NAACL 2021~\cite{pagnoni-etal-2021-understanding}. We explore TAC~2011 with NLI and sentence embedding attribution methods on human summaries for Task~1 (1 attributing sentence with preceding and following sentences for context) and Task~2 (3 attributing sentences), followed by an analysis of all three data sets with the sentence embedding attribution method on abstractive and hybrid machine summaries for Task~2.
\subsubsection{Typology of Factual Errors (Refutation)}
The typology of factual errors for abstractive summaries in~\cite{pagnoni-etal-2021-understanding} was modified for the refutation analysis as follows:
\begin{itemize}
\item Semantic Frame Errors:
\begin{itemize}
\item Predicate Error (PredE) : The predicate in the summary statement is inconsistent with the attribution(s).
\item Entity Error (EntE): The primary arguments (or their attributes) in the summary statement are inconsistent with the attribution(s).
\item Circumstance error: (CircE) Additional information (such as location or time) specifying the circumstance around a predicate in the summary statement is inconsistent with the attribution(s).
\end{itemize}
\item Content Verifiability Errors
\begin{itemize}
\item Out of Article Error (OutE): The summary statement contains information not present in the attribution(s); this includes extrinsic hallucinations.
\item Grammatical Error (GramE): The grammar mistakes in an \textit{attribution} is so wrong that it becomes meaningless.
\end{itemize}
\item Additional Categories
\begin{itemize}
\item Others (OthE): Refutation errors that do not correspond to any of the above categories.
\item Not an Error (NE): There are no refutation errors.
\end{itemize}
\end{itemize}
Examples from~\cite{pagnoni-etal-2021-understanding}, modified for refutations can be found in Table~\ref{table:refutation-examples}. The first 4 (PredE, EntE, CircE, OutE) are examples of \textit{summary} statements where errors are highlighted in red. The last (GramE) is an example where the grammar in the \textit{attribution} is so wrong that it is meaningless.
\begin{table}
\small
\centering
\begin{tabular}{llp{.3\textwidth}p{.3\textwidth}}
& \bf{Category} & \bf{Description} & \bf{Example}\\
\bf{PredE} & Predicate Error & The predicate in the summary statement is inconsistent with the attribution(s). & The Ebola vaccine \textcolor{red}{was rejected} by the FDA in 2019 [1].\\
\bf{EntE} & Entity Error & The primary arguments (or their attributes) in the summary statement are inconsistent with the attribution(s). & The \textcolor{red}{COVID-19 vaccine} was approved by the FDA in 2019 [1],[3]. \\
\bf{CircE} & Circumstance Error & Additional information (like location or time) specifying the circumstance around a predicate in the summary statement is inconsistent with the attribution(s). & The first vaccine for Ebola was approved by the FDA in \textcolor{red}{2014} [1]. \\
\bf{OutE} & Out of Article Error & The summary statement contains information not present in the attribution(s). & \textcolor{red}{China} has started clinical trials of the COVID-19 vaccine [3].\\
\bf{GramE} & Grammatical Error & The grammar of the \textit{attribution} is so wrong that it becomes meaningless. & The vaccine for Ebola \textcolor{red}{approved have already started}.
\end{tabular}
\caption{Examples of refutation errors, classified using the refutation typology. Source text sentences: \textit{[1]: The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. [2]: To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. [3]: Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started.} An additional attributing sentence which is meaningless: \textit{[4]: The vaccine for Ebola approved have already started.}} \label{table:refutation-examples}
\end{table}
\subsection{Human: TAC 2011: Task 1 versus Task 2}
We evaluated refutations for the NLI and T5 sentence embedding attribution methods on human-generated summaries from TAC 2011. Two tasks were examined. In Task~1, the attributing document sentence is presented in context with the sentence preceding and following it. In Task~2, the automatic attribution methods give three supporting sentences, since some summary sentences may be the fusion of two or more sentences. Figure~\ref{fig:refutation-tac2011-human-bar} provides bar charts summarizing the refutation results of three analysts' annotations of 30 summary sentences to attribution text pairings with the NLI and embedding automatic attribution methods (60 total pairings for TAC 2011).
\begin{figure}[H]
\centering
\includegraphics[width=0.8\linewidth]{./Figures/TAC2011_Human_BarCharts.png}
\caption{Refutation Error Counts for Human Summaries: TAC 2011: Task 1 and Task 2.}\label{fig:refutation-tac2011-human-bar}
\end{figure}
For Task~1, the 2 semantic refutations were time-shift errors, where the summary contained information inconsistent with an attribution because the attribution referred to a different period of time when fact(s) differed. The 8 content refutations were due to the summary containing information not found in the attribution. All refutations were for the embedding attribution method. The first annotator labeled 6 refutations (4 unique), the second labeled 3 refutations (1 unique), and the third labeled 0 (0 unique). See Table~\ref{tab:human-task-1} for more details.
The refutation profile changed for Task~2. The 4 semantic refutations were mostly but not entirely due to time-shift errors; for example, a summary and the highest ranked attributing sentence stated that the ``soles of shoes'' are considered the ultimate insult in Arab countries, while lower-ranked attributing sentences stated that ``throwing a shoe at someone'' or ``hitting someone with a shoe'' was the worst insult. The 4 content refutations were due to the summary containing additional information not found in the attributions (3) or bad attribution grammar (1). The 6 additional refutations were caused by lower-ranked attributing sentences not being directly relevant to the summary, as those attributions referenced related or unrelated topics instead. There were 5 NLI and 8 embedding refutations. The first annotator labeled 2 refutations (1 unique), the second labeled 12 refutations (9 unique) and the third labeled 0 (0 unique). It was unclear why one example was marked as a refutation and that example was not included in the counts. See Table~\ref{tab:tac2011-human-task2} for more details.
Combining the attribution and refutation analysis for TAC 2011 human summaries (Task~1 and Task~2), it appears that allowing 3 attributing sentences provided more accurate attribution but potentially add sentences of lower quality.
(Note that counting differs for the attribution pie charts versus the refutation bar charts: a summary sentence with its associated attributing sentence(s) are enumerated in the pie charts as often as the annotators mark it for that category, while counts have been deduplicated within a category for the typography-based bar charts.)
\subsection{Machine (Task~2): TAC 2011, Cyber, CrisisFACTS}
Next, we compared summaries generated by GPT with a hybrid approach where we asked GPT to paraphrase \texttt{occams} extracts. To limit the amount of annotation, we only evaluated the embedding attribution method and Task~2 approach of providing the top three attributing sentences. Figure~\ref{fig:refutation-tac2011-cti-crisisfacts-machine-bar} provides bar charts summarizing the refutation results of three analysts' annotations of 30 attribution text pairings for the TAC 2011, Cyber Threat Intelligence, and CrisisFACTS data sets with the abstractive and hybrid machine summarization methods (60 total pairings per data set).
\begin{figure}[H]
\centering
\includegraphics[width=0.8\linewidth]{./Figures/TAC2011_Cyber_CrisisFACTS_Machine_BarCharts.png}
\caption{Refutation Error Counts for Machine Summaries: TAC 2011, Cyber Threat Intelligence, CrisisFACTS: Task 2.}\label{fig:refutation-tac2011-cti-crisisfacts-machine-bar}
\end{figure}
TAC 2011 is a multi-document data set of fluent newswire articles over time, clustered by topic. There were 5 semantic, 4 content, and 3 additional/other refutations. The semantic refutations were due to time-shift errors, where the summary contained information inconsistent with an attributing sentence because the sentence referred to a different period of time when fact(s) differed. Two content refutations were due to summary hallucinations. In the first hallucination, the attributing sentence stated that a widow had been married to an alleged gunman for almost 10 years while the hybrid summary stated that the widow knew him for almost 10 years. In the second hallucination, 1 of 4 alleged terrorist attack masterminds were described in the abstractive summary as ``the'' mastermind. The remaining two content refutations were caused by a summary containing information not found in the attributions. The additional/other refutations were due to attributions containing related information.
There were 8 abstractive and 3 hybrid summary statements with attribution pairings that contained refutations. The first annotator labeled 5 refutations (4 unique), the second labeled 3 (1 unique), and the third labeled 8 (4 unique); it was unclear why one example was marked as a refutation and that example was not included in the counts. See Table~\ref{tab:tac2011-machine-task2} for more details.
Cyber Threat Intelligence is a single document, specialist data set. The annotators were divided into two groups: ``analysts'' for annotators with general knowledge and ``experts'' for annotators who were experts in cyber security. There were 2 semantic, 2 content, and 13 additional/other refutations in the analyst annotations, and 1 semantic, 5 content, and 12 additional/other refutations in the expert annotations. The Cyber data set had the highest proportion of additional/other refutations due to attributions containing related information. This data set contained complex technical sentences, and a summary sentence was more likely to summarize a smaller number of sentences from the source text.
There were no semantic time-shift errors: while a document may describe events over a time period, there was no additional complication from events split over different documents. Both groups identified a semantic refutation for a hybrid summary in which the LLM paraphrasing used stronger language (``highly likely'') compared to the source text (``likely'') to describe the probability of network compromise. The experts identified two summary hallucinations, while the analysts only identified one of the two summary hallucinations.
%One content refutation was due to a summary hallucination for technical reasons (a malware attack component was assigned to the incorrect stage) and another content refutation was due to a misleading summary because the attribution contained a technical detail that was missing in the summary (``incidents involving landing URLs'' versus ``incidents'').
There were 10 abstractive and 6 hybrid summary statements with attribution pairings that contained refutations in the analyst annotations. The first annotator labeled 16 refutations (13 unique), the second labeled 3 (0 unique), and the third labeled 0 (0 unique); it was unclear why one example was marked as a refutation and it is not included in the counts. There were 9 abstractive and 8 hybrid summary statements with attribution pairings that contained refutations in the expert annotations; 7 of those refutations were also marked as refutations by the analysts. The first annotator labeled 1 refutation, the second labeled 2, and the third labeled 14 refutations (including the two summary hallucinations). See Table~\ref{tab:cyber-machine-task2-analyst} and Table~\ref{tab:cyber-machine-task2-expert} for more details.
CrisisFACTS is a multi-stream data set and is closest to raw data of the three data sets. There were 6 semantic, 5 content, and 5 additional/other refutations. Content refutations were divided between 2 OutE refutations (the summary contained information not present in the attributions) and 3 GramE refutations (the grammar of an attributing sentence was so wrong that it became meaningless). TAC 2011 contains fluent newswire and was straightforward to parse into sentences for attribution. Cyber Threat Intelligence documents contained fluent text, lists of fluent text, and unformatted tables. An unformatted table was parsed as a very long sentence, and additional computing resources were required for the automatic embedding attribution method to accommodate very long sentences. Although cyber table-based sentences were not meaningful without formatting, these sentences did not appear in human evaluation. CrisisFACTS contained short text excerpts from multiple data streams (Twitter, Reddit, newswire). Text excerpts were not necessarily fluent text and might be truncated either in the original data set or afterwards in post-processing when multiple excerpts were concatenated and possibly truncated to fit within the LLM context window. CrisisFACTS was a challenging data set to automatically parse into ``sentences'' for attribution, and this was reflected in refutations where an attributing ``sentence'' was meaningless due to poor parsing and truncation.
The 6 CrisisFACTS semantic refutations were mostly due to time-shift errors leading to inconsistent information, with an additional semantic error due to additional location information in the summary being inconsistent with one of the attributing sentences. The additional/other refutations were due to attributions containing related information or the summary statement being so ambiguous that it was difficult to determine if the paired attributing sentences were relevant to the underlying disaster.
There were 7 abstractive and 5 hybrid summary statements with attribution pairings that contained refutations. The first annotator labeled 11 refutations (8 unique) the second labeled 4 (1 unique), and the third labeled 0 (0 unique); it was unclear why one example was marked as a refutation, and it is not included in the counts. See Table~\ref{tab:crisisfacts-machine-task2} for more details.
\section{Conclusions}
In this paper, we set up a series of experiments to measure how well automatic attribution methods worked for attributing information generated by a large language model. We first compared two automatic approaches, one based on a natural language inference model, trained on entailment data, and compared it with a baseline using a sentence embedding method. We found that the embedding model, which was the larger unsupervised model, provided better attribution. However, it had higher instances of refutations.
Next, we compared the embedding approach on two styles of summaries on three datasets: TAC 2011, a Cyber Threat Intelligence dataset, and the TREC 2022 CrisisFACTS dataset. Here, we found that the hybrid summary, a paraphrase of an extractive \texttt{occams} summary, generated text whose sentences were easier to attribute to information in the original documents for all three datasets.
Finally, we used a custom topology to identify the proportion of different categories of attribution-related errors. These refutations often differed from hallucinations in machine summaries. Proportions varied based on the underlying data, and parsing issues could lead to meaningless attributing sentences due to poorly formatted source text.
\vspace{6pt}
\section*{Acknowledgements}
The authors thank Ani Nenkova for her visit to the Summer Conference on Applied Data Science in 2023 and for her suggestions, which help initiate this work. We like to thank a colleague, who chose to remain anonymous for their critical contribution to this work. We also are very grateful to the analyst annotators and cyber experts for their evaluation of automatic attribution methods for three datasets.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% optional
%\supplementary{The following supporting information can be downloaded at: \linksupplementary{s1}, Figure S1: title; Table S1: title; Video S1: title.}
% Only for the journal Methods and Protocols:
% If you wish to submit a video article, please do so with any other supplementary material.
% \supplementary{The following supporting information can be downloaded at: \linksupplementary{s1}, Figure S1: title; Table S1: title; Video S1: title. A supporting video article is available at doi: link.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \authorcontributions{For research articles with several authors, a short paragraph specifying their contributions must be provided. The following statements should be used ``Conceptualization, X.X. and Y.Y.; methodology, X.X.; software, X.X.; validation, X.X., Y.Y. and Z.Z.; formal analysis, X.X.; investigation, X.X.; resources, X.X.; data curation, X.X.; writing---original draft preparation, X.X.; writing---review and editing, X.X.; visualization, X.X.; supervision, X.X.; project administration, X.X.; funding acquisition, Y.Y. All authors have read and agreed to the published version of the manuscript.'', please turn to the \href{http://img.mdpi.org/data/contributor-role-instruction.pdf}{CRediT taxonomy} for the term explanation. Authorship must be limited to those who have contributed substantially to the work~reported.}
% \funding{Please add: ``This research received no external funding'' or ``This research was funded by NAME OF FUNDER grant number XXX.'' and and ``The APC was funded by XXX''. Check carefully that the details given are accurate and use the standard spelling of funding agency names at \url{https://search.crossref.org/funding}, any errors may affect your future funding.}
% \institutionalreview{In this section, you should add the Institutional Review Board Statement and approval number, if relevant to your study. You might choose to exclude this statement if the study did not require ethical approval. Please note that the Editorial Office might ask you for further information. Please add “The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of NAME OF INSTITUTE (protocol code XXX and date of approval).” for studies involving humans. OR “The animal study protocol was approved by the Institutional Review Board (or Ethics Committee) of NAME OF INSTITUTE (protocol code XXX and date of approval).” for studies involving animals. OR “Ethical review and approval were waived for this study due to REASON (please provide a detailed justification).” OR “Not applicable” for studies not involving humans or animals.}
% \informedconsent{Any research article describing a study involving humans should contain this statement. Please add ``Informed consent was obtained from all subjects involved in the study.'' OR ``Patient consent was waived due to REASON (please provide a detailed justification).'' OR ``Not applicable'' for studies not involving humans. You might also choose to exclude this statement if the study did not involve humans.
% Written informed consent for publication must be obtained from participating patients who can be identified (including by the patients themselves). Please state ``Written informed consent has been obtained from the patient(s) to publish this paper'' if applicable.}
% \dataavailability{We encourage all authors of articles published in MDPI journals to share their research data. In this section, please provide details regarding where data supporting reported results can be found, including links to publicly archived datasets analyzed or generated during the study. Where no new data were created, or where data is unavailable due to privacy or ethical re-strictions, a statement is still required. Suggested Data Availability Statements are available in section “MDPI Research Data Policies” at \url{https://www.mdpi.com/ethics}.}
%\acknowledgments{In this section you can acknowledge any support given which is not covered by the author contribution or funding sections. This may include administrative and technical support, or donations in kind (e.g., materials used for experiments).}
%\conflictsofinterest{Declare conflicts of interest or state ``The authors declare no conflict of interest.'' Authors must identify and declare any personal circumstances or interest that may be perceived as inappropriately influencing the representation or interpretation of reported research results. Any role of the funders in the design of the study, in the collection, analyses or interpretation of data, and in the writing of the manuscecision to publish the results must be declared in this section. If there is no role, please state ``The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the~results''.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optional
%\sampleavailability{Samples of the coThe authors' mpounds ... are available from the authors.}
%% Only for journal Encyclopedia
%\entrylink{The Link to this entry published on the encyclopedia platform.}
%\abbreviations{Abbreviations}{
%The following abbreviations are used in this manuscript:\\
%\noindent
%\begin{tabular}{@{}ll}
%SCADS & Summer Conference on Applied Data Science\\
%LAS & Laboratory of Analytic Sciences
%\end{tabular}
%}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optional
\appendixtitles{yes} % Leave argument "no" if all appendix headings stay EMPTY (then no dot is printed after "Appendix A"). If the appendix sections contain a heading, change the argument to "yes."
\appendixstart
\appendix
\section[\appendixname~\thesection]{Annotation Guidelines}
This appendix show sample annotation guidelines for TAC 2011. We ask annotators to read these guidelines as part of the human evaluation. The SCADS Cyber Threat Intelligence Data Set and CrisisFACTS guidelines were similar (3 attributing sentences for Task 2: Machine Summaries).
\subsection[\appendixname~\thesubsection]{TAC 2011: Task 1: Human Summaries}
\begin{myquotation}
\input{AnnotationGuidelines/tac-task1}
\end{myquotation}
\subsection[\appendixname~\thesubsection]{TAC 2011: Task 2: Machine Summaries}
\begin{quotation}
\input{AnnotationGuidelines/tac-task2}
\end{quotation}
%\subsection[\appendixname~\thesubsection]{CrisisFACTS: Task 1}
%\begin{quotation}
%\input{AnnotationGuidelines/facts-task1}
%\end{quotation}
%\subsection[\appendixname~\thesubsection]{CrisisFACTS: Task 2}
%\begin{quotation}
%\input{AnnotationGuidelines/facts-task2}
%\end{quotation}
\section[\appendixname~\thesection]{Refutation Analysis Tables}
Tables used for the refutation analysis in Subsection~\ref{sec:refutation-analysis} are provided in this appendix. Duplicates are shown in a gray font.
\input{RefutationAnalysis/refutation-analysis-tables}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{adjustwidth}{-\extralength}{0cm}
%\printendnotes[custom] % Un-comment to print a list of endnotes
\reftitle{References}
% Please provide either the correct journal abbreviation (e.g., according to the “List of Title Word Abbreviations” http://www.issn.org/services/online-services/access-to-the-ltwa/) or the full name of the journal.
% Citations and References in Supplementary files are permitted provided that they also appear in the reference list here.
%=====================================
% References, variant A: external bibliography
%=====================================
\bibliography{references_human_eval_attrib}
% If authors have biography, please use the format below
%\section*{Short Biography of Authors}
%\bio
%{\raisebox{-0.35cm}{\includegraphics[width=3.5cm,height=5.3cm,clip,keepaspectratio]{Definitions/author1.pdf}}}
%{\textbf{Firstname Lastname} Biography of first author}
%
%\bio
%{\raisebox{-0.35cm}{\includegraphics[width=3.5cm,height=5.3cm,clip,keepaspectratio]{Definitions/author2.jpg}}}
%{\textbf{Firstname Lastname} Biography of second author}
% For the MDPI journals use author-date citation, please follow the formatting guidelines on http://www.mdpi.com/authors/references
% To cite two works by the same author: \citeauthor{ref-journal-1a} (\citeyear{ref-journal-1a}, \citeyear{ref-journal-1b}). This produces: Whittaker (1967, 1975)
% To cite two works by the same author with specific pages: \citeauthor{ref-journal-3a} (\citeyear{ref-journal-3a}, p. 328; \citeyear{ref-journal-3b}, p.475). This produces: Wong (1999, p. 328; 2000, p. 475)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% for journal Sci
%\reviewreports{\\
%Reviewer 1 comments and authors’ response\\
%Reviewer 2 comments and authors’ response\\
%Reviewer 3 comments and authors’ response
%}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\PublishersNote{}
\end{adjustwidth}
% Comments from annotators:
% I think may have other comments. This is one that Aaron passed along for the CrisisFACTS and I wanted to capture it somewhere.
% "One interesting point this time around was that the summaries added adverbs that made me question if it was partial or full support. Here is an example: "
% [SUMMARY] The fire has been ongoing for about a week, and containment is slowly increasing.
% [**EVAL**] https://t.co/q5zLkZhbwl⋯ “I’m just thinking we turned the bend here,” U.S. Forest Service spokeswoman Vickie Wright said of the wildfire that had burned out of control for nearly a week.
% [**EVAL**] As the smoke began to clear above Lake Elsinore, firefighters took heart Sunday as they appeared to gain control of the Holy fire, with containment at 51 percent and 22,714 acres consumed.
% [**EVAL**] Calif.’s #HolyFire, Sees Growth In Size, However Containment Also Increases https://t.co/3vnIMuKApJ https://t.co/3B87t3UyGH⋯ Day 6 Of So.
% A comment for one of the expert cyber annotators and our response.
% On 10/11/24 10:40, Aaron Wiechmann wrote:
% > John,
% > Sounds good. Also, I received the following feedback from one of the labelers in case it is helpful to you.
% >
% > "One thing I wanted to note that jumped out to me a lot. It was pretty common to have the summary be an almost exact word for word copy of one of the eval statements, with only a 1-2 words changed or the structure slightly altered. The other two eval statements would be further elaboration or unrelated to the summary entirely. I'd say a good third of the labels were like this. Felt like it was cheating a bit by doing since its trying to support itself by just directly repeating itself. There were also a few summaries that were either lacking necessary context to properly judge the eval statements, or just straight statements that didn't really apply to eval (i.e "Please see x report" as a summary). Overall though, there usually was at least 1 eval statement that did directly support the summary"
% >
% > Cheers,
% > Aaron
% Response:
% Thank you, for the feedback. I wouldn't call that cheating, I call that paraphrasing, which is what we asked the model to do to the occams extracts!
% John
% As to the second point, we made the decision to use three sentences. For the hybrid summaries the attribution was limited to the original occams extract sentences. Often it would be the case that only 1 or 2 would really be supportive of each of the paraphrased summary sentences.
% In short, I appreciate the feedback, but it is what we expected. In an operational system I think the user interface would take the user back to the the orginal passages in corresponding to one or more citations. We wouldn't want to always give 3.
\end{document}
This evaluation will use the CrisisFACTS data set. The data set consists of content snippets (approximately a sentence in length) from three content streams (Twitter, Reddit, online news sources) related to multiple crises (wildfires, hurricanes, floods). A summary is based on important facts for a day of a crisis event.
\subsection*{Examples}
The upcoming examples are from fictitious content snippets (in the style of news and Twitter) related to crises during \textit{The Wonderful Wizard of Oz}. The examples are based on the original book.
\subsubsection*{Example: Support, Refute=no}
\mysample{
[SUMMARY] A cyclone dropped a house on the Wicked Witch of the East and killed her.\newline
[**EVAL**] Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch
}
The sentence to evaluate for attribution is “Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch”. This sentence provides supporting evidence for a cyclone dropping a house on the Wicked Witch of the East.
The summary statement is not fully supported since no evidence is provided for the Wicked Witch of the East being killed. However, partial support meets the definition of support. You may mark “support” for an answer.
Consider the refute question next. The attribution sentence does not contain contradictory information. The answer is ``refute=no.''
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] A scarecrow was stuck on a pole in the middle of a river after a rafting incident, and was saved by a child, a tin man, a lion, and a small animal of unknown origin.\newline
[**EVAL**] A scarecrow stuck on a pole in the middle of a cornfield was rescued by an unknown good samaritan.
}
The attribution sentence does not provide supporting evidence for a river rescue because it describes a cornfield rescue. This example may be marked “no support”.
Consider the refute question next. While the attribution sentence describes a scarecrow rescue, this example will be marked as ``refute=yes'' because the type of rescue (cornfield versus river) differs.
This example could have also been marked as ``refute=no'' because the rescues differ too much to be related. There will be ambiguous cases like this where it is not clear if an attribution refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another (e.g., Winkies identified as Munchkins), or misinterprets the source document during text generation (e.g., child saves scarecrow misinterpreted as scarecrow saves child). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., cornfield rescue versus river rescue).
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with “refute=no”.
\subsubsection*{Example: No Support, Refute=no}
\mysample{
[SUMMARY] A sleeping lion was rescued from a deadly poppy field by a large group of field mice, aided by a tin man, scarecrow, child, and an animal of unknown origin.\newline
[**EVAL**] WARNING: Large spider approaching. Seek shelter immediately! \#giantspider
}
There will be a topic-based relationship between a statement and an attribution. CrisisFACTS content snippets will be based on the same crises event. The examples in this document are based on crises from a single book, \textit{The Wonderful Wizard of Oz}. However, the connection is too tenuous for this example (lion rescue, spider attack), and it should be marked “no support”.
\subsection*{Examples}
The upcoming examples are from fictitious content snippets (in the style of news and Twitter) related to crises during \textit{The Wonderful Wizard of Oz}. The examples are based on the original book.
\subsubsection*{Example: Full Support, Refute=no}
\mysample{
[SUMMARY] Winged Monkeys are feared due to frightful attacks, and are believed to be aligned with the Wicked Witch of the West.\newline
[**EVAL**] Winged Monkeys drop tin man from great heights to rocks below \#oznews\newline
[**EVAL**] Winged Monkeys tear out straw from scarecrow and leave clothes in tall tree. \#oznews\newline
[**EVAL**] Winged Monkeys kidnap child and small animal of unknown origin, imprison them in castle of Wicked Witch of West. \#oznews
}
You should consider all 3 sentences as part of the attribution for this task. The sentences in combination provide supporting evidence for multiple attacks by Winged Monkeys. The third sentence implies an affiliation between Winged Monkeys and the Wicked Witch of the West. The sentences fully support a statement that Winged Monkeys conduct fearful attacks and are aligned with the Wicked Witch of the West. This example can be marked “full support”.
Consider the refute question next. The three sentences support the summary statement and do not contain contradictory information. The answer is “refute=no”.
\subsubsection*{Example: Full support, Refute=no}
\mysample{
[SUMMARY] A cyclone dropped a house on the Wicked Witch of the East and killed her.\newline
[**EVAL**] The Wicked Witch of the East died due to injuries from a home dropped by a cyclone.\newline
[**EVAL**] Cyclone drops house on Wicked Witch of East \#cyclone \#eastwitch\newline
[**EVAL**] WARNING: tin man with sharp axe in area, attacked Tree Border Patrol. Considered very dangerous. Do not approach! \#livingforest
}
The first sentence fully supports the summary statement. The second sentence partially supports the statement. The third sentence is not related to this accident. While the third sentence does not provide supporting evidence, the three sentences when viewed as a group provide full support, and this example can be marked “full support”.
Consider the refute question next. The first two sentences support the summary statement and do not contain contradictory information. The third sentence is irrelevant. An irrelevant sentence is not a refutation. The answer is “refute=no”.
\subsubsection*{Example: Partial support, Refute=no}
\mysample{
[SUMMARY] The Wicked Witch of the West is aligned with Wolves, Crows, and Munchkins, who further her political agenda through acts which induce fear.\newline
[**EVAL**] Oz News Exclusive: Wicked Witch of West pays Wolves, Crows, and Black Bees to terrify populace. \#oznews\newline
[**EVAL**] Wolves attack and kill protesters outside of Wicked Witch of the West’s castle \#oznews\newline
[**EVAL**] WARNING: Crows approach. Seek shelter immediately! \#oznews \#landofthewest
}
The first two sentences provide supporting evidence that Wolves and Crows are aligned with the Wicked Witch of the West and use fear-based methods on her behalf. The third statement provides additional information that crows are viewed as dangerous in the Land of the West. None of the sentences support an alliance with Munchkins. Mark this example as “partial support” since the sentences partially support the summary.
The detail about Munchkins is a mistake in the summary. Munchkins are a peaceful people living in a region controlled by the Wicked Witch of the East, and are not agents of terror. Full support will not be possible due to the mistake in the summary.
Consider the “refute” question next. None of the attribution sentences contain information that refute the summary statement. The answer may be marked “refute=no”.
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] The King of the Winged Monkeys thanked the Good Witch of the South for freeing them from magical cap enslavement, and declared that they would return to the Great Forest.\newline
[**EVAL**] Dorothy thanked the Good Witch of the South for telling her about the magical silver shoes and used them to return to Kansas.
[**EVAL**] China cow breaks leg after startled by giants. \#oznews \#daintychina\newline
[**EVAL**] Insure your home with Oz Insurance. \#ozinsurance
}
The 3 attribution sentences do not provide supporting evidence for the summary statement about Winged Monkeys. This example may be marked “no support”.
Consider the refute question next. The summary statement and first attribution sentence are about two different cases where the Good Witch was thanked for magic-related repatriations (Winged Monkeys/magical cap versus Dororthy/magical shoes). This sentence will be treated as a refutation because it is related but different. The last two sentences are irrelevant and are not viewed as refutations.
Since there is one attribution sentence that refutes the summary statement, the question may be marked “refute=yes”.
This example could have also been marked as “refute=no” if you decided that the first sentence differed too much from the summary statement. There will be ambiguous cases like this where it is not clear if an attribution sentence is a refutation or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the
purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another (e.g., Winkies identified as Munchkins), or misinterprets the source document during text generation (e.g., Dorothy thanks Good Witch misinterpreted as Good Witch thanks Dorothy). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., magical cap versus magical shoes).
Another type of misfire is when the attribution method has low accuracy and
picks an irrelevant sentence. We would like to capture this with “refute=no”
\section*{Questions or Feedback?}
If you have questions about the task, or any feedback about how we could make it better or what your experience was like with it, please contact us on Slack using the channel \texttt{\#inf-attribution-evaluation} and we'll get back to you. Thanks!
\section*{Overview}
We are a team interested in evaluating attribution for summarization (how to link automatically generated summaries back to their source material.)
In this project, you will evaluate the quality of attributions for summaries that have been created by humans or by automatic methods that mimic what humans would write. We would like to evaluate if an attribution method can provide accurate attributions, and if the accuracy is influenced by the type of automatic summarization method used.
At a high level, the project breaks down into two tasks:
\begin{enumerate}
\item Given a statement in a summary, judge the quality of a single attribution.
\item Given a statement in a summary, judge whether the statement is fully supported by three (ranked) attributions.
\end{enumerate}
Examples of the task you have chosen will be provided in this document.
\section*{Preliminaries}
\textcolor{red}{@aaron - THIS SECTION IS FOR YOU TO PROVIDE INFINITYPOOL GUIDANCE}
\section*{Task 1: Judge the quality of a single attribution}
In this part, you will be provided a statement from the summary, and will be asked to judge the quality of a corresponding attribution in the source material. The statement will be a single sentence from the summary. The attribution will be a single (1) sentence from the source document, along with surrounding context sentences (sentence before and sentence after the attribution sentence in the source document, if available).
Carefully read the summary statement and the attribution sentence with the surrounding context. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding attribution in the source document that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Support}: Some or all of the information in the summary statement is supported by the attribution sentence.
\item \textbf{No Support}: The attribution sentence does not support the summary statement.
\item \textbf{Unclear (Can’t Make Judgement)}: The information provided does not make it possible to determine whether the attribution sentence supports or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is ``yes'' or ``no'' to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (``yes'') with ``support'' or ``no support''. Answer ``no'' if the information is unclear.
\end{itemize}
\section*{Task 1: Judging the quality of a single attribution}
In this part, you will be provided a statement from the summary, and will be asked to judge the quality of a corresponding attribution in the source material. The statement will be a single sentence from the summary. The attribution will be a single (1) sentence from the source document.
Carefully read the summary statement and the attribution sentence with the surrounding context. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding attribution in the source document that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Support}: Some or all of the information in the summary statement is supported by the attribution sentence.
\item \textbf{No Support}: The attribution sentence does not support the summary statement.
\item \textbf{Unclear (Can’t Make Judgement)}: The information provided does not make it possible to determine whether the attribution sentence supports or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is ``yes'' or ``no'' to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (“yes”) with “support” or “no support”. Answer “no” if the information is unclear.
\end{itemize}
\section*{Task 2: Judge the quality of a group of attributions}
In this part, you will be provided a statement from the summary, and will be asked to judge the amount of support by the attribution. The statement will be a single sentence from the summary. The attribution will be three (3) sentences from the source material.
Carefully read the summary statement and the 3 attribution sentences. The summary statement may appear to be very fluent and well-formed, and there may be slight differences compared to the corresponding 3 attribution sentences from the source data that is not easy to discern at first glance. Please pay close attention to the text, as if you were proofreading.
Your task will be to determine which category applies:
\begin{itemize}
\item \textbf{Full Support}: All of the information in the summary statement is supported by the attribution sentences.
\item \textbf{Partial Support}: Only some of the information in the summary statement is supported, but other parts of the information are missing.
\item \textbf{No Support}: The attribution sentences do not support the summary statement.
\item \textbf{Unclear (Can't Make Judgment)}: The information provided does not make it possible to determine whether the attribution sentence fully supports, partially supports, or does not support the summary statement. The information is unclear, incomprehensible, malformed, or deficient in some other way.
\end{itemize}
The second part of the task will be to determine if the answer is “yes” or “no” to the following question:
\begin{itemize}
\item \textbf{Refute}: Does the attribution contain information that refutes the summary statement? There can be a refutation (“yes”) with “full support”, “partial support”, or “no support”. Answer “no” if the information is unclear.
\end{itemize}
This evaluation will use the TAC 2011 (Text Analysis Conference) data set for multi-document summarization. The data set contains news articles grouped by topic (10 documents per topic). A summary is based on a single topic (10 documents).
\subsection*{Examples}
For TAC 2011, a summary is based on 10 documents for a single topic. The upcoming examples are from a single topic on threats to sea turtles.
\subsubsection*{Example: Support, Refute=no}
\mysample{
[SUMMARY] Sea turtles are threatened by increased poaching, habitat destruction for land development, illegal fishing, and pollution.\newline
[CONTEXT] The fishermen do not properly release the turtles and often kill them, leaving them to wash ashore. \newline
[**EVAL**] Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists. \newline
[CONTEXT] Mohammad Aminul Islam, the top administrator for the area, ordered local officials to teach people, from fishermen to tourists, to change their behavior.
}
The sentence to evaluate for attribution is “Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists.” This sentence provides supporting evidence that sea turtles are threatened by pollution. While the first context sentence supports illegal fishing, the decision for the evaluation should only be based on the sentence with the {\fontfamily{lmss}\selectfont[**EVAL**]} (evaluate) prefix.
The summary statement is not fully supported since no supporting evidence is provided for poaching, habitat destruction for land development, and illegal fishing by the attribution sentence. However, partial support meets the definition of support. You may mark “support” for an answer.
Consider the refute question next. The attribution sentence supports the summary statement and also contains supplemental information. It does not contain contradictory information. The answer is ``refute=no''.
\subsubsection*{Example: No Support, Refute=yes}
\mysample{
[SUMMARY] Malaysia has arrested a number of poachers, many from south China, for poaching turtles and other animals.\newline
[CONTEXT] At that price, a single nest containing about 100 eggs nets a poacher 1,000 pesos. Compare that, he says, to Mexico's minimum daily wage of 45 pesos. \newline
[**EVAL**] The Mexican government takes poaching seriously, he continued, assessing steep fines or jail time--``it's a worse offense than drugs''--but it lacks the manpower to prevent the crime in the first place or prosecute offenders in the second. \newline
[CONTEXT] Rapid development endangers the turtles.
}
The attribution sentence is the second of three consecutive sentences. This sentence does not provide supporting evidence for poaching in Malaysia because the attribution is about poaching in Mexico which is a different country. (The difference is due to the summary being based on a different news article than the attribution.) You may mark ``no support.''
Consider the refute question next. While the attribution sentence supports turtle poaching having legal repercussions, this example will be marked as “refute=yes” because the country differs.
This example could have also been marked as “refute=no” because the statement and attribution sentence are about different countries. There will be ambiguous cases like this where it is not clear if an attribution refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide between “refute=yes” and “refute=no”, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality for another, or misinterprets the source document during text generation (e.g., dog bites man misinterpreted as man bites dog). Or when an attribution method provides a link which is related to the summary statement but misses the mark (e.g., Mexican poaching instead of Malaysian poaching).
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with the “refute=no”.
\subsubsection*{Example: No Support, Refute=no}
\mysample{
[SUMMARY] 90 sea turtles were rescued in Mexico when they became comatose from a cold snap.\newline
[CONTEXT] Some sea turtle species eat jellyfish like they are going out of style.\newline
[**EVAL**] If the turtles weren't around to eat the jellyfish, the jellyfish population would explode.\newline
[CONTEXT] The hungry jellyfish hordes would gorge on zooplankton.
}
The attribution sentence is “If the turtles weren't around to eat the jellyfish, the jellyfish population would explode.” This sentence does not provide supporting evidence for a turtle rescue. You may mark ``no support.''
Consider the refute question. There will be a topic-based relationship between a statement and an attribution with TAC 2011 since the summary is based on news articles from the same topic (e.g., threats to sea turtles). However, the connection is too tenuous for this example, and it should be marked “refute=no”.
\subsection*{Examples}
For TAC 2011, a summary is based on 10 documents for a single topic. The upcoming examples are from a single topic on threats to sea turtles.
\subsubsection*{Example: Full support, Refute=no}
\mysample{
[SUMMARY] Some countries have passed laws to protect the turtles.\newline
[**EVAL**] The Mexican government takes poaching seriously, he continued, assessing steep fines or jail time--``it's a worse offense than drugs''--but it lacks the manpower to prevent the crime in the first place or prosecute offenders in the second.\newline
[**EVAL**] A Malaysian court Friday jailed a group of 13 Chinese fishermen for 18 months each for poaching nearly 80 protected turtles.\newline
[**EVAL**] Ahmed would not give any specific reason for the spike in deaths, but said the use of illegal fishing nets near the shoreline has apparently increased recently.
}
%You should consider all 3 sentences as part of the attribution for Task 2 of the evaluation.
The first sentence describes a poaching law in Mexico. Malaysian law can be inferred by the second sentence. The third sentence describes illegal fishing nets but does not specify the country (Bangladesh). Combined, the sentences are sufficient to fully support a statement that more than one country has laws to protect turtles, and the example can be marked ``full support''.
Consider the refute question next. Each sentence supports the summary statement and does not contain contradictory information. The answer is ``refute=no''.
\subsubsection*{Full support, Refute=no}
\mysample{
[SUMMARY] 90 sea turtles were rescued in Mexico when they became comatose from a cold snap.\newline
[**EVAL**] Nearly 90 sea turtles rescued after a cold snap left them comatose have been returned to the Gulf of Mexico.\newline
[**EVAL**] Dozens of rehabilitated sea turtles, threatened by cold, returned to Gulf of Mexico.\newline
[**EVAL**] Green turtles are born in southern Mexico and spend their early years feeding on turtle grass in shallow areas such as the Laguna Madre.
}
The first sentence fully supports the summary statement. The second sentence partially supports the statement. The third sentence is about turtles in Mexico but not a rescue. %While the third sentence does not provide supporting evidence, the three sentences when viewed as a group provide full support, and this example can be marked “full support.”
If one or two sentences fully support the summary statement, the question may be marked ``full support''. Support is not required from all 3 sentences.
Consider the refute question next. The first two sentences support the summary sentence and do not contain contradictory information. The third sentence is about Mexican turtles but is about a different topic (rescue versus early life). The connection is tenuous and will not be viewed as a refutation. Since none of the sentences contain a refutation, the answer to the question is “refute=no”.
This example could have also been marked as “refute=yes” because the third sentence is about Mexican turtles so there is a connection between that sentence and the summary statement, but the facts are different. There will be ambiguous cases like this where it is not clear if an attribution sentence refutes a statement or not. We want you to use your best judgment during the evaluation.
If it helps to decide if a sentence refutes the summary statement or not, the purpose of the “refute” choice is to identify more focused algorithmic misfires. For example, when an automatic summarization method generates an incorrect fact (“hallucination”) like substituting one nationality over another, or misinterprets the source document during text generation (e.g., dog bites man misinterpreted as man bites dog). Or when an attribution method provides a link which is related to the summary statement but misses the mark.
Another type of misfire is when the attribution method has low accuracy and picks an irrelevant sentence. We would like to capture this with ``refute=no''.
\subsubsection*{Full support, Refute=yes}
\mysample{
[SUMMARY] 17 Chinese nationals found with 274 protected turtles, most of them dead, were recently arrested in Malaysian waters.\newline
[**EVAL**] The poachers were caught by Malaysian marine police on March 28 in waters off the coast of eastern Sabah state with a catch of 274 protected Hawksbill and Green turtles, only 32 of which were still alive.\newline
[**EVAL**] They were the second group of fishermen from China's Hainan island jailed this week for turtle poaching off the waters of Malaysia's eastern Sabah state, with 17 men sentenced on Wednesday for catching 274 of the creatures.\newline
[**EVAL**] A Malaysian court Friday jailed a group of 13 Chinese fishermen for 18 months each for poaching nearly 80 protected turtles.
}
The first sentence provides supporting evidence for poachers being caught with 274 protected turtles, most of them dead. The second sentence provides support for the arrest of 17 Chinese nationals with 274 turtles. The third sentence is about a different arrest.
The first two sentences provide full support for the summary statement, and this example may be marked ``full support''.
Consider the refute question next. The third sentence is on the same subject: a Malaysian arrest of Chinese nationals for poaching protected turtles. However, it is about a different arrest. This is an example of a misfire where a sentence is related but misses the mark, and can be viewed as an instance of ``refute=yes''.
\subsubsection*{Partial support, Refute=no}
\mysample{
[SUMMARY] Sea turtles are threatened by increased poaching, habitat destruction for land development, illegal fishing, and pollution.\newline
[**EVAL**] The turtles--which can be found in other parts of Asia along the Andaman Coast and the South China Sea--have seen their numbers reduced drastically in recent years mostly due to poaching of their eggs, pollution and habitat loss.\newline
[**EVAL**] Turtles that come to lay eggs on the beaches may be killed by pollution, stray dogs or foxes, or captured by tourists.\newline
[**EVAL**] Sea turtles face a host of threats around the world.
}
The three sentences provide supporting evidence for a threat to sea turtles for several reasons but do not support illegal fishing. This example may be marked “partial support”.
None of the sentences refute the summary sentence. The answer is “refute=no”.
\subsubsection*{Example: No support, Refute=no}
\mysample{
[SUMMARY] Scientists say that the loss of sea turtles could have unforeseen, even catastrophic, consequences.\newline
[**EVAL**] Efforts in Mexico to protect sea turtles are a patchwork of official and unofficial endeavors, according to Sea Turtle, Inc., a nonprofit based in South Padre Island.\newline
[**EVAL**] ``We are trying to mobilize resources to make a bigger plan to save the sea turtles in the future.''\newline
[**EVAL**] Thai villagers have caught a river terrapin turtle that was thought to be extinct in the country.
}
Sentences will be related to a common topic (threats to sea turtles). However, the 3 attribution sentences do not support the summary statement. This example may be marked “no support”.
Consider the refute question next. Each attribution sentence is irrelevant to the summary statement. An irrelevant sentence is not a refutation. None of the sentences refute the summary statement and the question may be marked “refute=no”.
\newcounter{question}
\setcounter{question}{0}
\newcommand{\question}[1]{\item[Q\refstepcounter{question}\thequestion.] #1}
\newcommand{\answer}[1]{\item[A\thequestion.] #1}
\subsubsection*{FAQ}
\begin{itemize}
\question\textit{How do we mark a group of attribution sentences if one of the sentences fully supports the summary statement, no matter what the rest say? Do we mark this as ``full support'' or ``partial support''?}
\answer If one attribution sentence fully supports the summary statement, then mark the group as ``full support'' for the first question.
\question\textit{In InfinityPool (the annotator platform), choices were not pre-selected for the first question about support. However, an answer was already highlighted for the second refute question before I answered it. Should this happen?}
\answer The refute questions all default to ``no''. Change an answer to ``yes'' (when applicable).
\end{itemize}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{CrisisFACTS}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/FACTS/overview-data}
%\input{Shared/preliminaries}
\input{AnnotationGuidelines/Shared/task1-nocontext}
\input{AnnotationGuidelines/FACTS/task1-examples}
%\input{Shared/task2}
%\input{FACTS/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{CrisisFACTS}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/FACTS/overview-data}
%\input{Shared/preliminaries}
%\input{Shared/task1-nocontext}
%\input{FACTS/task1-examples}
\input{AnnotationGuidelines/Shared/task2}
\input{AnnotationGuidelines/FACTS/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%\title{Annotation Guidelines}
%\author{TAC 2011}
%\date{}
\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/TAC2011/overview-data}
%\input{Shared/preliminaries}
\input{AnnotationGuidelines/Shared/task1-context}
\input{AnnotationGuidelines/TAC2011/task1-examples}
%\input{Shared/task2}
%\input{TAC2011/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%%\documentclass[12pt]{article}
%%\usepackage{graphicx} % Required for inserting images
%%\usepackage{xcolor}
%%\usepackage[margin=1.25in]{geometry}
\newcommand{\mysample}[1]{
\begin{center}
\begin{minipage}{.80\textwidth}\fontfamily{lmss}\selectfont
#1
\end{minipage}
\end{center}
}
%%\title{Annotation Guidelines}
%%\author{TAC 2011}
%%\date{}
%%\begin{document}
%%\maketitle
\input{AnnotationGuidelines/Shared/overview}
\input{AnnotationGuidelines/TAC2011/overview-data}
%\input{Shared/preliminaries}
%\input{Shared/task1-context}
%\input{TAC2011/task1-examples}
\input{AnnotationGuidelines/Shared/task2}
\input{AnnotationGuidelines/TAC2011/task2-faq}
\input{AnnotationGuidelines/TAC2011/task2-examples}
\input{AnnotationGuidelines/Shared/feedback}
%%\end{document}
%\DeclareOption{}{\gdef\@journal{}\gdef\@journalfull{}\gdef\@journalshort{}\gdef\@doiabbr{}\gdef\@ISSN{}\gdef\@continuouspages{yes}}
\DeclareOption{scads}
{\gdef\@journal{scads}\gdef\@journalfull{Summer Conference on Applied Data Science (SCADS)}\gdef\@journalshort{SCADS}\gdef\@continuouspages{yes}}
% LaTeX support: latex@mdpi.com
% For support, please attach all files needed for compiling as well as the log file, and specify your operating system, LaTeX version, and LaTeX editor.
%=================================================================
\documentclass[scads,technicalreport,accept,pdftex,moreauthors]{Definitions/scads}
%--------------------
% Class Options:
%--------------------
% For posting an early version of this manuscript as a preprint, you may use "preprints" as the journal. Changing "submit" to "accept" before posting will remove line numbers.
%---------
% Manuscript Type
%---------
% The default type of manuscript is "technicalreport", but can be replaced by:
% abstract, addendum, article, book, bookreview, briefreport, casereport, comment, commentary, communication, conferenceproceedings, correction, conferencereport, entry, expressionofconcern, extendedabstract, datadescriptor, editorial, essay, erratum, hypothesis, interestingimage, obituary, opinion, projectreport, reply, retraction, review, perspective, protocol, shortnote, studyprotocol, systematicreview, supfile, technicalnote, technicalreport, viewpoint, guidelines, registeredreport, tutorial
% supfile = supplementary materials
%----------
% submit
%----------
% The class option "submit" will be changed to "accept" by the Editorial Office when the paper is accepted. This will only make changes to the frontpage (e.g., the logo of the journal will get visible), the headings, and the copyright information. Also, line numbering will be removed. Journal info and pagination for accepted papers will also be assigned by the Editorial Office.
%------------------
% moreauthors
%------------------
% If there is only one author the class option oneauthor should be used. Otherwise use the class option moreauthors.
%---------
% pdftex
%---------
% The option pdftex is for use with pdfLaTeX. Remove "pdftex" for (1) compiling with LaTeX & dvi2pdf (if eps figures are used) or for (2) compiling with XeLaTeX.
%=================================================================
% MDPI internal commands - do not modify
\firstpage{1}
\makeatletter
\setcounter{page}{\@firstpage}
\makeatother
\pubvolume{1}
\issuenum{1}
\articlenumber{0}
\pubyear{2023}
\copyrightyear{2023}
%\externaleditor{Academic Editor: Firstname Lastname}
\datereceived{ }
\daterevised{ } % Comment out if no revised date
\dateaccepted{ }
\datepublished{ }
%\datecorrected{} % For corrected papers: "Corrected: XXX" date in the original paper.
%\dateretracted{} % For corrected papers: "Retracted: XXX" date in the original paper.
\hreflink{https://doi.org/} % If needed use \linebreak
%\doinum{}
%\pdfoutput=1 % Uncommented for upload to arXiv.org
%=================================================================
% Add packages and commands here. The following packages are loaded in our class file: fontenc, inputenc, calc, indentfirst, fancyhdr, graphicx, epstopdf, lastpage, ifthen, float, amsmath, amssymb, lineno, setspace, enumitem, mathpazo, booktabs, titlesec, etoolbox, tabto, xcolor, colortbl, soul, multirow, microtype, tikz, totcount, changepage, attrib, upgreek, array, tabularx, pbox, ragged2e, tocloft, marginnote, marginfix, enotez, amsthm, natbib, hyperref, cleveref, scrextend, url, geometry, newfloat, caption, draftwatermark, seqsplit
% cleveref: load \crefname definitions after \begin{document}
\usepackage{ulem}
\usepackage{nopageno}
%\usepackage{draftwatermark}
\usepackage{datetime}
\mmddyydate
%\SetWatermarkLightness{0.9}
%\SetWatermarkText{\parbox{6em}
%{\centering{DRAFT\\\today}}}
%\SetWatermarkScale{0.7}
%=================================================================
% Please use the following mathematics environments: Theorem, Lemma, Corollary, Proposition, Characterization, Property, Problem, Example, ExamplesandDefinitions, Hypothesis, Remark, Definition, Notation, Assumption
%% For proofs, please use the proof environment (the amsthm package is loaded by the MDPI class).
\usepackage[indentfirst=false]{quoting}
\newenvironment{myquotation}{
\begin{quoting}
}
{
\end{quoting}
}
\def\gray#1{\textcolor{gray}{\textit{#1}}}
%=================================================================
% Full title of the paper (Capitalized)
\Title{Where did you get that? \\Towards Summarization Attribution for Analysts\footnote{An earlier version of this report appeared in the \textit{SCADS 2023 Proceedings}.}}
% MDPI internal command: Title for citation in the left column
\TitleCitation{Title}
% Author Orchid ID: enter ID or remove command
\newcommand{\orcidauthorA}{0000-0000-0000-000X} % Add \orcidA{} behind the author's name
%\newcommand{\orcidauthorB}{0000-0000-0000-000X} % Add \orcidB{} behind the author's name
% Authors, for the paper (add full first names)
\Author{Violet B, John M. Conroy$^{1}$, Sean Lynch$^{2}$, Danielle M, Neil P. Molino$^{1}$, Aaron Wiechmann$^{2}$, and Julia S. Yang$^{1}$}
%\longauthorlist{yes}
% MDPI internal command: Authors, for metadata in PDF
\AuthorNames{Violet B, John M. Conroy, Sean Lynch, Danielle M, Neil P. Molino, Aaron Wiechmann, and Julia S. Yang}
% MDPI internal command: Authors, for citation in the left column
\AuthorCitation{Lastname, F.; Lastname, F.; Lastname, F.}
% If this is a Chicago style journal: Lastname, Firstname, Firstname Lastname, and Firstname Lastname.
% Affiliations / Addresses (Add [1] after \address if there is only one affiliation.)
\address{%
$^{1}$ \quad IDA Center for Computing Sciences, Bowie, MD, USA; \{conroy,npmolin,jsyang\}@super.org\\
$^{2}$ \quad Laboratory for Analytic Sciences, Raleigh, NC; \{sclynch,awiechm\}@ncsu.edu
}
% Contact information of the corresponding author
%\corres{Correspondence: e-mail@e-mail.com; Tel.: (optional; include country code; if there are multiple corresponding authors, add author initials) +xx-xxxx-xxx-xxxx (F.L.)}
% Current address and/or shared authorship
%\firstnote{Current address: Affiliation 3.}
%\secondnote{These authors contributed equally to this work.}
% The commands \thirdnote{} till \eighthnote{} are available for further notes
%\simplesumm{} % Simple summary
%\conference{} % An extended version of a conference paper
% Abstract (Do not insert blank lines, i.e. \\)
\abstract{
Analysts require attribution, as nothing can be reported without knowing the source of the information. In this paper, we will focus on automatic methods for attribution, linking each sentence in the summary to a portion of the source text, which may be in one or more documents. We explore using a hybrid summarization, i.e., an automatic paraphrase of an extractive summary, to ease attribution. We also use a custom topology to identify the proportion of different categories of attribution-related errors.}
% Keywords
%\keyword{keyword 1; keyword 2; keyword 3 (List three to ten pertinent keywords specific to the article; yet reasonably common within the subject discipline.)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
Automatic text summarization methods have been studied for over sixty-five years \cite{10.1147/rd.22.0159} and have shown promise as a content selection and filtering tool. Evaluations from the National Institute of Standards (NIST), such as the Document Understanding Conferences (DUC) and Text Analysis Conferences (TAC), help scope the problem and advance techniques in text summarization. The methods developed in the time frame of the conferences 2001-2011 were largely extractive-based methods, i.e., algorithms for finding critical portions of one or more documents, typically sentences, which best served the user's need for a summary.
In the last six years, significant progress in abstractive summarization has been made with the advent of recurrent neural networks for abstractive summarization \cite{journals/corr/SeeLM17}, and then transformer models. The transformer model can encode or decode text to generate summaries \cite{liu-lapata-2019-text} \cite{lewis-etal-2020-bart}. The limits of transfer learning to perform multiple natural language processing (NLP) tasks were extended by \cite{T5:10.5555/3455716.3455856} to tackle dozens of tasks. OpenAI extended such approaches \cite{chatgpt}, and many research laboratories produce fluent text and multi-modal models (including images, speech, etc.)
We focus on automatic methods for attribution, i.e., linking each sentence in the summary to a portion of the source text, which may be in one or more documents.
Given that large language models too often \emph{hallucinate}, i.e., make statements that are either not supported by the text or even contradicted by the text, the problem of automatic attribution methods is particularly daunting.
Key to our approach is a fast extractive summarization method, \texttt{occams}~\cite{analytics2030030}, which produces extracts with strong content coverage. The \texttt{occams} summary, typically less than 1K tokens, is then paraphrased with an abstractive summarizer. In contrast to this \emph{hybrid} approach, the abstractive-only method requires applying the neural abstractive model to the entire set of input documents, increasing the computational cost and complicating the attribution problem.
Automatic methods can link summary sentences with likely attributing sentences in the source documents. To test the accuracy of these methods, we employ human evaluation. For each sentence in the abstract, one or more putative supporting sentences are presented to a human to judge if these sentences support, partially support, contradict, partially contradict, or are neutral.
To test if the hybrid approach to summarization makes the problem of attribution easier, we compare it to an abstractive method on three datasets representing a range of information tasks for an analyst. We describe these in section \ref{sec:data}.
We note that the recent paper ``Attribute First, then Generate'' by \citet{slobodkin-etal-2024-attribute} takes an alternate approach of identifying relevant source segments before conditioning the generation process. This approach like our hybrid improves the citation accuracy. Also, \citet{worledge2024extractive} introduce the extractive-abstractive spectrum, with search engines and large language models as extremes with multiple intermediate approaches. The authors found ``...we find that perceived utility improves by as much as 200\%, while the proportion of properly cited sentences decreases by as much as 50\% and users take up to 3 times as long to verify cited information.''
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Summarization Algorithms}
In this experiment, we evaluate two automatic summary methods: abstractive and hybrid summarization. While both ways utilize GPT-3.5 Turbo (GPT in the rest of this paper), we use different inputs and prompts to change how the summaries are generated. The abstractive method will use GPT on the entire document to generate the final summary, relying on how the model itself was trained to handle summarizing documents. The hybrid approach will use \texttt{occams} to extract the most important text from the document. Then GPT will be used to rewrite the resulting paragraph into a more human-readable final summary. This section will outline how we execute each of these methods.
\subsection{Abstractive Summarization}
To generate our abstractive summaries, we will provide basic instructions for GPT. This experiment compares a hybrid summary with a purely abstractive summary, so we want to allow GPT to behave according to its training. While some recent research has investigated the impact of different prompting styles on the quality and factuality of summaries generated by large language models, we are seeking to evaluate the summaries that the model generates without refinement.
With these goals in mind, we used the following prompt to generate the abstractive summaries for all three datasets:
{\fontfamily{qcr}\selectfont
\noindent``````
\noindent You are an abstractive summarizer that follows the output pattern:
\noindent Text:
\noindent \{text\}
\noindent Summary:
\noindent ''''''
}
We specify an output pattern to ensure no extraneous references to the source text or other words beyond the expected summary. Also, it is essential to note that we do not constrict the length of the abstractive summary, allowing GPT to make it as long as it would by default. With that, we do generate these summaries early in the pipeline because the lengths of these summaries will influence the length we specify to \texttt{occams}.
One consideration we made when generating these summaries is the token length of our input. To prevent over-complicating our pipeline, we sought to abstractively summarize each document all at once in a single API call to GPT. While this was possible for the DUC/TAC and Cyber Threat Intelligence datasets, given the length of their documents, CrisisFACTS had some events with too many documents to process all at once. To reduce the number of documents for each event to a number that can be passed to GPT, we filtered the documents by their provided importance score and removed duplicates. This way, we could ensure that we had enough tokens to process the entire summary without sacrificing the most essential information.
\subsection{Hybrid Summarization}
We generate the hybrid summaries for each document via a two-step process, with the first step being carried out using the \texttt{occams} package while the second is a rewrite using GPT.
\subsubsection{OCCAMS Extractive Summarization}
Unlike GPT, \texttt{occams} requires a word or character budget to generate summaries. We need to use a different character budget for our experiment depending on the dataset we are working with. This is because we want to be able to compare the hybrid and abstractive summaries, and length is a major contributing factor when determining how much of the information from the source text can be represented within the summary.
We will determine this character budget for the DUC/TAC and Cyber Threat Intelligence datasets based on the abstractive summaries from GPT. Because these summaries are of varying lengths, we decided to set the budget at the 75th percentile of the length of all abstractive summaries for each dataset. We decided to go for this instead of the median length because \texttt{occams} is very rigid with its budget and will not exceed it. It will get very close to this target length for the summary, with minimal variation. There is a lot of variation within the abstractive summary lengths, and we must ensure that outliers will not impact the length we choose for our extractive summaries.
For CrisisFACTS, the nature of the dataset does not require as much consideration for the length of the summary. Given that the discrete documents in the dataset tend to be much shorter, we expect that our extractive summary will pull individual documents to construct our fact summary. Additionally, our gold-standard summaries are already extractive and represent an ideal extractive summary. Consequently, we use the median summary from the facts to generate our extractive summaries.
Upon determining the character budget, we can generate the summaries for each dataset using \texttt{occams}. This will be directly used in the next step of our pipeline to determine our final hybrid summary.
\subsubsection{GPT Summarization}
We will use GPT to rewrite our abstractive summaries for the second step in our hybrid summarizer. Given that the advantages of abstractive summarization over extractive summarization are mostly related to readability and human preference, we decided to use a different prompt than that used for abstractive summaries. This is because we seek to rewrite our extractive summaries, generating a more cohesive paragraph rather than a collection of sentences. With that, we want to make sure that no additional information is being added or hallucinated by GPT. With this goal, we used the following prompt:
{\fontfamily{qcr}\selectfont
\noindent``````
\noindent Please rewrite the following into a coherent and readable paragraph. Do not deviate from the facts of these sentences or add any new information. Follow the output pattern:
\noindent Text:
\noindent \{text\}
\noindent Summary:
\noindent ''''''
}
Like the abstractive prompt, we provide an output format to ensure the generated summary has no extraneous words or sentences. We specify that no information should be added. It is also important to note that we are still not concerned with length and do not expect the final summary to be shorter than the extractive summary. The output returned by this prompt will be our final hybrid summary used for our evaluation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Three Exemplar Datasets}
\label{sec:data}
To sample a range of content, we performed our attribution study on three open datasets, which mimic specific analysts' needs. First, the CrisisFACTS data set\footnote{CrisisFACTS (https://crisisfacts.github.io) is a track at the NIST Text Retrieval Conference (TREC) meant to mimic the needs of crisis managers from the Federal Emergency Management Agency as they monitor an ongoing disaster, such as a fire or flood.} contains extracted content snippets (from Twitter, Reddit, and news articles) that are relevant to the current query. Second, a Cyber Threat Intelligence data set prepared by Elemendar for the Laboratory for Analytic Sciences (LAS) represents a specialist requirement, in this case, analysts who keep up with cyber threat activity. Third, the Text Analysis Conference (TAC) 2011 data set is a multi-document summarization data set of newswire documents where the documents are grouped by topic.
The CrisisFACTS dataset from the Text Retrieval Evaluation Conference (TREC) 2022 and 2023 is a good model of a task-based summary where an analyst, in this case, one working for FEMA, has a requirement to write a report on an ongoing disaster such as a fire or flood. For the 2022 dataset, there are eight events. Multiple report requests were made for these events, with the total number of reports (summaries) being 55. As some of the summaries and retrieved relevant documents were small, we filtered the EventID to remove those with too few facts or too many.\footnote{In particular, we filtered the data so that min\_sum\_len=200, max\_sum\_len=5000, min\_doc\_len=100, where length is measured in characters.} An EventID labels each report request. For each EventID, there is a list of facts, loosely an extracted summary for the period of the given EventID. Note there is an abstractive summary for each of the eight events from Wikipedia. We opted not to use these in our study.
The Cyber Threat Intelligence dataset was explicitly created for SCADS by Elemendar. These data contain reports on cyber events. Each document includes a summary. The document set consists of 59 training documents and 15 tests. We will sample the training data for our initial study. In addition to the documents, there are knowledge graphs and some sample questions and answers, which could be used for future content evaluation and fact-checking.
The Text Analysis Conference (TAC) 2011 dataset is a classic multi-document summarization task by the National Institute of Technology and Standards (NIST). The task of interest from 2011 is the guided summary task, which was to perform a multi-document summary of 10 documents on a given topic. The dataset has 45 such topics and an updated summary of each topic. For each topic, there are four human-written summaries.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Automatic Attribution}
SummaC is by no means perfect for attribution, as experimentation indicates the metric has an underlying bias toward extractive summaries. During most of our experiments where we computed scores for abstractive, hybrid, and extractive summaries, SummaC consistently scored extractive summaries higher than any other type. We believe this is due to the Natural Language Inference (NLI) used initially in the metric pipeline to score the entailments of document-summary sentence pairs, as these probabilities are run through the model to output the score.
After performing a document reduction experiment on every summary type where we ordered the document sentences by ascending neutrality, reduced the original document by removing the most neutral sentence, and recomputing the SummaC score, we found that only a few document sentences influenced the SummaC score for extractive summaries. Notably, the exact number of document sentences matching the total summary sentences contributed most heavily to the score, implying that source text facts are not evenly distributed among extractive summary sentences. On the other hand, for abstractive and hybrid summaries, more document sentences were relevant to the score. This indicated that source text facts are more evenly distributed for hybrid and abstractive. Because of this even distribution, no single summary sentence is highly entailed by a document sentence, so the entailment probabilities are typically low. By extension, the SummaC score is very low, but this output doesn’t necessarily mean the abstractive or hybrid summary is bad. Instead, SummaC is bad at capturing summary text's abstract nature. As a result, using solely SummaC as the attribution solution for an entire summary would cause potential unwanted disqualification of summaries from being marked as consistent with the original document.
For this human evaluation experiment, we propose using NLI for sentence-level attribution as opposed to the SummaC metric for the entire summary. Because NLI produces low entailment probabilities for abstractive and hybrid summary sentences, we need to account for any missing context for a single abstracted summary sentence. To do this, we combine a targeted document sentence with a previous and following sentence to provide a range of context. Then we present this three-sentence text to our human evaluators for attribution labeling.
To test the ability of automatic methods for attribution, we use a natural language inference (NLI) model to find good matching sentences for each sentence in the GPT and Hybrid summaries. Following \cite{laban-etal-2022-summac}, we use the NLI \texttt{roberta-large-mnli} model. We also use a T5 sentence embedding cosine similarity for a baseline, with the \texttt{sentence-t5-xxl} model.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Human Labeling Experimental Design}
\label{sec:experiment}
Our evaluation methods are motivated by a recent paper \cite{liu2023evaluating}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}
The three summary types are human-written and two machine-generated summaries.
The machine-generated methods are purely abstractive using OpenAI GPT-3.5 Turbo model
\texttt{gpt-3.5-turbo-16k-0613} (GPT) and hybrid, which is a paraphrase of an extractive summary generated by \texttt{occams}~\cite{analytics2030030} (hybrid). We note that the hybrid method is a paraphrased extract, and the extract's length was chosen to be comparable with the GPT summary to make the comparison easier. The input limit of the GPT model is 16K tokens, which generally allows GPT to summarize the entire document without repeatedly applying it. The only exception is the CrisisFACTS data.
\subsection{Content-Based}
We compare the two summarization approaches, GPT and hybrid, and benchmark their content coverage on the three datasets. Here, we measure the agreement of human-written summaries with the two automatically generated methods. To this end, we use both ROUGE \cite{lin-2004-rouge} and a recently proposed measure SMART \cite{amplayo2023smart} as a check to indicate how closely the automatic summaries match the human-written ones. Both these metrics correlate well with human assessment of content coverage. Table~\ref{tab:RS} gives the mean scores of these automatic evaluations. Broadly, the two summarization methods perform comparably regarding their content similarity with the human summaries for each dataset. Each dataset has about 50 documents (or document sets in the case of TAC 2011). In most instances, the differences are insignificant, meaning that both methods likely have comparable quality content.%\footnote{In follow-on work, we plan to conduct a human evaluation for content to confirm the automatic evaluation results.}
\begin{table}
\begin{center}
\begin{tabular}{l|c|c|c|c}
Dataset &
\multicolumn{2}{c|}{GPT} &
\multicolumn{2}{c}{Hybrid} \\
& {ROUGE-2} & {SMART-2} & {ROUGE-2} & {SMART-2}\\
\hline
CrisisFACTS & 3.37 & 15.3 & 3.90 & 15.3 \\
Cyber Data & 10.2 & 24.9 & 9.3 & 22.9 \\
TAC 2011 & 11.5 & 24.4 & 10.4 & 24.5 \\
\end{tabular}
\end{center}
\caption{Automatic Evaluation Scores.} \label{tab:RS}
\end{table}
\subsection{Attribution}
We now turn to the question of automatic attribution. We evaluated two approaches for attribution, one based on an NLI model and the second using a T5 sentence embedding. We were uncertain how important the context would be for allowing the analysts to judge if the automatic attribution gave any support or contradicted the summary sentence. To this end, we considered two approaches when presenting the sentences in the document that the NLI or embedding model chooses for attribution. We called these two approaches Task 1 and 2 in the study.
In Task 1, the attributing document sentence is presented in context with the sentence preceding and following it. In Task 2, we opted to allow the automatic attribution methods to give three supporting sentences, as some summary sentences may be a fusion of two or more sentences. Figures \ref{Task1:human} and \ref{Task2:human} give pie charts summarizing the results of the three analysts' annotations of 30 summary sentences to attribution text pairings for the NLI and embedding approaches. Here, the summaries are the human-generated summaries from TAC 2011. We note that in both tasks, the sentence embedding method gave attributions that the analysts judged better than the NLI model. Also, it appears that giving three possible supporting sentences gave more accurate attribution.
We next compared summaries generated by GPT on the TAC 2011 data with a hybrid approach where we asked GPT to paraphrase \texttt{occams} extracts. To limit the amount of annotation, we decided only to use the embedding model and the Task 2 approach of providing the top three sentences for the automatic attribution evaluation. In Figure~\ref{Task2:machine}, the hybrid summaries were easier to attribute automatically. Note that the attribution sentences for the hybrid were selected from the extracted summary. As the summaries are approximately 100 words long, the analysts were given a significant portion of the original extract, typically about 5 sentences.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Task1_NLI.png}
\includegraphics[height=5cm]{Figures/Task1_emb.png}
%\end{adjustwidth}
\caption{Using embeddings in Task~1 gives stronger automatic attribution as judged by analysts compared to the NLI model.\label{Task1:human}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Task2_NLI.png}
\includegraphics[height=5cm]{Figures/Task2_emb.png}
%\end{adjustwidth}
\caption{Using embeddings in Task~2 gives stronger automatic attribution as judged by analysts compared to the NLI model.\label{Task2:human}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/TAC2011_GPT.png}
\includegraphics[height=5cm]{Figures/TAC2011_Hybrid.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the TAC 2011 dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{Task2:machine}}
\end{figure}
For the Cyber Threat Intelligence data, we conducted a similar experiment. We generated GPT summaries and hybrid summaries (GPT paraphrases of \texttt{occams} extracts). As the Cyber data might be consumed by both a general audience and experts in cyber security, we separated the annotation into two groups. The first group with general knowledge is labeled as ``analysts,'' and the second group with expertise in cyber security is labeled as ``experts.'' Figures \ref{Cyber:analysts} and \ref{Cyber:experts} give the results. In both cases, the hybrid approach was more likely to provide attribution with full support. Of note is that the cyber experts judged the attributions to be of higher quality. Specifically, the cyber experts labeled the attributions as giving ``full support'' more often than analysts without a background in cyber security.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Cyber_GPT_Analysts.png}
\includegraphics[height=5cm]{Figures/Cyber_Hybrid_Analysts.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the Cyber dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{Cyber:analysts}}
\end{figure}
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/Cyber_GPT_Experts.png}
\includegraphics[height=5cm]{Figures/Cyber_Hybrid_Experts.png}
%\end{adjustwidth}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the Cyber dataset gives rise to stronger automatic attribution as judged by cyber experts compared to using GPT alone.\label{Cyber:experts}}
\end{figure}
For the CrisisFACTS automatic attribution results (Figure~\ref{CrisisFACTS}), we see again the hybrid summary sentences had better automatic attribution than those using GPT without the \texttt{occams} extracts. As this data set was the least fluent of the three, it is not surprising to see that there was a drop in the fraction of attributions that received \emph{full support} relative to the results in Cyber and TAC datasets. For example, with the CrisisFACTS hybrid summaries, 58.9\% of the sentences had full support attributions. In contrast, with the TAC and Cyber datasets, the corresponding summaries had 74\% full support or more.
\begin{figure}[H]
%\begin{adjustwidth}{-\extralength}{0cm}
\centering
\includegraphics[height=5cm]{Figures/CrisisFACTS_GPT.png}
\includegraphics[height=5cm]{Figures/CrisisFACTS_hybrid.png}
\caption{Using Hybrid (\texttt{occams}/GPT) summaries on the CrisisFACTS dataset gives rise to stronger automatic attribution as judged by analysts compared to using GPT alone.\label{CrisisFACTS}}
\end{figure}
\subsection{Refutation}
Finally, we tally the percentage of refutations found in each experiment. Table~\ref{tab:refT12} gives the percentage of refutations for Task~1 and 2. Here, the embedding attribution method is more likely to link to a refuting sentence than the NLI approach. %Further analysis is needed here, but one potential explanation is that the embedding method found information from an earlier time period e.g., the number of fatalities increased in a disaster.
\begin{table}[H]
\centering
\begin{tabular}{c|c|c}
&NLI & Embedding \\
\hline
Task 1 & 0.0 & 10\\
Task 2 & 7.0 & 10\\
\end{tabular}
\caption{Percentage refutations for Task 1 and 2.}
\label{tab:refT12}
\end{table}
Table~\ref{tab:hybridVSgpt} compiles the remaining refutation results for the TAC and Cyber datasets. Here, the hybrid approach had fewer refutations than the GPT purely abstractive method except for the CrisisFACTS data where they both had about 8.9\% refutations.
\begin{table}[H]
\centering
\begin{tabular}{c|c|c}
& Hybrid & GPT\\
\hline
TAC 2011 & 5.6 & 13.3\\
Cyber Analysts & 7.8 & 14.4 \\
Cyber Experts & 8.9 & 10.0 \\
CrisisFACTS & 8.9 & 8.9 \\
\end{tabular}
\caption{Percentages of refutations in all annotations comparing GPT and hybrid approaches.}
\label{tab:hybridVSgpt}
\end{table}
An in-depth analysis of the refutations follows.
\subsection{Refutation: In-Depth}\label{sec:refutation-analysis}
Three data sets -- TAC 2011 (multi-document over time), Cyber Threat Intelligence (single document, specialist) and CrisisFACTS (multi-stream, closest to raw data) -- were analyzed for different types of refutations. The findings were as follows:
\begin{itemize}
\item Allowing 3 attributing sentences instead of 1 improved support significantly, but has the potential to add sentences of lower quality.
\item Refutations often differ from LLM hallucinations in a machine summary.
\item The types of refutations reflect the underlying data. TAC 2011 and CrisisFACTS refutations include timing errors, where attributing sentences refer to different periods of time when facts differed, but Cyber Threat Intelligence did not. Attributing sentences for Cyber Threat Intelligence refutations were more likely to be about related, rather than directly relevant, topics for lower-ranked attributions. CrisisFACTS refutations were of a variety of types and included issues with ambiguity and parsing.
\item Parsing problems due to poorly formatted source text can lead to meaningless attributing sentences.
\item Annotators may differ on what was considered a refutation.
\end{itemize}
Details follow, starting with a new refutation typology based on an existing typology for factual errors in machine summaries from ``Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics'' by Artidoro Pangoni, Vidhisha Balachandran, and Yulia Tsetkov at NAACL 2021~\cite{pagnoni-etal-2021-understanding}. We explore TAC~2011 with NLI and sentence embedding attribution methods on human summaries for Task~1 (1 attributing sentence with preceding and following sentences for context) and Task~2 (3 attributing sentences), followed by an analysis of all three data sets with the sentence embedding attribution method on abstractive and hybrid machine summaries for Task~2.
\subsubsection{Typology of Factual Errors (Refutation)}
The typology of factual errors for abstractive summaries in~\cite{pagnoni-etal-2021-understanding} was modified for the refutation analysis as follows:
\begin{itemize}
\item Semantic Frame Errors:
\begin{itemize}
\item Predicate Error (PredE) : The predicate in the summary statement is inconsistent with the attribution(s).
\item Entity Error (EntE): The primary arguments (or their attributes) in the summary statement are inconsistent with the attribution(s).
\item Circumstance error: (CircE) Additional information (such as location or time) specifying the circumstance around a predicate in the summary statement is inconsistent with the attribution(s).
\end{itemize}
\item Content Verifiability Errors
\begin{itemize}
\item Out of Article Error (OutE): The summary statement contains information not present in the attribution(s); this includes extrinsic hallucinations.
\item Grammatical Error (GramE): The grammar mistakes in an \textit{attribution} is so wrong that it becomes meaningless.
\end{itemize}
\item Additional Categories
\begin{itemize}
\item Others (OthE): Refutation errors that do not correspond to any of the above categories.
\item Not an Error (NE): There are no refutation errors.
\end{itemize}
\end{itemize}
Examples from~\cite{pagnoni-etal-2021-understanding}, modified for refutations can be found in Table~\ref{table:refutation-examples}. The first 4 (PredE, EntE, CircE, OutE) are examples of \textit{summary} statements where errors are highlighted in red. The last (GramE) is an example where the grammar in the \textit{attribution} is so wrong that it is meaningless.
\begin{table}
\small
\centering
\begin{tabular}{llp{.3\textwidth}p{.3\textwidth}}
& \bf{Category} & \bf{Description} & \bf{Example}\\
\bf{PredE} & Predicate Error & The predicate in the summary statement is inconsistent with the attribution(s). & The Ebola vaccine \textcolor{red}{was rejected} by the FDA in 2019 [1].\\
\bf{EntE} & Entity Error & The primary arguments (or their attributes) in the summary statement are inconsistent with the attribution(s). & The \textcolor{red}{COVID-19 vaccine} was approved by the FDA in 2019 [1],[3]. \\
\bf{CircE} & Circumstance Error & Additional information (like location or time) specifying the circumstance around a predicate in the summary statement is inconsistent with the attribution(s). & The first vaccine for Ebola was approved by the FDA in \textcolor{red}{2014} [1]. \\
\bf{OutE} & Out of Article Error & The summary statement contains information not present in the attribution(s). & \textcolor{red}{China} has started clinical trials of the COVID-19 vaccine [3].\\
\bf{GramE} & Grammatical Error & The grammar of the \textit{attribution} is so wrong that it becomes meaningless. & The vaccine for Ebola \textcolor{red}{approved have already started}.
\end{tabular}
\caption{Examples of refutation errors, classified using the refutation typology. Source text sentences: \textit{[1]: The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. [2]: To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. [3]: Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started.} An additional attributing sentence which is meaningless: \textit{[4]: The vaccine for Ebola approved have already started.}} \label{table:refutation-examples}
\end{table}
\subsection{Human: TAC 2011: Task 1 versus Task 2}
We evaluated refutations for the NLI and T5 sentence embedding attribution methods on human-generated summaries from TAC 2011. Two tasks were examined. In Task~1, the attributing document sentence is presented in context with the sentence preceding and following it. In Task~2, the automatic attribution methods give three supporting sentences, since some summary sentences may be the fusion of two or more sentences. Figure~\ref{fig:refutation-tac2011-human-bar} provides bar charts summarizing the refutation results of three analysts' annotations of 30 summary sentences to attribution text pairings with the NLI and embedding automatic attribution methods (60 total pairings for TAC 2011).
\begin{figure}[H]
\centering
\includegraphics[width=0.8\linewidth]{./Figures/TAC2011_Human_BarCharts.png}
\caption{Refutation Error Counts for Human Summaries: TAC 2011: Task 1 and Task 2.}\label{fig:refutation-tac2011-human-bar}
\end{figure}
For Task~1, the 2 semantic refutations were time-shift errors, where the summary contained information inconsistent with an attribution because the attribution referred to a different period of time when fact(s) differed. The 8 content refutations were due to the summary containing information not found in the attribution. All refutations were for the embedding attribution method. The first annotator labeled 6 refutations (4 unique), the second labeled 3 refutations (1 unique), and the third labeled 0 (0 unique). See Table~\ref{tab:human-task-1} for more details.
The refutation profile changed for Task~2. The 4 semantic refutations were mostly but not entirely due to time-shift errors; for example, a summary and the highest ranked attributing sentence stated that the ``soles of shoes'' are considered the ultimate insult in Arab countries, while lower-ranked attributing sentences stated that ``throwing a shoe at someone'' or ``hitting someone with a shoe'' was the worst insult. The 4 content refutations were due to the summary containing additional information not found in the attributions (3) or bad attribution grammar (1). The 6 additional refutations were caused by lower-ranked attributing sentences not being directly relevant to the summary, as those attributions referenced related or unrelated topics instead. There were 5 NLI and 8 embedding refutations. The first annotator labeled 2 refutations (1 unique), the second labeled 12 refutations (9 unique) and the third labeled 0 (0 unique). It was unclear why one example was marked as a refutation and that example was not included in the counts. See Table~\ref{tab:tac2011-human-task2} for more details.
Combining the attribution and refutation analysis for TAC 2011 human summaries (Task~1 and Task~2), it appears that allowing 3 attributing sentences provided more accurate attribution but potentially add sentences of lower quality.
(Note that counting differs for the attribution pie charts versus the refutation bar charts: a summary sentence with its associated attributing sentence(s) are enumerated in the pie charts as often as the annotators mark it for that category, while counts have been deduplicated within a category for the typography-based bar charts.)
\subsection{Machine (Task~2): TAC 2011, Cyber, CrisisFACTS}
Next, we compared summaries generated by GPT with a hybrid approach where we asked GPT to paraphrase \texttt{occams} extracts. To limit the amount of annotation, we only evaluated the embedding attribution method and Task~2 approach of providing the top three attributing sentences. Figure~\ref{fig:refutation-tac2011-cti-crisisfacts-machine-bar} provides bar charts summarizing the refutation results of three analysts' annotations of 30 attribution text pairings for the TAC 2011, Cyber Threat Intelligence, and CrisisFACTS data sets with the abstractive and hybrid machine summarization methods (60 total pairings per data set).
\begin{figure}[H]
\centering
\includegraphics[width=0.8\linewidth]{./Figures/TAC2011_Cyber_CrisisFACTS_Machine_BarCharts.png}
\caption{Refutation Error Counts for Machine Summaries: TAC 2011, Cyber Threat Intelligence, CrisisFACTS: Task 2.}\label{fig:refutation-tac2011-cti-crisisfacts-machine-bar}
\end{figure}
TAC 2011 is a multi-document data set of fluent newswire articles over time, clustered by topic. There were 5 semantic, 4 content, and 3 additional/other refutations. The semantic refutations were due to time-shift errors, where the summary contained information inconsistent with an attributing sentence because the sentence referred to a different period of time when fact(s) differed. Two content refutations were due to summary hallucinations. In the first hallucination, the attributing sentence stated that a widow had been married to an alleged gunman for almost 10 years while the hybrid summary stated that the widow knew him for almost 10 years. In the second hallucination, 1 of 4 alleged terrorist attack masterminds were described in the abstractive summary as ``the'' mastermind. The remaining two content refutations were caused by a summary containing information not found in the attributions. The additional/other refutations were due to attributions containing related information.
There were 8 abstractive and 3 hybrid summary statements with attribution pairings that contained refutations. The first annotator labeled 5 refutations (4 unique), the second labeled 3 (1 unique), and the third labeled 8 (4 unique); it was unclear why one example was marked as a refutation and that example was not included in the counts. See Table~\ref{tab:tac2011-machine-task2} for more details.
Cyber Threat Intelligence is a single document, specialist data set. The annotators were divided into two groups: ``analysts'' for annotators with general knowledge and ``experts'' for annotators who were experts in cyber security. There were 2 semantic, 2 content, and 13 additional/other refutations in the analyst annotations, and 1 semantic, 5 content, and 12 additional/other refutations in the expert annotations. The Cyber data set had the highest proportion of additional/other refutations due to attributions containing related information. This data set contained complex technical sentences, and a summary sentence was more likely to summarize a smaller number of sentences from the source text.
There were no semantic time-shift errors: while a document may describe events over a time period, there was no additional complication from events split over different documents. Both groups identified a semantic refutation for a hybrid summary in which the LLM paraphrasing used stronger language (``highly likely'') compared to the source text (``likely'') to describe the probability of network compromise. The experts identified two summary hallucinations, while the analysts only identified one of the two summary hallucinations.
%One content refutation was due to a summary hallucination for technical reasons (a malware attack component was assigned to the incorrect stage) and another content refutation was due to a misleading summary because the attribution contained a technical detail that was missing in the summary (``incidents involving landing URLs'' versus ``incidents'').
There were 10 abstractive and 6 hybrid summary statements with attribution pairings that contained refutations in the analyst annotations. The first annotator labeled 16 refutations (13 unique), the second labeled 3 (0 unique), and the third labeled 0 (0 unique); it was unclear why one example was marked as a refutation and it is not included in the counts. There were 9 abstractive and 8 hybrid summary statements with attribution pairings that contained refutations in the expert annotations; 7 of those refutations were also marked as refutations by the analysts. The first annotator labeled 1 refutation, the second labeled 2, and the third labeled 14 refutations (including the two summary hallucinations). See Table~\ref{tab:cyber-machine-task2-analyst} and Table~\ref{tab:cyber-machine-task2-expert} for more details.
CrisisFACTS is a multi-stream data set and is closest to raw data of the three data sets. There were 6 semantic, 5 content, and 5 additional/other refutations. Content refutations were divided between 2 OutE refutations (the summary contained information not present in the attributions) and 3 GramE refutations (the grammar of an attributing sentence was so wrong that it became meaningless). TAC 2011 contains fluent newswire and was straightforward to parse into sentences for attribution. Cyber Threat Intelligence documents contained fluent text, lists of fluent text, and unformatted tables. An unformatted table was parsed as a very long sentence, and additional computing resources were required for the automatic embedding attribution method to accommodate very long sentences. Although cyber table-based sentences were not meaningful without formatting, these sentences did not appear in human evaluation. CrisisFACTS contained short text excerpts from multiple data streams (Twitter, Reddit, newswire). Text excerpts were not necessarily fluent text and might be truncated either in the original data set or afterwards in post-processing when multiple excerpts were concatenated and possibly truncated to fit within the LLM context window. CrisisFACTS was a challenging data set to automatically parse into ``sentences'' for attribution, and this was reflected in refutations where an attributing ``sentence'' was meaningless due to poor parsing and truncation.
The 6 CrisisFACTS semantic refutations were mostly due to time-shift errors leading to inconsistent information, with an additional semantic error due to additional location information in the summary being inconsistent with one of the attributing sentences. The additional/other refutations were due to attributions containing related information or the summary statement being so ambiguous that it was difficult to determine if the paired attributing sentences were relevant to the underlying disaster.
There were 7 abstractive and 5 hybrid summary statements with attribution pairings that contained refutations. The first annotator labeled 11 refutations (8 unique) the second labeled 4 (1 unique), and the third labeled 0 (0 unique); it was unclear why one example was marked as a refutation, and it is not included in the counts. See Table~\ref{tab:crisisfacts-machine-task2} for more details.
\section{Conclusions}
In this paper, we set up a series of experiments to measure how well automatic attribution methods worked for attributing information generated by a large language model. We first compared two automatic approaches, one based on a natural language inference model, trained on entailment data, and compared it with a baseline using a sentence embedding method. We found that the embedding model, which was the larger unsupervised model, provided better attribution. However, it had higher instances of refutations.
Next, we compared the embedding approach on two styles of summaries on three datasets: TAC 2011, a Cyber Threat Intelligence dataset, and the TREC 2022 CrisisFACTS dataset. Here, we found that the hybrid summary, a paraphrase of an extractive \texttt{occams} summary, generated text whose sentences were easier to attribute to information in the original documents for all three datasets.
Finally, we used a custom topology to identify the proportion of different categories of attribution-related errors. These refutations often differed from hallucinations in machine summaries. Proportions varied based on the underlying data, and parsing issues could lead to meaningless attributing sentences due to poorly formatted source text.
\vspace{6pt}
\section*{Acknowledgements}
The authors thank Ani Nenkova for her visit to the Summer Conference on Applied Data Science in 2023 and for her suggestions, which help initiate this work. We like to thank a colleague, who chose to remain anonymous for their critical contribution to this work. We also are very grateful to the analyst annotators and cyber experts for their evaluation of automatic attribution methods for three datasets.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% optional
%\supplementary{The following supporting information can be downloaded at: \linksupplementary{s1}, Figure S1: title; Table S1: title; Video S1: title.}
% Only for the journal Methods and Protocols:
% If you wish to submit a video article, please do so with any other supplementary material.
% \supplementary{The following supporting information can be downloaded at: \linksupplementary{s1}, Figure S1: title; Table S1: title; Video S1: title. A supporting video article is available at doi: link.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \authorcontributions{For research articles with several authors, a short paragraph specifying their contributions must be provided. The following statements should be used ``Conceptualization, X.X. and Y.Y.; methodology, X.X.; software, X.X.; validation, X.X., Y.Y. and Z.Z.; formal analysis, X.X.; investigation, X.X.; resources, X.X.; data curation, X.X.; writing---original draft preparation, X.X.; writing---review and editing, X.X.; visualization, X.X.; supervision, X.X.; project administration, X.X.; funding acquisition, Y.Y. All authors have read and agreed to the published version of the manuscript.'', please turn to the \href{http://img.mdpi.org/data/contributor-role-instruction.pdf}{CRediT taxonomy} for the term explanation. Authorship must be limited to those who have contributed substantially to the work~reported.}
% \funding{Please add: ``This research received no external funding'' or ``This research was funded by NAME OF FUNDER grant number XXX.'' and and ``The APC was funded by XXX''. Check carefully that the details given are accurate and use the standard spelling of funding agency names at \url{https://search.crossref.org/funding}, any errors may affect your future funding.}
% \institutionalreview{In this section, you should add the Institutional Review Board Statement and approval number, if relevant to your study. You might choose to exclude this statement if the study did not require ethical approval. Please note that the Editorial Office might ask you for further information. Please add “The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of NAME OF INSTITUTE (protocol code XXX and date of approval).” for studies involving humans. OR “The animal study protocol was approved by the Institutional Review Board (or Ethics Committee) of NAME OF INSTITUTE (protocol code XXX and date of approval).” for studies involving animals. OR “Ethical review and approval were waived for this study due to REASON (please provide a detailed justification).” OR “Not applicable” for studies not involving humans or animals.}
% \informedconsent{Any research article describing a study involving humans should contain this statement. Please add ``Informed consent was obtained from all subjects involved in the study.'' OR ``Patient consent was waived due to REASON (please provide a detailed justification).'' OR ``Not applicable'' for studies not involving humans. You might also choose to exclude this statement if the study did not involve humans.
% Written informed consent for publication must be obtained from participating patients who can be identified (including by the patients themselves). Please state ``Written informed consent has been obtained from the patient(s) to publish this paper'' if applicable.}
% \dataavailability{We encourage all authors of articles published in MDPI journals to share their research data. In this section, please provide details regarding where data supporting reported results can be found, including links to publicly archived datasets analyzed or generated during the study. Where no new data were created, or where data is unavailable due to privacy or ethical re-strictions, a statement is still required. Suggested Data Availability Statements are available in section “MDPI Research Data Policies” at \url{https://www.mdpi.com/ethics}.}
%\acknowledgments{In this section you can acknowledge any support given which is not covered by the author contribution or funding sections. This may include administrative and technical support, or donations in kind (e.g., materials used for experiments).}
%\conflictsofinterest{Declare conflicts of interest or state ``The authors declare no conflict of interest.'' Authors must identify and declare any personal circumstances or interest that may be perceived as inappropriately influencing the representation or interpretation of reported research results. Any role of the funders in the design of the study, in the collection, analyses or interpretation of data, and in the writing of the manuscecision to publish the results must be declared in this section. If there is no role, please state ``The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the~results''.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optional
%\sampleavailability{Samples of the coThe authors' mpounds ... are available from the authors.}
%% Only for journal Encyclopedia
%\entrylink{The Link to this entry published on the encyclopedia platform.}
%\abbreviations{Abbreviations}{
%The following abbreviations are used in this manuscript:\\
%\noindent
%\begin{tabular}{@{}ll}
%SCADS & Summer Conference on Applied Data Science\\
%LAS & Laboratory of Analytic Sciences
%\end{tabular}
%}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Optional
\appendixtitles{yes} % Leave argument "no" if all appendix headings stay EMPTY (then no dot is printed after "Appendix A"). If the appendix sections contain a heading, change the argument to "yes."
\appendixstart
\appendix
\section[\appendixname~\thesection]{Annotation Guidelines}
This appendix show sample annotation guidelines for TAC 2011. We ask annotators to read these guidelines as part of the human evaluation. The SCADS Cyber Threat Intelligence Data Set and CrisisFACTS guidelines were similar (3 attributing sentences for Task 2: Machine Summaries).
\subsection[\appendixname~\thesubsection]{TAC 2011: Task 1: Human Summaries}
\begin{myquotation}
\input{AnnotationGuidelines/tac-task1}
\end{myquotation}
\subsection[\appendixname~\thesubsection]{TAC 2011: Task 2: Machine Summaries}
\begin{quotation}
\input{AnnotationGuidelines/tac-task2}
\end{quotation}
%\subsection[\appendixname~\thesubsection]{CrisisFACTS: Task 1}
%\begin{quotation}
%\input{AnnotationGuidelines/facts-task1}
%\end{quotation}
%\subsection[\appendixname~\thesubsection]{CrisisFACTS: Task 2}
%\begin{quotation}
%\input{AnnotationGuidelines/facts-task2}
%\end{quotation}
\section[\appendixname~\thesection]{Refutation Analysis Tables}
Tables used for the refutation analysis in Subsection~\ref{sec:refutation-analysis} are provided in this appendix. Duplicates are shown in a gray font.
\input{RefutationAnalysis/refutation-analysis-tables}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{adjustwidth}{-\extralength}{0cm}
%\printendnotes[custom] % Un-comment to print a list of endnotes
\reftitle{References}
% Please provide either the correct journal abbreviation (e.g., according to the “List of Title Word Abbreviations” http://www.issn.org/services/online-services/access-to-the-ltwa/) or the full name of the journal.
% Citations and References in Supplementary files are permitted provided that they also appear in the reference list here.
%=====================================
% References, variant A: external bibliography
%=====================================
\bibliography{references_human_eval_attrib}
% If authors have biography, please use the format below
%\section*{Short Biography of Authors}
%\bio
%{\raisebox{-0.35cm}{\includegraphics[width=3.5cm,height=5.3cm,clip,keepaspectratio]{Definitions/author1.pdf}}}
%{\textbf{Firstname Lastname} Biography of first author}
%
%\bio
%{\raisebox{-0.35cm}{\includegraphics[width=3.5cm,height=5.3cm,clip,keepaspectratio]{Definitions/author2.jpg}}}
%{\textbf{Firstname Lastname} Biography of second author}
% For the MDPI journals use author-date citation, please follow the formatting guidelines on http://www.mdpi.com/authors/references
% To cite two works by the same author: \citeauthor{ref-journal-1a} (\citeyear{ref-journal-1a}, \citeyear{ref-journal-1b}). This produces: Whittaker (1967, 1975)
% To cite two works by the same author with specific pages: \citeauthor{ref-journal-3a} (\citeyear{ref-journal-3a}, p. 328; \citeyear{ref-journal-3b}, p.475). This produces: Wong (1999, p. 328; 2000, p. 475)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% for journal Sci
%\reviewreports{\\
%Reviewer 1 comments and authors’ response\\
%Reviewer 2 comments and authors’ response\\
%Reviewer 3 comments and authors’ response
%}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\PublishersNote{}
\end{adjustwidth}
% Comments from annotators:
% I think may have other comments. This is one that Aaron passed along for the CrisisFACTS and I wanted to capture it somewhere.
% "One interesting point this time around was that the summaries added adverbs that made me question if it was partial or full support. Here is an example: "
% [SUMMARY] The fire has been ongoing for about a week, and containment is slowly increasing.
% [**EVAL**] https://t.co/q5zLkZhbwl⋯ “I’m just thinking we turned the bend here,” U.S. Forest Service spokeswoman Vickie Wright said of the wildfire that had burned out of control for nearly a week.
% [**EVAL**] As the smoke began to clear above Lake Elsinore, firefighters took heart Sunday as they appeared to gain control of the Holy fire, with containment at 51 percent and 22,714 acres consumed.
% [**EVAL**] Calif.’s #HolyFire, Sees Growth In Size, However Containment Also Increases https://t.co/3vnIMuKApJ https://t.co/3B87t3UyGH⋯ Day 6 Of So.
% A comment for one of the expert cyber annotators and our response.
% On 10/11/24 10:40, Aaron Wiechmann wrote:
% > John,
% > Sounds good. Also, I received the following feedback from one of the labelers in case it is helpful to you.
% >
% > "One thing I wanted to note that jumped out to me a lot. It was pretty common to have the summary be an almost exact word for word copy of one of the eval statements, with only a 1-2 words changed or the structure slightly altered. The other two eval statements would be further elaboration or unrelated to the summary entirely. I'd say a good third of the labels were like this. Felt like it was cheating a bit by doing since its trying to support itself by just directly repeating itself. There were also a few summaries that were either lacking necessary context to properly judge the eval statements, or just straight statements that didn't really apply to eval (i.e "Please see x report" as a summary). Overall though, there usually was at least 1 eval statement that did directly support the summary"
% >
% > Cheers,
% > Aaron
% Response:
% Thank you, for the feedback. I wouldn't call that cheating, I call that paraphrasing, which is what we asked the model to do to the occams extracts!
% John
% As to the second point, we made the decision to use three sentences. For the hybrid summaries the attribution was limited to the original occams extract sentences. Often it would be the case that only 1 or 2 would really be supportive of each of the paraphrased summary sentences.
% In short, I appreciate the feedback, but it is what we expected. In an operational system I think the user interface would take the user back to the the orginal passages in corresponding to one or more citations. We wouldn't want to always give 3.
\end{document}
%%% TAC 2011 HUMAN TASK 1 TABLE %%%
\begin{table}[p]\small
\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.5cm}|p{1cm}|p{4cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annotator} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
13135 & Emb & 8d4ff & S & Content & OutE & Primary part of summary not present in attribution (=13245)\\ \hline
% China environmental issue
13181 & Emb & 8d4ff & N & Content & OutE & Summary contains information not present in attribution which is about a related topic \\ \hline
% Walter Reed controversy
13182 & Emb & 8f14c & U & Semantic & EntE & Time Shift led to inconsistent info (=13221) \\ \hline
% ferry sinking recovery efforts
13189 & Emb & 8d4ff & S & Semantic Content & CircE OutE & Time Shift led to inconsistent timing info; Summary contains info not in attributing sentence but in context \\ \hline
% liberian war crime trial
13190 & Emb & 8d4ff & S & Content & OutE & Timing information in summary not present in attribution \\ \hline
% Athens embassy attack
\gray{13221} & \gray{Emb} & \gray{8d4ff} & \gray{S} & \gray{Semantic} & \gray{EntE} & \gray{Time Shift led to inconsistent info (=13182)} \\ \hline % ferry sinking recovery efforts
13229 & Emb & 8d4ff & S & Content & OutE & Timing info stated differently (August versus Thursday) \\ \hline
% Mattel recall
\gray{13245} & \gray{Emb} & \gray{8f14c} & \gray{U} & \gray{Content} & \gray{OutE} & \gray{Primary part of summary not present in attribution (=13135)}\\ \hline
% China environmental issue
13258 & Emb & 8f14c & N & Content & OutE & Timing and supplemental location information in summary not present in attribution \\ \hline
% piracy seizure
\end{tabular}
\parbox[b]{4.85in}{}
\caption{TAC 2011: Human Summaries: Task 1: One Attributing Sentence with Context. L/Label: S=Support, N=No Support, U=Unclear.}
\label{tab:human-task-1}
\end{table}
%%% TAC 2011 HUMAN TASK 2 TABLE %%%
\begin{table}[h]\small
\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annotator} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
12981 & Emb & eed54 & P & Semantic & CircE & Time Shift error led to inconsistent timing info (2,3) \\ \hline
% south korea oil tanker accident
13022 & Emb & 415f6 & P & Content & OutE & Summary contains statistic not present in attributions\\ \hline
% china earthquake, stats (number dead) in news article but not in citations
13025 & Emb & 415f6 & F & Content Additional & GramE OthE & Bad attribution grammar (2); Attribution on related topic (3)\\ \hline
% internet redesign
13039 & NLI & eed54 & F & Semantic & PredE & Time Shift led to inconsistent info (3) (=13144)\\ \hline
% Menu Foods recall
13041 & NLI & 415f6 & F & Semantic & EntE & Attributions inconsistent but related (2,3) (=13046)\\ \hline
% Arabs and shoe soles insult
\gray{13046} & \gray{Emb} & \gray{415f6} & \gray{F} & \gray{Semantic} & \gray{EntE} & \gray{Attributions inconsistent but related (2,3) (=13041)}\\ \hline
% Arabs and shoe soles insult
13056 & NLI & 415f6 & F & Additional & OthE & Unrelated attribution (3)\\ \hline
% construction and missing people
13065 & Emb & 415f6 & F & Content & OutE & Summary statement contains additional timing information not present in attributions\\ \hline
% athens Embassy attack
13072 & Emb & 415f6 & F & Semantic & PredE & Inconsistent info, from time shift or different disaster (3)\\ \hline
% Bangledesh cyclone
13095 & Emb & 415f6 & P & Content Additional & OutE OthE & Summary contains info not present in attributions; Attribution for wrong location (3)\\ \hline
% SW China water shortage causes
13097 & NLI & 415f6 & F & Additional & OthE & Attribution unrelated (3)\\ \hline
% bombing in India
13098 & Emb & 415f6 & F & Additional & NE & Unclear why selected as refutation\\ \hline
% construction and missing people, same summary, different citations 13056
13104 & NLI & 415f6 & F & Additional & OthE & Attributions related (2,3) \\ \hline
% madoff lawsuit
13105 & Emb & 415f6 & F & Additional & OthE & Attributions related (2,3) \\ \hline
% madoff lawsuit, same summary, diff citations 13104
\gray{13144} & \gray{NLI} & \gray{415f6} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time Shift led to inconsistent info (3) (=13039)}\\ \hline
% menu foods, same summary, same citations 13039
\end{tabular}
\parbox[b]{4.85in}{}
\caption{TAC 2011: Human Summaries: Task 2: Three Attributing Sentences. L/Label: F=Full Support, P=Partial Support.}
\label{tab:tac2011-human-task2}
\end{table}
%%% TAC 2011 MACHINE TASK 2 %%%
\begin{table}[h]\small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annot.} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
13951 & Abstractive & 8d4ff & P & Content & OutE & Summary contains information not present in attributions but might be inferred\\ \hline
% children trama-> inflammation and disease
% summary about childhood impact, #2/#3 about adult disease from child trauma
13954 & Abstractive & 8d4ff & P & Semantic Content & CircE OutE & Time shift (1,2); Summary contained info not present in attributions\\ \hline
% Finnish PM expressed condolences, gov held emergency meeting
13957 & Abstractive & 8d4ff & P & Semantic & PredE & Time shift led to inconsistent info (2,3) \\ \hline
% Cyclone disaster
13961 & Abstractive & 8d4ff & F & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% Minneapolis bridge collapse with fatalities
13971 & Hybrid & 8d4ff & F & Content & OutE & Summary hallucination (1) (=14091)\\ \hline
% Amish children killed
% summary says "she had known for almost 10 years", source docs say married
% to for almost 10 years
\gray{14021} & \gray{Abstractive} & \gray{8f14c} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=13961, 14081)}\\ \hline
% Minneapolis bridge collapse with fatalities
14026 & Abstractive & 8f14c & F & Additional & OthE & Attribution contains related info (2)\\ \hline
% Chinese FM offers conolences to air crash victims
14045 & Hybrid & 8f14c & F & Semantic & PredE & Time shift led to inconsistent info (2) (=14105) \\ \hline
% Iraqi reporter throws shoes at Pres Bush, one citation says shoe
\gray{14077} & \gray{Abstractive} & \gray{36f2c} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (2,3) (=13597)}\\ \hline
% Cyclone related: summary evacuation, citation homelessness
\gray{14081} & \gray{Abstractive} & \gray{36f2c} & \gray{F} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=13961,14021)}\\ \hline
% Minneapolis bridge collapse = 14021
\gray{14091} & \gray{Hybrid} & \gray{36f2c} & \gray{P} & \gray{Content} & \gray{OutE} & \gray{Summary hallucination (1) (=13971)}\\ \hline
% Amish children killed, = 13971
14092 & Abstractive & 36f2c & F & Additional & OthE & Attribution same topic but different location (2) \\ \hline
% East Asian ivory anti-poaching education
14049 & Abstractive & 36f2c & F & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% Albarez decision - death or life in prison?
14102 & Abstractive & 36f2c & F & Content & OutE & Summary hallucination (summary used ``the'' versus 1 of 4)\\ \hline
\gray{14105} & \gray{Hybrid} & \gray{36f2c} & \gray{F} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (2) (=14045)}\\ \hline
% Iraqi reporter throws show at Pres Bush = 14045
14108 & Abstractive & 36f2c & F & Additional & NE & Unclear why selected as refutation \\ \hline
% Chinese computer virus arrest
14115 & Hybrid & 36f2c & F & Additional & OthE & Attributions contain related information (2,3)\\ \hline
% Lack of sleep -> weight gain
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{TAC 2011: Machine Summaries: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support.}
\label{tab:tac2011-machine-task2}
\end{table}
%%% CTI ANALYST MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
17040 & Abstractive & P & 73655 & Content & OutE & Summary hallucination (=17160)\\ \hline
% Inside TDrop2: Technical Analysis of new Dark Seoul Malware
% Source document: (Stage 4) "It proceeds to randomly select an
% executable file in the system32 folder, and performs process
% hollowing against it."
% Summary: In the fourth stage, the malware checked for parent processes
% and performed process hollowing again.
17041 & Abstractive & P & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 (=17161)\\ \hline
% Unit 42 has observed historical scans using IP addr shared by Volexity
% All info in citation 1, 2/3 related
17043 & Hybrid & P & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% VirusTotal graph. All info in citation 1, 2/3 related.
17044 & Hybrid & F & 73655 & Additional & OthE & Closely related information (2), related information (3), Summary $\approx$ Attribution~1\\ \hline
% Certificate-based attack described.
% Citation 1 is almost copy of summary sentence. Citation 2 describes vuln.
17047 & Abstractive & P & 73655 & Additional & OthE & Related information (2)\\ \hline
% Roshan_VC.iso, uses Brute Ratel C4 malware
% (2) related info
17048 & Abstractive & F & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% CVE-2022-26809
% Citation 1 ~ summary
17049 & Hybrid & P & 73655 & Semantic Additional & PredE OthE & Summary close to Attribution~1 but uses stronger language (``very likely'' versus ``likely''); Related information (2,3)\\ \hline
% Iranian govt network compromise
% Citation 1 ~ summary
17053 & Abstractive & F & 73655 & Additional & OthE & Related information (3)\\ \hline
% invest in awareness training, detection tool to reduce callback phish risk
% citations 1 and 2 provide full support and additional information
17056 & Hybrid & P & 73655 & Additional & OthE & Related information (3)\\ \hline
% JsonWebToken patch information
17057 & Abstractive & P & 73655 & Content & OutE & Summary contains info not present in attributions\\ \hline
% list of products to detect and prevednt Conti ransomware, subset of list in citations (due to parsing?)
17061 & Abstractive & P & 73655 & Additional & OthE & Related information (3) (=17181)\\ \hline
% Iranian govt network compromise, related 17049
% Summary: Playful Taurus continues to evolve its tactics...
17063 & Hybrid & F & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% VMWare advisory
17065 & Hybrid & P & 73655 & Additional & NE & Unclear why selected as refutation\\ \hline
% technical description malware
17066 & Abstractive & P & 73655 & Additional & OthE & Closely related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% ML to detect network abuse
17072 & Hybrid & F & 73655 & Additional & OthE & Closely related information (2), related information (3), Summary $\approx$ Attribution~1\\ \hline
% AWS phishing attack
17073 & Abstractive & P & 73655 & Semantic & EntE & Closely related information (2,3), Summary $\approx$ Attribution~1 but summary entity lacks sufficient information (``incidents'' versus ``incidents involving landing URLs'')\\ \hline
% URL-based attack analysis
17083 & Abstractive & P & 73655 & Additional & OthE & Closely related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% custom URI checksum to identify attack
\gray{17160} & \gray{Abstractive} & \gray{N} & \gray{8f14c} & \gray{Content} & \gray{OutE} & \gray{Summary hallucination (=17040)}\\ \hline
% 4th stage process hollowing
\gray{17161} & \gray{Abstractive} & \gray{P} & \gray{8f14c} & \gray{Additional} & \gray{OthE} & \gray{Related information (2,3) (=17041)}\\ \hline
% Unit 4 has observed historical scans targeting organizations using
% IP addresses shared by Volexity
\gray{17181} & \gray{Abstractive} & \gray{P} & \gray{8f14c} & \gray{Additional} & \gray{OthE} & \gray{Related information (3) (=17061)}\\ \hline
% Playful Taurus, Iranian govt network compromise
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{Cyber Threat Intelligence: Machine Summaries: Analysts: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support.}
\label{tab:cyber-machine-task2-analyst}
\end{table}
%%% CTI EXPERT MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2ndary} & \textbf{Comment} \\
\hline
16212 & Abstractive & P & 415f6 & Additional & OthE & Related information (3) (=17061, 17181)\\ \hline
% Playful Taurus continues to evolve its tactics, and the analysis suggests that Iranian govt network have likely been compromised.
% Interesting -- in this case, it is likely, not very likely
17592 & Hybrid & F & 5e2f7 & Semantic Additional & PredE OthE & Summary close to Attribution~1 but uses stronger language (``very likely'' versus ``likely''); Related information (2,3) (=17049) \\ \hline
% Iranian network compromise
17606 & Hybrid & F & 5e2f7 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 (=17063) \\ \hline
% advisory regarding 2 CVEs
17633 & Abstractive & F & 84863 & Additional & NE & Unclear why selected as refutation \\ \hline
% summary attributes action to threat actor x4k, unclear why selected
17639 & Abstractive & P & 84863 & Content & OutE & Summary hallucination \\ \hline
% class object vuln
% (2) hallucination: summary says vuln allows remote access class objects when (2) says the vuln is allows the search path of the system's class loader to be modified
17642 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution 1 \\ \hline
% where downloaded file written
% summary paraphrase of (1), (2,3) related info
17643 & Abstractive & P & 84863 & Content Additional & OutE OthE & Summary contains information not in attributions, Related information (3) \\ \hline
% OpenSSL suggested mitigations
% (3) related
% (2) missing entity: summary says OpenSSL suggests X, while the citation only suggests X without saying who suggested it
17645 & Abstractive & P & 84863 & Content & OutE & Summary hallucination (=17040, 17160) \\ \hline
% confusion on malware stage
17652 & Abstractive & N & 84863 & Additional & OthE & Related information (2) (=17047) \\ \hline
% sample file associated with Brute Ratel C4
% (2) related info, different sample file
17658 & Abstractive & N & 84863 & Additional & OthE & Related information (3) (=17053) \\ \hline
% mitigations for phishing
17660 & Abstractive & P & 84863 & Content & OutE & Summary contains named entity while (3) uses pronoun \\ \hline
% MS has provided mitigations & recs until a patch is released
% missing entity: summary says MS provided mitigations but (3) uses "they"
17661 & Hybrid & P & 84863 & Additional & OthE & Related information (3) (=17056) \\ \hline
% JsonWebToken patch info
17667 & Abstractive & F & 84863 & Content & GramE & Badly parsed attribution (2) \\ \hline
% Auth0 team handled disclosure, patch
% Summary paraphrase (1)
% (2) had parsing issues of timeline so included several items but readable and relevant, list it as the objection
% (3) provides supplemental info about the vuln
17679 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3) \\ \hline
% summary approx (1)
% (2,3) related
17680 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% summary paraphrase (1)
% (2,3) related
17684 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% summary = (1)
% (2,3) related
17687 & Hybrid & F & 84863 & Additional & OthE & Related information (3) \\ \hline
% URL requests similar to ProxyShell requests, reported by Mandiant
% (3) related
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{Cyber Threat Intelligence: Machine Summaries: Experts: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support.}
\label{tab:cyber-machine-task2-expert}
\end{table}
%%% CRISISFACTS MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
17341 & Abstractive & P & 73655 & Content Additional & GramE OthE & Badly parsed, related information (2,3); Not enough information in summary to identify which fire\\ \hline
% fire fought with planes and helicopters
17344 & Abstractive & U & 73655 & Additional & OthE & Not enough information in summary to identify which disaster\\ \hline
% generic: communities coming together in support
17347 & Hybrid & F & 73655 & Semantic Additional & PredE OthE & Time shift led to inconsistent info (3); Attribution~2 for a different fire (=17467)\\ \hline
% Holy fire burned 6200 acres, 5% contained
% Citation 2 for Carr Fire, not Holy Fire
17420 & Hybrid & P & 73555 & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% hurrican models trending west towards East Coast
% summary says east coast, #3 says not known if east coast
17425 & Abstractive & U & 73655 & Semantic Content & CircE GramE & Additional location information in summary inconsistent with attribution (2); Badly parsed attributions difficult to read (2,3)\\ \hline
% red cross opened 2 new shelters
17428 & Hybrid & F & 73655 & Semantic & PredE & Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17488)\\ \hline
% fire 200 acres -> 800 acres in one hour
17429 & Hybrid & P & 73655 & Semantic & PredE & Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17489)\\ \hline
% Lilacfire burned 4100 acres, 60% contained, citation 3 says 90% contained
17432 & Abstractive & P & 73655 & Content & OutE & Summary contains info not present in attributions\\ \hline
% the storm is projected to potentially make landfall, prompted concerns + prep
% no citations address concerns + prep
17436 & Abstractive & N & 73655 & Additional & OthE & Attribution provided option not listed in summary (3)\\ \hline
% consider evacuation plans
% citation 3 has suggestion not in summary
17439 & Hybrid & U & 73655 & Content Additional & OutE OthE & Summary contains world knowledge not included in attribution (1); Attributions on related info (2,3)\\ \hline
% 102h 12Z GFS model (Global Forecast System model)
17448 & Abstractive & F & 73655 & Semantic & PredE & Time shift error led to inconsistent info (1,3)\\ \hline
% fire fueled by strong Santa Ana winds, more favorable weather expected
% citation 3 before more favorable weather predicted, citation 1 after
17452 & Abstractive & F & 73655 & Additional & NE & Unclear why selected as refutation \\ \hline
% localized flooding and power outages
\gray{17467} & \gray{Hybrid} & \gray{F} & \gray{36f2c} & \gray{Semantic Additional} & \gray{PredE OthE} & \gray{Time shift led to inconsistent info (3); Attribution~2 for a different fire (=17347)}\\ \hline
% the fire has burned 6200 acres and is currently 5% contained
% Citation 2 for Carr Fire, not Holy Fire
\gray{17488} & \gray{Hybrid} & \gray{F} & \gray{36f2c} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=17428)}\\ \hline
% fire 200 acres -> 800 acres in one hour
\gray{17489} & \gray{Hybrid} & \gray{P} & \gray{36f2c} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17429)}\\ \hline
% fire burned 4,100 acres and 60% contained, citation 3 says 90% contained
17515 & Abstractive & F & 36f2c & Content & GramE & Badly parsed attribution (3)\\ \hline
% Riverside and Shasta counties state of Emergency
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{CrisisFACTS: Machine Summaries: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support, U=Unclear.}
\label{tab:crisisfacts-machine-task2}
\end{table}
%%% TAC 2011 HUMAN TASK 1 TABLE %%%
\begin{table}[p]\small
\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.5cm}|p{1cm}|p{4cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annotator} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
13135 & Emb & 8d4ff & S & Content & OutE & Primary part of summary not present in attribution (=13245)\\ \hline
% China environmental issue
13181 & Emb & 8d4ff & N & Content & OutE & Summary contains information not present in attribution which is about a related topic \\ \hline
% Walter Reed controversy
13182 & Emb & 8f14c & U & Semantic & EntE & Time Shift led to inconsistent info (=13221) \\ \hline
% ferry sinking recovery efforts
13189 & Emb & 8d4ff & S & Semantic Content & CircE OutE & Time Shift led to inconsistent timing info; Summary contains info not in attributing sentence but in context \\ \hline
% liberian war crime trial
13190 & Emb & 8d4ff & S & Content & OutE & Timing information in summary not present in attribution \\ \hline
% Athens embassy attack
\gray{13221} & \gray{Emb} & \gray{8d4ff} & \gray{S} & \gray{Semantic} & \gray{EntE} & \gray{Time Shift led to inconsistent info (=13182)} \\ \hline % ferry sinking recovery efforts
13229 & Emb & 8d4ff & S & Content & OutE & Timing info stated differently (August versus Thursday) \\ \hline
% Mattel recall
\gray{13245} & \gray{Emb} & \gray{8f14c} & \gray{U} & \gray{Content} & \gray{OutE} & \gray{Primary part of summary not present in attribution (=13135)}\\ \hline
% China environmental issue
13258 & Emb & 8f14c & N & Content & OutE & Timing and supplemental location information in summary not present in attribution \\ \hline
% piracy seizure
\end{tabular}
\parbox[b]{4.85in}{}
\caption{TAC 2011: Human Summaries: Task 1: One Attributing Sentence with Context. L/Label: S=Support, N=No Support, U=Unclear.}
\label{tab:human-task-1}
\end{table}
%%% TAC 2011 HUMAN TASK 2 TABLE %%%
\begin{table}[h]\small
\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annotator} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
12981 & Emb & eed54 & P & Semantic & CircE & Time Shift error led to inconsistent timing info (2,3) \\ \hline
% south korea oil tanker accident
13022 & Emb & 415f6 & P & Content & OutE & Summary contains statistic not present in attributions\\ \hline
% china earthquake, stats (number dead) in news article but not in citations
13025 & Emb & 415f6 & F & Content Additional & GramE OthE & Bad attribution grammar (2); Attribution on related topic (3)\\ \hline
% internet redesign
13039 & NLI & eed54 & F & Semantic & PredE & Time Shift led to inconsistent info (3) (=13144)\\ \hline
% Menu Foods recall
13041 & NLI & 415f6 & F & Semantic & EntE & Attributions inconsistent but related (2,3) (=13046)\\ \hline
% Arabs and shoe soles insult
\gray{13046} & \gray{Emb} & \gray{415f6} & \gray{F} & \gray{Semantic} & \gray{EntE} & \gray{Attributions inconsistent but related (2,3) (=13041)}\\ \hline
% Arabs and shoe soles insult
13056 & NLI & 415f6 & F & Additional & OthE & Unrelated attribution (3)\\ \hline
% construction and missing people
13065 & Emb & 415f6 & F & Content & OutE & Summary statement contains additional timing information not present in attributions\\ \hline
% athens Embassy attack
13072 & Emb & 415f6 & F & Semantic & PredE & Inconsistent info, from time shift or different disaster (3)\\ \hline
% Bangledesh cyclone
13095 & Emb & 415f6 & P & Content Additional & OutE OthE & Summary contains info not present in attributions; Attribution for wrong location (3)\\ \hline
% SW China water shortage causes
13097 & NLI & 415f6 & F & Additional & OthE & Attribution unrelated (3)\\ \hline
% bombing in India
13098 & Emb & 415f6 & F & Additional & NE & Unclear why selected as refutation\\ \hline
% construction and missing people, same summary, different citations 13056
13104 & NLI & 415f6 & F & Additional & OthE & Attributions related (2,3) \\ \hline
% madoff lawsuit
13105 & Emb & 415f6 & F & Additional & OthE & Attributions related (2,3) \\ \hline
% madoff lawsuit, same summary, diff citations 13104
\gray{13144} & \gray{NLI} & \gray{415f6} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time Shift led to inconsistent info (3) (=13039)}\\ \hline
% menu foods, same summary, same citations 13039
\end{tabular}
\parbox[b]{4.85in}{}
\caption{TAC 2011: Human Summaries: Task 2: Three Attributing Sentences. L/Label: F=Full Support, P=Partial Support.}
\label{tab:tac2011-human-task2}
\end{table}
%%% TAC 2011 MACHINE TASK 2 %%%
\begin{table}[h]\small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{Annot.} & \textbf{L} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
13951 & Abstractive & 8d4ff & P & Content & OutE & Summary contains information not present in attributions but might be inferred\\ \hline
% children trama-> inflammation and disease
% summary about childhood impact, #2/#3 about adult disease from child trauma
13954 & Abstractive & 8d4ff & P & Semantic Content & CircE OutE & Time shift (1,2); Summary contained info not present in attributions\\ \hline
% Finnish PM expressed condolences, gov held emergency meeting
13957 & Abstractive & 8d4ff & P & Semantic & PredE & Time shift led to inconsistent info (2,3) \\ \hline
% Cyclone disaster
13961 & Abstractive & 8d4ff & F & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% Minneapolis bridge collapse with fatalities
13971 & Hybrid & 8d4ff & F & Content & OutE & Summary hallucination (1) (=14091)\\ \hline
% Amish children killed
% summary says "she had known for almost 10 years", source docs say married
% to for almost 10 years
\gray{14021} & \gray{Abstractive} & \gray{8f14c} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=13961, 14081)}\\ \hline
% Minneapolis bridge collapse with fatalities
14026 & Abstractive & 8f14c & F & Additional & OthE & Attribution contains related info (2)\\ \hline
% Chinese FM offers conolences to air crash victims
14045 & Hybrid & 8f14c & F & Semantic & PredE & Time shift led to inconsistent info (2) (=14105) \\ \hline
% Iraqi reporter throws shoes at Pres Bush, one citation says shoe
\gray{14077} & \gray{Abstractive} & \gray{36f2c} & \gray{P} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (2,3) (=13597)}\\ \hline
% Cyclone related: summary evacuation, citation homelessness
\gray{14081} & \gray{Abstractive} & \gray{36f2c} & \gray{F} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=13961,14021)}\\ \hline
% Minneapolis bridge collapse = 14021
\gray{14091} & \gray{Hybrid} & \gray{36f2c} & \gray{P} & \gray{Content} & \gray{OutE} & \gray{Summary hallucination (1) (=13971)}\\ \hline
% Amish children killed, = 13971
14092 & Abstractive & 36f2c & F & Additional & OthE & Attribution same topic but different location (2) \\ \hline
% East Asian ivory anti-poaching education
14049 & Abstractive & 36f2c & F & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% Albarez decision - death or life in prison?
14102 & Abstractive & 36f2c & F & Content & OutE & Summary hallucination (summary used ``the'' versus 1 of 4)\\ \hline
\gray{14105} & \gray{Hybrid} & \gray{36f2c} & \gray{F} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (2) (=14045)}\\ \hline
% Iraqi reporter throws show at Pres Bush = 14045
14108 & Abstractive & 36f2c & F & Additional & NE & Unclear why selected as refutation \\ \hline
% Chinese computer virus arrest
14115 & Hybrid & 36f2c & F & Additional & OthE & Attributions contain related information (2,3)\\ \hline
% Lack of sleep -> weight gain
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{TAC 2011: Machine Summaries: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support.}
\label{tab:tac2011-machine-task2}
\end{table}
%%% CTI ANALYST MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
17040 & Abstractive & P & 73655 & Content & OutE & Summary hallucination (=17160)\\ \hline
% Inside TDrop2: Technical Analysis of new Dark Seoul Malware
% Source document: (Stage 4) "It proceeds to randomly select an
% executable file in the system32 folder, and performs process
% hollowing against it."
% Summary: In the fourth stage, the malware checked for parent processes
% and performed process hollowing again.
17041 & Abstractive & P & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 (=17161)\\ \hline
% Unit 42 has observed historical scans using IP addr shared by Volexity
% All info in citation 1, 2/3 related
17043 & Hybrid & P & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% VirusTotal graph. All info in citation 1, 2/3 related.
17044 & Hybrid & F & 73655 & Additional & OthE & Closely related information (2), related information (3), Summary $\approx$ Attribution~1\\ \hline
% Certificate-based attack described.
% Citation 1 is almost copy of summary sentence. Citation 2 describes vuln.
17047 & Abstractive & P & 73655 & Additional & OthE & Related information (2)\\ \hline
% Roshan_VC.iso, uses Brute Ratel C4 malware
% (2) related info
17048 & Abstractive & F & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% CVE-2022-26809
% Citation 1 ~ summary
17049 & Hybrid & P & 73655 & Semantic Additional & PredE OthE & Summary close to Attribution~1 but uses stronger language (``very likely'' versus ``likely''); Related information (2,3)\\ \hline
% Iranian govt network compromise
% Citation 1 ~ summary
17053 & Abstractive & F & 73655 & Additional & OthE & Related information (3)\\ \hline
% invest in awareness training, detection tool to reduce callback phish risk
% citations 1 and 2 provide full support and additional information
17056 & Hybrid & P & 73655 & Additional & OthE & Related information (3)\\ \hline
% JsonWebToken patch information
17057 & Abstractive & P & 73655 & Content & OutE & Summary contains info not present in attributions\\ \hline
% list of products to detect and prevednt Conti ransomware, subset of list in citations (due to parsing?)
17061 & Abstractive & P & 73655 & Additional & OthE & Related information (3) (=17181)\\ \hline
% Iranian govt network compromise, related 17049
% Summary: Playful Taurus continues to evolve its tactics...
17063 & Hybrid & F & 73655 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% VMWare advisory
17065 & Hybrid & P & 73655 & Additional & NE & Unclear why selected as refutation\\ \hline
% technical description malware
17066 & Abstractive & P & 73655 & Additional & OthE & Closely related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% ML to detect network abuse
17072 & Hybrid & F & 73655 & Additional & OthE & Closely related information (2), related information (3), Summary $\approx$ Attribution~1\\ \hline
% AWS phishing attack
17073 & Abstractive & P & 73655 & Semantic & EntE & Closely related information (2,3), Summary $\approx$ Attribution~1 but summary entity lacks sufficient information (``incidents'' versus ``incidents involving landing URLs'')\\ \hline
% URL-based attack analysis
17083 & Abstractive & P & 73655 & Additional & OthE & Closely related information (2,3), Summary $\approx$ Attribution~1\\ \hline
% custom URI checksum to identify attack
\gray{17160} & \gray{Abstractive} & \gray{N} & \gray{8f14c} & \gray{Content} & \gray{OutE} & \gray{Summary hallucination (=17040)}\\ \hline
% 4th stage process hollowing
\gray{17161} & \gray{Abstractive} & \gray{P} & \gray{8f14c} & \gray{Additional} & \gray{OthE} & \gray{Related information (2,3) (=17041)}\\ \hline
% Unit 4 has observed historical scans targeting organizations using
% IP addresses shared by Volexity
\gray{17181} & \gray{Abstractive} & \gray{P} & \gray{8f14c} & \gray{Additional} & \gray{OthE} & \gray{Related information (3) (=17061)}\\ \hline
% Playful Taurus, Iranian govt network compromise
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{Cyber Threat Intelligence: Machine Summaries: Analysts: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support.}
\label{tab:cyber-machine-task2-analyst}
\end{table}
%%% CTI EXPERT MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2ndary} & \textbf{Comment} \\
\hline
16212 & Abstractive & P & 415f6 & Additional & OthE & Related information (3) (=17061, 17181)\\ \hline
% Playful Taurus continues to evolve its tactics, and the analysis suggests that Iranian govt network have likely been compromised.
% Interesting -- in this case, it is likely, not very likely
17592 & Hybrid & F & 5e2f7 & Semantic Additional & PredE OthE & Summary close to Attribution~1 but uses stronger language (``very likely'' versus ``likely''); Related information (2,3) (=17049) \\ \hline
% Iranian network compromise
17606 & Hybrid & F & 5e2f7 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 (=17063) \\ \hline
% advisory regarding 2 CVEs
17633 & Abstractive & F & 84863 & Additional & NE & Unclear why selected as refutation \\ \hline
% summary attributes action to threat actor x4k, unclear why selected
17639 & Abstractive & P & 84863 & Content & OutE & Summary hallucination \\ \hline
% class object vuln
% (2) hallucination: summary says vuln allows remote access class objects when (2) says the vuln is allows the search path of the system's class loader to be modified
17642 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution 1 \\ \hline
% where downloaded file written
% summary paraphrase of (1), (2,3) related info
17643 & Abstractive & P & 84863 & Content Additional & OutE OthE & Summary contains information not in attributions, Related information (3) \\ \hline
% OpenSSL suggested mitigations
% (3) related
% (2) missing entity: summary says OpenSSL suggests X, while the citation only suggests X without saying who suggested it
17645 & Abstractive & P & 84863 & Content & OutE & Summary hallucination (=17040, 17160) \\ \hline
% confusion on malware stage
17652 & Abstractive & N & 84863 & Additional & OthE & Related information (2) (=17047) \\ \hline
% sample file associated with Brute Ratel C4
% (2) related info, different sample file
17658 & Abstractive & N & 84863 & Additional & OthE & Related information (3) (=17053) \\ \hline
% mitigations for phishing
17660 & Abstractive & P & 84863 & Content & OutE & Summary contains named entity while (3) uses pronoun \\ \hline
% MS has provided mitigations & recs until a patch is released
% missing entity: summary says MS provided mitigations but (3) uses "they"
17661 & Hybrid & P & 84863 & Additional & OthE & Related information (3) (=17056) \\ \hline
% JsonWebToken patch info
17667 & Abstractive & F & 84863 & Content & GramE & Badly parsed attribution (2) \\ \hline
% Auth0 team handled disclosure, patch
% Summary paraphrase (1)
% (2) had parsing issues of timeline so included several items but readable and relevant, list it as the objection
% (3) provides supplemental info about the vuln
17679 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3) \\ \hline
% summary approx (1)
% (2,3) related
17680 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% summary paraphrase (1)
% (2,3) related
17684 & Hybrid & F & 84863 & Additional & OthE & Related information (2,3), Summary $\approx$ Attribution~1 \\ \hline
% summary = (1)
% (2,3) related
17687 & Hybrid & F & 84863 & Additional & OthE & Related information (3) \\ \hline
% URL requests similar to ProxyShell requests, reported by Mandiant
% (3) related
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{Cyber Threat Intelligence: Machine Summaries: Experts: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support.}
\label{tab:cyber-machine-task2-expert}
\end{table}
%%% CRISISFACTS MACHINE TASK 2 TABLE %%%
\begin{table}[h] \small
%\parbox[b]{4.85in}{}
\centering
\begin{tabular}{|l|c|c|c|p{1.6cm}|p{1.1cm}|p{4.1cm}|}
\hline
\textbf{ID} & \textbf{Method} & \textbf{L} & \textbf{Annot.} &
\textbf{Primary} & \textbf{2nd} & \textbf{Comment} \\
\hline
17341 & Abstractive & P & 73655 & Content Additional & GramE OthE & Badly parsed, related information (2,3); Not enough information in summary to identify which fire\\ \hline
% fire fought with planes and helicopters
17344 & Abstractive & U & 73655 & Additional & OthE & Not enough information in summary to identify which disaster\\ \hline
% generic: communities coming together in support
17347 & Hybrid & F & 73655 & Semantic Additional & PredE OthE & Time shift led to inconsistent info (3); Attribution~2 for a different fire (=17467)\\ \hline
% Holy fire burned 6200 acres, 5% contained
% Citation 2 for Carr Fire, not Holy Fire
17420 & Hybrid & P & 73555 & Semantic & PredE & Time shift led to inconsistent info (3)\\ \hline
% hurrican models trending west towards East Coast
% summary says east coast, #3 says not known if east coast
17425 & Abstractive & U & 73655 & Semantic Content & CircE GramE & Additional location information in summary inconsistent with attribution (2); Badly parsed attributions difficult to read (2,3)\\ \hline
% red cross opened 2 new shelters
17428 & Hybrid & F & 73655 & Semantic & PredE & Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17488)\\ \hline
% fire 200 acres -> 800 acres in one hour
17429 & Hybrid & P & 73655 & Semantic & PredE & Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17489)\\ \hline
% Lilacfire burned 4100 acres, 60% contained, citation 3 says 90% contained
17432 & Abstractive & P & 73655 & Content & OutE & Summary contains info not present in attributions\\ \hline
% the storm is projected to potentially make landfall, prompted concerns + prep
% no citations address concerns + prep
17436 & Abstractive & N & 73655 & Additional & OthE & Attribution provided option not listed in summary (3)\\ \hline
% consider evacuation plans
% citation 3 has suggestion not in summary
17439 & Hybrid & U & 73655 & Content Additional & OutE OthE & Summary contains world knowledge not included in attribution (1); Attributions on related info (2,3)\\ \hline
% 102h 12Z GFS model (Global Forecast System model)
17448 & Abstractive & F & 73655 & Semantic & PredE & Time shift error led to inconsistent info (1,3)\\ \hline
% fire fueled by strong Santa Ana winds, more favorable weather expected
% citation 3 before more favorable weather predicted, citation 1 after
17452 & Abstractive & F & 73655 & Additional & NE & Unclear why selected as refutation \\ \hline
% localized flooding and power outages
\gray{17467} & \gray{Hybrid} & \gray{F} & \gray{36f2c} & \gray{Semantic Additional} & \gray{PredE OthE} & \gray{Time shift led to inconsistent info (3); Attribution~2 for a different fire (=17347)}\\ \hline
% the fire has burned 6200 acres and is currently 5% contained
% Citation 2 for Carr Fire, not Holy Fire
\gray{17488} & \gray{Hybrid} & \gray{F} & \gray{36f2c} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3) (=17428)}\\ \hline
% fire 200 acres -> 800 acres in one hour
\gray{17489} & \gray{Hybrid} & \gray{P} & \gray{36f2c} & \gray{Semantic} & \gray{PredE} & \gray{Time shift led to inconsistent info (3), Summary $\approx$ Attribution~1 (=17429)}\\ \hline
% fire burned 4,100 acres and 60% contained, citation 3 says 90% contained
17515 & Abstractive & F & 36f2c & Content & GramE & Badly parsed attribution (3)\\ \hline
% Riverside and Shasta counties state of Emergency
\end{tabular}
%\parbox[b]{4.85in}{}
\caption{CrisisFACTS: Machine Summaries: Task 2: 3 Attributing Sentences. L/Label: F=Full Support, P=Partial Support, N=No Support, U=Unclear.}
\label{tab:crisisfacts-machine-task2}
\end{table}
\begin{thebibliography}{999}
\bibitem[Luhn(1958)]{10.1147/rd.22.0159}
Luhn, H.P.
\newblock The Automatic Creation of Literature Abstracts.
\newblock {\em IBM J. Res. Dev.} {\bf 1958}, {\em 2},~159–165.
\newblock {\url{https://doi.org/10.1147/rd.22.0159}}.
\bibitem[See \em{et~al.}(2017)See, Liu, and Manning]{journals/corr/SeeLM17}
See, A.; Liu, P.J.; Manning, C.D.
\newblock Get To The Point: Summarization with Pointer-Generator Networks.
\newblock {\em CoRR} {\bf 2017}, {\em abs/1704.04368}.
\bibitem[Liu and Lapata(2019)]{liu-lapata-2019-text}
Liu, Y.; Lapata, M.
\newblock Text Summarization with Pretrained Encoders.
\newblock In Proceedings of the Proceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLP-IJCNLP); Association for
Computational Linguistics: Hong Kong, China, 2019; pp. 3730--3740.
\newblock {\url{https://doi.org/10.18653/v1/D19-1387}}.
\bibitem[Lewis \em{et~al.}(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed,
Levy, Stoyanov, and Zettlemoyer]{lewis-etal-2020-bart}
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.;
Stoyanov, V.; Zettlemoyer, L.
\newblock {BART}: Denoising Sequence-to-Sequence Pre-training for Natural
Language Generation, Translation, and Comprehension.
\newblock In Proceedings of the Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics; Association for Computational
Linguistics: Online, 2020; pp. 7871--7880.
\newblock {\url{https://doi.org/10.18653/v1/2020.acl-main.703}}.
\bibitem[Raffel \em{et~al.}(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, and Liu]{T5:10.5555/3455716.3455856}
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou,
Y.; Li, W.; Liu, P.J.
\newblock Exploring the Limits of Transfer Learning with a Unified Text-to-Text
Transformer.
\newblock {\em J. Mach. Learn. Res.} {\bf 2020}, {\em 21}.
\bibitem[OpenAI()]{chatgpt}
OpenAI.
\newblock {ChatGPT} Language Model (Version 3.5).
\newblock \url{https://openai.com}.
\newblock Accessed: December 21, 2022.
\bibitem[White \em{et~al.}(2023)White, Molino, Yang, and
Conroy]{analytics2030030}
White, C.T.; Molino, N.P.; Yang, J.S.; Conroy, J.M.
\newblock \texttt{occams}: A Text Summarization Package.
\newblock {\em Analytics} {\bf 2023}, {\em 2},~546--559.
\newblock {\url{https://doi.org/10.3390/analytics2030030}}.
\bibitem[Slobodkin \em{et~al.}(2024)Slobodkin, Hirsch, Cattan, Schuster, and
Dagan]{slobodkin-etal-2024-attribute}
Slobodkin, A.; Hirsch, E.; Cattan, A.; Schuster, T.; Dagan, I.
\newblock Attribute First, then Generate: Locally-attributable Grounded Text
Generation.
\newblock In Proceedings of the Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers);
Association for Computational Linguistics: Bangkok, Thailand, 2024; pp.
3309--3344.
\newblock {\url{https://doi.org/10.18653/v1/2024.acl-long.182}}.
\bibitem[Worledge \em{et~al.}(2024)Worledge, Hashimoto, and
Guestrin]{worledge2024extractive}
Worledge, T.; Hashimoto, T.; Guestrin, C.
\newblock The Extractive-Abstractive Spectrum: Uncovering Verifiability
Trade-offs in LLM Generations, 2024,
\href{http://xxx.lanl.gov/abs/2411.17375}{{\normalfont
[arXiv:cs.CL/2411.17375]}}.
\newblock {\url{https://doi.org/10.48550/arXiv.2411.17375}}.
\bibitem[Laban \em{et~al.}(2022)Laban, Schnabel, Bennett, and
Hearst]{laban-etal-2022-summac}
Laban, P.; Schnabel, T.; Bennett, P.N.; Hearst, M.A.
\newblock {S}umma{C}: Re-Visiting {NLI}-based Models for Inconsistency
Detection in Summarization.
\newblock {\em Transactions of the Association for Computational Linguistics}
{\bf 2022}, {\em 10},~163--177.
\newblock {\url{https://doi.org/10.1162/tacl_a_00453}}.
\bibitem[Liu \em{et~al.}(2023)Liu, Zhang, and Liang]{liu2023evaluating}
Liu, N.F.; Zhang, T.; Liang, P.
\newblock Evaluating Verifiability in Generative Search Engines, 2023,
\href{http://xxx.lanl.gov/abs/2304.09848}{{\normalfont
[arXiv:cs.CL/2304.09848]}}.
\bibitem[Lin(2004)]{lin-2004-rouge}
Lin, C.Y.
\newblock {ROUGE}: A Package for Automatic Evaluation of Summaries.
\newblock In Proceedings of the Text Summarization Branches Out; Association
for Computational Linguistics: Barcelona, Spain, 2004; pp. 74--81.
\bibitem[Amplayo \em{et~al.}(2023)Amplayo, Liu, Zhao, and
Narayan]{amplayo2023smart}
Amplayo, R.K.; Liu, P.J.; Zhao, Y.; Narayan, S.
\newblock {SMART}: Sentences as Basic Units for Text Evaluation.
\newblock In Proceedings of the The Eleventh International Conference on
Learning Representations, 2023.
\bibitem[Pagnoni \em{et~al.}(2021)Pagnoni, Balachandran, and
Tsvetkov]{pagnoni-etal-2021-understanding}
Pagnoni, A.; Balachandran, V.; Tsvetkov, Y.
\newblock Understanding Factuality in Abstractive Summarization with {FRANK}: A
Benchmark for Factuality Metrics.
\newblock In Proceedings of the Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies; Toutanova, K.; Rumshisky, A.; Zettlemoyer, L.;
Hakkani-Tur, D.; Beltagy, I.; Bethard, S.; Cotterell, R.; Chakraborty, T.;
Zhou, Y., Eds.; Association for Computational Linguistics: Online, 2021; pp.
4812--4829.
\newblock {\url{https://doi.org/10.18653/v1/2021.naacl-main.383}}.
\end{thebibliography}
\begin{thebibliography}{999}
\bibitem[Luhn(1958)]{10.1147/rd.22.0159}
Luhn, H.P.
\newblock The Automatic Creation of Literature Abstracts.
\newblock {\em IBM J. Res. Dev.} {\bf 1958}, {\em 2},~159–165.
\newblock {\url{https://doi.org/10.1147/rd.22.0159}}.
\bibitem[See \em{et~al.}(2017)See, Liu, and Manning]{journals/corr/SeeLM17}
See, A.; Liu, P.J.; Manning, C.D.
\newblock Get To The Point: Summarization with Pointer-Generator Networks.
\newblock {\em CoRR} {\bf 2017}, {\em abs/1704.04368}.
\bibitem[Liu and Lapata(2019)]{liu-lapata-2019-text}
Liu, Y.; Lapata, M.
\newblock Text Summarization with Pretrained Encoders.
\newblock In Proceedings of the Proceedings of the 2019 Conference on Empirical
Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLP-IJCNLP); Association for
Computational Linguistics: Hong Kong, China, 2019; pp. 3730--3740.
\newblock {\url{https://doi.org/10.18653/v1/D19-1387}}.
\bibitem[Lewis \em{et~al.}(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed,
Levy, Stoyanov, and Zettlemoyer]{lewis-etal-2020-bart}
Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.;
Stoyanov, V.; Zettlemoyer, L.
\newblock {BART}: Denoising Sequence-to-Sequence Pre-training for Natural
Language Generation, Translation, and Comprehension.
\newblock In Proceedings of the Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics; Association for Computational
Linguistics: Online, 2020; pp. 7871--7880.
\newblock {\url{https://doi.org/10.18653/v1/2020.acl-main.703}}.
\bibitem[Raffel \em{et~al.}(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, and Liu]{T5:10.5555/3455716.3455856}
Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou,
Y.; Li, W.; Liu, P.J.
\newblock Exploring the Limits of Transfer Learning with a Unified Text-to-Text
Transformer.
\newblock {\em J. Mach. Learn. Res.} {\bf 2020}, {\em 21}.
\bibitem[OpenAI()]{chatgpt}
OpenAI.
\newblock {ChatGPT} Language Model (Version 3.5).
\newblock \url{https://openai.com}.
\newblock Accessed: December 21, 2022.
\bibitem[White \em{et~al.}(2023)White, Molino, Yang, and
Conroy]{analytics2030030}
White, C.T.; Molino, N.P.; Yang, J.S.; Conroy, J.M.
\newblock \texttt{occams}: A Text Summarization Package.
\newblock {\em Analytics} {\bf 2023}, {\em 2},~546--559.
\newblock {\url{https://doi.org/10.3390/analytics2030030}}.
\bibitem[Slobodkin \em{et~al.}(2024)Slobodkin, Hirsch, Cattan, Schuster, and
Dagan]{slobodkin-etal-2024-attribute}
Slobodkin, A.; Hirsch, E.; Cattan, A.; Schuster, T.; Dagan, I.
\newblock Attribute First, then Generate: Locally-attributable Grounded Text
Generation.
\newblock In Proceedings of the Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers);
Association for Computational Linguistics: Bangkok, Thailand, 2024; pp.
3309--3344.
\newblock {\url{https://doi.org/10.18653/v1/2024.acl-long.182}}.
\bibitem[Worledge \em{et~al.}(2024)Worledge, Hashimoto, and
Guestrin]{worledge2024extractive}
Worledge, T.; Hashimoto, T.; Guestrin, C.
\newblock The Extractive-Abstractive Spectrum: Uncovering Verifiability
Trade-offs in LLM Generations, 2024,
\href{http://xxx.lanl.gov/abs/2411.17375}{{\normalfont
[arXiv:cs.CL/2411.17375]}}.
\newblock {\url{https://doi.org/10.48550/arXiv.2411.17375}}.
\bibitem[Laban \em{et~al.}(2022)Laban, Schnabel, Bennett, and
Hearst]{laban-etal-2022-summac}
Laban, P.; Schnabel, T.; Bennett, P.N.; Hearst, M.A.
\newblock {S}umma{C}: Re-Visiting {NLI}-based Models for Inconsistency
Detection in Summarization.
\newblock {\em Transactions of the Association for Computational Linguistics}
{\bf 2022}, {\em 10},~163--177.
\newblock {\url{https://doi.org/10.1162/tacl_a_00453}}.
\bibitem[Liu \em{et~al.}(2023)Liu, Zhang, and Liang]{liu2023evaluating}
Liu, N.F.; Zhang, T.; Liang, P.
\newblock Evaluating Verifiability in Generative Search Engines, 2023,
\href{http://xxx.lanl.gov/abs/2304.09848}{{\normalfont
[arXiv:cs.CL/2304.09848]}}.
\bibitem[Lin(2004)]{lin-2004-rouge}
Lin, C.Y.
\newblock {ROUGE}: A Package for Automatic Evaluation of Summaries.
\newblock In Proceedings of the Text Summarization Branches Out; Association
for Computational Linguistics: Barcelona, Spain, 2004; pp. 74--81.
\bibitem[Amplayo \em{et~al.}(2023)Amplayo, Liu, Zhao, and
Narayan]{amplayo2023smart}
Amplayo, R.K.; Liu, P.J.; Zhao, Y.; Narayan, S.
\newblock {SMART}: Sentences as Basic Units for Text Evaluation.
\newblock In Proceedings of the The Eleventh International Conference on
Learning Representations, 2023.
\bibitem[Pagnoni \em{et~al.}(2021)Pagnoni, Balachandran, and
Tsvetkov]{pagnoni-etal-2021-understanding}
Pagnoni, A.; Balachandran, V.; Tsvetkov, Y.
\newblock Understanding Factuality in Abstractive Summarization with {FRANK}: A
Benchmark for Factuality Metrics.
\newblock In Proceedings of the Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human
Language Technologies; Toutanova, K.; Rumshisky, A.; Zettlemoyer, L.;
Hakkani-Tur, D.; Beltagy, I.; Bethard, S.; Cotterell, R.; Chakraborty, T.;
Zhou, Y., Eds.; Association for Computational Linguistics: Online, 2021; pp.
4812--4829.
\newblock {\url{https://doi.org/10.18653/v1/2021.naacl-main.383}}.
\end{thebibliography}