IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
Extracted source text · arXiv 2209.03895
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% packages IDIAP
\usepackage[table]{xcolor}
\definecolor{Gray}{gray}{0.9}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{graphicx}
\usepackage{mathtools}
\usepackage{amsfonts}
% Remove the "review" option to generate the final version.
\usepackage{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% Smaller URLs
\usepackage{relsize}
\renewcommand{\UrlFont}{\smaller[0.1]}
\newcommand{\dashrule}[1][black]{%
\color{#1}\rule[\dimexpr.5ex-.2pt]{4pt}{.4pt}\xleaders\hbox{\rule{4pt}{0pt}\rule[\dimexpr.5ex-.2pt]{4pt}{.4pt}}\hfill\kern0pt%
}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% If the title and author information does not fit in the area allocated, uncomment the following
%
\setlength\titlebox{6cm}
%
% and set <dim> to something 5cm or larger.
\newcommand{\todo}[1]{{\color{red}\textbf{[[TODO: #1]]}}}
%tentative title, I think it's too long
%\title{IDIAPers @ Causal News Corpus 2022: Causal Relation Identification Using a Few-shot and Prompt-based Fine-tuning of Language Models}
\title{IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
%\thanks{Manuscript submitted to CASE@EMNLP}
}
% Author information can be set in various styles:
% For several authors from the same institution:
\author{Sergio Burdisso$^{*,1,2}$, Juan Zuluaga-Gomez$^{1,3}$, Esaú Villatoro-Tello$^{1,5}$, Martin Fajcik$^{1,4}$\\
\textbf{Muskaan Singh}$^{1}$\textbf{,} \textbf{Pavel Smrz}$^{4}$\textbf{,} \textbf{Petr Motlicek}$^{1}$\\
$^{1}$Idiap Research Institute, Martigny, Switzerland\\
$^{2}$Universidad Nacional de San Luis (UNSL), San Luis, Argentina\\
$^{3}$Ecole Polytechnique Fédérale de Lausanne, Switzerland\\
$^{4}$Brno University of Technology, Brno, Czech Republic\\
$^{5}$Universidad Autónoma Metropolitana Unidad Cuajimalpa, Mexico City, Mexico\\
$^{*}$\emph{corresponding author: sergio.burdisso@idiap.ch}
}
\begin{document}
\maketitle
\begin{abstract}
In this paper, we describe our participation in the subtask 1 of CASE-2022, Event Causality Identification with Casual News Corpus.
We address the Causal Relation Identification (CRI) task by exploiting a set of simple yet complementary techniques for fine-tuning language models (LMs) on a small number of annotated examples (i.e., a \textit{few-shot} configuration).
We follow a prompt-based prediction approach for fine-tuning LMs in which the CRI task is treated as a masked language modeling problem (MLM). This approach allows LMs natively pre-trained on MLM problems to directly generate textual responses to CRI-specific prompts.
We compare the performance of this method against ensemble techniques trained on the entire dataset.
Our best-performing submission was fine-tuned with only 256 instances per class, 15.7\% of the all available data, and yet obtained the second-best precision ($0.82$), third-best accuracy ($0.82$), and an F1-score ($0.85$) very close to what was reported by the winner team ($0.86$).\footnote{Code available at \url{https://github.com/idiap/cncsharedtask}.}
%the competition, i.e., F1=$0.86$.
%Our results indicate that the proposed architecture is able to leverage information about the crime categories for effectively detect the mention of a violent act within a tweet. Our approach obtained the best performance ($F1=0.7758$) among 11 different teams and a total of 32 different submissions.
\end{abstract}
%% Individual sections are listed here!
%%@ALL: I prefer to have separated files for each section, IMO, it's easier to edit and follow
\input{introduction}
\input{relatedWork}
\input{method}
\input{experimental_results}
\input{conclusions}
\input{acknowledgments}
% Entries for the entire Anthology, followed by custom entries
% \bibliography{anthology,custom}
\bibliography{references}
\bibliographystyle{acl_natbib}
\appendix
\newpage
\input{apendixA}
%\section{Example Appendix}
%\label{sec:appendix}
%This is an appendix.
\end{document}
\section{Introduction}
\label{sect:intro}
Causal relation identification aims to predict whether or not there exists a cause-effect relation between a pair of events mentioned in a given text. For example, in the sentence \textit{``\underline{Protests} spread to 15 towns and resulted in the \underline{destruction of property}''}, the automatic causal identification system must be able to realize that there is cause-effect relation between the events \textit{``protest''} and \textit{``destruction''}.%, i.e, \textit{``protest''} $\xrightarrow{\text{cause}}$ \textit{``destruction''}.
Hence, understanding causal relations within a text is an essential aspect of natural language processing (NLP) and understanding (NLU)~\cite{ayyanar2019causal,LI2021207, Fiona2022}. Once the causal information is identified within a text, such knowledge becomes beneficial for many other downstream NLP tasks, e.g., Information Extraction, Question Answering, Text Summarization \cite{ayyanar2019causal, man-etal-2022-event}.
However, due to the ambiguity and diversity in written documents, causality identification is not easy and remains a challenging problem.% in the NLP \& NLU fields.
The Event Causality Identification with Causal News Corpus (CASE-2022) shared task \cite{,tan-etal-2022-event} addresses this problem on a recently created corpus named the Causal News Corpus (CNC) \cite{tan-EtAl:2022:LREC}. Contrary to previous existing causality corpora, the CNC dataset, manually annotated by experts, incorporates a broader set of causal linguistic constructions, i.e., not only limited to %clause-based
explicit constructions, resulting in a more challenging dataset.
In this paper, we describe our followed methodology for addressing the causal event classification shared task (subtask 1) during the CASE-2022 competition~\cite{tan-etal-2022-event}.\footnote{We refer the reader to our standalone publication \cite{idiap_subtaskB} to know our results for subtask 2.} %We participated in both subtasks but report on subtask 1 in this paper. For subtask two, we refer the reader to our standalone publication \cite{idiap_subtaskB}.
Our primary method, based on a \textit{few-shot} configuration, follows a prompt-based approach for fine-tuning the language model (LM). The intuitive idea of this approach is to allow the LM to directly auto-complete natural language prompts. Following this technique, we leverage the LM's knowledge and let it decide the correct label of the input sequence. Additionally, we evaluate the performance of ensemble techniques trained using the entire dataset available. Our results demonstrate that our few-shot, prompt-based, fine-tuning approach can generalize well even when using as few as 256 samples per class %(i.e., 512 samples)
for training, outperforming ensemble techniques trained with the entire dataset, as well as most of other teams' submissions.
The rest of the paper is organized as follows. Section \ref{sect:rw} describes relevant related work, Section \ref{sub-sec:prompt} describes the components of our main method, namely the prompt-based approach. Section \ref{sect:exps} describes the experimental setup, i.e., datasets, additional baselines, experiments configuration and obtained results. Finally, Section \ref{sect:conclusions} depicts our main conclusions and future work directions.
% Ensemble figure
\begin{figure*}[t]
% \widefigure
\centering
\includegraphics[width=1\textwidth]{figures/lm-bff-causality.pdf}
\caption{Augmented prompt-based classification for causality identification task.
First, the input instance $x = \textit{``Soldiers were hurt in the attacks''}$ is converted into three different input prompts by applying $f_t'(x)$ three times. Then, these three prompts are given to a RoBERTa model, and one logit vector is obtained for each. These vectors are then averaged, and the word with the highest score, \emph{``causal''}, is selected. Finally, this word is mapped to its corresponding class, and $x$ is classified as positive. Note that, in this example, we have the following word-to-class label mapping $word(positive) = \textit{``causal''}$ and $word(negative) = \textit{``random''}$.}
\label{fig:lm-bff}
\end{figure*}
\section{Related Work}
\label{sect:rw}
%Causal relations in texts are expressed in arbitrary and complex ways. Detecting this type of relations from text becomes complicated when the cause-effect is not explicitly mentioned between events of interest \cite{KhetanEtAl_2022,Fiona2022}.
Previous work on causal relation identification varies from knowledge-based to deep neural network approaches (Deep-NN). %includes knowledge-based, statistical machine learning (ML), and, more recently, deep learning-based methods.
%On the one hand,
Knowledge-based systems rely on linguistic patterns extracted using an exhaustive exploration of the data, where lexico-semantic and syntactic
analysis lead to the identification of relevant structures and keywords that depict the presence of a causal relation in the text \cite{GarciaDaniela96, khoo-etal-2000-extracting}. Although interpretable, these methods %and understand,
require a lot of human effort to generate relevant patterns and result in models that are not readily applicable in different domains.
%On the other hand,
Statistical machine learning (ML) approaches leave to the selected algorithm to find patterns in the data on the basis of the manual annotation. Traditionally, using different NLP tools, it is possible to compute various features for a given collection and apply any ML pipeline to train a causality relation classifier, e.g., \cite{rutherford-xue-2014-discovering, hidey-mckeown-2016-identifying}.
%We refer the reader to \cite{rutherford-xue-2014-discovering, hidey-mckeown-2016-identifying} as examples of such an approach.
However, one main disadvantage of these techniques is the language dependency and error propagation of the NLP tools, e.g., syntactic parsers. % As known, these techniques are less labor-intensive; however, they are usually language dependent and might be prone to errors due to the NLP tools used during the syntactic parsing.
Finally, recent approaches based on Deep-NN have become popular, given their powerful representation learning ability. Typical approaches include convolutional neural networks \cite{9028985}, long short-term memory networks \cite{LI2021207}, and pre-trained transformer-based LMs such as BERT~\cite{devlin-etal-2019-bert}, where following a standard fine-tuning approach makes possible the detection of causality relations \cite{Fiona2022, KhetanEtAl_2022, fajcik-etal-2020-fit}.
Normally, these methods involve high computational costs and large amounts of labeled data. However, in this work, we show that pre-trained LMs can still be effective even when fine-tuned with very few instances.
Contrary to previous work, we evaluate the effectiveness of very recent prompt-based prediction approaches under a \textit{few-shot} configuration for causal relation identification.
% In this work, we show the effectiveness of fine-tuning a pre-trained language model using only a small subset of examples. Our proposed approach, based on prompt-based fine-tuning, was able to outperform other complex ensembles trained on the entire CNC dataset.
%Causal relation on news, some related work
%Baseline models: reference of each model (references are alread in Table 1)
%Experiments with Transformers and Datasets toolkit \cite{wolf2020transformers,lhoest2021datasets}
%\section{Methodology}
%\label{sect:method}
\section{Prompt-Based Approach}
\label{sub-sec:prompt}
In the ``pre-train, prompt, and predict'' paradigm, unlike the standard ``pre-train and fine-tune'' paradigm, instead of adapting pre-trained LMs to downstream tasks via objective engineering,\footnote{\textit{Objective engineering} referes to both the pre-training and
fine-tuning stages of LMs~\cite{liu2021pre}.} downstream tasks are reformulated to look more like those solved during the LM pre-training phase~\cite{liu2021pre}.
More precisely, prompt-based prediction treats the downstream task as a masked language modeling problem, where the model directly generates a textual response (referred to as a \emph{label word}) to a given prompt defined by a task-specific \emph{template}~\cite{gao2020making}.
For instance, when identifying the sentiment of a movie review like ``I love this movie.'' we may continue with ``Overall, it was a \texttt{[MASK]} movie.'' and ask the LM to fill the mask with a sentiment-bearing word.
In this example, the original input text $x$ (``I love this movie.'') is modified using the \emph{template} ``\texttt{[x]} Overall, it was a \texttt{[MASK]} movie.'' into a textual string prompt $x'$ in which the mask will be filled with a \emph{label word}.
Some examples of \emph{label words} for this example could be ``fantastic'' or ``boring''.
In the case of classification tasks, in addition to defining a set of possible \emph{label words}, it is necessary to define a mapping between each one and the actual output labels. For instance, if labels $+$ and $-$ refer to positive and negative sentiment, respectively, ``fantastic'' in previous example could be mapped to output label $+$, and ``boring'' to $-$.
Formally, let $\mathcal{L}$ be a pre-trained language model, $f_t(x)$ a function that converts the input $x$ into a prompt by instantiating template $t$ which contains one \texttt{[MASK]} token, $mask$. Let $word: \mathcal{Y} \rightarrow \mathcal{W}$ be a mapping from the task label space, $\mathcal{Y}$, to the \emph{label words} set, $\mathcal{W}$.
Then, the classification task is converted to a \emph{masked language modeling} (MLM) task in which the probability of predicting class $y \in \mathcal{Y}$ is modeled as:
\begin{equation}
\label{eq:prompt}
\begin{split}
p(y|x) & = p(mask = word(y) | f_t(x)) =\\
& = \frac{exp(\mathbf{w}_{word(y)} \cdot \mathbf{h}_{mask})}{\sum_{y'\in \mathcal{Y}}exp(\mathbf{w}_{word(y')} \cdot \mathbf{h}_{mask})},
\end{split}
\end{equation}
where $\mathbf{h}_{mask}$ is the hidden vector of \texttt{[MASK]} and $\mathbf{w}_v$ denotes the vector encoding word $v$.
Note that when fine-tuning $\mathcal{L}$ to minimize the cross-entropy loss, the pre-trained weights $\mathbf{w}_v$ are re-used, and there's no need to introduce any new parameter. On the contrary, with standard fine-tuning a task-specific head, $softmax(\mathbf{W}_o \mathbf{h}_{\texttt{[CLS]}})$, has to be added, with new task-specific learnable parameters $\mathbf{W}_o \in \mathbb{R}^{|\mathcal{Y}|\times d}$, which increases the gap between pre-training and fine-tuning.
Hereafter % and for simplicity's sake,
we will refer to the "causal" and "non-causal" classes as ``positive'' ($+$) and ``negative'' ($-$) respectively. %, or $+$ and $-$.
In addition, %in this work in particular,
and following previous work by~\citet{gao2020making}, we append one answered prompt for each class to the input prompt as \emph{demonstrations}.\footnote{These~\emph {demonstrations}~\cite{gao2020making} are used to demonstrate the LM, in-context, how it should provide the answer to the input prompt.}
More precisely, let $\mathcal{Y}=\{+,-\}$ be the set of labels for the binary causality identification task, let $t\leftarrow v$ be the template $t$ in which its \texttt{[MASK]} token has been filled with word $v$, and $w^y = word(y)$ the \emph{word label} for class $y \in\mathcal{Y}$, then we redefine $f_t(x)$ in \autoref{eq:prompt} as $f_t'(x)$ defined as:
\begin{equation}
\label{eq:demo}
f_t'(x) = f_t(x)\mathbin\Vert f_{t\leftarrow w^+}(x^+)\mathbin\Vert f_{t\leftarrow w^-}(x^-)
\end{equation}
where $\mathbin\Vert$ is the string concatenation operator, and $x^y$ is an instance of class $y$ randomly sampled from the training set.
%For instance, in
\autoref{fig:lm-bff}, depicts an example of three different input prompts are shown by applying $f_t'(x)$ three times to the input instance $x$.
%\vspace{0.1cm}
\noindent\textbf{Classification process:} the process is illustrated
%with an example
in \autoref{fig:lm-bff}. First, the input instance $x$ is converted into $d$ different input prompts by applying $f_t'(x)$, $d$ times. Then, each input prompt is given to the LM to obtain $d$ logit vectors holding the word scores for the mask in each prompt. A simple ensemble scheme is then applied by averaging all $d$ logit vectors, and the \emph{word label} with the highest score is selected, which is finally mapped to its corresponding class $y$ using mapping $word(y)$.
% Additional answered prompts like $f_{t\leftarrow w^+}(x^+)$ above are referred to as \emph{demonstrations}~\cite{gao2020making} since are used to in-context demonstrate the LM how it should provide the answer to the input prompt.
% The intuition behind is to take advantage of the ability some LMs have to learn repetitive patterns during pre-training. For instance, works related to \emph{in-context learning} have shown that providing a prompt along with some demonstrations alone, without fine-tuning, to the pre-trained LM could be enough to achieve strong performance on many NLP datasets~\cite{brown2020language}.
\begin{table*}[t!]
\centering
% \small
% \scalebox{0.99}{
\begin{tabular}{l ccc ccc ccc cccc}
\toprule
& & \multicolumn{2}{c}{\cellcolor{Gray} \textbf{Precision}} & & \multicolumn{2}{c}{\cellcolor{Gray} \textbf{Recall}} & & \multicolumn{2}{c}{\cellcolor{Gray} \textbf{Accuracy}} & & \multicolumn{2}{c}{\cellcolor{Gray} \textbf{F1-Score}} \\
\cline{3-4} \cline{6-7} \cline{9-10} \cline{12-13}
Submission & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} \\
\midrule
Ensemble-10m & & \textbf{88.46} & 82.78 & & 90.45 & 84.66 & & \textbf{88.26} & 81.35 & & \textbf{89.44} & 83.70 \\
Prompt-256 & & 85.49 & \textbf{82.80} & & \textbf{92.70} & 87.50 & & 87.30 & \textbf{82.64} & & 88.95 & \textbf{85.08} \\
Prompt-356e & & 82.72 & 80.41 & & 88.76 & \textbf{88.64} & & 83.60 & 81.35 & & 85.63 & 84.32 \\
Prompt-1000 & & 84.56 & 81.08 & & 91.57 & 85.22 & & 86.07 & 80.39 & & 87.87 & 83.10 \\
Ensemble-8p & & 86.10 & 81.15 & & 90.44 & 88.07 & & 86.69 & 81.67 & & 88.22 & 84.47 \\
\bottomrule
\end{tabular}
% }
\caption{Official performance metrics in percentages (\%) from the selected methods in dev and test partitions of the Causal News Corpus.}%Performance metrics with baseline and several ensembles (see Section~\ref{sub-sec:submissions}). Metrics of the output generated by the organizers' scoring script on the development and test sets.}
\label{tab:exp_submissions}
\end{table*}
\noindent\textbf{Training and model selection}: for developing our prompt-based models, we performed a simplified version of the process described in previous work by \citet{gao2020making}.
Namely, we carried out the following six steps:
\textbf{Step 1}: we created a new training set, $\tau_k$, by extracting $k$ instances per class from the original train partition, and used the remaining $2925 - 2\times k$ instances as a large evaluation set $\delta_{T-k}$ (dataset stats are given in \autoref{tab:dataset}).
%\textbf{Step 2}: in order to choose a word-to-class label mapping $word(y)$, first, we manually set the template to $t = \text{``\texttt{[x]} \textit{It was} \texttt{[MASK]}''}$ and then trained and evaluated the performance of different hand-crafted \emph{word label} options.
% After this stage, words \textit{``causal''} and \textit{``random''} were selected for positive and negative classes respectively. %\footnote{In fact, different hand-crafted templates were also tried.}
\textbf{Step 2}: in order to add \emph{demonstrations} to a given input $x$ (see \autoref{eq:prompt}), we uniformly sampled $x^-$ and $x^+$ from the top-$50\%$ most similar instances in $\tau_k$.\footnote{We tested different percentages, however $50\%$ was the best-performing one.} To do so, we pre-computed the sentence embeddings of training instances using a pre-trained SBERT \cite{reimers2019sentence} model, and cosine distance was used as a similarity metric.
\textbf{Step 3}: using \textit{``causal''} and \textit{``random''} as \emph{word labels},\footnote{We performed some simple preliminary tests using different words like ``coincidence'', ``choice'', ``causal'', ``cause'', with few trivial hand-crafted templates (e.g. $\text{``\texttt{[x]} \textit{It was} \texttt{[MASK]}''}$), from which ``random'' and ``casual'' where selected.} the next step was to generate candidate templates automatically using T5. First, each training instance $x$ of class $y$ in $\tau_{k}$ was converted to ``$\texttt{[x]}\texttt{<P>}word(y)\texttt{<S>}$'' where $\texttt{<P>}$ and $\texttt{<S>}$ are T5 mask tokens, and used a 100 wide beam search to decode multiple template candidates by filling $\texttt{<P>}$ and $\texttt{<S>}$ tokens.
\textbf{Step 4}: next step was sorting all 100 final candidate templates by F1 score. However, since this is a time-consuming step, a subset of the evaluation set was used by sampling $256$ unique positive and negative instances from $\delta_{T-k}$. Note that no fine-tuning is used at this point, just the out-of-the-box pre-trained LM.
\textbf{Step 5}: we selected the top-10 best-performing templates as final candidates. For each candidate template we fine-tuned the LM as a MLM task (see \autoref{eq:prompt}) on the training set, $\tau_{k}$, evaluating it on the complete evaluation set, $\delta_{T-k}$.
\textbf{Step 6}: finally, the model with the best F1 score on the official dev set was selected as a candidate for submission ---we also checked that the F1 score on $\delta_{T-k}$ was among the first ones too (if not first).
Note that in this step we're evaluating the model on unseen data since the official dev set is being used as an unofficial test set.
\
The above process was repeated varying the number $k$ of training instances, with $k=256$, $356$, $512$, and $1000$;\footnote{Inspired by evidence showing a performance saturation when $k=256$ (Figure 3 in \citet{gao2020making}), compared to standard fine-tuning on the entire dataset, we decided to start from this value.} the number $d$ of input prompts to ensemble during classification stage, with $d$ from $1$ to $9$; and using RoBERTa (large and base), and DeBERTa V3 (base) as pre-trained LMs.
In step 5, models were fine-tuned for a maximum of $1000$ steps using AdamW~\cite{loshchilov2018decoupled} optimizer ($\beta_1{=}0.9, \beta_2{=}0.999, \epsilon{=}1\mathrm{e}{-8}$) with a learning rate of $\gamma{=}1\mathrm{e}{-5}$ with no weight decay ($\lambda{=}0$). Models were evaluated every 100 steps and check-pointed when new best F1 scores were obtained.
%\subsection{Ensemble-based Approach}
%\label{subsect:ensemble}
%We also performed several ensembles of different fine-tuned LMs to increase the generalization and compensate for the overfitting of the models.
%We followed the approach described in~\citet{fajcik2019but}, called \textit{TOP-N} fusion. In this formulation, we first define a \textit{set} of $M$ pre-trained LMs, varying only the training seed. \textit{TOP-N} fusion starts by choosing one uniformly random model from the \textit{set}, which is added to the ensemble. Next, it randomly shuffles the rest of the models and tries adding them into the ensemble once, as long as the F1 score improves. Each time a model is added to the ensemble, its performance gets measured. The model would stay in the ensemble only and only if it improved the overall performance. This aims at an iterative optimization of the ensemble's F1 score by averaging the output probabilities. As the selection process is stochastic, we repeat the process $N{=}10000$ times. We construct a new ensemble for each iteration, independently of the previous ones. Finally, we select the best performing ensemble for submission. Further details are given in \autoref{appendix:ensemble} (\autoref{fig:ensemble}). %A comprehensive overview and further details of the proposed ensemble method are in \autoref{fig:ensemble} of \autoref{appendix:ensemble}.
\section{Results \& Discussion}
\label{sect:exps}
In this section we provide the details of the employed dataset, a set of additional experiments based on recent ensemble techniques, and the final configuration of our submitted runs to the subtask 1 of CASE 2022.
\subsection{Dataset}
\label{subsect:dataset}
As mentioned earlier, the main goal of subtask 1 of CASE-2022 is to classify whether or not a given sentence contains a \textit{cause-effect} relation. Thus, systems have to be able to predict \textit{Causal} or \textit{Non-causal} labels per sentence. Table \ref{tab:dataset} contains a few statistics regarding the distribution of the classes in the \textit{train, dev}, and \textit{test} partitions.
\begin{table}[]
\centering
\begin{tabular}{lcccc}
\toprule
Label & {\cellcolor{Gray}\textbf{Train}} & {\cellcolor{Gray}\textbf{Dev}} & {\cellcolor{Gray}\textbf{Test}} & {\cellcolor{Gray}\textbf{Total}}\\
\midrule
\textit{Causal} & 1603 & 178 & 176 & 1957 \\
\textit{Non-causal} & 1322 & 145 & 135 & 1602\\
\cline{2-5}
Total: & 2925 & 323 & 311 & \textbf{3559}\\
\bottomrule
\end{tabular}
\caption{Number of positive (causal) and negative (non-causal) instances in the \textit{train, dev,} and \textit{test} sets of the shared task. We refer the interested reader to \cite{tan-etal-2022-event} to know more details about the data and the labeling process.}
\label{tab:dataset}
\end{table}
\subsection{Ensemble-based Approach}
\label{subsect:ensemble}
We also performed several ensembles of different fine-tuned LMs to increase the generalization and compensate for the overfitting of the models.
We followed the approach described in~\citet{fajcik2019but}, called \textit{TOP-N} fusion. In this formulation, we first define a \textit{set} of $M$ pre-trained LMs, varying the training seed. \textit{TOP-N} fusion starts by choosing one uniformly random model from the \textit{set}, which is added to the ensemble. Next, it randomly shuffles the rest of the models and tries adding them into the ensemble once, as long as the F1 score improves. Each time a model is added to the ensemble, its performance gets measured. The model would stay in the ensemble only and only if it improved the overall performance. This aims at an iterative optimization of the ensemble's F1 score by averaging the output probabilities. As the selection process is stochastic, we repeat the process $N{=}10000$ times. We construct a new ensemble for each iteration, independently of the previous ones. Finally, we select the best performing ensemble for submission. Further details are given in \autoref{appendix:ensemble} (\autoref{fig:ensemble}). %A comprehensive overview and further details of the proposed ensemble method are in \autoref{fig:ensemble} of \autoref{appendix:ensemble}.
%All models were implemented using \emph{PyTorch} \cite{paszke2019pytorch}. We used \emph{Transformers}~\cite{wolf2020transformers} and \emph{Datasets}~\cite{lhoest2021datasets} toolkit for fetching the pre-trained weights of the models.
\subsection{Official Submissions}
\label{sub-sec:submissions}
Next, we describe each one of our submissions:%five submissions to the CASE 2022 task:
\noindent \textbf{Ensemble-10m:} ensemble model described in \autoref{subsect:ensemble} with 10 final models obtained from a set of 150 initial ones (50 fine-tuned \textit{bert-base-cased}, \textit{roberta-base}, and \textit{deberta-v3-base} models).
\noindent \textbf{Prompt-256:} prompt-based \textit{roberta-large} model with $k{=}256$ training instances per class, $d{=}3$ input prompts to ensemble during classification stage; and template $t = \text{``\texttt{[x]} \textit{This is not} \texttt{[MASK]}''}$.
\noindent \textbf{Prompt-1000:} The same previous model but with $t = \text{``\texttt{[x]} \textit{There were no} \texttt{[MASK]}\textit{ities in this}''}$, $k{=}1000$, and $d{=}1$.
\noindent \textbf{Ensemble-8p:} ensemble model described in \autoref{subsect:ensemble} with 8 final models obtained from the top-50 best performing prompt-based models as the initial set.
\noindent \textbf{Prompt-356e:} three prompt-base models trained with $k{=}356$ instances. The first two models have the same template as \emph{Prompt-1000} but with $d{=}2$ and $3$, respectively. The third one uses the template $t = \text{``\texttt{[x]} \textit{The incident is not} \texttt{[MASK]}''}$ with $d{=}1$.\footnote{Note that these prompts, as well as previous ones, were automatically generated as described in \autoref{sub-sec:prompt}.}
Finally, a simple majority voting ensemble among these three models generates the output. %classification output.
%We refer the reader to \autoref{sect:method} where further details about the ensemble are covered.
\subsection{Results}
\label{sub-sec:results}
\autoref{tab:exp_submissions} shows the official results, both in dev and test partitions, for our five submissions. As expected, the ensemble of several LMs (Ensemble-10m) was able to obtain outstanding performance across several metrics during the validation phase (i.e., dev partition\footnote{
We further performed a 5-cvf experiment on six different architectures, see the results on \autoref{tab:exp_k-fold} in \autoref{appendix:baseline_results}.}).
However, the performance dropped significantly in the test partition (F1$=89.44$ $\rightarrow$ F1$=83.70$).
On the contrary, our prompt-based approach trained on 256 instances per class (Prompt-256) could generalize better on the test partition. Such submission obtained 2nd place in terms of precision (82.80\%), 3rd in accuracy (82.64\%), and 5th in F1 (85.08\%) ---the best F1 was $86.19\%$.
% However, the main advantage of our method is that it does not require large amount of training data to fine-tune the model, i.e., a computationally less expensive approach, that can be easily adapted to different domains.
However, the main advantage of our approach is that it allows the LM to be trained in a few-shot setting, making it harder for the model to overfit the data.
Moreover, most of the available data can be kept and used for measuring the generalization power of the model instead.
For instance, our best-performing model (Prompt-256) was fine-tuned only on $15.7\%$ of all available data,\footnote{i.e. train + dev sets in \autoref{tab:dataset}} allowing the remaining $84.3\%$ to be used for evaluation and model selection ($74.3\%$ as evaluation set and $10\%$ as our own test set).
Therefore, model selection choice is more robust since the risk of performance drop on unseen data, such as the official test set, is expected to be lower.
%%%%%% TRASH
% \begin{table}[t]
% \caption{Performance metrics (first row: precision, recall, accuracy and F1-score, respectively) with several ensembles. Metrics of the output generated by the organizers' scoring script on the development set. $^\dagger$ensemble that contains 150 fine-tuned models: \textit{bert-base-cased}, \textit{roberta-base} and, \textit{deberta-v3-base}, 50 each. $^{\dagger\dagger}$ensemble of only \textit{deberta-v3-base}. $^{\mathsection}$ensemble of xx prompt-based models. $^{\ddagger}$mix of prompt-based and \textit{deberta-v3-base} LMs.}
% \label{tab:exp_ensemble}
% \centering
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c c c c c}
% \toprule
% Model & \multicolumn{1}{l}{\cellcolor{Gray} @Pre} & \multicolumn{1}{l}{\cellcolor{Gray} @Rec} & \multicolumn{1}{l}{\cellcolor{Gray} @Acc} & \multicolumn{1}{l}{\cellcolor{Gray} @F1}\\
% \midrule
% \textit{ENS\_BRD}$^\dagger$ & 87.91 & 89.89 & 87.61 & 88.89 \\
% \textit{ENS\_D-v3}$^{\dagger\dagger}$ & 88.46 & 90.45 & 88.26 & 89.44 \\
% \textit{ENS\_prompt}$^{\mathsection}$ & 86.10 & 90.44 & 86.69 & 88.22 \\
% \textit{ENS\_full}$^{\ddagger}$ & 86.70 & 91.57 & 87.62 & 89.07 \\
% \bottomrule
% \end{tabular}
% }
% \end{table}
% \begin{table}[t]
% \caption{Performance metrics (first row: precision, recall, accuracy and F1-score, respectively) with several ensembles. Metrics of the output generated by the organizers' scoring script on the test set. %$^\dagger$ensemble that contains 150 fine-tuned models: \textit{bert-base-cased}, \textit{roberta-base} and, \textit{deberta-v3-base}, 50 each. $^{\dagger\dagger}$ensemble of only \textit{deberta-v3-base}. $^{\mathsection}$ensemble of xx prompt-based models. $^{\ddagger}$mix of prompt-based and \textit{deberta-v3-base}
% LMs.}
% \label{tab:exp_test_set}
% \centering
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c c c c c}
% \toprule
% Model & \multicolumn{1}{l}{\cellcolor{Gray} @Pre} & \multicolumn{1}{l}{\cellcolor{Gray} @Rec} & \multicolumn{1}{l}{\cellcolor{Gray} @Acc} & \multicolumn{1}{l}{\cellcolor{Gray} @F1}\\
% \midrule
% \textit{ENS\_D-v3}$^{\dagger\dagger}$ & 82.78 & 84.66 & 81.35 & 83.70 \\
% \textit{ENS\_prompt}$^{\mathsection}$ & 81.15 & 88.07 & 81.67 & 84.47 \\
% \textit{ENS\_prompt\_256}$^{\mathsection}$ & 82.80 & 87.50 & 82.64 & 85.08 \\
% \textit{ENS\_prompt\_356}$^{\mathsection}$ & 80.41 & 88.64 & 81.35 & 84.32 \\
% \textit{ENS\_prompt\_1000}$^{\mathsection}$ & 81.08 & 85.22 & 80.39 & 83.10 \\
% \bottomrule
% \end{tabular}
% }
% \end{table}
% \begin{table*}[t!]
% \caption{Official obtained results from the selected methods in dev and test partitions.}%Performance metrics with baseline and several ensembles (see Section~\ref{sub-sec:submissions}). Metrics of the output generated by the organizers' scoring script on the development and test sets.}
% \label{tab:exp_submissions}
% \centering
% \begin{tabular}{l ccc ccc ccc cccc}
% \toprule
% & & \multicolumn{2}{c}{\cellcolor{Gray} Precision} & & \multicolumn{2}{c}{\cellcolor{Gray} Recall} & & \multicolumn{2}{c}{\cellcolor{Gray} Accuracy} & & \multicolumn{2}{c}{\cellcolor{Gray} F1-Score} \\
% \cline{3-4} \cline{6-7} \cline{9-10} \cline{12-13}
% Model & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} & & \textit{dev} & \textit{test} \\
% \midrule
% SubmissionA & & \textbf{88.46} & 82.78 & & 90.45 & 84.66 & & \textbf{88.26} & 81.35 & & \textbf{89.44} & 83.70 \\
% SubmissionB & & 86.10 & 81.15 & & 90.44 & 88.07 & & 86.69 & 81.67 & & 88.22 & 84.47 \\
% SubmissionC (256) & & 85.49 & \textbf{82.80} & & \textbf{92.70} & 87.50 & & 87.30 & \textbf{82.64} & & 88.95 & \textbf{85.08} \\
% SubmissionD (356) & & 82.72 & 80.41 & & 88.76 & \textbf{88.64} & & 83.60 & 81.35 & & 85.63 & 84.32 \\
% SubmissionE (1000) & & 84.56 & 81.08 & & 91.57 & 85.22 & & 86.07 & 80.39 & & 87.87 & 83.10 \\
% \bottomrule
% \end{tabular}
% \end{table*}
\section{Conclusions}
\label{sect:conclusions}
This paper describes our participation in the CASE-2022 subtask 1. %: Causal Event Classification challenge.
Our proposed approach uses a few-shot configuration in which a prompt-based model is fine-tuned using \emph{only 256 instances} per class and yet was able to obtain remarkable results among all 16 participant teams. The comparison against traditional fine-tuning techniques, ensemble approaches, as well as the other participating models, show the potential of the proposed approach for better generalizing the posed task.
For future work, we plan to perform further ablation studies when we have access to test set ground truth labels. For instance, measuring the dev-to-test performance drop in relation to $k$ or the robustness against different training and demonstration sampling given a fixed $k$.
\section*{Acknowledgements}
This work was supported by CRiTERIA, an EU project, funded under the Horizon 2020 programme, grant agreement no. 101021866 and the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90140).
Esaú Villatoro-Tello, was supported partially by Idiap, SNI CONACyT, and UAM-Cuajimalpa Mexico.
\section{Baseline results}
\label{appendix:baseline_results}
We performed standard cross-entropy fine-tuning on six different pre-trained LMs (see first column in~\autoref{tab:exp_k-fold}) to produce baselines. We perform 5-fold cross-validation for each architecture following the partitions proposed in~\citet{tan-EtAl:2022:LREC}. Each system is fine-tuned on the sequence classification task to discriminate between casual and non-causal text input sequences. We report the mean and standard deviation (mean $\pm$ std) on the official development set over several metrics, see~\autoref{tab:exp_k-fold}.
During experimentation, we use the same learning rate of $\gamma=5\mathrm{e}{-5}$ with a linear learning rate scheduler. Dropout is set to $dp=0.1$ for the attention and hidden layers, while Gaussian Error Linear Units (GELU) is used as activation function~\cite{hendrycks2016gaussian}. We fine-tune each model with an effective batch size of 32 for 50 epochs with AdamW \cite{loshchilov2018decoupled} optimizer ($\beta_1{=}0.9, \beta_2{=}0.999$, $\epsilon{=}1\mathrm{e}{-8}$). We noted that \textit{deberta-v3-base} performed systematically better in all metrics as shown in~\autoref{tab:exp_k-fold}.
%Thus, we made a submission by performing an ensemble using only this architecture (see~\autoref{sub-sec:results}).
\begin{figure*}[t]
\centering
\includegraphics[width=1\textwidth]{figures/Ensemble.pdf}
\caption{Our proposed method to ensemble $N$ fine-tuned LMs, based on~\citet{fajcik2019but} approach. We fine-tune several LMs by modifying only the training seed. Our implementation uses the sequence classification task from HuggingFace toolkit \cite{wolf2020transformers,lhoest2021datasets}.}
\label{fig:ensemble}
\end{figure*}
\begin{table*}[t]
\centering
\scalebox{0.9}{
\begin{tabular}{l c c c c c}
\toprule
\textbf{Model} & \multicolumn{1}{c}{\cellcolor{Gray} \textbf{Precision}} & \multicolumn{1}{c}{\cellcolor{Gray} \textbf{Recall}} & \multicolumn{1}{c}{\cellcolor{Gray} \textbf{Accuracy}} & \multicolumn{1}{c}{\cellcolor{Gray} \textbf{F1-score}} & \multicolumn{1}{c}{\cellcolor{Gray} \textbf{Reference}} \\
\midrule
\texttt{bert-base-cased} & 83.52 $\pm$ 1.01 & 87.88 $\pm$ 3.08 & 79.68 $\pm$ 1.83 & 81.03 $\pm$ 1.20 & {\small\cite{devlin-etal-2019-bert}} \\
\texttt{bart-base} & 84.21 $\pm$ 0.88 & 87.80 $\pm$ 2.26 & 80.99 $\pm$ 2.19 & 81.98 $\pm$ 0.95 & {\small\cite{lewis2020bart}} \\
\texttt{roberta-base} & 85.13 $\pm$ 1.11 & 87.86 $\pm$ 2.41 & 82.66 $\pm$ 2.35 & 83.21 $\pm$ 1.10 & {\small\cite{liu2019roberta}}\\
\texttt{distilroberta-base} & 84.41 $\pm$ 1.20 & 88.05 $\pm$ 1.69 & 81.12 $\pm$ 2.09 & 82.22 $\pm$ 1.12 & {\small\cite{sanh2019distilbert}} \\
\texttt{deberta-base} & 82.67 $\pm$ 2.76 & 85.74 $\pm$ 2.72 & 80.32 $\pm$ 6.49 & 80.31 $\pm$ 3.44 & {\small\cite{he2021deberta}}\\
\texttt{deberta-v3-base} & \textbf{85.87 $\pm$ 1.18} & \textbf{88.88 $\pm$ 1.74} & \textbf{83.18 $\pm$ 3.16} & \textbf{84.00 $\pm$ 1.18} & {\small\cite{he2021debertav3}}\\
\bottomrule
\end{tabular}
}
\caption{Mean and standard deviation (mean $\pm$ std) of different metrics on the dev set using a 5-fold cross validation scheme on the CNC dataset. We report results for six different architectures of pre-trained LMs.}%Mean and standard deviation (mean $\pm$ std) on several metrics for a 5-fold cross validation on the CNC database. We perform experiments by fine tuning six different pre-trained LMs. The metrics are obtained from the official development set.}
\label{tab:exp_k-fold}
\end{table*}
\section{Ensembling}
\label{appendix:ensemble}
We compose ensembles before submission to leaderboard in two manners. \texttt{Ensembling-type-1} and \texttt{Ensembling-type-2}:
\begin{itemize}
\item \texttt{Ensembling-type-1}: we define a \textit{set} of models, which contains only baseline LMs fine-tuned on the sequence classification task (see~\autoref{tab:exp_k-fold}). We fine-tune 50 LMs for each architecture from first column of~\autoref{tab:exp_k-fold}. Next, we run our \textit{TOP-N} fusion algorithm (see~\autoref{subsect:ensemble}) with the \textit{set} of models previously defined. The model submitted with \texttt{Ensembling-type-1} is \mbox{\textbf{Ensemble-10m}}, reporting its performance in~\autoref{tab:exp_submissions}.
\item \texttt{Ensembling-type-2}, we define a \textit{set} of models containing prompt-based LMs. We select the top models for leaderboard submission. The overall process for ensembling is illustrated in~\autoref{fig:ensemble}. Even though the figure only depicts our first approach (explained above), we perform exactly the same with the prompt-based models explained in~\autoref{sub-sec:prompt}. The model submitted with \mbox{\texttt{Ensembling-type-2}} is \textbf{Ensemble-8p}, reporting its performance in~\autoref{tab:exp_submissions}.
\end{itemize}
\begin{table}[ht]
\centering
\begin{tabular}{ll}
\toprule
Model & F1-score (\%) \\
\midrule
\texttt{bert-base-cased} & 85.15 \\
\texttt{roberta-base} & 86.76 \\
\texttt{deberta-v3-base} & 89.69 \\
\midrule
\textit{\textbf{Ensemble-10m}}$^{\dagger}$ & \textbf{89.7} \\
\bottomrule
\end{tabular}
\caption{Obtained F1-scores on the \textit{dev} partition of subtask 1 of the Causal News Corpus. Results depict the top performance of three models that belong to the \textit{\textbf{Ensemble-10m}} configuration. The last row corresponds to an ensemble model composed of ten independent LMs, namely, six \textit{deberta-v3-base}, two \textit{bert-base-cased}, and two \textit{roberta-base}. More details about the ensemble construction are described in ~\autoref{subsect:ensemble} and~\citet{fajcik2019but}.}
%(see~\autoref{subsect:ensemble} and~\citet{fajcik2019but}). We only list the top performance of each different architecture present in the ensemble. The ensemble's performance is listed in the last row. $^{\dagger}$%\textit{\textbf{Ensemble-10m}}
%this model is composed of ten independent LMs models, namely, six \textit{deberta-v3-base}, and two \textit{bert-base-cased} and \textit{roberta-base} each.}
\label{tab:exp_ensemble_only_lms}
\end{table}
\textbf{Details about the ensemble:} we select the best ensembles based on its F1-score performance on the dev set. For example, in~\autoref{tab:exp_ensemble_only_lms} we list the performance of the \texttt{Ensembling-type-1} system (i.e., \mbox{\textbf{Ensemble-10m}}) we used for our submission in the leaderboard.
%\subsection{Baseline language models}
%\label{subsect:baselines}
%As an exploratory analysis
%We performed the traditional LM fine-tuning on six different pre-trained architectures (first column in \autoref{tab:exp_k-fold}, \autoref{appendix:baseline_results}). We perform 5-fold cross-validation on each architecture following the partitions proposed in~\citet{tan-EtAl:2022:LREC}. All architectures are fine-tuned on the sequence classification task to discriminate between casual and non-causal text input sequences. Refer to \autoref{appendix:baseline_results} for more details about the training process. %During experimentation, we use the same learning rate\footnote{We use a linear learning rate scheduler.} of $lr=5e^{-5}$. Dropout is set to $dp=0.1$ for the attention and hidden layers, while Gaussian Error Linear Units (GELU) as activation function~\cite{hendrycks2016gaussian}. We fine-tune each model with an effective batch size of 32 for 50 epochs with AdamW \cite{loshchilov2018decoupled} optimizer (betas=0.9, 0.999).
\begin{thebibliography}{28}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Ayyanar et~al.(2019{\natexlab{a}})Ayyanar, Koomullil, and
Ramasangu}]{ayyanar2019causal}
Raja Ayyanar, George Koomullil, and Hariharan Ramasangu. 2019{\natexlab{a}}.
\newblock Causal relation classification using convolutional neural networks
and grammar tags.
\newblock In \emph{2019 IEEE 16th India Council International Conference
(INDICON)}, pages 1--3. IEEE.
\bibitem[{Ayyanar et~al.(2019{\natexlab{b}})Ayyanar, Koomullil, and
Ramasangu}]{9028985}
Raja Ayyanar, George Koomullil, and Hariharan Ramasangu. 2019{\natexlab{b}}.
\newblock \href {https://doi.org/10.1109/INDICON47234.2019.9028985} {Causal
relation classification using convolutional neural networks and grammar
tags}.
\newblock In \emph{2019 IEEE 16th India Council International Conference
(INDICON)}, pages 1--3.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Fajcik et~al.(2020)Fajcik, Jon, Docekal, and
Smrz}]{fajcik-etal-2020-fit}
Martin Fajcik, Josef Jon, Martin Docekal, and Pavel Smrz. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.semeval-1.53} {{BUT}-{FIT} at
{S}em{E}val-2020 task 5: Automatic detection of counterfactual statements
with deep pre-trained language representation models}.
\newblock In \emph{Proceedings of the Fourteenth Workshop on Semantic
Evaluation}, pages 437--444, Barcelona (online). International Committee for
Computational Linguistics.
\bibitem[{Fajcik et~al.(2022)Fajcik, Singh, Zuluaga-Gomez, Villatoro-Tello,
Burdisso, Motlicek, and Smrz}]{idiap_subtaskB}
Martin Fajcik, Muskaan Singh, Juan Zuluaga-Gomez, Esaú Villatoro-Tello, Sergio
Burdisso, Petr Motlicek, and Pavel Smrz. 2022.
\newblock Idiapers @ causal news corpus 2022: Extracting cause-effect-signal
triplets via pre-trained autoregressive language model.
\newblock In \emph{The 5th Workshop on Challenges and Applications of Automated
Extraction of Socio-political Events from Text (CASE @ EMNLP 2022)}.
Association for Computational Linguistics.
\bibitem[{Fajcik et~al.(2019)Fajcik, Smrz, and Burget}]{fajcik2019but}
Martin Fajcik, Pavel Smrz, and Lukas Burget. 2019.
\newblock But-fit at semeval-2019 task 7: Determining the rumour stance with
pre-trained deep bidirectional transformers.
\newblock In \emph{Proceedings of the 13th International Workshop on Semantic
Evaluation}, pages 1097--1104.
\bibitem[{Gao et~al.(2020)Gao, Fisch, and Chen}]{gao2020making}
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.
\newblock Making pre-trained language models better few-shot learners.
\newblock \emph{arXiv preprint arXiv:2012.15723}.
\bibitem[{Garcia(1997)}]{GarciaDaniela96}
Daniela Garcia. 1997.
\newblock Coatis, an nlp system to locate expressions of actions connected by
causality links.
\newblock In \emph{Knowledge Acquisition, Modeling and Management}, pages
347--352, Berlin, Heidelberg. Springer Berlin Heidelberg.
\bibitem[{He et~al.(2021{\natexlab{a}})He, Gao, and Chen}]{he2021debertav3}
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021{\natexlab{a}}.
\newblock Debertav3: Improving deberta using electra-style pre-training with
gradient-disentangled embedding sharing.
\newblock \emph{arXiv preprint arXiv:2111.09543}.
\bibitem[{He et~al.(2021{\natexlab{b}})He, Liu, Gao, and Chen}]{he2021deberta}
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021{\natexlab{b}}.
\newblock \href {https://openreview.net/forum?id=XPZIaotutsD} {Deberta:
Decoding-enhanced bert with disentangled attention}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Hendrycks and Gimpel(2016)}]{hendrycks2016gaussian}
Dan Hendrycks and Kevin Gimpel. 2016.
\newblock Gaussian error linear units (gelus).
\newblock \emph{arXiv preprint arXiv:1606.08415}.
\bibitem[{Hidey and McKeown(2016)}]{hidey-mckeown-2016-identifying}
Christopher Hidey and Kathy McKeown. 2016.
\newblock \href {https://doi.org/10.18653/v1/P16-1135} {Identifying causal
relations using parallel {W}ikipedia articles}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1424--1433,
Berlin, Germany. Association for Computational Linguistics.
\bibitem[{Khetan et~al.(2022)Khetan, Ramnani, Anand, Sengupta, and
Fano}]{KhetanEtAl_2022}
Vivek Khetan, Roshni Ramnani, Mayuresh Anand, Subhashis Sengupta, and Andrew~E.
Fano. 2022.
\newblock Causal bert: Language models for causality detection between events
expressed in text.
\newblock In \emph{Intelligent Computing}, pages 965--980, Cham. Springer
International Publishing.
\bibitem[{Khoo et~al.(2000)Khoo, Chan, and Niu}]{khoo-etal-2000-extracting}
Christopher S.~G. Khoo, Syin Chan, and Yun Niu. 2000.
\newblock \href {https://doi.org/10.3115/1075218.1075261} {Extracting causal
knowledge from a medical database using graphical patterns}.
\newblock In \emph{Proceedings of the 38th Annual Meeting of the Association
for Computational Linguistics}, pages 336--343, Hong Kong. Association for
Computational Linguistics.
\bibitem[{Lewis et~al.(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis2020bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
\newblock Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7871--7880.
\bibitem[{Lhoest et~al.(2021)Lhoest, del Moral, Jernite, Thakur, von Platen,
Patil, Chaumond, Drame, Plu, Tunstall et~al.}]{lhoest2021datasets}
Quentin Lhoest, Albert~Villanova del Moral, Yacine Jernite, Abhishek Thakur,
Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu,
Lewis Tunstall, et~al. 2021.
\newblock Datasets: A community library for natural language processing.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 175--184.
\bibitem[{Li et~al.(2021)Li, Li, Zou, and Ren}]{LI2021207}
Zhaoning Li, Qi~Li, Xiaotian Zou, and Jiangtao Ren. 2021.
\newblock \href {https://doi.org/https://doi.org/10.1016/j.neucom.2020.08.078}
{Causality extraction based on self-attentive bilstm-crf with transferred
embeddings}.
\newblock \emph{Neurocomputing}, 423:207--219.
\bibitem[{Liu et~al.(2021)Liu, Yuan, Fu, Jiang, Hayashi, and
Neubig}]{liu2021pre}
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and
Graham Neubig. 2021.
\newblock Pre-train, prompt, and predict: A systematic survey of prompting
methods in natural language processing.
\newblock \emph{arXiv preprint arXiv:2107.13586}.
\bibitem[{Liu et~al.(2019)Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis,
Zettlemoyer, and Stoyanov}]{liu2019roberta}
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
\newblock Roberta: A robustly optimized bert pretraining approach.
\newblock \emph{arXiv preprint arXiv:1907.11692}.
\bibitem[{Loshchilov and Hutter(2019)}]{loshchilov2018decoupled}
Ilya Loshchilov and Frank Hutter. 2019.
\newblock \href {https://openreview.net/forum?id=Bkg6RiCqY7} {Decoupled weight
decay regularization}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Man et~al.(2022)Man, Nguyen, and Nguyen}]{man-etal-2022-event}
Hieu Man, Minh Nguyen, and Thien Nguyen. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.starsem-1.28} {Event
causality identification via generation of important context words}.
\newblock In \emph{Proceedings of the 11th Joint Conference on Lexical and
Computational Semantics}, pages 323--330, Seattle, Washington. Association
for Computational Linguistics.
\bibitem[{Reimers and Gurevych(2019)}]{reimers2019sentence}
Nils Reimers and Iryna Gurevych. 2019.
\newblock Sentence-bert: Sentence embeddings using siamese bert-networks.
\newblock \emph{arXiv preprint arXiv:1908.10084}.
\bibitem[{Rutherford and Xue(2014)}]{rutherford-xue-2014-discovering}
Attapol Rutherford and Nianwen Xue. 2014.
\newblock \href {https://doi.org/10.3115/v1/E14-1068} {Discovering implicit
discourse relations through brown cluster pair representation and coreference
patterns}.
\newblock In \emph{Proceedings of the 14th Conference of the {E}uropean Chapter
of the Association for Computational Linguistics}, pages 645--654,
Gothenburg, Sweden. Association for Computational Linguistics.
\bibitem[{Sanh et~al.(2019)Sanh, Debut, Chaumond, and
Wolf}]{sanh2019distilbert}
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.
\newblock Distilbert, a distilled version of bert: smaller, faster, cheaper and
lighter.
\newblock \emph{arXiv preprint arXiv:1910.01108}.
\bibitem[{Tan et~al.(2022{\natexlab{a}})Tan, Hürriyetoğlu, Caselli, Oostdijk,
Hettiarachchi, Nomoto, Uca, and Liza}]{tan-etal-2022-event}
Fiona~Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk, Hansi
Hettiarachchi, Tadashi Nomoto, Onur Uca, and Farhana~Ferdousi Liza.
2022{\natexlab{a}}.
\newblock Event causality identification with causal news corpus - shared task
3, {CASE} 2022.
\newblock In \emph{Proceedings of the 5th Workshop on Challenges and
Applications of Automated Extraction of Socio-political Events from Text
(CASE 2022)}, Online. Association for Computational Linguistics.
\bibitem[{Tan et~al.(2022{\natexlab{b}})Tan, Hürriyetoğlu, Caselli, Oostdijk,
Nomoto, Hettiarachchi, Ameer, Uca, Liza, and Hu}]{tan-EtAl:2022:LREC}
Fiona~Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk,
Tadashi Nomoto, Hansi Hettiarachchi, Iqra Ameer, Onur Uca, Farhana~Ferdousi
Liza, and Tiancheng Hu. 2022{\natexlab{b}}.
\newblock \href {https://aclanthology.org/2022.lrec-1.246} {The causal news
corpus: Annotating causal relations in event sentences from news}.
\newblock In \emph{Proceedings of the Language Resources and Evaluation
Conference}, pages 2298--2310, Marseille, France. European Language Resources
Association.
\bibitem[{Tan et~al.(2022{\natexlab{c}})Tan, Zuo, and Ng}]{Fiona2022}
Fiona~Anting Tan, Xinyu Zuo, and See-Kiong Ng. 2022{\natexlab{c}}.
\newblock \href {https://doi.org/10.48550/ARXIV.2208.09163} {Unicausal: Unified
benchmark and model for causal text mining}.
\bibitem[{Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz et~al.}]{wolf2020transformers}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et~al.
2020.
\newblock Transformers: State-of-the-art natural language processing.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 38--45.
\end{thebibliography}
\begin{thebibliography}{28}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Ayyanar et~al.(2019{\natexlab{a}})Ayyanar, Koomullil, and
Ramasangu}]{ayyanar2019causal}
Raja Ayyanar, George Koomullil, and Hariharan Ramasangu. 2019{\natexlab{a}}.
\newblock Causal relation classification using convolutional neural networks
and grammar tags.
\newblock In \emph{2019 IEEE 16th India Council International Conference
(INDICON)}, pages 1--3. IEEE.
\bibitem[{Ayyanar et~al.(2019{\natexlab{b}})Ayyanar, Koomullil, and
Ramasangu}]{9028985}
Raja Ayyanar, George Koomullil, and Hariharan Ramasangu. 2019{\natexlab{b}}.
\newblock \href {https://doi.org/10.1109/INDICON47234.2019.9028985} {Causal
relation classification using convolutional neural networks and grammar
tags}.
\newblock In \emph{2019 IEEE 16th India Council International Conference
(INDICON)}, pages 1--3.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Fajcik et~al.(2020)Fajcik, Jon, Docekal, and
Smrz}]{fajcik-etal-2020-fit}
Martin Fajcik, Josef Jon, Martin Docekal, and Pavel Smrz. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.semeval-1.53} {{BUT}-{FIT} at
{S}em{E}val-2020 task 5: Automatic detection of counterfactual statements
with deep pre-trained language representation models}.
\newblock In \emph{Proceedings of the Fourteenth Workshop on Semantic
Evaluation}, pages 437--444, Barcelona (online). International Committee for
Computational Linguistics.
\bibitem[{Fajcik et~al.(2022)Fajcik, Singh, Zuluaga-Gomez, Villatoro-Tello,
Burdisso, Motlicek, and Smrz}]{idiap_subtaskB}
Martin Fajcik, Muskaan Singh, Juan Zuluaga-Gomez, Esaú Villatoro-Tello, Sergio
Burdisso, Petr Motlicek, and Pavel Smrz. 2022.
\newblock Idiapers @ causal news corpus 2022: Extracting cause-effect-signal
triplets via pre-trained autoregressive language model.
\newblock In \emph{The 5th Workshop on Challenges and Applications of Automated
Extraction of Socio-political Events from Text (CASE @ EMNLP 2022)}.
Association for Computational Linguistics.
\bibitem[{Fajcik et~al.(2019)Fajcik, Smrz, and Burget}]{fajcik2019but}
Martin Fajcik, Pavel Smrz, and Lukas Burget. 2019.
\newblock But-fit at semeval-2019 task 7: Determining the rumour stance with
pre-trained deep bidirectional transformers.
\newblock In \emph{Proceedings of the 13th International Workshop on Semantic
Evaluation}, pages 1097--1104.
\bibitem[{Gao et~al.(2020)Gao, Fisch, and Chen}]{gao2020making}
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.
\newblock Making pre-trained language models better few-shot learners.
\newblock \emph{arXiv preprint arXiv:2012.15723}.
\bibitem[{Garcia(1997)}]{GarciaDaniela96}
Daniela Garcia. 1997.
\newblock Coatis, an nlp system to locate expressions of actions connected by
causality links.
\newblock In \emph{Knowledge Acquisition, Modeling and Management}, pages
347--352, Berlin, Heidelberg. Springer Berlin Heidelberg.
\bibitem[{He et~al.(2021{\natexlab{a}})He, Gao, and Chen}]{he2021debertav3}
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021{\natexlab{a}}.
\newblock Debertav3: Improving deberta using electra-style pre-training with
gradient-disentangled embedding sharing.
\newblock \emph{arXiv preprint arXiv:2111.09543}.
\bibitem[{He et~al.(2021{\natexlab{b}})He, Liu, Gao, and Chen}]{he2021deberta}
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021{\natexlab{b}}.
\newblock \href {https://openreview.net/forum?id=XPZIaotutsD} {Deberta:
Decoding-enhanced bert with disentangled attention}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Hendrycks and Gimpel(2016)}]{hendrycks2016gaussian}
Dan Hendrycks and Kevin Gimpel. 2016.
\newblock Gaussian error linear units (gelus).
\newblock \emph{arXiv preprint arXiv:1606.08415}.
\bibitem[{Hidey and McKeown(2016)}]{hidey-mckeown-2016-identifying}
Christopher Hidey and Kathy McKeown. 2016.
\newblock \href {https://doi.org/10.18653/v1/P16-1135} {Identifying causal
relations using parallel {W}ikipedia articles}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1424--1433,
Berlin, Germany. Association for Computational Linguistics.
\bibitem[{Khetan et~al.(2022)Khetan, Ramnani, Anand, Sengupta, and
Fano}]{KhetanEtAl_2022}
Vivek Khetan, Roshni Ramnani, Mayuresh Anand, Subhashis Sengupta, and Andrew~E.
Fano. 2022.
\newblock Causal bert: Language models for causality detection between events
expressed in text.
\newblock In \emph{Intelligent Computing}, pages 965--980, Cham. Springer
International Publishing.
\bibitem[{Khoo et~al.(2000)Khoo, Chan, and Niu}]{khoo-etal-2000-extracting}
Christopher S.~G. Khoo, Syin Chan, and Yun Niu. 2000.
\newblock \href {https://doi.org/10.3115/1075218.1075261} {Extracting causal
knowledge from a medical database using graphical patterns}.
\newblock In \emph{Proceedings of the 38th Annual Meeting of the Association
for Computational Linguistics}, pages 336--343, Hong Kong. Association for
Computational Linguistics.
\bibitem[{Lewis et~al.(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis2020bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
\newblock Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7871--7880.
\bibitem[{Lhoest et~al.(2021)Lhoest, del Moral, Jernite, Thakur, von Platen,
Patil, Chaumond, Drame, Plu, Tunstall et~al.}]{lhoest2021datasets}
Quentin Lhoest, Albert~Villanova del Moral, Yacine Jernite, Abhishek Thakur,
Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu,
Lewis Tunstall, et~al. 2021.
\newblock Datasets: A community library for natural language processing.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 175--184.
\bibitem[{Li et~al.(2021)Li, Li, Zou, and Ren}]{LI2021207}
Zhaoning Li, Qi~Li, Xiaotian Zou, and Jiangtao Ren. 2021.
\newblock \href {https://doi.org/https://doi.org/10.1016/j.neucom.2020.08.078}
{Causality extraction based on self-attentive bilstm-crf with transferred
embeddings}.
\newblock \emph{Neurocomputing}, 423:207--219.
\bibitem[{Liu et~al.(2021)Liu, Yuan, Fu, Jiang, Hayashi, and
Neubig}]{liu2021pre}
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and
Graham Neubig. 2021.
\newblock Pre-train, prompt, and predict: A systematic survey of prompting
methods in natural language processing.
\newblock \emph{arXiv preprint arXiv:2107.13586}.
\bibitem[{Liu et~al.(2019)Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis,
Zettlemoyer, and Stoyanov}]{liu2019roberta}
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
\newblock Roberta: A robustly optimized bert pretraining approach.
\newblock \emph{arXiv preprint arXiv:1907.11692}.
\bibitem[{Loshchilov and Hutter(2019)}]{loshchilov2018decoupled}
Ilya Loshchilov and Frank Hutter. 2019.
\newblock \href {https://openreview.net/forum?id=Bkg6RiCqY7} {Decoupled weight
decay regularization}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Man et~al.(2022)Man, Nguyen, and Nguyen}]{man-etal-2022-event}
Hieu Man, Minh Nguyen, and Thien Nguyen. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.starsem-1.28} {Event
causality identification via generation of important context words}.
\newblock In \emph{Proceedings of the 11th Joint Conference on Lexical and
Computational Semantics}, pages 323--330, Seattle, Washington. Association
for Computational Linguistics.
\bibitem[{Reimers and Gurevych(2019)}]{reimers2019sentence}
Nils Reimers and Iryna Gurevych. 2019.
\newblock Sentence-bert: Sentence embeddings using siamese bert-networks.
\newblock \emph{arXiv preprint arXiv:1908.10084}.
\bibitem[{Rutherford and Xue(2014)}]{rutherford-xue-2014-discovering}
Attapol Rutherford and Nianwen Xue. 2014.
\newblock \href {https://doi.org/10.3115/v1/E14-1068} {Discovering implicit
discourse relations through brown cluster pair representation and coreference
patterns}.
\newblock In \emph{Proceedings of the 14th Conference of the {E}uropean Chapter
of the Association for Computational Linguistics}, pages 645--654,
Gothenburg, Sweden. Association for Computational Linguistics.
\bibitem[{Sanh et~al.(2019)Sanh, Debut, Chaumond, and
Wolf}]{sanh2019distilbert}
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.
\newblock Distilbert, a distilled version of bert: smaller, faster, cheaper and
lighter.
\newblock \emph{arXiv preprint arXiv:1910.01108}.
\bibitem[{Tan et~al.(2022{\natexlab{a}})Tan, Hürriyetoğlu, Caselli, Oostdijk,
Hettiarachchi, Nomoto, Uca, and Liza}]{tan-etal-2022-event}
Fiona~Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk, Hansi
Hettiarachchi, Tadashi Nomoto, Onur Uca, and Farhana~Ferdousi Liza.
2022{\natexlab{a}}.
\newblock Event causality identification with causal news corpus - shared task
3, {CASE} 2022.
\newblock In \emph{Proceedings of the 5th Workshop on Challenges and
Applications of Automated Extraction of Socio-political Events from Text
(CASE 2022)}, Online. Association for Computational Linguistics.
\bibitem[{Tan et~al.(2022{\natexlab{b}})Tan, Hürriyetoğlu, Caselli, Oostdijk,
Nomoto, Hettiarachchi, Ameer, Uca, Liza, and Hu}]{tan-EtAl:2022:LREC}
Fiona~Anting Tan, Ali Hürriyetoğlu, Tommaso Caselli, Nelleke Oostdijk,
Tadashi Nomoto, Hansi Hettiarachchi, Iqra Ameer, Onur Uca, Farhana~Ferdousi
Liza, and Tiancheng Hu. 2022{\natexlab{b}}.
\newblock \href {https://aclanthology.org/2022.lrec-1.246} {The causal news
corpus: Annotating causal relations in event sentences from news}.
\newblock In \emph{Proceedings of the Language Resources and Evaluation
Conference}, pages 2298--2310, Marseille, France. European Language Resources
Association.
\bibitem[{Tan et~al.(2022{\natexlab{c}})Tan, Zuo, and Ng}]{Fiona2022}
Fiona~Anting Tan, Xinyu Zuo, and See-Kiong Ng. 2022{\natexlab{c}}.
\newblock \href {https://doi.org/10.48550/ARXIV.2208.09163} {Unicausal: Unified
benchmark and model for causal text mining}.
\bibitem[{Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz et~al.}]{wolf2020transformers}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et~al.
2020.
\newblock Transformers: State-of-the-art natural language processing.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 38--45.
\end{thebibliography}