Open Domain Event Extraction Using Neural Latent Variable Models
Extracted source text · arXiv 1906.06947
%
% File acl2019.tex
%
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%% taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2019}
\usepackage{times}
\usepackage{latexsym}
%\usepackage{stix}
% for math formulars
\usepackage{mathrsfs} % \mathscr{E}
\usepackage{amsmath} % \mathcal{E}
\usepackage{amsfonts} % \mathbb{E}
% for lists
\usepackage{enumerate}
% for tables
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{floatrow}
\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}
\newfloatcommand{capbtabbox}{table}[][\FBwidth]
% for algorithms
\usepackage{algorithmic}
\usepackage{algorithm}
% Change Text 'Algorithm' to 'Procedure'
%\floatname{algorithm}{Generative Story}
% Change Text 'Require' to 'Input'
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
% Change Text 'Ensure' to 'Output'
\renewcommand{\algorithmicensure}{\textbf{Output:}}
% for graphs
\usepackage{graphicx}
\usepackage{subfigure}
% for url links
%\usepackage{url}
\aclfinalcopy % Uncomment this line for the final submission
\def\aclpaperid{1839} % Enter the acl Paper ID here
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\newcommand\BibTeX{B\textsc{ib}\TeX}
\title{Open Domain Event Extraction Using Neural Latent Variable Models}
%\author{
% Xiao Liu\textsuperscript{$\dag$} \and Yue Zhang\textsuperscript{$\ddagger$} \and Heyan Huang\textsuperscript{$\dag$}\thanks{\textsuperscript{*}Corresponding author.} \\
% \textsuperscript{$\dag$}School of Computer Science and Technology, Beijing Institute of Technology \\
% 100081 Beijing, China \\
% {\tt \{xiaoliu,hhy63\}@bit.edu.cn} \\
% \textsuperscript{$\ddagger$}School of Engineering, Westlake University \\
% 310012 Hangzhou, Zhejiang, China \\
% {\tt yue.zhang@wias.org.cn}
%}
\author{
Xiao Liu\textsuperscript{1,2} \and Heyan Huang\textsuperscript{1,2} \and Yue Zhang\textsuperscript{3,4}\thanks{\ \ Corresponding author.} \\
\textsuperscript{1}School of Computer Science and Technology, Beijing Institute of Technology \\
% 100081 Beijing, China \\
\textsuperscript{2}Zhejiang Lab, China \\
{\tt \{xiaoliu,hhy63\}@bit.edu.cn} \\
\textsuperscript{3}School of Engineering, Westlake University \\
\textsuperscript{4}Institute of Advanced Technology, Westlake Institute for Advanced Study \\
% 310012 Hangzhou, Zhejiang, China \\
{\tt yue.zhang@wias.org.cn}
}
%\author{
% Xiao Liu\textsuperscript{$\spadesuit\clubsuit$} \and Heyan Huang\textsuperscript{$\spadesuit\clubsuit$} \and Yue Zhang\textsuperscript{$\vardiamondsuit$}\thanks{\ \ Corresponding author.} \\
% \textsuperscript{$\spadesuit$}School of Computer Science and Technology, Beijing Institute of Technology \\
%% 100081 Beijing, China \\
% \textsuperscript{$\clubsuit$}Zhejiang Lab, China \\
% {\tt \{xiaoliu,hhy63\}@bit.edu.cn} \\
% \textsuperscript{$\vardiamondsuit$}School of Engineering, Westlake University \\
%% 310012 Hangzhou, Zhejiang, China \\
% {\tt yue.zhang@wias.org.cn}
%}
\date{}
\begin{document}
\maketitle
\begin{abstract}
We consider open domain event extraction, the task of extracting unconstraint types of events from news clusters.
A novel latent variable neural model is constructed, which is scalable to very large corpus.
A dataset is collected and manually annotated, with task-specific evaluation metrics being designed.
Results show that the proposed unsupervised model gives better performance compared to the state-of-the-art method for event schema induction.
%There is a rapidly growing sets of news reports published by miscellaneous posts in the real world, leading to textual redundancy.
%However, most open domain schema induction models rely on sparse features and restrain the event types beforehand, training with learning methods which is not scalable in large corpus.
%In this paper, we propose a model to extract unconstraint types of events and induce universal schemas from news clusters by continuous contextual features, modeling latent event types and leveraging redundant ratios.
%Besides, our learning framework combines both the strengths of neural networks and latent variable models and is scalable to large corpus.
%For evaluation, we construct the GNBusiness dataset by merging news reports about the same events and introduce slot coherence as a new metrics to evaluate the intrinsic qualities quantitatively of learned slots and schemas.
%The experiment results demonstrate that the proposed model achieves competitive results compared with state-of-the-art methods.
\end{abstract}
\section{Introduction}
% introducing what is ESI (traditional)
Extracting events from news text has received much research attention.
The task typically consists of two subtasks, namely {\it schema induction}, which is to extract event templates that specify argument slots for given event types \cite{DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15,DBLP:conf/naacl/ShaLCS16,DBLP:conf/acl/HuangCFJVHS16,DBLP:conf/acl/Ahn17,DBLP:conf/cikm/0001RHZGHJL018}, and {\it event extraction}, which is to identify events with filled slots from a piece of news \cite{DBLP:conf/naacl/NguyenCG16,DBLP:conf/aaai/ShaQCS18,DBLP:conf/aaai/Liu00018,DBLP:conf/emnlp/0001Y00J18,DBLP:conf/acl/ChenXLZ015,DBLP:conf/acl/FengHTJQL16,DBLP:conf/emnlp/NguyenG16,DBLP:conf/emnlp/LiuLH18}.
Previous work focuses on extracting events from single news documents according to a set of pre-specified event types, such as arson, attack or earthquakes.
%Extracting events and inducing event schemas in the open-domain is a hot research topic.
%Inducing schemas is extracting templates that define specific types of events, each associated with slots held by entities as slot realizations.
%While extracting events is finding which types of event exist and filling in corresponding schemas according to texts.
%Traditionally, previous works on Event Schema Induction can be divided into two categories with the main idea of clustering potential event arguments as slots.
%One approach is using latent variable models with MCMC sampling to find schemas that best explain the text by treating slots as topics \cite{DBLP:conf/acl/ChambersJ11,DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15}.
%The other approach focuses on ad-hoc clustering algorithms in which similar entity mentions are clustered as slots \cite{DBLP:conf/acl/FilatovaHM06,DBLP:conf/acl/ChambersJ11,DBLP:conf/naacl/ShaLCS16,DBLP:conf/acl/Ahn17,DBLP:conf/cikm/0001RHZGHJL018}.
% use a paragraph to state the limitation.
While useful for tracking highly specific types of events from news, the above setting can be relatively less useful for decision making in security and financial markets, which can require comprehensive knowledge on broad-coverage, fine-grained and dynamically-evolving event categories.
In addition, given the fact that different news agencies can report the same events, redundancy can be leveraged for better event extraction.
In this paper, we investigate {\it open domain event extraction} (ODEE), which is to extract unconstraint types of events and induce universal event schemas from clusters of news reports.
%However, there is an explosive growing rate of news reports with various and unconstraint event types published by miscellaneous news posts, which leads to a rich textual redundancy in news reports.
%Both of the two approaches are not suitable for this situation because they restrain the event types as a fixed set in their methods and datasets beforehand, and rely on hand-crafted discrete features which is sparse in texts.
%Besides, the slots in schemas are assumed to subject to an identical uniform distribution regardless the actual event types also without specially considering the textual redundancy.
%Therefore, exploring unconstraint types of events and utilizing textual redundancy remains a problem.
%Besides, most works use collapsed Gibbs sampling to perform inference, which is also expensive and not scalable in a large dataset.
% use a paragraph to describe your task definition
% Need to be careful not to offend any reviewer who has similar task settings. So be very precise where you are different.
%Inspired by FrameNet \cite{DBLP:conf/acl/BakerFL98} and works on semantics role labeling,
%To remedy these problems, we propose the \textbf{O}pen \textbf{D}omain \textbf{E}vent \textbf{E}xtraction (ODEE) task to extract unconstraint types of events and induce universal event schemas from clusters of news reports that are about the same main event.
%There are two main differences between ODEE and the previous settings.
%First, open-domain event types are unlimited thus too many to enumerate, while previous works on MUC 4 \cite{DBLP:conf/muc/Sundheim92} only focus on four template types, e.g. \textit{arson}, \textit{attack}, \textit{bombing}, and \textit{kidnapping}, and \newcite{DBLP:conf/cikm/0001RHZGHJL018} crawl and classify five event types, e.g. \textit{business mergers \& acquisitions}, \textit{airplane crashes}, \textit{product announcements}, \textit{terrorist bombings}, and \textit{earthquakes}.
%Second, events are extracted from clusters of news in ODEE with sufficient redundant information to explore, while the documents in previous work have less redundant information.
% introduce my dataset and evaluation metric immediately after the task definition paragraph
%We find the lack of large public datasets with well-annotated and various typed events and schemas in the open domain.
%Therefore, we publish the GNBusiness dataset which has the largest volume ever for open domain event schema induction with diverse event types, explainable event schemas and rich redundant information.
%As for the evaluation metrics, despite the popular schema matching evaluation, we also adapt \textit{slot coherence} based on the normalized point-wise mutual information (NPMI) \cite{DBLP:conf/eacl/LauNB14} to provide a quantitative measure about the intrinsic qualities of learned slots and schema without annotations.
% talk about your model in a single high-level paragraph
%Previous works on event schema induction use latent variable models with discrete and head-crafted features, which are pretty sparse in the training corpus.
%In these works, the slots in schemas are assumed to subject to an identical uniform distribution regardless the actual event types.
%Additionally, none of them explicitly models the textual redundancy.
%Besides, most works use collapsed Gibbs sampling to perform inference, which is also expensive and not scalable in a large dataset.
%In this paper, we propose a scalable neural latent variable model for event schema induction and assemble events using dependency parse trees.
%To be scalable in large corpus and combine the power of neural inference network, we adapt AVITM \cite{srivastava17lda} to perform inference and learn our model.
%First, we use ELMo \cite{DBLP:conf/naacl/PetersNIGCLZ18}, the pre-trained language representation model, to encode the contextualized textual features of entity mentions.
%Second, we sample a special latent event type vector for each news cluster from a global parameterized normal distribution to model its unique slot distribution.
%Last but not least, we leverage the textual redundancy from news clusters by computing redundant ratios for each entity.
%The learning framework of our model is also extendable and offers a pilot example of combining the power of learning complex transformation with neural networks and explaining the dataset with latent variable models.
\begin{figure}
\centering
\includegraphics[width=75mm]{imgs/setting_comparison.pdf}
\caption{Comparison between MUC 4 and ODEE.}
\label{fig:related_work}
\end{figure}
As shown in Figure \ref{fig:related_work}, compared with traditional event extraction task exemplified by MUC 4 \cite{DBLP:conf/muc/Sundheim92}, the task of ODEE poses additional challenges to modeling, which have not been considered in traditional methods.
First, more than one event can be extracted from a news cluster, where events can be flexible in having varying numbers of slots in the open domain, and slots can be flexible without identical distributions regardless of the event type, which has been assumed by previous work on schema induction.
%event types are unconstrained
Second, mentions of the same entities from different reports in a news cluster should be taken into account for improved performance.
%redundancy information
We build an unsupervised generative model to address these challenges.
While previous work on generative schema induction \cite{DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15} relies on hand-crafted indicator features, we introduce latent variables produced by neural networks for better representation power.
A novel graph model is designed, with a latent event type vector for each news cluster from a global parameterized normal distribution, and textual redundancy features for entities.
Our model takes advantage of contextualized pre-trained language model (ELMo, \newcite{DBLP:conf/naacl/PetersNIGCLZ18}) and scalable neural variational inference \cite{srivastava17lda}.
To evaluate model performance, we collect and annotate a large-scale dataset from Google Business News\footnote{\url{https://news.google.com/?hl=en-US&gl=US&ceid=US:en}, crawled from Oct. 2018 to Jan. 2019.} with diverse event types and explainable event schemas.
In addition to the standard metrics for schema matching, we adapt {\it slot coherence} based on NPMI \cite{DBLP:conf/eacl/LauNB14} for quantitatively measuring the intrinsic qualities of slots and schemas, which are inherently clusters.
% talk about results
%As far as we know, it is the first model combining neural networks with latent variable models for event schema induction.
%By comparing our approach with the state-of-the-art event schema induction methods, our model achieve the best results.
%Additionally, visualization on learned parameters also shows that our model can learn reasonable latent event type vectors.
%To our knowledge, we are the first to use neural latent variable model for inducing event schemas.
%We release our code and data at \url{https://}\footnote{We will make the URL public after being accpeted.}.
Results show that our neural latent variable model outperforms state-of-the-art event schema induction methods.
In addition, redundancy is highly useful for improving open domain event extraction.
Visualizations of learned parameters show that our model can give reasonable latent event types.
To our knowledge, we are the first to use neural latent variable model for inducing event schemas and extracting events.
We release our code and dataset at \url{https://github.com/lx865712528/ACL2019-ODEE}.
%\footnote{We will make the URL public after being accepted.}
%As a summary, our major contributions are:
%\begin{enumerate}
% \item We propose a novel model for learning event schemas and extracting events without the constraints of event types from news reports using continuous contextual features, modeling latent event types and textual redundancy.
% \item Our learning framework is scalable to large corpus and combines both the strengths of neural networks and latent variable models.
% \item We also publish the GNBusiness dataset for our ODEE task and introduce slot coherence as evaluation metrics for evaluating intrinsic qualities for learned slots and schemas.
%\end{enumerate}
\section{Related Work}
%Our work is related to the following four different research topics.
The most popular schema induction and event extraction task setting is MUC 4, in which four event types - \textit{Arson}, \textit{Attack}, \textit{Bombing} and \textit{Kidnapping} - and four slots - \textit{Perpetrator}, \textit{Instrument}, \textit{Target} and \textit{Victim} - are defined.
We compare the task settings of MUC 4 and ODEE in Figure \ref{fig:related_work}.
%For each input news document for MUC 4, the output is four types of events with schemas consisting of fixed slots.
For MUC 4, the inputs are single news documents, and the output belongs to four types of events with schemas consisting of fixed slots.
%For ODEE, in contrast, for each input news cluster, the output is unconstrained types of open domain events and unique schemas with various slot combinations.
For ODEE, in contrast, the inputs are news clusters rather than the individual news, and the output is unconstrained types of open domain events and unique schemas with various slot combinations.
\textbf{Event Schema Induction} seminal work studies patterns \cite{DBLP:conf/naacl/ShinyamaS06,DBLP:conf/acl/FilatovaHM06,DBLP:conf/ijcnlp/QiuKC08} and event chains \cite{DBLP:conf/acl/ChambersJ11} for template induction.
For MUC 4, the current dominant methods include probabilistic generative methods \cite{DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15} that jointly model predicate and argument assignment, and ad-hoc clustering algorithms for inducing slots \cite{DBLP:conf/naacl/ShaLCS16,DBLP:conf/acl/HuangCFJVHS16,DBLP:conf/acl/Ahn17,DBLP:conf/cikm/0001RHZGHJL018}.
These methods all rely on hand-crafted discrete features without fully model the textual redundancy.
There are also works on modeling event schemas and scripts using neural language models \cite{DBLP:conf/conll/ModiT14,DBLP:conf/emnlp/RudingerRFD15,DBLP:conf/acl/PichottaM16}, but they do not explore neural latent variables and redundancy.
%However, these works restrain the event types as a fixed set beforehand and they assume that the slots in schemas subject to an identical uniform distribution regardless the actual event types.
%normalised cut approach to maximize intra-class similarity within slots \cite{DBLP:conf/naacl/ShaLCS16}, and reversed schema induction steps that distinguish functions and themes among words \cite{DBLP:conf/acl/Ahn17}.
%For specific domain or corpus, \newcite{DBLP:conf/acl/HuangCFJVHS16} combine the merits of symbolic semantics and distributed semantics to cluster event type and argument roles.
%While \newcite{DBLP:conf/cikm/0001RHZGHJL018} propose a new task called Open-Schema Event Profiling in which they extract event profiling slot-value pairs and induce schemas of pre-defined five event types by slot expressions and patterns at a corpus level with pre-limited event types.
%Previous works on automatic pattern extraction \cite{DBLP:conf/naacl/ShinyamaS06,DBLP:conf/acl/FilatovaHM06,DBLP:conf/ijcnlp/QiuKC08} and event chain pattern \cite{DBLP:conf/acl/ChambersJ11} develop the early stage of template induction.
%Then probabilistic generative methods that jointly model the assignment of predicates to event schemas and arguments to event roles become mainstream.
%There are methods using entity-driven model to link coreferring arguments to the same event role \cite{DBLP:conf/emnlp/Chambers13}, a pair of HMMs focusing on event clauses and model transitions \cite{DBLP:conf/naacl/CheungPV13}, phrase modifiers to argument similarity scores \cite{DBLP:conf/acl/NguyenTFB15}, normalised cut approach to maximize intra-class similarity within slots \cite{DBLP:conf/naacl/ShaLCS16}, and reversed schema induction steps that distinguish functions and themes among words \cite{DBLP:conf/acl/Ahn17}.
%For specific domain or corpus, \newcite{DBLP:conf/acl/HuangCFJVHS16} combine the merits of symbolic semantics and distributed semantics to cluster event type and argument roles.
%While \newcite{DBLP:conf/cikm/0001RHZGHJL018} propose a new task called Open-Schema Event Profiling in which they extract event profiling slot-value pairs and induce schemas of pre-defined five event types by slot expressions and patterns at a corpus level with pre-limited event types.
%However, our work is different from the above mentioned works because we benefit from the redundant information that is complementary to the single documents in news clusters.
%Besides, our work is able to explore the unlimited event types in the open domain by using real-valued latent event type vectors.
%%%%%%%% Discard the following materials.
%\newcite{DBLP:conf/naacl/ShinyamaS06} use ``basic patterns'' to cluster different entities pairs to generate IE results.
%Their evaluation method is a kind of manual evaluation.
%This work is more like an unsupervised work.
%
%\newcite{DBLP:conf/acl/FilatovaHM06} introduce their work on creating domain templates in a collection of documents with the same topic.
%Their evaluation method can produce a precent score.
%
%\newcite{DBLP:conf/ijcnlp/QiuKC08} construct graphs for documents with predicate-argument tuples as vertices and their contextual relations as edges.
%They use these the similarities of the graphs to measure the similarities of contexts, and then cluster the similar contexts to induce templates.
%Their evaluation method is manually comparing to MUC-4 and a balanced news corpus.
%\newcite{DBLP:conf/emnlp/BalasubramanianSME13} use cooccurrence statistics of semantically typed relational triples, which they call Rel-grams (relational n-grams).
%In a human evaluation, their schemas outperform Chambers’s schemas by wide margins on several evaluation criteria.
%\newcite{DBLP:conf/eacl/FerreroPQ17}
%\section{Narrative Chains and Schemas}
%\newcite{DBLP:conf/acl/ChambersJ08} is the first work on extracting Narrative Chains.
%The narrative chains are not self-contained \textit{scripts} introduced in \newcite{schank2013scripts}.
%A narrative event in a narrative chain is consisted of two parts: event trigger (most simply a verb) and its participants, represented as typed dependencies (subject or object).
%
%\newcite{DBLP:conf/acl/ChambersJ09} extends the definition of narrative chains to types narrative chains, and induce narrative chain schema by chain linking and clustering.
%%%%%%%% Discard the above materials.
\textbf{Event Extraction} work typically assumes that event schemas are given, recognizing event triggers and their corresponding arguments.
This can be regarded as a subtask of ODEE.
%There are several approaches exploiting various sentence-level features, such as ranking dependencies \cite{DBLP:conf/acl/McCloskySM11}, combinational features of triggers and arguments \cite{DBLP:conf/acl/LiJH13}, probabilistic soft logic information \cite{DBLP:conf/acl/LiuCHL016}, trigger-specific features and relational features \cite{DBLP:conf/naacl/YangM16,DBLP:conf/emnlp/KeithHPMMO17}.
%Others also seek features in document-level methods \cite{DBLP:conf/acl/LiaoG10,DBLP:conf/acl/JiG08,DBLP:conf/acl/HongZMYZZ11,DBLP:conf/naacl/ReichartB12}.
%Besides, there are also a group of deep learning methods using RNNs \cite{DBLP:conf/naacl/NguyenCG16,DBLP:conf/aaai/ShaQCS18,DBLP:conf/aaai/Liu00018,DBLP:conf/emnlp/0001Y00J18}, CNNs \cite{DBLP:conf/acl/ChenXLZ015,DBLP:conf/acl/FengHTJQL16,DBLP:conf/emnlp/NguyenG16} and GCNs \cite{DBLP:conf/emnlp/LiuLH18} capturing the associations between events.
Existing work exploits sentence-level \cite{DBLP:conf/acl/McCloskySM11,DBLP:conf/acl/LiJH13,DBLP:conf/acl/LiuCHL016,DBLP:conf/naacl/YangM16} and document-level statistics \cite{DBLP:conf/acl/LiaoG10,DBLP:conf/acl/JiG08,DBLP:conf/acl/HongZMYZZ11,DBLP:conf/naacl/ReichartB12}.
There has also been work using RNNs \cite{DBLP:conf/naacl/NguyenCG16,DBLP:conf/aaai/ShaQCS18,DBLP:conf/aaai/Liu00018,DBLP:conf/emnlp/0001Y00J18}, CNNs \cite{DBLP:conf/acl/ChenXLZ015,DBLP:conf/acl/FengHTJQL16,DBLP:conf/emnlp/NguyenG16} and GCNs \cite{DBLP:conf/emnlp/LiuLH18} to represent sentences of events.
Event extraction has been treated as a supervised or semi-supervised \cite{DBLP:conf/coling/LiaoG10,DBLP:conf/eacl/HuangR12} task.
In contrast, ODEE is a fully unsupervised setting.
%Additionally, research has also been done to learn events in semi-supervised ways using bootstrapping solutions \cite{DBLP:conf/coling/LiaoG10,DBLP:conf/eacl/HuangR12}.
%We assume that in our work the event schemas are not given in advance and we need to learn them by analyzing the open-domain corpus, which is the biggest difference between our work the most of the works on event extraction.
\textbf{Event Discovery in Tweet Streams} extracts news-worthy clusters of words, segments and frames.
Both supervised and unsupervised methods have been used.
The former \cite{DBLP:conf/www/SakakiOM10,DBLP:conf/acl/BensonHB11} are typically designed to monitor certain event types, while the latter cluster features according to their burstiness \cite{DBLP:conf/icwsm/BeckerNG11,DBLP:conf/cikm/CuiZLMZ12,DBLP:conf/cikm/LiSD12,DBLP:conf/kdd/RitterMEC12,DBLP:conf/ijcnlp/QinZZZ13,DBLP:conf/www/IfrimSB14,DBLP:conf/clef/McMinnJ15,DBLP:conf/ialp/QinZZZ17}.
%Different feature representations have been investigated, including words \cite{DBLP:conf/clef/McMinnJ15,DBLP:conf/cikm/CuiZLMZ12}, segments \cite{DBLP:conf/cikm/LiSD12,DBLP:conf/ijcnlp/QinZZZ13}, tweets themselves \cite{DBLP:conf/icwsm/BeckerNG11,DBLP:conf/www/IfrimSB14} and semantic frames \cite{DBLP:conf/ialp/QinZZZ17}.
This line of work is similar to our work in using information redundancy, but different because we focus on formal news texts and induce structural event schemas.
%The most significant difference between our work and this topic is that our work focuses on open domain news clusters which have different genres from tweets, and thus we need to develop different methods.
%Our work is also related to Event Discovery in Tweet Streams which can assist public opinion monitoring, advertising and brand image maintenance.
%Both supervised and unsupervised methods have been used for tweet event discovery.
%While supervised methods \cite{DBLP:conf/www/SakakiOM10,DBLP:conf/acl/BensonHB11} are typically designed to monitor certain event types, such as earthquakes and concerts, unsupervised methods can be used for open domain events.
%\newcite{DBLP:conf/kdd/RitterMEC12} propose a semi-supervised event detection method focusing on extracting open domain events in specified structures including named entities, event phrases, time and event category.
%As for the unsupervised methods, existing methods typically cluster feature representations via their burstiness, and filter out more news worthy clusters as outputs.
%Different feature representations have been investigated, which include words \cite{DBLP:conf/clef/McMinnJ15,DBLP:conf/cikm/CuiZLMZ12}, segments \cite{DBLP:conf/cikm/LiSD12,DBLP:conf/ijcnlp/QinZZZ13} , tweets themselves \cite{DBLP:conf/icwsm/BeckerNG11,DBLP:conf/www/IfrimSB14} and semantic frames \cite{DBLP:conf/ialp/QinZZZ17}.
%
%The most significant difference between our work and Event Discovery in Tweet Streams is that our work focuses on open domain news clusters which have different genres from tweets, and thus we need to develop different methods.
\textbf{First Story Detection} (FSD) systems aim to identify news articles that discuss events not reported before.
Most work on FSD detects first stories by finding the nearest neighbors of new documents \cite{DBLP:conf/naacl/KumaranA05,DBLP:conf/sigir/MoranMMO16,DBLP:conf/coling/PanagiotouATKG16,DBLP:conf/sigir/VuurensV16}.
This line of work exploits textual redundancy in massive streams predicting whether or not a document contains a new event as a classification task.
In contrast, we study the event schemas and extract detailed events.
% using locality sensitive hashing, paraphrases and entities information, for each news document
\section{Task and Data}
%We propose the task of Open Domain Event Extraction (ODEE) to extract open-domain events with unlimited types and inducting schemas from clusters of news that each cluster contains news reports about the same event in the real world.
%\begin{figure}
% \centering
% \includegraphics[width=75mm]{imgs/data_annotation.pdf}
% \caption{Data annotation for a news cluster about \textit{UnitedHealth shares rise}. The blue entities and the yellow ones consist of two events respectively.}
% \label{fig:data_annotation}
%\end{figure}
\noindent\textbf{Task Definition.}
In ODEE, the input consists of news clusters, each containing reports about the same event.
The output is a bag of open-domain events, each consisting of an event trigger and a list of event arguments in its own schema.
In most cases, one event is semantically sufficient to represent the output.
Formally, given an open-domain news corpus $\mathcal{N}$ containing a set of news clusters $\{ c \in \mathcal{N} \}$, suppose that there are $M_c$ news reports $\{ d_i \in c | i = 1, \dotsb, M_c \}$ in the news cluster $c$ focusing on the same event $ \mathcal{E}_c $.
The output is a pair ($ \mathcal{E}_c,\mathcal{T}_{\mathcal{E}}$), where $ \mathcal{E}_c $ is the aforementioned set of open-domain events and $\mathcal{T}_{\mathcal{E}}$ is a set of schemas that define the semantic slots for this set of events.
% using semantic slots $\{ s \in \mathcal{T}_{\mathcal{E}} \}$.
\noindent\textbf{Data Collection.}
We crawl news reports from Google Business News, which offers news clusters about the same events from different sources.
In each news cluster, there are no more than five news reports.
For each news report, we obtain the title, publish timestamp, download timestamp, source URL and full text.
In total, we obtain 55,618 business news reports with 13,047 news clusters in 288 batches from Oct. 17, 2018, to Jan. 22, 2019.
The crawler is executed about three times per day.
%5. our steps 3 - manually annotation and schema introduction
%To keep the redundant information in each news cluster, we treat each news cluster as the document in other dataset like MUC 4, ACE 2005\footnote{\url{https://catalog.ldc.upenn.edu/LDC2006T06}}, ERE\footnote{\url{https://catalog.ldc.upenn.edu/LDC2013E64}}, etc.
The full text corpus is released as \textit{GNBusiness-Full-Text}.
For this paper, we trim the news reports in each news cluster by keeping the title and first paragraph, releasing as \textit{GNBusiness-All}.
% annotation schema
Inspired by the general slots in FrameNet \cite{DBLP:conf/acl/BakerFL98}, we design reference event schemas for open domain event types, which include eight possible slots: \textit{Agent}, \textit{Patient}, \textit{Time}, \textit{Place}, \textit{Aim}, \textit{Old Value}, \textit{New Value} and \textit{Variation}.
\textit{Agent} and \textit{Patient} are the semantic agent and patient of the trigger, respectively;
\textit{Aim} is the target or reason for the event.
%entity why \textit{Agent} does something to \textit{Patient}.
If the event involves value changes, \textit{Old Value} serves the old value, \textit{New Value} serves the new value and \textit{Variation} is the variation between \textit{New Value} and \textit{Old Value}.
Note that the roles that we define are more thematic and less specific to detailed events as some of the existing event extraction datasets do \cite{DBLP:conf/muc/Sundheim92,DBLP:conf/lrec/NguyenTFB16}, because we want to make our dataset general and useful for a wide range of open domain conditions.
We leave finer-grained role typing to future work.
% annotation and data split
We randomly select 18 batches of news clusters, with 680 clusters in total, dividing them into a development set and a test set by a ratio of $1:5$.
The development set, test set and the rest unlabeled clusters are released as \textit{GNBusiness-Dev}, \textit{GNBusiness-Test} and \textit{GNBusiness-Unlabeled}, respectively.
One coauthor and an external annotator manually label the events in the news clusters as gold standards.
For each news cluster, they assign each entity which participants in the event or its head word a beforehand slot.
%Figure \ref{fig:data_annotation} gives an annotation example for a news cluster.
The inter-annotator agreement (IAA) for each slot realization in the development set has a Cohen's kappa \cite{cohen1960coefficient} $\kappa=0.7$.
%a Fleiss' kappa \cite{fleiss1971measuring}
\begin{table}
\centering
\small
\setlength{\tabcolsep}{1mm}{
\begin{tabular}{l|c|c|c|c}
%\toprule[1pt]
\hline
\textbf{Split} & \textbf{\#C} & \textbf{\#R} & \textbf{\#S} & \textbf{\#W} \\
\hline
Test & 574 & 2,433 & 5,830 & 96,745 \\
Dev & 106 & 414 & 991 & 16,839 \\
Unlabelled & 12,305 & 52,464 & 127,416 & 2,101,558 \\
\hline
All & 12,985 & 55,311 & 134,237 & 2,215,142 \\
\hline
Full-Text & 12,985 & 55,311 & 1,450,336 & 31,103,698 \\
\hline
%\bottomrule[1pt]
\end{tabular}
}
\vspace{-0.2cm}
\caption{\label{tab:statistics_data_splits} Data split statistics. ($C$ news clusters; $R$ news reports; $S$ sentences; $W$ words.)}
\end{table}
\begin{table}
\centering
\small
\begin{tabular}{l|c|c|c|c}
%\toprule[1pt]
\hline
\textbf{Dataset} & \textbf{\#D} & \textbf{\#L} & \textbf{\#T} & \textbf{\#S} \\
\hline
MUC 4 & 1700 & 400 & 4 & 4 \\
ACE 2005 & 599 & 599 & 33 & 36 \\
ERE & 562 & 562 & 38 & 27 \\
ASTRE & 1038 & 100 & 12 & 18 \\
\textbf{GNBusiness} & 12,985 & 680 & -- & 8 \\
\hline
%\bottomrule[1pt]
\end{tabular}
\vspace{-0.2cm}
\caption{\label{tab:dataset_statistics_comp} Comparison with existing datasets. ($D$ documents or news clusters; $L$ labeled documents or news clusters; $T$ event types; $S$ slots.)}
\end{table}
\begin{figure*}
\centering
%\subfigure[]{
%\begin{minipage}{8cm}
%\centering
%\includegraphics[width=5cm]{imgs/fig1.eps}
%\label{fig1.1}
%\end{minipage}%
%}%
\subfigure[\textit{ODEE-F}]{
%\begin{minipage}{6cm}
\centering
\includegraphics[width=4cm]{imgs/fig2.eps}
\label{fig:odee-f}
%\end{minipage}%
}%
\subfigure[\textit{ODEE-FE}]{
%\begin{minipage}{4cm}
\centering
\includegraphics[width=4cm]{imgs/fig3.eps}
\label{fig:odee-fe}
%\end{minipage}%
}%
\subfigure[\textit{ODEE-FER}]{
%\begin{minipage}{6cm}
\centering
\includegraphics[width=4cm]{imgs/fig4.eps}
\label{fig:odee-fer}
%\end{minipage}%
}
%\vspace{-2mm}
\caption{Plate notations for models.
($S$ -- \# of slots; $E$ -- \# of entities; $C$ -- \# of news clusters; $V$ -- head word vocabulary size; the grey circles are observed variables and the white circles are hidden variables.)}
%(a) \textit{ODEE-F}: replacing discrete features by distributional features learned in pre-trained neural modules;
%(b) \textit{ODEE-FE}: exploiting latent event types;
%(c) \textit{ODEE-FER}: leveraging redundant information.
%The letter $S$ stands for the assumed slot numbers, while the letter $E$ represents the number of different entities.
%$D$ and $C$ means the number of documents and the clusters in the corpus.
%$V$ represents the head word vocabulary size and $n$ represents the vector dimension of $f$.
\label{fig1}
\end{figure*}
%6. our steps 4 - data split statistics and comparison
The statistics of each data split is shown in Table \ref{tab:statistics_data_splits}, and a comparison with existing event extraction and event schema induction datasets, including ASTRE \cite{DBLP:conf/lrec/NguyenTFB16}, MUC 4, ACE 2005\footnote{\url{https://catalog.ldc.upenn.edu/LDC2006T06}} and ERE\footnote{\url{https://catalog.ldc.upenn.edu/LDC2013E64}}, is shown in Table \ref{tab:dataset_statistics_comp}.
Compared with the other datasets, GNBusiness has a much larger number of documents (i.e., news clusters in GNBusiness), and a comparable number of labeled documents.
%The most promising is that there are unlimited event types in our open domain setting while the event types are fixed in others.
\section{Method}
%As described in Section 3, there are two main differences between ODEE and the task in MUC-4: unlimited event types in the open domain and textual redundancy.
%Therefore, some modifications and improvements must be accomplished from the models in MUC-4.
%Figure \ref{fig1} shows the step-by-step improvements of model variants.
%Let us follow the figure to introduce our neural network latent variable extractor and start with the baseline \cite{DBLP:conf/acl/NguyenTFB15} in MUC-4.
%\subsection{Baseline in MUC-4}
%
%We choose the state-of-the-art model \cite{DBLP:conf/acl/NguyenTFB15} under the setting of MUC-4 as our baseline model, which is shown in Figure \ref{fig1.1}.
%In this model, an entity is represented as a triple containing a head word $h$, a list $A$ of attribute relations and a list $T$ of predicate relations.
%The input of the model is a sequence of entity triples while document boundary is only used in a post-processing step.
%Head words are extracted from noun phrases.
%Attribute relations and predicate relations are extracted from the certain edges in dependency parse trees.
%
%\begin{algorithm}
%\caption{Baseline in Figure \ref{fig1.1}}
%\label{alg1}
%\algsetup{indent=1em,linenosize=\footnotesize}
%\footnotesize
%\begin{algorithmic}[1]
% \STATE Sample the slot-head distribution $\lambda \sim \textrm{Dirichlet}(\alpha)$
% \STATE Sample the slot-attribute distribution $\theta \sim \textrm{Dirichlet}(\beta)$
% \STATE Sample the slot-predicate distribution $\phi \sim \textrm{Dirichlet}(\gamma)$
% \FOR{each entity $e \in E$}
% \STATE Sample a slot $s \sim \textrm{Uniform}(1, S)$
% \STATE Sample a head $h \sim \textrm{Multinomial}(1, \lambda_s)$
% \FOR{$i \gets 1 \textrm{ to } A$}
% \STATE Sample an attribute $a \sim \textrm{Multinomial}(1, \theta_s)$
% \ENDFOR
% \FOR{$i \gets 1 \textrm{ to } T$}
% \STATE Sample a predicate $t \sim \textrm{Multinomial}(1, \phi_s)$
% \ENDFOR
% \ENDFOR
%\end{algorithmic}
%\end{algorithm}
%
%The generative story is in the generative story \ref{alg1}.
%And the joint probability of the entire entity set $E$ in the whole corpus is defined as
%\begin{align*}
%p_{\lambda,\theta,\phi}(E) = & \prod_{e \in E} p_{\lambda,\theta,\phi}(e) \tag{1}\\
%p_{\lambda,\theta,\phi}(e) = &P(s) \times p_{\lambda}(h|s) \times \\
%& \prod_{a \in A} p_{\theta}(a|s) \times \prod_{t \in T} p_{\phi}(t|s) \tag{2}
%\end{align*}
%By comparing the settings in MUC-4 and ODEE, We find that the baseline model is not able to do well in ODEE because of several reasons.
%First, as our ODEE-GN dataset is several times bigger than the MUC-4 dataset, extracting sparse and discrete features will dramatically affect the model’s size and performance and is not desirable when considering scalability.
%Second, the baseline model lacks the ability to determine event types, and it relies on an ordinary assumption that slots subject to the uniform distribution regardless of the influence of certain event types, which ignores the characteristics of different event types.
%The last but not least, it does not fully model the textual redundancy that could help to induce schemas and extract events better.
We investigate three incrementally more complex neural latent variable models for ODEE.
\subsection{Model 1}
Our first model is shown in Figure \ref{fig:odee-f}.
It can be regarded as a neural extension of \newcite{DBLP:conf/acl/NguyenTFB15}.
Given a corpus $\mathcal{N}$, we sample a slot $s$ for each entity $e$ from a uniform distribution of $S$ slots, and then a head word $h$ from a multinomial distribution, as well as a continuous feature vector $f \in \mathbb{R}^n$ produced by a contextual encoder.
% as $f=\textrm{encoder}(\textrm{contexts}(h))$, where the function $\textrm{contexts}(\cdot)$ means getting the context around a head word.
For simplicity, we assume that $f$ follows a multi-variable normal distribution whose covariance matrix is a diagonal matrix.
We mark all the parameters (mean vectors and diagonal vectors of covariance matrixes) for the $S$ different normal distributions for $f$ as $\beta \in \mathbb{R}^{S \times 2n}$, where $n$ represents the dimension of $f$, treating the probability matrix $\lambda \in \mathbb{R}^{S \times V}$ in the slot-head distribution as parameters under the row-wise simplex constraint, where $V$ is the head word vocabulary size.
% that each row is a simplex over the vocabulary length $V$ of head words.
We call this model \textit{ODEE-F}.
\begin{algorithm}[t]
\caption{\textit{ODEE-F}}
\label{alg2}
\algsetup{indent=1em,linenosize=\footnotesize}
\footnotesize
\begin{algorithmic}[1]
\FOR{each entity $e \in E$}
\STATE Sample a slot $s \sim \textrm{Uniform}(1, S)$
\STATE Sample a head $h \sim \textrm{Multinomial}(1, \lambda_s)$
\STATE Sample a feature vector $f \sim \textrm{Normal}(\beta)$
\ENDFOR
\end{algorithmic}
\end{algorithm}
%As the features used in previous models are discrete and hard-coded, which are not suitable for exploring more and broader feature space and are not scalable to bigger datasets, we prefer using distributed feature representations from a certain semantics space as our contextual features.
Pre-trained contextualized embeddings such as ELMo \cite{DBLP:conf/naacl/PetersNIGCLZ18}, GPTs \cite{radford2018improving,radford2019language} and BERT \cite{DBLP:journals/corr/abs-1810-04805} give improvements on a range of natural language processing tasks by offering rich language model information.
%They can extract the strong and effective contextualized features for each token in sentences.
%Therefore, we can use them as encoders to get contextualized features for each token in sentences.
We choose ELMo\footnote{In practice, we use the ``small'' ELMo model with $2\times128$-d output in \url{https://allennlp.org/elmo} as initial parameters and fine-tune it on \textit{GNBusiness-Full-Text}.} as our contextual feature encoder, which manipulates unknown words by using character representations.
%because it is more robust when manipulating unknown words by using character representations in the open domain.
The generative story is shown in Algorithm \ref{alg2}.
The joint probability of an entity $e$ is
\begin{align*}
%p_{\lambda,\beta}(E) = & \prod_{e \in E} p_{\lambda,\beta}(e) \tag{1}\\
p_{\lambda,\beta}(e) = p(s) \times p_{\lambda}(h|s) \times p_{\beta}(f|s) \tag{1}
\end{align*}
%As the features used in previous models are discrete and hard-coded, which are not suitable for exploring more and broader feature space and are not scalable to bigger datasets, we prefer using distributed feature representations from a certain semantics space as our contextual features.
\begin{algorithm}[t]
\caption{\textit{ODEE-FE}}
\label{alg3}
\algsetup{indent=1em,linenosize=\footnotesize}
\footnotesize
\begin{algorithmic}[1]
\FOR{each news cluster $c \in \mathcal{N}$}
\STATE Sample a latent event type vector $t \sim \textrm{Normal}(\alpha)$
\FOR{each entity $e \in E_c$}
\STATE Sample a slot $s \sim \textrm{Multinomial}(\textrm{MLP}(t;\theta))$
\STATE Sample a head $h \sim \textrm{Multinomial}(1, \lambda_s)$
\STATE Sample a feature vector $f \sim \textrm{Normal}(\beta_s)$
\ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}
\subsection{Model 2}
A limitation of \textit{ODEE-F} is that sampling slot assignment $s$ from a global uniform distribution does not sufficiently model the fact that different events may have different slot distributions.
%The characteristics of different event types are important in the open domain because there are unlimited event types and their characteristics can somehow encode their preferences of slots.
Thus, in Figure \ref{fig:odee-fe}, we further sample a latent event type vector $t \in \mathbb{R}^n$ for each news cluster from a global normal distribution parameterized by $\alpha$.
We then use $t$ and a multi-layer perceptron (MLP) with parameters $\theta$ to encode the corresponding slot distribution logits, sampling a discrete slot assignment $s \sim \textrm{Multinomial}(\textrm{MLP}(t;\theta))$.
The output of the MLP is passed through a softmax layer before being used.
%Considering the implement of learning the model, we find that if we use gradient-based inference and training methods, it will be possible to learn the model.
We name this model as \textit{ODEE-FE}.
The generative story is shown in Algorithm \ref{alg3}.
The joint probability of a news cluster $c$ is
\begin{align*}
%p_{\alpha,\beta,\theta,\lambda}(\mathcal{N}) = & \prod_{d \in \mathcal{N}} p_{\alpha,\beta,\theta,\lambda}(d) \tag{3}\\
p_{\alpha,\beta,\theta,\lambda}(c) = & \,p_{\alpha}(t) \times \prod_{e \in E_c} p_{\theta}(s|t) \\
\times & \,p_{\lambda}(h|s) \times p_{\beta}(f|s) \tag{2}
\end{align*}
\subsection{Model 3}
Intuitively, the more frequently a coreferential entity shows up in a news cluster, the more likely it is with an important slot.
Beyond that, different news agencies focus on different aspects of event arguments, which can offer complementary information through textual redundancy.
%if extremely rare a coreferential entity is, it also may be linked to a complementary slot.
One intuition is that occurrence frequency is a straightforward measure for word-level redundancy.
Thus, in Figure \ref{fig:odee-fer}, we additionally bring in the normalized occurrence frequency of a coreferential slot realization as an observed latent variable $r \sim \textrm{Normal}(\gamma_s)$.
We call this model \textit{ODEE-FER}.
%\footnote{For each news cluster, we use the maximum occurrence frequency to normalize the occurrence frequencies to $[0, 1]$.}
\begin{algorithm}[t]
\caption{\textit{ODEE-FER}}
\label{alg4}
\algsetup{indent=1em,linenosize=\footnotesize}
\footnotesize
\begin{algorithmic}[1]
\FOR{each news cluster $c \in \mathcal{N}$}
\STATE Sample a latent event type vector $t \sim \textrm{Normal}(\alpha)$
\FOR{each entity $e \in E_c$}
\STATE Sample a slot $s \sim \textrm{Multinomial}(\textrm{MLP}(t;\theta))$
\STATE Sample a head $h \sim \textrm{Multinomial}(1, \lambda_s)$
\STATE Sample a feature vector $f \sim \textrm{Normal}(\beta_s)$
\STATE Sample a redundancy ratio $r \sim \textrm{Normal}(\gamma_s)$
\ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}
Formally, a news cluster $c$ receives a latent event type vector $t$ where each entity $e \in E_c$ receives a slot type $s$.
The generative story is shown in Algorithm \ref{alg4}.
The joint distribution of a news cluster with head words, redundant contextual features and latent event type is
\begin{align*}
%p_{\alpha,\beta,\gamma,\theta,\lambda}(\mathcal{N}) = & \prod_{c \in \mathcal{N}} p_{\alpha,\beta,\gamma,\theta,\lambda}(c) \tag{5}\\
p_{\alpha,\beta,\gamma,\theta,\lambda}(c) = & \,p_{\alpha}(t) \times \prod_{e \in E_c} p_{\theta}(s|t) \\
\times & \,p_{\lambda}(h|s) \times p_{\beta}(f|s) \times p_{\gamma}(r|s) \tag{3}
\end{align*}
\subsection{Inference}
We now consider two tasks for \textit{ODEE-FER}: (1) learning the parameters
%$\alpha, \beta, \gamma, \theta, \lambda$,
and (2) performing inference to obtain the posterior distribution of the latent variables $s$ and $t$, given a news cluster $c$.
%These two tasks are intimately connected because learning parameters often uses inference as a subroutine.
%Instead of collapsed Gibbs sampling,
We adapt the amortized variational inference method of \newcite{srivastava17lda}, using neural inference network to learn the variational parameters.
For simplicity, we concatenate $f$ with $r$ as a new observed feature vector $f'$ in \textit{ODEE-FER} and merge their parameters as $\beta' \in \mathbb{R}^{S \times (2n+2)}$.
Following \newcite{srivastava17lda}, we collapse the discrete latent variable $s$ to obtain an Evidence Lower BOund (ELBO) \cite{kingma2013auto} of the log marginal likelihood:
\begin{align*}
&\textrm{log}\,p_{\alpha,\beta',\theta,\lambda}(c)\\
&=\textrm{log}\,\int_{t}[\prod_{e \in E_c} \, p_{\lambda,\theta}(h|t) \, p_{\beta',\theta}(f'|t)] \, p_{\alpha}(t) \, dt \\
&\geq \textrm{ELBO}_{c}(\alpha,\beta',\theta,\lambda,\omega)\\
%&=\textrm{log}p_{\alpha,\beta',\theta,\lambda}(c) - D_{\textrm{KL}}[ q_{\omega}(t) \| p_{\alpha,\beta',\theta,\lambda}(t_c|c) ] \tag{7}\label{eq12}\\
&=\mathbb{E}_{q_{\omega}(t)}\textrm{log}\,p_{\beta',\theta,\lambda}(c|t) - D_{\textrm{KL}}[ q_{\omega}(t) \| p_{\alpha}(t) ] \tag{4}\label{eq13}
\end{align*}
where $D_{\textrm{KL}}[q_{\omega} \| p_{\alpha}]$ is the KL divergence between the variational posterior $q_{\omega}$ and the prior $p_{\alpha}$.
Due to the difficulty in computing the KL divergence between different categories of distributions and the existence of simple and effective reparameterization tricks for normal distributions, we choose $q_{\omega}(t)$ to be a normal distribution parameterized by $\omega$, which is learned by a neural inference network.
As shown in Figure \ref{fig:inference_network}, our inference network takes the head word histograms $h$ (the times of each head word appears in a news cluster) and contextual features $f'$ as inputs, and computes the mean vector $\mu$ and the variance vector $\sigma^2$ of $q_{\omega}(t)$.
\begin{figure}
\centering
\includegraphics[width=65mm]{imgs/inference_network.pdf}
\caption{The framework of our inference network.}
\label{fig:inference_network}
\end{figure}
Equation \ref{eq13} can be solved by obtaining a Monte Carlo sample and applying reparameterization tricks for the first term, and using the closed-form for the KL divergence term.
We then use the ADAM optimizer \cite{DBLP:journals/corr/KingmaB14} to maximumize the ELBO.
In addition, to alleviate the component collapsing problem \cite{dinh2016training},
% in which the inference network becomes stuck in a bad local optimum where all slots are identical
we follow \newcite{srivastava17lda} and use high moment weight ($>0.8$) and learning rate (in $[0.001, 0.1]$) in the ADAM optimizer, performing batch normalization \cite{DBLP:conf/icml/IoffeS15} and dropout \cite{DBLP:journals/jmlr/SrivastavaHKSS14}.
After learning the model, we make slot assignment for each entity mention by MLE, choosing the slot $s$ that maximizes the likelihood
\begin{align*}
p_{\beta',\theta,\lambda}(s|e,t) \propto& \,p_{\beta',\theta,\lambda}(s,h,f',t) \\
=& \,p_{\theta}(s|t) \times p_{\lambda}(h|s) \times p_{\beta'}(f'|s) \tag{5}\label{eq14}
%&\textrm{log}\,p_{\alpha,\beta',\theta,\lambda}(c)\\
%&=\textrm{log}\,\int_{t}[\prod_{e \in E_c} \, p_{\lambda,\theta}(h|t) \, p_{\beta',\theta}(f'|t)] \, p_{\alpha}(t) \, dt \\
%&\geq \textrm{ELBO}_{c}(\alpha,\beta',\theta,\lambda,\omega)\\
%%&=\textrm{log}p_{\alpha,\beta',\theta,\lambda}(c) - \textrm{D}_{\textrm{KL}}[ q_{\omega}(t) \| p_{\alpha,\beta',\theta,\lambda}(t_c|c) ] \tag{7}\label{eq12}\\
%&=\mathbb{E}_{q_{\omega}(t)}\textrm{log}\,p_{\beta',\theta,\lambda}(c|t) - \textrm{D}_{\textrm{KL}}[ q_{\omega}(t) \| p_{\alpha}(t) ] \tag{5}\label{eq13}
\end{align*}
\subsection{Assembling Events for Output}
To assemble the events in a news cluster $c$ for final output, we need to find the predicate for each entity, which now has a slot value.
We use POS-tags and parse trees produced by the Stanford dependency parser \cite{DBLP:conf/acl/KleinM03} to extract the predicate for the head word of each entity mention.
The following rules are applied:
(1) if the governor of a head word is \textit{VB}, or
(2) if the governor of a head word is \textit{NN} and belongs to the \textit{noun.ACT} or \textit{noun.EVENT} category of WordNet, then it is regarded as a predicate.
%\footnote{\url{http://wordnetcode.princeton.edu/wn3.1.dict.tar.gz}}
We merge the predicates of entity mentions in the same coreference chain as a predicate set.
For each predicate $v$ in these sets, we find the entities whose predicate set contains $v$, treating the entities as arguments of the event triggered by $v$.
Finally, by ranking the numbers of arguments, we obtain top-N open-domain events as the output $\mathcal{E}_c$.
\section{Experiments}
We verify the effectiveness of neural latent variable modeling and redundancy information for ODEE, and conduct case analysis.
All our experiments are conducted on the GNBusiness dataset.
Note that we do not compare our models and existing work on MUC 4 or ACE 2005 due to the fact that these datasets do not consist of news clusters.
%study examples to illustrate our output and analyze the drawbacks.
%We compare our works with state-of-the-art event schema induction methods in the schemas matching experiment and slot coherence experiment.
%By conducting development experiments, we determine the hyper-parameter slot number $S$.
%To examine the learned latent event types vectors, we also visualize them and divide them into clusters with manually estimated themes.
%Then we study examples to illustrate our output and analyze the drawbacks.
%We follow the previous works to use \textit{precision}, \textit{recall} and \textit{F1-score} as conventional metrics for schema matching.
%However, this kind of evaluation needs comparison with some labeled data.
%Our task is extracting events and inducing schemas under the open-domain setting where sometimes there does not exist or are expensive to produce annotated data.
%To further evaluate the intrinsic value of schemas, we also use \textit{slot coherence} which is previously introduced as \textit{topic coherence} \cite{DBLP:conf/wsdm/RoderBH15,DBLP:conf/eacl/LauNB14} to value the intrinsic quality of the induced schemas without the need of external annotated data.
%Then we also visualize the latent event types vectors and analysis some cases.
%REWRITE
\vspace{1mm}
\noindent\textbf{Settings.}
The hyper-parameters in our models and inference network are shown in Table \ref{tab:setting}.
Most of the hyper-parameters directly follow \newcite{srivastava17lda}, while the slot number $S$ is chosen according to development experiments.
\begin{table}
\centering
\small
\begin{tabular}{c|c}
%\toprule[1pt]
\hline
\textbf{Name} & \textbf{Value} \\
\hline
Slots number $S$ & 30 \\
Feature Dimension $n$ & 256 \\
Fully connected layer size & 100 \\
MLP layer number & 1 \\
Activation function & softplus \\
Learning rate & 0.002 \\
Momentum & 0.99 \\
Dropout rate & 0.2 \\
Batch size & 200 \\
\hline
%\bottomrule[1pt]
\end{tabular}
\vspace{-0.1cm}
\caption{\label{tab:setting} Hyper-parameters setting.}
\end{table}
\subsection{Evaluation Metrics}
\noindent\textbf{Schemas Matching.}
We follow previous work and use \textit{precision}, \textit{recall} and \textit{F1-score} as the metrics for schema matching \cite{DBLP:conf/acl/ChambersJ11,DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15,DBLP:conf/naacl/ShaLCS16,DBLP:conf/acl/Ahn17}.
The matching between model answers and references is based on the head word.
Following previous work, we regard as the head word the right-most word of an entity phrase or the right-most word before the first ``of'', ``that'', ``which'' and ``by'' if any.
In addition, we also perform slot mapping, between slots that our model learns and slots in the annotation.
Following previous work on MUC 4 \cite{DBLP:conf/emnlp/Chambers13,DBLP:conf/naacl/CheungPV13,DBLP:conf/acl/NguyenTFB15,DBLP:conf/naacl/ShaLCS16,DBLP:conf/acl/Ahn17}, we implement automatic greedy slot mapping.
Each reference slot is mapped to a learned slot that ranks the best according to the \textit{F1-score} metric on \textit{GNBusiness-Dev}.
%We compare model performances on \textit{GNBusiness-Test}.
\vspace{1mm}
\noindent\textbf{Slot Coherence.}
Several metrics of qualitative topic coherence evaluation have been proposed.
\newcite{DBLP:conf/eacl/LauNB14} showed that normalized point-wise mutual information (NPMI) between all the pairs of words in a set of topics the most closely matches human judgment among all the competing metrics.
We thus adopt it as \textit{slot coherence}\footnote{We use the implementation in \url{https://github.com/jhlau/topic_interpretability}.}.
Formally, the slot coherence $C_{\textrm{NPMI}}(s)$ of a slot $s$ is calculated by using its top-N head words as
\begin{align*}
&C_{\textrm{NPMI}}(s)=\frac{2}{N^2-N}\sum_{i=2}^{N}\sum_{j=1}^{i-1}\textrm{NPMI}(w_i, w_j) \tag{6}\label{eq17} \\
&\textrm{NPMI}(w_i, w_j)=\frac{\textrm{log}\frac{p(w_i, w_j)+\epsilon}{p(w_i)\cdot p(w_j)}}{-\textrm{log}(p(w_i, w_j)+\epsilon)} \tag{7}\label{eq18}
\end{align*}
where $p(w_j)$ and $p(w_i, w_j)$ are estimated based on word co-occurrence counts derived within a sliding window over external reference documents and $\epsilon$ is added to avoid zero logarithm.
Previous work on topic coherence uses Wikipedia and Gigaword as the reference corpus to calculate word frequencies \cite{DBLP:conf/naacl/NewmanLGB10,DBLP:conf/eacl/LauNB14}.
We use \textit{GNBusiness-Full-Text}, in which there are 1.45M sentences and 31M words, which is sufficient for estimating the probabilities.
%Additionally, we find that the top-N words are more sparse than the topic words in topic models because we use nearly all words expect for stop words in a topic model, while in our schema induction model, we use only the head words of entity mentions.
To reduce sparsity, for each news report, we count word co-occurrences in the whole document instead of a sliding window.
In addition, for each slot, we keep the top-5, top-10, top-20, and top-100 head words, averaging the $4 \times S$ coherence results over a test set.
\begin{figure}
\centering
\includegraphics[width=65mm]{imgs/slot_number.eps}
\caption{$F_1$ scores of schemas matching and averaged slot coherences $C_{\textrm{NPMI}}$ of the five models with different numbers of slots $S$.
% The red line is for $F_1$ score, while the blue one for slot coherence. The number of slots achieves best results for both metrics are marked with dashed vertical lines.
}
\label{fig:hyps}
\end{figure}
\subsection{Development Experiments}
We learn the models on \textit{GNBusiness-All} and use \textit{GNBusiness-Dev} to determine the slot number $S$ by grid search in $[10, 50]$ with the step equals to $5$.
Figure \ref{fig:hyps} shows the $F_1$ scores of schemas matching and averaged slot coherences of the five models we introduce in the next subsection with different numbers of slots $S$ ranging from $10$ to $50$.
We can see that for the best $F_1$ score of \textit{ODEE-FER}, the optimal number of slots is 30, while for the best slot coherence, the optimal number of slots is 25.
A value of $S$ larger than 30 or smaller than 25 gives lower results on both $F_1$ score and slot coherence.
%Both on the left part and the right part of the dashed lines, the metrics values are lower than the best results, but the changes are not simply monotonous.
Considering the balance between $F_1$ score and slot coherence, we chose $S=30$ as our final $S$ value for the remaining experiments.
\subsection{Final Results}
\begin{table}
\centering
\small
\begin{tabular}{l|ccc}
%\toprule[1pt]
\hline
\textbf{Method} & \multicolumn{3}{c}{\textbf{Schema Matching (\%)}} \\
& $P$ & $R$ & $F_1$ \\
\hline
\newcite{DBLP:conf/acl/NguyenTFB15} & 41.5 & 53.4 & 46.7 \\
Clustering & 41.2 & 50.6 & 45.4 \\
\hline
ODEE-F & 41.7 & 53.2 & 46.8 \\
ODEE-FE & 42.4 & 56.1 & 48.3 \\
ODEE-FER & \textbf{43.4} & \textbf{58.3} & \textbf{49.8} \\
\hline
%\bottomrule[1pt]
\end{tabular}
\vspace{-0.1cm}
\caption{\label{tab:slot_matching} Overall performance of schema matching.}
\end{table}
\begin{table}
\centering
\small
\begin{tabular}{l|c}
%\toprule[1pt]
\hline
\textbf{Method} & \textbf{Ave Slot Coherence} \\
\hline
\newcite{DBLP:conf/acl/NguyenTFB15} & 0.10 \\
%Clustering & / \\
\hline
ODEE-F & 0.10 \\
ODEE-FE & 0.16 \\
ODEE-FER & \textbf{0.18} \\
\hline
%\bottomrule[1pt]
\end{tabular}
\vspace{-0.1cm}
\caption{\label{tab:slot_coherence} Averaged slot coherence results.}
\end{table}
Table \ref{tab:slot_matching} and Table \ref{tab:slot_coherence} show the final results.
The $p$ values based on the appropriate t-test are provided below in cases where the compared values are close.
We compare our work with \newcite{DBLP:conf/acl/NguyenTFB15}, the state-of-the-art model on MUC 4 representing each entity as a triple containing a head word, a list of attribute relation features and a list of predicate relation features.
Features in the model are discrete and extracted from dependency parse trees.
The model structure is identical to our \textit{ODEE-F} except for the features.
To test the strengths of our external features in isolation, we build another baseline model by taking the continuous features of each entity in \textit{ODEE-F} and runing spectral clustering \cite{DBLP:journals/sac/Luxburg07}.
We call it \textit{Clustering}.
%\footnote{We use the implementation in \url{https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html}.}
\vspace{1mm}
\noindent\textbf{Schemas Matching.}
Table \ref{tab:slot_matching} shows the overall performance of schema matching on \textit{GNBusiness-Test}.
From the table, we can see that \textit{ODEE-FER} achieves the best $F_1$ scores among all the methods.
By comparing \textit{\newcite{DBLP:conf/acl/NguyenTFB15}} and \textit{ODEE-F} ($p=0.01$), we can see that using continuous contextual features gives better performance than discrete features.
This demonstrates the advantages of continuous contextual features for alleviating the sparsity of discrete features in texts.
We can also see from the result of \textit{Clustering} that using only the contextual features is not sufficient for ODEE, while combining with our neural latent variable model in \textit{ODEE-F} can achieve strong results ($p=6 \times 10^{-6}$).
This shows that the neural latent variable model can better explain the observed data.
These results demonstrate the effectivenesses of our method in incorporating with contextual features, latent event types and redundancy information.
Among ODEE models, \textit{ODEE-FE} gives a $2\%$ gain in $F_1$ score against \textit{ODEE-F}, which shows that the latent event type modeling is beneficial and the slot distribution relies on the latent event type.
Additionally, there is a $1\%$ gain in $F_1$ score by comparing \textit{ODEE-FER} and \textit{ODEE-FE} ($p=2 \times 10^{-6}$), which confirms that leveraging redundancy is also beneficial in exploring which slot an entity should be assigned.
\vspace{1mm}
\noindent\textbf{Slot Coherence.}
Table \ref{tab:slot_coherence} shows the comparison of averaged slot coherence results over all the slots in the schemas.
Note that we do not report the slot coherence for the \textit{Clustering} model because it does not output the top-N head words in each slot.
The averaged slot coherence of \textit{ODEE-FER} is the highest, which is consistent with the conclusion from Table \ref{tab:slot_matching}.
The averaged slot coherence of \textit{ODEE-F} is comparable to that of \textit{\newcite{DBLP:conf/acl/NguyenTFB15}} ($p=0.3415$), which again demonstrates that the contextual features are a strong alternative to discrete features.
The scores of \textit{ODEE-FE} ($p=0.06$) and \textit{ODEE-FER} ($p=10^{-5}$) are both higher than that of \textit{ODEE-F}, which proves that the latent event type is critical in ODEE.
\subsection{Latent Event Type Analysis}
\begin{figure}
\centering
\includegraphics[width=75mm]{imgs/event_type.pdf}
\caption{T-SNE visualization results of the latent event type vectors in the test set with colored labels produced by spectral clustering.}
\label{fig:t-SNE}
\end{figure}
%The motivation of our \textit{ODEE-FE} model is to encode different slot distributions for different events by using the latent event type vectors, as there are infinite event types in the open domain.
%In practice, we adapt the AVITM inference method and propose an inference network for learning the mean vector $\mu$ and diagonal variance vector $\sigma$ of the latent event type vectors subjects to a normal distribution.
We are interested in learning how well the latent event type vectors can be modeled.
To this end, for each news cluster in \textit{GNBusiness-Dev}, we use our inference network in Figure \ref{fig:inference_network} to calculate the mean $\mu$ for the latent event type vector $t$.
T-SNE transformation \cite{maaten2008visualizing} of the mean vectors are shown in Figure \ref{fig:t-SNE}.
%\footnote{We use the implementation in \url{https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html}.}
Spectral clustering is further applied, and the number of clusters is chosen by the Calinski-Harabasz Score \cite{calinski1974dendrite} in grid search.
In Figure \ref{fig:t-SNE}, there are four main clusters marked in different colors.
Representative titles of news reports are shown as examples.
We find that the vectors show salient themes for each main cluster.
For example, the red cluster contains news reports about rise and drop of stocks such as \textit{Netflix shares surge}, \textit{IBM drops}, \textit{Intel shares gain}, etc;
the news reports in the purple cluster are mostly about product related activities, such as \textit{Boston Dynamics' reveals its robodog Spot dancing}, \textit{Arby's will debut sous vide duck sandwich}, \textit{Wendy's Offering \$1 Any Size Fry}, etc.
The green cluster and the orange cluster are also interpretable.
The former is about organization reporting changes, while the latter is about service related activities.
%For the sake of saving room, we do not display examples.
%From the demonstration, we can see that our latent event type vectors actually encode theme information of each news cluster and surrounding latent event type vectors have similar themes in each displayed cluster.
\subsection{Case Study}
\begin{figure}
\centering
\includegraphics[width=75mm]{imgs/case_study.pdf}
\caption{Extracted open domain events for \textit{UnitedHealth shares rise}.}
\label{fig:case_study}
\end{figure}
We further use the news cluster \textit{UnitedHealth shares rise} in Figure \ref{fig:t-SNE} for case study.
Figure \ref{fig:case_study} shows the top-3 open-domain events extracted from the news cluster, where four input news reports are shown on the left and three system-generated events are shown on the right with mapped slots.
%where the slot names are mapped from learned slots to the annotation schema in the dev set.
By comparing the plain news reports and the extracted events, we can see that the output events give a reasonable summary for the news cluster with three events triggered by ``raise'', ``report'' and ``predict'', respectively.
Most of the slots are meaningful and closely related to the trigger, while covering most key aspects.
However, this example also contains several incorrect slots.
In the event 1, the slot ``Variation'' and its realization ``28\%'' are only related to the entity ``better-than-expected profits'', but there are three slot realizations in the event, which causes confusion.
In addition, the slot ``Aim'' does not appear in the first event, whose realization should be ``third-quarter profit'' in document 1.
The reason may be that we assemble an event only using entities with the same predicate, which introduces noise.
% and fails to merge entities with other predicates.
Besides, due to the preprocessing errors in resolving coreference chains, some entity mentions are missing from the output.
There are also cases where one slot realization is semantically related to one trigger but eventually appears in a different event.
One example is the entity ``better-than-expected profits'', which is related to the predicate word ``report'' but finally appears in the ``raise'' event.
The cause can be errors propagated from parsing dependency trees, which confuse the syntactic predicate of the head word of an entity.
\section{Conclusion}
We presented the task of open domain event extraction, extracting unconstraint types of events from news clusters.
A novel latent variable neural model was investigated, which explores latent event type vectors and entity mention redundancy.
% achieves better results than the state-of-the-art event schema induction methods in the
In addition, GNBusiness dataset, a large-scale dataset annotated with diverse event types and explainable event schemas, is released along with this paper.
%In addition, redundancy is highly useful for improving open domain event extraction.
%Visualization of learned parameters show that our model can give reasonable latent event types.
To our knowledge, we are the first to use neural latent variable model for inducing event schemas and extracting events.
%We make our code and data public for research community.
%We consider open domain event extraction, the task of extracting unconstraint types of events from news clusters.
%A novel latent variable neural model is constructed, which is scalable to very large corpus.
%A dataset is collected and manually annotated, with task specific evaluation metrics being designed.
%Results show that the proposed model gives better performances compared to the state-of-the-art methods.
%
%scalable model that extracts unconstraint types of events and induces universal schemas from news clusters by continuous contextual features, modeling latent event types and leveraging redundant ratios in the open-domain.
%We adapt the AVITM inference methods and propose a pilot example for combining neural networks and latent variable models.
%By comparing with the state-of-the-art models in our GNBusiness dataset, our model achieve better results whether in traditional schema matching evaluation with annotation from experts or newly introduced slot coherence evaluation with the intrinsic qualities of learned slots and schemas.
%This is the very first work to combine neural networks with latent variable models for open domain event extraction and schema induction and there still exist pretty interesting and worthy-to-explore directions for future works.
%The proposed framework offers extendable configurations for exploring more broader unsupervised feature learning, which is a very exciting direction for attempts.
%Besides, we also need to find a better cut-off strategy for pruning entities in slots to improve the precision of the schema matching.
%The last but not least, as there is still noises when collecting news clusters, figuring out how to reduce the noises is also very important.
\section*{Acknowledgments}
We thank the anonymous reviewers for their valuable comments and suggestions.
%We would thank Zhunchen Luo for their insightful suggestions.
We thank Kiem-Hieu Nguyen from Hanoi University of Science and Technology for providing source code and solving confusions for their work.
We thank Katherine Keith from University of Massachusetts at Amherst for sharing valuable experiences on probabilistic models.
This work is supported by National Natural Science Foundation of China No. 61751201, National Key Research and Development Plan No. 2016QY03D0602, Research Foundation of Beijing Municipal Science and Technology Commission No. Z181100008918002, the funding from Rxhui Inc\footnote{\url{https://rxhui.com}} and China Scholarship Council No. 201806030142.
The work is done when Xiao Liu is visiting Yue Zhang.
\bibliography{acl2019}
\bibliographystyle{acl_natbib}
%\appendix
%\section{Appendices}
%\label{sec:appendix}
%Appendices are material that can be read, and include lemmas, formulas, proofs, and tables that are not critical to the reading and understanding of the paper.
%Appendices should be \textbf{uploaded as supplementary material} when submitting the paper for review. Upon acceptance, the appendices come after the references, as shown here. Use
%\verb|\appendix| before any appendix section to switch the section
%numbering over to letters.
%
%
%\section{Supplemental Material}
%\label{sec:supplemental}
%Submissions may include non-readable supplementary material used in the work and described in the paper. Any accompanying software and/or data should include licenses and documentation of research review as appropriate. Supplementary material may report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper. Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.
%
%Nonetheless, supplementary material should be supplementary (rather
%than central) to the paper. \textbf{Submissions that misuse the supplementary
%material may be rejected without review.}
%Supplementary material may include explanations or details
%of proofs or derivations that do not fit into the paper, lists of
%features or feature templates, sample inputs and outputs for a system,
%pseudo-code or source code, and data. (Source code and data should
%be separate uploads, rather than part of the paper).
%
%The paper should not rely on the supplementary material: while the paper
%may refer to and cite the supplementary material and the supplementary material will be available to the
%reviewers, they will not be asked to review the
%supplementary material.
\end{document}
\begin{thebibliography}{66}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Ahn(2017)}]{DBLP:conf/acl/Ahn17}
Natalie Ahn. 2017.
\newblock \href {https://aclanthology.info/papers/W17-2710/w17-2710} {Inducing
event types and roles in reverse: Using function to discover theme}.
\newblock In \emph{Proceedings of the Events and Stories in the News
Workshop@ACL 2017}, pages 66--76.
\bibitem[{Baker et~al.(1998)Baker, Fillmore, and
Lowe}]{DBLP:conf/acl/BakerFL98}
Collin~F. Baker, Charles~J. Fillmore, and John~B. Lowe. 1998.
\newblock \href {http://aclweb.org/anthology/P/P98/P98-1013.pdf} {The berkeley
framenet project}.
\newblock In \emph{Proceedings of the 36th Annual Meeting of the Association
for Computational Linguistics and 17th International Conference on
Computational Linguistics}, pages 86--90.
\bibitem[{Becker et~al.(2011)Becker, Naaman, and
Gravano}]{DBLP:conf/icwsm/BeckerNG11}
Hila Becker, Mor Naaman, and Luis Gravano. 2011.
\newblock \href
{http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2745} {Beyond
trending topics: Real-world event identification on twitter}.
\newblock In \emph{Proceedings of the 5th International Conference on Weblogs
and Social Media}, pages 438--441.
\bibitem[{Benson et~al.(2011)Benson, Haghighi, and
Barzilay}]{DBLP:conf/acl/BensonHB11}
Edward Benson, Aria Haghighi, and Regina Barzilay. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1040} {Event discovery in
social media feeds}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 389--398.
\bibitem[{Cali{\'n}ski and Harabasz(1974)}]{calinski1974dendrite}
Tadeusz Cali{\'n}ski and Jerzy Harabasz. 1974.
\newblock A dendrite method for cluster analysis.
\newblock \emph{Communications in Statistics-theory and Methods}, 3(1):1--27.
\bibitem[{Chambers(2013)}]{DBLP:conf/emnlp/Chambers13}
Nathanael Chambers. 2013.
\newblock \href {http://aclweb.org/anthology/D/D13/D13-1185.pdf} {Event schema
induction with a probabilistic entity-driven model}.
\newblock In \emph{Proceedings of the 2013 Conference on Empirical Methods in
Natural Language Processing}, pages 1797--1807.
\bibitem[{Chambers and Jurafsky(2011)}]{DBLP:conf/acl/ChambersJ11}
Nathanael Chambers and Dan Jurafsky. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1098} {Template-based
information extraction without the templates}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 976--986.
\bibitem[{Chen et~al.(2015)Chen, Xu, Liu, Zeng, and
Zhao}]{DBLP:conf/acl/ChenXLZ015}
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.
\newblock \href {http://aclweb.org/anthology/P/P15/P15-1017.pdf} {Event
extraction via dynamic multi-pooling convolutional neural networks}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing}, pages 167--176.
\bibitem[{Chen et~al.(2018)Chen, Yang, Liu, Zhao, and
Jia}]{DBLP:conf/emnlp/0001Y00J18}
Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018.
\newblock \href {https://aclanthology.info/papers/D18-1158/d18-1158}
{Collective event detection via a hierarchical and bias tagging networks with
gated multi-level attention mechanisms}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 1267--1276.
\bibitem[{Cheung et~al.(2013)Cheung, Poon, and
Vanderwende}]{DBLP:conf/naacl/CheungPV13}
Jackie Chi~Kit Cheung, Hoifung Poon, and Lucy Vanderwende. 2013.
\newblock \href {http://aclweb.org/anthology/N/N13/N13-1104.pdf} {Probabilistic
frame induction}.
\newblock In \emph{Proceedings of the 2013 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 837--846.
\bibitem[{Cohen(1960)}]{cohen1960coefficient}
Jacob Cohen. 1960.
\newblock \href {https://journals.sagepub.com/doi/10.1177/001316446002000104}
{A coefficient of agreement for nominal scales}.
\newblock \emph{Educational and psychological measurement}, 20(1):37--46.
\bibitem[{Cui et~al.(2012)Cui, Zhang, Liu, Ma, and
Zhang}]{DBLP:conf/cikm/CuiZLMZ12}
Anqi Cui, Min Zhang, Yiqun Liu, Shaoping Ma, and Kuo Zhang. 2012.
\newblock \href {https://doi.org/10.1145/2396761.2398519} {Discover breaking
events with popular hashtags in twitter}.
\newblock In \emph{Proceedings of the 21st {ACM} International Conference on
Information and Knowledge Management}, pages 1794--1798.
\bibitem[{Devlin et~al.(2018)Devlin, Chang, Lee, and
Toutanova}]{DBLP:journals/corr/abs-1810-04805}
Jacob Devlin, Ming{-}Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.
\newblock \href {http://arxiv.org/abs/1810.04805} {{BERT:} pre-training of deep
bidirectional transformers for language understanding}.
\newblock \emph{CoRR}, abs/1810.04805.
\bibitem[{Dinh and Dumoulin(2016)}]{dinh2016training}
Laurent Dinh and Vincent Dumoulin. 2016.
\newblock \href
{http://www.iro.umontreal.ca/~bengioy/cifar/NCAP2014-summerschool/slides/Laurent_dinh_cifar_presentation.pdf}
{Training neural bayesian nets}.
\newblock Technical report.
\bibitem[{Feng et~al.(2016)Feng, Huang, Tang, Ji, Qin, and
Liu}]{DBLP:conf/acl/FengHTJQL16}
Xiaocheng Feng, Lifu Huang, Duyu Tang, Heng Ji, Bing Qin, and Ting Liu. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-2011.pdf} {A
language-independent neural network for event detection}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 66--71.
\bibitem[{Filatova et~al.(2006)Filatova, Hatzivassiloglou, and
McKeown}]{DBLP:conf/acl/FilatovaHM06}
Elena Filatova, Vasileios Hatzivassiloglou, and Kathleen~R. McKeown. 2006.
\newblock \href {http://aclweb.org/anthology/P06-2027} {Automatic creation of
domain templates}.
\newblock In \emph{Proceedings of the 44th Annual Meeting of the Association
for Computational Linguistics and the 21st International Conference on
Computational Linguistics}, pages 207--214.
\bibitem[{Hong et~al.(2011)Hong, Zhang, Ma, Yao, Zhou, and
Zhu}]{DBLP:conf/acl/HongZMYZZ11}
Yu~Hong, Jianfeng Zhang, Bin Ma, Jian{-}Min Yao, Guodong Zhou, and Qiaoming
Zhu. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1113} {Using cross-entity
inference to improve event extraction}.
\newblock In \emph{roceedings of the 49th Annual Meeting of the Association for
Computational Linguistics}, pages 1127--1136.
\bibitem[{Huang et~al.(2016)Huang, Cassidy, Feng, Ji, Voss, Han, and
Sil}]{DBLP:conf/acl/HuangCFJVHS16}
Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare~R. Voss, Jiawei Han,
and Avirup Sil. 2016.
\newblock \href {http://www.aclweb.org/anthology/P16-1025} {Liberal event
extraction and event schema induction}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 258--268.
\bibitem[{Huang and Riloff(2012)}]{DBLP:conf/eacl/HuangR12}
Ruihong Huang and Ellen Riloff. 2012.
\newblock \href {http://aclweb.org/anthology/E/E12/E12-1029.pdf} {Bootstrapped
training of event extraction classifiers}.
\newblock In \emph{Proceedings of the 13th Conference of the European Chapter
of the Association for Computational Linguistics}, pages 286--295.
\bibitem[{Ifrim et~al.(2014)Ifrim, Shi, and Brigadir}]{DBLP:conf/www/IfrimSB14}
Georgiana Ifrim, Bichen Shi, and Igor Brigadir. 2014.
\newblock \href {http://ceur-ws.org/Vol-1150/ifrim.pdf} {Event detection in
twitter using aggressive filtering and hierarchical tweet clustering}.
\newblock In \emph{Proceedings of the {SNOW} 2014 Data Challenge co-located
with 23rd International World Wide Web Conference}, pages 33--40.
\bibitem[{Ioffe and Szegedy(2015)}]{DBLP:conf/icml/IoffeS15}
Sergey Ioffe and Christian Szegedy. 2015.
\newblock \href {http://jmlr.org/proceedings/papers/v37/ioffe15.html} {Batch
normalization: Accelerating deep network training by reducing internal
covariate shift}.
\newblock In \emph{Proceedings of the 32nd International Conference on Machine
Learning}, pages 448--456.
\bibitem[{Ji and Grishman(2008)}]{DBLP:conf/acl/JiG08}
Heng Ji and Ralph Grishman. 2008.
\newblock \href {http://www.aclweb.org/anthology/P08-1030} {Refining event
extraction through cross-document inference}.
\newblock In \emph{Proceedings of the 46th Annual Meeting of the Association
for Computational Linguistics}, pages 254--262.
\bibitem[{Kingma and Ba(2014)}]{DBLP:journals/corr/KingmaB14}
Diederik~P. Kingma and Jimmy Ba. 2014.
\newblock \href {http://arxiv.org/abs/1412.6980} {Adam: {A} method for
stochastic optimization}.
\newblock \emph{CoRR}, abs/1412.6980.
\bibitem[{Kingma and Welling(2014)}]{kingma2013auto}
Diederik~P Kingma and Max Welling. 2014.
\newblock \href {https://arxiv.org/abs/1312.6114} {Auto-encoding variational
bayes}.
\newblock In \emph{Proceedings of the 2014 International Conference on Learning
Representations}.
\bibitem[{Klein and Manning(2003)}]{DBLP:conf/acl/KleinM03}
Dan Klein and Christopher~D. Manning. 2003.
\newblock \href {http://aclweb.org/anthology/P/P03/P03-1054.pdf} {Accurate
unlexicalized parsing}.
\newblock In \emph{Proceedings of the 41st Annual Meeting of the Association
for Computational Linguistics}, pages 423--430.
\bibitem[{Kumaran and Allan(2005)}]{DBLP:conf/naacl/KumaranA05}
Giridhar Kumaran and James Allan. 2005.
\newblock \href {http://aclweb.org/anthology/H/H05/H05-1016.pdf} {Using names
and topics for new event detection}.
\newblock In \emph{Proceedings of the 2005 Conference on Empirical Methods in
Natural Language Processing}, pages 121--128.
\bibitem[{Lau et~al.(2014)Lau, Newman, and Baldwin}]{DBLP:conf/eacl/LauNB14}
Jey~Han Lau, David Newman, and Timothy Baldwin. 2014.
\newblock \href {http://aclweb.org/anthology/E/E14/E14-1056.pdf} {Machine
reading tea leaves: Automatically evaluating topic coherence and topic model
quality}.
\newblock In \emph{Proceedings of the 14th Conference of the European Chapter
of the Association for Computational Linguistics}, pages 530--539.
\bibitem[{Li et~al.(2012)Li, Sun, and Datta}]{DBLP:conf/cikm/LiSD12}
Chenliang Li, Aixin Sun, and Anwitaman Datta. 2012.
\newblock \href {https://doi.org/10.1145/2396761.2396785} {Twevent:
segment-based event detection from tweets}.
\newblock In \emph{Proceedings of the 21st {ACM} International Conference on
Information and Knowledge Management}, pages 155--164.
\bibitem[{Li et~al.(2013)Li, Ji, and Huang}]{DBLP:conf/acl/LiJH13}
Qi~Li, Heng Ji, and Liang Huang. 2013.
\newblock \href {http://aclweb.org/anthology/P/P13/P13-1008.pdf} {Joint event
extraction via structured prediction with global features}.
\newblock In \emph{Proceedings of the 51st Annual Meeting of the Association
for Computational Linguistics}, pages 73--82.
\bibitem[{Liao and Grishman(2010{\natexlab{a}})}]{DBLP:conf/coling/LiaoG10}
Shasha Liao and Ralph Grishman. 2010{\natexlab{a}}.
\newblock \href {http://aclweb.org/anthology/C10-1077} {Filtered ranking for
bootstrapping in event extraction}.
\newblock In \emph{Proceedings of the 23rd International Conference on
Computational Linguistics}, pages 680--688.
\bibitem[{Liao and Grishman(2010{\natexlab{b}})}]{DBLP:conf/acl/LiaoG10}
Shasha Liao and Ralph Grishman. 2010{\natexlab{b}}.
\newblock \href {http://www.aclweb.org/anthology/P10-1081} {Using document
level cross-event inference to improve event extraction}.
\newblock In \emph{Proceedings of the 48th Annual Meeting of the Association
for Computational Linguistics}, pages 789--797.
\bibitem[{Liu et~al.(2018{\natexlab{a}})Liu, Chen, Liu, and
Zhao}]{DBLP:conf/aaai/Liu00018}
Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2018{\natexlab{a}}.
\newblock \href
{https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16371} {Event
detection via gated multilingual attention mechanism}.
\newblock In \emph{Proceedings of the 32nd {AAAI} Conference on Artificial
Intelligence}, pages 4865--4872.
\bibitem[{Liu et~al.(2016)Liu, Chen, He, Liu, and
Zhao}]{DBLP:conf/acl/LiuCHL016}
Shulin Liu, Yubo Chen, Shizhu He, Kang Liu, and Jun Zhao. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-1201.pdf} {Leveraging
framenet to improve automatic event detection}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 2134--2143.
\bibitem[{Liu et~al.(2018{\natexlab{b}})Liu, Luo, and
Huang}]{DBLP:conf/emnlp/LiuLH18}
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018{\natexlab{b}}.
\newblock \href {https://aclanthology.info/papers/D18-1156/d18-1156} {Jointly
multiple events extraction via attention-based graph information
aggregation}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 1247--1256.
\bibitem[{von Luxburg(2007)}]{DBLP:journals/sac/Luxburg07}
Ulrike von Luxburg. 2007.
\newblock \href {https://doi.org/10.1007/s11222-007-9033-z} {A tutorial on
spectral clustering}.
\newblock \emph{Statistics and Computing}, 17(4):395--416.
\bibitem[{Maaten and Hinton(2008)}]{maaten2008visualizing}
Laurens van~der Maaten and Geoffrey Hinton. 2008.
\newblock \href
{http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf}
{Visualizing data using t-sne}.
\newblock \emph{Journal of machine learning research}, 9(Nov):2579--2605.
\bibitem[{McClosky et~al.(2011)McClosky, Surdeanu, and
Manning}]{DBLP:conf/acl/McCloskySM11}
David McClosky, Mihai Surdeanu, and Christopher~D. Manning. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1163} {Event extraction as
dependency parsing}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 1626--1635.
\bibitem[{McMinn and Jose(2015)}]{DBLP:conf/clef/McMinnJ15}
Andrew~James McMinn and Joemon~M. Jose. 2015.
\newblock \href {https://doi.org/10.1007/978-3-319-24027-5\_6} {Real-time
entity-based event detection for twitter}.
\newblock In \emph{Proceedings of the Experimental {IR} Meets Multilinguality,
Multimodality, and Interaction - 6th International Conference of the {CLEF}
Association}, pages 65--77.
\bibitem[{Modi and Titov(2014)}]{DBLP:conf/conll/ModiT14}
Ashutosh Modi and Ivan Titov. 2014.
\newblock \href {http://aclweb.org/anthology/W/W14/W14-1606.pdf} {Inducing
neural models of script knowledge}.
\newblock In \emph{Proceedings of the Eighteenth Conference on Computational
Natural Language Learning}, pages 49--57.
\bibitem[{Moran et~al.(2016)Moran, McCreadie, Macdonald, and
Ounis}]{DBLP:conf/sigir/MoranMMO16}
Sean Moran, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2016.
\newblock \href {https://doi.org/10.1145/2911451.2914719} {Enhancing first
story detection using word embeddings}.
\newblock In \emph{Proceedings of the 39th International {ACM} {SIGIR}
conference on Research and Development in Information Retrieval}, pages
821--824.
\bibitem[{Newman et~al.(2010)Newman, Lau, Grieser, and
Baldwin}]{DBLP:conf/naacl/NewmanLGB10}
David Newman, Jey~Han Lau, Karl Grieser, and Timothy Baldwin. 2010.
\newblock \href {http://www.aclweb.org/anthology/N10-1012} {Automatic
evaluation of topic coherence}.
\newblock In \emph{Proceedings of the 2010 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 100--108.
\bibitem[{Nguyen et~al.(2015)Nguyen, Tannier, Ferret, and
Besan{\c{c}}on}]{DBLP:conf/acl/NguyenTFB15}
Kiem{-}Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besan{\c{c}}on.
2015.
\newblock \href {http://aclweb.org/anthology/P/P15/P15-1019.pdf} {Generative
event schema induction with entity disambiguation}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing}, pages 188--197.
\bibitem[{Nguyen et~al.(2016{\natexlab{a}})Nguyen, Tannier, Ferret, and
Besan{\c{c}}on}]{DBLP:conf/lrec/NguyenTFB16}
Kiem{-}Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besan{\c{c}}on.
2016{\natexlab{a}}.
\newblock \href
{http://www.lrec-conf.org/proceedings/lrec2016/summaries/837.html} {A dataset
for open event extraction in english}.
\newblock In \emph{Proceedings of the 10th International Conference on Language
Resources and Evaluation}, pages 1939--1943.
\bibitem[{Nguyen et~al.(2016{\natexlab{b}})Nguyen, Cho, and
Grishman}]{DBLP:conf/naacl/NguyenCG16}
Thien~Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016{\natexlab{b}}.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1034.pdf} {Joint event
extraction via recurrent neural networks}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 300--309.
\bibitem[{Nguyen and Grishman(2016)}]{DBLP:conf/emnlp/NguyenG16}
Thien~Huu Nguyen and Ralph Grishman. 2016.
\newblock \href {http://aclweb.org/anthology/D/D16/D16-1085.pdf} {Modeling
skip-grams for event detection with convolutional neural networks}.
\newblock In \emph{Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing}, pages 886--891.
\bibitem[{Panagiotou et~al.(2016)Panagiotou, Akkaya, Tsioutsiouliklis,
Kalogeraki, and Gunopulos}]{DBLP:conf/coling/PanagiotouATKG16}
Nikolaos Panagiotou, Cem Akkaya, Kostas Tsioutsiouliklis, Vana Kalogeraki, and
Dimitrios Gunopulos. 2016.
\newblock \href {http://aclweb.org/anthology/C/C16/C16-1305.pdf} {First story
detection using entities and relations}.
\newblock In \emph{Proceedings of the 26th International Conference on
Computational Linguistics}, pages 3237--3244.
\bibitem[{Peters et~al.(2018)Peters, Neumann, Iyyer, Gardner, Clark, Lee, and
Zettlemoyer}]{DBLP:conf/naacl/PetersNIGCLZ18}
Matthew~E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark,
Kenton Lee, and Luke Zettlemoyer. 2018.
\newblock \href {https://aclanthology.info/papers/N18-1202/n18-1202} {Deep
contextualized word representations}.
\newblock In \emph{Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 2227--2237.
\bibitem[{Pichotta and Mooney(2016)}]{DBLP:conf/acl/PichottaM16}
Karl Pichotta and Raymond~J. Mooney. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-1027.pdf} {Using
sentence-level {LSTM} language models for script inference}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}.
\bibitem[{Qin et~al.(2013)Qin, Zhang, Zhang, and
Zheng}]{DBLP:conf/ijcnlp/QinZZZ13}
Yanxia Qin, Yue Zhang, Min Zhang, and Dequan Zheng. 2013.
\newblock \href {http://aclweb.org/anthology/I/I13/I13-1035.pdf} {Feature-rich
segment-based news event detection on twitter}.
\newblock In \emph{Proceedings of the 6th International Joint Conference on
Natural Language Processing}, pages 302--310.
\bibitem[{Qin et~al.(2017)Qin, Zhang, Zhang, and
Zheng}]{DBLP:conf/ialp/QinZZZ17}
Yanxia Qin, Yue Zhang, Min Zhang, and Dequan Zheng. 2017.
\newblock \href {https://doi.org/10.1109/IALP.2017.8300594} {Semantic-frame
representation for event detection on twitter}.
\newblock In \emph{Proceedings of the 2017 International Conference on Asian
Language Processing}, pages 264--267.
\bibitem[{Qiu et~al.(2008)Qiu, Kan, and Chua}]{DBLP:conf/ijcnlp/QiuKC08}
Long Qiu, Min{-}Yen Kan, and Tat{-}Seng Chua. 2008.
\newblock \href {http://aclweb.org/anthology/I/I08/I08-1021.pdf} {Modeling
context in scenario template creation}.
\newblock In \emph{Proceedings of the 3rd International Joint Conference on
Natural Language Processing}, pages 157--164.
\bibitem[{Radford et~al.(2018)Radford, Narasimhan, Salimans, and
Sutskever}]{radford2018improving}
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.
\newblock \href
{https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf}
{Improving language understanding by generative pre-training}.
\bibitem[{Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, and
Sutskever}]{radford2019language}
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. 2019.
\newblock \href
{https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf}
{Language models are unsupervised multitask learners}.
\bibitem[{Reichart and Barzilay(2012)}]{DBLP:conf/naacl/ReichartB12}
Roi Reichart and Regina Barzilay. 2012.
\newblock \href {http://www.aclweb.org/anthology/N12-1008} {Multi-event
extraction guided by global constraints}.
\newblock In \emph{Proceedings of the 2012 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 70--79.
\bibitem[{Ritter et~al.(2012)Ritter, Mausam, Etzioni, and
Clark}]{DBLP:conf/kdd/RitterMEC12}
Alan Ritter, Mausam, Oren Etzioni, and Sam Clark. 2012.
\newblock \href {https://doi.org/10.1145/2339530.2339704} {Open domain event
extraction from twitter}.
\newblock In \emph{Proceedings of the 18th {ACM} {SIGKDD} International
Conference on Knowledge Discovery and Data Mining}, pages 1104--1112.
\bibitem[{Rudinger et~al.(2015)Rudinger, Rastogi, Ferraro, and
Durme}]{DBLP:conf/emnlp/RudingerRFD15}
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin~Van Durme.
2015.
\newblock \href {http://aclweb.org/anthology/D/D15/D15-1195.pdf} {Script
induction as language modeling}.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 1681--1686.
\bibitem[{Sakaki et~al.(2010)Sakaki, Okazaki, and
Matsuo}]{DBLP:conf/www/SakakiOM10}
Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010.
\newblock \href {https://doi.org/10.1145/1772690.1772777} {Earthquake shakes
twitter users: real-time event detection by social sensors}.
\newblock In \emph{Proceedings of the 19th International Conference on World
Wide Web}, pages 851--860.
\bibitem[{Sha et~al.(2016)Sha, Li, Chang, and Sui}]{DBLP:conf/naacl/ShaLCS16}
Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1049.pdf} {Joint
learning templates and slots for event schema induction}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 428--434.
\bibitem[{Sha et~al.(2018)Sha, Qian, Chang, and Sui}]{DBLP:conf/aaai/ShaQCS18}
Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018.
\newblock \href
{https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16222} {Jointly
extracting event triggers and arguments by dependency-bridge {RNN} and
tensor-based argument interaction}.
\newblock In \emph{Proceedings of the 32nd {AAAI} Conference on Artificial
Intelligence}, pages 5916--5923.
\bibitem[{Shinyama and Sekine(2006)}]{DBLP:conf/naacl/ShinyamaS06}
Yusuke Shinyama and Satoshi Sekine. 2006.
\newblock \href {http://aclweb.org/anthology/N/N06/N06-1039.pdf} {Preemptive
information extraction using unrestricted relation discovery}.
\newblock In \emph{Proceedings of the 2006 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 304--311.
\bibitem[{Srivastava and Sutton(2017)}]{srivastava17lda}
Akash Srivastava and Charles Sutton. 2017.
\newblock \href {https://arxiv.org/abs/1703.01488} {Autoencoding variational
inference for topic models}.
\newblock In \emph{Proceedings of the 2017 International Conference on Learning
Representations}.
\bibitem[{Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and
Salakhutdinov}]{DBLP:journals/jmlr/SrivastavaHKSS14}
Nitish Srivastava, Geoffrey~E. Hinton, Alex Krizhevsky, Ilya Sutskever, and
Ruslan Salakhutdinov. 2014.
\newblock \href {http://dl.acm.org/citation.cfm?id=2670313} {Dropout: a simple
way to prevent neural networks from overfitting}.
\newblock \emph{Journal of Machine Learning Research}, 15(1):1929--1958.
\bibitem[{Sundheim(1992)}]{DBLP:conf/muc/Sundheim92}
Beth Sundheim. 1992.
\newblock \href {https://doi.org/10.1145/1072064.1072066} {Overview of the
fourth message understanding evaluation and conference}.
\newblock In \emph{Proceedings of the 4th Conference on Message Understanding},
pages 3--21.
\bibitem[{Vuurens and de~Vries(2016)}]{DBLP:conf/sigir/VuurensV16}
Jeroen B.~P. Vuurens and Arjen~P. de~Vries. 2016.
\newblock \href {https://doi.org/10.1145/2911451.2914761} {First story
detection using multiple nearest neighbors}.
\newblock In \emph{Proceedings of the 39th International {ACM} {SIGIR}
conference on Research and Development in Information Retrieval}, pages
845--848.
\bibitem[{Yang and Mitchell(2016)}]{DBLP:conf/naacl/YangM16}
Bishan Yang and Tom~M. Mitchell. 2016.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1033.pdf} {Joint
extraction of events and entities within a document context}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 289--299.
\bibitem[{Yuan et~al.(2018)Yuan, Ren, He, Zhang, Geng, Huang, Ji, Lin, and
Han}]{DBLP:conf/cikm/0001RHZGHJL018}
Quan Yuan, Xiang Ren, Wenqi He, Chao Zhang, Xinhe Geng, Lifu Huang, Heng Ji,
Chin{-}Yew Lin, and Jiawei Han. 2018.
\newblock \href {https://doi.org/10.1145/3269206.3271674} {Open-schema event
profiling for massive news corpora}.
\newblock In \emph{Proceedings of the 27th {ACM} International Conference on
Information and Knowledge Management}, pages 587--596.
\end{thebibliography}
\begin{thebibliography}{66}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Ahn(2017)}]{DBLP:conf/acl/Ahn17}
Natalie Ahn. 2017.
\newblock \href {https://aclanthology.info/papers/W17-2710/w17-2710} {Inducing
event types and roles in reverse: Using function to discover theme}.
\newblock In \emph{Proceedings of the Events and Stories in the News
Workshop@ACL 2017}, pages 66--76.
\bibitem[{Baker et~al.(1998)Baker, Fillmore, and
Lowe}]{DBLP:conf/acl/BakerFL98}
Collin~F. Baker, Charles~J. Fillmore, and John~B. Lowe. 1998.
\newblock \href {http://aclweb.org/anthology/P/P98/P98-1013.pdf} {The berkeley
framenet project}.
\newblock In \emph{Proceedings of the 36th Annual Meeting of the Association
for Computational Linguistics and 17th International Conference on
Computational Linguistics}, pages 86--90.
\bibitem[{Becker et~al.(2011)Becker, Naaman, and
Gravano}]{DBLP:conf/icwsm/BeckerNG11}
Hila Becker, Mor Naaman, and Luis Gravano. 2011.
\newblock \href
{http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2745} {Beyond
trending topics: Real-world event identification on twitter}.
\newblock In \emph{Proceedings of the 5th International Conference on Weblogs
and Social Media}, pages 438--441.
\bibitem[{Benson et~al.(2011)Benson, Haghighi, and
Barzilay}]{DBLP:conf/acl/BensonHB11}
Edward Benson, Aria Haghighi, and Regina Barzilay. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1040} {Event discovery in
social media feeds}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 389--398.
\bibitem[{Cali{\'n}ski and Harabasz(1974)}]{calinski1974dendrite}
Tadeusz Cali{\'n}ski and Jerzy Harabasz. 1974.
\newblock A dendrite method for cluster analysis.
\newblock \emph{Communications in Statistics-theory and Methods}, 3(1):1--27.
\bibitem[{Chambers(2013)}]{DBLP:conf/emnlp/Chambers13}
Nathanael Chambers. 2013.
\newblock \href {http://aclweb.org/anthology/D/D13/D13-1185.pdf} {Event schema
induction with a probabilistic entity-driven model}.
\newblock In \emph{Proceedings of the 2013 Conference on Empirical Methods in
Natural Language Processing}, pages 1797--1807.
\bibitem[{Chambers and Jurafsky(2011)}]{DBLP:conf/acl/ChambersJ11}
Nathanael Chambers and Dan Jurafsky. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1098} {Template-based
information extraction without the templates}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 976--986.
\bibitem[{Chen et~al.(2015)Chen, Xu, Liu, Zeng, and
Zhao}]{DBLP:conf/acl/ChenXLZ015}
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015.
\newblock \href {http://aclweb.org/anthology/P/P15/P15-1017.pdf} {Event
extraction via dynamic multi-pooling convolutional neural networks}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing}, pages 167--176.
\bibitem[{Chen et~al.(2018)Chen, Yang, Liu, Zhao, and
Jia}]{DBLP:conf/emnlp/0001Y00J18}
Yubo Chen, Hang Yang, Kang Liu, Jun Zhao, and Yantao Jia. 2018.
\newblock \href {https://aclanthology.info/papers/D18-1158/d18-1158}
{Collective event detection via a hierarchical and bias tagging networks with
gated multi-level attention mechanisms}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 1267--1276.
\bibitem[{Cheung et~al.(2013)Cheung, Poon, and
Vanderwende}]{DBLP:conf/naacl/CheungPV13}
Jackie Chi~Kit Cheung, Hoifung Poon, and Lucy Vanderwende. 2013.
\newblock \href {http://aclweb.org/anthology/N/N13/N13-1104.pdf} {Probabilistic
frame induction}.
\newblock In \emph{Proceedings of the 2013 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 837--846.
\bibitem[{Cohen(1960)}]{cohen1960coefficient}
Jacob Cohen. 1960.
\newblock \href {https://journals.sagepub.com/doi/10.1177/001316446002000104}
{A coefficient of agreement for nominal scales}.
\newblock \emph{Educational and psychological measurement}, 20(1):37--46.
\bibitem[{Cui et~al.(2012)Cui, Zhang, Liu, Ma, and
Zhang}]{DBLP:conf/cikm/CuiZLMZ12}
Anqi Cui, Min Zhang, Yiqun Liu, Shaoping Ma, and Kuo Zhang. 2012.
\newblock \href {https://doi.org/10.1145/2396761.2398519} {Discover breaking
events with popular hashtags in twitter}.
\newblock In \emph{Proceedings of the 21st {ACM} International Conference on
Information and Knowledge Management}, pages 1794--1798.
\bibitem[{Devlin et~al.(2018)Devlin, Chang, Lee, and
Toutanova}]{DBLP:journals/corr/abs-1810-04805}
Jacob Devlin, Ming{-}Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.
\newblock \href {http://arxiv.org/abs/1810.04805} {{BERT:} pre-training of deep
bidirectional transformers for language understanding}.
\newblock \emph{CoRR}, abs/1810.04805.
\bibitem[{Dinh and Dumoulin(2016)}]{dinh2016training}
Laurent Dinh and Vincent Dumoulin. 2016.
\newblock \href
{http://www.iro.umontreal.ca/~bengioy/cifar/NCAP2014-summerschool/slides/Laurent_dinh_cifar_presentation.pdf}
{Training neural bayesian nets}.
\newblock Technical report.
\bibitem[{Feng et~al.(2016)Feng, Huang, Tang, Ji, Qin, and
Liu}]{DBLP:conf/acl/FengHTJQL16}
Xiaocheng Feng, Lifu Huang, Duyu Tang, Heng Ji, Bing Qin, and Ting Liu. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-2011.pdf} {A
language-independent neural network for event detection}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 66--71.
\bibitem[{Filatova et~al.(2006)Filatova, Hatzivassiloglou, and
McKeown}]{DBLP:conf/acl/FilatovaHM06}
Elena Filatova, Vasileios Hatzivassiloglou, and Kathleen~R. McKeown. 2006.
\newblock \href {http://aclweb.org/anthology/P06-2027} {Automatic creation of
domain templates}.
\newblock In \emph{Proceedings of the 44th Annual Meeting of the Association
for Computational Linguistics and the 21st International Conference on
Computational Linguistics}, pages 207--214.
\bibitem[{Hong et~al.(2011)Hong, Zhang, Ma, Yao, Zhou, and
Zhu}]{DBLP:conf/acl/HongZMYZZ11}
Yu~Hong, Jianfeng Zhang, Bin Ma, Jian{-}Min Yao, Guodong Zhou, and Qiaoming
Zhu. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1113} {Using cross-entity
inference to improve event extraction}.
\newblock In \emph{roceedings of the 49th Annual Meeting of the Association for
Computational Linguistics}, pages 1127--1136.
\bibitem[{Huang et~al.(2016)Huang, Cassidy, Feng, Ji, Voss, Han, and
Sil}]{DBLP:conf/acl/HuangCFJVHS16}
Lifu Huang, Taylor Cassidy, Xiaocheng Feng, Heng Ji, Clare~R. Voss, Jiawei Han,
and Avirup Sil. 2016.
\newblock \href {http://www.aclweb.org/anthology/P16-1025} {Liberal event
extraction and event schema induction}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 258--268.
\bibitem[{Huang and Riloff(2012)}]{DBLP:conf/eacl/HuangR12}
Ruihong Huang and Ellen Riloff. 2012.
\newblock \href {http://aclweb.org/anthology/E/E12/E12-1029.pdf} {Bootstrapped
training of event extraction classifiers}.
\newblock In \emph{Proceedings of the 13th Conference of the European Chapter
of the Association for Computational Linguistics}, pages 286--295.
\bibitem[{Ifrim et~al.(2014)Ifrim, Shi, and Brigadir}]{DBLP:conf/www/IfrimSB14}
Georgiana Ifrim, Bichen Shi, and Igor Brigadir. 2014.
\newblock \href {http://ceur-ws.org/Vol-1150/ifrim.pdf} {Event detection in
twitter using aggressive filtering and hierarchical tweet clustering}.
\newblock In \emph{Proceedings of the {SNOW} 2014 Data Challenge co-located
with 23rd International World Wide Web Conference}, pages 33--40.
\bibitem[{Ioffe and Szegedy(2015)}]{DBLP:conf/icml/IoffeS15}
Sergey Ioffe and Christian Szegedy. 2015.
\newblock \href {http://jmlr.org/proceedings/papers/v37/ioffe15.html} {Batch
normalization: Accelerating deep network training by reducing internal
covariate shift}.
\newblock In \emph{Proceedings of the 32nd International Conference on Machine
Learning}, pages 448--456.
\bibitem[{Ji and Grishman(2008)}]{DBLP:conf/acl/JiG08}
Heng Ji and Ralph Grishman. 2008.
\newblock \href {http://www.aclweb.org/anthology/P08-1030} {Refining event
extraction through cross-document inference}.
\newblock In \emph{Proceedings of the 46th Annual Meeting of the Association
for Computational Linguistics}, pages 254--262.
\bibitem[{Kingma and Ba(2014)}]{DBLP:journals/corr/KingmaB14}
Diederik~P. Kingma and Jimmy Ba. 2014.
\newblock \href {http://arxiv.org/abs/1412.6980} {Adam: {A} method for
stochastic optimization}.
\newblock \emph{CoRR}, abs/1412.6980.
\bibitem[{Kingma and Welling(2014)}]{kingma2013auto}
Diederik~P Kingma and Max Welling. 2014.
\newblock \href {https://arxiv.org/abs/1312.6114} {Auto-encoding variational
bayes}.
\newblock In \emph{Proceedings of the 2014 International Conference on Learning
Representations}.
\bibitem[{Klein and Manning(2003)}]{DBLP:conf/acl/KleinM03}
Dan Klein and Christopher~D. Manning. 2003.
\newblock \href {http://aclweb.org/anthology/P/P03/P03-1054.pdf} {Accurate
unlexicalized parsing}.
\newblock In \emph{Proceedings of the 41st Annual Meeting of the Association
for Computational Linguistics}, pages 423--430.
\bibitem[{Kumaran and Allan(2005)}]{DBLP:conf/naacl/KumaranA05}
Giridhar Kumaran and James Allan. 2005.
\newblock \href {http://aclweb.org/anthology/H/H05/H05-1016.pdf} {Using names
and topics for new event detection}.
\newblock In \emph{Proceedings of the 2005 Conference on Empirical Methods in
Natural Language Processing}, pages 121--128.
\bibitem[{Lau et~al.(2014)Lau, Newman, and Baldwin}]{DBLP:conf/eacl/LauNB14}
Jey~Han Lau, David Newman, and Timothy Baldwin. 2014.
\newblock \href {http://aclweb.org/anthology/E/E14/E14-1056.pdf} {Machine
reading tea leaves: Automatically evaluating topic coherence and topic model
quality}.
\newblock In \emph{Proceedings of the 14th Conference of the European Chapter
of the Association for Computational Linguistics}, pages 530--539.
\bibitem[{Li et~al.(2012)Li, Sun, and Datta}]{DBLP:conf/cikm/LiSD12}
Chenliang Li, Aixin Sun, and Anwitaman Datta. 2012.
\newblock \href {https://doi.org/10.1145/2396761.2396785} {Twevent:
segment-based event detection from tweets}.
\newblock In \emph{Proceedings of the 21st {ACM} International Conference on
Information and Knowledge Management}, pages 155--164.
\bibitem[{Li et~al.(2013)Li, Ji, and Huang}]{DBLP:conf/acl/LiJH13}
Qi~Li, Heng Ji, and Liang Huang. 2013.
\newblock \href {http://aclweb.org/anthology/P/P13/P13-1008.pdf} {Joint event
extraction via structured prediction with global features}.
\newblock In \emph{Proceedings of the 51st Annual Meeting of the Association
for Computational Linguistics}, pages 73--82.
\bibitem[{Liao and Grishman(2010{\natexlab{a}})}]{DBLP:conf/coling/LiaoG10}
Shasha Liao and Ralph Grishman. 2010{\natexlab{a}}.
\newblock \href {http://aclweb.org/anthology/C10-1077} {Filtered ranking for
bootstrapping in event extraction}.
\newblock In \emph{Proceedings of the 23rd International Conference on
Computational Linguistics}, pages 680--688.
\bibitem[{Liao and Grishman(2010{\natexlab{b}})}]{DBLP:conf/acl/LiaoG10}
Shasha Liao and Ralph Grishman. 2010{\natexlab{b}}.
\newblock \href {http://www.aclweb.org/anthology/P10-1081} {Using document
level cross-event inference to improve event extraction}.
\newblock In \emph{Proceedings of the 48th Annual Meeting of the Association
for Computational Linguistics}, pages 789--797.
\bibitem[{Liu et~al.(2018{\natexlab{a}})Liu, Chen, Liu, and
Zhao}]{DBLP:conf/aaai/Liu00018}
Jian Liu, Yubo Chen, Kang Liu, and Jun Zhao. 2018{\natexlab{a}}.
\newblock \href
{https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16371} {Event
detection via gated multilingual attention mechanism}.
\newblock In \emph{Proceedings of the 32nd {AAAI} Conference on Artificial
Intelligence}, pages 4865--4872.
\bibitem[{Liu et~al.(2016)Liu, Chen, He, Liu, and
Zhao}]{DBLP:conf/acl/LiuCHL016}
Shulin Liu, Yubo Chen, Shizhu He, Kang Liu, and Jun Zhao. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-1201.pdf} {Leveraging
framenet to improve automatic event detection}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 2134--2143.
\bibitem[{Liu et~al.(2018{\natexlab{b}})Liu, Luo, and
Huang}]{DBLP:conf/emnlp/LiuLH18}
Xiao Liu, Zhunchen Luo, and Heyan Huang. 2018{\natexlab{b}}.
\newblock \href {https://aclanthology.info/papers/D18-1156/d18-1156} {Jointly
multiple events extraction via attention-based graph information
aggregation}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 1247--1256.
\bibitem[{von Luxburg(2007)}]{DBLP:journals/sac/Luxburg07}
Ulrike von Luxburg. 2007.
\newblock \href {https://doi.org/10.1007/s11222-007-9033-z} {A tutorial on
spectral clustering}.
\newblock \emph{Statistics and Computing}, 17(4):395--416.
\bibitem[{Maaten and Hinton(2008)}]{maaten2008visualizing}
Laurens van~der Maaten and Geoffrey Hinton. 2008.
\newblock \href
{http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf}
{Visualizing data using t-sne}.
\newblock \emph{Journal of machine learning research}, 9(Nov):2579--2605.
\bibitem[{McClosky et~al.(2011)McClosky, Surdeanu, and
Manning}]{DBLP:conf/acl/McCloskySM11}
David McClosky, Mihai Surdeanu, and Christopher~D. Manning. 2011.
\newblock \href {http://www.aclweb.org/anthology/P11-1163} {Event extraction as
dependency parsing}.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics}, pages 1626--1635.
\bibitem[{McMinn and Jose(2015)}]{DBLP:conf/clef/McMinnJ15}
Andrew~James McMinn and Joemon~M. Jose. 2015.
\newblock \href {https://doi.org/10.1007/978-3-319-24027-5\_6} {Real-time
entity-based event detection for twitter}.
\newblock In \emph{Proceedings of the Experimental {IR} Meets Multilinguality,
Multimodality, and Interaction - 6th International Conference of the {CLEF}
Association}, pages 65--77.
\bibitem[{Modi and Titov(2014)}]{DBLP:conf/conll/ModiT14}
Ashutosh Modi and Ivan Titov. 2014.
\newblock \href {http://aclweb.org/anthology/W/W14/W14-1606.pdf} {Inducing
neural models of script knowledge}.
\newblock In \emph{Proceedings of the Eighteenth Conference on Computational
Natural Language Learning}, pages 49--57.
\bibitem[{Moran et~al.(2016)Moran, McCreadie, Macdonald, and
Ounis}]{DBLP:conf/sigir/MoranMMO16}
Sean Moran, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2016.
\newblock \href {https://doi.org/10.1145/2911451.2914719} {Enhancing first
story detection using word embeddings}.
\newblock In \emph{Proceedings of the 39th International {ACM} {SIGIR}
conference on Research and Development in Information Retrieval}, pages
821--824.
\bibitem[{Newman et~al.(2010)Newman, Lau, Grieser, and
Baldwin}]{DBLP:conf/naacl/NewmanLGB10}
David Newman, Jey~Han Lau, Karl Grieser, and Timothy Baldwin. 2010.
\newblock \href {http://www.aclweb.org/anthology/N10-1012} {Automatic
evaluation of topic coherence}.
\newblock In \emph{Proceedings of the 2010 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 100--108.
\bibitem[{Nguyen et~al.(2015)Nguyen, Tannier, Ferret, and
Besan{\c{c}}on}]{DBLP:conf/acl/NguyenTFB15}
Kiem{-}Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besan{\c{c}}on.
2015.
\newblock \href {http://aclweb.org/anthology/P/P15/P15-1019.pdf} {Generative
event schema induction with entity disambiguation}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing of the Asian Federation of Natural Language
Processing}, pages 188--197.
\bibitem[{Nguyen et~al.(2016{\natexlab{a}})Nguyen, Tannier, Ferret, and
Besan{\c{c}}on}]{DBLP:conf/lrec/NguyenTFB16}
Kiem{-}Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besan{\c{c}}on.
2016{\natexlab{a}}.
\newblock \href
{http://www.lrec-conf.org/proceedings/lrec2016/summaries/837.html} {A dataset
for open event extraction in english}.
\newblock In \emph{Proceedings of the 10th International Conference on Language
Resources and Evaluation}, pages 1939--1943.
\bibitem[{Nguyen et~al.(2016{\natexlab{b}})Nguyen, Cho, and
Grishman}]{DBLP:conf/naacl/NguyenCG16}
Thien~Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016{\natexlab{b}}.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1034.pdf} {Joint event
extraction via recurrent neural networks}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 300--309.
\bibitem[{Nguyen and Grishman(2016)}]{DBLP:conf/emnlp/NguyenG16}
Thien~Huu Nguyen and Ralph Grishman. 2016.
\newblock \href {http://aclweb.org/anthology/D/D16/D16-1085.pdf} {Modeling
skip-grams for event detection with convolutional neural networks}.
\newblock In \emph{Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing}, pages 886--891.
\bibitem[{Panagiotou et~al.(2016)Panagiotou, Akkaya, Tsioutsiouliklis,
Kalogeraki, and Gunopulos}]{DBLP:conf/coling/PanagiotouATKG16}
Nikolaos Panagiotou, Cem Akkaya, Kostas Tsioutsiouliklis, Vana Kalogeraki, and
Dimitrios Gunopulos. 2016.
\newblock \href {http://aclweb.org/anthology/C/C16/C16-1305.pdf} {First story
detection using entities and relations}.
\newblock In \emph{Proceedings of the 26th International Conference on
Computational Linguistics}, pages 3237--3244.
\bibitem[{Peters et~al.(2018)Peters, Neumann, Iyyer, Gardner, Clark, Lee, and
Zettlemoyer}]{DBLP:conf/naacl/PetersNIGCLZ18}
Matthew~E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark,
Kenton Lee, and Luke Zettlemoyer. 2018.
\newblock \href {https://aclanthology.info/papers/N18-1202/n18-1202} {Deep
contextualized word representations}.
\newblock In \emph{Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 2227--2237.
\bibitem[{Pichotta and Mooney(2016)}]{DBLP:conf/acl/PichottaM16}
Karl Pichotta and Raymond~J. Mooney. 2016.
\newblock \href {http://aclweb.org/anthology/P/P16/P16-1027.pdf} {Using
sentence-level {LSTM} language models for script inference}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}.
\bibitem[{Qin et~al.(2013)Qin, Zhang, Zhang, and
Zheng}]{DBLP:conf/ijcnlp/QinZZZ13}
Yanxia Qin, Yue Zhang, Min Zhang, and Dequan Zheng. 2013.
\newblock \href {http://aclweb.org/anthology/I/I13/I13-1035.pdf} {Feature-rich
segment-based news event detection on twitter}.
\newblock In \emph{Proceedings of the 6th International Joint Conference on
Natural Language Processing}, pages 302--310.
\bibitem[{Qin et~al.(2017)Qin, Zhang, Zhang, and
Zheng}]{DBLP:conf/ialp/QinZZZ17}
Yanxia Qin, Yue Zhang, Min Zhang, and Dequan Zheng. 2017.
\newblock \href {https://doi.org/10.1109/IALP.2017.8300594} {Semantic-frame
representation for event detection on twitter}.
\newblock In \emph{Proceedings of the 2017 International Conference on Asian
Language Processing}, pages 264--267.
\bibitem[{Qiu et~al.(2008)Qiu, Kan, and Chua}]{DBLP:conf/ijcnlp/QiuKC08}
Long Qiu, Min{-}Yen Kan, and Tat{-}Seng Chua. 2008.
\newblock \href {http://aclweb.org/anthology/I/I08/I08-1021.pdf} {Modeling
context in scenario template creation}.
\newblock In \emph{Proceedings of the 3rd International Joint Conference on
Natural Language Processing}, pages 157--164.
\bibitem[{Radford et~al.(2018)Radford, Narasimhan, Salimans, and
Sutskever}]{radford2018improving}
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.
\newblock \href
{https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf}
{Improving language understanding by generative pre-training}.
\bibitem[{Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, and
Sutskever}]{radford2019language}
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. 2019.
\newblock \href
{https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf}
{Language models are unsupervised multitask learners}.
\bibitem[{Reichart and Barzilay(2012)}]{DBLP:conf/naacl/ReichartB12}
Roi Reichart and Regina Barzilay. 2012.
\newblock \href {http://www.aclweb.org/anthology/N12-1008} {Multi-event
extraction guided by global constraints}.
\newblock In \emph{Proceedings of the 2012 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 70--79.
\bibitem[{Ritter et~al.(2012)Ritter, Mausam, Etzioni, and
Clark}]{DBLP:conf/kdd/RitterMEC12}
Alan Ritter, Mausam, Oren Etzioni, and Sam Clark. 2012.
\newblock \href {https://doi.org/10.1145/2339530.2339704} {Open domain event
extraction from twitter}.
\newblock In \emph{Proceedings of the 18th {ACM} {SIGKDD} International
Conference on Knowledge Discovery and Data Mining}, pages 1104--1112.
\bibitem[{Rudinger et~al.(2015)Rudinger, Rastogi, Ferraro, and
Durme}]{DBLP:conf/emnlp/RudingerRFD15}
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin~Van Durme.
2015.
\newblock \href {http://aclweb.org/anthology/D/D15/D15-1195.pdf} {Script
induction as language modeling}.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 1681--1686.
\bibitem[{Sakaki et~al.(2010)Sakaki, Okazaki, and
Matsuo}]{DBLP:conf/www/SakakiOM10}
Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010.
\newblock \href {https://doi.org/10.1145/1772690.1772777} {Earthquake shakes
twitter users: real-time event detection by social sensors}.
\newblock In \emph{Proceedings of the 19th International Conference on World
Wide Web}, pages 851--860.
\bibitem[{Sha et~al.(2016)Sha, Li, Chang, and Sui}]{DBLP:conf/naacl/ShaLCS16}
Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1049.pdf} {Joint
learning templates and slots for event schema induction}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 428--434.
\bibitem[{Sha et~al.(2018)Sha, Qian, Chang, and Sui}]{DBLP:conf/aaai/ShaQCS18}
Lei Sha, Feng Qian, Baobao Chang, and Zhifang Sui. 2018.
\newblock \href
{https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16222} {Jointly
extracting event triggers and arguments by dependency-bridge {RNN} and
tensor-based argument interaction}.
\newblock In \emph{Proceedings of the 32nd {AAAI} Conference on Artificial
Intelligence}, pages 5916--5923.
\bibitem[{Shinyama and Sekine(2006)}]{DBLP:conf/naacl/ShinyamaS06}
Yusuke Shinyama and Satoshi Sekine. 2006.
\newblock \href {http://aclweb.org/anthology/N/N06/N06-1039.pdf} {Preemptive
information extraction using unrestricted relation discovery}.
\newblock In \emph{Proceedings of the 2006 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 304--311.
\bibitem[{Srivastava and Sutton(2017)}]{srivastava17lda}
Akash Srivastava and Charles Sutton. 2017.
\newblock \href {https://arxiv.org/abs/1703.01488} {Autoencoding variational
inference for topic models}.
\newblock In \emph{Proceedings of the 2017 International Conference on Learning
Representations}.
\bibitem[{Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and
Salakhutdinov}]{DBLP:journals/jmlr/SrivastavaHKSS14}
Nitish Srivastava, Geoffrey~E. Hinton, Alex Krizhevsky, Ilya Sutskever, and
Ruslan Salakhutdinov. 2014.
\newblock \href {http://dl.acm.org/citation.cfm?id=2670313} {Dropout: a simple
way to prevent neural networks from overfitting}.
\newblock \emph{Journal of Machine Learning Research}, 15(1):1929--1958.
\bibitem[{Sundheim(1992)}]{DBLP:conf/muc/Sundheim92}
Beth Sundheim. 1992.
\newblock \href {https://doi.org/10.1145/1072064.1072066} {Overview of the
fourth message understanding evaluation and conference}.
\newblock In \emph{Proceedings of the 4th Conference on Message Understanding},
pages 3--21.
\bibitem[{Vuurens and de~Vries(2016)}]{DBLP:conf/sigir/VuurensV16}
Jeroen B.~P. Vuurens and Arjen~P. de~Vries. 2016.
\newblock \href {https://doi.org/10.1145/2911451.2914761} {First story
detection using multiple nearest neighbors}.
\newblock In \emph{Proceedings of the 39th International {ACM} {SIGIR}
conference on Research and Development in Information Retrieval}, pages
845--848.
\bibitem[{Yang and Mitchell(2016)}]{DBLP:conf/naacl/YangM16}
Bishan Yang and Tom~M. Mitchell. 2016.
\newblock \href {http://aclweb.org/anthology/N/N16/N16-1033.pdf} {Joint
extraction of events and entities within a document context}.
\newblock In \emph{Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 289--299.
\bibitem[{Yuan et~al.(2018)Yuan, Ren, He, Zhang, Geng, Huang, Ji, Lin, and
Han}]{DBLP:conf/cikm/0001RHZGHJL018}
Quan Yuan, Xiang Ren, Wenqi He, Chao Zhang, Xinhe Geng, Lifu Huang, Heng Ji,
Chin{-}Yew Lin, and Jiawei Han. 2018.
\newblock \href {https://doi.org/10.1145/3269206.3271674} {Open-schema event
profiling for massive news corpora}.
\newblock In \emph{Proceedings of the 27th {ACM} International Conference on
Information and Knowledge Management}, pages 587--596.
\end{thebibliography}