Dependency Parsing with Bottom-up Hierarchical Pointer Networks
Extracted source text · arXiv 2105.09611
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
%
% File acl2021.tex
%
%% Based on the style files for EMNLP 2020, which were
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%% taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{acl2021}
\usepackage{times}
\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
%ADDED
\usepackage{tikz-dependency}
\usepackage{tikz}
\usepackage{tikz-qtree}
\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{arydshln}
%\defcitealias{fernandez-gonzalez-gomez-rodriguez-2019-left}{\scshape FG\&GR19}
\defcitealias{fernandez-gonzalez-gomez-rodriguez-2019-left}{Fdez-G \& G\'omez-R (2019)}
\defcitealias{multipointer}{Fdez-G \& G\'omez-R (2020)}
%SAVE SPACE.
\addtolength{\dbltextfloatsep}{-3mm}
\addtolength{\textfloatsep}{-3mm}
\renewcommand{\floatpagefraction}{.99}%
\renewcommand{\dblfloatpagefraction}{.99}%
\renewcommand{\topfraction}{0.85}
\renewcommand{\bottomfraction}{0.85}
\renewcommand{\textfraction}{0.15}
\newcommand{\dani}[1]{\textcolor{black}{{#1}}}
\newcommand{\carlos}[1]{\textcolor{black}{{#1}}}
\newcommand{\new}[1]{\textcolor{black}{{#1}}}
\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} % Enter the acl Paper ID here
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\newcommand\BibTeX{B\textsc{ib}\TeX}
\title{Dependency Parsing with Bottom-up Hierarchical Pointer Networks}
\author{Daniel Fern\'{a}ndez-Gonz\'{a}lez \and Carlos G\'{o}mez-Rodr\'{i}guez\\
Universidade da Coru\~{n}a, CITIC\\
FASTPARSE Lab, LyS Group \\
Depto. de Ciencias de la Computaci\'{o}n y Tecnolog\'{i}as de la Informaci\'{o}n \\
Campus de Elvi\~{n}a, s/n, 15071 A Coru\~{n}a, Spain \\
{\tt d.fgonzalez@udc.es}, {\tt carlos.gomez@udc.es}\\}
\date{}
\begin{document}
\maketitle
\begin{abstract}
\dani{Dependency parsing is a crucial step towards deep language understanding and, therefore, widely demanded by numerous Natural Language Processing applications. In particular,}
left-to-right and top-down transition-based algorithms that rely on Pointer Networks are among the most accurate approaches for
\dani{performing}
dependency parsing. Additionally, it has been observed for the top-down algorithm that Pointer Networks' sequential decoding can be improved by implementing a hierarchical variant, more adequate to model dependency structures. Considering all this, we develop a bottom-up-oriented Hierarchical Pointer Network
for the left-to-right parser and propose two novel transition-based alternatives: an approach that parses a sentence in right-to-left order and a variant that does it from the outside in.
We empirically test the proposed neural architecture with the different algorithms on a wide variety of languages, outperforming the original approach in
practically
all of them and
setting new state-of-the-art
results on the English and Chinese Penn Treebanks for non-contextualized and BERT-based embeddings.
\end{abstract}
\section{Introduction}
\textit{Dependency parsing} consists in representing the grammatical structure of a given sentence by attaching each word to another (which will be considered its \textit{head} or \textit{parent}), to finally gather all these directed links into a dependency tree as the one in Figure~\ref{fig:tree}. \dani{Additionally, these directed links or dependencies are enhanced with \textit{labels} that describe syntactic functions.}
\begin{figure}
\centering
\begin{dependency}[theme = simple]
\small
\begin{deptext}[column sep=1em]
John$_1$ \& and$_2$ \& Mary$_3$ \& play$_4$ \& tennis$_5$ \& together$_6$ \\
\end{deptext}
\deproot[edge unit distance=2ex]{4}{root$_0$}
\depedge{4}{1}{nsubj}
\depedge{3}{2}{cc}
\depedge{1}{3}{conj}
\depedge[edge unit distance=3ex]{4}{5}{obj}
\depedge[edge unit distance=3ex]{4}{6}{advmod}
\end{dependency}
\caption{Dependency tree for a given sentence.}
\label{fig:tree}
\end{figure}
\dani{This syntactic information accurately provided by parsers as \textit{dependency trees} has been demonstrated highly useful for a huge variety of Natural Language Processing (NLP) tasks.
%and still is highly demanded by a huge variety of Natural Language Processing applications.
In particular, dependency parsing has been recently used for machine translation \cite{zhang-etal-2019-syntax-enhanced,YANG2020105042,ZHANG2021103427}, opinion mining \cite{zhang-etal-2020-syntax}, relation and event extraction \cite{Nguyen2019}, question answering \cite{CAOPMID:31562071}, sentiment classification \cite{bai-etal-2021-syntax}, sentence classification \cite{ZHANG2021103427}, summarization \cite{balachandran-etal-2021-structsum} or semantic role labeling and named entity recognition \cite{sachan-etal-2021-syntax}, among others.}
\textit{Pointer Networks} \cite{Vinyals15} have notably succeeded in implementing \carlos{highly accurate versions of one of the most widely used dependency parsing paradigms: \textit{transition-based} dependency parsers.}
In particular, two different algorithms have been proposed: a \textit{top-down} approach \cite{ma-etal-2018-stack} that, at each step and starting from the root node, connects each word to one of its children; and a \textit{left-to-right} variant \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}
that, starting from the left, attaches each word of the sentence to its parent. Apart from doubling the speed of the former, the latter outperforms the top-down variant in terms of accuracy in practically all datasets tested so far.
As transition-based algorithms \cite{nivre-2003-efficient}, both perform the parsing process as a sequential decoding where, at each step or parsing state, all permissible actions (which do not violate the single-head or acyclicity constraints) are evaluated and the highest-scoring one is greedily applied, generating a new state. This sequential decoding differs from the approach followed by their main competitors:
\textit{graph-based} algorithms \cite{mcdonald-etal-2005-online}. These previously score all possible arcs (or sets of arcs) and then, during decoding, search for the highest-scoring valid dependency tree. While recent graph-based models use a head-selection strategy \cite{Zhang17,DozatM17}
similar to that followed by the left-to-right transition-based parser \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}, the training and decoding processes are completely different: at each decoding step, only permissible
arc/transitions
are evaluated in the transition-based approach, and scores are computed taking into account a sequence of previous decisions encoded through the decoder. On the contrary, these simplified graph-based models independently score all possible parents for each word and then, during decoding, only the highest-scoring ones are kept regardless of the parents chosen for the other words. Then, at the end of the process, a maximum spanning tree algorithm is applied (if necessary) to output a well-formed tree.\footnote{Additionally, while in transition-based parsers like \cite{ma-etal-2018-stack} and \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}, the sequential decoding makes it possible to easily add high-order features and projectivity constraints without increasing runtime complexity, these enhancements in graph-based models lead to a performance penalty.
} This difference can be also seen empirically, as the left-to-right transition-based parser achieves higher accuracies than the approach by \citet{DozatM17} in several widely-known benchmarks.
While this sequential decoding (typically implemented by a recurrent neural network) seems to be beneficial for
dependency parsing, it might lead to accuracy losses due to error propagation:
mistaken past decisions
will affect future actions,
especially harming performance on long-range dependencies and attachments created in final steps. This limitation
present in classic transition-based parsers also affects recent algorithms based on Pointer Networks, as decoder states located at the end of the sequence tend to forget relevant information from the past and are more prone to suffer from error propagation.
In fact, \citet{liu-etal-2019-hierarchical} propose a \textit{hierarchical} decoding for the top-down algorithm \cite{ma-etal-2018-stack} by having access, at each step, to information about the focus word's parent and siblings created in the past, introducing not only knowledge about distant decoder states relevant for future decisions, but also an underlying tree structure to the decoding process (more appropriate for modelling dependency graphs).
In this paper, we initially develop a general Hierarchical Pointer Network architecture with a bottom-up structured decoding for the left-to-right transition-based algorithm \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}. This will allow the decoder to have access to information about partial structures created in the past and will help to make better future decisions. Alternatively to this algorithm, we design two novel bottom-up-oriented transition systems that can be easily implemented on the proposed neural model.
Finally, we empirically show that the presented architecture\footnote{Source code available at \url{https://github.com/danifg/BottomUp-Hierarchical-PtrNet}.}
with any transition-based algorithm
provides improvements in accuracy on ten different languages from Universal Dependencies \cite{nivre-etal-2016-universal} and achieves state-of-the-art scores on the widely-known English and Chinese Penn Treebanks \cite{marcus93,Xue2005}.
\dani{The remainder of this article is organized as follows: Section~\ref{sec:preliminaries} introduces the baseline left-to-right parser by \citet{fernandez-gonzalez-gomez-rodriguez-2019-left} and briefly presents the top-down Hierarchical Pointer Network by \citet{liu-etal-2019-hierarchical}. In Section~\ref{sec:proposal}, we describe in detail the novel bottom-up Hierarchical Pointer Network and how it is adapted to the left-to-right algorithm. Section~\ref{sec:newalgos} presents new transition systems and how they are implemented on the novel neural architecture. In Section~\ref{sec:experiments}, we extensively evaluate the proposed neural model with each parsing strategy on numerous datasets, as well as include a thorough analysis of their performance. Lastly, Section~\ref{sec:conclusion} contains a final discussion.}
\section{Preliminaries}
\label{sec:preliminaries}
\subsection{Left-to-Right Transition System}
\citet{fernandez-gonzalez-gomez-rodriguez-2019-left} propose an efficient transition system that is defined by a \textit{focus word pointer} $i$, which is used to point to the word currently being processed, and a single \textsc{Shift-Attach}-$p$ transition, which attaches the current focus word $w_i$ to its parent word $w_p$ in position $p$ (producing the dependency arc $w_p \rightarrow w_i$) and then moves the focus to the next word $w_{i+1}$. This algorithm starts at the first word ($w_1$) of a sentence of length $n$ and sequentially (from left to right) connects each word $w_i$ to its parent in just $n$ steps.\footnote{The top-down transition system by \citet{ma-etal-2018-stack} needs 2$n$-1 steps to parse a sentence of length $n$.} In Figure~\ref{fig:algos}(a), we depict the sequential prediction of \textsc{Shift-Attach}-$p$ transitions that produces all arcs in the dependency tree in Figure~\ref{fig:tree} following a left-to-right transition system.
\begin{figure}[h]
\centering
\includegraphics[width=0.49\textwidth]{algo.png}
\caption{Attachment order and leftmost (\texttt{lm}), rightmost (\texttt{rm}), last-attached left (\texttt{la}) and right (\texttt{ra}) dependents that are available at each decoding step when parsing the sentence in Figure~\ref{fig:tree} with the three available transition-based algorithms. Note that words $w_i$ are represented by the positional index $i$.}
\label{fig:algos}
\end{figure}
In order to incrementally build a well-formed dependency tree during decoding, only \textsc{Shift-Attach}-$p$ transitions that do not generate cycles in the already-built dependency graph are allowed. Additionally, the left-to-right transition system can efficiently produce unrestricted dependency graphs, including non-projective structures; however, the projectivity constraint can be enforced, discarding transitions that produce crossing dependencies in those treebanks with a negligible presence of non-projectivity \cite{multipointer}.
\subsection{Pointer Networks for Left-to-Right Dependency Parsing}
\label{sec:l2r}
The left-to-right algorithm can be easily implemented by a Pointer Network \cite{Vinyals15} to perform non-projective dependency parsing. These sequence-to-sequence neural networks are able to learn the conditional probability of an output sequence of discrete numbers that correspond to positions from an input sequence. In our case, the output is the sequence $\mathbf{p} = p_1, p_2, \dots, p_n$
of
values
necessary to parameterize the \textsc{Shift-Attach}-$p$ transition and build a dependency graph for the input sentence $\mathbf{w} = w_1, \dots ,w_n$. These positions (or values of $p$ in the left-to-right transition system) are selected by a mechanism of attention \cite{Bahdanau2014} over the input sentence. This avoids having to keep a fixed size for the output dictionary, which will vary for each input sentence with respect to its length.
More specifically, the encoder-decoder architecture of the original left-to-right parser
\cite{fernandez-gonzalez-gomez-rodriguez-2019-left} is designed as follows:
\paragraph{Encoder} Given an input sentence $\mathbf{w} = w_1, \dots ,w_n$, each word $w_i$ is initially represented as $x_i$: i.e., a concatenation of character-level representations ($\mathbf{e}^c_i$) \dani{(extracted by max-pooling-based Convolutional Neural Networks \cite{Ma2016})}, POS tag embeddings ($\mathbf{e}^p_i$) and word embeddings ($\mathbf{e}^w_i$):
$$\mathbf{x}_i = \mathbf{e}^c_i \oplus \mathbf{e}^w_i \oplus \mathbf{e}^p_i$$
\noindent Then, each $\mathbf{x}_i$ is fed
one-by-one
into a BiLSTM to generate the vector representation $\mathbf{h}_i$.
$$ \mathbf{h}_i = \mathbf{h}_{li}\oplus\mathbf{h}_{ri} = \mathbf{BiLSTM}(\mathbf{x}_i)$$
\noindent This results in a sequence of encoder hidden states $\mathbf{h}_0 \dots \mathbf{h}_n$, where $\mathbf{h}_0$ is a special vector representation for the ROOT node.
\dani{We also report accuracies with our encoder enhanced with deep contextualized word embeddings extracted from language model BERT \cite{devlin-etal-2019-bert}. In those cases, BERT-based word embeddings are directly concatenated to the resulting basic word representation before feeding the BiLSTM-based encoder:
$$\mathbf{x}'_i = \mathbf{x}_i \oplus \mathbf{e}^{BERT}_i;\ \mathbf{h}_i = \mathbf{BiLSTM}(\mathbf{x}'_i)$$}
\paragraph{Decoder} The sequential decoding that models the transition-based behaviour is implemented by a unidirectional LSTM. At each time step $t$, a new \textit{decoder hidden state} $\mathbf{s}_t$ is generated,
receiving as input the encoder hidden state $\mathbf{h}_i$ of the current focus word $w_i$ and, as a
recurrent neural network, being also conditioned by the previous decoder state $\mathbf{s}_{t-1}$:
$$\mathbf{s}_t = \mathbf{LSTM}(\mathbf{h}_i)$$
%$$\mathbf{s}_t = f(\mathbf{s}_{t-1},\mathbf{h}_i)$$
\noindent In the original work, authors added to $\mathbf{h}_i$ the previous and next encoder hidden states as extra features; however, in a followup paper, they removed those additions since the bidirectional LSTM is already encoding such information and, therefore, they do not lead to significant improvements \cite{multipointer}.
The resulting decoder state $\mathbf{s}_t$ (which represents the current focus word plus the past decisions made so far) is used for computing scores of all possible words $w_j$ from the input (represented by encoder hidden representations $\mathbf{h}_j$ with $j \in [0,n]$ and $j \neq i$) as parent of $w_i$. These scores are obtained by a biaffine scoring function \cite{DozatM17} to finally compute the attention distribution in vector $\mathbf{a}_t$:
\begin{align*}
\mathbf{v}_{tj} = \mathbf{score}(\mathbf{s}_t, \mathbf{h}_j)= f_1(\mathbf{s}_t)^T \mathbf{W} f_2(\mathbf{h}_j)\\
+\mathbf{U}^Tf_1(\mathbf{s}_t) + \mathbf{V}^Tf_2(\mathbf{h}_j) + \mathbf{b};
\end{align*}
\noindent $$\mathbf{a}_t = \mathbf{softmax}(\mathbf{v}_t)$$
where $\mathbf{W}$, $\mathbf{U}$ and $\mathbf{V}$ are the weights and $f_1(\cdot)$ and $f_2(\cdot)$ are two single-layer multilayer perceptrons (MLP) with ELU activation.
The attention vector $\mathbf{a}_t$ is then used as a pointer over the input, selecting, at each step, the highest-scoring position (where the parent word $w_p$ is located) and, therefore, providing a value $p$ necessary for applying a \textsc{Shift-Attach}-$p$ transition and building the arc $w_p \rightarrow w_i$. Finally, as proposed by \cite{ma-etal-2018-stack}, a biaffine classifier \cite{DozatM17} is separately trained to predict the arc label of each dependency created by the pointer.
This transition-based algorithm can process unrestricted non-projective sentences in
$O(n^2)$
time complexity, since the decoding requires $n$ attachments to successfully parse a sentence with length $n$,
and at each step the attention vector $\mathbf{a}_t$ is computed over the whole input.
\subsection{Top-down Hierarchical Pointer Networks}
\citet{liu-etal-2019-hierarchical} introduced Hierarchical Pointer Networks for the top-down transition-based algorithm developed by \citet{ma-etal-2018-stack}. The top-down transition system consists of two actions and a stack: one transition for connecting the word on top of the stack to one of its children and push it into the stack, and another action for popping the current focus word on top.
\citet{liu-etal-2019-hierarchical} design a structured decoding for the original Pointer Network, where not only the immediately-previous decoder state $\mathbf{s}_{t-1}$ of the LSTM is considered for choosing the next action to be applied on the current word on top,
but also decoder hidden states generated when the current focus word's parent
and last sibling
were assigned following the top-down transition system. More graphically, in Figure~\ref{fig:tree}, when, for instance, the current focus word to be processed is \textit{John}, decoder hidden states of its parent \textit{play} and last-assigned sibling \textit{together}\footnote{Assuming that in the top-down transition system, right children are attached first in an inside-out order.} are
used
for choosing \textit{John}'s children. With this strategy, they manage to introduce an explicit structural inductive bias into the original linear decoder, allowing a more adequate modelling for the generation of dependency graphs. \citet{liu-etal-2019-hierarchical} also empirically show that their top-down Hierarchical Pointer Networks are beneficial for increasing the accuracy on arcs created in final steps, which is especially crucial on long sentences.
\section{Bottom-up Hierarchical Pointer Networks}
\label{sec:proposal}
Based on the research work by \citet{liu-etal-2019-hierarchical}, we propose a Hierarchical Pointer Network for the left-to-right transition-based algorithm \cite{fernandez-gonzalez-gomez-rodriguez-2019-left} and other related parsing strategies where dependents of the focus word may already have been attached (as we will design more such algorithms later). In particular, instead of parent or sibling information, we design a bottom-up structured tree decoding by considering decoder hidden states of the focus word's relevant dependents. More concretely, to alleviate the gradual loss of relevant information during the sequential decoding, we propose to consider decoder hidden states of the leftmost and rightmost dependents already attached to the current focus word, as well as the most recently attached left and right dependents. Note that not all this information is available in all strategies, e.g., the left-to-right algorithm has no access to right dependencies of the focus word.
All this will not only provide valuable knowledge about words processed in the past (especially those attached by long-range dependencies), but will also keep an underlying tree structure that will certainly help the decoder to make good decisions throughout the parsing process. For instance, when the sentence from Figure~\ref{fig:tree} is parsed following a purely bottom-up strategy\footnote{Please note that the left-to-right transition system is not purely bottom-up since, when a word is attached to its parent, all its children might have not been assigned yet.} and current focus word \textit{play} must be attached to its parent, it would be helpful to have access to information about its leftmost and rightmost dependents (\textit{John} and \textit{together}, respectively), as well as the more recently-attached right dependent (\textit{tennis}).
\subsection{Structured Tree Decoder}
To implement this novel bottom-up Hierarchical Pointer Network variant, we keep the same encoder as the original approach \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}, but use a bottom-up hierarchical decoder instead.
More in detail, at each step $t$ with current focus word $w_i$, the generated decoder hidden state $\mathbf{s}_t$ is directly conditioned, apart from by the previous decoder state $\mathbf{s}_{t-1}$ and the encoder hidden state $\mathbf{h}_i$, by the decoder states of the already-attached leftmost ($\mathbf{s}_{lm(t)}$) and rightmost ($\mathbf{s}_{rm(t)}$) dependents of $w_i$, as well as the decoder states of the left ($\mathbf{s}_{la(t)}$) and right ($\mathbf{s}_{ra(t)}$) last-attached dependents:\footnote{Note that $\mathbf{s}_{lm(t)}$ and $\mathbf{s}_{rm(t)}$ might have the same values as $\mathbf{s}_{la(t)}$ and $\mathbf{s}_{ra(t)}$, respectively, when just one left or right dependent were assigned for the current focus word.}
$$\mathbf{s}_t = f(\mathbf{s}_{lm(t)},\mathbf{s}_{rm(t)},\mathbf{s}_{la(t)},\mathbf{s}_{ra(t)},\mathbf{s}_{t-1},\mathbf{h}_i)$$
\noindent where $f(\cdot)$ is a fusion function that combines all components into a single decoder state.
Following \citet{liu-etal-2019-hierarchical}, we do not directly feed these six components to the decoder, but utilize a \textit{gating mechanism} to allow our model to adequately extract the most useful information at each decoding step. In particular, we implement two different gating functions:
\begin{equation}
\tag{\textsc{Gate1}}
\begin{split}
\mathbf{g}_t = \mathbf{sigmoid}(\mathbf{W}_{gp}\mathbf{s}_{t-1}\\
+ \mathbf{W}_{glm}\mathbf{s}_{lm(t)}\\
+ \mathbf{W}_{grm}\mathbf{s}_{rm(t)}\\
+ \mathbf{W}_{gla}\mathbf{s}_{la(t)}\\
+ \mathbf{W}_{gra}\mathbf{s}_{ra(t)} + \mathbf{b}_g)
\end{split}
\end{equation}
\begin{equation}
\tag{\textsc{Gate2}}
\begin{split}
\mathbf{g}_t = \mathbf{sigmoid}(\mathbf{W}_{glm}(\mathbf{s}_{t-1} \odot \mathbf{s}_{lm(t)})\\
+ \mathbf{W}_{grm}(\mathbf{s}_{t-1} \odot \mathbf{s}_{rm(t)})\\
+ \mathbf{W}_{gla}(\mathbf{s}_{t-1} \odot \mathbf{s}_{la(t)})\\
+ \mathbf{W}_{gra}(\mathbf{s}_{t-1} \odot \mathbf{s}_{ra(t)}) + \mathbf{b}_g)
\end{split}
\end{equation}
\noindent where $\mathbf{W}_{gp}$, $\mathbf{W}_{glm}$, $\mathbf{W}_{grm}$, $\mathbf{W}_{gla}$, $\mathbf{W}_{gra}$ and $\mathbf{b}_g$ are gating weights. While all decoder states are equally weighted in \textsc{Gate1}, the element-wise product used in \textsc{Gate2} has the effect of similarity comparison among the previous decoder state $\mathbf{s}_{t-1}$ and the other components.
These gating functions are then used to define the fusion function $f$ as follows:
\begin{align*}
\mathbf{h}_t' = tanh(\mathbf{W}_{p}\mathbf{s}_{t-1} + \mathbf{W}_{lm}\mathbf{s}_{lm(t)}\\
+ \mathbf{W}_{rm}\mathbf{s}_{rm(t)}
+ \mathbf{W}_{la}\mathbf{s}_{la(t)}\\
+ \mathbf{W}_{ra}\mathbf{s}_{ra(t)});
\end{align*}
$$\mathbf{h}_t'' = \mathbf{g}_t \odot \mathbf{h}_t';$$
$$\mathbf{s}_t = \mathbf{LSTM}(\mathbf{h}_t'',\mathbf{h}_i)$$
\noindent where $\mathbf{W}_{lm}$, $\mathbf{W}_{rm}$, $\mathbf{W}_{la}$ and $\mathbf{W}_{ra}$ are the weights for combining the five different decoder states into one intermediate hidden state $\mathbf{h}_t'$. After that, the gating mechanism $\mathbf{g}_t$ is applied to control the information flow and generate the hidden state $\mathbf{h}_t''$, which will be fed into the decoder together with the encoder hidden state $\mathbf{h}_i$.
As shown in Section~\ref{sec:l2r}, the resulting decoder state $\mathbf{s}_t$ is then used for computing the attention vector $\mathbf{a}_t$
that will work as a pointer over the input sentence.
\subsection{Model Specifics for the Left-to-Right Transition System}
While the left-to-right transition system can be directly implemented on the described Hierarchical Pointer Network, it cannot use its full potential. This transition-based approach does not follow a fully bottom-up strategy during the left-to-right decoding\footnote{Left dependents are added bottom–up and right dependents top–down.} and, therefore, right dependents are not available when they are needed. In the example, when the word \textit{play} is under processing, only the leftmost dependent \textit{John} is available since no right dependents were created yet and, when the words \textit{tennis} and \textit{together} will be attached, they are no longer needed since the word \textit{play} will have already been processed. This led us to simplify the definition of the gating-based fusion function $f$ for the left-to-right transition system by removing decoder states of right dependents:
\begin{equation}
\tag{l-adapted}
\mathbf{s}_t = f(\mathbf{s}_{lm(t)},\mathbf{s}_{la(t)},\mathbf{s}_{t-1},\mathbf{h}_i)
\end{equation}
\noindent Since, in some languages, it might be the case that words have either only one left dependent ($\mathbf{s}_{lm(t)}$ and $\mathbf{s}_{la(t)}$ being the same) or the left last-attached dependent was created in the previous time step ($\mathbf{s}_{t-1}$ and $\mathbf{s}_{la(t)}$ being the same), we also experiment with a simpler variant of $f$ that just considers decoder states $\mathbf{s}_{t-1}$ and $\mathbf{s}_{lm(t)}$:
\begin{equation}
\tag{l-simple}
\mathbf{s}_t = f(\mathbf{s}_{lm(t)},\mathbf{s}_{t-1},\mathbf{h}_i)
\end{equation}
Finally, it is worth mentioning that the hierarchical decoder does not penalize the quadratic runtime complexity of the left-to-right parser and, as the original work \cite{fernandez-gonzalez-gomez-rodriguez-2019-left}, it is trained by minimizing the total log loss (cross entropy) for choosing the correct sequence of \textsc{Shift-Attach}-$p$ transitions to build a gold dependency tree for the input sentence $\mathbf{w}$ (i.e. predicting the correct sequence of indices $p$, with each decision ($p_t$) being conditioned by previous ones ($p_{<t}$)):
$$\mathcal{L}(\theta) = - \sum_{t=1}^{T} log P_\theta (p_t | p_{<t},\mathbf{w})$$
\noindent The labeler is simultaneously trained by optimizing the sum of their objectives.
\begin{figure*}
\centering
\includegraphics[width=0.8\textwidth]{network2.png}
\caption{Neural architecture for the Hierarchical Pointer Network and decoding steps to parse the sentence in Figure~\ref{fig:tree} with the outside-in transition system. Note that some dependent influences, such as the one generated by arc $1 \rightarrow 3$ in decoder state $\mathbf{s_5}$, will not be used as the word in position $1$ was already processed.}
\label{fig:network}
\end{figure*}
\section{Alternative Transition-based Algorithms}
\label{sec:newalgos}
Apart from the existing left-to-right algorithm, other transition systems can be implemented on the proposed bottom-up Hierarchical Pointer Network. In particular, we develop two alternative approaches to sequentially parse a sentence by attaching each word to its parent.
\paragraph{Right-to-Left}
While a left-to-right hierarchical decoding might be more adequate for modelling
\new{left-branching}
languages where long-range dependencies tend to be leftward arcs (such as Turkish and Korean), for \new{right-branching}
languages with a high predominance of long rightward arcs (such as Arabic and Hebrew),
a right-to-left transition system could be a perfect fit.\footnote{\dani{Please note that, following common usage in the parsing literature, naming conventions in this paper use the terms ``left'' and ``right'' to refer to temporal order (under the convention that a word is to the ``left'' of another if it comes first), not to spatial order when the words are written in a given script. This means that in languages with a right-to-left writing system, such as Arabic and Hebrew, the original left-to-right parser will begin reading each line of text from the right (i.e., the beginning), and the right-to-left algorithm will begin from the left (i.e., the end). Similarly, the graphical representation of what we call a ``leftward arc'' would be an arrow pointing to the right in those languages.}}
This parses a sentence starting from the last word $w_n$ and it provides a \textsc{Shift-Attach}-$p$ transition that, at each step, assigns a parent to the current focus word $w_i$ and moves $i$ to point to word $w_{i-1}$. In Figure~\ref{fig:algos}(b), we describe the order in which arcs are created with the right-to-left transition system to parse the sentence in Figure~\ref{fig:tree} and the available dependents at each decoding step. Symmetrically to the left-to-right algorithm, we adjust the fusion function $f$ to process only decoder states of right dependents:
\begin{equation}
\tag{r-adapted}
\mathbf{s}_t = f(\mathbf{s}_{rm(t)},\mathbf{s}_{ra(t)},\mathbf{s}_{t-1},\mathbf{h}_i)
\end{equation}
\noindent Additionally, following the same reasoning as for the left-to-right algorithm, we also implement a simplified alternative that removes $\mathbf{s}_{ra(t)}$ from the equation:
\begin{equation}
\tag{r-simple}
\mathbf{s}_t = f(\mathbf{s}_{rm(t)},\mathbf{s}_{t-1},\mathbf{h}_i)
\end{equation}
\paragraph{Outside-in}
In order to fully use the proposed neural architecture and, therefore, have access to left and right dependents during the decoding process, we design a novel \textit{outside-in} transition system: it parses a sentence starting from the leftmost word ($w_1$) and continuing with the rightmost word ($w_n$) towards the middle of the sentence, alternating words from the left and the right. To that end, a canonical order $w_1, w_n, w_2, w_{n-1}, \dots , w_{\lfloor \frac{n+1}{2} \rfloor}$ is determined at the beginning and at each decoding step the \textsc{Shift-Attach}-$p$ transition, apart from attaching $w_i$ to its parent, moves $i$ according to that order. In Figure~\ref{fig:algos}(c), we can see an example of how this outside-in algorithm works and how left and right dependents are available during the decoding process. Since this strategy allows us to use decoder states of left and right dependents, the fusion function $f$ can be fully used. For the same reasons as stated for the other transition systems, we also propose a simplification that discards last-attached dependents as follows:
\begin{equation}
\tag{simple}
\mathbf{s}_t = f(\mathbf{s}_{lm(t)},\mathbf{s}_{rm(t)},\mathbf{s}_{t-1},\mathbf{h}_i)
\end{equation}
\noindent Note that both alternative transition systems are guaranteed to parse a sentence of length $n$ in just $n$ steps, keeping the same runtime complexity as the left-to-right algorithm, and are likewise trained, except for the order in which transitions (values of $p$) are predicted.
Figure~\ref{fig:network} depicts the proposed Hierarchical Pointer Network architecture and the decoding procedure for parsing the sentence in Figure~\ref{fig:tree} with the novel outside-in transition system. In this sketch, it can be graphically seen how decoder hidden states of dependents influence the decoding process.
Finally, it is worth mentioning that
it is not possible to design a purely bottom-up transition system on the Pointer Network framework using $n$ transitions (one per word)
as, regardless of the order in which words are considered, it is not possible to guarantee that dependents are always processed before heads (e.g., the first node considered will not necessarily be a leaf).
\section{Experiments}
\label{sec:experiments}
\subsection{Data}
We conduct experiments on a wide variety of languages from Universal Dependencies v2.6 \cite{nivre-etal-2016-universal}. Following \cite{kulmizev-etal-2019-deep}, we choose ten treebanks
from different language families, with different morphological complexity and with different predominances of long-range dependencies. These are detailed in Table~\ref{tab:treebanks}.
\begin{table}[h]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 0pt}l@{\hskip 8pt}l@{\hskip 4pt}c@{\hskip 8pt}c@{\hskip 8pt}c@{\hskip 6pt}c@{\hskip 4pt}l@{\hskip 0pt}}
\scriptsize{\textbf{Language}} & \scriptsize{\textbf{Treebank}} & \scriptsize{\textbf{Family}} & \scriptsize{\textbf{Order}} & \scriptsize{\textbf{Size}} & \scriptsize{\textbf{\%long}} & \scriptsize{\textbf{\%left}} \\
\hline
Arabic & PADT & AA & VSO & 6.1k & 16.62 & 10.30 \\
Basque & BDT & LI & SOV & 5.4k &
20.17 & 44.46
\\
Chinese & GSD & ST & SVO & 4.0k &
25.31 & 59.44
\\
English & EWT & IE & SVO & 12.5k &
16.43 & 25.49
\\
Finnish & TDT & UR & SVO & 12.2k &
17.45 & 26.83
\\
Hebrew & HTB & AA & SVO & 5.2k &
17.82 & 20.60
\\
Italian & ISDT & IE & SVO & 13.1k &
16.20 & 22.69
\\
Korean & GSD & KO & SOV & 4.4k &
14.76 & 78.40
\\
Swedish & Talbanken & IE & SVO & 4.3k &
19.22 & 29.51
\\
Turkish & IMST & TU & SOV & 3.7k &
14.84 & 70.69
\\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Detailed treebanks used in our experiments. Family = \dani{Afro-Asiatic (AA), Indo-European (IE), Koreanic (KO), Language isolate (LI), Sino-Tibetan (ST), Turkic (TU) or Uralic (UR)}. Order = dominant word order according to WALS \cite{haspelmath2005world}. Size = number of training sentences. \%long = percentage of long arcs (length $>$ 4) in the dev split. \%left = percentage of long arcs that are leftward. }
%\%longL and \%longR = percentage of leftward and rightward long arcs (length $>$ 4) in the development set.}
\label{tab:treebanks}
\end{center}
\end{small}
\end{table}
Additionally, we test our approach on two widely-used benchmarks: the Stanford Dependencies \cite{de-marneffe-manning-2008-stanford} conversion (using Stanford parser v3.3.0)\footnote{\url{https://nlp.stanford.edu/software/lex-parser.shtml}} of the English Penn Treebank (PTB) \cite{marcus93}, with standard splits
and, following \cite{multipointer}, without any PoS tags;\footnote{It has been shown that using predicted PoS tags does not lead to accuracy gains in parsers built on Pointer Networks \cite{ma-etal-2018-stack}.}
and the converted dependency variant \cite{zhang-clark-2008-tale} of the Chinese Penn Treebank 5.1 (CTB) \cite{Xue2005} with gold POS tags.
Following standard practice, we just exclude punctuation for evaluating on PTB and CTB, and, due to random initializations, we report the average Labelled and Unlabelled Attachment Scores (LAS and UAS) over 3 repetitions for each experiment.
\subsection{Settings}
We use the code of the original parser by
\citet{fernandez-gonzalez-gomez-rodriguez-2019-left}
(with the modifications proposed for the single-task parser in \cite{multipointer}) and implement the bottom-up Hierarchical Pointer Network plus the two novel transition systems, allowing a homogeneous comparison against the baseline. Similarly to \cite{liu-etal-2019-hierarchical}, we experiment with the two available gate mechanisms in combination with the proposed transition systems and fusion function implementations.
Character and PoS tag embeddings are randomly initialized and
word vectors are initialized with pre-trained structured-skipgram embeddings \cite{Ling2015} for English and Chinese, and Polyglot embeddings \cite{al-rfou-etal-2013-polyglot} for other languages, being all of them fine-tuned during training.
For PTB and CTB, we additionally concatenate deep
contextualized word embeddings from
the pre-trained language model
BERT \cite{devlin-etal-2019-bert}. We follow the greener and less resource-consuming
approach undertaken by \cite{multipointer} and directly feed
fixed
weights
extracted
from BERT as described in Section~\ref{sec:l2r},
without any fine-tuning to our specific task. More specifically, we extract a combination of weights from 17-20 layers of BERT$_\textsc{LARGE}$ for PTB and weights from the second-to-last layer of BERT$_\textsc{BASE}$ for CTB, \dani{averaging BERT-based embeddings when a word is tokenized into more than one subword.}
Finally, we use beam size 10 for PTB and CTB, and 1 for UD, the Adam optimizer \cite{Adam} and follow \cite{ma-etal-2018-stack,DozatM17} for
parameter optimization and
hyper-parameter selection. These are detailed in Table~\ref{tab:hyper}.
\begin{table}[h]
\begin{footnotesize}
\centering
\begin{tabular}{@{\hskip 0pt}lc@{\hskip 0pt}}
%\hline
\textbf{Architecture hyper-parameters} & \\
\hline
CNN window size & 3 \\
CNN number of filters & 50 \\
BiLSTM encoder layers & 3 \\
BiLSTM encoder size & 512 \\
LSTM decoder layers & 1 \\
LSTM decoder size & 512 \\
LSTM layers dropout & 0.33 \\
Word/POS/Char. embedding dimension & 100\\
English BERT embedding dimension & 1024\\
Chinese BERT embedding dimension & 768\\
Embeddings dropout & 0.33 \\
MLP layers & 1 \\
MLP activation function & ELU \\
Arc MLP size & 512 \\
Label MLP size & 128 \\
UNK replacement probability & 0.5 \\
Beam size & 10 \\
\hline
\textbf{Adam optimizer hyper-parameters} &\\
\hline
Initial learning rate & 0.001 \\
$\beta_1$, $\beta_2$ & 0.9 \\
Batch size & 32 \\
Decay rate & 0.75 \\
Gradient clipping & 5.0 \\
\hline
\end{tabular}
\setlength{\abovecaptionskip}{4pt}
\caption{Model hyper-parameters.}
\label{tab:hyper}
\end{footnotesize}
\end{table}
\begin{table*}[tbp]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 1pt}l@{\hskip 4pt}c@{\hskip 4pt}c@{\hskip 4pt}|cccccccccc|c@{\hskip 1pt}}
\textbf{tran.} & \textbf{$f$} & \textbf{gate} & \textbf{ar} & \textbf{en} & \textbf{eu} & \textbf{fi} & \textbf{he} & \textbf{it} & \textbf{ko} & \textbf{sv} & \textbf{tr} & \textbf{zh} & \textbf{Avg.}\\
\hline
\multicolumn{14}{c}{\textbf{dev}} \\
\hline
\textbf{L2R} & - & - & 83.82 & 90.29 & 83.80 & 88.46 & 89.11 & 92.52 & 83.29 & 86.78 & 64.75 & 82.54 & 84.54 \\
\hline
%L2R2
\textbf{L2R} & l-adapted & \textsc{Gate1} & 84.15 & 90.51 & 84.67 & \textbf{88.97} & 89.27 & \textbf{92.63} & 83.81 & 87.41 & 65.35 & 83.62 & 85.04 \\
& l-adapted & \textsc{Gate2} & 84.05 & 90.44 & 84.53 & 88.95 & 89.22 & 92.60 & 83.79 & 87.32 & 65.24 & 83.48 & 84.96 \\
%L2R
& l-simple & \textsc{Gate1} & 84.06 & 90.49 & 84.84 & 88.90 & 89.30 & 92.59 & 83.84 & \textbf{87.44} & \textbf{65.40} & 83.65 & \textbf{85.05} \\
& l-simple & \textsc{Gate2} & 84.05 & 90.29 & 84.77 & 88.89 & 89.20 & 92.62 & \textbf{83.86} & 87.43 & 65.12 & 83.57 & 84.98 \\
\hline
%R2L2
\textbf{R2L} & r-adapted & \textsc{Gate1} & 84.21 & 90.41 & 84.83 & 88.95 & 89.21 & 92.47 & 83.66 & 87.28 & 65.25 & 83.50 & 84.98 \\
& r-adapted & \textsc{Gate2} & 84.20 & 90.26 & \textbf{84.93} & 88.81 & 89.21 & 92.52 & 83.68 & 87.14 & 65.09 & 83.53 & 84.94 \\
%R2L
& r-simple & \textsc{Gate1} & \textbf{84.28} & \textbf{90.52} & 84.75 & 88.86 & 89.28 & 92.56 & 83.71 & 87.41 & 65.33 & \textbf{83.69} & \textbf{85.04} \\
& r-simple & \textsc{Gate2} & 84.26 & 90.46 & 84.88 & 88.85 & 89.20 & 92.40 & 83.69 & 87.26 & 65.36 & 83.52 & 84.99 \\
\hline
%HOI2
\textbf{O-I} & full & \textsc{Gate1} & 84.07 & 90.43 & 84.59 & 88.74 & 89.28 & 92.51 & 83.46 & 86.91 & 65.14 & 83.54 & 84.87 \\
& full & \textsc{Gate2} & 84.00 & 90.33 & 84.42 & 88.80 & 89.09 & 92.38 & 83.45 & 86.96 & 65.07 & 83.47 & 84.80 \\
%HOI
& simple & \textsc{Gate1} & 84.13 & 90.48 & 84.54 & 88.77 & \textbf{89.36} & 92.55 & 83.56 & 87.06 & 65.12 & 83.49 & \textbf{84.91} \\
& simple & \textsc{Gate2} & 84.04 & 90.35 & 84.53 & 88.77 & 89.10 & 92.33 & 83.53 & 87.07 & 65.14 & 83.28 & 84.81 \\
\hline
\multicolumn{14}{c}{\textbf{test}} \\
\hline
\textbf{L2R} & - & - & 84.20 & 89.32 & 84.53 & 88.45 & 87.75 & 92.46 & 86.01 & 88.92 & 66.12 & 83.68 & 85.14 \\
\textbf{L2R} & l-simple & \textsc{Gate1} & 84.45 & \textbf{89.43} & 85.24 & \textbf{89.00} & 88.23 & 92.50 & \textbf{86.21} & 89.16 & 66.63 & \textbf{84.99} & 85.58 \\
\textbf{R2L} & r-simple & \textsc{Gate1} & \textbf{84.51} & \textbf{89.43} & \textbf{85.46} & 88.77 & 88.26 & \textbf{92.62} & 86.16 & \textbf{89.17} & \textbf{67.05} & 84.75 & \textbf{85.62} \\
\textbf{O-I} & simple & \textsc{Gate1} & 84.31 & 89.33 & 85.33 & 88.88 & \textbf{88.27} & 92.54 & 86.14 & 88.99 & 66.60 & 84.43 & 85.48 \\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{LAS comparison of the
original left-to-right parser with sequential decoding against the available three transition systems on Hierarchical Pointer Networks combined with different fusion functions
implementations
and gating mechanisms on ten treebanks from UD. Only best models on average on dev splits are evaluated on test sets. In Appendix~\ref{sec:standev}, we report the standard deviations over 3 runs on test splits. We use ISO 639-1 codes to represent languages.}
\label{tab:ud}
\end{center}
\end{small}
\end{table*}
\subsection{Results}
In Table~\ref{tab:ud}, we report LAS on dev splits of UD treebanks obtained by the proposed bottom-up Hierarchical Pointer Networks with different transition systems and parser configurations (gate and fusion function) and choose
the best configuration on average to be evaluated on test splits. As shown in the reported results,
the proposed architecture with the three algorithms
improves over the baseline parser in practically all datasets, obtaining higher gains on those languages with a larger amount of long arcs (such as Basque and Chinese). We can also observe that
the simplified fusion functions improve over the adapted and full versions on average, meaning that long-distance dependents are more valuable than last-assigned ones and, in some cases, the usage of the latter harms parsing accuracy. Regarding the novel transition systems,
the right-to-left parser outperforms other algorithms on languages with a significant predominance of \carlos{long} rightward arcs (such as Arabic),
and
the outside-in algorithm clearly underperforms the other transition systems on average in spite of having access to left and right relevant dependents, which probably means that a sequential human-like strategy is more suitable for parsing natural languages, or that the outside-in order is too complex to learn effectively. Finally, \textsc{Gate1} obtains higher accuracy in general.
\begin{table}[tbp]
%\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 1pt}lc|cc@{\hskip 1pt}}
\textbf{tran.} & \textbf{$f$} & \textbf{UAS} & \textbf{LAS} \\
\hline
%L2R2
\textbf{L2R} & \textbf{l-adapted} & \textbf{96.03} & \textbf{94.15} \\
%L2R
& l-simple & 96.01 & \textbf{94.15} \\
\hline
%R2L2
\textbf{R2L} & r-adapted & 96.00 & 94.14 \\
%R2L
& \textbf{r-simple} & \textbf{96.04} & \textbf{94.16} \\
\hline
%HOI2
\textbf{O-I} & full & 95.96 & 94.10 \\
%HOI
& \textbf{simple} & \textbf{96.00} & \textbf{94.14} \\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Accuracy performance on PTB dev splits. We mark in bold the chosen configurations.}
\label{tab:ptbdev}
\end{center}
%\end{small}
\end{table}
\begin{table}[tbp]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 1pt}l@{\hskip 2pt}c@{\hskip 8pt}cc@{\hskip 8pt}c@{\hskip 1pt}}
& \multicolumn{2}{@{\hskip 0pt}c@{\hskip 0pt}}{\textbf{PTB}} & \multicolumn{2}{@{\hskip 0pt}c@{\hskip 0pt}}{\textbf{CTB}} \\
\textbf{Parser} & \textbf{UAS} & \textbf{LAS} & \textbf{UAS} & \textbf{LAS} \\
\hline
%\citet{andor-etal-2016-globally} & 94.61 & 92.79 & - & - \\
%\citet{Wang2016} & 94.08 & 91.82 & 87.55 & 86.23\\
%\citet{Cheng2016} & 94.10 & 91.49 & 88.1\hphantom{0} & 85.7\hphantom{0} \\
%\citet{Kuncoro2016} & 94.26 & 92.06 & 88.87 & 87.30 \\
%\citet{zhang-etal-2016-probabilistic} & 93.42 & 91.29 & 87.65 & 86.17 \\
\citet{Zhang17} & 94.10 & 91.90 & 87.84 & 86.15 \\
\citet{Ma2017} & 94.88 & 92.96 & 89.05 & 87.74\\
\citet{DozatM17} & 95.74 & 94.08 & 89.30 & 88.23 \\
\citet{li-etal-2018-seq2seq} & 94.11 & 92.08 & 88.78 & 86.23 \\
\citet{ma-etal-2018-stack} & 95.87 & 94.19 & 90.59 & 89.29 \\
\citet{ji-etal-2019-graph}$^\dagger$ & 95.97 & 94.31 & - & - \\
\citetalias{fernandez-gonzalez-gomez-rodriguez-2019-left} & 96.04 & 94.43 & - & - \\
\citet{li2019global} & 95.83 & 94.54 & 90.47 & 89.44 \\
\citetalias{multipointer} & 96.06 & 94.50 & 90.61 & 89.51 \\
\citet{zhang-etal-2020-efficient}$^\dagger$ & 96.14 & 94.49 & - & - \\
\citet{wang2020secondorder} & 95.98 & 94.34 & \textbf{90.81} & 89.57 \\
\textbf{Hier. Ptr. Net. L2R} & \textbf{96.18} & \textbf{94.59} & 90.76 & \textbf{89.67} \\
\textbf{Hier. Ptr. Net. R2L} & 96.14 & 94.53 & 90.72 & 89.62 \\
\textbf{Hier. Ptr. Net. O-I} & 96.07 & 94.48 & 90.64 & 89.50 \\
\hdashline[1pt/1pt]
\textit{+BERT} \\
\ \ \ \citet{li2019global} & 96.44 & 94.63 & 90.89 & 89.73 \\
\ \ \ \citet{li2019global}$^*$ & 96.57 & 95.05 & - & - \\
\ \ \defcitealias{mohammadshahi2020recursive}{Moham. \& Hend. (2020)}\citetalias{mohammadshahi2020recursive}$^*$ & 96.66 & 95.01 & \textbf{92.86} & 91.11 \\
\ \ \ \citet{wang2020secondorder}$^*$ & 96.91 & 95.34 & 92.55 & 91.38 \\
\ \ \ \citetalias{multipointer} & 96.91 & 95.35 & 92.58 & 91.42 \\
\ \ \ \textbf{Hier. Ptr. Net. L2R} & \textbf{97.05} & 95.47 & 92.70 & 91.50 \\
\ \ \ \textbf{Hier. Ptr. Net. R2L} & 97.01 & \textbf{95.48} & 92.75 & \textbf{91.62} \\
\ \ \ \textbf{Hier. Ptr. Net. O-I} & 96.95 & 95.36 & 92.65 & 91.47\\
\hline
\hline
\citet{zhou-zhao-2019-head} & 96.09 & \textbf{94.68} & - & - \\
\citetalias{multipointer} & \textbf{96.25} & 94.64 & \textbf{90.79} & \textbf{89.69} \\
\hdashline[1pt/1pt]
\textit{+BERT} \\
%\hdashline[1pt/1pt]
\ \ \ \citet{zhou-zhao-2019-head}$^*$ & \textbf{97.00} & 95.43 & 91.21 & 89.15 \\
\ \ \ \citetalias{multipointer} & 96.97 & \textbf{95.46} & \textbf{92.78} & \textbf{91.65} \\
\hline
\hline
\citet{liu-etal-2019-hierarchical} & 96.09 & 95.03 & - & - \\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Accuracy comparison of state-of-the-art dependency parsers on PTB and CTB. Second block gathers approaches that are enhanced with constituent information and, the last block, includes the performance of the top-down transition-based model with Hierarchical Pointer Networks, since only scores with gold PoS tags are reported. Models that fine-tune BERT are marked with $^*$. Those parsers marked with $^\dagger$ report scores based on the best single run on the development set, instead of reporting the average score on the test set over several runs, \carlos{i.e., instead of averaging to mitigate the effect of random seeds in reported accuracy, they use model selection to choose the most promising seed using the dev set} (following this method, our best model \textit{Hier.Ptr.Net L2R} w/o BERT obtains UAS 96.19 LAS 94.61 on PTB).}
\label{tab:penn}
\end{center}
\end{small}
\end{table}
\dani{We also compare our novel approach with the three implemented transition systems
against other state-of-the-art dependency parsers on PTB and CTB. In order to choose the best parser configuration for the Penn Treebanks, we use scores on PTB development splits reported in Table~\ref{tab:ptbdev}. Since \textsc{Gate1} outperforms \textsc{Gate2} in general (as shown for UD treebanks in Table~\ref{tab:ud}), we simply use \textsc{Gate1} in all experiments and vary the fusion function implementation. Lastly, for BERT augmentations, we directly use simplified fusion functions, significantly reducing training time.
As we can see in Table~\ref{tab:penn}, the
left-to-right approach
achieves the highest accuracy obtained so far on PTB and CTB test splits} with neither
contextualized word embeddings nor extra constituent information. Regarding BERT augmentations, all transition systems not only outperform the baseline \cite{multipointer}, but also the right-to-left model achieves
the best LAS to date among approaches that use (or fine-tune) BERT (even improving over those enhanced with constituent information on PTB). These results provide some evidences that our proposed neural architecture has access to some structural information not learnt by the pre-trained language model BERT.
\begin{table}[tbp]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 0pt}lcccccc@{\hskip 0pt}}
& \multicolumn{2}{r}{\textbf{Left2Right}} & \multicolumn{2}{r}{\textbf{Right2Left}} & \multicolumn{2}{r}{\textbf{Outside-in}} \\
\textbf{Lang.} & \textbf{all} & \textbf{long} & \textbf{all} & \textbf{long} & \textbf{all} & \textbf{long} \\
\hline
Arabic & 10.07 & 1.20 & \textbf{22.20} & \textbf{8.00} & 14.59 & 3.75 \\
Basque & 6.65 & 2.29 & 5.75 & 2.17 & \textbf{7.10} & \textbf{2.81} \\
Chinese & \textbf{16.34} & \textbf{6.17} & 7.99 & 3.83 & 12.60 & 5.34
\\
English & \textbf{7.02} & 1.66 & 4.54 & \textbf{2.54} & 6.10 & 2.33
\\
Finnish & \textbf{7.06} & 1.69 & 5.36 & \textbf{2.70} & 6.61 & 2.45
\\
Hebrew & 10.80 & 1.69 & \textbf{11.77} & \textbf{5.47} & 11.69 & 3.95
\\
Italian & \textbf{11.15} & 1.74 & 8.97 & \textbf{4.88} & 10.64 & 3.73
\\
Korean & \textbf{6.94} & \textbf{2.45} & 4.64 & 0.79 & 5.77 & 1.61
\\
Swedish & \textbf{10.63} & 2.70 & 7.81 & \textbf{4.18} & 9.61 & 3.71
\\
Turkish & \textbf{5.71} & \textbf{1.85} & 3.46 & 0.74 & 5.07 & 1.27
\\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Number of all and long-range dependents per sentence that are available when processing gold trees in UD dev splits with each proposed transition system. Note that the number of long-range dependents (with arc lengths $>$ 4) per sentence is notably low since short sentences are also considered for the computation.}
\label{tab:deps}
\end{center}
\end{small}
\end{table}
\subsection{Dependent Information Availability}
\dani{To further understand the availability of dependent information for each transition system and its impact on parsing performance,
we report in Table~\ref{tab:deps} the number of dependents per sentence that are accessible for each transition system on UD datasets. While, at a first glance, we might think that the outside-in algorithm is the strategy that leverages dependent information the most since it receives both left and right dependents, it seems that, on overall and according to gold trees, the left-to-right parser is the strategy that has access to a larger amount of (in this case, left) dependents and, the right-to-left approach, the one that uses more long-range (right) dependents (typically present
in \new{right-branching} languages, where rightward dependency arcs tend to be longer). This might explain the good results of the right-to-left approach on UD datasets on average (since \new{the majority of tested languages have a higher percentage of long rightward arcs and} long-range dependents are considered more valuable for reducing error propagation) and the fact that the l-adapted fusion function has a better performance on the left-to-right parser (since this function gathers information about closer dependents and this algorithm has access to a notable amount of dependent information).
Finally, we can also observe that\new{, as expected,} the right-to-left algorithm is more adequate to model languages with a high predominance of rightward arcs such as Hebrew and especially Arabic, where it has access to a remarkable amount of right dependents not seen by the other transition systems throughout the parsing process.}
\begin{figure*}[h]
\centering
\includegraphics[width=0.45\textwidth]{imga.png}
\vspace{0.5cm}
\includegraphics[width=0.45\textwidth]{imgb.png}
\includegraphics[width=0.45\textwidth]{imgc.png}
\vspace{0.5cm}
\includegraphics[width=0.45\textwidth]{imgd.png}
\caption{Parsing performance of the original left-to-right parser and the proposed variants implemented on Hierarchical Pointer Networks
relative to sentence length and
word position.}
\label{fig:analysis}
\end{figure*}
\subsection{Error Analysis}
We study the mitigation of error propagation by
the proposed
neural
architecture and characterize errors relative to
sentence length and word position\footnote{Errors relative to dependency length (as described in \cite{mcdonald-nivre-2007-characterizing}) are no longer an effective method to measure the impact of error propagation in transition systems designed for Pointer Networks (except for the top-down algorithm \cite{ma-etal-2018-stack}), since these require a single \textsc{Shift-Attach}-$p$ transition to directly build any arc (regardless of its length), not involving several decoder states and, therefore,
not directly reflecting the presence of error propagation.}
for a concatenation of all ten UD datasets\footnote{
In order to prevent data sparsity,
we randomly select the number of sentences that
approximately gather 10,000 tokens per treebank.} and for the PTB with BERT-based embeddings.
In particular, we show in Figures~\ref{fig:analysis}(a) and (b) the accuracy relative to sentence lengths of the three novel
transition systems
plus the baseline parser on UD
and
PTB, respectively. On UD, we see accuracy gains regardless of the sentence length; however, while the improvements tend to be
higher as sentences are longer, these narrow when the length is larger than 40 (especially affecting the left-to-right and outside-in variants). On PTB,
the novel approaches not only outperform the baseline on long sentences (as expected), but also surprisingly on the shortest ones. We also note that,
while the right-to-left parser
obtains the highest performance on the longest sentences (better dealing with error-propagation), the outside-in algorithm is suffering a drop
on sentences with a length greater than 40 on UD
and, with length between 31 and 40, on PTB (possibly
because the information about left and right dependents is exclusively used by this transition system and it tends to be available at final steps
-- as shown
in Figure~\ref{fig:algos}(c) --, where it might be difficult to manage when sentences are substantially long and the amount of information is probably significantly large).
The reduction of error propagation can be seen more clearly in
Figures~\ref{fig:analysis}(c) and (d), where we report the LAS relative to word positions within the sentence. In comparison to the sequential variant, the left-to-right transition system with structured decoding obtains the highest gains in accuracy on attachments made on words at final positions of the sentence (the most affected by error propagation). We can also observe how the right-to-left and outside-in approaches significantly outperform the left-to-right algorithm on words at the end of the sentence, since these transition systems attach those words in initial steps of the parsing process. Lastly, there is a significant drop in accuracy by the outside-in model in words at middle positions on PTB (from 21 to 30) and, less significant, on UD datasets (from 31 to 40). This is probably due to the fact that arcs in those positions are created in final steps following the outside-in strategy, being the most affected by error-propagation.
\section{Conclusions}
\label{sec:conclusion}
We manage to introduce structural knowledge to the sequential decoding of the
left-to-right dependency parser with Pointer Networks. We additionally
implement two novel transition systems that can be added to the proposed neural architecture.
Finally, we extensively test Hierarchical Pointer Networks on a wide range of languages, proving that they lead to significant accuracy gains
as well as
providing evidence that error propagation is mitigated with the presented
structured
decoding.
\section*{Acknowledgments}
We acknowledge the European Research Council (ERC), which has funded this research under the European Union’s Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150), MINECO (ANSWER-ASAP, TIN2017-85160-C2- 1-R), Xunta de Galicia (ED431C 2020/11), and Centro de Investigaci\'on de Galicia ``CITIC'', funded by Xunta de Galicia and the European Union (ERDF - Galicia 2014-2020 Program), by grant ED431G 2019/01.
\bibliography{anthology,main}
\bibliographystyle{acl_natbib}
\clearpage
\appendix
\section{Appendices}
\label{sec:appendix}
\subsection{Standard deviations}
\label{sec:standev}
\begin{table}[ht]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 1pt}l@{\hskip 2pt}c@{\hskip 8pt}cc@{\hskip 8pt}c@{\hskip 1pt}}
& \multicolumn{2}{@{\hskip 0pt}c@{\hskip 0pt}}{\textbf{PTB}} & \multicolumn{2}{@{\hskip 0pt}c@{\hskip 0pt}}{\textbf{CTB}} \\
\textbf{Parser} & \textbf{UAS} & \textbf{LAS} & \textbf{UAS} & \textbf{LAS} \\
\hline
\textbf{Hier. Ptr. Net. L2R} & $\pm$0.02 & $\pm$0.03 & $\pm$0.05 & $\pm$0.06 \\
\textbf{Hier. Ptr. Net. R2L} & $\pm$0.04 & $\pm$0.03 & $\pm$0.06 & $\pm$0.08 \\
\textbf{Hier. Ptr. Net. O-I} & $\pm$0.03 & $\pm$0.05 & $\pm$0.04 & $\pm$0.06 \\
\hdashline[1pt/1pt]
\textit{+BERT} \\
\ \ \ \textbf{Hier. Ptr. Net. L2R} & $\pm$0.01 & $\pm$0.02 & $\pm$0.04 & $\pm$0.05 \\
\ \ \ \textbf{Hier. Ptr. Net. R2L} & $\pm$0.03 & $\pm$0.02 & $\pm$0.06 & $\pm$0.05 \\
\ \ \ \textbf{Hier. Ptr. Net. O-I} & $\pm$0.02 & $\pm$0.02 & $\pm$0.03 & $\pm$0.04 \\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Standard deviations over 3 runs on test splits for scores reported in Table~\ref{tab:penn}.}
\label{tab:ptbstandev}
\end{center}
\end{small}
\end{table}
\begin{table*}[ht]
\begin{small}
\begin{center}
\centering
\begin{tabular}{@{\hskip 1pt}l@{\hskip 4pt}c@{\hskip 4pt}c@{\hskip 4pt}|cccccccccc@{\hskip 1pt}}
\textbf{tran.} & \textbf{$f$} & \textbf{gate} & \textbf{ar} & \textbf{en} & \textbf{eu} & \textbf{fi} & \textbf{he} & \textbf{it} & \textbf{ko} & \textbf{sv} & \textbf{tr} & \textbf{zh} \\
\hline
\textbf{L2R} & - & - & $\pm$0.06 & $\pm$0.01 & $\pm$0.05 & $\pm$0.06 & $\pm$0.14 & $\pm$0.04 & $\pm$0.06 & $\pm$0.03 & $\pm$0.13 & $\pm$0.04 \\
%L2R
\textbf{L2R} & l-simp. & \textsc{Gate1} & $\pm$0.02 & $\pm$0.04 & $\pm$0.06 & $\pm$0.02 & $\pm$0.08 & $\pm$0.02 & $\pm$0.04 & $\pm$0.04 & $\pm$0.08 & $\pm$0.04 \\
%R2L
\textbf{R2L} & r-simp. & \textsc{Gate1} & $\pm$0.06 & $\pm$0.04 & $\pm$0.02 & $\pm$0.06 & $\pm$0.12 & $\pm$0.02 & $\pm$0.06 & $\pm$0.05 & $\pm$0.11 & $\pm$0.08
\\
%HOI
\textbf{O-I} & simple & \textsc{Gate1} & $\pm$0.02 & $\pm$0.08 & $\pm$0.05 & $\pm$0.04 & $\pm$0.06 & $\pm$0.07 & $\pm$0.09 & $\pm$0.02 & $\pm$0.12 & $\pm$0.08 \\
\hline
\end{tabular}
\centering
\setlength{\abovecaptionskip}{4pt}
\caption{Standard deviations over 3 runs on test splits for scores reported in Table~\ref{tab:ud}.}
\label{tab:standev}
\end{center}
\end{small}
\end{table*}
\end{document}
\begin{thebibliography}{40}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Al-Rfou{'} et~al.(2013)Al-Rfou{'}, Perozzi, and
Skiena}]{al-rfou-etal-2013-polyglot}
Rami Al-Rfou{'}, Bryan Perozzi, and Steven Skiena. 2013.
\newblock \href {https://www.aclweb.org/anthology/W13-3520} {{P}olyglot:
Distributed word representations for multilingual {NLP}}.
\newblock In \emph{Proceedings of the Seventeenth Conference on Computational
Natural Language Learning}, pages 183--192, Sofia, Bulgaria. Association for
Computational Linguistics.
\bibitem[{Bahdanau et~al.(2014)Bahdanau, Cho, and Bengio}]{Bahdanau2014}
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.
\newblock Neural machine translation by jointly learning to align and
translate.
\newblock \emph{CoRR}, abs/1409.0473.
\bibitem[{Bai et~al.(2021)Bai, Wang, Chen, Yang, Bai, Yu, and
Tong}]{bai-etal-2021-syntax}
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and
Yunhai Tong. 2021.
\newblock \href {https://www.aclweb.org/anthology/2021.eacl-main.262}
{Syntax-{BERT}: Improving pre-trained transformers with syntax trees}.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
3011--3020, Online. Association for Computational Linguistics.
\bibitem[{Balachandran et~al.(2021)Balachandran, Pagnoni, Lee, Rajagopal,
Carbonell, and Tsvetkov}]{balachandran-etal-2021-structsum}
Vidhisha Balachandran, Artidoro Pagnoni, Jay~Yoon Lee, Dheeraj Rajagopal, Jaime
Carbonell, and Yulia Tsvetkov. 2021.
\newblock \href {https://www.aclweb.org/anthology/2021.eacl-main.220}
{{S}truct{S}um: Summarization via structured representations}.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
2575--2585, Online. Association for Computational Linguistics.
\bibitem[{Cao et~al.(2021)Cao, Liang, Li, and Lin}]{CAOPMID:31562071}
Qingxing Cao, Xiaodan Liang, Bailin Li, and Liang Lin. 2021.
\newblock \href {https://doi.org/10.1109/tpami.2019.2943456} {Interpretable
visual question answering by reasoning on dependency trees}.
\newblock \emph{IEEE transactions on pattern analysis and machine
intelligence}, 43(3):887—901.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Dozat and Manning(2017)}]{DozatM17}
Timothy Dozat and Christopher~D. Manning. 2017.
\newblock Deep biaffine attention for neural dependency parsing.
\newblock In \emph{{ICLR}}. OpenReview.net.
\bibitem[{Fern{\'a}ndez-Gonz{\'a}lez and
G{\'o}mez-Rodr{\'\i}guez(2019)}]{fernandez-gonzalez-gomez-rodriguez-2019-left}
Daniel Fern{\'a}ndez-Gonz{\'a}lez and Carlos G{\'o}mez-Rodr{\'\i}guez. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1076} {Left-to-right
dependency parsing with pointer networks}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 710--716, Minneapolis,
Minnesota. Association for Computational Linguistics.
\bibitem[{Fernández-González and Gómez-Rodríguez(2020)}]{multipointer}
Daniel Fernández-González and Carlos Gómez-Rodríguez. 2020.
\newblock \href {http://arxiv.org/abs/2009.09730} {Multitask pointer network
for multi-representational parsing}.
\bibitem[{Haspelmath et~al.(2005)Haspelmath, Dryer, University, Bibiko, Press,
f{\"u}r Evolution{\"a}re~Anthropologie, Hagen, Gil, Comrie, and
Schmidt}]{haspelmath2005world}
M.~Haspelmath, M.S. Dryer, Oxford University, H.J. Bibiko, Oxford~University
Press, Max-Planck-Institut f{\"u}r Evolution{\"a}re~Anthropologie, J.~Hagen,
D.~Gil, B.~Comrie, and C.~Schmidt. 2005.
\newblock \href {https://books.google.es/books?id=7rkSDAAAQBAJ} {\emph{The
World Atlas of Language Structures}}.
\newblock Oxford linguistics. OUP Oxford.
\bibitem[{Ji et~al.(2019)Ji, Wu, and Lan}]{ji-etal-2019-graph}
Tao Ji, Yuanbin Wu, and Man Lan. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1237} {Graph-based dependency
parsing with graph neural networks}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 2475--2485, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Kingma and Ba(2014)}]{Adam}
Diederik~P. Kingma and Jimmy Ba. 2014.
\newblock \href {http://arxiv.org/abs/1412.6980} {Adam: A method for stochastic
optimization}.
\newblock Published as a conference paper at the 3rd International Conference
for Learning Representations, San Diego, 2015.
\bibitem[{Kulmizev et~al.(2019)Kulmizev, de~Lhoneux, Gontrum, Fano, and
Nivre}]{kulmizev-etal-2019-deep}
Artur Kulmizev, Miryam de~Lhoneux, Johannes Gontrum, Elena Fano, and Joakim
Nivre. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1277} {Deep contextualized
word embeddings in transition-based and graph-based dependency parsing - a
tale of two parsers revisited}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 2755--2768, Hong Kong,
China. Association for Computational Linguistics.
\bibitem[{Li et~al.(2018)Li, Cai, He, and Zhao}]{li-etal-2018-seq2seq}
Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao. 2018.
\newblock \href {https://www.aclweb.org/anthology/C18-1271} {Seq2seq dependency
parsing}.
\newblock In \emph{Proceedings of the 27th International Conference on
Computational Linguistics}, pages 3203--3214, Santa Fe, New Mexico, USA.
Association for Computational Linguistics.
\bibitem[{Li et~al.(2020)Li, Zhao, and Parnow}]{li2019global}
Zuchao Li, Hai Zhao, and Kevin Parnow. 2020.
\newblock Global greedy dependency parsing.
\newblock In \emph{the Thirty-Fourth AAAI Conference on Artificial Intelligence
(AAAI-2020)}.
\bibitem[{Ling et~al.(2015)Ling, Dyer, Black, and Trancoso}]{Ling2015}
Wang Ling, Chris Dyer, Alan~W. Black, and Isabel Trancoso. 2015.
\newblock \href {https://doi.org/10.3115/v1/N15-1142} {Two/too simple
adaptations of {W}ord2{V}ec for syntax problems}.
\newblock In \emph{Proceedings of the 2015 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics}, pages 1299--1304,
Denver, Colorado. Association for Computational Linguistics.
\bibitem[{Liu et~al.(2019)Liu, Lin, Joty, Han, and
Bing}]{liu-etal-2019-hierarchical}
Linlin Liu, Xiang Lin, Shafiq Joty, Simeng Han, and Lidong Bing. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1093} {Hierarchical pointer
net parsing}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 1007--1017, Hong Kong,
China. Association for Computational Linguistics.
\bibitem[{Ma and Hovy(2016)}]{Ma2016}
Xuezhe Ma and Eduard Hovy. 2016.
\newblock \href {https://doi.org/10.18653/v1/P16-1101} {End-to-end sequence
labeling via bi-directional lstm-cnns-crf}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics}, pages 1064--1074. Association for
Computational Linguistics.
\bibitem[{Ma and Hovy(2017)}]{Ma2017}
Xuezhe Ma and Eduard Hovy. 2017.
\newblock \href {http://aclweb.org/anthology/I17-1007} {Neural probabilistic
model for non-projective mst parsing}.
\newblock In \emph{Proceedings of the Eighth International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 59--69. Asian
Federation of Natural Language Processing.
\bibitem[{Ma et~al.(2018)Ma, Hu, Liu, Peng, Neubig, and
Hovy}]{ma-etal-2018-stack}
Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard
Hovy. 2018.
\newblock \href {https://doi.org/10.18653/v1/P18-1130} {Stack-pointer networks
for dependency parsing}.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1403--1414,
Melbourne, Australia. Association for Computational Linguistics.
\bibitem[{Marcus et~al.(1993)Marcus, Santorini, and Marcinkiewicz}]{marcus93}
Mitchell~P. Marcus, Beatrice Santorini, and Mary~Ann Marcinkiewicz. 1993.
\newblock Building a large annotated corpus of {E}nglish: The {P}enn
{T}reebank.
\newblock \emph{Computational Linguistics}, 19:313--330.
\bibitem[{de~Marneffe and Manning(2008)}]{de-marneffe-manning-2008-stanford}
Marie-Catherine de~Marneffe and Christopher~D. Manning. 2008.
\newblock \href {https://www.aclweb.org/anthology/W08-1301} {The {S}tanford
typed dependencies representation}.
\newblock In \emph{Coling 2008: Proceedings of the workshop on Cross-Framework
and Cross-Domain Parser Evaluation}, pages 1--8, Manchester, UK. Coling 2008
Organizing Committee.
\bibitem[{McDonald et~al.(2005)McDonald, Crammer, and
Pereira}]{mcdonald-etal-2005-online}
Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005.
\newblock \href {https://doi.org/10.3115/1219840.1219852} {Online large-margin
training of dependency parsers}.
\newblock In \emph{Proceedings of the 43rd Annual Meeting of the Association
for Computational Linguistics ({ACL}{'}05)}, pages 91--98, Ann Arbor,
Michigan. Association for Computational Linguistics.
\bibitem[{McDonald and Nivre(2007)}]{mcdonald-nivre-2007-characterizing}
Ryan McDonald and Joakim Nivre. 2007.
\newblock \href {https://www.aclweb.org/anthology/D07-1013} {Characterizing the
errors of data-driven dependency parsing models}.
\newblock In \emph{Proceedings of the 2007 Joint Conference on Empirical
Methods in Natural Language Processing and Computational Natural Language
Learning ({EMNLP}-{C}o{NLL})}, pages 122--131, Prague, Czech Republic.
Association for Computational Linguistics.
\bibitem[{Mohammadshahi and Henderson(2020)}]{mohammadshahi2020recursive}
Alireza Mohammadshahi and James Henderson. 2020.
\newblock \href {http://arxiv.org/abs/2003.13118} {Recursive non-autoregressive
graph-to-graph transformer for dependency parsing with iterative refinement}.
\bibitem[{Nguyen and Verspoor(2019)}]{Nguyen2019}
Dat~Quoc Nguyen and Karin Verspoor. 2019.
\newblock \href {https://doi.org/10.1186/s12859-019-2604-0} {From pos tagging
to dependency parsing for biomedical event extraction}.
\newblock \emph{BMC Bioinformatics}, 20(1):72.
\bibitem[{Nivre(2003)}]{nivre-2003-efficient}
Joakim Nivre. 2003.
\newblock \href {https://www.aclweb.org/anthology/W03-3017} {An efficient
algorithm for projective dependency parsing}.
\newblock In \emph{Proceedings of the Eighth International Conference on
Parsing Technologies}, pages 149--160, Nancy, France.
\bibitem[{Nivre et~al.(2016)Nivre, de~Marneffe, Ginter, Goldberg, Haji{\v{c}},
Manning, McDonald, Petrov, Pyysalo, Silveira, Tsarfaty, and
Zeman}]{nivre-etal-2016-universal}
Joakim Nivre, Marie-Catherine de~Marneffe, Filip Ginter, Yoav Goldberg, Jan
Haji{\v{c}}, Christopher~D. Manning, Ryan McDonald, Slav Petrov, Sampo
Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.
\newblock \href {https://www.aclweb.org/anthology/L16-1262} {Universal
dependencies v1: A multilingual treebank collection}.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation ({LREC} 2016)}, pages 1659--1666,
Portoro{\v{z}}, Slovenia. European Language Resources Association (ELRA).
\bibitem[{Sachan et~al.(2021)Sachan, Zhang, Qi, and
Hamilton}]{sachan-etal-2021-syntax}
Devendra Sachan, Yuhao Zhang, Peng Qi, and William~L. Hamilton. 2021.
\newblock \href {https://www.aclweb.org/anthology/2021.eacl-main.228} {Do
syntax trees help pre-trained transformers extract information?}
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter
of the Association for Computational Linguistics: Main Volume}, pages
2647--2661, Online. Association for Computational Linguistics.
\bibitem[{Vinyals et~al.(2015)Vinyals, Fortunato, and Jaitly}]{Vinyals15}
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015.
\newblock \href {http://papers.nips.cc/paper/5866-pointer-networks.pdf}
{Pointer networks}.
\newblock In C.~Cortes, N.~D. Lawrence, D.~D. Lee, M.~Sugiyama, and R.~Garnett,
editors, \emph{Advances in Neural Information Processing Systems 28}, pages
2692--2700. Curran Associates, Inc.
\bibitem[{Wang and Tu(2020)}]{wang2020secondorder}
Xinyu Wang and Kewei Tu. 2020.
\newblock \href {http://arxiv.org/abs/2010.05003} {Second-order neural
dependency parsing with message passing and end-to-end training}.
\bibitem[{Xue et~al.(2005)Xue, Xia, Chiou, and Palmer}]{Xue2005}
Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005.
\newblock \href {https://doi.org/10.1017/S135132490400364X} {The penn chinese
treebank: Phrase structure annotation of a large corpus}.
\newblock \emph{Nat. Lang. Eng.}, 11(2):207--238.
\bibitem[{Yang et~al.(2020)Yang, Wong, Chao, and Zhang}]{YANG2020105042}
Baosong Yang, Derek~F. Wong, Lidia~S. Chao, and Min Zhang. 2020.
\newblock \href {https://doi.org/https://doi.org/10.1016/j.knosys.2019.105042}
{Improving tree-based neural machine translation with dynamic lexicalized
dependency encoding}.
\newblock \emph{Knowledge-Based Systems}, 188:105042.
\bibitem[{Zhang et~al.(2020{\natexlab{a}})Zhang, Zhang, Wang, Li, and
Zhang}]{zhang-etal-2020-syntax}
Bo~Zhang, Yue Zhang, Rui Wang, Zhenghua Li, and Min Zhang. 2020{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.297} {Syntax-aware
opinion role labeling with dependency graph convolutional networks}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 3249--3258, Online. Association for
Computational Linguistics.
\bibitem[{Zhang et~al.(2019)Zhang, Li, Fu, and
Zhang}]{zhang-etal-2019-syntax-enhanced}
Meishan Zhang, Zhenghua Li, Guohong Fu, and Min Zhang. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1118} {Syntax-enhanced neural
machine translation with syntax-aware word representations}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 1151--1161,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Zhang et~al.(2021)Zhang, Li, Fu, and Zhang}]{ZHANG2021103427}
Meishan Zhang, Zhenghua Li, Guohong Fu, and Min Zhang. 2021.
\newblock \href {https://doi.org/https://doi.org/10.1016/j.artint.2020.103427}
{Dependency-based syntax-aware word representations}.
\newblock \emph{Artificial Intelligence}, 292:103427.
\bibitem[{Zhang et~al.(2017)Zhang, Cheng, and Lapata}]{Zhang17}
Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017.
\newblock \href {https://aclanthology.info/papers/E17-1063/e17-1063}
{Dependency parsing as head selection}.
\newblock In \emph{Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics, {EACL} 2017, Valencia,
Spain, April 3-7, 2017, Volume 1: Long Papers}, pages 665--676.
\bibitem[{Zhang et~al.(2020{\natexlab{b}})Zhang, Li, and
Zhang}]{zhang-etal-2020-efficient}
Yu~Zhang, Zhenghua Li, and Min Zhang. 2020{\natexlab{b}}.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.302} {Efficient
second-order {T}ree{CRF} for neural dependency parsing}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 3295--3305, Online. Association for
Computational Linguistics.
\bibitem[{Zhang and Clark(2008)}]{zhang-clark-2008-tale}
Yue Zhang and Stephen Clark. 2008.
\newblock \href {https://www.aclweb.org/anthology/D08-1059} {A tale of two
parsers: {I}nvestigating and combining graph-based and transition-based
dependency parsing}.
\newblock In \emph{Proceedings of the 2008 Conference on Empirical Methods in
Natural Language Processing}, pages 562--571, Honolulu, Hawaii. Association
for Computational Linguistics.
\bibitem[{Zhou and Zhao(2019)}]{zhou-zhao-2019-head}
Junru Zhou and Hai Zhao. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1230} {Head-driven phrase
structure grammar parsing on {P}enn treebank}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 2396--2408, Florence, Italy.
Association for Computational Linguistics.
\end{thebibliography}