Location Attention for Extrapolation to Longer Sequences
Extracted source text · arXiv 1911.03872
%
% File acl2020.tex
%
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%% taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
\documentclass[11pt,a4paper]{article}
\usepackage[table,xcdraw]{xcolor} % ADDED
\usepackage[hyperref]{acl2020}
\usepackage{times}
\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}
% Our packages
\usepackage{graphicx}
\graphicspath{ {images/} }
\usepackage[autostyle]{csquotes}
\usepackage{amsmath,amssymb,amsfonts, mathtools}
\usepackage{placeins}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\aclfinalcopy % Uncomment this line for the final submission
\def\aclpaperid{3207} % Enter the acl Paper ID here
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\newcommand\BibTeX{B\textsc{ib}\TeX}
\title{Location Attention for Extrapolation to Longer Sequences}
\author{Yann Dubois \\
University of Cambridge \\
\texttt{yanndubois96@gmail.com} \\ \And
Gautier Dagan \\
University of Amsterdam \\
\texttt{gautier.dagan@gmail.com} \\\AND
Dieuwke Hupkes\thanks{\ \ Shared senior authorship} \\
ILLC, University of Amsterdam \\
\texttt{d.hupkes@uva.nl} \\\And
Elia Bruni\textsuperscript{*} \\
Universitat Pompeu Fabra \\
\texttt{elia.bruni@gmail.com}
}
\date{}
\begin{document}
\maketitle
\input{text/abstract}
\input{text/introduction.tex}
\input{text/extrapolation_definition.tex}
\label{sec:extrap_def}
\input{text/desiderata.tex}
\label{sec:desiderata}
\input{text/background_attention.tex}
\label{sec:back_attn}
\input{text/method.tex}
\label{sec:method}
\section{Experiments}
\label{sec:experiments}
\input{text/experiments.tex}
\section{Results}
\label{ref:results}
\input{text/results.tex}
\section{Discussion}
\input{text/conclusion.tex}
\section*{Acknowledgments}
Dieuwke Hupkes is funded by the Netherlands Organization
for Scientific Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium. Elia Bruni is funded by the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie
grant agreement No 790369 (MAGIC).
\bibliographystyle{acl_natbib}
\bibliography{acl2020}
\appendix
\end{document}
\begin{abstract}
%Most machine learning tools, especially neural networks, are unable to extrapolate beyond the range defined by their training data.
Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set.
However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple.
In this paper, we first review the notion of extrapolation, why it is important, and how one could hope to tackle it.
We then focus on a specific type of extrapolation, which is especially useful for natural language processing: generalization to sequences longer than those seen during training.
We hypothesize that models with a separate content- and location-based attention are more likely to extrapolate than those with common attention mechanisms.
We empirically support our claim for recurrent \textit{seq2seq} models with our proposed attention on variants of the \textit{Lookup Table} task.
This sheds light on some striking failures of neural models for sequences and on possible methods to approaching such issues.
\end{abstract}
\section{Introduction}
It is indisputable that, in recent years, neural network research has made stunning progress on a wide variety of tasks that require to process sequential inputs, such as machine translation \citep{sutskever2014sequence} and speech recognition \citep{graves2013speech}.
However, many researchers have questioned the forms of generalization that neural networks exhibit, which significantly diverges from human-like generalization \citep{baroni2018still,geirhos2018generalisation}.
%However, recent research has questioned the forms of generalization that neural networks exhibit, which significantly diverge from human-like generalization (e.g., \citealp{baroni2018still}).
This discrepancy with human-like generalization is particularly true when it comes to extrapolating ``outside'' the training space \citep{delosh1997extrapolation,marcus1998rethinking}.
% CAN DELETE
%As a result, the majority of successes of neural networks only require interpolating ``between'' the training data points, even though extrapolation is arguably more important in human-like cognition \cite{marcus2018deep}.
%This is particularly true when it comes to extrapolation \cite{marcus2018deep}, where% the notion of \emph{extrapolation} is counterposed to that of \emph{interpolation}: The latter refers to model predictions that are made ``between'' the training data points, while the first refers to model predictions ``outside'' the training space.
%Having drawn such distinction, we observe that the majority of successes of neural networks only require interpolation, even though extrapolation is arguably more important in human-like cognition \cite{marcus2018deep}.
As neural networks are powerful memorizers \cite{zhang2016understanding} and easily learn superficial statistical cues \cite{jo2017measuring}, testing extrapolation and generalization to samples from the long tails of a distribution %\footnote{Extrapolation as defined we define it, corresponds to testing on samples from the long tails of the distribution rather than samples from an other distribution.}
might be the only way of quantifying their capacity of abstract reasoning \cite{barrett2018measuring}.
%Methods that are capable of abstract reasoning could be key to improve sample efficiency and interpretability as this is akin to how humans learn.
%This argument is well exemplified by \citet{mitchell2018extrapolation}, who state that extrapolation requires exploiting the global structure of a problem such as the physical laws.
%\end{description}
Despite this benefit, little work has been done in extrapolation. A possible explanation is that the probability of encountering a test example in the extrapolation setting seems low when the training set $\mathcal{D}$ is large.\footnote{Extrapolation is still prevalent in practical scenarios as high-dimensional problems would typically require an exponentially large $\mathcal{D}$ to be representative, and the underlying distribution may vary over time \cite{hooker2004diagnostics}.} However, such an argument fails to consider the high cost of error in extrapolation settings, and this can be a barrier for real-world scenarios (e.g., self-driving cars).
In this paper, we focus on extrapolation in sequences. More precisely, how to generalize sequence-to-sequence predictors to inputs of length $n_{*}>n_{\mathcal{D}}$, where $n_{\mathcal{D}}$ denotes the length of the longest sequence in the training set.
Such extrapolation is crucial for language acquisition, where humans have limited learning resources to account for the unbounded nature of language.
To successfully generalize, a language learner needs to process new and potentially longer sentences than previously encountered ones \cite{chomsky1956three}.
Accounting for this unbounded nature of language is challenging for neural networks.
This issue has recently been uncovered for seq2seq models by looking at simple artificial tasks \citep{lake2017generalization,liska2018memorize,weber2018fine}.
\citet{liska2018memorize} find that seq2seq architectures can converge to local minima that generalize, but rarely do.
This suggests that neural networks could generalize but lack inductive biases that favor extrapolatable behavior.
%Even in weak extrapolation settings, when the task requires generalization to slightly longer sequences, neural networks fail drastically \cite{liska2018memorize} if not carefully assisted \cite{hupkes2018learning}.
%
%One of the most common examples where neural network fail at extrapolating is generalizing to longer sequences in a sequence-to-sequence task \cite{}. \textcolor{blue}{Talk about SCAN and lookup tables, but also about something more NLPy}
%Recurrent neural networks such as long-short memory and gated recurrent neural networks are now among the established methods in modelling and transducing sequential data [cite].
%
%
%An important contribution to their success comes from attention mechanisms, which allow recurrent models to keep track of long distance dependencies more effectively.
%
%However, current implementations of the attention mechanism do not typically disentangle content from positional information. On the other hand, in cases such as natural language, it would be useful to have a location based attention in addition to the content one. For example, in English sentences being able to attend to the source words from left to right seems crucial.
%
%Most attention mechanisms are a function of an embedded key and query (e.g. source and target word embedding in NLP) and can thus be seen as content-based addressing. Such attention mechanism can also perform location based addressing in an interpolation setting, as the key could learn to encode the location of the word. The problem with content attention arises when the model has to extrapolate to sequences longer than any of the ones in the training set. Indeed, this would require the embeddings to learn to encode the location of the word in an extrapolatable fashion, which is improbable due to the non-linear behaviour and the large degree of freedom of neural models.
%
%Some recent work have discussed the importance of other types of extrapolation in Natural Language Processing (NLP) and suggested tackling those by incorporate inductive biases to steer the model towards an extrapolatable behavior without diminishing its expressiveness \citep{mitchell2018extrapolation, weyde2018modelling}.
%This paper follows recent works that have suggested the importance of extrapolation in Natural Language Processing (NLP).
%Recent work by \citet{mitchell2018extrapolation} have considered using additional constraints to model global structures in language, which they argue to be pivotal for extrapolation.
%Our approach is more similar to \citet{weyde2018modelling}, who incorporate an inductive bias for learning identity rules without diminishing the models' expressiveness.
%In this paper, we focus on extrapolation %in sequences. More precisely, how to generalize
%to sequences of length $n_{*}>n_{\mathcal{D}}$, where $n_{\mathcal{D}}$ denotes the length of the longest sequence in the training set.
%We believe that this is crucial type for NLP as humans have no issues in such setting.
%
In the following sections, we review the concepts of attention and extrapolation.
We then argue that current attention mechanisms, which are mainly responsible for recent successes in natural language processing (NLP), are unlikely to extrapolate as they depend on the \emph{content} of trained embeddings.
This leads us to introduce a novel \emph{location}-based attention that is loosely inspired by human visual attention.
To avoid gaining extrapolation capabilities at the cost of expressivity, we introduce an \emph{attention mixer} that combines content- and position-based attentions.
Finally, we show that recurrent models equipped with this new attention mechanism can extrapolate to longer sequences.
%In the remainder of this paper, we will first briefly discuss related work and discuss the definition of extrapolation.
%We then discuss our proposed model, which we contextualise in the light
\section{Extrapolation}
%\textcolor{blue}{We need a little paragraph saying that here, as a first contribution, we formalize the notion of extrapolation}
Extrapolation is often used but rarely formally defined.
\citet{ebert2014interpolation} have found that when extrapolation is explicitly defined, it often refers to points outside a hull delimited by the training set.
E.g., rectangular hull, concave hull, or convex hull.
In this work we use the rectangle hull definition \cite{brooks1988characterizing}, as any model which is extrapolatable for this region would also be extrapolatable for the convex and concave definition.
Given any finite training dataset $\mathcal{D} \coloneqq \{\mathbf{x}^{(n)}\}_{n=1}^N \subset \mathbb{R}^d$, we define the interpolation domain to be the $d$-dimensional interval $\mathcal{I}_{inter} \coloneqq \prod_{i=1}^d [\min_n x_i^{(n)}, \max_n x_i^{(n)}]$ and the \textbf{extrapolation domain} its complement $\mathcal{I}_{extra} \coloneqq \mathbb{R}^d \setminus \mathcal{I}_{inter}$.
In other words, we define a test example $\mathbf{x}^*$ to be in the extrapolation setting if \textit{at least} one of its features $x^*_j$ is larger or smaller than any values it took during training (Figure~\ref{fig:convhull}).
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{convexhull}
\caption{Schematic extrapolation setting for $d=2$.}
\label{fig:convhull}
\end{figure}
Throughout this paper, we assume that neural networks with inputs or temporary representations in $\mathcal{I}_{extra}$ will break.
Indeed, for a given target function $t : \mathbb{R}^d \to \mathbb{R}$ to approximate, there is an infinite amount of predictors that satisfy $f(\mathbf{x}) = t(\mathbf{x}),\ \forall \mathbf{x} \in \mathcal{I}_{inter} \subset \mathbb{R}^d$.
Without any additional constraints, it is thus extremely unlikely that $f(\mathbf{x}) = t(\mathbf{x}),\ \forall \mathbf{x} \in \mathbb{R}^d$.
This could explain why neural networks have empirically been found to break in extrapolation settings \cite{lohninger1999teach,hettiarachchi2005extrapolation,mitchell2018extrapolation}.
The rest of the paper discusses how to constrain representations used by our neural models \footnote{Although the sentence length is a scalar, the temporary representations (outputs of a hidden layer) are high dimensional.} to be in $\mathcal{I}_{inter}$ regardless of the source sentence length, without decreasing their expressivity.
%Indeed, models have not been trained to output a coherent prediction when the input or any of their temporary representation.
%As a result, there is usually an infinite amount of possible predictions for the model and it is extremely unlikely that the model at hand
%Indeed, for a given target function $t$ to approximate $f(\mathbf{x}^*) \approx t(\mathbf{x}^*), \ \forall \mathbf{x}^* \in \mathbb{R}^d$, there is an infinite amount of predictors that satisfy $f(\mathbf{x}^*) = t(\mathbf{x}^*),\ \forall \mathbf{x} \in \mathcal{I}_{inter}$.
%but only $f=t$ satisfies $f(\mathbf{x}^*) = t(\mathbf{x}^*),\ \forall \mathbf{x} \in \mathbb{R}^d$.
%Current machine learning systems are usually evaluated in the simpler interpolation setting, but recovering the \enquote{true} predictor $t$ thus requires considering extrapolation.
\section{Desiderata}
First and foremost, we would like a model that can extrapolate to sequences longer than the longest training one $n_{\mathcal{D}}$ (\emph{Extrapolation Constraint}).
As previously discussed, models with inputs or temporary representations in $\mathcal{I}_{extra}$ will very likely break.
To satisfy the extrapolation constraint, neural models should thus not depend on features that take values in $\mathcal{I}_{extra}$ for sequences longer than $n_{\mathcal{D}}$.
%In other words, we want each feature extracted by our neural model, to live in bounded intervals.
%We posit that recurrent seq2seq models with content attention will not be extrapolatable.
%Indeed, such models must learn to attend to certain indices by encoding a positional embedding in the hidden states of the encoder through some sort of internal ``counter''.
%Once in the extrapolation regime, it is unlikely that this counter will work, thereby breaking the model.\footnote{This assumption depends on the architecture and the inductive bias it provides as shown by \citet{weiss2018practical}. For our task, we found that the assumption held regardless of the architecture of the RNN. }
Second, our model should be able to learn very complex positional attention patterns (\emph{Positional Patterns Constraint}).
%This is the main issue we foresee with the sinusoidal encodings as they lack the ability to model location patterns that depend on general word position such as ``look at the $i^{th}$ word (after ...)'' in the extrapolation setting.
%Indeed, the sinusoidal encoding for any fixed offset $\mathbf{p}_{t+k}$ is linear in $\mathbf{p}_{t}$ but not in $k$.\footnote{Reminder: $\sin(x+y) = \sin(x)\cos(y)+\cos(x)\sin(y)$.}
Finally, although the position of words in a sentence is important, many tasks depend on their semantics.
The model should thus still be able to learn content-based attention patterns (\emph{Content Patterns Constraint}).
In the following section, we review previously proposed attention-mechanism and discuss why they do not fulfill the three aforementioned desired properties.
%For example, a simple way of enforcing the extrapolation constraint would be to normalize features by the current input length $n_s$.
%Such model would not satisfy the positional patterns constraint, as it would only allow patterns of attention based on percentile position rather than absolute positions. E.g. it could attend at the 10\% percentile word but not at the $27^{th}$ word.
%In summary, we want a model to be able to : extrapolate, learn content dependent attention patterns, and learn complex position dependent attention patterns.
%In the next section, we discuss our proposed solution to these constraints.
\section{Attention Mechanisms}
%In the following, we discuss attention mechanisms in their most general form and review state of the art attention mechanisms that will be used as baselines.
%Although the idea of having a soft attention over input features started in computer vision \cite{larochelle2010learning,denil2012learning}, modern attention mechanisms usually follow the framework proposed in Neural Machine Translation \cite{bahdanau2014neural}.
%to jointly learn the soft alignment over words and their translation
%An attention mechanism is a differentiable module that generates attention.
%Let us call \textit{the attender} an end-to-end differentiable module that generates attention.
An attention mechanism (or attender) takes as input a matrix of keys $\mathrm{K} \coloneqq \{\mathbf{k}_s^T\}_{s=1}^{n_s} \in \mathbb{R}^{n_s \times d}$ and a query $\mathbf{q}_t \in \mathbb{R}^d$, and outputs a probability mass function $\pmb{\alpha}_t \in \mathbb{R}^{n_s}$ that will weight a set of values $\mathrm{V} \coloneqq \{\mathbf{v}_s^T\}_{s=1}^{n_s} \in \mathbb{R}^{n_s \times d_v}$ to generate a glimpse vector $ \mathbf{g}_t \in \mathbb{R}^{d_v}$ used for downstream tasks.
Following \citet{graves2014neural}, it is useful to think of the attender as a memory access module, $\pmb{\alpha}_t$ as the soft address and $ \mathbf{g}_t $ as the accessed vector.
\begin{equation}\label{eq:attender}
\mathbf{g}_t \coloneqq \sum_{s=1}^{n_s} \mathbf{v}_s \mathrm{attender}(\mathbf{k}_s,\mathbf{q}_t) = \mathrm{V} \pmb{\alpha}_t
\end{equation}
Figure~\ref{fig:attender_seq2seq} illustrates attention in a recurrent seq2seq \cite{cho2014learning}, which we will use for our experiments.
Both the keys and the values correspond to the set of encoder hidden states $\mathrm{K}=\mathrm{V} = \mathrm{E} \coloneqq \{\mathbf{e}^T_s\}_{s=1}^{n_s}$, while the query corresponds to the current decoder hidden state $\mathbf{q}_t = \mathbf{d}_t$.
%\footnote{The glimpse vector $\mathbf{g}_t$ is often called context vector in such framework \cite{luong2015effective}.}
\begin{figure}[h!]
\centering
\includegraphics[width=\columnwidth]{spattn_highlevel}
\caption{Attender in a recurrent seq2seq.}
\label{fig:attender_seq2seq}
\end{figure}
\subsection{Content Attention}
Most attention mechanisms compute ``content-based addressing'' (associative memory) that depend on (partial) matches of the key and query.
They take as input $\mathrm{K}$ and $\mathbf{q}_t$ and output a semantic-based attention $\pmb{\gamma}_t \in \mathbb{R}^{n_s}$.
%Intuitively, content attention enables the network to keep track of high-level concepts and then accesses the details through partial semantic matching.
For example, if you wanted to translate a scientific paper, you could understand the main point of the text without remembering the specific technical terms that were used.
When translating, you would go back to the text and translate the jargon by knowing what to look for.
A number of content-based have been proposed, they usually differ in a $\mathrm{score}$ that quantifies the match between $\mathbf{k}_s,\mathbf{q}_t$ through multinomial logits:
\begin{gather}\label{eq:gamma}
\pmb\gamma_{t} \coloneqq \{\mathrm{softmax}(\mathrm{score}(\mathbf{k}_s,\mathbf{q}_t ))\}_{s=0}^{n_s-1} \\[10pt]
\begin{aligned}\label{eq:content}
&\mathrm{score}(\mathbf{k}_s,\mathbf{q}_t ) \coloneqq \\[10pt]
&\begin{cases}
\mathbf{u}^T \tanh([\widetilde{\mathbf{k}}_s ; \widetilde{\mathbf{q}}_t]) & \text{\small \textit{Additive} } \text{\scriptsize \citet{bahdanau2014neural}} \\[5pt]
\mathbf{k}_s^T \widetilde{\mathbf{q}}_t & \text{\small \textit{Multiplicative} } \text{\scriptsize \citet{luong2015effective}} \\[5pt]
\frac{\mathbf{k}_s^T \mathbf{q}_t}{\sqrt{d}} & \text{\small \textit{S. Dot Prod.} } \text{\scriptsize \citet{vaswani2017attention}} \\
\end{cases}
\end{aligned}
\end{gather}
Where $\widetilde{\mathbf{x}}$ is a shorthand for $W\mathbf{x}$.
\subsection{Location Attention}
A location (or position) attention mechanism computes ``location-based addressing'' (random access memory) that depend on the index of the key.
It takes as input $\mathbf{q}_t$ and outputs a location attention $\pmb{\lambda}_t \in \mathbb{R}^{n_s}$.
Intuitively, it decides which value to retrieve based on its index.
For example, in German sentences, the verb goes at the end of the sentence, after a subordinate clause.
When translating from German to English, it might thus make sense to directly attend to the last word in the German source sentence after encoding a subordinate clause.
There are many other cases where attending to words based on their positions seems important. E.g. translating from subject-object-verb to subject-verb-object languages, or understanding the emphasis in some languages.
Despite the importance of word ordering in natural language, location-based attention is not common in \textit{seq2seq} frameworks.
This is probably because content-based attention can emulate location-based attention in the usual interpolation setting.
Indeed, it can learn to encode a positional embedding in the hidden states of the encoder through some internal ``counter''.
This counter is unlikely to work in the extrapolation regime,\footnote{This assumption can depend on the architecture and the inductive bias it provides \cite{weiss2018practical}. For our task, we found that the assumption held for both LSTM and GRU. } we, therefore, investigate other types of location-attention that could satisfy the extrapolation constraint.
\citet{luong2015effective} proposed a location-based attention by using Equation \ref{eq:gamma} with a score that is independent of the key $\mathrm{score}(\mathbf{k}_s,\mathbf{q}_t )=\mathbf{w}^T \mathbf{q}_t$.
They restrict themselves to sequences of the same length, which is not of interest to our work.
Such a mechanism could be extended to sequences of varying lengths but would still lack extrapolation capacity as the model still has to learn to embed the location of the index it wants to retrieve.
%It also does not satisfy the positional patterns constraint as it cannot easily retrieve ``the ith word after ...''.
%In other frameworks, some models augment content attention to help location addressing.
The Neural Turing Machine
\cite{graves2014neural}, post-processes the content attention by shifting its location by a predicted number of steps.
We use a similar mechanism, which is extrapolatable due to the independence of the sequence length.
Nevertheless, on its own, it does not allow positional-only patterns in variable-length sentences.
For example, it cannot attend to the i\textsuperscript{th} word irrespective of the sentence length.
The same argument holds for other location-based attention developed for architectures with an external memory \cite{sukhbaatar2015end}.
More recently, many location-based attention have been proposed in self-attention mechanism.
These methods are usually based on sinusoidal encodings (SE), which have been proposed to take into account the word positions while bypassing the need for recurrences in encoder-decoder frameworks.
In this paper, we will consider the \textit{transformer} and \textit{transformerXL} (relative SE) attention, which are computed as follows.
\begin{equation}\label{eq:location}
\setlength{\jot}{10pt}
\begin{aligned}
&\mathrm{score}(\mathbf{k}_s,\mathbf{q}_t ) \coloneqq \\
&\begin{cases}
\frac{(\mathbf{k}_s + \mathbf{p}_s)^T (\mathbf{q}_t + \mathbf{p}_t)}{\sqrt{d}} & \text{\small \textit{Transformer} \cite{vaswani2017attention}} \\[10pt]
\frac{(\widetilde{\mathbf{k}}_s + \widetilde{\mathbf{p}}_{s-t})^T (\widetilde{\mathbf{q}}_t + b)}{\sqrt{d}} & \text{\small\textit{TransformerXL} \cite{dai2019transformer}} \\
\end{cases}
\end{aligned}
\end{equation}
Where $\mathbf{p}_{t}$ is a positional encoding with sinusoidals of different frequencies at every dimension.
Although powerful, the sinusoidal encoding and its variants \cite{shaw2018self,dai2019transformer} lack the ability to model location patterns that depend on general word position such as ``look at the $i^{th}$ word (after ...)'' in the extrapolation setting.
Indeed, the sinusoidal encoding for any fixed offset $\mathbf{p}_{t+k}$ is linear in $\mathbf{p}_{t}$ but not in $k$.
Location-based processing of attention has also been proposed as a way of constraining content-based attention to some (soft) window. \citet{yang2018modeling} achieve it by multiplying the content attention by the weights of a predicted Gaussian such that the model has an inductive bias towards attending to words that are close to each other.
\citet{sukhbaatar2019adaptive} use a piece-wise window to decrease the computational complexity of the model.
These methods nevertheless solve a fundamentally different problem and do not allow location-only extrapolatable patterns of attention.
\section{Model}
In this section, we propose a location attender that can satisfy the extrapolation and positional patterns constraint.
We then discuss how to incorporate content attention to satisfy the content patterns constraint.
%
%To illustrate how one can add inductive biases to enforce extrapolation, in Subsection \ref{subsec:location_attender} we introduce a new type of location-based attention which does not suffer from the limitations individuated in the previous section.
%As inductive biases should not decrease the expressivity of the model, in Subsection \ref{subsection:mix_attender} we propose a simple method for combining the location and the content attention to yield a model which is at least as expressive as its constituents.
\subsection{Location Attender}
\label{subsec:location_attender}
%The Location Attender computes \enquote{location based addressing}
%(random access memory) based on the query and the index of the key.
%It takes as input $\mathbf{q}_t$ and outputs a location based attention $\pmb{\lambda}_t \in \mathbb{R}^{n_s}$.
%Intuitively, it decides which value to retrieve only based on its index.
%For example at the second decoding step of a German to English translation, you might want to attend to the last word in the German source sentence (the verb) regardless of its identity.
We would like our position attention to be loosely reminiscent of human attention, whereby we sequentially focus on a single area of the input (e.g., words or pixels) but vaguely perceive neighboring inputs due to the eccentricity effect \cite{carrasco1995eccentricity}.
The visual acuity of humans is uni-modal, symmetric, and spikes at the fovea, which corresponds to a $0^\circ$ retinal eccentricity.
We model this visual acuity using a Gaussian Probability Density Function (PDF) similarly to \citet{mnih2014recurrent}.\footnote{Visual acuity is distributed in a Laplace-like distribution, but initial experiments were more encouraging using a Gaussian.}
I.e. for each step, the Location Attender models a Gaussian attention over the relative word positions.
Specifically, it generates a mean $\mu_t$ and standard deviation $\sigma_t$, which are used to compute the location attention given the values of the PDFs at the relative indices $r_s \coloneqq \frac{s}{n_s-1}$ of the keys:
$$\pmb{\lambda}_{t} \coloneqq \{\mathrm{PDF}_{\mu_t, \sigma_t}(r_s)\}_{s=0}^{n_s-1}$$
Using relative indices $r_s$ instead of the absolute ones $s$ is crucial such that the generated $\mu_t$ is bounded (in $ [0,1]$), thereby satisfying the extrapolation constraint.
This model, unfortunately, fails to satisfy the positional patterns constraint, as it only allows patterns of attention based on percentile positions. E.g., it can decide to attend to the 10\%-percentile word but not to the $2^{nd}$ word.
This incapacity to satisfy the position pattern constraint is a general issue with commonly used attention mechanisms (including sinusoidal-based) that only becomes apparent when dealing with complex extrapolation patterns.
To have a general attention mechanism, we need a $\mu_t$ that can: i) attend to locations based on absolute positions; ii) attend to locations based on percentile positions;
iii) attend to positions based on the previous attention.
We achieve this by defining one building block for each of those requirements ($\mathbf{b}_t$) such that their weighted average forms $\mu_t$, and the weights $\pmb{\rho}_t$ are bounded outputs of the model. The three building blocks are:
\begin{itemize}
\item The step size $\frac{1}{n_s-1}$ between words allows the attention mechanism to depend on absolute positions. The generated weight is an integer, which dictates the additional number of steps to take.
\item The bias term $1$ enables the model to use percentile positions. The generated weight gates it (on or off). %allows $\rho_t^{(1)} \in \{0,0.5,1\}$ is used to attend to the end/middle/start of a sequence.
\item The average position of the previous attention $\bar{\pmb{\alpha}}_{t-1}$ that is gated by the generated weight. This ensures that the model can attend using absolute positions to words at indices not seen during training. E.g., attending to index $n_\mathcal{D}+5$ by first attending to $n_\mathcal{D}$ then $\bar{\pmb{\alpha}}_{t-1} + 5$.
\end{itemize}
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{spattn_position}
\caption{Proposed Location Attender. Given a resized query, the Weighter outputs the standard deviation $\sigma_t$ and $\pmb{\rho}_t$ which will weight the building blocks $\mathbf{b}_t$ to compute the mean $\mu_t$. $\mu_t$ and $\sigma_t$ parametrize a Gaussian PDF used to compute the location attention $\pmb{\lambda}_t$.}
\label{fig:position}
\end{figure}
The weights $\pmb{\rho}_t$ are generated using a Gated Recurrent Unit (GRU) \cite{cho2014learning}. $\mu_t$ is clamped to $[0,1]$ by a linear function to yield interpretable and extrapolatable behaviour. We also force $\sigma_t > min_\sigma$ and normalize it by $n_s$ which respectively avoids division by $0$ and makes $\sigma_t$ comparable regardless of $n_s$. A graphical overview of the Location Attender can be seen in Figure~\ref{fig:position}. Formally:
\begin{align*}
\pmb{\omega}_t &\coloneqq \mathrm{GRU}\left(\mathrm{ReLU}\left(\mathrm{W}^{(resize)} \mathbf{q}_t\right)\right) \\
\sigma_t &\coloneqq \frac{\mathrm{ReLU}(\mathrm{W}^{(\sigma)}\pmb{\omega}_t) + min_\sigma}{n_s} \\
\pmb{\rho}_t &\coloneqq a(\mathrm{W}^{(\rho)}\pmb{\omega}_t) \\
\mathbf{b}_t &\coloneqq \{\bar{\pmb{\alpha}}_{t-1}; \frac{1}{n_s-1}; 1\} \\
\mu_t &\coloneqq \mathrm{clamp}(\pmb{\rho}_t^T \mathbf{b}_t) \\
\lambda_{st} &\coloneqq \frac{1}{\sqrt{2 \pi \sigma_t^2}}\exp \left(\frac{-(\frac{s}{n_s -1}-\mu_t)^2}{2 \sigma_t^2} \right)
\end{align*}
Where $clamp$ is a leaky clamping (2 leaky ReLUs) and $\min_\sigma = 0.27$.
%$$\mathcal{G}^{(all)}(\mathbf{x}) \coloneqq \{\mathcal{G}^{(\alpha)}(x_1); \mathcal{G}^{(1/n_s)}(x_2); \mathcal{G}^{(1)}(x_3) \}$$
$a$ is the activation function that forces each of the three dimensions of $\pmb{\rho}_t$ to take on the desired values.
Namely a sigmoid activation for the gates, and the following ``soft-staircase''
\footnote{Straight-through estimators \citep{bengio2013estimating} and Gumbel-Softmax \citep{jang2016categorical,maddison2016concrete} performed slightly worst and required predefining the maximum number of steps.}
to force the weights of the step size to be approximately integers (Figure~\ref{fig:soft_staircase}):
\[
softstair(x) \coloneqq \lfloor x \rfloor + \mathrm{sigmoid}(20 (x - 0.5 - \lfloor x \rfloor))
\]
%we tried
%using a straight through estimator \cite{bengio2013estimating} and
%the Gumbel-Softmax \cite{jang2016categorical,maddison2016concrete}. Ultimately we found that using a deterministic continuous relaxation of the required step function worked better. We call this the \enquote{soft-staircase} activation function (Figure~\ref{fig:soft_staircase}):
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{soft_staircase}
\caption{Soft staircase activation function.}
\label{fig:soft_staircase}
\end{figure}
%The query $\mathbf{q}_t$ first passes through a linear layer to downsize it. This makes the location attender independent of the size of $\mathbf{q}_t$ such that it could be pretrained and transferred across tasks.
\subsection{Mix Attender}
\label{subsection:mix_attender}
We enforce the content patterns constraint, by using a convex combination of content and location attention (Figure~\ref{fig:mix}):
\begin{align*}
\pmb{\alpha}_t &\coloneqq \%^{(\lambda)}_t \pmb\lambda_t + (1-\%^{(\lambda)}_t) \pmb\gamma_t\\
\%^{(\lambda)}_t &\coloneqq \mathrm{sigmoid}(\mathrm{W}^{(\%)}\mathbf{q}_t)
\end{align*}
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{spattn_mixer}
\caption{Mix Attender. The output $\pmb \alpha_t$ is a convex combination of the content and location attention.}
\label{fig:mix}
\end{figure}
%In order to focus on the different types of attention, a standard encoder-decoder architecture was used for all experiments (Figure~\ref{fig:attender_seq2seq}).
%The architecture and training process has never been modified during our experimentation's.
%More details can be found in Appendix~\ref{sec:architecture}.
\subsection{Datasets}
The fact that humans generate and understand unbounded sentences with a finite experience is often used as proof of the \textit{principle of compositionality} \citep{sep-compositionality}.
Following this argument, methods that can extrapolate to longer sequences should exhibit some compositionality.
%which is one of the missing components in NLP \cite{lake2017generalization,baroni2019linguistic}.
Based on this observation, we evaluate on a compositionality-specific artificial task, \emph{lookup tables} \citep{liska2018memorize}, but extend it to better quantify extrapolation. \footnote{ The extended datasets as well as scripts to generate them can be found at \url{https://github.com/i-machine-think/machine-tasks/tree/master/LongLookupTables}}
This task is especially interesting to us, as there is a clear notion of what a good attention pattern should look like, making it easy to qualitatively and quantitatively analyze attentive models.
It is a well-controlled task, which allows us to uncover challenges that prevent models from extrapolating on real-world data.
%This task is especially interesting to us, as the targets are generated by a known function $t(\mathbf{x})$, which enables us to analyse the predictor $f$ and quantify how far it is from $t$.
%Such property is only possible with artificial tasks that are not affected by external sources of noise.
%Furthermore, well concocted artificial tasks prevent the network to ``hack'' the task through superficial cues.
%We believe that understanding why current models are not able to solve seemingly straightforward tasks is an important step towards the end goal of having NLP systems that can extrapolate on real world data.
%
%Building upon the insights provided by \citeauthor{hupkes2018learning} \shortcite{hupkes2018learning}, namely the importance of attention for compositionality, we investigate a new type of attention to enable extrapolation.
%To test the effectiveness of the proposed attention we use an extended version of the \textit{lookup tables} task \cite{livska2018memorize} which has the advantage of being easy to test for extrapolation and compositionality. Additionally, there is a clear notion of what a good attention pattern should look like, making it easy to qualitatively analyse the model. We consider three variants of this task.\footnote{The datasets and code to generate them can be found online \url{anonymous}}
%github.com/i-machine-think/machine-tasks
\subsubsection{Long Lookup Tables}
\begin{table}[h!]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|c|c|c|}
\hline
\rowcolor[HTML]{EFEFEF}
{\color[HTML]{000000} \textbf{Input}} & {\color[HTML]{000000} \textbf{Target}} & {\color[HTML]{000000} \textbf{Target Attention}} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 000 t1 .} & {\color[HTML]{000000} 000 110 \texttt{<eos>}} & {\color[HTML]{000000} 0 1 2} \\ \hline
{\color[HTML]{000000} 110 t1 .} & {\color[HTML]{000000} 110 110 \texttt{<eos>}} & {\color[HTML]{000000} 0 1 2} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 110 t2 .} & {\color[HTML]{000000} 110 100 \texttt{<eos>}} & {\color[HTML]{000000} 0 1 2} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 000 t1 t1 t2 .} & {\color[HTML]{000000} 000 110 110 100 \texttt{<eos>}} & {\color[HTML]{000000} 0 1 2 3 4} \\ \hline
\end{tabular}
}
\caption{Long lookup table examples.}
\label{table:lookup}
\end{table}
The \textit{lookup tables} task consists in sequentially applying $k$ pre-defined lookup table functions.
The lookup tables are bijective mappings on the set of all 3-bit strings
$t_i: \{0,1\}^3 \to \{0,1\}^3$.
For example, if $t_1(000) = 110$ and $t_2(110) = 100$ then $t_2(t_1(000))=t_2(110)=100$.
Following \citet{hupkes2018learning}, we write the operations from left to right, as well as add the inputs and temporary steps to the targets.
E.g. the previous example corresponds to the input \texttt{000 t1 t2} and the target \texttt{000 110 100}.
General extrapolatable seq2seq models should be able to terminate by outputting an end of sentence token \texttt{<eos>}.
We thus append \texttt{<eos>} to the targets and a full stop \texttt{.} to the inputs. \footnote{This makes the task harder than the one in \citet{hupkes2018learning}, who force termination after the right amount of steps.}
At each decoding step, the target only depends on the previous output and the current lookup table.
E.g. the last decoding step of \texttt{000 t1 t2}, only depends on the previous output $110 = t_1(000)$ and the current table $t_2$.
The network thus has to learn the lookup table mappings and use the correct one at each step.
The gold standard attention, therefore, corresponds to the position of the current lookup table. Table~\ref{table:lookup} illustrates a longer example and its correct attention.
The various train and test sets are generated by composing 6 random lookup tables ${t_1, \dots, t_6}$ that have as input and output one of the $2^3=8$ possible 3-bit strings.
Specifically, we use $k=1\dots 4$ composed tables in the training set, $k=2 \dots 4$ for the interpolation test sets, and $k=5 \dots 9$ for the extrapolation test sets.
There are 5 different extrapolation test sets, depending on their additional lengths compared to the maximum training examples (\texttt{long 1}, \dots, \texttt{long 5}).
We randomly select only 5000 possible examples for each of these test sets.
For the interpolation test sets, we select 3000 examples from all possible input-output pairs.
The training set contains all other possible input-output pairs, approximately 10000 examples.
%For the interpolation test sets, we used the ones proposed by \citet{hupkes2018learning}.
%The \texttt{heldout inputs} test set is created by selecting 2 inputs for each sequence of composed tables in the training data (e.g. $\{000, 001\}$ for $t_3 \circ t_5$ and $\{010, 101\}$ for $t_6
%\circ t_2 \circ t_3$).
%Similarly, the \texttt{heldout composition} takes 50 different sequence of compositions (e.g. $t_3 \circ t_5 \circ t_1$ for each of the 8 possible 3-bit strings).
%Finally, the \texttt{new compositions} test set consists of compositions of two new tables $t_7$, $t_8$ that have only been seen as atomic tables in the training set but never composed.
%I.e. the training set only contains the $2*8=16$ inputs with atomic tables $t_7$, $t_8$ (e.g. \texttt{000 t7} but not \texttt{000 t7 t1} or \texttt{010 t2 t3 t7}).
%Although the 3 aforementioned datasets require compositionality for solving them, \texttt{heldout inputs} and \texttt{heldout composition} are simpler conditions that only require the model to learn how to compose atomic table that it has already seen composed in various ways.
%\begin{table}[h!]
%\centering
%\begin{tabular}{|c|c|c|}
%\hline
%\rowcolor[HTML]{EFEFEF}
%{\color[HTML]{000000} \textbf{Dataset}} & {\color[HTML]{000000} \textbf{\# Examples}} & {\color[HTML]{000000} \textbf{Extrapolation}} \\ \hline
%\rowcolor[HTML]{FFFFFF}
%{\color[HTML]{000000} Train} & {\color[HTML]{000000} 9081} & {\color[HTML]{000000} False} \\ \hline
%\rowcolor[HTML]{FFFFFF}
%{\color[HTML]{000000} Validation} & {\color[HTML]{000000} 475} & {\color[HTML]{000000} False}\\ \hline
%\rowcolor[HTML]{FFFFFF}
%{\color[HTML]{000000} Heldout Inputs} & {\color[HTML]{000000} 2492}& {\color[HTML]{000000} False} \\ \hline
%\rowcolor[HTML]{FFFFFF}
%{\color[HTML]{000000} Heldout Compositions} & {\color[HTML]{000000} 400} & {\color[HTML]{000000} False}\\ \hline
%\rowcolor[HTML]{FFFFFF}
%{\color[HTML]{000000} New Compositions} & {\color[HTML]{000000} 128} & {\color[HTML]{000000} False}\\ \hline
%{\color[HTML]{000000} Long 1, \dots Long 5} & {\color[HTML]{000000} 5000 each}& {\color[HTML]{000000} True} \\ \hline
%\end{tabular}
%\caption{Number of examples per datasets for our lookup table task.}
%\label{table:nexamples}
%\end{table}
%Finally, a validation set is also created by randomly selecting 5\% of the train set.
%The number of examples for each splits is summarized in Table~\ref{table:nexamples}.
%
%The focus is to quantify extrapolation, we thus use $k=1\dots 4$ composed tables in the training set but test on $k=2 \dots 9$.
%
%we use $\mathcal{B}=\{0,1\}$, $M=6$, $j=3$, randomly generate $\{t_i\}_{i=1}^M$ and add all intermediate steps to the target. Examples of the training data are given in Table~\ref{table:lookup}. % (Appendix~\ref{sec:appdx_datasets}).
%
%As the focus is to quantify extrapolation, we use compositions of length $k=1\dots 4$ in the training set but test on $k=2 \dots 9$ ($k=5 \dots 9$ will be referred to as \texttt{Long i} where $i=k-4$).
\subsubsection{Reversed Lookup Tables}
To test whether the attention can generate more complex patterns (investigating the Positional Patterns Constraint), we also introduce a dataset which reverses the order of the inputs in the previous dataset.
E.g. the last example in Table~\ref{table:lookup}, would be written as \texttt{t2 t1 t1 000 .}, the target would not change, and the attention pattern should be \texttt{3 2 1 0 4} (attend to \texttt{.} when outputting \texttt{<eos>}).
Although the change seems minor, we hypothesize that such a setting will be much more complicated as the attention pattern is not monotonic and does not follow the encoding nor the decoding steps.
Indeed, in the previous task, the model %can learn the same transformation in the encoder and decoder which will always be matched, as the
only needs to learn to match the $i^{th}$ decoding step with the $i^{th}$ encoding step.
\subsubsection{Lookup Tables with Noisy Start}
\begin{table}[h!]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|c|c|c|}
\hline
\rowcolor[HTML]{EFEFEF}
{\color[HTML]{000000} \textbf{Input}} & {\color[HTML]{000000} \textbf{Target}} & {\color[HTML]{000000} \textbf{Target Attention}} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 000 t2 ! t1 .} & {\color[HTML]{000000} 000 110 \texttt{<eos>}} & {\color[HTML]{000000} 0 2 3 4} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 110 t5 t3 t1 ! t1 .} & {\color[HTML]{000000} 110 110 \texttt{<eos>}} & {\color[HTML]{000000} 0 4 5 6} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 110 ! t2 .} & {\color[HTML]{000000} 110 100 \texttt{<eos>}} & {\color[HTML]{000000} 0 2 3} \\ \hline
\rowcolor[HTML]{FFFFFF}
{\color[HTML]{000000} 000 t6 t3 ! t1 t1 t2 .} & {\color[HTML]{000000} 000 110 110 100 \texttt{<eos>}} & {\color[HTML]{000000} 0 3 4 5 6 7} \\ \hline
\end{tabular}
}
\caption{Lookup tables with noisy start examples}
\label{table:noisy}
\end{table}
Finally, we introduce another variant that also requires content attention (investigating the Content Patterns Constraint).
To do so, we augment each training example with a start token ``\texttt{!}'' between the input and the tables in the source sequence. We then add $m \sim \mathcal{U}\{0,10\}$ tables $t_i$ before the start token. The target outputs were not modified and are thus independent of the added tables. Solving this task requires to first attend to the input, then to the token which follows ``\texttt{!}'' (content attention) and finally proceed with incremental location attention. Examples of the training data are given in Table~\ref{table:noisy}.
\subsection{Metrics}
\label{ssec:metrics}
The main metric is \textit{sequence accuracy} (\textit{seqAcc}), which corresponds to the accuracy of predicting the entire sequence correctly (including its length). To get insights about how the model works, we will also use two other losses.
\textit{Sequence Accuracy Before Eos} (\textit{seqAccBE}), which only evaluates the accuracy of the sub-sequence before the model generated a \texttt{<eos>}.
%is computed as the \textit{seqAcc} between the output and the initial sub-sequence of the target with the same length. This will only be small if the model predicts erroneous sequences but not if it outputs the \texttt{<eos>} token too soon.
\textit{Attention Loss} (\textit{attnLoss}), which quantifies the quality of the attention pattern before \texttt{<eos>}. It is computed as the mean squared error between the predicted and gold standard attention. \footnote{The loss is overly simplistic as it is symmetric around $\bar{\pmb\alpha}_t$ even though errors in the temporal direction are less serious as the embeddings contain past information.} The attention loss gives an indication of how far the model is to the ideal attention patterns required to solve the sequence. %The lower the better.
%\item \textit{Attention Loss} (\textit{attnLoss}), which quantifies the dissimilarity is the attention pattern compared to the target one of the same length. This is computed as the mean squared error between the outputted and gold standard attention (the lower the better). \footnote{The loss is overly simplistic as it is symmetric around $\bar{\pmb\alpha}_t$ even though errors in the temporal direction are less serious as the embeddings contains past information}.
\subsection{Architecture and Baselines}
Concerning baselines, we use three content attention: additive, multiplicative, scaled dot product (Eq.\ref{eq:content}).
We also have two mixed content-location attention baselines: Transformer and TransformerXL (Eq.\ref{eq:location}).
%Our proposed solution and the baselines only differed in the type of attention.
%For the baselines, we used the three content attention from equation (additive, multiplicative, scaled dot product) as well as the mixed content-location based attention from equation (Transformer and TransformerXL).
To focus on the attention mechanisms, our model and the baselines all use a smaller version of the best performing recurrent seq2seq architecture on the \textit{lookup table} task \cite{hupkes2018learning}.
The model has never been modified during our experimentation and is schematized in Figure~\ref{fig:attender_seq2seq}.
The embeddings are of dimension $64$, the recurrent network is a GRU \cite{cho2014learning} with a hidden size of $128$, $50\%$ dropout \cite{srivastava2014dropout} is applied on the encoder-decoder bottleneck, and a residual connection is used between the inputs (embeddings) and outputs of the encoder.
Training consists of 50 epochs with the Adam \cite{kingma2014adam} optimizer.
\subsection{Interpolation}
For sanity check, we tested all the baselines and our models (with and without attention mix) on the interpolation setting of the three tasks.
Our models and the best baseline (transformer attention) achieved $100\%$ sequence accuracy (\textit{seqAcc}).
\subsection{Extrapolation Constraint}
The major desired property of our model is to be able to extrapolate.
We tested the extrapolation capacity
of our location attender by evaluating its \textit{seqAcc} on the long lookup table extrapolation test sets.
Figure~\ref{fig:res_loc} shows the \textit{seqAcc} of the location attender against the strongest baseline (transformer attention).
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{fig_forward}
\caption{\textit{SeqAcc} for the Location Attender and best baseline on the Long Lookup Tables task ($10$ runs).}
\label{fig:res_loc}
\end{figure}
As hypothesized, the transformer attention has some extrapolation capacity, but our location attender substantially outperforms it in this simple task.
Importantly, the loss in performance in the extrapolation setting for the best baseline is abrupt and goes from $100\%$ to $0\%$ by adding only three tokens to the inputs.
This suggests that commonly used models are brittle and cannot even extrapolate by a small amount.
Although the previous results are encouraging, we would like to understand what is holding back our model from perfectly extrapolating (Figure~\ref{fig:res_loc}).
\begin{figure}[h]
\centering
\includegraphics[width=\columnwidth]{fig_forward_before}
\caption{\textit{SeqAccBE} for the Location Attender and best baseline on the Long Lookup Tables task ($10$ runs).}
\label{fig:res_loc_before}
\end{figure}
To do so, we computed the sequence accuracy before \texttt{<eos>} (\textit{SeqAccBE}).
Figure~\ref{fig:res_loc_before} shows that the model outputs are always correct but that it often terminates decoding too soon, which we will refer to as \textbf{the \texttt{<eos>} problem}.
This suggests that the decoder keeps an internal ``counter'' to increase the probability of outputting \texttt{<eos>} when the decoding step is greater than the ones seen at training time.
The model learns this heuristic, which is always correct during training time and can be thought of as a metric hacking.
Importantly, it is not a ``hard'' boundary: the model is often able to extrapolate a couple of steps but usually stops before the correct number of steps.
\subsection{Positional and Content Patterns Constraint}
Having shown that our model can extrapolate well on a simple task, we would like to investigate whether it can do so for tasks that require more complicated attention patterns such as the reversed and noisy task.
Although the Mix Attender, outperformed all baselines on both tasks, it was not able to get more than $40\%$ and $5\%$ sequence accuracy for \texttt{long 1} and \texttt{long 2} respectively.
\begin{figure}[h!]
\centering
\centerline{\includegraphics[width=\columnwidth]{fig_mixattn.png}}
\caption{\textit{SeqAccBE} ($5$ runs) for the Mix Attender and best baseline on the reversed lookup tables (\textit{reverse}) and lookup tables with noisy start (\textit{noisy}).}
\label{fig:res_mix}
\end{figure}
Figure~\ref{fig:res_mix} shows that when considering \textit{seqAccBE}, the Mix Attender is able to extrapolate well in the noisy setting and a little in the reverse setting.
This suggests that it is not able to extrapolate well when considering sequence accuracy because it strongly suffers from the \texttt{<eos>} problem.
This is a recurrent problem in our experiments and is more likely to happen in harder tasks and larger models.
\subsection{Attention Pattern}
As previously discussed, variants of the lookup table task are especially interesting as we know the gold standard attention pattern.
This enables evaluation of attention patterns through the MSE attention loss (\textit{attnLoss}).
\begin{table}[h]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{|
>{\columncolor[HTML]{EFEFEF}}c |
>{\columncolor[HTML]{FFFFFF}}c |
>{\columncolor[HTML]{FFFFFF}}c |
>{\columncolor[HTML]{FFFFFF}}c |
>{\columncolor[HTML]{FFFFFF}}c |
>{\columncolor[HTML]{FFFFFF}}c |
>{\columncolor[HTML]{FFFFFF}}c |}
\hline
{\color[HTML]{000000} \textbf{Attention}} & \cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Interp.}} & \cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Long 1}} & \cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Long 2}} & \cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Long 3}} &
\cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Long 4}} &
\cellcolor[HTML]{EFEFEF}{\color[HTML]{000000} \textbf{Long 5}} \\ \hline
{\color[HTML]{000000} \textbf{Scaled Dot}} & {\color[HTML]{000000} 5.3} & {\color[HTML]{000000} 6.3} & {\color[HTML]{000000} 8.1} & {\color[HTML]{000000} 10.2} &
{\color[HTML]{000000} 12.6} &
{\color[HTML]{000000} 15.4} \\ \hline
{\color[HTML]{000000} \textbf{Multiplicative}} & {\color[HTML]{000000} 3.1} &
{\color[HTML]{000000} 4.6} & {\color[HTML]{000000} 6.3} & {\color[HTML]{000000} 7.9} & {\color[HTML]{000000} 9.9} & {\color[HTML]{000000} 12.4} \\ \hline
{\color[HTML]{000000} \textbf{Additive}} & {\color[HTML]{000000} 3.1} & {\color[HTML]{000000} 8.4} & {\color[HTML]{000000} 15.6} & {\color[HTML]{000000} 22.2} &
{\color[HTML]{000000} 28.7} &
{\color[HTML]{000000} 34.8} \\ \hline
{\color[HTML]{000000} \textbf{Transformer}} & {\color[HTML]{000000} 2.8} & {\color[HTML]{000000} 3.5} & {\color[HTML]{000000} 6.1} & {\color[HTML]{000000} 9.1} &
{\color[HTML]{000000} 11.7} &
{\color[HTML]{000000} 13.9} \\ \hline
{\color[HTML]{000000} \textbf{TransformerXL}} & {\color[HTML]{000000} 3.0} & {\color[HTML]{000000} 3.9} & {\color[HTML]{000000} 5.3} & {\color[HTML]{000000} 7.1} &
{\color[HTML]{000000} 9.1} &
{\color[HTML]{000000} 11.4} \\ \hline
{\color[HTML]{000000}
\textbf{Mix Attention}} & {\color[HTML]{000000} \textbf{2.1}} & {\color[HTML]{000000} \textbf{2.2}} & {\color[HTML]{000000} \textbf{2.9}} & {\color[HTML]{000000} \textbf{4.1}} &
{\color[HTML]{000000} \textbf{5.3}} &
{\color[HTML]{000000} \textbf{6.7}} \\ \hline
\end{tabular}
}
\caption{\textit{AttnLoss} for various attention models averaged over the three datasets and 5 runs.}
\label{tab:mean_attn}
\end{table}
Table~\ref{tab:mean_attn} shows the attention loss averaged over the three tasks.
Although not perfect, the Mix Attender performs on average the best across all settings. \footnote{Some baselines outperformed it in the interpolation settings of specific tasks. Namely, the additive attention in the reversed task and transformer in the noisy task.}
Crucially, it performs similarly in an interpolation setting and simple extrapolation setting (\texttt{long 1}), while all other baselines perform significantly worse after adding a single token.
Even in \texttt{long 2}, it is competitive with all other attention mechanisms in their interpolation domain.
This indicates that the model is indeed able to extrapolate by being more precise with its attention pattern.
%interstingly additive is the least extrapolatable but also the model with most trainable parameters. Suggesting that as hypothesized, the larger the degrees of freedom the least extrapolatable
\subsection{Qualitative Analysis}
\begin{figure*}[h]
\centering
\centerline{\includegraphics[width=1.8\columnwidth]{example_output}}
\caption{Example output by the attention-mixer for the lookup tables with noisy start task (\texttt{Long 4} test set).}
\label{fig:example_output}
\end{figure*}
In addition to enabling extrapolation, the temporary variables such as the weight given to each building block are very helpful for debugging the model and improving interpretability.
%having a better interpretation of what is happening.
Figure~\ref{fig:example_output} shows the output of a Mix Attender for the \emph{lookup tables with noisy start} task.
The input was sampled from the \texttt{Long 4} test set.
The top-left image shows the final attention.
The top-right table shows the value of some interpretable variables at every decoding step.
The bottom images correspond to the content and location attention.
The first decoding step uses location attention to attend to the first input.
For the next three steps, the model outputs a mixing weight $\%^{(\lambda)} \approx 0$ to focus on content attention.
The content attention successfully finds the first non-noisy table (after \texttt{!}). \footnote{A single step of content attention should be sufficient, but the model seems to consistently use three steps.}
It then goes back to using the location attention with $\rho^{(\alpha)}=1$ and $\rho^{(1/n)}=1$ to generate a diagonal attention.
Finally, it predicts \texttt{<eos>} when attending to the end of the input \enquote{\texttt{.}}.
At each step, $\sigma = min_\sigma$ as it does not need to attend to neighboring words for this task.
$\%^{(\lambda)}$ is never exactly 0 or 1, such that the model can easily learn to switch between content and location attention as it does not collapse to using a single form of attention.
%In summary, the results suggest that the location attender extrapolates well in simple tasks and can learn various patterns. The bottleneck for extrapolation is now due to premature outputs of the \texttt{<eos>}, suggesting that solving this issue is key to reaching perfect extrapolation.
%Extrapolation is a hard setting that has not been much investigated by the machine learning community.
%We argue that it is important to consider such setting to enable more robust, efficient and interpretable methods.
%As current methods that memorize and learn superficial cues are unable to extrapolate while humans are, we posit that such a setting might help (and force) the field to come up with more human-like computational models that are capable of abstract reasoning.
% that are: capable of abstract reasoning, compositional, robust, efficient.
%\dots
%To build a common ground for future research in extrapolation, we formalized its definition.
%Following previous work,
%We argued that extrapolation should be solved by adding inductive biases in computational models of language without hindering their expressiveness.
%To show how one could go about it,
%We then focused on the task of generalizing to longer sequences, a form of extrapolation that enables productivity in language learners.
In this paper, we focused on one type of extrapolation, which is especially important in NLP: generalization to longer sequences.
%tasks that require extrapolation to test sequences longer than the training ones and used an attention mechanism that only depends on the position of the key as an inductive bias.
%\footnote{Importantly, we also saw the same trend when training on much larger datasets.}.
We propose a new location-based attention, and show that it can extrapolate better than previous models while learning various attention patterns.
Despite promising initial results, our model is still unable to extrapolate perfectly for harder tasks.
By analyzing its behavior, we uncovered an interesting heuristic used by seq2seq models, namely that they keep track of a decoding ``counter'' to know when to output the \texttt{<eos>} token.
This is a bottleneck for extrapolation, suggesting that removing this heuristic is key to reaching perfect extrapolation and should be investigated in future work.
Once the \texttt{<eos>} problem is solved, we could test the model on real-world datasets.
It would also be interesting to test such attention mechanisms in self-attentive seq2seq models without recurrence.
Finally, as the location attender is not model dependent, it could be pretrained on complex location patterns and incorporated as a plug-and-play module to get extrapolatable position attention.
%Other future research directions include:
%testing whether the location attender is general enough to learn any pattern generated by a finite state automaton (FSA),
%pre-training the location attender on complex location patterns and using it as a plug-and-play module to incorporate extrapolatable position attention, investigating the efficiency of the model to real-world datasets.
%Although the method has shown promising initial results, there is still much to be done in future work, including:
%\begin{itemize}
%\item Showing that the location attender is general enough to learn any pattern generated by a finite state automaton (FSA).
%\item Pre-training the location attender on complex location patterns (e.g. generated by random FSA) and using it as a plug-and-play module to incorporate extrapolatable position attention.
%\item Showing the efficiency of the model to real world datasets such as NMT.
%\item Test the use of the attention-mixer in stacked self-attention layers (e.g. transformer \cite{vaswani2017attention}) by replacing the time dependent building blocks by layer wise ones, to keep the model fully parallelizable.
%\end{itemize}
Taking a step back, we have shown that current deep learning models with common attention mechanisms are unable to extrapolate well on seemingly straightforward tasks.
This tends to be overlooked by the field due to standard benchmarks that can be solved using only interpolation.
We hope that this paper acts as a reminder that extrapolation is a hard setting that has not been much investigated by the machine learning community.
%We argue that it is important to consider such setting to enable more robust, efficient and interpretable methods.
As current methods that memorize and learn superficial cues are unable to extrapolate while humans are, we believe that such a setting might help (and force) the field to come up with more human-like computational models that are capable of abstract reasoning.
\begin{thebibliography}{39}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Bahdanau et~al.(2015)Bahdanau, Cho, and Bengio}]{bahdanau2014neural}
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.
\newblock \href {http://arxiv.org/abs/1409.0473} {Neural machine translation by
jointly learning to align and translate}.
\newblock In \emph{3rd International Conference on Learning Representations,
{ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings}.
\bibitem[{Bengio et~al.(2013)Bengio, L{\'{e}}onard, and
Courville}]{bengio2013estimating}
Yoshua Bengio, Nicholas L{\'{e}}onard, and Aaron~C. Courville. 2013.
\newblock \href {http://arxiv.org/abs/1308.3432} {Estimating or propagating
gradients through stochastic neurons for conditional computation}.
\newblock \emph{CoRR}, abs/1308.3432.
\bibitem[{Brooks et~al.(1988)Brooks, Carroll, and
Verdini}]{brooks1988characterizing}
Daniel~G Brooks, Steven~S Carroll, and William~A Verdini. 1988.
\newblock Characterizing the domain of a regression model.
\newblock \emph{The American Statistician}, 42(3):187--190.
\bibitem[{Carrasco et~al.(1995)Carrasco, Evert, Chang, and
Katz}]{carrasco1995eccentricity}
Marisa Carrasco, Denise~L Evert, Irene Chang, and Svetlana~M Katz. 1995.
\newblock The eccentricity effect: Target eccentricity affects performance on
conjunction searches.
\newblock \emph{Perception \& Psychophysics}, 57(8):1241--1261.
\bibitem[{Cho et~al.(2014)Cho, van Merrienboer, G{\"{u}}l{\c{c}}ehre, Bahdanau,
Bougares, Schwenk, and Bengio}]{cho2014learning}
Kyunghyun Cho, Bart van Merrienboer, {\c{C}}aglar G{\"{u}}l{\c{c}}ehre, Dzmitry
Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.
\newblock \href {https://doi.org/10.3115/v1/d14-1179} {Learning phrase
representations using {RNN} encoder-decoder for statistical machine
translation}.
\newblock In \emph{Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing, {EMNLP} 2014, October 25-29, 2014, Doha, Qatar,
{A} meeting of SIGDAT, a Special Interest Group of the {ACL}}, pages
1724--1734. {ACL}.
\bibitem[{Chomsky(1956)}]{chomsky1956three}
Noam Chomsky. 1956.
\newblock \href {https://doi.org/10.1109/TIT.1956.1056813} {Three models for
the description of language}.
\newblock \emph{{IRE} Trans. Information Theory}, 2(3):113--124.
\bibitem[{Dai et~al.(2019)Dai, Yang, Yang, Carbonell, Le, and
Salakhutdinov}]{dai2019transformer}
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime~G. Carbonell, Quoc~Viet Le, and
Ruslan Salakhutdinov. 2019.
\newblock \href {https://doi.org/10.18653/v1/p19-1285} {Transformer-xl:
Attentive language models beyond a fixed-length context}.
\newblock In \emph{Proceedings of the 57th Conference of the Association for
Computational Linguistics, {ACL} 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers}, pages 2978--2988. Association for Computational
Linguistics.
\bibitem[{DeLosh et~al.(1997)DeLosh, Busemeyer, and
McDaniel}]{delosh1997extrapolation}
Edward~L DeLosh, Jerome~R Busemeyer, and Mark~A McDaniel. 1997.
\newblock Extrapolation: The sine qua non for abstraction in function learning.
\newblock \emph{Journal of Experimental Psychology: Learning, Memory, and
Cognition}, 23(4):968.
\bibitem[{Ebert et~al.(2014)Ebert, Belz, and Nelles}]{ebert2014interpolation}
Tobias Ebert, Julian Belz, and Oliver Nelles. 2014.
\newblock \href {https://doi.org/10.1109/CIDM.2014.7008683} {Interpolation and
extrapolation: Comparison of definitions and survey of algorithms for convex
and concave hulls}.
\newblock In \emph{2014 {IEEE} Symposium on Computational Intelligence and Data
Mining, {CIDM} 2014, Orlando, FL, USA, December 9-12, 2014}, pages 310--314.
{IEEE}.
\bibitem[{Geirhos et~al.(2018)Geirhos, Temme, Rauber, Sch{\"{u}}tt, Bethge, and
Wichmann}]{geirhos2018generalisation}
Robert Geirhos, Carlos R.~Medina Temme, Jonas Rauber, Heiko~H. Sch{\"{u}}tt,
Matthias Bethge, and Felix~A. Wichmann. 2018.
\newblock \href
{http://papers.nips.cc/paper/7982-generalisation-in-humans-and-deep-neural-networks}
{Generalisation in humans and deep neural networks}.
\newblock In \emph{Advances in Neural Information Processing Systems 31: Annual
Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8
December 2018, Montr{\'{e}}al, Canada}, pages 7549--7561.
\bibitem[{Graves et~al.(2013)Graves, Mohamed, and Hinton}]{graves2013speech}
Alex Graves, Abdel{-}rahman Mohamed, and Geoffrey~E. Hinton. 2013.
\newblock \href {https://doi.org/10.1109/ICASSP.2013.6638947} {Speech
recognition with deep recurrent neural networks}.
\newblock In \emph{{IEEE} International Conference on Acoustics, Speech and
Signal Processing, {ICASSP} 2013, Vancouver, BC, Canada, May 26-31, 2013},
pages 6645--6649. {IEEE}.
\bibitem[{Graves et~al.(2014)Graves, Wayne, and Danihelka}]{graves2014neural}
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014.
\newblock \href {http://arxiv.org/abs/1410.5401} {Neural turing machines}.
\newblock \emph{CoRR}, abs/1410.5401.
\bibitem[{Hettiarachchi et~al.(2005)Hettiarachchi, Hall, and
Minns}]{hettiarachchi2005extrapolation}
P~Hettiarachchi, MJ~Hall, and AW~Minns. 2005.
\newblock The extrapolation of artificial neural networks for the modelling of
rainfall—runoff relationships.
\newblock \emph{Journal of Hydroinformatics}, 7(4):291--296.
\bibitem[{Hooker(2004)}]{hooker2004diagnostics}
Giles Hooker. 2004.
\newblock \emph{Diagnostics and extrapolation in machine learning}.
\newblock stanford university.
\bibitem[{Hupkes et~al.(2018)Hupkes, Singh, Korrel, Kruszewski, and
Bruni}]{hupkes2018learning}
Dieuwke Hupkes, Anand Singh, Kris Korrel, Germ{\'{a}}n Kruszewski, and Elia
Bruni. 2018.
\newblock \href {http://arxiv.org/abs/1805.09657} {Learning compositionally
through attentive guidance}.
\newblock \emph{CoRR}, abs/1805.09657.
\bibitem[{Jang et~al.(2017)Jang, Gu, and Poole}]{jang2016categorical}
Eric Jang, Shixiang Gu, and Ben Poole. 2017.
\newblock \href {https://openreview.net/forum?id=rkE3y85ee} {Categorical
reparameterization with gumbel-softmax}.
\newblock In \emph{5th International Conference on Learning Representations,
{ICLR} 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings}. OpenReview.net.
\bibitem[{Jo and Bengio(2017)}]{jo2017measuring}
Jason Jo and Yoshua Bengio. 2017.
\newblock \href {http://arxiv.org/abs/1711.11561} {Measuring the tendency of
cnns to learn surface statistical regularities}.
\newblock \emph{CoRR}, abs/1711.11561.
\bibitem[{Kingma and Ba(2015)}]{kingma2014adam}
Diederik~P. Kingma and Jimmy Ba. 2015.
\newblock \href {http://arxiv.org/abs/1412.6980} {Adam: {A} method for
stochastic optimization}.
\newblock In \emph{3rd International Conference on Learning Representations,
{ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings}.
\bibitem[{Lake and Baroni(2017)}]{baroni2018still}
Brenden~M. Lake and Marco Baroni. 2017.
\newblock \href {http://arxiv.org/abs/1711.00350} {Still not systematic after
all these years: On the compositional skills of sequence-to-sequence
recurrent networks}.
\newblock \emph{CoRR}, abs/1711.00350.
\bibitem[{Lake and Baroni(2018)}]{lake2017generalization}
Brenden~M. Lake and Marco Baroni. 2018.
\newblock \href {http://proceedings.mlr.press/v80/lake18a.html} {Generalization
without systematicity: On the compositional skills of sequence-to-sequence
recurrent networks}.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning, {ICML} 2018, Stockholmsm{\"{a}}ssan, Stockholm, Sweden, July 10-15,
2018}, volume~80 of \emph{Proceedings of Machine Learning Research}, pages
2879--2888. {PMLR}.
\bibitem[{Liska et~al.(2018)Liska, Kruszewski, and Baroni}]{liska2018memorize}
Adam Liska, Germ{\'{a}}n Kruszewski, and Marco Baroni. 2018.
\newblock \href {http://arxiv.org/abs/1802.06467} {Memorize or generalize?
searching for a compositional {RNN} in a haystack}.
\newblock \emph{CoRR}, abs/1802.06467.
\bibitem[{Lohninger(1999)}]{lohninger1999teach}
Hans Lohninger. 1999.
\newblock \emph{Teach/me: Data Analysis}.
\newblock Springer.
\bibitem[{Luong et~al.(2015)Luong, Pham, and Manning}]{luong2015effective}
Thang Luong, Hieu Pham, and Christopher~D. Manning. 2015.
\newblock \href {https://doi.org/10.18653/v1/d15-1166} {Effective approaches to
attention-based neural machine translation}.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing, {EMNLP} 2015, Lisbon, Portugal, September 17-21,
2015}, pages 1412--1421. The Association for Computational Linguistics.
\bibitem[{Maddison et~al.(2017)Maddison, Mnih, and Teh}]{maddison2016concrete}
Chris~J. Maddison, Andriy Mnih, and Yee~Whye Teh. 2017.
\newblock \href {https://openreview.net/forum?id=S1jE5L5gl} {The concrete
distribution: {A} continuous relaxation of discrete random variables}.
\newblock In \emph{5th International Conference on Learning Representations,
{ICLR} 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings}. OpenReview.net.
\bibitem[{Marcus(1998)}]{marcus1998rethinking}
Gary~F Marcus. 1998.
\newblock Rethinking eliminative connectionism.
\newblock \emph{Cognitive psychology}, 37(3):243--282.
\bibitem[{Mitchell et~al.(2018)Mitchell, Minervini, Stenetorp, and
Riedel}]{mitchell2018extrapolation}
Jeff Mitchell, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel.
2018.
\newblock \href {http://arxiv.org/abs/1805.06648} {Extrapolation in {NLP}}.
\newblock \emph{CoRR}, abs/1805.06648.
\bibitem[{Mnih et~al.(2014)Mnih, Heess, Graves, and
Kavukcuoglu}]{mnih2014recurrent}
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014.
\newblock \href
{http://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention}
{Recurrent models of visual attention}.
\newblock In \emph{Advances in Neural Information Processing Systems 27: Annual
Conference on Neural Information Processing Systems 2014, December 8-13 2014,
Montreal, Quebec, Canada}, pages 2204--2212.
\bibitem[{Santoro et~al.(2018)Santoro, Hill, Barrett, Morcos, and
Lillicrap}]{barrett2018measuring}
Adam Santoro, Felix Hill, David G.~T. Barrett, Ari~S. Morcos, and Timothy~P.
Lillicrap. 2018.
\newblock \href {http://proceedings.mlr.press/v80/santoro18a.html} {Measuring
abstract reasoning in neural networks}.
\newblock In \emph{Proceedings of the 35th International Conference on Machine
Learning, {ICML} 2018, Stockholmsm{\"{a}}ssan, Stockholm, Sweden, July 10-15,
2018}, volume~80 of \emph{Proceedings of Machine Learning Research}, pages
4477--4486. {PMLR}.
\bibitem[{Shaw et~al.(2018)Shaw, Uszkoreit, and Vaswani}]{shaw2018self}
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.
\newblock \href {https://doi.org/10.18653/v1/n18-2074} {Self-attention with
relative position representations}.
\newblock In \emph{Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume
2 (Short Papers)}, pages 464--468. Association for Computational Linguistics.
\bibitem[{Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever, and
Salakhutdinov}]{srivastava2014dropout}
Nitish Srivastava, Geoffrey~E. Hinton, Alex Krizhevsky, Ilya Sutskever, and
Ruslan Salakhutdinov. 2014.
\newblock \href {http://dl.acm.org/citation.cfm?id=2670313} {Dropout: a simple
way to prevent neural networks from overfitting}.
\newblock \emph{J. Mach. Learn. Res.}, 15(1):1929--1958.
\bibitem[{Sukhbaatar et~al.(2019)Sukhbaatar, Grave, Bojanowski, and
Joulin}]{sukhbaatar2019adaptive}
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019.
\newblock \href {https://doi.org/10.18653/v1/p19-1032} {Adaptive attention span
in transformers}.
\newblock In \emph{Proceedings of the 57th Conference of the Association for
Computational Linguistics, {ACL} 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers}, pages 331--335. Association for Computational
Linguistics.
\bibitem[{Sukhbaatar et~al.(2015)Sukhbaatar, Szlam, Weston, and
Fergus}]{sukhbaatar2015end}
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015.
\newblock \href {http://papers.nips.cc/paper/5846-end-to-end-memory-networks}
{End-to-end memory networks}.
\newblock In \emph{Advances in Neural Information Processing Systems 28: Annual
Conference on Neural Information Processing Systems 2015, December 7-12,
2015, Montreal, Quebec, Canada}, pages 2440--2448.
\bibitem[{Sutskever et~al.(2014)Sutskever, Vinyals, and
Le}]{sutskever2014sequence}
Ilya Sutskever, Oriol Vinyals, and Quoc~V. Le. 2014.
\newblock \href
{http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks}
{Sequence to sequence learning with neural networks}.
\newblock In \emph{Advances in Neural Information Processing Systems 27: Annual
Conference on Neural Information Processing Systems 2014, December 8-13 2014,
Montreal, Quebec, Canada}, pages 3104--3112.
\bibitem[{Szabó(2017)}]{sep-compositionality}
Zoltán~Gendler Szabó. 2017.
\newblock Compositionality.
\newblock In Edward~N. Zalta, editor, \emph{The Stanford Encyclopedia of
Philosophy}, summer 2017 edition. Metaphysics Research Lab, Stanford
University.
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.
\newblock \href {http://papers.nips.cc/paper/7181-attention-is-all-you-need}
{Attention is all you need}.
\newblock In \emph{Advances in Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems 2017, 4-9 December 2017,
Long Beach, CA, {USA}}, pages 5998--6008.
\bibitem[{Weber et~al.(2018)Weber, Shekhar, and
Balasubramanian}]{weber2018fine}
Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-1004} {The fine line between
linguistic generalization and failure in {S}eq2{S}eq-attention models}.
\newblock In \emph{Proceedings of the Workshop on Generalization in the Age of
Deep Learning}, pages 24--27, New Orleans, Louisiana. Association for
Computational Linguistics.
\bibitem[{Weiss et~al.(2018)Weiss, Goldberg, and Yahav}]{weiss2018practical}
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018.
\newblock \href {https://doi.org/10.18653/v1/P18-2117} {On the practical
computational power of finite precision rnns for language recognition}.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics, {ACL} 2018, Melbourne, Australia, July 15-20,
2018, Volume 2: Short Papers}, pages 740--745. Association for Computational
Linguistics.
\bibitem[{Yang et~al.(2018)Yang, Tu, Wong, Meng, Chao, and
Zhang}]{yang2018modeling}
Baosong Yang, Zhaopeng Tu, Derek~F. Wong, Fandong Meng, Lidia~S. Chao, and Tong
Zhang. 2018.
\newblock \href {https://doi.org/10.18653/v1/d18-1475} {Modeling localness for
self-attention networks}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, Brussels, Belgium, October 31 - November 4,
2018}, pages 4449--4458. Association for Computational Linguistics.
\bibitem[{Zhang et~al.(2017)Zhang, Bengio, Hardt, Recht, and
Vinyals}]{zhang2016understanding}
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
2017.
\newblock \href {https://openreview.net/forum?id=Sy8gdB9xx} {Understanding deep
learning requires rethinking generalization}.
\end{thebibliography}