DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence
Extracted source text ยท arXiv 2110.04821
%
% File acl2021.tex
%
%% Based on the style files for EMNLP 2020, which were
%% Based on the style files for ACL 2020, which were
%% Based on the style files for ACL 2018, NAACL 2018/19, which were
%% Based on the style files for ACL-2015, with some improvements
%% taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith
% \documentclass[11pt,a4paper]{article}
\documentclass[letterpaper]{article}
\usepackage[hyperref]{acl2021}
\usepackage{times}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{CJKutf8}
\usepackage{multirow}
\usepackage{titlesec}
\usepackage{caption}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{times}
\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}
\frenchspacing
\setlength{\pdfpagewidth}{8.5in}
\setlength{\pdfpageheight}{11in}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
\special{papersize=210mm,297m}
\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} % Enter the acl Paper ID here
%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.
\newcommand\BibTeX{B\textsc{ib}\TeX}
\title{DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence}
\author{Kai-Po Chang \\
National Tsing Hua University \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
\texttt{irving870810@gapp.nthu.edu.tw} \\\And
Wei-Yun Ma \\
Academia Sinica \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
\texttt{ma@iis.sinica.edu.tw} \\}
\date{}
\begin{document}
\maketitle
\begin{abstract}
In this paper, we propose Dynamic Compressive Transformer (DCT), a transformer-based framework for modeling the unbounded sequence. In contrast to the previous baselines which append every sentence representation to memory, conditionally selecting and appending them is a more reasonable solution to deal with unlimited long sequences. Our model uses a policy that determines whether the sequence should be kept in memory with a compressed state or discarded during the training process. With the benefits of retaining semantically meaningful sentence information in the memory system, our experiment results on Enwik8 benchmark show that DCT outperforms the previous state-of-the-art (SOTA) model.
\end{abstract}
\section{Introduction}
When studying a textbook with tens of thousands of words, humans used to selectively memorize the important relevant knowledge from what we have learned. With this habit, most of the trivial details of text are discarded by us automatically, and we only retain the most essential knowledge in our brain.
Just like humans, artificial neural network also has memory systems to record what it has learned before. Early models like Recurrent Neural Networks (RNN, \cite{hochreiter1997long}) such as Long Short-Term Memory (LSTM, \cite{rumelhart1986learning}) use various learning gates and cell states to emulate a memory \cite{staudemeyer2019understanding} to store information from earlier time steps. Although these models are good for processing sequential data, they are difficult to optimize due to the gradient vanishing and explosion problem on long-range context \cite{hochreiter2001gradient}. Furthermore, previous work has \cite{khandelwal2018sharp} found that LSTM could only take advantage of 200 prior tokens on average.
Afterwards, the transformer-based language model has achieved a wide range of impressive success \cite{devlin2018bert} by increasing the context length on language modeling \cite{dai2019transformer, rae2019compressive}, machine translation \cite{vaswani2017attention} and a large number of natural language processing (NLP) benchmarks \cite{devlin2018bert, yang2019xlnet}. The self-attention mechanism \cite{vaswani2017attention} contributes to the success of transformer, while the huge amount of parameters for integrating the hidden states in each layer makes itself limited to a fixed context length. With the increasing length of context cost bringing rising computational cost, the model has its limitation about how much information it can store in memory, which arises the long-range text modeling issue.
To address issues from modeling long-range text, a dizzying number of \emph{"X-former"} models have been proposed \cite{tay2020efficient}. The main ideas on these models are fixed patterns \cite{child2019generating, ho2019axial, qiu2019blockwise}, learnable patterns \cite{kitaev2020reformer, tay2020sparse, roy2021efficient}, low rank methods \cite{choromanski2020rethinking, wang2020linformer, katharopoulos2020transformers}, and recurrence \cite{dai2019transformer, rae2019compressive}. In addition, all these solutions are either making improvements around computational and memory complexity or caching past representations in memory. Instead, the dynamically selecting mechanism of the compression judger also prevents us from excessively cost on self-attention mechanism \cite{vaswani2017attention} by discarding semantically meaningless sentences, which is also an approach to reduce the computational and memory complexity. Although these models make the transformer more efficient, they still cannot deal with the infinite context.
In this paper, we propose DCT, a dynamic compressive transformer for unbounded sequence modeling. Our framework consists of two models, a compressive transformer \cite{rae2019compressive} and a deep reinforcement-learning (RL) based compression judger, which determines the past sentence representation should be either kept in the memory system or discarded. In DCT, the compression judger is in charge of selecting the semantically meaningful sentence representation and thus the transformer could optimize task loss with the maximum information. As opposed to transformerXL \cite{dai2019transformer} and compressive transformer \cite{rae2019compressive} which append all past sentence representations to their memory system, our memory system can be much helpful for optimizing task loss. Also, we conduct our experiment on a character-level language modeling benchmark Enwik8 taken from the Hutter Prize \cite{kriesi2012political}, to demonstrate that our framework applies to learning long-range sequence representation. In summary, our contributions can be summarized as follows:
\begin{itemize}
\item We propose DCT, Dynamic Compressive Transformer, that focuses on modeling sequence in unlimited long-range sequence.
\item We present a transformer-based framework with an RL method that allows models to learn a more informatic memory system. DCT would drop the memories which are considered less important.
\item We show that our method is a selective memorization process instead of keeping the latest sentence representation. This approach is applicable in searching the critical features in the dataset, and it could contribute to other NLP task where long context comprehension is needed.
\end{itemize}
\section{Related Work}
Many researches \cite{tay2020efficient} have attempted to capture the unbounded long-range context problem in language modeling. In recent years, improving transformer-based language models with memory mechanisms and reducing the computational requirements of attention mechanism are both of the most popular directions in particular. Previous studies about self-attention computational optimization include the sparse attention mechanism \cite{child2019generating, beltagy2020longformer, tay2020sparse, roy2021efficient} and self-attention time and memory complexity reduction models \cite{kitaev2020reformer, choromanski2020rethinking, peng2021random}. Although these models and techniques can reduce the computational requirements and memory cost, all memories need to be kept during training. And their performances on Enwik8 dataset do not exceed transformerXL \cite{dai2019transformer}.
On the other hand, leveraging a side memory module that caching past representations \cite{dai2019transformer, rae2019compressive} has been regarded as the most potent approach to address the computational problem on the unbounded input sequences. Since \citeauthor{dai2019transformer} (\citeyear{dai2019transformer}) proposed the TransformerXL that uses a novel relative position embedding and a segment-level recurrence mechanism to keep past information in memory, they reached SOTA results on multiple benchmarks including WikiText-103 \cite{merity2016pointer} and Enwik8 \cite{hutter2012human}. Following this, \citeauthor{rae2019compressive} (\citeyear{rae2019compressive}) proposed compressive transformer which incorporated the ideas of transformerXL \cite{rae2019compressive} to preserve prior sentences and furthermore improves its memory system through both short-term granular memory and longer-term coarse compressed memory. One problem of these approaches is they would eventually drop the oldest memories regardless of the importance of the memory in the current circumstance whenever the memory system is full. In comparison, our framework has a procedure to decide the oldest memory when the oldest sentence is beneficial to task performance. Recently, the infinity transformer \cite{martins2021infty} leverages continuous attention \cite{martins2020sparse} to read unbounded context. Although it is demonstrated its ability on their synthetic sorting task, it (24.41 ppl) only gains slight improvement on Wikitext-103 \cite{merity2016pointer} than the previous SOTA model (24.22 ppl) on their experiment. Finally, inspired by compressive transformer \cite{rae2019compressive}, DCT uses it as part of our framework to preserve both the short-term and long-term information.
% Compared to previous works that must eventually drop some useful but old sentence-level information in the memory system, our framework could choose not to discard the old sentences when they are beneficial to task performance.
\section{Models}
The task we cope with in this paper is language modeling whose metrics for evaluating are Perplexity (PPL) and Bits-Per-Character (BPC). Given a tokenized sequence $\mathbf{w} = \{w_0, w_1, ..., w_T\}$, and the perplexity of $\mathbf{w}$ is
\begin{equation}
\operatorname{PPL}(\mathbf{w})=\exp \left\{-\frac{1}{T} \sum_{i}^{T} \log p_{\theta}\left(w_{i} \mid w_{<i}\right)\right\} \label{eq:one}
\end{equation}
where $\log p_{\theta}\left(w_{i} \mid w_{<i}\right)$ is the log-likelihood of the i-th token conditioned on the preceding tokens $w_{<i}$ \cite{wolf-etal-2020-transformers}, while BPC is defined as the average cross-entropy (used with log base 2) \cite{graves2013generating}. A language model is a probability distribution over entire context, so these two metrics could be considered as an evaluation of the model's ability to predict next specified token according to the conditional probability (like in Eq.~\ref{eq:one} ).
We propose DCT, a transformer-based framework that determines whether to keep or discard the compacted past activations into a compressed memory. As illustrated in Fig.~\ref{fig:one}, DCT is composed of the compression judger and compressive transformer \cite{rae2019compressive}. When compressive transformer \cite{rae2019compressive} eliminates these oldest past activations from memory, the compression judger will make a binary decision on each input evicted memory to decide whether they should be appended to compressed memory or be discarded.
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{DCT.png}
\caption{DCT: Dynamic compressive transformer. The process in this diagram is executed per step. The above model has four layers with a sequence length $n_{s} = 128$, a memory size $n_{m}=128$, a compressed memory size $n_{cm}=256$, and a compression ratio of $c$. First, the yellow highlighted memories will be compressed through a compression function $f_{c}$ in each layer. Second, the compression judger would decide whether the entire layers should be kept in compressed memory or be discarded.}
\label{fig:one}
\end{figure*}
\subsection{Compressive Tranasformer}
Compressive transformer \cite{rae2019compressive} is a long-range sequence model which compresses and append all past activations into compressed memory. They applies a \emph{compression operation} $f_{c}: \mathbf{R}^{n_{s} \times d} \rightarrow \mathbf{R}^{\left[\frac{n_{s}}{c}\right\rfloor \times d}$ which is an 1D convolution layer for mapping the $n_s$ oldest memories to $\left\lfloor\frac{n_{s}}{c}\right\rfloor$ compressed memories \cite{rae2019compressive}. In addition, those compacted momories will be stored in secondary FIFO compressed memory. However, it still needs to discard these compressed memories when the compressed memory is full.
Therefore, we modify and improve its original operation to be \emph{selectively compression operation}. Besides compressing and appending all the eliminated memories to secondary compressed memory, we also apply a compression judger that determines whether the compressed memories are beneficial for the transformer to minimize the task loss depends on the content of compressed memory, memory, and sequence. In other words, the compression judger could select and append the meaningful compacted sentence representation to compressed memory, instead of preserving all text including meaningless sentences.
\subsection{Compression Judger}
We interpret the problem of how to identify the semantically important sentences based on reinforcement learning. The state \emph{s} is concatenated tensors composed of the last layer of compressed memory, memory, and sequence [($n_{cm}$ $+$ $n_{m}$ $+$ $n_{s}$) $\times$ dimension]. The action space \emph{a} is discrete in our framework and given as \{0 (\emph{Discard}), 1 (\emph{Keep})\}, which means to append or discard the compacted memories, respectively. Depends on state, the actor, parameterized by $\theta$, follows a policy $p_{\theta}(a \mid s)$ to sample an action from action set. Although the policy $p_{\theta}(a \mid s)$ is stochastic, the state transition is deterministic after an action has been choosen, i.e. $\delta_{s, s^{\prime}}^{a} = 1$. That is, the problem could be formulated as Markov Decision Process (MDP), which is well aligned with RL problem \cite{barto2017some}.
Therefore, we propose the RL-based compression judger which is composed of two components, actor and evaluator. The actor makes a binary decision for determining whether to append the evicted memories to compressed memory, and the evaluator is a freezing pre-trained language model to estimate the perplexity of the input sentence. The details are described as following:
\subsubsection{Actor}
The main architecture of the actor is a BiLSTM encoder \cite{schuster1997bidirectional}. To model the actor as a policy of selecting the evicted old memory more accurately, we treat the actor as an agent and adopt the policy gradient method \cite{sutton2000policy} to optimize the actor by maximizing the objective function $J(\theta)$:
\begin{equation}
J(\theta)=E_{\tau \sim p_{\theta}(\tau)}\left[R(\tau)\right]=\sum_{\tau} R(\tau) p_{\theta}(\tau)\label{eq:two}
\end{equation}
where $\tau$ is given as $\{s_1, a_1, r_1,..., s_n, a_n, r_n, ..., s_N, $ $a_N, r_N\}$ which is known as trajectory or episode, $\theta$ is the parameter of the actor, $p_{\theta}(\tau)$ represents the softmax probability of the sampled determination, and $R(\tau)$ will be discussed later. The trajectory begins when both the compressed memory and memory are full so that the tensor shape of state would be fixed. In timestep $\emph{n}$, the actor takes specific action $a_n$ (keep or discard) according to current $s_n$ (the last layer of memory system and input sequence), and then obtains the corresponding reward $r_n$.
The next question is how to estimate the total reward of trajectory $R(\tau)$. In this paper, we consider the perplexity from compressive transformer as the reward. However, the property of perplexity is the lower the better, which doesn't conform to the standard reward definition in RL problems like OpenAIGym \cite{brockman2016openai}, so we apply the perplexity (ppl) with the following function $f(.)$:
\begin{equation}
f(ppl)= m \cdot a^{ppl} \label{eq:three}
\end{equation}
where the base nubmer $a$ and slope $m$ are hyperparameters, and $a$ must be between $0$ and $1$. With this function, we could make the reward and the transformed perplexity with a positive correlated relation. Thus, we would make the reward the higher the better by redesigning it as follows if we sample a trajectory $\tau$ with $N$ state-action pairs.
\begin{equation}
R(\tau) =\frac{1}{N} \sum_{t=1}^{N} \left\{r_{t} = f\left(ppl_{t}\right) \label{eq:four} \right\}
\end{equation}
% To choose the optimal policy which maximizing the objective function $J(\theta)$, we apply the REINFORCE \cite{williams1992simple} algorithm, which relies on Monte-Carlo based policy gradient methods; the formula of updating parameters of the actor (denoted as $\theta$) is as follows. $A^{\theta}\left(s_{t}, a_{t}\right)$ stands for the estimated return, and we only calculate return after one batch is finished. And the state-action pairs are collected by mini-batch gradient ascent in implementation (Details in Section~\ref{sec:three_three}).
% \begin{equation}
% \nabla_{\theta} J(\theta)=\sum_{t=1}^{T} A^{\theta}\left(s_{t}, a_{t}\right)\nabla \log p_{\theta}\left(a_{t} \mid s_{t}\right) \label{eq:six}
% \end{equation}
% \begin{equation}
% \theta \leftarrow \theta+\eta \nabla_{\theta} J(\theta) \label{eq:seven}
% \end{equation}
\subsubsection{Evaluator}
The evaluator is an arbitrary freezing pre-trained language model. Due to the high variance in evaluating the perplexity between different sequences when using REINFORCE algorithm \cite{williams1992simple}, we need to set a baseline $b$ to mitigate this problem to maximize the reward $R(\tau)$.
% \begin{equation}
% \begin{split}
% J(\theta)&=\sum_{\tau} \left( R(\tau) - \beta \right) p_{\theta}\left(a_{t} \mid s_{t}\right) \\
% &= \sum_{t=1}^{n} \left( f\left(pred\mbox{-}ppl_{t}\right) - f\left(eval\mbox{-}ppl_{t}\right) \right) p_{\theta}\left(a_{t} \mid s_{t}\right)
% \label{eq:seven}
% \end{split}
% \end{equation}
Once a trajectory has been processed by the DCT, we are ready to update the actor. In fact, each state-action pair in a trajectory is collected through one batch, and the details will be discussed in Section~\ref{sec:three_three}. Following \citeauthor{sutton2000policy} (\citeyear{sutton2000policy}), to maximize the total reward of trajectory $R(\tau)$, the gradient of the objective can be derived as:
\begin{equation}
\begin{multlined}
\nabla_{\theta} J(\theta) = E_{\tau \sim p_{\theta}(\tau)}\Big[ (R(\tau)-b+\alpha \cdot S[p_{\theta}]) \\
\nabla \log p_{\theta}\left(s_{t}, a_{t}\right)\Big] \label{eq:five}
\end{multlined}
\end{equation}
where $b$ is also an averaged transformed perplexity score from a freezing pre-trained language model in one trajectory like $R\left(\tau\right)$, $\alpha$ is a coffeicent and $S[p_{\theta}]$ denotes an entropy of the action probability from actor.
Since it is not possible to enumerate all possibilities of the real samples, we apply the REINFORCE algorithm \cite{williams1992simple} which relies on Monte-Carlo based policy gradient methods; it works because the expectation of the sample gradient as follows is approximated to the actual gradient in Eq.~\ref{eq:five}:
\begin{equation}
\begin{multlined}
\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{t=1}^{N} \Big[(r_{t}-b+\alpha \cdot S[p_{\theta}]) \\ \nabla \log p_{\theta}\left(s_{t}, a_{t}\right)\Big] \label{eq:six}
\end{multlined}
\end{equation}
The above formula is the process during optimization. We use the gradient ascent to find the best actor $\theta$ that produces the highest reward. Then we update the actor's parameters with $\eta$ denotes the learning rate as:
\begin{equation}
\theta \leftarrow \theta+\eta \nabla_{\theta} J(\theta) \label{eq:seven}
\end{equation}
\subsection{Training Description and Details} \label{sec:three_three}
To train the proposed DCT, we need to optimize the two components, compressive transformer \cite{rae2019compressive} and compression judger, simultaneously. We propose the following training procedures during each batch. In one batch, we divide it into mini-batches and go through them one by one. In this way, we can append past hidden states in the unit of minibatch, reduce the computational cost, and synchronize the updating frequency of compressive transformer \cite{rae2019compressive} and compression judger (see Figure ~\ref{fig:one}).
\begin{enumerate}
\item \textbf{DCT training :}
We first pretrain the compressive transformer \cite{rae2019compressive} for nearly one epoch, and then combine it with the training of the compression judger. At beginning of each step, we split the batch of samples into several smaller mini-batches of samples; then train both of the Compressive transformer \footnote{We use and modify the author's implementation available at https://github.com/lucidrains/compressive-transformer-pytorch} \cite{rae2019compressive} and freezing evaluator in compression judger with the same mini-batch as input sample at the same time.
\item \textbf{Compression judger prediction :}
The policy (agent behavior strategy) $p_{\theta}\left(a \mid s\right)$ is predicted by the actor in compression judger: $$p_{\theta}\left(a \mid s\right) = \mbox{softmax}( \mbox{FC}( \mbox{BiLSTM}(s))) \label{eq:eight}$$ where $s$ is the concatenated sentence representations of compressed memory, memory, and sequence. The reason to use BiLSTM encoder \cite{schuster1997bidirectional} is that \citeauthor{parisotto2020stabilizing} (\citeyear{parisotto2020stabilizing}) demonstrated the effectiveness of LSTM \cite{rumelhart1986learning} agent in RL problem outperform transformer-XL \cite{dai2019transformer} agent.
In compression judger, the actor is in charge of predicting an action probability distribution $p_{\theta}\left(a \mid s\right)$ after the compressed memory and memory are filled with the pasted hidden states. And we wouldn't discard the past activations unless the memory system is full.
\item \textbf{Compression judger decision :}
Depends on the policy $p_{\theta}\left(a \mid s\right)$ of this binary decision in Eq.~\ref{eq:eight}, the compression judger sample an action from the action probability to determine evicted memories would be discarded or appended to compressed memory in a compressed state.
\item \textbf{Reward calculation :}
After sequentially passing each mini-batch to both of the compressive transformer \cite{rae2019compressive} and evaluator, we get two lists that contain the perplexity scores from each individual models. In order to decrease the high variance of sentences, we do more than using per scores from transformer as reward. Instead, we decrease those by a baseline $b$, the perplexity scores from evaluator, to make the final return values.
$$b =\mbox{cross-entropy}(\mathrm{logits}, \mathrm{target})$$
$$\small{\mathrm{logits} = \mbox{FC}(\mbox{Freezing-GPT2}(\mathrm{input}\textendash \mathrm{smaples}))}$$
where input-samples are the mini-batch of samples, and target is the shifted mini-batch with one token difference. In our implementation, the freezing pre-trained language model is HuggingFace's pre-trained OpenAI-GPT2\footnote{https://github.com/huggingface/transformers} \cite{radford2019language}.
\item \textbf{Policy gradient :}
A trajectory begins after the memory system arefull, and it will be finished when one step is finished. The actor will optimize its parameters $\theta$ through policy gradient \cite{sutton2000policy} to maximize the reward $r_{t}$ in Eq.~\ref{eq:six}.
\end{enumerate}
\section{Experiments}
\subsection{Dataset}
Our method is evaluated on Enwik8 dataset, a standard character-level language modeling benchmark taken from Hutter Prize \cite{hutter2012human}, which contains 100M bytes of unprocessed Wikipedia text. The first 90MB is the training dataset, the following 5MB is the validation dataset, and the last 5MB is the testing dataset.
\subsection{Implementation Settings and Details}
We set the batch size to 32 and sequence length to 128. For the memory system, the compressed memory length is 64, memory length and sequence length are 128 with a compression ratio of 4. As for the learning rate, we use fixed $1\times 10^{-4}$ when pretraining compressive transformer, and then co-training with compression judger with fixed $1\times 10^{-5}$ on both models. Also, we train our baseline model, compressive transformer \cite{rae2019compressive}, with the same hyperparameters.
% % At beginning of training, the memory system are not full unless the fourth mini-batch is passed to model. To make the state transition stable and maintain its tensor shape, the actor need choose to append the evicted memories to compressed memory.
% A trajectory begins after the memory system are full. State $s_t$, action $a_t$, reward $r_t$ at time step $t$ of one trajectory are generated through the actor choosing an action (keep or discard the past activation) based on the state (current memory system and input sequence) and the transformed BPC (reward) is obtained after the next sequence pass to model. The trajectory will be finished when one step is finished, and the actor will optimize its parameters through REINFORCE \cite{williams1992simple} algorithm with the samples collected at every transition step.
\subsection{Results}
The testing results of DCT are shown in Table~\ref{table:eval}. As we can see, Ours-8L DCT outperforms other models in BPC on Wiki8 dataset \cite{hutter2012human}. The results show that increasing the reading length by the revised memory mechanism leads to better perplexity.
\begin{table}[h]
\setlength{\arrayrulewidth}{0.5mm}
% \setlength{\tabcolsep}{18pt}
\renewcommand{\arraystretch}{1.2}
\begin{center}
\begin{tabular}{p{6cm}|c}
\hline
Models & BPC \\
\hline \hline
7L LSTM \cite{graves2013generating} & 1.67 \\
12L Transformer \cite{al2019character} & 1.11\\
12L Transformer-XL \cite{dai2019transformer} & 1.06\\
24L Transformer-XL \cite{dai2019transformer} & 0.99\\
24L Compressive Transformer \cite{rae2019compressive} & 0.97\\
\hline
Ours-8L Compressive Transformer & 0.978\\
Ours-8L DCT & \bf{0.963}\\
\hline
\end{tabular}
\end{center}
\caption{State-of-the-art testing results on Enwik8.}
\label{table:eval}
\end{table}
We observe that DCT is able to outperform the compressive transformer \cite{rae2019compressive} in EnWik8. However, we only trained our model for at most 10 days on a single GPU in small sequence length and small-batch settings. It would be advantageous to train our DCT with the large batch size and sequence length for its memory selecting mechanism.
\subsection{Analysis}
To know whether the compression judger contributes to the reduction in complexity, we show the Figure.~\ref{fig:longest} which is the results of the longest length read by DCT for each batch. The definition of longest length is the most distant tokens of all hidden vectors corresponding to the corpus in the memory system. The shorter the reading distance of the model in a certain range, the more the model chooses to keep the evicted memories in compressed memory. In Figure.~\ref{fig:longest}, we observe that the reading distance of DCT is shorter than average between the 2500-th and 3500-th step, which means the compression judger tends to compress the evicted memories in this range instead of discarding them. This represents the compression judger regards this range of text as more highly significant information for optimizing the task loss, which demonstrates the ability of DCT to identify the critical information in a corpus.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{train_longest.png}
\caption{The reading distance for each batch.}
\label{fig:longest}
\end{figure}
Inspired by SeqGAN \cite{yu2017seqgan}, we first pretrain the compressive transformer to generate the more semantically sentence embedding, and then co-train both of the compressive transformer and compression agent to reduce the BPC. Besides, the entropy bonus $S[p_{\theta}]$ could also help training because it avoids the overly lopsided view of compression judger that everything should be discarded or kept forever. These two training skills help us construct a better language modeling method, the RL-based co-training framework of the compressive transformer and compression judger.
\section{Conclusion}
In this paper, we propose the DCT, a framework extended from compressive transformer. By making use of an RL-based compression judger which is composed of actor and evaluator, the DCT could capture the further long context and even the unbounded sequence with computation budget. Instead of keeping all information in the memory system, the actor in DCT determines whether to append the past activations or discard them based on the current memory system and input samples. Namely, the actor judge whether the past activations are beneficial to our language modeling task in the attention calculation. Our experiments show significant improvement in BPC compared with previous SOTA models on Enwik8 dataset.
\section{Future Work}
Inspired by SimCSE \cite{gao2021simcse}, our next step would be pretraining compressive transformer in DCT on Natural language inference (NLI) dataset \cite{bowman2015large} with contrastive learning objective. Thus, the actor in DCT could base on the more semantically meaningful sentence representations to judge the importance of the past activation to the language modeling task. Another future work is to apply our approach to other NLP tasks such as multiple-document summarization or medical information problem tasks based on long DNA sequences.
\bibliographystyle{acl_natbib}
\bibliography{anthology,acl2021}
%\appendix
\end{document}
\begin{thebibliography}{40}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Al-Rfou et~al.(2019)Al-Rfou, Choe, Constant, Guo, and
Jones}]{al2019character}
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019.
\newblock Character-level language modeling with deeper self-attention.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~33, pages 3159--3166.
\bibitem[{Barto et~al.(2017)Barto, Thomas, and Sutton}]{barto2017some}
Andrew~G Barto, Philip~S Thomas, and Richard~S Sutton. 2017.
\newblock Some recent applications of reinforcement learning.
\newblock In \emph{Proceedings of the Eighteenth Yale Workshop on Adaptive and
Learning Systems}.
\bibitem[{Beltagy et~al.(2020)Beltagy, Peters, and
Cohan}]{beltagy2020longformer}
Iz~Beltagy, Matthew~E Peters, and Arman Cohan. 2020.
\newblock Longformer: The long-document transformer.
\newblock \emph{arXiv preprint arXiv:2004.05150}.
\bibitem[{Bowman et~al.(2015)Bowman, Angeli, Potts, and
Manning}]{bowman2015large}
Samuel~R Bowman, Gabor Angeli, Christopher Potts, and Christopher~D Manning.
2015.
\newblock A large annotated corpus for learning natural language inference.
\newblock \emph{arXiv preprint arXiv:1508.05326}.
\bibitem[{Brockman et~al.(2016)Brockman, Cheung, Pettersson, Schneider,
Schulman, Tang, and Zaremba}]{brockman2016openai}
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba. 2016.
\newblock Openai gym.
\newblock \emph{arXiv preprint arXiv:1606.01540}.
\bibitem[{Child et~al.(2019)Child, Gray, Radford, and
Sutskever}]{child2019generating}
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019.
\newblock Generating long sequences with sparse transformers.
\newblock \emph{arXiv preprint arXiv:1904.10509}.
\bibitem[{Choromanski et~al.(2020)Choromanski, Likhosherstov, Dohan, Song,
Gane, Sarlos, Hawkins, Davis, Mohiuddin, Kaiser
et~al.}]{choromanski2020rethinking}
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song,
Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin,
Lukasz Kaiser, et~al. 2020.
\newblock Rethinking attention with performers.
\newblock \emph{arXiv preprint arXiv:2009.14794}.
\bibitem[{Dai et~al.(2019)Dai, Yang, Yang, Carbonell, Le, and
Salakhutdinov}]{dai2019transformer}
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc~V Le, and Ruslan
Salakhutdinov. 2019.
\newblock Transformer-xl: Attentive language models beyond a fixed-length
context.
\newblock \emph{arXiv preprint arXiv:1901.02860}.
\bibitem[{Devlin et~al.(2018)Devlin, Chang, Lee, and
Toutanova}]{devlin2018bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.
\newblock Bert: Pre-training of deep bidirectional transformers for language
understanding.
\newblock \emph{arXiv preprint arXiv:1810.04805}.
\bibitem[{Gao et~al.(2021)Gao, Yao, and Chen}]{gao2021simcse}
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
\newblock Simcse: Simple contrastive learning of sentence embeddings.
\newblock \emph{arXiv preprint arXiv:2104.08821}.
\bibitem[{Graves(2013)}]{graves2013generating}
Alex Graves. 2013.
\newblock Generating sequences with recurrent neural networks.
\newblock \emph{arXiv preprint arXiv:1308.0850}.
\bibitem[{Ho et~al.(2019)Ho, Kalchbrenner, Weissenborn, and
Salimans}]{ho2019axial}
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019.
\newblock Axial attention in multidimensional transformers.
\newblock \emph{arXiv preprint arXiv:1912.12180}.
\bibitem[{Hochreiter et~al.(2001)Hochreiter, Bengio, Frasconi, Schmidhuber
et~al.}]{hochreiter2001gradient}
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, J{\"u}rgen Schmidhuber, et~al.
2001.
\newblock Gradient flow in recurrent nets: the difficulty of learning long-term
dependencies.
\bibitem[{Hochreiter and Schmidhuber(1997)}]{hochreiter1997long}
Sepp Hochreiter and J{\"u}rgen Schmidhuber. 1997.
\newblock Long short-term memory.
\newblock \emph{Neural computation}, 9(8):1735--1780.
\bibitem[{Hutter(2012)}]{hutter2012human}
Marcus Hutter. 2012.
\newblock The human knowledge compression contest.
\newblock \emph{URL http://prize. hutter1. net}, 6.
\bibitem[{Katharopoulos et~al.(2020)Katharopoulos, Vyas, Pappas, and
Fleuret}]{katharopoulos2020transformers}
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran{\c{c}}ois
Fleuret. 2020.
\newblock Transformers are rnns: Fast autoregressive transformers with linear
attention.
\newblock In \emph{International Conference on Machine Learning}, pages
5156--5165. PMLR.
\bibitem[{Khandelwal et~al.(2018)Khandelwal, He, Qi, and
Jurafsky}]{khandelwal2018sharp}
Urvashi Khandelwal, He~He, Peng Qi, and Dan Jurafsky. 2018.
\newblock Sharp nearby, fuzzy far away: How neural language models use context.
\newblock \emph{arXiv preprint arXiv:1805.04623}.
\bibitem[{Kitaev et~al.(2020)Kitaev, Kaiser, and Levskaya}]{kitaev2020reformer}
Nikita Kitaev, {\L}ukasz Kaiser, and Anselm Levskaya. 2020.
\newblock Reformer: The efficient transformer.
\newblock \emph{arXiv preprint arXiv:2001.04451}.
\bibitem[{Kriesi et~al.(2012)Kriesi, Grande, Dolezal, Helbling, H{\"o}glinger,
Hutter, and W{\"u}est}]{kriesi2012political}
Hanspeter Kriesi, Edgar Grande, Martin Dolezal, Marc Helbling, Dominic
H{\"o}glinger, Swen Hutter, and Bruno W{\"u}est. 2012.
\newblock \emph{Political conflict in western Europe}.
\newblock Cambridge University Press.
\bibitem[{Martins et~al.(2020)Martins, Farinhas, Treviso, Niculae, Aguiar, and
Figueiredo}]{martins2020sparse}
Andr{\'e}~FT Martins, Ant{\'o}nio Farinhas, Marcos Treviso, Vlad Niculae,
Pedro~MQ Aguiar, and M{\'a}rio~AT Figueiredo. 2020.
\newblock Sparse and continuous attention mechanisms.
\newblock \emph{arXiv preprint arXiv:2006.07214}.
\bibitem[{Martins et~al.(2021)Martins, Marinho, and Martins}]{martins2021infty}
Pedro~Henrique Martins, Zita Marinho, and Andr{\'e}~FT Martins. 2021.
\newblock $infty$-former: Infinite memory transformer.
\newblock \emph{arXiv preprint arXiv:2109.00301}.
\bibitem[{Merity et~al.(2016)Merity, Xiong, Bradbury, and
Socher}]{merity2016pointer}
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.
\newblock Pointer sentinel mixture models.
\newblock \emph{arXiv preprint arXiv:1609.07843}.
\bibitem[{Parisotto et~al.(2020)Parisotto, Song, Rae, Pascanu, Gulcehre,
Jayakumar, Jaderberg, Kaufman, Clark, Noury
et~al.}]{parisotto2020stabilizing}
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre,
Siddhant Jayakumar, Max Jaderberg, Raphael~Lopez Kaufman, Aidan Clark, Seb
Noury, et~al. 2020.
\newblock Stabilizing transformers for reinforcement learning.
\newblock In \emph{International Conference on Machine Learning}, pages
7487--7498. PMLR.
\bibitem[{Peng et~al.(2021)Peng, Pappas, Yogatama, Schwartz, Smith, and
Kong}]{peng2021random}
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah~A Smith, and
Lingpeng Kong. 2021.
\newblock Random feature attention.
\newblock \emph{arXiv preprint arXiv:2103.02143}.
\bibitem[{Qiu et~al.(2019)Qiu, Ma, Levy, Yih, Wang, and
Tang}]{qiu2019blockwise}
Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang.
2019.
\newblock Blockwise self-attention for long document understanding.
\newblock \emph{arXiv preprint arXiv:1911.02972}.
\bibitem[{Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, Sutskever
et~al.}]{radford2019language}
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
Sutskever, et~al. 2019.
\newblock Language models are unsupervised multitask learners.
\newblock \emph{OpenAI blog}, 1(8):9.
\bibitem[{Rae et~al.(2019)Rae, Potapenko, Jayakumar, and
Lillicrap}]{rae2019compressive}
Jack~W Rae, Anna Potapenko, Siddhant~M Jayakumar, and Timothy~P Lillicrap.
2019.
\newblock Compressive transformers for long-range sequence modelling.
\newblock \emph{arXiv preprint arXiv:1911.05507}.
\bibitem[{Roy et~al.(2021)Roy, Saffar, Vaswani, and
Grangier}]{roy2021efficient}
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021.
\newblock Efficient content-based sparse attention with routing transformers.
\newblock \emph{Transactions of the Association for Computational Linguistics},
9:53--68.
\bibitem[{Rumelhart et~al.(1986)Rumelhart, Hinton, and
Williams}]{rumelhart1986learning}
David~E Rumelhart, Geoffrey~E Hinton, and Ronald~J Williams. 1986.
\newblock Learning representations by back-propagating errors.
\newblock \emph{nature}, 323(6088):533--536.
\bibitem[{Schuster and Paliwal(1997)}]{schuster1997bidirectional}
Mike Schuster and Kuldip~K Paliwal. 1997.
\newblock Bidirectional recurrent neural networks.
\newblock \emph{IEEE transactions on Signal Processing}, 45(11):2673--2681.
\bibitem[{Staudemeyer and Morris(2019)}]{staudemeyer2019understanding}
Ralf~C. Staudemeyer and Eric~Rothstein Morris. 2019.
\newblock \href {http://arxiv.org/abs/1909.09586} {Understanding {LSTM} - a
tutorial into long short-term memory recurrent neural networks}.
\newblock \emph{CoRR}, abs/1909.09586.
\bibitem[{Sutton et~al.(2000)Sutton, McAllester, Singh, and
Mansour}]{sutton2000policy}
Richard~S Sutton, David~A McAllester, Satinder~P Singh, and Yishay Mansour.
2000.
\newblock Policy gradient methods for reinforcement learning with function
approximation.
\newblock In \emph{Advances in neural information processing systems}, pages
1057--1063.
\bibitem[{Tay et~al.(2020{\natexlab{a}})Tay, Bahri, Yang, Metzler, and
Juan}]{tay2020sparse}
Yi~Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan.
2020{\natexlab{a}}.
\newblock Sparse sinkhorn attention.
\newblock In \emph{International Conference on Machine Learning}, pages
9438--9447. PMLR.
\bibitem[{Tay et~al.(2020{\natexlab{b}})Tay, Dehghani, Bahri, and
Metzler}]{tay2020efficient}
Yi~Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020{\natexlab{b}}.
\newblock Efficient transformers: A survey.
\newblock \emph{arXiv preprint arXiv:2009.06732}.
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin. 2017.
\newblock Attention is all you need.
\newblock In \emph{Advances in neural information processing systems}, pages
5998--6008.
\bibitem[{Wang et~al.(2020)Wang, Li, Khabsa, Fang, and Ma}]{wang2020linformer}
Sinong Wang, Belinda~Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.
\newblock Linformer: Self-attention with linear complexity.
\newblock \emph{arXiv preprint arXiv:2006.04768}.
\bibitem[{Williams(1992)}]{williams1992simple}
Ronald~J Williams. 1992.
\newblock Simple statistical gradient-following algorithms for connectionist
reinforcement learning.
\newblock \emph{Machine learning}, 8(3):229--256.
\bibitem[{Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu,
Le~Scao, Gugger, Drame, Lhoest, and Rush}]{wolf-etal-2020-transformers}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le~Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-demos.6} {Transformers:
State-of-the-art natural language processing}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 38--45, Online.
Association for Computational Linguistics.
\bibitem[{Yang et~al.(2019)Yang, Dai, Yang, Carbonell, Salakhutdinov, and
Le}]{yang2019xlnet}
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ~R Salakhutdinov,
and Quoc~V Le. 2019.
\newblock Xlnet: Generalized autoregressive pretraining for language
understanding.
\newblock \emph{Advances in neural information processing systems}, 32.
\bibitem[{Yu et~al.(2017)Yu, Zhang, Wang, and Yu}]{yu2017seqgan}
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017.
\newblock Seqgan: Sequence generative adversarial nets with policy gradient.
\newblock In \emph{Proceedings of the AAAI conference on artificial
intelligence}, volume~31.
\end{thebibliography}