Online Learning for Neural Machine Translation Post-editing
Extracted source text · arXiv 1706.03196
%\title{emnlp 2017 instructions}
% File emnlp2017.tex
%
\documentclass[11pt,letterpaper]{article}
\usepackage{emnlp2017}
\usepackage{times}
\usepackage{latexsym}
\usepackage[utf8]{inputenc}
\usepackage{amssymb,amsfonts,amsmath}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{enumitem}
%\usepackage{showframe}% http://ctan.org/pkg/showframe
\usepackage{float}
\usepackage{wrapfig}
\restylefloat{figure}
\usepackage{booktabs}
\usepackage[english]{babel}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{csquotes}
\usepackage{setspace}
\usepackage{array}
\usepackage{calc}
\usepackage{multicol}
\usepackage[subpreambles=true]{standalone}
\usepackage{pgfplots}
\usepackage{pgfkeys}
\usepgfplotslibrary{dateplot}
\pgfplotsset{compat=newest}
\pagestyle{empty}
\usetikzlibrary{matrix}
\usepgfplotslibrary{groupplots}
\usetikzlibrary{patterns}
\usepgfplotslibrary{external}
\pgfplotsset{
invoke before crossref tikzpicture={\tikzexternaldisable},
invoke after crossref tikzpicture={\tikzexternalenable},
}
\usepackage[capitalize]{cleveref}
% Uncomment this line for the final submission:
\emnlpfinalcopy
% Enter the EMNLP Paper ID here:
%\def\emnlppaperid{***}
% To expand the titlebox for more authors, uncomment
% below and set accordingly.
% \addtolength\titlebox{.5in}
\newcommand\BibTeX{B{\sc ib}\TeX}
\crefformat{appendix}{#2#1#3}
\DeclareMathOperator*{\argmin}{arg\,min~}
\DeclareMathOperator*{\argmax}{arg\,max~}
\newcommand{\x}{\mathbf{x}}
\newcommand{\y}{\mathbf{y}}
\newcommand{\h}{\mathbf{h}}
\newcommand{\VTheta}{\mathbf{\Theta}}
\newcommand{\Vlambda}{\mathbf{\lambda}}
\newcommand{\vm}{\mathbf{m}}
\newcommand{\vv}{\mathbf{v}}
\newcommand{\w}{\mathbf{w}}
\newcommand{\z}{\mathbf{z}}
\newcommand{\ve}{\mathbf{v}}
\newcommand{\me}{\mathbf{c}}
\newcommand{\out}{\mathbf{o}}
\newcommand{\inp}{\mathbf{i}}
\newcommand{\fo}{\mathbf{f}}
\newcommand{\bias}{\mathbf{b}}
\newcommand{\prob}{\mathbf{p}}
\newcommand{\g}{\mathbf{g}}
\newcommand{\f}{\mathbf{f}}
\newcommand{\A}{\mathbf{A}}
\newcommand{\B}{\mathbf{B}}
\newcommand{\comentario}[1]{\textcolor{red}{#1}}
\newcommand{\texto}[1]{\textcolor{blue}{#1}}
\newcommand{\comentarioalvaro}[1]{\textcolor{orange}{\'Alvaro: #1}}
\title{Online Learning for Neural\\ Machine Translation Post-editing}
% Author information can be set in various styles:
% For several authors from the same institution:
\author{\'{A}lvaro Peris \and Luis Cebri\'{a}n \and Francisco Casacuberta \\
Pattern Recognition and Human Language Technology Research Center \\ Universitat Politècnica de València, València, Spain\\
\texttt{\{lvapeab, fcn\}@prhlt.upv.es} \\
\texttt{luicebch@inf.upv.es } }
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
% If the title and author information does not fit in the area allocated,
% place \setlength\titlebox{<new height>} right after
% at the top, where <new height> can be something larger than 2.25in
\date{}
\begin{document}
\maketitle
\begin{abstract}
Neural machine translation has meant a revolution of the field. Nevertheless, post-editing the outputs of the system is mandatory for tasks requiring high translation quality.
Post-editing offers a unique opportunity for improving neural machine translation systems, using online learning techniques and treating the post-edited translations as new, fresh training data.
%We delve into the application of online learning in the neural machine translation post-editing process, aiming to reduce the required effort.
We review classical learning methods and propose a new optimization algorithm. We thoroughly compare online learning algorithms in a post-editing scenario. Results show significant improvements in translation quality and effort reduction.
\end{abstract}
\input{1_introduction.tex}
\input{2_related_work.tex}
\input{3_NMT.tex}
\input{4_PA.tex}
\input{5_exp_setup.tex}
\input{6_results.tex}
\input{7_conclusions.tex}
\section*{Acknowledgments}
The research leading to these results has received funding from
the Generalitat Valenciana under grant PROMETEOII/2014/030. We also acknowledge NVIDIA for the donation of a GPU used in this work.
\bibliography{emnlp2017}
\bibliographystyle{emnlp_natbib}
\end{document}
\section{Introduction} \label{sec:introducion}
In recent years, the statistical machine translation (SMT) field has evolved by leaps and bounds. Nevertheless, SMT systems still require human supervision in order to produce high-quality translations. This correction process is known as post-editing. In the machine translation (MT) industry, the reduction of the post-editing effort has a great interest, as it leads to larger productivity \citep{Arenas08,Plitt10}.
The text obtained during the post-editing process can be converted into new training data, useful for adapting the system to a different domain or to a changing environment.
Online learning (OL) techniques allow the system to learn from post-edited samples and, aiming to avoid the committed mistakes.
The application of OL to the classical phrase-based SMT systems has been thoroughly studied~\citep{Ortiz16}. Nevertheless, over the last years, the new neural machine translation (NMT) technology has shaken the machine translation field~\citep{Sutskever14,Bahdanau15,Wu16}. Therefore, there is a need to reduce the post-editing effort in NMT systems.
The neural approach to MT has a short but meteoric history. The first fully neural translation system was introduced by~\citet{Kalchbrenner13}, being the first competitive NMT systems simultaneously developed by~\citet{Cho14} and \citet{Sutskever14}. From there to now, NMT has surpassed the classical phrase-based systems in many tasks and currently is one the most active research topics (e.g. \citet{Erk16}). Furthermore, the translation industry is also shifting to this new translation paradigm~\citep{Crego16,Wu16}.
NMT relies on the encoder-decoder framework. An encoder recurrent neural network (RNN) processes the source sentence, computing a dense, continuous representation of it. From this representation, a decoder RNN generates, word by word, the translated sentence. In order to cope with the vanishing gradient problem~\citep{Bengio94}, such RNNs feature gated units, such as long short-term memory (LSTM) \citep{Hochreiter97} or gated recurrent units (GRU)~\citep{Cho14}.
The vanilla encoder-decoder model has been steadily improved. The inclusion of attention mechanisms~\citep{Bahdanau15,Luong15a} overcame the problem of processing long source sentences. The management of out-of-vocabulary words has also been explored~\citep{Jean15,Luong15b}. Closely related to this, the NMT vocabulary size limitation has been tackled either developing strategies for taking into account large vocabularies~\citep{Jean15} or shifting from word-level translations to character \citep{Chung16} or sub-word-level \citep{Sennrich16} translations.
Such NMT systems are usually trained from parallel corpora by means of stochastic gradient descent (SGD)~\citep{Robbins51}. SGD can be applied sample by sample, fitting into the OL paradigm.
In this paper, we evaluate the most common SGD optimizers. Moreover, we propose a new SGD variant for NMT, based on passive-aggressive (PA) techniques~\citep{Crammer06}. This method aims to apply the minimum modification to the model (passiveness) required to satisfy a correctness criterion (aggressiveness). The contributions of this work are the following:
\begin{itemize}[noitemsep]
\item We study the application of OL to the NMT framework. To the best of our knowledge, this is the first work that applies OL to NMT in a post-editing scenario.
\item We present a new OL algorithm, inspired by PA techniques and implemented using subgradient methods.
\item We conduct a wide experimentation, testing the NMT engine in three different scenarios. We compare well-established learning algorithms together with the newly proposed PA algorithms.
\item Results show that OL algorithms are able to significantly reduced the post-editing effort required by an NMT system. Moreover, we demonstrate the capability of NMT for adapting to different domains by means of OL techniques.
\item In order to make research reproducible, we make public the source code of the NMT system and the training algorithms\footnote{ \url{https://github.com/lvapeab/nmt-keras/tree/interactive_NMT}}.
\end{itemize}
The rest of the paper is structured as follows: after this introduction,~\cref{sec:related-work} reviews the related work. An NMT system is briefly described in~\cref{sec:nmt}. The classical SGD methods are revisited in~\cref{sec:online-sgd}.~\cref{sec:pa-subgrad} details the newly proposed online algorithm. \cref{sec:exp-setup} presents the experimental framework designed to assess our method. Results are shown and discussed in~\cref{sec:results}. Finally, we conclude in~\cref{sec:conclusions}, tracing future research lines.
\section{Related work}
\label{sec:related-work}
Online learning is a profusely studied topic in machine learning. In the field of SMT, OL techniques are mainly employed for tuning the weights of the log-linear model~\citep{Zens02,Och02} from classical phrase-based SMT systems~\citep{Koehn10}. More specifically, PA-based techniques, such as the margin infuse relaxed algorithm (MIRA)~\citep{Crammer01} are especially well-suited when dealing with a large number of features. Therefore, MIRA is usually applied to models dealing with lots of sparse features~\citep{Watanabe07,Chiang12}.
Estimating the model features---in addition to the weights of the log-linear model---is a less explored application of OL techniques. Most relating this, were developed under the umbrella of the CasMaCat project~\cite{Alabau13}. As colophon of it,~\citet{Ortiz16} developed an incremental derivation of the EM algorithm.
Another application of OL techniques in SMT is the adaptation of a trained system to a post-editing scenario. \citet{Martinez-Gomez12} and \citet{Mathur13} used OL for adapting a SMT under a CAT framework. These works are close to ours in the scenarios on which the MT systems are applied. Nevertheless, in our work, we employ an NMT system.
\section{Background: NMT}
\label{sec:nmt}
In this work, we use an attentional NMT system similar to the one described by~\citet{Bahdanau15}, but using LSTM networks. The input is a sequence of words $\x$ in the source language. Each word is linearly projected to a continuous space by means of an embedding matrix.
The sequence of word embeddings feeds a bidirectional LSTM, which analyzes the input sequence in both directions, from left to right and vice versa. This network computes a sequence of annotations by concatenating the hidden states from the forward and backward layers.
At each decoding timestep, an attention mechanism weights each element from the sequence of annotations, according to the previous decoding state, and computes a joint context vector.
The decoder is another LSTM network that takes into account the context vector, the previously generated word and its previous hidden state. Finally, a deep output layer~\citep{Pascanu14} is employed to compute a distribution over the target language vocabulary.
At decoding time, the model approximates the target sentence with a beam-search method~\citep{Sutskever14}.
\subsection{Training}
To estimate the model parameters $\VTheta$ (i.e. the weight matrices), the training objective is to minimize a loss function $L_\VTheta$, usually the minus log-likelihood over a bilingual parallel corpus ${\cal T} = \{(\x_t, \y_t)\}_{t=1}^T$, consisting of $T$ source--target sentence pairs ($\x_t$ and $\y_t$, respectively), with respect to $\VTheta$:
%
\begin{equation}
\begin{gathered}
\label{eq:training-obj}
\widehat{\VTheta} = \argmin_{\VTheta} L_\VTheta = \\
\argmin_{\VTheta} \sum_{t=1}^T \sum_{i=1}^{I_t}
-\log(p(\y_t \mid {\y_{t, <i}, \x_t}; \VTheta))
\end{gathered}
\end{equation}
%
where $I_t$ is the length of the $t$-th target sentence.
SGD techniques are the predominant way of optimizing neural networks. The goal is to minimize a given loss function $L_\VTheta$, parametrized by the network parameters.
%In the case of NMT, the loss is the conditional probability of target sentences given the source sentences (see~\cref{eq:training-obj}).}
SGD updates the parameters in the opposite direction of the gradient of $L_\VTheta$. The update size is controlled by a learning rate. According to when this updates occur, SGD techniques can be classified in batch, minibatch or online modes. Moreover, depending on how the gradient is computed and applied, a large variety of SGD variants have been proposed~\citep{Duchi11,Zeiler12,Kingma14}.
\section{Online gradient descent}
\label{sec:online-sgd}
Online SGD updates the model parameters after each sample. Therefore, for a single sample ($\x_t, \y_t$) the parameters update is computed as:
%
\begin{equation}
\label{eq:grad}
\Delta \VTheta_t = - \rho~ \nabla L_{\VTheta_t}(\x_t,\y_t)
\end{equation}
%
where $\rho$ is the learning rate that controls the step size and $\nabla L_{\VTheta_t}$ is the gradient of the objective function $L$ with respect to $\VTheta_t$. Most optimizers used in deep learning, are variants of this vanilla SGD. Therefore, they can also be applied under the OL framework.
In this work, we compared Adagrad, Adadelta and Adam, whose update rules are briefly commented below.
Adagrad~\citep{Duchi11} aims to perform larger updates for infrequent parameters, defining its update rule as:
\begin{displaymath}
\label{eq:grad}
\Delta \VTheta_t = - \frac{\rho}{\sqrt{\mathbf{G}_t + \epsilon}} \odot \nabla L_{\VTheta_t}(\x_t,\y_t)
\end{displaymath}
%
where $\mathbf{G}_t$ is a vector that contains the sum of the squares of the past gradients with respect to $\VTheta$, $\odot$ is the element-wise multiplication, the division is also applied element-wise and $\epsilon$ is a small value, introduced for numerical stability reasons.
Adadelta~\citep{Zeiler12} is a less aggressive variant of Adadelta, which avoids its constant learning rate decay. Adadelta updates the parameters according to the root mean square (RMS) of the gradients and corrects these updates according to the RMS of the previous update:
\begin{displaymath}
\label{eq:grad}
\Delta \VTheta_t = - \rho \frac{RMS(\Delta \VTheta_{t-1})}{RMS(\nabla L_{\VTheta_t}(\x_t,\y_t))}\nabla L_{\VTheta_t}(\x_t,\y_t)
\end{displaymath}
The default value for $\rho$ is 1. Nevertheless, in an OL scenario, this value must be lowered, in order to perform less aggressive updates (see~\cref{sec:results}).
Finally, Adam~\citep{Kingma14} computes decaying averages for the past gradients ($\vv_t$) and the past squared gradients ($\vm_t$). Such averages are computed following:
%
$$\vm_t = \beta_1 \vm_{t-1} + (1-\beta_1)\nabla L_{\VTheta_t}(\x_t,\y_t)$$
$$\vv_t = \beta_2 \vv_{t-1} + (1- \beta_2 \nabla L_{\VTheta_t}^2(\x_t,\y_t))$$
%
where $\beta_1$ and $\beta_2$ are two constant values.
The parameter update performed by Adam is:
\begin{displaymath}
\label{eq:grad}
\Delta \VTheta_t = - \frac{\rho}{\sqrt{\hat{\vv}_t} + \epsilon}\hat{\vm}_t
\end{displaymath}
%
where $\hat{\vv}_t = \frac{\vv_t}{1-\beta_2^t}$ and $\hat{\vm}_t = \frac{\vm_t}{1-\beta_1^t}$. The divisions and the square root are computed element-wise and $\beta^t$ denotes $\beta$ powered to $t$.
\section{Passive-aggressive via subgradient techniques}
\label{sec:pa-subgrad}
We propose a new version of SGD, inspired by PA techniques such as MIRA~\citep{Crammer01}. The method aims to perform the minimum modification to the model parameters for compelling with a correctness criterion. Our criterion is directly based on the model loss function (\cref{eq:training-obj}). Nevertheless, we could employ other loss functions, such as BLEU, although this could be costly. We declined towards this loss due to efficiency reasons.
Let %$\x_t$ be a source sentence,
$\h_t$ be the hypothesis generated by the NMT (using the current parameters $\VTheta_t$) % and $\y_t$ a post-edited target sentence
of the source sentence $\x_t$. We consider that $\VTheta_t$ is incorrect if the model assigns a lower probability to the target reference sentence $\y_t$ than to $\h_t$:
\begin{equation}
p_{\VTheta_t}(\y_t\mid\x_t) < p_{\VTheta_t}(\h_t\mid\x_t)
\end{equation}
In this case, we want to search for a $\widehat\VTheta$ such that $\widehat\VTheta$ is close to $\VTheta_t$ and $p_{\widehat\VTheta}(\y_t\mid\x_t) > p_{\widehat\VTheta}(\h_t\mid\x_t)$. This can be expressed with the loss function $\ell$:
%
\begin{equation}
\begin{gathered}
\ell(\widehat\VTheta,\x_t,\y_t,\h_t) = \\
\log p_{\widehat\VTheta}(\h_t\mid\x_t)-\log p_{\widehat\VTheta}(\y_t\mid\x_t) \leq \xi
\end{gathered}
\end{equation}
%
being $\xi \geq 0$ a slack variable, included for providing more flexibility to the method.
This can be formulated as a minimization problem with constraints:
%
\begin{equation}
\begin{gathered}
\label{eq:optimizaton-subgrad}
\widehat\VTheta=\argmin_{\VTheta} \frac{1}{2} \|\VTheta-\VTheta_t\|^2+C~\xi \\
\mbox{s.t.}~~\ell(\VTheta,\x_t,\y_t,\h_t) \leq \xi~~\mbox{and}~~\xi\geq 0
\end{gathered}
\end{equation}
%
where $C$ is a parameter that controls the aggressiveness of the algorithm \citep{Crammer06}.
From~\cref{eq:optimizaton-subgrad}, $\displaystyle\xi\geq\max(0,\ell(\VTheta,\x_t,\y_t,\h_t))$. We define $F_t$ as the function to optimize:
\begin{displaymath}
\begin{gathered}
F_t(\VTheta,\x_t,\y_t,\h_t) = \\
\frac{1}{2}
\|\VTheta-\VTheta_t\|^2
+C~\max(0,\ell(\VTheta,\x_t,\y_t,\h_t))
\end{gathered}
\end{displaymath}
%
aiming to find the set of parameters that minimize this function, i.e.:
%
\begin{equation}
\widehat\VTheta=\argmin_{\VTheta}F_t(\VTheta,\x_t,\y_t,\h_t)
\end{equation}
For obtaining $\widehat\VTheta$, we use a subgradient method \citep{Shor03}. Subgradient methods are iterative algorithms aimed to solve minimization problems with non-differentiable objective functions. Since our function has discontinuity points, such techniques are well-suited.
Therefore, we iteratively apply the following weight update:
\begin{equation}
\label{eq:update}
\Delta \VTheta_t = -\rho~ \partial_\VTheta F_t(\VTheta,\x_t,\y_t,\h_t)|_{\VTheta^{k}}
\end{equation}
%
where $\rho$ is the learning rate and $\partial_\VTheta F_t$ is the subgradient of $F_t$ with respect to $\VTheta$. We initialize $\VTheta^{k=0}=\VTheta_t$. This update is applied $k$ times, until reaching some convergence criterion. We denote this passive-aggressive via subgradient methods update rule as PAS.
\subsection{Projected subgradient}
\label{sec:ppa-subgrad}
An extension of the PAS method is the projected subgradient method (PPAS), in which the optimization problem is reformulated~\citep{Boyd03}. Let us define $G_t(\VTheta,\x_t,\y_t,\h_t)$ as $\max(0,\ell(\VTheta,\x_t,\y_t,\h_t))$. Then, \cref{eq:optimizaton-subgrad} can be rewritten as:
\begin{equation}
\begin{gathered}
\widehat\VTheta~=~
\argmin_{\VTheta} G_t(\VTheta,\x_t,\y_t,\h_t)) \\
~~\mbox{s.t.}~~ \|\VTheta-\VTheta_t\|^2 \leq C
\end{gathered}
\end{equation}
With this expression, we can compute the intermediate weight update:
\begin{gather*}
\bar\VTheta^{k+1}=
\VTheta^{k} -
\rho~
\partial_\VTheta G_t(\VTheta,\x_t,\y_t,\h_t)|_{\VTheta^{k}}
\end{gather*}
As in the previous case, we initialize $\VTheta^{k=0}=\VTheta_t$ and iterate following some convergence criterion. The final weight update defined as:
\begin{equation}
\Delta \VTheta_t =
\frac{\bar\VTheta^{k+1}-\VTheta_t}
{\|\bar\VTheta^{k+1}-\VTheta_t\|}~C
\end{equation}
\section{Experiments and results}
\label{sec:exp-setup}
In this section, we describe experimental setup used to conduct the experimentation. We define the assessment metrics, describe the corpora and detail the configuration of the NMT systems.
\subsection{Experimental framework}
We tested our OL methods in a post-editing task. Within this scope we define three different scenarios: 1. We are post-editing samples from a different domain to that on which our system has been trained. 2. We are post-editing the outputs of a system trained on the same domain of our task, but we have available out-of-domain data. 3. We only have the in-domain data, either for training and testing.
For each scenario, we adapt the system on-the-fly, applying OL to the post-edited sentences from the test set. The final goal is to reduce the post-editing effort required by upcoming samples. The main difference is that in the first case, we exclusively rely on the OL techniques for performing an adaptation of the system to the translation domain; while in the later cases, the NMT systems are already adapted. We measure up to what extent can OL refine this adapted system. Note that we work under a pure OL framework, i.e., each training sample is seen only once.
In a real post-editing scenario, the system produces a translation hypothesis. The user reviews it and corrects the mistakes committed by the system. To test our proposals in this real post-editing scenario it would require real users, which makes it prohibitively costly. Therefore, we simulate the post-editing process by using the reference sentences as post-edited ones.
In this simulation, the system generates a hypothesis for a given source sentence. Next, this hypothesis is corrected, producing a new training sample (i.e., the reference sentence). The system updates its parameters with this new sample and advances to the next sentence. This procedure is repeated until the translation process is finished.
\subsection{Metrics and evaluation}
For approximating the post-editing effort, we used the translation edit rate (TER)~\citep{Snover06}. This metric computes the edit distance between hypotheses and reference sentences. The edit operations considered are addition, deletion, substitution and shifting. Moreover, we used BLEU~\citep{Papineni02} and Meteor~\citep{Lavie09} for assessing the translation quality of the systems. BLEU compares the ratio of n-gram structures shared between the system hypotheses and reference sentences, while Meteor computes the F1 score of precision and recall between hypotheses and references, additionally considering exact, stemmed, synonyms and paraphrase matches. For all results shown in this work, we compute $95\%$ confidence intervals by means of bootstrap resampling \citep{Koehn04}.
The evaluation procedure is as follows: for a given source sentence, we produce its translation. This translation is stored for the posterior evaluation. Next, we take the post-edited sentence (i.e. the reference sentence) and update the NMT system with it. This is repeated until the full set of sentences are translated. We evaluate the stored hypotheses, hoping for a steady improvement as the translation process goes on.
\subsection{Corpora}
\label{sec:corpora}
As out-of-domain data (scenarios 1 and 2), we use the well-known, publicly available Europarl~\citep{Koehn05} corpus, using the partition \texttt{newstest2013} as development set. As in-domain corpora (scenarios 2 and 3), we use the Emea~\citep{Tiedemann09}, XRCE~\citep{Barrachina09} and TED~\citep{Federico11} corpora. The domains of them are medical, printer manuals and TED talks, respectively. We used the standard partitions for all corpora. We used the English--French language pair for all experiments. \cref{tab:corpora} shows the main figures of the corpora.
\begin{table*} [!ht]
\caption{\label{tab:corpora} Corpora number of sentences ($\vert S \vert $) and vocabulary sizes ($\vert V \vert $). k stands for thousands and M for millions.}
\centering
\small
\begin{tabular}{l lll lll lll}
\toprule
& \multicolumn{3}{c}{Training} & \multicolumn{3}{c}{Development} & \multicolumn{3}{c}{Test} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
& \multirow{2}{*}{$\vert S \vert $} & \multicolumn{2}{c}{$\vert V \vert $} & \multirow{2}{*}{$\vert S \vert $} & \multicolumn{2}{c}{$\vert V \vert $} & \multirow{2}{*}{$\vert S \vert $} & \multicolumn{2}{c}{$\vert V \vert $} \\
\cmidrule(lr){3-4} \cmidrule(lr){6-7} \cmidrule(lr){9-10}
& & \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} & & \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} && \multicolumn{1}{c}{En} & \multicolumn{1}{c}{Fr} \\
\midrule
Europarl & 2.0M &134k &153k & 3k & 10.5k & 11.9k & -- & -- & -- \\
Emea & 319k & 52.0k & 59.3k & 500 &2.8k & 2.9k & 1.0k &4.5k & 4.5k\\
TED & 159k & 46.7k & 58.2k&887 &3.4k & 3.9k &1.7k & 4.9k& 4.9k\\
XRCE & 52k & 14.0k &15.5k &984 &1.8k& 1.9k& 994 &1.7k &1.8k\\
\bottomrule
\end{tabular}
\end{table*}
We tokenized the text using the script from Moses,
keeping sentences truecase. For all tasks, we use the joint byte pair encoding (BPE) algorithm for translating at a sub-word level, as described by~\citet{Sennrich16}. We learned 32,000 merge operations.
In addition to address the unknown word problem, sub-word-level translation allows us to naturally apply an NMT system from one domain to another. We extracted the BPE codes from the out-of-domain corpus (Europarl) and applied them to each in-domain corpus. Therefore, each corpus is segmented according to the out-of-domain corpus. By using this strategy, the vocabulary coverage is extremely high (more than 99.5\% in all cases).
\subsection{NMT systems}
The NMT\footnote{Implementation publicly available at: \url{https://github.com/lvapeab/nmt-keras/tree/interactive_NMT}} systems and OL algorithms were implemented with the Keras and Theano~\citep{Theano16} libraries.
For choosing the main hyperparameters of the system, we take advantage of the vast exploration made by~\citet{Britz17}. Therefore, as described in~\cref{sec:nmt}, the system consists in an encoder-decoder LSTM network equipped with the attention mechanism described by \citet{Bahdanau15}. For practical reasons, we used single-layered LSTMs. The size of each LSTM is 512, as well as the word embedding size and the attention mechanism layer. As regularization methods, we applied layer normalization~\citep{Ba16} and Gaussian noise to the weights ($\sigma =0.01$)~\citep{Graves11}. We also early stopped the training if the BLEU on the development set did not improve in $100,000$ updates. The original NMT systems were trained using Adadelta~\citep{Zeiler12}, with the default parameters. The norm of the gradients was clipped to 1 in order to avoid the exploding gradient problem~\citep{Pascanu13}. The size of the beam was set to 6 in all experiments.
Online learning hyperparameters were estimated on the Europarl validation set through a grid search, with $\rho$ and $C$ taking the values $10^{-a}, a \in {\{0, -1, -2, -3, -4, -5, -6\}}$. We set $\epsilon = 10^{-8}$ and the rest of the Adam values to their default ($\beta_1=0.9$, $\beta_2=0.999$). \cref{tab:hyperparams} shows the value of the hyperparameters for each algorithm.
\begin{table}
\caption{\label{tab:hyperparams} Algorithm hyperparameters. $\rho$ refers to the learning rate and $C$ to the aggressiveness of PA algorithms.}
\centering
\footnotesize
{\setlength{\tabcolsep}{4pt}
\begin{tabular}{lllllll}
\toprule
& SGD & Adagrad & Adadelta & Adam & PAS & PPAS \\
$\rho$ & $10^{-3}$ & $10^{-4}$ & $10^{-1}$ & $10^{-3}$ & $1$ & $10^{-2}$ \\
$C$ & -- &-- & --& --& $10^{-2}$ & $10^{-2}$\\
\bottomrule
\end{tabular}}
\end{table}
\section{Results and discussion}
\label{sec:results}
In this section, we show and analyze the experimentation conducted. We applied OL to the NMT system with the post-edited samples from the test set, with the goal of reducing the post-editing effort required by the following samples. As stated in \cref{sec:corpora}, we pose three different scenarios:
\begin{enumerate}[noitemsep]
\item We have a general corpus and no in-domain data. Therefore, in order to reduce the post-editing effort, we must take advantage exclusively from the in-domain test samples by means of OL.
\item We have the general corpus, but we also have enough in-domain data for training a system. In this case, we first train with the out-of-domain corpus and next with the in-domain corpus. OL can also fine-tune the resulting system, for obtaining a more tailored system and consequently, reducing the post-editing effort.
\item We only have in-domain corpora. This situation is more artificial than the previous ones, but illustrates up to which extent the NMT system can be improved and adapted to a test set by means of OL.
\end{enumerate}
Since the final scope of this work is a real application, we should check whether the OL algorithms consume an excessive amount of time. The mean update time\footnote{Tested on a Nvidia GTX1080 GPU with CuDNN 5005.} was similar for each algorithm (approximately 65 ms), which is affordable.
\subsection{Adapting without in-domain}
In this scenario, we assume that we only have an out-of-domain corpus. The text to translate comes from an unknown domain. Therefore, we must adapt the system on-the-fly, according to the post-edited sentences: as soon as a sentence is translated and corrected, we modify our system, in order to take this sample into account. As baseline, we take an offline system, which remains static along the translation process.
First, we study the evolution of the different learning algorithms. \cref{fig:evol-bleu1} shows the difference in terms of BLEU between the baseline and the online systems, for the Emea test set as a function of the number of processed sentences. We show the average of BLEU up to the $n$-th sentence.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{bleu_dif.pdf}
\caption{\label{fig:evol-bleu1} Evolution of a system trained on Europarl with the Emea test set. We show the BLEU difference (averaged up to the $n$-th sentence) with respect to an offline system.}
\end{figure}
As we can see in~\cref{fig:evol-bleu1}, the vanilla SGD algorithm remained steady along the learning process. Adaptive algorithms (Adagrad, Adadelta, Adam) had a more unstable behavior at the early learning stages. This is because these algorithms rely on the accumulation of the past gradients for updating the model weights. Therefore, at the beginning, they present a chaotic behavior, but once they are stabilized, the learning improves. The Adam algorithm clearly exhibits this: it took 400 sentences to improve the baseline, but from here, the enhancement was acute, eventually yielding the best overall performance. On the other hand, PA algorithms remain close to the original system, which denotes an excessive passivity.
\cref{fig:emea-europarl} shows the overall differences between OL algorithms in terms of BLEU. The dashed line represents our baseline. All three adaptive SGD algorithms were able to significantly improve over the baseline. Vanilla SGD also yielded small enhancements, although they were non-significant. PA-like algorithms obtained a similar performance to the baseline.
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{emea_europarl.pdf}
\caption{\label{fig:emea-europarl} Effect of tuning an NMT system trained on Europarl with the Emea test set in terms of BLEU. The horizontal line refers to an offline system. We show $95\%$ confidence intervals.}
\end{figure}
Finally, \cref{tab:no-in-domain-nmt} condenses the results of the adaptation without an in-domain corpus. For the sake of clarity, we only show results for the best performing algorithm for each task. In almost every case, the online system performed significantly better than the offline one. The Emea and TED tasks had a similar difficulty for the out-of-domain system and the improvements obtained through OL were alike.
The NMT system performed much worse in the XRCE task. This is probably due to the corpus characteristics: since it is extracted from printer manuals, many of its structures are nonexistent in the out-of-domain corpus. Therefore, the performance drops. OL also succeeded in adapting the model to this task, but its contribution was limited by the size of the test set: OL could adapt the model to the test set, but only to some extent.
\noindent\begin{table}[h]
\caption{\label{tab:no-in-domain-nmt} Offline vs online systems, trained without in-domain data. Bold results indicate a significant improvement over the offline system. We also show the best performing algorithm for each task.}
\centering
\footnotesize
{\setlength{\tabcolsep}{4pt}
\begin{tabular}{lllll}
\toprule
& & Emea & TED & XRCE \\
\cmidrule(lr){3-5}
\multirow{3}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Offline}}} & BLEU & $20.5 \pm 1.0$ & $20.9\pm 0.8$ & $8.3 \pm 0.9$\\
& Meteor & $42.4 \pm 1.0$ & $41.1 \pm 0.9$ & $19.7 \pm 1.0$\\
& TER & $69.0 \pm 1.6$ & $67.8 \pm 1.2$ & $103.8 \pm 3.5$ \\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Online}}} & BLEU & $\mathbf{24.0 \pm 1.0}$ & $\mathbf{24.0 \pm 0.9}$ & $ \mathbf{11.8 \pm 1.2}$\\
& Meteor & $\mathbf{46.5 \pm 1.0}$ & $\mathbf{43.0 \pm 0.9}$& $19.8 \pm 1.3$\\
& TER & $\mathbf{63.1 \pm 1.5}$ & $\mathbf{62.4 \pm 1.1}$ &$\mathbf{83.2 \pm 3.9}$ \\
\cmidrule(lr){2-5}
& Algorithm & Adam & Adadelta& Adam \\
\bottomrule
\end{tabular}}
\end{table}
\subsection{Adapting with an in-domain}
In this case, we assume that we have a collection of in-domain data, in addition to our out-of-domain corpus. Therefore, we first train a general system on the out-of-domain data and fine-tune it with the in-domain training set. Next, we study if OL can still improve the model, although it is already adapted to the in-domain task. As before, we apply OL techniques to the test set.
\begin{figure}[!ht]
\centering
\includegraphics[width=0.5\textwidth]{bleu_dif2.pdf}
\caption{\label{fig:evol-bleu_europarl_emea_emea} Evolution of a system trained on Europarl and Emea with the Emea test set. We show the BLEU difference (averaged up to the $n$-th sentence) with respect to an offline system.}
\end{figure}
\cref{fig:evol-bleu_europarl_emea_emea} shows the BLEU differences between OL algorithms. In this case, the behavior was more stable for all algorithms. Although some variance can still be observed at the early stages of the process, the system learned faster and better than in the previous case. Since the effective learning started early, the cumulated enhancements were larger. From~\cref{fig:evol-bleu_europarl_emea_emea} we can also extract the importance of the adaptive SGD implementations, which make the learning process to converge faster and to a better point.
Absolute BLEU improvement is shown in~\cref{fig:europarl_emea_emea}. Again, the dashed line represents the offline baseline. In this case, the gains obtained by the adaptive SGD algorithms are even higher than in the previous experiment. Adagrad, Adadelta and Adam yielded to almost the same value, producing improvements of more than 7 BLEU points with respect to the baseline.
\begin{figure}[!h]
\centering
\includegraphics[width=0.5\textwidth]{europarl_emea_emea.pdf}
\caption{\label{fig:europarl_emea_emea} Effect of OL in an NMT system trained on Europarl and Emea. We use the Emea test set for applying OL. We show BLEU and $95\%$ confidence intervals. The horizontal line refers to the offline system baseline.}
\end{figure}
\cref{tab:in-domain-nmt} shows the overall results for this scenario. Again, we only show figures of the best method for each task. The use of in-domain training data allowed the NMT system to yield much better translation quality than in the previous case. Especially dramatics were the improvements in the TED and XRCE corpora (+10.4 and +26.4 BLEU points, respectively). For such tasks, the use of OL only affected the system to a negligible extent.
On the other hand, the improvements observed in the Emea task were smaller than those obtained in other corpora. Nevertheless, this is clearly complemented by the online system. In this case, OL was able to adapt the system to the test set. The system was significantly improved. The post-editing effort was reduced approximately by 15\%.
\noindent\begin{table} [h]
\caption{\label{tab:in-domain-nmt}Offline vs online systems, trained with out-of-domain data and adapted with additional in-domain data. Bold results indicate a significant improvement of the online system with respect to the offline. We also indicate the best performing algorithm for each task.}
\centering
\footnotesize
{\setlength{\tabcolsep}{4pt}
\begin{tabular}{lllll}
\toprule
& & Emea & TED & XRCE \\
\cmidrule(lr){3-5}
\multirow{3}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Offline}}}& BLEU & $21.3\pm 1.0$ & $31.3\pm 1.0$ & $ 34.7 \pm 2.2$\\
& Meteor & $39.5 \pm 1.1$ & $52.0 \pm 1.0$ & $52.2 \pm 2.3$\\
& TER & $66.9 \pm 1.5$ & $52.3 \pm 1.0$ & $55.0 \pm 2.4$ \\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Online}}} & BLEU & $\mathbf{28.6 \pm 1.2}$ & $31.4 \pm 1.0$& $35.1 \pm 2.2$\\
& Meteor & $\mathbf{48.8 \pm 1.1}$ & $51.8 \pm 0.9$ & $53.0\pm 2.1$\\
& TER & $\mathbf{57.0 \pm 1.5}$ & $52.2 \pm 1.0$ & $55.8 \pm 2.2$\\
\cmidrule(lr){2-5}
& Algorithm & Adam & PPAS & PPAS \\
\bottomrule
\end{tabular}}
\end{table}
\subsection{Refining an NMT system}
Finally, we move to the scenario in which we have enough in-domain data for training an NMT system. The system is trained exclusively on the in-domain data. OL learning techniques can be used for refining the system at test time.
\cref{fig:evol-bleu_emea} shows the evolution of the BLEU scores along the online learning process. Again, algorithms with adaptive learning rate perform better than the rest of them. Nevertheless, such differences are lower than in other tasks. The PAS algorithm is especially effective at the first iterations of the process. %TODO: Why?
\begin{figure}[!ht]
\centering
\includegraphics[width=0.5\textwidth]{bleu_dif3.pdf}
\caption{\label{fig:evol-bleu_emea} Evolution of a system trained on Europarl and Emea with the own Emea test set. We show the BLEU difference (averaged up to the $n$-th sentence) with respect to an offline system.}
\end{figure}
All OL algorithms significantly outperform the offline baseline, as shown in~\cref{fig:emea}. The performance of the PAS algorithm is comparable to the best-performing algorithms. The projected PA version is slightly better than vanilla SGD, although it yields worse results than the other algorithms.
\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{emea.pdf}
\caption{\label{fig:emea} Effect of tuning an NMT system trained on Emea and refined with the own Emea test, in terms of BLEU. The horizontal line refers to an offline system. We show $95\%$ confidence intervals.}
\end{figure}
\cref{tab:only-in-domain} collapses the performance of the systems in the different tasks.
\begin{table} [h]
\caption{\label{tab:only-in-domain} Offline vs online systems, trained solely on the in-domain data. Bold results indicate a significant improvement of the online system with respect to the offline. We also indicate the best performing algorithm for each task.}
\centering
\footnotesize
{\setlength{\tabcolsep}{4pt}
\begin{tabular}{lllll}
\toprule
& & Emea & TED & XRCE \\
\cmidrule(lr){3-5}
\multirow{3}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Offline}}} & BLEU & $17.6 \pm 0.9$ & $27.1\pm 0.9$ & $31.0\pm 2.2$\\
& Meteor & $37.1 \pm 0.9$ & $47.6 \pm 1.0$ & $49.8 \pm 2.0$\\
& TER & $74.2 \pm 2.2$ & $57.9 \pm 1.0$& $62.8 \pm 2.7$\\
\midrule
\multirow{4}{*}{\rotatebox[origin=c]{90}{\parbox[c]{1cm}{\centering Online}}} & BLEU & $\mathbf{22.4 \pm 1.0}$ & $28.2 \pm 0.9$ & $\mathbf{36.7 \pm 2.3}$\\
& Meteor & $\mathbf{41.1 \pm 0.9}$ & $49.1 \pm 1.0$ & $\mathbf{54.2 \pm 2.0}$ \\
& TER & $\mathbf{63.8 \pm 1.4}$ & $56.0\pm 1.1$ & $\mathbf{56.5 \pm 2.6}$\\
\cmidrule(lr){2-5}
& Algorithm & Adadelta & Adadelta & Adagrad \\
\bottomrule
\end{tabular}}
\end{table}
Even when the systems have been exclusively trained on in-domain data, OL helps to perform a more fine-grained adaptation to the text to translate. In the Emea and XRCE tasks we could observe significant improvements (approximately 5 points of BLEU). This is probably due to regularities in the test set, which allow to exploit OL to its full. OL also improved the performance in TED task, although to a lower extent.
\section{Conclusions and future work}
\label{sec:conclusions}
In this work, we studied the implementation of OL strategies for NMT. We empirically demonstrated the capacity of NMT for adapting to new domains, by means of gradient descent. Moreover, we discussed and compared a wide range of gradient descent algorithms. In addition, we proposed two novel methods, inspired by a PA strategy and tackled by means of subgradient optimization methods. PA-based methods offer a competitive performance. In almost every case, they outperformed vanilla SGD. Nevertheless, adaptive SGD algorithms performed generally better than PA. We found two cases in which the performance of the PA method was superior to adaptive SGD algorithms, although differences were small.
One of the main drawbacks relating the NMT is the independence (up to some point) of the network objective function (probability of the target sentence given the source) with respect to the evaluation metric (e.g. BLEU or TER). Minimum risk training aims to overcome this gap and it has successfully been applied to classical SMT~\citep{Och03,Chiang12} and recently to NMT~\citep{Shen16}.
In this work, we set the basis for using BLEU or other non-differentiable loss function, since our PAS and PPAS algorithms cope with non-differentiable functions. As future work, we intend to directly optimize the evaluation metric, integrating it into the online learning framework~\citep{Martinez-Gomez12}. A major challenge is keeping the update time at a reasonable time.
Finally, interactive machine translation (IMT) is tightly related with this work. IMT consists in an evolution of the classical post-editing scenario, striving for profiting the human effort made in the post-editing process. Under the IMT paradigm, human and computer collaborate in order to minimize the user effort. The development of interactive NMT systems has been recently addressed~\citep{Knowles16,Peris16}. The inclusion of OL techniques into the interactive framework is the next natural step to take, in order to develop more adaptive and productive translation systems.
\begin{thebibliography}{}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Alabau et~al.(2013)Alabau, Bonk, Buck, Carl, Casacuberta,
Garc{\'i}a-Mart{\'i}nez, Gonz{\'a}lez-Rubio, Koehn, Leiva, Mesa-Lao,
Ortiz-Mart{\'i}nez, Saint-Amand, Sanchis-Trilles, and Tsoukala}]{Alabau13}
Vicent Alabau, Ragnar Bonk, Christian Buck, Michael Carl, Francisco
Casacuberta, Mercedes Garc{\'i}a-Mart{\'i}nez, Jes{\'u}s Gonz{\'a}lez-Rubio,
Philipp Koehn, Luis~A. Leiva, Bartolom{\'e} Mesa-Lao, Daniel
Ortiz-Mart{\'i}nez, Herv{\'e} Saint-Amand, Germ{\'a}n Sanchis-Trilles, and
Chara Tsoukala. 2013.
\newblock {CASMACAT}: An open source workbench for advanced computer aided
translation.
\newblock {\em The Prague Bulletin of Mathematical Linguistics\/} 100:101--112.
\bibitem[{Arenas(2008)}]{Arenas08}
Ana~Guerberof Arenas. 2008.
\newblock Productivity and quality in the post-editing of outputs from
translation memories and machine translation.
\newblock {\em Localisation Focus\/} 7(1):11--21.
\bibitem[{Ba et~al.(2016)Ba, Kiros, and Hinton}]{Ba16}
Jimmy~Lei Ba, Jamie~Ryan Kiros, and Geoffrey Hinton. 2016.
\newblock Layer normalization.
\newblock \textit{arXiv:1607.06450}.
\bibitem[{Bahdanau et~al.(2015)Bahdanau, Cho, and Bengio}]{Bahdanau15}
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.
\newblock Neural machine translation by jointly learning to align and
translate.
\newblock \textit{arXiv:1409.0473}.
\bibitem[{Barrachina et~al.(2009)Barrachina, Bender, Casacuberta, Civera,
Cubel, Khadivi, Lagarda, Ney, Tom\'{a}s, Vidal, and Vilar}]{Barrachina09}
Sergio Barrachina, Oliver Bender, Francisco Casacuberta, Jorge Civera, Elsa
Cubel, Shahram Khadivi, Antonio Lagarda, Hermann Ney, Jes\'{u}s Tom\'{a}s,
Enrique Vidal, and Juan-Miguel Vilar. 2009.
\newblock Statistical approaches to computer-assisted translation.
\newblock {\em Computational Linguistics\/} 35:3--28.
\bibitem[{Bengio et~al.(1994)Bengio, Simard, and Frasconi}]{Bengio94}
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994.
\newblock Learning long-term dependencies with gradient descent is difficult.
\newblock {\em IEEE transactions on neural networks\/} 5(2):157--166.
\bibitem[{Boyd et~al.(2003)Boyd, Xiao, and Mutapcic}]{Boyd03}
Stephen Boyd, Lin Xiao, and Almir Mutapcic. 2003.
\newblock Subgradient methods.
\newblock In {\em lecture notes of EE392o, Stanford University, Autumn
Quarter\/}.
\bibitem[{Britz et~al.(2017)Britz, Goldie, Luong, and Le}]{Britz17}
Denny Britz, Anna Goldie, Thang Luong, and Quoc Le. 2017.
\newblock Massive exploration of neural machine translation architectures.
\newblock \textit{arXiv:1703.03906}.
\bibitem[{Chiang(2012)}]{Chiang12}
David Chiang. 2012.
\newblock Hope and fear for discriminative training of statistical translation
models.
\newblock {\em Journal of Machine Learning Research\/} 13(Apr):1159--1187.
\bibitem[{Cho et~al.(2014)Cho, van Merri{\"e}nboer, Bahdanau, and
Bengio}]{Cho14}
Kyunghyun Cho, Bart van Merri{\"e}nboer, Dzmitry Bahdanau, and Yoshua Bengio.
2014.
\newblock On the properties of neural machine translation: Encoder-decoder
approaches.
\newblock In {\em Proceedings of the Workshop on Syntax, Semantic and Structure
in Statistical Translation\/}. pages 103--111.
\bibitem[{Chung et~al.(2016)Chung, Cho, and Bengio}]{Chung16}
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016.
\newblock A character-level decoder without explicit segmentation for neural
machine translation.
\newblock \textit{arXiv:1603.06147}.
\bibitem[{Crammer et~al.(2006)Crammer, Dekel, Keshet, Shalev-Shwartz, and
Singer}]{Crammer06}
Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer.
2006.
\newblock Online passive-aggressive algorithms.
\newblock {\em Journal of Machine Learning Research\/} 7:551--585.
\bibitem[{Crammer and Singer(2001)}]{Crammer01}
Koby Crammer and Yoram Singer. 2001.
\newblock Ultraconservative online algorithms for multiclass problems.
\newblock In {\em Proceedings of the Annual Conference on Computational
Learning Theory\/}. pages 99--115.
\bibitem[{Crego et~al.(2016)Crego, Kim, Klein, Rebollo, Yang, Senellart,
Akhanov, Brunelle, Coquard, Deng, Enoue, Geiss, Johanson, Khalsa, Khiari, Ko,
Kobus, Lorieux, Martins, Nguyen, Priori, Riccardi, Segal, Servan, Tiquet,
Wang, Yang, Zhang, Zhou, and Zoldan}]{Crego16}
Josep~Maria Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean
Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng,
Satoshi Enoue, Chiyo Geiss, Joshua Johanson, Ardas Khalsa, Raoum Khiari,
Byeongil Ko, Catherine Kobus, Jean Lorieux, Leidiana Martins, Dang{-}Chuan
Nguyen, Alexandra Priori, Thomas Riccardi, Natalia Segal, Christophe Servan,
Cyril Tiquet, Bo~Wang, Jin Yang, Dakun Zhang, Jing Zhou, and Peter Zoldan.
2016.
\newblock {SYSTRAN}'s pure neural machine translation systems.
\newblock \textit{arXiv:1610.05540}.
\bibitem[{Duchi et~al.(2011)Duchi, Hazan, and Singer}]{Duchi11}
John Duchi, Elad Hazan, and Yoram Singer. 2011.
\newblock Adaptive subgradient methods for online learning and stochastic
optimization.
\newblock {\em Journal of Machine Learning Research\/} 12:2121--2159.
\bibitem[{Erk and Smith(2016)}]{Erk16}
Katrin Erk and Noah~A. Smith, editors. 2016.
\newblock {\em Proceedings of the 54th Annual Meeting of the Association for
Computational Linguistics\/}.
\bibitem[{Federico et~al.(2011)Federico, Bentivogli, Paul, and
St{\"{u}}ker}]{Federico11}
Marcello Federico, Luisa Bentivogli, Michael Paul, and Sebastian St{\"{u}}ker.
2011.
\newblock Overview of the {IWSLT} evaluation campaign.
\newblock In {\em Proceedings of the International Workshop on Spoken Language
Translation\/}. pages 11--27.
\bibitem[{Graves(2011)}]{Graves11}
Alex Graves. 2011.
\newblock Practical variational inference for neural networks.
\newblock In {\em Advances in Neural Information Processing Systems\/}. pages
2348--2356.
\bibitem[{Hochreiter and Schmidhuber(1997)}]{Hochreiter97}
Sepp Hochreiter and J{\"u}rgen Schmidhuber. 1997.
\newblock Long short-term memory.
\newblock {\em Neural Computation\/} 9(8):1735--1780.
\bibitem[{Jean et~al.(2015)Jean, Cho, Memisevic, and Bengio}]{Jean15}
S\'{e}bastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015.
\newblock On using very large target vocabulary for neural machine translation.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics and the International Joint Conference on Natural
Language Processing\/}. pages 1--10.
\bibitem[{Kalchbrenner and Blunsom(2013)}]{Kalchbrenner13}
Nal Kalchbrenner and Phil Blunsom. 2013.
\newblock Recurrent continuous translation models.
\newblock In {\em Proceedings of the Conference on Empirical Methods in Natural
Language Processing\/}. pages 1700--1709.
\bibitem[{Kingma and Ba(2014)}]{Kingma14}
Diederik Kingma and Jimmy Ba. 2014.
\newblock Adam: A method for stochastic optimization.
\newblock \textit{arXiv:1412.6980}.
\bibitem[{Knowles and Koehn(2016)}]{Knowles16}
Rebecca Knowles and Philipp Koehn. 2016.
\newblock Neural interactive translation prediction.
\newblock In {\em Proceedings of the Association for Machine Translation in the
Americas\/}. pages 107--120.
\bibitem[{Koehn(2004)}]{Koehn04}
Philipp Koehn. 2004.
\newblock Statistical significance tests for machine translation evaluation.
\newblock In {\em Proceedings of the Conference on Empirical Methods in Natural
Language Processing\/}. pages 388--395.
\bibitem[{Koehn(2005)}]{Koehn05}
Philipp Koehn. 2005.
\newblock Europarl: A parallel corpus for statistical machine translation.
\newblock In {\em Proceedings of the Machine Translation Summit\/}. pages
79--86.
\bibitem[{Koehn(2010)}]{Koehn10}
Philipp Koehn. 2010.
\newblock A process study of computer-aided translation.
\newblock {\em Machine Translation\/} 23(4):241--263.
\bibitem[{Lavie and Denkowski(2009)}]{Lavie09}
Alon Lavie and Michael~J Denkowski. 2009.
\newblock The {METEOR} metric for automatic evaluation of machine translation.
\newblock {\em Machine translation\/} 23(2-3):105--115.
\bibitem[{Luong et~al.(2015{\natexlab{a}})Luong, Pham, and Manning}]{Luong15a}
Thang Luong, Hieu Pham, and Christopher~D. Manning. 2015{\natexlab{a}}.
\newblock Effective approaches to attention-based neural machine translation.
\newblock In {\em Proceedings of the Conference on Empirical Methods in Natural
Language Processing\/}. pages 1412--1421.
\bibitem[{Luong et~al.(2015{\natexlab{b}})Luong, Sutskever, Le, Vinyals, and
Zaremba}]{Luong15b}
Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba.
2015{\natexlab{b}}.
\newblock Addressing the rare word problem in neural machine translation.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics and the International Joint Conference on Natural
Language Processing\/}. pages 11--19.
\bibitem[{Mart\'{i}nez-G\'{o}mez et~al.(2012)Mart\'{i}nez-G\'{o}mez,
Sanchis-Trilles, and Casacuberta}]{Martinez-Gomez12}
Pascual Mart\'{i}nez-G\'{o}mez, Germ\'{a}n Sanchis-Trilles, and Francisco
Casacuberta. 2012.
\newblock Online adaptation strategies for statistical machine translation in
post-editing scenarios.
\newblock {\em Pattern Recognition\/} 45(9):3193--3203.
\bibitem[{Mathur et~al.(2013)Mathur, Cettolo, Federico, and Kessler}]{Mathur13}
Prashant Mathur, Mauro Cettolo, Marcello Federico, and FBK-Fondazione~Bruno
Kessler. 2013.
\newblock Online learning approaches in computer assisted translation.
\newblock In {\em Proceedings of the Eighth Workshop on Statistical Machine
Translation\/}. pages 301--308.
\bibitem[{Och(2003)}]{Och03}
Franz~Josef Och. 2003.
\newblock Minimum error rate training in statistical machine translation.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics\/}. pages 160--167.
\bibitem[{Och and Ney(2002)}]{Och02}
Franz~Josef Och and Hermann Ney. 2002.
\newblock Discriminative training and maximum entropy models for statistical
machine translation.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics\/}. pages 295--302.
\bibitem[{Ortiz-Mart{\'i}nez(2016)}]{Ortiz16}
Daniel Ortiz-Mart{\'i}nez. 2016.
\newblock Online learning for statistical machine translation.
\newblock {\em Computational Linguistics\/} 42(1):121--161.
\bibitem[{Papineni et~al.(2002)Papineni, Roukos, Ward, and Zhu}]{Papineni02}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
\newblock {BLEU}: a method for automatic evaluation of machine translation.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics\/}. pages 311--318.
\bibitem[{Pascanu et~al.(2014)Pascanu, Gulcehre, Cho, and Bengio}]{Pascanu14}
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014.
\newblock How to construct deep recurrent neural networks.
\newblock \textit{arXiv:1312.6026}.
\bibitem[{Pascanu et~al.(2012)Pascanu, Mikolov, and Bengio}]{Pascanu13}
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012.
\newblock On the difficulty of training recurrent neural networks.
\newblock \textit{arXiv:1211.5063}.
\bibitem[{Peris et~al.(2017)Peris, Domingo, and Casacuberta}]{Peris16}
{\'A}lvaro Peris, Miguel Domingo, and Francisco Casacuberta. 2017.
\newblock Interactive neural machine translation.
\newblock {\em Computer Speech \& Language\/} 45:201--220.
\bibitem[{Plitt and Masselot(2010)}]{Plitt10}
Mirko Plitt and Fran{\c{c}}ois Masselot. 2010.
\newblock A productivity test of statistical machine translation post-editing
in a typical localisation context.
\newblock {\em The Prague bulletin of mathematical linguistics\/} 93:7--16.
\bibitem[{Robbins and Monro(1951)}]{Robbins51}
Herbert Robbins and Sutton Monro. 1951.
\newblock A stochastic approximation method.
\newblock {\em The Annals of Mathematical Statistics\/} pages 400--407.
\bibitem[{Sennrich et~al.(2016)Sennrich, Haddow, and Birch}]{Sennrich16}
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.
\newblock Neural machine translation of rare words with subword units.
\newblock In {\em Proceedings of the Annual Meeting of the Association for
Computational Linguistics\/}. pages 1715--1725.
\bibitem[{Shen et~al.(2016)Shen, Cheng, He, He, Wu, Sun, and Liu}]{Shen16}
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu.
2016.
\newblock Minimum risk training for neural machine translation.
\newblock In {\em Proceedings of the 54th Annual Meeting of the Association for
Computational Linguistics\/}. Association for Computational Linguistics,
pages 1683--1692.
\bibitem[{Shor et~al.(2003)Shor, Zhurbenko, Likhovid, and Stetsyuk}]{Shor03}
N.~Z. Shor, N.~G. Zhurbenko, A.~P. Likhovid, and P.~I. Stetsyuk. 2003.
\newblock Algorithms of nondifferentiable optimization: Development and
application.
\newblock {\em Cybernetics and Systems Analysis\/} 39(4):537--548.
\bibitem[{Snover et~al.(2006)Snover, Dorr, Schwartz, Micciulla, and
Makhoul}]{Snover06}
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John
Makhoul. 2006.
\newblock A study of translation edit rate with targeted human annotation.
\newblock In {\em Proceedings of the Association for Machine Translation in the
Americas\/}. pages 223--231.
\bibitem[{Sutskever et~al.(2014)Sutskever, Vinyals, and Le}]{Sutskever14}
Ilya Sutskever, Oriol Vinyals, and Quoc~V. Le. 2014.
\newblock Sequence to sequence learning with neural networks.
\newblock In {\em Proceedings of the Advances in Neural Information Processing
Systems\/}, volume~27, pages 3104--3112.
\bibitem[{{Theano Development Team}(2016)}]{Theano16}
{Theano Development Team}. 2016.
\newblock Theano: A {Python} framework for fast computation of mathematical
expressions.
\newblock \textit{arXiv:1605.02688}.
\bibitem[{Tiedemann(2009)}]{Tiedemann09}
J{\"o}rg Tiedemann. 2009.
\newblock News from {OPUS} - {A} collection of multilingual parallel corpora
with tools and interfaces.
\newblock In {\em Recent Advances in Natural Language Processing\/}, volume~5,
pages 237--248.
\bibitem[{Watanabe et~al.(2007)Watanabe, Suzuki, Tsukada, and
Isozaki}]{Watanabe07}
Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007.
\newblock Online large-margin training for statistical machine translation.
\newblock In {\em Proceedings of the 2007 Joint Conference on Empirical Methods
in Natural Language Processing and Computational Natural Language
Learning\/}. Association for Computational Linguistics.
\bibitem[{{Wu} et~al.(2016){Wu}, {Schuster}, {Chen}, {Le}, {Norouzi},
{Macherey}, {Krikun}, {Cao}, {Gao}, {Macherey}, {Klingner}, {Shah},
{Johnson}, {Liu}, {Kaiser}, {Gouws}, {Kato}, {Kudo}, {Kazawa}, {Stevens},
{Kurian}, {Patil}, {Wang}, {Young}, {Smith}, {Riesa}, {Rudnick}, {Vinyals},
{Corrado}, {Hughes}, and {Dean}}]{Wu16}
Y.~{Wu}, M.~{Schuster}, Z.~{Chen}, Q.~V. {Le}, M.~{Norouzi}, W.~{Macherey},
M.~{Krikun}, Y.~{Cao}, Q.~{Gao}, K.~{Macherey}, J.~{Klingner}, A.~{Shah},
M.~{Johnson}, X.~{Liu}, {\L}.~{Kaiser}, S.~{Gouws}, Y.~{Kato}, T.~{Kudo},
H.~{Kazawa}, K.~{Stevens}, G.~{Kurian}, N.~{Patil}, W.~{Wang}, C.~{Young},
J.~{Smith}, J.~{Riesa}, A.~{Rudnick}, O.~{Vinyals}, G.~{Corrado},
M.~{Hughes}, and J.~{Dean}. 2016.
\newblock {Google's Neural Machine Translation System: Bridging the Gap between
Human and Machine Translation}.
\newblock \textit{arXiv:1609.08144}.
\bibitem[{Zeiler(2012)}]{Zeiler12}
Matthew~D. Zeiler. 2012.
\newblock {ADADELTA}: An adaptive learning rate method.
\newblock \textit{arXiv:1212.5701}.
\bibitem[{Zens et~al.(2002)Zens, Och, and Ney}]{Zens02}
Richard Zens, Franz~Josef Och, and Hermann Ney. 2002.
\newblock Phrase-based statistical machine translation.
\newblock In {\em Proceedings of the Annual German Conference on Advances in
Artificial Intelligence\/}. volume 2479, pages 18--32.
\end{thebibliography}