"Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks
Extracted source text · arXiv 2104.08384
% BARR
% File tacl2018v2.tex
% Sep 20, 2018
% The English content of this file was modified from various *ACL instructions
% by Lillian Lee and Kristina Toutanova
%
% LaTeXery is mostly all adapted from acl2018.sty.
\documentclass[11pt]{article}
\usepackage{arabtex}
\usepackage{emnlp2021}
\usepackage{times,latexsym}
\usepackage{url}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{utf8}
\usepackage{pgf}
\usepackage{tikz-dependency}
\usepackage[export]{adjustbox}
\usepackage{verbatim}
%\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{***} % Enter the acl Paper ID here
\usepackage{graphicx,wrapfig,lipsum}
\usepackage{multirow}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{algorithmicx}
\usepackage[ruled]{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{tikz}
\usepackage{xcolor}
\usepackage{pgfplots}
\usetikzlibrary{patterns}
\usepackage{subcaption}
%\usepackage{todonotes}
%%%% Material in this block is specific to generating TACL instructions
% \usepackage{xspace,mfirstuc,tabulary}
% \newcommand{\dateOfLastUpdate}{Sept. 20, 2018}
% \newcommand{\styleFileVersion}{tacl2018v2}
\newcommand*{\affaddr}[1]{#1} % No op here. Customize it for different styles.
\newcommand*{\affmark}[1][*]{\textsuperscript{#1}}
\newcommand{\ex}[1]{{\sf #1}}
\newcommand{\printfnsymbol}[1]{%
\textsuperscript{\@fnsymbol{#1}}%
}
\usepackage{microtype}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
%\title{``Wikily'' Neural Machine Translation without Explicit Supervision}
\title{``Wikily'' Supervised Neural Translation Tailored to Cross-Lingual Tasks}
\author{Mohammad Sadegh Rasooli$^{1}$\thanks{~~Research was conducted at The University of Pennsylvania.}\ \ Chris Callison-Burch$^{2}$\ \ Derry Tanti Wijaya$^{3}$ \\
\normalsize{~$^1$Microsoft}\\[-.05cm]
\normalsize{~$^2$Department of Computer and Information Science, University of Pennsylvania}\\[-.05cm]
\normalsize{~$^3$Department of Computer Science, Boston University}\\[-.05cm]
{\small\tt mrasooli@microsoft.com,\ ccb@seas.upenn.edu,\ wijaya@bu.edu} \\
}
\date{}
\begin{document}
\maketitle
\begin{abstract}
We present a simple but effective approach for leveraging Wikipedia for neural machine translation as well as cross-lingual tasks of image captioning and dependency parsing without using any direct supervision from external parallel data or supervised models in the target language. We show that first sentences and titles of linked Wikipedia pages, as well as cross-lingual image captions, are strong signals for a seed parallel data to extract bilingual dictionaries and cross-lingual word embeddings for mining parallel text from Wikipedia. Our final model achieves high BLEU scores that are close to or sometimes higher than strong \emph{supervised} baselines in low-resource languages; e.g. supervised BLEU of 4.0 versus 12.1 from our model in English-to-Kazakh. Moreover, we tailor our \emph{``wikily'' supervised} translation models to unsupervised image captioning, and cross-lingual dependency parser transfer. In image captioning, we train a multi-tasking machine translation and image captioning pipeline for Arabic and English from which the Arabic training data is a translated version of the English captioning data, using our wikily-supervised translation models. Our captioning results on Arabic are slightly \emph{better} than that of its supervised model. In dependency parsing, we translate a large amount of monolingual text, and use it as artificial training data in an \emph{annotation projection} framework. We show that our model outperforms recent work on cross-lingual transfer of dependency parsers.
\end{abstract}
\input{intro}
\input{background}
\input{approach}
\input{experiments}
\input{related}
\input{conclusion}
\section*{Acknowledgments}
We would like to thank reviewers and the editor for their useful comments. We also would like to thank Alireza Zareian, Daniel (Joongwon) Kim, Qing Sun, and Afra Feyza Akyurek for their help and useful comments througout this project. This work is
supported in part by the DARPA HR001118S0044 (the LwLL program),
and the Department of the Air Force FA8750-19-
2-3334 (Semi-supervised Learning of Multimodal
Representations). The U.S. Government is authorized
to reproduce and distribute reprints for Governmental
purposes. The views and conclusions
contained in this publication are those of the authors
and should not be interpreted as representing
official policies or endorsements of DARPA, the
Air Force, and the U.S. Government.
%\appendix
%\input{analysis}
\bibliography{refs}
\bibliographystyle{acl_natbib}
\appendix
\input{appendix}
\end{document}
\section{Introduction}
Developing machine translation models without using bilingual parallel text is an intriguing research problem with real applications: obtaining a large volume of parallel text for many languages is hard if not impossible. Moreover, translation models could be used in downstream cross-lingual tasks in which annotated data does not exist for some languages. There has recently been a great deal of interest in unsupervised neural machine translation~(e.g.~\newcite{artetxe-etal-2018-unsupervised,lample2018unsupervised,lample-etal-2018-phrase,xlm_paper,song2019mass,kim-etal-2020-unsupervised,tae2020meta}). Unsupervised neural machine translation models often perform nearly as well as supervised models when translating between similar languages, but they fail to perform well in low-resource or distant languages~\cite{kim-etal-2020-unsupervised} or out-of-domain monolingual data~\cite{marchisio-etal-2020-unsupervised}. In practice, the highest need for unsupervised models is to expand beyond high resource, similar European language pairs.
There are two key goals in this paper: Our first goal is developing accurate translation models for low-resource distant languages \emph{without} any supervision from a supervised model or gold-standard parallel data. Our second goal is to show that our machine translation models can be directly tailored to downstream natural language processing tasks. In this paper, we showcase our claim in cross-lingual image captioning and cross-lingual transfer of dependency parsers, but this idea is applicable to a wide variety of tasks.
We present a fast and accurate approach for learning translation models using Wikipedia. Unlike \emph{unsupervised machine translation} that solely relies on raw monolingual data, we believe that we should not neglect the availability of incidental supervisions from online resources such as Wikipedia. Wikipedia contains articles in nearly 300 languages and more languages might be added in the future, including indigenous languages and dialects of different regions in the world. Different from similar recent work~\cite{schwenk2019wikimatrix}, we do not rely on any supervision from supervised translation models. Instead, we leverage the fact that many first sentences in linked Wikipedia pages are rough translations, and furthermore, many captions of the same images are similar sentences, sometimes translations. Figure~\ref{fig:wiki_image} shows a real example of a pair of linked Wikipedia pages in Arabic and English in which the titles, first sentences, and also the image captions are rough translations of each other. Our method learns a seed bilingual dictionary from a small collection of first sentence pairs, titles and captions, and then learns cross-lingual word embeddings. We make use of cross-lingual word embeddings to extract parallel sentences from Wikipedia. Our experiments show that our approach improves over strong unsupervised translation models for low-resource languages: we improve the BLEU score of English$\rightarrow$Gujarati from $0.6$ to $15.2$, and English$\rightarrow$Kazakh from $0.8$ to $12.1$. %We also show that our approach can reach a very high BLEU score of $24.4$ in Arabic-English language pair as a high-resource language pair.
\input{wiki_figure}
In the realm of downstream tasks, we show that we can easily use our translation models to generate high-quality translations of MS-COCO~\cite{chen2015microsoft} and Flickr~\cite{hodosh2013framing} datasets, and train a cross-lingual image captioning model in a multi-task pipeline paired with machine translation in which the model is initialized by the parameters from our translation model. Our results on Arabic captioning show a BLEU score of $5.72$ that is slightly better than a supervised captioning model with a BLEU score of $5.22$. As another task, in \emph{dependency parsing}, we first translate a large amount of monolingual data using our translation models and then apply transfer using the \emph{annotation projection} method~\cite{Yarowsky:2001:IMT:1072133.1072187,hwa2005bootstrapping}. Our results show that our approach performs similarly compared to using gold-standard parallel text in high-resource scenarios, and significantly better in low-resource languages. %: our parser has an unlabeled attachment score (UAS) of $46.2$ as opposed to a UAS of $21.8$ when we use gold-standard parallel text. Moreover, our performance is very close to that of a supervised parser.
%Moreover, unlike previous work~\cite{song2019mass,tran2020cross} that heavily relies on using multiple large-memory GPUs, we only use a single GPU with 11GB of RAM.
A summary of our contribution is as follows: 1) We propose a simple, fast and effective approach towards using the Wikipedia monolingual data for machine translation without any explicit supervision. Our mining algorithm easily scales on large comparable data using limited computational resources. We achieve very high BLEU scores for distant languages, especially those in which current unsupervised methods perform very poorly. 2) We propose novel methods for leveraging our current translation models in image captioning. We show that how a combination of translating caption training data, and multi-task learning with English captioning as well as translation improves the performance. Our results on Arabic shows results slightly superior to that of a supervised captioning model trained on gold-standard datasets. 3) We propose a novel modification to the annotation projection method to be able to leverage our translation models. Our results on dependency parsing performs better than previous work in most cases, and performs similarly to using gold-standard parallel datasets.
%Our code will be publicly available online.
Our translation and captioning code and models are publicly available online\footnote{Our code: \url{https://github.com/rasoolims/ImageTranslate}. Our modification to Stanza for training on partially projected trees: \url{https://github.com/rasoolims/stanza}.}.
\begin{figure}[t!]
\centering
\includegraphics[width=0.45\textwidth]{wiki.pdf}
% \begin{subfigure}{.25\textwidth}
% \centering
% % include first image
% \includegraphics[height=0.8\linewidth]{images/wiki_ar.png}
% \end{subfigure}\begin{subfigure}{.25\textwidth}
% \centering
% % include second image
% \includegraphics[height=0.8\linewidth]{images/wiki_en}
% \end{subfigure}
% ~\\~\\~
% \begin{subfigure}{.25\textwidth}
% \centering
% % include first image
% \includegraphics[height=0.8\linewidth]{images/image_ar.png}
% \end{subfigure}\begin{subfigure}{.25\textwidth}
% \centering
% % include second image
% \includegraphics[height=0.8\linewidth]{images/image_en}
% \end{subfigure}
\caption{A pair of Wikipedia documents in Arabic and English, along with a same image with two captions.}
\label{fig:wiki_image}
\vspace{-1em}
\end{figure}
\section{Background}
%In this section, we briefly describe the main concepts that we repeatedly use throughout the paper.
\paragraph{Supervised neural machine translation}
Supervised machine translation uses a parallel text ${\cal P}=\{(s_i,t_i)\}_{i=1}^{n}$ in which each sentence $s_i \in l_1$ is a translation of $t_i \in l_2$. \emph{Neural machine translation} uses sequence-to-sequence models with attention~\cite{cho-etal-2014-learning,Bahdanau2015NeuralMT,vaswani2017attention} for which the likelihood of training data is maximized by maximizing the log-likelihood of predicting each target word given its previous predicted words and source sequence:
\[
{\cal L}({\cal P}) = \sum_{i=1}^{n} \sum_{j=1}^{|t_i|} \log p(t_{i,j} | t_{i,k<j}, s_i; \theta)
\]
where $\theta$ is a collection of parameters to be learned. %The model $\theta$ can be extended to bidirectional---translating in both language directions--- as well as multilingual~\cite{firat-etal-2016-multi,johnson-etal-2017-googles,siddhant-etal-2020-leveraging,tang2020multilingual}.
\paragraph{Unsupervised neural machine translation}
Unsupervised neural machine translation does not have access to any parallel data. Instead, it tailors monolingual datasets ${\cal M}_{l_1}$ and ${\cal M}_{l_2}$ for learning multilingual language models. These language models usually mask parts of every input sentence, and try to uncover the masked words~\cite{devlin-etal-2019-bert}. The monolingual language models are used along with iterative back-translation~\cite{hoang-etal-2018-iterative} to learn unsupervised translation. An input sentence $s$ is translated to $t'$ using current model $\theta$, then the model assumes that $(t', s)$ is a gold-standard translation, and uses the same training objective as of supervised translation. %The main assumption here is that languages have distributional similarities and these similarities can be captured by pretrained multilingual language models~\cite{conneau-etal-2020-unsupervised}.
\input{ap_image}
\paragraph{Dependency parsing}
Dependency parsing algorithms capture the best scoring dependency trees for sentences among an exponential number of possible dependency trees. A valid dependency tree for a sentence $s=s_1,\ldots,s_n$ assigns heads $h_i$ for each for word $s_i$ where $1 \leq i\leq n$, $0\leq h_i\leq n$ and $h_i \neq i$. The zeroth word represents a dummy root token as an indicator for the root of the sentence. For more details about efficient parsing algorithms, we encourage the reader to see \newcite{kubler2009dependency}.
\paragraph{Annotation projection}
Annotation projection is an effective method for transferring supervised annotation from a rich-resource language to a low-resource language through translated text~\cite{Yarowsky:2001:IMT:1072133.1072187}. Having a parallel data ${\cal P} = \{(s_i, t_i)\}_{i=1}^{n}$, and supervised source annotations for source sentences $s_i$, we transfer those annotations through word translation links $0 \leq a_{i}^{(j)} \leq |t_i|$ for $1 \leq j\leq |s_i|$ where $ a_{i}^{(j)} =0$ shows a {\tt null} alignment. The alignment links are learned in an unsupervised fashion using unsupervised word alignment algorithms \cite{giza}. In dependency parsing, if $h_i = j$ and $a^{(j)} = k$ and $a^{(i)}=m$, we project a dependency $k\rightarrow m$ (i.e. $h_m=k$) to the target side. Previous work~\cite{rasooli-collins-2017-cross,rasooli-collins-2019-low} has shown that annotation projection only works when a large amount of translation data exists. In the absence of parallel data, we create artificial parallel data using our translation models. Figure~\ref{fig:annotation_projection} shows an example of annotation projection using translated text.
\begin{figure*}
\centering
%\includegraphics[width=.9\textwidth]{images/ap_example.pdf}
% \scalebox{0.65}{
% \begin{dependency}[theme = simple]
% \begin{deptext}
% These \& regulations \& detail \& various \& tasks \& provided \& by \& the \& law \& .
% \\
% \\
% \\
% \\
% Aceste \& reglementari \& detaliaza \& diverse \& sarcini \& prevazute \& de \& lege \& . \\
% \end{deptext}
% \depedge[arc angle=30]{2}{1}{det}
% \depedge[arc angle=30]{3}{2}{nsubj}
% \deproot[arc angle=30]{3}{root}
% \depedge[arc angle=30]{5}{4}{amod}
% \depedge[arc angle=30]{3}{5}{obj}
% \depedge[arc angle=30]{5}{6}{acl}
% \depedge[arc angle=30]{9}{7}{case}
% \depedge[arc angle=30]{9}{8}{det}
% \depedge[arc angle=30]{6}{9}{obl}
% \depedge[arc angle=30]{3}{10}{punct}
% \depedge[dashed,edge below,arc angle=30,color=purple]{2}{1}{det}
% \depedge[dashed,edge below,arc angle=30,color=purple]{3}{2}{nsubj}
% \deproot[dashed,edge below,arc angle=30,color=purple]{3}{root}
% \depedge[dashed,edge below,arc angle=30,color=purple]{5}{4}{amod}
% \depedge[dashed,edge below,arc angle=30,color=purple]{3}{5}{obj}
% \depedge[dashed,edge below,arc angle=30,color=purple]{5}{6}{acl}
% \depedge[dashed,edge below,arc angle=30,color=purple]{8}{7}{case}
% \depedge[dashed,edge below,arc angle=30,color=purple]{6}{8}{obl}
% \depedge[dashed,edge below,arc angle=30,color=purple]{3}{9}{obj}
% % 2-2 41 5-7 7-8 15-16 20-21 0-0 11-12 17-19 3-4 6-6 10-11 13-13 12-14 18-20 16-18 21-22 14-15 9-10 8-9
% \path[dashed,color=blue] (\wordref{1}{1}) edge (\wordref{5}{1});
% \path[dashed,color=blue] (\wordref{1}{2}) edge (\wordref{5}{2});
% \path[dashed,color=blue] (\wordref{1}{3}) edge (\wordref{5}{3});
% \path[dashed,color=blue] (\wordref{1}{4}) edge (\wordref{5}{4});
% \path[dashed,color=blue] (\wordref{1}{5}) edge (\wordref{5}{5});
% \path[dashed,color=blue] (\wordref{1}{6}) edge (\wordref{5}{6});
% \path[dashed,color=blue] (\wordref{1}{7}) edge (\wordref{5}{7});
% \path[dashed,color=blue] (\wordref{1}{9}) edge (\wordref{5}{8});
% \path[dashed,color=blue] (\wordref{1}{10}) edge (\wordref{5}{9});
% \end{dependency}
% }
% \begin{comment}
% \scalebox{0.5}{
% \begin{dependency}[theme = simple]
% \begin{deptext}
% The \& International \& Crisis \& Group \& recently \& suggested \& moving \& responsibility \& for \& pension \& to \& state \& level \& , \& to \& eliminate \& some \& of \& the \& problems \& . \\
% \\
% \\
% \\
% Grupul \& International \& de \& Criza \& a \& sugerat \& recent \& mutarea \& responsabilitatii \& pentru \& pensii \& la \& nivelul \& statului \& , \& pentru \& a \& elimina \& unele \& dintre \& probleme \& . \\
% \end{deptext}
% \depedge[arc angle=30]{4}{1}{det}
% \depedge[arc angle=30]{4}{2}{amod}
% \depedge[arc angle=30]{4}{3}{compound}
% \depedge[arc angle=30]{6}{4}{nsubj}
% \depedge[arc angle=30]{6}{5}{advmod}
% \deproot[arc angle=30]{6}{root}
% \depedge[arc angle=30]{6}{7}{xcomp}
% \depedge[arc angle=30]{7}{8}{obj}
% \depedge[arc angle=30]{10}{9}{case}
% \depedge[arc angle=30]{8}{10}{nmod}
% \depedge[arc angle=30]{13}{11}{case}
% \depedge[arc angle=30]{13}{12}{compound}
% \depedge[arc angle=30]{7}{13}{obl}
% \depedge[arc angle=30]{16}{14}{punct}
% \depedge[arc angle=30]{16}{15}{mark}
% \depedge[arc angle=30]{7}{16}{advcl}
% \depedge[arc angle=30]{16}{17}{obj}
% \depedge[arc angle=30]{20}{18}{case}
% \depedge[arc angle=30]{20}{19}{det}
% \depedge[arc angle=30]{17}{20}{nmod}
% \depedge[arc angle=30]{6}{21}{punct}
% \depedge[dashed,edge below,arc angle=30,color=purple]{6}{1}{nsubj}
% \depedge[dashed,edge below,arc angle=30,color=purple]{1}{2}{amod}
% \depedge[dashed,edge below,arc angle=30,color=purple]{1}{4}{compound}
% \depedge[dashed,edge below,arc angle=30,color=purple]{6}{7}{advmod}
% \depedge[dashed,edge below,arc angle=30,color=purple]{6}{8}{xcomp}
% \depedge[dashed,edge below,arc angle=30,color=purple]{8}{9}{obj}
% \depedge[dashed,edge below,arc angle=30,color=purple]{11}{10}{case}
% \depedge[dashed,edge below,arc angle=30,color=purple]{9}{11}{nmod}
% \depedge[dashed,edge below,arc angle=30,color=purple]{13}{12}{case}
% \depedge[dashed,edge below,arc angle=30,color=purple]{8}{13}{obl}
% \depedge[dashed,edge below,arc angle=30,color=purple]{13}{14}{compound}
% \depedge[dashed,edge below,arc angle=30,color=purple]{18}{15}{punct}
% \depedge[dashed,edge below,arc angle=30,color=purple]{18}{16}{mark}
% \depedge[dashed,edge below,arc angle=30,color=purple]{8}{18}{advcl}
% \depedge[dashed,edge below,arc angle=30,color=purple]{18}{19}{obj}
% \depedge[dashed,edge below,arc angle=30,color=purple]{21}{20}{case}
% \depedge[dashed,edge below,arc angle=30,color=purple]{19}{21}{nmod}
% \depedge[dashed,edge below,arc angle=30,color=purple]{6}{22}{punct}
% \deproot[dashed,edge below,arc angle=30,color=purple]{6}{root}
% % 2-2 4-1 5-7 7-8 15-16 20-21 0-0 11-12 17-19 3-4 6-6 10-11 13-13 12-14 18-20 16-18 21-22 14-15 9-10 8-9
% \path[dashed,color=blue] (\wordref{1}{2}) edge (\wordref{5}{2});
% \path[dashed,color=blue] (\wordref{1}{4}) edge (\wordref{5}{1});
% \path[dashed,color=blue] (\wordref{1}{5}) edge (\wordref{5}{7});
% \path[dashed,color=blue] (\wordref{1}{7}) edge (\wordref{5}{8});
% \path[dashed,color=blue] (\wordref{1}{15}) edge (\wordref{5}{16});
% \path[dashed,color=blue] (\wordref{1}{20}) edge (\wordref{5}{21});
% \path[dashed,color=blue] (\wordref{1}{11}) edge (\wordref{5}{12});
% \path[dashed,color=blue] (\wordref{1}{17}) edge (\wordref{5}{19});
% \path[dashed,color=blue] (\wordref{1}{3}) edge (\wordref{5}{4});
% \path[dashed,color=blue] (\wordref{1}{6}) edge (\wordref{5}{6});
% \path[dashed,color=blue] (\wordref{1}{10}) edge (\wordref{5}{11});
% \path[dashed,color=blue] (\wordref{1}{13}) edge (\wordref{5}{13});
% \path[dashed,color=blue] (\wordref{1}{12}) edge (\wordref{5}{14});
% \path[dashed,color=blue] (\wordref{1}{18}) edge (\wordref{5}{20});
% \path[dashed,color=blue] (\wordref{1}{16}) edge (\wordref{5}{18});
% \path[dashed,color=blue] (\wordref{1}{21}) edge (\wordref{5}{22});
% \path[dashed,color=blue] (\wordref{1}{14}) edge (\wordref{5}{15});
% \path[dashed,color=blue] (\wordref{1}{9}) edge (\wordref{5}{10});
% \path[dashed,color=blue] (\wordref{1}{8}) edge (\wordref{5}{9});
% \end{dependency}
% }
% \end{comment}
\includegraphics[width=\textwidth]{images/ap_example.pdf}
\vspace{0.0cm}
\caption[skip=0pt]{An example of annotation projection for which the source on top is a translation of the Romanian target via our \emph{wikily} translation model. The supervised source tree is projected using intersected word alignments. }%A longer example with have missing dependencies is shown in the supplementary material.}
\label{fig:annotation_projection}
\end{figure*}
\newcommand{\docs}{w}
\newcommand{\captions}{{\cal C}}
\newcommand{\images}{{\cal I}}
\newcommand{\seed}{{\cal S}}
\newcommand{\comp}{\texttt{COMP}}
\newcommand{\first}{{\cal F}}
\newcommand{\dict}{{\cal D}}
\newcommand{\bitext}{{\cal P}}
\newcommand{\titles}{{\cal T}}
\newcommand{\embed}{v}
\section{Learning Translation from Wikipedia}
The key component of our approach is to leverage the multilingual cues from linked Wikipedia pages across languages. Wikipedia is a great comparable data in which many of its pages explain entities in the world in different languages.
In most cases, first sentences define or introduce the mentioned entity in that page (e.g. Figure~\ref{fig:wiki_image}). Therefore, we observe that many first sentence pairs in linked Wikipedia documents are rough translations of each other. Moreover, captions of images in different languages are usually similar but not necessarily direct translations of each other. We leverage this information to extract many parallel sentences from Wikipedia without using any external supervision. In this section, we describe our algorithm which is briefly shown in Figure~\ref{fig:alg_wiki}.
\subsection{Data Definitions}
For languages $e$ and $f$ in which $e$ is English and $f$ is a low-resource target language of interest, there are Wikipedia documents $\docs_e= \{\docs^{(e)}_1 \ldots \docs^{(e)}_n\}$ and $\docs_f= \{\docs^{(f)}_1 \ldots \docs^{(f)}_m\}$. We refer to $\docs^{(l)}_{(i,j)}$ as the $j$th sentence in the $i$th document for language $l$. A subset of these documents are aligned (using Wikipedia \emph{languages links}). Thus we have an aligned set of document pairs in which we can easily extract many sentence pairs that are potentially translations of each other. A smaller subset $\first$ is the set of first sentences in Wikipedia $(\docs^{(e)}_{(i,1)},\docs^{(f)}_{(i',1)})$ in which documents $i$ and $i'$ are linked and their first sentence lengths are in a similar range. In addition to text content, Wikipedia has a large set of images. Each image comes along with one or more captions, sometimes in different languages. A small subset of these images have captions both in English and the target language. We refer to this set as $\captions$. We use the set of all caption pairs ($\captions$), title pairs ($\titles$), and first sentences ($\first$) as the seed parallel data: $\seed = \first \cup \captions \cup \titles$. %We use this seed parallel data to extract bilingual dictionaries, and learn cross-lingual word embeddings.
\subsection{Bilingual Dictionary Extraction and Cross-Lingual Word Embeddings}
Having the seed parallel data $\seed$, we run unsupervised word alignment~\cite{dyer-etal-2013-simple} in both English-to-target, and target-to-English directions. We use the intersected alignments to extract highly confident word-to-word connections. Finally, we pick the most frequently aligned word for each word in English as translation. This set serves as a bilingual dictionary $\dict$.
Given two monolingual trained word embeddings $\embed_e \in \mathbb{R}^{N_e\times d}$ and $\embed_f \in \mathbb{R}^{N_f\times d}$, and the extracted bilingual dictionary $\dict$, we use the method of \newcite{faruqui-dyer-2014-improving} to project these two embedding vectors to a shared cross-lingual space.\footnote{There are more recent approaches such as \cite{lample2018word}. Comparing different embedding methods is not the focus of this paper, thereby we leave further investigation to future work.} This method uses a bilingual dictionary along with canonical correlation analysis (CCA) to learn two projection matrices to map each embedding vector to a shared space $\embed'_e \in \mathbb{R}^{N_e\times d'}$ and $\embed'_f \in \mathbb{R}^{N_f\times d'}$ where $d'\leq d$.
\input{alg}
\subsection{Mining Parallel Sentences}
We use cross-lingual embedding vectors $\embed'_e \in \mathbb{R}^{N_e\times d}$ and $\embed'_f \in \mathbb{R}^{N_f\times d'}$ for calculating the cosine similarity between pairs of words. Moreover, we use the extracted bilingual dictionary to boost the accuracy of the scoring function. For a pair of sentences $(s,t)$ where $s=s_1 \ldots s_n$ and $t=t_1 \ldots t_m$, after filtering sentence pairs with different numerical values (e.g. sentences containing $2019$ in the source and $1987$ in the target), we use a modified version of cosine similarity between words:
\[
sim(s_i, t_j)=
\begin{cases}
1.0,& \text{if } (s_i,t_j)\in \dict \\
cos(s_i,t_j), & \text{otherwise}
\end{cases}
\]
Using the above definition of word similarity, we use the average-maximum similarity between pairs of sentences.
\[
score(s, t) = \frac{\sum_{i=1}^{n}\max_{j=1}^{m} sim(s_i, t_i)}{n}
\]
From a pool of candidates, we pick those pairs that have the highest score in both directions. %$(s, t)$ as translations, and add this pair to the mined parallel data. %We can then use a set of extracted parallel sentences to train a neural machine translation model.
% \begin{equation}\label{eq1}
% \begin{cases}
% t = \arg\max_{y \in {\cal G}(s)} score(s, y) \\
% s = \arg\max_{x \in {\cal G}(t)} score(x, t)
% \end{cases}
% \end{equation}
% where ${\cal G}$ is the translation candidate generation function from linked Wikipedia pages based on sentences with similar lengths.
\subsection{Leveraging Similar Languages}
In many low-resource scenarios, the number of paired documents is very small, leading to a small number and often noisy extracted parallel sentences. To alleviate this problem to some extent, we assume to have another language $g$ in which $g$ has a large lexical overlap with the target language $f$ (such as $g$=Russian and $f$=Kazakh). We assume that a parallel data exists between language $g$ and English, and we can use it both as an auxiliary parallel data in training, and also for extracting extra lexical entries for the bilingual dictionaries: as shown in Figure~\ref{fig:alg_wiki}, we supplement the extracted bilingual dictionary from seed parallel data with the bilingual dictionary extracted from related language parallel data.
\subsection{Translation Model}\label{sec:model}
We use a standard sequence-to-sequence transformer-based translation model~\cite{vaswani2017attention} with a six-layer BERT-based~\cite{devlin-etal-2019-bert} encoder-decoder architecture from HuggingFace~\cite{wolf2019huggingface} and Pytorch~\cite{paszke2019pytorch} with a shared SentencePiece~\cite{sentencepiece} vocabulary. All input and output token embeddings are summed up with the language id embedding. First tokens of every input and output sentence are shown by the language ID. Our training pipeline assumes that the encoder and decoder are shared across different languages, except that we use a separate output layer for each language in order to prevent input copying~\cite{artetxe2018unsupervised,sen-etal-2019-multilingual}. We pretrain the model on a tuple of three Wikipedia datasets for the three languages $g$, $f$, and $e$ using the MASS model~\cite{song2019mass}. The MASS model masks a contiguous span of input tokens, and recovers that span in the output sequence.
To facilitate multi-task learning with image captioning, our model has an image encoder that is used in cases of image captioning (more details in \S\ref{sec:captioning}). In other words, the decoder is shared between the translation and captioning tasks. We use the pretrained ResNet-152 model~\cite{resnet} from Pytorch to encode every input image. We extract the final layer as a $7\times 7$ grid vector ($g \in \mathbb{R}^{7 \times 7 \times d_g}$), and project it to a new space by a linear transformation ($g' \in \mathbb{R}^{49\times d_t}$), and then add location embeddings ($l \in \mathbb{R}^{49 \times d_t}$) by using entry-wise addition. Afterwards, we assume that the $49$ vectors are encoded text representations as if a sentence with $49$ words occurs. This is similar to but not exactly the same as the Virtex model~\cite{virtex}.
\subsection{Back-Translation: One-shot and Iterative}
Finally, we use the back-translation technique to improve the quality of our models. Back-translation is done by translating a large amount of monolingual text to and from the target language. The translated texts serve as noisy input text along with the monolingual data as the silver-standard translations. Previous work~\cite{sennrich-etal-2016-improving,edunov-etal-2018-understanding} has shown that back-translation is a very simple but effective technique to improve the quality of translation models. Henceforth, we refer to this method as \emph{one-shot back-translation}. Another approach is to use \emph{iterative back-translation}~\cite{hoang-etal-2018-iterative}, the most popular approach in unsupervised translation~\cite{artetxe2018unsupervised,xlm_paper,song2019mass}. The main difference from one-shot translation is that the model uses an online approach, and updates its parameters in every batch. %Thus a back-translation of a particular sentence can vary across different iterations.
We empirically find \emph{one-shot back-translation} faster to train but with much less potential to reach a high translation accuracy. A simple and effective way to have both a reliable and accurate model is to first initialize a model with one-shot back-translation, and then apply iterative back-translation. The model that is initialized with a more accurate model reaches a higher accuracy.
\section{Cross-Lingual Tasks}
In this section, we describe our approaches for tailoring our translation models to cross-lingual tasks. Note that henceforth we assume that our translations model training is finished, and we have access to trained translation models for cross-lingual tasks.
\subsection{Cross-Lingual Image Captioning}\label{sec:captioning}
Having gold-standard image captioning training data ${\cal I} = \{(I_i, c_i)\}_{i=1}^{n}$ where $I_i$ is the image as pixel values, and $c_i = c_i^{(1)}, \ldots, c_i^{k_i}$ as the textual description with $k_i$ words, our goal is to learn a captioning model that is able to describe new (unseen) images. As described in \S\ref{sec:model}, we use a transformer decoder from our translation model and a ResNet image encoder~\cite{resnet} for our image captioning pipeline. Unfortunately, annotated image captioning datasets do not exist in many languages. Having our translation model parameter $\theta^*_{\rightleftarrows}$, we can use its translation functionality to translate each caption $c_i$ to $c'_i = translate(c_i | \theta^*_{\rightleftarrows})$. Afterwards, we will have a translated annotated dataset ${\cal I'} = \{(I_i, c'_i)\}_{i=1}^{n}$ in which the textual descriptions are not gold-standard but translations from the English captions. Figure~\ref{fig:caption_example} shows a real example from MS-Coco~\cite{chen2015microsoft} in which Arabic translations are provided by our translation model. Furthermore, to augment our learning capability, we initialize our decoder with decoding parameters of $\theta^*_{\rightleftarrows}$, and also continue training with both English captioning and translation.
\begin{figure}[t!]
\centering
\includegraphics[width=0.48\textwidth]{images/cucumber_image.pdf}
\caption{An image from MS-Coco~\protect\cite{chen2015microsoft} with gold-standard English captions, and Arabic translations from our \emph{wikily} translation model.}
\label{fig:caption_example}
\end{figure}
\subsection{Cross-Lingual Dependency Parsing}
Assuming that we have a large body of monolingual text, we translate that monolingual text to create artificial parallel data. We run unsupervised word alignments on the artificial parallel text. Following previous work~\cite{rasooli-collins-2015-density,ma-xia:2014:P14-1}, we run Giza++~\cite{och2000giza} alignments on both source-to-target and target-to-source directions, and extract intersected alignments to keep high-precision one-to-one alignments. We run a supervised dependency parser of English as our rich-resource language. Then, we project dependencies to the target language sentences via word alignment links. Inspired by previous work~\cite{rasooli-collins-2015-density}, to remove noisy projections, we keep those sentences that at least $50\%$ of words or $5$ consecutive words in the target side have projected dependencies. %Selecting dense projections guarantees us that many noisy projections from sparse alignments get ignored. %Figure~\ref{fig:annotation_projection} shows such an example in which a Romanian monolingual text is first translated to English, then the English sentences are parsed and its parse tree is projected using word alignment links.
\makeatletter
\def\algbackskip{\hskip\dimexpr-\algorithmicindent+\labelsep}
\def\LState{\State \algbackskip}%
\makeatother
\makeatletter
\def\algbackskip{\hskip\dimexpr-\algorithmicindent+\labelsep}
\def\LComment{\Comment \algbackskip}%
\makeatother
\algnewcommand\algorithmicforeach{\textbf{for each}}
\algdef{S}[FOR]{ForEach}[1]{\algorithmicforeach\ #1\ \algorithmicdo}
\begin{figure}[t!]
\hrule width \hsize \kern 1mm \hrule width \hsize height 2pt
\vspace{0.1cm}
%\noindent\rule{0.5\textwidth}{0.4pt}
\begin{algorithmic}[0]
\scriptsize
\LState {{\bf Definitions: }{ 1) $e$ is English, $f$ is the foreign language, and $g$ is a language similar to $f$}, 2) $\colorbox{red!30}{learn\_dict}(P)$ extracts a bilingual dictionary from parallel data $P$, 3) $\colorbox{orange!30}{t}(x| m)$ translates input $x$ given model $m$, , 4) $\colorbox{yellow!30}{pretrain}(x)$ pretrains on monolingual data $x$ using MASS~\protect\cite{song2019mass}, 5) $\colorbox{green!30}{train}(P | m)$ trains on parallel data $P$ initialized by model $m$, 6) $\colorbox{blue!10}{bt\_train}(x_1, x_2 | m)$ trains iterative back-translation on monolingual data $x_1 \in e$ and $x_2 \in f$ initialized by model $m$. }
\LState {{\bf Inputs: }{ 1) Wikipedia documents $\docs^{(e)}$, $\docs^{(f)}$, and $\docs^{(g)}$}, 2) Monolingual word embedding vectors $\embed_e$ and $\embed_f$, 3) Set of linked pages from Wikipedia \colorbox{pink!30}{$\comp$}, their aligned titles $\titles$, and their first sentence pairs $\first$, 4) Set of paired image captions $\captions$, and 5) Gold-standard parallel data $\bitext^{(e,g)}$.}
\vspace{0.1cm}
\LState{\bf Algorithm:}
%~\\~~
\LState{\bf $\rightarrow$ Learn bilingual dictionary and embeddings}
\State $\seed = \first \cup \captions \cup \titles$
\State $\dict^{(f,e)} = \colorbox{red!30}{learn\_dict}(\seed)$
\State $\dict^{(g,e)} = \colorbox{red!30}{learn\_dict}(\bitext^{(e,g)})$ \Comment{\colorbox{gray!15}{Related language}}
\State Learn $\embed_e \rightarrow \embed'_e$ and $\embed_f\rightarrow \embed'_f$ using $\dict^{(f,e)} \cup \dict^{(g,e)}$
%~\\~~
\LState{\bf $\rightarrow$ Mine parallel data}
\State Extract comparable sentences ${\cal Z}$ from \colorbox{pink!30}{$\comp$}
\State Extract $\bitext^{(f,e)}$ from ${\cal Z}$.% using Eq.~\ref{eq1}. \Comment{\colorbox{gray!15}{Cosine sim.}}
\State $\bitext^{(f,e)} = \bitext^{(f,e)} \cup \titles$ \Comment{\colorbox{gray!15}{Mined Data}}
%~\\~~
\LState{\bf $\rightarrow$ Train MT with pretraining and back-translation}
\State $\theta_0= \colorbox{yellow!30}{pretrain}(\docs^{(e)} \cup \docs^{(f)} \cup \docs^{(g)})$ \Comment{\colorbox{gray!15}{MASS Training}}
\State $\theta_{\rightleftarrows} = \colorbox{green!30}{train}(\bitext^{(f,e)} \cup \bitext^{(g,e)} | \theta_0 )$ \Comment{\colorbox{gray!15}{NMT Training}}
\State $\bitext^{(e\rightarrow f)} = (\colorbox{orange!30}{t}(\docs^{(f)} | \theta_{\rightleftarrows}),\docs^{(f)})$
\State $\bitext^{(f\rightarrow e)} = (\colorbox{orange!30}{t}(\docs^{(e)} | \theta_{\rightleftarrows}),\docs^{(e)})$
\State $\bitext'^{(f,e)} = \bitext^{(e\rightarrow f)} \cup \bitext^{(f\rightarrow e)} \cup \bitext^{(f,e)}$
\State $\theta'_{\rightleftarrows}= \colorbox{green!30}{train}(\bitext'^{(f,e)} | \theta_0)$
\State $\theta*_{\rightleftarrows}= \colorbox{blue!10}{bt\_train}(\docs^{(e)}, \docs^{(f)}|\theta'_{\rightleftarrows})$
%~\\~~
\LState {{\bf Output: } $\theta^*_{\rightleftarrows}$}
\end{algorithmic}
%\vspace{0.2cm}
\hrule width \hsize \kern 1mm \hrule width \hsize height 2pt
\caption{A brief depiction of the training pipeline.}\label{fig:alg_wiki}
\vspace{-1em}
\end{figure}
\section{Experiments}\label{sec:expriments}
In this section, we provide details about our experimental settings and results for translation, captioning, and dependency parsing. We put more details about our settings as well as thorough analysis of our results in the supplementary material.
\subsection{Datasets and Settings}
\paragraph{Languages} We focus on four language pairs: Arabic-English, Gujarati-English, Kazakh-English, and Romanian-English. We choose these pairs to provide enough evidence that our model works in distant languages, morphologically-rich languages, as well as similar languages. As for similar languages, we use Persian for Arabic (written with very similar scripts and have many words in common), Hindi for Gujarati (similar languages), Russian for Kazakh (written with the same script), and Italian for Romanian (Romance languages).
\paragraph{Monolingual and Translation Datasets}
%We use regular expressions to tokenize sentences of Wikipedia dump text.
We use a shared SentencePiece vocabulary~\cite{sentencepiece} with size 60K. Table~\ref{tab:par_sizes} shows the sizes of Wikipedia data in different languages. For evaluation, we use the Arabic-English UN data~\cite{ziemski2016united}, WMT 2019 data~\cite{barrault-etal-2019-findings} for Gujarati-English and Kazakh-English, and WMT 2016 shared task data~\cite{bojar-etal-2016-findings} for Romanian-English. Following previous work~\cite{sennrich-etal-2016-edinburgh}, diacritics are removed from the Romanian data. More details about other datasets and their sizes, we refer the reader to the supplementary material.
% We use an off-the-shelf Indic-transliteration library\footnote{\url{https://pypi.org/project/indic-transliteration}} to convert the Devanagari script to Hindi script to make the Hindi documents look like Gujarati by removing the graphical vertical bars from Hindi letters, thus increasing the chance of capturing more words in common. We boost the Romanian, Gujarati, and Kazakh monolingual data with newstext dataset from WMT. For parallel data in similar languages, we use the Mizan parallel data for Persian~\cite{kashefi2018mizan} with one million sentences, the IITB data~\cite{kunchukuttan-etal-2018-iit} and HindiEnCorp 0.5~\cite{bojar2014hindencorp} for Hindi with a total of 367K sentences, ParaCrawl for Russian~\cite{espla-etal-2019-paracrawl} with 12M sentences, and Europarl for Italian~\cite{koehn2005europarl} with 2M sentences. We use the Arabic-English UN data~\cite{ziemski2016united}, WMT 2019 data~\cite{barrault-etal-2019-findings} for Gujarati-English and Kazakh-English, and WMT 2016 shared task data~\cite{bojar-etal-2016-findings} for Romanian-English. Following previous work~\cite{sennrich-etal-2016-edinburgh}, diacritics are removed from the Romanian data.
% \paragraph{Cross-Lingual Embedding}
% We use the off-the-shelf 300-dimensional FastText embeddings~\cite{grave-etal-2018-learning} as monolingual embedding vectors. We run FastAlign~\cite{dyer-etal-2013-simple} on the seed parallel text from both source-to-target and target-to-source directions, run alignment intersection to get intersected alignments, and extract the highest occurring alignment for every word as the dictionary entry. We make use of the cross-lingual CCA tool~\cite{faruqui-dyer-2014-improving} to extract 150-dimensional vectors. This tool can be run on a single CPU within a few hours.
\input{parallel_sizes}
\paragraph{Pretraining} We pretrain four models on 3-tuples of languages via a single NVIDIA Geforce RTX 2080 TI with 11GB of memory. We create batches of 4K words, run pretraining for two million iterations where we alternate between language batches, and accumulate gradients for 8 steps. We use the apex library\footnote{\url{https://github.com/NVIDIA/apex}} to use FP-16 tensors. This whole process takes four weeks in a single GPU. We use the Adam optimizer~\cite{adam_kingma} with inverse square root and learning rate of $10^{-4}$, $4000$ warm-up steps, and dropout probability of $0.1$. %Due to GPU memory limitation, this whole process takes about four weeks: in theory, with 8 high-memory GPUs, we could obtain higher-quality pretrained models in a few days. %We believe that using a high-memory multi-GPU setting would help, but we have not had access to them at the time of experimentation.
\paragraph{Translation Training} Table~\ref{tab:par_sizes} shows the sizes of different types of datasets in our experiments. We pick comparable candidates for sentence pairs whose lengths are within a range of half to twice of each other. As we see, the final size of mined datasets heavily depends on the number of paired English-target language Wikipedia documents. We train our translation models initialized by pretrained models. More details about our hyper-parameters are in the supplementary material. All of our evaluations are conducted using SacreBLEU~\cite{post-2018-call} except for en$\leftrightarrow$ro in which we use BLEU score~\cite{papineni2002bleu} from Moses decoder scripts~\cite{koehn2007moses} for the sake of comparison to previous work.
% Table~\ref{tab:par_sizes} shows the sizes of different types of datasets in our experiments. We pick comparable candidates for sentence pairs whose lengths are within a range of half to twice of each other. As we see, the final size of mined datasets heavily depends on the number of paired English-target language Wikipedia documents. We train our translation models initialized by pretrained models. Each batch has roughly 4K tokens. Except for Arabic, for which the size of mined data significantly outnumbers the size of Persian-English parallel data, we use the related language data before using iterative back-translation in which we only use the source and target monolingual datasets. We use similar learning hyper-parameters to pretraining except for iterative back-translation in which we accumulate gradients for 100 steps, and use a dropout probability of $0.2$ and $10000$ warmup steps since we find smaller dropout and warmup make the model diverge. Our one-shot back-translation experiments use a beam size of 4, but we use a beam size of one for iterative back-translation since we have not seen much gains in using beam-based iterative back-translation except for purely unsupervised settings. All of our translations are performed with a beam size of $4$ and $max\_len\_a=1.3$ and $max\_len\_b=5$. We alternate between supervised parallel data of a similar language paired with English and the mined data.
% We train translation models for roughly 400K batches except for Gujarati that has smaller mined data for which we train for 200K iterations. We have seen a quick divergence in Kazakh iterative back-translation, thereby we stopped it early after running it for one epoch of all monolingual data. Most likely, the mined data for Kazakh-English has lower quality (see the supplementary material for more details), and that leads to very noisy translations in back-translation outputs. All of our evaluations are conducted using SacreBLEU~\cite{post-2018-call} except for en$\leftrightarrow$ro in which we use BLEU score~\cite{papineni2002bleu} from Moses decoder scripts~\cite{koehn2007moses} for the sake of comparison to previous work. %We have also tried SacreBLEU~\cite{post-2018-call} and have seen a very slight increase in BLEU scores, but for the sake of comparison, we only report the multi-bleu scores from Moses decoder scripts.
\paragraph{Image Captioning}
We use the Flickr~\cite{hodosh2013framing} and MS-Coco~\cite{chen2015microsoft} datasets for English\footnote{We have also tried Conceptual Captions~\cite{sharma2018conceptual} in our initial experiments but we have observed drops in performance. Previous work~\cite{singh2020we} have also observed a similar problem with Conceptual Captions as a noisy crawled caption dataset.}, and the gold-standard Arabic Flickr dataset~\cite{visapp20} for evaluation. The Arabic test set has $1000$ images with $3$ captions per image. We translate all the training datasets to Arabic for having translated caption data. The final training data contains $620K$ captions for about $125K$ unique images. Throughout experiments, we use the pretrained Resnet-152 models~\cite{resnet} from Pytorch~\cite{paszke2019pytorch}, and let it fine-tune during our training pipeline. Each training batch contains $20$ images. We accumulate gradients for $16$ steps, and use a dropout of $0.1$ for the projected image output representations. Other training parameters are the same as our translation training. To make our pipeline fully unsupervised, we use translated development sets to pick the best checkpoint during training.
\paragraph{Dependency Parsing}
We use the Universal Dependencies v2.7 collection~\cite{universal2_7} for Arabic, Kazakh, and Romanian. We use the Stanza~\cite{stanza} pretrained supervised models for getting supervised parse trees for Arabic and Romanian, and use the UDPipe~\cite{straka2016udpipe} pretrained model for Kazakh. We translate about 2 million sentences from each language to English, and also 2 million English sentences to Arabic. We use a simple modification to Stanza to facilitate training on partially projected trees by masking dependency and label assignments for words with missing dependencies. All of our training on projected dependencies is blindly conducted with $100k$ training steps with default parameters of Stanza~\cite{stanza}. As for gold-standard parallel data, we use our supervised translation training data for Romanian-English and Kazakh-English and use a sample of 2 million sentences from the UN Arabic-English data due to its large size that causes word alignment significant slow-down. For Kazakh \emph{wikily} projections, due to low supervised POS accuracy, we use the projected POS tags for projected words and supervised tags for unprojected words. We observe a two percent increase in performance by using projected tags.
\input{main_results}
\input{projection_experiments}
\input{captioning_experiments}
\subsection{Translation Results}
Table~\ref{tab:main_results} shows the results of different settings in addition to baseline and state-of-the-art results. We see that Arabic as a clear exception needs more rounds of training: we train our Arabic model once again on mined data by initializing it by our back-translation model.\footnote{We have seen that during multi-tasking with image captioning, the translation BLEU score for Arabic-English significantly improves. We initially thought that multi-tasking is improving both translation and captioning, but our further investigation shows that it is actually due to lack of training for Arabic. We have tried the same procedure for other languages but have not observed any further gains.} We have not seen further improvement by back-translation. To have a fair comparison, we list the best supervised models for all language pairs (to the best of our knowledge). In low-resource settings, we outperform strong supervised models that are boosted by back-translation. In high-resource settings, our Arabic models achieve very high performance but regarding the fact that the parallel data for Arabic has 18M sentences, it is quite impossible to reach that level of accuracy.%: our Arabic Wikipedia data is much smaller than the UN parallel data.
\input{examples}
Figure~\ref{fig:output_example} shows a randomly chosen example from the Gujarati-English development data. As depicted, we see that the model after back-translation reaches to somewhat the core meaning of the sentence with a bit of divergence from exactly matching the reference. The final iterative back-translation output almost catches a correct translation. We also see that the use of the word ``creative'' is seen in Google Translate output, a model that is most likely trained on much larger parallel data than what is currently available for public use. In general, unsupervised translation performs very poorly compared to our approach in all directions.
\subsection{Captioning Results}
Table~\ref{tab:caption_results} shows the final results on the Arabic test set using the SacreBLEU measure~\cite{post-2018-call}. First, we should note that similar to \newcite{visapp20}, we see lower scales of BLEU scores due to morphological richness in Arabic. We see that if we initialize our model with the translation model and multi-task it with translation and also English captioning, we achieve much higher performance. It is interesting to observe that translating the English output on the test data to Arabic achieves a much lower result. %by translating the English output on the test data to Arabic, we achieve a much lower result.
This is a strong indicator of the strength of our approach. We also see that supervised translation fails to perform well. This might due to the UN translation training dataset which has a different domain from the caption dataset. Furthermore, we see that our model outperforms Google Translate which is a strong machine translation system, and that is actually what is being used as seed data for manual revision in the Arabic dataset. Finally, it is interesting to see that our model outperforms supervised captioning. Multi-tasking make translation performance slightly worse.
\begin{figure*}[t!]
\centering
\setlength{\tabcolsep}{0pt}
\begin{tabular}{c c}
\includegraphics[width=0.36\textwidth]{images/1490670858_e122df2560.jpg} & \includegraphics[width=0.58\textwidth]{arabic_captions.pdf} \\
\end{tabular}
\caption{An example of different outputs in our captioning experiments both for English and Arabic, as well as Arabic translations of English outputs on the Arabic Flickr dataset~\protect\cite{visapp20}.}
\label{fig:caption_outputs}
\end{figure*}
Figure~\ref{fig:caption_outputs} shows a randomly picked example with different model outputs. We see that the two outputs from our approach with multi-tasking are roughly the same but one of them as more syntactic order overlap with the reference while both orders are correct in Arabic as a free-word order language. The word \setcode{utf8}
\<برتقالية > means ``orange'' which is close to \setcode{utf8}
\<حمراء > that means ``red''. The word \setcode{utf8}
\<شريحة > means ``slide'' which is correct but other meanings of this word exist in the reference. In general, we observe that although superficially the BLEU scores for Arabic is low, it is mostly due to its lexical diversity, free-word order, and morphological complexity.
\subsection{Dependency Parsing Results}
Table~\ref{tab:projection_results} shows the results for dependency parsing experiments. We see that our model performs very high in Romanian with a UAS of $74$ which is much higher than that of \newcite{ahmad-etal-2019-difficulties} and slightly lower than that of \newcite{rasooli-collins-2019-low} which uses a combination of multi-source annotation projection and direct model transfer. Our work on Arabic outperforms all previous work and performs even better than using gold-standard parallel data. One clear highlight is our result in Kazakh. As mentioned before, by projecting the part-of-speech tags, we achieve roughly 2 percent absolute improvement. Our final results on Kazakh are significantly higher than that of using gold-standard parallel text (7K sentences).
\begin{table}[]
\centering \tiny
\setlength{\tabcolsep}{5pt}
\begin{tabular}{l|c c c c}
\hline\hline
Direction & ar$\rightleftarrows$en & gu$\rightleftarrows$en & kk$\rightleftarrows$en & ro$\rightleftarrows$en\\ \hline
Foreign docs & 1.0m & 28k & 230k & 400k \\
Paired docs & 745k & 7.3k & 80k & 270k \\
First sents. & 205k & 3.2k & 52k & 78k\\
Captions & 92k & 2.2k & 1.9k & 35k \\
Comparable pairs & 0.1b & 14m & 32m & 64m \\
Mined sents. & 1.7m & 49k & 183k & 675k\\
BT & 2.1m & 1.5m & 2.2m & 2.1m\\
Iterative BT & 4.0m & 3.8m & 4.0m & 6.1m\\
\hline\hline
\end{tabular}
\caption{Data sizes for different pairs. We use a sample of English sentences with similar sizes to each data.} %English has 6 million documents.
\label{tab:par_sizes}
\end{table}
\begin{table*}[t!]
\centering \small
\setlength{\tabcolsep}{5pt}
\begin{tabular}{l l |c c c c c c c c}
\hline\hline
& Model & ar$\rightarrow$en & en$\rightarrow$ar & gu$\rightarrow$en & en$\rightarrow$gu & kk$\rightarrow$en & en$\rightarrow$kk & ro$\rightarrow$en & en$\rightarrow$ro \\ \hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\tiny UNMT}}}
& \newcite{xlm_paper} & -- & -- & -- & -- &-- & -- & 31.8 & 33.3\\
& \newcite{song2019mass} (MASS; 8 GPUs) & -- &-- &-- & -- &-- & -- & 33.1 & 35.2\\
% & Our experiments (MASS; 1 GPU) & 10.3 & 9.3 & 0.1 & 0.3 & 2.8& 3.3 & & 21.7 \\
& Best published results & 11.0* & 9.4* & $0.6^{1}$ & $0.6^{1}$ & $2.0^{1}$ & $0.8^{1}$ &$37.6^{4}$ & $36.3^{2}$ \\ \hline
\parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{\tiny Wikily UNMT}}} & First sentences + captions + titles & 6.1 & 3.1 & 0.7 & 1.1 & 2.3 & 1.0 & 2.0 & 1.9\\
& Mined Corpora & 23.1 & 19.7 & 4.2 & 4.9 & 2.8 & 1.6 & 22.1 & 21.6 \\
& + Related Language & -- & -- & 9.1 & 7.8 & 7.3 & 2.3 & 23.2 & 21.5 \\
& + One-shot back-translation (bt-beam=4) & 23.0 & 18.8 & {\bf 13.8} & 13.9 & 7.0 & \colorbox{blue!20}{\bf 12.1} & 25.2 & 28.1 \\
& + Iterative back-translation (bt-beam=1) & 24.4 & 18.9 & 13.3 & \colorbox{blue!20}{\bf 15.2} &{\bf 9.0 }& 10.8 & 32.5 & 33.0 \\
& + Retrain on mined data & {\bf 30.6} & {\bf 23.4} & -- & -- & -- & -- & -- & -- \\\hline
& (Semi-)Supervised & 48.9* & 40.6* & $14.2^{1}$ & $4.0^{1}$ & $12.5^{1}$ & $3.1^{1}$ & $39.9^{3}$ & $38.5^{3}$ \\
\hline\hline
\end{tabular}
\caption{BLEU scores for different models. Reference results are from *: Our implementation, 1:~\protect\newcite{kim-etal-2020-unsupervised}, 2: \protect\newcite{Li2020Data-dependent}, 3: \protect\newcite{liu2020multilingual} (supervised), 4: \protect\newcite{tran2020cross} (unsupervised with mined parallel data).}
\label{tab:main_results}
\end{table*}
% \begin{table*}[t!]
% \centering \small
% \setlength{\tabcolsep}{5pt}
% \begin{tabular}{l|c c c c c c c c}
% \hline\hline
% Model & ar$\rightarrow$en & en$\rightarrow$ar & gu$\rightarrow$en & en$\rightarrow$gu & kk$\rightarrow$en & en$\rightarrow$kk & ro$\rightarrow$en & en$\rightarrow$ro \\ \hline
% Best Unsupervised & 7.6 & 7.4 & $0.6^{1}$ & $0.6^{1}$ & $2.0^{1}$ & $0.8^{1}$ &$33.1^{2}$ & $35.2^{2}$ \\ \hline
% First sentences + captions + titles & 5.0 & 2.8 & 0.7 & 0.5 & 0.2 & 0.3 & 1.6 & 1.2 \\
% Mined Corpora & 21.3 & 18.3 & 4.0 & 1.7 & 2.4 & 1.0 & 19.0 & 12.0 \\
% + Related Language & -- & -- & 9.7 & 4.2 & 6.6 & 2.3 & 21.1 & 18.0 \\
% + Back-translation & \\ \hline
% Best Supervised & TBD & TBD & $14.2^{1}$ & $4.0^{1}$ & $12.5^{1}$ & $3.1^{1}$ & $39.9^{3}$ & $38.5^{3}$ \\
% \hline\hline
% \end{tabular}
% \caption{All our models are initialized by a pretrained MASS model. Reference results from 1:~\protect\newcite{kim-etal-2020-unsupervised}, 2: \protect\newcite{song2019mass}, and 3: \protect\newcite{liu2020multilingual}.}
% \label{tab:main_results}
% \end{table*}
% MASS & 7.63 (8.06) & 7.43 (7.43) & 0.10 (0.10) & 0.04 (0.04) &\\ \hline
% Mined Corpora & 23.70 (25.00) & 19.90 (19.90) & 4.00 (4.50) & 1.72 (1.71) \\
% + Related Language & 22.38 (23.64) & 17.09 (17.09) & 9.69 (10.49) & 4.21 (4.17) \\
\begin{table*}[]
\centering \small
\setlength{\tabcolsep}{3pt}
\begin{tabular}{|l l | c| c | c c c | c c c | c c c | }
\hline
\multicolumn{2}{|c|}{\multirow{2}{*}{Method}} & \multirow{2}{*}{Version} & \multirow{2}{*}{Token and POS} & \multicolumn{3}{c|}{Arabic} & \multicolumn{3}{c|}{Kazakh} & \multicolumn{3}{c|}{Romanian} \\ \cline{5-13}
& & & & UAS & LAS& BLEX & UAS & LAS & BLEX& UAS &LAS & BLEX \\ \hline \hline
\parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{\tiny Previous}}} & \newcite{rasooli-collins-2019-low} & 2.0 & gold/supervised & 61.2 & 48.8 & -- & -- & -- & -- & 76.3 & 64.3 & -- \\
& \newcite{ahmad-etal-2019-difficulties} & 2.2 & gold & 38.1 & 28.0 & -- & -- & -- & -- & 65.1 & 54.1 & -- \\
& \newcite{kurniawan2021ppt} & 2.2 & gold & 48.3 & 29.9 & -- & -- & -- & -- & -- & -- & -- \\ \hline
\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{\tiny Projection}}} & \multirow{2}{*}{\emph{Wikily} translation} & \multirow{5}{*}{2.7} & gold & 62.5 & 50.7 & 46.3 & 46.8 & 28.5 & 25.0 & 74.1 & 57.7 & 52.6 \\
& & & supervised & 60.2 & 48.7 & 42.1 & 46.2 & 27.8 & 14.1 & 73.6 & 57.4 & 50.9 \\ \cline{2-2}\cline{4-13}
& \multirow{2}{*}{ Gold-standard Parallel data} & & gold & 61.5 & 47.3 & 42.4 & 22.2 & 9.3 & 7.9 & 75.9 & 62.4 & 57.3 \\
& & & supervised & 59.1 & 45.3 & 38.5 & 21.8 & 9.2 & 3.8 & 75.6 & 62.0 & 55.6 \\ \cline{1-2}\cline{4-13}
\multicolumn{2}{|c|}{Supervised} & & supervised & 84.2 & 79.8 & 72.7 & 48.0 & 29.8 & 13.7 & 90.8 & 86.0 & 80.0 \\ \hline \hline
\end{tabular}
\caption{Dependency parsing results on the Universal Dependencies dataset~\protect\cite{universal2_7}. Previous work has used different sub-versions of the Universal Dependencies data in which slight differences are expected.}
\label{tab:projection_results}
\end{table*}
\begin{comment}
\begin{table*}[]
\centering \tiny
\setlength{\tabcolsep}{2pt}
\begin{tabular}{|l| c| c | c c c c c | c c c c c | c c c c c | }
\hline \hline
& \multirow{2}{*}{Version} & \multirow{2}{*}{Token and POS} & \multicolumn{5}{c|}{Arabic} & \multicolumn{5}{c|}{Kazakh} & \multicolumn{5}{c|}{Romanian} \\ \cline{4-18}
& & & UAS & LAS& CLAS& MLAS & BLEX & UAS & LAS & CLAS &MLAS &BLEX& UAS &LAS & CLAS& MLAS& BLEX \\ \hline \hline
\multirow{2}{*}{AP Translation} & \multirow{5}{*}{2.7} & gold & 62.5 & 50.7 & 46.3 & 43.6 & 46.3 & 46.8 (44.8) & 28.5 (27.0) & 25.0 (24.0) & 22.7 (21.8) & 25.0 (24.0) & 74.1 & 57.7 & 52.6 & 44.9 & 52.6 \\
& & supervised & 60.2 & 48.7 & 44.1 & 38.6 & 42.1 & 46.2 (44.8) & 27.8 (27.1) & 25.3 (24.2) & 6.6 (6.3) & 14.1 (13.8) & 73.6 & 57.4 & 51.2 & 43.3 & 50.9 \\ \cline{1-1}\cline{3-18}
\multirow{2}{*}{ AP Gold} & & gold & 61.6 & 47.1 & 42.5 & 39.6 & 42.5 & 22.2 & 9.3 & 7.9 & 6.5 & 7.9 & 75.9 & 62.4 & 57.3 & 49.5 & 57.3 \\
& & supervised & 58.9 & 45.6 & 40.8 & 35.1 & 39.0 & 21.8 & 9.2 & 7.7 & 2.0 & 3.8 & 75.6 & 62.0 & 57.0 & 47.8 & 55.6 \\ \cline{1-1}\cline{3-18}
Supervised & & supervised & 84.2 & 79.8 & 76.2 & 69.2 & 72.7 & 48.0 & 29.8 & 24.3 & 11.2 & 13.7 & 90.8 & 86.0 & 81.7 & 77.9 & 80.0 \\ \hline
\end{tabular}
\caption{Caption}
\label{tab:projection_results}
\end{table*}
\end{comment}
\begin{table}[t!]
\centering \small
\setlength{\tabcolsep}{3pt}
\begin{tabular}{c c c c c c c c c c }
\hline
& \multirow{2}{*}{Supervision} & \multirow{2}{*}{Pretrained} & \multicolumn{2}{c}{Multi-task}& \multicolumn{2}{c}{BLEU} \\
& & & EN & MT & @1 & @4 \\ \hline \hline
\parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{\tiny Translate train data}}} & \emph{wikily} & \xmark & \xmark & \xmark & 33.1 & 4.57 \\
& \emph{wikily} & \cmark & \xmark & \xmark & 32.9 & 5.28 \\
& \emph{wikily} & \cmark & \cmark & \xmark & 32.8 & 4.37 \\
& \emph{wikily} & \cmark & \xmark & \cmark & 33.3 & \colorbox{green!50}{5.72} \\
& \emph{wikily} & \cmark & \cmark & \cmark & \colorbox{green!50}{36.8} & 5.60 \\
& \emph{supervised} & \cmark & \xmark & \xmark & 17.7 & 1.26 \\ \hline
\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{\tiny Translate test}}} &
& \multicolumn{3}{c}{English test performance$\rightarrow$} & 68.7 & 20.42 \\
& \emph{wikily} & \cmark & \xmark &\xmark & 30.6 & 4.20 \\
& supervised & \cmark & \xmark& \xmark& 15.8 & 0.92\\
& Google & \cmark & \xmark &\xmark & 31.8 & 5.56 \\ \hline
& \multirow{2}{*}{Gold} & \cmark & \xmark & \xmark & 33.7 & 3.76 \\
& & \cmark & \cmark & \xmark & \colorbox{blue!30}{37.9} & 5.22 \\ \hline
\end{tabular}
\caption{Image captioning results evaluated on the Arabic Flickr dataset~\protect\cite{visapp20} using SacreBLEU~\protect\cite{post-2018-call}. ``pretrained'' indicates initializing our captioning model with our translation parameters.}
\label{tab:caption_results}
\end{table}
\begin{figure*}[t!]
\centering
\includegraphics[width=0.95\textwidth]{images/example_gu.pdf}
\caption{An example of a Gujarati sentence and its outputs from different models, as well as Google Translate.}
\label{fig:output_example}
\end{figure*}
%\vspace{-.15cm}
\section{Related Work}
%In the last three years, we have seen a surge of interest in unsupervised neural machine translation~\cite{lample2018unsupervised,artetxe2018iclr,lample-etal-2018-phrase,yang-etal-2018-unsupervised,xlm_paper,song2019mass,artetxe-etal-2019-effective,wei-etal-2019-unsupervised,wu-etal-2019-extract,ren-etal-2019-explicit,xu2019polygon,sun2020self,sun2020robust,li2020reference,xu2020improving,tae2020meta,sun2020knowledge,bai2020unsupervised,marie2020iterative} including some using multi-modal models~\cite{su2019unsupervised}. Our work is based on the method of \newcite{song2019mass} with small differences in model architecture.
\newcite{kim-etal-2020-unsupervised} has shown that unsupervised translation models often fail to provide good translation systems for distant languages. Our work solves this problem by leveraging the Wikipedia data. Using pivot languages has been used in previous work~\cite{al-shedivat-parikh-2019-consistency}, as well as using related languages~\cite{zoph-etal-2016-transfer,nguyen-chiang-2017-transfer}. Our work only explores a simple idea of adding one similar language pair. Most likely, adding more language pairs and using ideas from recent work might improve the performance.
%Despite being different in details, most of unsupervised neural models have a two stage process, in which the first stage involves pretraining multilingual language models using masked language models, and the second stage involves methods such as back-translation and denoising autoencoders.
Wikipedia is an interesting dataset for solving NLP problems including machine translation~\cite{li2012wiki,patry-langlais-2011-identifying,lin2011unsupervised,tufics2013wikipedia,barron-cedeno-etal-2015-factory,wijaya2017learning,ruiter-etal-2019-self,srinivasan2021wit}. The WikiMatrix data~\cite{schwenk2019wikimatrix} is the most similar effort to ours in terms of using Wikipedia, but with using supervised translation models. Bitext mining has a longer history of research~\cite{resnik1998parallel,resnik2003web} in which most efforts are spent on using a seed supervised translation model~\cite{guo-etal-2018-effective,schwenk2019ccmatrix,artetxe-schwenk-2019-margin,schwenk2019wikimatrix,jones2021majority}. Recently, a number of papers have focused on unsupervised extraction of parallel data~\cite{ruiter-etal-2019-self,hangya-fraser-2019-unsupervised,keung2020unsupervised,tran2020cross,kuwanto2021lowresource}. \newcite{ruiter-etal-2019-self} focus on using vector similarity of sentences to extract parallel text from Wikipedia. Their work does not leverage structural signals from Wikipedia. %It is worth noting that recent work has considered using a huge number of small Bible parallel data for translation~\cite{mueller-etal-2020-analysis}: we think this line of work can be combined with ours.
Cross-lingual and unsupervised image captioning has been studied in previous work~\cite{gu2018unpaired,feng2019unsupervised,song2019unpaired,gu2019unpaired,gao2020unsupervised,burns2020learning}. Unlike previous work, we do not have a supervised translation model. Cross-lingual transfer of dependency parser have a long history. We encourage the reader to read a recent survey on this topic~\cite{das2020survey}. Our work does not use gold-standard parallel data or even supervised translation models to apply annotation projection.% but we still see that our models performs similarly or sometimes better than using gold-standard parallel text.
\vspace{-.15cm}
\section{Conclusion}\label{sec:conclude}
We have described a fast and effective algorithm for learning translation systems using Wikipedia. We show that by wisely choosing what to use as seed data, we can have very good seed parallel data to mine more parallel text from Wikipedia. We have also shown that our translation models can be used in downstream cross-lingual natural language processing tasks. In the future, we plan to extend our approach beyond Wikipedia to other comparable datasets like the BBC World Service. A clear extension of this work is to try our approach on other cross-lingual tasks. Moreover, as many captions of the same images in Wikipedia are similar sentences and sometimes translations, multimodal machine translation \cite{specia2016shared,caglayan2019probing,hewitt2018learning, yao2020multimodal} based on this data or the analysis of the data, such as whether more similar languages may share more similar captions \cite{khani2021cultural} are other interesting avenues.
\section{Cross-Lingual Embedding}
We use the off-the-shelf 300-dimensional FastText embeddings~\cite{grave-etal-2018-learning} as monolingual embedding vectors. We run FastAlign~\cite{dyer-etal-2013-simple} on the seed parallel text from both source-to-target and target-to-source directions, run alignment intersection to get intersected alignments, and extract the highest occurring alignment for every word as the dictionary entry. We make use of the cross-lingual CCA tool~\cite{faruqui-dyer-2014-improving} to extract 150-dimensional vectors. This tool can be run on a single CPU within a few hours.
\section{Monolingual and Translation Datasets}
We use an off-the-shelf Indic-transliteration library\footnote{\url{https://pypi.org/project/indic-transliteration}} to convert the Devanagari script to Hindi script to make the Hindi documents look like Gujarati by removing the graphical vertical bars from Hindi letters, thus increasing the chance of capturing more words in common. We boost the Romanian, Gujarati, and Kazakh monolingual data with newstext dataset from WMT. For parallel data in similar languages, we use the Mizan parallel data for Persian~\cite{kashefi2018mizan} with one million sentences, the IITB data~\cite{kunchukuttan-etal-2018-iit} and HindiEnCorp 0.5~\cite{bojar2014hindencorp} for Hindi with a total of 367K sentences, ParaCrawl for Russian~\cite{espla-etal-2019-paracrawl} with 12M sentences, and Europarl for Italian~\cite{koehn2005europarl} with 2M sentences. We use the Arabic-English UN data~\cite{ziemski2016united}, WMT 2019 data~\cite{barrault-etal-2019-findings} for Gujarati-English and Kazakh-English, and WMT 2016 shared task data~\cite{bojar-etal-2016-findings} for Romanian-English. Following previous work~\cite{sennrich-etal-2016-edinburgh}, diacritics are removed from the Romanian data.
% \input{examples}
\section{Translation Training Parameters}
We pick comparable candidates for sentence pairs whose lengths are within a range of half to twice of each other. As we see, the final size of mined datasets heavily depends on the number of paired English-target language Wikipedia documents. We train our translation models initialized by pretrained models. Each batch has roughly 4K tokens. Except for Arabic, for which the size of mined data significantly outnumbers the size of Persian-English parallel data, we use the related language data before using iterative back-translation in which we only use the source and target monolingual datasets. We use similar learning hyper-parameters to pretraining except for iterative back-translation in which we accumulate gradients for 100 steps, and use a dropout probability of $0.2$ and $10000$ warmup steps since we find smaller dropout and warmup make the model diverge. Our one-shot back-translation experiments use a beam size of 4, but we use a beam size of one for iterative back-translation since we have not seen much gains in using beam-based iterative back-translation except for purely unsupervised settings. All of our translations are performed with a beam size of $4$ and $max\_len\_a=1.3$ and $max\_len\_b=5$. We alternate between supervised parallel data of a similar language paired with English and the mined data.
We train translation models for roughly 400K batches except for Gujarati that has smaller mined data for which we train for 200K iterations. We have seen a quick divergence in Kazakh iterative back-translation, thereby we stopped it early after running it for one epoch of all monolingual data. Most likely, the mined data for Kazakh-English has lower quality (see the supplementary material for more details), and that leads to very noisy translations in back-translation outputs. All of our evaluations are conducted using SacreBLEU~\cite{post-2018-call} except for en$\leftrightarrow$ro in which we use BLEU score~\cite{papineni2002bleu} from Moses decoder scripts~\cite{koehn2007moses} for the sake of comparison to previous work.
\input{compare_sup}
\input{compare_pret}
\input{compare_tran}
% \begin{figure*}[t!]
% \centering
% \includegraphics[width=\textwidth]{images/ap_example.pdf}
% \caption[skip=0pt]{An example of annotation projection for which the source on top is a translation of the Romanian target via our \emph{wikily} translation model. The source side is parsed with a supervised parser and its parse tree is projected using intersected word alignments. A longer example with have missing dependencies is shown in the supplementary material.}
% \label{fig:annotation_projection}
% \end{figure*}
\input{analysis}
% \begin{table}[]
% \centering
% \small
% % \setlength{\tabcolsep}{3pt}
% \begin{tabular}{c|c | c c}
% Model & Ours & WikiMatrix$^*$ & Supervised\\ \hline
% ar$\rightarrow$en & 22.7 & 0 & 0 \\
% en$\rightarrow$ar & 19.7 & 0 & 0 \\
% kk$\rightarrow$en & 7.2 & 0 & 0 \\
% en$\rightarrow$kk & 2.2 & 0 & 0 \\
% ro$\rightarrow$en & 23.2 & 0 & 0 \\
% en$\rightarrow$ro & 21.5 & 0 & 0 \\
% \end{tabular}
% \caption{Comparing our results to supervised models. We trained our pipeline with the same pretrained models on WikiMatrix~\protect\cite{schwenk2019wikimatrix}. The Arabic results are trained on a sample of the UN parallel corpus~\protect\cite{ziemski2016united} with about 2.6 million sentences.}
% \label{tab:my_label}
% \end{table}
\begin{figure}[t!]
\centering
\scalebox{0.4}{
\pgfplotstableread[row sep=\\,col sep=&]{
direction & ours & wikimatrix & sup \\
ar$\rightarrow$en & 23.1 & 26.2 & 48.9 \\
en$\rightarrow$ar & 19.7 & 20.4 & 40.6
\\
kk$\rightarrow$en & 7.3 & 0.3 & 3.9 \\
en$\rightarrow$kk & 2.3 & 0.7 & 3.3 \\
ro$\rightarrow$en & 23.2 & 25.3 & 25.2 \\
en$\rightarrow$ro & 21.5 & 24.8 & 25.4 \\
}\mydata
\begin{tikzpicture}
\begin{axis}[
ybar,
bar width=.6cm,
width=\textwidth,
height=.5\textwidth,
legend style={at={(0.7,0.95)},
anchor=north,legend columns=-1},
symbolic x coords={ar$\rightarrow$en,en$\rightarrow$ar,kk$\rightarrow$en,en$\rightarrow$kk,ro$\rightarrow$en,en$\rightarrow$ro},
xtick=data,
nodes near coords,
nodes near coords align={vertical},
ymin=0,ymax=51,
ylabel={BLEU},
]
\addplot[black,postaction={pattern=north east lines},fill=yellow!80!red!10!] table[x=direction,y=ours]{\mydata};
\addplot[black,,postaction={pattern=north west lines},fill=green!60!blue] table[x=direction,y=wikimatrix]{\mydata};
\addplot[black,fill=blue!50!] table[x=direction,y=sup]{\mydata};
\legend{Ours, WikiMatrix, Supervised}
\end{axis}
\end{tikzpicture}
}
\caption{Results using our mined data versus WikiMatrix~\protect\cite{schwenk2019wikimatrix} and gold-standard data.}
\label{fig:compare_sup}
\end{figure}
\begin{figure}[t!]
\centering
\scalebox{0.4}{
\pgfplotstableread[row sep=\\,col sep=&]{
direction & nopret & withpret \\
ar$\rightarrow$en & 18.6 & 23.0 \\
en$\rightarrow$ar & 14.9 & 19.7 \\
gu$\rightarrow$en & 2.9 & 9.1\\
en$\rightarrow$gu & 3.6 & 7.8 \\
kk$\rightarrow$en & 2.4 & 7.3\\
en$\rightarrow$kk & 0.7 & 2.3\\
ro$\rightarrow$en & 19.5 & 23.2\\
en$\rightarrow$ro & 17.8 & 21.5 \\
}\mydata
\begin{tikzpicture}
\begin{axis}[
ybar,
bar width=.5cm,
width=\textwidth,
height=.5\textwidth,
legend style={at={(0.5,0.9)},
anchor=north,legend columns=-1},
symbolic x coords={ar$\rightarrow$en,en$\rightarrow$ar,gu$\rightarrow$en,en$\rightarrow$gu,kk$\rightarrow$en,en$\rightarrow$kk,ro$\rightarrow$en,en$\rightarrow$ro},
xtick=data,
nodes near coords,
nodes near coords align={vertical},
ymin=0,ymax=27,
ylabel={BLEU},
]
\addplot[black,postaction={pattern=north east lines},fill=yellow!80!red!10!] table[x=direction,y=nopret]{\mydata};
\addplot[black,fill=green!60!blue] table[x=direction,y=withpret]{\mydata};
\legend{No Pretraining, With Pretraining}
\end{axis}
\end{tikzpicture}
}
\caption{Results using mined data (no back-translation) with and without pretraining.}
\label{fig:compare_pret}
\end{figure}
\begin{figure}[t!]
\centering
\scalebox{0.4}{
\pgfplotstableread[row sep=\\,col sep=&]{
direction & CRISS & Ours \\
gu$\rightarrow$en & 18.0 & 13.8 \\
en$\rightarrow$gu & 16.9 & 15.2 \\
kk$\rightarrow$en & 13.2 & 9.0 \\
en$\rightarrow$kk & 4.3 & 10.8 \\
}\mydata
\begin{tikzpicture}
\begin{axis}[
ybar,
bar width=.6cm,
width=\textwidth,
height=.5\textwidth,
legend style={at={(0.7,0.95)},
anchor=north,legend columns=-1},
symbolic x coords={gu$\rightarrow$en,en$\rightarrow$gu,kk$\rightarrow$en,en$\rightarrow$kk},
xtick=data,
nodes near coords,
nodes near coords align={vertical},
ymin=0,ymax=20,
ylabel={BLEU},
]
\addplot[black,postaction={pattern=north east lines},fill=blue] table[x=direction,y=CRISS]{\mydata};
\addplot[black,,postaction={pattern=north west lines},fill=red] table[x=direction,y=Ours]{\mydata};
\legend{CRISS, Ours}
\end{axis}
\end{tikzpicture}
}
\caption{Our best results versus the \emph{supervised} model of \protect\newcite{tran2020cross}.}
\label{fig:compare_tran}
\end{figure}
%\vspace{-.2cm}
%\section{Analysis and Discussion}\label{sec:analysis}
%In this section, we provide more detailed analysis of our results by running some more experiments.
%\section{Manual Observation} Figure~\ref{fig:output_example} shows a randomly chosen example from the Gujarati-English development data. As depicted, we see that the model after back-translation reaches to somewhat the core meaning of the sentence with a bit of divergence from exactly matching the reference. The final iterative back-translation output almost catches a correct translation. We also see that the use of the word ``creative'' is seen in Google Translate output, a model that is most likely trained on much larger parallel data than what is currently available for public use. In general, unsupervised translation performs very poorly compared to our approach in all directions. %In cases of Gujarati and Kazakh, the pure unsupervised results are very low, especially for Gujarati for which we see random gibberish such as the one shown in Figure~\ref{fig:output_example} .
\section{Quality of Mined Data} The quality of parallel data matters a lot for getting high-accuracy. For example, we manually observe that the quality of mined data for all languages are very good except for Kazakh. Our hypothesis is that the Kazakh Wikipedia data is less aligned with the English content. We compare our mined data to that of the supervised mined data from WikiMatrix~\cite{schwenk2019wikimatrix} as well as gold-standard data. Figure~\ref{fig:compare_sup} shows the difference between the three datasets of three language pairs (WikiMatrix does not contain Gujarati). As we see, our data has BLEU scores near to WikiMatrix in all languages, and in the case of Kazakh, the model trained on our data performs higher than WikiMatrix. In other words, in the case of having very noisy comparable data, as is the case for Kazakh-English, our model even outperforms a contextualized supervised model. It is also interesting to see that our model outperforms the supervised model for Kazakh that has only 7.7K gold-standard training data. These are all strong evidences of the strength of our approach in truly low-resource settings.
\section{Pretraining Matters} It is a truth universally acknowledged, that a single model in possession of a small training data and high learning capacity, must be in want of a pretrained model. To prove this, we run our translation experiments with and without pretraining. In this case, all models with the same training data and parameters are equal, but some models are more equal. Figure~\ref{fig:compare_pret} shows the results on the mined data. Clearly, there is a significant gain by using pre-trained models. For Gujarati, which is our the lowest-resource language in our experiments, the distance is more notable: from BLEU score of $2.9$ to $9.0$. If we had access to a cluster of high-memory GPUs, we could potentially obtain even higher results throughout all of our experiments. Therefore, we believe that part of the blame for our results in English-Romanian is on pretraining. As we see in Figure~\ref{fig:compare_sup}, our supervised results without back-translation are also low for English-Romanian. %There has recently been critiques on the excessive resource consumption by neural models that leads to a huge amount of energy consumption, and potential environmental effects~\cite{strubell-etal-2019-energy}. We also observe this trend in recent parallel data mining approaches. We have instead proposed a simple but very effective model that does not require a lot of GPUs. Moreover, similar to many academic researchers, we are financially unable to make use of the amount of resources that some few research labs and big technology companies use.
%Due to limited access to GPUs, we leave further in-depth analysis of why this is the case to future work.
\section{Comparing to CRISS} The recent work of \newcite{tran2020cross} shows impressive gains using high-quality pretrained models and iterative parallel data mining from a larger comparable data than that of Wikipedia. Their pretrained model is trained using 256 Nvidia V100 GPUs in approximately 2.5 weeks~\cite{liu2020multilingual}. Figure~\ref{fig:compare_tran} shows that by considering all these facts, our model still outperforms their \emph{supervised} model in English-to-Kazakh with a big margin ($4.3$ cs $10.8$) and gets close to their performance in other directions. We should emphasize on the fact that \newcite{tran2020cross} explores a much bigger comparable data than ours. One clear addition to our work is exploring parallel data from other available comparable datasets. Due to limited computational resources, we skip this part but we do believe that using our current unsupervised models can help extract even more high-quality parallel data from comparable datasets, and this might lead to further gains for low-resource languages.
%\paragraph{Mining other comparable data:}
%We will release our models for future use by others.
% \section{Image Captioning Quality}
% % Figure~\ref{fig:caption_outputs} shows a randomly picked example with different model outputs. We see that the two outputs from our approach with multi-tasking are roughly the same but one of them as more syntactic order overlap with the reference while both orders are correct in Arabic as a free-word order language. The word \setcode{utf8}
% \<برتقالية > means ``orange'' which is close to \setcode{utf8}
% \<حمراء > that means ``red''. The word \setcode{utf8}
% \<شريحة > means ``slide'' which is correct but other meanings of this word exist in the reference. In general, we observe that although superficially the BLEU scores for Arabic is low, it is mostly due to its lexical diversity, free-word order, and morphological complexity.
% \section{Annotation Projection}
% Figure~\ref{fig:annotation_projection} shows an example of annotation projection for Romanian in which there are missing dependencies.
\begin{thebibliography}{94}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Ahmad et~al.(2019)Ahmad, Zhang, Ma, Hovy, Chang, and
Peng}]{ahmad-etal-2019-difficulties}
Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun
Peng. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1253} {On difficulties of
cross-lingual transfer with order differences: A case study on dependency
parsing}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 2440--2452,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Al-Shedivat and Parikh(2019)}]{al-shedivat-parikh-2019-consistency}
Maruan Al-Shedivat and Ankur Parikh. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1121} {Consistency by
agreement in zero-shot neural machine translation}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 1184--1197,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Artetxe et~al.(2018{\natexlab{a}})Artetxe, Labaka, and
Agirre}]{artetxe-etal-2018-unsupervised}
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/D18-1399} {Unsupervised
statistical machine translation}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 3632--3642, Brussels, Belgium.
Association for Computational Linguistics.
\bibitem[{Artetxe et~al.(2018{\natexlab{b}})Artetxe, Labaka, Agirre, and
Cho}]{artetxe2018unsupervised}
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho.
2018{\natexlab{b}}.
\newblock \href {https://openreview.net/forum?id=Sy2ogebAW} {Unsupervised
neural machine translation}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Artetxe and Schwenk(2019)}]{artetxe-schwenk-2019-margin}
Mikel Artetxe and Holger Schwenk. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1309} {Margin-based parallel
corpus mining with multilingual sentence embeddings}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 3197--3203, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Bahdanau et~al.(2015)Bahdanau, Cho, and
Bengio}]{Bahdanau2015NeuralMT}
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.
\newblock Neural machine translation by jointly learning to align and
translate.
\newblock \emph{CoRR}, abs/1409.0473.
\bibitem[{Barrault et~al.(2019)Barrault, Bojar, Costa-juss{\`a}, Federmann,
Fishel, Graham, Haddow, Huck, Koehn, Malmasi, Monz, M{\"u}ller, Pal, Post,
and Zampieri}]{barrault-etal-2019-findings}
Lo{\"\i}c Barrault, Ond{\v{r}}ej Bojar, Marta~R. Costa-juss{\`a}, Christian
Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp
Koehn, Shervin Malmasi, Christof Monz, Mathias M{\"u}ller, Santanu Pal, Matt
Post, and Marcos Zampieri. 2019.
\newblock \href {https://doi.org/10.18653/v1/W19-5301} {Findings of the 2019
conference on machine translation ({WMT}19)}.
\newblock In \emph{Proceedings of the Fourth Conference on Machine Translation
(Volume 2: Shared Task Papers, Day 1)}, pages 1--61, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Barr{\'o}n-Cede{\~n}o et~al.(2015)Barr{\'o}n-Cede{\~n}o,
Espa{\~n}a-Bonet, Boldoba, and M{\`a}rquez}]{barron-cedeno-etal-2015-factory}
Alberto Barr{\'o}n-Cede{\~n}o, Cristina Espa{\~n}a-Bonet, Josu Boldoba, and
Llu{\'\i}s M{\`a}rquez. 2015.
\newblock \href {https://doi.org/10.18653/v1/W15-3402} {A factory of comparable
corpora from {W}ikipedia}.
\newblock In \emph{Proceedings of the Eighth Workshop on Building and Using
Comparable Corpora}, pages 3--13, Beijing, China. Association for
Computational Linguistics.
\bibitem[{Bojar et~al.(2016)Bojar, Chatterjee, Federmann, Graham, Haddow, Huck,
Jimeno~Yepes, Koehn, Logacheva, Monz, Negri, N{\'e}v{\'e}ol, Neves, Popel,
Post, Rubino, Scarton, Specia, Turchi, Verspoor, and
Zampieri}]{bojar-etal-2016-findings}
Ond{\v{r}}ej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry
Haddow, Matthias Huck, Antonio Jimeno~Yepes, Philipp Koehn, Varvara
Logacheva, Christof Monz, Matteo Negri, Aur{\'e}lie N{\'e}v{\'e}ol, Mariana
Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia
Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.
\newblock \href {https://doi.org/10.18653/v1/W16-2301} {Findings of the 2016
conference on machine translation}.
\newblock In \emph{Proceedings of the First Conference on Machine Translation:
Volume 2, Shared Task Papers}, pages 131--198, Berlin, Germany. Association
for Computational Linguistics.
\bibitem[{Bojar et~al.(2014)Bojar, Diatka, Rychl{\`y}, Stran{\'a}k, Suchomel,
Tamchyna, and Zeman}]{bojar2014hindencorp}
Ondrej Bojar, Vojtech Diatka, Pavel Rychl{\`y}, Pavel Stran{\'a}k, V{\'\i}t
Suchomel, Ales Tamchyna, and Daniel Zeman. 2014.
\newblock Hindencorp-hindi-english and hindi-only corpus for machine
translation.
\newblock In \emph{LREC}, pages 3550--3555.
\bibitem[{Burns et~al.(2020)Burns, Kim, Wijaya, Saenko, and
Plummer}]{burns2020learning}
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan~A Plummer.
2020.
\newblock Learning to scale multilingual representations for vision-language
tasks.
\newblock In \emph{European Conference on Computer Vision}, pages 197--213.
Springer.
\bibitem[{Caglayan et~al.(2019)Caglayan, Madhyastha, Specia, and
Barrault}]{caglayan2019probing}
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Lo{\"\i}c Barrault. 2019.
\newblock Probing the need for visual context in multimodal machine
translation.
\newblock \emph{arXiv preprint arXiv:1903.08678}.
\bibitem[{Chen et~al.(2015)Chen, Fang, Lin, Vedantam, Gupta, Doll{\'a}r, and
Zitnick}]{chen2015microsoft}
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr
Doll{\'a}r, and C~Lawrence Zitnick. 2015.
\newblock Microsoft coco captions: Data collection and evaluation server.
\newblock \emph{arXiv preprint arXiv:1504.00325}.
\bibitem[{Cho et~al.(2014)Cho, van Merri{\"e}nboer, Gulcehre, Bahdanau,
Bougares, Schwenk, and Bengio}]{cho-etal-2014-learning}
Kyunghyun Cho, Bart van Merri{\"e}nboer, Caglar Gulcehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.
\newblock \href {https://doi.org/10.3115/v1/D14-1179} {Learning phrase
representations using {RNN} encoder{--}decoder for statistical machine
translation}.
\newblock In \emph{Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing ({EMNLP})}, pages 1724--1734, Doha, Qatar.
Association for Computational Linguistics.
\bibitem[{Conneau and Lample(2019)}]{xlm_paper}
Alexis Conneau and Guillaume Lample. 2019.
\newblock \href
{http://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf}
{Cross-lingual language model pretraining}.
\newblock In \emph{Advances in Neural Information Processing Systems 32}, pages
7059--7069. Curran Associates, Inc.
\bibitem[{Das and Sarkar(2020)}]{das2020survey}
Ayan Das and Sudeshna Sarkar. 2020.
\newblock A survey of the model transfer approaches to cross-lingual dependency
parsing.
\newblock \emph{ACM Transactions on Asian and Low-Resource Language Information
Processing (TALLIP)}, 19(5):1--60.
\bibitem[{Desai and Johnson(2021)}]{virtex}
Karan Desai and Justin Johnson. 2021.
\newblock {VirTex: Learning Visual Representations from Textual Annotations}.
\newblock In \emph{CVPR}.
\bibitem[{Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova}]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1423} {{BERT}: Pre-training of
deep bidirectional transformers for language understanding}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 4171--4186,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Dyer et~al.(2013)Dyer, Chahuneau, and Smith}]{dyer-etal-2013-simple}
Chris Dyer, Victor Chahuneau, and Noah~A. Smith. 2013.
\newblock \href {https://www.aclweb.org/anthology/N13-1073} {A simple, fast,
and effective reparameterization of {IBM} model 2}.
\newblock In \emph{Proceedings of the 2013 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 644--648, Atlanta, Georgia. Association for
Computational Linguistics.
\bibitem[{Edunov et~al.(2018)Edunov, Ott, Auli, and
Grangier}]{edunov-etal-2018-understanding}
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-1045} {Understanding
back-translation at scale}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 489--500, Brussels, Belgium. Association
for Computational Linguistics.
\bibitem[{ElJundi. et~al.(2020)ElJundi., Dhaybi., Mokadam., Hajj., and
Asmar.}]{visapp20}
Obeida ElJundi., Mohamad Dhaybi., Kotaiba Mokadam., Hazem Hajj., and Daniel
Asmar. 2020.
\newblock \href {https://doi.org/10.5220/0008881202330241} {Resources and
end-to-end neural network models for arabic image captioning}.
\newblock In \emph{Proceedings of the 15th International Joint Conference on
Computer Vision, Imaging and Computer Graphics Theory and Applications -
Volume 5: VISAPP,}, pages 233--241. INSTICC, SciTePress.
\bibitem[{Espl{\`a} et~al.(2019)Espl{\`a}, Forcada, Ram{\'\i}rez-S{\'a}nchez,
and Hoang}]{espla-etal-2019-paracrawl}
Miquel Espl{\`a}, Mikel Forcada, Gema Ram{\'\i}rez-S{\'a}nchez, and Hieu Hoang.
2019.
\newblock \href {https://www.aclweb.org/anthology/W19-6721} {{P}ara{C}rawl:
Web-scale parallel corpora for the languages of the {EU}}.
\newblock In \emph{Proceedings of Machine Translation Summit XVII Volume 2:
Translator, Project and User Tracks}, pages 118--119, Dublin, Ireland.
European Association for Machine Translation.
\bibitem[{Faruqui and Dyer(2014)}]{faruqui-dyer-2014-improving}
Manaal Faruqui and Chris Dyer. 2014.
\newblock \href {https://doi.org/10.3115/v1/E14-1049} {Improving vector space
word representations using multilingual correlation}.
\newblock In \emph{Proceedings of the 14th Conference of the {E}uropean Chapter
of the Association for Computational Linguistics}, pages 462--471,
Gothenburg, Sweden. Association for Computational Linguistics.
\bibitem[{Feng et~al.(2019)Feng, Ma, Liu, and Luo}]{feng2019unsupervised}
Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2019.
\newblock Unsupervised image captioning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pages 4125--4134.
\bibitem[{Gao et~al.(2020)Gao, Zhou, Yu, and Gu}]{gao2020unsupervised}
Jiahui Gao, Yi~Zhou, Philip~LH Yu, and Jiuxiang Gu. 2020.
\newblock Unsupervised cross-lingual image captioning.
\newblock \emph{arXiv preprint arXiv:2010.01288}.
\bibitem[{Grave et~al.(2018)Grave, Bojanowski, Gupta, Joulin, and
Mikolov}]{grave-etal-2018-learning}
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas
Mikolov. 2018.
\newblock \href {https://www.aclweb.org/anthology/L18-1550} {Learning word
vectors for 157 languages}.
\newblock In \emph{Proceedings of the Eleventh International Conference on
Language Resources and Evaluation ({LREC} 2018)}, Miyazaki, Japan. European
Language Resources Association (ELRA).
\bibitem[{Gu et~al.(2018)Gu, Joty, Cai, and Wang}]{gu2018unpaired}
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, and Gang Wang. 2018.
\newblock Unpaired image captioning by language pivoting.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
(ECCV)}, pages 503--519.
\bibitem[{Gu et~al.(2019)Gu, Joty, Cai, Zhao, Yang, and Wang}]{gu2019unpaired}
Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Handong Zhao, Xu~Yang, and Gang Wang.
2019.
\newblock Unpaired image captioning via scene graph alignments.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 10323--10332.
\bibitem[{Guo et~al.(2018)Guo, Shen, Yang, Ge, Cer, Hernandez~Abrego, Stevens,
Constant, Sung, Strope, and Kurzweil}]{guo-etal-2018-effective}
Mandy Guo, Qinlan Shen, Yinfei Yang, Heming Ge, Daniel Cer, Gustavo
Hernandez~Abrego, Keith Stevens, Noah Constant, Yun-Hsuan Sung, Brian Strope,
and Ray Kurzweil. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-6317} {Effective parallel
corpus mining using bilingual sentence embeddings}.
\newblock In \emph{Proceedings of the Third Conference on Machine Translation:
Research Papers}, pages 165--176, Brussels, Belgium. Association for
Computational Linguistics.
\bibitem[{Hangya and Fraser(2019)}]{hangya-fraser-2019-unsupervised}
Viktor Hangya and Alexander Fraser. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1118} {Unsupervised parallel
sentence extraction with parallel segment detection helps machine
translation}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 1224--1234, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{{He} et~al.(2016){He}, {Zhang}, {Ren}, and {Sun}}]{resnet}
K.~{He}, X.~{Zhang}, S.~{Ren}, and J.~{Sun}. 2016.
\newblock \href {https://doi.org/10.1109/CVPR.2016.90} {Deep residual learning
for image recognition}.
\newblock In \emph{2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, pages 770--778.
\bibitem[{Hewitt et~al.(2018)Hewitt, Ippolito, Callahan, Kriz, Wijaya, and
Callison-Burch}]{hewitt2018learning}
John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry~Tanti Wijaya,
and Chris Callison-Burch. 2018.
\newblock Learning translations via images with a massively multilingual image
dataset.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 2566--2576.
\bibitem[{Hoang et~al.(2018)Hoang, Koehn, Haffari, and
Cohn}]{hoang-etal-2018-iterative}
Vu~Cong~Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-2703} {Iterative
back-translation for neural machine translation}.
\newblock In \emph{Proceedings of the 2nd Workshop on Neural Machine
Translation and Generation}, pages 18--24, Melbourne, Australia. Association
for Computational Linguistics.
\bibitem[{Hodosh et~al.(2013)Hodosh, Young, and
Hockenmaier}]{hodosh2013framing}
Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013.
\newblock Framing image description as a ranking task: Data, models and
evaluation metrics.
\newblock \emph{Journal of Artificial Intelligence Research}, 47:853--899.
\bibitem[{Hwa et~al.(2005)Hwa, Resnik, Weinberg, Cabezas, and
Kolak}]{hwa2005bootstrapping}
Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005.
\newblock Bootstrapping parsers via syntactic projection across parallel texts.
\newblock \emph{Natural language engineering}, 11(03):311--325.
\bibitem[{Jones and Wijaya(2021)}]{jones2021majority}
Alex Jones and Derry~Tanti Wijaya. 2021.
\newblock \href {http://arxiv.org/abs/2103.06369} {Majority voting with
bidirectional pre-translation for bitext retrieval}.
\bibitem[{Kashefi(2018)}]{kashefi2018mizan}
Omid Kashefi. 2018.
\newblock Mizan: a large {Persian-English} parallel corpus.
\newblock \emph{arXiv preprint arXiv:1801.02107}.
\bibitem[{Keung et~al.(2020)Keung, Salazar, Lu, and
Smith}]{keung2020unsupervised}
Phillip Keung, Julian Salazar, Yichao Lu, and Noah~A Smith. 2020.
\newblock Unsupervised bitext mining and translation via self-trained
contextual embeddings.
\newblock \emph{arXiv preprint arXiv:2010.07761}.
\bibitem[{Khani et~al.(2021)Khani, Tourni, Rasooli, Callison-Burch, and
Wijaya}]{khani2021cultural}
Nikzad Khani, Isidora Tourni, Mohammad~Sadegh Rasooli, Chris Callison-Burch,
and Derry~Tanti Wijaya. 2021.
\newblock Cultural and geographical influences on image translatability of
words across languages.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 198--209.
\bibitem[{Kim et~al.(2020)Kim, Gra{\c{c}}a, and
Ney}]{kim-etal-2020-unsupervised}
Yunsu Kim, Miguel Gra{\c{c}}a, and Hermann Ney. 2020.
\newblock \href {https://www.aclweb.org/anthology/2020.eamt-1.5} {When and why
is unsupervised neural machine translation useless?}
\newblock In \emph{Proceedings of the 22nd Annual Conference of the European
Association for Machine Translation}, pages 35--44, Lisboa, Portugal.
European Association for Machine Translation.
\bibitem[{Kingma and Ba(2015)}]{adam_kingma}
Diederik~P. Kingma and Jimmy Ba. 2015.
\newblock \href {http://arxiv.org/abs/1412.6980} {Adam: {A} method for
stochastic optimization}.
\newblock In \emph{3rd International Conference on Learning Representations,
{ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings}.
\bibitem[{Koehn(2005)}]{koehn2005europarl}
Philipp Koehn. 2005.
\newblock Europarl: A parallel corpus for statistical machine translation.
\newblock In \emph{MT summit}, volume~5, pages 79--86. Citeseer.
\bibitem[{Koehn et~al.(2007)Koehn, Hoang, Birch, Callison-Burch, Federico,
Bertoldi, Cowan, Shen, Moran, Zens et~al.}]{koehn2007moses}
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello
Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, et~al. 2007.
\newblock Moses: Open source toolkit for statistical machine translation.
\newblock In \emph{Proceedings of the 45th annual meeting of the ACL on
interactive poster and demonstration sessions}, pages 177--180. Association
for Computational Linguistics.
\bibitem[{K{\"u}bler et~al.(2009)K{\"u}bler, McDonald, and
Nivre}]{kubler2009dependency}
Sandra K{\"u}bler, Ryan McDonald, and Joakim Nivre. 2009.
\newblock Dependency parsing.
\newblock \emph{Synthesis lectures on human language technologies},
1(1):1--127.
\bibitem[{Kudo and Richardson(2018)}]{sentencepiece}
Taku Kudo and John Richardson. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-2012} {{S}entence{P}iece: A
simple and language independent subword tokenizer and detokenizer for neural
text processing}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 66--71, Brussels,
Belgium. Association for Computational Linguistics.
\bibitem[{Kunchukuttan et~al.(2018)Kunchukuttan, Mehta, and
Bhattacharyya}]{kunchukuttan-etal-2018-iit}
Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018.
\newblock \href {https://www.aclweb.org/anthology/L18-1548} {The {IIT} {B}ombay
{E}nglish-{H}indi parallel corpus}.
\newblock In \emph{Proceedings of the Eleventh International Conference on
Language Resources and Evaluation ({LREC} 2018)}, Miyazaki, Japan. European
Language Resources Association (ELRA).
\bibitem[{Kurniawan et~al.(2021)Kurniawan, Frermann, Schulz, and
Cohn}]{kurniawan2021ppt}
Kemal Kurniawan, Lea Frermann, Philip Schulz, and Trevor Cohn. 2021.
\newblock Ppt: Parsimonious parser transfer for unsupervised cross-lingual
adaptation.
\newblock \emph{arXiv preprint arXiv:2101.11216}.
\bibitem[{Kuwanto et~al.(2021)Kuwanto, Akyürek, Tourni, Li, and
Wijaya}]{kuwanto2021lowresource}
Garry Kuwanto, Afra~Feyza Akyürek, Isidora~Chara Tourni, Siyang Li, and Derry
Wijaya. 2021.
\newblock \href {http://arxiv.org/abs/2103.13272} {Low-resource machine
translation for low-resource languages: Leveraging comparable data,
code-switching and compute resources}.
\bibitem[{Lample et~al.(2018{\natexlab{a}})Lample, Conneau, Denoyer, and
Ranzato}]{lample2018unsupervised}
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato.
2018{\natexlab{a}}.
\newblock \href {https://openreview.net/forum?id=rkYTTf-AZ} {Unsupervised
machine translation using monolingual corpora only}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Lample et~al.(2018{\natexlab{b}})Lample, Conneau, Ranzato, Denoyer,
and Jégou}]{lample2018word}
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and
Hervé Jégou. 2018{\natexlab{b}}.
\newblock \href {https://openreview.net/forum?id=H196sainb} {Word translation
without parallel data}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Lample et~al.(2018{\natexlab{c}})Lample, Ott, Conneau, Denoyer, and
Ranzato}]{lample-etal-2018-phrase}
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc{'}Aurelio
Ranzato. 2018{\natexlab{c}}.
\newblock \href {https://doi.org/10.18653/v1/D18-1549} {Phrase-based {\&}
neural unsupervised machine translation}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 5039--5049, Brussels, Belgium.
Association for Computational Linguistics.
\bibitem[{Li et~al.(2012)Li, Gra{\c{c}}a, and Taskar}]{li2012wiki}
Shen Li, Joao~V Gra{\c{c}}a, and Ben Taskar. 2012.
\newblock Wiki-ly supervised part-of-speech tagging.
\newblock In \emph{Proceedings of the 2012 Joint Conference on Empirical
Methods in Natural Language Processing and Computational Natural Language
Learning}, pages 1389--1398. Association for Computational Linguistics.
\bibitem[{Li et~al.(2020)Li, Wang, Chen, Utiyama, Sumita, Zhang, and
Zhao}]{Li2020Data-dependent}
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng
Zhang, and Hai Zhao. 2020.
\newblock \href {https://openreview.net/forum?id=S1efxTVYDr} {Data-dependent
gaussian prior objective for language generation}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Lin et~al.(2011)Lin, Snover, and Ji}]{lin2011unsupervised}
Wen-Pin Lin, Matthew Snover, and Heng Ji. 2011.
\newblock Unsupervised language-independent name translation mining from
wikipedia infoboxes.
\newblock In \emph{Proceedings of the First workshop on Unsupervised Learning
in NLP}, pages 43--52.
\bibitem[{Liu et~al.(2020)Liu, Gu, Goyal, Li, Edunov, Ghazvininejad, Lewis, and
Zettlemoyer}]{liu2020multilingual}
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan
Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.
\newblock Multilingual denoising pre-training for neural machine translation.
\newblock \emph{arXiv cs.CL 2001.08210}.
\bibitem[{Ma and Xia(2014)}]{ma-xia:2014:P14-1}
Xuezhe Ma and Fei Xia. 2014.
\newblock \href {http://www.aclweb.org/anthology/P14-1126} {Unsupervised
dependency parsing with transferring distribution via parallel guidance and
entropy regularization}.
\newblock In \emph{Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1337--1348,
Baltimore, Maryland. Association for Computational Linguistics.
\bibitem[{Marchisio et~al.(2020)Marchisio, Duh, and
Koehn}]{marchisio-etal-2020-unsupervised}
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020.
\newblock \href {https://www.aclweb.org/anthology/2020.wmt-1.68} {When does
unsupervised machine translation work?}
\newblock In \emph{Proceedings of the Fifth Conference on Machine Translation},
pages 571--583, Online. Association for Computational Linguistics.
\bibitem[{Nguyen and Chiang(2017)}]{nguyen-chiang-2017-transfer}
Toan~Q. Nguyen and David Chiang. 2017.
\newblock \href {https://www.aclweb.org/anthology/I17-2050} {Transfer learning
across low-resource, related languages for neural machine translation}.
\newblock In \emph{Proceedings of the Eighth International Joint Conference on
Natural Language Processing (Volume 2: Short Papers)}, pages 296--301,
Taipei, Taiwan. Asian Federation of Natural Language Processing.
\bibitem[{Och and Ney(2003{\natexlab{a}})}]{giza}
Franz~Josef Och and Hermann Ney. 2003{\natexlab{a}}.
\newblock A systematic comparison of various statistical alignment models.
\newblock \emph{Computational linguistics}, 29(1):19--51.
\bibitem[{Och and Ney(2003{\natexlab{b}})}]{och2000giza}
Franz~Josef Och and Hermann Ney. 2003{\natexlab{b}}.
\newblock A systematic comparison of various statistical alignment models.
\newblock \emph{Computational Linguistics}, 29(1):19--51.
\bibitem[{Papineni et~al.(2002)Papineni, Roukos, Ward, and
Zhu}]{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In \emph{Proceedings of the 40th annual meeting of the Association
for Computational Linguistics}, pages 311--318.
\bibitem[{Paszke et~al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga et~al.}]{paszke2019pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et~al.
2019.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock In \emph{Advances in neural information processing systems}, pages
8026--8037.
\bibitem[{Patry and Langlais(2011)}]{patry-langlais-2011-identifying}
Alexandre Patry and Philippe Langlais. 2011.
\newblock \href {https://www.aclweb.org/anthology/W11-1212} {Identifying
parallel documents from a large bilingual collection of texts: Application to
parallel article extraction in {W}ikipedia.}
\newblock In \emph{Proceedings of the 4th Workshop on Building and Using
Comparable Corpora: Comparable Corpora and the Web}, pages 87--95, Portland,
Oregon. Association for Computational Linguistics.
\bibitem[{Post(2018)}]{post-2018-call}
Matt Post. 2018.
\newblock \href {https://doi.org/10.18653/v1/W18-6319} {A call for clarity in
reporting {BLEU} scores}.
\newblock In \emph{Proceedings of the Third Conference on Machine Translation:
Research Papers}, pages 186--191, Brussels, Belgium. Association for
Computational Linguistics.
\bibitem[{Qi et~al.(2020)Qi, Zhang, Zhang, Bolton, and Manning}]{stanza}
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher~D. Manning.
2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-demos.14} {{S}tanza: A
python natural language processing toolkit for many human languages}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics: System Demonstrations}, pages 101--108,
Online. Association for Computational Linguistics.
\bibitem[{Rasooli and Collins(2015)}]{rasooli-collins-2015-density}
Mohammad~Sadegh Rasooli and Michael Collins. 2015.
\newblock \href {https://doi.org/10.18653/v1/D15-1039} {Density-driven
cross-lingual transfer of dependency parsers}.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
Natural Language Processing}, pages 328--338, Lisbon, Portugal. Association
for Computational Linguistics.
\bibitem[{Rasooli and Collins(2017)}]{rasooli-collins-2017-cross}
Mohammad~Sadegh Rasooli and Michael Collins. 2017.
\newblock \href {https://doi.org/10.1162/tacl_a_00061} {Cross-lingual syntactic
transfer with limited resources}.
\newblock \emph{Transactions of the Association for Computational Linguistics},
5:279--293.
\bibitem[{Rasooli and Collins(2019)}]{rasooli-collins-2019-low}
Mohammad~Sadegh Rasooli and Michael Collins. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1385} {Low-resource syntactic
transfer with unsupervised source reordering}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers)}, pages 3845--3856,
Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Resnik(1998)}]{resnik1998parallel}
Philip Resnik. 1998.
\newblock Parallel strands: A preliminary investigation into mining the web for
bilingual text.
\newblock In \emph{Conference of the Association for Machine Translation in the
Americas}, pages 72--82. Springer.
\bibitem[{Resnik and Smith(2003)}]{resnik2003web}
Philip Resnik and Noah~A Smith. 2003.
\newblock The web as a parallel corpus.
\newblock \emph{Computational Linguistics}, 29(3):349--380.
\bibitem[{Ruiter et~al.(2019)Ruiter, Espa{\~n}a-Bonet, and van
Genabith}]{ruiter-etal-2019-self}
Dana Ruiter, Cristina Espa{\~n}a-Bonet, and Josef van Genabith. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1178} {Self-supervised neural
machine translation}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 1828--1834, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Schwenk et~al.(2019{\natexlab{a}})Schwenk, Chaudhary, Sun, Gong, and
Guzmán}]{schwenk2019wikimatrix}
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco
Guzmán. 2019{\natexlab{a}}.
\newblock Wikimatrix: Mining 135m parallel sentences in 1620 language pairs
from wikipedia.
\newblock \emph{arXiv cs.CL 1907.05791}.
\bibitem[{Schwenk et~al.(2019{\natexlab{b}})Schwenk, Wenzek, Edunov, Grave, and
Joulin}]{schwenk2019ccmatrix}
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand
Joulin. 2019{\natexlab{b}}.
\newblock Ccmatrix: Mining billions of high-quality parallel sentences on the
web.
\newblock \emph{arXiv preprint arXiv:1911.04944}.
\bibitem[{Sen et~al.(2019)Sen, Gupta, Ekbal, and
Bhattacharyya}]{sen-etal-2019-multilingual}
Sukanta Sen, Kamal~Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1297} {Multilingual
unsupervised {NMT} using shared encoder and language-specific decoders}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 3083--3089, Florence, Italy.
Association for Computational Linguistics.
\bibitem[{Sennrich et~al.(2016{\natexlab{a}})Sennrich, Haddow, and
Birch}]{sennrich-etal-2016-edinburgh}
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016{\natexlab{a}}.
\newblock \href {https://doi.org/10.18653/v1/W16-2323} {{E}dinburgh neural
machine translation systems for {WMT} 16}.
\newblock In \emph{Proceedings of the First Conference on Machine Translation:
Volume 2, Shared Task Papers}, pages 371--376, Berlin, Germany. Association
for Computational Linguistics.
\bibitem[{Sennrich et~al.(2016{\natexlab{b}})Sennrich, Haddow, and
Birch}]{sennrich-etal-2016-improving}
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016{\natexlab{b}}.
\newblock \href {https://doi.org/10.18653/v1/P16-1009} {Improving neural
machine translation models with monolingual data}.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 86--96, Berlin,
Germany. Association for Computational Linguistics.
\bibitem[{Sharma et~al.(2018)Sharma, Ding, Goodman, and
Soricut}]{sharma2018conceptual}
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.
\newblock Conceptual captions: A cleaned, hypernymed, image alt-text dataset
for automatic image captioning.
\newblock In \emph{Proceedings of ACL}.
\bibitem[{Singh et~al.(2020)Singh, Goswami, and Parikh}]{singh2020we}
Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. 2020.
\newblock Are we pretraining it right? digging deeper into visio-linguistic
pretraining.
\newblock \emph{arXiv preprint arXiv:2004.08744}.
\bibitem[{Song et~al.(2019{\natexlab{a}})Song, Tan, Qin, Lu, and
Liu}]{song2019mass}
Kaitao Song, Xu~Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019{\natexlab{a}}.
\newblock {MASS}: Masked sequence to sequence pre-training for language
generation.
\newblock \emph{arXiv cs.CL 1905.02450}.
\bibitem[{Song et~al.(2019{\natexlab{b}})Song, Chen, Zhao, and
Jin}]{song2019unpaired}
Yuqing Song, Shizhe Chen, Yida Zhao, and Qin Jin. 2019{\natexlab{b}}.
\newblock Unpaired cross-lingual image caption generation with self-supervised
rewards.
\newblock In \emph{Proceedings of the 27th ACM International Conference on
Multimedia}, pages 784--792.
\bibitem[{Specia et~al.(2016)Specia, Frank, Sima’An, and
Elliott}]{specia2016shared}
Lucia Specia, Stella Frank, Khalil Sima’An, and Desmond Elliott. 2016.
\newblock A shared task on multimodal machine translation and crosslingual
image description.
\newblock In \emph{Proceedings of the First Conference on Machine Translation:
Volume 2, Shared Task Papers}, pages 543--553.
\bibitem[{Srinivasan et~al.(2021)Srinivasan, Raman, Chen, Bendersky, and
Najork}]{srinivasan2021wit}
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc
Najork. 2021.
\newblock Wit: Wikipedia-based image text dataset for multimodal multilingual
machine learning.
\newblock \emph{arXiv preprint arXiv:2103.01913}.
\bibitem[{Straka et~al.(2016)Straka, Hajic, and
Strakov{\'a}}]{straka2016udpipe}
Milan Straka, Jan Hajic, and Jana Strakov{\'a}. 2016.
\newblock Udpipe: trainable pipeline for processing conll-u files performing
tokenization, morphological analysis, pos tagging and parsing.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation (LREC'16)}, pages 4290--4297.
\bibitem[{Tae et~al.(2020)Tae, Park, Kim, Yang, Khan, Park, Qin, and
Choo}]{tae2020meta}
Yunwon Tae, Cheonbok Park, Taehee Kim, Soyoung Yang, Mohammad~Azam Khan,
Eunjeong Park, Tao Qin, and Jaegul Choo. 2020.
\newblock Meta-learning for low-resource unsupervised neural machine
translation.
\newblock \emph{arXiv preprint arXiv:2010.09046}.
\bibitem[{Tran et~al.(2020)Tran, Tang, Li, and Gu}]{tran2020cross}
Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020.
\newblock Cross-lingual retrieval for iterative self-supervised training.
\newblock \emph{Advances in Neural Information Processing Systems}, 33.
\bibitem[{Tufi{\c{s}} et~al.(2013)Tufi{\c{s}}, Ion, Dumitrescu, and
Stefanescu}]{tufics2013wikipedia}
Dan Tufi{\c{s}}, Radu Ion, Stefan~Daniel Dumitrescu, and Dan Stefanescu. 2013.
\newblock Wikipedia as an {SMT} training corpus.
\newblock In \emph{Proceedings of the International Conference Recent Advances
in Natural Language Processing RANLP 2013}, pages 702--709.
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin. 2017.
\newblock Attention is all you need.
\newblock In \emph{Advances in neural information processing systems}, pages
5998--6008.
\bibitem[{Wijaya et~al.(2017)Wijaya, Callahan, Hewitt, Gao, Ling, Apidianaki,
and Callison-Burch}]{wijaya2017learning}
Derry~Tanti Wijaya, Brendan Callahan, John Hewitt, Jie Gao, Xiao Ling, Marianna
Apidianaki, and Chris Callison-Burch. 2017.
\newblock Learning translations via matrix completion.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing}, pages 1452--1463.
\bibitem[{Wolf et~al.(2019)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz et~al.}]{wolf2019huggingface}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, R{\'e}mi Louf, Morgan Funtowicz,
et~al. 2019.
\newblock Huggingface's transformers: State-of-the-art natural language
processing.
\newblock \emph{ArXiv}, pages arXiv--1910.
\bibitem[{Yao and Wan(2020)}]{yao2020multimodal}
Shaowei Yao and Xiaojun Wan. 2020.
\newblock Multimodal transformer for multimodal machine translation.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 4346--4350.
\bibitem[{Yarowsky et~al.(2001)Yarowsky, Ngai, and
Wicentowski}]{Yarowsky:2001:IMT:1072133.1072187}
David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001.
\newblock \href {https://doi.org/10.3115/1072133.1072187} {Inducing
multilingual text analysis tools via robust projection across aligned
corpora}.
\newblock In \emph{Proceedings of the First International Conference on Human
Language Technology Research}, HLT '01, pages 1--8, Stroudsburg, PA, USA.
Association for Computational Linguistics.
\bibitem[{Zeman et~al.(2020)Zeman, Nivre et~al.}]{universal2_7}
Daniel Zeman, Joakim Nivre, et~al. 2020.
\newblock \href {http://hdl.handle.net/11234/1-3424} {Universal dependencies
2.7}.
\newblock {LINDAT}/{CLARIAH}-{CZ} digital library at the Institute of Formal
and Applied Linguistics ({{\'U}FAL}), Faculty of Mathematics and Physics,
Charles University.
\bibitem[{Ziemski et~al.(2016)Ziemski, Junczys-Dowmunt, and
Pouliquen}]{ziemski2016united}
Micha{\l} Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.
\newblock The {United Nations} parallel corpus v1.0.
\newblock In \emph{Proceedings of the Tenth International Conference on
Language Resources and Evaluation (LREC'16)}, pages 3530--3534.
\bibitem[{Zoph et~al.(2016)Zoph, Yuret, May, and
Knight}]{zoph-etal-2016-transfer}
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.
\newblock \href {https://doi.org/10.18653/v1/D16-1163} {Transfer learning for
low-resource neural machine translation}.
\newblock In \emph{Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing}, pages 1568--1575, Austin, Texas. Association
for Computational Linguistics.
\end{thebibliography}