Overcoming Vocabulary Constraints with Pixel-level Fallback
Extracted source text · arXiv 2504.02122
\documentclass{article} %
\usepackage[final]{colm2025_conference}
\usepackage{subcaption}
\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{lineno}
\usepackage{times}
\usepackage{latexsym}
\usepackage[utf8]{inputenc}
\usepackage{tabularx}
\usepackage{wrapfig}
\usepackage{xspace}
\usepackage{multirow}
\usepackage{xspace}
\usepackage{tablefootnote}
\usepackage{mathtools}
\usepackage{algorithm}
\usepackage{inconsolata}
\usepackage{microtype}
\usepackage{euflag}
\usepackage{wrapfig} %
\usepackage{pifont}%
\usepackage{color,soul}
\usepackage{xcolor}
\usepackage{dirtytalk}
\usepackage[hang,flushmargin]{footmisc}
\usepackage{comment}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{lmodern}
\usepackage{colortbl}
\usepackage{siunitx}
\usepackage{physics}
\AtBeginDocument{\RenewCommandCopy\qty\SI}
\colorlet{tableheadcolor}{gray!75} %
\newcommand{\headcol}{\rowcolor{tableheadcolor}} %
\colorlet{tablerowcolor}{gray!10} %
\newcommand{\rowcol}{\rowcolor{tablerowcolor}} %
\newcommand{\tc}{\textcolor{tableheadcolor}} %
\newcommand{\myul}[2][black]{\setulcolor{#1}\ul{#2}\setulcolor{black}}
\newcommand{\ots}{\textsc{ots}\xspace}
\newcommand{\nohn}{\textsc{base}\xspace}
\newcommand{\vfes}{fallback networks\xspace}
\newcommand{\vfe}{fallback network\xspace}
\newcommand{\vexp}{\textsc{vocab+}\xspace}
\newcommand{\hien}{\textsc{hi}$\rightarrow$\textsc{en}\xspace}
\newcommand{\uken}{\textsc{uk}$\rightarrow$\textsc{en}}
\newcommand{\esen}{\textsc{es}$\rightarrow$\textsc{en}\xspace}
\newcommand{\then}{\textsc{th}$\rightarrow$\textsc{en}\xspace}
\newcommand{\ruen}{\textsc{ru}$\rightarrow$\textsc{en}\xspace}
\newcommand*{\mystrut}{\rule[-0.0\baselineskip]{0pt}{0.00\baselineskip}}
\newcommand*{\patch}[1]{\framebox{\mystrut \texttt{#1}\vphantom{Hq}}}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}
\author{Jonas F. Lotz\thanks{Work done during an internship at Apple.} \\
{University of Copenhagen, Denmark \&} \\
{ROCKWOOL Foundation Research Unit} \\
\texttt{jonasf.lotz@di.ku.dk} \\
\And
Hendra Setiawan \& Stephan Peitz \\
Apple \\
\texttt{\{hendra, speitz\}@apple.com} \\
\AND
Yova Kementchedjhieva \\
MBZUAI, UAE \\
\texttt{yova.kementchedjhieva@mbzuai.ac.ae}
}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\title{Overcoming Vocabulary Constraints with Pixel-level Fallback}
\begin{document}
\ifcolmsubmission
\linenumbers
\fi
\maketitle
\begin{abstract}
Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized during training.
We propose to augment pretrained language models with a vocabulary-free encoder that generates input embeddings from text rendered as pixels.
Through experiments on English-centric language models, we demonstrate that our approach substantially improves machine translation performance
and facilitates effective cross-lingual transfer, outperforming tokenizer-based methods.
Furthermore, we find that pixel-based representations outperform byte-level approaches and standard vocabulary expansion.
Our approach enhances the multilingual capabilities of monolingual language models without extensive retraining and reduces decoding latency via input compression.
\end{abstract}
\section{Introduction}
Subword tokenization is an intrinsic part of the modern language modeling pipeline \citep{schuster-japanese-2012,sennrich-etal-2016-neural,kudo-2018-subword}.
Tokenizers are trained to strike a balance between computational efficiency and vocabulary coverage.
While larger tokenizer vocabularies offer better input coverage,
the expanded embedding matrix significantly increases resource requirements.
Consequently, language models typically adopt a moderate-sized vocabulary optimized for representational efficiency on the training corpus.
Byte-level BPE \citep{wang2019neuralmachinetranslationbytelevel,radford2019language} addresses the open vocabulary-problem,
allowing, in principle, for the processing of any text without loss of information.
However, fine-grained tokenization, down to the level of bytes,
can lead to suboptimal performance, a problem particularly pronounced for languages and scripts that are underrepresented or absent from the training data \citep{muller-etal-2021-unseen,rust-etal-2021-good,pfeiffer-etal-2021-unks}.
The effectiveness of most large language models is constrained to English and a few high-resource languages
\citep{touvron2023llama2openfoundation,jiang2023mistral7b,gemmateam2024gemmaopenmodelsbased},
limiting the benefits of modern language technology for millions of users worldwide \citep{van-esch-etal-2022-writing}. Meanwhile, English-centric language models possess latent linguistic capabilities applicable across languages \citep{brinkmann2025largelanguagemodelsshare}.
A viable alternative to costly training on massive, multilingual data
is thus to adapt pretrained English-centric models to new languages, leveraging their knowledge and capabilities \citep{peters-etal-2019-tune}.
Various approaches have been explored to extend
language models to new languages and scripts, each with its drawbacks.
\emph{Vocabulary expansion} requires additional training to align new tokens with existing parameters \citep{wang-etal-2020-extending,chau-etal-2020-parsing,lin2024mala500massivelanguageadaptation}, potentially at the cost of catastrophic forgetting \citep{MCCLOSKEY1989109}, especially after post-training steps such as supervised fine-tuning (SFT) or direct preference optimization (DPO).
\emph{Adapter modules} do not address the issue of suboptimal tokenization \citep{pfeiffer-etal-2020-mad, pfeiffer-etal-2021-unks, ansell-etal-2022-composable}.
Finally, \emph{transliteration}
sacrifices the original representation and relies on heuristics which may not be available for all languages \citep{durrani-etal-2014-integrating,muller-etal-2021-unseen,j-etal-2024-romansetu}. All of these methods operate within the vocabulary-based framework and as such remain limited by its constraints.
\begin{figure*}[t!]
\centering
\begin{subfigure}[b]{0.64\textwidth}
\centering
\includegraphics[width=0.99\textwidth]{figures/fig1a.pdf}
\caption{Proposed pipeline.
}
\end{subfigure}
\quad
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=0.99\textwidth]{figures/fig1b.pdf}
\caption{Pixel-based fallback network.}
\end{subfigure}
\caption{
Illustration of our proposed NLP pipeline for Hindi-to-English machine translation. The decoder-only language model is instructed,
encodes the source text using the \vfe, and autoregressively generates an English translation (left).
Inside the \vfe the text is segmented into a list of words, rendered
into image patches containing character bigrams, and projected into patch embeddings $\vb{z}_{i,j}$.
The encoder outputs single-vector word representations $\vb{y}_i$, mapped as input embeddings to the language model (right).
}
\label{fig:architecture}
\end{figure*}
We therefore propose augmenting the language modeling pipeline with a \textit{\vfe}, which maps inputs suboptimally covered by the vocabulary directly into the embedding space of the language model \citep{pinter-etal-2017-mimicking, schick-schutze-2019-attentive}, circumventing the tokenizer.
We base our \vfe on the demonstrated effectiveness of pixel-based language encoding for vocabulary-free modeling where text is rendered to an image \citep{salesky-etal-2021-robust, rust2023language, lotz-etal-2023-text}.
Unlike recent approaches focusing on vocabulary embeddings \citep{gee-etal-2022-fast, dobler-de-melo-2023-focus, liu-etal-2024-ofa}, the \vfe
does not depend on complex heuristics or model-specific information.
It is language-agnostic by design, and can be trained end-to-end jointly with any language model.
Since the \vfe exclusively improves input representations without modifying the vocabulary or output generation, we evaluate its effectiveness across tasks involving inputs in unseen scripts.
We find that pixel-based \vfes allow a 360M-parameter language model to exceed the performance of a 1.7B-parameter baseline and similarly push the 1.7B model beyond a 3.8B one.
When trained on identical data, our pixel-based \vfe consistently outperforms standard vocabulary expansion and a byte-based fallback network.
Additionally, the \vfe reduces inference time by up to $4\times$, particularly for larger language models and on languages prone to over-segmentation, by compressing input sequences.
Strong transfer effects across visually similar scripts further emphasize the potential of pixel-based \vfes for low-resource language modeling.
\section{Proposed Approach}
We propose to replace conventional input tokenization for unseen scripts with input embeddings generated by an external fallback network.
Figure \ref{fig:architecture} exemplifies the proposed modeling pipeline in the context of machine translation with a decoder-only model.
First, the language model is instructed with a prompt, which is embedded using the model's vocabulary.
Next, the source text is rendered to an image and encoded by the \vfe.
The concatenated representations from both the vocabulary and the \vfe are then passed to the decoder, which autoregressively predicts the English translation of the source text.
Although our primary focus is on decoder-only architectures, we also evaluate \vfes for encoder-only models, following the same logic of mapping inputs into the embedding space of the language model.
Importantly, our approach treats the image-encoded source text the same as text embeddings, without converting it into discrete tokens \citep{rolfe2017discrete, vandenoord2017neuraldiscrete, yu2024titok} or connecting the image encoder and the text decoder via layers of cross-attention \citep{alayrac2022flamingo, li2023trocr, aria}.
\subsection{Fallback Network: A Vocabulary-free Encoder}
\label{sec:hypernetwork}
Our \vfe is based on an encoder architecture that extends the Vision Transformer \citep[ViT;][]{dosovitskiy2021an} to text rendered as images, similar to PIXEL \citep{rust2023language}.
Following ViT, the rendered image is split into patches $\vb{x} \in \mathbb{R}^{N \times \left(P^2 \cdot \, C \right)}$, where $N$ is the number of patches, $(P,P)$ is the resolution per patch, and $C$ is the number of channels.
These image patches are then linearly projected into patch embeddings $\vb{z} = \vb{x}\mathbf{E} + \mathbf{E}_{pos}$, where $\mathbf{E} \in \mathbb{R}^{\left( P^2 \cdot \, C \right) \times d}$ is a 2D-convolutional layer with kernel size and stride of size $P$, $d$ is the latent dimension size, and $\mathbf{E}_{pos} \in \mathbb{R}^{N \times d}$ are positional embeddings.
Because inputs are linear sequences of patches rather than full 2D grids, we encode only horizontal (1D) positional information.
Finally, the patch embeddings are processed through a stack of Transformer layers \citep{NIPS2017_vaswani_attention}.
A final linear layer projects the average over patch encodings
from $d$ to the dimension of the language model input embeddings.
The fallback network is designed to function similarly to a vocabulary lookup, providing non-contextual embeddings which the language model can later contextualize. Specifically, we (1) pretokenize inputs into words,\footnote{Splitting on whitespace is one simple \textit{pretokenization} strategy; for languages without clear word boundaries, more appropriate segmentation methods can be utilized.} (2) encode words independently of one another, and (3) apply average pooling over the patch encodings corresponding to a word to obtain a single word-level representation $\vb{y}_i \in \mathbb{R}^d$. Two key adjustments enable the efficient handling of multiple rendered words in a single forward pass: we concatenate the patches of individual words into a single sequence, resetting positional embeddings at each word boundary; and we restrict attention so that patches only attend to other patches within the same word.
\paragraph{Text Compression}
Average-pooling the encoder representations leads to
improved downstream efficiency by compressing subword-level information into a single embedding vector, shortening the input sequences provided to the language model.
This advantage is particularly pronounced for non-Latin scripts prone to over-segmentation with an English-centric tokenizer.
This compression effectively increases the amount of content that can fit within a language model’s fixed context window.
\paragraph{Interleaving Text and Image Representations}
The flexibility of our method allows words from the input text to be selectively embedded via the vocabulary or encoded as visual representations.
For instance, non-Latin segments can be passed to the \vfe, while Latin (ASCII) segments go through the tokenizer.
This selective encoding enables the language model to process only those parts of the input that align with its pretrained vocabulary, delegating more complex segments to the \vfe.
We hypothesize that interleaving modalities within sentences is particularly advantageous for tasks involving \emph{code-switching},
where a monolingual tokenizer may suboptimally represent parts of the input that the \vfe can be trained to handle.
\section{Experiments with Decoder-only Models}
\label{sec:decoder-experiments}
To demonstrate the efficacy of our proposed \vfe, we focus on the task of machine translation from languages written in non-Latin scripts into English.
Since English-centric models handle English generation reliably, this setup isolates the impact of improved input representation on the downstream task.
We conduct experiments using three decoder-only language models, namely SmolLM2-360M, SmolLM2-1.7B, and Phi-3-mini (3.8B parameters).
These models are all based on the same underlying architecture \citep{touvron2023llama2openfoundation} and finetuned for chat applications.
SmolLM2 models have a vocabulary size of 49,152, whereas Phi-3-mini has 32,064 tokens.
The linguistic capacity of all three models is mostly restricted to English text \citep{allal2024SmolLM2, abdin2024phi3technicalreporthighly}.
We follow the language models' default chat template.
\subsection{Data and Experimental Setup}
\label{sec:setup}
We train the models on parallel data from the OPUS corpus \citep{TIEDEMANN_OPUS2012} and evaluate them on the FLORES+ benchmark \citep{nllb-22}.
Specifically, we consider translations into English from Hindi (\textsc{hi}), Russian (\textsc{ru}), Spanish (\textsc{es}), Thai (\textsc{th}), and Ukrainian (\textsc{uk}).\footnote{We word-tokenize Thai with DeepCut \citep{Kittinaradorn2019} for \vfe modeling.}
Additional details are provided in Table~\ref{tab:languages} (Appendix \ref{app:details}).
Translation quality is measured using \textsc{chrF++} \citep{popovic-2015-chrf},
a character $n$-gram $F$-score incorporating word unigrams and bigrams of the hypothesis with respect to the reference translation.
\textsc{chrF++} is the standard primary metric for assessing performance on FLORES benchmarks \citep{goyal-etal-2022-flores, nllb-22, nllb_nature}.
We render input text as images using the PangoCairo rendering software,\footnote{
\url{https://docs.gtk.org/PangoCairo}}
segmenting each word into patches containing character bigrams, following \citet{lotz-etal-2023-text}.
Based on preliminary experiments, we apply a sliding window with one-character overlap between patches, analogous to overlapping frames in speech modeling.
For instance, the word \textit{Happy} is segmented into patches of:
\patch{Ha}, \patch{ap}, \patch{pp}, and \patch{py}.\footnote{Not illustrated in Figure \ref{fig:architecture} for simplicity.}
We use the Google Noto font family for comprehensive script coverage.\footnote{\url{https://fonts.google.com/noto}}
Following \citet{salesky-etal-2023-multilingual}, each patch is rendered as a $24 \times 24$ pixel image at 120 DPI with a font size of 10.
The maximum sequence length of the \vfe is 529 image patches.
We constrain the \vfe{} to fewer than 100M parameters, approximately matching the embedding layer of SmolLM2-1.7B and Phi-3-mini.
Based on preliminary experiments, we select a 92M-parameter configuration with $n_{\text{layers}}=4$, $d_{\text{model}}=1536$, and $n_{\text{heads}}=16$.
Section \ref{sec:ablation} explores alternative \vfe configurations.
Following the standard pretrain-then-finetune paradigm \citep{Li2020Rethinking},
training proceeds in two stages:
first, we pretrain the randomly initialized \vfe while freezing the language model, aligning the \vfe features to the language model \citep{peters-etal-2019-tune,kumar2022finetuning,ren2023how};
next, we perform joint finetuning on the downstream task.
During finetuning, we apply parameter-efficient updates using Weight-Decomposed Low-Rank Adaptation \citep[DoRA;][]{liu2024dora}, employing reduced rank for the decoder and full rank for the \vfe{}.
The learning rate is linearly warmed up to \num{3e-4} during the first 10\% of training, followed by cosine decay to \num{3e-5}.
Additional experimental details are provided in Table \ref{tab:details} (Appendix \ref{app:details}).
Results for all experiments are averaged over three runs.
Standard deviations are reported
in Appendix \ref{app:std}.
\subsection{Competing Methods}
We evaluate the pixel-based \vfe (\textsc{pixels}) against default model tokenization (\nohn), vocabulary expansion (\vexp), and a byte-based \vfe (\textsc{bytes}).
\paragraph{Vocabulary Expansion}
To improve the language coverage of the language model, we train a new tokenizer and merge it into the original one, $\mathcal{V}_{\textsc{+}} = \mathcal{V}_{\textsc{base}} \cup \mathcal{V}_{new}$.
Specifically, we train another byte-level BPE tokenizer with a vocabulary size of 32k on either Hindi, Russian, or Thai.
This results in expanded vocabulary sizes falling between the typical 30k-60k range of monolingual models
\citep{brown-et-al-2020-gpt3, touvron2023llamaopenefficientfoundation}
and the 100k+ token range of multilingual models \citep{workshop2023bloom176bparameteropenaccessmultilingual, palm, dubey2024llama3herdmodels}.
This adds approximately 25M parameters to SmolLM2-360M, 50M parameters to SmolLM2-1.7B, and 90M parameters to Phi-3-mini.
Following common practice, we randomly initialize the new vocabulary embeddings
\citep{choi-etal-2024-optimizing,yamaguchi2024effectivelyexpandvocabularyllms}.\footnote{Mean instead of random initialization for vocabulary expansion does not change the conclusions.}
Training is done in two stages, with the new embeddings being pretrained in a first stage, followed by a stage of model finetuning, for a fair comparison to the \vfe.
\paragraph{Byte-based Fallback Network}
Vocabulary-free modeling can alternatively be achieved by representing text at the byte level \citep{xue-etal-2022-byt5,yu2023megabyte,kallini2025mrt},
decomposing inputs into a discrete set of 256 embeddings.
Unlike byte-level BPE, which uses byte sequences as subword units,
treating text atomically as individual bytes enables complete vocabulary coverage without a large embedding matrix.
However, byte-based modeling significantly increases sequence lengths, as each character may require multiple bytes depending on its Unicode encoding
\citep{libovický2022dontpeopleusecharacterlevel}.
For instance, the source text shown in Figure \ref{fig:architecture} occupies six image patches but requires 59 bytes to represent.
For byte-based fallback encoding, the maximum sequence length of the \vfe is therefore extended to 2048 bytes,
significantly increasing GPU memory requirements.
To compare pixels to bytes as basis for vocabulary-free encoding,
we train parallel fallback networks differing only in input modality and corresponding embedding layers.\footnote{The embedding layer within the \vfe comprises 13M parameters for pixel-based encoding and 11M parameters for byte-based encoding.}
Conceptually, this sets up a key trade-off for the \vfe: byte-level inputs yield longer sequences drawn from a discrete input space, whereas pixel-based inputs produce shorter sequences characterized by a continuous representation.
This comparison also quantifies the benefit to the language model derived from the added encoder capacity of the \vfe.
\subsection{Machine Translation Results}
\label{sec:end-to-end}
\begin{table}[t]
\centering
\setlength\tabcolsep{3pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
& \multicolumn{4}{c}{\hien} & \multicolumn{4}{c}{\ruen} & \multicolumn{4}{c}{\then} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels} \\
\midrule
SmolLM2-360M
& $53.2$ & $48.3$ & $53.2$ & $\bm{56.8}$
& $53.9$ & $53.0$ & $55.0$ & $\bm{56.0}$
& $36.5$ & $34.8$ & $46.9$ & $\bm{48.6}$ \\
SmolLM2-1.7B
& $56.8$ & $54.4$ & $57.6$ & $\bm{59.0}$
& $57.0$ & $56.7$ & $57.4$ & $\bm{57.8}$
& $40.4$ & $39.4$ & $50.2$ & $\bm{52.1}$ \\
Phi-3-mini
& $57.3$ & $54.7$ & $59.5$ & $\bm{60.9}$
& $57.9$ & $57.8$ & $57.8$ & $\bm{58.2}$
& $51.1$ & $50.4$ & $52.0$ & $\bm{53.1}$ \\
\bottomrule
\end{tabular}%
}
\caption{
\textsc{chrF++} scores for \textsc{xx}$\rightarrow$\textsc{en} translation
after finetuning for one epoch.
}
\label{tab:end-to-end}
\end{table}
Translation performances after one epoch of pretraining and finetuning are shown in Table \ref{tab:end-to-end}.
We observe that pixel-based representations (\textsc{pixels}) consistently outperform
the other methods, including the
byte-based \vfe (\textsc{bytes}),
with differences exceeding multiple run-to-run standard deviations (Table \ref{tab:end-to-end2}).
Vocabulary expansion (\vexp) falls below even default tokenizer modeling (\nohn),
likely due to insufficient training to effectively integrate the newly added vocabulary tokens in this setup
\citep{yamaguchi2024effectivelyexpandvocabularyllms, zhao2024llamaenglishempiricalstudy}.
The SmolLM2-360M model particularly benefits from the \vfe,
showing improvements ranging from 2 to 12 points.
Notably, pixel-augmented SmolLM2-360M surpasses the larger SmolLM2-1.7B baseline on \then (48.6 vs. 40.4), a trend also evident between SmolLM2-1.7B and Phi-3-mini (52.1 vs. 51.1).
\subsection{Cross-lingual Transfer Results}
\label{sec:transfer}
To evaluate how effectively pixel-based representations facilitate positive language transfer \citep{conneau-etal-2020-unsupervised,chau-etal-2020-parsing,pfeiffer-etal-2021-unks},
particularly relevant for low-resource scenarios,
we pretrain the \vfes on 11M samples of \ruen, \esen, or \then, and subsequently finetune on \uken~for $k$ steps,
where the number of steps simulates constraints on available training data.
As a comparison, we follow the same procedure for continued training of the language model embedding matrix.
We compare performance to default modeling without continued embedding training (\textsc{base*}) and setups without \vfe pretraining (\textsc{pixels*}, \textsc{bytes*}).
We omit comparisons to vocabulary expansion due to its non-competitive effectiveness in Section \ref{sec:end-to-end}.
Table \ref{tab:transfer-to_uk} shows that integrating a pixel-based \vfe generally yields the strongest transfer effects,
particularly benefiting the SmolLM2-360M model.
We attribute this improvement to the ViT’s convolutional layer, which embeds inputs directly at the pixel level and enables updates to all encoder parameters at each training step.
This promotes cross-lingual transfer as the \vfe can exploit shared visual cues among languages
\citep{rahman-etal-2023-token,salesky-etal-2023-multilingual}, and most notably so with pretraining on Russian, which uses the same script as Ukrainian (Cyrillic.)
Positive transfer for \textsc{bytes} with Russian likely arises from the overlap in byte sequences encoding Cyrillic characters.
\begin{table}[t]
\centering
\setlength\tabcolsep{4pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{rcccccccccccc}
\toprule
& \multicolumn{3}{c}{Only \uken}
& \multicolumn{3}{c}{\ruen~then \uken}
& \multicolumn{3}{c}{\esen~then \uken}
& \multicolumn{3}{c}{\then~then \uken} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
\textit{Steps}
& \textsc{base*} & \textsc{bytes*} & \textsc{pixels*}
& \nohn & \textsc{bytes} & \textsc{pixels}
& \nohn & \textsc{bytes} & \textsc{pixels}
& \nohn & \textsc{bytes} & \textsc{pixels} \\
\midrule
\rowcol\multicolumn{13}{c}{\textit{SmolLM2-360M}} \\
10
& $18.8$ & $11.7$ & $13.3$
& $21.1$ & $25.6$ & $\bm{31.2}$
& $18.9$ & $15.0$ & $14.6$
& $19.9$ & $14.6$ & $13.5$ \\
50
& $23.3$ & $12.9$ & $13.4$
& $24.5$ & $34.2$ & $\bm{40.2}$
& $23.3$ & $16.8$ & $20.9$
& $23.5$ & $16.8$ & $18.0$ \\
100
& $26.0$ & $15.4$ & $15.2$
& $26.8$ & $39.2$ & $\bm{44.4}$
& $25.9$ & $19.3$ & $29.8$
& $25.9$ & $18.6$ & $25.0$ \\
1000
& $38.9$ & $19.3$ & $41.6$
& $40.1$ & $49.6$ & $\bm{52.6}$
& $39.1$ & $46.1$ & $50.6$
& $39.3$ & $42.5$ & $49.1$ \\
\rowcol\multicolumn{13}{c}{\textit{SmolLM2-1.7B}} \\
10
& $35.7$ & $\phantom{0}5.3$ & $\phantom{0}8.3$
& $\bm{39.8}$ & $30.1$ & $35.9$
& $36.5$ & $15.1$ & $14.9$
& $36.5$ & $14.9$ & $15.2$ \\
50
& $42.2$ & $14.7$ & $14.3$
& $44.0$ & $39.6$ & $\bm{45.5}$
& $42.6$ & $17.0$ & $22.9$
& $41.5$ & $17.3$ & $20.9$ \\
100
& $43.8$ & $15.8$ & $15.8$
& $45.9$ & $44.0$ & $\bm{48.9}$
& $44.1$ & $20.7$ & $34.2$
& $43.7$ & $19.8$ & $30.4$ \\
1000
& $51.2$ & $27.0$ & $46.9$
& $52.1$ & $53.2$ & $\bm{55.7}$
& $51.1$ & $48.9$ & $53.2$
& $51.5$ & $46.7$ & $52.4$ \\
\rowcol\multicolumn{13}{c}{\textit{Phi-3-mini}} \\
10
& $43.3$ & $\phantom{0}9.5$ & $11.3$
& $\bm{44.4}$ & $30.3$ & $12.4$
& $41.6$ & $14.1$ & $13.0$
& $43.9$ & $13.3$ & $12.7$ \\
50
& $49.8$ & $15.3$ & $14.9$
& $49.1$ & $46.8$ & $\bm{51.1}$
& $48.5$ & $20.6$ & $29.0$
& $49.2$ & $18.5$ & $26.1$ \\
100
& $51.2$ & $17.0$ & $15.7$
& $50.8$ & $50.3$ & $\bm{53.8}$
& $50.2$ & $31.3$ & $44.2$
& $50.7$ & $27.2$ & $41.7$ \\
1000
& $56.6$ & $36.1$ & $54.5$
& $56.6$ & $57.5$ & $\bm{58.8}$
& $55.8$ & $55.4$ & $57.3$
& $56.1$ & $54.0$ & $56.9$ \\
\bottomrule
\end{tabular}%
}
\caption{
\textsc{chrF++} scores on \uken~translation after $k$ training steps,
starting from weights initially trained on \textsc{xx}$\rightarrow$\textsc{en}.
The ``Only \uken'' setting involves no prior training.
}
\label{tab:transfer-to_uk}
\end{table}
\subsection{Cross-task Transfer Results}
\label{sec:cross-task}
\begin{wraptable}{r}{0.35\textwidth}
\vspace{-1em}
\centering
\small
\setlength\tabcolsep{6pt}
\begin{tabular}{lcc}
\toprule
& \textsc{base} & \textsc{pixels} \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{SmolLM2-360M}} \\
Hindi & $41.0$ & $\bm{78.1}$ \\
Avg. Deva. & $40.1$ & $\bm{65.1}$ \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{SmolLM2-1.7B}} \\
Hindi & $70.8$ & $\bm{77.0}$ \\
Avg. Deva. & $70.0$ & $\bm{72.2}$ \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{Phi-3-mini}} \\
Hindi & $\bm{72.5}$ & $70.3$ \\
Avg. Deva. & $\bm{69.3}$ & $45.6$ \\
\bottomrule
\end{tabular}
\caption{
Topic classification.
}
\label{tab:cross-task}
\vspace{-1em}
\end{wraptable}
Beyond machine translation,
we evaluate the potential of transfer across tasks by adapting a \vfe pretrained for \hien~machine translation (from Section \ref{sec:end-to-end}) to topic
classification on the 10 languages from the SIB200 dataset \citep{adelani-etal-2024-sib} written in the Devanagari script.
Since pixel-based augmentation consistently outperformed the byte-based alternative in prior experiments, we now focus exclusively on \textsc{pixels}.
See Table \ref{tab:cross_task-details} (Appendix \ref{app:details}) for experimental details.
Table \ref{tab:cross-task} compares test set accuracies from finetuning the three language models with default tokenization (\nohn) and with our \vfe (\textsc{pixels}).
We find that augmenting Phi-3-mini results in reduced performance,
potentially due to the \vfe overfitting during its machine translation pretraining.
The SmolLM2 models, on the other hand, consistently benefit from the augmentation, especially so on the Hindi articles.
\subsection{Efficiency Analysis}
\label{sec:ablation}
We observe that the relative computational overhead during training, introduced by the fallback network, varies with model scale and decreases for larger models (Table \ref{tab:stats}, based on experiments in Section \ref{sec:end-to-end}).
Although the first generation step incurs increased computational cost (measured in FLOPs), subsequent steps reuse cached fallback encodings.
Crucially, for a similar number of generated tokens (``Gen len''), the shorter input sequences
from \vfe compression
significantly reduce total sequence-level inference time, particularly for Phi-3-mini and on Thai. On the FLORES+ dev set, the \vfe leads to average compression ratios for Hindi, Russian, and Thai of 5.1, 4.7, and 8.6, respectively, relative to the SmolLM2 tokenizer, and 5.1, 2.2, and 5.1 relative to the Phi-3-mini tokenizer.
To address the higher relative overhead incurred by the SmolLM2 models, we evaluate performance after machine translation pretraining on \hien for one epoch using scaled-down fallback network configurations (Table \ref{tab:capacity}).
Even at reduced capacity, the fallback networks largely retain their performance, indicating that the demonstrated benefits of pixel-augmented modeling are achievable at a reduced cost.
\begin{table*}[t]
\centering
\begin{minipage}[t]{0.49\textwidth}
\centering
\small
\setlength\tabcolsep{3pt}
\begin{tabular}[t]{lcccc}
\toprule
& Train (s) & Gen (s) & Gen len & FLOPs \\
\midrule
\rowcol\multicolumn{5}{c}{\textit{SmolLM2 360M}} \\
\hien & 1.74 & 0.96 & 0.97 & 1.41 \\
\ruen & 1.76 & 0.98 & 0.98 & 1.41 \\
\then & 1.75 & 0.61 & 0.88 & 1.41 \\
\midrule
\rowcol\multicolumn{5}{c}{\textit{SmolLM2 1.7B}} \\
\hien & 1.42 & 0.92 & 1.00 & 1.09 \\
\ruen & 1.43 & 0.97 & 1.00 & 1.09 \\
\then & 1.42 & 0.68 & 0.93 & 1.09 \\
\rowcol\multicolumn{5}{c}{\textit{Phi-3-mini}} \\
\hien & 1.18 & 0.36 & 0.98 & 1.05 \\
\ruen & 1.19 & 0.40 & 1.00 & 1.05 \\
\then & 1.19 & 0.26 & 0.98 & 1.05 \\
\bottomrule
\end{tabular}%
\caption{
Metric ratios
(\textsc{pixels}/\nohn).
}
\label{tab:stats}
\end{minipage}
\hfill
\begin{minipage}[t]{0.49\textwidth}
\centering
\small
\setlength\tabcolsep{4.8pt}
\begin{tabular}[t]{ccccc}
\toprule
$n_{\text{params}}$ & $n_{\text{layers}}$ & $d_{\text{model}}$ & $n_{\text{heads}}$ & \vphantom{F}\textsc{hi}$\rightarrow$\textsc{en} \\ \midrule
\rowcol\multicolumn{5}{c}{\textit{SmolLM2 360M}} \\
92M & 4 & 1536 & 16 & 43.8 \\
65M & 6 & \phantom{0}960 & 12 & 43.1 \\
27M & 2 & \phantom{0}960 & 12 & 41.5 \\
\midrule
\rowcol\multicolumn{5}{c}{\textit{SmolLM2 1.7B}} \\
92M & 4 & 1536 & 16 & 51.8\\
51M & 4 & 1024 & 16 & 50.8 \\
31M & 2 & 1024 & 16 & 50.1 \\
\bottomrule
\end{tabular}%
\caption{Fallback network configurations.
Performance is measured as \hien~translation quality after one epoch of pretraining when only updating the network parameters.}
\label{tab:capacity}
\end{minipage}
\end{table*}
\section{Interleaving Images and Text}
\label{sec:gap}
The flexibility to interleave visual and textual representations is broadly relevant in multimodal scenarios such as multi-image applications and visual storytelling \citep{li2025llavanextinterleave}.
To explore this flexibility within our proposed framework,
we evaluate performance on a machine translation task involving Hindi-English code-switched source text and English target text from \citet{tarunesh-etal-2021-machine}.
When interleaving representations, ASCII text is embedded using the vocabulary, while all other segments are delegated to the \hien pretrained fallback network from Section \ref{sec:end-to-end}.
We compare the performance of interleaved modeling against default tokenization and uni-modal pixel processing, with which the entire input sequence is encoded by the \vfe.
See Table \ref{tab:code_switching-details} (Appendix \ref{app:details}) for experimental details.
Table \ref{tab:code-switching} shows that the \vfe again offers considerable gains over tokenization.
Yet, mixing input modalities (\textsc{pixels}\textsuperscript{\RIGHTcircle}) at best leads to the same performance as
encoding the entire input via the fallback network (\textsc{pixels}).
While the majority of the code-switched source text is indeed in Hindi (75\%),
this result raises questions about how compatible the two latent representation spaces are.
Intuitively, handling English text via the tokenizer should be easier than having the \vfe learning a new language, especially given the limited amount of training data.
We next explore this observation.
\paragraph{Modality Gap}
We hypothesize that a disconnect between the latent spaces of images and text limits effective utilization of both modalities within a sequence.
We therefore train a linear classifier on the FLORES+ dev set to distinguish Hindi words encoded by the \hien~fallback network from English words embedded by the vocabulary.
The classifier achieves perfect accuracy on a held-out subset,
indicating fully disjoint latent spaces \citep{wang-isola-2020-align-uniform,shi2023towards}.
Additionally, we measure the distance between the centers of these spaces
\citep{modality_gap}, $\vert\vert \mu_I - \mu_T \vert\vert_2$.
For SmolLM2-360M this distance is 40.7.
While it is unclear whether narrowing this gap would lead to better downstream performance \citep{Al-Jaff_2023,yaras2024explainingmitigatingmodalitygap,fahim2025its},
as the gap might arise from learning dynamics rather than representation quality,
we propose new pretraining strategies aimed at better aligning image and text representations to facilitate effective mixed-modality modeling:
mixing input representations during pretraining of the fallback network and employing an auxiliary loss based on word alignments.\footnote{All fallback networks in this section share the same initialization, as initial randomness could affect the representation space \citep{modality_gap}.}
\vspace{-0.5em}
\paragraph{Pretraining on Modality-switched Data}
We explore two distinct pretraining strategies on the \hien machine translation data.
(1) We obtain word alignments between source and target text in the \hien data and use those to
synthesize code-switched data with the methodology outlined in \citet{jalili-sabet-etal-2020-simalign},
based on XLM-R\textsubscript{\textsc{large}} \citep{conneau-etal-2020-unsupervised},
matching the downstream Hindi-English ratio of 75:25
(\textsc{synthesized}).
(2) We extend the former approach by adding modality-indicating prefix tokens
\citep{wang2024emu3nexttokenpredictionneed,nguyen-etal-2025-spirit, tschannen2025jetformer} to explicitly mark segment modality (\textsc{prefix}).
\vspace{-0.5em}
\paragraph{Auxiliary Alignment Loss}
Related work has found explicit signals to aid the alignment of untied embedding spaces \citep{minixhofer2024zeroshot}.
We therefore propose to include an auxiliary training objective during pretraining that forces the \vfe $h(w_k)$ to mimic the vocabulary embeddings $e_{w_k}$ for aligned words \citep{pinter-etal-2017-mimicking}
\begin{equation*}
\mathcal{L}^{\text{align}} = \frac{1}{n}\sum_{k=1}^n \vert\vert h(w_k) - e_{w_k}\vert\vert^2_2 \: .
\end{equation*}
Based on the word alignments from pretraining with modality-switched data,
we combine $\mathcal{L}^{\text{align}}$
with the cross entropy loss $\mathcal{L}^{\text{CE}}$ to obtain the new loss (\textsc{alignment}).
\begin{equation*}
\mathcal{L} = \mathcal{L}^\text{CE} + \mathcal{L}^{\text{align}} \: .
\end{equation*}
\begin{table}[t]
\centering
\setlength\tabcolsep{3pt}
\begin{minipage}[t]{0.48\textwidth}
\centering
\small
\begin{tabular}{lccc}
\toprule
& \nohn & \textsc{pixels}\textsuperscript{\RIGHTcircle} & \textsc{pixels}\\
\midrule
SmolLM2-360M & $32.7$ & $\bm{43.3}$ & $\bm{43.3}$ \\
SmolLM2-1.7B & $42.3$ & $\bm{45.8}$ & $\bm{45.8}$ \\
Phi-3-mini & $44.9$ & $45.9$ & $\bm{47.8}$ \\
\bottomrule
\end{tabular}
\caption{\textsc{chrF++} scores on Hindi-English code-switched data. ``\RIGHTcircle''~indicates mixed input modality sequences.}
\label{tab:code-switching}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\small
\setlength\tabcolsep{6pt}
\begin{tabular}{lcc}
\toprule
& $\vert\vert \mu_I - \mu_T \vert\vert_2$ & \textsc{pixels}\textsuperscript{\RIGHTcircle} \\
\midrule
\textsc{synthesized} & $\phantom{0}77.3$ & $42.5$ \\
\textsc{prefix} & $126.8$ & $37.4$ \\
\textsc{alignment} & $\phantom{0}\phantom{0}2.6$ & $38.4$ \\
\bottomrule
\end{tabular}
\caption{Distance between latent space centers
and downstream performance on mixed-modality sequences.
All experiments are based on SmolLM2-360M.}
\label{tab:explore_gap}
\end{minipage}
\end{table}
\paragraph{Results Using Alignment Strategies} Table \ref{tab:explore_gap} shows that none of the proposed strategies outperform the baseline from Table \ref{tab:code-switching} (43.3).
In all settings, we again find that a linear classifier can perfectly separate the two modalities.
Notably, pretraining and finetuning with the auxiliary loss (\textsc{alignment}) reduces the distance between centers (2.6 vs. 40.7) but leads to substantially worse performance.
These findings indicate that neither simple alignment strategies nor reducing latent-space distance alone effectively improves performance or bridges the latent spaces.
Future work could explore more sophisticated methods for effectively interleaving text and image representations.
\section{Experiments with Encoder-only Models}
\label{bert}
To explore whether the benefits of a pixel-based \vfe generalize to different architectures, we experiment with BERT \citep{devlin-etal-2019-bert},
which unlike BPE-based models suffers from out-of-vocabulary constraints on unseen scripts \citep{rust-etal-2021-good}.
Bypassing the tokenizer with a \vfe avoids potential \texttt{[UNK]} token substitution and thereby loss of information.
Specifically, we augment BERT\textsubscript{\textsc{base}} with a 24M-parameter pixel-based \vfe.\footnote{$n_{\text{layers}}=4$, $d_{\text{model}}=768$, and $n_{\text{heads}}=12$.}
We evaluate on named entity recognition
in Indic languages from the Naamapadam dataset \citep{mhaske-etal-2023-naamapadam},\footnote{We exclude Assamese since its run-to-run variance across all models exceeds that of the other languages by more than an order of magnitude.
}
a semantic sequence-level classification task.
The models are fully finetuned, encoding the entire input via the \vfe.
We compare performance with a randomly initialized \vfe (BERT+\textsc{pixels*}) and after pretraining on the Hindi portion of the dataset (BERT+\textsc{pixels}).
Table \ref{tab:NER} shows that integrating a \vfe substantially alleviates BERT's representational limitations,
outperforming the equally constrained BERT\textsubscript{\textsc{large}}.
For these tasks, pretraining the \vfe provides no additional benefit,
likely because finetuning on enough data sufficiently adapts these smaller models to
a comparatively simpler task than open-ended text generation \citep{liang2023holistic}.
However, BERT+\textsc{pixels*}, while competitive, does not surpass the multilingual mBERT, which was pretrained on 104 languages.
We observe a significant correlation between the proportion of \texttt{[UNK]} tokens and the gap in performance between BERT and BERT+\textsc{pixels*}.\footnote{Pearson correlation $r=0.67$, $p<0.05$.}
These findings reinforce that pixel-based \vfes provide an effective approach to overcoming the vocabulary constraints of monolingual models in multilingual scenarios.
\begin{table*}[t]
\centering
\small
\setlength\tabcolsep{3pt}
\resizebox{0.98\columnwidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
& $\vert \theta \vert$ & \textsc{bn} & \textsc{gu} & \textsc{hi} & \textsc{kn} & \textsc{ml} & \textsc{mr} & \textsc{or} & \textsc{pa} & \textsc{ta} & \textsc{te} & Avg. \\ \midrule
\tc{mBERT\textsubscript{\textsc{base}}} & \tc{179M} & \tc{$77.5$} & \tc{$78.7$} & \tc{$79.7$} & \tc{$76.5$} & \tc{$78.6$} & \tc{$79.1$} & \tc{$23.8$} & \tc{$68.1$} & \tc{$67.5$} & \tc{$79.5$} & \tc{$70.9$} \\
BERT\textsubscript{\textsc{base}} & 110M & $62.2$ & $24.3$ & $62.5$ & $25.7$ & $32.0$ & $65.7$ & $23.8$ & $13.1$ & $15.2$ & $26.8$ & $35.1$ \\
BERT+\textsc{pixels*} & 134M & $\bm{69.8}$ & $\bm{73.5}$ & $\bm{74.9}$ & $71.1$ & $71.0$ & $\bm{76.5}$ & $24.6$ & $\bm{65.8}$ & $51.6$ & $\bm{73.1}$ & $\bm{65.2}$ \\
BERT+\textsc{pixels} & 134M & $66.8$ & $72.7$ & -- & $\bm{72.4}$ & $\bm{72.8}$ & $75.3$ & $\bm{26.4}$ & $63.7$ & $\bm{57.3}$ & $71.8$ & $64.4$ \\
BERT\textsubscript{\textsc{large}} & 340M & $62.6$ & $24.3$ & $63.7$ & $25.6$ & $31.8$ & $66.5$ & $22.7$ & $13.6$ & $15.3$ & $25.8$ & $35.2$ \\
\midrule
\multicolumn{2}{l}{BERT \texttt{[UNK]}\%} & $9.4\%$ & $85.6\%$ & $14.8\%$ & $81.0\%$ & $79.5\%$ & $11.4\%$ & $85.8\%$ & $85.4\%$ & $62.7\%$ & $80.6\%$ & $59.6\%$ \\
\multicolumn{2}{l}{mBERT \texttt{[UNK]}\%} & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $85.8\%$ & $0.2\%$ & $0.0\%$ & $0.0\%$ & $8.6\%$ \\
\bottomrule
\end{tabular}%
}
\caption{Test set $F_1$ scores for
BERT models
on Naamapadam.
$\vert \theta \vert$ denotes parameter count.
The bottom two rows report the proportion of \texttt{[UNK]} tokens for BERT and mBERT.
}
\label{tab:NER}
\end{table*}
\section{Related Work}
In multilingual modeling, computational constraints often prohibit adequately representing a large number of languages \citep{conneau-etal-2020-unsupervised, rust-etal-2021-good}. Such vocabulary constraints result in lower downstream performance for languages underrepresented during pretraining \citep{bostrom-durrett-2020-byte, toraman-etal-2023-impact, fujii-etal-2023-different}.
Recent approaches to vocabulary-free NLP typically fall into one of two categories: byte-based or pixel-based methods.
While overlapping byte sequences are not necessarily semantically related \citep{choi-etal-2024-optimizing, cui2024efficienteffectivetextencoding}, shared sequences can enhance robustness and facilitate cross-lingual transfer via parameter sharing \citep{xue-etal-2022-byt5}.
\citet{de-souza-etal-2024-measuring} rely on bytes for quantifying also the language-agnostic component to cross-lingual transfer.
To alleviate the overhead from modeling non-Latin characters as bytes \citep{arnett-etal-2024-bit},
patch-based and dynamic token-merging strategies can improve the computational efficiency
\citep{yu2023megabyte,kallini2024mrt5dynamictokenmerging}.
As a promising outlook,
Byte Latent Transformer \citep{pagnoni2024bytelatenttransformerpatches} and EvaByte \citep{evabyte} demonstrate comparable performance to subword LLMs.
Recent advances in pixel-based language modeling have demonstrated
visual language understanding through pixels alone \citep{lee-etal-2023-pix2struct}.
In a similar vein, CLIPPO shows that a single encoder can effectively handle both image and text modalities by treating text as pixels \citep{tschannen2023clippo}.
Our work builds upon the concept of a general-purpose pixel-based language encoder introduced in PIXEL \citep{rust2023language}.
\citet{lotz-etal-2023-text} further explored text rendering strategies for PIXEL to reduce input redundancy, while recent efforts by \citet{chai-etal-2024-autoregressive} and \citet{tai-etal-2024-pixar} investigated autoregressive pretraining directly on pixel representations, with \citet{chai-etal-2024-autoregressive} finding benefits to multimodal over unimodal (text or image) pretraining.
Additionally, \citet{salesky-etal-2021-robust, salesky-etal-2023-multilingual} trained encoder-decoder models for machine translation using pixels as inputs.
In contrast, our approach enables pretrained and post-trained language models to benefit from pixel-based modeling without altering the underlying language model weights.
\section{Conclusion}
We introduced a fallback network that alleviates the vocabulary constraints of monolingual language models in multilingual settings by encoding text as pixels.
Our experiments show that pixel-based encodings outperform default tokenization, standard vocabulary expansion, and byte-based methods, resulting in improved performance, shorter input sequences, and faster decoding compared to modeling without a \vfe.
Notably, a pixel-augmented 360M-parameter model can surpass an unmodified 1.7B-parameter baseline on machine translation.
Our \vfe also enables effective cross-task transfer, and cross-lingual transfer based on visual similarities between scripts.
Interleaving text and image
representations is an exciting direction and future work could explore more sophisticated
methods for effectively and seamlessly mixing modalities within a sequence.
While our study focuses on English-centric models,
we argue that our approach remains relevant also for
multilingual models.
Any model relying on a single, fixed-size vocabulary is faced with a trade-off between its vocabulary coverage and the granularity of its representations.
The pixel-based \vfe is designed to circumvent the issue of over-segmentation, moving beyond vocabulary constraints instead of optimizing within them.
\section*{Acknowledgements}
We thank
Matthias Sperber,
Sarthak Garg,
Desmond Elliott,
Elizabeth Salesky,
Phillip Rust,
Kentaro Inui,
Benjamin Heinzerling, and
Ana Brassard
for helpful discussions and feedback.
Jonas F. Lotz is funded by the ROCKWOOL Foundation (grant 1242).
\bibliography{colm2025_conference}
\bibliographystyle{colm2025_conference}
\newpage
\appendix
\onecolumn
\section{Training Details}
\label{app:details}
\begin{table}[h]
\centering
\small
\begin{tabular}{llll} \toprule
Language & ISO 639-1 & Language Family & Script \\ \midrule
Bengali & \textsc{bn} & Indo-Aryan & Bengali \\
English & \textsc{en} & Indo-European & Latin \\
Gujarati & \textsc{gu} & Indo-European & Gujarati \\
Hindi & \textsc{hi} & Indo-European & Devanagari \\
Kannada & \textsc{kn} & Dravidian & Kannada \\
Malayalam & \textsc{ml} & Dravidian & Malayalam \\
Marathi & \textsc{mr} & Indo-European & Devanagari \\
Oriya & \textsc{or} & Indo-European & Oriya \\
Punjabi & \textsc{pa} & Indo-European & Gurmukhi \\
Russian & \textsc{ru} & Indo-European & Cyrillic \\
Spanish & \textsc{es} & Indo-European & Latin \\
Tamil & \textsc{ta} & Dravidian & Tamil \\
Telugu & \textsc{te} & Dravidian & Telugu \\
Thai & \textsc{th} & Kra-Dai & Thai \\
Ukrainian & \textsc{uk} & Indo-European & Cyrillic \\ \bottomrule
\end{tabular}%
\caption{Overview of languages used in our experiments.}
\label{tab:languages}
\end{table}
\begin{table}[h]
\centering
\resizebox{0.98\columnwidth}{!}{%
\begin{tabular}{ll} \toprule
Parameter & Value \\ \midrule
Optimizer & AdamW \citep{loshchilov2018decoupled, kingma-ba-2015-adam} \\
Adam $\beta$ & (0.9; 0.999) \\
Adam $\epsilon$ & \num{1e-8} \\
Weight decay & 0.0 \\
Dropout probability & 0.0 \\
Maximum source length & 256 \\
Maximum target length & 256 \\
Learning rate schedule & Cosine Decay \citep{DBLP:conf/iclr/LoshchilovH17} \\
Warmup ratio & 10\% \\
Peak learning rate & \num{3e-4} \\
Minimum learning rate & \num{3e-5} \\
Batch size & SmolLM2: 256; Phi-3-mini: 512 \\
Number of training samples in 1 epoch & Hindi: 14M, Russian: 14M, Spanish: 14M, Thai: 11M \\
(DoRA) Rank $r$ & 32 \\
(DoRA) $\alpha$ & 64 \\
(DoRA) dropout & 0.05 \\
(DoRA) Modules & Q, K, V, O and \vfe or LM embedding matrix \\ \midrule
Beam size & 2 \\
Length penalty & 1.0 \\
Repetition penalty & 1.0 \\
Temperature & 1.0 \\
Top-K sampling & 50 \\
Top-P sampling & 1.0 \\
\bottomrule
\end{tabular}%
}
\caption{Parameters and their values for the machine translation experiments in Section \ref{sec:end-to-end} and \ref{sec:transfer}. The top section covers training and the bottom covers inference.}
\label{tab:details}
\end{table}
Pretrained language model weights are downloaded from Hugging Face.\footnote{\url{https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct}}\textsuperscript{,}\footnote{\url{https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct}}\textsuperscript{,}\footnote{\url{https://huggingface.co/microsoft/Phi-3-mini-4k-instruct}}
\begin{table}[h]
\centering
\small
\begin{tabular}{ll} \toprule
Parameter & Value \\ \midrule
Batch size & 64 \\
Max number of epochs & 10 \\
Early stopping & \checkmark \\
\bottomrule
\end{tabular}%
\caption{Parameters and their values for the topic classification experiments in Section \ref{sec:cross-task}.
Only the batch size and and number of epochs are different from the experiments in Section \ref{sec:end-to-end} and \ref{sec:transfer}. We apply early stopping to check for convergence before the maximum number of epochs.
We instruct the models using the template:
\texttt{Would you classify the topic of this article as "science/technology", "travel", "politics", "sports", "health", "entertainment", or "geography"? \{INPUT\}}.
}
\label{tab:cross_task-details}
\end{table}
\begin{table}[h]
\centering
\small
\begin{tabular}{ll} \toprule
Parameter & Value \\ \midrule
Batch size & 64 \\
Epochs & 2 (342 steps) \\
\bottomrule
\end{tabular}%
\caption{Parameters and their values for the code-switching experiments in Section \ref{sec:gap}.
Only the batch size and and number of epochs are different from the experiments in Section \ref{sec:end-to-end} and \ref{sec:transfer}.
}
\label{tab:code_switching-details}
\end{table}
\begin{table}[h]
\centering
\small
\begin{tabular}{ll} \toprule
Parameter & Value \\ \midrule
Optimizer & AdamW \\
Adam $\beta$ & (0.9; 0.999) \\
Adam $\epsilon$ & \num{1e-8} \\
Weight decay & 0.0 \\
DoRA dropout & 0.05 \\
Maximum sequence length & 192 \\
Learning rate schedule & Linear Decay \\
Warmup steps & 1000 \\
Learning rate & \num{3e-4} \\
Batch size & 64 \\
Max number of training samples & 100,000 \\
Max steps & 15,000 \\
Eval steps & 500 \\
Early stopping & \checkmark \\
\bottomrule
\end{tabular}%
\caption{Parameters and their values for the NER experiments in Section \ref{bert}. }
\label{tab:encoder-details}
\end{table}
\clearpage
\section{Detailed Experimental Results}
\label{app:std}
Standard deviations are reported using subscript notation.
\begin{table}[h]
\centering
\setlength\tabcolsep{3pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
& \multicolumn{4}{c}{\hien} & \multicolumn{4}{c}{\ruen} & \multicolumn{4}{c}{\then} \\
\cmidrule(lr){2-5} \cmidrule(lr){6-9} \cmidrule(lr){10-13}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels}
& \nohn & \vexp & \textsc{bytes} & \textsc{pixels} \\
\midrule
SmolLM2-360M
& $53.2_{0.36}$ & $48.3_{0.26}$ & $53.2_{0.13}$ & $\bm{56.8}_{0.49}$
& $53.9_{0.12}$ & $53.0_{0.17}$ & $55.0_{0.12}$ & $\bm{56.0}_{0.18}$
& $36.5_{0.22}$ & $34.8_{0.05}$ & $46.9_{0.41}$ & $\bm{48.6}_{0.18}$ \\
SmolLM2-1.7B
& $56.8_{0.15}$ & $54.4_{0.41}$ & $57.6_{0.08}$ & $\bm{59.0}_{0.10}$
& $57.0_{0.13}$ & $56.7_{0.17}$ & $57.4_{0.08}$ & $\bm{57.8}_{0.09}$
& $40.4_{0.18}$ & $39.4_{0.04}$ & $50.2_{0.10}$ & $\bm{52.1}_{0.16}$ \\
Phi-3-mini
& $57.3_{0.14}$ & $54.7_{0.22}$ & $59.5_{0.13}$ & $\bm{60.9}_{0.20}$
& $57.9_{0.13}$ & $57.8_{0.03}$ & $57.8_{0.11}$ & $\bm{58.2}_{0.12}$
& $51.1_{0.26}$ & $50.4_{0.32}$ & $52.0_{0.37}$ & $\bm{53.1}_{0.35}$ \\
\bottomrule
\end{tabular}%
}
\caption{
Copy of Table \ref{tab:end-to-end} including standard deviations.
}
\label{tab:end-to-end2}
\end{table}
\vfill
\begin{table}[h]
\centering
\setlength\tabcolsep{4pt}
\resizebox{\textwidth}{!}{%
\begin{tabular}{rcccccccccccc}
\toprule
& \multicolumn{3}{c}{Only \uken}
& \multicolumn{3}{c}{\ruen~then \uken}
& \multicolumn{3}{c}{\esen~then \uken}
& \multicolumn{3}{c}{\then~then \uken} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
\textit{Steps}
& \nohn & \textsc{bytes*} & \textsc{pixels*}
& \nohn & \textsc{bytes} & \textsc{pixels}
& \nohn & \textsc{bytes} & \textsc{pixels}
& \nohn & \textsc{bytes} & \textsc{pixels} \\
\midrule
\rowcol\multicolumn{13}{c}{\textit{SmolLM2-360M}} \\
10
& $18.8_{0.18}$ & $11.7_{1.61}$ & $13.3_{0.25}$
& $21.1_{0.23}$ & $25.6_{0.16}$ & $\bm{31.2}_{0.18}$
& $18.9_{0.63}$ & $15.0_{0.05}$ & $14.6_{0.16}$
& $19.9_{0.16}$ & $14.6_{0.21}$ & $13.5_{0.21}$ \\
50
& $23.3_{0.14}$ & $12.9_{0.36}$ & $13.4_{0.35}$
& $24.5_{0.29}$ & $34.2_{0.10}$ & $\bm{40.2}_{0.17}$
& $23.3_{0.18}$ & $16.8_{0.11}$ & $20.9_{0.06}$
& $23.5_{0.03}$ & $16.8_{0.13}$ & $18.0_{0.09}$ \\
100
& $26.0_{0.15}$ & $15.4_{0.20}$ & $15.2_{0.11}$
& $26.8_{0.09}$ & $39.2_{0.06}$ & $\bm{44.4}_{0.07}$
& $25.9_{0.14}$ & $19.3_{0.11}$ & $29.8_{0.07}$
& $25.9_{0.18}$ & $18.6_{0.11}$ & $25.0_{0.25}$ \\
1000
& $38.9_{0.16}$ & $19.3_{0.13}$ & $41.6_{0.91}$
& $40.1_{0.15}$ & $49.6_{0.08}$ & $\bm{52.6}_{0.08}$
& $39.1_{0.46}$ & $46.1_{0.38}$ & $50.6_{0.18}$
& $39.3_{0.50}$ & $42.5_{0.32}$ & $49.1_{0.32}$ \\
\rowcol\multicolumn{13}{c}{\textit{SmolLM2-1.7B}} \\
10
& $35.7_{0.31}$ & $5.3_{1.29}$ & $8.3_{0.31}$
& $\bm{39.8}_{0.28}$ & $30.1_{0.13}$ & $35.9_{0.11}$
& $36.5_{0.37}$ & $15.1_{0.22}$ & $14.9_{0.09}$
& $36.5_{0.20}$ & $14.9_{0.13}$ & $15.2_{0.17}$ \\
50
& $42.2_{0.25}$ & $14.7_{0.28}$ & $14.3_{0.60}$
& $44.0_{0.37}$ & $39.6_{0.29}$ & $\bm{45.5}_{0.11}$
& $42.6_{0.31}$ & $17.0_{0.03}$ & $22.9_{0.22}$
& $41.5_{0.01}$ & $17.3_{0.06}$ & $20.9_{0.03}$ \\
100
& $43.8_{0.26}$ & $15.8_{0.27}$ & $15.8_{0.29}$
& $45.9_{0.07}$ & $44.0_{0.10}$ & $\bm{48.9}_{0.13}$
& $44.1_{0.42}$ & $20.7_{0.36}$ & $34.2_{0.10}$
& $43.7_{0.48}$ & $19.8_{0.18}$ & $30.4_{0.13}$ \\
1000
& $51.2_{0.27}$ & $27.0_{0.26}$ & $46.9_{0.17}$
& $52.1_{0.18}$ & $53.2_{0.40}$ & $\bm{55.7}_{0.15}$
& $51.1_{0.34}$ & $48.9_{0.03}$ & $53.2_{0.13}$
& $51.5_{0.32}$ & $46.7_{0.07}$ & $52.4_{0.12}$ \\
\rowcol\multicolumn{13}{c}{\textit{Phi-3-mini}} \\
10
& $43.3_{0.04}$ & $9.5_{0.57}$ & $11.3_{0.54}$
& $\bm{44.4}_{0.25}$ & $30.3_{1.01}$ & $12.4_{0.98}$
& $41.6_{0.02}$ & $14.1_{0.33}$ & $13.0_{0.54}$
& $43.9_{0.41}$ & $13.3_{0.40}$ & $12.7_{0.50}$ \\
50
& $49.8_{0.16}$ & $15.3_{0.05}$ & $14.9_{0.08}$
& $49.1_{0.42}$ & $46.8_{0.34}$ & $\bm{51.1}_{0.29}$
& $48.5_{0.33}$ & $20.6_{0.23}$ & $29.0_{0.96}$
& $49.2_{0.09}$ & $18.5_{0.18}$ & $26.1_{0.29}$ \\
100
& $51.2_{0.12}$ & $17.0_{0.09}$ & $15.7_{0.56}$
& $50.8_{0.28}$ & $50.3_{0.33}$ & $\bm{53.8}_{0.29}$
& $50.2_{0.16}$ & $31.3_{0.21}$ & $44.2_{0.24}$
& $50.7_{0.16}$ & $27.2_{1.09}$ & $41.7_{0.06}$ \\
1000
& $56.6_{0.17}$ & $36.1_{0.52}$ & $54.5_{0.09}$
& $56.6_{0.03}$ & $57.5_{0.13}$ & $\bm{58.8}_{0.21}$
& $55.8_{0.15}$ & $55.4_{0.16}$ & $57.3_{0.16}$
& $56.1_{0.21}$ & $54.0_{0.14}$ & $56.9_{0.15}$ \\
\bottomrule
\end{tabular}%
}
\caption{
Copy of Table \ref{tab:transfer-to_uk} including standard deviations.
}
\label{tab:transfer-to_uk2}
\end{table}
\vfill
\begin{table}[h]
\centering
\small
\setlength\tabcolsep{6pt}
\begin{tabular}{lcc}
\toprule
& \textsc{base} & \textsc{pixels} \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{SmolLM2-360M}} \\
Hindi & $41.0_{2.32}$ & $\bm{78.1}_{3.19}$ \\
Avg. Deva. & $40.1$ & $\bm{65.1}$ \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{SmolLM2-1.7B}} \\
Hindi & $70.8_{0.75}$ & $\bm{77.0}_{1.30}$ \\
Avg. Deva. & $70.0$ & $\bm{72.2}$ \\
\midrule
\rowcol\multicolumn{3}{c}{\textit{Phi-3-mini}} \\
Hindi & $\bm{72.5}_{1.30}$ & $70.3_{1.72}$ \\
Avg. Deva. & $\bm{69.3}$ & $45.6$ \\
\bottomrule
\end{tabular}
\caption{
Copy of Table \ref{tab:cross-task} including standard deviation.
}
\label{tab:cross-task2}
\end{table}
\begin{table}[h]
\setlength\tabcolsep{2pt}
\centering
\small
\begin{tabular}{lccc} \toprule
& \nohn & \textsc{pixels}\textsuperscript{\RIGHTcircle} & \textsc{pixels} \\ \midrule
SmolLM2-360M & $32.7_{0.06}$ & $\bm{43.3}_{0.08}$ & $\bm{43.3}_{0.22}$ \\
SmolLM2-1.7B & $42.3_{0.09}$ & $\bm{45.8}_{0.24}$ & $\bm{45.8}_{0.33}$ \\
Phi-3-mini & $44.9_{0.10}$ & $45.9_{0.17}$ & $\bm{47.8}_{0.17}$ \\
\bottomrule
\end{tabular}%
\caption{
Copy of Table \ref{tab:code-switching} including standard deviations.
}
\label{tab:code-switching2}
\end{table}
\vfill
\begin{table}[h]
\setlength\tabcolsep{3pt}
\centering
\small
\begin{tabular}{lcc} \toprule
& $\vert\vert \mu_I - \mu_T \vert\vert_2$ & \textsc{pixels}\textsuperscript{\RIGHTcircle} \\ \midrule
\textsc{synthesized} & $77.3$ & $42.5_{0.37}$ \\
\textsc{prefix} & $126.8$ & $37.4_{0.02}$ \\
\textsc{alignment} & $2.6$ & $38.4_{0.16}$ \\
\bottomrule
\end{tabular}%
\caption{
Copy of Table \ref{tab:explore_gap} including standard deviations.
}
\label{tab:explore_gap2}
\end{table}
\vfill
\begin{table*}[h]
\centering
\setlength\tabcolsep{3pt}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{lcccccccccccc}
\toprule
& $\vert \theta \vert$ & \textsc{bn} & \textsc{gu} & \textsc{hi} & \textsc{kn} & \textsc{ml} & \textsc{mr} & \textsc{or} & \textsc{pa} & \textsc{ta} & \textsc{te} & Avg. \\ \midrule
\tc{mBERT\textsubscript{\textsc{base}}} & \tc{179M} & \tc{$77.5_{1.12}$} & \tc{$78.7_{0.74}$} & \tc{$79.7_{1.02}$} & \tc{$76.5_{1.27}$} & \tc{$78.6_{0.16}$} & \tc{$79.1_{0.77}$} & \tc{$23.8_{2.34}$} & \tc{$68.1_{0.50}$} & \tc{$67.5_{0.10}$} & \tc{$79.5_{0.76}$} & \tc{$70.9$} \\
BERT\textsubscript{\textsc{base}} & 110M & $62.2_{0.42}$ & $24.3_{0.70}$ & $62.5_{0.56}$ & $25.7_{1.31}$ & $32.0_{0.57}$ & $65.7_{0.63}$ & $23.8_{2.36}$ & $13.1_{0.62}$ & $15.2_{0.88}$ & $26.8_{0.32}$ & $35.1$ \\
BERT+24M\textsc{*} & 134M & $\bm{69.8}_{1.01}$ & $\bm{73.5}_{1.13}$ & $\bm{74.9}_{0.10}$ & $71.1_{1.33}$ & $71.0_{1.25}$ & $\bm{76.5}_{0.32}$ & $24.6_{2.44}$ & $\bm{65.8}_{0.59}$ & $51.6_{2.20}$ & $\bm{73.1}_{2.74}$ & $\bm{65.2}$ \\
BERT+24M & 134M & $66.8_{1.01}$ & $72.7_{0.60}$ & -- & $\bm{72.4}_{0.09}$ & $\bm{72.8}_{0.72}$ & $75.3_{0.86}$ & $\bm{26.4}_{1.00}$ & $63.7_{0.88}$ & $\bm{57.3}_{0.15}$ & $71.8_{0.62}$ & $64.4$ \\
BERT\textsubscript{\textsc{large}} & 340M & $62.6_{0.60}$ & $24.3_{0.79}$ & $63.7_{0.43}$ & $25.6_{1.67}$ & $31.8_{0.43}$ & $66.5_{1.65}$ & $22.7_{0.41}$ & $13.6_{0.24}$ & $15.3_{0.68}$ & $25.8_{0.06}$ & $35.2$ \\
\midrule
\multicolumn{2}{l}{BERT \texttt{[UNK]}\%} & $9.4\%$ & $85.6\%$ & $14.8\%$ & $81.0\%$ & $79.5\%$ & $11.4\%$ & $85.8\%$ & $85.4\%$ & $62.7\%$ & $80.6\%$ & $59.6\%$ \\
\multicolumn{2}{l}{mBERT \texttt{[UNK]}\%} & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $0.0\%$ & $85.8\%$ & $0.2\%$ & $0.0\%$ & $0.0\%$ & $8.6\%$ \\
\bottomrule
\end{tabular}%
}
\caption{
Copy of Table \ref{tab:NER} including standard deviations.
}
\label{tab:NER2}
\end{table*}
\end{document}
\begin{thebibliography}{98}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abdin et~al.(2024)Abdin, Aneja, Awadalla, Awadallah, Awan, Bach, Bahree, Bakhtiari, Bao, Behl, Benhaim, Bilenko, Bjorck, Bubeck, Cai, Cai, Chaudhary, Chen, Chen, Chen, Chen, Chen, Cheng, Chopra, Dai, Dixon, Eldan, Fragoso, Gao, Gao, Gao, Garg, Giorno, Goswami, Gunasekar, Haider, Hao, Hewett, Hu, Huynh, Iter, Jacobs, Javaheripi, Jin, Karampatziakis, Kauffmann, Khademi, Kim, Kim, Kurilenko, Lee, Lee, Li, Li, Liang, Liden, Lin, Lin, Liu, Liu, Liu, Liu, Liu, Luo, Madan, Mahmoudzadeh, Majercak, Mazzola, Mendes, Mitra, Modi, Nguyen, Norick, Patra, Perez-Becker, Portet, Pryzant, Qin, Radmilac, Ren, de~Rosa, Rosset, Roy, Ruwase, Saarikivi, Saied, Salim, Santacroce, Shah, Shang, Sharma, Shen, Shukla, Song, Tanaka, Tupini, Vaddamanu, Wang, Wang, Wang, Wang, Wang, Wang, Ward, Wen, Witte, Wu, Wu, Wyatt, Xiao, Xu, Xu, Xu, Xue, Yadav, Yang, Yang, Yang, Yang, Yu, Yuan, Zhang, Zhang, Zhang, Zhang, Zhang, Zhang, Zhang, and Zhou]{abdin2024phi3technicalreporthighly}
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar~Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie~Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell~J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam~Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young~Jin Kim, Lev Kurilenko, James~R. Lee, Yin~Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce~Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César~Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen,
Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de~Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu~Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu~Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li~Lyna Zhang, Yi~Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou.
\newblock Phi-3 technical report: A highly capable language model locally on your phone, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2404.14219}.
\bibitem[Adelani et~al.(2024)Adelani, Liu, Shen, Vassilyev, Alabi, Mao, Gao, and Lee]{adelani-etal-2024-sib}
David~Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba~O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun~Annie Lee.
\newblock {SIB}-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.
\newblock In Yvette Graham and Matthew Purver (eds.), \emph{Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 226--245, St. Julian{'}s, Malta, March 2024. Association for Computational Linguistics.
\newblock URL \url{https://aclanthology.org/2024.eacl-long.14/}.
\bibitem[Al-Jaff(2023)]{Al-Jaff_2023}
Mohammad Al-Jaff.
\newblock \emph{Messing With The Gap: On The Modality Gap Phenomenon In Multimodal Contrastive Representation Learning}.
\newblock PhD thesis, Uppsala University, 2023.
\newblock URL \url{https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-517811}.
\bibitem[Alayrac et~al.(2022)Alayrac, Donahue, Luc, Miech, Barr, Hasson, Lenc, Mensch, Millican, Reynolds, Ring, Rutherford, Cabi, Han, Gong, Samangooei, Monteiro, Menick, Borgeaud, Brock, Nematzadeh, Sharifzadeh, Binkowski, Barreira, Vinyals, Zisserman, and Simonyan]{alayrac2022flamingo}
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
\newblock Flamingo: a visual language model for few-shot learning.
\newblock In Alice~H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), \emph{Advances in Neural Information Processing Systems}, 2022.
\newblock URL \url{https://openreview.net/forum?id=EbMuimAbPbs}.
\bibitem[Allal et~al.(2025)Allal, Lozhkov, Bakouch, Blázquez, Penedo, Tunstall, Marafioti, Kydlíček, Lajarín, Srivastav, Lochner, Fahlgren, Nguyen, Fourrier, Burtenshaw, Larcher, Zhao, Zakka, Morlon, Raffel, von Werra, and Wolf]{allal2024SmolLM2}
Loubna~Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel~Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín~Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf.
\newblock Smollm2: When smol goes big -- data-centric training of a small language model, 2025.
\newblock URL \url{https://doi.org/10.48550/arXiv.2502.02737}.
\bibitem[Ansell et~al.(2022)Ansell, Ponti, Korhonen, and Vuli{\'c}]{ansell-etal-2022-composable}
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vuli{\'c}.
\newblock Composable sparse fine-tuning for cross-lingual transfer.
\newblock In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), \emph{Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1778--1796, Dublin, Ireland, May 2022. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2022.acl-long.125}.
\newblock URL \url{https://aclanthology.org/2022.acl-long.125/}.
\bibitem[Arnett et~al.(2024)Arnett, Chang, and Bergen]{arnett-etal-2024-bit}
Catherine Arnett, Tyler~A. Chang, and Benjamin Bergen.
\newblock A bit of a problem: Measurement disparities in dataset sizes across languages.
\newblock In Maite Melero, Sakriani Sakti, and Claudia Soria (eds.), \emph{Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024}, pp.\ 1--9, Torino, Italia, May 2024. ELRA and ICCL.
\newblock URL \url{https://aclanthology.org/2024.sigul-1.1/}.
\bibitem[{BigScience Workshop} et~al.(2023){BigScience Workshop}, :, Scao, Fan, Akiki, Pavlick, Ilić, Hesslow, Castagné, Luccioni, Yvon, Gallé, Tow, Rush, Biderman, Webson, Ammanamanchi, Wang, Sagot, Muennighoff, del Moral, Ruwase, Bawden, Bekman, McMillan-Major, Beltagy, Nguyen, Saulnier, Tan, Suarez, Sanh, Laurençon, Jernite, Launay, Mitchell, Raffel, Gokaslan, Simhi, Soroa, Aji, Alfassy, Rogers, Nitzav, Xu, Mou, Emezue, Klamm, Leong, van Strien, Adelani, Radev, Ponferrada, Levkovizh, Kim, Natan, Toni, Dupont, Kruszewski, Pistilli, Elsahar, Benyamina, Tran, Yu, Abdulmumin, Johnson, Gonzalez-Dios, de~la Rosa, Chim, Dodge, Zhu, Chang, Frohberg, Tobing, Bhattacharjee, Almubarak, Chen, Lo, Werra, Weber, Phan, allal, Tanguy, Dey, Muñoz, Masoud, Grandury, Šaško, Huang, Coavoux, Singh, Jiang, Vu, Jauhar, Ghaleb, Subramani, Kassner, Khamis, Nguyen, Espejel, de~Gibert, Villegas, Henderson, Colombo, Amuok, Lhoest, Harliman, Bommasani, López, Ribeiro, Osei, Pyysalo, Nagel, Bose, Muhammad, Sharma, Longpre,
Nikpoor, Silberberg, Pai, Zink, Torrent, Schick, Thrush, Danchev, Nikoulina, Laippala, Lepercq, Prabhu, Alyafeai, Talat, Raja, Heinzerling, Si, Taşar, Salesky, Mielke, Lee, Sharma, Santilli, Chaffin, Stiegler, Datta, Szczechla, Chhablani, Wang, Pandey, Strobelt, Fries, Rozen, Gao, Sutawika, Bari, Al-shaibani, Manica, Nayak, Teehan, Albanie, Shen, Ben-David, Bach, Kim, Bers, Fevry, Neeraj, Thakker, Raunak, Tang, Yong, Sun, Brody, Uri, Tojarieh, Roberts, Chung, Tae, Phang, Press, Li, Narayanan, Bourfoune, Casper, Rasley, Ryabinin, Mishra, Zhang, Shoeybi, Peyrounette, Patry, Tazi, Sanseviero, von Platen, Cornette, Lavallée, Lacroix, Rajbhandari, Gandhi, Smith, Requena, Patil, Dettmers, Baruwa, Singh, Cheveleva, Ligozat, Subramonian, Névéol, Lovering, Garrette, Tunuguntla, Reiter, Taktasheva, Voloshina, Bogdanov, Winata, Schoelkopf, Kalo, Novikova, Forde, Clive, Kasai, Kawamura, Hazan, Carpuat, Clinciu, Kim, Cheng, Serikov, Antverg, van~der Wal, Zhang, Zhang, Gehrmann, Mirkin, Pais, Shavrina, Scialom, Yun,
Limisiewicz, Rieser, Protasov, Mikhailov, Pruksachatkun, Belinkov, Bamberger, Kasner, Rueda, Pestana, Feizpour, Khan, Faranak, Santos, Hevia, Unldreaj, Aghagol, Abdollahi, Tammour, HajiHosseini, Behroozi, Ajibade, Saxena, Ferrandis, McDuff, Contractor, Lansky, David, Kiela, Nguyen, Tan, Baylor, Ozoani, Mirza, Ononiwu, Rezanejad, Jones, Bhattacharya, Solaiman, Sedenko, Nejadgholi, Passmore, Seltzer, Sanz, Dutra, Samagaio, Elbadri, Mieskes, Gerchick, Akinlolu, McKenna, Qiu, Ghauri, Burynok, Abrar, Rajani, Elkott, Fahmy, Samuel, An, Kromann, Hao, Alizadeh, Shubber, Wang, Roy, Viguier, Le, Oyebade, Le, Yang, Nguyen, Kashyap, Palasciano, Callahan, Shukla, Miranda-Escalada, Singh, Beilharz, Wang, Brito, Zhou, Jain, Xu, Fourrier, Periñán, Molano, Yu, Manjavacas, Barth, Fuhrimann, Altay, Bayrak, Burns, Vrabec, Bello, Dash, Kang, Giorgi, Golde, Posada, Sivaraman, Bulchandani, Liu, Shinzato, de~Bykhovetz, Takeuchi, Pàmies, Castillo, Nezhurina, Sänger, Samwald, Cullan, Weinberg, Wolf, Mihaljcic, Liu, Freidank,
Kang, Seelam, Dahlberg, Broad, Muellner, Fung, Haller, Chandrasekhar, Eisenberg, Martin, Canalli, Su, Su, Cahyawijaya, Garda, Deshmukh, Mishra, Kiblawi, Ott, Sang-aroonsiri, Kumar, Schweter, Bharati, Laud, Gigant, Kainuma, Kusa, Labrak, Bajaj, Venkatraman, Xu, Xu, Xu, Tan, Xie, Ye, Bras, Belkada, and Wolf]{workshop2023bloom176bparameteropenaccessmultilingual}
{BigScience Workshop}, :, Teven~Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra~Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander~M. Rush, Stella Biderman, Albert Webson, Pawan~Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert~Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz~Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro~Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham~Fikri Aji, Amit Alfassy, Anna Rogers, Ariel~Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David~Ifeoluwa Adelani, Dragomir Radev, Eduardo~González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal~Bar Natan, Francesco~De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran,
Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de~la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro~Von Werra, Leon Weber, Long Phan, Loubna~Ben allal, Ludovic Tanguy, Manan Dey, Manuel~Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh~Chien Vu, Mohammad~A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de~Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto~Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen~Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago~Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina,
Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut~Emre Taşar, Elizabeth Salesky, Sabrina~J. Mielke, Wilson~Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason~Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M~Saiful Bari, Maged~S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen~H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung~Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von
Platen, Pierre Cornette, Pierre~François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta~Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica~Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van~der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona
Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos~Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong~A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio~Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav~Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo~Wang, Caio Brito,
Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel~León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena~U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose~David Posada, Karthik~Rangasai Sivaraman, Lokesh Bulchandani, Lu~Liu, Luisa Shinzato, Madeleine~Hahn de~Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria~A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel~De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas~Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok~S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya
Kainuma, Wojciech Kusa, Yanis Labrak, Yash~Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu~Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf.
\newblock Bloom: A 176b-parameter open-access multilingual language model, 2023.
\newblock URL \url{https://doi.org/10.48550/arXiv.2211.05100}.
\bibitem[Bostrom \& Durrett(2020)Bostrom and Durrett]{bostrom-durrett-2020-byte}
Kaj Bostrom and Greg Durrett.
\newblock Byte pair encoding is suboptimal for language model pretraining.
\newblock In Trevor Cohn, Yulan He, and Yang Liu (eds.), \emph{Findings of the Association for Computational Linguistics: EMNLP 2020}, pp.\ 4617--4624, Online, November 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.findings-emnlp.414}.
\newblock URL \url{https://aclanthology.org/2020.findings-emnlp.414/}.
\bibitem[Brinkmann et~al.(2025)Brinkmann, Wendler, Bartelt, and Mueller]{brinkmann2025largelanguagemodelsshare}
Jannik Brinkmann, Chris Wendler, Christian Bartelt, and Aaron Mueller.
\newblock Large language models share representations of latent grammatical concepts across typologically diverse languages, 2025.
\newblock URL \url{https://doi.org/10.48550/arXiv.2501.06346}.
\bibitem[Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei]{brown-et-al-2020-gpt3}
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared~D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
\newblock Language models are few-shot learners.
\newblock In H.~Larochelle, M.~Ranzato, R.~Hadsell, M.F. Balcan, and H.~Lin (eds.), \emph{Advances in Neural Information Processing Systems}, volume~33, pp.\ 1877--1901. Curran Associates, Inc., 2020.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf}.
\bibitem[Chai et~al.(2024)Chai, Liu, Xiao, Wang, Sun, and Wu]{chai-etal-2024-autoregressive}
Yekun Chai, Qingyi Liu, Jingwu Xiao, Shuohuan Wang, Yu~Sun, and Hua Wu.
\newblock Autoregressive pre-training on pixels and texts.
\newblock In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 3106--3125, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.182}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.182/}.
\bibitem[Chau et~al.(2020)Chau, Lin, and Smith]{chau-etal-2020-parsing}
Ethan~C. Chau, Lucy~H. Lin, and Noah~A. Smith.
\newblock Parsing with multilingual {BERT}, a small corpus, and a small treebank.
\newblock In Trevor Cohn, Yulan He, and Yang Liu (eds.), \emph{Findings of the Association for Computational Linguistics: EMNLP 2020}, pp.\ 1324--1334, Online, November 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.findings-emnlp.118}.
\newblock URL \url{https://aclanthology.org/2020.findings-emnlp.118/}.
\bibitem[Choi et~al.(2024)Choi, Jeong, Park, Won, Lim, Kim, Kang, Yoon, Park, Lee, Lee, Hahm, Kim, and Lim]{choi-etal-2024-optimizing}
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, and KyungTae Lim.
\newblock Optimizing language augmentation for multilingual large language models: A case study on {K}orean.
\newblock In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), \emph{Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, pp.\ 12514--12526, Torino, Italia, May 2024. ELRA and ICCL.
\newblock URL \url{https://aclanthology.org/2024.lrec-main.1095/}.
\bibitem[Chowdhery et~al.(2023)Chowdhery, Narang, Devlin, Bosma, Mishra, Roberts, Barham, Chung, Sutton, Gehrmann, Schuh, Shi, Tsvyashchenko, Maynez, Rao, Barnes, Tay, Shazeer, Prabhakaran, Reif, Du, Hutchinson, Pope, Bradbury, Austin, Isard, Gur-Ari, Yin, Duke, Levskaya, Ghemawat, Dev, Michalewski, Garcia, Misra, Robinson, Fedus, Zhou, Ippolito, Luan, Lim, Zoph, Spiridonov, Sepassi, Dohan, Agrawal, Omernick, Dai, Pillai, Pellat, Lewkowycz, Moreira, Child, Polozov, Lee, Zhou, Wang, Saeta, Diaz, Firat, Catasta, Wei, Meier-Hellstern, Eck, Dean, Petrov, and Fiedel]{palm}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi~Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew~M. Dai, Thanumalayan~Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.
\newblock Palm: scaling language modeling with pathways.
\newblock \emph{J. Mach. Learn. Res.}, 24\penalty0 (1), January 2023.
\newblock ISSN 1532-4435.
\bibitem[Conneau et~al.(2020)Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzm{\'a}n, Grave, Ott, Zettlemoyer, and Stoyanov]{conneau-etal-2020-unsupervised}
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm{\'a}n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.
\newblock Unsupervised cross-lingual representation learning at scale.
\newblock In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), \emph{Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, pp.\ 8440--8451, Online, July 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.acl-main.747}.
\newblock URL \url{https://aclanthology.org/2020.acl-main.747/}.
\bibitem[Costa-juss{\`a} et~al.(2024)Costa-juss{\`a}, Cross, {\c C}elebi, Elbayad, Heafield, Heffernan, Kalbassi, Lam, Licht, Maillard, Sun, Wang, Wenzek, Youngblood, Akula, Barrault, Gonzalez, Hansanti, Hoffman, Jarrett, Sadagopan, Rowe, Spruit, Tran, Andrews, Ayan, Bhosale, Edunov, Fan, Gao, Goswami, Guzm{\'a}n, Koehn, Mourachko, Ropers, Saleem, Schwenk, Wang, and Team]{nllb_nature}
Marta~R. Costa-juss{\`a}, James Cross, Onur {\c C}elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al~Youngblood, Bapi Akula, Loic Barrault, Gabriel~Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik~Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip~Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm{\'a}n, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang, and NLLB Team.
\newblock Scaling neural machine translation to 200 languages.
\newblock \emph{Nature}, 630\penalty0 (8018):\penalty0 841--846, 2024.
\newblock \doi{10.1038/s41586-024-07335-x}.
\newblock URL \url{https://doi.org/10.1038/s41586-024-07335-x}.
\bibitem[Cui et~al.(2024)Cui, Yang, and Yao]{cui2024efficienteffectivetextencoding}
Yiming Cui, Ziqing Yang, and Xin Yao.
\newblock Efficient and effective text encoding for chinese llama and alpaca, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2304.08177}.
\bibitem[De~Souza et~al.(2024)De~Souza, Almeida, Lotufo, and Frassetto~Nogueira]{de-souza-etal-2024-measuring}
Leandro De~Souza, Thales Almeida, Roberto Lotufo, and Rodrigo Frassetto~Nogueira.
\newblock Measuring cross-lingual transfer in bytes.
\newblock In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), \emph{Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, pp.\ 7526--7537, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.naacl-long.418}.
\newblock URL \url{https://aclanthology.org/2024.naacl-long.418}.
\bibitem[Devlin et~al.(2019)Devlin, Chang, Lee, and Toutanova]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock {BERT}: Pre-training of deep bidirectional transformers for language understanding.
\newblock In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), \emph{Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}, pp.\ 4171--4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/N19-1423}.
\newblock URL \url{https://aclanthology.org/N19-1423}.
\bibitem[Dobler \& de~Melo(2023)Dobler and de~Melo]{dobler-de-melo-2023-focus}
Konstantin Dobler and Gerard de~Melo.
\newblock {FOCUS}: Effective embedding initialization for monolingual specialization of multilingual models.
\newblock In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 13440--13454, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.emnlp-main.829}.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.829/}.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{dosovitskiy2021an}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=YicbFdNTTy}.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, Goyal, Hartshorn, Yang, Mitra, Sravankumar, Korenev, Hinsvark, Rao, Zhang, Rodriguez, Gregerson, Spataru, Roziere, Biron, Tang, Chern, Caucheteux, Nayak, Bi, Marra, McConnell, Keller, Touret, Wu, Wong, Ferrer, Nikolaidis, Allonsius, Song, Pintz, Livshits, Esiobu, Choudhary, Mahajan, Garcia-Olano, Perino, Hupkes, Lakomkin, AlBadawy, Lobanova, Dinan, Smith, Radenovic, Zhang, Synnaeve, Lee, Anderson, Nail, Mialon, Pang, Cucurell, Nguyen, Korevaar, Xu, Touvron, Zarov, Ibarra, Kloumann, Misra, Evtimov, Copet, Lee, Geffert, Vranes, Park, Mahadeokar, Shah, van~der Linde, Billock, Hong, Lee, Fu, Chi, Huang, Liu, Wang, Yu, Bitton, Spisak, Park, Rocca, Johnstun, Saxe, Jia, Alwala, Upasani, Plawiak, Li, Heafield, Stone, El-Arini, Iyer, Malik, Chiu, Bhalla, Rantala-Yeary, van~der Maaten, Chen, Tan, Jenkins, Martin, Madaan, Malo, Blecher, Landzaat, de~Oliveira, Muzzi, Pasupuleti, Singh, Paluri, Kardas, Oldham, Rita,
Pavlova, Kambadur, Lewis, Si, Singh, Hassan, Goyal, Torabi, Bashlykov, Bogoychev, Chatterji, Duchenne, Çelebi, Alrassy, Zhang, Li, Vasic, Weng, Bhargava, Dubal, Krishnan, Koura, Xu, He, Dong, Srinivasan, Ganapathy, Calderer, Cabral, Stojnic, Raileanu, Girdhar, Patel, Sauvestre, Polidoro, Sumbaly, Taylor, Silva, Hou, Wang, Hosseini, Chennabasappa, Singh, Bell, Kim, Edunov, Nie, Narang, Raparthy, Shen, Wan, Bhosale, Zhang, Vandenhende, Batra, Whitman, Sootla, Collot, Gururangan, Borodinsky, Herman, Fowler, Sheasha, Georgiou, Scialom, Speckbacher, Mihaylov, Xiao, Karn, Goswami, Gupta, Ramanathan, Kerkez, Gonguet, Do, Vogeti, Petrovic, Chu, Xiong, Fu, Meers, Martinet, Wang, Tan, Xie, Jia, Wang, Goldschlag, Gaur, Babaei, Wen, Song, Zhang, Li, Mao, Coudert, Yan, Chen, Papakipos, Singh, Grattafiori, Jain, Kelsey, Shajnfeld, Gangidi, Victoria, Goldstand, Menon, Sharma, Boesenberg, Vaughan, Baevski, Feinstein, Kallet, Sangani, Yunus, Lupu, Alvarado, Caples, Gu, Ho, Poulton, Ryan, Ramchandani, Franco, Saraf,
Chowdhury, Gabriel, Bharambe, Eisenman, Yazdan, James, Maurer, Leonhardi, Huang, Loyd, Paola, Paranjape, Liu, Wu, Ni, Hancock, Wasti, Spence, Stojkovic, Gamido, Montalvo, Parker, Burton, Mejia, Wang, Kim, Zhou, Hu, Chu, Cai, Tindal, Feichtenhofer, Civin, Beaty, Kreymer, Li, Wyatt, Adkins, Xu, Testuggine, David, Parikh, Liskovich, Foss, Wang, Le, Holland, Dowling, Jamil, Montgomery, Presani, Hahn, Wood, Brinkman, Arcaute, Dunbar, Smothers, Sun, Kreuk, Tian, Ozgenel, Caggioni, Guzmán, Kanayet, Seide, Florez, Schwarz, Badeer, Swee, Halpern, Thattai, Herman, Sizov, Guangyi, Zhang, Lakshminarayanan, Shojanazeri, Zou, Wang, Zha, Habeeb, Rudolph, Suk, Aspegren, Goldman, Damlaj, Molybog, Tufanov, Veliche, Gat, Weissman, Geboski, Kohli, Asher, Gaya, Marcus, Tang, Chan, Zhen, Reizenstein, Teboul, Zhong, Jin, Yang, Cummings, Carvill, Shepard, McPhie, Torres, Ginsburg, Wang, Wu, U, Saxena, Prasad, Khandelwal, Zand, Matosich, Veeraraghavan, Michelena, Li, Huang, Chawla, Lakhotia, Huang, Chen, Garg, A, Silva, Bell,
Zhang, Guo, Yu, Moshkovich, Wehrstedt, Khabsa, Avalani, Bhatt, Tsimpoukelli, Mankus, Hasson, Lennie, Reso, Groshev, Naumov, Lathi, Keneally, Seltzer, Valko, Restrepo, Patel, Vyatskov, Samvelyan, Clark, Macey, Wang, Hermoso, Metanat, Rastegari, Bansal, Santhanam, Parks, White, Bawa, Singhal, Egebo, Usunier, Laptev, Dong, Zhang, Cheng, Chernoguz, Hart, Salpekar, Kalinli, Kent, Parekh, Saab, Balaji, Rittner, Bontrager, Roux, Dollar, Zvyagina, Ratanchandani, Yuvraj, Liang, Alao, Rodriguez, Ayub, Murthy, Nayani, Mitra, Li, Hogan, Battey, Wang, Maheswari, Howes, Rinott, Bondu, Datta, Chugh, Hunt, Dhillon, Sidorov, Pan, Verma, Yamamoto, Ramaswamy, Lindsay, Lindsay, Feng, Lin, Zha, Shankar, Zhang, Zhang, Wang, Agarwal, Sajuyigbe, Chintala, Max, Chen, Kehoe, Satterfield, Govindaprasad, Gupta, Cho, Virk, Subramanian, Choudhury, Goldman, Remez, Glaser, Best, Kohler, Robinson, Li, Zhang, Matthews, Chou, Shaked, Vontimitta, Ajayi, Montanez, Mohan, Kumar, Mangla, Albiero, Ionescu, Poenaru, Mihailescu, Ivanov, Li, Wang,
Jiang, Bouaziz, Constable, Tang, Wang, Wu, Wang, Xia, Wu, Gao, Chen, Hu, Jia, Qi, Li, Zhang, Zhang, Adi, Nam, Yu, Wang, Hao, Qian, He, Rait, DeVito, Rosnbrick, Wen, Yang, and Zhao]{dubey2024llama3herdmodels}
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian~Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric~Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia~Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu~Xu, Hugo Touvron, Iliyan Zarov,
Imanol~Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van~der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan~Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke~Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van~der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de~Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh~Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier
Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit~Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo~Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun~Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu,
Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing~Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi~Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie~Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto~De Paola, Bhargavi Paranjape, Bing Liu, Bo~Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl
Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela~Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James
Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam~Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael~L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel~Jubert Hermoso, Mo~Metanat, Mohammad Rastegari, Munish Bansal, Nandhini
Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay~Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai~Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin~Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta,
Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy~Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay~Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad~Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye~Hu, Ye~Jia, Ye~Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint}, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2407.21783}.
\bibitem[Durrani et~al.(2014)Durrani, Sajjad, Hoang, and Koehn]{durrani-etal-2014-integrating}
Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn.
\newblock Integrating an unsupervised transliteration model into statistical machine translation.
\newblock In Shuly Wintner, Stefan Riezler, and Sharon Goldwater (eds.), \emph{Proceedings of the 14th Conference of the {E}uropean Chapter of the Association for Computational Linguistics, volume 2: Short Papers}, pp.\ 148--153, Gothenburg, Sweden, April 2014. Association for Computational Linguistics.
\newblock \doi{10.3115/v1/E14-4029}.
\newblock URL \url{https://aclanthology.org/E14-4029/}.
\bibitem[Fahim et~al.(2025)Fahim, Murphy, and Fyshe]{fahim2025its}
Abrar Fahim, Alex Murphy, and Alona Fyshe.
\newblock It's not a modality gap: Characterizing and addressing the contrastive gap, 2025.
\newblock URL \url{https://openreview.net/forum?id=wE8wJXgI9T}.
\bibitem[Fujii et~al.(2023)Fujii, Shibata, Yamaguchi, Morishita, and Sogawa]{fujii-etal-2023-different}
Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa.
\newblock How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in {J}apanese.
\newblock In Vishakh Padmakumar, Gisela Vallejo, and Yao Fu (eds.), \emph{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)}, pp.\ 39--49, Toronto, Canada, July 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.acl-srw.5}.
\newblock URL \url{https://aclanthology.org/2023.acl-srw.5/}.
\bibitem[Gee et~al.(2022)Gee, Zugarini, Rigutini, and Torroni]{gee-etal-2022-fast}
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni.
\newblock Fast vocabulary transfer for language model compression.
\newblock In Yunyao Li and Angeliki Lazaridou (eds.), \emph{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track}, pp.\ 409--416, Abu Dhabi, UAE, December 2022. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2022.emnlp-industry.41}.
\newblock URL \url{https://aclanthology.org/2022.emnlp-industry.41/}.
\bibitem[{Gemma Team} et~al.(2024){Gemma Team}, Mesnard, Hardin, Dadashi, Bhupatiraju, Pathak, Sifre, Rivière, Kale, Love, Tafti, Hussenot, Sessa, Chowdhery, Roberts, Barua, Botev, Castro-Ros, Slone, Héliou, Tacchetti, Bulanova, Paterson, Tsai, Shahriari, Lan, Choquette-Choo, Crepy, Cer, Ippolito, Reid, Buchatskaya, Ni, Noland, Yan, Tucker, Muraru, Rozhdestvenskiy, Michalewski, Tenney, Grishchenko, Austin, Keeling, Labanowski, Lespiau, Stanway, Brennan, Chen, Ferret, Chiu, Mao-Jones, Lee, Yu, Millican, Sjoesund, Lee, Dixon, Reid, Mikuła, Wirth, Sharman, Chinaev, Thain, Bachem, Chang, Wahltinez, Bailey, Michel, Yotov, Chaabouni, Comanescu, Jana, Anil, McIlroy, Liu, Mullins, Smith, Borgeaud, Girgin, Douglas, Pandya, Shakeri, De, Klimenko, Hennigan, Feinberg, Stokowiec, hui Chen, Ahmed, Gong, Warkentin, Peran, Giang, Farabet, Vinyals, Dean, Kavukcuoglu, Hassabis, Ghahramani, Eck, Barral, Pereira, Collins, Joulin, Fiedel, Senter, Andreev, and Kenealy]{gemmateam2024gemmaopenmodelsbased}
{Gemma Team}, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir~Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier~Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline~Le Lan, Christopher~A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars~Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem,
Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel~L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu~hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.
\newblock Gemma: Open models based on gemini research and technology, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2403.08295}.
\bibitem[Goyal et~al.(2022)Goyal, Gao, Chaudhary, Chen, Wenzek, Ju, Krishnan, Ranzato, Guzm{\'a}n, and Fan]{goyal-etal-2022-flores}
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da~Ju, Sanjana Krishnan, Marc{'}Aurelio Ranzato, Francisco Guzm{\'a}n, and Angela Fan.
\newblock The {F}lores-101 evaluation benchmark for low-resource and multilingual machine translation.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 10:\penalty0 522--538, 2022.
\newblock \doi{10.1162/tacl_a_00474}.
\newblock URL \url{https://aclanthology.org/2022.tacl-1.30}.
\bibitem[J et~al.(2024)J, Dabre, M, Gala, Jayakumar, Puduppully, and Kunchukuttan]{j-etal-2024-romansetu}
Jaavid J, Raj Dabre, Aswanth M, Jay Gala, Thanmay Jayakumar, Ratish Puduppully, and Anoop Kunchukuttan.
\newblock {R}oman{S}etu: Efficiently unlocking multilingual capabilities of large language models via {R}omanization.
\newblock In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 15593--15615, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.acl-long.833}.
\newblock URL \url{https://aclanthology.org/2024.acl-long.833/}.
\bibitem[Jalili~Sabet et~al.(2020)Jalili~Sabet, Dufter, Yvon, and Sch{\"u}tze]{jalili-sabet-etal-2020-simalign}
Masoud Jalili~Sabet, Philipp Dufter, Fran{\c{c}}ois Yvon, and Hinrich Sch{\"u}tze.
\newblock {S}im{A}lign: High quality word alignments without parallel training data using static and contextualized embeddings.
\newblock In Trevor Cohn, Yulan He, and Yang Liu (eds.), \emph{Findings of the Association for Computational Linguistics: EMNLP 2020}, pp.\ 1627--1643, Online, November 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.findings-emnlp.147}.
\newblock URL \url{https://aclanthology.org/2020.findings-emnlp.147/}.
\bibitem[Jiang et~al.(2023)Jiang, Sablayrolles, Mensch, Bamford, Chaplot, de~las Casas, Bressand, Lengyel, Lample, Saulnier, Lavaud, Lachaux, Stock, Scao, Lavril, Wang, Lacroix, and Sayed]{jiang2023mistral7b}
Albert~Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra~Singh Chaplot, Diego de~las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio~Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven~Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William~El Sayed.
\newblock Mistral 7b, 2023.
\newblock URL \url{https://doi.org/10.48550/arXiv.2310.06825}.
\bibitem[Kallini et~al.(2024)Kallini, Murty, Manning, Potts, and Csordás]{kallini2024mrt5dynamictokenmerging}
Julie Kallini, Shikhar Murty, Christopher~D. Manning, Christopher Potts, and Róbert Csordás.
\newblock Mrt5: Dynamic token merging for efficient byte-level language models, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2410.20771}.
\bibitem[Kallini et~al.(2025)Kallini, Murty, Manning, Potts, and Csord{\'a}s]{kallini2025mrt}
Julie Kallini, Shikhar Murty, Christopher~D Manning, Christopher Potts, and R{\'o}bert Csord{\'a}s.
\newblock Mrt5: Dynamic token merging for efficient byte-level language models.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=VYWBMq1L7H}.
\bibitem[Kingma \& Ba(2015)Kingma and Ba]{kingma-ba-2015-adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: {A} method for stochastic optimization.
\newblock In Yoshua Bengio and Yann LeCun (eds.), \emph{Proceedings of the 3rd International Conference on Learning Representations ({ICLR})}, San Diego, CA, USA, 2015.
\newblock URL \url{http://arxiv.org/abs/1412.6980}.
\bibitem[Kittinaradorn et~al.(2019)Kittinaradorn, Achakulvisut, Chaovavanich, Srithaworn, Chormai, Kaewkasi, Ruangrong, and Oparad]{Kittinaradorn2019}
Rakpong Kittinaradorn, Titipat Achakulvisut, Korakot Chaovavanich, Kittinan Srithaworn, Pattarawat Chormai, Chanwit Kaewkasi, Tulakan Ruangrong, and Krichkorn Oparad.
\newblock {DeepCut: A Thai word tokenization library using Deep Neural Network}, September 2019.
\newblock URL \url{http://doi.org/10.5281/zenodo.3457707}.
\bibitem[Kudo(2018)]{kudo-2018-subword}
Taku Kudo.
\newblock Subword regularization: Improving neural network translation models with multiple subword candidates.
\newblock In Iryna Gurevych and Yusuke Miyao (eds.), \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 66--75, Melbourne, Australia, July 2018. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/P18-1007}.
\newblock URL \url{https://aclanthology.org/P18-1007/}.
\bibitem[Kumar et~al.(2022)Kumar, Raghunathan, Jones, Ma, and Liang]{kumar2022finetuning}
Ananya Kumar, Aditi Raghunathan, Robbie~Matthew Jones, Tengyu Ma, and Percy Liang.
\newblock Fine-tuning can distort pretrained features and underperform out-of-distribution.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\newblock URL \url{https://openreview.net/forum?id=UYneFzXSJWh}.
\bibitem[Lee et~al.(2023)Lee, Joshi, Turc, Hu, Liu, Eisenschlos, Khandelwal, Shaw, Chang, and Toutanova]{lee-etal-2023-pix2struct}
Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.
\newblock Pix2struct: screenshot parsing as pretraining for visual language understanding.
\newblock In \emph{Proceedings of the 40th International Conference on Machine Learning}, ICML'23. JMLR.org, 2023.
\bibitem[Li et~al.(2024)Li, Liu, Wu, Wang, Shen, Qu, Niu, Wang, Chen, and Li]{aria}
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li.
\newblock Aria: An open multimodal native mixture-of-experts model.
\newblock \emph{arXiv preprint}, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2410.05993}.
\bibitem[Li et~al.(2025)Li, Zhang, Zhang, Zhang, Li, Li, MA, and Li]{li2025llavanextinterleave}
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo~Li, Wei Li, Zejun MA, and Chunyuan Li.
\newblock {LL}a{VA}-ne{XT}-interleave: Tackling multi-image, video, and 3d in large multimodal models.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=oSQiao9GqB}.
\bibitem[Li et~al.(2020)Li, Chaudhari, Yang, Lam, Ravichandran, Bhotika, and Soatto]{Li2020Rethinking}
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto.
\newblock Rethinking the hyperparameters for fine-tuning.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\newblock URL \url{https://openreview.net/forum?id=B1g8VkHFPH}.
\bibitem[Li et~al.(2023)Li, Lv, Chen, Cui, Lu, Florencio, Zhang, Li, and Wei]{li2023trocr}
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei.
\newblock Trocr: Transformer-based optical character recognition with pre-trained models.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~37, pp.\ 13094--13102, 2023.
\bibitem[Liang et~al.(2023)Liang, Bommasani, Lee, Tsipras, Soylu, Yasunaga, Zhang, Narayanan, Wu, Kumar, Newman, Yuan, Yan, Zhang, Cosgrove, Manning, Re, Acosta-Navas, Hudson, Zelikman, Durmus, Ladhak, Rong, Ren, Yao, WANG, Santhanam, Orr, Zheng, Yuksekgonul, Suzgun, Kim, Guha, Chatterji, Khattab, Henderson, Huang, Chi, Xie, Santurkar, Ganguli, Hashimoto, Icard, Zhang, Chaudhary, Wang, Li, Mai, Zhang, and Koreeda]{liang2023holistic}
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce~Zhang, Christian~Alexander Cosgrove, Christopher~D Manning, Christopher Re, Diana Acosta-Navas, Drew~Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri~S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan~Andrew Chi, Sang~Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.
\newblock Holistic evaluation of language models.
\newblock \emph{Transactions on Machine Learning Research}, 2023.
\newblock ISSN 2835-8856.
\newblock URL \url{https://openreview.net/forum?id=iO4LZibEqW}.
\newblock Featured Certification, Expert Certification.
\bibitem[Liang et~al.(2022)Liang, Zhang, Kwon, Yeung, and Zou]{modality_gap}
Victor~Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James~Y Zou.
\newblock Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.
\newblock In S.~Koyejo, S.~Mohamed, A.~Agarwal, D.~Belgrave, K.~Cho, and A.~Oh (eds.), \emph{Advances in Neural Information Processing Systems}, volume~35, pp.\ 17612--17625. Curran Associates, Inc., 2022.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf}.
\bibitem[Libovický et~al.(2022)Libovický, Schmid, and Fraser]{libovický2022dontpeopleusecharacterlevel}
Jindřich Libovický, Helmut Schmid, and Alexander Fraser.
\newblock Why don't people use character-level machine translation?, 2022.
\newblock URL \url{https://doi.org/10.48550/arXiv.2110.08191}.
\bibitem[Lin et~al.(2024)Lin, Ji, Tiedemann, Martins, and Schütze]{lin2024mala500massivelanguageadaptation}
Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F.~T. Martins, and Hinrich Schütze.
\newblock Mala-500: Massive language adaptation of large language models.
\newblock \emph{arXiv preprint}, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2401.13303}.
\bibitem[Liu et~al.(2024{\natexlab{a}})Liu, Wang, Yin, Molchanov, Wang, Cheng, and Chen]{liu2024dora}
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang~Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.
\newblock Dora: Weight-decomposed low-rank adaptation.
\newblock In \emph{Proceedings of the 41 International Conference on Machine Learning}, 2024{\natexlab{a}}.
\newblock URL \url{https://doi.org/10.48550/arXiv.2402.09353}.
\bibitem[Liu et~al.(2024{\natexlab{b}})Liu, Lin, Wang, and Schuetze]{liu-etal-2024-ofa}
Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze.
\newblock {OFA}: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining.
\newblock In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), \emph{Findings of the Association for Computational Linguistics: NAACL 2024}, pp.\ 1067--1097, Mexico City, Mexico, June 2024{\natexlab{b}}. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.findings-naacl.68}.
\newblock URL \url{https://aclanthology.org/2024.findings-naacl.68/}.
\bibitem[Loshchilov \& Hutter(2017)Loshchilov and Hutter]{DBLP:conf/iclr/LoshchilovH17}
Ilya Loshchilov and Frank Hutter.
\newblock {SGDR:} stochastic gradient descent with warm restarts.
\newblock In \emph{5th International Conference on Learning Representations, {ICLR} 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings}. OpenReview.net, 2017.
\newblock URL \url{https://openreview.net/forum?id=Skq89Scxx}.
\bibitem[Loshchilov \& Hutter(2019)Loshchilov and Hutter]{loshchilov2018decoupled}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled weight decay regularization.
\newblock In \emph{Proceedings of the 7th International Conference on Learning Representations ({ICLR})}, New Orleans, LA, USA, 2019. OpenReview.net.
\newblock URL \url{https://openreview.net/forum?id=Bkg6RiCqY7}.
\bibitem[Lotz et~al.(2023)Lotz, Salesky, Rust, and Elliott]{lotz-etal-2023-text}
Jonas Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott.
\newblock Text rendering strategies for pixel language models.
\newblock In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 10155--10172, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.emnlp-main.628}.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.628}.
\bibitem[McCloskey \& Cohen(1989)McCloskey and Cohen]{MCCLOSKEY1989109}
Michael McCloskey and Neal~J. Cohen.
\newblock Catastrophic interference in connectionist networks: The sequential learning problem.
\newblock \emph{Elsevier}, 24:\penalty0 109--165, 1989.
\bibitem[Mhaske et~al.(2023)Mhaske, Kedia, Doddapaneni, Khapra, Kumar, Murthy, and Kunchukuttan]{mhaske-etal-2023-naamapadam}
Arnav Mhaske, Harshit Kedia, Sumanth Doddapaneni, Mitesh~M. Khapra, Pratyush Kumar, Rudra Murthy, and Anoop Kunchukuttan.
\newblock Naamapadam: A large-scale named entity annotated data for {I}ndic languages.
\newblock In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), \emph{Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 10441--10456, Toronto, Canada, July 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.acl-long.582}.
\newblock URL \url{https://aclanthology.org/2023.acl-long.582}.
\bibitem[Minixhofer et~al.(2024)Minixhofer, Ponti, and Vuli{\'c}]{minixhofer2024zeroshot}
Benjamin Minixhofer, Edoardo Ponti, and Ivan Vuli{\'c}.
\newblock Zero-shot tokenizer transfer.
\newblock In \emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}, 2024.
\newblock URL \url{https://openreview.net/forum?id=RwBObRsIzC}.
\bibitem[Muller et~al.(2021)Muller, Anastasopoulos, Sagot, and Seddah]{muller-etal-2021-unseen}
Benjamin Muller, Antonios Anastasopoulos, Beno{\^i}t Sagot, and Djam{\'e} Seddah.
\newblock When being unseen from m{BERT} is just the beginning: Handling new languages with multilingual language models.
\newblock In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz~Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), \emph{Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, pp.\ 448--462, Online, June 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.naacl-main.38}.
\newblock URL \url{https://aclanthology.org/2021.naacl-main.38/}.
\bibitem[Nguyen et~al.(2025)Nguyen, Muller, Yu, Costa-jussa, Elbayad, Popuri, Ropers, Duquenne, Algayres, Mavlyutov, Gat, Williamson, Synnaeve, Pino, Sagot, and Dupoux]{nguyen-etal-2025-spirit}
Tu~Anh Nguyen, Benjamin Muller, Bokai Yu, Marta~R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Beno{\^i}t Sagot, and Emmanuel Dupoux.
\newblock {S}pi{R}it-{LM}: Interleaved spoken and written language model.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 13:\penalty0 30--52, 2025.
\newblock \doi{10.1162/tacl_a_00728}.
\newblock URL \url{https://aclanthology.org/2025.tacl-1.2/}.
\bibitem[{NLLB Team} et~al.(2022){NLLB Team}, Costa-jussà, Cross, Çelebi, Elbayad, Heafield, Heffernan, Kalbassi, Lam, Licht, Maillard, Sun, Wang, Wenzek, Youngblood, Akula, Barrault, Mejia-Gonzalez, Hansanti, Hoffman, Jarrett, Sadagopan, Rowe, Spruit, Tran, Andrews, Ayan, Bhosale, Edunov, Fan, Gao, Goswami, Guzmán, Koehn, Mourachko, Ropers, Saleem, Schwenk, and Wang]{nllb-22}
{NLLB Team}, Marta~R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al~Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik~Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip~Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang.
\newblock No language left behind: Scaling human-centered machine translation.
\newblock \emph{arXiv preprint}, 2022.
\bibitem[Pagnoni et~al.(2024)Pagnoni, Pasunuru, Rodriguez, Nguyen, Muller, Li, Zhou, Yu, Weston, Zettlemoyer, Ghosh, Lewis, Holtzman, and Iyer]{pagnoni2024bytelatenttransformerpatches}
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer.
\newblock Byte latent transformer: Patches scale better than tokens, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2412.09871}.
\bibitem[Peters et~al.(2019)Peters, Ruder, and Smith]{peters-etal-2019-tune}
Matthew~E. Peters, Sebastian Ruder, and Noah~A. Smith.
\newblock To tune or not to tune? adapting pretrained representations to diverse tasks.
\newblock In Isabelle Augenstein, Spandana Gella, Sebastian Ruder, Katharina Kann, Burcu Can, Johannes Welbl, Alexis Conneau, Xiang Ren, and Marek Rei (eds.), \emph{Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)}, pp.\ 7--14, Florence, Italy, August 2019. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/W19-4302}.
\newblock URL \url{https://aclanthology.org/W19-4302/}.
\bibitem[Pfeiffer et~al.(2020)Pfeiffer, Vuli{\'c}, Gurevych, and Ruder]{pfeiffer-etal-2020-mad}
Jonas Pfeiffer, Ivan Vuli{\'c}, Iryna Gurevych, and Sebastian Ruder.
\newblock {MAD-X}: {A}n {A}dapter-{B}ased {F}ramework for {M}ulti-{T}ask {C}ross-{L}ingual {T}ransfer.
\newblock In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), \emph{Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, pp.\ 7654--7673, Online, November 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.emnlp-main.617}.
\newblock URL \url{https://aclanthology.org/2020.emnlp-main.617/}.
\bibitem[Pfeiffer et~al.(2021)Pfeiffer, Vuli{\'c}, Gurevych, and Ruder]{pfeiffer-etal-2021-unks}
Jonas Pfeiffer, Ivan Vuli{\'c}, Iryna Gurevych, and Sebastian Ruder.
\newblock {UNK}s everywhere: {A}dapting multilingual language models to new scripts.
\newblock In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), \emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pp.\ 10186--10203, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.emnlp-main.800}.
\newblock URL \url{https://aclanthology.org/2021.emnlp-main.800/}.
\bibitem[Pinter et~al.(2017)Pinter, Guthrie, and Eisenstein]{pinter-etal-2017-mimicking}
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein.
\newblock Mimicking word embeddings using subword {RNN}s.
\newblock In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), \emph{Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing}, pp.\ 102--112, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/D17-1010}.
\newblock URL \url{https://aclanthology.org/D17-1010/}.
\bibitem[Popovi{\'c}(2015)]{popovic-2015-chrf}
Maja Popovi{\'c}.
\newblock chr{F}: character n-gram {F}-score for automatic {MT} evaluation.
\newblock In Ond{\v{r}}ej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), \emph{Proceedings of the Tenth Workshop on Statistical Machine Translation}, pp.\ 392--395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/W15-3049}.
\newblock URL \url{https://aclanthology.org/W15-3049}.
\bibitem[Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, and Sutskever]{radford2019language}
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
\newblock Language models are unsupervised multitask learners, 2019.
\bibitem[Rahman et~al.(2023)Rahman, Sakib, Faisal, and Anastasopoulos]{rahman-etal-2023-token}
Md~Mushfiqur Rahman, Fardin~Ahsan Sakib, Fahim Faisal, and Antonios Anastasopoulos.
\newblock To token or not to token: A comparative study of text representations for cross-lingual transfer.
\newblock In Duygu Ataman (ed.), \emph{Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)}, pp.\ 67--84, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.mrl-1.6}.
\newblock URL \url{https://aclanthology.org/2023.mrl-1.6/}.
\bibitem[Ren et~al.(2023)Ren, Guo, Bae, and Sutherland]{ren2023how}
Yi~Ren, Shangmin Guo, Wonho Bae, and Danica~J. Sutherland.
\newblock How to prepare your task head for finetuning.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\newblock URL \url{https://openreview.net/forum?id=gVOXZproe-e}.
\bibitem[Rolfe(2017)]{rolfe2017discrete}
Jason~Tyler Rolfe.
\newblock Discrete variational autoencoders.
\newblock In \emph{International Conference on Learning Representations}, 2017.
\newblock URL \url{https://openreview.net/forum?id=ryMxXPFex}.
\bibitem[Rust et~al.(2021)Rust, Pfeiffer, Vuli{\'c}, Ruder, and Gurevych]{rust-etal-2021-good}
Phillip Rust, Jonas Pfeiffer, Ivan Vuli{\'c}, Sebastian Ruder, and Iryna Gurevych.
\newblock How good is your tokenizer? on the monolingual performance of multilingual language models.
\newblock In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), \emph{Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, pp.\ 3118--3135, Online, August 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.acl-long.243}.
\newblock URL \url{https://aclanthology.org/2021.acl-long.243}.
\bibitem[Rust et~al.(2023)Rust, Lotz, Bugliarello, Salesky, de~Lhoneux, and Elliott]{rust2023language}
Phillip Rust, Jonas~F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de~Lhoneux, and Desmond Elliott.
\newblock Language modelling with pixels.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\newblock URL \url{https://openreview.net/forum?id=FkSp8VW8RjH}.
\bibitem[Salesky et~al.(2021)Salesky, Etter, and Post]{salesky-etal-2021-robust}
Elizabeth Salesky, David Etter, and Matt Post.
\newblock Robust open-vocabulary translation from visual text representations.
\newblock In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), \emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pp.\ 7235--7252, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.emnlp-main.576}.
\newblock URL \url{https://aclanthology.org/2021.emnlp-main.576/}.
\bibitem[Salesky et~al.(2023)Salesky, Verma, Koehn, and Post]{salesky-etal-2023-multilingual}
Elizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post.
\newblock Multilingual pixel representations for translation and effective cross-lingual transfer.
\newblock In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 13845--13861, Singapore, December 2023. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2023.emnlp-main.854}.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.854}.
\bibitem[Schick \& Sch{\"u}tze(2019)Schick and Sch{\"u}tze]{schick-schutze-2019-attentive}
Timo Schick and Hinrich Sch{\"u}tze.
\newblock Attentive mimicking: Better word embeddings by attending to informative contexts.
\newblock In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), \emph{Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}, pp.\ 489--494, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/N19-1048}.
\newblock URL \url{https://aclanthology.org/N19-1048/}.
\bibitem[Schuster \& Nakajima(2012)Schuster and Nakajima]{schuster-japanese-2012}
Mike Schuster and Kaisuke Nakajima.
\newblock Japanese and korean voice search.
\newblock In \emph{2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pp.\ 5149--5152, 2012.
\newblock \doi{10.1109/ICASSP.2012.6289079}.
\bibitem[Sennrich et~al.(2016)Sennrich, Haddow, and Birch]{sennrich-etal-2016-neural}
Rico Sennrich, Barry Haddow, and Alexandra Birch.
\newblock Neural machine translation of rare words with subword units.
\newblock In Katrin Erk and Noah~A. Smith (eds.), \emph{Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1715--1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/P16-1162}.
\newblock URL \url{https://aclanthology.org/P16-1162/}.
\bibitem[Shi et~al.(2023)Shi, Welle, Bj{\"o}rkman, and Kragic]{shi2023towards}
Peiyang Shi, Michael~C. Welle, M{\r{a}}rten Bj{\"o}rkman, and Danica Kragic.
\newblock Towards understanding the modality gap in {CLIP}.
\newblock In \emph{ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls}, 2023.
\newblock URL \url{https://openreview.net/forum?id=8W3KGzw7fNI}.
\bibitem[Tai et~al.(2024)Tai, Liao, Suglia, and Vergari]{tai-etal-2024-pixar}
Yintao Tai, Xiyang Liao, Alessandro Suglia, and Antonio Vergari.
\newblock {PIXAR}: Auto-regressive language modeling in pixel space.
\newblock In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), \emph{Findings of the Association for Computational Linguistics: ACL 2024}, pp.\ 14673--14695, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.findings-acl.874}.
\newblock URL \url{https://aclanthology.org/2024.findings-acl.874/}.
\bibitem[Tarunesh et~al.(2021)Tarunesh, Kumar, and Jyothi]{tarunesh-etal-2021-machine}
Ishan Tarunesh, Syamantak Kumar, and Preethi Jyothi.
\newblock From machine translation to code-switching: Generating high-quality code-switched text.
\newblock In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), \emph{Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, pp.\ 3154--3169, Online, August 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.acl-long.245}.
\newblock URL \url{https://aclanthology.org/2021.acl-long.245}.
\bibitem[Tiedemann(2012)]{TIEDEMANN_OPUS2012}
Jörg Tiedemann.
\newblock Parallel data, tools and interfaces in {OPUS}.
\newblock In \emph{Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12)}, Istanbul, Turkey, may 2012. European Language Resources Association (ELRA).
\newblock ISBN 978-2-9517408-7-7.
\bibitem[Toraman et~al.(2023)Toraman, Yilmaz, \c{S}ahinu\c{c}, and Ozcelik]{toraman-etal-2023-impact}
Cagri Toraman, Eyup~Halit Yilmaz, Furkan \c{S}ahinu\c{c}, and Oguzhan Ozcelik.
\newblock Impact of tokenization on language models: An analysis for turkish.
\newblock \emph{ACM Trans. Asian Low-Resour. Lang. Inf. Process.}, 22\penalty0 (4), March 2023.
\newblock ISSN 2375-4699.
\newblock \doi{10.1145/3578707}.
\newblock URL \url{https://doi.org/10.1145/3578707}.
\bibitem[Touvron et~al.(2023{\natexlab{a}})Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave, and Lample]{touvron2023llamaopenefficientfoundation}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
\newblock Llama: Open and efficient foundation language models, 2023{\natexlab{a}}.
\newblock URL \url{https://doi.org/10.48550/arXiv.2302.13971}.
\bibitem[Touvron et~al.(2023{\natexlab{b}})Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale, Bikel, Blecher, Ferrer, Chen, Cucurull, Esiobu, Fernandes, Fu, Fu, Fuller, Gao, Goswami, Goyal, Hartshorn, Hosseini, Hou, Inan, Kardas, Kerkez, Khabsa, Kloumann, Korenev, Koura, Lachaux, Lavril, Lee, Liskovich, Lu, Mao, Martinet, Mihaylov, Mishra, Molybog, Nie, Poulton, Reizenstein, Rungta, Saladi, Schelten, Silva, Smith, Subramanian, Tan, Tang, Taylor, Williams, Kuan, Xu, Yan, Zarov, Zhang, Fan, Kambadur, Narang, Rodriguez, Stojnic, Edunov, and Scialom]{touvron2023llama2openfoundation}
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian~Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit~Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric~Michael Smith, Ranjan Subramanian, Xiaoqing~Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian~Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.
\newblock Llama 2: Open foundation and fine-tuned chat models, 2023{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2307.09288}.
\bibitem[Tschannen et~al.(2023)Tschannen, Mustafa, and Houlsby]{tschannen2023clippo}
Michael Tschannen, Basil Mustafa, and Neil Houlsby.
\newblock Image-and-language understanding from pixels only.
\newblock \emph{2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2023.
\newblock URL \url{https://doi.org/10.48550/arXiv.2212.08045}.
\bibitem[Tschannen et~al.(2025)Tschannen, Pinto, and Kolesnikov]{tschannen2025jetformer}
Michael Tschannen, Andr{\'e}~Susano Pinto, and Alexander Kolesnikov.
\newblock Jetformer: An autoregressive generative model of raw images and text.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\newblock URL \url{https://openreview.net/forum?id=sgAp2qG86e}.
\bibitem[van~den Oord et~al.(2017)van~den Oord, Vinyals, and Kavukcuoglu]{vandenoord2017neuraldiscrete}
Aaron van~den Oord, Oriol Vinyals, and Koray Kavukcuoglu.
\newblock Neural discrete representation learning.
\newblock In \emph{Proceedings of the 31st International Conference on Neural Information Processing Systems}, NIPS'17, pp.\ 6309–6318, Red Hook, NY, USA, 2017. Curran Associates Inc.
\newblock ISBN 9781510860964.
\bibitem[van Esch et~al.(2022)van Esch, Lucassen, Ruder, Caswell, and Rivera]{van-esch-etal-2022-writing}
Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, and Clara Rivera.
\newblock Writing system and speaker metadata for 2,800+ language varieties.
\newblock In Nicoletta Calzolari, Fr{\'e}d{\'e}ric B{\'e}chet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, H{\'e}l{\`e}ne Mazo, Jan Odijk, and Stelios Piperidis (eds.), \emph{Proceedings of the Thirteenth Language Resources and Evaluation Conference}, pp.\ 5035--5046, Marseille, France, June 2022. European Language Resources Association.
\newblock URL \url{https://aclanthology.org/2022.lrec-1.538/}.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{NIPS2017_vaswani_attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan~N Gomez, \L~ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In I.~Guyon, U.~Von Luxburg, S.~Bengio, H.~Wallach, R.~Fergus, S.~Vishwanathan, and R.~Garnett (eds.), \emph{Advances in Neural Information Processing Systems}, volume~30. Curran Associates, Inc., 2017.
\newblock URL \url{https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf}.
\bibitem[Wang et~al.(2019)Wang, Cho, and Gu]{wang2019neuralmachinetranslationbytelevel}
Changhan Wang, Kyunghyun Cho, and Jiatao Gu.
\newblock Neural machine translation with byte-level subwords, 2019.
\newblock URL \url{https://doi.org/10.48550/arXiv.1909.03341}.
\bibitem[Wang \& Isola(2020)Wang and Isola]{wang-isola-2020-align-uniform}
Tongzhou Wang and Phillip Isola.
\newblock Understanding contrastive representation learning through alignment and uniformity on the hypersphere.
\newblock In \emph{Proceedings of the 37th International Conference on Machine Learning}, ICML'20. JMLR.org, 2020.
\bibitem[Wang et~al.(2024)Wang, Zhang, Luo, Sun, Cui, Wang, Zhang, Wang, Li, Yu, Zhao, Ao, Min, Li, Wu, Zhao, Zhang, Wang, Liu, He, Yang, Liu, Lin, Huang, and Wang]{wang2024emu3nexttokenpredictionneed}
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo~Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi~Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang.
\newblock Emu3: Next-token prediction is all you need, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2409.18869}.
\bibitem[Wang et~al.(2020)Wang, K, Mayhew, and Roth]{wang-etal-2020-extending}
Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth.
\newblock Extending multilingual {BERT} to low-resource languages.
\newblock In Trevor Cohn, Yulan He, and Yang Liu (eds.), \emph{Findings of the Association for Computational Linguistics: EMNLP 2020}, pp.\ 2649--2656, Online, November 2020. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2020.findings-emnlp.240}.
\newblock URL \url{https://aclanthology.org/2020.findings-emnlp.240/}.
\bibitem[Xue et~al.(2022)Xue, Barua, Constant, Al-Rfou, Narang, Kale, Roberts, and Raffel]{xue-etal-2022-byt5}
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel.
\newblock {B}y{T}5: Towards a token-free future with pre-trained byte-to-byte models.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 10:\penalty0 291--306, 2022.
\newblock \doi{10.1162/tacl_a_00461}.
\newblock URL \url{https://aclanthology.org/2022.tacl-1.17}.
\bibitem[Yamaguchi et~al.(2024)Yamaguchi, Villavicencio, and Aletras]{yamaguchi2024effectivelyexpandvocabularyllms}
Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras.
\newblock How can we effectively expand the vocabulary of llms with 0.01gb of target language text?, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2406.11477}.
\bibitem[Yaras et~al.(2024)Yaras, Chen, Wang, and Qu]{yaras2024explainingmitigatingmodalitygap}
Can Yaras, Siyi Chen, Peng Wang, and Qing Qu.
\newblock Explaining and mitigating the modality gap in contrastive multimodal learning, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2412.07909}.
\bibitem[Yu et~al.(2023)Yu, Simig, Flaherty, Aghajanyan, Zettlemoyer, and Lewis]{yu2023megabyte}
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis.
\newblock {MEGABYTE}: Predicting million-byte sequences with multiscale transformers.
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing Systems}, 2023.
\newblock URL \url{https://openreview.net/forum?id=JTmO2V9Xpz}.
\bibitem[Yu et~al.(2024)Yu, Weber, Deng, Shen, Cremers, and Chen]{yu2024titok}
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen.
\newblock An image is worth 32 tokens for reconstruction and generation.
\newblock \emph{arXiv preprint}, 2024.
\bibitem[Zhao et~al.(2024)Zhao, Zhang, Gao, Zhang, Gui, and Huang]{zhao2024llamaenglishempiricalstudy}
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi~Zhang, Tao Gui, and Xuanjing Huang.
\newblock Llama beyond english: An empirical study on language capability transfer, 2024.
\newblock URL \url{https://doi.org/10.48550/arXiv.2401.01055}.
\bibitem[Zheng et~al.(2025)Zheng, Zhao, Wang, Wu, Dong, Wang, Wang, Du, Bo, Sharma, Li, Zhang, Hu, Thakker, and Kong]{evabyte}
Lin Zheng, Xueliang Zhao, Guangtao Wang, Chen Wu, David Dong, Angela Wang, Mingran Wang, Yun Du, Haige Bo, Amol Sharma, Bo~Li, Kejie Zhang, Changran Hu, Urmish Thakker, and Lingpeng Kong.
\newblock Evabyte: Efficient byte-level language models at scale, 2025.
\newblock URL \url{https://hkunlp.github.io/blog/2025/evabyte}.
\end{thebibliography}