From Language Models over Tokens to Language Models over Characters
Extracted source text · arXiv 2412.03719
\pdfoutput=1
\documentclass{article} %
\usepackage{iclr2024_conference}
\iclrfinaltrue
\usepackage{times}
\usepackage{cancel}
\usepackage{amssymb}
\usepackage{mleftright}
\usepackage{latexsym}
\usepackage{inconsolata}
\usepackage{xspace}
\usepackage{multicol}
\usepackage{setspace}
\usepackage{framed}
\usepackage{tocloft}
\usepackage{wrapfig}
\usepackage{enumitem}
\setitemize{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt,leftmargin=*}
%\usepackage{minted}
\usepackage[frozencache,cachedir=minted-cache]{minted}
\setminted{linenos, firstnumber=last, escapeinside=@@, numbersep=4pt}
\usepackage{tikz}
\usetikzlibrary{tikzmark}
\usetikzlibrary{arrows}
\usetikzlibrary{arrows.meta}
\usetikzlibrary{shapes}
\usetikzlibrary{shapes.arrows}
\usetikzlibrary{shapes.misc}
\usetikzlibrary{shapes.geometric}
\usetikzlibrary{automata,arrows}
\usetikzlibrary{positioning}
\usetikzlibrary{backgrounds}
\usetikzlibrary{decorations.pathreplacing}
\usetikzlibrary{decorations.pathmorphing}
\usetikzlibrary{decorations.markings}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[flushmargin,hang]{footmisc}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{bbm}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{dsfont}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{pgf}
\usepackage{fp}
\usepackage[hypertexnames=false, bookmarks=false]{hyperref}
\usepackage{adjustbox}
\usepackage{booktabs}
\usepackage{makecell}
\usepackage{nicefrac}
\usepackage{svg}
\usepackage{lipsum}
\usepackage{thmtools}
\usepackage{thm-restate}
\usepackage{scalerel}
\usepackage{float}
\usepackage{flafter}
\usepackage{relsize}
\usepackage{mathtools}
\usepackage{tcolorbox}
\usepackage{colortbl}
\input{macros}
\hypersetup{
colorlinks=true,
linkcolor=blue!35!black, %
citecolor=blue!35!black, %
urlcolor=blue!35!black %
}
\usepackage{colortbl}
\definecolor{ETHBlue}{RGB}{33,92,175} %
\definecolor{ETHGreen}{RGB}{98,115,19} %
\definecolor{ETHPurpleDark}{RGB}{140,10,89} %
\definecolor{ETHPurple}{RGB}{163,7,116} %
\definecolor{ETHGray}{RGB}{111,111,111} %
\definecolor{ETHRed}{RGB}{183,53,45} %
\definecolor{ETHPetrol}{RGB}{0,120,148} %
\definecolor{ETHBronze}{RGB}{142,103,19} %
\definecolor{ETHOrange}{RGB}{230, 100, 50}
\colorlet{MacroColor}{ETHPetrol}
\colorlet{MACROCOLOR}{MacroColor}
\usepackage{tikz}
\usetikzlibrary{automata, fit, positioning, matrix, arrows.meta, decorations.pathmorphing, shapes, shapes.multipart, chains,shadows.blur}
\usepackage{tikz-qtree}
\usepackage{tikz-dependency}
\tikzset{
transition/.style={-{Latex[length=1.8mm, width=1.4mm]}},
every state/.style={thick, fill=gray!10},
every loop/.append style=-{Latex[length=1.8mm, width=1.4mm]},
initial text=$ $,
line width=0.7pt,
node distance=10mm,
minimum size=4mm,
}
\setlength{\tabcolsep}{8pt} %
\renewcommand{\arraystretch}{1.} %
\title{%
From Language Models over Tokens \\
to Language Models over Characters%
}
\newcommand{\ethz}{\ensuremath{{}^{{\color{black}1}}}}
\newcommand{\cuny}{\ensuremath{{}^{{\color{black}4}}}}
\newcommand{\mcgill}{\ensuremath{{}^{{\color{black}3}}}}
\newcommand{\mila}{\ensuremath{{}^{{\color{black}2}}}}
\newcommand{\cifar}{\ensuremath{{}^{{\color{black}5}}}}
\usepackage{emoji}
\author{%
\textbf{Tim Vieira}\thanks{Please direct correspondence to \texttt{\href{mailto:tim.f.vieira@gmail.com}{tim.f.vieira@gmail.com}} and
\texttt{\href{mailto:ryan.cotterell@inf.ethz.ch}{ryan.cotterell@inf.ethz.ch}}.
}${\,\,\,}^{,}${\ethz} \quad
\textbf{Ben LeBrun}\mila \quad
\textbf{Mario Giulianelli}\ethz \quad
\textbf{Juan Luis Gastaldi}\ethz \quad
\textbf{Brian DuSell}\ethz \quad \\
\textbf{John Terilla}\cuny \quad
\textbf{Timothy J. O'Donnell}\mcgill$^,$\mila$^,$\cifar \quad
\textbf{Ryan Cotterell}\ethz
\\
{\ethz}ETH Zürich \quad
{\mila}Mila -- Quebec Artificial Intelligence Institute\quad
{\mcgill}McGill University \quad \\
{\cuny}City University of New York \quad
{\cifar}Canada CIFAR AI Chair, Mila
}
\begin{document}
\maketitle
\begin{abstract}
Modern language models are internally---and mathematically---distributions over \emph{token} strings rather than \emph{character} strings, posing numerous challenges for programmers building user applications on top of them. For example, if a prompt is specified as a character string, it must be tokenized before passing it to the token-level language model. Thus, the tokenizer and consequent analyses are very sensitive to the specification of the prompt (e.g., if the prompt ends with a space or not). This paper presents algorithms for converting token-level language models to character-level ones. We present both exact and approximate algorithms.
In the empirical portion of the paper, we benchmark the practical runtime and approximation quality. We find that---even with a small computation budget---our method is able to accurately approximate the character-level distribution (less than 0.00021 excess bits / character) at reasonably fast speeds (46.3 characters / second) on the Llama 3.1 8B language model.
\\
\\
\emoji[twitter]{github} \url{https://github.com/rycolab/token-to-char-lm}
\\[.5\baselineskip]
\end{abstract}
\section{Introduction}
\label{sec:introduction}
For various technical reasons, modern language models are engineered as probability distributions over strings of \emph{tokens} rather than strings of \emph{characters}.
However, this leads to a fundamental tension between the users of large language models and the engineers who build them.
Specifically, token-level models are rife with unintuitive behaviors that---without a technical fix---baffle users.
As an illustrative example of a common user complaint, we exhibit the prompt boundary problem (see below). This paper provides a principled solution to the prompt boundary problem as well as other oddities that make interfacing with token-level language models with character-level prompts hard for users.\looseness=-1
\paragraph{Tokenized language models: A brief overview.}
Let $\charAlphabet$ be an alphabet of characters, and let $\charStrings$ denote the set of all strings that can be built from it.
Suppose there is a true distribution $\pchars^*$ over $\charStrings$ that we seek to model.
We observe a training corpus of character strings: $\charseq^{(1)}, \ldots, \charseq^{(M)} \simIID \pchars^*$. However, rather than estimating a language model that approximates $\pchars^*$ directly, we employ a (possibly stochastic)\footnote{The tokenizer $\tokenizer$ is often deterministic (e.g., byte-pair encoding \citep{sennrich-etal-2016-neural}), but stochastic tokenizers (e.g., UnigramLM \citep{kudo-2018-subword}) are also of interest to the community.
} tokenizer $\tokenizer$ that transforms the training corpus into a corpus of \emph{token} strings.
$\tokseq^{(1)} \sim \tokenizer(\cdot \mid \charseq^{(1)}),
\ldots,
\tokseq^{(M)} \sim \tokenizer(\cdot \mid \charseq^{(M)})$.
Next, we estimate a token-level language model to fit the strings $\tokseq^{(1)}, \ldots, \tokseq^{(M)}$.
Lastly, we use $\ptokens$ to generate character strings by means of the following generative process: (i) sample $\tokseq \sim \ptokens$ and (ii) return $\charseq = \cotokenizer(\tokseq)$ where $\cotokenizer$ is a decoding function. Let $\pchars$ denote the resulting distribution of this process. Practically, we hope that the choice of $\tokenizer$ and $\cotokenizer$ should aid in our ability to estimate $\pchars^*$ in the sense that $\pchars^* \approx \pchars$.\footnote{\citet{gastaldi2024foundations} give a general characterization of which tokenizers allow consistent statistical estimation.}
Commonly used tokenizers in the realm of LLMs use tokenizers $\tokenizer$ that break long strings into chunks.
Intuitively, generating chunks instead of individual characters helps because it effectively shortens strings without obfuscating them using a complicated encoding.
\paragraph{The prompt boundary problem.}
Consider the case of \texttt{GPT2} \citep{radford2019language}, which was trained over token strings created from byte-pair encoding (BPE; \citet{sennrich-etal-2016-neural,gage1994bpe}). Suppose we wish to generate continuations of the prompt:
\quad$\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the}$
Unfortunately, the language model $\ptokens$ does not accept a \emph{character} string; thus, it is common to encode it as a \emph{token} string with an encoding function $\tokenizerBPE$:\footnote{In practice, $\tokenizerBPE$ outputs an integer sequence; however, we provide the substring {\tt\scriptsize\color{TokenColor} gloss} for readability.}$^,$\footnote{Here we write $\tokseq = \tokenizer(\chars)$ instead of $\tokseq \sim \tokenizer(\cdot \mid \charseq)$ since $\tokenizer$ is deterministic (i.e., a function).}
\quad$\tokenizerBPE(\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the}) = [%
\tokenfont[1]{"},
\tokenfont[818]{In},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[13239]{{\textvisiblespace}kingdom},
\tokenfont[286]{{\textvisiblespace}of},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[7770]{{\textvisiblespace}blind},
\tokenfont[11]{,},
\tokenfont[262]{{\textvisiblespace}the}%
]$
If we complete the prompt by taking the most likely next token (also known as \emph{greedy completion}), we generate the following continuation:
\quad$[%
\tokenfont[530]{{\textvisiblespace}one},
\tokenfont[12]{-},
\tokenfont[18834]{eyed},
\tokenfont[582]{{\textvisiblespace}man},
\tokenfont[318]{{\textvisiblespace}is},
\tokenfont[5822]{{\textvisiblespace}king},
\tokenfont[526]{."}%
]$
Now that we have our generated output, we can apply the decoding function $\cotokenizer$ that maps token strings to character strings as part of the tokenization protocol:
\quad$\cotokenizerBPE([%
\tokenfont[1]{"},
\tokenfont[818]{In},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[13239]{{\textvisiblespace}kingdom},
\tokenfont[286]{{\textvisiblespace}of},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[7770]{{\textvisiblespace}blind},
\tokenfont[11]{,},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[530]{{\textvisiblespace}one},
\tokenfont[12]{-},
\tokenfont[18834]{eyed},
\tokenfont[582]{{\textvisiblespace}man},
\tokenfont[318]{{\textvisiblespace}is},
\tokenfont[5822]{{\textvisiblespace}king},
\tokenfont[526]{."}%
])$
\quad$=
\charfont{"}
\charfont{In}
\charfont{{\textvisiblespace}the}
\charfont{{\textvisiblespace}kingdom}
\charfont{{\textvisiblespace}of}
\charfont{{\textvisiblespace}the}
\charfont{{\textvisiblespace}blind}
\charfont{,}
\charfont{{\textvisiblespace}the}
\charfont{{\textvisiblespace}one}
\charfont{-}
\charfont{eyed}
\charfont{{\textvisiblespace}man}
\charfont{{\textvisiblespace}is}
\charfont{{\textvisiblespace}king}
\charfont{."}
$
This is a good completion, as the string is a well-known proverb. However, if we tweak the prompt \emph{ever so slightly} by inserting a trailing whitespace:
\quad$\tokenizerBPE(\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the{\textvisiblespace}}) = [
\tokenfont[1]{"},
\tokenfont[818]{In},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[13239]{{\textvisiblespace}kingdom},
\tokenfont[286]{{\textvisiblespace}of},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[7770]{{\textvisiblespace}blind},
\tokenfont[11]{,},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[220]{{\textvisiblespace}}
]$
greedy completion returns%
\quad$[
\tokenfont[2171]{ills},
\tokenfont[286]{{\textvisiblespace}of},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[995]{{\textvisiblespace}world},
\tokenfont[389]{{\textvisiblespace}are},
\tokenfont[1775]{{\textvisiblespace}seen},
\cdots
]$
This happens because the conditional probability ($\prefixTokens$) of generating the characters we want becomes extremely unlikely:
\begin{center}
\vspace{-\baselineskip}
\begin{tabular}{rll}
Before: & $0.98$ & $= \prefixTokens(\tokenfont[530]{{\textvisiblespace}one} \, \mid \tokenizerBPE(
\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the}
))$
\\
After: & $0.0000005$ & $= \prefixTokens(\tokenfont[505]{one} \, \mid \tokenizerBPE(
\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the{\textvisiblespace}}
))$
\end{tabular}
\vspace{-\baselineskip}
\end{center}
Hence, the trailing white space undesirably impacts the output, which no longer corresponds to the proverb. This behavior is odd, as we would expect the most likely continuation (shown above) to be invariant to whether or not the whitespace character (\charfont{\textvisiblespace}) is in the prompt or the continuation. More broadly, the \emph{prompt boundary problem} may be generally characterized as an unwanted sensitivity to the characters at the boundary and the continuation. Our perspective is that the prompt boundary problem arises from incorrectly conditioning the \emph{token-level} language model on a string of \emph{characters} by using $\tokenizer(\charseq)$ rather than finding token strings that best match the prompt $\charseq$.
$\hookrightarrow$ \emph{The token healing heuristic:} \citet{token-healing} present \emph{token healing} as a heuristic that mitigates the prompt boundary problem. Token healing works as follows:
(1) Tokenize (i.e., encode) the prompt.
(2) Backup the tokenized prompt to the penultimate token.
(3) Generate the next token subject to the constraint that it starts with the unmatched substring at the end of the prompt.
(4) Continue generating as usual.
Now, we can see how token healing patches the running example:
\quad$\tokenizerBPE(\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind{\textvisiblespace}}) = [
\tokenfont[1]{"},
\tokenfont[818]{In},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[13239]{{\textvisiblespace}kingdom},
\tokenfont[286]{{\textvisiblespace}of},
\tokenfont[262]{{\textvisiblespace}the},
\tokenfont[7770]{{\textvisiblespace}blind},
\tokenfont[11]{,},
\tokenfont[262]{{\textvisiblespace}the},
\cancel{\tokenfont[220]{{\textvisiblespace}}}
]$
The most probable next token starting with \charfont{{\textvisiblespace}} is \tokenfont[530]{{\textvisiblespace}one}. From here, generating the remaining tokens recovers the desired output because it matches the prompt before we added the whitespace.
Unfortunately, backing up one token is insufficient for the general case, as the following example will illustrate. Consider generating from \texttt{GPT2} using the prompt \charfont{Hello,{\textvisiblespace}worl}:
\quad$\tokenizerBPE(\charfont{Hello,{\textvisiblespace}worl}) = [%
\tokenfont[15496]{Hello},
\tokenfont[11]{,},
\tokenfont[476]{{\textvisiblespace}wor},
\tokenfont[75]{l}
]$
The most likely next \emph{character} ought to be \charfont{d} as \charfont{Hello,{\textvisiblespace}world} is a common expression (popularized in educational material). However, the most likely token is \charfont{Hello,{\textvisiblespace}worlwide}, an apparent misspelling of \charfont{worldwide}.\footnote{The misspelling is a testament to the extent to which the tokenized prompt is out-of-distribution.} Unfortunately, token healing's strategy of backing up by a single token cannot salvage the poor tokenization as \tokenfont[75]{l} is still the most common next token that is consistent with the string \charfont{l}.
Thus, after generating \tokenfont[75]{l}, we are back where we started, generating $[%
\tokenfont[15496]{Hello},
\tokenfont[11]{,},
\tokenfont[476]{{\textvisiblespace}wor},
\tokenfont[75]{l},
\tokenfont[4421]{wide}
]$.
$\hookrightarrow$ \emph{Getting it right:}
A simple ``probability 101'' expression tells us the correct solution to the prompt boundary problem.
Consider a $\tokStrings$-valued random variable $\Y$, distributed
according to $\ptokens$.
Then, the correct way to sample from $\ptokens$ conditioned on a character string $\charseq$ is according to
\begin{equation}
\charCond(\tokseq \mid \charseq) \defeq \Pr[\Y \sim \ptokens]{ \Y \!=\! \tokseq \,\middle|\, \cotokenizer(\Y) \succeq \charseq}
\end{equation}
where we have conditioned on the event that the decoded string $\cotokenizer(\Y)$ has $\charseq$ as a prefix (i.e., $\cotokenizer(\Y) \succeq \charseq$).
While innovative with respect to the literature, the expression $\Pr[\Y \sim \ptokens]{ \Y \!=\! \tokseq \,\middle|\, \cotokenizer(\Y) \succeq \charseq}$ conveys the probability we are interested in precisely and concisely.
For the more procedurally minded, this corresponds to the following generation process:\footnote{We adopt the common notational convention of probability theory where random variables are denoted by upper-case symbols (e.g., $\Y$), and values by lower-case symbols (e.g., $\tokseq$).\looseness=-1}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def conditional_token_generation(@$\charseq$@):
while True:
sample @$\tokseq \sim \ptokens$@
if @$\cotokenizer(\tokseq) \succeq \charseq$@: # accept
return @$\tokseq$@
else:
pass # try again...
\end{minted}
\end{minipage}
\end{center}
This is, of course, very inefficient, but fear not---we will provide an equivalent, efficient algorithm.
\begin{wrapfigure}{r}{5.25cm} %
\vspace{-\baselineskip} %
\custombox{\begin{minipage}{5cm}\vspace{0pt} %
{\scriptsize
\begin{tabular}{@{\hspace{1.25cm}}r@{\hspace{.25em}}l}
\probbar{0.9820714} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}worl\underline{d}}] \\
\probbar{0.0106702} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[11621]{{\textvisiblespace}worl\underline{ds}}] \\
\probbar{0.0070749} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[8688]{{\textvisiblespace}worl\underline{dwide}}] \\
\probbar{0.0000830} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[43249]{{\textvisiblespace}worl\underline{dly}}] \\
\probbar{0.0000369} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[29081]{{\textvisiblespace}worl\underline{dview}}] \\
\probbar{0.0000225} &: [\tokenfont[28254]{Hell}, \tokenfont[78]{o}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}worl\underline{d}}] \\
\probbar{0.0000179} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[476]{{\textvisiblespace}wor}, \tokenfont[75]{l\underline{}}] \\
\probbar{0.0000036} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[476]{{\textvisiblespace}wor}, \tokenfont[7278]{l\underline{s}}] \\
\end{tabular}}
\end{minipage}}
\vspace{-1.25\baselineskip} %
\end{wrapfigure}
Our method finds a set of token strings that form a \emph{covering}, a key technical concept we introduce in this paper. Once we have the covering, we can pick a member of the covering in proportion to its (normalized) prefix probability\footnote{In the diagram, we have normalized the prefix probability to make it a distribution over the covering.} and then repeatedly sample subsequent tokens until the end-of-string event $\eos$ is generated. We give a complete algorithm with correctness guarantees in \cref{sec:prompt-boundary-problem}.
Notice that each token string in the covering (shown on the right) includes the character string \charfont{Hello,{\textvisiblespace}worl} as a prefix, but it may have some extra characters (marked by {\color{TokenColor}\underline{underlining}}) in the partially matched last token---just like token healing.
The size of the complete covering may, in general, be exponential in the length of the character string. However, we can enumerate its high-probability members very quickly in practice. Since the worst-case time to find the top-$K$ elements of the covering might be exponential, we provide a more aggressive approximation based on beam search that gives a good approximation even with small beam sizes. Unlike token healing, the sequence $[\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}worl\underline{d}}]$ is the highest probability member of the covering; thus, it is not pruned away by the aggressive heuristics based on $\tokenizer$, as in token healing. The covering is defined only in terms of $\cotokenizer$, and the pruning is based on the language model probability, which typically prioritizes the kinds of token strings that $\tokenizer$ generates, as they are the most reflective of the language model's training data. We provide an algorithm for correctly conditioning a token-level model on a character string in \cref{sec:prompt-boundary-problem}.
\paragraph{Character-level model.}
Reasoning at the character level is intuitive. Consider again our example illustrating the prompt boundary problem. Appending whitespace behaves intuitively at the character level, as it satisfies the \emph{probabilistic chain rule}:\looseness=-1
\begin{align*}
&\prefixChars(\charfont{{\textvisiblespace}one} \mid
\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the}
) \\
&=
\prefixChars(\charfont{{\textvisiblespace}}\mid
\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the})
\cdot \prefixChars(\charfont{one} \mid
\charfont{"In{\textvisiblespace}the{\textvisiblespace}kingdom{\textvisiblespace}of{\textvisiblespace}the{\textvisiblespace}blind,{\textvisiblespace}the{\textvisiblespace}}
)
\end{align*}
Here $\prefixChars$ denotes the character-level model's conditional distribution.
Similarly, recall the \charfont{Hello,{\textvisiblespace}world} example above.
The character-level model correctly infers that \charfont{d} is the most likely next character given \charfont{Hello,{\textvisiblespace}worl}. The computation of this conditional probability is simply the total probability of the covering of \charfont{Hello,{\textvisiblespace}world} divided by the total probability of the covering of \charfont{Hello,{\textvisiblespace}worl}. These quantities are derived from our concept of covering, which directly leads to an algorithm for determining the distribution over possible next characters. Beyond the prompt boundary problem, computing the conditional probability of a character string given a token-level language model has many applications.
\paragraph{Applications.} Aside from making character-level conditioning well-behaved, we highlight a few applications of language models requiring careful reasoning about character strings.
$\hookrightarrow$ \emph{Character-level constraints:}
Enforcing character-level constraints on allowed strings is a promising area that has received much recent attention \citep[e.g.,][]{picard,synchromesh,geng2023grammar,guidance,outlines,koo2024automatabased}.
$\hookrightarrow$ \emph{Computational psycholinguistics:}
Computing the contextual surprisal (negative log probability) of a character substring to predict human reading times \citep{hale2001probabilistic,levy-2008}. Two recent papers \citep{oh2024leading,pimentel2024compute} have given algorithms for computing the surprisal of whitespace-separated \emph{words} under a number of strong assumptions.
Our algorithms can compute the contextual surprisal of arbitrary character strings. \citet{giulianelli-etal-2024-proper} show experimentally that having the flexibility to compute character substring surprisals leads to a more predictive model of reading behavior than a fixed notion of a word.
\paragraph{Does it work?}
In the experimental portion of our paper (\cref{sec:experiments}), we report the empirical runtime of our algorithm for converting token-level language models to character-level ones and quantify its accuracy in estimating the conditional distribution over characters. We find that even with a limited computational budget, our method is able to provide an accurate estimate of the conditional distribution over the next character under two publicly available language models (GPT2-large and Llama 3.1 8B). We defer further discussion to \cref{sec:experiments}.
s
\section{Background}
\label{sec:definitions}
\subsection{Alphabets and Strings}
\label{subsec:alphabets-and-strings}
An \defn{alphabet} $\abstractalphabet$ is a non-empty, finite set of elements called \defn{symbols}.
A \defn{string} $\abstractstring$ over alphabet $\abstractalphabet$ is a finite sequence $\abstractstring = \abstractsymbol_1 \cdots \abstractsymbol_N$ for some $0 \le N < \infty$ of symbols where $\abstractsymbol_1, \ldots, \abstractsymbol_N \in \abstractalphabet$.
Let $\card{\abstractstring}$ denote the string's length $N$.
We denote the empty string as $\varepsilon$.
For any alphabet $\abstractalphabet$, let $\abstractStrings$ denote the set of all strings over $\abstractalphabet$, and let $\kleeneplus{\abstractalphabet}$ denote the set of all non-empty strings over $\abstractalphabet$.
For any two strings $\abstractstring', \abstractstring'' \in \abstractStrings$, we denote their concatenation as $\abstractstring' \c \abstractstring''$.
Additionally, we define $S \c S' \defeq \{ \abstractstring \c \abstractstring' \mid \abstractstring \in S, \abstractstring' \in S' \}$ for any $S, S' \subseteq \abstractStrings$.
Given a string $\abstractstring$ such that $\card{\abstractstring} \geq t$, let $\abstractstring_{<t}$ denote the string of the first $t\!-\!1$ characters of $\abstractstring$.
We write $\abstractstring \preceq \abstractstring'$ if $\abstractstring$ is a prefix\footnote{%
Formally, $(\abstractstring \preceq \abstractstring') \defeq (\exists \abstractstring'' \in \abstractStrings \colon \abstractstring' = \abstractstring \c \abstractstring'')$\label{def:prefix} and $(\abstractstring \prec \abstractstring') \defeq (\exists \abstractstring'' \in \abstractalphabet^{+} \colon \abstractstring' = \abstractstring \c \abstractstring'')$.
} of $\abstractstring'$ and $\abstractstring \prec \abstractstring'$ if $\abstractstring$ is a proper prefix of $\abstractstring'$.
The relation $\preceq$ defines a partial order on $\abstractStrings$.
We write $\succeq$ and $\succ$ to refer to the relations $\preceq$ and $\prec$ with their respective arguments transposed.
\subsection{Language Models and Prefix Probability}
\label{sec:problem-formulation-prefixprobs}
A \defn{language model} $\pabstract$ is a probability distribution over $\abstractalphabet^*$ where $\abstractalphabet$ is an alphabet.
Let $Y$ be a $\abstractStrings$-valued random variable distributed according to $\pabstract$ and $\abstractstring \in \abstractStrings$.
We define the \defn{prefix probability} $\prefixAbstract(\abstractstring)$ as the probability that $Y$ has $\abstractstring$ as a prefix:
\begin{equation}\label{eq:prefix-prob}
\prefixAbstract(\abstractstring) \defeq
\Pr[Y \sim \pabstract]{ Y \succeq \abstractstring}
= \sum_{ \abstractstring' \in \abstractStrings } \indicator{\abstractstring' \succeq \abstractstring} \, \pabstract(\abstractstring')
\end{equation}
We also define the following shorthand for the \defn{conditional prefix probability} $\prefixAbstract(\abstractstring' \mid \abstractstring)$ as the probability of the event $Y \succeq \abstractstring\c\abstractstring'$ provided that $Y \succeq \abstractstring$:
\begin{equation}
\label{eq:ratio}
\prefixAbstract(\abstractstring' \mid \abstractstring)
\defeq
\Pr[Y \sim \pabstract]{ Y \succeq \abstractstring \c \abstractstring' \,\middle|\, Y \succeq \abstractstring }
= \frac{\prefixAbstract(\abstractstring\c \abstractstring')}{\prefixAbstract(\abstractstring)}
= \frac{\sum_{ \abstractstring'' \in \abstractStrings } \indicator{\abstractstring'' \succeq \abstractstring\c \abstractstring'} \, \pabstract(\abstractstring'')}{\sum_{ \abstractstring'' \in \abstractStrings } \indicator{\abstractstring'' \succeq \abstractstring} \, \pabstract(\abstractstring'')}
\end{equation}
\noindent Note that the conditional prefix probability above is only well-defined when $\prefixAbstract(\abstractstring) > 0$.\footnote{Such a caveat is common of conditional probabilities. We note that the condition is always satisfied in practice for softmax-normalized language models, as they place nonzero probability on all elements of $\abstractStrings$; hence, every string in $\abstractStrings$ has a nonzero prefix probability.}
Usefully, we may express the probability of $\abstractstring$ as a product of conditional prefix probabilities:\footnote{
Note that conditional prefix probabilities satisfy the following chain rules:
$\prefixAbstract(\abstractstring\c\abstractstring')
=
\prefixAbstract(\abstractstring) \,
\prefixAbstract(\abstractstring' \mid \abstractstring)$
for all $\abstractstring, \abstractstring' \in \abstractStrings$, and
$\prefixAbstract(\abstractstring'\c\abstractstring'' \mid \abstractstring)
=
\prefixAbstract(\abstractstring' \mid \abstractstring) \,
\prefixAbstract(\abstractstring'' \mid \abstractstring\c\abstractstring')
$ for all $\abstractstring, \abstractstring', \abstractstring'' \in \abstractStrings$.
}
\begin{equation}
\label{eq:conditional-prob-product}
\pabstract(\abstractstring) = \prefixAbstract(\eos \mid \abstractstring) \prod_{t = 1}^{\card{\abstractstring}} \prefixAbstract\mleft(\abstractsymbol_t \mid \abstractstring_{<t}\mright)
\end{equation}
where each $\prefixAbstract\mleft(\abstractsymbol_t \mid \abstractstring_{<t}\mright)$ is an instance of \cref{eq:ratio}, and
\begin{equation}
\prefixAbstract(\eos \mid \abstractstring) \defeq \frac{\pabstract(\abstractstring)}{\prefixAbstract(\abstractstring)}
\label{eq:prefix-eos}
\end{equation}
Here, $\eos$ is a distinguished \defn{end-of-string symbol} that cannot appear in any alphabet. Of particular interest are the single-symbol conditional prefix distributions $\prefixAbstract\mleft(\cdot \mid \abstractstring_{<t}\mright)$, as they may each be interpreted as a probability distribution over the set $\abstractalphabet \cup \{\eos\}$---in fact, modern language models\footnote{E.g., transformers \citep{vaswani2017attention}, RNNs \citep[e.g.,][]{mikolov10_interspeech,sundermeyer2015feedforward}, and $n$-gram models \citep[e.g.,][]{shannon1948mathematical}. These models are often called \emph{autoregressive}, as they directly predict the probability of the next symbol in the string based on its previous symbols using a parametric model.} are \emph{defined} via the product in \Cref{eq:conditional-prob-product} where each single-symbol conditional prefix probability comes from the learned parametric model.\footnote{Note that this can lead to the issue of \emph{tightness} (see, for example, \citet{cotterell2023formal}) where probability is lost to infinite sequences---leading to a probability model summing to less than one over $\abstractStrings$.}
\paragraph{An apparent chicken and egg problem.}
The reader may notice that the equations for
$\pabstract(\abstractstring)$, $\prefixAbstract(\abstractstring' \mid \abstractstring)$, and $\prefixAbstract(\eos \mid \abstractstring)$ may appear cyclical. The key to resolving the concern is to recognize that the apparent cycle just needs a base case. In our presentation, we assumed that $\pabstract$ is the base case. Some readers may view \cref{eq:conditional-prob-product} as the \emph{definition} of the language model $\pabstract$, i.e., they take the components on the right-hand side of \cref{eq:conditional-prob-product} as the base case.
\subsection{Tokenization}
\label{sec:problem-formulation-tokenization}
We now discuss our basic formalization for tokenization.
\begin{definition}
An \defn{(exact) tokenization model} is a tuple $(\charAlphabet, \tokAlphabet, \tokenizer, \cotokenizer)$ where
\begin{itemize}
\item $\charAlphabet$ is an alphabet of \defn{character} symbols
\item $\tokAlphabet$ is an alphabet of \defn{token} symbols
\item $\tokenizer$ is a (possibly) stochastic \defn{encoder}: $\tokenizer(\cdot \mid \charseq)$ is a probability distribution over $\tokStrings$ for each $\charseq \in \charStrings$
\item $\cotokenizer\colon \tokStrings \to \charStrings$ is a \defn{decoder} function mapping token strings to character strings satisfying
\end{itemize}
\begin{equation}\label{eq:exact}
\sum_{\tokseq \in \tokStrings} \indicator{\cotokenizer(\tokseq) = \charseq} \tokenizer(\tokseq \mid \charseq) = 1,\quad\text{ for all }\charseq \in \charStrings
\end{equation}
\end{definition}
The condition between $\tokenizer$ and $\cotokenizer$ expressed in \cref{eq:exact} is called \defn{exactness}. One may consider more general tokenization models in which $\cotokenizer$ is possibly stochastic and for which \cref{eq:exact} may not hold.
All tokenization models considered in this paper will be exact and will be called simply \emph{tokenization models} rather than \emph{exact tokenization models}. Exactness is satisfied by lossless tokenizers, including BPE \citep{sennrich-etal-2016-neural}, UnigramLM \citep{kudo-2018-subword}), and SentencePiece \citep{kudo-richardson-2018-sentencepiece}. Lossy tokenizers, such as those leveraging out-of-vocabulary (OOV) tokens, are not exact.
See \citet{gastaldi2024foundations} for the general case; in particular, for a proof that \cref{eq:exact} implies that an \emph{a priori} stochastic $\cotokenizer$ must, in fact, be deterministic.
\begin{definition}
A \defn{tokenized language model} $\pchars$ is a language model over $\charStrings$ that is \emph{parameterized} by a language model $\ptokens$ over $\tokStrings$ and a decoding function $\cotokenizer\colon \tokStrings \to \charStrings$.
This tokenized language model generates character strings via the following process: (i) $\tokseq \sim \ptokens$, (ii) $\charseq \gets \cotokenizer(\tokseq)$.
Thus, the character strings $\charseq$ generated have the following distribution:
\begin{equation}
\label{eq:character-level-model}
\pchars(\charseq)
\defeq \Pr[\Y \sim \ptokens]{ \cotokenizer(\Y) = \charseq },
\qquad \forall \charseq \in \charStrings
\end{equation}
\end{definition}
Note that $\pchars(\charseq)$ accounts for the fact that many token strings may be associated with a given character string through $\cotokenizer$.\footnote{Many authors \citep{cao-rimell-2021-evaluate,chirkova-etal-2023-marginalize,phan2024understanding} have discussed the particular complications introduced by the fact that for a given character string of length $N$, there are many token strings that decode to it. Marginalizing over the token strings that generate a given character string improves perplexity, but sometimes the improvements are only modest \citep{chirkova-etal-2023-marginalize}. However, prior work has not given algorithms for inferring the distribution over the next character.
} To describe that association, we define $\mathcal{E}(\charseq) \defeq \set{\tokseq \in \tokStrings \colon \charseq = \cotokenizer(\tokseq)}$ denote the set of \defn{encodings} for any character string $\charseq \in \charStrings$.\footnote{Note that $|\mathcal{E}(\charseq)|$ can be very large, e.g., infinite in the worst case. In the case of BPE, it is exponential in $|\charseq|$.}\looseness=-1
\paragraph{What about $\tokenizer$?}
The reader may notice that $\tokenizer$ does not appear in \cref{eq:character-level-model}. Although $\tokenizer$ is essential for generating training data, once the model $\ptokens$ has been trained, the information in $\tokenizer$ is not of immediate practical use.
Moreover, attempts to leverage $\tokenizer$ seem to lead to faulty heuristics, as we discussed in the introduction.
We note that under exactness (\cref{eq:exact}), $\tokenizer(\charseq)$ must be present in $\mathcal{E}(\charseq)$. This is because exactness \emph{indirectly} implies that $\mathcal{E}(\charseq) \supseteq \{ \tokseq \in \tokStrings \colon \tokenizer(\tokseq \mid \charseq) > 0 \}$ for all $\charseq \in \charStrings$. In the common case where $\tokenizer$ is \defn{deterministic} (i.e., $\tokenizer(\tokseq \mid \charseq) \in \set{0,1}$ for all $\tokseq \in \tokStrings, \charseq \in \charStrings$), we emphasize that $\mathcal{E}(\charseq)$ is only \emph{lower bounded} by $\{ \tokenizer(\charseq) \}$. The tokenization model would need to be bijective for $\mathcal{E}(\charseq) = \{ \tokenizer(\charseq) \}$. Unfortunately, common tokenizers (e.g., BPE) are \emph{not} bijective because they do \emph{not} satisfy $\tokenizer(\cotokenizer(\tokseq)) = \tokseq$ for all $\tokseq \in \tokStrings$. We illustrate this in \cref{ex:encodings} below.
\paragraph{The mirage of the canonical tokenization.}
Consider the case when the encoder $\tokenizer$ is deterministic.
In that case, we write $\tokseq = \tokenizer(\charseq)$, and we call this $\tokseq$ the \defn{canonical} tokenization of $\charseq$.
Note that even if $\tokenizer$ is deterministic, there may exist many \defn{noncanonical} tokenizations $\tokseq' \in \mathcal{E}(\charseq)$ such that $\tokseq' \ne \tokenizer(\charseq)$ with nonzero probability $\ptokens(\tokseq') > 0$. Thus, the character string generation process includes a mix of canonical and noncanonical token strings---making it incorrect to only consider a character string's canonical tokenization when assessing its probability. In practice, the conditional probability $\Pr[\Y \sim \ptokens]{\Y = \tokseq \,\middle|\, \cotokenizer(\Y) = \charseq}$ over the encodings $\tokseq$ of a character string $\charseq$ tends to be highly concentrated around the canonical tokenizations, as illustrated in \cref{ex:encodings} below.\looseness=-1
\begin{myexample}
\label{ex:encodings}
\phantom{.}
\begin{center}
\vspace{-.85\baselineskip}
\begin{minipage}[t]{.61\linewidth}
On the right, we show the top-8 encodings $\mathcal{E}(\charfont{Hello,{\textvisiblespace}world})$ under \texttt{GPT2} ranked by their conditional probability.
This short string has 78 tokenizations, which arise because there are many ways to break up \charfont{Hello,{\textvisiblespace}world} into substrings from the tokenization alphabet $\tokAlphabet$:
\begin{align*}
\tokAlphabet \supseteq \{
&\tokenfont[11]{,},
\tokenfont[39]{H},
\tokenfont[78]{o},
\tokenfont[220]{{\textvisiblespace}},
\tokenfont[266]{{\textvisiblespace}w},
\tokenfont[335]{ld},
\tokenfont[476]{{\textvisiblespace}wor},
\tokenfont[695]{ell},
\tokenfont[995]{{\textvisiblespace}world}, \\
& \tokenfont[1544]{He},
\tokenfont[1764]{orld},
\tokenfont[6894]{world},
\tokenfont[11109]{ello},
\tokenfont[28254]{Hell},
\tokenfont[15496]{Hello},
\tokenfont[18798]{llo}
\}
\end{align*}
On the right, we see that the probability assigned to these tokenizations is heavily concentrated on the canonical tokenization: $\tokenizerBPE(\charfont{Hello,{\textvisiblespace}world}) = [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}]$.
\end{minipage}
\begin{minipage}[t]{.38\linewidth}
{\normalfont
\vspace{-1\baselineskip}
\begin{center}
\custombox{{\scriptsize
\hspace{.75cm}
\begin{minipage}{4.1cm}
\begin{center}
\begin{tabular}{r@{\hspace{0pt}}l}
\probbar{0.9999719} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}] \\
\probbar{0.0000229} &: [\tokenfont[28254]{Hell}, \tokenfont[78]{o}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}] \\
\probbar{0.0000024} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[476]{{\textvisiblespace}wor}, \tokenfont[335]{ld\underline{}}] \\
\probbar{0.0000017} &: [\tokenfont[1544]{He}, \tokenfont[18798]{llo}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}] \\
\probbar{0.0000004} &: [\tokenfont[39]{H}, \tokenfont[695]{ell}, \tokenfont[78]{o}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}] \\
\probbar{0.0000002} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[266]{{\textvisiblespace}w}, \tokenfont[1764]{orld\underline{}}] \\
\probbar{0.0000002} &: [\tokenfont[39]{H}, \tokenfont[11109]{ello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}world\underline{}}] \\
\probbar{0.0000001} &: [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[220]{{\textvisiblespace}}, \tokenfont[6894]{world\underline{}}]
\end{tabular}
\end{center}
\end{minipage}
}}
\end{center}}
\end{minipage}
\end{center}
\vspace{-1\baselineskip}
\end{myexample}
\paragraph{A character-level interface.}
A character-level interface to the token-level language model $\ptokens$ is available in the following equations, which hold $\forall \charseq, \charseq' \in \charStrings$:
\begin{align}
\prefixChars(\charseq)
&= \Pr[\Y \sim \ptokens]{\cotokenizer(\Y) \succeq \charseq}
\label{eq:prefix-chars}
\\
\prefixChars(\charseq' \mid \charseq)
&= \frac{\prefixChars(\charseq\c\charseq') }{\prefixChars(\charseq) }
\label{eq:prefix-chars-conditional}
\\
\prefixChars(\eos \mid \charseq)
&= \frac{\pchars(\charseq)}{\prefixChars(\charseq) }
\label{eq:prefix-chars-conditional-eos}
\end{align}
These equations show that we can have a complete character-level language model derived from the tokenized language model if we can compute---or approximate---the necessary summations implied by \cref{eq:character-level-model,eq:prefix-chars}; specifically,
\begin{align}
\pchars(\charseq) = \sum_{\tokseq \in \tokStrings} \indicator{\cotokenizer(\tokseq) = \charseq} \, \ptokens(\tokseq)
\label{eq:p-chars-sum}
\\
\prefixChars(\charseq) = \sum_{\tokseq \in \tokStrings} \indicator{\cotokenizer(\tokseq) \succeq \charseq} \, \ptokens(\tokseq)
\label{eq:prefix-chars-sum}
\end{align}
We will develop effective methods for these summations in the remainder of the paper. We will study a family of strict-prefix monotone decoders $\cotokenizer$ (described in \cref{sec:tokenizer-properties}) where \cref{eq:p-chars-sum} and \cref{eq:prefix-chars-sum} admit a finite summation. In \cref{sec:prompt-boundary-problem}, we give an algorithm for the prompt boundary problem using these concepts.\looseness=-1
\subsection{Useful Properties of \cotokenizer}
\label{sec:tokenizer-properties}
This section provides a collection of definitions and basic results that are useful for characterizing tokenizers. We aim to capture only the essential properties of tokenizers, such as BPE, while committing to as few of its nuances as possible, as it will allow us to future-proof our methods.
\begin{definition}\label{def:strict-monotonicity}
We say that $\cotokenizer\colon \tokStrings \to \charStrings$ is \defn{prefix monotone} if $\tokseq \preceq \tokseq' \implies \cotokenizer(\tokseq) \preceq \cotokenizer(\tokseq')$ and \defn{strict-prefix monotone} if $\tokseq \prec \tokseq' \implies \cotokenizer(\tokseq) \prec \cotokenizer(\tokseq')$.
\end{definition}
In simpler terms, strict-prefix monotonicity says that concatenating a token to the encoding necessarily concatenates at least one character to the decoded character string.
\begin{myexample}
The diagrams below illustrate how {\tt GPT2}'s strict-prefix monotone $\cotokenizer$ gives rise to a certain alignment between three token strings and the character string \charfont{Hello,{\textvisiblespace}world}:
\begin{center}
\vspace{-\baselineskip}
\begin{tabular}{c@{\hskip 1cm}c@{\hskip 1cm}c}
\begin{tikzpicture}
\begin{scope}[node distance=-0.1cm]
\node (source1) {\strut\charfont{H}};
\node[right=of source1] (source2) {\strut\charfont{e}};
\node[right=of source2] (source3) {\strut\charfont{l}};
\node[right=of source3] (source4) {\strut\charfont{l}};
\node[right=of source4] (source5) {\strut\charfont{o}};
\node[right=of source5] (source6) {\strut\charfont{,}};
\node[right=of source6] (source7) {\strut\charfont{\textvisiblespace}};
\node[right=of source7] (source8) {\strut\charfont{w}};
\node[right=of source8] (source9) {\strut\charfont{o}};
\node[right=of source9] (source10) {\strut\charfont{r}};
\node[right=of source10] (source11) {\strut\charfont{l}};
\node[right=of source11] (source12) {\strut\charfont{d}};
\end{scope}
\begin{scope}
\node[below=of source3] (target1) {\tokenfont[15496]{Hello}};
\node[below=of source6] (target2) {\tokenfont[11]{,}};
\node[below=of source10] (target3) {\tokenfont[995]{{\textvisiblespace}world}};
\end{scope}
\draw[black!30] (source1.south) -- (target1.north);
\draw[black!30] (source2.south) -- (target1.north);
\draw[black!30] (source3.south) -- (target1.north);
\draw[black!30] (source4.south) -- (target1.north);
\draw[black!30] (source5.south) -- (target1.north);
\draw[black!30] (source6.south) -- (target2.north);
\draw[black!30] (source7.south) -- (target3.north);
\draw[black!30] (source8.south) -- (target3.north);
\draw[black!30] (source9.south) -- (target3.north);
\draw[black!30] (source10.south) -- (target3.north);
\draw[black!30] (source11.south) -- (target3.north);
\draw[black!30] (source12.south) -- (target3.north);
\end{tikzpicture}
&
\begin{tikzpicture}
\begin{scope}[node distance=-0.1cm]
\node (source1) {\strut\charfont{H}};
\node[right=of source1] (source2) {\strut\charfont{e}};
\node[right=of source2] (source3) {\strut\charfont{l}};
\node[right=of source3] (source4) {\strut\charfont{l}};
\node[right=of source4] (source5) {\strut\charfont{o}};
\node[right=of source5] (source6) {\strut\charfont{,}};
\node[right=of source6] (source7) {\strut\charfont{\textvisiblespace}};
\node[right=of source7] (source8) {\strut\charfont{w}};
\node[right=of source8] (source9) {\strut\charfont{o}};
\node[right=of source9] (source10) {\strut\charfont{r}};
\node[right=of source10] (source11) {\strut\charfont{l}};
\node[right=of source11] (source12) {\strut\charfont{d}};
\end{scope}
\begin{scope}
\node[below=of source1] (target1) {\tokenfont[39]{H}};
\node[below=of source3] (target2) {\tokenfont[695]{ell}};
\node[below=of source5] (target3) {\tokenfont[78]{o}};
\node[below=of source6] (target4) {\tokenfont[11]{,}};
\node[below=of source10] (target5) {\tokenfont[995]{{\textvisiblespace}world}};
\end{scope}
\draw[black!30] (source1.south) -- (target1.north);
\draw[black!30] (source2.south) -- (target2.north);
\draw[black!30] (source3.south) -- (target2.north);
\draw[black!30] (source4.south) -- (target2.north);
\draw[black!30] (source5.south) -- (target3.north);
\draw[black!30] (source6.south) -- (target4.north);
\draw[black!30] (source7.south) -- (target5.north);
\draw[black!30] (source8.south) -- (target5.north);
\draw[black!30] (source9.south) -- (target5.north);
\draw[black!30] (source10.south) -- (target5.north);
\draw[black!30] (source11.south) -- (target5.north);
\draw[black!30] (source12.south) -- (target5.north);
\end{tikzpicture}
&
\begin{tikzpicture}
\begin{scope}[node distance=-0.1cm]
\node (source1) {\strut\charfont{H}};
\node[right=of source1] (source2) {\strut\charfont{e}};
\node[right=of source2] (source3) {\strut\charfont{l}};
\node[right=of source3] (source4) {\strut\charfont{l}};
\node[right=of source4] (source5) {\strut\charfont{o}};
\node[right=of source5] (source6) {\strut\charfont{,}};
\node[right=of source6] (source7) {\strut\charfont{\textvisiblespace}};
\node[right=of source7] (source8) {\strut\charfont{w}};
\node[right=of source8] (source9) {\strut\charfont{o}};
\node[right=of source9] (source10) {\strut\charfont{r}};
\node[right=of source10] (source11) {\strut\charfont{l}};
\node[right=of source11] (source12) {\strut\charfont{d}};
\end{scope}
\begin{scope}
\node[below=of source3] (target1) {\tokenfont[15496]{Hello}};
\node[below=of source6] (target2) {\tokenfont[11]{,}};
\node[below=of source9] (target3) {\tokenfont[476]{{\textvisiblespace}wor}};
\node[below=of source11] (target4) {\tokenfont[75]{l}};
\node[below=of source12] (target5) {\tokenfont[67]{d}};
\end{scope}
\draw[black!30] (source1.south) -- (target1.north);
\draw[black!30] (source2.south) -- (target1.north);
\draw[black!30] (source3.south) -- (target1.north);
\draw[black!30] (source4.south) -- (target1.north);
\draw[black!30] (source5.south) -- (target1.north);
\draw[black!30] (source6.south) -- (target2.north);
\draw[black!30] (source7.south) -- (target3.north);
\draw[black!30] (source8.south) -- (target3.north);
\draw[black!30] (source9.south) -- (target3.north);
\draw[black!30] (source10.south) -- (target3.north);
\draw[black!30] (source11.south) -- (target4.north);
\draw[black!30] (source12.south) -- (target5.north);
\end{tikzpicture}
\end{tabular}
\vspace{-\baselineskip}
\end{center}
\end{myexample}
More formally, every application $\chars_1 \cdots \chars_M = \cotokenizer(\token_1 \cdots \token_M)$ of a strict-prefix monotone mapping has the following properties. Each token in $\token_1 \cdots \token_M$ maps to one or more contiguous characters in $\chars_1 \cdots \chars_M$. Moreover, the mappings do not exclude any characters, and no edges of the mapping cross one another. \emph{Strict} prefix monotonicity, in contrast to prefix monotonicity, ensures that there are no deletions of tokens in the mapping, i.e., each token maps to \emph{at least one} character.\looseness=-1
Strict-prefix monotonicity is the key structural property that we require for the algorithms in \cref{sec:algorithms}, as it allows us to replace an infinite sum with a finite sum in \cref{prop:cover-prefix-prob}. Before moving on to algorithms, we briefly mention multiplicative non-erasing decoders (defined below) as they are a common special case of strict-prefix monotone decoders, as they include BPE \citep{sennrich-etal-2016-neural}, WordPiece \citep{devlin-etal-2019-bert}, and SentencePiece \citep{kudo-richardson-2018-sentencepiece}. However, we will not leverage multiplicativity beyond the remainder of this subsection.
\begin{itemize}
\item We say that a decoder $\cotokenizer$ is \defn{multiplicative} if
$\cotokenizer(\tokseq\c \tokseq') = \cotokenizer(\tokseq) \c \cotokenizer(\tokseq')$ for all $\tokseq, \tokseq' \in \tokStrings$, and
\defn{non-erasing} if $\cotokenizer(\token) = \varepsilon \Longrightarrow \token = \varepsilon$.
\item If $\cotokenizer$ is multiplicative, then $\cotokenizer$ is prefix monotone.
\item If $\cotokenizer$ is multiplicative and non-erasing, then $\cotokenizer$ is strict-prefix monotone.
\item If $\cotokenizer$ is multiplicative, then $\cotokenizer(\token_1 \cdots \token_N) = \cotokenizer(\token_1) \cdots \cotokenizer(\token_N)$ for all $\token_1 \cdots \token_N \in \tokStrings$.
\end{itemize}
\section{Algorithms
}
\label{sec:algorithms}
This section gives algorithms for $\pchars(\charseq)$, $\prefixChars(\charseq)$, $\prefixChars(\charseq' \mid \charseq)$, $\prefixChars(\eos \mid \charseq)$, and conditional token generation (i.e., our solution to the prompt boundary problem).
Unlike prior papers \citep{cao-rimell-2021-evaluate,chirkova-etal-2023-marginalize,phan2024understanding} on the topic of marginalizing tokenization strings to compute, or estimate, the probability of character strings, i.e., compute $\pchars(\charseq)$, we will provide algorithms for computing conditional probabilities $\pchars(\charseq' \mid \charseq)$ for any strings $\charseq, \charseq' \in \charStrings$. Our approach is based on estimating the prefix probability $\prefixChars(\charseq)$, and it can be used in conjunction with \Cref{eq:ratio} to compute $\prefixChars(\charseq' \mid \charseq)$. Throughout this section, we assume that $\cotokenizer$ is strict-prefix monotone.\looseness-1
\subsection{Covering}
Our primary tool for the prefix probability $\prefixChars(\charseq)$ is the covering of $\charseq$. It is a subset of tokens associated with the given string $\charseq$ that is sufficient to evaluate its prefix probability. As we will see experimentally (\cref{sec:experiments}), this set's subset of high-probability token strings is often small enough to enumerate.\looseness=-1
\Cref{eq:prefix-chars-sum} shows that we can, in principle, compute the prefix probability $\prefixChars(\charseq)$ by summing over \defn{prefix-encodings} of $\charseq$, $\prefixEncodings(\charseq) \defeq \set{ \tokseq \in \tokStrings\colon \cotokenizer(\tokseq) \succeq \charseq }$. Unfortunately, $\prefixEncodings(\charseq)$ is infinitely large. Fortunately, we can exploit the prefix monotone structure of $\cotokenizer$ to find a different way to perform the summation by summing over a \emph{finite} set.
Let $\tokseq \in \tokStrings$, and $\charseq \in \charStrings$.
We say that $\tokseq$ \defn{covers} $\charseq$ if and only if $\cotokenizer(\tokseq) \succeq \charseq$.
Monotonicity ensures that for all $\tokseq \in \prefixEncodings(\charseq)$, we have that $\forall \tokseq' \in \tokStrings\colon \cotokenizer(\tokseq\c\tokseq') \succeq \charseq$. In other words, any $\tokseq$ that decodes to an extension of $\charseq$ (i.e., $\cotokenizer(\tokseq) \succeq \charseq$) will continue to do so if we append tokens to it.
Thus, we may additionally qualify the relationship as $\tokseq$ \defn{minimally covers} $\charseq$ if and only if $\cotokenizer(\token_1 \cdots \token_{M-1}) \prec \charseq$. With that in mind, we define $\minCover{\charseq}(\tokseq)$ as the \emph{shortest} prefix $\tokseq' \preceq \tokseq$ such that $\cotokenizer(\tokseq') \succeq \charseq$, i.e., it maps any $\tokseq$ that covers $\charseq$ to a (possibly equal) token string that minimally covers $\charseq$. Next, we define the set of \emph{minimal} prefix encodings of $\cover(\charseq)$, which we call the \defn{covering} of $\charseq$, $\cover(\charseq) \defeq \{ \minCover{\charseq}(\tokseq) \mid \tokseq \in \prefixEncodings(\charseq) \}$.
A more convenient expression for the covering $\cover(\charseq)$ of a string $\charseq \in \charStrings$ is equal to the following subset of $\tokStrings$:\looseness=-1
\begin{equation}\label{eq:covering-monotone}
\cover(\charseq)
=
\begin{cases}
\{ \varepsilon\} &\textbf{if } \charseq = \varepsilon\\
\{ \token_1 \cdots \token_M \in \kleeneplus{\tokAlphabet} \colon
\cotokenizer(\token_1 \cdots \token_{M-1}) \prec \charseq \preceq \cotokenizer(\token_1 \cdots \token_M) \} &\textbf{otherwise}
\end{cases}
\end{equation}
\begin{myexample}
\phantom{.}
\begin{center}
\vspace{-.75\baselineskip}
\begin{minipage}[t]{.74\linewidth}
Recall the example of a covering of $\charseq = \charfont{Hello,{\textvisiblespace}worl}$ from the introduction. We have repeated it on the right and couched it in our terminology. Note that the complete covering for this string contains 36,608 token strings; we only show the top-$8$ according to their respective $\prefixTokens$.
\vspace{.25\baselineskip}
\begin{itemize}
\item The covering for any given string $\charseq$ always has the property that each non-empty token string $\tokseq$ in the covering decodes to some string that is an extension of the character string $\charseq \preceq \cotokenizer(\tokseq)$. This is illustrated by the {\tt\small\color{TokenColor}gloss} string in the tokenization.
\item However, it may include a {\tt\small\color{TokenColor}par\underline{tially}} matched token at its end (i.e., $\token_M$ in \cref{eq:covering-monotone}). We have marked the extra characters by {\tt\small\color{TokenColor}\underline{underlining}} them. We note that the 7\textsuperscript{th} member does not have a partially matched last token.
\item Each token string in the covering has at most one partially matched token thanks to the condition $\cotokenizer(\tokseq) \prec \charseq \preceq \cotokenizer(\tokseq \c \token)$. The 7\textsuperscript{th} member of the cover has a completely matched last token; hence, there is no underlining.
\item We see that if we were to extend any member $\tokseq \in \cover(\charseq)$ with an arbitrary string of additional tokens $\tokseq'$, it would continue to decode to a string such that $\cotokenizer(\tokseq\c\tokseq') \succeq \charseq$. Moreover, $\tokseq$ is minimal (i.e., $\minCover{\charseq}(\tokseq) = \tokseq$).\looseness=-1
\end{itemize}
\end{minipage}
\begin{minipage}[t]{.25\linewidth}
\vspace{-\baselineskip}
{\scriptsize \quad$\cover(\charfont{Hello,{\textvisiblespace}worl})=$}
$$
\custombox{\begin{minipage}{2.5cm}\vspace{-10pt} %
{\scriptsize
\begin{align*}
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}worl\underline{d}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[11621]{{\textvisiblespace}worl\underline{ds}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[8688]{{\textvisiblespace}worl\underline{dwide}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[43249]{{\textvisiblespace}worl\underline{dly}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[29081]{{\textvisiblespace}worl\underline{dview}}] \\
& [\tokenfont[28254]{Hell}, \tokenfont[78]{o}, \tokenfont[11]{,}, \tokenfont[995]{{\textvisiblespace}worl\underline{d}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[476]{{\textvisiblespace}wor}, \tokenfont[75]{l\underline{}}] \\
& [\tokenfont[15496]{Hello}, \tokenfont[11]{,}, \tokenfont[476]{{\textvisiblespace}wor}, \tokenfont[7278]{l\underline{s}}]
\end{align*}}
\end{minipage}}
$$
\vspace{-2\baselineskip} %
\end{minipage}
\end{center}
\end{myexample}
The notion of a covering is used to derive an algorithm for computing character-level probabilities given a token-level language model. We first show how it gives us the prefix probability and subsequently give equations for the remaining quantities of the character=level language model.
\clearpage
\begin{restatable}[]{proposition}{monotone}
\label{prop:cover-prefix-prob}
Suppose $(\charAlphabet, \tokAlphabet, \tokenizer, \cotokenizer)$ is a tokenization model where $\cotokenizer$ is strict-prefix monotone
and $\ptokens$ is a token-level language model.
Then, the prefix probability $\prefixChars(\charseq)$ for the character-level model \cref{eq:character-level-model} is given by the equation below.
\begin{align}
\prefixChars(\charseq)
&= \smashoperator{\sum_{\tokseq \in \cover(\charseq)}}
\prefixTokens(\tokseq),
\qquad \forall \charseq \in \charStrings
\label{eq:prefix-chars-monotone}
\end{align}
\end{restatable}
\vspace{-\baselineskip}
\begin{proof}
We prove the proposition directly through the following manipulations.
\begin{align}
\prefixChars(\charseq) %
&= \sum_{\tokseq' \in \tokStrings} \indicator{\charseq \preceq \cotokenizer(\tokseq')} \ptokens(\tokseq')
\\
&= \indicator{\charseq = \varepsilon} \ptokens(\varepsilon) + \smashoperator{\sum_{\tokseq' \in \kleeneplus{\tokAlphabet}}} \indicator{\charseq \preceq \cotokenizer(\tokseq')} \, \ptokens(\tokseq')
\\
&= \indicator{\charseq = \varepsilon} \ptokens(\varepsilon)
+ \smashoperator{\sum_{\tokseq \c \token' \c \tokseq'' \in \kleeneplus{\tokAlphabet}}} \indicator{\cotokenizer(\tokseq) \prec \charseq \preceq \cotokenizer(\tokseq \c \token' \c \cancel{\tokseq''})}\, \ptokens(\tokseq \c \token' \c \tokseq'')%
\\
&= \indicator{\charseq = \varepsilon} \ptokens(\varepsilon)
+ \smashoperator{\sum_{\tokseq \c \token' \in \kleeneplus{\tokAlphabet}}} \indicator{\cotokenizer(\tokseq) \prec \charseq \preceq \cotokenizer(\tokseq \c \token')}
\smashoperator{\sum_{\tokseq'' \in \tokStrings}}
\ptokens(\tokseq \c \token' \c \tokseq'')
\\
&= \indicator{\charseq = \varepsilon} \ptokens(\varepsilon)
+ \smashoperator{\sum_{\tokseq \c \token' \in \kleeneplus{\tokAlphabet}}} \indicator{\cotokenizer(\tokseq) \prec \charseq \preceq \cotokenizer(\tokseq \c \token')}
\,\prefixTokens(\tokseq \c \token')
\\
&= \smashoperator{\sum_{\tokseq\in \cover(\charseq)}} \prefixTokens(\tokseq)
\end{align}
About the steps above:
We start with the summation expression for the character-level prefix probability (i.e., \cref{eq:prefix-chars-sum}).
We expand the summation into two cases (so that it will eventually match the two cases in the expression for the covering \cref{eq:covering-monotone}).
Next, for each summand, we consider its \emph{unique} minimal prefix $\tokseq\c \token'$ covering $\charseq$.
We see why $\varepsilon$ is handled separately, as it cannot be covered by a token sequence of that form.
We exploit the key property of prefix monotone tokenizers (i.e., that once $\tokseq \c \token'$ covers $\charseq$, each extension $\tokseq \c \token' \tokseq''$ continues to cover it). This allows us to rearrange the summation to sum over the extension $\tokseq''$, which is conveniently equal to the prefix probability of $\tokseq\c\token'$. The final step is to recognize that the summands can all be indexed by the covering $\cover(\charseq)$.
\end{proof}
\Cref{eq:prefix-chars-monotone} is a substantial improvement over \Cref{eq:prefix-chars} for computing $\prefixChars(\charseq)$. Specifically, we now have a \emph{finite} sum, as $\card{\cover(\charseq)}$ is finite for all $\charseq \in \charStrings$. Bear in mind that the covering's size is likely too large to be practical, as there may still be a large number of summands; however, the set of high-prefix-probability elements of the covering tends to be reasonably small, an observation that we verify in \cref{sec:experiments}, and leverage to develop practical algorithms in \cref{sec:algorithms}.
Given a covering, we filter it to the case of string equality, as \begin{align}
\mathcal{E}(\charseq) = \set{\tokseq \in \cover(\charseq) \colon \cotokenizer(\tokseq) = \charseq},
\end{align}
to give an expression of the string's probability
\begin{align}
\pchars(\charseq)
= \smashoperator{\sum_{\tokseq \in \cover(\charseq)}}
\indicator{\cotokenizer(\tokseq) = \charseq} \, \ptokens(\tokseq)
= \smashoperator{\sum_{\tokseq \in \mathcal{E}(\charseq)}}
\ptokens(\tokseq)
\end{align}
\subsection{Algorithms for \protect\ensuremath{\prefixChars(\charseq)} and \protect\ensuremath{\pchars(\charseq)}}
The enumeration algorithm will enumerate elements of the covering along with their prefix probability (for convenience). It filters prefixes of token strings that cannot eventually cover the target string $\charseq$. The strict-prefix monotonicity property is essential for this filtering.
Our algorithm \mintinline{python}{enumerate_cover} performs recursive enumeration of the members of the covering $\cover(\charseq)$ along with some metadata. Specifically, the algorithm returns a collection of triples where each triple $(p', \charseq', \tokseq')$ satisfies $\tokseq' \in \cover(\charseq)$, $p' = \prefixTokens(\tokseq')$, and $\charseq' = \cotokenizer(\tokseq')$.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def enumerate_cover(@$\chars_1 \cdots \chars_N$@):
if @$N = 0$@: return [@$(1, \varepsilon, \varepsilon)$@] # base case
result = []
for @$(p^{\prime}, \charseqp, \tokseqp)$@ in enumerate_cover(@$\chars_1 \cdots \chars_{N-1}$@): # recurse
if @$|\charseqp| < N$@: # extend: @$\tokseqp$@ needs to be longer to cover @$\chars_1 \cdots \chars_N$@
for @$\token^{\prime\prime} \in \tokAlphabet$@:
@$\charseq^{\prime\prime} \gets \cotokenizer(\tokseq^{\prime} \c \token^{\prime\prime})$@
if @$\charseq^{\prime\prime}_N = \chars_N$@: # filter: @$N\textsuperscript{th}$@ character matches
result.append(@$(p^{\prime} \cdot \prefixTokens(\token^{\prime\prime} \mid \tokseq^{\prime}), \charseq^{\prime\prime}, \tokseq^{\prime} \c \token^{\prime\prime})$@)
elif @$\chars^\prime_N = \chars_N$@: # filter: @$N\textsuperscript{th}$@ character matches
result.append(@$(p^{\prime}, \charseqp, \tokseqp)$@)
return prune(@$\chars_1 \cdots \chars_N$@, result)
\end{minted}
\end{minipage}
\end{center}
Note that this method has an additional parameter, the function \mintinline{python}{prune}, which is used on the last line. This method, as the name suggests, is used to limit the size of the covering to prevent excessive growth. We will discuss this parameter shortly. For now, consider the following definition:
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def prune_nothing(@$\chars_1 \cdots \chars_N$@, result):
return result
\end{minted}
\end{minipage}
\end{center}
Below, we show how to use the output of the enumeration algorithm to compute several key quantities in the character-level interface.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def @$\cover(\charseq)$@:
return {@$\tokseqp$@ for @$(\_, \_, \tokseqp)$@ in enumerate_cover(@$\charseq$@)}
\end{minted}
\end{minipage}
\end{center}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def @$\prefixChars(\charseq)$@:
return sum(@$p^\prime$@ for @$(p^\prime, \_, \_)$@ in enumerate_cover(@$\charseq$@))
\end{minted}
\end{minipage}
\end{center}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def @$\mathcal{E}(\charseq)$@:
return {@$\tokseqp$@ for @$(\_, \charseqp, \tokseqp)$@ in enumerate_cover(@$\charseq$@) if @$\charseqp = \charseq$@}
\end{minted}
\end{minipage}
\end{center}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def @$\pchars(\charseq)$@:
return sum(@$p^\prime \cdot \ptokens(\eos \mid \tokseqp)$@ for @$(p^\prime, \charseqp, \tokseqp)$@ in enumerate_cover(@$\charseq$@) if @$\charseqp = \charseq$@)
\end{minted}
\end{minipage}
\end{center}
This algorithm is specified at a fairly high level; thus, in order to meaningfully discuss its running time, we require some assumptions:
\begin{itemize}
\item $\cotokenizer$: Our analysis assumes that $\cotokenizer(\tokseq' \c \token'')$ can be evaluated in constant time given $\cotokenizer(\tokseq')$.
\item $\prefixTokens$: Our analysis assumes that the cost of evaluating $\prefixTokens(\token_t \mid \tokseq_{<t})$ is constant given that $\prefixTokens(\token_s \mid \tokseq_{<s})$ has been computed for $0 \le s < t$.\footnote{In the case of the common transformer language model \citep{vaswani2017attention}, this can be achieved with efficient caching and limiting context windows to a constant size.}
\end{itemize}
Under these assumptions, the running time of $\mbox{\mintinline{python}{enumerate_cover}}(\chars_1 \cdots \chars_N)$ can be exponential in $N$ when no pruning is used. We provide detailed bounds on the covering's size in \cref{sec:cover-size}. It is straightforward to verify that the space complexity is $\mathcal{O}(|\cover(\charseq)|)$, and the running time is $\mathcal{O}(|\tokAlphabet| \cdot \sum_{t=1}^{|\charseq|} |\cover(\charseq_{<t})|)$ which is dominated by the $|\tokAlphabet| \cdot |\cover(\charseq)|$ term; thus, $\mathcal{O}(|\tokAlphabet| \cdot |\cover(\charseq)|)$.
\paragraph{Pruning.}
We now consider some useful pruning heuristics for the algorithm, which make it an \emph{approximation} but substantially improve its running time. We propose a heuristic based on beam search. This heuristic is very effective: it gives us a linear running time as a function of the character strings's length. It has a parameter $K$ that controls the approximation quality. Larger $K$ makes the approximation more accurate, and the approximation becomes exact as $K$ approaches the size of the (largest intermediate) covering.
For simplicity, we take $K$ to be a global variable in the pseudocode.
$\hookrightarrow$ \emph{Our pruning heuristic:}
Our pruning heuristic enumerates $\le K$ distinct token strings modulo their last token. This choice allows up to $|\tokAlphabet|$ versions of the last token to be enumerated. Thus, the work done at each step is $\mathcal{O}(K \cdot |\tokAlphabet|)$, and the size of the \mintinline{python}{result} list is at most that size. Therefore, the overall running time is $\mathcal{O}(N \cdot K \cdot |\tokAlphabet|)$ for a character string of length $N$.\footnote{%
We note that finding the (unordered) collection of top-$K$ elements from a set of size $n$ is possible in $\mathcal{O}(n)$ time via the median-of-medians algorithm \citep{mom-topk}. This is faster asymptotically than the commonly used $\mathcal{O}(n \log K)$ heap-based strategy. In practice, $K$ is small enough that this is not an important detail. However, we sought to clarify why there is no $\log$ factor in our running time-bound. We also note that the cost of hashing a sequence of length $N$ can be amortized to constant time in this pseudocode using the hashconsing pattern \citep[see,][for an overview]{goubault1994implementing}.}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def prune_top_K_buckets(@$\chars_1 \cdots \chars_N$@, results):
# Put items into buckets based on completely matched prefix
buckets = {}
for item in results:
@$(p, \charseqp, \token_1 \cdots \token_M)$@ = item
# Exclude a @${\tt\color{TokenColor}part\underline{ially}}$@ matched last token from the key, if one exists
key = @$\token_1 \cdots \token_{M-1}$@ if @$|\charseqp| > N$@ else @$\token_1 \cdots \token_M$@
buckets[key].append(item) # don't count last token
pruned = []
for bucket in (top @$K$@ buckets according to their total probability):
for item in bucket:
pruned.append(item)
return pruned
\end{minted}
\end{minipage}
\end{center}
\paragraph{Bundled beam summing implementation.}
\Cref{sec:trie-alg} describes an implementation strategy that improves the constant factors associated with the pseudocode above.
The key idea is to group the token sequences that fall into the same bucket in \mintinline{python}{prune_top_K_buckets} into a \emph{bundle} that represents them compactly. In particular, we can use a trie to efficiently filter out the next tokens that disagree with the next character. This improves the per-iteration cost of that filter as it can via the data structure, as it does the organizational work ahead of time in bulk. We can regard the trie as a local language model that generates the next token character-by-character according to the probability assigned by $\prefixTokens(\cdot \mid \tokseq)$. Each bundle can be unbundled (if necessary) into the respective tuples that the \mintinline{python}{enumerate_cover} algorithm maintains.
\subsection{Algorithms for
\protect\ensuremath{\prefixChars(\charseq' \mid \charseq)} and \protect\ensuremath{\prefixChars(\eos \mid \charseq)}}
This section gives algorithms for computing the character-level conditional prefix probability. Recall the definition of the character-level conditional prefix probability, that is, \cref{eq:prefix-chars-conditional,eq:prefix-chars-conditional-eos}, can be computed from a certain ratio of calls to $\prefixChars$ (and $\pchars$ in the case of $\eos$).
Below is a direct translation of this into an algorithm that computes the distribution over $\charAlphabet \cup \set{\eos}$ given $\charseq \in \charStrings$:
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def next_character_probability(@$\charseq$@): # slower version
@$Z$@ = @$\prefixChars(\charseq)$@; @$\overline{p} = \{\}$@
for @$\charsp \in \charAlphabet$@:
@$\overline{p}(\charsp)$@ = @$\prefixChars(\charseq \c \charsp)$@
@$\overline{p}(\eos)$@ = @$\pchars(\charseq)$@
return @$\overline{p} / Z$@
\end{minted}
\end{minipage}
\end{center}
The algorithm below is semantically equivalent to the one above. However, it is more efficient because it reuses computations between iterations of the loop over $\charsp \in \charAlphabet$. The duplicated work was caused by making separate calls to $\prefixChars$ and $\pchars$ for many related arguments; they are now fused in the method below.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def next_character_probability(@$\charseq$@): # faster version
@$N \gets |\charseq|$@; @$Z$@ = @$0$@; @$\overline{p}$@ = {@$\charsp$@: @$0$@ for @$\charsp \in \charAlphabet \cup \set{\eos}$@}
for @$(p^{\prime}, \charseqp, \tokseqp) \in \mbox{\texttt{{\color{blue}enumerate\_cover}}}(\charseq)$@:
@$Z$@ += @$p^{\prime}$@
if @$|\charseqp| = N$@: # i.e., @$\charseqp = \charseq$@
@$\overline{p}(\eos)$@ += @$p^{\prime} \cdot \prefixTokens(\eos \mid \tokseq)$@
for @$\token^{\prime\prime} \in \tokAlphabet$@: # extend
@$\charseq^{\prime\prime} \gets \cotokenizer(\tokseqp \c \token^{\prime\prime})$@
@$\overline{p}(\charseq^{\prime\prime}_{N+1})$@ += @$p^{\prime} \cdot \prefixTokens(\token^{\prime\prime} \mid \tokseqp)$@ # sum prefix prob. of @$(N\!+\!1)\textsuperscript{th}$@ character
else: # i.e., @$\charseqp \succeq \charseq$@
@$\overline{p}(\charseqp_{N+1})$@ += @$p^{\prime}$@ # sum prefix prob. of @$(N\!+\!1)\textsuperscript{th}$@ character
return @$\overline{p} / Z$@ # @$Z$@ = @$\prefixChars(\charseq)$@
\end{minted}
\end{minipage}
\end{center}
\subsection{Conditional Generation \protect\ensuremath{\charCond(\tokseq \mid \charseq)}}
\label{sec:prompt-boundary-problem}
This section gives a simple algorithm for correctly generating a token string $\Y$ that has a given character-level prompt $\charseq$ as its prefix. This algorithm is equivalent to the algorithm in the introduction but significantly faster.
The algorithm works by enumerating the covering $\cover(\charseq)$, drawing a token string from it in proportion to its prefix probability, and finishing the token string by sampling a completion, which can be done from the token-level model.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def conditional_token_generation(@$\charseq$@):
@$Z$@ = @$\prefixChars(\charseq)$@
# Sample from the covering
@$\tokseqp \sim \mathrm{Categorical}$@({@$\tokseqp\colon p^{\prime}/Z$@ for @$(p^{\prime}, \_, \tokseq^{\prime})$@ in enumerate_cover(@$\charseq$@)})
return sample_completion(@$\tokseqp$@)
\end{minted}
\end{minipage}
\end{center}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def sample_completion(@$\tokseqp$@):
@$\tokseq^{\prime\prime} \gets \varepsilon$@
while True:
@$\token \sim \prefixTokens(\cdot \mid \tokseqp \c \tokseq^{\prime\prime})$@
if @$\token = \eos$@: break
@$\tokseq^{\prime\prime} \gets \tokseq^{\prime\prime} \c \token$@
return @$\tokseqp\c\tokseq^{\prime\prime}$@
\end{minted}
\end{minipage}
\end{center}
The following proposition shows that the samples generated by the algorithm correctly condition on any character string prefix:
\begin{proposition}
\label{prop:token-healing}
The algorithm $\mbox{\mintinline{python}{conditional_token_generation}}(\charseq)$ generates $\Y \sim \charCond(\cdot \mid \charseq)$ for all $\charseq \in \charStrings$.
\end{proposition}
\vspace{-\baselineskip}
\begin{proof}[Proof (Sketch)]
Choose an arbitrary $\tokseq \in \tokStrings$.
\begin{align}
\charCond(\tokseq \mid \charseq)
&=
\Pr[\Y \sim \ptokens]{ \Y = \tokseq \,\middle|\, \cotokenizer(\Y) \succeq \charseq }
\\
&= \ptokens(\tokseq) \frac{\indicator{\cotokenizer(\tokseq) \succeq \charseq}}{\displaystyle\Pr[\Y \sim \ptokens]{\cotokenizer(\Y) \succeq \charseq}}
\\
&= \ptokens(\tokseq)\frac{\indicator{\cotokenizer(\tokseq) \succeq \charseq}}{\prefixChars(\charseq)}
\\
\intertext{Let $\tokseq' = \minCover{\charseq}(\tokseq)$ (i.e., the shortest prefix of $\tokseq$ such that $\cotokenizer(\tokseq') \succeq \charseq$). Choose $\tokseq''$ such that $\tokseq'\c\tokseq'' = \tokseq$.}
&= \underbrace{\prefixTokens(\eos \mid \tokseq' \c \tokseq'') \prefixTokens(\tokseq'' \mid \tokseq')}_{\text{sample completion}} \underbrace{ \prefixTokens(\tokseq') \frac{\indicator{\cotokenizer(\tokseq') \succeq \charseq}}{\prefixChars(\charseq)}}_{\text{sample from covering}}\label{eq:two-steps-to-healing}
\end{align}
We can see that the algorithm samples from this distribution because it samples a token string $\tokseq'$ from the covering in proportion to the right factor in \cref{eq:two-steps-to-healing} and then samples a completion $\tokseq''$ of $\tokseq'$ in proportion to the left factor of the equation. Thus, the sample $\tokseq = \tokseq' \c \tokseq''$ has probability $\charCond(\tokseq \mid \charseq)$, and $\mbox{\mintinline{python}{conditional_token_generation}}(\charseq)$ is a correct sampling procedure for it.
\end{proof}
We also note the following corollary, as it gives an interpretation for the categorical distribution in the efficient \mintinline{python}{conditional_token_generation} algorithm.
\begin{corollary}
\label{prop:sample-cover}
For all $\charseq \in \charStrings$, $\tokseq \in \tokStrings$,
\begin{align}
\Pr[\Y \sim \ptokens]{ \minCover{\charseq}(\Y) = \tokseq \,\middle|\, \cotokenizer(\Y) \succeq \charseq }
=
\frac{\prefixTokens(\tokseq)}{\prefixChars(\charseq)} \indicator{\tokseq \in \cover(\charseq)}
\end{align}
\end{corollary}
Thus, we have provided an efficient solution to the prompt boundary problem. We also note that generating from $\pchars$ a character at a time is also a correct solution to the prompt boundary problem; however, it is slower because it does not benefit from the fact that the generated string is shorter in token space. This is because once the minimally covering token string has been sampled, the method \mintinline{python}{sample_completion} will generate a complete sequence more efficiently than the character-at-a-time sample algorithm, as it does not have the overhead of marginalization that $\prefixChars(\cdot \mid \cdot)$ does.
\section{Experiments}
\label{sec:experiments}
In this section, we investigate our proposed algorithm's running time and accuracy.
\paragraph{Setup.}
\begin{itemize}
\item We use {Llama 3.1 8B} \citep{dubey2024llama} and {GPT2 large} \citep{radford2019language}
from the \huggingface\kern1pt {\tt transformers} library \citep{wolf-etal-2020-transformers}. Both models were trained over token strings created from byte-pair encoding (BPE; \citet{sennrich-etal-2016-neural,gage1994bpe}).
\item We use the \texttt{wikitext-103-v1} corpus as a source of character strings; we used the version in the \huggingface\kern1pt {\tt datasets} library. Specifically, we use the \emph{test} portion.
\item We use {\scriptsize\emoji[twitter]{vllm}} \citep{vllm} to perform the efficient, batched evaluation of transformer language models on GPUs.
We batch the evaluation of all sequence extensions. All experiments were run an A100 GPU with 80GB of memory.
\item Our implementation uses a trie to represent all items in each bucket efficiently (see \cref{sec:trie-alg}). We use the bucket-based pruning heuristic described in \cref{sec:algorithms}.
\end{itemize}
In the discussion below, let $N$ denote $|\charseq|$ and let $K$ denote the beam-size parameter.
\paragraph{Error vs.\@ speed.}
To better understand the quality of the approximation that our method provides, we perform the following experiment. We use a large beam $K=128$ as a reference model, and we measure the average per-character Jensen-Shannon distance (JSD) to the reference model's conditional distribution over the next character. The table below shows the average JSD between the character-level conditional distributions with beam sizes $K \in \set{4,8,16,32,64}$ and the reference model.\footnote{We note that if the beam size is too small, it is possible for the beam summing procedure to hit a \emph{dead end} where no extension of the token sequences on the beam exist that match the next character of the input string. Indeed, for the very small beam size of $K=2$, GPT2-large hits such a dead end after approximately 1700 characters; Llama 3.1 8B, on the other hand, manages to successfully complete the whole corpus with $K=2$.} The average is computed across the first 4000 characters of the \texttt{wikitext-103-v1} test corpus. To quantify the speed of our method, we show the number of characters per second for each value of $K$, including the reference model.\footnote{Since the running time is linear, we can gain a good understanding of the total running time to process a string of any length $N$ based on the measurement of the average characters per second.} In parentheses, we provide a 95\% confidence interval computed using bootstrapping. To aid in interpretation, we also show the same information graphically below.\looseness=-1
\begin{center}
\begin{minipage}[c]{\linewidth}
\resizebox{\textwidth}{!}{
\begin{tabular}{c|cc|cc}
\toprule
& \multicolumn{2}{c}{\textbf{GPT2 large}} & \multicolumn{2}{c}{\textbf{Llama 3.1 8B}} \\
\midrule
$K$ & average JSD / char & char / sec & average JSD / char & char / sec\\
\midrule
4 & 0.00330 (0.00235, 0.00435) & 47.51 (46.86, 48.17) & 0.00021 (0.00008, 0.00041) & 46.31 (45.82, 46.82) \\
8 & 0.00075 (0.00053, 0.00102) & 28.45 (27.99, 28.94) & 0.00008 (0.00006, 0.00010) & 29.45 (29.03, 29.87) \\
16 & 0.00040 (0.00020, 0.00064) & 15.23 (14.94, 15.54) & 0.00004 (0.00003, 0.00005) & 15.37 (15.13, 15.62) \\
32 & 0.00009 (0.00005, 0.00015) & 7.90 (7.74, 8.06) & 0.00005 (0.00004, 0.00006) & 7.55 (7.42, 7.69) \\
64 & 0.00002 (0.00001, 0.00003) & 3.88 (3.80, 3.96) & 0.00003 (0.00003, 0.00003) & 3.77 (3.70, 3.84) \\
128 & \textcolor{black!40}{(not applicable)} & 1.81 (1.77, 1.85) & \textcolor{black!40}{(not applicable)} & 1.90 (1.87, 1.94) \\
\bottomrule
\end{tabular}
}
\end{minipage}
\end{center}
\begin{minipage}[c]{\linewidth}
\includegraphics[width=\linewidth]{img/gpt2-llama-4000-tradeoff-jsd.png}
\end{minipage}
\paragraph{Discussion.}
As expected, we observe that the character per second decreases with $K$. We observe an inverse relationship between error (JSD) and speed (chars/sec): as the processing speed (characters/sec) decreases, the JSD also decreases. Notably, this tradeoff is non-linear, with JSD increasing more sharply at higher processing speeds compared to lower speeds. This trend is particularly evident for Llama 3.1 8B, where the JSD stabilizes around $K \geq 8$. This indicates diminishing returns in reducing error as $K$ gets large. We hypothesize that this occurs because the language model's probability mass is concentrated around a limited set of tokenizations, which are adequately covered even with smaller beam sizes. A plausible explanation is that Llama 3.1 8B's larger model and training set size has made it better able to learn to assign much lower probabilities to non-canonical tokenizations; thus, mass tends to be concentrated around fewer tokenizations, making it possible to capture the distribution with a smaller value of $K$.
Comparing the two models, we observe that GPT2-large generally exhibits larger JSD from the reference distribution compared to Llama 3.1 8B across all $K$, with the exception of $K=64$. This difference can be explained by GPT2-large spreading its probability mass across more tokenizations, requiring larger $K$ to cover all probable tokenizations. %
\section{Conclusions, Limitations, and Future Work}
We have developed an effective method for ameliorating tensions between tokens and characters faced by engineers and users. We gave theory and algorithms that provide a character-level interface to tokenized language models. We characterized and resolved the prompt boundary problem. We investigated the empirical speed and error rates of our method on a modern language model.
The primary limitation of our beam summing method is that it will require a very large beam size $K$ when the language model does not favor a small number of tokenizations. The models that we explored in our experiments concentrate mass on a few tokenizations; thus, we did not require large $K$ to estimate their character-level prefix probabilities accurately. Future work may wish to investigate sampling-based estimation methods and possibly derive upper and lower bounds on the true values of the character-level prefix probability.
\section*{Acknowledgments}
The authors would like to thank Andreas Opedal, Alex Lew, Jacob Hoover Vigly, Luca Malagutti, Manuel de Prada Corral, and V{\'e}steinn Sn{\ae}bjarnarson for their helpful feedback and discussions.
JT would like to thank Rycolab for its hospitality during a recent visit. The authors JLG and JT would like to thank Institut des Hautes \'Etudes Scientifiques (IHES) for their hospitality while revising this paper.
MG was supported by an ETH Z\"{u}rich Postdoctoral Fellowship. This research was enabled in part by compute resources provided by Mila (mila.quebec).
\bibliographystyle{iclr2024_conference}
\bibliography{main}
\appendix
\clearpage
\section{The Size of the Covering}
\label{sec:cover-size}
We now set about to bound the worst-case size of the covering function.
To do so, we introduce additional definitions that characterize the different growth factors.
We define the $\cotokenizer$'s \defn{fertility} as
\begin{equation}
F \defeq \max_{\tokseq \in \tokStrings} \max_{\charseq \in \charStrings} \card{ \{ \token' \in \tokAlphabet\colon \charseq = \cotokenizer(\tokseq \c \token') \} } \leq \card{\tokAlphabet}
\label{def:fertility}
\end{equation}
\begin{myexample}
The BPE tokenizer has $F_{\mathrm{BPE}}=1$ because it is multiplicative, and its tokens represent \emph{distinct} substrings. More formally,
\begin{align}
F_{\mathrm{BPE}}
&= \max_{\tokseq \in \tokStrings} \max_{\charseq \in \charStrings} | \{ \token' \in \tokAlphabet\colon \charseq = \cotokenizer(\tokseq \c \token') \} |
& \proofExplain{def of fertility}
\\
&= \max_{\tokseq \in \tokStrings} \max_{\charseq \in \charStrings} | \{ \token' \in \tokAlphabet\colon \charseq = \cotokenizer(\tokseq) \c \cotokenizer(\token') \} |
& \proofExplain{def multiplicativity}
\\
&= | \{ \token' \in \tokAlphabet\colon \charseq' = \cotokenizer(\token') \}
& \proofExplain{def function}
\\
&= 1
& \proofExplain{distinctness}
\end{align}
\end{myexample}
Additionally, we define a $\cotokenizer$'s \defn{munch} as follows.
\begin{equation}
M \defeq \max_{\tokseq \in \tokStrings}
\max_{\token \in \tokAlphabet} |\cotokenizer(\tokseq \c \token)| - |\cotokenizer(\tokseq)|
\label{def:munch}
\end{equation}
In words, the munch measures the length of the largest number of characters that can be introduced by adding one more token to any given context.
\begin{myexample}
The munch of a multiplicative $\cotokenizer$, such as BPE, is $\max_{\token \in \tokAlphabet} |\cotokenizer(\token)|$. Put in words, it is the length of the longest detokenization. The munch for \texttt{GPT-2} is surprisingly long (128), as they are common in, for example, markdown syntax.
\end{myexample}
\begin{restatable}[]{proposition}{recurrence}\label{prop:recurrence}
Let $F$ and $M$ be the fertility and munch of $\cotokenizer$. Then,
for all $\charseq \in \charStrings$,
\begin{align}
|\cover(\charseq)| \le C(\card{\charseq})
\end{align}
where
\begin{align}
C(n) &\defeq \begin{cases}
0 & \text{if } n < 0 \\
1 & \text{if } n = 0 \\
F \smashoperator{\sum_{j = n-M}^{n-1}} C(j) & \text{otherwise)}
\end{cases}
\end{align}
\end{restatable}
\vspace{-\baselineskip}
\begin{proof}
The base cases $N \le 0$ are straightforward. Consider the case of a string of length $N \ge 0$. \emph{Inductive hypothesis}: Suppose for all strings $\charseq'$ with $|\charseq'| < N$, $|\cover(\charseq')| \le C(|\charseq'|)$.
Let $\charseq$ be an arbitrary string with length $N > 0$.
\begin{align}
& \!\!\!\!\!\card{\cover(\chars_1 \cdots \chars_N)}
\\
&= \card{\set{ \tokseq \c \token \in \tokAlphabet^+ \colon \cotokenizer(\tokseq) \prec \chars_1 \cdots \chars_N \preceq \cotokenizer(\tokseq \c \token) }}
\\
&= \card{\bigcup_{j=0}^N
\underbrace{
\set{ \tokseq \c \token \in \tokAlphabet^+ \colon \cotokenizer(\tokseq) = \chars_1 \cdots \chars_{j}, \chars_1 \cdots \chars_N \preceq \cotokenizer(\tokseq\c\token) }
}_{= \emptyset \text{ if }N-(j+1) > M \text{ or } N=j+1}
}
\\
&\le
\sum_{j=N-M}^{N-1}
\card{
\underbrace{\set{ \tokseq \in \tokStrings \colon \cotokenizer(\tokseq) = \chars_1 \cdots \chars_{j} }}_{\subseteq \cover(\chars_1 \cdots \chars_{j})}
}
\cdot
\underbrace{
\card{\set{ \token \in \tokAlphabet \colon \tokseq \in \tokStrings, \chars_{1} \cdots \chars_N \preceq \cotokenizer(\tokseq\c\token) }
}}_{\le F}
\\
&\le F \cdot \smashoperator{\sum_{j=N-M}^N} \quad
\underbrace{\card{\cover(\chars_1 \cdots \chars_{j}) }}_{\text{inductive hypothesis}}
\\
&\le F \cdot \smashoperator{\sum_{j=N-M}^N} C(j)
\\
&= C(N)
\end{align}
Thus, the proposition holds true by the principle of induction.
\end{proof}
\begin{restatable}[]{corollary}{cover-bound}\label{cor:cover-bound}
Let $N = \card{\charseq}$. Consider the following cases:
\begin{itemize}
\item When $M=N$ and $F = 1$, $C(N) = 2^N$.
\item When $M=N$ and $F \ge 0$, $C(N) = F(1+F)^N$.
\item Otherwise, $C(N) = F^N \mathrm{Fib}(N,M)$ where $\mathrm{Fib}(N,M)$ is $N$\textsuperscript{th} $M$\textsuperscript{th}-order Fibonacci number.\footnote{$M$\textsuperscript{th}-order Fibonacci numbers are a variation of the well-known Fibonacci (i.e., $M=2$) that sums the previous $M$ numbers in the sequence instead of the previous two.}
\end{itemize}
In all cases, $C^F_M(N) < \infty$.
\end{restatable}
In the proposition below, we show that the covering can easily be exponential in size:
\begin{proposition}
\begin{equation}
|\cover(\charseq)| \in \Omega(2^{|\charseq|})
\end{equation}
\end{proposition}
\vspace{-\baselineskip}
\begin{proof}
We prove the proposition by constructing an example that achieves the lower bound.
\begin{itemize}
\item Let $\charAlphabet = \set{\charfont{a}}$, $\tokAlphabet = \set{\tokenfont[1]{a}, \tokenfont[2]{aa}}$.
\item Let $\cotokenizer$ be multiplicative, and define $\cotokenizer(\tokenfont[1]{a}) \defeq \charfont{a}, \cotokenizer(\tokenfont[2]{aa}) \defeq \charfont{aa}$.
\item Let $\charseq$ be an arbitrary string from $\charStrings$. Let $N = |\charseq|$.
\end{itemize}
Then, $|\mathcal{E}(\charseq)|$ equals the number of nonnegative integer solutions $(n,m)$ to $1 \, m + 2 \, n = N$. This is because we can build the $\charfont{a}^N$ using a sequence of $\tokenfont[1]{a}$ or $\tokenfont[2]{aa}$, but each $\tokenfont[1]{a}$ accounts for 1 \charfont{a} and each $\tokenfont[2]{aa}$ accounts for 2. So if $n$ is the number of token 1 and $m$ is the number of token 2, we must have that $1 \, m + 2 \, n = N$. The number of solutions grows like $\Omega(2^N)$.
Lastly, because $\cover(\charseq) \supseteq \mathcal{E}(\charseq)$, we have that $|\cover(\charseq)| \in \Omega(2^{|\charseq|})$. Thus, the proposition holds.
\end{proof}
\clearpage
\section{Bundled Beam Summing Implementation}
\label{sec:trie-alg}
\newcommand{\trie}[0]{\mathit{trie}}
Upon implementing this scheme, we observed that it is possible to efficiently reason about all the next tokens that extend a given token sequence in the cover in bulk. The key idea is to group the token sequences that fall into the same bucket in \mintinline{python}{prune_top_K_buckets} into a \mintinline{python}{Bundle} (see below) that represents them compactly. In particular, we can use a probability trie to efficiently filter out the next tokens that disagree with the next character. This improves the per-iteration cost of that filter as it can via the data structure, as it does the organizational work ahead of time in bulk. We can regard the probability trie as a local language model that generates the next token character-by-character according to the probability assigned to it by $\prefixTokens(\cdot \mid \tokseq)$. Each bundle can be unbundled (see method \mintinline{python}{unbundle}) into the respective tuples that the \mintinline{python}{enumerate_cover} algorithm maintains. The algorithms are otherwise equivalent.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def beam(@$\chars_1 \cdots \chars_N$@):
if @$N = 0$@: return [Bundle(@$1$@, @$\varepsilon$@, @$\varepsilon$@, build_trie(@$\varepsilon$@))]
candidates = []
for bundle in beam(@$\chars_1 \cdots \chars_{N-1}$@):
filtered_bundle = bundle.filter(@$\chars_N$@)
if filtered_bundle is not None:
candidates.append(filtered_bundle)
for extended_bundle in bundle.extend():
candidates.append(extended_bundle.filter(@$\chars_N$@))
# Keep top-@$K$@ bundles according to their prefix probability
return top@$_K$@(candidates, key=lambda bundle: -bundle.@$p$@)
\end{minted}
\end{minipage}
\end{center}
Each bundle is an instance of the following class with four member variables: the prefix probability $p$, token string $\tokseq$, character string $\charseq$, and a reference to a local probability trie $\trie$. The trie provides the character-level probabilities of the distribution over possible next tokens: $\ptokens(\cdot \mid \tokseq)$. The trie is also augmented with a special symbol $\eot$ to denote the end of this next token.\footnote{This is completely analogous to how $\eos$ marks the end of a string.}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
class Bundle(@$p$@, @$\tokseq$@, @$\charseq$@, @$\trie$@):
def filter(@$\charsp$@):
if @$\trie.p(\charsp \mid \charseq) = 0$@: return # no tokens give prefix @$\charseq \c \charsp$@ prob
return Bundle(@$p \cdot trie(\charsp \mid \charseq)$@, @$\tokseq$@, @$\charseq\c\charsp$@, @$\trie$@)
def extend():
@$Z$@ = @$\trie.p(\textsc{eot} \mid \charseq)$@
if @$Z > 0$@: # emit tokens that decode to @$\charseq$@
for @$(\tokenp, p^{\prime})$@ in @$\trie.\mathit{tokens}[\charseq]$@:
yield Bundle(@$\frac{p}{Z} \cdot p^{\prime}$@, @$\tokseq\c\tokenp$@, @$\varepsilon$@, build_trie(@$\tokseq\c\tokenp$@))
\end{minted}
\end{minipage}
\end{center}
The code below builds the probability trie from the possible next tokens. It figures out the character strings associated with those next tokens and puts them into the trie with the respective probabilities.\footnote{Note that $\eos$ is handled as an indivisible symbol in the probability trie, whereas other strings are predicted one character at a time in the trie.}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def build_trie(@$\tokseq$@):
@$\charseq$@ = @$\cotokenizer(\tokseq)$@ # remember common prefix
@$\trie$@ = ProbabilityTrie() # uses @{\normalfont\small\textsc{EOT}}@ to mark the end of a token
for @$\tokenp \in \tokAlphabet$@:
@$\charseq \c \charseqp = \cotokenizer(\tokseq\c\tokenp)$@ # use new characters, i.e., ignore prefix @$\charseq$@
@$\trie.\mathit{add}(\charseqp, \tokenp, \ptokens(\tokenp \mid \tokseq))$@ # add @$\charseqp$@ and @$\tokenp$@ to @$\trie$@ with this probability
return @$\trie$@
\end{minted}
\end{minipage}
\end{center}
The probability trie has the following functionality:
\begin{itemize}
\item $\trie.\mathit{tokens}[\charseqp]$ stores the set of tokens that decode to $\charseqp$ along with their respective.
\item $\trie.p(\charsp \mid \charseq)$ returns the probability of the character $\charsp$ given $\charseq$, it is equal to
\begin{equation}
\trie.p(\charsp \mid \charseq)
\propto
\sum_{\tokenp \in \tokAlphabet} \ptokens(\tokenp \mid \tokseq) \,
\begin{cases}
\indicator{\cotokenizer(\tokseq \c \tokenp) = \charseq}&
\textbf{if } \charsp = \eot
\\
\indicator{\cotokenizer(\tokseq \c \tokenp) \succeq \charseq \c \charsp}& \textbf{otherwise}
\end{cases}
\end{equation}
\end{itemize}
We note that the trie implicitly depends on the token string $\tokseq$ used in its creation.
To aid in understanding (and testing) how the bundled algorithm relates to the original algorithm, we give the following methods.
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
class Bundle(@$p$@, @$\tokseq$@, @$\charseq$@, @$\trie$@):
...
def unbundle():
agenda = [@$\charseq$@]
while agenda:
@$\charseqp$@ = agenda.pop()
@$Z$@ = @$\trie.p(\eot \mid \charseq)$@
if @$Z$@ > @$0$@:
for @$(\tokenp, p^\prime) \in \trie.\mathit{tokens}[\charseqp]$@:
yield (@$p \cdot p^\prime$@, @$\cotokenizer(\tokseq \c \tokenp)$@, @$\tokseq \c \tokenp$@)
for @$\chars^{\prime\prime}$@ in @$\trie.p(\cdot \mid \charseqp)$@:
if @$\chars^{\prime\prime} \ne \eot$@:
agenda.append(@$\charseqp\c\chars^{\prime\prime}$@)
\end{minted}
\end{minipage}
\end{center}
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def unbundle_beam(beam):
return [item for bundle in beam for item in bundle.unbundle()]
\end{minted}
\end{minipage}
\end{center}
We note that \mintinline{python}{unbundle_beam(beam(@$\charseq$@))} gives precisely the same set of elements as the unbundled algorithm (i.e., \mintinline{python}{enumerate_cover(@$\charseq$@)}) run on the same string $\charseq$ and the bucket-based pruning scheme with parameter $K$ (up to reordering).
To compute the next-character probability, we use the following algorithm:
\begin{center}
\begin{minipage}{.9\linewidth}
\begin{minted}{python}
def next_character_probability(@$\charseq$@):
@$\overline{p}$@ = {@$\charsp$@: @$0$@ for @$\charsp \in \charAlphabet \cup \set{\eos}$@}
for bundle in beam(@$\charseq$@):
for @$\charsp \in \charAlphabet$@:
@$\overline{p}(\charsp)$@ += bundle.@$p$@ @$\cdot$@ @$\texttt{bundle}.\trie.p(\charsp \mid \texttt{bundle}.\charseq)$@
for ext_bundle in bundle.extend():
for @$\charsp \in \charAlphabet$@:
@$\overline{p}(\charsp)$@ += ext_bundle.@$p$@ @$\cdot$@ @$\texttt{ext\_bundle}.\trie.p(\charsp \mid \texttt{ext\_bundle}.\charseq)$@
@$Z$@ = sum(@$\overline{p}$@.values())
return {@$\charsp$@: @$\overline{p}(\charsp) / Z$@ for @$\charsp \in \charAlphabet$@}
\end{minted}
\end{minipage}
\end{center}
\end{document}
\newcommand{\mymacro}[1]{#1}
\newcommand{\huggingface}[0]{\emoji[twitter]{huggingface-LaTeX}}
\usepackage{mathtools}
\usepackage{ellipsis}
\makeatletter
\renewcommand*{\mathellipsis}{%
\mathinner{%
\kern\ellipsisbeforegap%
{\ldotp}\kern\ellipsisgap%
{\ldotp}\kern\ellipsisgap%
{\ldotp}\kern\ellipsisaftergap%
}%
}
\renewcommand*{\dotsb@}{%
\mathinner{%
\kern\ellipsisbeforegap%
{\cdotp}\kern\ellipsisgap%
{\cdotp}\kern\ellipsisgap%
{\cdotp}\kern\ellipsisaftergap%
}%
}
\renewcommand*{\@cdots}{%
\mathinner{%
\kern\ellipsisbeforegap%
{\cdotp}\kern\ellipsisgap%
{\cdotp}\kern\ellipsisgap%
{\cdotp}\kern\ellipsisaftergap%
}%
}
\renewcommand*{\ellipsis@default}{%
\ellipsis@before
\kern\ellipsisbeforegap
.\kern\ellipsisgap
.\kern\ellipsisgap
.\kern\ellipsisgap
\ellipsis@after\relax}
\renewcommand*{\ellipsis@centered}{%
\ellipsis@before
\kern\ellipsisbeforegap
.\kern\ellipsisgap
.\kern\ellipsisgap
.\kern\ellipsisaftergap
\ellipsis@after\relax}
\AtBeginDocument{%
\DeclareRobustCommand*{\dots}{%
\ifmmode\@xp\mdots@\else\@xp\textellipsis\fi}}
\def\ellipsisgap{-.05em}
\def\ellipsisbeforegap{-.05em}
\def\ellipsisaftergap{.05em}
\makeatother
\newcommand{\mathhl}[2][yellow!80]{\mathchoice%
{\colorbox{#1}{$\displaystyle#2$}}%
{\colorbox{#1}{$\textstyle#2$}}%
{\colorbox{#1}{$\scriptstyle#2$}}%
{\colorbox{#1}{$\scriptscriptstyle#2$}}}%
\newcommand{\paroutline}[1]{\reversemarginpar\marginpar{\vspace{\baselineskip}{\scriptsize\color{ETHPurpleDark}\begin{raggedright}#1\end{raggedright}}}\normalmarginpar}
\usepackage[normalem]{ulem} %
\definecolor{ETHBlue}{RGB}{33,92,175} %
\definecolor{ETHGreen}{RGB}{98,115,19} %
\definecolor{ETHPurpleDark}{RGB}{140,10,89} %
\definecolor{ETHPurple}{RGB}{163,7,116} %
\definecolor{ETHGray}{RGB}{111,111,111} %
\definecolor{ETHRed}{RGB}{183,53,45} %
\definecolor{ETHPetrol}{RGB}{0,120,148} %
\definecolor{ETHBronze}{RGB}{142,103,19} %
\definecolor{ETHOrange}{RGB}{230, 100, 50}
\colorlet{MacroColor}{ETHPetrol}
\colorlet{MACROCOLOR}{MacroColor}
\definecolor{WildStrawberry}{rgb}{1.0, 0.26, 0.64}
\definecolor{Wisteria}{rgb}{0.79, 0.63, 0.86}
\definecolor{Olive}{rgb}{0.5, 0.5, 0.0}
\definecolor{Plum}{rgb}{0.56, 0.27, 0.52}
\definecolor{NavyBlue}{rgb}{0.0, 0.0, 0.5}
\definecolor{OliveGreen}{rgb}{0.33, 0.42, 0.18}
\definecolor{RoyalPurple}{rgb}{0.47, 0.32, 0.66}
\renewcommand{\qedsymbol}{$\blacksquare$}
\usepackage[capitalize,noabbrev]{cleveref}
\crefname{section}{\S}{\S\S}
\crefname{table}{Tab.}{Tab.}
\crefname{figure}{Fig.}{Figs.}
\crefname{algorithm}{Alg.}{}
\crefname{equation}{Eq.}{Eq.}
\crefname{appendix}{App.}{}
\crefname{theorem}{Theorem}{}
\crefname{prop}{Proposition}{}
\crefname{definition}{Def.}{}
\crefname{cor}{Corollary}{}
\crefname{observation}{Observation}{}
\crefname{assumption}{Assumption}{}
\crefname{hyp}{Hyp.}{Hypotheses}
\crefformat{section}{\S#2#1#3}
\crefname{namedtheorem}{Hyp.}{Hypotheses}
\newtheorem{lemma}{Lemma}
\newtheorem{remark}{Remark}
\newtheorem{theorem}{Theorem}
\newtheorem{construction}{Construction}
\newtheorem{proposition}{Proposition}
\newtheorem{conjecture}{Conjecture}
\newtheorem{definition}{Definition}
\newtheorem{myexample}{Example}
\newtheorem{mytransform}{Transform}
\newtheorem{corollary}{Corollary}
\newcommand{\marked}[0]{{}^{\#}}
\newcommand{\justification}[1]{%
\refstepcounter{equation}%
\tag{\theequation \textcolor{black!50}{, \footnotesize{#1}}}
}
\newcommand{\prooftext}[1]{\text{\color{black!50}#1}}
\newcommand{\proofExplain}[1]{\text{\scriptsize\color{black!50}[#1]}}
\newcommand{\proofExplainLong}[2][1.5in]{%
{\scriptsize\color{black!50}%
\left[\begin{minipage}[c]{#1}%
#2%
\end{minipage}\right]}%
}
\newcommand{\defn}[1]{\textbf{#1}}
\newcommand{\defeq}{\mathrel{\overset{\raisebox{-0.25ex}{\textnormal{\tiny def}}}{=}}}
\newcommand{\card}[1]{\left|#1\right|}
\newcommand{\simIID}[0]{\mathrel{\overset{{\text{\scriptsize i.i.d.}}}{\sim}}}
\usepackage{stmaryrd} %
\newcommand{\sem}[1]{\left\llbracket #1 \right\rrbracket}
\newcommand{\set}[1]{\left\{ #1 \right\}}
\newcommand{\indicator}[1]{\mathbbm{1}\mleft\{#1\mright\}}
\newcommand{\idfun}{\mymacro{\text{Id}}}
\newcommand{\bigo}{{\mymacro{ \mathcal{O}}}}
\newcommand{\bigO}[1]{{\mymacro{ \bigo\mleft(#1\mright)}}}
\newcommand{\bigOmega}[1]{{\mymacro{ \Omega\mleft(#1\mright)}}}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\exactagg}{{\mymacro{\textsc{Exact}}}}
\newcommand{\sumagg}{{\mymacro{\textsc{Sum}}}}
\newcommand{\meanagg}{{\mymacro{\textsc{Mean}}}}
\newcommand{\firstagg}{{\mymacro{\textsc{First}}}}
\newcommand{\surprisal}{{\mymacro{h}}}
\newcommand{\entropy}{\mymacro{\mathrm{H}}}
\newcommand{\setX}{{\mymacro{\mathcal{X}}}}
\newcommand{\setY}{{\mymacro{\mathcal{Y}}}}
\newcommand{\setZ}{{\mymacro{\mathcal{Z}}}}
\newcommand{\elemx}{{\mymacro{x}}}
\newcommand{\elemy}{{\mymacro{y}}}
\newcommand{\elemz}{{\mymacro{z}}}
\newcommand{\mapf}{{\mymacro{f}}}
\newcommand{\mapg}{{\mymacro{g}}}
\newcommand{\prefixEncodings}[0]{{\color{TokenColor}\mathcal{P}}}
\newcommand{\encodings}[0]{{\color{TokenColor}\mathcal{E}}}
\newcommand{\cover}{{\color{TokenColor}\mathcal{C}}}
\newcommand{\leakage}{\mymacro{\lambda}}
\newcommand{\minCover}[1]{{\color{TokenColor}\phi}_{#1}}
\newcommand{\abstractalphabet}{{\mymacro{ \Gamma}}}
\newcommand{\abstractStrings}{{\mymacro{ \Gamma^*}}}
\newcommand{\abstractstring}{{\mymacro{ \boldsymbol{\gamma}}}}
\newcommand{\abstractsymbol}{{\mymacro{\gamma}}}
\newcommand{\abstracteosalphabet}{{\mymacro{\overline{\Gamma}}}}
\newcommand{\wordalphabet}{{\mymacro{\Omega}}}
\newcommand{\eoswordalphabet}{{\mymacro{\overline{\wordalphabet}}}}
\newcommand{\prewords}{{\mymacro{\widetilde{\Omega}}}}
\newcommand{\word}{{\mymacro{w}}}
\newcommand{\wordseq}{{\mymacro{\boldsymbol{w}}}}
\newcommand{\tokAlphabetwhite}{\tokAlphabet_1}
\newcommand{\tokAlphabetnowhite}{\tokAlphabet_2}
\newcommand{\kleene}[1]{#1^*}
\newcommand{\kleeneplus}[1]{#1^+}
\newcommand{\eos}{{\mymacro{\textsc{eos}}}}
\newcommand{\eot}{{\mymacro{\textsc{eot}}}}
\newcommand{\normconstant}{\mymacro{Z}}
\DeclareTextCommandDefault{\textvisiblespace}{%
\mbox{\kern.06em\vrule \@height.3ex}%
\vbox{\hrule \@width.3em}%
\hbox{\vrule \@height.3ex}}
\newcommand{\pcontext}{\mymacro{c_{\charAlphabet}}}
\renewcommand{\c}[0]{{\cdot}}
\definecolor{CharacterColor}{RGB}{98,115,19}
\definecolor{TokenColor}{RGB}{140,10,89}
\newcommand{\probbar}[1]{\makebox[1cm][r]{\rule{#1\linewidth/6}{5pt} {\scriptsize\color{black!50} ({#1})}} }
\newcommand{\mylogprob}[1]{%
{\color{red}%
\FPeval{\result}{exp(#1)}%
\pgfmathprintnumber[sci, precision=1]{\result}%
}%
}
\newcommand{\custombox}[1]{%
\begin{tikzpicture}[baseline={(current bounding box.center)}]
\node[anchor=north west] (box) at (0,0) {#1};
\draw[thin, black!25] (box.south west) -- (box.north west) -- (box.north east) -- (box.south east);
\draw[thin, black!25]
(box.south west) .. controls +(0.3cm,0.15cm) and +(-0.3cm,-0.15cm) .. ++(0.6cm,0)
.. controls +(0.3cm,0.15cm) and +(-0.3cm,-0.15cm) .. ++(0.6cm,0)
.. controls +(0.3cm,0.15cm) and +(-0.3cm,-0.15cm) .. ++(0.6cm,0)
.. controls +(0.3cm,0.15cm) and +(-0.3cm,-0.15cm) .. (box.south east); %
\end{tikzpicture}%
}
\newcommand{\tokenfont}[2][]{%
\mbox{{\scriptsize\ensuremath{%
\stackrel{\raisebox{-2ex}{{\color{TokenColor}\texttt{\strut#2}}}}%
{\raisebox{-1ex}{{\color{black!50}\texttt{\strut#1}}}}%
}}}%
}
\newcommand{\charfont}[1]{{\color{CharacterColor}\texttt{\small#1}}}
\newcommand{\X}[0]{{\color{CharacterColor}X}}
\newcommand{\Y}[0]{{\color{TokenColor}Y}}
\newcommand{\tokmodel}{\mymacro{\mathcal{T}}}
\newcommand{\tokenizer}{\ensuremath{{\color{TokenColor}\tau}}\xspace}
\newcommand{\cotokenizer}{\ensuremath{{\color{CharacterColor}\kappa}}\xspace}
\newcommand{\tokenizerBPE}[0]{\ensuremath{\tokenizer_{{\color{TokenColor}\text{BPE}}}}\xspace}
\newcommand{\cotokenizerBPE}[0]{\ensuremath{\cotokenizer_{{\color{CharacterColor}\text{BPE}}}}\xspace}
\newcommand{\D}[0]{{\color{TokenColor}D}}
\newcommand{\notD}[0]{{\color{TokenColor}\overline{D}}}
\newcommand{\preD}[0]{{\color{TokenColor}\prefixLanguageOp{D}}}
\newcommand{\mytok}[0]{\delta}
\newcommand{\token}{{\color{TokenColor}\mytok}}
\newcommand{\tokseq}{\ensuremath{{\color{TokenColor}\boldsymbol{\mytok}}}\xspace}
\newcommand{\tokAlphabet}{{\color{TokenColor}\Delta}}
\newcommand{\tokStrings}[0]{{\color{TokenColor}\kleene{\tokAlphabet}}}
\newcommand{\charCond}[0]{p_{\tokAlphabet\mid\charAlphabet}}
\newcommand{\mychar}[0]{\sigma}
\newcommand{\chars}{{\color{CharacterColor}\mychar}}
\newcommand{\charseq}{{\color{CharacterColor}\boldsymbol{\mychar}}}
\newcommand{\charAlphabet}{{\color{CharacterColor}\Sigma}}
\newcommand{\charStrings}[0]{{\color{CharacterColor}\kleene{\charAlphabet}}}
\newcommand{\charsp}[0]{\chars'}
\newcommand{\charseqp}[0]{\charseq'}
\newcommand{\tokenp}[0]{\token'}
\newcommand{\tokseqp}[0]{\tokseq'}
\newcommand{\ptokens}{\mymacro{p_{\tokAlphabet}}}
\newcommand{\pchars}{\mymacro{p_{\charAlphabet}}}
\newcommand{\pabstract}{\mymacro{p_{\abstractalphabet}}}
\newcommand{\prefixLanguageOp}[1]{\overrightarrow{#1}}
\newcommand{\prefixClosure}[1]{\prefixLanguageOp{#1}}
\newcommand{\prefixAbstract}{\mymacro{\prefixLanguageOp{\pabstract}}}
\newcommand{\prefixTokens}{{\mymacro{\prefixLanguageOp{\ptokens}}}}
\newcommand{\prefixChars}{{\mymacro{\prefixLanguageOp{\pchars}}}}
\renewcommand{\Pr}[2][]{\mathop{\mathbb{P}}_{\substack{#1}}\mleft[#2\mright]}
\begin{thebibliography}{32}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Blum et~al.(1973)Blum, Floyd, Pratt, Rivest, and Tarjan]{mom-topk}
Manuel Blum, Robert~W. Floyd, Vaughan~R. Pratt, Ronald~L. Rivest, and
Robert~Endre Tarjan.
\newblock Time bounds for selection.
\newblock \emph{Journal of Computer and System Sciences}, 7\penalty0 (4), 1973.
\newblock \doi{10.1016/S0022-0000(73)80033-9}.
\newblock URL \url{https://doi.org/10.1016/S0022-0000(73)80033-9}.
\bibitem[Cao \& Rimell(2021)Cao and Rimell]{cao-rimell-2021-evaluate}
Kris Cao and Laura Rimell.
\newblock You should evaluate your language model on marginal likelihood over
tokenisations.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural Language Processing}, 2021.
\newblock \doi{10.18653/v1/2021.emnlp-main.161}.
\newblock URL \url{https://aclanthology.org/2021.emnlp-main.161}.
\bibitem[Chirkova et~al.(2023)Chirkova, Kruszewski, Rozen, and
Dymetman]{chirkova-etal-2023-marginalize}
Nadezhda Chirkova, Germ{\'a}n Kruszewski, Jos Rozen, and Marc Dymetman.
\newblock Should you marginalize over possible tokenizations?
\newblock In \emph{Proceedings of the Annual Meeting of the Association for
Computational Linguistics}, 2023.
\newblock \doi{10.18653/v1/2023.acl-short.1}.
\newblock URL \url{https://aclanthology.org/2023.acl-short.1}.
\bibitem[Cotterell et~al.(2024)Cotterell, Svete, Meister, Liu, and
Du]{cotterell2023formal}
Ryan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, and Li~Du.
\newblock Formal aspects of language modeling, 2024.
\newblock URL \url{https://arxiv.org/abs/2311.04329}.
\bibitem[Devlin et~al.(2019)Devlin, Chang, Lee, and
Toutanova]{devlin-etal-2019-bert}
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock {BERT}: Pre-training of deep bidirectional transformers for language
understanding.
\newblock In \emph{Proceedings of the Conference of the North {A}merican
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, 2019.
\newblock \doi{10.18653/v1/N19-1423}.
\newblock URL \url{https://aclanthology.org/N19-1423}.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman,
Mathur, Schelten, Yang, Fan, et~al.]{dubey2024llama}
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,
et~al.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint arXiv:2407.21783}, 2024.
\bibitem[Gage(1994)]{gage1994bpe}
Philip Gage.
\newblock A new algorithm for data compression.
\newblock \emph{C Users Journal}, 12\penalty0 (2), 1994.
\newblock ISSN 0898-9788.
\newblock URL
\url{https://web.archive.org/web/20230319172720/https://www.derczynski.com/papers/archive/BPE_Gage.pdf}.
\bibitem[Gastaldi et~al.(2024)Gastaldi, Terilla, Malagutti, DuSell, Vieira, and
Cotterell]{gastaldi2024foundations}
Juan~Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, and
Ryan Cotterell.
\newblock The foundations of tokenization: Statistical and computational
concerns, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.11606}.
\bibitem[Geng et~al.(2023)Geng, Josifoski, Peyrard, and West]{geng2023grammar}
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West.
\newblock Grammar-constrained decoding for structured {NLP} tasks without
finetuning.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural Language Processing}, 2023.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.674.pdf}.
\bibitem[Giulianelli et~al.(2024)Giulianelli, Malagutti, Gastaldi, DuSell,
Vieira, and Cotterell]{giulianelli-etal-2024-proper}
Mario Giulianelli, Luca Malagutti, Juan~Luis Gastaldi, Brian DuSell, Tim
Vieira, and Ryan Cotterell.
\newblock On the proper treatment of tokenization in psycholinguistics.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural Language Processing}, 2024.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.1032}.
\bibitem[Goubault(1994)]{goubault1994implementing}
Jean Goubault.
\newblock Implementing functional languages with fast equality, sets and maps:
An exercise in hash consing.
\newblock \emph{Journ{\'e}es Francophones des Langages Applicatifs}, 1994.
\bibitem[Hale(2001)]{hale2001probabilistic}
John Hale.
\newblock A probabilistic {E}arley parser as a psycholinguistic model.
\newblock In \emph{Meeting of the North American Chapter of the Association for
Computational Linguistics}, 2001.
\newblock URL \url{https://aclanthology.org/N01-1021.pdf}.
\bibitem[Koo et~al.(2024)Koo, Liu, and He]{koo2024automatabased}
Terry Koo, Frederick Liu, and Luheng He.
\newblock Automata-based constraints for language model decoding.
\newblock In \emph{Conference on Language Modeling}, 2024.
\newblock URL \url{https://openreview.net/forum?id=BDBdblmyzY}.
\bibitem[Kudo(2018)]{kudo-2018-subword}
Taku Kudo.
\newblock Subword regularization: Improving neural network translation models
with multiple subword candidates.
\newblock In \emph{Proceedings of the Annual Meeting of the Association for
Computational Linguistics}, 2018.
\newblock \doi{10.18653/v1/P18-1007}.
\newblock URL \url{https://aclanthology.org/P18-1007}.
\bibitem[Kudo \& Richardson(2018)Kudo and
Richardson]{kudo-richardson-2018-sentencepiece}
Taku Kudo and John Richardson.
\newblock {S}entence{P}iece: {A} simple and language independent subword
tokenizer and detokenizer for neural text processing.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, 2018.
\newblock \doi{10.18653/v1/D18-2012}.
\newblock URL \url{https://aclanthology.org/D18-2012}.
\bibitem[Kwon et~al.(2023)Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang,
and Stoica]{vllm}
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody~Hao Yu,
Joseph~E. Gonzalez, Hao Zhang, and Ion Stoica.
\newblock Efficient memory management for large language model serving with
{PagedAttention}.
\newblock In \emph{Proceedings of the ACM SIGOPS Symposium on Operating Systems
Principles}, 2023.
\newblock URL \url{https://arxiv.org/abs/2309.06180}.
\bibitem[Levy(2008)]{levy-2008}
Roger Levy.
\newblock Expectation-based syntactic comprehension.
\newblock \emph{Cognition}, 106\penalty0 (3), 2008.
\newblock ISSN 0010-0277.
\newblock \doi{https://doi.org/10.1016/j.cognition.2007.05.006}.
\newblock URL
\url{https://www.sciencedirect.com/science/article/pii/S0010027707001436}.
\bibitem[Lundberg \& Ribeiro(2023)Lundberg and Ribeiro]{token-healing}
Scott Lundberg and Marco~Tulio Ribeiro.
\newblock The art of prompt design: Prompt boundaries and token healing.
\newblock \emph{Medium}, 2023.
\newblock URL
\url{https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38}.
\bibitem[Microsoft(2023)]{guidance}
Microsoft.
\newblock Guidance.
\newblock \url{https://github.com/microsoft/guidance}, 2023.
\bibitem[Mikolov et~al.(2010)Mikolov, Karafiát, Burget, Černocký, and
Khudanpur]{mikolov10_interspeech}
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev
Khudanpur.
\newblock Recurrent neural network based language model.
\newblock In \emph{Proceedings of INTERSPEECH}, 2010.
\newblock \doi{10.21437/Interspeech.2010-343}.
\newblock URL
\url{https://www.isca-archive.org/interspeech_2010/mikolov10_interspeech.html}.
\bibitem[Oh \& Schuler(2024)Oh and Schuler]{oh2024leading}
Byung-Doh Oh and William Schuler.
\newblock Leading whitespaces of language models' subword vocabulary poses a
confound for calculating word probabilities, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.10851}.
\bibitem[Phan et~al.(2024)Phan, Havasi, Muckley, and
Ullrich]{phan2024understanding}
Buu Phan, Marton Havasi, Matthew~J. Muckley, and Karen Ullrich.
\newblock Understanding and mitigating tokenization bias in language models.
\newblock In \emph{ICML Workshop on Theoretical Foundations of Foundation
Models}, 2024.
\newblock URL \url{https://openreview.net/forum?id=OqfdrBj1y1}.
\bibitem[Pimentel \& Meister(2024)Pimentel and Meister]{pimentel2024compute}
Tiago Pimentel and Clara Meister.
\newblock How to compute the probability of a word, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.14561}.
\bibitem[Poesia et~al.(2022)Poesia, Polozov, Le, Tiwari, Soares, Meek, and
Gulwani]{synchromesh}
Gabriel Poesia, Alex Polozov, Vu~Le, Ashish Tiwari, Gustavo Soares, Christopher
Meek, and Sumit Gulwani.
\newblock Synchromesh: Reliable code generation from pre-trained language
models.
\newblock In \emph{The Tenth International Conference on Learning
Representations, {ICLR} 2022, Virtual Event, April 25-29, 2022}.
OpenReview.net, 2022.
\newblock URL \url{https://openreview.net/forum?id=KmtVD97J43e}.
\bibitem[Radford et~al.(2019)Radford, Wu, Child, Luan, Amodei, Sutskever,
et~al.]{radford2019language}
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
Sutskever, et~al.
\newblock Language models are unsupervised multitask learners.
\newblock \emph{OpenAI blog}, 1\penalty0 (8), 2019.
\newblock URL
\url{https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf}.
\bibitem[Scholak et~al.(2021)Scholak, Schucher, and Bahdanau]{picard}
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau.
\newblock {PICARD:} parsing incrementally for constrained auto-regressive
decoding from language models.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural}, 2021.
\newblock \doi{10.18653/V1/2021.EMNLP-MAIN.779}.
\newblock URL \url{https://doi.org/10.18653/v1/2021.emnlp-main.779}.
\bibitem[Sennrich et~al.(2016)Sennrich, Haddow, and
Birch]{sennrich-etal-2016-neural}
Rico Sennrich, Barry Haddow, and Alexandra Birch.
\newblock Neural machine translation of rare words with subword units.
\newblock In \emph{Proceedings of the Annual Meeting of the Association for
Computational Linguistics}, 2016.
\newblock \doi{10.18653/v1/P16-1162}.
\newblock URL \url{https://aclanthology.org/P16-1162}.
\bibitem[Shannon(1948)]{shannon1948mathematical}
Claude~E. Shannon.
\newblock A mathematical theory of communication.
\newblock \emph{The Bell System Technical Journal}, 27\penalty0 (4), 1948.
\newblock URL \url{https://doi.org/10.1002/j.1538-7305.1948.tb00917.x}.
\bibitem[Sundermeyer et~al.(2015)Sundermeyer, Ney, and
Schl{\"u}ter]{sundermeyer2015feedforward}
Martin Sundermeyer, Hermann Ney, and Ralf Schl{\"u}ter.
\newblock From feedforward to recurrent {LSTM} neural networks for language
modeling.
\newblock \emph{IEEE/ACM Transactions on Audio, Speech, and Language
Processing}, 23\penalty0 (3), 2015.
\newblock URL \url{https://ieeexplore.ieee.org/document/7050391}.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock \emph{Advances in Neural Information Processing Systems}, 30, 2017.
\newblock URL \url{https://arxiv.org/abs/1706.03762}.
\bibitem[Willard \& Louf(2023)Willard and Louf]{outlines}
Brandon~T. Willard and R{\'{e}}mi Louf.
\newblock Efficient guided generation for large language models.
\newblock \emph{CoRR}, abs/2307.09702, 2023.
\newblock \doi{10.48550/ARXIV.2307.09702}.
\newblock URL \url{https://doi.org/10.48550/arXiv.2307.09702}.
\bibitem[Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu,
Le~Scao, Gugger, Drame, Lhoest, and Rush]{wolf-etal-2020-transformers}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le~Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush.
\newblock \huggingface\ {T}ransformers: {S}tate-of-the-art natural language
processing.
\newblock In \emph{Proceedings of the Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, 2020.
\newblock \doi{10.18653/v1/2020.emnlp-demos.6}.
\newblock URL \url{https://aclanthology.org/2020.emnlp-demos.6}.
\end{thebibliography}