Modeling Emergent Lexicon Formation with a Self-Reinforcing Stochastic Process
Extracted source text · arXiv 2206.11146
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass{article} % For LaTeX2e
\usepackage{iclr2022_conference,times}
\usepackage{hyperref}
\usepackage{url}
% Standard package includes
\usepackage{latexsym,amsmath}
\usepackage{booktabs}
\usepackage{algorithm,listings}
\usepackage{graphicx}
\usepackage{subcaption}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
\pdfsuppresswarningpagegroup=1
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{Modeling Emergent Lexicon Formation with a Self-Reinforcing Stochastic Process}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Brendon Boldt, David Mortensen \\
Language Technologies Institute \\
Carnegie Mellon University\\
Pittsburgh, PA 15213, USA \\
\texttt{\{bboldt,dmortens\}@cs.cmu.edu} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand\theProcess{\textsc{FiLex}}
\newcommand\ee[1]{\times10^{#1}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
We introduce \theProcess{}, a self-reinforcing stochastic process which models finite lexicons in emergent language experiments.
The central property of \theProcess{} is that it is a \emph{self-reinforcing} process, parallel to the intuition that the more a word is used in a language, the more its use will continue.
As a theoretical model, \theProcess{} serves as a way to both explain and predict the behavior of the emergent language system.
We empirically test \theProcess{}'s ability to capture the relationship between the emergent language's hyperparameters and the lexicon's Shannon entropy.
\end{abstract}
\section{Introduction}\label{sec:intro}
The methods of emergent language provide a uniquely powerful way to study the foundational questions of how language arises and develops.
While many papers have used emergent language to empirically study such properties, there has been a relative lack of theoretical models accounting for the empirical data.
Such models are important insofar as they can both \emph{explain} and make \emph{predictions} about the behavior of the actual system.
To this end, we introduce \theProcess{}, a self-reinforcing stochastic process based on the Chinese restaurant process, as a model of the Shannon entropy of an emergent language's lexicon.
This simple process models a comparatively complex emergent language system\footnotemark{} (ELS), comprising environment dynamics, neural networks, gradient descent, and reinforcement learning.
\footnotetext{\emph{Emergent language system} or ELS refers to the combination of agents (neural networks), the environment, and the training procedure used as part of an emergent language experiment.}
While the complex system cannot be reasoned about directly, \theProcess{} is simple enough for this.
In Section~\ref{sec:system}, we explain the emergent language system which we are modeling.
Section~\ref{sec:process} introduces \theProcess{} and how it corresponds to presented ELS\@.
Finally, we empirically test \theProcess{}'s ability to capture the relationship between the emergent language's hyperparameters and the lexicon's Shannon entropy in Section~\ref{sec:eval}.\footnote{Code is available at\\ \url{https://github.com/brendon-boldt/filex-emergent-language}}
\paragraph{Related Work}
For a survey of deep learning-based emergent language work, please see \citet{lazaridou2020survey}.
The environment of this paper is most analogous to the near-continuous color-naming environment of \citet{chaabouni2021color} because both environments deal with the discretization of a (near) continuous signal into a smaller number of atomic, discrete symbols.
Insofar as this paper studies the entropy of emergent language, it works in a similar space to \citet{kharitonov2020entropy,chaabouni2021color}.
\citet{mordatch2018emergence} use the Chinese restaurant process as an auxiliary reward to inductively bias the emergent language learning process.
Although this partly inspired the model presented in this work, the methodological roles of an auxiliary reward and theoretical model are distinct.
On a methodological level, this paper is similar to \citet{resnick_capacity_2020} which also develops a theoretical model for an emergent language phenomenon.
Namely, their model predicts the compositionality of an emergent communication protocol dependent on the capacity of the neural-network agents.
Analogously, out model predicts the entropy of the emergent communication protocol dependent on learning rate, replay buffer size, bottleneck size, or training steps.
\section{Emergent Language System}\label{sec:system}
\paragraph{Environment}
In this paper we use a simple $2$-dimensional, obstacle-free navigation environment.
A sender agent observes the position of a receiver agent, sends a message to the receiver, and the receiver takes a step.
For a given episode, the receiver is initialized uniformly at random within a circle and must navigate towards a smaller circular goal region at the center.
An illustration is provided in Appendix~\ref{sec:diagrams}.
The receiver's location and action are continuous variables.
\paragraph{Agent Architecture}
Our architecture comprises two agents, conceptually speaking, but in practice, they are a single neural network.
The \emph{sender} is a disembodied agent which observes the location of the receiver and passes a message in order to guide it towards the goal.
The \emph{receiver} is an agent which receives the message as its only input and takes a step solely based on that message.
The sender and receiver are randomly initialized at the start of training, trained together, and tested together.
The observation of the sender is a pair of floating-point values representing the receiver's location.
The sender itself is a $2$-layer perceptron with tanh activations.
The output of the second layer is passed to a Gumbel-Softmax bottleneck layer \citep{maddison2017concrete,jang2017categorical} which enables learning a discrete, one-hot representation.%
\footnote{Using a Gumbel-Softmax bottleneck layer allows for end-to-end backpropagation, making optimization faster and more consistent than using a backpropagation-free method like REINFORCE \citep{kharitonov2020entropy,williams1992simple}.}
The activations of this layer can be thought of as the words forming the lexicon of the emergent language.
At evaluation time, the bottleneck layer functions deterministically as an argmax layer, emitting one-hot vectors.
The receiver is a $1$-layer perceptron which takes the output of the Gumbel-Softmax layer as input.
The output is a pair of floating-point values which determine the step direction and magnitude of the receiver.
An illustration and precise specification are provided in Appendices~\ref{sec:diagrams} and~\ref{sec:hps}.
\paragraph{Optimization}
Since our environment involves multi-step rewards, we use proximal policy optimization (PPO) \citep{schulman2017ppo} paired with Adam to optimize the neural networks; we use PPO specifically because it is more stable than a vanilla policy gradient method.
We the PPO implementation of Stable Baselines 3 built on PyTorch \citep{stable-baselines3,pytorch}.
\paragraph{Rewards}
We make use of two different rewards in our configuration, a \emph{base} reward and an \emph{shaped} reward.
The base reward is simply a positive reward of $1$ given if the receiver reaches to the goal region before the episode ends and no reward otherwise.
The shaped reward, given at every timestep, is the decrease in distance to the goal, that is $|s|\cdot\cos\theta$, where $|s|$ is the size of the step, and $\theta$ is the bearing to the goal; for example, taking the maximum step size directly towards/away from the goal yields a reward of $1$/$-1$.
\section{\theProcess{} Stochastic Process}\label{sec:process}
\theProcess{}\footnote{Short for ``finite lexicon stochastic process''} is a mathematical model based on the Chinese restaurant process (CRP) \citep{blei2007crp,aldous1985exchangeability}, an iterative stochastic process which yields a probability distribution over the positive integers.
The basic idea of the CRP is that in a hypothetical restaurant, each table corresponds to a positive integer; as each customer walks in, they sit at a random table with a probability proportional to the number of people already at that table.
The key property here is that the process is \emph{self-reinforcing}; tables with many people are likely to get even more.
By analogy to language, the more a word is used the more likely it is to continue to be used.
For example, speakers may develop a cognitive preference for it, or it gets passed along to subsequent generations as a higher rate \citep{Francis2021}.
Furthermore, self-reinforcing processes based on the CRP have been used to model the power-law distributions found in natural language \citep{goldwater2011pitmanyor}.
\paragraph{Formulation}
The pseudocode describing \theProcess{} is given in Figure~\ref{alg:the-process}.
The process starts with an array of weights, length $S$, initialized to $\alpha/S$.
$\alpha$ is a positive, real-valued hyperparameter controlling the uniformity of the final result (high $\alpha$, high uniformity).
At each iteration, we select $\beta$ samples (i.e., indices) with replacement from a categorical distribution parameterized by the weights.
For each, index, we increment the weight at the index by $1/\beta$ so the total update for every iteration is always $1$.
This proceeds $N$ times after which the weights are normalized to $1$ and returned.
The two key differences between \theProcess{} and the CRP are the hyperparameters $S$ and $\beta$.
\theProcess{} only has finitely many parameters so as to match the fact that the agents in the ELS have a fixed-size bottleneck layer, that is, a fixed, finite lexicon.
Secondly, $\beta$ is introduced to account for the fact that certain RL algorithms like PPO accumulate a buffer of steps from the environment with the same parameters before performing gradient descent.
\begin{figure}
\centering
\begin{subfigure}[t]{0.52\linewidth}
\lstset{escapeinside={(*@}{@*)}}
\hrulefill
\vspace{-1ex}
\begin{lstlisting}[
language=Python,
basicstyle=\ttfamily\footnotesize,
numbers=left,
xleftmargin=2.0em,
]
alpha: float > 0
beta: int > 0
N: int >= 0
S: int > 0
weights = array(size = S)
weights.fill(alpha / S)
for _ in range(N):
w_copy = weights.copy()
for _ in range(beta):
i = sample_categorical(w_copy) (*@\label{lst:sample}@*)
weights[i] += 1 / beta
return weights / sum(weights)
\end{lstlisting}
\vspace{-2ex}
\hrulefill{}
\vspace{1ex}
\subcaption{\theProcess{} written in Python pseudocode.}\label{alg:the-process}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.35\linewidth}
\begin{table}[H]
\centering
\begin{tabular}{crr}
Ind.\ var. & ELS & \theProcess{} \\
\midrule
$1/\alpha$ & $-0.86$ & $-0.87$ \\
$\beta$ & $+0.84$ & $+0.95$ \\
$N$ & $-0.61$ & $-0.53$ \\
$S$ & $+0.65$ & $+0.77$ \\
\end{tabular}
\caption{Kendall rank correlation coefficient for each hyperparameter in both the ELS and \theProcess{}. All values have $p<0.005$}\label{tab:quant-eval}
\end{table}
\end{subfigure}
\caption{}
\end{figure}
\paragraph{Analogy}
Identifying the correspondence between the emergent language system and \theProcess{} is key to its explanatory value.
Firstly, the weights of \theProcess{} correspond the learned likelihood with which a given bottleneck unit is used in the ELS\@; in turn, both of these correspond to the frequency with which a word is used in a language.
Each iteration of \theProcess{} is analogous to a whole cycle in the ELS of simulating episodes in the environment, receiving the rewards, and performing gradient descent with respect to the rewards.
In light of this, we can specifically draw connections between the four hyperparameters of \theProcess{}.
As mentioned before $\beta$ corresponds directly to the number of steps in the environment the agent takes before performing an update of the parameters.
$S$ corresponds the size of the bottleneck layer in the ELS\@.
$N$ corresponds the number of steps taken in the environment throughout the course of training the ELS\@.
$\alpha$ controls the initialization of weights of \theProcess{}, and it corresponds, in part, to the \emph{inverse} of the size of the parameter updates in the ELS, that is, the learning rate.
We can see this by looking at the sum of the weights in \theProcess{}:
the sum is equal to $\alpha+n$ after the $n$\textsuperscript{th} iteration.
Since the both the categorical distribution and the final result normalize the weights, they are invariant to scaling by a positive number.
Hence, we could scale by $1/\alpha$ and say that the initial weights sum to $1$ with the sum being $1+n/\alpha$ after $n$ iterations.
Thus, varying $\alpha$ is equivalent to scaling the parameter updates by $1/\alpha$.
\paragraph{Role of the Model}
The key feature of a theoretical model is that it allows us to reason directly about how the phenomena in question work in order to direct subsequent empirical research.
To achieve this, the model must make simplifications from the full system while still accurately describing some aspect of that system; in \theProcess{}'s case, we remove the environment and reward dynamics from consideration while specifically aiming to explain trends in lexicon frequency.
On the other hand, the full, simulated system is necessary to verify that the behavior described by the model is, in fact, accurate.
Wherever the model is inaccurate, it must either be revised or have its scope limited so as to remain in accord with the real system.
In this way, the theoretical model and the ELS work in tandem to better the understanding of various phenomena in emergent language.
\section{Empirical Evaluation}\label{sec:eval}
Our evaluations use the Shannon entropy \citep{shannon} of the lexicon as the dependent variable.
For the ELS, the entropy is calculated from how frequently each of the bottleneck units are used across $3000$ episodes; for \theProcess{}, the entropy is calculated directly over the returned weights.
Hyperparameter details are given in Appendix~\ref{sec:hps}.
The quantitative evaluation is performed by comparing the Kendall rank correlation coefficient between each hyperparameter and entropy for both \theProcess{} and the ELS, shown in Figure~\ref{tab:quant-eval}.
We see that \theProcess{} both produces the correct rank correlation as well as approximating the degree of correlation.
Beyond measuring the correlation, we present the plots of entropy vs.\@ the hyperparameters for both \theProcess{} and the ELS\@ in Figure~\ref{fig:plots} for qualitative evaluation.
With the exception of the experiments varying $S$, $S=2^6=64$, meaning that the maximum entropy is $6$ bits.
Although there is a conceptual correspondence between the hyperparameters of \theProcess{} and ELS, the simplifications of the theoretical model mean that the numerical values of the hyperparameters are not in one-to-one correspondence with the ELS\@.
In light of this, we choose hyperparameters for which \theProcess{} best matches the behavior of the ELS\@; as a result, the $x$-axes of the plots are not identical.
\begin{figure}
\def\plotheight{0.95in}
\def\subfigwidth{0.49}
\centering
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/model-alpha.pdf}
\includegraphics[height=\plotheight]{assets/learning_rate_log-entropy-default.pdf}
\caption{$1/\alpha$ and learning rate}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/model-beta.pdf}
\includegraphics[height=\plotheight]{assets/n_steps_log-entropy-default.pdf}
\caption{$\beta$ and rollout buffer size}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/model-n_params.pdf}
\includegraphics[height=\plotheight]{assets/bottleneck_size_log-entropy-default.pdf}
\caption{$S$ and bottleneck size}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/model-n_iters.pdf}
\includegraphics[height=\plotheight]{assets/total_timesteps_log-entropy-default.pdf}
\caption{$N$ and total training steps}
\end{subfigure}
\caption{
Plots for entropy vs.\@ various hyperparameters.
\theProcess{} is plotted in orange and the ELS in blue.
}\label{fig:plots}
\end{figure}
\paragraph{$\boldsymbol\alpha$ and Learning Rate}
\theProcess{}'s primary prediction regarding $\alpha$ is that as $\alpha$ increases, the entropy of the final distribution increases because for a high $\alpha$, the updates do not move the distribution far from uniformity whereas with a low $\alpha$, the updates make more a difference and can concentrate the distribution, leading to a lower entropy.
This correlation (with $1/\alpha$) is borne out in the ELS experiment varying learning rate; additionally, both plots how a decreasing sigmoid pattern starting from the maximum entropy and going to a non-minimal entropy.
Although the adaptive nature of the Adam optimizer weakens the analogy between these two hyperparameter, their correspondence is still evident empirically.
\paragraph{$\boldsymbol\beta$ and Rollout Buffer}
\theProcess{}'s primary prediction regarding $\beta$ is that as $\beta$ increases, the entropy of the distribution increases because the updates to the weights at each iteration are ``smoother,'' lessening the accumulation on particular weights.
With one exception, this trend is matched by ELS, including the asymptotic shape as $\beta$/buffer size increases.
The ELS plot, though, has a minimum entropy of $2$ bits instead of $0$ bits because the navigation requires $3$ or more ``words'' (each corresponding to one direction) to span a $2$-dimensional space\footnote{$2$ bits of entropy corresponds to $4$ actions; we do not currently have explanation for why the ELS never settles on $3$, the theoretical minimum.}, whereas no such restriction exists in \theProcess{}.
\paragraph{$\boldsymbol S$ and Bottleneck Size}
Note that in this experiment, $\alpha$ is increased in proportion to $S$ so that each individual weight has the same initialization for every $S$, otherwise, a constant $\alpha$ would be ``spread thin.''
\theProcess{}'s primary prediction regarding $S$ is that entropy will simply increase in proportion to $S$ which is matched by ELS experiment.
\paragraph{$\boldsymbol N$ and Training Steps}
\theProcess{}'s primary prediction regarding $N$ is that for low values, the entropy is maximized as the weights are close to their uniform initialization, but, as $N$ increases, the entropy decreases asymptotically to some non-minimal value.
These predictions are generally matched by the ELS experiment.
\section{Conclusion}\label{sec:conclusion}
In this paper we have introduced \theProcess{}, based on the Chinese restaurant process, as a way to explain and predict the behavior of lexicon entropy in a simple emergent language system.
\theProcess{} is a model simple enough to reason about yet nuanced enough to capture the behavior the more complex system as borne out by experimentation.
Building such theoretical models, generally speaking, is an important step in emergent language research as it moves the field toward developing general scientific accounts of the phenomena observed through empirical investigation.
These scientific accounts are what will prove most useful in applying emergent language to linguistics and beyond.
\subsubsection*{Acknowledgements}
This material is based on research sponsored in part by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
% Entries for the entire Anthology, followed by custom entries
\bibliography{custom}
\bibliographystyle{iclr2022_conference}
\appendix
\section{Environment and Agent Architecture}\label{sec:diagrams}
\begin{figure}[H]
\centering
\def\figheight{20ex}
\begin{subfigure}[t]{0.40\linewidth}
\centering
\includegraphics[height=\figheight]{assets/navigation.pdf}
\caption{The receiver (pictured) is rewarded for moving towards the goal region at the center.}\label{fig:env}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.40\linewidth}
\centering
\includegraphics[height=\figheight]{assets/agent-architecture.pdf}
\caption{The agent architecture is a single neural network although conceptually two separate agents: the sender and the receiver.}\label{fig:arch}
\end{subfigure}
\caption{}
\end{figure}
\section{Hyperparameters for Experiments}\label{sec:hps}
\subsection{ELS Experiment Configurations}
\paragraph{Environment}
The environment is a circle $9$ units wide with a circular goal region at the center which has a radius of $1$ unit.
The receiver can move up to $1$ unit per step.
The episode terminates after the $27$\textsuperscript{th} if the agent has not reached the goal.
\paragraph{Agent Architecture}
\begin{itemize}
\item {Bottleneck size}: $64$
\item Architecture; sender is 1-3 and receiver is 5; bottleneck size is $N$
\begin{enumerate}
\item Linear w/ bias: $2$ in, $32$ out
\item Tanh activation
\item Linear w/ bias: $32$ in, $N$ out
\item Gumbel-Softmax: $N$ in, $N$ out
\item Linear w/ bias: $N$ in, $2$ out (action) and $1$ out (value for PPO)
\end{enumerate}
\item {Bottleneck (Gumbel-Softmax) temperature}: $1.5$
\item {Weight initialization}: $\mathcal U\left(-\sqrt \frac1n, \sqrt \frac1n\right)$, where $n$ is the input size of the layer (PyTorch 1.10 default)
\end{itemize}
\paragraph{Optimization}
Unless otherwise noted, we use the default hyperparameters for PPO as specified at \url{https://stable-baselines3.readthedocs.io/en/v1.0/modules/ppo.html}.
Hyperparameters in Table~\ref{tab:opt-hps} were changed from their default.
\begin{table}[H]
\centering
\begin{tabular}{lr}
Hyperparameter & Value \\
\midrule
Total training steps & $5\ee{4}$ \\
Evaluation episodes & $3\ee{3}$ \\
Learning rate & $3\ee{-3}$ \\
Rollout buffer size & $256$ \\
Batch size & $64$ \\
Temporal discount factor ($\gamma$) & $0.9$ \\
\end{tabular}
\caption{Optimization hyperparameters changed from their default value.}\label{tab:opt-hps}
\end{table}
Learning rate corresponds applies to the whole end-to-end network.
\paragraph{Experiments}
Each experiment uses a logarithmic sweep across hyperparameters; the sweep is defined by Equation~\ref{eq:log-sweep}, where $x$ and $y$ are the inclusive upper and lower bounds respectively and $n$ is the number steps to divide the interval into.
The floor function is applied if the elements must be integers.
\begin{equation}
\text{LS}(x,y,n)=
\left\{x\cdot\left(\frac{y}{x}\right)^{\frac{i}{n-1}} \,\middle|\, i \in \{0, 1, \dots, n-1\}\right\}
\label{eq:log-sweep}
\end{equation}
Table~\ref{tab:els-hps} provides the varied hyperparameters for the ELS experiments.
\begin{table}[H]
\centering
\begin{tabular}{lrrr}
ELS & Low & High & Steps \\
\midrule
learning rate & $10^{-4}$ & $10^{-1}$ & $200$ \\
buffer size & $2^{3}$ & $2^{15}$ & $600$ \\
bottleneck size & $2^{3}$ & $2^{8}$ & $400$ \\
training steps & $10^{2}$ & $10^{6}$ & $400$ \\
\end{tabular}
\caption{%
Logarithmic sweeps for each of the ELS hyperparameters.
All other hyperparameters keep their default values.
}\label{tab:els-hps}
\end{table}
\subsection{\theProcess{} Experiment Configurations}
The hyperparameters for the \theProcess{} experiments are provided in Table~\ref{tab:filex-hps}.
\begin{table}[H]
\centering
\begin{tabular}{rrrr}
$\alpha$ & $\beta$ & $S$ & $N$ \\
\midrule
$\text{LS}(10^{-4}, 10^{-1}, 200)$ & $10$ & $64$ & $10^3$ \\
$10^{-3}$ & $\text{LS}(2^3, 2^{15}, 600)$ & $64$ & $10^4$ \\
$5\ee{-3}\cdot S$\textsuperscript{\textdagger} & $10$ & $\text{LS}(2^3, 2^{8}, 400)$ & $10^3$ \\
$1$ & $5$ & $64$ & $\text{LS}(10^{2}, 10^{6}, 400)$ \\
\end{tabular}
\caption{%
Hyperparameters for the empirical evaluation of \theProcess{}.
Each row corresponds to one experiment (which has one varying hyperparameter, specified by $\text{LS}(\cdot,\cdot,\cdot)$ from Equation~\ref{eq:log-sweep}).
\textsuperscript\textdagger{}$\alpha$ is multiplied by the hyperparameter $S$ so that the individual weights are initialized to $5\ee{-3}$.
}\label{tab:filex-hps}
\end{table}
\section{Further Experimental Results}
\begin{figure}[H]
\def\plotheight{1in}
\def\subfigwidth{0.24}
\centering
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/learning_rate_log-steps-default.pdf}
\caption{Learning rate}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/n_steps_log-steps-default.pdf}
\caption{Rollout buffer size}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/bottleneck_size_log-steps-default.pdf}
\caption{Bottleneck size}
\end{subfigure}
\begin{subfigure}[t]{\subfigwidth\linewidth}
\centering
\includegraphics[height=\plotheight]{assets/total_timesteps_log-steps-default.pdf}
\caption{Total training steps}
\end{subfigure}
\caption{
Plots for average steps to complete episodes (lower is better) vs.\@ various hyperparameters for the ELS\@.
All data points have a $100\%$ success rate.
}\label{fig:appendix-plots}
\end{figure}
\end{document}
%%% END DOCUMENT %%%
%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
\begin{thebibliography}{16}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Aldous(1985)]{aldous1985exchangeability}
David~J. Aldous.
\newblock Exchangeability and related topics.
\newblock In P.~L. Hennequin (ed.), \emph{{\'E}cole d'{\'E}t{\'e} de
Probabilit{\'e}s de Saint-Flour XIII --- 1983}, pp.\ 1--198, Berlin,
Heidelberg, 1985. Springer Berlin Heidelberg.
\newblock ISBN 978-3-540-39316-0.
\bibitem[Blei(2007)]{blei2007crp}
David Blei.
\newblock The chinese restaurant process, 2007.
\newblock URL
\url{https://www.cs.princeton.edu/courses/archive/fall07/cos597C/scribe/20070921.pdf}.
\bibitem[Chaabouni et~al.(2021)Chaabouni, Kharitonov, Dupoux, and
Baroni]{chaabouni2021color}
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni.
\newblock Communicating artificial neural networks develop efficient
color-naming systems.
\newblock \emph{Proceedings of the National Academy of Sciences}, 118\penalty0
(12), Mar 2021.
\newblock ISSN 0027-8424, 1091-6490.
\newblock \doi{10.1073/pnas.2016569118}.
\newblock URL \url{https://www.pnas.org/content/118/12/e2016569118}.
\bibitem[Francis et~al.(2021)Francis, Rabinovich, Samir, Mortensen, and
Stevenson]{Francis2021}
David Francis, Ella Rabinovich, Farhan Samir, David Mortensen, and Suzanne
Stevenson.
\newblock {Quantifying Cognitive Factors in Lexical Decline}.
\newblock \emph{Transactions of the Association for Computational Linguistics},
9:\penalty0 1529--1545, 12 2021.
\newblock ISSN 2307-387X.
\newblock \doi{10.1162/tacl_a_00441}.
\newblock URL \url{https://doi.org/10.1162/tacl\_a\_00441}.
\bibitem[Goldwater et~al.(2011)Goldwater, Griffiths, and
Johnson]{goldwater2011pitmanyor}
Sharon Goldwater, Thomas~L. Griffiths, and Mark Johnson.
\newblock Producing power-law distributions and damping word frequencies with
two-stage language models.
\newblock \emph{J. Mach. Learn. Res.}, 12:\penalty0 2335--2382, 2011.
\bibitem[Jang et~al.(2017)Jang, Gu, and Poole]{jang2017categorical}
Eric Jang, Shixian Gu, and Ben Poole.
\newblock Categorical reparameterization with gumbel-softmax.
\newblock In \emph{Proceedings of the 2017 International Conference on Learning
Representations (ICLR)}, 2017.
\newblock URL \url{https://openreview.net/forum?id=rkE3y85ee}.
\bibitem[Kharitonov et~al.(2020)Kharitonov, Chaabouni, Bouchacourt, and
Baroni]{kharitonov2020entropy}
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni.
\newblock Entropy minimization in emergent languages.
\newblock In Hal~Daumé III and Aarti Singh (eds.), \emph{Proceedings of the
37th International Conference on Machine Learning}, volume 119 of
\emph{Proceedings of Machine Learning Research}, pp.\ 5220--5230. PMLR,
13--18 Jul 2020.
\newblock URL \url{http://proceedings.mlr.press/v119/kharitonov20a.html}.
\bibitem[Lazaridou \& Baroni(2020)Lazaridou and Baroni]{lazaridou2020survey}
Angeliki Lazaridou and Marco Baroni.
\newblock Emergent multi-agent communication in the deep learning era.
\newblock \emph{arXiv:2006.02419 [cs]}, Jul 2020.
\newblock URL \url{http://arxiv.org/abs/2006.02419}.
\newblock arXiv: 2006.02419.
\bibitem[Maddison et~al.(2017)Maddison, Mnih, and Teh]{maddison2017concrete}
Chris~J. Maddison, Andriy Mnih, and Yee~Whye Teh.
\newblock The concrete distribution: A continuous relaxation of discrete random
variables.
\newblock In \emph{Proceedings of the 2017 International Conference on Learning
Representations (ICLR)}, 2017.
\newblock URL \url{https://openreview.net/forum?id=S1jE5L5gl}.
\bibitem[Mordatch \& Abbeel(2018)Mordatch and Abbeel]{mordatch2018emergence}
Igor Mordatch and Pieter Abbeel.
\newblock Emergence of grounded compositional language in multi-agent
populations, 2018.
\bibitem[Paszke et~al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison,
Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala]{pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban
Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan
Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu~Fang, Junjie Bai, and Soumith
Chintala.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer, F.~d\textquotesingle
Alch\'{e}-Buc, E.~Fox, and R.~Garnett (eds.), \emph{Advances in Neural
Information Processing Systems 32}, pp.\ 8024--8035. Curran Associates,
Inc., 2019.
\newblock URL
\url{http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf}.
\bibitem[Raffin et~al.(2019)Raffin, Hill, Ernestus, Gleave, Kanervisto, and
Dormann]{stable-baselines3}
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi
Kanervisto, and Noah Dormann.
\newblock Stable baselines3.
\newblock \url{https://github.com/DLR-RM/stable-baselines3}, 2019.
\bibitem[Resnick et~al.(2020)Resnick, Gupta, Foerster, Dai, and
Cho]{resnick_capacity_2020}
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew~M. Dai, and Kyunghyun
Cho.
\newblock Capacity, {Bandwidth}, and {Compositionality} in {Emergent}
{Language} {Learning}.
\newblock \emph{International Conference on Autonomous Agents and Multi-Agent
Systems}, April 2020.
\newblock URL \url{http://arxiv.org/abs/1910.11424}.
\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and
Klimov]{schulman2017ppo}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.
\bibitem[Shannon(1948)]{shannon}
C.~E. Shannon.
\newblock A mathematical theory of communication.
\newblock \emph{The Bell System Technical Journal}, 27\penalty0 (3):\penalty0
379--423, 1948.
\newblock \doi{10.1002/j.1538-7305.1948.tb01338.x}.
\bibitem[Williams(1992)]{williams1992simple}
Ronald~J Williams.
\newblock Simple statistical gradient-following algorithms for connectionist
reinforcement learning.
\newblock \emph{Machine learning}, 8\penalty0 (3):\penalty0 229--256, 1992.
\end{thebibliography}