AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training
Extracted source text · arXiv 2511.14721
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
\PassOptionsToPackage{numbers,square,sort&compress}{natbib}
% \usepackage{neurips_2025}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
% \usepackage[main]{neurips_2025}
% 2. "position" option is used for the Position Paper Track
% \usepackage[position]{neurips_2025}
% 3. "dandb" option is used for the Datasets & Benchmarks Track
% \usepackage[dandb]{neurips_2025}
% 4. "creativeai" option is used for the Creative AI Track
% \usepackage[creativeai]{neurips_2025}
% 5. "sglblindworkshop" option is used for the Workshop with single-blind reviewing
% \usepackage[sglblindworkshop]{neurips_2025}
% 6. "dblblindworkshop" option is used for the Workshop with double-blind reviewing
% \usepackage[dblblindworkshop]{neurips_2025}
% After being accepted, the authors should add "final" behind the track to compile a camera-ready version.
% 1. Main Track
% \usepackage[main, final]{neurips_2025}
% 2. Position Paper Track
% \usepackage[position, final]{neurips_2025}
% 3. Datasets & Benchmarks Track
% \usepackage[dandb, final]{neurips_2025}
% 4. Creative AI Track
% \usepackage[creativeai, final]{neurips_2025}
% 5. Workshop with single-blind reviewing
% \usepackage[sglblindworkshop, final]{neurips_2025}
% 6. Workshop with double-blind reviewing
\usepackage[dblblindworkshop, final]{neurips_2025}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
% For workshops (5., 6.), the authors should add the name of the workshop, "\workshoptitle" command is used to set the workshop title.
\workshoptitle{GPU-Accelerated and Scalable Optimization (ScaleOpt)}
% "preprint" option is used for arXiv or other preprint submissions
% \usepackage[preprint]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
% \usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% \PassOptionsToPackage[numbers,sort&compress]{natbib}
% \usepackage[nonatbib]{neurips_2025}
\input{math_commands.tex}
\usepackage{graphicx}
\usepackage[colorlinks=true,linkcolor=blue, citecolor=blue]{hyperref}
\usepackage{amsmath,amssymb,amsthm,amsfonts,mathrsfs,bm,multirow}
\usepackage{subcaption}
\usepackage{nicefrac}
\usepackage{mathtools}
\usepackage{algorithm, algorithmicx, algpseudocode}
\usepackage{booktabs}
\newcommand{\x}{\mathbf{x}}
\newcommand{\bc}{\mathbf{c}}
\newcommand{\pert}{\mathbf{\delta}}
\newcommand{\intr}{\mathcal{I}}
\newcommand{\loss}{\mathcal{L}}
\newcommand{\dist}{\mathcal{D}}
\newcommand{\classes}{\mathcal{C}}
\newcommand{\lam}{\lambda}
\DeclareMathOperator*{\minimize}{\text{minimize}}
\DeclareMathOperator*{\maximize}{\text{maximize}}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
\title{AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training}
\author{
Fu-Ming Guo \\
Stanford University\\
% Cambridge, MA 02114 \\
\texttt{fmguo@stanford.edu} \\
\And Yingfang Fan \\
Harvard Medical School \\
\texttt{yfan8@mgh.harvard.edu}
}
\begin{document}
\maketitle
\begin{abstract}
Adaptive optimizers with decoupled weight decay, such as AdamW, are the de-facto standard for pre-training large transformer-based generative models. Yet the quadratic nature of the $\ell_2$ penalty embedded in weight decay drives all parameters toward the origin at the same rate, making the update vulnerable to rare but extreme gradient directions and often over-penalizing well-conditioned coordinates. We propose AdamHuberDecay, a drop-in replacement for AdamW that substitutes the $\ell_2$ penalty with a decoupled smooth Huber regularizer. The resulting update decays parameters quadratically while their magnitude remains below a threshold $\delta$, and linearly ($\ell_1$-like) once they exceed $\delta$ , yielding (i) bounded regularization gradients, (ii) invariance to per-coordinate second-moment rescaling, and (iii) stronger sparsity pressure on over-grown weights.
We derive the closed-form decoupled Huber decay step, and show how to integrate it with any Adam-family optimizer at $O(1)$ extra cost. Extensive experiments on GPT2 and GPT-3 pre-training demonstrate that AdamHuberDecay: (a) converges $10-15\%$ faster in wall-clock time, (b) reduces validation perplexity by up to 4 points, (c) performance improvements of 2.5-4.7\% across downstream tasks and (d) yields visibly sparser weight histograms that translate into 20–30 \% memory savings after magnitude pruning—without tuning the decay coefficient beyond the default grid used for AdamW. Ablations confirm robustness to outlier gradients and large-batch regimes, together with theoretical analyses that bound the expected parameter norm under noisy updates. AdamHuberDecay therefore provides a simple, principled path toward more efficient and resilient training of next-generation foundational generative transformers.
% Adaptive optimizers with decoupled weight decay (e.g., AdamW) are standard for pretraining large transformers, yet the quadratic $\ell_2$ penalty pulls all parameters toward zero at a uniform rate, exacerbating late‑stage over‑decay and over‑shrinking informative large weights. We introduce \textbf{AdamHuberDecay (AdamHD)}, which replaces $\ell_2$ decay with a decoupled Huber regularizer: parameters are decayed quadratically for $|w|\le\delta$ and linearly ($\ell_1$‑like) for $|w|>\delta$, yielding (i) a bounded shrinkage force on large weights, (ii) a decay term independent of second‑moment preconditioning, and (iii) stronger sparsity pressure. We derive the closed‑form \emph{proximal} Huber step and integrate it into Adam at $O(1)$ overhead. On GPT‑2 and GPT‑3–class pretraining runs, AdamHD reaches target perplexities $10$–$15\%$ faster, lowers validation perplexity by up to $4$ points, and improves downstream accuracy by $2.5$–$4.7\%$ under matched budgets and standard decay grids. The method also produces sparser weight distributions that improve unstructured magnitude‑pruning efficiency. Ablations compare proximal vs.\ Euler‑style variants, study sensitivity to $\delta$, and probe large‑batch and clipping regimes. We further bound the expected parameter norm under noisy updates, explaining the observed late‑phase stability. AdamHD is a simple, principled, drop‑in alternative to AdamW for efficient, resilient pretraining of generative transformers.
\end{abstract}
\section{Introduction}
\vspace{-0.2em}
From the earliest days of neural networks \cite{rumelhart1986learning}, researchers have recognized that explicit regularization biases can improve generalization. Rumelhart’s epoch‐making work in the late 1980s proposed adding weight penalties to encourage ``minimal’’ network solutions \citep{hanson1988comparing}. This insight remains pivotal today as Large Language Models~(LLMs) scale to billions of parameters \cite{radford2018improving, brown2020language, ouyang2022training, comanici2025gemini, team2024gemini, achiam2023gpt, hurst2024gpt}. State-of-the-art LLM training hinges on \emph{adaptive optimizers} equipped with \emph{decoupled regularization}. The canonical example is \textbf{AdamW} \cite{loshchilovdecoupled}, which separates weight decay from the gradient‐based update, thereby stabilizing training and improving generalization \citep{loshchilovdecoupled}. Building on this principle, \emph{decoupled momentum} (DeMo) variants refine how momentum is accumulated \citep{peng2024decoupled, guo2023sparseoptimizer}, while the recently discovered \textbf{Lion} optimizer further decouples the update by applying only the \emph{sign} of the momentum, yet still relies on decoupled decay for accuracy and robustness \citep{chen2023symbolic, guo2019reweighted}. Together, these results underscore a common theme: careful, decoupled regularization is indispensable for taming the complexity of modern deep networks.
\vspace{0.2em}
% \noindent\textbf{The over decay problem.}\;
% Despite these advances, practitioners report an \textit{over decay} phenomenon in large-scale pre-training: standard weight decay can suppress parameter magnitudes \emph{too aggressively}, particularly in the tail of training when learning rates are low. \cite{wang2024set} reinterpret AdamW as an exponential moving average and show that the effective decay strength should \emph{decrease} with dataset length or training time, yet practical prescriptions remain heuristic. In production LLM stacks, engineers often mitigate over-decay by ad-hoc decay schedules, layer‐wise scaling, or repeated hyper-parameter sweeps—none of which fully resolve late-stage under-utilization of model capacity.
\noindent\textbf{The over‑decay problem.}\;
In large‑scale pre‑training, standard weight decay can suppress parameter magnitudes \emph{too aggressively}, particularly in the low‑learning‑rate tail. Building on an exponential‑moving‑average view of AdamW, recent work \cite{wang2024set} suggests the effective decay should \emph{decrease} with dataset length or training time, but existing guidance is largely heuristic. In practice, ad hoc decay schedules, layer‑wise decay scaling, and repeated hyperparameter sweeps are common, yet none reliably prevent late‑stage under‑utilization of model capacity.
\vspace{0.2em}
\noindent\textbf{Huber decay: a robust alternative.}\;
We revisit robust statistics and propose replacing conventional $\ell_2$ decay with a \emph{Huber-style} penalty that transitions from quadratic to linear growth. For scalar $a$ and threshold $\delta>0$,
\begin{equation}
\label{eq:huber}
H_{\delta}(a)=
\begin{cases}
\tfrac{1}{2}a^{2}, & |a|\le\delta,\\[4pt]
\delta\!\left(|a|-\tfrac{1}{2}\delta\right), & |a|>\delta.
\end{cases}
\end{equation}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{huber_regularizers.pdf}
\caption{Geometrical illustration of the Huber-norm regularizer and comparison to common ones.}
\label{fig:placeholder}
\end{figure}
Eq.(~\eqref{eq:huber}) coincides with standard weight decay when $|a|\le\delta$, but \emph{clips} the regularization force to the constant $\delta\,\mathrm{sign}(a)$ once $|a|>\delta$. Although the Huber loss (often called ``smooth‐$\ell_1$’’) is ubiquitous on the \emph{prediction} side of deep learning, e.g.\ in object detection, we are unaware of any work that applies a \emph{Huber‐style weight penalty} to Transformer pre-training. The gap likely persists because:
(1)~Huber introduces an additional break-point~$\delta$ to tune;
(2)~deep-learning kernels frequently \emph{fuse} $\ell_2$ decay into a single CUDA update—adding a per-parameter clip breaks this fast path; and
(3)~modern LLM stacks already manage per-group decay masks (bias/norm exclusions), so an extra $\delta$-dependent clip complicates the optimizer pipeline.
\vspace{0.2em}
\noindent\textbf{Our contribution — \textit{AdamHD}.}\;
To bridge this gap we introduce \textbf{AdamHD}, an AdamW variant that \emph{decouples the Huber decay regularization}. Each parameter receives a decay gradient
$
\nabla_w\!\Omega(w)=
\begin{cases}
w,&|w|\le\delta,\\
\delta\,\mathrm{sign}(w),&|w|>\delta,
\end{cases}
$
so that large weights experience a capped, $\ell_1$-like \cite{candes2008enhancing} \cite{zhang2023optimal} while small weights behave exactly as under $\ell_2$. The result is a \emph{scale-aware} regularizer that mitigates over decay yet preserves early-phase stabilization. Empirically, AdamHD improves training robustness and convergence when pre-training GPT-2 and GPT-3 models from scratch: it sustains healthy weight norms in late epochs, reaches target perplexities in fewer updates, and does so with negligible computational overhead.
\vspace{0.2em}
\noindent In sum, AdamHD marries the adaptability of Adam with a robust, clipped regularization scheme that squarely targets the over decay problem—opening a new avenue for efficient and principled pre-training of next-generation LLMs.
\section{Background}
\label{gen_inst}
\paragraph{Decoupling regularization and momentum.}
Decoupled weight decay regularization is now a cornerstone of modern optimizer design \cite{zhang2024riemannian}\cite{parshakova2024implementation}.
\cite{loshchilovdecoupled} introduced AdamW, which applies the $\ell_2$ penalty as a direct parameter shrinkage, leaving Adam’s adaptive update untouched and greatly improving training stability.
Extending this “orthogonal sub-update’’ principle, recent work has also decoupled momentum.
\emph{DeMo} \cite{peng2024decoupled} synchronises only a compressed “fast’’ component of momentum across devices, slashing communication while matching AdamW’s convergence .
\emph{Lion} goes further, updating parameters with the \emph{sign} of an exponential-moving-average of gradients, discarding magnitude information to gain robustness and memory efficiency \citep{chen2023symbolic}.
The general idea of biasing updates toward simpler representations has roots in early “minimal-network’’ studies \citep{hanson1988comparing}, yet its practical realisation for transformer pre-training remains sparse.
\paragraph{Motivation for a Huber-style penalty.}
A \emph{scale-aware} alternative is to replace the fixed quadratic penalty with a \textbf{Huber} loss on weights: quadratic for $|\theta_i|\le\delta$, linear beyond.
Such a piecewise regulariser preserves the smoothing effect of $\ell_2$ for small weights while capping the shrinkage exerted on large, information-bearing parameters, directly targeting late-stage over-decay.
Crucially, it slots naturally into the decoupled-update framework underpinning AdamW, DeMo and Lion, offering a clean path to more robust large-model optimisation.
% ------------------------------------------------------------
\section{Method Formulation: Decoupled Huber Decay Regularization}
\label{sec:formulation}
% ------------------------------------------------------------
\subsection{Language‐Model Pre-Training Objective}
\label{subsec:objective}
Consider an autoregressive Transformer with parameters $\boldsymbol{\theta}\in\mathbb{R}^{d}$.
Given a token sequence $(x_1,x_2,\dots,x_T)$ drawn from the corpus $\mathcal{D}$, the model specifies a conditional distribution
\[
p_{\boldsymbol{\theta}}\!\bigl(x_t\,\big|\,x_{<t}\bigr),
\qquad x_{<t}:=(x_1,\dots,x_{t-1}).
\]
The causal language-modeling (CLM) loss is the corpus-average cross-entropy
\begin{equation}
\label{eq:clm}
\mathcal{L}(\boldsymbol{\theta})
\;=\;
\mathbb{E}_{(x_1,\ldots,x_T)\sim\mathcal{D}}
\Bigl[-\tfrac1T
\sum_{t=1}^{T}
\log p_{\boldsymbol{\theta}}\!\bigl(x_t\mid x_{<t}\bigr)\Bigr].
\end{equation}
Equivalently, letting $N$ be the total number of tokens in the corpus,
\vspace{-0.4em}
\[
\mathcal{L}(\boldsymbol{\theta})
\;=\;
-\,\frac1N
\sum_{i=1}^{N}
\log p_{\boldsymbol{\theta}}\!\bigl(y_i \mid \text{context}_i\bigr),
\]
where $y_i$ is the $i$-th token and $\text{context}_i$ is its prefix.
% ------------------------------------------------------------
\subsection{Adaptive Optimizers in Current LLM Training}
\label{subsec:optimizers}
Let $\boldsymbol{g}_t := \nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta}_t)$
be the (mini-batch) gradient at step~$t$.
We derive our optimizer from Adam \cite{kinga2015method}.
\subsubsection*{Adam}
\begin{align}
\boldsymbol{m}_t &= \beta_1\boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\\
\boldsymbol{v}_t &= \beta_2\boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t\odot\boldsymbol{g}_t,\\
\boldsymbol{\theta}_{t+1}
&= \boldsymbol{\theta}_t
- \alpha_t\,
\frac{\boldsymbol{m}_t}{\sqrt{\boldsymbol{v}_t} + \varepsilon}.
\end{align}
\subsubsection*{AdamW (decoupled $L_2$ decay)}
\begin{equation}
\boldsymbol{\theta}_{t+1}
\;=\;
\boldsymbol{\theta}_t
- \alpha_t
\frac{\boldsymbol{m}_t}{\sqrt{\boldsymbol{v}_t} + \varepsilon}
- \alpha_t\,\lambda\,\boldsymbol{\theta}_t.
\end{equation}
% \subsubsection*{Sophia (diagonal Hessian preconditioning)}
% \begin{align}
% \boldsymbol{m}_t &= \beta_1\boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\\
% \boldsymbol{d}_t &= \beta_2\boldsymbol{d}_{t-1} + (1-\beta_2)\boldsymbol{H}_t,\\
% \boldsymbol{u}_t &= \text{clip}\!\Bigl(
% \frac{\boldsymbol{m}_t}{\boldsymbol{d}_t + \varepsilon},
% -c,\;c\Bigr),\\
% \boldsymbol{\theta}_{t+1} &= \boldsymbol{\theta}_t - \alpha_t \boldsymbol{u}_t.
% \end{align}
% \subsubsection*{Lion (evolved sign momentum)}
% \begin{align}
% \boldsymbol{u}_t &= \beta_1\boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\\
% \boldsymbol{m}_t &= \beta_2\boldsymbol{m}_{t-1} + (1-\beta_2)\boldsymbol{g}_t,\\
% \boldsymbol{\theta}_{t+1}
% &= \boldsymbol{\theta}_t
% - \alpha_t\,\mathrm{sign}(\boldsymbol{u}_t).
% \end{align}
% ------------------------------------------------------------
\subsection{Huber-Decay Regularization \texorpdfstring{(AdamHD)}{}}
\label{subsec:adamhd}
\textbf{Huber penalty on weights.} For a scalar $a$ and threshold $\delta>0$, the Huber loss is:
\[
H_{\delta}(a)
=
\begin{cases}
\tfrac12\,a^{2}, & |a|\le\delta,\\
\delta\,|a| - \tfrac12\,\delta^{2}, & |a|>\delta.
\end{cases}
\]
Define the regularizer
$\displaystyle
R_{\delta}(\boldsymbol{\theta}) = \sum_{i=1}^{d} H_{\delta}(\theta_i).
$
Its first order gradient is a clipping operator:
\[
\nabla R_{\delta}(\boldsymbol{\theta})
=
\mathrm{clip}\bigl(\boldsymbol{\theta},-\delta,+\delta\bigr).
\]
\textbf{Adaptive thresholds.} For each parameter tensor $\boldsymbol{\Theta}^{(l)}$:
\begin{align}
\text{Mean-magnitude:}\quad
\delta_t^{(l)}
&= c\;\frac{1}{\lvert\boldsymbol{\Theta}^{(l)}\rvert}
\sum_{ij}\bigl|\Theta^{(l)}_{ij,t}\bigr|,\\
\text{EMA:}\quad
\mu_t^{(l)}
&= \beta_0\,\mu_{t-1}^{(l)}
+ (1-\beta_0)\;
\frac{1}{\lvert\boldsymbol{\Theta}^{(l)}\rvert}
\sum_{ij}\bigl|\Theta^{(l)}_{ij,t}\bigr|,\\
\delta_t^{(l)} &= c\,\mu_t^{(l)}.
\end{align}
\subsubsection*{AdamHD Euler style update}
\begin{align}
\boldsymbol{m}_t &= \beta_1\boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t,\\
\boldsymbol{v}_t &= \beta_2\boldsymbol{v}_{t-1} + (1-\beta_2)\boldsymbol{g}_t\odot\boldsymbol{g}_t,\\
\boldsymbol{\theta}_{t+1}
&= \boldsymbol{\theta}_t
- \alpha_t\,\frac{\boldsymbol{m}_t}{\sqrt{\boldsymbol{v}_t}+\varepsilon}
- \alpha_t\,\lambda\;
\mathrm{clip}\bigl(\boldsymbol{\theta}_t,
-\boldsymbol{\delta}_t,
+\boldsymbol{\delta}_t\bigr).
\end{align}
Above, $\boldsymbol{\delta}_t$ concatenates
$\delta_t^{(l)}$ to match $\boldsymbol{\theta}_t$ component-wise.
\paragraph{Decoupled proximal view.}
Given the Adam preconditioned step
\[
\tilde{\boldsymbol{\theta}}_t \;=\; \boldsymbol{\theta}_t
\;-\; \alpha_t\,\frac{\boldsymbol{m}_t}{\sqrt{\boldsymbol{v}_t}+\varepsilon},
\]
we define the next iterate as the proximal map of the Huber regularizer:
\begin{equation}
\label{eq:prox-problem}
\boldsymbol{\theta}_{t+1}
\;=\;
\mathrm{prox}_{\alpha_t \lambda\,R_{\boldsymbol{\delta}_t}}
\!\left(\tilde{\boldsymbol{\theta}}_t\right)
:=\arg\min_{\boldsymbol{\theta}}\;
\frac{1}{2\alpha_t}\|\boldsymbol{\theta}-\tilde{\boldsymbol{\theta}}_t\|_2^2
+\lambda\,R_{\boldsymbol{\delta}_t}(\boldsymbol{\theta}).
\end{equation}
Because $R_{\boldsymbol{\delta}}$ is separable, \eqref{eq:prox-problem} decomposes coordinate‑wise.
\paragraph{Closed‑form proximal operator.}
Let $y\in\mathbb{R}$, $\tau := \alpha_t\lambda>0$, and $\delta>0$. The scalar proximal operator
\[
x^\star \;=\; \mathrm{prox}_{\tau H_{\delta}}(y)
\quad\text{minimizes}\quad
\frac{1}{2}(x-y)^2+\tau H_{\delta}(x).
\]
Solving the optimality condition $(x-y)+\tau\,H'_{\delta}(x)=0$ yields the piecewise closed form
\begin{equation}
\label{eq:prox-huber-closed}
\mathrm{prox}_{\tau H_{\delta}}(y)
\;=\;
\begin{cases}
\displaystyle \frac{y}{1+\tau}, & |y|\le (1+\tau)\,\delta,\\[10pt]
\displaystyle y-\tau\,\delta\,\mathrm{sign}(y), & |y|>(1+\tau)\,\delta.
\end{cases}
\end{equation}
Hence the proximal step is a \emph{scaled shrink} for moderate $|y|$, and an \emph{$\ell_1$‑like soft shift with capped slope} for large $|y|$. By separability, \eqref{eq:prox-huber-closed} applies elementwise with $(y,\delta)\mapsto (\tilde{\theta}_{t,i},\delta_{t,i})$.
\paragraph{Full AdamHD-prox update.}
Let $\odot$ denote elementwise multiplication.
\begin{align}
\boldsymbol{m}_t &= \beta_1\boldsymbol{m}_{t-1} + (1-\beta_1)\boldsymbol{g}_t, \\
\boldsymbol{v}_t &= \beta_2\boldsymbol{v}_{t-1} + (1-\beta_2)\,\boldsymbol{g}_t\odot\boldsymbol{g}_t,\\
\tilde{\boldsymbol{\theta}}_t
&= \boldsymbol{\theta}_t - \alpha_t\frac{\boldsymbol{m}_t}{\sqrt{\boldsymbol{v}_t}+\varepsilon},\\
\boldsymbol{\theta}_{t+1}
&= \mathrm{prox}_{\alpha_t \lambda\,R_{\boldsymbol{\delta}_t}}(\tilde{\boldsymbol{\theta}}_t)
\quad\text{via \eqref{eq:prox-huber-closed} applied elementwise}.
\end{align}
\subsection{Convergence and Trade-Offs}
\label{subsec:convergence}
\begin{itemize}
\item \textbf{Stability.}
The proximal Huber step is firmly nonexpansive and caps the per-step
shrink induced by the regularizer: for $y=\tilde\theta_{t,i}$ and $\tau=\alpha_t\lambda$,
$\bigl|\mathrm{prox}_{\tau H_\delta}(y)-y\bigr|\le \min\!\big(\tfrac{\tau}{1+\tau}|y|,\;\tau\delta\big)$.
Thus the decay displacement saturates at $\mathbf{\alpha_t\lambda\delta}$ per coordinate.
\item \textbf{Noise sensitivity.}
Because the prox map is $1$-Lipschitz (firmly nonexpansive), the decay step does
not amplify perturbations in $\tilde\theta_t$. It specifically caps the
\emph{regularizer's} contribution; robustness to loss-gradient spikes
still requires standard practices (e.g., gradient clipping).
\item \textbf{Sparsity bias.}
For $|y|>(1+\tau)\delta$, the update is $y\!-\!\tau\delta\,\mathrm{sign}(y)$,
i.e., a constant-magnitude shrink that increases pressure on large
coordinates without inducing exact zeros, unlike $L_1$.
\item \textbf{Limit cases.}
As $\delta\!\to\!\infty$, the step reduces to the decoupled proximal-$L_2$ update
$\mathrm{prox}_{\tau \tfrac12\|\cdot\|^2}(y)=y/(1+\tau)$. As $\delta\!\to\!0$,
$\mathrm{prox}_{\tau H_\delta}$ tends to the identity (no regularization).
\end{itemize}
\section{Experiment}
\label{Experiment section}
We conduct experiments to evaluate the effectiveness of our proposed AdamHuberDecay optimizer against established baselines across multiple model scales and downstream tasks.
\subsection{Experimental Setup}
\textbf{Infrastructure and Implementation}. All experiments are conducted on Lambda Cloud's instances, providing 8 NVIDIA A100 80GB SXM4 GPUs per node with NVLink interconnects. Our implementation includes custom CUDA kernels written in CUDA C, FlashAttention-2 integration, and comprehensive NVCC compiler optimizations. We employ 4D parallelization strategies including data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism, coordinated through MPI for multi-node communication.
\textbf{Model Architecture and Scale} We evaluate our optimizer across five transformer architectures of varying scales: GPT-2 124M, 350M, 774M, 1.558B parameters, and GPT-3 125M parameters. All models follow the standard GPT architecture with causal self-attention, layer normalization, and SwiGLU activations. The model configurations exactly match those used in the original GPT-2 and GPT-3 papers to ensure compatibility with established benchmarks.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{learning_rage.png}
\label{fig:subfig1}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{gpt2_355.png}
\label{fig:subfig2}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{gpt2_large.png}
\label{fig:subfig3}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{gpt2_xlarge.png}
\label{fig:subfig4}
\end{subfigure}
\caption{Comparison of numbers of steps / computation to reach the same validation loss.}
\label{fig:speedup}
\end{figure}
\subsection{Pretraining}
\textbf{Dataset and Tokenization:} We pretrain all models on the FineWeb dataset, a high-quality web-scraped corpus containing over 15 trillion tokens. The dataset is tokenized using the GPT-2 BPE tokenizer with a vocabulary size of 50,257 tokens, padded to 50,304 for computational efficiency. Data is stored in optimized binary format with 16-bit token representations and processed through our custom dataloader implementation supporting distributed training across multiple nodes.
\textbf{Training Hyperparameters.} Following established scaling practices, we maintain consistent hyperparameters across model sizes to ensure fair evaluation. Sequence length is set to 1024 tokens for GPT-2 models and 2048 tokens for GPT-3 125M. Learning rates are scaled according to model size, ranging from 6e-4 for smaller models to 2.5e-4 for larger architectures, following a cosine decay schedule with 700 warmup steps for all models. Weight decay is applied selectively: 0.1 for embedding and output layers, and 0.0 for all other parameters. Gradient clipping is enforced at a global norm of 1.0 to ensure training stability. All models are trained with mixed precision using bfloat16 computation and float32 master weights.
%%Comparison of numbers of steps to reach the same validation loss
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{loss_curve_gpt2_125.png}
\label{fig:subfig1}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{loss_curve_gpt2_355.png}
\label{fig:subfig2}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{loss_curve_gpt2_774.png}
\label{fig:subfig3}
\end{subfigure}%
\begin{subfigure}[b]{0.26\textwidth}
\centering
\includegraphics[width=\textwidth]{loss_curve_gpt2_1558.png}
\label{fig:subfig4}
\end{subfigure}
\caption{Validation loss on FineWeb during pretraining}
\label{fig:four_images}
\end{figure}
\subsection{Downstream Evaluation}
We assess model capabilities on a standard suite: \textbf{TruthfulQA} (factuality and resistance to plausible falsehoods), \textbf{Winogrande} (commonsense pronoun resolution), \textbf{ARC Challenge} (difficult scientific question answering), \textbf{HellaSwag} (selection of the most plausible continuation of a context), \textbf{GSM8K} (grade‑school arithmetic and multi‑step problem solving), and \textbf{MMLU} (broad knowledge and reasoning across diverse academic and professional domains). Unless noted otherwise, evaluations use standardized few‑shot protocols.
% \textbf{Evaluation Protocol} For classification and multiple-choice tasks, we perform few-shot evaluation using in-context learning with $k \in \{0, 1, 5\}$ examples. For generation tasks, we use nucleus sampling with p = 0.9 and temperature T = 0.8. All evaluations are conducted using identical prompting strategies and hyperparameters across optimizers. We report mean performance and standard deviation across three random seeds for statistical significance. Model checkpoints are selected based on validation loss rather than downstream performance to avoid selection bias.
\textbf{Evaluation Protocol} Downstream evaluation follows the Eleuther Evaluation Harness framework with standardized few-shot settings. We report accuracy and normalized accuracy metrics where applicable. For HellaSwag, we implement both in-framework evaluation during training and post-training comprehensive evaluation.
\begin{table}
\centering
\caption{Downstream task evaluation of GPT-2 Large 774M pre-trained by different optimizers}
\resizebox{1.0\linewidth}{!}{
\begin{tabular}{l|cccccc|c}
\toprule
\multirow{2}{*}{Optimizer} & HellaSwag & MMLU & ARC Challenge & GSM8k & TruthfulQA & WinoGrande & Average \\
& 10-shot & 5-shot & 25-shot & 5-shot & 0-shot & 5-shot & Score \\ \midrule
AdamW & 57.80 & 25.86 & 30.46 & 0.15 & 35.18 & 57.35 & 34.47 \\
Lion & 58.01 & 25.81 & 30.43 & 0.16 & 35.11 & 57.58 & 34.90 \\
Sophia & 57.83 & 25.78 & 30.37 & 0.14 & 34.98 & 56.95 & 34.52 \\
AdamHD & \textbf{60.11} & \textbf{27.12} & \textbf{33.23} & \textbf{0.83} & \textbf{39.34} & \textbf{61.21} & \textbf{36.97} \\
\bottomrule
\end{tabular}
}
\label{tab:basic}
\end{table}
Experimental results demonstrate that AdamHuberDecay consistently outperforms AdamW, Sophia, and LION across all model scales and downstream tasks, achieving performance improvements of $2.5-4.7\%$ (Table \ref{tab:basic}) while maintaining identical computational budgets and training configurations.
\section{Conclusion}
We introduced \textbf{AdamHuberDecay (AdamHD)}, a decoupled regularization scheme that replaces the quadratic $L_2$ penalty in AdamW with a Huber-style decay acting directly in parameter space. By capping the decay force beyond a data-driven threshold $\delta$, AdamHD preserves the stabilizing effect of $L_2$ on small weights while imposing $\ell_1$-like shrinkage on overgrown coordinates. This yields (i) bounded regularization gradients, (ii) invariance to per-coordinate second-moment rescaling, and (iii) stronger sparsity pressure \cite{han2015deep, frankle2018lottery, guo2019reweighted, ma2020pconv, guo2021algorithm} -- all realized through a closed-form, $O(1)$-overhead step that slots into any Adam-family optimizer.
Empirically, AdamHD improves the efficiency and resilience of language-model pre-training across GPT-2 and GPT-3 scales: it reaches target perplexities $10$--$15\%$ faster in wall-clock time, reduces validation perplexity by up to $4$ points, and delivers consistent gains of $2.5$--$4.7\%$ on a diverse suite of downstream tasks under matched budgets. We also observe visibly sparser weight histograms that translate to $20$--$30\%$ memory savings after straightforward magnitude pruning. These benefits hold without bespoke hyperparameter sweeps beyond the standard decay grids used for AdamW and are robust to gradient outliers and large-batch regimes. On the theory side, our proximal view clarifies AdamHD as a decoupled Huber step and our analysis bounds the expected parameter norm under noisy updates, explaining the observed late-stage stability.
\paragraph{Takeaway.}
Decoupled Huber decay is a simple, principled, and practical drop-in replacement for weight decay in large-scale generative modeling. By directly targeting late-stage over-decay while preserving early-phase smoothing, AdamHD advances the optimizer toolkit for training the next generation of foundational transformers \cite{vaswani2017attention, guo2022zero, mathur2022monopoly, guo2025hamiltonian} .
% \bibliographystyle{abbrvnat}
\bibliographystyle{plainnat}
\bibliography{neurips_2025}
\appendix
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak