Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling
Extracted source text · arXiv 2510.03638
\documentclass{amsart} % For LaTeX2e
% \usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{url}
\usepackage{amsmath,amsfonts,amssymb,amsthm}
% \usepackage{txfonts}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{wrapfig}
\usepackage{longtable}
\usepackage{amsrefs}
\usepackage{indentfirst}
\usepackage{mathrsfs}
\usepackage{xcolor}
\usepackage[margin=1.2in]{geometry}
\usepackage{nicefrac}
\captionsetup[subfigure]{justification=centering}
\renewcommand{\eqref}[1]{\textup{(\ref{#1})}}
\usepackage{tikz}
\usetikzlibrary{shapes, arrows.meta, positioning, matrix}
\usetikzlibrary{backgrounds}
\usetikzlibrary{fit}
\usetikzlibrary{calc,decorations.pathreplacing}
\usepackage{pgfplots}
\pgfplotsset{compat=1.18}
\usepgfplotslibrary{groupplots}
\usepackage{hyperref}
\title[Implicit Models: Expressive Power Scales with Test-Time Compute]{Implicit Models: Expressive Power Scales with Test-Time Compute}
\author{Jialin Liu}
\address{(JL) Department of Statistics and Data Science, University of Central Florida, Orlando, FL 32826.}
\email{jialin.liu@ucf.edu}
\author{Lisang Ding}
\address{(LD) Department of Mathematics, University of California, Los Angeles, CA 90095}
\email{lisangding@ucla.edu}
\author{Stanley Osher}
\address{(SO) Department of Mathematics, University of California, Los Angeles, CA 90095}
\email{sjo@math.ucla.edu}
\author{Wotao Yin}
\address{(WY) Decision Intelligence Lab, DAMO Academy, Alibaba US, Bellevue, WA 98004.}
\email{wotao.yin@alibaba-inc.com}
\date{\today}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\begin{document}
\begin{abstract}
Implicit models, an emerging model class, compute outputs by iterating a single parameter block to a fixed point. This architecture realizes an infinite-depth, weight-tied network that trains with constant memory, significantly reducing memory needs for the same level of performance compared to explicit models. While it is empirically known that these compact models can often match or even exceed the accuracy of larger explicit networks by allocating more test-time compute, the underlying mechanism remains poorly understood.
We study this gap through a nonparametric analysis of expressive power. We provide a strict mathematical characterization, showing that a simple and regular implicit operator can, through iteration, progressively express more complex mappings. We prove that for a broad class of implicit models, this process lets the model's expressive power scale with test-time compute, ultimately matching a much richer function class.
The theory is validated across four domains: image reconstruction, scientific computing, operations research, and LLM reasoning, demonstrating that as test-time iterations increase, the complexity of the learned mapping rises, while the solution quality simultaneously improves and stabilizes.
\end{abstract}
\maketitle
\section{Introduction}
\label{sec:intro}
Many machine-learning tasks can be cast as learning a mapping $\cF$ from input $\vx$ to the desired output $\vy_\ast$, i.e., $\vy_\ast = \cF(\vx)$. An emerging alternative is the \textit{implicit models} (or named deep equilibrium models, also fixed-point models): train an operator $\cG$ whose fixed point matches the target, $\vy_\ast = \cG(\vy_\ast, \vx)$ \cites{bai2019deep,el2021implicit}. At inference time, the fixed point is obtained via a root-finding solver. While advanced algorithms (e.g., Anderson acceleration or Broyden's method) exist, the canonical approach is the Picard iteration:
\begin{equation}
\label{eq:implicit-iterate}
\vy_1 = \cG(\vy_0, \vx), ~~ \vy_2 = \cG(\vy_1, \vx), ~~ \vy_3 = \cG(\vy_2, \vx), ~~ \cdots,
\end{equation}
and expect $\vy_t(\vx) \to \vy_\ast(\vx)= \cF(\vx)$ for all $\vx$. Rather than producing $\vy_\ast$ in a single feedforward pass, implicit models reach the target through gradual equilibrium-seeking updates by reusing the operator $\cG$ over and over.
Here, ``\textit{test-time compute}" refers to the computational budget spent at inference—primarily the number of iterations; increasing this budget increases runtime but not the number of learned parameters.
By tailoring the structure of $\cG$, implicit models have shown strong results across many domains (e.g., imaging \cite{gilton2021deep}, scientific computing \cite{marwah2023deep}, generative modeling \cite{pokle2022deep,geng2023one}, LLM reasoning \cite{geiping2025scaling}, etc.).
Behind these successes, the advantages of implicit models include: \textbf{(i)} they realize an infinite-depth, weight-tied network trainable with constant memory, which yields efficient training \cite{fung2022jfb,geng2021training}; \textbf{(ii)} they allow us to explicitly impose or ``implicitly bake in" domain constraints and structure (e.g., physics, geometry, safety), see \cite{xie2022optimization,gungor2023deq,oshin2024differentiable}; and, \textit{most surprisingly,} \textbf{(iii)} \textit{they can often match or even exceed the accuracy of larger explicit networks by allocating more iterations} \cite{marwah2023deep,wang2024lion,geiping2025scaling,yang2024looped}. Point (i) stems from the weight-tied architecture and avoiding full back-propagation. Point (ii) arises from the inherently implicit nature of many real-world, equation-based constraints. In contrast, the mechanism underlying the surprising effectiveness of (iii) remains less well understood.
We study this through the lens of {expressive power}—the set of input–output maps a model family can represent. We ask two questions. First, as a baseline:
\textbf{(Q1)} \textit{Do implicit models (at least) match the expressive power of explicit ones?} Concretely, for a target map \(\cF:\vx\mapsto \vy_\ast\), does there always exist an implicit operator \(\cG\) such that the iterates of \eqref{eq:implicit-iterate} satisfy $\vy_t(\vx) \to \cF(\vx)$ for all $\vx$?
If yes, a more insightful question follows:
\textbf{(Q2)} \textit{Do implicit models offer an expressive advantage?} In particular, can a relatively \textit{simple} implicit operator $\cG$, through iteration, represent a \textit{complex} explicit map $\cF$? A positive answer to (Q2) would directly explain phenomenon (iii).
To our knowledge, these questions remain largely open. While universality has been touched upon in specific settings \cite{bai2019deep,marwah2023deep} and separation results have demonstrated advantages over explicit models \cite{wu2024separation}, a complete characterization of the representable function class of implicit models (and hence a direct answer to questions (Q1) and (Q2)) is still missing.
Unlike studies focusing on infinite-width limits and kernel connections \cite{gao2022a,feng2023neural,ling2024deep}, our work fills this gap from a \textit{nonparametric, function-space perspective}, establishing that an implicit model's expressive power scales with test-time compute. (See Appendix \ref{app:refs} for broader contextual discussions.)
Specifically:
\begin{itemize}[leftmargin=5mm]
\item \textbf{Expressive boundary.} We identify locally Lipschitz mappings as a natural target class and prove: every such map $\cF$ can be expressed as the fixed point of a ``regular" (simple and well-behaved) $\cG$, and conversely, every such fixed-point map is locally Lipschitz.
\item \textbf{Emergent expressive power.} Our theory, combined with the root-finding solvers' dynamics, yields a new viewpoint on implicit models: the expressive power of a regular implicit operator is not static but \textit{progressively unlocked by iterations} (i.e., scales with test-time compute), and finally matches a much richer function class.
\item \textbf{Validation across domains.} We validate our theory with case studies in a wide range of applications (e.g., image reconstruction, scientific computing, operations research, and LLM reasoning). For a representative problem in each domain, we demonstrating that as test-time iterations increase, the empirical complexity of the iterate $\vy_t(\vx)$ rises while solution quality improves and stabilizes.
\end{itemize}
Note that, while explicit networks are capable of expressing locally Lipschitz target maps \cite{beneventano2021deep} by scaling up the model size, implicit models are able to scale expressivity with test-time iterations and represent increasingly complex functions without adding parameters.
\section{Main Results}
\label{sec:main-results}
We now return to (Q1): given a target map \(\cF\), does there exist an implicit operator \(\cG\) whose fixed-point iteration yields $\vy_t(\vx)\to\cF(\vx)$? A naive construction answers “yes”: define, for \(0<\eta<1\),
\begin{equation}
\label{eq:naive}
\cG(\vy,\vx) := (1-\eta) \vy + \eta \cF(\vx).
\end{equation}
Then the fixed-point iteration reduces to \(\vy_t=(1-\eta)\vy_{t-1}+\eta\,\cF(\vx)\), hence \(\vy_t - \cF(\vx)= (1-\eta) (\vy_{t-1}-\cF(\vx))\). As \(0<\eta<1\), it holds that, for all \(\vx\), \(\vy_t(\vx) - \cF(\vx) \to \vzero \) as \(t\to\infty\).
However, \eqref{eq:naive} is merely a trivial averaging of \(\vy\) and \(\cF(\vx)\); learning such an implicit model is no different from learning \(\cF\) directly. This prompts the natural follow-up: is there any \emph{nontrivial} implicit representation that is able to indicate the expressive benefits of implicit models?
\textbf{An illustrative example.} Let \(\cF(x)=1/x\) on \([-1,1]\setminus\{0\}\). This function is smooth (differentiable to any order) almost everywhere, but blows up near the singular point $x=0$:
\[ |\cF(x)|= \left| \frac{1}{x} \right| \to \infty,\quad \left|\frac{\mathrm{d}\cF}{\mathrm{d}x}\right|=\left|-\frac{1}{x^2}\right| \to \infty, \quad \textup{as }x \to 0.
\]
Neural networks approximating $1/x$ on $[-1,-\delta)\cup(\delta,1]$ typically demands higher network complexity—i.e., increasing depth/width as $\delta \to 0$ to capture the growing steepness near the singularity \cite{telgarsky2017neural}.
If we adopt the naive implicit form \eqref{eq:naive}, \(\cG(y,x)=(1-\eta)y+\eta/x\), nothing is gained: the model still inherits the singular behavior \(|\partial\cG/\partial x|= \eta/x^2\to\infty\).
What would be a nontrivial implicit representation in this setting? Instead of writing $(1/x)$ explicitly, we can regard it as the solution of the equation $xy-1=0$ \textbf{(implicit representation)}. Inspired by this, we apply a fixed-point iteration to $xy-1=0$: $\cG(y,x)=y-\eta(xy-1)$. Using the general scheme in \eqref{eq:implicit-iterate}, we have $y_t = y_{t-1} - \eta (x y_{t-1}-1)$. Subtracting the true solution gives
\[ y_t -\frac{1}{x} = y_{t-1} - \frac{1}{x} - \eta x \left( y_{t-1} - \frac{1}{x} \right) = \left(1-\eta x\right)\left( y_{t-1} - \frac{1}{x} \right)
\]
For any \(0<\eta<1\) and any $x \in (0,1]$, we have $0 < (1-\eta x) < 1$ which implies $y_t \to 1/x$. (For $x<0$, simply flip the stepsize sign, $\eta$ to $-\eta$.) This implicit formulation is much simpler and more elegant: the operator $\cG(y,x)=y-\eta(xy-1)$ has \textit{no singularity} and \textit{no blow-up}.
The example indicates: intuitively, an implicit representation can realize a complicated map with singularities via a much simpler, smoother update operator \(\cG\). Next, we make it precise: we formally define what we mean by “simple” versus “complex,” and characterize—beyond the \(1/x\) example—the class of target functions for which an implicit representation admits such a simple form.
\begin{definition}[Lipschitz continuity]
\label{def:lip}
Let $(\sX,\|\cdot\|)$ and $(\sY,\|\cdot\|)$ be normed spaces, and let $\cQ:\sX\to\sY$.
We say $\cQ$ is \emph{$L$-Lipschitz} (globally Lipschitz) on $\sX$ if there exists $L>0$ such that
\[
\|\cQ(\vx_1)-\cQ(\vx_2)\| \;\le\; L\,\|\vx_1-\vx_2\|\quad\text{for all }\vx_1,\vx_2\in\sX,
\]
and the smallest such $L$ is the \emph{Lipschitz constant} (or \emph{Lipschitz modulus}), denoted as $\textup{Lip}(\cQ)$. If the Lipschitz constant $L < 1$, we say $\cQ$ is a \textit{contraction} with modulus $L$ (or $L$-\textit{contractive}) on $\sX$.
Given $\vx\in\sX$, we say $\cQ$ is \emph{locally Lipschitz at $\vx$} if there exists a neighborhood $\sU$ of $\vx$ on which $\cQ$ is $L_\sU$-Lipschitz continuous for some $L_\sU>0$.
If $\cQ$ is locally Lipschitz at every $\vx\in\sX$, we say $\cQ$ is \emph{locally Lipschitz on $\sX$}.
\end{definition}
Intuitively, Lipschitz continuity limits how quickly a function’s value can change. When a function is differentiable, its Lipschitz modulus can be characterized by the norm of its first derivative via the mean-value theorem. For example, \(\cF(x)=1/x\) is locally Lipschitz on \([-1,1]\setminus\{0\}\) but not globally Lipschitz there, since \(|\mathrm{d}\cF/\mathrm{d}x|=1/x^2\) is unbounded as \(x\to 0\), causing local Lipschitz constants to blow up near the singularity. In contrast, the implicit update \(\cG(y,x)=y-\eta(xy-1)\) has simple partial derivatives \(|\partial\cG/\partial x|=|\eta\,y|\) and \(|\partial\cG/\partial y|=|1-\eta x|\) without singularity.
Locally Lipschitz mappings form a much richer class than globally Lipschitz ones. Typical examples (locally Lipschitz everywhere in their domains but not globally Lipschitz on the whole set) include: $\log x$ in $(0,1]$, $\tan x$ in $\bigl(-\tfrac{\pi}{2},\tfrac{\pi}{2}\bigr)$, $\sqrt{x}$ in $(0,1]$, $\Gamma(x)$ in $\sR \setminus \{0, -1, -2,\cdots\}$, etc.
For this reason, we refer to globally Lipschitz maps as “simple” operators and locally Lipschitz maps (which may exhibit large local slopes near certain inputs) as “complex.” Next, we formally state our main result: identifying a broad family of target functions for which implicit representations provide such simple update operators while expressing complex fixed-point mappings.
\begin{assumption}
\label{asmp:f}
Let $\sX \subset \sR^d$ and $\cF:\sX\to\sR^n$ be locally Lipschitz on $\sX$.
\end{assumption}
We do not assume the domain $\sX$ to be bounded, compact, closed, or connected.
For instance, $\sX=\sR \backslash \{0\}$ excludes the singular point and permits $\cF(x)=1/x$ to blow up at the interior gap $x=0$ while remaining locally Lipschitz on $\sX$.
Another example is $\sX = \bigcup_{k \in \sZ} \bigl(k\pi -\tfrac{\pi}{2}, k\pi + \tfrac{\pi}{2}\bigr)$, where $\cF(x)=\tan x$ remains locally Lipschitz despite blowing up at the singularity points $\{k\pi + \tfrac{\pi}{2}\}_{k \in \sZ}$.
We now formalize what we mean by “simple” update rules—namely, \emph{regular implicit operators}.
\begin{definition}[Regular implicit operator]
\label{def:regular-g}
Let $\sX\subset\sR^d$ be bounded. An operator $\cG:\sR^n\times\sX\to\sR^n$ is \emph{regular} if: (i) For any $\vy \in \sR^n$, the map $\vx\mapsto \cG(\vy,\vx)$ is {globally} Lipschitz (w.r.t. $\vx$) on $\sX$, and the Lipschitz constant grows linearly w.r.t. $\|\vy\|$, and (ii) For each $\vx \in \sX$, there exists $\mu(\vx)\in (0,1)$, the map $\vy \mapsto \cG(\vy,\vx)$ is $\mu(\vx)$-{contractive} on $\sR^n$, and $\mu(\vx)$ is continuous w.r.t. $\vx$.
\end{definition}
% \textbf{Explanation.}~A regular $\cG$ satisfies: (i) Fixing $\vy$, \textit{$\cG(\vy,\cdot)$ is globally Lipschitz in $\vx$}, this makes it a ``simple" operator, and (ii) Fixing $\vx$, \textit{$\cG(\cdot,\vx)$ is contractive in $\vy$}; by Banach’s theorem, this yields a unique fixed point $\vy_\ast(\vx)$ for each \(\vx\) and guarantees that iterates of \eqref{eq:implicit-iterate} converge to it: $\vy_t(\vx) \to \vy_\ast(\vx)$. \textbf{An example} of such a regular $\cG$ is the aforementioned $\cG(y,x)=y-\eta(xy-1)$ on $x \in (0,1]$ with $0<\eta<1$.
% With this definition, we present our main results.
A regular $\cG$ satisfies: (i) Fixing $\vy$, \textit{$\cG(\vy,\cdot)$ is globally Lipschitz in $\vx$}, this makes it a ``simple" operator, and (ii) Fixing $\vx$, \textit{$\cG(\cdot,\vx)$ is contractive in $\vy$}; by Banach’s theorem, this yields a unique fixed point $\vy_\ast(\vx)$ and guarantees that iterates of \eqref{eq:implicit-iterate} converge to it: $\vy_t(\vx) \to \vy_\ast(\vx)$. An example of regular $\cG$ is the aforementioned $\cG(y,x)=y-\eta(xy-1)$ on $x \in (0,1]$ with $0<\eta<1$. Moreover, regularity does not require joint Lipschitz properties.
With this definition, we present our main results.
\begin{theorem}[Sufficiency]
\label{thm:lip}
Under Assumption~\ref{asmp:f}, for any $\cF$ there exists a \emph{regular} implicit operator $\cG:\sR^n\times\sX\to\sR^n$ whose fixed-point map reproduces $\cF$: $\mathrm{Fix}\big(\cG(\cdot,\vx)\big)\;=\;\cF(\vx)$ for all $\vx\in\sX$.
\end{theorem}
\begin{theorem}[Necessity]
\label{thm:lip-2}
Let $\sX\subset\sR^d$ and let $\cG:\sR^n\times\sX\to\sR^n$ be \emph{regular}. For every $\vx\in\sX$, $\cG(\cdot,\vx)$ has a unique fixed point $\vy_\ast$, and the fixed-point map $\vx\mapsto\vy_\ast(\vx)$ is locally Lipschitz on $\sX$.
\end{theorem}
Proofs are deferred to Appendix \ref{sec:app-main}. Theorem~\ref{thm:lip} provides an affirmative answer to (Q1) and (Q2) posed in the introduction. It proves that for any locally Lipschitz target $\cF$ on a bounded domain, there exists a \textit{regular} implicit operator $\cG$, whose iterations converge to the target $\vy_t(\vx)\to\cF(\vx)$ for all $\vx$. This demonstrates that the expressive power of implicit models \textbf{not only matches} that of explicit models \textbf{but also provides a distinct expressive benefit}: \textit{a relatively simple (regular) implicit representation can yield a complex fixed-point mapping}. Complementarily, Theorem~\ref{thm:lip-2} shows the boundary is tight: fixed points induced by any regular $\cG$ are \emph{necessarily} locally Lipschitz. Together, the two results give an exact expressivity characterization for regular implicit models.
% \textbf{What does our theory imply?}
% Take any locally Lipschitz target $\cF$ (e.g., the curve in Fig.~\ref{fig:toy-diagram}). Our results guarantee the existence of a \emph{regular} implicit operator $\cG$ such that the iteration $\vy_t=\cG(\vy_{t-1},\vx)$ with $\vy_0=\vzero$ converges: $\vy_t(\vx)\to\cF(\vx)$. Consider the first iterate:
% \[\vy_1(\vx)=\cG(\vzero,\vx) \quad \implies \quad \textup{Lip}(\vy_1) = \sup_{\vx,\vx^\prime} \frac{\|\cG(\vzero,\vx)-\cG(\vzero,\vx^\prime)\|}{\|\vx-\vx^\prime\|} = \textup{Lip}(\cG(\vzero,\cdot)).
% \]
% Thus, \textit{after a single iteration, $\vy_1(\cdot)$ is exactly as smooth as $\cG(\vzero,\cdot)$}. By Definition~\ref{def:regular-g}, $\cG(\vzero,\cdot)$ is {globally Lipschitz in $\vx$} with a \emph{uniform} constant, hence the map $\vy_1(\cdot)$ inherits the same property.
% With more iterations, $\vy_t$ approaches $\cF$, so the Lipschitz constant of $\vy_t(\cdot)$ approaches that of $\cF(\cdot)$:
% \[
% \lim_{t\to\infty}\frac{\|\vy_t(\vx)-\vy_t(\vx^\prime)\|}{\|\vx-\vx^\prime\|} = \frac{\|\cF(\vx)-\cF(\vx^\prime)\|}{\|\vx-\vx^\prime\|}.
% \]
% If $\cF$ has singularities (where local slopes are large or even unbounded), then the effective Lipschitz constants of $\vy_t(\cdot)$ necessarily grow with $t$ to match that complexity—illustrated in Fig.~\ref{fig:toy-diagram}. In short: \textit{a regular operator $\cG$ gains expressive power through iteration (“test-time compute”)}, ultimately matching a richer function class that can include singularities and unbounded Lipschitz behavior.
\textbf{What does our theory imply?}
Take a locally Lipschitz target $\cF$ (e.g., the curve in Fig.~\ref{fig:toy-diagram}). Our results guarantee the existence of a \emph{regular} implicit operator $\cG$ such that the iteration $\vy_t=\cG(\vy_{t-1},\vx)$ with $\vy_0=\vzero$ converges: $\vy_t(\vx)\to\cF(\vx)$. Consider the first iterate and its Lipschitz property:
\[\vy_1(\vx)=\cG(\vzero,\vx) \quad \implies \quad \textup{Lip}(\vy_1) = \sup_{\vx,\vx^\prime} \frac{\|\cG(\vzero,\vx)-\cG(\vzero,\vx^\prime)\|}{\|\vx-\vx^\prime\|} = \textup{Lip}(\cG(\vzero,\cdot)).
\]
Because a regular operator $\mathcal{G}$ is globally Lipschitz by definition, $\vy_1(\cdot)$ is restricted to representing ``simple," globally smooth mappings. However, as iterations progress, $\vy_t$ converges toward $\mathcal{F}$. If the target $\mathcal{F}$ features singularities (regions where local slopes become large or unbounded), the effective Lipschitz constant of the iterate $\mathbf{y}_t(\cdot)$ naturally grows with $t$ to match that complexity:
\[
\lim_{t\to\infty}\frac{\|\vy_t(\vx)-\vy_t(\vx^\prime)\|}{\|\vx-\vx^\prime\|} = \frac{\|\cF(\vx)-\cF(\vx^\prime)\|}{\|\vx-\vx^\prime\|}.
\]
This dynamic highlights a fundamental distinction: While explicit networks scale their \textit{model size} to approximate locally Lipschitz targets \cite{beneventano2021deep}, implicit models can scale expressivity with \textit{test-time compute}. Since our theory guarantees a regular operator can define a complex equilibrium, iterating this single operator realizes this complexity \textit{without adding parameters}.
\begin{figure*}[t] % {0.4\textwidth}
\centering
\resizebox{\linewidth}{!}{\input{toy-diagram.tikz}}
\caption{(Conceptual diagram) A simple implicit update expresses a complex map via iteration.}
\label{fig:toy-diagram}
\end{figure*}
\textbf{Generalization.} Someone may ask: {does a large Lipschitz constant of the fixed-point map $\vy_\ast(\vx)$ imply sensitivity or poor generalization} (cf. \cite{pabbaraju2021estimating})? Our view is that this sensitivity is \emph{inherent to the target} $\cF$, not to the implicit representation—any faithful model, explicit or implicit, must track $\cF$’s sharp variations. Our case studies in Section \ref{sec:cases} confirm this: the target $\cF$ in many tasks is indeed steep somewhere and the effective Lipschitz grows as accuracy improves. Crucially, the implicit formulation can realize such targets with a \textit{simple} operator $\cG$, which regularizes training and supports good generalization in practice.
\textbf{Insights for practitioners.} A substantial line of work (e.g., \cite{el2021implicit,winston2020monotone,jafarpour2021robust,revay2020lipschitz,havens2023exploiting}) enforces a global Lipschitz bound on the fixed-point map $\vy_\ast(\vx)$. Typically, the model is parameterized as $\cG(\vy,\vx)=\sigma(\mA\vy+\mB\vx+\vb)$, and by imposing specific algebraic structure on $\mA$ and $\mB$,
one ensures that $\vy_\ast(\vx)$ is globally Lipschitz in $\vx$. While this indeed improves robustness, our theory shows it \textit{constrains expressivity and undercuts the unique advantage of implicit models}. Our recommendation is different: rather than imposing uniform Lipschitz constraints, incorporate case-by-case \textit{domain-specific knowledge, priors, or constraints} (as illustrated in our case studies Sec. \ref{sec:cases}). This method provides effective regularization, leading to robustness and strong test performance while unlocking the full power of implicit models—representing complex maps with relatively simple operators.
\section{Case Studies}
\label{sec:cases}
In this section, we present four case studies. For the first three tasks, we (i) verify that the target satisfies Assumption~\ref{asmp:f}; (ii) specify a domain-informed architecture for $\cG$; (iii) confirm empirically that, under standard training without explicitly enforcing $\cG$ to be regular, \textit{the learned operators $\cG$ exhibit these properties}---i.e., $\cG$ is Lipschitz in $\vx$ and iterates $\vy_t$ converge (see Appendix~\ref{app:training} for training strategies and discussions regarding regularity guarantees); and (iv) demonstrate that expressive power scales with test-time iterations. Finally, we extend this analysis to LLM reasoning to validate our predictions in a domain where strict mathematical definitions are less applicable.
\subsection{Case Study 1: Image Reconstruction (Inverse problems)}
Inverse problems in imaging seek to recover an image \(\vy_\ast\in\sR^n\) from partial, noisy measurements $\vx = \mA \vy_\ast + \vn \in \sR^d ~ (d<n)$, where \(\mA\) is a known linear operator and \(\vn\) is noise. A common prior is that \(\vy_\ast\) lies near a smooth data manifold \(\sM\subset\sR^n\). To recover \(\vy_\ast\), a standard estimator solves
\begin{equation}
\label{eq:mapping-1a}
\min_{\vy\in\sR^n}\;\frac{1}{2}\|\vx-\mA\vy\|^2 \;+\; \frac{\alpha}{2}\,\mathrm{dist}^2(\vy,\sM),
\end{equation}
or, equivalently, a variable–splitting surrogate
\begin{equation}
\label{eq:mapping-1b}
\min_{\vy,\vz\in\sR^n}\;\frac{1}{2}\|\vx-\mA\vy\|^2 \;+\; \frac{\alpha}{2}\,\mathrm{dist}^2(\vz,\sM) \;+\; \frac{\beta}{2}\|\vy-\vz\|^2.
\end{equation}
Next we will show that, under mild assumptions, both \eqref{eq:mapping-1a} and \eqref{eq:mapping-1b} admit a unique minimizer for each \(\vx\) in a bounded set, and the solution map \(\vx\mapsto\hat{\vy}(\vx)\) is \emph{locally Lipschitz}. Hence the reconstruction target falls within Assumption~\ref{asmp:f} and is covered by our expressivity results in Section~\ref{sec:main-results}.
\begin{assumption}
\label{assume:M}
Let $\sM\subset\sR^n$ be a compact, $\cC^2$, embedded (possibly nonconvex) submanifold with positive reach $\tau>0$. Assume the forward operator $\mA:\sR^n\!\to\!\sR^d$ is $(\mu,L)$–bi-Lipschitz when restricted to $\sM$ and let $\sigma_{\textup{max}}$ denote the maximal singular value of $\mA$.
\end{assumption}
These assumptions are modest: they are standard in prior work and supported by existing theory. Formal definitions (reach and bi-Lipschitz continuity) and relevant literature appear in Appendix~\ref{sec:app-inv}.
\begin{definition}
\label{def:X-inv-prob}
Define the admissible set of observations $\vx$ for \eqref{eq:mapping-1a} and \eqref{eq:mapping-1b}:
\[ \sX := \left\{ \vx: \vx = \mA \vy_\ast + \vn, \quad \textup{for some }\vy_\ast \in \sM, ~~\|\vn\| < \frac{1}{80}\frac{\mu^5}{\sigma^2_{\textup{max}}L^2} \tau. \right\}
\]
\end{definition}
\begin{theorem}
\label{thm:inverse-lip-1a}
Under Assumption \ref{assume:M}, there exists $\alpha > 0$ for all $\vx \in \sX$ such that the minimization problem \eqref{eq:mapping-1a} yields a unique minimizer $\hat{\vy}$. Let $\cF_{1a}$: $\vx \mapsto \hat{\vy}$ denote the associated solution map from input $\vx$ to the recovery $\hat{\vy}$. Then $\cF_{1a}$ is locally Lipschitz continuous on $\sX$.
\end{theorem}
\begin{theorem}
\label{thm:inverse-lip-1b}
Under Assumption \ref{assume:M}, there exist $\alpha,\beta > 0$ for all $\vx \in \sX$ such that the minimization problem \eqref{eq:mapping-1b} yields a unique minimizer $(\hat{\vy},\hat{\vz})$. Let $\cF_{1b}$: $\vx \mapsto \hat{\vy}$ denote the associated solution map from input $\vx$ to the recovery $\hat{\vy}$. Then $\cF_{1b}$ is locally Lipschitz continuous on $\sX$.
\end{theorem}
\begin{corollary}
\label{cor:inverse-prob}
There must be a regular implicit operator $\cG(\vy,\vx)$ such that \(\textup{Fix}(\cG(\cdot,\vx)) = \cF_{1a}(\vx)\) for all $\vx\in\sX$. The same conclusion holds for $\cF_{1b}(\vx)$.
\end{corollary}
Proofs of the theorems are deferred to Appendix \ref{sec:app-inv}, and Corollary~\ref{cor:inverse-prob} follows immediately from Theorems~\ref{thm:lip}, \ref{thm:inverse-lip-1a}, and \ref{thm:inverse-lip-1b}. This corollary guarantees the existence of regular implicit models $\cG$ for image reconstruction. Next, we present how to implement $\cG$ in this context.
\textbf{Problem-specific $\cG$.}
We adopt \emph{algorithm-inspired} designs that mirror classical solvers for \eqref{eq:mapping-1a} and \eqref{eq:mapping-1b}.
% (e.g., PGD- and HQS-style modules).
Parameterizing these iterative solvers gives {problem-tailored implicit models}. In particular,
\begin{itemize}[leftmargin=5mm]
\item {Option I (PGD-style).}
To solve \eqref{eq:mapping-1a}, if \(\sM\) were known, one would use proximal gradient descent (PGD): $\vy_{t+1} = \textup{prox}_{\sigma}\!(\vy_t - \gamma\,\mA^\top(\mA\vy_t-\vx))$, with parameters \(\sigma,\gamma>0\), where \(\textup{prox}_{\sigma}\) is the proximal map of \(({\sigma}/{2}) \mathrm{dist}^2(\vy,\sM)\) (see Appendix~\ref{sec:app-prox}).
In practice, we replace \(\textup{prox}_{\sigma}\) by a learnable neural network denoiser \(\cH_{\theta,\sigma}\) (parameters $\theta$ and noise level input $\sigma$) and obtain
\begin{equation}
\label{eq:pgd-deq}
\cG_{\Theta}(\vy,\vx)
= \cH_{\theta,\sigma}\!\Big(\vy - \gamma\,\mA^\top(\mA\vy-\vx)\Big),
\qquad
\Theta=\{\theta,\sigma,\gamma\}.
\end{equation}
\item {Option II (HQS-style).}
For \eqref{eq:mapping-1b}, a standard solver is half–quadratic splitting (HQS, see Appendix~\ref{sec:app-pnp}).
Similar to Option~I, we replace the proximal map by a learned module and obtain
\begin{equation}
\label{eq:hqs-deq}
\cG_{\Theta}(\vy,\vx)
\;=\;
\cH_{\theta,\sigma}\!\left(\,\big(\mA^\top\mA+\beta\mI\big)^{-1}\!\big(\mA^\top\vx+\beta\,\vy\big)\right),
\qquad
\Theta=\{\theta,\sigma,\beta\}.
\end{equation}
\end{itemize}
Here we follow the long-standing “plug-in denoiser” idea from Plug-and-Play (PnP) methods \cite{venkatakrishnan2013plug}, which replaces a proximal operator with an off-the-shelf denoiser inside an iterative solver (see brief bibliography in Appendix \ref{sec:app-pnp}). Unlike PnP, one can also train the \emph{entire} $\cG_\Theta$ as an implicit model, in both PGD-style \cite{gilton2021deep,winston2020monotone,zou2023deep,yu2024msdc,daniele2025deep,shenoy2025recovering} and HQS-style \cite{gkillas2023optimization} formulations. We adopt the latter.
\textbf{Questions.} Given the parameterizations in \eqref{eq:pgd-deq} and \eqref{eq:hqs-deq}, we examine: (i) are these $\cG_{\Theta}$ operators Lipschitz with respect to $\vx$; and (ii) do they, as our theory predicts, realize progressively more complex input–output mappings over iterations despite having simple per-iteration operators?
\textbf{Experiment settings.} We study image deblurring, $\vx = \mA(\vy_\ast) + \vn$, where $\mA$ is a motion-blur operator and $\vn$ is additive Gaussian noise. Using BSDS500 \cite{MartinFTM01}, we construct 200 training, 100 validation, and 200 test pairs $(\vx,\vy_\ast)$, yielding datasets $\sD_{\textup{inv,train}}$, $\sD_{\textup{inv,val}}$, and $\sD_{\textup{inv,test}}$. Implementation details (data preprocessing, model choices, and training) are in Appendix~\ref{sec:app-exp-inv}.
For evaluation, we analyze 100 iterations of the learned dynamics, $\vy_{t+1}(\vx)=\cG_{\Theta}(\vy_t(\vx),\vx), 0 \leq t \leq 99$ and $\vy_0 = \vzero$, on the test set $\sD_{\textup{inv,test}}=\{(\vx_i,\vy^\ast_{i})\}_{i=1}^{200}$. For each $i$, we create 5 perturbed ground truths $\vy^\ast_{i,j}, 1 \leq j \leq 5$, and for each $\vy^\ast_{i,j}$, we apply $\mA$, add noise, and then obtain $\vx_{i,j}$. The perturbed pairs $\{(\vx_{i,j},\vy^\ast_{i,j})\}_{i,j}$ form the perturbed dataset $\sD^\prime_{\textup{inv,test}}$. Details appear in Appendix \ref{sec:app-exp-inv}. We track two metrics, including an empirical Lipschitz estimate and reconstruction quality in PSNR (i.e., Peak Signal-to-Noise Ratio, higher PSNR means more accurate reconstruction, see appendix):
\[L_t := \max_{1 \leq i \leq 200}\max_{1 \leq j \leq 5} \frac{\| \vy_t(\vx_{i}) - \vy_t(\vx_{i,j}) \|}{ \|\vx_{i} - \vx_{i,j}\| }, \quad \textup{and} \quad P_t(i,j) := \textup{PSNR}(\vy_t(\vx_{i,j}), \vy^\ast_{i,j}),
\]
for $1 \leq i \leq 200, 0 \leq j \leq 5$,
where $j=0$ means the original (unperturbed) sample, $\vx_{i,0}:=\vx_i,\vy^\ast_{i,0}:=\vy^\ast_i$.
Here, $L_t$ estimates how complex the t-th iterate map $\vy_t(\cdot)$ is, while $P_t$ measures the reconstruction quality on \textit{both} the original dataset $\sD_{\textup{inv,test}}$ and the perturbed set $\sD^\prime_{\textup{inv,test}}$.
\textbf{Experiment results.}
\textbf{\underline{\textit{(i)}}} Results in Figure~\ref{fig:inv-theory} support our theory.
Figure~\ref{fig:inv-lip} plots \(L_t\) versus \(t\), while Figure~\ref{fig:inv-psnr} reports the mean \(\pm\) std of \(\{P_t(i,j)\}_{i,j}\) versus \(t\).
At \(t=1\), the mapping \(\vy_1(\vx)=\cG_{\Theta}(\vzero,\vx)\) reflects a \textit{single application of \(\cG_\Theta\) and exhibits low Lipschitz constant}: \(L_1=0.140\) for PGD and \(L_1=0.436\) for HQS.
As \(t\) increases, \(\vy_t\) approaches the fixed point and \(L_t\) grows substantially, saturating around \(\approx 5.0\) for both models (Figure~\ref{fig:inv-lip}).
Meanwhile, the PSNR rises and stabilizes, indicating that \(\vy_t(\vx)\) converges toward the ground truth (Figure~\ref{fig:inv-psnr}).
Thus, the increase in \(L_t\) does not reflect divergence or instability; rather, \textit{it captures the greater complexity of the underlying target mapping \(\vx \mapsto \vy_\ast\), which is progressively expressed through iteration}.
\textbf{\underline{\textit{(ii)}}} We also provide a comparison (both visually and
quantitatively) to an explicit model in Figure \ref{fig:inv-visualize}.
This baseline uses the identical DRUnet and is trained on the deblurring dataset with an end-to-end MSE loss.
A visual inspection reveals that implicit models, particularly implicit HQS \eqref{eq:hqs-deq}, produce sharper images with better-recovered textures and fewer artifacts than the explicit baseline. This perceptual advantage is corroborated by the quantitative metrics, where the DEQ-HQS model achieves a significant PSNR gain of over 2dB on average across the entire test set.
\textbf{\underline{\textit{(iii)}}} Additional experiments showing a small implicit model outperforming larger explicit ones appear in Appendix~\ref{sec:app-exp-inv}.
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{lip_plot_inv.pdf}
\caption{Empirical Lipschitz \(L_t\) of \(\vy_t(\cdot)\) vs.\ iteration. \(L_t\) starts small at \(t{=}1\) and grows to a plateau (\(\sim\!5\)), indicating increasing expressivity of \(\vy_t(\cdot)\).}
\label{fig:inv-lip}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{psnr_plot_inv.pdf}
\caption{Reconstruction quality \(P_t\) (mean \(\pm\) std over the original and perturbed test samples) increases and stabilizes: \(\vy_t(\vx)\) converges toward the truth.}
\label{fig:inv-psnr}
\end{subfigure}
\caption{Validation on image deblurring. Iterating a simple operator \(\cG_{\Theta}\) produces a complex fixed-point mapping: Lipschitz (a) grows, while accuracy (b) improves and stabilizes.}
\label{fig:inv-theory}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{id_3_truth.png}
\centering
Ground Truth \\ - \\ -
\label{fig:inv-truth}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{id_3_blured.png}
\centering
Observation \\ - \\ -
\label{fig:inv-blur}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{id_3_recon_unet.png}
\centering
Explicit:28.49dB \\ 26.97 $\pm$ 3.54 dB \\ Params: 32.64 M
\label{fig:inv-unet}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{id_3_recon_PGD.png}
\centering
Imp-PGD:30.03dB \\ 28.20 $\pm$ 3.70 dB \\ Params: 32.64 M
\label{fig:inv-pgd}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{id_3_recon_HQS.png}
\centering
Imp-HQS:31.53dB \\ 29.18 $\pm$ 3.66 dB \\ Params: 32.64 M
\label{fig:inv-hqs}
\end{subfigure}
\caption{Visual results for deblurring. The top PSNR values (28.49, 30.03, or 31.53 dB) correspond to the single visualized image; the second line shows the average ($\pm$ std) over all test samples.}
\label{fig:inv-visualize}
\end{figure}
\subsection{Case Study 2: Scientific Computing}
\label{sec:ns}
The Navier--Stokes (NS) equations are foundational to computational fluid dynamics. We focus on the 2D steady-state incompressible case on a periodic domain $\Omega:=[0,2\pi]^2$:
\begin{equation}
\label{eq:ns-velocity}
(u \cdot \nabla) u + \nabla p = \nu \Delta u + f, \quad \nabla \cdot u = 0 \quad \textup{on $\Omega$}
\end{equation}
where $u:\Omega \to \sR^2$ is the velocity field, $p:\Omega \to \sR$ is the pressure, $\nu > 0$ is the viscosity, $f:\Omega \to \sR^2$ is the external force. Solving NS equations refers to determining $u$ given $f$.
Although global existence/smoothness of the solution given general forcings is famously open, classical results guarantee well-posedness under suitable conditions on $f$.
\begin{theorem}[\cite{temam1995navier}]
\label{thm:ns-existence}
There exists a constant $c>0$ depending only on $\Omega$ such that, if $\|f\|_{L^2(\Omega)} \le c\,\nu^2$, then \eqref{eq:ns-velocity} admits a unique solution $u_\ast(f)$.
Let $\sH$ denote the space of admissible forcings\footnote{Details regarding the function spaces are provided in Appendix \ref{sec:app-ns}.}, and set $\sB_\nu:=\{f\in\sH:\ \|f\|_{L^2(\Omega)}\le c \nu^2\}$.
Then there exists a subset $\sH_\nu\subset\sB_\nu$ that is dense in $\sB_\nu$, on which the solution map $f\mapsto u_\ast(f)$ is locally Lipschitz.
\end{theorem}
\textbf{Vorticity form.} Let $\omega := \nabla \times u$ (and hence $\omega_\ast := \nabla \times u_\ast$). Under periodic boundary and zero-mean conditions, one can recover the velocity $u$ from vorticity $\omega$ by solving a Poisson equation \cite{majda2001vorticity}. We hence focus on the solution map in vorticity: $f \mapsto \omega_\ast$.
While Theorem \ref{thm:ns-existence} gives a local Lipschitz result in function spaces, our expressivity results (Section \ref{sec:main-results}) are stated for finite-dimensional spaces. To bridge this gap, we discretize the NS equations.
\textbf{Discretization.}
Partition $\Omega$ into $N_h$ cells $\Omega_h := \{C_i\}_{i=1}^{N_h}$ and define the cell–average restriction $\cR_h (f)|_{C} := \frac{1}{|C|}\int_{C} f(\xi) \mathrm{d}\xi$ (similarly for $\omega$). We work with the discrete forcings and vorticities:
\[\vx := \cR_h (f) \in \sR^{N_h \times 2}, \quad \vy:=\cR_h (\omega) \in \sR^{N_h}
\]
and aim to learn $\vx \mapsto \vy_\ast$ where $\vy_\ast := \cR_h(\omega_\ast)$ is the discrete solution in vorticity form. Back to the continuum setting, let the lifting operator $\cE_h$ be the piecewise–constant reconstruction $\cE_h(\vx):= \sum_{C \in \Omega_h} x_C \vone_{C}$, and let $\cP$ be the orthogonal projection onto divergence–free, zero–mean fields.
\begin{corollary}
\label{coro:ns-discrete}
$\cF_{2}:\vx \mapsto \vy_\ast$ is locally Lipschitz on $\sX_{\nu,h}:=\{\vx \in \sR^{N_h \times 2}: \cP(\cE_{h}(\vx)) \in \sH_{\nu}\}$ and there exists a \textit{regular} implicit operator $\cG(\vy,\vx)$ satisfying $\textup{Fix}(\cG(\cdot,\vx)) = \cF_{2}(\vx)$ on $\sX_{\nu,h}$.
\end{corollary}
The corollary instantiates our expressivity theory for steady-state NS, guaranteeing the existence of a \emph{regular} implicit model $\cG$. As in the image–reconstruction case, we now (i) choose a problem-specific parameterization of $\cG$ and (ii) verify our theory numerically on this architecture.
\textbf{Problem-tailored parameterization.}
We use \cite{marwah2023deep} as our code base. In particular,
\begin{equation}
\label{eq:imp-fno}
\vz_\ast = \cG_{\Theta}\!\big(\vz_\ast,\ \cQ_{\Phi}(\vx)\big),
\qquad
\vy_\ast = \cQ_{\Psi}(\vz_\ast).
\end{equation}
The core $\cG_\Theta$ is implemented as a Fourier Neural Operator (FNO) \cite{li2021fourier}, and both the encoder $\cQ_{\Phi}$ and decoder $\cQ_{\Psi}$ use pointwise MLPs\footnote{Introducing additional encoder and decoder is common in practice. Compared to the vanilla formulation $\vy_\ast=\cG(\vy_\ast,\vx)$, it does not change our expressivity results in Section \ref{sec:main-results}. Details appear in Appendix \ref{sec:app-general}.}. Details appear in Appendix \ref{sec:app-ns-exp}.
There is a growing literature on AI for scientific computing (e.g., \cites{raissi2019pinn,lu2021deeponet,li2021fourier,rahman2022uno,kovachki2023neural,li2023geofno}; see \cites{karniadakis2021piml,kovachki2024operatorreview} for surveys). We adopt \eqref{eq:imp-fno} because it learns solution \emph{operators} rather than per-instance solutions, yielding discretization/mesh-resolution invariance, fast global mixing via Fourier layers, and strong parameter efficiency.
\textbf{Experiments.} We use the dataset of \cite{marwah2023deep} with viscosity \(\nu=0.01\), which provides \(4500\) training pairs and \(500\) test pairs \((\vx,\vy^\ast)\), where \(\vx\) is the discretized force and \(\vy^\ast\) is the corresponding vorticity; we denote these sets by \(\sD_{\textup{pde,train}}\) and \(\sD_{\textup{pde,test}}\). Details are given in Appendix~\ref{sec:app-ns-exp}.
We test iteration-wise behavior for \(50\) steps starting from \(\vz_{0}=\vzero\): $\vz_{t+1}=\cG_{\Theta}\big(\vz_t,\cQ_{\Phi}(\vx)\big)$ for $0\le t\le 49$, and $\vy_t(\vx)=\cQ_{\Psi}(\vz_t)$.
Analogous to the inverse-problem study, we augment the test set with perturbations.
For each \((\vx_i,\vy_i^\ast)\in\sD_{\textup{pde,test}}\), we construct \(15\) perturbed vorticities \(\{\vy^\ast_{i,j}\}_{j=1}^{15}\); we then compute compatible forces \(\{\vx_{i,j}\}_{j=1}^{15}\) by evaluating the NS operator (see Appendix~\ref{sec:app-ns-exp} for details).
The perturbed test set is $\sD'_{\textup{pde,test}}=\{(\vx_{i,j},\vy^\ast_{i,j}) : 1\le i\le 500,\ 1\le j\le 15\}$.
Across iterations we report an empirical Lipschitz estimate $L_t$ and relative reconstruction error $E_t$:
\[L_t := \max_{1 \leq i \leq 500} \max_{1 \leq j \leq 15} \frac{\| \vy_t(\vx_{i}) - \vy_t(\vx_{i,j}) \|}{ \|\vx_{i} - \vx_{i,j}\| }, \quad \textup{and}\quad E_t(i,j) := \frac{\|\vy_t(\vx_{i,j})-\vy^\ast_{i,j}\|}{\|\vy^\ast_{i,j}\|+\epsilon},\]
for $1 \leq i \leq 500, 0 \leq j \leq 15$,
where $j=0$ means the original (unperturbed) sample, $\vx_{i,0}:=\vx_i,\vy^\ast_{i,0}:=\vy^\ast_i$. Therefore, \(E_t\) evaluates accuracy on both \(\sD_{\textup{pde,test}}\) and \(\sD'_{\textup{pde,test}}\).
The results in Figure~\ref{fig:ns-theory} align with our theory.
At \(t=1\), the mapping \(\vy_1(\vx)\) reflects a single application of \(\cG_{\Theta}\) and exhibits low Lipschitz constant: \(L_1=23.1\).
As iterations proceed toward the fixed point, the complexity grows markedly: \(L_t\) increases to \(\approx 367\) by \(t=50\) (Figure~\ref{fig:ns-lip}).
Meanwhile, the relative error \(E_t\) decreases monotonically and stabilizes at \(0.078 \pm 0.028\) (Figure~\ref{fig:ns-err}), indicating convergence to a good approximation of \(\vy_\ast\).
Thus, \textit{the learned operator $\cG_\Theta$ is simple (Lipschitz in $\vx$), while additional test-time iterations let $\vy_t$ realize progressively more complex mappings.}
In addition, a comparison with an explicit baseline (vanilla FNO) in Figure~\ref{fig:ns-visualize} shows the implicit model produces more accurate solutions, both visually and quantitatively.
Additional experiments showing a small implicit model outperforming larger explicit ones appear in Appendix~\ref{sec:app-ns-exp}.
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{lip_plot_ns.pdf}
\caption{Empirical Lipschitz \(L_t\) of the \(t\)-step map \(\vy_t(\cdot)\) vs.\ iteration. \(L_t\) starts small at \(t{=}1\) (\(23.1\)) and grows substantially, reaching \(\sim 367\) by \(t{=}50\).}
\label{fig:ns-lip}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{err_plot_ns.pdf}
\caption{Relative error \(E_t\) (mean \(\pm\) std over original and perturbed inputs) vs.\ iteration. \(E_t\) decreases to \(0.078 \pm 0.028\)---$\vy_t$ converges towards \(\vy_\ast\).}
\label{fig:ns-err}
\end{subfigure}
\caption{Validation on the steady Navier--Stokes task. Iterating a simple operator \(\cG_{\Theta}\) yields a complex fixed-point mapping: Lipschitz constant (a) increases, while error (b) decreases.}
\label{fig:ns-theory}
\end{figure}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{deq_sample_3_ground_truth.png}
\centering
Ground Truth $\vy_\ast$ \\ -
\label{fig:ns-truth}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{non-wt-no-inj_sample_3_prediction.png}
\centering
Prediction by \\ FNO
\label{fig:ns-fno}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{non-wt-no-inj_sample_3_difference_shared_scale.png}
\centering
Error: 0.260 \\ 0.179 $\pm$ 0.050 \\
\label{fig:ns-f2}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{deq_sample_3_prediction.png}
\centering
Prediction by \\ implicit FNO
\label{fig:ns-deq}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{deq_sample_3_difference_shared_scale.png}
\centering
Error: 0.112 \\ 0.078 $\pm$ 0.028
\label{fig:ns-f1}
\end{subfigure}
\caption{Visual results for NS equations. The top value (0.260 and 0.112) means the relative error (between the prediction and the ground truth) on the single visualized sample; the second line shows the average relative error ($\pm$ std) over all test samples. Both models have 2.376 M parameters.}
\label{fig:ns-visualize}
\end{figure}
\subsection{Case Study 3: Operations Research}
Linear programming (LP) is fundamental to operations research, of which a general form is given by
\begin{equation}
\label{eq:lp}
\min_{\vy \in \sR^n} \vc^\top \vy, \quad \textup{s.t. } \mA \vy \circ \vb, ~~ \vl \leq \vy \leq \vu.
\end{equation}
Here, $\mA \in \sR^{m \times n}, \vb \in \sR^{m}, \vc \in \sR^n, \vl \in \sR^n, \vu \in \sR^n$, and $\circ\in\{=,\le\}^{m}$ denotes component-wise relations, i.e., each $\circ_i\in\{=,\le\}$ specifies whether $(\mA\vy)_i$ equals or is bounded above by $b_i$.
Let $\vx := (\mA,\vb,\vc,\circ,\vl,\vu)$ as the input that describes the LP in \eqref{eq:lp}. To define the solution mapping $\cF_3$ that maps $\vx$ to the solution of LP, we require feasibility and boundedness (which ensure an optimal solution \cite{bertsimas1997introduction}). Accordingly, let
\[ \sX:= \{(\mA,\vb,\vc,\circ,\vl,\vu): \textup{The resulting LP is feasible and bounded}\}
\]
Within $\sX$, there are some LPs where the solution mapping is not single-valued or not continuous. By excluding these LPs, it forms a subset $\sX_{\mathrm{sub}} \subset \sX$ on which $\cF_3$ is single-valued and locally Lipschitz. The strict definition of $\sX_{\mathrm{sub}}$ and the proof of Theorem \ref{thm:lp} are provided in Appendix \ref{sec:app-lp}.
\begin{theorem}
\label{thm:lp}
There is a subset $\sX_{\mathrm{sub}}\subset \sX$ that is dense in $\sX$, on which each LP admits a unique solution $\vy_\ast$, and the solution map $\cF_3:\vx\mapsto\vy_\ast$ is locally Lipschitz continuous on $\sX_{\mathrm{sub}}$.
\end{theorem}
\begin{corollary}
\label{coro:lp}
There exists a regular implicit model $\cG(\vy,\vx)$ with $\textup{Fix}(\cG(\cdot,\vx)) = \cF_3(\vx)$ on $\sX_{\mathrm{sub}}$.
\end{corollary}
Corollary \ref{coro:lp} follows immediately from Theorems \ref{thm:lip} and \ref{thm:lp}. It indicates the existence of implicit models with desired properties that solves LP. As in the previous case studies, we now (i) choose a problem-specific parameterization of $\cG$ and (ii) verify the theory numerically on this architecture.
\textbf{Implicit GNN parameterization.} We model the implicit operator $\cG$ for LP with a \textit{graph neural network (GNN)}. First, express an LP instance $\vx=(\mA,\vb,\vc,\circ,\vl,\vu)$ as a bipartite graph (Figure \ref{fig:gnn-diagram}). We create $n$ variable nodes $\{V_j\}_{j=1}^n$ and $m$ constraint nodes $\{W_{i}\}_{i=1}^m$. Node features collect the data of the LP: each $V_j$ stores $(c_j,l_j,u_j)$; each $W_i$ stores $(b_i,\circ_i)$. We connect $W_i$ to $V_j$ if $A_{ij}\neq 0$, and place $A_{ij}$ on that edge as its feature.
Given this representation, an (explicit) GNN can map the LP to a solution, i.e., $\vy_\ast=\textup{GNN}(\vx)$ where $\vx$ denotes the graph-encoded LP.
This approach was proposed in \cite{gasse2019exact} in the context of mixed-integer linear programs, and \cite{chen2023lp} subsequently showed that (explicit) GNNs offer a universal framework for representing LPs. Built on this, we propose an \emph{implicit} GNN:
\begin{equation}
\label{eq:gnn-structure}
\vz_\ast = \cG_{\Theta}(\vz_\ast,\cQ_{\Phi}(\vx)), \quad \vy_\ast = \cQ_{\Psi}(\vz_\ast)
\end{equation}
where $\cG_{\Theta}$ is the core GNN, $\cQ_{\Phi}$ encodes instance-specific (static) features from $\vx$, and $\cQ_{\Psi}$ decodes per-variable states to the solution. Both $\cQ_{\Phi}$ and $\cQ_{\Psi}$ are small MLPs shared across all nodes.
At inference, we repeatedly call $\cG_{\Theta}$ with initialization $\vz_0=\vzero$: $\vz_{t}=\cG_{\Theta}(\vz_{t-1},\cQ_{\Phi}(\vx))$ for $t=1,2,\cdots,T$, and finally output $\vy_t=\cQ_{\Psi}(\vz_t)$. Relative to prior work, our only modification is to attach to each variable node an additional \emph{dynamic} state $z_j^{(t)} \in \sR$. Details appear in Appendix \ref{sec:app-lp-exp}.
\begin{figure*}[t] % {0.4\textwidth}
\centering
\resizebox{\linewidth}{!}{\input{gnn-deq.tikz}}
\caption{The graph representation of LP and implicit GNN applied on this graph}
\label{fig:gnn-diagram}
\end{figure*}
There is a rapidly growing literature on implicit GNNs with diverse applications and theories \cite{gu2020implicit,park2022convergent,chen2022efficient,chen2022optimization,baker2023implicit,lin2024ignn,nastorg2024implicit,zhong2024efficient,yang2025implicit}. Our LP case study is complementary to this line of work: rather than adopting a particular implicit-GNN architecture, we start from a standard explicit GNN for LP and convert it into a fixed-point formulation tailored to linear programs.
In addition, recent work on ML for LP has progressed quickly (e.g., \cite{chen2023lp,fan2023smart,sakaue2024generalization,li2024on,li2024pdhgunrolled,qian2024exploring,kuang2025accelerate,liu2024learning}); for broader context, see overviews on Learning to Optimize \cite{chen2024learning} and AI for Operations Research \cite{fan2024artificial}. Our goal here is not to outperform all state-of-the-arts; rather, we aim to establish a foundational point: converting a standard LP-GNN into an implicit form yields the benefits predicted by our theory—implicit models can be simple at the update level yet expressive at the fixed-point map, and their performance improves predictably with test-time iteration.
\textbf{Experiments.} We sample LP instances \(\vx=(\mA,\vb,\vc,\circ,\vl,\vu)\), solve it to obtain an optimal solution \(\vy_\ast\), and form \(2{,}500\) training pairs and \(1{,}000\) test pairs like $(\vx,\vy_\ast)$, denoted \(\sD_{\textup{LP,train}}\) and \(\sD_{\textup{LP,test}}\).
We also create five perturbed test sets \(\{\sD^{(j)}_{\textup{LP,test}}\}_{j=1}^{5}\) by altering exactly one block among
(\(\mA\), \(\vb\), \(\vc\), \(\vl\), or \(\vu\)).
For each \((\vx_i,\vy_i^\ast)\in\sD_{\textup{LP,test}}\) and each perturbation type \(j\), we form a perturbed instance \(\vx_{i,j}\), solve it to obtain \(\vy^\ast_{i,j}\), and collect
\(\sD^{(j)}_{\textup{LP,test}}=\{(\vx_{i,j},\vy^\ast_{i,j})\}_{i=1}^{1000}\).
Details in Appendix \ref{sec:app-lp-exp}.
We report:
\[L_t(j) := \max_{1 \leq i \leq 1000} \frac{\| \vy_t(\vx_{i}) - \vy_t(\vx_{i,j}) \|}{ \|\vx_{i} - \vx_{i,j}\| }, \quad \textup{and}\quad E_t(i,j) := \frac{\|\vy_t(\vx_{i,j})-\vy^\ast_{i,j}\|}{\|\vy^\ast_{i,j}\|+\epsilon},
\]
for $1 \leq i \leq 1000, 0 \leq j \leq 5$, where $j=0$ denotes the unperturbed pair $(\vx_{i,0},\vy^\ast_{i,0}):=(\vx_i,\vy^\ast_i)$.
Results support our theory.
{\textbf{\textit{(i)}}} Figure~\ref{fig:lp-lip} plots the five curves \(L_t(j)\) (one for each perturbation type). At \(t=1\), a single application of \eqref{eq:gnn-structure} yields relatively small empirical Lipschitz constants for \textit{all} perturbation modes. As iterations proceed toward the fixed point, Lipschitz constants grow markedly.
{\textbf{\textit{(ii)}}} Figure~\ref{fig:lp-err} reports the mean\(\pm\)std of \(E_t(i,j)\): \(E_t\) decreases and stabilizes at \(0.146\), indicating that the growth of \(L_t\) reflects the higher intrinsic complexity of the solution mapping \(\vy_\ast(\vx)\) rather than divergence or instability.
{\textbf{\textit{(iii)}}} Table \ref{tab:gnn-compare} contrasts implicit and explicit GNNs. At matched embedding sizes, implicit GNNs match or beat explicit ones—most clearly at small/mid sizes (4/8/16). In addition, a smaller implicit model can outperform a larger explicit model on training error. For example, implicit–4 vs. explicit–8 (0.203 vs. 0.233) and implicit–8 vs. explicit–16 (0.162 vs. 0.183). This supports our theory that iterating a simple implicit operator can yield strong expressivity.
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{lip_plot_lp.pdf}
\caption{Empirical Lipschitz \(L_t\) of the \(t\)-step map \(\vy_t(\cdot)\) vs.\ iteration. \(L_t\) starts small at \(t{=}1\) and grows substantially by \(t{=}8\) for all perturbation modes.}
\label{fig:lp-lip}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
\includegraphics[width=\linewidth]{err_plot_lp.pdf}
\caption{Relative error \(E_t\) (mean \(\pm\) std over original and perturbed inputs) vs.\ iteration. \(E_t\) decreases to \(0.146 \pm 0.091\)---$\vy_t$ converges towards \(\vy_\ast\).}
\label{fig:lp-err}
\end{subfigure}
\caption{Numerical validation on the linear-program task.}
\label{fig:lp-theory}
\end{figure}
\textbf{Discussion on generalization.}
While generalization is not our main focus, a trend in Table~\ref{tab:gnn-compare} is informative: explicit GNNs improve as width increases from 4 to 8 but then \textit{overfit} (test error significantly rises at 16/32), whereas implicit GNNs improve from 4 to 8 to 16 and only tick up slightly at 32. We attribute this to: (i) LP constraints $\mA \vy \circ \vb$ in \eqref{eq:lp} are specified implicitly rather than as an explicit set; implicit models align naturally with such a structure, and (ii) while fixed-point maps \(\vy_\ast(\vx)\) can be sensitive to inputs $\vx$, the implicit formulation allows us realize them via a simpler, smaller operator $\cG$, which ``implicitly" regularizes training and support good generalization in practice.
\begin{table}[t]
\centering
\caption{Comparison between explicit GNNs and implicit GNNs on the LP task.}
\label{tab:gnn-compare}
{\small
\begin{tabular}{l|l|l|l|l|l}
\hline
\multirow{4}{*}{Exp-GNNs} & Emb. size & 4 & 8 & 16 & 32 \\ \cline{2-6}
& \# Params. & 580 & 2,088 & 7,888 & 30,624 \\ \cline{2-6}
& Err (Train) & 0.387 $\pm$ 0.103 & 0.233 $\pm$ 0.084 & 0.183 $\pm$ 0.070 & 0.112 $\pm$ 0.049 \\ \cline{2-6}
& Err (Test) & 0.397 $\pm$ 0.107 & 0.273 $\pm$ 0.104 & 0.283 $\pm$ 0.111 & 0.318 $\pm$ 0.122 \\ \hline\hline
\multirow{4}{*}{Imp-GNNs} & Emb. size & 4 & 8 & 16 & 32 \\ \cline{2-6}
& \# Params. & 722 & 2,350 & 8,390 & 31,606 \\ \cline{2-6}
& Err (Train) & 0.203 $\pm$ 0.107 & 0.162 $\pm$ 0.094 & 0.131 $\pm$ 0.080 & 0.118 $\pm$ 0.073 \\ \cline{2-6}
& Err (Test) & 0.218 $\pm$ 0.117 & 0.177 $\pm$ 0.105 & 0.152 $\pm$ 0.098 & 0.156 $\pm$ 0.109 \\ \hline
\end{tabular}
}
\end{table}
\subsection{Case Study 4: LLM Reasoning}
While previous case studies focused on domains with strict mathematical definitions (inverse problems, PDEs, LPs), we now investigate if our theory extends to broader applications where metrics like ``smoothness" and ``Lipschitz continuity" are less formally defined. We examine the looped transformer for LLM reasoning, utilizing the pre-trained model from \cite{geiping2025scaling}. Unlike standard feed-forward transformers, this architecture recycles a shared block $\mathcal{G}_{\Theta}$ to iteratively update a latent ``thought" vector $\vz$:
$
\vz_{t} = \mathcal{G}_{\Theta}(\vz_{t-1}, \mathcal{Q}_{\Phi}(\vx)), ~ \vy_{t} = \mathcal{Q}_{\Psi}(\vz_{t})
$
where $\mathcal{Q}_{\Phi}$ encodes the input $\vx$, and $\mathcal{Q}_{\Psi}$ decodes the latent state into the output sequence $\vy_t$ obtained after $t$ recurrent blocks.
Strictly extending our Lipschitz theory to the discrete space of language tokens is challenging, as standard norms do not apply. However, we can empirically test the core prediction of our theory: can the model express increasingly complex mappings as iterations increase? In this context, complexity implies: \textit{subtle differences in the input correspond to substantial shifts in context.} A capable model must effectively distinguish these semantic nuances and produce vastly different responses.
\textbf{Qualitative Results.} Table \ref{tab:charge_context_shift} visualizes the evolution of reasoning on a typical example. At early iterations ($t=2, 4$), the model fails to differentiate context (Physics vs. Finance), producing repetitions or shallow associations. Conversely, with more iterations ($t=6,8$ or more), the model utilizes increased test-time compute to resolve this ambiguity, correctly defining ``charge" as electric potential versus financial debt. This confirms the implicit operator's ability to progressively realize complex, context-sensitive mappings.
\begin{table}[t]
\centering
\caption{Evolution of model outputs for the inputs ``charge and voltage" vs. ``charge and pay", showing how the semantic difference emerges over iterations.}
\label{tab:charge_context_shift}
\scriptsize
\renewcommand{\arraystretch}{1.5}
\begin{tabular}{p{0.34\textwidth} p{0.34\textwidth} p{0.21\textwidth}}
\hline
\textbf{Input/Output 1} & \textbf{Input/Output 2} & \textbf{Comments} \\ \hline
\textit{$\vx$:} explain the difference between charge and \textbf{voltage}. & \textit{$\vx'$:} explain the difference between charge and \textbf{pay}. & Prompts differ in a \emph{single word}, but the context shifts from Physics to Finance. \\ \hline
\textit{$\vy_2(\vx)$:} explain the difference between charge and {voltage}. & \textit{$\vy_2(\vx')$:} explain the difference between charge and {pay}. & Both outputs merely echo inputs. \\ \hline
\textit{$\vy_4(\vx)$:} explain the difference between charge and {voltage}. explain the difference between current and voltage. &
\textit{$\vy_4(\vx')$:} explain the difference between charge and {pay}. explain the difference between the two. & Outputs repeat the inputs with slight variations; contexts are not yet separated. \\ \hline
\textit{$\vy_6(\vx)$:} The difference between voltage and charge is that voltage is the difference between the potential difference between two points, while charge is the difference between the electric field between two points. & \textit{$\vy_6(\vx')$:} explain the difference between charge and {pay}. Explain the difference between charge and pay. & {Separation begins:} Output~1 moves into a Physics explanation (potential difference), while Output~2 still stays near the prompt. \\ \hline
\textit{$\vy_8(\vx)$:} The difference between charge and voltage is that voltage is the difference in electric potential between two points in an electric field, while charge is the amount of electric charge in a system. & \textit{$\vy_8(\vx')$:} Charge is the amount of money that a person owes to a company or organization. Pay is the amount of money that a person receives from a company or organization. & Full separation: Output~1 gives a Physics definition; Output~2 correctly adopts the Financial interpretation (money owed vs.\ received). \\ \hline
\textit{$\vy_{32}(\vx)$:} Charge is the amount of electric charge present in a body. Voltage is the difference in electric potential between two points. & \textit{$\vy_{32}(\vx')$:} Charge is the amount of money that a person or company owes to a credit card company. Pay is the amount of money that a person or company has paid to the credit card company. & Refinement: both domains have stable, concise, and accurate definitions specialized to Physics versus Finance. \\ \hline
\end{tabular}
\end{table}
\textbf{Quantitative Results.} To quantitatively measure this, we define an ``Empirical Lipschitz" constant $L_t$ using Levenshtein distance $d(\cdot, \cdot)$:
\[
L_t(i):=\frac{d(\vy_t(\vx_i),\vy_t(\vx^\prime_i))}{d(\vx_i,\vx^\prime_i)}
\]
We construct $\{(\vx_i, \vx'_i)\}_{i=1}^{200}$, a dataset of 200 pairs where inputs differ by only 1–2 words but require vastly different semantic contexts\footnote{\scriptsize Available at: \url{https://github.com/liujl11git/semantic_contrast_pairs/blob/main/semantic_contrast_pairs_200.csv}}.
Figure \ref{fig:llm-Lip} plots the geometric mean of $L_t$, which rises from $\approx 29.2$ at $t=2$ (indicating relative insensitivity) to saturate at $\approx 52.5$ by $t=16$. Consistent with our theory, this growth reflects the model's emergent capacity to map proximal inputs to semantically distinct outputs.
Even in the discrete domain of language reasoning, iterating a fixed operator allows the model to scale its expressive power, evolving from simple surface-level processing to complex, context-aware reasoning.
\begin{figure}
% \vspace{-10mm}
\begin{center}
\includegraphics[width=0.48\textwidth]{lip_plot_llm.pdf}
\end{center}
% \vspace{-5mm}
\caption{Empirical Lipschitz of the output sequence $\vy_t(\cdot)$ generated by \cite{geiping2025scaling} using $t$ recurrent blocks. $L_t$ grows as $t$ increases.}
\label{fig:llm-Lip}
% \vspace{-1mm}
\end{figure}
\section{Conclusions}
We establish a sharp expressivity boundary for regular implicit models: as iterations increase, their expressive power grows, and the resulting fixed points can represent exactly the class of locally Lipschitz maps. In three case studies, per-iteration Lipschitz estimates grew toward the target’s complexity while accuracy improved and stabilized. Overall, both theory and evidence show that iterating a simple operator is a principled route to powerful models, clarifying how fixed-point architectures can match or surpass large explicit networks.
%\newpage
\bibliography{refs}
\bibliographystyle{amsxport}
%\newpage
\appendix
\section{Proofs of main results}
\label{sec:app-main}
The core intuition behind our proofs is an extension of the $1/x$ example discussed in the introduction.
For Theorem 2.4 (Sufficiency), we construct the implicit operator $\mathcal{G}$ as a dynamic interpolation: $\mathcal{G}(\vy,\vx) = (1-\varepsilon(\vx))\vy + \varepsilon(\vx)\mathcal{F}(\vx)$, which iteratively pulls the state $\vy$ toward the target $\mathcal{F}(\vx)$ with a step size $\varepsilon(\vx)$. The key theoretical innovation is making this step size adaptive: we construct $\varepsilon(\vx)$ to be inversely proportional to the local steepness (Lipschitz constant) of the target $\vy_\ast(\vx)$. In regions where the target function becomes extremely steep or singular (like $x \to 0$ for $1/x$), our constructed $\varepsilon(\vx)$ naturally vanishes. This effectively ``slows down" the dynamics, ensuring the operator $\mathcal{G}$ itself remains globally smooth and contractive.
Theorem 2.5 (Necessity) establishes the converse: we show that for any regular operator, the local steepness of the fixed point is mathematically bounded by the operator's parameters ($y$-contraction modulus $\mu(\vx)$); and the fixed point map $\vy_\ast(\vx)$ can only become singular if the convergence rate slows down (contraction modulus $\to 1$), perfectly matching the mechanism used in our sufficiency construction.
\subsection{Proof of sufficiency}
\begin{proof}[Proof of Theorem \ref{thm:lip}]
Given any $\cF$ satisfying Assumption \ref{asmp:f}, the existence of $\cG$ is proved by the following construction:
\begin{equation}
\label{eq:g-definition}
\cG(\vy,\vx) = \cF(\vx) + (1-\varepsilon(\vx)) (\vy - \cF(\vx)).
\end{equation}
The proof will be done by choosing a function $\varepsilon: \sX\to\sR$ such that
\begin{itemize}
\item Functions $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ are both globally Lipschitz continuous on $\sX$.
\item $0 < \varepsilon(\vx) < 1$ for any $\vx \in \sX$.
\end{itemize}
The existence of such a $\varepsilon$ function is deferred to Theorem \ref{thm:eps-unbdd}. Now, let's suppose such a $\varepsilon(\vx)$ is given and finish the whole proof. First, let's check the contractivity of $\cG$ in \eqref{eq:g-definition} as $\vx$ fixed. For any $\vy,\hat{\vy}\in\sR^n$, it holds that
\[ \cG(\vy,\vx) - \cG(\hat{\vy}, \vx) = (1-\varepsilon(\vx)) (\vy - \cF(\vx)) - (1-\varepsilon(\vx)) (\hat{\vy} - \cF(\vx)) = (1-\varepsilon(\vx)) (\vy - \hat{\vy}). \]
Since $0<\varepsilon(\vx) < 1$ for $\vx \in \sX$, we conclude that $\cG(\cdot,\vx)$ is contractive for $\vx \in \sX$. In addition, the continuity of the contraction modulus $(1-\varepsilon(\vx))$ is a direct result of the continuity of $\varepsilon(\vx)$.
Finally, we check the Lipschitz continuity as $\vy$ fixed. For any $\vx,\hat{\vx}\in\sX$ and any $\vy \in \sR^n$, it holds that
\begin{align*}
& \cG(\vy,\vx) - \cG(\vy, \hat{\vx})\\
= & \Big(\cG(\vy,\vx) - \vy\Big) - \Big(\cG(\vy, \hat{\vx}) - \vy\Big) \\
= & \Big(\cF(\vx) - \vy + (1-\varepsilon(\vx)) (\vy - \cF(\vx)) \Big) - \Big(\cF(\hat{\vx}) - \vy + (1-\varepsilon(\hat{\vx})) (\vy - \cF(\hat{\vx}))\Big) \\
= & - \varepsilon(\vx) (\vy - \cF(\vx)) + \varepsilon(\hat{\vx}) (\vy - \cF(\hat{\vx}))\\
= & \Big( - \varepsilon(\vx) + \varepsilon(\hat{\vx}) \Big) \vy + \Big( \varepsilon(\vx)\cF(\vx) - \varepsilon(\hat{\vx}) \cF(\hat{\vx}) \Big)
\end{align*}
With a fixed $\vy \in \sR^n$, the Lipschitz continuity of $\cG(\vy,\cdot)$ follows from the Lipschitz continuity of $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$. In particular, by denoting the Lipschitz constants of $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ as $L_{\varepsilon}$ and $L_{\varepsilon \cF}$ respectively, we have
\[ \|\cG(\vy,\vx) - \cG(\vy, \hat{\vx})\| \leq L_{\varepsilon} \|\vx - \hat{\vx}\| \cdot \|\vy\| + L_{\varepsilon \cF} \|\vx - \hat{\vx}\| \leq \Big(L_{\varepsilon} \|\vy\| + L_{\varepsilon \cF}\Big) \|\vx - \hat{\vx}\| \]
where the Lipschitz constant of $\cG$, $L:=L_{\varepsilon} \|\vy\| + L_{\varepsilon \cF}$, grows linearly w.r.t. $\|\vy\|$, which finishes the whole proof.
\end{proof}
% Below we provide the core theorem used in the proof of Theorem \ref{thm:lip} and its proof.
% \begin{theorem}
% \label{thm:eps}
% For any $\cF$ and $\sX$ satisfying Assumption \ref{asmp:f}, there exists a function $\varepsilon: \sX\to\sR$ such that $0 < \varepsilon(\vx) < 1$ for $\vx \in \sX$, and $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ are both globally Lipschitz continuous on $\sX$.
% \end{theorem}
Below we provide the core theorems used in the proof of Theorem \ref{thm:lip}. We first consider $\sX$ to be bounded (Theorem \ref{thm:eps}) and then extend the results to the unbounded domain (Theorem \ref{thm:eps-unbdd}).
\begin{theorem}
\label{thm:eps}
For any $\cF$ satisfying Assumption \ref{asmp:f} defined on a bounded domain $\sX \subset \sR^d$, there exists a function $\varepsilon: \sX\to\sR$ such that $0 < \varepsilon(\vx) < 1$ for $\vx \in \sX$, and $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ are both globally Lipschitz continuous on $\sX$.
\end{theorem}
\begin{proof} Let $\overline{\sX}$ be the closure of set $\sX$. In this proof, we will first extend $\cF$ to $\overline{\sX}$, construct the $\varepsilon$ function on $\overline{\sX}$, and finally prove the global Lipschitz continuity of $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ on $\overline{\sX}$.
\textbf{Step 1: Extension to $\overline{\sX}$.} First we extend $\cF$ to $\bar \vx \in \overline{\sX} \setminus \sX$ by the limit relative to $\sX$:
\[
\cF(\bar \vx )=\left\{
\begin{aligned}
& \lim_{\sX\ni \vx\to\bar \vx}\cF(\vx), \quad && \textup{if $\lim_{\sX\ni \vx\to\bar \vx}\cF(\vx)$ exists}, \\
& 0, && \textup{otherwise}.
\end{aligned}\right.
\]
Note that even if $\cF$ is continuously extendable to $\bar \vx$, it is still possible that $\cF$ is not locally Lipschitz continuous at the point $\bar \vx$. A simple example is the function $\sqrt{x}$, which is continuous as $x\geq 0$ and locally Lipschitz continuous for all points $x >0$ but NOT locally Lipschitz at $x=0$. We collect all these points (where $\cF$ is not locally Lipschitz) into the set $\sD(\cF)$:
\[ \sD(\cF):= \left\{ \vx \in \overline{\sX}: \textup{$\cF$ is not locally Lipschitz continuous at $\vx$} \right\} \]
For brevity, we will use $\sD$ to denote $\sD(\cF)$. It holds that $\sD$ is a closed set (ref. to Lemma \ref{lemma:local-lip-closed}) and $\sD \subset \overline{\sX}\setminus\sX$.
\textbf{Step 2: Constructing a function $\varepsilon: \overline{\sX} \to \sR_{\ge 0}$.}Now let's define a set including all points that are very ``safe", i.e., sufficiently far from the discontinuity set $\sD$. In particular, given a positive real number $r>0$, the set $\sD_r$ is define by
\[\sD_r:= \big\{ \vx \in \overline{\sX}: d(\vx, \sD) \geq r \big\},\]
where $d(\vx, \sD)$ means the distance of $\vx$ and $\sD$, and the closedness of $\sD_r$ can be derived from the continuity of the distance function. Since $\sD_r \subset \overline{\sX}$ and $\overline{\sX}$ is compact, $\sD_r$ must be compact. Note that $\sD_r$ and $\sD$ are disjoint, hence $\cF$ is locally Lipschitz continuous everywhere on $\sD_r$. Thanks to the fact that local Lipschitz continuity on a compact set implies global Lipschitz continuity (ref to Lemma \ref{lemma:local-Lip}), we can conclude that $\cF$ is bounded and globally Lipschitz continuous on $\sD_r$ for all $r > 0$. Therefore, the following two supremums exist, as long as the cardinality (number of elements) of $\sD_r$ is large enough:
\begin{align*}
& h_1(r)=\left\{
\begin{aligned}
& \sup_{\vx_1,\vx_2\in\sD_r,\vx_1\not=\vx_2}\frac{ \|\cF(\vx_1)-\cF(\vx_2) \| }{\|\vx_1-\vx_2\|}, \quad && \mathrm{card}(\sD_r) \geq 2, \\
& 0, && \textup{otherwise}.
\end{aligned}\right.
\\
& h_2(r)= \left\{
\begin{aligned}
& \sup_{\vx\in\sD_r}\|\cF(\vx)\|, \quad && \mathrm{card}(\sD_r) \geq 1,\\
& 0, && \textup{otherwise}.
\end{aligned}\right.
\end{align*}
Here, both $h_1$ and $h_2$ are non-negative and monotone non-increasing on $(0,+\infty)$. Then we define:
\[ \hat{h}(r) = \frac{1}{h_1(r) + h_2(r) + 1}. \]
It has the following properties:
\begin{itemize}[leftmargin=5mm]
\item Bounded: $ 0 < \hat{h}(r) \leq 1$ as $r > 0$.
\item Monotone : $\hat{h}(r_1) \leq \hat{h}(r_2)$ as $0 < r_1 \leq r_2$. (Due to the monotonicity of $h_1$ and $h_2$)
\item Naturally extended to $r=0$: $\lim_{r\to 0_{+}}\hat{h}(r)$ exists. (Due to the monotonicity of $\hat{h}$)
\item $\hat{h}(r) h_i(r) < 1$ for $r \geq 0$ and $i=1,2$.
\end{itemize}
These properties implies that $\hat{h}$ is Riemann integrable on $[0,+\infty)$. Then we can define the following function:
\[ \hat{\varepsilon}(r) := \int_{0}^r \hat{h}(s) \mathrm{d}s \]
with the following properties:
\begin{itemize}[leftmargin=5mm]
\item $\hat{\varepsilon}(0)=0$.
\item Monotone increasing. This is a straightforward result of the fact that $\hat{h}(s) > 0$ for $s > 0$.
\item Strictly positive as $r>0$. This is also straightforward as $\hat{h}(s) > 0$ for $s > 0$.
\item $1$-Lipschitz continuous on $[0,+\infty)$. For any $r_1,r_2$ with $0 \leq r_1 \leq r_2 < +\infty$, we have
\[ | \hat{\varepsilon}(r_1) - \hat{\varepsilon}(r_2) | = \hat{\varepsilon}(r_2) - \hat{\varepsilon}(r_1) = \int_{r_1}^{r_2} \hat{h}(s)\mathrm{d}s \leq \left( \sup_{r \geq 0} \hat{h}(r) \right) |r_1 - r_2| = |r_1 - r_2|. \]
\end{itemize}
With such a $\hat{\varepsilon}(r)$, we can define $\varepsilon(\vx)$ by
\[ \varepsilon(\vx) = \frac{\hat{\varepsilon} \Big( d(\vx,\sD) \Big)}{1+\hat{\varepsilon} \Big( d(\vx,\sD) \Big)} . \]
It holds that $\varepsilon(\vx)=0$ for $\vx \in \sD$ and $0 < \varepsilon(\vx) < 1$ for $\vx \in \overline{\sX} \setminus \sD$. As $\sD \subset \overline{\sX}\setminus\sX$, we have $0 < \varepsilon(\vx) < 1$ for $\vx \in \sX$.
\textbf{Step 3: Establishing Lipschitz continuity.} Since the distance function $d(\vx,\sD)$ is $1$-Lipschitz continuous \cite[Theorem 4.8 (1)]{federer1959curvature}, the Lipschitz continuity of $\hat{\varepsilon}$ implies the Lipschitz continuity of $\varepsilon$. In particular, for all $\vx_1,\vx_2 \in \overline{\sX}$, it holds that
\begin{align*}
& \Big|\varepsilon(\vx_1) - \varepsilon(\vx_2)\Big| \\
= & \left| \frac{\hat{\varepsilon} \Big( d(\vx_1,\sD) \Big)}{1+\hat{\varepsilon} \Big( d(\vx_1,\sD) \Big)} - \frac{\hat{\varepsilon} \Big( d(\vx_2,\sD) \Big)}{1+\hat{\varepsilon} \Big( d(\vx_2,\sD) \Big)} \right| && \left(\textup{$\frac{x}{1+x}$ is 1-Lipschitz as $\left(\frac{x}{1+x}\right)^\prime=\frac{1}{(1+x)^2}$}\right) \\
\leq & \left| \hat{\varepsilon} \Big( d(\vx_1,\sD) \Big) - \hat{\varepsilon} \Big( d(\vx_2,\sD) \Big) \right| && (\textup{Lipschitz continuity of $\hat{\varepsilon}$}) \\
\leq & \Big| d(\vx_1,\sD) - d(\vx_2,\sD) \Big| \leq \|\vx_1 - \vx_2\| && (\textup{Lipschitz continuity of $d$})
\end{align*}
Therefore, to complete the whole proof, it's enough to show the global Lipschitz continuity of $\varepsilon \cF$ on $\overline{\sX}$. As $\overline{\sX}$ is compact, and thanks to Lemma \ref{lemma:local-Lip}, it's enough to show $\varepsilon \cF$ is locally Lipschitz everywhere on $\overline{\sX}$.
First, we consider local Lipschitz continuity of $\varepsilon\cF$ on $\overline{\sX}\setminus\sD$. Due to Lemma \ref{lemma:local-lip-closed}, $\overline{\sX}\setminus\sD$ must be open relative to $\overline{\sX}$.
For any $\vx \in \overline{\sX}\setminus\sD$, there must be a small enough $r>0$ such that $\sU:=\sB(\vx,r) \cap \overline{\sX} \subset \overline{\sX}\setminus\sD$. Pick $\vx_1,\vx_2 \in \sU$. For any $\vx_1,\vx_2$, it holds that
\begin{equation}
\label{eq:proof-lip-epsf}
\begin{aligned}
& \|\varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_2)\cF(\vx_2)\| \\
= & \|\varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_1)\cF(\vx_2) + \varepsilon(\vx_1)\cF(\vx_2) -\varepsilon(\vx_2)\cF(\vx_2)\| \\
\leq & \varepsilon(\vx_1) \big\|\cF(\vx_1) - \cF(\vx_2)\big\| + |\varepsilon(\vx_1) -\varepsilon(\vx_2)| \cdot \big \|\cF(\vx_2)\big \|.
\end{aligned}
\end{equation}
Since both $\varepsilon$ and $\cF$ are locally Lipschitz and locally bounded everywhere on $\overline{\sX}\setminus\sD$, they must be Lipschitz and bounded within $\sU$. Then the local Lipschitz continuity of $\varepsilon \cF$ at $\vx$ immediately follows from \eqref{eq:proof-lip-epsf}. Note that $\vx$ is arbitrarily picked from $\overline{\sX}\setminus\sD$, hence $\varepsilon \cF$ is locally Lipschitz everywhere on $\overline{\sX}\setminus\sD$.
Next, we consider local Lipschitz continuity of $\varepsilon\cF$ on $\sD$. For any $\vx\in\sD$, we consider its neighborhood $\sU:=\sB(\vx,1)\cap\overline{\sX}$ and pick $\vx_1,\vx_2\in \sU$. Then we need to consider three cases. The first case is both $\vx_1,\vx_2$ belong to the discontinuity set $\sD$: $\vx_1,\vx_2\in \sD$. In this case, it holds that $\varepsilon(\vx_1)=\varepsilon(\vx_2)=0$ and hence
$$
\Big\| \varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_2)\cF(\vx_2) \Big\| = 0 \leq \|\vx_1 - \vx_2\|.
$$
The second case is that one of the point is in $\sD$ while the other is not, we suppose $\vx_1\in\sD,\vx_2\in \overline{\sX}\setminus\sD$, then
\begin{align*}
& \Big\| \varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_2)\cF(\vx_2) \Big\| \\
= & \Big\|\varepsilon(\vx_2)\cF(\vx_2) \Big\| \leq \hat{\varepsilon}\Big(d(\vx_2,\sD)\Big)\|\cF(\vx_2)\| \\
=& \left(\int_{0}^{d(\vx_2,\sD)}\hat{h}(s) \mathrm{d}s \right) \|\cF(\vx_2)\|\\
\leq &\hat{h}(d(\vx_2,\sD)) \cdot d(\vx_2,\sD) \cdot \|\cF(\vx_2)\| && (\textup{Monotonicity of $\hat{h}$}) \\
\leq& \hat{h}(d(\vx_2,\sD)) \cdot d(\vx_2,\sD) \cdot h_2(d(\vx_2,\sD)) && (\textup{Definition of $h_2$}) \\
<& d(\vx_2,\sD) && (\hat{h}(r) \cdot h_2(r) < 1 \textup{ as } r \geq 0)\\
=& d(\vx_2,\sD) - d(\vx_1,\sD) \leq \|\vx_1-\vx_2\|
\end{align*}
Finally, we consider the last case where $\vx_1,\vx_2\in\overline{\sX}\setminus\sD$. Without loss of generality, we assume \[0<d(\vx_1,\sD)\leq d(\vx_2,\sD).\]
Then the definition of $h_1$ and $h_2$ implies that
\[
\begin{aligned}
& \|\cF(\vx_1) - \cF(\vx_2)\| \\
\leq & \max\Big( h_1(d(\vx_1,\sD)), h_1(d(\vx_2,\sD)) \Big) \cdot \|\vx_1- \vx_2\| \\
= & h_1(d(\vx_1,\sD)) \cdot \|\vx_1- \vx_2\| ,
\end{aligned}
\]
and
\[
\|\cF(\vx_2)\| \leq h_2(d(\vx_2,\sD)).
\]
Consequently, applying \eqref{eq:proof-lip-epsf} and the above inequalities, we have
\begin{align*}
& \|\varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_2)\cF(\vx_2)\| \\
\leq & \varepsilon(\vx_1) \big\|\cF(\vx_1) - \cF(\vx_2)\big\| + |\varepsilon(\vx_1) -\varepsilon(\vx_2)| \cdot \big \|\cF(\vx_2)\big \| \\
\leq & \varepsilon(\vx_1) \cdot h_1(d(\vx_1,\sD)) \cdot \|\vx_1- \vx_2\| + |\varepsilon(\vx_1) -\varepsilon(\vx_2)| \cdot h_2(d(\vx_2,\sD)) \\
\leq & \hat{\varepsilon}\Big(d(\vx_1,\sD)\Big) \cdot h_1(d(\vx_1,\sD)) \cdot \|\vx_1- \vx_2\| + \left |\hat{\varepsilon}\Big(d(\vx_1,\sD)\Big) -\hat{\varepsilon}\Big(d(\vx_2,\sD)\Big) \right| \cdot h_2(d(\vx_2,\sD)) \\
= & \left( \int_{0}^{d(\vx_1,\sD)} \hat{h}(s) \mathrm{d}s \right) \cdot h_1(d(\vx_1,\sD)) \cdot \|\vx_1- \vx_2\| + \left( \int_{d(\vx_1,\sD)}^{d(\vx_2,\sD)} \hat{h}(s) \mathrm{d}s \right) \cdot h_2(d(\vx_2,\sD))\\
\leq & d(\vx_1,\sD) \cdot \hat{h}(d(\vx_1,\sD)) \cdot h_1(d(\vx_1,\sD)) \cdot \|\vx_1- \vx_2\| \\
& \qquad + \Big| d(\vx_1,\sD) - d(\vx_2,\sD) \Big| \cdot \hat{h}(d(\vx_2,\sD)) \cdot h_2(d(\vx_2,\sD)) \\
< & d(\vx_1,\sD) \cdot \|\vx_1- \vx_2\| + \Big| d(\vx_1,\sD) - d(\vx_2,\sD) \Big| \\
\leq & d(\vx_1,\sD) \cdot \|\vx_1- \vx_2\| + \|\vx_1- \vx_2\|
\end{align*}
The last inequality results from $\hat{h}(r) \cdot (h_1(r) + h_2(r)) < 1$ for all $r > 0$. And the above inequalities imply
\[ \|\varepsilon(\vx_1)\cF(\vx_1) - \varepsilon(\vx_2)\cF(\vx_2)\| \leq (\textup{diam}(\overline{\sX})+1)\cdot
\|\vx_1-\vx_2\|.\]
Combining all the results together, we have $\varepsilon \cF$ is locally $(\textup{diam}(\overline{\sX})+1)$-Lipschitz at any $\vx \in \overline{\sX}$. Then the compactness of $\overline{\sX}$ concludes the global Lipschitz continuous of $\varepsilon \cF$, which finishes the whole proof.
\end{proof}
Follows are some lemmas (as well as their proofs) that we used in the proof of Theorem \ref{thm:eps}.
\begin{lemma}
\label{lemma:local-lip-closed}
Let $\sT \subset\sR^d$ be closed and let $\cF:\sT\to\sR^n$.
Denote by $\sD(\cF)\subset\sT$ the set of points at which $\cF$ is \emph{not}
locally Lipschitz. Then $\sD(\cF)$ is closed (in $\sT$, hence in $\sR^d$).
\end{lemma}
\begin{proof}
Recall that $\cF$ is locally Lipschitz (relative to $\sT$) at $\vx\in\sT$ if there exist $r>0$ and $L>0$ such that
\[
\|\cF(\vu)-\cF(\vv)\|\le L\,\|\vu-\vv\|\qquad\text{for all }\vu,\vv\in \sT\cap \sB(\vx,r).
\]
Let $\sG:=\sT\setminus\sD(\cF)$ be the set of points where $\cF$ \emph{is} locally Lipschitz.
We first show that $\sG$ is relatively open in $\sT$. Fix $\vx\in\sG$ and choose $r,L$ as above.
If $\vx'\in\sT\cap \sB(\vx,r/2)$, then $\sB(\vx',r/2)\subset \sB(\vx,r)$; hence the same $L$ works on
$\sT\cap \sB(\vx',r/2)$, so $\cF$ is locally Lipschitz at $\vx'$. Therefore
$\sT\cap \sB(\vx,r/2)\subset\sG$, proving that $\sG$ is open in $\sT$.
Consequently, $\sD(\cF)=\sT\setminus\sG$ is closed in $\sT$.
Since $\sT$ is closed in $\sR^d$, every set closed in $\sT$ is also closed in $\sR^d$.
Hence $\sD(\cF)$ is closed in $\sR^d$ as well.
\end{proof}
\begin{lemma}
\label{lemma:local-Lip}
Let $\sT$ be a compact set. If $\cF$ is locally Lipschitz everywhere on $\sT$, then it must be globally Lipschitz on $\sT$.
\end{lemma}
\begin{proof}
Assume, to the contrary, that $\cF$ is not globally Lipschitz on $\sT$. Then we can choose sequences $\{\vx_k\}_{k\ge1},\{\vy_k\}_{k\ge1}\subset\sT$ such that
\begin{equation}\label{eq:blowup}
\frac{\|\cF(\vx_k)-\cF(\vy_k)\|}{\|\vx_k - \vy_k\|}\xrightarrow[]{k\to\infty}\infty.
\end{equation}
Local Lipschitzness implies continuity of $\cF$ on $\sT$, so by compactness $\cF$ is bounded:
there exists $C<\infty$ with $\|\cF(\vz)\|\le C$ for all $\vz\in\sT$. Consequently,
\[
\|\cF(\vx_k)-\cF(\vy_k)\|\le 2C\qquad\text{for all }k,
\]
and therefore \eqref{eq:blowup} forces $\|\vx_k - \vy_k\|\to 0$.
By sequential compactness of $\sT$, passing to a subsequence (not relabeled) we may assume $\vx_k\to \vx\in\sT$; since $\|\vx_k - \vy_k\|\to0$, we also have $\vy_k\to \vx$. Since $\cF$ is locally Lipschitz at $\vx$, for $k$ large enough we have
\[
\frac{\|\cF(\vx_k)-\cF(\vy_k)\|}{\|\vx_k - \vy_k\|}\le L,
\]
for some $L>0$, which contradicts \eqref{eq:blowup}. Therefore $\cF$ must be globally Lipschitz on $\sT$.
\end{proof}
Note that Lemmas \ref{lemma:local-lip-closed} and \ref{lemma:local-Lip} are standard in real analysis, we provide concise proofs of them for the sake of completeness.
Next, we relax the condition in Theorem \ref{thm:eps} and extend it to unbounded domains.
\begin{theorem}
\label{thm:eps-unbdd}
For any $\sX\subset \sR^d$ (not necessarily bounded) and any locally Lipschitz function $\cF:\sX\to \sR$, there exists a function $\varepsilon: \sX\to\sR$ such that $0 < \varepsilon(\vx) < 1$ for $\vx \in \sX$, and $\varepsilon(\vx)$ and $\varepsilon(\vx) \cF(\vx)$ are both globally Lipschitz continuous on $\sX$.
\end{theorem}
\begin{proof}
We consider the following grids in $\sR^d$:
$$
\vk = (2k_1,2k_2,\ldots,2k_d),\quad k_i\in \sZ, \, i\in [d].
$$
We define the closed d-dimensional cubic centered at $\vk$ by
$$
\cC_\vk =\left[ 2k_1-\frac{3}{2}, 2k_1+\frac{3}{2} \right]\times\left[ 2k_2-\frac{3}{2}, 2k_2+\frac{3}{2} \right]\times \cdots \times \left[ 2k_d-\frac{3}{2}, 2k_d+\frac{3}{2} \right].
$$
According to Theorem \ref{thm:eps}, there exists a function $\varepsilon_\vk$ defined on $\cC_\vk\cap \sX$, so that
\begin{itemize}[leftmargin=5mm]
\item $0<\varepsilon_\vk(\vx)<1$ on $\cC_\vk\cap \sX$.
\item $\varepsilon_\vk(\vx)$ is 1-Lipschitz on $\cC_\vk\cap \sX$.
\item $\varepsilon_\vk(\vx)\cF(\vx)$ is $\left(\mathrm{diam}(\cC_\vk) + 1\right)$-Lipschitz on $\cC_\vk\cap \sX$.
\end{itemize}
Next, we concatenate all these $\varepsilon_\vk$ functions and get a global $\varepsilon:\sX\to\sR$. We define the following concatenation function in 1-dimensional space:
\begin{equation*}
\rho(x) = \left\{
\begin{aligned}
& x + \frac{3}{2}, \quad && x\in\left[ -\frac{3}{2},-\frac{1}{2} \right], \\
& 1, && x\in\left[ -\frac{1}{2},\frac{1}{2} \right],\\
& -x+\frac{3}{2}, && x\in\left[ \frac{1}{2},\frac{3}{2} \right],\\
& 0, && \textup{otherwise}.
\end{aligned}\right.
\end{equation*}
Then we define the d-dimensional concatenation function. Let $\vx=(x_1, x_2,\ldots, x_d)$.
$$
\rho^{(d)}(\vx) = \prod_{i=1}^d \rho(x_i).
$$
We define the shifting function $\rho^{(d)}_\vk(\vx)=\rho^{(d)}(\vx-\vk)$. On $\sX$, we construct
$$
\varepsilon(\vx) = \sum_{\vk\in\{\vk:\vx\in\cC_\vk\}} \rho^{(d)}_\vk(\vx)\varepsilon_\vk(\vx).
$$
Given the constructed $\varepsilon(\vx)$, it's enough to prove that: $0<\varepsilon(\vx)<1$, $\varepsilon(\vx)$ and $\varepsilon(\vx)\cF(\vx)$ are globally Lipschitz over $\sX$. We will show these claims one by one and finish the proof.
First, let's show $0<\varepsilon(\vx)<1$. As $\rho^{(d)}$ is non-negative and $0<\varepsilon_\vk(\vx)<1$, we have
\[ 0 < \varepsilon(\vx) < \sum_{\vk} \rho^{(d)}_\vk(\vx) = 1 \]
where $\sum_{\vk} \rho^{(d)}_\vk(\vx) = 1$ comes from the fact that each term $\rho(x_i - 2k_i)$ depends only on its specific index $k_i$ and not on the others, and hence we can distribute the summation as $\sum_i \sum_j a_i b_j = (\sum_i a_i)(\sum_j b_j)$. That is,
\[
\begin{aligned}
\sum_{\mathbf{k}} \rho^{(d)}_\mathbf{k}(\vx)
= & \left( \sum_{k_1 \in \mathbb{Z}} \rho(x_1 - 2k_1) \right) \times \left( \sum_{k_2 \in \mathbb{Z}} \rho(x_2 - 2k_2) \right) \times \cdots \times \left( \sum_{k_d \in \mathbb{Z}} \rho(x_d - 2k_d) \right) \\
= & 1 \times 1 \times \cdots \times 1 = 1.
\end{aligned}
\]
Second, we prove the Lipschitz continuity of $\varepsilon(\vx)$. In particular, it holds that
$$\begin{aligned}
|\varepsilon(\vx) - \varepsilon(\hat{\vx})|
&= \left| \sum_{\mathbf{k}} \rho^{(d)}_\mathbf{k}(\vx)\varepsilon_\mathbf{k}(\vx) - \sum_{\mathbf{k}} \rho^{(d)}_\mathbf{k}(\hat{\vx})\varepsilon_\mathbf{k}(\hat{\vx}) \right| \\
&= \left| \sum_{\mathbf{k}} \rho^{(d)}_\mathbf{k}(\vx)(\varepsilon_\mathbf{k}(\vx) - \varepsilon_\mathbf{k}(\hat{\vx})) + \sum_{\mathbf{k}} (\rho^{(d)}_\mathbf{k}(\vx) - \rho^{(d)}_\mathbf{k}(\hat{\vx}))\varepsilon_\mathbf{k}(\hat{\vx}) \right| \\
&\le \sum_{\mathbf{k}} \rho^{(d)}_\mathbf{k}(\vx) \underbrace{|\varepsilon_\mathbf{k}(\vx) - \varepsilon_\mathbf{k}(\hat{\vx})|}_{\le \|\vx-\hat{\vx}\|} + \sum_{\vk\in\{\vk:\vx\in\cC_\vk\}} \underbrace{|\rho^{(d)}_\mathbf{k}(\vx) - \rho^{(d)}_\mathbf{k}(\hat{\vx})|}_{\le \sqrt{d}\|\vx-\hat{\vx}\|} \underbrace{|\varepsilon_\mathbf{k}(\hat{\vx})|}_{\le 1}\\
&\leq \|\vx-\hat{\vx}\| + 2^d \sqrt{d} \|\vx-\hat{\vx}\| = (1+2^d \sqrt{d}) \|\vx-\hat{\vx}\|
\end{aligned}$$
where $2^d$ comes from the fact at most $2^d$ grid cubes overlap at each point $\mathbf{x}$.
Third, using the same argument, we can show that $\varepsilon(\vx)\cF(\vx)$ is globally Lipschitz over $\sX$, and the Lipschitz constant is bounded by $\left(3\sqrt{d} + 1 + 2^d \sqrt{d}\right)$ (Recall that $\mathrm{diam}(\cC_\vk) = 3\sqrt{d}$), which finishes the proof.
\end{proof}
\subsection{Proof of necessity}
For Theorem \ref{thm:lip-2}, we adopt a similar idea: first considering a bounded domain and then extending the results to unbounded domains.
\begin{theorem}
\label{thm:lip-2-bdd}
Let $\sX\subset\sR^d$ be a \emph{bounded} domain and let $\cG:\sR^n\times\sX\to\sR^n$ be \emph{regular}. Then, for every $\vx\in\sX$, the map $\vy\mapsto \cG(\vy,\vx)$ has a unique fixed point $\vy_\ast(\vx)$, and the resulting fixed-point map $\vy_\ast(\vx)$ must be locally Lipschitz on $\sX$.
\end{theorem}
\begin{proof}%[Proof of Theorem \ref{thm:lip-2}]
Let $\overline{\sX}$ be the closure of $\sX$. In this proof, we will first extend the operator $\cG$ to $\sR^n \times \overline{\sX}$, and then analyze its properties on this closed domain.
\textbf{Step 1: Extension to $\overline{\sX}$.} For any $\vy \in \sR^n$, $\cG(\vy,\vx)$ is globally Lipschitz continuous on $\sX$, hence its extension is naturally define by
\[ \cG(\vy,\bar \vx) := \lim_{\sX\ni \vx\to\bar \vx}\cG(\vy, \vx) ,\quad \textup{for all }\bar \vx \in \overline{\sX} \setminus \sX. \]
Different from the proof of Theorem \ref{thm:eps} where $\cF$ might be not locally Lipschitz at $\bar \vx$ even if it is continuous at $\bar \vx$, here the extended $\cG$ must be Lipschitz at $\bar \vx$ and hence Lipschitz on the overall set $\overline{\sX}$. This can be verified by examining the difference quotient for $\vx_1 \neq \vx_2$ and $\vy \in \sR^n$:
\[ \Delta \cG[\vy; \vx_1,\vx_2]:= \frac{\|\cG(\vy,\vx_1)-\cG(\vy,\vx_2)\|}{\|\vx_1-\vx_2\|} \]
Let $\cG(\vy,\cdot)$'s Lipschitz constant on $\sX$ be $L(\vy):= \sup_{\vx_1\neq\vx_2 \in \sX}\Delta \cG[\vy; \vx_1,\vx_2]$. For any $\vx_1 \in \sX$ and $\bar{\vx}_2 \in \overline{\sX}\setminus\sX$, it holds that
\[
\Delta \cG[\vy; \vx_1,\bar{\vx}_2] = \lim_{\sX\ni \vx_2\to \bar{\vx}_2}\Delta \cG[\vy; \vx_1,\vx_2] \leq \sup_{\vx_2 \in \sX:\vx_2 \neq \vx_1}\Delta \cG[\vy; \vx_1,\vx_2] \leq L(\vy)
\]
For any $\bar{\vx}_1 \neq \bar{\vx}_2 \in \overline{\sX}\setminus\sX$, we have
\[
\Delta \cG[\vy; \bar{\vx}_1,\bar{\vx}_2] = \lim_{\sX\ni \vx_1\to \bar{\vx}_1}\lim_{\sX\ni \vx_2\to \bar{\vx}_2}\Delta \cG[\vy; \vx_1,\vx_2] \leq \sup_{\vx_1,\vx_2 \in \sX:\vx_2 \neq \vx_1}\Delta \cG[\vy; \vx_1,\vx_2] = L(\vy)
\]
Therefore, we obtain an upper bound for $\cG(\vy,\cdot)$'s Lipschitz constant on $\overline{\sX}$:
\begin{align*}
& \sup_{\vx_1\neq\vx_2\in\overline{\sX}} \Delta \cG[\vy; \vx_1,\vx_2] \\
=& \max\left(
\sup_{\vx_1\neq\vx_2 \in \sX} \Delta \cG[\vy; \bar{\vx}_1,\bar{\vx}_2],
\sup_{\vx_1\in\sX,\vx_2\in\overline{\sX}\setminus\sX} \Delta \cG[\vy; \vx_1,\bar{\vx}_2],
\sup_{\vx_1\neq\vx_2 \in \overline{\sX}\setminus\sX} \Delta \cG[\vy; \bar{\vx}_1,\bar{\vx}_2]
\right) \\
\leq & \max\left( L(\vy),L(\vy),L(\vy) \right) = L(\vy)
\end{align*}
That is, for any $\vy \in \sR^n$, $\cG(\vy,\cdot)$ is globally Lipschitz on $\overline{\sX}$, and the Lipschitz constant is the same as that of $\sX$.
On the other hand, let's consider the Lipschitz constant (contraction modulus) w.r.t. $\vy$ when fixing $\bar \vx \in \overline{\sX}\setminus\sX$:
\[\mu(\bar\vx) = \lim_{\sX \ni \vx\to \bar\vx} \mu(\vx)\]
Since $0 < \mu(\vx) < 1$ for $\vx \in \sX$, by taking limit, we have $0 \leq \mu(\bar\vx) \leq 1$. For those $\bar\vx$ with $\mu(\bar\vx) < 1$, the operator $\cG(\cdot,\bar\vx)$ is still contractive. But if $\mu(\bar\vx) =1$, the operator $\cG(\cdot,\bar\vx)$ is not contractive.
\textbf{Step 2: Defining $\sD$ and $\sD_r$.} We collect all points $\vx\in \overline{\sX}$ where the operator $\cG(\cdot,\vx)$ is not contractive:
\[ \sD:=\left\{ \vx \in \overline{\sX}: \mu(\vx)=1 \right\} \]
and define a ``safe" set that is sufficiently far from $\sD$:
\[\sD_r:= \big\{ \vx \in \overline{\sX}: d(\vx, \sD) \geq r \big\}.\]
Note that $\overline{\sX}\setminus\sD = \bigcup_{r > 0} \sD_r$ and $\sX \subset \overline{\sX}\setminus\sD$. We obtain
\[\sX \subset \bigcup_{r > 0} \sD_r.\]
For any $\sD_r$ with $r > 0$, we can obtain a uniform contraction modulus of the operator $\cG(\cdot,\vx)$: There is a constant $\mu_r \in (0,1)$ such that
\begin{equation}
\label{eq:local-contract}
\|\cG(\vy_1, \vx) - \cG(\vy_2,\vx)\| \leq \mu_r \|\vy_1 - \vy_2\|
\end{equation}
for all $\vy_1,\vy_2 \in \sR^n$ and $\vx \in \sD_r$, which follows immediately from the continuity of $\mu(\vx)$ and the compactness of $\sD_r$. By the Banach fixed-point theorem, the operator $\cG(\cdot,\vx)$ must have a unique fixed point $\vy_\ast$ for each $\vx\in\sD_r$.
To complete the proof of Theorem \ref{thm:lip-2}, thanks to the fact that $\sX \subset \bigcup_{r > 0} \sD_r$, it's enough to show that: For any $\sD_r$ with $r > 0$, there is a constant $C_r$ such that
\begin{equation}
\label{eq:local-lip}
\| \vy_\ast(\vx_1) - \vy_\ast(\vx_2) \| \leq C_r \| \vx_1 - \vx_2 \|
\end{equation}
holds for all $\vx_1, \vx_2 \in \sD_r$. In the following steps, we will show \eqref{eq:local-lip}.
\textbf{Step 3: A controllable sequence.} Fix $\vx \in \sD_r$.
By defining a sequence $\{\vy_k(\vx)\}_{k \geq 0} \subset \sR^n$:
\[ \vy_{k+1}(\vx) = \cG(\vy_{k}(\vx),\vx), \quad \vy_0 \textup{ is constant for all }\vx, \]
we are able to estimate the upper bound of $\|\vy_\ast(\vx)\|$. In particular, we decompose $\vy_0 - \vy_\ast$ by a series:
\[ \vy_0 - \vy_\ast = \lim_{k \to \infty} (\vy_0 - \vy_k) = \sum_{k=0}^{\infty} (\vy_k - \vy_{k+1}) \]
Thanks to \eqref{eq:local-contract}, we have
\begin{align*}
\|\vy_k(\vx) - \vy_{k+1}(\vx)\| \leq \mu_r \|\vy_{k-1}(\vx) - \vy_{k}(\vx)\| \cdots \leq \mu_r^k \|\vy_0 - \vy_{1}(\vx)\| = \mu_r^k \|\vy_0 - \cG(\vy_0,\vx)\|
\end{align*}
for all $\vx \in \sD_r$. Therefore, it holds that
\begin{align*}
\|\vy_0 - \vy_\ast(\vx)\|
\leq& \sum_{k=0}^{\infty} \|\vy_k(\vx) - \vy_{k+1}(\vx)\| \\
\leq& \left(\sum_{k=0}^{\infty} \mu_r^k \right) \|\vy_0 - \cG(\vy_0,\vx)\|
= \frac{1}{1-\mu_r} \|\vy_0 - \cG(\vy_0,\vx)\|
\end{align*}
Now we can conclude the boundedness of $\|\vy_\ast(\vx)\|$ for $\vx \in \sD_r$ by the compactness of $\sD_r$:
\[ \|\vy_\ast(\vx)\| \leq \underbrace{\|\vy_0\| + \frac{1}{1-\mu_r} \sup_{\vx \in \sD_r}\|\vy_0 - \cG(\vy_0,\vx)\|}_{\textup{defined as $M_r \geq 0$.}} \]
With the same argument, we have $\|\vy_k(\vx)\| \leq M_r$ for all $k \geq 0$ and $\vx \in \sD_r$. It implies that
\[ L(\vy_k(\vx)) \leq L_1 + L_2 M_r \]
for some $L_1,L_2>0$ as $L(\vy)$ grows linearly w.r.t. $\|\vy\|$.
Consequently, we can estimate an upper bound for the Lipschitz constant of $\vy_k(\vx)$. In particular, for $\vx_1,\vx_2 \in \sD_r$, it holds that
\begin{align*}
& \| \vy_{k+1}(\vx_1) - \vy_{k+1}(\vx_2) \|\\
= & \| \cG(\vy_{k}(\vx_1),\vx_1) - \cG(\vy_{k}(\vx_2),\vx_2) \| \\
= & \| \cG(\vy_{k}(\vx_1),\vx_1) - \cG(\vy_{k}(\vx_2),\vx_1) + \cG(\vy_{k}(\vx_2),\vx_1) - \cG(\vy_{k}(\vx_2),\vx_2) \| \\
\leq & \| \cG(\vy_{k}(\vx_1),\vx_1) - \cG(\vy_{k}(\vx_2),\vx_1)\| + \| \cG(\vy_{k}(\vx_2),\vx_1) - \cG(\vy_{k}(\vx_2),\vx_2) \|\\
\leq & \mu_r \| \vy_{k}(\vx_1) - \vy_{k}(\vx_2)\| + (L_1 + L_2 M_r) \| \vx_1 - \vx_2 \|
\end{align*}
For simplicity, let $L_r:=L_1 + L_2 M_r$, $a_k := \| \vy_{k}(\vx_1) - \vy_{k}(\vx_2)\|$, and $h := \| \vx_1 - \vx_2 \|$. By recursively applying $a_{k+1} \leq \mu_r a_{k} + Lh$ and $a_0 = 0$, we have
\[ \| \vy_{k}(\vx_1) - \vy_{k}(\vx_2)\| = a_k \leq (\mu_r)^k a_0 + (\mu_r^{k-1} + \cdots + \mu_r + 1)L_r h \leq \frac{1}{1-\mu_r}L_r h = \frac{L_r}{1-\mu_r} \| \vx_1 - \vx_2 \|.\]
\textbf{Step 4: Final proof.}
As $\cG(\cdot,\vx)$ is contractive in $\vy$ for any $\vx \in \sD_r$, it holds that $\vy_k(\vx) \to \vy_\ast(\vx)$ for any $\vx \in \sD_r$. (Here, as for the ``convergence," we mean the pointwise convergence, which is enough here. We don't need stronger conditions like the uniform convergence.) For the above $\vx_1,\vx_2$, there is a $K$ such that
\[ \|\vy_k(\vx_1) - \vy_\ast(\vx_1)\| \leq \frac{L_r}{1-\mu_r} \| \vx_1 - \vx_2 \|, \quad \|\vy_k(\vx_2) - \vy_\ast(\vx_2)\| \leq \frac{L_r}{1-\mu_r} \| \vx_1 - \vx_2 \| \]
for $k \geq K$. Combining the above results, we obtain
\begin{align*}
\| \vy_\ast(\vx_1) - \vy_\ast(\vx_2) \|
\leq & \|\vy_\ast(\vx_1) - \vy_k(\vx_1)\| + \| \vy_{k}(\vx_1) - \vy_{k}(\vx_2)\| + \|\vy_k(\vx_2) - \vy_\ast(\vx_2)\| \\
\leq & \frac{3 L_r}{1-\mu_r} \| \vx_1 - \vx_2 \|
\end{align*}
By letting $C_r = 3L_r / (1-\mu_r)$, we get \eqref{eq:local-lip}, which completes the proof.
\end{proof}
\begin{remark}
Our result relaxes two uniformity requirements in \cite[Thm. 1A.4]{dontchev2009implicit}: (i) the contraction modulus $\mu(\vx)$ is allowed to vary with $\vx$ (it only needs to be continuous in $\vx$), rather than being a single global constant; and (ii) for each $\vy$, the mapping $\vx \mapsto \cG(\vy,\vx)$ is Lipschitz on $\sX$ with a constant that may grow linearly in $\|\vy\|$, instead of being uniformly bounded in $\vy$. Because these bounds are not uniform, we conclude only local (as opposed to global) Lipschitz continuity of the fixed-point map $\vx \mapsto \vy_\ast(\vx)$ on $\sX$.
\end{remark}
Now we relax the condition in Theorem \ref{thm:lip-2-bdd} to unbounded domains and prove Theorem \ref{thm:lip-2} based on Theorem \ref{thm:lip-2-bdd}.
\begin{proof}[Proof of Theorem \ref{thm:lip-2}] We cover the domain $\sR^d$ using the grid $\vk = (2k_1,\ldots,2k_d)$ for $k_i\in \sZ$, defining closed cubic regions $\cC_\vk$ of side length 3 centered at each $\vk$:
$$
\cC_\vk =\left[ 2k_1-\frac{3}{2}, 2k_1+\frac{3}{2} \right]\times\left[ 2k_2-\frac{3}{2}, 2k_2+\frac{3}{2} \right]\times \cdots \times \left[ 2k_d-\frac{3}{2}, 2k_d+\frac{3}{2} \right].
$$
By applying Theorem \ref{thm:lip-2-bdd} to the bounded set $\cC_\vk\cap \sX$, we guarantee the existence of a unique fixed-point map $\vy_{\vk,*}: \cC_\vk\cap \sX \to \sR^n$ which is locally Lipschitz continuous on its domain.
Consider any $\vx$ in the intersection of two regions $\cC_\vk \cap \cC_{\vk'}$. Since $\cG(\cdot,\vx)$ is a contraction, it admits a unique fixed point in $\sR^n$. Therefore, the local solutions must coincide:
$$\vy_{\vk,*}(\vx)= \vy_{\vk',*}(\vx).$$
This consistency allows us to define a global fixed-point map $\vy_*: \sX \to \sR^n$ by setting $\vy_*(\vx) = \vy_{\vk,*}(\vx)$ for any $\vk$ such that $\vx \in \cC_\vk$. Since $\vy_*$ coincides with a locally Lipschitz function $\vy_{\vk,*}$ on every compact neighborhood $\cC_\vk$, $\vy_*$ is locally Lipschitz continuous on $\sX$. \end{proof}
\section{A variant architecture}
\label{sec:app-general}
In practice, many works use a variant of the vanilla model $\vy_\ast=\cG(\vy_\ast,\vx)$:
\begin{equation}
\label{eq:imp-pq}
\vz_\ast = \cG(\vz_\ast,\cQ_1(\vx)),\quad \vy_\ast=\cQ_2(\vz_\ast)
\end{equation}
where $\cG$ is the core implicit model, $\cQ_1$ is a encoding network and $\cQ_2$ is a decoding (readout).
At inference, one iterates $\vz_{t}=\cG(\vz_{t-1},\cQ_1(\vx))$ for $1 \leq t \leq T$ and finally $\vy_T = \cQ_{2}(\vz_T)$. This often improves empirical performance but does not alter the expressivity in Theorems~\ref{thm:lip}–\ref{thm:lip-2}.
\begin{corollary}
\label{coro:imp-pq}
Under Assumption~\ref{asmp:f}, for any $\cF$ there exists a {regular} implicit operator $\cG$ and globally Lipschitz maps $\cQ_1,\cQ_2$ such that $\cQ_2\left(\mathrm{Fix}\big(\cG(\cdot,\cQ_1(\vx))\big)\right)=\cF(\vx)$ for all $\vx\in\sX$. Conversely, for any {regular} implicit operator $\cG$ any globally Lipschitz $\cQ_1,\cQ_2$, the fixed point $\vz_\ast$ defined by \eqref{eq:imp-pq} exists uniquely and the induced map $\vx \mapsto \vy_\ast$ must be locally Lipschitz on $\sX$.
\end{corollary}
\begin{proof}
The claim follows directly from Theorems~\ref{thm:lip}–\ref{thm:lip-2}.
\emph{Sufficiency.} Given any locally Lipschitz target $\cF$ on $\sX$, Theorem~\ref{thm:lip} ensures the existence of a regular $\cG$ whose fixed-point map equals $\cF$. Taking $\cQ_1,\cQ_2$ as both identity maps recovers the sufficiency statement with globally Lipschitz $\cQ_1,\cQ_2$.
\emph{Necessity.} Suppose $\cG$ is regular and $\cQ_1,\cQ_2$ are globally Lipschitz. Then the composite update $\cG(\vz,\cQ_1(\vx))$ is still regular in $\vz$ and $\vx$. By Theorem~\ref{thm:lip-2}, for every $\vx\in\sX$, there is a unique fixed point $\vz_\ast(\vx)$ and the map $\vx \mapsto \vz_\ast(\vx)$ is locally Lipschitz on $\sX$. Finally, applying the globally Lipschitz readout $\cQ_2$ preserves local Lipschitz continuity, so $\vx \mapsto \vy_\ast$ is locally Lipschitz as claimed.
The proof is finished.
\end{proof}
\section{Proofs of Theorems for Inverse Problems}
\label{sec:app-inv}
This section proves that the target solution mappings, $\cF_{1a}$ and $\cF_{1b}$, are single-valued and locally Lipschitz on their domain, as stated in Theorems \ref{thm:inverse-lip-1a} and \ref{thm:inverse-lip-1b}. Before the proofs, we first provide some definitions that used in Assumption \ref{assume:M}.
Given a close subset $\sM \subset \sR^n$, its \textit{reach} $\tau$ is defined in \cite{federer1959curvature}:
\begin{align*}
\tau := \sup \{ r>0: & \forall \vy \in \sR^n \textup{ with dist}(\vy,\sM)<r, \\
& \textup{there exists a unique }\vz \in \sM \textup{ such that }\|\vy - \vz\| = \textup{dist}(\vy,\sM) \}.
\end{align*}
A set with positive reach is also called a ``prox-regular" set in the literature \cite{poliquin2000local}.
The bi-Lipschitz condition refers to: for some $0 < \mu \leq L < +\infty$, it holds that
\begin{equation}
\label{eq:JL-condition}
\mu \|\vy_1-\vy_2\| \leq \|\mA \vy_1 - \mA \vy_2\| \leq L \|\vy_1-\vy_2\| \quad \forall \vy_1,\vy_2 \in \sM.
\end{equation}
According to the definition, it holds that $0 < \mu \leq L \leq \sigma_{\textup{max}} < +\infty$.
This condition ensures $\mA$ can be viewed as an injective mapping when restricted to $\sM$, which is important for the recovery guarantee.
\textit{Remark for Assumption \ref{assume:M}.} The assumption that data (particularly images) lies on a smooth manifold has a long and influential history \cite{roweis2000nonlinear,donoho2005image}, and it is still widely used in recent literature. The compactness of the data manifold can be achieved by standard techniques like normalization. In addition, reach is an important concept for manifold to ensure the uniqueness of its projection \cite{federer1959curvature,aamari2019estimating}. The overall assumptions on manifolds, smoothness, compactness, and positive reach, is typically used in recent literature regarding image and signal processing \cite{tang2024adaptivity,potaptchik2024linear,azangulov2024convergence}.
The on-manifold bi-Lipschitz condition does \emph{not} require $\mA$ to be globally invertible; it merely rules out ill-posedness \emph{restricted to} $\sM$.
This is closely related to Johnson–Lindenstrauss (JL)–type embeddings in compressive sensing: e.g., \cite{baraniuk2009random} shows that random matrices are bi-Lipschitz on low-dimensional manifolds with high probability, and JL-style conditions are widely analyzed and used \cite{candes2006near,clarkson2008tighter,wakin2010manifold,iwen2013approximation,hegde2012signal}.
\begin{proof}[Proof of Theorem \ref{thm:inverse-lip-1a}]
For simplicity, we first denote the objective functions in \eqref{eq:mapping-1a} as $F_{1a}(\vy)$:
\[
F_{1a}(\vy):= \frac{1}{2}\|\vx - \mA \vy\|^2 + \frac{\alpha}{2} \textup{dist}^2(\vy,\sM)
\]
Then we introduce some definitions that will be useful in our proof:
\[ \sU_r(\sM) := \{\vy \in \sR^n: \textup{dist}(\vy,\sM) < r\}, \quad \overline{\sU}_r(\sM) := \{\vy \in \sR^n: \textup{dist}(\vy,\sM) \leq r\} \]
Here, $\sU_r(\sM)$ is an open tubular neighborhood of the manifold $\sM$ and $\overline{\sU}_r(\sM)$ is its closure. As $r = \tau$, the open set $\sU_r(\sM)$ is named as the \textit{reach tube} of $\sM$, denoted as $\sU_{\tau}(\sM)$. As introduced in \cite{federer1959curvature}, within the reach tube, some nice properties of the distance function and projection mapping can be utilized. For any $\vy \in \sU_{\tau}(\sM)$ or any $\vy \in \overline{\sU}_r(\sM)$ with $r < \tau$, the projection mapping
\[\vp(\vy):=\argmin_{\vz \in \sM} \|\vz-\vy\|\]
is single valued and well defined, and $\textup{dist}(\vy,\sM)=\|\vy - \vp(\vy)\|$.
\textbf{Step 1: Existence of minimizers of $F_{1a}$.} As $\vx \in \sX$, there must be an underlying $\vy_\ast \in \sM$ (hence $\vy_\ast \in \overline{\sU}_r(\sM)$) and $\vn$ such that $\|\vx - \mA\vy_\ast\| = \|\vn\|$. Therefore, it holds that
\[F_{1a}(\vy_\ast) = \frac{1}{2}\|\vx - \mA \vy_\ast\|^2 + \frac{\alpha}{2} \textup{dist}^2(\vy_\ast,\sM) = \frac{1}{2} \|\vn\|^2 + 0 = \frac{1}{2} \|\vn\|^2 \]
In the other hand, for any point outside the tube: $\vy \not\in \overline{\sU}_r(\sM)$, the objective value is lower bounded by:
\[ F_{1a}(\vy) \geq 0 + \frac{\alpha}{2} \textup{dist}^2(\vy,\sM) > \frac{\alpha}{2} r^2 \]
As long as we have large enough $\alpha$:
\begin{equation}
\label{eq:proof-alpha}
\alpha \geq \frac{\|\vn\|^2}{r^2},
\end{equation}
we can ensure $F_{1a}(\vy)>F_{1a}(\vy_\ast)$ for all $\vy \not\in \overline{\sU}_r(\sM)$, which implies $\inf_{\vy\in\sR^n}F_{1a}(\vy)=\inf_{\vy \in \overline{\sU}_r(\sM)}F_{1a}(\vy)$. As $\sM$ is compact, $\overline{\sU}_r(\sM)$ must be compact as well. Consequently, the infimum of $F$ is attainable, which concludes the existence of the minimizer of $F_{1a}$, denoted by $\hat{\vy}$, and $\hat{\vy} \in \overline{\sU}_r(\sM)$. Finally, we have the conclusion: {It holds for all $r > 0$ that, condition \eqref{eq:proof-alpha} ensures the existence of $\hat{\vy}$ and $\hat{\vy} \in \overline{\sU}_r(\sM)$.}
\textbf{Step 2: Bound of minimizers of $F_{1a}$.} For any $\vy \in \sU_{\tau}(\sM)$, the projection $\vp(\vy)$ is uniquely defined, hence we have
\[
\begin{aligned}
\Big\|\mA\vy - \vx\Big\|
= \Big\|\mA\vy - \mA\vy_\ast - \vn\Big\|
= & \Big\|\mA\vy - \mA \vp(\vy) + \mA \vp(\vy) - \mA\vy_\ast - \vn\Big\| \\
\geq & \Big\|\mA \vp(\vy) - \mA\vy_\ast \Big\| - \Big\|\mA\vy - \mA \vp(\vy) \Big\| - \|\vn\| \\
\geq & \mu \|\vp(\vy) - \vy_\ast \| - \sigma_{\textup{max}} \|\vy - \vp(\vy) \| - \|\vn\|
\end{aligned}
\]
According to the conclusion in Step 1, as long as
\begin{equation}
\label{eq:proof-alpha-2}
\alpha \geq \frac{\|\vn\|^2}{r^2} > \frac{\|\vn\|^2}{\tau^2},
\end{equation}
it holds that the minimizer $\hat{\vy}$ exists and $\hat{\vy} \in \overline{\sU}_r(\sM)$ for some $r < \tau$ and hence $\hat{\vy} \in \sU_\tau(\sM)$, which allows us to use the above inequalities at the beginning of Step 2. Now we aim to establish an upper bound for $\|\vp(\hat{\vy}) - \vy_\ast \|$ by contradiction. Suppose
\[ \mu \|\vp(\hat{\vy}) - \vy_\ast \| > \sigma_{\textup{max}} \|\hat{\vy} - \vp(\hat{\vy}) \| + 2 \|\vn\|\]
we will obtain
\[\|\mA\hat{\vy} - \vx\| \geq \mu \|\vp(\hat{\vy}) - \vy_\ast \| - \sigma_{\textup{max}} \|\hat{\vy} - \vp(\hat{\vy}) \| - \|\vn\| > \|\vn\|,\]
which implies
\[ F_{1a}(\hat{\vy}) = \frac{1}{2} \|\mA\hat{\vy} - \vx\|^2 + \frac{\alpha}{2} \textup{dist}^2(\hat{\vy},\sM) > \frac{1}{2} \|\vn\|^2 + 0 = F_{1a}(\vy_\ast). \]
This contradicts with the definition of $\hat{\vy}$: the minimizer of function $F_{1a}$. Therefore, we obtain:
\[
\mu \|\vp(\hat{\vy}) - \vy_\ast \| \leq \sigma_{\textup{max}} \|\hat{\vy} - \vp(\hat{\vy}) \| + 2 \|\vn\| \leq \sigma_{\textup{max}} r + 2 \|\vn\|
\]
which is equivalent to
\[
\|\vp(\hat{\vy}) - \vy_\ast \| \leq \frac{\sigma_{\textup{max}}}{\mu} r + \frac{2}{\mu} \|\vn\|
\]
and implies that
\begin{equation}
\label{eq:proof-error-bound}
\|\hat{\vy} - \vy_\ast\| \leq \|\hat{\vy} - \vp(\hat{\vy}) \| + \|\vp(\hat{\vy}) - \vy_\ast \| \leq \left(1 + \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\|
\end{equation}
holds for all $\hat{\vy}$ that minimizes $F_{1a}(\vy)$.
\textbf{Step 3: Positive definiteness of the Hessian of $F_{1a}$.}
To prove the uniqueness of the solution, we will establish the strict convexity of the objective function $F_{1a}(\vy)$ within a neighborhood around any point of $\sM$. To achieve this, we establish the positive definiteness of the Hessian of $F_{1a}(\vy)$ in this step.
For any $\vy \in \sU_{\tau}(\sM)$, the projection mapping is single valued and the objective function can be written as \[F_{1a}(\vy) = \frac{1}{2} \underbrace{\|\vx - \mA \vy\|^2}_{f(\vy)} + \frac{\alpha}{2}\underbrace{\|\vy - \vp(\vy)\|^2}_{g(\vy)} \]
The smoothness of $\sM$ implies the smoothness of $g$ and of the projection mapping, and hence we can take first and second orders of derivatives on $g$ \cite[Theorem 2]{leobacher2021existence}. Thanks to \cite[Theorem 4.8]{federer1959curvature}, the gradient and Hessian of $g$ are given by
\[ \nabla g(\vy) = 2 \left( \vy - \vp(\vy) \right), \quad \nabla^2 g(\vy) = 2 \left( \mI - D \vp(\vy) \right), \]
where $D \vp$ denotes the Jacobian of the projection mapping. The overall Hessian of $F_{1a}$ is provided by
\begin{equation}
\label{eq:proof-hessian}
\nabla^2 F_{1a}(\vy) = \mA^\top\mA + \alpha \left( \mI - D \vp(\vy) \right).
\end{equation}
To further present the properties of the above Hessian, we introduce a space decomposition according to $\vp(\vy)$:
\[\sR^n = \sT_{\vp(\vy)}(\sM) \oplus \sN_{\vp(\vy)}(\sM)\]
where $\sT_{\vp(\vy)}(\sM)$ denotes the tangent space of $\sM$ at the point $\vp(\vy) \in \sM$, and $\sN_{\vp(\vy)}(\sM)$ represents the normal space. According to \cite[Theorem C and Definition 7]{leobacher2021existence}, the matrix $D \vp(\vy)$ is actually restricted to the tangent space. In other words, for any decomposition $\vh$ with $\vh = \vh_{\mathrm{T}} + \vh_{\mathrm{N}}$ where $\vh_{\mathrm{T}} \in \sT_{\vp(\vy)}(\sM)$ and $\vh_{\mathrm{N}} \in \sN_{\vp(\vy)}(\sM)$, it holds that
\begin{equation}
\label{eq:proof-Dp-block-diag}
D \vp(\vy) \vh_{\mathrm{N}} = \vzero, \quad D \vp(\vy) \vh_{\mathrm{T}} \in \sT_{\vp(\vy)}(\sM).
\end{equation}
In addition, function $g(\vy)$ is $(\frac{s}{\tau - s})$-weakly convex where $\tau$ is the reach of $\sM$ and $s=\textup{dist}(\vy,\sM)$ \cite[Section 5]{nacry2022distance}, and hence the spectrum of $\nabla^2 g$ can be lower bounded by
\begin{equation}
\label{eq:proof-weak-convex}
\langle \vh_{\mathrm{T}}, \nabla^2 g(\vy) \vh_{\mathrm{T}} \rangle \geq - \frac{2 s}{\tau - s} \|\vh_{\mathrm{T}} \|^2,
\end{equation}
Now, let's turn to the first term in the Hessian: $\mA^\top\mA$. It can be shown using the JL condition \eqref{eq:JL-condition} that, the spectrum of $\mA^\top\mA$ restricted to the tangent space can also be lower bounded. In particular, we pick an arbitrary tangent vector $\vh_{\mathrm{T}} \in \sT_{\vp(\vy)}(\sM)$. According to the definition of tangent space, there must be a curve $\gamma: (-\delta, \delta) \to \sM$ with $\delta>0$, $\gamma(0) = \vp(\vy)$, and $\gamma^\prime(0) = \vh_{\mathrm{T}}$. For any $0 \leq t < \delta$, $\gamma(t) \in \sM$. By applying condition \eqref{eq:JL-condition} with the pair $(\gamma(t),\gamma(0))$ and divide by $t^2$, we have
\[ \mu^2 \frac{\|\gamma(t)-\gamma(0)\|^2}{t^2} \leq \frac{\|\mA \gamma(t) - \mA\gamma(0)\|^2}{t^2} \leq L^2 \frac{\|\gamma(t)-\gamma(0)\|^2}{t^2} \]
By differentiability and the continuity of the operator $\mA$, it holds that
\[\lim_{t \to 0} \frac{\gamma(t)-\gamma(0)}{t} = \vh_{\mathrm{T}} , \quad \lim_{t \to 0} \frac{\mA\gamma(t)-\mA\gamma(0)}{t} = \mA\vh_{\mathrm{T}}\]
which implies
\begin{equation}
\label{eq:proof-A-local-bound}
\mu^2 \|\vh_{\mathrm{T}} \|^2 \leq \| \mA\vh_{\mathrm{T}} \|^2 \leq L^2 \| \vh_{\mathrm{T}} \|^2.
\end{equation}
Combining \eqref{eq:proof-hessian}, \eqref{eq:proof-Dp-block-diag}, \eqref{eq:proof-weak-convex}, and \eqref{eq:proof-A-local-bound}, we have
\begin{align*}
& \langle \vh, \nabla^2 F_{1a}(\vy) \vh \rangle \\
= & \underbrace{\langle \vh_{\mathrm{T}}, \mA^\top\mA \vh_{\mathrm{T}} \rangle}_{\geq \mu^2 \|\vh_{\mathrm{T}} \|^2}
+ 2 \langle \vh_{\mathrm{T}}, \mA^\top\mA \vh_{\mathrm{N}} \rangle
+ \underbrace{\langle \vh_{\mathrm{N}}, \mA^\top\mA \vh_{\mathrm{N}} \rangle}_{\geq 0} \\
& + \alpha \underbrace{\langle \vh_{\mathrm{T}}, \left( \mI - D \vp(\vy) \right) \vh_{\mathrm{T}} \rangle}_{\geq - \frac{s}{\tau - s} \|\vh_{\mathrm{T}} \|^2 }
+ 2 \alpha \underbrace{ \langle \vh_{\mathrm{T}}, \left( \mI - D \vp(\vy) \right) \vh_{\mathrm{N}} \rangle}_{=\langle \vh_{\mathrm{T}}, \vh_{\mathrm{N}} \rangle = 0 }
+ \alpha \underbrace{ \langle \vh_{\mathrm{N}}, \left( \mI - D \vp(\vy) \right) \vh_{\mathrm{N}} \rangle}_{=\|\vh_{\mathrm{N}} \|^2} \\
\geq & \left(\mu^2 - \alpha \frac{s}{\tau - s} \right) \|\vh_{\mathrm{T}} \|^2 + \alpha \|\vh_{\mathrm{N}} \|^2 - 2 \|\mA\vh_{\mathrm{T}}\| \cdot \|\mA\vh_{\mathrm{N}}\| \\
\geq & \left(\mu^2 - \alpha \frac{s}{\tau - s} \right) \|\vh_{\mathrm{T}} \|^2 + \alpha \|\vh_{\mathrm{N}} \|^2 - 2 L \|\vh_{\mathrm{T}}\| \cdot \sigma_{\textup{max}} \|\vh_{\mathrm{N}}\| \\
= & \begin{bmatrix}
\|\vh_{\mathrm{T}}\| & \|\vh_{\mathrm{N}}\|
\end{bmatrix}
\begin{bmatrix}
\mu^2 - \alpha \frac{s}{\tau - s} & - \sigma_{\textup{max}} L \\
- \sigma_{\textup{max}} L & \alpha
\end{bmatrix}
\begin{bmatrix}
\|\vh_{\mathrm{T}}\| \\ \|\vh_{\mathrm{N}}\|
\end{bmatrix}
\end{align*}
Therefore, to ensure $\langle \vh, \nabla^2 F_{1a}(\vy) \vh \rangle > 0$ for any $\vh \neq \vzero$, it's enough to ensure the $2\times 2$ matrix to be positive definite:
\begin{equation}
\label{eq:proof-param-strict-convex}
\mu^2 - \alpha \frac{s}{\tau - s} > 0 \quad \textup{and} \quad \alpha \left(\mu^2 - \alpha \frac{s}{\tau - s}\right) - \sigma^2_{\textup{max}}L^2 > 0.
\end{equation}
In other words, \eqref{eq:proof-param-strict-convex} will guarantee the positive definiteness of $\nabla^2 F_{1a}(\vy)$ for all $\vy \in \overline{\sU}_s(\sM)$ and any $s < \tau$.
\textbf{Step 4: Uniqueness of minimizers of $F_{1a}$.} In this step, we will combine the results from Steps 2 and 3. Then we are able to prove that the objective function $F_{1a}(\vy)$ is strictly convex in a neighborhood of its minimizers, which implies the uniqueness of the minimizer. To achieve this, it's enough to ensure
\begin{equation}
\label{eq:proof-error-bound-2}
\|\hat{\vy} - \vy_\ast\| \leq s
\end{equation}
for all $\hat{\vy} \in \argmin_{\vy} F_{1a}(\vy)$, where $s$ satisfies \eqref{eq:proof-param-strict-convex}. With this condition \eqref{eq:proof-error-bound-2}, it holds that
\[ \hat{\vy} \in \sB(\vy_\ast, s) \subset \overline{\sU}_s(\sM). \]
Along with the fact that $\sB(\vy_\ast, s)$ is convex and that $\nabla^2 F_{1a}(\vy)$ is positive definite for all $\vy \in \overline{\sU}_s(\sM)$, $F_{1a}$ is strictly convex within $\sB(\vy_\ast, s)$ \cite[Section 3.1.4]{boyd2004convex}. As all minimizers of the strict convex function belong to this convex set, $\sB(\vy_\ast, s)$, the minimizer $\hat{\vy}$ must be unique.
Now the question is: How to guarantee \eqref{eq:proof-error-bound-2}? According to \eqref{eq:proof-error-bound}, Condition \eqref{eq:proof-alpha-2} along with
\begin{equation}
\label{eq:proof-param}
\left(1 + \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\| \leq s
\end{equation}
can guarantee \eqref{eq:proof-error-bound-2}. Finally, it's enough to choose $\alpha$, $s$, and $r$ such that \eqref{eq:proof-alpha-2}, \eqref{eq:proof-param-strict-convex}, and \eqref{eq:proof-param} are satisfied together. In particular, we choose
\[ s = \frac{4}{\mu} \|\vn\|, \quad r = \frac{1}{\sigma_{\textup{max}}} \|\vn\| , \quad \alpha = \frac{2\sigma^2_{\textup{max}}L^2}{\mu^2}\]
where $\alpha$ merely depends on $\mA$ and $\sM$ but is independent of $\vx$. Such a parameter choice implies \eqref{eq:proof-param}:
\[ \left(1 + \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\| \leq 2 \frac{\sigma_{\textup{max}}}{\mu} r + \frac{2}{\mu} \|\vn\| = \frac{2}{\mu} \|\vn\| + \frac{2}{\mu} \|\vn\| = s. \]
As $\|\vn\| < \frac{1}{20}\frac{\mu^5}{\sigma^2_{\textup{max}}L^2} \tau$, it holds that
\[ s = \frac{4}{\mu} \|\vn\| < \frac{\mu^4}{5 \sigma^2_{\textup{max}}L^2} \tau
~~ \implies ~~
\frac{s}{\tau-s}
< \frac{\frac{\mu^4}{5 \sigma^2_{\textup{max}}L^2} \tau}{\tau - \frac{\mu^4}{5 \sigma^2_{\textup{max}}L^2} \tau}
\leq \frac{\frac{\mu^4}{5 \sigma^2_{\textup{max}}L^2} \tau}{\tau - \frac{1}{5} \tau} = \frac{\mu^4}{4 \sigma^2_{\textup{max}}L^2}
\]
and therefore \eqref{eq:proof-param-strict-convex} is satisfied:
\[
\mu^2 - \alpha \frac{s}{\tau - s} > \mu^2 - \frac{2\sigma^2_{\textup{max}}L^2}{\mu^2} \frac{\mu^4}{4 \sigma^2_{\textup{max}}L^2} = \frac{1}{2}\mu^2 > 0 \]
and
\[\alpha \left(\mu^2 - \alpha \frac{s}{\tau - s}\right) > \frac{2\sigma^2_{\textup{max}}L^2}{\mu^2} \cdot \frac{1}{2}\mu^2 = \sigma^2_{\textup{max}}L^2.
\]
Finally, by choosing $\alpha$ as before, condition \eqref{eq:proof-alpha-2} is satisfied:
\[ \alpha = 2\sigma^2_{\textup{max}} \cdot \frac{L^2}{\mu^2} \geq 2\sigma^2_{\textup{max}} = \frac{2 \|\vn\|^2}{r^2},
\quad
r = \frac{1}{\sigma_{\textup{max}}} \|\vn\| < \frac{1}{\sigma_{\textup{max}}} \cdot \frac{1}{20}\frac{\mu^5}{\sigma^2_{\textup{max}}L^2} \tau < \tau,
\]
which finishes the proof of the uniqueness of minimizers of $F_{1a}$.
\textbf{Step 5: Local Lipschitz continuity of $\cF_{1a}$.} Previous results from Steps 1-4 indicate that, for any $\vx \in \sX$, there is a unique $\hat{\vy}(\vx)$ that minimizes $F_{1a}$, but the continuity of $\hat{\vy}$ w.r.t. $\vx$ has not been established. In this step, we will show this continuity via the implicit function theorem.
Firstly, as $\hat{\vy}$ minimizes $F_{1a}$, by first-order optimality conditions for smooth minimization, it holds that
\[ \nabla F_{1a}(\hat{\vy}) = \underbrace{\mA^\top(\mA\hat{\vy}-\vx) + \alpha ( \hat{\vy} - \vp(\hat{\vy}) )}_{=:\cH(\vx,\hat{\vy})} = \vzero \]
Now, let's pick a point $\vx_0$ from $\sX$. Previous results from Steps 1-4 indicate that, operator $\cH(\vx,\vy)$ is continuously differentiable within a neighborhood of $(\vx_0,\hat{\vy}(\vx_0))$, and its Jacobian matrix w.r.t. $\vy$
\[D_{\vy}\cH(\vx,\vy) = \nabla^2 F_{1a}(\vy)\]
is positive definite within that neighborhood of $(\vx_0,\hat{\vy}(\vx_0))$. Therefore, we are able to apply the implicit function theorem \cite[Theorem 3.9]{Folland2023-AdvCalcPDF} and conclude that $\hat{\vy}(\vx)$ is Lipschitz continuous within a neighborhood of $\vx_0$. This argument applies for any points $\vx_0$ in $\sX$. Therefore, $\hat{\vy}=\cF_{1a}(\vx)$ is locally Lipschitz continuous on $\sX$.
\end{proof}
The proof line of Theorem \ref{thm:inverse-lip-1b} largely follows the proof of Theorem \ref{thm:inverse-lip-1a}. Here we will highlight the difference of proofs between the two theorems, so that Theorem \ref{thm:inverse-lip-1b} will be rigorously proved without too much redundancy.
\begin{proof}[Proof of Theorem \ref{thm:inverse-lip-1b}]
For simplicity, we denote the objective function in \eqref{eq:mapping-1b} as $F_{1b}(\vy,\vz)$:
\[
F_{1b}(\vy,\vz) := \frac{1}{2}\|\vx - \mA \vy\|^2 + \frac{\alpha}{2} \textup{dist}^2(\vz,\sM) + \frac{\beta}{2} \|\vz-\vy\|^2,
\]
and we will study its properties analogously to $F_{1a}$.
\textbf{Step 1: Existence of minimizers of $F_{1b}$.} For any $r>0$, as
\[ \alpha \geq \frac{\|\vn\|^2}{r^2}, \quad \beta \geq \frac{\|\vn\|^2}{r^2}, \]
it holds that
\begin{equation}
\label{eq:proof-thm-mapping-1b-step1}
\inf_{\vy,\vz}F_{1b}(\vy,\vz) = \inf_{ (\vy,\vz): ~ \textup{dist}(\vz,\sM)\leq r \textup{ and } \|\vz-\vy\|\leq r }F_{1b}(\vy,\vz).
\end{equation}
This can be proved by contradiction: (I) Suppose $F_{1b}(\hat{\vy},\hat{\vz})$ is lower than the right-hand-side of \eqref{eq:proof-thm-mapping-1b-step1} and $\textup{dist}(\hat\vz,\sM) > r$, we have
\[ F_{1b}(\hat{\vy},\hat{\vz}) \geq 0 + \frac{\|\vn\|^2}{2 r^2} \textup{dist}^2(\hat{\vz},\sM) + 0 > \frac{1}{2} \|\vn\|^2 = F_{1b}(\vy_\ast,\vy_\ast)\]
which contradicts with the hypothesis regarding $(\hat{\vy},\hat{\vz})$. (II) Suppose $F_{1b}(\hat{\vy},\hat{\vz})$ is lower than the right-hand-side of \eqref{eq:proof-thm-mapping-1b-step1} and $\|\hat{\vz}-\hat{\vy}\| > r$, we have
\[ F_{1b}(\hat{\vy},\hat{\vz}) \geq 0 + 0 + \frac{\|\vn\|^2}{2 r^2}\|\hat{\vz}-\hat{\vy}\|^2 > \frac{1}{2}\|\vn\|^2 = F_{1b}(\vy_\ast,\vy_\ast)\]
which also derives a contradiction. Arguments in (I) and (II) together prove \eqref{eq:proof-thm-mapping-1b-step1}. Similar to the proof of Theorem \ref{thm:inverse-lip-1a}, \eqref{eq:proof-thm-mapping-1b-step1} implies the existence of minimizers of $F_{1b}$ (i.e., minimizers are attainable.)
\textbf{Step 2: Bound of minimizers of $F_{1b}$.} To extend the proof regarding $F_{1a}$ to $F_{1b}$, we consider the following inequality that holds for all $\vy,\vz \in \sU_{\tau}(\sM)$
\[ \|\vy-\vp(\vy)\| \leq \|\vy-\vp(\vz)\| \leq \|\vy-\vz\| + \|\vz-\vp(\vz)\| = \|\vy-\vz\| + \textup{dist}(\vz,\sM) \leq 2r. \]
Therefore, we need $2r < \tau$ and
\begin{equation}
\label{eq:proof-alpha-3}
\alpha \geq \frac{\|\vn\|^2}{r^2} > \frac{4\|\vn\|^2}{\tau^2}, \quad \beta \geq \frac{\|\vn\|^2}{r^2} > \frac{4\|\vn\|^2}{\tau^2}
\end{equation}
to ensure $\hat{\vy},\hat{\vz} \in \sU_{\tau}(\sM)$. Following the same argument as the proof of Theorem \ref{thm:inverse-lip-1a}, the above condition \eqref{eq:proof-alpha-3} implies
\[
\|\vp(\hat{\vy}) - \vy_\ast \| \leq \frac{\sigma_{\textup{max}}}{\mu} (2r) + \frac{2}{\mu} \|\vn\|
\]
and hence
\begin{equation}
\label{eq:proof-error-bound-1b}
\|\hat{\vy} - \vy_\ast\| \leq \|\hat{\vy} - \vp(\hat{\vy}) \| + \|\vp(\hat{\vy}) - \vy_\ast \| \leq 2 \left(1 + \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\|
\end{equation}
and
\begin{equation}
\label{eq:proof-error-bound-1b-z}
\|\hat{\vz} - \vy_\ast\| \leq \|\hat{\vz} - \hat{\vy} \| + \|\hat{\vy} - \vy_\ast \| \leq \left(3 + 2 \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\|
\end{equation}
holds for all $(\hat{\vy},\hat{\vz})$ that minimizes $F_{1b}(\vy,\vz)$.
\textbf{Step 3: Positive definiteness of the Hessian of $F_{1b}$.} Function $F_{1b}(\vy,\vz)$'s Hessian matrix is of size $2n \times 2n$ and can be written as a $2\times 2$ block w.r.t. $\vy$ and $\vz$:
\[ \nabla^2F_{1b}(\vy,\vz) =
\begin{bmatrix}
\mA^\top \mA & \vzero \\
\vzero & \vzero
\end{bmatrix} +
\alpha \begin{bmatrix}
\vzero & \vzero \\
\vzero & \mI - D\vp(\vz)
\end{bmatrix} +
\beta \begin{bmatrix}
\mI & -\mI \\
-\mI & \mI
\end{bmatrix}
\]
For any $\vh = [\vu^\top~~ \vv^\top]^\top \in \sR^{2n}$, the quadratic form $\langle \vh, \nabla^2F_{1b}(\vy,\vz) \vh\rangle$ can be calculated through:
\[
\langle \vh, \nabla^2F_{1b}(\vy,\vz) \vh\rangle
= \vu^\top\mA^\top \mA\vu + \alpha \vv^\top(\mI - D\vp(\vz))\vv + \beta \|\vu-\vv\|^2
\]
Decompose $\vu = \vu_{\mathrm{T}} + \vu_{\mathrm{N}}$ and $\vv = \vv_{\mathrm{T}} + \vv_{\mathrm{N}}$ in $\sT_{\vp(\vz)}(\sM) \oplus \sN_{\vp(\vz)}(\sM)$. Using the same argument as the proof of Theorem \ref{thm:inverse-lip-1a}, we have
\begin{align*}
& \langle \vh, \nabla^2F_{1b}(\vy,\vz) \vh\rangle \\
\geq & \bigg( \mu^2 \|\vu_{\mathrm{T}} \|^2 - 2 \sigma_{\textup{max}} L\|\vu_{\mathrm{T}}\|\|\vu_{\mathrm{N}}\| \bigg)
+ \alpha \left(- \frac{s}{\tau - s} \|\vv_{\mathrm{T}} \|^2 + \|\vv_{\mathrm{N}} \|^2 \right)
+ \beta \|\vu-\vv\|^2
\end{align*}
which implies
\begin{align*}
& \langle \vh, \nabla^2F_{1b}(\vy,\vz) \vh\rangle \\
\geq & \bigg( \mu^2 \|\vu_{\mathrm{T}} \|^2 - 2 \sigma_{\textup{max}} L\|\vu_{\mathrm{T}}\|\|\vu_{\mathrm{N}}\| \bigg) \\
& \quad + \alpha \left(- \frac{s}{\tau - s} \|\vv_{\mathrm{T}} \|^2 + \|\vv_{\mathrm{N}} \|^2 \right)
+ \beta \bigg( \|\vu_{\mathrm{T}}-\vv_{\mathrm{T}}\|^2 + \|\vu_{\mathrm{N}}-\vv_{\mathrm{N}}\|^2 \bigg) \\
\geq & \bigg( \mu^2 \|\vu_{\mathrm{T}} \|^2 - 2 \sigma_{\textup{max}} L\|\vu_{\mathrm{T}}\|\|\vu_{\mathrm{N}}\| \bigg) \\
& \quad + \alpha \left(- \frac{s}{\tau - s} \|\vv_{\mathrm{T}} \|^2 + \|\vv_{\mathrm{N}} \|^2 \right)
+ \beta \bigg( ( \|\vu_{\mathrm{T}}\| - \|\vv_{\mathrm{T}}\| )^2 + ( \|\vu_{\mathrm{N}}\| - \| \vv_{\mathrm{N}}\| )^2 \bigg) \\
= & \begin{bmatrix}
\|\vu_{\mathrm{T}}\| & \|\vu_{\mathrm{N}}\| & \|\vv_{\mathrm{T}}\| & \|\vv_{\mathrm{N}}\|
\end{bmatrix}
\underbrace{\begin{bmatrix}
\mu^2 + \beta & - \sigma_{\textup{max}} L & - \beta & \\
- \sigma_{\textup{max}} L & \beta & & -\beta \\
-\beta & & \beta - \alpha \frac{s}{\tau-s} & \\
& - \beta & & \alpha + \beta
\end{bmatrix}}_{=:\mB}
\begin{bmatrix}
\|\vu_{\mathrm{T}}\| \\ \|\vu_{\mathrm{N}}\| \\ \|\vv_{\mathrm{T}}\| \\ \|\vv_{\mathrm{N}}\|
\end{bmatrix}
\end{align*}
To ensure the positive definiteness of $\nabla^2F_{1b}(\vy,\vz)$, it's enough to ensure $\mB \succ \vzero$. For simplicity, we define
\[
\theta := \alpha \frac{s}{\tau-s}, \quad
\mB_{1}:= \begin{bmatrix}
\mu^2 + \beta & - \sigma_{\textup{max}} L \\
- \sigma_{\textup{max}} L & \beta
\end{bmatrix}
\quad \mB_{2}:= \begin{bmatrix}
- \beta & \\
& -\beta
\end{bmatrix}
\quad \mB_{3}:= \begin{bmatrix}
\beta - \theta & \\
& \alpha + \beta
\end{bmatrix}\]
Then $\mB = \begin{bmatrix}
\mB_1 & \mB_2 \\
\mB_2^\top & \mB_3
\end{bmatrix}$ is positive definite if and only if $\mB_3$ and its Schur complement $\mS$ are both positive definite:
\[ \mB_3 \succ \vzero ,\quad \mS = \mB_1 - \mB_2 \mB_3^{-1} \mB_2^\top \succ \vzero\]
As $\mB_2$ and $\mB_3$ are both diagonal, so $\mB_2 \mB_3^{-1} \mB_2^\top$ is straight forward to calculate: $\mB_2 \mB_3^{-1} \mB_2^\top = \textup{diag}\left(\frac{\beta^2}{\beta-\theta}, \frac{\beta^2}{\alpha+\beta}\right)$. Then the Schur complement can be calculated:
\[ \mS = \begin{bmatrix}
\mu^2 + \beta - \frac{\beta^2}{\beta-\theta} & - \sigma_{\textup{max}} L \\
- \sigma_{\textup{max}} L & \beta - \frac{\beta^2}{\alpha+\beta}
\end{bmatrix} =
\begin{bmatrix}
\mu^2 - \frac{\beta \theta}{\beta-\theta} & - \sigma_{\textup{max}} L \\
- \sigma_{\textup{max}} L & \frac{\alpha\beta}{\alpha+\beta}
\end{bmatrix}
\]
Note that $\mB_3 \succ \vzero$ if.f $\beta > \theta$. Therefore, $\mB \succ \vzero$ if.f.
\begin{equation}
\label{eq:proof-B-PD}
\beta > \theta, \quad \mu^2 > \frac{\beta \theta}{\beta-\theta}, \quad \left(\mu^2 - \frac{\beta \theta}{\beta-\theta}\right)\frac{\alpha\beta}{\alpha+\beta} > \sigma^2_{\textup{max}} L^2,
\end{equation}
where $\theta = \alpha \frac{s}{\tau-s}$. Finally, we obtain that \eqref{eq:proof-B-PD} ensures $\nabla^2F_{1b}(\vy,\vz)\succ \vzero$ for all $\vy \in \sR^n$ and all $\vz \in \overline{\sU}_s(\sM)$ with $s < \tau$.
\textbf{Step 4: Uniqueness of minimizers of $F_{1b}$.} Comparable to the Step 4 in Theorem \ref{thm:inverse-lip-1a}, we need $\|\hat{\vz}-\vy_\ast\| \leq s$ for all $(\hat{\vy},\hat{\vz}) \in \argmin F_{1b}(\vy,\vz)$. Based on \eqref{eq:proof-error-bound-1b-z}, it's enough to guarantee
\begin{equation}
\label{eq:proof-1b-s-condition}
\left(3 + 2 \frac{\sigma_{\textup{max}}}{\mu} \right) r + \frac{2}{\mu} \|\vn\| \leq s
\end{equation}
Now we choose
\[ s = \frac{4}{\mu} \|\vn\|, \quad r = \frac{2}{5\sigma_{\textup{max}}} \|\vn\|\]
which directly satisfies \eqref{eq:proof-1b-s-condition}. As $\|\vn\| < \frac{1}{76}\frac{\mu^5}{\sigma^2_{\textup{max}}L^2} \tau$, we have
\[ s = \frac{4}{\mu} \|\vn\| < \frac{1}{19} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2} \tau, \quad \frac{s}{\tau-s} < \frac{\frac{1}{19} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2} \tau}{\tau - \frac{1}{19} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2} \tau} \leq \frac{\frac{1}{19} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2} \tau}{\tau - \frac{1}{19} \tau} = \frac{1}{18} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2}\]
As long as we take
\[\alpha = \frac{9\sigma^2_{\textup{max}}L^2}{\mu^2}, \quad \beta \geq \max\left(\alpha, \frac{3}{2} \mu^2\right)\]
it holds that
\[
\theta = \alpha \frac{s}{\tau-s} < \frac{9\sigma^2_{\textup{max}}L^2}{\mu^2} \frac{1}{18} \frac{\mu^4}{\sigma^2_{\textup{max}}L^2} = \frac{1}{2} \mu^2
\]
which implies $\beta > 3 \theta$ and hence $\beta > \theta$. Moreover, we can verify the remaining part of \eqref{eq:proof-B-PD}:
\[
\begin{aligned}
\frac{\beta \theta}{\beta-\theta} <& \frac{\beta \theta}{\beta - \beta/3} = \frac{3}{2}\theta < \frac{3}{4}\mu^2 < \mu^2, \\
\left(\mu^2 - \frac{\beta \theta}{\beta-\theta}\right)\frac{\alpha\beta}{\alpha+\beta} >& \left(\mu^2 - \frac{3}{4}\mu^2\right) \frac{\alpha\beta}{\beta+\beta} = \frac{1}{8} \mu^2 \alpha = \frac{1}{8} \mu^2 \cdot \frac{9\sigma^2_{\textup{max}}L^2}{\mu^2} > \sigma^2_{\textup{max}}L^2.
\end{aligned}
\]
which finishes the proof of \eqref{eq:proof-B-PD}. Finally, it's enough to verify \eqref{eq:proof-alpha-3}:
\[2r \leq \frac{\|\vn\|}{\sigma_{\textup{max}}} \leq \frac{1}{76}\frac{\mu^5}{\sigma^3_{\textup{max}}L^2} \tau < \tau, \quad \frac{\|\vn\|^2}{r^2} = \frac{25}{4} \sigma^2_{\textup{max}} \leq \alpha \leq \beta,\]
which finishes Step 4, and concludes the uniqueness of $(\hat{\vy},\hat{\vz})$.
\textbf{Step 5: Local Lipschitz continuity of $\cF_{1b}$.} By largely following Step 5 in the proof of Theorem \ref{thm:inverse-lip-1a} and changing $\nabla^2 F_{1a}(\vy)$ to $\nabla^2 F_{1b}(\vy,\vz)$, one can directly conclude that the mapping $\cF_{1b}$ is locally Lipschitz continuous on $\sX$.
\end{proof}
\subsection{Proximal operator near a manifold}
\label{sec:app-prox}
We collect here the definition and basic properties of the proximal map used in the main text and relate them to the convergence condition proposed in \cite{ryu2019plug}.
\begin{theorem}[Contractivity of the proximal residual near a $\cC^2$ manifold]
\label{thm:prox-residual}
Let $\sM\subset\sR^n$ be a compact $\cC^2$ embedded submanifold with reach $\tau>0$.
For $\sigma>0$ define, for each $\vz\in\Utt$,
\[
\phi_\sigma(\vy,\vz):= \frac{\sigma}{2} \textup{dist}^2(\vy,\sM) + \frac{1}{2}\|\vy - \vz \|^2.
\]
Then $\phi_\sigma$ must yield a unique minimizer, and hence we are able to define
\[
\textup{prox}_{\sigma}(\vz):=\argmin_{\vy} \phi_\sigma(\vy,\vz),
\qquad
\cS_\sigma(\vz):=\textup{prox}_\sigma(\vz)-\vz.
\]
Then $\cS_\sigma$ is contractive within a tubular neighborhood of $\sM$. In particular, it holds that
\begin{equation}
\label{eq:proximal-residual-lip}
\|\cS_\sigma(\vz)-\cS_\sigma(\vz^\prime)\| \leq \frac{\sigma}{1+\sigma} \|\vz - \vz^\prime\|
\end{equation}
for all $\vz,\vz^\prime \in \Urt$ where $r \leq \tau/4$ and $\|\vz - \vz^\prime\|\leq\tau/4$.
\end{theorem}
Relation to plug-and-play (PnP):
Condition~(A) of \cite{ryu2019plug} assumes a (nearly) contractive \emph{denoiser residual}—precisely the kind of property \eqref{eq:proximal-residual-lip} guarantees for the proximal residual $\textup{prox}_{\sigma}-\mI$ on a neighborhood of $\sM$.
In practice, $\sM$ is unknown; one therefore learns a parameterized operator (e.g., a neural network) whose residual is constrained to be (nearly) $\sigma$-contractive and plugs it into PGD/HQS in place of the exact proximal map.
Whereas \cite{ryu2019plug} posits Condition~(A) to ensure convergence, Theorem~\ref{thm:prox-residual} shows this condition arises naturally when the prior corresponds to the manifold-penalty $\tfrac{\sigma}{2}\,\mathrm{dist}^2(\cdot,\sM)$.
\begin{proof}[Proof of Theorem \ref{thm:prox-residual}]
We first note that, for any $\vy$, if $\|\vy - \vz\| > \|\vz - \vp(\vz)\|$, then it holds that
\[ \phi_\sigma(\vp(\vz),\vz) = 0 + \frac{1}{2} \|\vz - \vp(\vz)\|^2 < \frac{\sigma}{2} \textup{dist}^2(\vy,\sM) + \frac{1}{2}\|\vy - \vz \|^2 = \phi_\sigma(\vy,\vz)\]
which implies
\[ \inf_{\vy} \phi_\sigma(\vy,\vz) = \inf_{\vy: \|\vy - \vz\| \leq \|\vz - \vp(\vz)\|} \phi_\sigma(\vy,\vz) \]
Let $r = \|\vz - \vp(\vz)\|$. We further notice that, for any $\vy$ with $\|\vy - \vz\| = s \leq r$, we are able to define $\tilde{\vy}$
\[ \tilde{\vy} := \frac{r-s}{r} \vz + \frac{s}{r} \vp(\vz) \]
which satisfies $\vp(\tilde{\vy})=\vp(\vz)$ and hence it holds that
\begin{align*}
\dist(\tilde{\vy},\sM) = \|\tilde{\vy} - \vp(\vz)\| = & \|\vz - \vp(\vz)\| - \|\tilde{\vy} - \vz\| \\
< & \|\vz - \vp(\vy)\| - \|\tilde{\vy} - \vz\| \\
\leq & \|\vz - \vy\| + \|\vy - \vp(\vy)\| - \|\tilde{\vy} - \vz\| \\
= & s + \|\vy - \vp(\vy)\| - s = \|\vy - \vp(\vy)\| = \dist(\vy,\sM)
\end{align*}
which implies
\[ \phi_\sigma(\tilde{\vy},\vz) = \frac{\sigma}{2} \textup{dist}^2(\tilde{\vy},\sM) + \frac{1}{2}\|\tilde{\vy} - \vz \|^2 < \frac{\sigma}{2} \textup{dist}^2(\vy,\sM) + \frac{1}{2}\|\vy - \vz \|^2 = \phi_\sigma(\vy,\vz)\]
Consequently, we conclude that minimizing $\phi_\sigma$ is equal to minimizing it over the line segment between $\vz$ and its projection $\vp(\vz)$:
\[\inf_{\vy} \phi_\sigma(\vy,\vz) = \inf_{\xi \in [0,1]} \phi_\sigma(\xi \vz + (1-\xi)\vp(\vz),\vz).\]
Now define $\psi(\xi)=\phi_\sigma(\xi \vz + (1-\xi)\vp(\vz),\vz)$. We have
\[
\begin{aligned}
\psi(\xi)=& \frac{\sigma}{2} \left\| \Big(\xi \vz + (1-\xi)\vp(\vz)\Big) - \vp(\vz)\right\|^2 + \frac{1}{2} \left\| \Big(\xi \vz + (1-\xi)\vp(\vz)\Big) - \vz\right\|^2\\
=& \frac{\sigma}{2} \xi^2 \|\vz-\vp(\vz)\|^2 + \frac{1}{2} (1-\xi)^2 \|\vz-\vp(\vz)\|^2\\
= & \Big(\sigma \xi^2 + (1-\xi)^2\Big) \cdot \frac{1}{2} \|\vz-\vp(\vz)\|^2
\end{aligned}
\]
Therefore, $\inf_{\xi\in[0,1]}\psi(\xi)$ is attainable, and the minimizer is $\xi_\ast = \frac{1}{1+\sigma}$, which implies $\phi_\sigma$ must yield a unique minimizer at
\[\vy_\ast = \frac{\vz + \sigma \vp(\vz)}{1+\sigma}.\]
Consequently, we have
\[ \cS_\sigma(\vz) = \vy_\ast-\vz = \frac{\sigma}{1+\sigma} (\vp(\vz)-\vz) \]
and hence
\[ D \cS_\sigma(\vz) = \frac{\sigma}{1+\sigma} (D\vp(\vz)-\mI). \]
According to \cite[Theorem C]{leobacher2021existence}, $D\vp(\vz)$ is actually restricted to the tangent space $\sT_{\vp(\vz)(\sM)}$:
\[ D\vp(\vz) = \Big(\mI_{\sT_{\vp(\vz)(\sM)}} - r \cL_{\vp(\vz),\vv} \Big)^{-1} P_{\sT_{\vp(\vz)(\sM)}} \]
where $r = \|\vp(\vz)-\vz\|$, $\vv = (\vp(\vz)-\vz)/r$, and $\cL_{\vp(\vz),\vv}$ is the shape operator in direction $\vv$ at $\vp(\vz)$. The shape operator's eigenvalues $\kappa_1,\cdots,\kappa_d$ (In this context, $d$ means the dimension of the tangent space) are the principal curvatures of $\sM$ \cite{do2016differential}, which implies the eigenvalues of $D\vp(\vz)$, when restricted to the tangent space, are
\[ \frac{1}{1-r\kappa_1}, \cdots, \frac{1}{1-r\kappa_d}. \]
All the curvatures are bounded by the reciprocal of the reach: $|\kappa_i| \leq 1/\tau$ \cite{aamari2019estimating}. Therefore, it holds that
\[ \frac{\tau}{\tau+r}\mI \Big|_{\sT_{\vp(\vz)(\sM)}} \preceq D\vp(\vz) \Big|_{\sT_{\vp(\vz)(\sM)}} \preceq \frac{\tau}{\tau-r}\mI \Big|_{\sT_{\vp(\vz)(\sM)}}.\]
Moreover, as $D\vp(\vz)$ is restricted to and acts only on the tangent space $\sT_{\vp(\vz)(\sM)}$, we have $\vzero \preceq D\vp(\vz) \preceq \frac{\tau}{\tau-r}\mI$, which implies
\[ -\mI \preceq D\vp(\vz) - \mI \preceq \frac{r}{\tau-r}\mI. \]
For $r \leq \tau/2$, we have $\frac{r}{\tau-r} \leq 1$ and hence $\|D \cS_\sigma(\vz)\| \leq \frac{\sigma}{1+\sigma}$. As long as $\vz,\vz^\prime \in \Urt$ where $r \leq \tau/4$ and $\|\vz - \vz^\prime\|\leq\tau/4$, the two points $\vz,\vz^\prime$ can be included in a convex subset (actually a ball) of $\Urt$ with $r=\tau/2$. By the mean value theorem, we finish the proof of \eqref{eq:proximal-residual-lip}.
\end{proof}
\subsection{Discussions regarding PnP}
\label{sec:app-pnp}
\paragraph{Derivation of HQS.}
Consider \eqref{eq:mapping-1b}:
\[
\min_{\vy,\vz\in\sR^n}\;\frac{1}{2}\|\vx-\mA\vy\|^2 \;+\; \frac{\alpha}{2}\,\mathrm{dist}^2(\vz,\sM) \;+\; \frac{\beta}{2}\|\vy-\vz\|^2.
\]
A typically method to solve it is applying block coordinate descent on it, which is also named ``Half-quadratic-splitting (HQS)" in the literature \cite{geman1995nonlinear}:
\[
\begin{aligned}
\vy_{t+1} = & \argmin_{\vy\in\sR^n} \frac{1}{2} \|\mA \vy - \vx\|^2 + \frac{\beta}{2} \|\vy - \vz_t\|^2 = \big(\mA^\top\mA + \beta \mI \big)^{-1} \Big( \mA^\top \vx + \beta \vz_t \Big)\\
\vz_{t+1} = & \argmin_{\vz\in\sR^n} \frac{\alpha}{2} \textup{dist}^2(\vz,\sM) + \frac{\beta}{2} \|\vz - \vy_{t+1}\|^2 = \textup{prox}_{\sigma} (\vy_{t+1}) \quad (\textup{let } \sigma=\alpha/\beta)
\end{aligned}
\]
Similarly, we can parameterize $\textup{prox}_{\sigma}$ as a neural network $\cH_{\theta,\sigma}$. Therefore, HQS suggests an implicit model
\[
\cG_{\Theta}(\vz,\vx) = \cH_{\theta,\sigma}\bigg( \big(\mA^\top\mA + \beta \mI \big)^{-1} \Big( \mA^\top \vx + \beta \vz \Big) \bigg)
\]
where $\Theta = \{\theta,\sigma,\beta\}$ includes all trainable parameters, which derives \eqref{eq:hqs-deq}.
\textbf{Bibliographical notes.} Here we adopt the long‐standing “plug-in denoiser” idea. It originated with Plug-and-Play (PnP) ADMM, which replaces a proximal operator with an off-the-shelf denoiser inside ADMM \cite{venkatakrishnan2013plug}. The framework has since been developed and analyzed extensively—see, e.g., \cite{chan2016plug,kamilov2017plug,buzzard2018plug,sun2019online} and the recent survey \cite{kamilov2023plug}. In the PGD setting, one pretrains $\cH$ for Gaussian denoising and plugs it into \eqref{eq:pgd-deq} \cite{ryu2019plug,gavaskar2020plug,liu2021recovery,hurault2022proximal}. The same plug-in idea applies to HQS via \eqref{eq:hqs-deq} \cite{zhang2021plug,hurault2022gradient,rasti2023plug}. In contrast to training a denoiser off-the-shelf and plugging it in, one can train the \emph{entire} $\cG_\Theta$ via deep equilibrium methods for the target task (the approach closest to this paper) in both PGD-style \cite{gilton2021deep,winston2020monotone,zou2023deep,yu2024msdc,daniele2025deep,shenoy2025recovering} and HQS-style \cite{gkillas2023optimization}.
\section{Proofs regarding NS Equations}
\label{sec:app-ns}
To rigorously state and prove the theorems, we present some definitions here.
First, We denote by $H^{m}(\Omega)$ the Sobolev space of functions which are in $L^2(\Omega)$ together with all their derivatives of order $\leq m$. Then $H^{m}_{\mathrm{p}}(\Omega)\subset H^{m}(\Omega)$ is the collection of functions in $H^{m}(\Omega)$ that satisfies the periodic boundary condition on $\Omega$ with zero mean (ref. to \cite[Remark 1.1]{temam1995navier}). Then, we can define the spaces considered in this paper:
\[ \sH := \left\{u \in \left\{ H^{0}_{\mathrm{p}}(\Omega) \right\}^2: \nabla \cdot u = 0 \right\},
\quad
\sV := \left\{u \in \left\{ H^{1}_{\mathrm{p}}(\Omega) \right\}^2: \nabla \cdot u = 0 \right\}
\]
For the NS equation \eqref{eq:ns-velocity}, we consider $f \in \sH$ and $u \in \sV$. Moreover, we denote $\sV^\prime$ as the dual space of $\sV$ and have
\[ \sV \subset \sH \subset \sV^\prime. \]
We then equip $\sH$ with the standard $L^2$ inner product and norm for vector fields:
\[ \langle u, v \rangle_{\sH} := \int_{\Omega} \langle u(\xi), v(\xi) \rangle \mathrm{d}\xi, \quad \| u \|_{\sH} := \sqrt{\langle u, u \rangle_{\sH}} = \left( \int_{\Omega} \|u(\xi)\|^2 \mathrm{d}\xi \right)^{1/2}= \|u\|_{L^2(\Omega)}\]
The space $\sV$ is equipped with the $L^2$ norm on the first-order derivatives of $u$. In particular,
\[
\begin{aligned}
\langle u, v \rangle_{\sV} := & \sum_{i=1}^2 \int_{\Omega} \left\langle \frac{\partial u}{\partial \xi_i}(\xi), \frac{\partial v}{\partial \xi_i}(\xi) \right\rangle \mathrm{d}\xi \\
\| u \|_{\sV} := & \sqrt{\langle u, u \rangle_{\sV}} = \left( \sum_{i=1}^2 \int_{\Omega} \left\|\frac{\partial u}{\partial \xi_i}(x) \right\|^2 \mathrm{d}\xi \right)^{1/2} = \|\nabla u\|_{L^2(\Omega)}
\end{aligned}
\]
and $\|\cdot\|_{\sV^\prime}$ is defined as the dual norm of $\|\cdot\|_{\sV}$. By Poincare and Cauchy-Shwartz inequalities, we have
\[ \|v\|_{\sH} \leq c_1 \|v\|_{\sV} , \quad \forall v \in \sV\]
and
\[ \|v\|_{\sV^\prime} \leq c_2 \|v\|_{\sH} , \quad \forall v \in \sH \]
where $c_1,c_2$ are constants depending on the domain $\Omega$. The above definitions and results are standard in the literature and we largely follow the notation in \cite[Section 2]{temam1995navier}.
\begin{proof}[Proof of Theorem \ref{thm:ns-existence}]
\cite[Theorem 10.1]{temam1995navier} states that, for any $f \in \sV^\prime$, if $\|f\|_{\sV^\prime} \le c_0 \nu^2$ (with $c_0>0$ depending only on $\Omega$), then the steady NS problem \eqref{eq:ns-velocity} has a unique solution $u_\ast$.
Since $\sH \subset \sV^\prime$ and $\|f\|_{\sV^\prime} \le c_2 \|f\|_{\sH}$, this yields uniqueness on
\[
\sH_{\nu}^{(1)} := \left\{ f \in \sH : \|f\|_{\sH} \le \frac{c_0}{c_2}\,\nu^2 \right\}.
\]
Moreover, by \cite[Theorem 10.4]{temam1995navier}, there exists an open dense set $\sH_{\nu}^{(2)} \subset \sH$ such that, on each connected component of $\sH_{\nu}^{(2)}$, the solution $u_\ast$ depends $C^\infty$ on $f$; in particular, $f \mapsto u_\ast$ is locally Lipschitz there.
Define $\sH_{\nu}:=\sH_{\nu}^{(1)} \cap \sH_{\nu}^{(2)}$. Since $\sH_{\nu}^{(2)}$ is open and dense in $\sH$, the set $\sH_{\nu}$ is dense in $\sH_{\nu}^{(1)}$. On $\sH_{\nu}$ the solution is unique and the map $f \mapsto u_\ast$ is locally Lipschitz. This completes the proof.
\end{proof}
Before moving to Corollary \ref{coro:ns-discrete}, let's reclarify lifting and projection operators: Let the lifting (or extension) operator $\cE_h: \sR^{N_h \times 2} \to \left\{L^2(\Omega)\right\}^2 $ be the piecewise–constant reconstruction $\cE_h(\vx):= \sum_{C \in \Omega_h} x_C \vone_{C}$, and let $\cP: \left\{L^2(\Omega)\right\}^2 \to \sH$ be the orthogonal projection onto divergence–free, zero–mean fields. Then we move on to Corollary \ref{coro:ns-discrete}.
\begin{proof}[Proof of Corollary \ref{coro:ns-discrete}]
The mapping $\cF_{2}:\vx \mapsto \vy_\ast$ can be viewed as a composition of multiple mappings: We first map $\vx \in \sR^{N_h \times 2}$ to a continuous version $f \in \sH$ by $\cP \circ \cE_h$, then $f$ can be mapped to its corresponding solution $u_\ast$ by a Locally Lipschitz operator as stated in Theorem \ref{thm:ns-existence}. Here we denote this mapping by $\cS: f \mapsto u_\ast$. Then $u_\ast$ is mapped to $\omega_\ast$ by vorticity: $\nabla \times u_\ast$, and finally $\omega_\ast$ can be mapped to $\vy_\ast$ by a restriction operator $\cR_h$:
\[ \cF_2 = \cR_h \circ (\nabla \times) \circ \cS \circ \cP \circ \cE_h. \]
Then let's analyze the norm of the above operators one by one. Firstly, the restriction operator $\cR_h$ has a norm no greater than $1$ as:
\begin{align*}
\| \cR_h(\omega) \|^2_{\ell^2_h}
= & \sum_{C \in \Omega_h} |C| \left| \frac{1}{|C|}\int_{C} \omega(\xi) \mathrm{d}\xi \right|^2 \\
\leq & \sum_{C \in \Omega_h} \frac{1}{|C|} \left( \int_{C} |\omega(\xi)|\mathrm{d}\xi \right)^2
\leq \sum_{C \in \Omega_h} \int_{C} |\omega(\xi)|^2 \mathrm{d}\xi = \|\omega\|^2_{L^2(\Omega)}
\end{align*}
Note that $\cR_h$ is a linear operator, hence its bounded norm immediately leads to its bounded Lipschitz constant:
\[ \| \cR_h(\omega) - \cR_h(\omega^\prime) \|^2_{\ell^2_h}
= \| \cR_h(\omega - \omega^\prime) \|^2_{\ell^2_h}
\leq \|\omega - \omega^\prime\|^2_{L^2(\Omega)}. \]
Second, the curl operator $\nabla \times$ must be a bounded linear operator because the solution $u_\ast \in \sV$, where first-order derivatives must be $L^2$. Third, the solution mapping $\cS$ has been discussed in Theorem \ref{thm:ns-existence}, it is a nonlinear operator, but it is locally Lipschitz continuous. Fourth, the projection operator $\cP$ must be linear and have a norm no greater than $1$. Finally, the lifting operator is linear and has a bounded norm as:
\[ \|\cE_h(\vx)\|^2_{L^2(\Omega)} = \sum_{C \in \Omega_h} |C| ~ |x_C|^2 = \|\vx\|^2_{\ell^2_h} \]
Therefore, except for the nonlinear operator $\cS$, the other four operators are all linear and bounded and hence are globally Lipschitz continuous. As long as we can show that the input of $\cS$ must be taken from the unique solution regime $\sH_{\nu}$, we will complete the proof that $\cF_2$ is locally Lipschitz everywhere on $\sX_{\nu,h}$. This can be proved because $\vx \in \sX_{\nu,h}$ implies $\cP(\cE_{h}(\vx)) \in \sH_{\nu}$. Finally, by applying Theorem \ref{thm:lip}, we conclude the existence of $\cG$ described in Corollary \ref{coro:ns-discrete}, which finishes the entire proof.
\end{proof}
\section{Proofs regarding linear programming}
\label{sec:app-lp}
Although Lipschitz continuity of LP solution maps has been studied (e.g., \cite{mangasarian1987lipschitz,dontchev2009implicit}), we are not aware of a reference that states Theorem \ref{thm:lp} in the precise form needed here—particularly allowing perturbations of $\mA$ (rather than treating $\mA$ as fixed). For completeness, we therefore include a self-contained discussion and proof.
To work with a standard form, we rewrite the general-form problem \eqref{eq:lp} in standard form. Suppose there are $p$ equality constraints and $q$ inequality constraints. Without loss of generality, we assume $\circ_i$ equals to ``$=$" for $1 \leq i \leq p$ and $\circ_i$ equals to ``$\leq$" for $p+1 \leq i \leq m$. Then we denote $\mA_p$ as the first $p$ rows of matrix $\mA$ and $\mA_q$ as the remaining part:
\[\mA_p := \mA[1:p, ~:],\qquad \mA_q := \mA[p+1:m, ~ :]\]
And therefore the general form LP \eqref{eq:lp} can be written as
\[\min_{\vy \in \sR^n} \vc^\top \vy, \quad \textup{s.t. } \mA_p \vy = \vb_p, ~~ \mA_q \vy \leq \vb_q, ~~ \vl \leq \vy \leq \vu. \]
Let $\hat{\vy}:=\vy-\vl$, $\vs := \vb_q - \mA_q \vy$, and $\vt := \vu - \vy$, the above problem can be transformed to
\[
\min_{\vy \in \sR^n} \vc^\top \hat{\vy}, \quad \textup{s.t. }
\begin{bmatrix}
\mA_p & & \\
\mA_q & \mI & \\
\mI & & \mI
\end{bmatrix}
\begin{bmatrix}
\hat{\vy} \\
\vs \\
\vt
\end{bmatrix}
=
\begin{bmatrix}
\vb_p - \mA_p \vl \\
\vb_q - \mA_q \vl \\
\vu - \vl
\end{bmatrix},
~~
\hat{\vy} \geq \vzero, \vs \geq \vzero, \vt \geq \vzero
\]
By letting
\[
\tilde{\vc}:=\begin{bmatrix}
\vc \\ \vzero \\ \vzero
\end{bmatrix}, ~~
\tilde{\mA}:=\begin{bmatrix}
\vb_p - \mA_p \vl \\
\vb_q - \mA_q \vl \\
\vu - \vl
\end{bmatrix},~~
\tilde{\vb}:=\begin{bmatrix}
\vb_p - \mA_p \vl \\
\vb_q - \mA_q \vl \\
\vu - \vl
\end{bmatrix},~~
\tilde{\vy}:=\begin{bmatrix}
\hat{\vy} \\
\vs \\
\vt
\end{bmatrix}
\]
The problem is equivalently expressed in standard form as
\[ \min_{\tilde{\vy}} \tilde{\vc}^\top \tilde{\vy}, ~~ \textup{s.t. } \tilde{\mA}\tilde{\vy}=\tilde{\vb},~~ \tilde{\vy} \geq \vzero. \]
In fact, every LP can be rewritten in an equivalent standard form. While concepts such as basic feasible solutions, degeneracy, and complementary slackness are most naturally and cleanly stated in standard form, each admits a closely related analogue (with minor adjustments) for the general form. Accordingly—without loss of generality and to keep the focus on core ideas—we carry out the proof in the standard-form setting:
\[
\min_{{\vy}} {\vc}^\top {\vy}, ~~ \textup{s.t. } {\mA}{\vy}={\vb},~~ {\vy} \geq \vzero,
\]
with dual
\[
\min_{{\vz}} {\vb}^\top {\vz}, ~~ \textup{s.t. } {\mA^\top}{\vz}\leq{\vc}.
\]
Here, we follow the standard settings in the literature: $\vy,\vc \in \sR^n$, $\vz,\vb \in \sR^m$, $\mA \in \sR^{m \times n}$, $\textup{rank}(\mA)=m$ (ensured by preprocessing with removing redundant equalities), and $m \leq n$. In this context, we define the domain of LP that we work on:
\[ \sX:= \{(\mA,\vb,\vc): \textup{The resulting standard LP is feasible and bounded}\} \]
Note that, to match the rest of the paper, we reserve $\vx$ for machine learning model inputs (in this context, it is $\vx=(\mA,\vb,\vc)$) and hence write the primal LP variable as $\vy$ and the dual LP variable as $\vz$. This departs from the common $(\vx,\vy)$ convention. Note also that in the main text the symbol $\vz$ denotes a latent variable; here, in the appendix regarding LP's technical details, it denotes the dual variable. These meanings are unrelated and should be clear from context.
Now let's present some definitions used in this appendix. Fix a \textit{basis} by selecting an index set $B \subset \{1,2,\cdots,n\}$ with $|B|=m$ such that the $m \times m$ submatrix $\mB:=\mA[:,B]$ is \textit{nonsingular}. Let $N=\{1,2,\cdots,n\} \setminus B$ be the complement of the basis and let $\mN:=\mA[:,N]$. Then the equality constraints read
\[\mB \vy_{B} + \mN \vy_{N} = \vb\]
Setting $\vy_{N}=\vzero$ yields $\vy_{B} = \mB^{-1}\vb$. Such a $\vy = [\vy_{B}, \vzero]$ is called a \textit{basic solution}. If additionally $\vy_{B} \geq \vzero$, this basic solution is feasible, then it is called a \textit{basic feasible solution (BFS)}. On the dual side, we define the slack variable $\vs$ and its sub-vector restricted to $B$ and $N$:
\[ \vs := \vc - \mA^\top \vz, \quad \vs_{B} := \vc_{B} - \mB^\top \vz, \quad \vs_{N} := \vc_{N} - \mN^\top \vz. \]
A pair $(\vy,\vz)$ is primal–dual optimal (i.e., satisfies KKT for LP) iff
\begin{equation}
\label{eq:lp-complementary}
\mA \vy = \vb,\quad \vc = \mA^\top \vz + \vs, \quad \vy\odot\vs=\vzero, \quad \vy \geq \vzero,~~\vs \geq \vzero
\end{equation}
for some $\vs \in \sR^n$. If, in addition, there exists a basis $B$ such that
\begin{equation}
\label{eq:lp-regularity}
\vy_{B} \geq \vzero, ~~\vy_{N} = \vzero, ~~\vs_{B} = \vzero, ~~\vs_{N} \geq \vzero,
\end{equation}
then the tuple $(\vy,\vz,\vs)$ is called an optimal BFS with a complementary dual. By the fundamental theorem of linear programming, any feasible instance with finite optimal value $(\mA,\vb,\vc) \in \sX$ admits an optimal BFS with a complementary dual satisfying \eqref{eq:lp-complementary} and \eqref{eq:lp-regularity} together \cite{bertsimas1997introduction}.
While conditions \eqref{eq:lp-complementary} and \eqref{eq:lp-regularity} are enough to ensure the existence of the optimal basic solutions, they are not enough to ensure that the optimal solution is unique and locally Lipschitz continuous w.r.t. the inputs $(\mA,\vb,\vc)$. To ensure these points, we present two additional conditions based on \eqref{eq:lp-complementary} and \eqref{eq:lp-regularity}:
\begin{align}
\vy_B > \vzero & \quad (\textup{Non-degeneracy}) \label{eq:lp-nondegenerate} \\
\vs_N > \vzero & \quad (\textup{Strict complementary slackness}) \label{eq:lp-sc}
\end{align}
All the conditions together are enough to the uniquenss and local Lischitz continuity. Let's introduce a set consisting of all ``good" LP instances:
\[ \sX_{\mathrm{sub}}:= \{(\mA,\vb,\vc) \in \sX: \textup{The LP yields a tuple $(\vy,\vz,\vs)$ satisfying \eqref{eq:lp-complementary}, \eqref{eq:lp-regularity}, \eqref{eq:lp-nondegenerate} and \eqref{eq:lp-sc}.}\} \]
With all the preparations, we can prove Theorem \ref{thm:lp} now. Actually, proving Theorem \ref{thm:lp} in the context of standard-form LP is equivalent to proving the following two theorems.
\begin{theorem}
\label{thm:lp-regular}
For any LP $(\mA,\vb,\vc) \in \sX_{\mathrm{sub}}$, it must yield a unique optimal solution $\vy_\ast$, and the solution mapping $(\mA,\vb,\vc) \mapsto \vy_\ast$ is locally Lipschitz continuous everywhere on $\sX_{\mathrm{sub}}$.
\end{theorem}
\begin{theorem}
\label{thm:lp-dense}
$\sX_{\mathrm{sub}}$ is a dense subset of $\sX$.
\end{theorem}
Theorem \ref{thm:lp-regular} follows from \cite{dontchev1996characterizations}, which develops Robinson’s notion of strong regularity \cite{robinson1980strongly} for nonlinear programs. For completeness—and to keep notation consistent with linear programming—we restate the relevant lemma in an LP-adapted form and then verify its hypotheses for LP. We begin by quoting the result from \cite{dontchev1996characterizations}.
\begin{lemma}[\cite{dontchev1996characterizations}]
\label{lemma:lp-regular}
Consider a parameteric nonlinear program:
\begin{align*}
\min_{\vy\in\sR^n} & \vc^\top \vy + g_0(\vw,\vy) \\
\textup{s.t. } & g_i(\vw,\vy)=u_i, ~~ 1 \leq i \leq r \\
& g_i(\vw,\vy)\leq u_i, ~~ r+1 \leq i \leq d
\end{align*}
where $g_i (0 \leq i \leq d)$ are all $\cC^2$ functions, and $\vc,\vw$ and $\vu=[u_1,\cdots,u_d]^\top$ are parameters to describe the program, and consider its Lagrangian with multipliers $\lambda=[\lambda_1,\cdots,\lambda_d] \in \sR^d$ given by
\[L(\vw,\vy,\lambda) = g_0(\vw,\vy) + \sum_{i=1}^{d} \lambda_i g_i(\vw, \vy).\]
Let $(\bar{\vy},\bar{\lambda})$ be a KKT point at $(\bar{\vc},\bar{\vw},\bar{\vu})$, and define the index sets at $(\bar{\vy},\bar{\lambda})$
\[
\begin{aligned}
I_1 = & \Big\{r+1 \leq i \leq d: g_i(\bar{\vw},\bar{\vy})=u_i, \bar{\lambda}_i > 0 \Big\} \cup \Big\{1,\cdots,r\Big\}, \\
I_2 = & \Big\{r+1 \leq i \leq d: g_i(\bar{\vw},\bar{\vy})=u_i, \bar{\lambda}_i = 0 \Big\}, \\
I_3 = & \Big\{r+1 \leq i \leq d: g_i(\bar{\vw},\bar{\vy}) < u_i, \bar{\lambda}_i = 0 \Big\}.
\end{aligned}
\]
If the following conditions hold:
\begin{itemize}
\item The constraint gradients $\nabla_\vy g_i(\bar{\vw},\bar{\vy})$ for $i \in I_1 \cup I_2$ are linearly independent; and
\item It holds that
\[ \langle \vy^\prime, \nabla^2_{\vy\vy}L(\bar{\vw},\bar{\vy},\bar{\lambda}) \vy^\prime \rangle > 0 \]
for all $\vy^\prime \neq \vzero$ in the subspace $\sM=\Big\{ \vy^\prime: \vy^\prime \perp \nabla_\vy g_i(\bar{\vw},\bar{\vy}) \textup{ for all } i \in I_1 \Big\}$,
\end{itemize}
then the KKT solution map $(\vc,\vw,\vu) \mapsto (\vy,\lambda)$ is locally single-valued and Lipschitz around $(\bar{\vc},\bar{\vw},\bar{\vu},\bar{\vy},\bar{\lambda})$.
\end{lemma}
\begin{proof}[Proof of Theorem \ref{thm:lp-regular}]
Taking $r=m$ and $d=m+n$. Let $\va^\top_i$ be the i-th row of $\mA$ in standard LP, and let
\[
g_i(\vw,\vy)=
\begin{cases}
\va_i^\top \vy,& i=1,\dots,m,\\[2pt]
-\,y_{i-m},& i=m+1,\dots,m+n,
\end{cases}
\quad
u_i=
\begin{cases}
b_i,& i=1,\dots,m,\\[2pt]
0,& i=m+1,\dots,m+n,
\end{cases}
\]
with \(\vw\) collecting the coefficients of \(\mA\).
The Lagrangian in Lemma \ref{lemma:lp-regular} becomes
\[
L(\vw,\vy,\lambda)=\vc^\top \vy+\sum_{i=1}^m \lambda_i\,\va_i^\top \vy+\sum_{j=1}^n \lambda_{m+j}(-y_j).
\]
Introduce the usual dual/primal–slack variables
\[
\vz:=-\lambda_{1:m}\in\sR^m,\qquad \vs:=\lambda_{m+1:m+n}\in\sR^n_{\ge 0},
\]
to rewrite stationarity as \(\nabla_{\vy} L=\vc-\mA^\top \vz-\vs=\vzero\), i.e., \(\vs=\vc-\mA^\top \vz\).
Primal feasibility is \(\mA\vy=\vb,\ \vy\ge 0\); dual feasibility is \(\vs\ge \vzero\); and complementarity is \(\vy\odot \vs=\vzero\).
Thus the KKT system in Lemma \ref{lemma:lp-regular} coincides with the standard LP KKT conditions.
Assume \((\mA,\vb,\vc)\in\sX_{\mathrm{sub}}\), i.e., the LP admits a tuple $(\bar\vy,\bar\vz,\bar\vs)$ satisfying \eqref{eq:lp-complementary}, \eqref{eq:lp-regularity}, \eqref{eq:lp-nondegenerate} and \eqref{eq:lp-sc} (A nondegenerate and strict complementary basic point). In this context, the index sets \(I_1,I_2,I_3\) at \((\bar\vy,\bar\vz,\bar\vs)\) become:
\[
\begin{aligned}
I_1 &= \{1,\ldots,m\}\ \cup\ \{\,m+j:\ \bar y_j=0,\ \bar s_j>0\,\},\\
I_2 &= \{\,m+j:\ \bar y_j=0,\ \bar s_j=0\,\},\\
I_3 &= \{\,m+j:\ \bar y_j>0,\ \bar s_j=0\,\}.
\end{aligned}
\]
which implies:
\begin{itemize}
\item For each \(j\), either \(\bar y_j>0\) or \(\bar s_j>0\), which implies \(I_2=\varnothing\).
\item $I_3$ is substantially the basis set: $I_3 = \{m + j:j \in B\}$
\item $I_1$ includes all the indices in the complement of basis: $I_1 = \{1,\ldots,m\}\ \cup\ \{\,m+j: j \in N\}$
\end{itemize}
To verify the hypotheses of Lemma \ref{lemma:lp-regular}, we examine the gradients:
\[\{\nabla_\vy g_i\}_{i\in I_1}=\{\va_i\}_{i=1}^m \cup \{-\ve_j\}_{j\in N} \]
In the context of standard LP, $|N|=n-m$. Hence, \(\{\nabla_\vy g_i\}_{i\in I_1}\) consists of $n$ vectors in $\sR^n$. Now we create a matrix $\mG$ by stacking these vectors as rows:
\[\mG:=\begin{bmatrix}
\va^\top_1 \\
\cdots \\
\va^\top_m \\
\ve^\top_{j_1} \\
\cdots \\
\ve^\top_{j_{n-m}}
\end{bmatrix}\]
By properly permuting the columns of $\mG$, it becomes
\[
\tilde{\mG} = \begin{bmatrix}
\mB & \mN \\
\vzero & \mI
\end{bmatrix}
\]
where $\mI$ represents the identity matrix in $\sR^{n-m}$. Since $\mB$ (the basis matrix) and $\mI$ are both nonsingular, $\tilde{\mG}$ (and hence $\mG$) must be nonsingular. Therefore, the rows of $\mG$ are linearly independent, i.e., \(\{\nabla_\vy g_i\}_{i\in I_1}\) is linearly independent. With \(I_2=\varnothing\), the first hypothesis of Lemma \ref{lemma:lp-regular} holds. Moreover, because these gradients \(\{\nabla_\vy g_i\}_{i\in I_1}\) span $\sR^n$, the $\sM$ subspace must be trivial: $\sM=\{\vzero\}$. Therefore, the second hypothesis of Lemma \ref{lemma:lp-regular} is automatically satisfied.
By Lemma \ref{lemma:lp-regular}, the KKT solution map is locally single-valued and Lipschitz around the given point, which yields the desired local uniqueness and Lipschitz dependence of \(\vy_\ast\) on \((\mA,\vb,\vc)\) for every \((\mA,\vb,\vc)\in\sX_{\mathrm{sub}}\).
\end{proof}
Theorem \ref{thm:lp-dense} can be proved by fundamental concepts in real analysis.
\begin{proof}[Proof of Theorem \ref{thm:lp-dense}]
To prove $\sX_{\mathrm{sub}}$ is dense in $\sX$, it's enough to show that: For any $(\mA,\vb,\vc)\in\sX$, one can always create a sequence of LP $\{(\mA_k,\vb_k,\vc_k)\}_{k\ge 1} \subset \sX_{\mathrm{sub}}$ such that
\[ \mA_k \to \mA, ~~ \vb_k \to \vb, ~~ \vc_k \to \vc. \]
Now let's fix $(\mA,\vb,\vc)\in\sX$. As we previously discussed, there must be a tuple $(\vy,\vz,\vs)$ satisfying \eqref{eq:lp-complementary} and \eqref{eq:lp-regularity}. Define:
\[ \vy_k := \vy + \frac{1}{k}\ve_{B},\quad \vs_{k} := \vs + \frac{1}{k}\ve_{N}, \quad \vz_k := \vz\]
so that $(\vy_k,\vz_k, \vs_k)$ must satisfy the nondegeneracy and strict complementary slackness: \eqref{eq:lp-regularity}, \eqref{eq:lp-nondegenerate}, and \eqref{eq:lp-sc}. Accordingly, define
\[ \mA_k := \mA, \quad \vb_k := \mA_k \vy_k, \quad \vc_k := \mA^\top_k \vz_k + \vs_k \]
Then one can verify that the tuple $(\vy,\vz,\vs)$ satisfies \eqref{eq:lp-complementary}, \eqref{eq:lp-regularity}, \eqref{eq:lp-nondegenerate} and \eqref{eq:lp-sc} for the LP instance $(\mA_k,\vb_k,\vc_k)$, hence $(\mA_k,\vb_k,\vc_k) \in \sX_{\mathrm{sub}}$ for all $k \geq 1$. Finally, such a perturbed LP instance can be arbitrarily close to $(\mA,\vb,\vc)$ as $k \to \infty$:
\begin{align*}
\|\mA_k-\mA\| = & 0 \\
\|\vb_k-\vb\| = & \left\| \mA\left( \frac{1}{k}\ve_{B} \right) \right\| \leq \frac{1}{k} \|\mA\| \|\ve_{B}\| = \frac{\sqrt{m}}{k}\|\mA\| \to 0 \\
\|\vc_k-\vc\| = & \left\|\frac{1}{k}\ve_{N} \right\| = \frac{\sqrt{n-m}}{k} \to 0
\end{align*}
which finishes the proof.
\end{proof}
\section{Training Strategies}
\label{app:training}
\textbf{Unrolling vs implicit differentiation.} There are two training strategies adopted in this paper. One is named ``unrolling" (minimizing $\ell(\vy_T)$):
$$\min_\theta \ell(\vy_T), ~~~ \vy_{t+1} = \mathcal{G}_{\theta}(\vy_{t},\vx),~~ t=0,1,2,\cdots,T-1$$
and the other is named ``implicit differentiation" (minimizing $\ell(\vy_*)$):
$$\min_\theta \ell(\vy_\ast), ~~~ \vy_{\ast} = \mathcal{G}_{\theta}(\vy_{\ast},\vx).$$
These two strategies are closely related. In particular,
\begin{itemize}[leftmargin=5mm]
\item As established in prior literature, unrolled training is mathematically equivalent to a Neumann series approximation of the implicit gradient \cite{geng2021training}. Specifically, implicit differentiation requires inverting the Jacobian $(\mI - \mJ_{\mathcal{G}_\theta})^{-1}$; finite unrolling effectively approximates this inverse via a Neumann series expansion. This is a widely adopted technique in the implicit model community to avoid the instability and cost of exact inversion.
\item Implicit training is simply the limit of unrolled training: as $T \to \infty$, the gradient $\nabla_\theta \ell(\vy_T)$ converges to the implicit gradient $\nabla_\theta \ell(\vy_\ast)$ \cite{geng2021training}.
\end{itemize}
Overall, unrolling and root-finding are merely two numerical implementations for approximating the same fixed point, $\vy_\ast(\vx)$, and technically speaking, there is no significant gap or distinction between the two. Theoretically, infinite unrolling converges exactly to $\vy_\ast(\vx)$. In practice, unrolling depth simply controls the trade-off between accuracy and computational cost: a dynamic strictly analogous to setting the error tolerance in implicit root-finding solvers.
Particularly in our paper, for Case Studies 1 \& 2, we employ implicit differentiation (minimizing $\ell(\vy_\ast)$) via root-finding; for Case Study 3, we adopt unrolling to train implicit GNNs, which serves as a truncated Neumann approximation of the implicit GNN gradient; for Case Study 4, we directly use the pretrained model from \cite{geiping2025scaling}.
\textbf{Guarantees of Regularity and the Expressivity Trade-off.} While our experiments demonstrate that standard training (either unrolling and implicit differentiation defined above) empirically results in regular implicit operators, \textit{we do not explicitly enforce this property in the loss function.} Designing training mechanisms that theoretically guarantee regularity without sacrificing the model's unique expressive capabilities remains an open and interesting future topic.
Recall that regularity (Definition \ref{def:regular-g}) comprises two conditions: the Lipschitz continuity of the map $\vx \mapsto \cG_\theta(\vy,\vx)$ and the contractivity of the map $\vy \mapsto \cG_\theta(\vy,\vx)$. The first condition is largely inherent to standard deep learning architectures; compositions of affine layers with bounded weights and 1-Lipschitz activations (e.g., ReLU) naturally preserve Lipschitz continuity with respect to the input \cite{miyato2018spectral,virmaux2018lipschitz}. Therefore, the critical challenge lies in guaranteeing the second condition: contractivity with respect to the state $\vy$.
A substantial body of literature has sought to enforce this contractivity by construction (e.g., \cite{el2021implicit,winston2020monotone,jafarpour2021robust,revay2020lipschitz,havens2023exploiting}). These approaches typically impose rigid structural constraints, such as parameterizing the model as a one-layer nonlinear MLP: $\cG_\theta(\vy,\vx)=\sigma(\mA\vy+\mB\vx+\vb)$ and strictly bounding the spectral norm of $\mA$, or enforcing global monotonicity.
However, these methods generally enforce a \textit{uniform} contraction modulus $\mu$ across the entire domain $\vx \in \sX$. Our theoretical analysis suggests that such uniformity fundamentally undercuts the unique expressive advantage of implicit models. As illustrated in Figure \ref{fig:toy-diagram}, for a sequence of continuous iterates $\vy_t(\vx)$ to converge to a target $\cF(\vx)$ that is discontinuous or has singularities, the convergence \textit{cannot} be uniform. This implies that the convergence rate—and consequently the operator's contraction modulus $\mu(\vx)$—must be adaptive, varying with $\vx$ to allow for slower convergence in complex regions. Enforcing a globally uniform $\mu$ severs this adaptive capability, thereby severely constraining the model's expressive power.
Therefore, developing novel regularization techniques that can guarantee \textit{adaptive} contractivity (ensuring $0 < \mu(\vx) < 1$ locally while allowing it to vary over $\vx$) is a critical direction for future research to balance theoretical stability with maximal expressivity.
\section{Experiment Details regarding Image Reconstruction}
\label{sec:app-exp-inv}
This section complements the main text with additional implementation and dataset details for the inverse-problem experiments.
\textbf{Experiment settings.} We consider an image deblurring task, $\vx = \mA(\vy_\ast) + \vn$, where $\mA$ is the blur operator and $\vn$ is the Gaussian noise ($\sigma=0.03$). We use a motion-blur operator, and the blur kernel is the first of the eight kernels from \cite{Levin_2009_CVPR}. Ground-truth images $\vy_\ast$ come from BSDS500 \cite{MartinFTM01}. We follow the official splits (200 train / 100 validation / 200 test) and apply a random $128 \times 128$ crop to each image. For each $\vy_\ast$, we generate the corresponding $\vx$ by applying $\mA$ and adding noise. The resulting pairs $(\vx,\vy_\ast)$ form three datasets $\sD_{\textup{inv,train}}$, $\sD_{\textup{inv,val}}$, and $\sD_{\textup{inv,test}}$ for training, validation, and testing, respectively.
In both PGD and HQS style parameterizations (\eqref{eq:pgd-deq} and \eqref{eq:hqs-deq}), the operator $\cH$ is implemented with DRUNet \cite{zhang2021plug}.
\textbf{Training.} We initialize $\cH$ using pretrained weights from the Deepinv library \cite{tachella2025deepinverse} and then fine-tune the full implicit models on the BSDS500 training set for this deblurring task. Training follows the vanilla Jacobian-based implicit differentiation and is implemented on top of the official Deepinv framework.
All models were trained with Adam (learning rate \(10^{-4}\), batch size \(3\)).
Explicit baselines were trained for \(20\) epochs, and the implicit models for \(10\) epochs.
After each epoch we evaluated on the validation set and saved the checkpoint; the final model used for testing is the one with the lowest validation loss. These epoch budgets were sufficient for validation-loss convergence.
\textbf{PSNR.} PSNR (Peak Signal-to-Noise Ratio) is defined between a reference $\vy^\ast$ and reconstructed image $\vy$ as
\[\textup{PSNR}(\vy,\vy^\ast):= 10 \log_{10}\left(\frac{n \cdot \textup{MAX}^2}{\|\vy-\vy^\ast\|^2}\right)\]
where $n$ is the dimension of $\vy$ and $\vy^\ast$, and MAX means the max possible pixel value (e.g., 255 for 8-bit, or 1 if images are in $[0,1]$. In our context, it is 1. Higher PSNR means better (more accurate) reconstruction.
\textbf{Standard test set.}
Evaluation uses the \(200\) images from the official BSDS500 test split, randomly cropped to \(128\times128\).
Let \(\sD_{\textup{inv,test}}=\{(\vx_i,\vy_i^\ast)\}_{i=1}^{200}\), where
\[
\vx_i \;=\; \mA(\vy_i^\ast) + \vn_i,\qquad \vn_i \sim \cN(\vzero,\sigma^2 \mI),\ \ \sigma=0.03.
\]
Here \(\vy_i^\ast\) denotes the clean (ground-truth) image and \(\vx_i\) its corresponding blurred–noisy observation under the forward model \(\mA\).
\textbf{Perturbed test set.} To empirically validate our theory, we created a perturbed version of the test set. To create a diverse and representative set of perturbations, we generate perturbations that correspond to different frequency levels. Image frequencies represent different levels of detail, where low frequencies capture smooth, large-scale areas, and high frequencies capture sharp edges and fine textures. By probing the model with perturbations across this spectrum, we can comprehensively evaluate its behavior.
Specifically, we construct each perturbation by targeting a singular vector of the forward operator $\mA$. Because \(\mA\) is (circular) convolution, its singular vectors are Fourier modes. For each image \(\vy^\ast_i\) and each frequency magnitude \(f\in\{0.1,0.3,0.5,0.7,0.9\}\), we first identify the 2D discrete Fourier frequencies and sort them by their geometric distance from the origin. We then select the frequency coordinate $(u,v)$ at the $f$-th percentile of this sorted list. A one-hot tensor is created in the Fourier domain with a value of 1.0 at the chosen $(u,v)$ position and zeros elsewhere. This sparse frequency representation is transformed back into the image domain by applying the adjoint of the blur operator, $\mA^\top$. These perturbations are visualized in Figure \ref{fig:inv-visualize-delta}. Adding them to \(\vy_i^\ast\) respectively yields perturbed clean images \(\vy_{i,j}^\ast\) (\(j=1,\dots,5\)); we then form the corresponding observation
\[
\vx_{i,j} \;=\; \mA(\vy_{i,j}^\ast) + \vn_{i},\qquad \vn_{i}\sim\cN(\vzero,\sigma^2 \mI).
\]
The perturbed evaluation set is
\[
\sD'_{\textup{inv,test}}
\;=\;
\big\{\,(\vx_{i,j},\vy_{i,j}^\ast)\ :\ 1\le i\le 200,\ 1\le j\le 5\,\big\}.
\]
For convenience we also define the unperturbed index \(j=0\) by \(\vx_{i,0}:=\vx_i\) and \(\vy_{i,0}^\ast:=\vy_i^\ast\).
\textbf{Platform.}
All experiments were run on a workstation with eight Quadro RTX~6000 GPUs.
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{method1_by_freq_0.10.png}
\centering
frequency=0.1
\label{fig:delta-1}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{method1_by_freq_0.30.png}
\centering
frequency=0.3
\label{fig:delta-2}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{method1_by_freq_0.50.png}
\centering
frequency=0.5
\label{fig:delta-3}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{method1_by_freq_0.70.png}
\centering
frequency=0.7
\label{fig:delta-4}
\end{subfigure}
\begin{subfigure}[b]{0.19\textwidth}
\includegraphics[width=\linewidth]{method1_by_freq_0.90.png}
\centering
frequency=0.9
\label{fig:delta-5}
\end{subfigure}
\caption{Visualized perturbations for inverse problems.}
\label{fig:inv-visualize-delta}
\end{figure}
\textbf{Additional Experiments.}
Implicit models often excel on imaging tasks, but a natural question is whether simply stacking more explicit layers (i.e., deepening the model) can close the gap. To probe this, we construct explicit counterparts to implicit models by untying the parameters across iterations:
\[
\min_{\Theta} \E_{\vx} {\ell(\vy_T,\vy_\ast)}, \quad \textup{s.t. } \vy_t = \cG_{\Theta^{(t)}}(\vy_{t-1},\vx), ~~ t=1,\cdots,T
\]
where each block $\cG_{\Theta^{(t)}}$ has the same architecture as in the implicit case (PGD or HQS), but $\Theta^{(t)}$ are separate for each $t$. This is equivalent to stacking $T$ blocks to form a deeper explicit model with more learnable parameters. Unlike implicit models (which can use different iteration counts at train vs.\ test), these explicit models must use the same \(T\) for both training and testing. We evaluated $T \in \{1, 2, 4, 8, 16, 32\}$ to compare against the corresponding implicit models.
\begin{table}[t]
\small
\centering
\caption{Deeper explicit models vs implicit models for image deblurring (PGD architecture). ``Exp ($\times T$)'' represents an explicit model $T$ times deeper than the implicit baseline. O/M denotes CUDA Out of Memory during training. }
\label{tab:inv-deeper}
\begin{tabular}{l|l|l|l|l|l|l|l}
\hline
& Exp ($\times 1$) & Exp ($\times 2$) & Exp ($\times 4$) & Exp ($\times 8$) & Exp ($\times 16$) & Exp ($\times 32$) & Implicit \\ \hline\hline
Params. & 32.641 M & 65.282 M & 130.56 M & 261.13 M & 522.26 M & 1044.5 M & 32.641 M \\ \hline
PSNR & 27.14 dB & 27.64 dB & 27.89 dB & 28.11 dB & 28.27 dB & O/M & 28.21 dB \\ \hline
\end{tabular}
\vspace{5mm}
\caption{Deeper explicit models vs implicit models for image deblurring (HQS architecture). ``Exp ($\times T$)'' represents an explicit model $T$ times deeper than the implicit baseline. O/M denotes CUDA Out of Memory during training. }
\label{tab:inv-wider}
\begin{tabular}{l|l|l|l|l|l|l|l}
\hline
& Exp ($\times 1$) & Exp ($\times 2$) & Exp ($\times 4$) & Exp ($\times 8$) & Exp ($\times 16$) & Exp ($\times 32$) & Implicit \\ \hline\hline
Params. & 32.641 M & 65.282 M & 130.56 M & 261.13 M & 522.26 M & 1044.5 M & 32.641 M \\ \hline
PSNR & 26.94 dB & 28.02 dB & 28.35 dB & 28.69 dB & 28.87 dB & O/M & 29.18 dB\\ \hline
\end{tabular}
\end{table}
Tables~\ref{tab:inv-deeper} and \ref{tab:inv-wider} report results on image deblurring. Across both PGD and HQS settings, deepening explicit models increases parameter counts massively (up to $\sim 1$ billion) but yields diminishing returns in PSNR. Crucially, the implicit models achieve performance comparable to or better than explicit models that are $16\times$ deeper, while using a fraction of the parameters (32.6 M vs. 522 M). For instance, in the HQS setting, the implicit model (29.18 dB) outperforms the explicit model with 16 unrolled blocks (28.87 dB).
Furthermore, training extremely deep explicit models (e.g., $T=32$) becomes infeasible due to memory constraints (O/M). This highlights the distinct efficiency advantage of the weight-tied implicit approach: it theoretically allows for infinite depth (realized here as 100 test-time iterations) while maintaining constant parameter counts ($32.6$ M) and memory usage.
\section{Experiment Details regarding Scientific Computing}
\label{sec:app-ns-exp}
\textbf{Model structure and training.}
Given cell-averaged forces $\vx\!\in\!\sR^{H\times W\times2}$ and vorticities $\vy\!\in\!\sR^{H\times W\times1}$, where $H$ means the height and $W$ means the width, we learn
\[
\vz_\ast = \cG_{\Theta}\!\big(\vz_\ast,\ \cQ_{\Phi}(\vx)\big),
\qquad
\vy_\ast = \cQ_{\Psi}(\vz_\ast),
\]
where $\vz_\ast\!\in\!\sR^{H\times W\times C}$ is a latent field with $C$ channels. At inference, we iterate
\[\vz_{t}=\cG_{\Theta}\!\big(\vz_{t-1},\ \cQ_{\Phi}(\vx)\big),\]
for $1 \leq t \leq T$ and finally call $\vy_T=\cQ_{\Phi}(\vz_T)$.
The projection $\cQ_{\Phi}$ is a \emph{pointwise} linear encoder applied at each grid cell to lift into $C$ channels. In particular, $\vg=\cQ_{\Phi}(\vx)$ reads
\[ \vg = \mW_{1} \vx + \vb_{1} \in \sR^{H\times W\times C} \]
where $\Phi=(\mW_{1},\vb_{1})$ are learnable parameters.
The core map $\cG_{\Theta}(\vz,\vg)$ stacks $L$ identical FNO layers with input injection:
\begin{align*}
&\vz^{(0)} = \vz \\
&\vz^{(l)} = \sigma\left( \vg + \mW^{(l)}_{2} \vz^{(l-1)} + \vb^{(l)}_2 + \textup{IFFT} (\mR^{(l)} \cdot \textup{FFT}(\vz^{(l-1)}) ) \right) , ~~ l = 1,2,\cdots,L,\\
&\cG_{\Theta}(\vz,\vg) = \vz^{(L)}
\end{align*}
where $\Theta = \{\mW^{(l)}_{2}, \vb^{(l)}_2, \mR^{(l)}\}_{l=1}^L$ are learnable parameters.
Each layer:
(i) performs a {global spectral convolution} on $\vz$: take an FFT of the $C$-channel tensor, keep only a small set of low Fourier modes. Suppose the number of retained Fourier modes is $K \times K$ (2D FFT), $\textup{FFT}(\vz)\in\sC^{K \times K \times C}$. For each retained mode $(k_1,k_2)$ multiply the $C$-dimensional channel vector by a learnable dense matrix $\mR^{(l)}_{k_1,k_2}\in\sC^{C \times C}$ (mixing channels) and hence the overall matrix is of size $\mR^{(l)}\in\sC^{K \times K \times C \times C}$, then apply an inverse FFT;
(ii) adds a {local pointwise transform}, adds the injected encoder features $\cQ_{\Phi}(\vx)$, and applies a nonlinearity. This realizes a resolution-invariant, globally receptive operator that naturally respects periodic boundary conditions.
Finally, we decode with the pointwise readout $\cQ_{\Psi}$ (a small per-cell two-layer MLP) to produce $\vy \in \sR^{H\times W\times1}$ where $\Psi$ are learnable parameters.
All samples use \(H=W=128\). Unless stated otherwise, we set the latent width \(C=32\), retain \(K=12\) Fourier modes per dimension in the FNO blocks, and use \(L=3\) FNO layers inside \(\cG_\Theta\). Training differentiates implicitly through the fixed point, and the fixed-point solver uses Anderson acceleration. We optimize with Adam (learning rate \(5\times10^{-3}\), batch size \(16\)). For explicit baselines, we train for \(500\) epochs, which suffices for the training loss to converge.
\textbf{Perturbed data generation.}
In this paragraph, we describe how we generate perturbed samples in $\sD^\prime_{\textup{pde,test}}$. We take the dataset of \cite{marwah2023deep} as the unperturbed set $\sD_{\textup{pde,test}}$ and create perturbations by linearizing the steady NS equation \eqref{eq:ns-velocity}. Each sample $(f,\omega)$ comprises a forcing term $f$ and its vorticity solution $\omega$. Directly prescribing $f$ and solving for $\omega$ is computationally costly; following \cite{marwah2023deep}, we instead prescribe $\omega$ and obtain the corresponding $f$ by evaluating the PDE operator (not by solving the PDE). In our setting, the base samples are given; thus we first construct a solution perturbation $\delta\omega$ and then compute the induced forcing perturbation $\delta f$ via the linearization, yielding the perturbed pair $(f+\delta f,\ \omega+\delta\omega)$.
Note that, while the dataset is discrete, we use the continuous notation $f,\omega,u$ in this section to ease reading and to remain consistent with the PDE literature. In addition, we use $\xi = (\xi_1,\xi_2)$ as the special domain variable to keep consistent with our main text, and use $k=(k_1,k_2)$ as the frequency domain variable.
(Generate $\delta\omega$).
Fix a target wavenumber $k_\ast\in\sN$ and a desired $L^2$–magnitude $\eta>0$. We construct $\delta \omega$ by
\[
\delta\omega(\xi_1,\xi_2)=A\,\sin\!\big(k_\ast \xi_1 + k_\ast \xi_2\big),
\qquad
A\ \text{chosen so that}\ \ \|\delta\omega\|_{L^2(\Omega)}=\eta.
\]
The wavenumber is selected from a user–specified frequency percentile $p_{\textup{freq}}$ relative to the maximum resolvable frequency $k_{\mathrm{max}}=H/2=W/2$, namely
\[
k_\ast = p_{\textup{freq}} \times k_{\mathrm{max}}
\quad\text{(rounded to the nearest integer mode).}
\]
In our code we set the grid size $H=W=128$, the perturbation strength $\eta=0.01$, and choose
\[
p_{\textup{freq}} \in \{0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.96, 0.97, 0.98, 0.99\}.
\]
Accordingly, each original sample yields $15$ perturbed samples.
(Generate velocity from vorticity).
Given a scalar vorticity field $\omega$ (and its perturbation $\delta\omega$), we recover the corresponding velocities $u$ via a streamfunction $\psi$ given by
\[
u = (\partial_2 \psi,\,-\partial_1 \psi),
\qquad
\omega = -\,\Delta \psi .
\]
Hence $\psi$ is obtained by solving the Poisson equation $\Delta \psi = -\,\omega$, after which we obtain $u$.
On a periodic grid, these operators are implemented efficiently in the Fourier domain.
(Linearization and the perturbed vorticity forcing $\delta g$).
Applying the (scalar) curl ``$\nabla\times$'' to both sides of \eqref{eq:ns-velocity} yields the steady vorticity form
\[
(u\!\cdot\!\nabla)\,\omega \;-\; \nu\,\Delta \omega = g,
\qquad
g = \nabla\times f = \partial_1 f_2 - \partial_2 f_1 ,
\]
where $f=(f_1,f_2)$ is the body force and $g$ is its curl. Introducing perturbations $(\delta u,\delta\omega,\delta g)$ and expanding
\[
(u+\delta u)\!\cdot\!\nabla\,(\omega+\delta\omega) \;-\; \nu\,\Delta(\omega+\delta\omega)
= g+\delta g,
\]
then subtracting the base equation and discarding higher–order terms gives the first–order relation
\[
\delta g
=
(u\!\cdot\!\nabla)\,\delta\omega
\;+\;
(\delta u\!\cdot\!\nabla)\,\omega
\;-\;
\nu\,\Delta\,\delta\omega .
\]
Again, for numerical implementation on a periodic grid, the differential operators are applied efficiently in the Fourier domain.
(Recover the vector force $\delta f$ from its curl $\delta g$).
We recover a periodic $\delta f=(\delta f_1,\delta f_2)$ satisfying $\nabla\times\delta f=\delta g$ by solving a Poisson equation for an auxiliary streamfunction $\psi$ and obtain $\delta f$ exactly as in ``Generate velocity from vorticity.''
\textbf{Additional Experiments: Scaling Explicit Models.} A natural question is whether the performance gap between implicit and explicit models can be bridged simply by scaling up the explicit architecture (i.e., stacking more layers or increasing channel width). To investigate this, we compared the implicit FNO against explicit baselines scaled significantly in two dimensions: depth (up to $32\times$) and width (up to $8\times$). The results, summarized in Table~\ref{tab:fno-deeper} and Table~\ref{tab:fno-wider}, demonstrate that while scaling explicit models yields modest accuracy gains, it faces severe diminishing returns and computational bottlenecks (eventually leading to CUDA Out-of-Memory errors). Crucially, the implicit model achieves markedly better performance (lowest relative error of $0.0785$) than even the largest viable explicit models, despite the explicit counterparts using over $10\times$ the number of parameters (e.g., $29.06$ M for Exp($\times 16$) vs. $2.376$ M for Implicit). This confirms that the implicit formulation provides an expressive advantage that cannot be efficiently replicated by simply allocating more capacity to an explicit solver.
\begin{table}[t]
\centering
\caption{Implicit FNO vs deeper Explicit FNO. ``Exp($\times l$)" denotes an explicit model that is $l$ times deeper than the implicit FNO. O/M indicates a CUDA out-of-memory error during training.}
\label{tab:fno-deeper}
\begin{tabular}{c|c|c|c|c|c|c|c}
\hline
& Exp($\times 1$) & Exp($\times 2$) & Exp($\times 4$) & Exp($\times 8$) & Exp($\times 16$) & Exp($\times 32$) & Implicit \\ \hline\hline
Params. & 2.376 M & 4.155 M & 7.713 M & 14.83M & 29.06M & 57.52M & 2.376 M \\ \hline
Rel. Err.
% & 0.1851
& 0.1787
& 0.1526 & 0.1410 & 0.1380 & 0.1360 & O/M & 0.0785 \\ \hline
\end{tabular}
\end{table}
\begin{table}[t]
\centering
\caption{Implicit FNO vs wider Explicit FNO. ``Exp($\times w$)" denotes an explicit model that is $w$ times wider than the implicit FNO. O/M indicates a CUDA out-of-memory error during training.}
\label{tab:fno-wider}
\begin{tabular}{c|c|c|c|c|c}
\hline
& Exp($\times 1$) & Exp($\times 2$) & Exp($\times 4$) & Exp($\times 8$) & Implicit \\ \hline\hline
Params. & 2.376 M & 9.504 M & 38.01 M & 152.0M & 2.376 M \\ \hline
Rel. Err.
% & 0.1851
& 0.1787 & 0.1555 & 0.1401 & O/M & 0.0785 \\ \hline
\end{tabular}
\end{table}
Note: These findings are broadly consistent with \cite{marwah2023deep}. We follow their setup with two minor deviations: we use a smaller training batch size (16) due to hardware limits, and while we keep $T=24$ training iterations for the implicit model, at inference we run $T=50$, because we observe that the trained implicit models remain stable and often benefit from additional fixed-point iterations at test time.
\section{Experiment Details regarding LP}
\label{sec:app-lp-exp}
\textbf{GNN model details.}
We implement \eqref{eq:gnn-structure}:
\[\vz_\ast = \cG_{\Theta}(\vz_\ast,\cQ_{\Phi}(\vx)), \quad \vy_\ast = \cQ_{\Psi}(\vz_\ast) \]
with an $L$-layer message-passing GNN \cite{scarselli2008graph,xu2018how} on the bipartite graph. Let $\cN(i)$ (resp. $\cN(j)$) be the neighbors of constraint node $W_i$ (resp. variable node $V_j$). With shared MLPs across all nodes and edges, the GNN structure is given by:
\begin{align*}
& \textup{Input-embedding: } && W_i^{(0)} = \textup{MLP}_{\phi_1}(b_i,\circ_i), \\
& && V_j^{(0)} = \textup{MLP}_{\phi_2}(c_j,l_j, u_j, z_{\mathrm{in},j}) \\
& \textup{Message-passing }(1\leq l \leq L-1): && W_i^{(l)} = \textup{MLP}_{\theta_1^{(l)}}\left( W_i^{(l)}, \sum_{j \in \cN(i)} A_{ij} \cdot \textup{MLP}_{\theta_2^{(l)}}\left( V_j^{(l-1)}\right) \right), \\
& && V_j^{(l)} = \textup{MLP}_{\theta_3^{(l)}}\left( V_j^{(l)}, \sum_{i \in \cN(j)} A_{ij} \cdot \textup{MLP}_{\theta_4^{(l)}}\left( W_i^{(l-1)}\right) \right) \\
& \textup{Output-embedding: } && z_{\mathrm{out},j} = \textup{MLP}_{\theta_5}\left( V_j^{(L)} \right)
\end{align*}
We write this compactly as follows.
\[ \vz_{\mathrm{out}} = \cG_{\Theta}(\vz_{\mathrm{in}},\cQ_{\Phi}(\vx)) \]
where $\Theta = \left\{ \{\theta_1^{(l)}\}_{l=1}^{L-1},\{\theta_2^{(l)}\}_{l=1}^{L-1},\{\theta_3^{(l)}\}_{l=1}^{L-1},\{\theta_4^{(l)}\}_{l=1}^{L-1}, \theta_5 \right\}$ are trainable parameters in the GNN, $\Phi = \{\phi_1,\phi_2\}$ includes the trainable parameters of the input embedding. The input $\vx$ includes all static information $\vx := (\mA,\vb,\vc,\circ,\vl,\vu)$. Finally, the output embedding $\vy = \cQ_{\Psi}(\vz)$ is given by
\[ y_j = \textup{MLP}_{\Psi}(z_j) \]
for every variable node $j$. All MLPs in $\cG_{\Theta}$, $\cQ_{\Phi}$, and $\cQ_{\Psi}$ use two layers with ReLU activations. We sweep widths (or embedding sizes) in $\{4,8,16,32\}$ and report results in the main text.
Note that $l$ is the layer index within the GNN structure, not the iteration number $t$. All parameters in $\Theta$ are independent of the iteration number, so this GNN can be applied iteratively. $\vx$ is the static features and $\vz$ is the dynamic feature.
In addition, removing the dynamic input $z_{\mathrm{in}}$ and decoding directly to $\vy$ recovers the standard (explicit) GNN baseline.
\textbf{Dataset generation.}
We largely follow \cite{chen2023lp} to construct the training set $\sD_{\textup{LP,train}}$ and test set $\sD_{\textup{LP,test}}$, drawing $(\mA,\vb,\vc,\circ,\vl,\vu)$ i.i.d.\ from the same distribution. Each LP has $50$ variables and $10$ constraints. The matrix $\mA$ is sparse with $100$ nonzeros whose locations are chosen uniformly at random and whose values are sampled from a standard normal distribution. Entries of $\vb$ and $\vc$ are sampled i.i.d.\ from $\mathrm{Unif}[-1,1]$, after which $\vc$ is scaled by $0.01$. Variable bounds $\vl,\vu$ are sampled coordinatewise from $\mathcal{N}(0,10)$; whenever $l_j>u_j$ we swap them. Constraint types are sampled independently with $\Pr(\circ_i=``\leq")=0.7$ and $\Pr(\circ_i=``=")=0.3$. Under this generator, the feasibility probability is approximately $0.53$; we retain only feasible instances, yielding $2{,}500$ LPs for training and $1{,}000$ for testing. Solutions are computed with \texttt{scipy.optimize}.
To build the perturbed datasets $\sD^{(j)}_{\textup{LP,test}}$, we perturb one component at a time while holding the others fixed. For $\vc$, draw $\delta\vc$ with i.i.d.\ standard normal entries, normalize, and scale to magnitude $10^{-4}$:
\[
\vc^\prime=\vc+10^{-4}\times\frac{\delta\vc}{\|\delta\vc\|}.
\]
We apply the same procedure to $\vb$, $\vl$, and $\vu$. For $\mA$, we perturb only existing nonzeros to preserve the sparsity pattern: let $\sS=\{(i_k,j_k)\}_{k=1}^{\mathrm{nnz}(\mA)}$ be the nonzero locations and draw $\delta\va\in\sR^{|\sS|}$ i.i.d.\ standard normal; normalize and scale so $\|\delta\va\|=10^{-4}$, then set
\[
\mA^\prime_{i_k,j_k}=\mA_{i_k,j_k}+(\delta\va)_k\ \ \text{for }(i_k,j_k)\in\sS,\qquad
\mA^\prime_{i,j}=\mA_{i,j}\ \text{otherwise}.
\]
This yields five perturbed versions (perturbing $\mA$, $\vb$, $\vc$, $\vl$, or $\vu$ separately). We evaluate the estimated Lipschitz constants $L_t$ and relative errors $E_t$ on each version and report the results in the main text.
\textbf{Training method.} To train our implicit GNNs, we employ a two-stage curriculum strategy. The model is trained by unrolling its iterative updates for a fixed number of steps, T, and minimizing the loss on the final output:
\begin{align*}
\min_{\Theta,\Phi,\Psi} & \sum_{(\vx,\vy_\ast) \in \sD_{\textup{LP,train}}} \ell(\vy_T, \vy_\ast) \\
\textup{s.t. } & \vz_0 = \vzero \\
& \vz_{t} = \cG_\Theta(\vz_{t-1},\cQ_{\Phi}(\vx)), \quad t=1,2,\cdots,T \\
& \vy_T = \cQ_\Psi(\vz_T)
\end{align*}
We set the final unroll horizon to $T=6$, as we observed no significant improvements with longer sequences. Training directly with $T=6$ is inefficient, so we adopt a two-stage curriculum. This approach is a standard practice in the Learning to Optimize field for training implicit or unrolled models that solve optimization problems \cite{chen2022learning}. This approach begins with a shorter unroll horizon and a larger learning rate, using the trained model to warm-start the subsequent stage with a longer horizon and a reduced learning rate. This strategy is often described as ``layerwise training" \cite{chen2018theoretical,liu2019alista} or ``curriculum learning" \cite{chen2020training,liu2023towards,he2024mathematics}. In our settings: Stage 1 uses $T=3$ with a learning rate $0.01$; Stage 2 uses $T=6$ with a learning rate $10^{-4}$. Both stages use Adam optimizer.
For a fair comparison, the non-iterative explicit GNNs are trained using the same two-stage learning rate schedule. This regimen proved effective, as the training errors for our explicit baselines surpassed those reported in prior work \cite{chen2023lp}.
At the inference time, $T$ can be chosen as the unroll length in the training stage, or moderately longer. In our experiments, we use $T=8$ at the inference time, as we do not observe significant improvement with a larger number of iterations.
\textit{Remark.} While we employ unrolled training rather than the vanilla Jacibian-based implicit differentiation, we classify our approach as an “implicit model” because the underlying architecture, a weight-tied update $\vz_{t} = \cG_\Theta(\vz_{t-1},\cQ_{\Phi}(\vx))$, remains identical. The distinction lies solely in the numerical implementation: as established by \cite{geng2021training}, unrolled training is mathematically equivalent to a Neumann series approximation of the implicit gradient. Thus, unrolling and root-finding are simply two valid strategies for approximating the same fixed point, $\vy_\ast(\vx)$. This equivalence is widely recognized in the Implicit GNN literature, where Neumann approximations are standard for scaling to large graphs (e.g., \cite{baker2023implicit}). Since our focus is on expressivity rather than optimization mechanics, we treat both formulations as belonging to the same model class.
\section{Broader contextual discussions}
\label{app:refs}
While our work primarily establishes the expressive power of implicit models through the lens of fixed-point iterations, we situate our contributions within the broader landscape of implicit model theory in this section.
\textbf{Universality and expressivity.} Despite its foundational importance, a general and systematic theory of expressive power for implicit models remains largely open. To our knowledge, existing results address only specific facets of the problem.
% For instance, \cite{bai2019deep} demonstrated that for an explicit deep network $f$, an operator $g$ exists whose fixed point reproduces $f$. However, this proof of existence does not guarantee that the fixed-point iteration \eqref{eq:implicit-iterate} will actually converge to the fixed point. Other works have established universality for specific, structured classes of implicit models tailored to problems like steady-state PDEs \cite{marwah2023deep}, or have proven separation results showing that some implicit models can express functions that large explicit models cannot \cite{wu2024separation}. While insightful, none of these results provide a complete characterization of the representable function class of implicit models or direct answers to questions (Q1) and (Q2) in the introduction.
For example, while \cite{bai2019deep} demonstrated that an operator $g$ exists such that its fixed point reproduces any explicit network $f$, this existence result crucially does not guarantee that the fixed-point iteration \eqref{eq:implicit-iterate} actually converges. Other works have established universality within restricted domains, such as steady-state PDEs \cite{marwah2023deep}, or proven separation results where implicit models outperform explicit counterparts in specific settings \cite{wu2024separation}. While insightful, none of these studies provide a complete characterization of the general function class representable by implicit models, nor do they directly address the fundamental questions (Q1) and (Q2) raised in our introduction.
\textbf{Contrast with explicit models.} First, consider the setting where the model is subject to a global Lipschitz constraint (e.g., $\text{Lip}(f_\theta) \le 1$), which is common for robustness and stability. In this case, explicit feedforward networks are mathematically strictly limited to representing globally 1-Lipschitz maps \cite{murari2025approximation}. Consequently, they are fundamentally incapable of expressing locally Lipschitz targets whose gradients become arbitrarily large (such as $1/x$ near zero). In contrast, our work demonstrates that implicit models break this barrier: a simple, globally regular operator (Lipschitz in $\vx$, contractive in $\vy$) can generate complex, locally Lipschitz fixed-point maps via iteration. This ``Simple Operator $\to$ Complex Fixed Point" mechanism is the core difference claimed in our paper.
Second, if we remove constraints, \cite{beneventano2021deep} have shown that deep ReLU networks can indeed approximate locally Lipschitz functions on arbitrary compact sets. However, achieving high precision for such complex targets requires the explicit model size (depth/width) to grow arbitrarily large. Here lies the crucial distinction: explicit models scale expressivity with model size, whereas implicit models are able to scale expressivity with test-time iterations. This allows implicit models to represent increasingly complex functions dynamically without adding parameters.
\textbf{Training Dynamics and Convergence.} A significant body of work focuses on the optimization mechanics of implicit models. \cite{geng2021training} rigorously established the equivalence between unrolled training and implicit differentiation via Neumann series approximations, validating the training methodologies used in our case studies. \cite{ling2023global,truong2025global} provide global convergence guarantees and rate analyses for the training in over-parameterized deep equilibrium models. While these studies ensure that training algorithms can successfully minimize the loss, our work addresses the fundamental antecedent question: whether a model exists that is capable of representing the target function in the first place.
\textbf{Generalization.} Distinct from expressivity, \cite{fung2024generalization} derive generalization bounds for families of implicit networks, characterizing their ability to perform on unseen data. Our analysis focuses on approximation capacity—the ability to construct an operator that exactly reproduces a target map—which is orthogonal to the sample complexity and generalization bounds discussed in their work.
\textbf{Infinite-Width Limits and Kernel Connections.} Recent research has sought to bridge the gap between implicit models, explicit deep networks, and kernel methods. \cite{gao2022a} extend the over-parameterization theory of explicit networks to implicit models, establishing well-posedness and convergence even in finite-width regimes where standard infinite-depth results do not directly apply. In the infinite-width limit, \cite{feng2023neural} formally derive the Neural Tangent Kernel (NTK) for equilibrium models, characterizing their training dynamics in the linear regime. On the architectural side, \cite{ling2024deep} show that for high-dimensional Gaussian mixtures, deep equilibrium models can be functionally equivalent to shallow explicit networks. In contrast to these kernel-based or distribution-specific analyses, \textit{our work adopts a non-parametric function-space perspective}; we demonstrate that for general locally Lipschitz targets, the expressive power of implicit models is not static but scales dynamically with test-time computation, a property distinct from the linear regimes often studied in kernel theory.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\DeclareMathOperator*{\mean}{mean}
% Graph
\def\cA{{\mathcal{A}}}
\def\cB{{\mathcal{B}}}
\def\cC{{\mathcal{C}}}
\def\cD{{\mathcal{D}}}
\def\cE{{\mathcal{E}}}
\def\cF{{\mathcal{F}}}
\def\cG{{\mathcal{G}}}
\def\cH{{\mathcal{H}}}
\def\cI{{\mathcal{I}}}
\def\cJ{{\mathcal{J}}}
\def\cK{{\mathcal{K}}}
\def\cL{{\mathcal{L}}}
\def\cM{{\mathcal{M}}}
\def\cN{{\mathcal{N}}}
\def\cO{{\mathcal{O}}}
\def\cP{{\mathcal{P}}}
\def\cQ{{\mathcal{Q}}}
\def\cR{{\mathcal{R}}}
\def\cS{{\mathcal{S}}}
\def\cT{{\mathcal{T}}}
\def\cU{{\mathcal{U}}}
\def\cV{{\mathcal{V}}}
\def\cW{{\mathcal{W}}}
\def\cX{{\mathcal{X}}}
\def\cY{{\mathcal{Y}}}
\def\cZ{{\mathcal{Z}}}
\newcommand{\Urt}{\sU_r(\sM)}
\newcommand{\Utt}{\sU_\tau(\sM)}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\Id}{\mathrm{Id}}
% \newtheorem{theorem}{Theorem}
% \newtheorem{lemma}[theorem]{Lemma}
% \newtheorem{remark}[theorem]{Remark}
% \newtheorem{proposition}[theorem]{Proposition}
% \newtheorem{assumption}[theorem]{Assumption}
% \newtheorem{definition}[theorem]{Definition}
% \newtheorem{corollary}[theorem]{Corollary}
\allowdisplaybreaks
\usepackage{amsthm}
% Number theorems within sections so appendix gets A.1, A.2, ...
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% (optional but nice)
% \usepackage[hidelinks]{hyperref}
% \usepackage[nameinlink,capitalize,noabbrev]{cleveref}