Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inference
Extracted source text · arXiv 2511.21223
\documentclass[a4paper,10pt]{article}
\usepackage{amsmath,amssymb,amsthm,mathtools,bm}
\usepackage[inline]{enumitem}
\usepackage{MnSymbol}
\usepackage[dvipsnames]{xcolor}
\usepackage{hyperref}
\usepackage{booktabs,tabularx}
\usepackage{mathtools}
\mathtoolsset{showonlyrefs}
\PassOptionsToPackage{sort,compress}{natbib}
\usepackage[preprint]{eiml_style_2025}
\urlstyle{same}
\definecolor{darkred}{rgb}{0.55, 0.0, 0.0}
\definecolor{darkblue}{rgb}{0.0, 0.0, 0.55}
\hypersetup{
colorlinks=true,
linkcolor=darkblue,
citecolor=darkred,
urlcolor=darkblue
}
\newcolumntype{L}{>{\raggedright\arraybackslash}X}
\DeclareMathOperator{\conv}{conv}
\DeclareMathOperator*{\opt}{opt}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argsup}{argsup}
\DeclareMathOperator*{\arginf}{arginf}
\newcommand{\ubar}[1]{\text{\b{$#1$}}}
\newcommand{\leg}[1]{#1^{\dagger}} % Legendre of command input, e.g. \leg{A}
\newcommand{\probexp}{\mathbb{E}} % Probabilistic expectation
\newcommand{\possexp}{\mathbb{E}^{\star}} % Possibilistic expectation
\newcommand{\mode}[1]{#1^{\star}} % Mode of a random variable, e.g. \mode{\theta}
\newcommand{\objmin}[1]{\ubar{#1}} % Minimizer of the objective -l(θ)-log g(θ), argmin for lower CBO
\newcommand{\objmax}[1]{\bar{#1}} % Maximizer of the objective -l(θ)-log g(θ), argmax for upper CBO
\newcommand{\loss}{\hat{\ell}} % Regularized loss
\newcommand{\given}{\,|\,} % given |
% Auto-adjust size of paranthesis
\newcommand{\lr}[3]{\left#1#2\right#3}
\newcommand{\rb}[1]{\lr{(}{#1}{)}}
\renewcommand{\sb}[1]{\lr{[}{#1}{]}}
\newcommand{\cb}[1]{\lr{\{}{#1}{\}}}
\newcommand{\innerprod}[1]{\lr{\langle}{#1}{\rangle}}
\newtheorem{corollary}{Corollary}
\theoremstyle{remark}
\newtheorem*{remark}{Remark}
\theoremstyle{definition}
\newtheorem{example}{Example}
\title{Maxitive Donsker–Varadhan Formulation for Possibilistic Variational Inference}
\author{
Jasraj Singh\thanks{Equal contribution.} \\
Nanyang Technological University \\
\And Shelvia Wongso$^*$ \\
Nanyang Technological University
\AND Jeremie Houssineau \\
Nanyang Technological University \\
\And Badr-Eddine Ch\'erief-Abdellatif \\
Sorbonne Universit\'e
}
\date{}
\newtheorem{theorem}{Theorem}
\newtheorem{proposition}{Proposition}
\begin{document}
\maketitle
\begin{abstract}
Variational inference (VI) is a cornerstone of modern Bayesian learning, enabling approximate inference in complex models that would otherwise be intractable. However, its formulation depends on expectations and divergences defined through high-dimensional integrals, often rendering analytical treatment impossible and necessitating heavy reliance on approximate learning and inference techniques. Possibility theory, an imprecise probability framework, allows to directly model epistemic uncertainty instead of leveraging subjective probabilities.
While this framework provides robustness and interpretability under sparse or imprecise information, adapting VI to the possibilistic setting requires rethinking core concepts such as entropy and divergence, which presuppose additivity. In this work, we develop a principled formulation of possibilistic variational inference and apply it to a special class of exponential-family functions, highlighting parallels with their probabilistic counterparts and revealing the distinctive mathematical structures of possibility theory.
\end{abstract}
\section{Introduction}
Variational inference (VI) has become a cornerstone of modern Bayesian learning, enabling approximate inference in complex models that were previously intractable. For many years, purely Bayesian methods were hampered by computational cost, but advances in Monte Carlo and especially variational approximations have brought them within reach \citep{blei2017vi-review,salimans2015mcmc-vi-bridge}. In VI, one typically posits a simple (parametric) variational family $\mathcal{Q}$ and then finds the probability distribution $q \in \mathcal{Q}$ that best approximates the true posterior $
\mode{q}_{\mathrm{add}}$ by maximizing the evidence lower bound (ELBO). %, which involves the entropy of $q$ and its KL-divergence from $p$.
This framework underlies many tools today -- e.g.\ variational autoencoders \citep{kingma2022autoencodingvariationalbayes}, some Bayesian neural networks \citep{blundell2015weight-uncertainty,kingma2015reparameterization-trick} -- and blurs the line between tractable and intractable Bayesian analyses.
Introduced by \cite{zadeh1978fuzzy}, possibility theory, an imprecise probability framework, offers a complementary approach for modelling uncertainty. Unlike probability, possibility does not require additivity: events have a degree of plausibility ($\leq 1$) and a dual \textit{necessity}, with logical rules based on min/max operators. This allows handling epistemic uncertainty naturally. For example, the analogue of probability distributions, referred to as possibility functions, can easily be made fully uninformative \citep{hieu2025decoupling}, a difficult endeavour in probability theory.
The flexibility and scalability of possibility theory makes it suitable for challenging applications such as control problems under uncertainty \citep{xue2025orbit} and point process-based inference \citep{houssineau2021linear}.
A key challenge arises when we try to adapt variational methods to this possibilistic setting. Standard VI relies on notions like Shannon entropy, which do not directly carry over: for example, the ELBO includes an entropy term, but there is no obvious ``possibilistic entropy'' analogous to the Shannon entropy of a probability distribution \citep{shannon1949mathematical}. Similarly, divergences like Kullback–Leibler (KL) require additive measures \citep{kullback1951kld}. As a result, traditional VI objectives cannot be used out-of-the-box. Nonetheless, recent work has sought approximations in the possibilistic context. For instance, \cite{cella2025vi-im} developed a variational-like approximation for inferential models \citep{martin2013im}, and use a Monte Carlo-based strategy to search over the chosen family of possibility functions. In that spirit, our goal is to formulate a VI framework for general possibilistic models: we define an objective that aligns a tractable candidate possibility function with the (target) posterior possibility function, and which can be optimized in lieu of maximizing a probabilistic ELBO.
We present a novel maxitive analogue of the classical Donsker–Varadhan (DV) in Section~\ref{sec:possibilistic-vi}, with Theorem\,\ref{thm:maxitive_DV} providing our main theoretical result. In Section~\ref{sec:ExpFam}, we develop a practical framework for possibilistic VI %(PVI)
within a special class of exponential-family functions, highlighting their structural parallels with the probabilistic case and giving rise to special mathematical structures.
These developments provide a foundation for extending variational reasoning to possibility theory, offering new avenues for inference under epistemic uncertainty.
\section{Primer on Possibility Theory}
Possibility theory provides a dedicated representation of epistemic uncertainty where, similarly to probabilities, each event is assigned a degree of possibility between $0$ and $1$ and, unlike probabilities, the main operation on possibilities is the maximum/supremum rather than the sum/integral.
To define epistemic uncertainty formally, we consider a sample space $\Omega$ whose elements characterise all the possible values of the relevant unknown quantities. Instead of equipping $\Omega$ with a probabilistic structure, we simply describe an unknown parameter $\theta_0$ in set $\Theta$ via a (deterministic) \emph{uncertain variable} $\bm{\theta} : \Omega \to \Theta$. If an element $\omega \in \Omega$ were the correct one, then $\bm{\theta}(\omega)$ would be true value of the parameter $\theta_0$. To \emph{describe} the available information about $\bm{\theta}$, we define a \emph{possibility function} $f_{\bm{\theta}}$ (a.k.a.\ possibility distribution) on $\Theta$ as a non-negative function that verifies $\sup_{\theta \in \Theta} f(\theta)=1$. The possibility of an event $\bm{\theta} \in A$ for some $A \subseteq \Theta$ is then $\sup_{\theta \in A} f_{\bm{\theta}}(\theta)$.
\paragraph{Marginalization and conditioning:} Possibility functions behave similar to probability mass functions (p.m.f.), except that summation is replaced by a maximum. For instance, if $\bm{\theta}$ and another uncertain variable $\bm{\psi}$ on a set $\Psi$ are jointly described by a possibility function $f_{\bm{\theta},\bm{\psi}}$, then the marginal possibility function describing $\bm{\theta}$ is characterised by
\begin{equation}
f_{\bm{\theta}}(\theta) =\sup_{\psi \in \Psi} f_{\bm{\theta},\bm{\psi}}(\theta,\psi), \qquad \forall \theta \in \Theta.
\end{equation}
Similarly, for a fixed $\psi \in \Psi$ satisfying $f_{\bm{\psi}}(\psi) > 0$, the conditional possibility function of $\bm{\theta}$ given $\bm{\psi} = \psi$ is characterised by
\begin{equation}
f_{\bm{\theta}}(\theta \given \bm{\psi} = \psi) = \frac{f_{\bm{\theta},\bm{\psi}}(\theta,\psi)}{f_{\bm{\psi}}(\psi)}, \qquad \forall \theta \in \Theta. \label{eqn:conditioning}
\end{equation}
As is standard with probability distributions, we will often omit which uncertain variable is being described by a possibility function and simply write, e.g. $f$ instead of $f_{\bm{\theta}}$.
\paragraph{Possibilistic moments:} Based on \cite{hieu2025decoupling}, the %A useful notion of
expected value for a possibility function $f$ is given by
\begin{align}
\possexp_{f}[\bm{\theta}] \doteq \argmax_{\theta \in \Theta} f(\theta),
\end{align}
which is a set in general, and satisfies $\possexp_f[T(\bm{\theta})] = T( \possexp_f[\bm{\theta}] )$ for any mapping $T$ -- a property shared with the maximum likelihood estimate. When $\possexp_{f}[\bm{\theta}]$ is a singleton $\{\mode{\theta} \}$, we do not make a distinction between this singleton and the element $\mode{\theta}$. In this case, if $f$ is twice differentiable at $\mode{\theta}$, we define the precision as $\mathcal{I}_f( \bm{\theta} ) \doteq \possexp_f[ -\nabla^2 \log f(\bm{\theta}) ] = -\nabla^2 \log f(\theta)|_{\theta = \mode{\theta}}$, where the operator $\nabla^2$ is the Hessian and \emph{precision} is to be understood as inverse covariance matrix.
\begin{example}
The normal possibility function with expected value $\mu$ and positive definite covariance matrix $\Sigma$ is defined as
\begin{equation}
f(\theta) = \overline{\mathrm{N}}\left(\theta;\mu,\Sigma \right) \doteq \exp \left( -\frac{1}{2} (\theta-\mu)^{\intercal}\Sigma^{-1}(\theta-\mu) \right).
\end{equation}
It verifies $\possexp_f[\bm{\theta}] = \mu$ and $\mathcal{I}_f(\bm{\theta}) = \Sigma^{-1}$, as is usually leveraged in the Gaussian approximation (a.k.a.\ Laplace approximation).
\end{example}
\paragraph{Notations:} We define the set $\mathcal{F}(\Theta) \doteq \{ f : \Theta \to [0,1] \; : \; \sup_{\theta \in \Theta} f(\theta) = 1 \}$ of possibility functions over $\Theta$. As opposed to the set $\mathcal{P}(\Theta)$ of probability distributions, $\mathcal{F}(\Theta)$ is a pre-ordered set when equipped with the partial order $\preceq$ defined as
$f \preceq g \iff f(\theta) \leq g(\theta), \forall \theta \in \Theta$.
In particular, any two elements $f$ and $g$ of $\mathcal{F}(\Theta)$ have a least upper bound $f \lor g$ defined for any $\theta \in \Theta$ as $(f \lor g)(\theta) = \max\{f(\theta),\, g(\theta)\}$. There is a greatest element in $\mathcal{F}(\Theta)$, the function equal to $1$ everywhere, which we denote by $\bm{1}$. For any subset $\mathcal{G}$ of $\mathcal{F}(\Theta)$, $\max \mathcal{G}$ denotes the maximal element of $\mathcal{G}$. The minimal element $\min \mathcal{G}$ can also be considered when it exists.
Table~\ref{tab:notations} summarizes other key notations.
\section{Bayesian Inference and the Donsker-Varadhan Variational Formula}
\label{sec:probabilistic-vi}
We first provide a review of probabilistic variational formulation of Bayesian inference and its variational approximations. We consider the problem of learning about an unknown parameter $\theta_0$ in a set $\Theta$, which can be thought of either as the true parameter in a statistical procedure or as the solution of an optimisation problem. In this context, it is usual to alternate between
\begin{enumerate*}[label=\roman*)]
\item the optimisation viewpoint where the main objects are a loss $\ell$ and a regulariser $R$ on $\Theta$, and
\item the inference viewpoint where the main objects are a likelihood $L \propto \exp(- \ell)$, that is, $L(\theta) \propto \exp(- \ell(\theta))$ for all $\theta \in \Theta$, and a prior $\pi$ with density $\propto \exp(- R)$ with respect to Lebesgue's measure.
\end{enumerate*}
This leads to the following formula for the (generalized) Bayesian posterior $q^{\star}_{\textnormal{add}}$ with respect to $\pi$:
\begin{equation}
q^{\star}_{\textnormal{add}}(\mathrm{d}\theta) \;\doteq\; \dfrac{\exp(-\ell(\theta))\,\pi(\mathrm{d}\theta)}{\int \exp(-\ell(\theta'))\pi(\mathrm{d}\theta')} = \dfrac{\exp\left(-\loss(\theta)\right)\mathrm{d}\theta}{\int \exp\left(-\loss(\theta')\right) \mathrm{d}\theta'}\,,
\end{equation}
where $\loss \doteq \ell + R$ is the regularised loss.
We start with the following classical variational formula, known since at least the 1950s (see e.g.\ \citet[Exercise 8.28]{kullback1959information} for the finite case), generally attributed in the general setting to \cite{donsker1983vi}, and later rediscovered in statistics by \cite{zellner1988optimal}. We refer the interested reader to \citet[Page~160]{catoni2004} for a proof.
\begin{theorem}[Donsker and Varadhan's variational formula]
\label{additive_DV_general}
Let $(\Theta,\mathcal{T})$ be a measurable space and let $\nu$ be a probability measure on $\Theta$.
For any measurable function $h:\Theta\to\mathbb{R}$ such that $\int e^{h} \mathrm{d}\nu<+\infty$, we have
\begin{equation}
\log \int e^{h} \mathrm{d}\nu
= \sup_{\rho\in\mathcal{P}(\Theta)}\left\{ \int h \mathrm{d}\rho - \mathrm{KL}\left(\rho\Vert\nu\right)\right\},
\end{equation}
where $\mathcal{P}(\Theta)$ denotes the set of probability measures on $\Theta$ and
$\mathrm{KL}(\rho\Vert\nu)$ is the Kullback--Leibler divergence of $\rho$ with respect to $\nu$, with the convention $\infty-\infty=-\infty$. \newline
Moreover, the supremum on the right-hand side is achieved exactly at the Gibbs measure $\nu_h$ whose density with respect to $\nu$ is
\begin{align}
\frac{\mathrm{d}\nu_h}{\mathrm{d}\nu}(\theta)
\doteq \frac{e^{h(\theta)}}{\int e^{h} \mathrm{d}\nu} .
\end{align}
\end{theorem}
\paragraph{(Generalized) Bayesian inference via Donsker-Varadhan.}
Theorem~\ref{additive_DV_general} provides an information-theoretic characterization of the Bayesian posterior. By setting $h=-\ell$ and $\nu=\pi$ (the prior), we obtain that the normalizing constant $Z_{\textnormal{add}} = \int \exp(-\ell)\,\mathrm{d}\pi$ (also called the \emph{evidence} in the statistical setting) satisfies
\begin{align}
\log Z_{\textnormal{add}} = \sup_{q\in\mathcal{P}(\Theta)} \mathrm{ELBO}(q),
\qquad
\mathrm{ELBO}(q) \doteq - \mathbb{E}_{q}[\ell(\bm{\theta})] + \mathrm{KL}(q \Vert \pi) .
\end{align}
The ELBO (\emph{Evidence Lower Bound}) provides a lower bound on the log-evidence.
Furthermore, the exact posterior can then be expressed as the solution of the infinite-dimensional optimization problem
\begin{align}
\label{eq:bayes_variational}
q^\star_{\textnormal{add}} = \argmin_{q\in\mathcal{P}(\Theta)} \Big\{ \mathbb{E}_{\theta\sim q}[\ell(\theta)] + \mathrm{KL}(q\Vert \pi) \Big\} .
\end{align}
Thus, Donsker and Varadhan's lemma simultaneously identifies the Bayesian posterior as the optimizer of a regularized expected loss (the negative ELBO) and expresses the log-evidence as the supremum of the ELBO over all probability measures.
\paragraph{(Generalized) Variational inference via Donsker-Varadhan.}
In practice, computing $q^\star_{\textnormal{add}}$ exactly is generally intractable. \emph{Generalized Variational Inference} (GVI) consists in restricting the optimization to a tractable family $\mathcal{Q}\subset\mathcal{P}(\Theta)$, turning the infinite-dimensional problem \eqref{eq:bayes_variational} into a simpler, typically parametric, optimization problem.
This restriction naturally induces a gap between the true log-evidence and the ELBO achievable within $\mathcal{Q}$, which can be quantified via
\begin{equation}
\log Z_{\textnormal{add}} = \mathrm{ELBO}(q) + \mathrm{KL}(q \,\Vert\, q^\star_{\textnormal{add}}),
\qquad q\in\mathcal{Q} .
\end{equation}
Maximizing the ELBO within $\mathcal{Q}$ therefore corresponds exactly to minimizing the KL divergence to the exact posterior restricted to the chosen family. In this sense, GVI can be interpreted both as the best tractable approximation (in the reverse KL sense) of the exact posterior within $\mathcal{Q}$, and as the optimal information-processing rule within $\mathcal{Q}$.
\section{Possibilistic Inference and a Maxitive Donsker--Varadhan Analogue}
\label{sec:possibilistic-vi}
In this section, we establish a maxitive analogue of the Donsker–Varadhan variational formula and use it to derive a possibilistic inference framework. We first observe that many standard regularizers naturally attain their minimum at $0$, so that the prior $\pi = \exp(- R)$ is directly a possibility function whereas $\pi$ could be improper as a probability distribution if $\exp(-R)$ is not integrable. This leads to the following formula for the (generalized) posterior $g^{\star}_{\textnormal{max}}$ in possibility theory:
\begin{equation}
\label{def:gibbs:max}
g^{\star}_{\textnormal{max}}(\theta) \;\doteq\; \dfrac{\exp(-\ell(\theta))\pi(\theta)}{\sup_{\theta'} \exp(-\ell(\theta')) \pi(\theta')} = \dfrac{\exp(-\loss(\theta))}{\sup_{\theta'} \exp(-\loss(\theta'))} \, .
\end{equation}
The function $\pi(\theta)$ now denotes a prior possibility function over $\Theta$, typically $\pi(\theta)=\exp(-R(\theta))$.
The normalizing constant in the maxitive posterior is defined as
\begin{equation}
Z_{\textnormal{max}} \doteq \sup_{\theta'\in\Theta} \exp(-\ell(\theta')) \pi(\theta') \, ,
\end{equation}
and is referred to as the the maxitive marginal likelihood. % or possibilistic evidence.
Contrary to its probabilistic analogue $Z_{\textnormal{add}}$, which measures the overall fit of the model to the data, the quantity $Z_{\textnormal{max}}$ quantifies the \emph{consistency} between the prior and the likelihood. This notion is central in robust inference and can be used, for instance, to detect outliers \citep{houssineau2022robust}.
\begin{theorem}[Maxitive Donsker-Varadhan formula]
\label{thm:maxitive_DV}
Let $\pi:\Theta\to[0,1]$ be a possibility function. For any function $\ell:\Theta\to\mathbb{R}_+$, we have
\begin{subequations}
\begin{align}
\label{eq:dv:max:supinf}
\log \sup_{\theta \in \Theta} e^{-\ell(\theta)} \pi(\theta)
&= \sup_{g\in\mathcal{F}(\Theta)} \inf_{\theta \in \Theta}
\left\{ -\ell(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\},\\
\label{eq:dv:max:infsup}
&= \inf_{g\in\mathcal{F}(\Theta)} \sup_{\theta \in \Theta}
\left\{ -\ell(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\},
\end{align}
\end{subequations}
where $\mathcal{F}(\Theta)$ denotes the set of all possibility functions on $\Theta$, with the convention $\infty\times 0=0$.
Moreover, the supremum in \eqref{eq:dv:max:supinf} is achieved by any possibility function lower-bounding the posterior $g^{\star}_{\textnormal{max}}$, that is
\begin{equation}
\label{eq:gibbs:max:supinf}
\argmax_{g\in\mathcal{F}(\Theta)}
\inf_{\theta\in\Theta}
\left\{ -\ell(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\}
= \big\{ g \in \mathcal{F}(\Theta) : g \preceq g^{\star}_{\textnormal{max}} \big\}.
\end{equation}
Similarly, the infimum in \eqref{eq:dv:max:infsup} is achieved by any possibility function upper-bounding $g^{\star}_{\textnormal{max}}$:
\begin{equation}
\label{eq:gibbs:max:infsup}
\argmin_{g\in\mathcal{F}(\Theta)}
\sup_{\theta\in\Theta}
\left\{ -\ell(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\}
= \big\{ g \in \mathcal{F}(\Theta) : g^{\star}_{\textnormal{max}} \preceq g \big\}.
\end{equation}
\end{theorem}
The proof is provided in Appendix~\ref{proof:thm:maxitive_DV}.
\medskip
On the left-hand side of \eqref{eq:dv:max:supinf} appears the log-consistency $\log Z_{\textnormal{max}}$, while the quantity optimized on the right-hand side defines a \emph{consistency bound} (CBO).
Two dual versions can be introduced:
\begin{align}
\underline{\mathrm{CBO}}(g)
&\doteq \inf_\theta \left\{ -\ell(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \right\}, \qquad %\\
\overline{\mathrm{CBO}}(g)
%&
\doteq \sup_\theta \left\{ -\ell(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \right\}.
\end{align}
These provide lower and upper bounds on the log-consistency, as shown in Theorem~\ref{thm:maxitive_DV}.
For any $g\in\mathcal{F}(\Theta)$, we even have the decomposition
\begin{align}
\log Z_{\textnormal{max}}
&= \underline{\mathrm{CBO}}(g)
+ D_{\textnormal{max}}(g \,\|\, g^{\star}_{\textnormal{max}}) %\\
%&
= \overline{\mathrm{CBO}}(g)
- D_{\textnormal{max}}(g^{\star}_{\textnormal{max}} \,\|\, g),
\end{align}
where the \emph{max-relative entropy} between two possibility functions is defined as
\begin{align}
D_{\textnormal{max}}(g \,\|\, f)
\doteq \sup_{\theta \in \Theta} \log \frac{g(\theta)}{f(\theta)} \geq 0.
\end{align}
Maximizing (resp.\ minimizing) the lower (resp.\ upper) CBO with respect to $g$ is thus equivalent to minimizing the max-relative entropy to the Gibbs possibility function $g^{\star}_{\textnormal{max}}$.
Any maximizer $g$ in \eqref{eq:dv:max:supinf} satisfies $g \preceq g^{\star}_{\textnormal{max}}$, whereas any minimizer $g$ in \eqref{eq:dv:max:infsup} satisfies $g \succeq g^{\star}_{\textnormal{max}}$.
In particular, the standard posterior $g^{\star}_{\textnormal{max}}$ is always an optimizer, and pointwise bounds all other solutions (below or above, depending on the formulation).
\paragraph{Canonical choice of maximizer:} Among all possibility functions $g$ that optimize the CBO, a natural canonical representative is the standard possibilistic posterior $g^{\star}_{\textnormal{max}}$. However, one must be careful with terminology: $g^{\star}_{\textnormal{max}}$ is an extremum of the CBO, which means it assigns the largest/smallest possibility degrees and is therefore the most/least \emph{permissive} (i.e. informative) representative of the equivalence class of solutions. Choosing $g^{\star}_{\textnormal{max}}$ is natural because it is the sup/inf-envelope of all maximizers/minimizers, stable under maxitive combination, and
\begin{enumerate*}[label=\roman*)]
\item \emph{for the lower CBO}, does not introduce any extra, arbitrary information,
\item \emph{for the upper CBO}, does not forgo any of the available information in the likelihood and/or in the prior.
\end{enumerate*}
\paragraph{Recovering the posterior:} Possibilistic Bayesian inference can thus be understood as the optimisation of a possibilistic analogue of the ELBO: the CBO. Unlike the probabilistic case, this optimisation generally admits multiple solutions, but the standard maxitive posterior $g^{\star}_{\textnormal{max}}$ stands out as the natural choice -- it is an extremum among the optimisers and serves as the canonical representative of the updated possibility distribution.
Recall that the standard Bayesian posterior is $\left\{ q^{\star}_{\textnormal{add}} \right\} = \argmax_{q\in\mathcal{P}(\Theta)} \mathrm{ELBO}(q)$, while the two characterizations of the Bayesian posterior in possibility theory are:
\begin{subequations}
\label{eq:maxELBO}
\begin{align}
\label{eq:maxLowerELBO}
g^{\star}_{\textnormal{max}} & = \max\Big( \argmax_{g\in\mathcal{F}(\Theta)} \underline{\mathrm{CBO}}(g) \Big), \\
\label{eq:minUpperELBO}
& = \min\Big( \argmin_{g\in\mathcal{F}(\Theta)} \overline{\mathrm{CBO}}(g) \Big).
\end{align}
\end{subequations}
The additional $\max$/$\min$ operations in \eqref{eq:maxLowerELBO} and \eqref{eq:minUpperELBO} add a layer of complexity to the recovery of the posterior possibility function. However, it holds that the posterior possibility function $g^{\star}_{\textnormal{max}}$ is the only element in
\begin{align}
\argmax_{g\in\mathcal{F}(\Theta)} \underline{\mathrm{CBO}}(g) \cap \argmin_{g\in\mathcal{F}(\Theta)} \overline{\mathrm{CBO}}(g).
\end{align}
To exploit this, we can set a scalar $\alpha \in (0,1)$, and express both optimisation problems over $\mathcal{F}(\Theta)$ as, e.g.\ an $\argmin$. It is then easy to see that
\begin{equation}
\label{eq:EvidenceSandwich}
\left\{ g^{\star}_{\textnormal{max}} \right\} = \argmin_{g\in\mathcal{F}(\Theta)} \alpha \overline{\mathrm{CBO}}(g) - (1-\alpha) \underline{\mathrm{CBO}}(g),
\end{equation}
where the first term penalises underestimation of the posterior and the second term penalises overestimation. $\alpha$ balances the degree of caution in the approximation -- $\alpha \approx 1$ biases the optimization towards highly plausible but less informative estimates, and vice-versa when $\alpha \approx 0$. This achieves a similar effect as symmetrising the KL divergence.
\paragraph{Relationship with VI:} The first two formulations, \eqref{eq:maxLowerELBO} and \eqref{eq:minUpperELBO}, to recover the posterior possibility function from an optimisation problem \emph{on the set of possibility functions}, provide two different schemes for defining approximations of the posterior when we restrict the optimization to be on a subset $\mathcal{G}$ of $\mathcal{F}(\Theta)$. While \eqref{eq:maxLowerELBO} is qualitatively related to standard VI and will typically underestimate the possibility when restricted to $\mathcal{G}$, \eqref{eq:minUpperELBO} provides an alternative that will instead overestimate possibilities. This latter behaviour is more in line with general statistical principles which tend to prefer pessimistic uncertainty estimates over optimistic ones. The third formulation, \eqref{eq:EvidenceSandwich}, strikes a balance between the two above extremes, with over/under-estimation being preferred as needed, depending on the value of $\alpha$.
\paragraph{Relationship with Generalized VI:} In classical VI, the optimization is typically framed through the decomposition of the ELBO, which involves the reverse KL divergence, $\mathrm{KL}\!\left(q\Vert q^{\star}_{\textnormal{add}}\right)$. This yields a lower bound on the model evidence and has well-known properties such as the mode-seeking effect. Yet, in the probabilistic literature, many alternative divergences and bound constructions have been explored, leading to a variety of upper and lower bounds on the marginal likelihood. Notable examples include the $\chi$-divergence upper bounds (CUBO) \citep{dieng2017variational} and variational Rényi bounds \citep{li2016renyi}. In each of these cases, one can define dual optimization problems -- minimisation to the left or to the right of the divergence -- corresponding to lower or upper bounds providing a sandwiching of the true model evidence.
Remarkably, in the possibilistic framework, an analogous structure arises naturally, but with the max-relative entropy replacing the KL divergence. The lower and upper consistency bounds (CBOs) defined above correspond to two dual optimisation perspectives: maximizing the lower CBO or minimizing the upper CBO, which yield, respectively, under- and over-estimation of the posterior possibility degrees. This is conceptually analogous to probabilistic VI, where reverse or forward KL leads to mode-seeking versus mass-covering behaviours. In fact, one can interpret the upper CBO as a limiting case of a Rényi- or $\chi^n$-type bound when the divergence order tends to infinity, producing an extreme mass-covering (resp.\ “hyper mode-seeking”) effect.
Finally, these dual bounds can be combined to define successive, balanced objectives, similar to approaches in probabilistic VI that mix ELBO and CUBO to control approximation error \citep{huggins2020validated}. In the possibilistic setting, expression \eqref{eq:EvidenceSandwich} provides a principled way to interpolate between the two extremes, penalising over- and under-estimation of possibility degrees in a single step. This construction thus generalises the notion of variational bounds to possibility theory, mirroring the rich family of bounds in probabilistic inference and offering a natural trade off between conservative and optimistic uncertainty estimates.
\section{Possibilistic VI with Exponential Families}
\label{sec:ExpFam}
We develop a general framework for possibilistic variational inference based on exponential families by considering their conjugate priors. We begin by defining the structure of exponential families in the possibilistic setting. Assuming that $\Theta$ is a subset of $\mathbb{R}^{d_{\theta}}$ and $\Lambda$ is a convex subset of $\mathbb{R}^{d_{\lambda}}$, we follow the probabilistic approach and define an exponential family as a set $\mathcal{G} = \{g_{\lambda} \in \mathcal{F}(\Theta): \lambda \in \Lambda\}$ where $g_{\lambda}$ is of the form
\begin{align}
g_{\lambda}(\theta) = \exp\big( \lambda^{\intercal} T(\theta) - A(\lambda) - B(\theta) \big), \qquad \forall \theta \in \Theta,
\end{align}
for a given base measure $B$ and sufficient statistics $T$ on $\Theta$ and the corresponding log-partition function $A$ on $\Lambda$, which ensures proper normalisation, that is
\begin{align}
A(\lambda) & = \log \sup_{\theta \in \Theta} \exp\big( \lambda^{\intercal} T(\theta) - B(\theta) \big) = \sup_{\theta \in \Theta} \lambda^{\intercal} T(\theta) - B(\theta).
\end{align}
We start by clarifying some properties of possibilistic exponential families:
\begin{enumerate}[label=\textbf{Fact \arabic*:}]
\item There is no general correspondence between the components $T_+$ and $B_+$ of a probabilistic (canonical) exponential family and their analogue in possibility theory.
\item Although possibility functions are not densities and, therefore, do not require the definition of a reference measure, the base measure of a possibilistic exponentially family need not be an indicator function for the support of the family.
\item The log-partition function $A$ is simpler when compared to the expression of the log-partition $A_+$ for a probabilistic exponential family, that is
\begin{align}
A_+(\lambda) = \log \int \exp\big( \lambda^{\intercal} T_+(\theta) - B_+(\theta) \big) \mathrm{d} \theta,
\end{align}
due to the fact that $\sup$ and $\log$ can be exchanged when defining $A$.
\end{enumerate}
These properties are illustrated in the examples below.
\begin{example}
Consider a Bernoulli-style possibility function: set $\Theta = \{0,1\}$ and define the possibility of $\bm{\theta} = 0$ as $\alpha_0$ and the possibility of $\bm{\theta} = 1$ as $\alpha_1$, with $\max\cb{\alpha_0,\alpha_1} = 1$ by construction. The corresponding possibility function is $f(\theta \given \alpha_0, \alpha_1)= \alpha_0^{1-\theta}\alpha_1^{\theta}$. This can be expressed in an exponential family form as
\begin{align}
g_{\lambda}(\theta) = \exp\big( \lambda^{\intercal}T(\theta)
\big),
\end{align}
with $T(\theta) = (1-\theta, \theta)$ and $\lambda = (\log \alpha_0, \log \alpha_1) \in \Lambda$, where $\Lambda = \{(\lambda_1,\lambda_2) \in \mathbb{R}^2 : \max\{\lambda_1,\lambda_2\} = 0 \}$.
These components differ significantly from the ones of a Bernoulli distribution. Although the parameters $\alpha_0$ and $\alpha_1$ are not directly related, the fact that their maximum is equal to $1$ allows to express this exponential family with a single variable: the odd ratio $r = \alpha_0/\alpha_1$. Indeed, it holds that
\begin{align}
\alpha_0 = (r - 1)^- + 1 \qquad \text{and} \qquad \alpha_1 = (r - 1)^+ + 1,
\end{align}
with $(\cdot)^-$ and $(\cdot)^+$ denoting the negative part and positive part, respectively. Despite its simplicity, the Bernoulli possibility function is useful to express, for instance, possibilities of detection in tracking problems, see e.g.\ \cite{tracking2020ristic}.
\end{example}
\begin{example}
Consider a Poisson-style possibility function: set $\Theta = \mathbb{N}_0$ and consider a parameter $\alpha \in \mathbb{R}^+$. The corresponding possibility function is
\begin{align}
f(\theta \given \alpha) = \frac{\alpha^{\theta - \lfloor\alpha\rfloor}\lfloor{\alpha}\rfloor!}{\theta!},
\end{align}
which can be expressed in an exponential family form as
\begin{align}
g_{\lambda}(\theta) = \exp\big( \lambda^{\intercal} \theta - A(\lambda) - B(\theta) \big).
\end{align}
with $\lambda = \log \alpha$, $B(\theta) = \log(\theta!)$, and $A(\lambda) = \lfloor e^{\lambda}\rfloor\lambda - \log\rb{\lfloor e^{\lambda}\rfloor!}$.
\end{example}
\begin{example}
Consider the univariate normal possibility function with known variance $\sigma^2$, $\overline{\mathrm{N}}(\theta;\mu,\sigma^2)$, for which we have $B(\theta) = \theta^2/(2\sigma^2)$, whereas the base measure of the corresponding univariate normal distribution is
\begin{align}
\frac{\theta^2}{2\sigma^2} + \frac{1}{2}\log (2\pi\sigma^2).
\end{align}
\end{example}
\subsection{Bayesian Inference with Conjugate Family}
\label{sec:ExpFamBayes}
We consider a likelihood $p(\cdot \given \theta)$, $\theta \in \Theta$, that is part of a regular and minimal exponential family with log-partition $A$, that is
\begin{equation}
\label{eq:probExFamily}
p(x\given \theta) = \exp( \theta^{\intercal}T(x) - A(\theta) - B(x) ).
\end{equation}
The following proposition rephrases known results on the relationship between exponential families and the Bregman divergence \citep{banerjee2005clustering} in the context of possibilistic inference.
\begin{proposition}
\label{prop:posteriorPossibilty}
The posterior possibility function under the uninformative prior on $\Theta$ is of the form
\begin{subequations}
\begin{align}
\label{eq:possConjPrior}
g_{\lambda}(\theta) & = \exp\big( \lambda^{\intercal}\theta - A^{\dagger}(\lambda) - A(\theta) \big) \\
\label{eq:PosteriorAsBregman}
& = \exp\big (-D_A(\theta \| \mode{\theta}(\lambda)) \big),
\end{align}
\end{subequations}
with $\mode{\theta}(\lambda) \doteq \possexp_{g_{\lambda}}[\bm{\theta}]$ the maximum likelihood estimator (MLE), $A^{\dagger}$ the Legendre transform of the log-partition $A$, and $D_A(\theta \| \theta') = A(\theta) - A(\theta') - \nabla_{\theta} A(\theta')^{\intercal}(\theta -\theta')$ the Bregman divergence.
\end{proposition}
Identifying the Bregman divergence in $g_{\lambda}$ allows us to leverage its properties. For instance, it is obvious from \eqref{eq:PosteriorAsBregman} that $\pi(\cdot\given \lambda)$ is a possibility function with expected value $\mode{\theta}(\lambda)$. We can also use the known relationship between $D_A$ and the KL divergence to see that $-\log g_{\lambda}(\theta)$ is the KL divergence between the likelihood at $\theta$ and the one at the MLE. We now highlight some important properties of possibility functions of the same form as $g_{\lambda}$ in the following proposition.
\begin{proposition}
\label{prop:PropertiesOfConjugatePriors}
Let $\mathcal{G}_A(\Theta)$ be the subset of $\mathcal{F}(\Theta)$ defined as
\begin{align}
\mathcal{G}_A(\Theta) \doteq \big\{ \theta \mapsto \exp(\lambda^{\intercal}\theta - A^{\dagger}(\lambda) -A(\theta) ) : \lambda \in \nabla_{\theta}A(\Theta) \big\},
\end{align}
where $\Lambda \doteq \nabla_{\theta}A(\Theta)$ is the image of $\Theta$ by $\nabla_{\theta}A$, then, for any $g_{\lambda} \in \mathcal{G}_A(\Theta)$, it holds that
\begin{enumerate}[itemsep=.1em,parsep=.1em,topsep=.1em]
\item For any $\nu \geq 0$, $g_{\lambda}^{\nu}$ is a conjugate prior for $p(\cdot \given \theta)$, and $g_{\lambda}^{\nu} \in \mathcal{G}_{\nu A}(\Theta)$.
\item The dual possibility function $f_{\theta}:\lambda \mapsto g_{\lambda}(\theta)$ is in $\mathcal{G}_{A^{\dagger}}(\Lambda)$.
\end{enumerate}
\end{proposition}
The first point in Proposition~\ref{prop:PropertiesOfConjugatePriors} is a possibilistic analogue of the result of \cite{diaconis1979conjugate}, with the additional features that the log-partition function is simply the Legendre transform of $A$ and that the $\nu$ parameter is a discount factor rather than an arbitrary parameter. This property is intuitive: $\nu$ can be interpreted as how many observations the prior is worth in terms of information content; in particular, when $\nu = 0$, it holds that $g_{\lambda}^{\nu} = \bm{1}$ and the prior is uninformative. The second point is a specific property of the considered class of conjugate prior possibility functions. Although it has additional properties, the possibilistic conjugate prior is often simply the renormalised version of its probabilistic counterpart.
We have the following properties for a given $g_{\lambda} \in \mathcal{G}_A(\Theta)$ as a possibility function describing the uncertain variable $\bm{\theta}$ in $\Theta$:
\begin{enumerate}[leftmargin=*]
\item The expected value $\possexp_{g_{\lambda}}[\bm{\theta}] = \nabla_{\lambda} \leg{A}(\lambda)$ is assumed to be the singleton $\{\mode{\theta}(\lambda)\}$. This assumption is weak as exponential families often have a unique mode except when they become fully uninformative, i.e.\ when $g_{\lambda} = \bm{1}$.
\item The precision $\mathcal{I}_{\lambda} \doteq \mathcal{I}_{g_{\lambda}}(\bm{\theta})$ verifies
\begin{align}
\mathcal{I}_{\lambda} = \possexp_{g_{\lambda}}\big[-\nabla^2_{\theta} \log g_{\lambda}(\bm{\theta}) \big] = -\nabla^2_{\theta} \log g_{\lambda}(\mode{\theta}(\lambda)) = \nabla_{\theta}^2 A\rb{\mode{\theta}(\lambda)} = (\nabla^2_{\lambda} \leg{A}(\lambda))^{-1},
\end{align}
where $\nabla^2_{\theta} \log g_{\lambda}(\mode{\theta}(\lambda))$ stands for $\nabla^2_{\theta} \log g_{\lambda}(\theta)|_{\theta = \mode{\theta}(\lambda)}$.
\end{enumerate}
\subsection{Variational Inference with Conjugate Family}
We now show how to perform VI for possibilistic conjugate families, which is exact when the likelihood is in an exponential family distribution, as is commonly the case, e.g.\ Gaussian distribution for regression, and categorical distribution for classification. The secondary objective is to recover known optimisation techniques as pioneered by \cite{khan2023bayesian}. We focus on the lower CBO since it is the one that is the closest to standard VI.
\begin{proposition}
\label{prop:updateLowerCBO}
Given a variational family $\mathcal{G}_A(\Theta)$, let $g_{\lambda_t}$ be the possibility function at the $t$-th step of the maximisation of $\underline{\mathrm{CBO}}$. Then a valid update rule based on sub-gradients is
\begin{align}
\label{eqn:lower-cbo-updates}
\lambda_{t+1} = \lambda_t - \rho_t \big( \mode{\theta}(\lambda_t + \nabla_{\theta} \loss(\objmin{\theta}_t)) - \mode{\theta}(\lambda_t) \big),
\end{align}
where $\objmin{\theta}_t \in \argmax_{\theta\in\Theta} \loss(\theta) + \lambda_t^{\intercal} \theta - A(\theta)$.
An approximate explicit expression
\begin{align}
\label{eq:approximateUpdate}
\lambda_{t+1} \approx \lambda_t - \rho_t \mathcal{I}_{\lambda_t}^{-1} \nabla_{\theta} \loss(\mode{\theta}(\lambda_t)),
\end{align}
follows from the approximations $\objmin{\theta}_t \approx \mode{\theta}(\lambda_t) + \mathcal{I}_{\lambda_t}^{-1} \nabla_{\theta} \loss(\objmin{\theta}_t)$ and $\objmin{\theta}_t \approx \mode{\theta}(\lambda_t)$ in the argument of $\nabla_{\theta} \loss$.
\end{proposition}
\begin{corollary}
\label{cor:normal_fixed_var}
Let $A(\theta)=\frac{1}{2}\theta^{\intercal}\Sigma\theta$ be the log-partition function of the multivariate normal distribution with known variance $\Sigma$,
and let $\loss_{\mathrm{s}}(\mu)$ be the regularised loss parametrised by the standard parameter $\mu = \Sigma \theta$, then
\begin{align}
\label{eq:AlmostGradientDescent}
\lambda_{t+1} \approx \lambda_t - \rho_t \nabla_{\mu} \loss_{\mathrm{s}}( \lambda_t ),
\end{align}
which is a standard gradient descent update.
\end{corollary}
Although similar to \cite{khan2023bayesian}, our result leverages a different approximation: instead of using the delta-method \citep{dorfman1938note}, we simply assume that the gradients of the regularised loss are small.
\begin{example}
Suppose that $\exp(-\loss_{\mathrm{s}})$ is the normal distribution $\mathrm{N}(x; \mu, \Sigma+\Sigma_0)$ for some positive definite covariance $\Sigma_0$, then $\nabla_{\lambda} \loss_{\mathrm{s}}(\lambda_t) = (\Sigma+\Sigma_0)^{-1} (\lambda_t - x)$, so that the update rule becomes
\begin{align}
\lambda_{t+1} \approx \lambda_t - \rho_t (\Sigma+\Sigma_0)^{-1} (\lambda_t - x).
\end{align}
This expression can be interpreted as follows: $\lambda_t$ is the current estimate for the sufficient statistics $x$, so that $\lambda_t - x$ is an estimation error term, which is scaled by the precision $(\Sigma+\Sigma_0)^{-1}$; directions in which the precision is low should receive smaller updates, and this is naturally achieved here.
\end{example}
\begin{corollary}
\label{cor:binomial}
Let $A(\theta) = n \log(1+\exp(\theta))$ be the log-partition function of the binomial distribution with known number of trials $n$, then the approximate update rule \eqref{eq:approximateUpdate} becomes
\begin{align}
\label{eq:binomialUpdate}
\lambda_{t+1} \approx \lambda_t - \rho_t\frac{1}{\lambda_t(1-\lambda_t/n)} \nabla_{\theta} \loss\bigg( \log \frac{\lambda_t}{n - \lambda_t} \bigg).
\end{align}
Reparameterizing the loss in terms of the standard parameter $p = (1+\exp(-\theta))^{-1}$, the approximate update rule is given by
\begin{align}
\label{eq:binomialUpdateStandard}
\lambda_{t+1} \approx \lambda_t - \frac{\rho_t}{n} \nabla_{p} \loss_s( \lambda_t/n ).
\end{align}
\end{corollary}
\begin{example}
Suppose that $\exp(-\loss_{\mathrm{s}})$ is a binomial distribution with standard parameter $p \in (0,1)$ and $n$ trials, for which $x$ successes have been observed. It is useful in this case to use the parameter $\hat{p}_t = \lambda_t / n$ instead of $\lambda_t$, so we have $\hat{p}_{t+1} \approx \hat{p}_t - \rho_t/n^2 \nabla_{p} \loss_{\mathrm{s}}( \hat{p}_t )$. Computing $\nabla_{p} \loss_{\mathrm{s}}( \hat{p}_t )$ yields the update rule
\begin{align}
\hat{p}_{t+1} \approx \hat{p}_t - \rho_t\frac{\hat{p}_t - x/n}{\mathbb{V}(X \given \hat{p}_t)},
\end{align}
where $\mathbb{V}(X \given p) = np(1-p)$ is the variance of the observation when the probability of success is $p$. This expression makes sense: the current estimate $\hat{p}_t$ is compared to the probability of success $x/n$ induced by the observation $x$, and %$p_t(1-p_t)$ is proportional to the inverse of the Fisher information, which
$\mathbb{V}(X \given \hat{p}_t)$ scales the update accordingly. Indeed, if $\hat{p}_t$ is close to $0$ (or close to $1$) then the estimation error $\hat{p}_t - x/n$ should be smaller since $x$ has lower variance.
\end{example}
\section{Discussion}
Possibilistic Bayesian inference is already an optimisation problem, which seems to limit the interest in the development of possibilistic VI at first sight. On the contrary, we have shown that there is a surprising amount of subtleties to be understood and of insights to be gained when exploring the properties of possibilistic VI in general as well as specifically through the lens of a possibilistic version of exponential families and conjugate priors. And indeed, even when closely following existing treatments of VI and its applications, many differences emerge, such as regularities in conjugate families and a new route leading to gradient descent.
\bibliographystyle{bib_style}
\bibliography{bibliography}
\newpage
\appendix
\section{Notations}
\begingroup
\renewcommand{\arraystretch}{1.3}
\setlength{\tabcolsep}{7pt}
\begin{table}[h!]
\caption{Summary of key notations}
\centering
\begin{tabularx}{0.8\linewidth}{@{} >{$}l<{$} L @{}}
\toprule
\text{Notation} & \text{Description} \\
\midrule
\possexp_{f} & Possibilistic expectation (mode) of $f$ \\
\mathcal{I}_f & Possibilistic precision of $f$ \\
A^{\dagger} & Legendre transform of $A$ \\
\loss & Regularized loss: $-\log L(\theta) - \log \pi(\theta)$ \\
\overline{\mathrm{CBO}} & Upper consistency bound \\
\underline{\mathrm{CBO}} & Lower consistency bound \\
\mode{\theta}(\lambda) & Mode / expected value of $g_\lambda$, that is $\possexp_{g_{\lambda}}[\bm{\theta}]$\\
\objmin{\theta}(\lambda) & Set of minimizers of $\underline{\mathrm{CBO}}(g_{\lambda})$ \\
\bottomrule
\label{tab:notations}
\end{tabularx}
\end{table}
\endgroup
\section{Proofs}
In this section, we present the proofs for the main theoretical results in the main text.
\subsection{Theorem \ref{thm:maxitive_DV}}
\label{proof:thm:maxitive_DV}
We prove each equation of the theorem one after the other.
\begin{proof}[Proof of Equation \eqref{eq:dv:max:supinf}]
Consider some fixed possibility function $g$.
Let us first assume that there exists some $\mode{\theta}_g\in\argsup_{\theta\in\Theta} g(\theta)$. We then have $g(\mode{\theta}_g)=1$, so that:
\begin{subequations}
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) & \geq \log e^{h(\mode{\theta}_g)} \pi(\mode{\theta}_g) \\ % definition of supremum
& = \log \frac{e^{h(\mode{\theta}_g)} \pi(\mode{\theta}_g)}{g(\mode{\theta}_g)} \\ % since g(θ*) = 1
& \geq \inf_{\theta \in \Theta} \log \frac{e^{h(\theta)} \pi(\theta)}{g(\theta)} \\ % definition of infimum
&= \inf_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, .
\end{align}
\end{subequations}
Now, if $\argsup_{\theta\in\Theta} g(\theta)=\emptyset$, then one can still define a sequence $(\mode{\theta}_{g,n})_{n=1}^{\infty}\in\Theta^\mathbb{N}$ such that for any integer $n>0$, $g(\mode{\theta}_{g,n})\geq 1-1/n$. We can then write:
\begin{subequations}
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) & \geq \log e^{h(\mode{\theta}_{g,n})} \pi(\mode{\theta}_{g,n}) \\ % definition of supremum
& = \log \frac{e^{h(\mode{\theta}_{g,n})} \pi(\mode{\theta}_{g,n})}{1-\frac{1}{n}} + \log \left(1-\frac{1}{n}\right) \\
& \geq \log \frac{e^{h(\mode{\theta}_{g,n})} \pi(\mode{\theta}_{g,n})}{g(\mode{\theta}_{g,n})} + \log \left(1-\frac{1}{n}\right) \\ % assumption on the sequence
& \geq \inf_{\theta \in \Theta} \log \frac{e^{h(\theta)} \pi(\theta)}{g(\theta)} + \log \left(1-\frac{1}{n}\right) \\ % definition of infimum
& = \inf_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} + \log \left(1-\frac{1}{n}\right) \, ,
\end{align}
\end{subequations}
so by letting $n\to+\infty$, we have
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) \geq \inf_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, .
\end{align}
Consequently, the inequality above holds for any possibility function $g\in\mathcal{F}(\Theta)$, and by taking the supremum over $g\in\mathcal{F}(\Theta)$ in the right-hand side leads to:
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) \geq \sup_{g\in\mathcal{F}(\Theta)} \inf_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, .
\end{align}
Furthermore, the choice of possibility function $g(\theta)=g^{\star}_{\textnormal{max}}(\theta) = {e^{h(\theta)} \pi(\theta)}/{\sup_{\theta'} e^{h(\theta')} \pi(\theta')}$ transforms the inequality into an equality, which finally gives Formula \eqref{eq:dv:max:supinf}:
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) = \sup_{g\in\mathcal{F}(\Theta)} \inf_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, .
\end{align}
\end{proof}
\begin{proof}[Proof of Equation \eqref{eq:dv:max:infsup}]
Any possibility function $g$ takes its values in $[0,1]$, so that for any possibility function $g$ and any parameter $\theta\in\Theta$, we have $\log g(\theta)\leq 0$. Thus, for any possibility function $g$ and any parameter $\theta$,
\begin{align}
\log e^{h(\theta)} \pi(\theta) \leq h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \, ,
\end{align}
so taking the supremum over $\theta$ in both sides leads for any $g$ to:
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) \leq \sup_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, ,
\end{align}
and taking the infimum over possibility functions $g$ in the right-hand side leads to
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) \leq \inf_{g\in\mathcal{F}(\Theta)} \sup_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, ,
\end{align}
Once again, $g=g^{\star}_{\textnormal{max}}$ transforms the inequality into an equality, which finally gives Formula \eqref{eq:dv:max:infsup}
\begin{align}
\log \sup_{\theta \in \Theta} e^{h(\theta)} \pi(\theta) = \inf_{g\in\mathcal{F}(\Theta)} \sup_{\theta \in \Theta} \Big\{ h(\theta) - \log \frac{g(\theta)}{\pi(\theta)} \Big\} \, .
\end{align}
\end{proof}
\begin{proof}[Proof of Equation \eqref{eq:gibbs:max:supinf}]
We have already mentioned in the proof of Equation \eqref{eq:dv:max:supinf} that
\begin{align}
g^{\star}_{\textnormal{max}}(\theta) \doteq \dfrac{e^{h(\theta)} \pi(\theta)}{\sup\limits_{\theta'\in\Theta} e^{h(\theta')} \pi(\theta')} \in \argmax_{g\in\mathcal{F}(\Theta)} \inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, ,
\end{align}
so that
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g^{\star}_{\textnormal{max}}(\theta)}{\pi(\theta)}\right) \right\} = \sup_{g\in\mathcal{F}(\Theta)}\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Furthermore, any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g'\preceq g^{\star}_{\textnormal{max}}$ satisfies by monotonicity of the logarithmic function:
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \geq \inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g^{\star}_{\textnormal{max}}(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Hence, combining the two lines above, we get for any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g'\preceq g^{\star}_{\textnormal{max}}$:
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \geq \sup_{g\in\mathcal{F}(\Theta)}\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Since by definition of the supremum, we have the reverse inequality:
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \leq \sup_{g\in\mathcal{F}(\Theta)}\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, ,
\end{align}
we finally have for any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g'\preceq g^{\star}_{\textnormal{max}}$:
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} = \sup_{g\in\mathcal{F}(\Theta)}\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
This provides half of the proof, namely
\begin{align}
\left\{ g\in\mathcal{F}(\Theta) \, : \, g\preceq g^{\star}_{\textnormal{max}} \right\} \subset \argmax_{g\in\mathcal{F}(\Theta)} \inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
To get the equality, we still have to show that any possibility function $g$ not satisfying $g\preceq g^{\star}_{\textnormal{max}}$ does not belong to the argmax.
To show this, let us consider some possibility function $g'\notin \left\{ g\in\mathcal{F}(\Theta) \, : \, g\preceq g^{\star}_{\textnormal{max}} \right\}$, and show that
\begin{align}
\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} < \sup_{g\in\mathcal{F}(\Theta)}\inf_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
To see this, recall that since $g' \not\preceq g^{\star}_{\textnormal{max}}$, there exists at least one $\theta' \in \Theta$ such that $g'(\theta') > g^{\star}_{\textnormal{max}}(\theta')$. Therefore,
\begin{align}
h(\theta') - \log\!\left(\frac{g'(\theta')}{\pi(\theta')}\right) < h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right),
\end{align}
Using the definition of the infimum, we have
\begin{align}
\inf_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} \le h(\theta') - \log\!\left(\frac{g'(\theta')}{\pi(\theta')}\right) < h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right) .
\end{align}
However, notice that by definition of $g^{\star}_{\textnormal{max}}$, the quantity in the right-hand side above does not depend on $\theta'$ since:
\begin{align}
h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right) = h(\theta') - \log\!\left(\frac{\dfrac{e^{h(\theta')} \pi(\theta')}{\sup e^{h(\cdot)} \pi(\cdot)}}{\pi(\theta')}\right) = \log \sup_{\theta\in\Theta} e^{h(\theta)} \pi(\theta) ,
\end{align}
so combining the two lines above, we get
\begin{align}
\inf_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} < \log \sup_{\theta\in\Theta} e^{h(\theta)} \pi(\theta) ,
\end{align}
By using Formula \eqref{eq:dv:max:supinf}, we can rewrite the quantity in the right-hand side:
\begin{align}
\inf_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} < \sup_{g\in\mathcal{F}(\Theta)} \inf_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right)\right\},
\end{align}
which is exactly what we wanted to show. Hence, $g'$ cannot belong to the set of maximisers. This proves that
\begin{align}
\argmax_{g\in\mathcal{F}(\Theta)} \inf_{\theta\in\Theta} \left\{h(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right)\right\} = \left\{ g\in\mathcal{F}(\Theta) : g\preceq g^{\star}_{\textnormal{max}} \right\},
\end{align}
which concludes the proof.
\end{proof}
\begin{proof}[Proof of Equation \eqref{eq:gibbs:max:infsup}]
The proof of Equation \eqref{eq:gibbs:max:infsup} is very similar to the proof of Equation \eqref{eq:gibbs:max:supinf}, and is only provided for the sake of completeness.
Once again, we start from the following fact mentioned in the proof \eqref{eq:dv:max:infsup}:
\begin{align}
g^{\star}_{\textnormal{max}}(\theta) \doteq \dfrac{e^{h(\theta)} \pi(\theta)}{\sup\limits_{\theta'\in\Theta} e^{h(\theta')} \pi(\theta')} \in \argmin_{g\in\mathcal{F}(\Theta)} \sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, ,
\end{align}
so that
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g^{\star}_{\textnormal{max}}(\theta)}{\pi(\theta)}\right) \right\} = \inf_{g\in\mathcal{F}(\Theta)}\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Furthermore, any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g^{\star}_{\textnormal{max}} \preceq g'$ satisfies by monotonicity of the logarithmic function:
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \leq \sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g^{\star}_{\textnormal{max}}(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Hence, combining the two lines above, we get for any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g^{\star}_{\textnormal{max}} \preceq g'$:
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \leq \inf_{g\in\mathcal{F}(\Theta)}\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
Since by definition of the infimum, we have the reverse inequality:
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} \geq \inf_{g\in\mathcal{F}(\Theta)}\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, ,
\end{align}
we finally have for any possibility function $g'\in\mathcal{F}(\Theta)$ such that $g^{\star}_{\textnormal{max}}\preceq g'$:
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} = \inf_{g\in\mathcal{F}(\Theta)}\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
This provides half of the proof, namely
\begin{align}
\left\{ g\in\mathcal{F}(\Theta) \, : \, g^{\star}_{\textnormal{max}} \preceq g \right\} \subset \argmin_{g\in\mathcal{F}(\Theta)} \sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
To get the equality, we now have to show that any possibility function $g$ not satisfying $g^{\star}_{\textnormal{max}}\preceq g$ does not belong to the argmin.
To show this, let us consider some possibility function $g'\notin \left\{ g\in\mathcal{F}(\Theta) \, : \, g^{\star}_{\textnormal{max}} \preceq g \right\}$, and show that
\begin{align}
\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g'(\theta)}{\pi(\theta)}\right) \right\} > \inf_{g\in\mathcal{F}(\Theta)}\sup_{\theta \in \Theta} \left\{ h(\theta) - \log\left(\frac{g(\theta)}{\pi(\theta)}\right) \right\} \, .
\end{align}
To see this, recall that since $g^{\star}_{\textnormal{max}} \not\preceq g'$, there exists at least one $\theta' \in \Theta$ such that $g'(\theta') < g^{\star}_{\textnormal{max}}(\theta')$. Therefore,
\begin{align}
h(\theta') - \log\!\left(\frac{g'(\theta')}{\pi(\theta')}\right) > h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right),
\end{align}
Using the definition of the supremum, we now have
\begin{align}
\sup_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} \ge h(\theta') - \log\!\left(\frac{g'(\theta')}{\pi(\theta')}\right) > h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right) .
\end{align}
Again, the quantity in the right-hand side above does not depend on $\theta'$:
\begin{align}
h(\theta') - \log\!\left(\frac{g^{\star}_{\textnormal{max}}(\theta')}{\pi(\theta')}\right) = h(\theta') - \log\!\left(\frac{\dfrac{e^{h(\theta')} \pi(\theta')}{\sup e^{h(\cdot)} \pi(\cdot)}}{\pi(\theta')}\right) = \log \sup_{\theta\in\Theta} e^{h(\theta)} \pi(\theta) ,
\end{align}
so combining the two lines above leads to
\begin{align}
\sup_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} > \log \sup_{\theta\in\Theta} e^{h(\theta)} \pi(\theta) ,
\end{align}
Now using Formula \eqref{eq:dv:max:infsup}, we can rewrite the quantity in the right-hand side:
\begin{align}
\sup_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g'(\theta)}{\pi(\theta)}\right)\right\} > \inf_{g\in\mathcal{F}(\Theta)} \sup_{\theta\in\Theta}\left\{h(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right)\right\},
\end{align}
which is precisely what we wanted to show. Hence, $g'$ cannot belong to the set of minimisers. This proves that
\begin{align}
\argmin_{g\in\mathcal{F}(\Theta)} \sup_{\theta\in\Theta} \left\{h(\theta) - \log\!\left(\frac{g(\theta)}{\pi(\theta)}\right)\right\} = \left\{ g\in\mathcal{F}(\Theta) : g^{\star}_{\textnormal{max}} \preceq g \right\},
\end{align}
which concludes the proof.
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Proofs of Results in Section~\ref{sec:ExpFam}}
\begin{proof}[Proof of Proposition \ref{prop:posteriorPossibilty}]
Given the probabilistic likelihood $p(x\given \theta) = \exp( \theta^{\intercal}T(x) - A(\theta) - B(x) )$, under the uninformative prior on $\Theta$, $\pi\rb{\theta} = 1$, the posterior possibility is given by
\begin{subequations}
\begin{align}
\pi\rb{\theta \given x} &= \frac{p(x\given \theta) \pi\rb{\theta}}{\max_{\theta'} p(x\given \theta') \pi\rb{\theta'}} = \frac{p(x\given \theta)}{\max_{\theta'} p(x\given \theta')} \\
&= \exp\rb{\rb{\theta-\mode{\theta}}^\intercal T(x) - \rb{A\rb{\theta}-A\rb{\mode{\theta}}}},
\end{align}
\end{subequations}
where we denote the MLE by $\mode{\theta} \doteq \argmax_{\theta} p\rb{x\given \theta} = \argmax_{\theta} \theta^{\intercal}T(x) - A(\theta)$. The first-order condition for the maximum tells us that $T(x) = \nabla_{\theta} A\rb{\theta^{\thinstar}}$. Plugging this in the equation above, we get the result in \eqref{eq:PosteriorAsBregman}:
\begin{subequations}
\begin{align}
\pi\rb{\theta \given x} &= \exp\rb{\rb{\theta-\mode{\theta}}^\intercal \nabla_{\theta} A\rb{\theta^{\thinstar}} - \rb{A\rb{\theta}-A\rb{\mode{\theta}}}} \\
&= \exp\rb{-D_A\rb{\theta \| \mode{\theta}}}.
\end{align}
\end{subequations}
Next, we rewrite the posterior as
\begin{subequations}
\begin{align}
\pi\rb{\theta \given x} &= \exp\rb{\theta^\intercal T(x) - A\rb{\theta} -\rb{\theta^{\thinstar\intercal} T(x) - A\rb{\mode{\theta}}}} \\
&= \exp\rb{\theta^\intercal \lambda - A\rb{\theta} - A'\rb{\lambda}},
\end{align}
\end{subequations}
which is indeed of the same form as $g_{\lambda}$, with $\lambda \doteq T(x)$. From the definition of $\mode{\theta}$, we get the result in Equation \eqref{eq:possConjPrior}, that $A' = \leg{A}$.
\end{proof}
\begin{proof}[Proof of Proposition \ref{prop:PropertiesOfConjugatePriors}]
Consider the likelihood $p(x\given \theta) = \exp( \theta^{\intercal}T(x) - A(\theta) - B(x) )$, and a choice of prior $g_{\lambda, \nu} \doteq g_{\lambda}^{\nu}$, with $g_{\lambda} \in \mathcal{G}_A(\Theta)$ defined as $g_{\lambda}\rb{\theta} = \exp\rb{\lambda^{\intercal}\theta - \leg{A}\rb{\lambda} - A\rb{\theta}}$, for some $\lambda \in \Lambda$. Denoting the maximum \textit{a posteriori} (MAP) estimate by $\mode{\theta} \doteq \argmax_{\theta} p\rb{x\given\theta} g_{\lambda,\nu}\rb{\theta}$, the posterior possibility is given by
\begin{align*}
g_{\lambda,\nu}\rb{\theta\given x} &= \frac{p(x\given \theta) g_{\lambda}^{\nu}\rb{\theta}}{\max_{\theta'} p(x\given \theta') g_{\lambda}^{\nu}\rb{\theta'}} \\
&= \frac{\exp\rb{\theta^\intercal\rb{T(x)+\nu\lambda} - \rb{\nu+1} A\rb{\theta}}}{\max_{\theta'} \exp\rb{\theta'^\intercal\rb{T(x)+\nu\lambda} - \rb{\nu+1} A\rb{\theta'}}} \\
&= \exp\rb{\theta^\intercal\rb{T(x)+\nu\lambda} - \rb{\nu+1} A\rb{\theta} - \rb{\nu+1}\leg{A}\rb{\frac{T\rb{x}+\nu\lambda}{\nu+1}}} \\
&= g_{\frac{T(x)+\nu\lambda}{\nu+1},\nu+1}\rb{\theta},
\end{align*}
As a sanity check, note that with $\nu=0$, $g$ is the uninformative prior, and we recover the posterior in Proposition~\ref{prop:posteriorPossibilty}.\footnote{We note here that when $\nu$ is an integer, it can be understood as encoding the number of \textit{pseudo-observations}, as also in probabilistic conjugate priors.} Therefore, $g_{\lambda,\nu}$ is a valid conjugate prior for the likelihood $p(\cdot \given \theta)$.
Next, we show that $g_{\lambda}^{\nu} \in \mathcal{G}_{\nu A}(\Theta)$. It holds that
\begin{subequations}
\begin{align}
g_{\lambda}^{\nu}\rb{\theta} &= \exp\rb{\nu\lambda^\intercal\theta - \nu \leg{A}\rb{\lambda} - \nu A\rb{\theta}} \\
&= \exp\rb{\nu\lambda^\intercal\theta - \leg{\rb{\nu A}}\rb{\nu\lambda} - \nu A\rb{\theta}},
\end{align}
\end{subequations}
where the second equality follows from the following property of the convex conjugate:
\begin{align}
\nu\leg{A}\rb{\lambda} &= \nu\sup_{\theta} \lambda^\intercal\theta - A\rb{\theta} = \sup_{\theta} \nu\lambda^\intercal\theta - \nu A\rb{\theta} = \leg{\rb{\nu A}}\rb{\nu\lambda}.
\end{align}
We conclude that $g_{\lambda}^{\nu} \in \mathcal{G}_{\nu A}(\Theta)$ as required. Finally, we have
\begin{subequations}
\begin{align}
g_{\lambda}\rb{\theta} &= \exp\rb{\lambda^\intercal\theta - \leg{A}\rb{\lambda} - A\rb{\theta}} \\
&= \exp\rb{\theta^\intercal \lambda - A^{\dagger\dagger}\rb{\theta} - A^{\dagger}\rb{\lambda}},
\end{align}
\end{subequations}
so the mapping $f_{\theta}: \lambda \to g_{\lambda}\rb{\theta}$, which can be written as
\begin{align}
f_{\theta}: \lambda \to \exp\rb{\theta^\intercal \lambda - A^{\dagger\dagger}\rb{\theta} - A^{\dagger}\rb{\lambda}}
\end{align}
is in $\mathcal{G}_{\leg{A}}\rb{\Lambda}$.
\end{proof}
\begin{proof}[Proof of Proposition~\ref{prop:updateLowerCBO}]
Let $\objmin{\theta}(\lambda) \doteq \argmax_{\theta\in\Theta} \loss(\theta) + \lambda^{\intercal} \theta - A(\theta)$ denote the solution set of the optimisation problem within $\underline{\mathrm{CBO}}$. A sub-gradient of $\underline{\mathrm{CBO}}$ is
\begin{align}
\nabla_{\lambda} \underline{\mathrm{CBO}}(g_{\lambda_t}) & = -\nabla_{\lambda} \log g_{\lambda_t}(\theta)|_{\theta = \objmin{\theta}_t} = \nabla_{\lambda}\leg{A}(\lambda_t) - \objmin{\theta}_t,
\end{align}
where $\objmin{\theta}_t$ is an optimiser in $\objmin{\theta}(\lambda_t)$, which satisfies $\nabla_{\theta} \loss(\objmin{\theta}_t) + \lambda_t - \nabla_{\theta}A(\objmin{\theta}_t) = 0$, so that
\begin{align*}
\objmin{\theta}_t = \nabla_{\lambda}\leg{A} (\lambda_t + \nabla_{\theta} \loss(\objmin{\theta}_t)) = \mode{\theta}(\lambda_t + \nabla_{\theta}\loss(\objmin{\theta}_t)).
\end{align*}
Therefore, recalling that $\underline{\mathrm{CBO}}(g_{\lambda_t})$ is maximised, a valid update rule for the parameter $\lambda$ is
\begin{align}
\lambda_{t+1} &= \lambda_t + \rho_t \nabla_{\lambda} \underline{\mathrm{CBO}}(g_{\lambda_t}) = \lambda_t - \rho_t \big[ \mode{\theta}\big(\lambda_t + \nabla_{\theta}\loss(\objmin{\theta}_t)\big) - \mode{\theta}(\lambda_t) \big].
\end{align}
We consider the first-order Taylor approximation of $\theta^{\thinstar}$:
\begin{align}
% \objmin{\theta}_t =
\mode{\theta}\big(\lambda_t + \nabla_{\theta}\loss(\objmin{\theta}_t)\big) &\approx \mode{\theta}(\lambda_t) + \nabla^2_{\lambda}\leg{A}(\lambda_t) \nabla_{\theta} \loss(\objmin{\theta}_t) \\
&\approx \mode{\theta}(\lambda_t) + \nabla^2_{\lambda}\leg{A}(\lambda_t) \nabla_{\theta} \loss(\mode{\theta}(\lambda_t)),
\end{align}
where a zeroth-order approximation has been made in the argument of $\nabla_{\theta} \loss$. Identifying the term $\nabla^2\leg{A}(\lambda_t)$ with the inverse of the information $\mathcal{I}_{\lambda_t}$ completes the proof of the proposition.
\end{proof}
\begin{proof}[Proof of Corollary~\ref{cor:normal_fixed_var}]
We consider the log-partition function $A(\theta)=\frac{1}{2} \,\theta^{\intercal}\Sigma\theta$ of the normal distribution with unknown mean and known variance $\Sigma$. The convex conjugate of $A(\theta)$ is
\begin{align}
\leg{A}(\lambda)
= \sup_{\theta\in\Theta}\left\{\theta^{\intercal}\lambda-\tfrac{1}{2}\theta^{\intercal}\Sigma\theta\right\}
= \tfrac{1}{2}\,\lambda^{\intercal}\Sigma^{-1}\lambda.
\end{align}
Possibility functions in $\mathcal{G}_A(\Theta)$ are of the form
\begin{subequations}
\begin{align}
g_{\lambda}(\theta)
&= \exp\!\left(\lambda^{\intercal}\theta-\tfrac{1}{2}\lambda^{\intercal}\Sigma^{-1}\lambda
-\tfrac{1}{2}\theta^{\intercal}\Sigma\theta\right) \\
&= \exp\!\left(-\tfrac{1}{2}(\theta-\Sigma^{-1}\lambda)^{\intercal}\Sigma(\theta-\Sigma^{-1}\lambda)\right) \\
&= \overline{\mathrm{N}}(\theta;\Sigma^{-1}\lambda,\Sigma^{-1}).
\end{align}
\end{subequations}
A convenient property of these exponential families, as in the probabilistic case, is that $\mathcal{I}_{\lambda} = \Sigma$ which does not depend on $\lambda$.
We also obtain from the properties of the normal possibility function that $\mode{\theta}(\lambda) = \Sigma^{-1}\lambda$.
Therefore, the approximate update rule can be expressed as
\begin{align}
\lambda_{t+1} & \approx \lambda_t - \rho_t \Sigma^{-1} \nabla_{\theta} \loss(\Sigma^{-1}\lambda_t).
\end{align}
Finally, noticing that
\begin{align}
\nabla_{\theta} \loss(\theta) = \nabla_{\theta} \loss_{\mathrm{s}}(\mu(\theta)) = \nabla_{\theta} \mu(\theta) \nabla_{\mu} \loss_{\mathrm{s}}( \mu(\theta) ) = \Sigma \nabla_{\mu} \loss_{\mathrm{s}}( \Sigma \theta ),
\end{align}
the update rule could also be expressed as
\begin{align}
\lambda_{t+1} \approx \lambda_t - \rho_t \nabla_{\mu} \loss_{\mathrm{s}}( \lambda_t ).
\end{align}
\end{proof}
\begin{proof}[Proof of Corollary~\ref{cor:binomial}]
We consider the log-partition function $A(\theta) = n \log(1+\exp(\theta))$ of the binomial distribution with known number of trials $n$. The convex conjugate of $A(\theta)$ is
\begin{align}
\leg{A}(\lambda)
&= \sup_{\theta\in\Theta}\left\{\theta^{\intercal}\lambda-n \log(1+\exp(\theta))\right\} \\
&= \lambda \log \tfrac{\lambda}{n} + (n-\lambda)\log\left(1-\tfrac{\lambda}{n} \right).
\end{align}
Subsequently, we can also obtain the following:
\begin{align}
\mode{\theta} &= \nabla_\lambda\leg{A}(\lambda) = \log \tfrac{\lambda}{n-\lambda} \\
\mathcal{I}_{\lambda}^{-1} &= \nabla^2_\lambda\leg{A}(\lambda) = \tfrac{1}{\lambda(1-\lambda/n)}
\end{align}
Therefore, the approximate update rule can be expressed as:
\begin{align}
\lambda_{t+1} \approx \lambda_t - \rho_t\frac{1}{\lambda_t(1-\lambda_t/n)} \nabla_{\theta} \loss\bigg( \log \frac{\lambda_t}{n - \lambda_t} \bigg).
\end{align}
Recall that $p(\theta) = (1+\exp(-\theta))^{-1} = \sigma(\theta)$, where $\sigma$ is the \textit{sigmoid} function. Chain rule gives us
\begin{align}
\nabla_{\theta} \loss(\theta) &= \nabla_{\theta} p(\theta) \nabla_{p} \loss_{\mathrm{s}} (\sigma(\theta)) \\
&= \sigma(\theta)(1-\sigma(\theta)) \nabla_{p} \loss_{\mathrm{s}} (\sigma(\theta))
\end{align}
At $\theta = \log(\lambda/(n-\lambda))$, it holds that $\sigma(\theta) = \lambda/n$. Hence, the update is given by
\begin{align}
\lambda_{t+1} \approx \lambda_t - \frac{\rho_t}{n} \nabla_{p} \loss_s( \lambda_t/n ).
\end{align}
\end{proof}
\end{document}