A Latent Source Model for Patch-Based Image Segmentation
Extracted source text ยท arXiv 1510.01648
% This is LLNCS.DEM the demonstration file of
% the LaTeX macro package from Springer-Verlag
% for Lecture Notes in Computer Science,
% version 2.4 for LaTeX2e as of 16. April 2010
%
\documentclass{llncs}
\usepackage{makeidx} % allows for indexgeneration
\usepackage{amsmath}
\usepackage{amstext}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{wrapfig}
\begin{document}
\frontmatter % for the preliminaries
\pagestyle{headings} % switches on printing of running heads
\addtocmark{Patch-Based Image Segmentation} % additional mark in the TOC
\mainmatter % start of the contributions
\title{A Latent Source Model for \\ Patch-Based Image Segmentation}
\titlerunning{Patch-Based Image Segmentation}
\author{George H.~Chen \and Devavrat Shah \and Polina Golland}
% index{Chen, George}
% index{Shah, Devavrat}
% index{Golland, Polina}
\authorrunning{G.~H.~Chen et al.} % abbreviated author list (for running head)
%%%% list of authors for the TOC (use if author list has to be modified)
\tocauthor{George H.~Chen, Devavrat Shah, and Polina Golland}
\institute{Massachusetts Institute of Technology, Cambridge MA 02139, USA}
\maketitle
\begin{abstract}
Despite the popularity and empirical success of patch-based nearest-neighbor
and weighted majority voting approaches to medical image segmentation, there
has been no theoretical development on when, why, and how well these
nonparametric methods work. We bridge this gap by providing a theoretical
performance guarantee for nearest-neighbor and weighted majority voting
segmentation under a new probabilistic model for patch-based image
segmentation. Our analysis relies on a new local property for how similar
nearby patches are, and fuses existing lines of work on modeling natural
imagery patches and theory for nonparametric classification. We use the model
to derive a new patch-based segmentation algorithm that iterates between
inferring local label patches and merging these local segmentations to produce
a globally consistent image segmentation. Many existing patch-based algorithms
arise as special cases of the new algorithm.
\end{abstract}
\section{Introduction}
Nearest-neighbor and weighted majority voting methods have been widely used
in medical image segmentation, originally at the pixel or voxel level
% \cite{left_atrium_segmentation_2010,label_fusion_generative_model_2010}
\cite{label_fusion_generative_model_2010}
and more recently for image patches
\cite{rueckert_patches_2013,coupe_2011,rousseau_2011,wachinger_2013}.
Perhaps the primary reason for the popularity of these nonparametric methods
is that standard label fusion techniques for image segmentation require robust
nonrigid registration whereas patch-based methods sidestep nonrigid image
alignment altogether. Thus, patch-based approaches provide a promising
alternative to registration-based methods for problems that present alignment
challenges, as in the case of whole body scans or other applications
characterized by large anatomical variability.
A second reason for patch-based methods' growing popularity lies in their
efficiency of computation: fast approximate nearest-neighbor search
algorithms, tailored for patches \cite{patchmatch} and for high-dimensional
spaces more generally
% \cite{ailon_2006_ann_jl,andoni_2006_euclidean_lsh,datar_2004_good_intro_lsh,%
% gionis_1999_original_lsh,muja_flann_2009},
(e.g.,
\cite{ailon_2006_ann_jl,muja_flann_2009}),
can rapidly find similar patches, and can readily parallelize across
search queries. For problems where the end goal is segmentation or a
decision based on segmentation, solving numerous nonrigid registration
subproblems required for
standard label fusion could be a computationally expensive detour that, even if
successful, might not produce better solutions than a patch-based approach.
Many patch-based image segmentation methods can be viewed as variations of the
following simple algorithm. To determine whether a pixel in the new image
should be foreground (part of the object of interest) or background, we
consider the patch centered at that pixel. We compare this image patch to
patches in a training database, where each training patch is labeled either
foreground or background depending on the pixel at the center of the training
patch. We transfer the label from the closest patch in the training database
to the pixel of interest in the new image. A plethora of embellishments
improve this algorithm, such as, but not limited to, using $K$ nearest
neighbors or weighted majority voting instead of only the nearest neighbor
\cite{coupe_2011,rousseau_2011,wachinger_2013}, incorporating hand-engineered
or learned feature descriptors \cite{wachinger_2013}, cleverly choosing the
shape of a patch \cite{wachinger_2013}, and enforcing consistency among
adjacent pixels by assigning each training intensity image patch to a label
patch rather than a single label \cite{rousseau_2011,wachinger_2013}, or by
employing a Markov random field \cite{mrf_superresolution}.
Despite the broad popularity and success of nonparametric patch-based image
segmentation and the smorgasbord of tricks to enhance its performance, the
existing work has been empirical with no theoretical justification for when
and why such methods should work and, if so, how well and with how much
training data. In this paper, we bridge this gap between theory and practice
for nonparametric patch-based image segmentation algorithms. We propose a
probabilistic model for image segmentation that draws from recent work on
modeling natural imagery patches \cite{epll,patch_gmm}. We begin in Section
\ref{sec:myopic} with a simple case of our model that corresponds to inferring
each pixel's label separately from other pixels. For this special case of
so-called pointwise segmentation, we provide a theoretical performance
guarantee for patch-based nearest-neighbor and weighted majority voting
segmentation in terms of the available training data. Our analysis borrows
from existing theory on nonparametric time series classification
\cite{georgehc_2013_time_series_nips} and crucially relies on a new structural
property on neighboring patches being sufficiently similar. We present our
full model in Section \ref{sec:full} and derive a new iterative patch-based
image segmentation algorithm that combines ideas from patch-based image
restoration \cite{epll} and distributed optimization \cite{boyd_admm_2011}.
This algorithm alternates between inferring label patches separately and
merging these local estimates to form a globally consistent segmentation. We
show how various existing patch-based algorithms are special cases of this new
algorithm.
\section{Pointwise Segmentation and a Theoretical Guarantee\label{sec:myopic}}
% \textbf{Notation.}
For an image $A$, we use $A(i)$ to denote the value of image~$A$ at pixel $i$,
and $A[i]$ to denote the patch of image $A$ centered at pixel $i$ based on a
pre-specified patch shape; $A[i]$ can include feature descriptors in addition
to raw intensity values. Each pixel $i$ belongs to a finite, uniformly sampled
lattice~$\mathcal{I}$.
\textbf{Model.}
Given an intensity image $Y$, we infer its label image~$L$ that delineates an
object of interest in $Y$. In particular, for each pixel $i \in \mathcal{I}$,
we infer label $L(i) \in \{+1, -1\}$, where $+1$ corresponds to foreground
(object of interest) \mbox{and $-1$} to background. We make this inference
using patches of image $Y$, each patch of dimensionality~$d$ (e.g., for 2D
images and 5-by-5 patches, \mbox{$d=5^2=25$}). We model the joint distribution
$p(L(i),Y[i])$ of label $L(i)$ and image patch $Y[i]$ as a generalization of a
Gaussian mixture model (GMM) with diagonal \mbox{covariances, where} each
mixture component corresponds to either \mbox{$L(i)=+1$} or \mbox{$L(i)=-1$}.
We define this generalization, called a \textit{diagonal sub-Gaussian mixture
model}, shortly.
First, we provide a concrete example where label $L(i)$ and patch $Y[i]$ are
related through a GMM with $\mathcal{C}_i$ mixture components. Mixture
component $c \in \{1, \dots, \mathcal{C}_i\}$ occurs with probability
$\rho_{ic} \in [0,1]$ and has mean vector $\mu_{ic} \in \mathbb{R}^d$ and
label $\lambda_{ic} \in \{+1,-1\}$. In this example, we assume that all
covariance matrices are $\sigma^2 \mathbf{I}_{d\times d}$, and that there
exists constant $\rho_{\min}>0$ such that $\rho_{ic} \ge \rho_{\min}$ for all
$i,c$. Thus, image patch $Y[i]$ belongs to mixture component $c$ with
probability $\rho_{ic}$, in which case $Y[i] = \mu_{ic} + W_i$, where vector
$W_i \in \mathbb{R}^d$ consists of white Gaussian noise with variance
$\sigma^2$, and $L(i)=\lambda_{ic}$. Formally,
\[
p(L(i),Y[i])
=\sum_{c=1}^{\mathcal{C}_i}
\rho_{ic}~\!
\mathcal{N}(Y[i];\mu_{ic},\sigma^2 \mathbf{I}_{d\times d})
% \ind
\delta(L(i)=\lambda_{ic}),
\]
where $\mathcal{N}(\cdot; \mu, \Sigma)$ is a Gaussian density with mean $\mu$
and covariance $\Sigma$, and $\delta(\cdot)$ is the indicator function. % Each
% pixel~$i$ has its own mixture model with parameters
% $(\rho_{ic}, \mu_{ic}, \lambda_{ic})$ for $c = 1,\dots,\mathcal{C}_i$.
The \textit{diagonal sub-Gaussian mixture model} refers to a % simple
generalization where noise vector $W_i$ consists of zero-mean i.i.d.~random
entries whose distribution has tails that decay at least as fast as that of a
Gaussian random variable. Formally, a zero-mean random variable $X$ is
sub-Gaussian %~\cite{subgaussian}
with parameter $\sigma$ if its moment
generating function $\mathbb{E}[e^{s X}]$ satisfies $\mathbb{E}[e^{s X}]
\le e^{s^2\sigma^2/2}$ for all $s \in \mathbb{R}$. Examples of such random
variables include
$\mathcal{N}(0,\sigma^2)$ and $\text{Uniform}[-\sigma,\sigma]$.
Every pixel is associated with its own diagonal sub-Gaussian mixture model
whose parameters
$(\rho_{ic}, \mu_{ic}, \lambda_{ic})$ for $c = 1,\dots,\mathcal{C}_i$
are fixed but unknown. Similar to recent work on modeling
generic natural image patches \cite{epll,patch_gmm}, we do not model how
different overlapping patches behave jointly and instead only model how each
individual patch, viewed alone, behaves. We suspect that medical image patches
have even more structure than generic natural image patches, which are very
accurately modeled by a GMM~\cite{patch_gmm}.
Rather than learning the mixture model components, we instead take a
nonparametric approach, using available training data in nearest-neighbor or
weighted majority voting schemes to infer label $L(i)$ from image patch
$Y[i]$. To this end, we assume we have access to~$n$ i.i.d.~training
intensity-label \textit{image} pairs $(Y_1,L_1), \dots, (Y_n,L_n)$ that obey
our probabilistic model above.
\textbf{Inference.}
We consider two simple segmentation methods that operate on each pixel $i$
separately, inferring label $L(i)$ only based on image patch $Y[i]$.
\textit{Pointwise nearest-neighbor segmentation} first finds which training
intensity image $Y_u$ has a patch centered at pixel $j$ that is closest
to observed patch $Y[i]$. This amounts to solving
$
(\widehat{u},\widehat{j})
=\text{argmin}_{u\in\{1,2,\dots,n\}, j \in N(i)}
\|Y_u[j]-Y[i]\|^2,
$
where $\|\cdot\|$ denotes Euclidean norm, and $N(i)$ refers to a
user-specified finite set of pixels that are neighbors of pixel $i$. Label
$L(i)$ is estimated to be $L_{\widehat{u}}(\,\widehat{j}\,)$.
\textit{Pointwise weighted majority voting segmentation} first computes the
following weighted votes for labels $\ell \in \{+1, -1\}$:
\[
V_{\ell}(i|Y[i]; \theta)
\triangleq
\sum_{u=1}^n
\sum_{j\in N(i)}
\exp\big(-\theta\|Y_u[j]-Y[i]\|^2\big)
\delta(L_u(j)= \ell),
\]
where $\theta$ is a scale parameter, and $N(i)$ again refers to
user-specified neighboring pixels of pixel $i$. Label $L(i)$ is estimated to
be the label $\ell$ with the higher vote $V_\ell(i|Y[i];\theta)$.
Pointwise nearest-neighbor segmentation can be viewed as this weighted voting
with $\theta \rightarrow \infty$. Pointwise weighted majority voting
has been used extensively for patch-based segmentation
\cite{rueckert_patches_2013,coupe_2011,rousseau_2011,wachinger_2013}, where we
note that our formulation readily allows for one to choose which training
image patches are considered neighbors, what the patch shape is, and whether
feature descriptors are part of the intensity patch vector $Y[i]$. % For
% example, a simple choice of feature at pixel $i$ is the coordinate for pixel
% $i$ itself. Thus, we can encode as part of the exponentially decaying weight
% $\exp(-\theta\|Y_u[j] - Y[i]\|^2)$ how far apart pixels $i$ and $j \in N(i)$
% are, yielding a weighted voting algorithm previously derived from a Bayesian
% model that explicitly models this displacement \cite{rueckert_patches_2013}.
\textbf{Theoretical guarantee.}
The model above allows nearby pixels to be associated with dramatically
different mixture models. However, real images are ``smooth'' with patches
centered at two adjacent pixels likely similar. We incorporate this smoothness
via a structural property on the sub-Gaussian mixture model parameters
associated with nearby pixels.
% To build some intuition, we consider two extremes. First, it could be
% that the $|\mathcal{I}|$ mixture models (one per pixel) are actually all
% identical. This means that every intensity patch comes from the same
% distribution. If we know this, then when we do pointwise nearest-neighbor or
% weighted majority voting segmentation, we could compare an observed intensity
% patch $Y[i]$ with a training image patch centered \textit{anywhere} in the
% training image, since the patches all follow the same distribution. On the
% opposite extreme, the $|\mathcal{I}|$ mixture models could have no
% commonalities, and so when we do pointwise nearest-neighbor or weighted
% majority voting segmentation, it only makes sense to compare intensity patch
% $Y[i]$ with training image patches also centered at pixel $i$. We can
% interpolate between these two extremes by saying how far away two pixels have
% to be while still sharing mixture component parameters, which we formalize
% as follows:
%
% \noindent
We refer to this property as the \textit{jigsaw condition},
% Our structural property on mixture model components depends on some true
% neighborhood $N^*(i)$ for each pixel $i$, where the mixture model associated
% with pixel $i$ shares mixture components with mixture models
% associated with pixels in $N^*(i)$. More precisely,
which holds
if for every mixture component
$(\rho_{ic},\mu_{ic},\lambda_{ic})$ of the diagonal sub-Gaussian mixture model
associated with pixel $i$, there exists a neighbor $j \in N^*(i)$ such that
the diagonal sub-Gaussian mixture model associated with pixel $j$ also has a
mixture component with mean $\mu_{ic}$, label $\lambda_{ic}$, and mixture
weight at least $\rho_{\min}$; this weight need not be equal to $\rho_{ic}$.
The shape and size of neighborhood $N^*(i)$, which is fixed and unknown,
control how similar the mixture models are across image pixels.
%
% For the two extremes discussed previously, the true neighborhood $N^*(i)$
% corresponds to the full space of pixels in the first case, and
% $N^*(i) = \{i\}$ in the second case. As we already hinted at, what $N^*(i)$ is
Note that $N^*(i)$ affects how far from pixel $i$ we should look for training
patches, i.e., how to choose neighborhood $N(i)$ in pointwise nearest-neighbor
and weighted majority voting segmentation, where ideally $N(i) = N^*(i)$.
% Naturally, setting $N(i)=N^*(i)$
% would be ideal but if we do not know $N^*(i)$, setting $N(i)$ to at least
% conntain $N^*(i)$
\emph{Separation gap.}
Our theoretical result also depends on the separation ``gap'' between training
intensity image patches that correspond to the two different labels:
\[
\mathcal{G}
\triangleq\min_{\substack{u,v\in\{1,\dots,n\},\\
i\in\mathcal{I},j\in N(i)\text{ s.t. }L_u(i)\ne L_v(j)}
}\|Y_u[i]-Y_v[j]\|^2.
\]
Intuitively, a small separation gap corresponds to the case of two training
intensity image patches that are very similar but one corresponds to
foreground and the other to background. In this case, a nearest-neighbor
approach may easily select a patch with the wrong label, resulting in an
error.
We now state our main theoretical guarantee. The proof is left to the
supplementary material and builds on existing time series classification
analysis~\cite{georgehc_2013_time_series_nips}.
\begin{theorem}
\label{thm:1nn-wmv-patches}
Under the model above with $n$ training intensity-label image pairs and
provided that the jigsaw condition holds with neighborhood $N^*$, pointwise
nearest-neighbor and weighted majority voting segmentation (with
user-specified neighborhood $N$ satisfying $N(i)\supseteq N^*(i)$ for every pixel $i$ and
with parameter $\theta = \frac{1}{8\sigma^2}$ for weighted majority
voting) achieve expected pixel labeling error rate
\begin{align*}
\mathbb{E}\bigg[
\frac{1}{|\mathcal{I}|}
\sum_{i\in\mathcal{I}}
% \ind\{\text{mislabel pixel }i\}
\delta(\text{mislabel pixel }i)
\bigg]
&\le
|\mathcal{I}|\mathcal{C}_{\max}\exp\Big(-\frac{n\rho_{\min}}{8}\Big)
+
|N|n\exp\Big(-\frac{\mathcal{G}}{16\sigma^{2}}\Big),
\end{align*}
where
$\mathcal{C}_{\max}$ is the maximum number of mixture components of any
diagonal sub-Gaussian mixture model associated with a pixel, and
$|N|$ is the largest user-specified neighborhood of any pixel.
\end{theorem}
\noindent
To interpret this theorem, we consider sufficient conditions for each term on
the right-hand side bound to be at most $\varepsilon/2$ for $\varepsilon \in
(0,1)$. For the first term, the number of training intensity-label image pairs
$n$ should be sufficiently large so that we see all the different mixture model
components in our training data:
$
n
\ge
\frac{8}{\rho_{\min}}
\log( 2|\mathcal{I}|\mathcal{C}_{\max}/\varepsilon )
$.
For the second term, the gap $\mathcal{G}$ should be sufficiently large so
that the nearest training intensity image patch found does not produce a
segmentation error:
$\mathcal{G}
\ge
16\sigma^2
\log( 2|N|n/\varepsilon )$.
There are different ways to change the gap, such as changing the patch shape
and including hand-engineered or learned patch features. % For
% example, if the mixture models are all Gaussian mixture models, then provided
% that no true mean vectors of opposite labels are the same, then gap
% $\mathcal{G}$ turns out to grow as $\Omega(\sigma^2 d)$.
Intuitively, using
larger patches $d$ should widen the gap, but using larger patches also means
that the maximum number of mixture components $\mathcal{C}_{\max}$ needed to
represent a patch increases, possibly quite dramatically. % (In the extreme case,
% the patch is the entire image.)
% We conjecture that the dependence on $n$ in the second term is loose.
The dependence on $n$ in the second term results from a worst-case analysis.
To keep the gap from having to grow as $\log(n)$, we could subsample the
training data so that $n$ is large enough to capture the diversity of mixture
model components yet not so large that it overcomes the gap. In particular,
treating $\mathcal{C}_{\max}$, $\sigma^2$, and $\rho_{\min}$ as constants that
depend on the application of interest and could potentially be estimated from
data, collecting $n = \Theta(\log(|\mathcal{I}|/\varepsilon))$ training image
pairs and with a gap
$\mathcal{G}
= \Omega\big(\log(
(|N|\log |\mathcal{I}|)/\varepsilon
)\big)$,
both algorithms achieve an expected error rate of at most $\varepsilon$. The
intuition is that as $n$ grows large, if we continue to consider all training
subjects, even those that look very different from the new subject, we are
bound to get unlucky (due to noise in intensity images) and, in the worst
case, encounter a training image patch that is close to a test image patch but
has the wrong label. Effectively, outliers in training data muddle
nearest-neighbor inference, and more training data means possibly more
outliers.
\section{Multipoint Segmentation}
\label{sec:full}
\textbf{Model.}
We generalize the basic model to infer label patches $L[i]$ rather than just a
single pixel's label $L(i)$. Every label patch $L[i]$ is assumed to have
dimensionality $d'$, where $d$ and $d'$ need not be equal. For example, for 2D
images, $Y[i]$ could be a 5-by-5 patch ($d=25$) whereas $L[i]$ could be a
3-by-3 patch ($d'=9$). When $d'>1$, estimated label patches must be merged to
arrive at a globally consistent estimate of label image~$L$. This case is
referred to as multipoint segmentation.
In this general case, we assume there to be $k$ latent label images
$\Lambda_1, \dots, \Lambda_k$ that occur with probabilities $\pi_1, \dots,
\pi_k$. To generate intensity image $Y$, we first sample label
image $\Lambda \in \{\Lambda_1, \dots, \Lambda_k\}$ according to probabilities
$\pi_1, \dots, \pi_k$. Then we sample label image $L$ to be a perturbed
version of $\Lambda$ such that
$
p(L|\Lambda) \propto \exp(-\alpha\mathbf{d}(L, \Lambda))
$
for some constant $\alpha \ge 0$ and differentiable ``distance'' function
$\mathbf{d}(\cdot, \cdot)$. For example, $\mathbf{d}(L, \Lambda)$ could relate
to volume overlap between the segmentations represented by label images $L$
and $\Lambda$ with perfect overlap yielding distance 0. Finally, intensity
image~$Y$ is generated so that for each pixel $i\in\mathcal{I}$, patch $Y[i]$
is a sample from a mixture model patch prior $p(Y[i]|L[i])$. If $\alpha=0$,
$d'=1$, and the mixture model is diagonal sub-Gaussian, we get our
earlier~model.
We refer to this formulation as a \textit{latent source model} since the
intensity image patches could be thought of as generated from the latent
``canonical'' label images $\Lambda_1, \dots, \Lambda_k$ combined with the
latent mixture model clusters linking label patches $L[i]$ to intensity
patches $Y[i]$. This hierarchical structure enables local appearances around a
given pixel to be shared across the canonical label~images.
\textbf{Inference.}
We outline an iterative algorithm based on the expected patch log-likelihood
(EPLL) framework \cite{epll}, deferring details to the supplementary material.
The EPLL framework seeks a label image $L$ by solving
\begin{equation*}
\widehat{L}
=\!\!
\underset{L \in \{+1,-1\}^{|\mathcal{I}|}}
{\text{argmax}}
\bigg\{\!
\log\bigg(
\sum_{g=1}^k
\pi_g
\exp(-\alpha\mathbf{d}(L,\Lambda_g))
\bigg)
+
\frac{1}{|\mathcal{I}|}
\sum_{i\in\mathcal{I}}\log p(Y[i]|L[i])\!
\bigg\}.
\end{equation*}
The first term in the objective function encourages label image $L$ to be
close to the true label images $\Lambda_1, \dots, \Lambda_k$. The second term
is the ``expected patch log-likelihood'', which favors solutions whose local
label patches agree well on average with the local intensity patches according
to the patch priors. Since latent label images $\Lambda_1, \dots, \Lambda_k$
are unknown, we use training label images $L_1, \dots, L_n$ as proxies
instead, replacing the first term in the objective function with
$F(L;\alpha)
\triangleq
\log
\big(
\frac{1}{n}
\sum_{u=1}^n
\exp(-\alpha\mathbf{d}(L,L_u))\big)$.
Next, we approximate the unknown patch prior $p(Y[i]|L[i])$ with a kernel
density estimate
\[
\widetilde{p}(Y[i]|L[i];\gamma)
\propto\sum_{u=1}^{n}\sum_{j \in N(i)}
\mathcal{N}\Big(
Y[i];
Y_u[j],
\frac{1}{2\gamma}\mathbf{I}_{d\times d}
\Big)
\, %\cdot
% \ind\{L[i]=L_u[j]\},
\delta(L[i]=L_u[j]),
\]
where the user specifies a neighborhood $N(i)$ of pixel $i$, and constant
$\gamma>0$ that controls the Gaussian kernel's bandwidth. We group the pixels
so that nearby pixels within a small block all share the same kernel density
estimate. This approximation assumes a stronger version of the
jigsaw condition from Section \ref{sec:myopic} since the algorithm operates as
if nearby pixels have the same mixture model as a patch prior. We thus
maximize objective
$F(L;\alpha) +
\frac{1}{|\mathcal{I}|}
\sum_{i \in \mathcal{I}} \log \widetilde{p}(Y[i]|L[i];\gamma)$. % to determine
% label image $L$.
Similar to the original EPLL method~\cite{epll}, we introduce an auxiliary
variable $\xi_{i}\in\mathbb{R}^{d'}$ for each patch $L[i]$, where $\xi_i$ acts
as a local estimate for $L[i]$. Whereas two patches $L[i]$ and $L[j]$ that
overlap in label image~$L$ must be consistent across the overlapping pixels,
there is no such requirement on their local estimates $\xi_i$ and $\xi_j$. In
summary, we maximize the objective
$
F(L;\alpha)+
\frac{1}{|\mathcal{I}|}
\sum_{i \in \mathcal{I}} \log \widetilde{p}(Y[i]|\xi_i;\gamma)
-\frac{\beta}{2}\sum_{i \in \mathcal{I}} \|L[i] - \xi_i\|^2
$ for $\beta>0$, subject to constraints $L[i] = \xi_i$ that are enforced
using Lagrange multipliers. We numerically optimize this cost function using the
Alternating Direction Method of Multipliers for
distributed optimization \cite{boyd_admm_2011}. Given the current estimate of
label image $L$, the algorithm produces estimate $\xi_i$ for $L[i]$ given
$Y[i]$ in parallel across $i$. Next, it updates $L$ based on $\xi_i$ via a
gradient method. Finally, the Lagrange multipliers are updated to penalize
large discrepancies between $\xi_i$ and $L[i]$.
Fixing $\xi_i$ and updating $L$ corresponds to merging local patch estimates
to form a globally consistent segmentation. This is the only step that
involves expression $F(L;\alpha)$. With $\alpha=0$ and forcing the Lagrange
multipliers to always be zero, the merging becomes a simple averaging of
overlapping label patch estimates $\xi_i$. This algorithm corresponds to
existing multipoint patch-based segmentation algorithms
\cite{coupe_2011,rousseau_2011,wachinger_2013} and the in-painting technique
achieved by the original EPLL method. Setting $\alpha=\beta=0$ and $d'=1$
yields pointwise weighted majority voting with parameter $\theta = \gamma$.
When $\alpha > 0$, a global correction is applied, shifting the label image
estimate closer to the training label images. This should produce better
estimates when the full training label images can, with small perturbations
as measured by $\mathbf{d}(\cdot, \cdot)$, explain new intensity images.
% For example, a distance based on aligning label images
% may allow for more flexible yet plausible overall label images, with the hope
% that aligning binary label images is cheaper and far more robust than aligning
% intensity images. An alternative is to use a cost function like Dice overlap
% that is efficient to compute but not based on image alignment, so the
% perturbation it allows for is limited.
\begin{wrapfigure}[14]{r}[0pt]{3.7cm}
\includegraphics[height=3.5cm,clip=true,trim=1em 1.6em 0 1.4em]{dice-overlap-liver-miccai}
\caption{Liver segmentation results.}
\label{fig:dice}
\end{wrapfigure}
\textbf{Experimental results.}
We empirically explore the new iterative algorithm on 20 labeled
thoracic-abdominal contrast-enhanced CT scans from the Visceral
\textsc{anatomy3} dataset \cite{anatomy3}. We train the model on 15 scans and
test on the remaining 5 scans. The training procedure amounted to using 10 of
the 15 training scans to estimate the algorithm parameters in an exhaustive
sweep, using the rest of the training scans to evaluate parameter settings.
Finally, the entire training dataset of 15 scans is used to segment the test
dataset of 5 scans using the best parameters found during training. For each
test scan, we first use a fast affine registration to roughly align each
training scan to the test scan. Then we apply four different algorithms: a
baseline majority voting algorithm (denoted ``MV") that simply averages the
training label images that are now roughly aligned to the test scan, pointwise
nearest neighbor (denoted ``1NN") and weighted majority voting (denoted
``WMV") segmentation that both use approximate nearest patches, and finally
our proposed iterative algorithm (denoted ``ADMM''), setting distance
$\mathbf{d}$ to one minus Dice overlap. Note that Dice overlap can be
reduced to a differentiable function by relaxing our optimization to allow
each label to take on a value in $[-1, 1]$. By doing so, the Dice overlap of
label images $L$ and $\Lambda$ is given by
$2\langle \widetilde{L}, \widetilde{\Lambda}\rangle/
(\langle \widetilde{L}, \widetilde{L} \rangle +
\langle \widetilde{\Lambda}, \widetilde{\Lambda} \rangle)$,
where $\widetilde{L} = (L+1)/2$ and $\widetilde{\Lambda} = (\Lambda+1)/2$.
We segmented the liver, spleen, left kidney, and right kidney. We report Dice
overlap scores for the liver in Fig.~\ref{fig:dice} using the four algorithms.
Our results for segmenting the other organs follow a similar trend where the
proposed algorithm outperforms pointwise weighted majority voting, which
outperforms both pointwise nearest-neighbor segmentation and the baseline
majority voting. For the organs segmented, there was little benefit to having
$\alpha>0$, suggesting the local patch estimates to already be quite
consistent and require no global correction.
\section{Conclusions}
We have established a new theoretical performance guarantee for two
nonparametric patch-based segmentation algorithms, uniting recent lines of
work on modeling patches in natural imagery and on theory for nonparametric
time series classification. Our result indicates that if nearby patches behave
as mixture models with sufficient similarity, then a myopic segmentation works
well, where its quality is stated in terms of the available training data.
Our main performance bound provides insight into how one should
approach building a training dataset for patch-based segmentation.
The looseness in the bound could be attributed to outliers in training data.
Detecting and removing these outliers should lead to improved segmentation
performance.
% While
% %we suspect
% our main performance bound is likely loose, it provides insight into how one should
% approach building a training dataset for patch-based segmentation.
From a modeling standpoint, understanding the joint behavior of patches could
yield substantial new insights into exploiting macroscopic structure in images
rather than relying only on local properties. In a related direction, while we
have modeled the individual behavior of patches, an interesting theoretical
problem is to find joint distributions on image pixels that lead to such
marginal distributions on patches. Do such joint distributions exist? If not,
is there a joint distribution whose patch marginals approximate the mixture
models we use? These questions outline rich areas for future research.
\smallskip{}
\noindent
\textbf{Acknowledgments.} This work was supported in part by the NIH NIBIB
NAC P41EB015902 grant, MIT Lincoln Lab, and Wistron Corporation.
GHC was supported by an NDSEG fellowship.
\bibliographystyle{plain}
\bibliography{latent_source_short}
% THE SUPPLEMENTARY MATERIAL IS BELOW
% \newpage
%
% \appendix
%
% \section{Proof of Theorem \ref{thm:1nn-wmv-patches}}
%
% We import proof techniques used to establish performance guarantees for
% nearest-neighbor and weighted majority voting time series classification
% \cite{georgehc_2013_time_series_nips}. Our adaptation of these techniques to
% our patch-based image segmentation setting crucially relies on the jigsaw
% condition. In what follows, we abbreviate ``diagonal sub-Gaussian mixture
% model'' by ``MM'', letting $\text{MM}(i)$ denote the diagonal sub-Gaussian
% mixture model associated with pixel $i$.
%
% We begin with three lemmas that progressively build on each other and that
% provide a sufficient condition on how many training intensity-label image
% pairs are sufficient for segmentation. The first two are stated in terms of
% categorical random variables in general and then the third lemma ties the
% categorical random variable results with our simplified latent source model
% for pointwise segmentation.
%
% The first lemma effectively allows us to say how many training samples
% from a mixture model are sufficient so that we will, with high probability,
% see a particular mixture component.
% \begin{lemma}
% \label{lem:enough-samples-from-single-category}
% With $n$ i.i.d.~samples from a categorical distribution with category weights
% $\pi_1, \dots, \pi_k$,
% \[
% \mathbb{P}
% \big(
% \text{category }g\text{ has }
% \le
% \frac{1}{2} n \pi_g
% \text{ occurrences}
% \big)
% \le
% \exp\Big( -\frac{n\pi_g}{8} \Big).
% \]
% \end{lemma}
% \begin{proof}
% This follows from a Chernoff bound applied to the binomial distribution. In
% particular, the $g$-th category occurs $A \sim \text{Bin}(n,\pi_g)$ times.
% Then
% \[
% \mathbb{P}
% \big(
% A \le \frac{1}{2} n \pi_g
% \big)
% \le
% \exp\Big(
% -\frac{1}{2}\cdot\frac{(n\pi_g - \frac{1}{2} n \pi_g)^2}{n \pi_g}
% \Big)
% = \exp\Big( -\frac{n \pi_g}{8} \Big). \tag*{\qed} % \qedhere
% \]
% \end{proof}
% The shortcoming of the first lemma is that it only looks at a single
% mixture component. We want to ensure that we see all mixture
% components enough times, which the next lemma provides a result
% for.
% \begin{lemma}
% \label{lem:enough-samples-from-every-category}
% With $n$ i.i.d.~samples from a categorical distribution with category weights
% $\pi_1, \dots, \pi_k$,
% \[
% \mathbb{P}
% \big(
% \exists
% \text{ a category with }
% \le
% \frac{1}{2} n \pi_{\min}
% \text{ occurrences}
% \big)
% \le
% k \exp\Big( -\frac{n\pi_{\min}}{8} \Big),
% \]
% where $\pi_{\min} \triangleq \min\{\pi_1, \dots, \pi_k\}$.
% \end{lemma}
% \begin{proof}
% This follows from a union bound:
% \begin{align*}
% &\mathbb{P}
% \Big(
% \bigcup_{g=1}^k
% \{\text{category }g\text{ has }
% \le
% \frac{1}{2}n\pi_{\min}
% \text{ occurrences}\}
% \Big) \\
% &\le
% \sum_{g=1}^k
% \mathbb{P}
% \Big(
% \text{category }g\text{ has }
% \le
% \frac{1}{2}n\pi_{\min}
% \text{ occurrences}
% \Big) \\
% &\le
% \sum_{g=1}^k
% \mathbb{P}
% \Big(
% \text{category }g\text{ has }
% \le
% \frac{1}{2}n\pi_g
% \text{ occurrences}
% \Big) \\
% &\le
% \sum_{g=1}^k
% \exp\Big( -\frac{n\pi_g}{8} \Big)
% \le
% \sum_{g=1}^k
% \exp\Big( -\frac{n\pi_{\min}}{8} \Big)
% =k \exp\Big( -\frac{n\pi_{\min}}{8} \Big). \tag*{\qed} % \qedhere
% \end{align*}
% \end{proof}
% Finally, in the third lemma, we tie these results with the pointwise
% segmentation model. Effectively, this lemma says how many training
% intensity-label image pairs are
% sufficient so that for every mixture model (associated with every pixel in
% $\mathcal{I}$), we see every mixture component enough times.
% \begin{lemma}
% \label{lem:every-cluster}
% For the pointwise segmentation model with $n$ training image pairs,
% \begin{align*}
% &\mathbb{P}
% \Big(
% \exists
% \text{ a pixel }i
% \text{ for which a mixture component of MM}(i)
% \text{ occurs}
% \le
% \frac{n\rho_{\min}}{2}
% \text{ times}
% \Big) \\
% &\le
% |\mathcal{I}|\mathcal{C}_{\max}
% \exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
% \end{lemma}
% \begin{proof}
% Fix pixel $i\in\mathcal{I}$. For training intensity images
% $Y_1, \dots, Y_n$, note that
% patches $Y_1[i], \dots,$ $Y_n[i]$ behave as if they were drawn from
% $\text{MM}(i)$, which has at most $\mathcal{C}_{\max}$ mixture components each
% with weight at least $\rho_{\min}$. Thus, by Lemma
% \ref{lem:enough-samples-from-every-category}, we have
% \begin{align*}
% &\mathbb{P}
% \Big(
% \exists
% \text{ mixture component of MM}(i)
% \text{ occurring}
% \le
% \frac{n\rho_{\min}}{2}
% \text{ times}
% \Big) \\
% &\le
% \mathcal{C}_{\max}\exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
% Union-bounding over all $|\mathcal{I}|$ pixels yields the claim.\hfill$\qed$
% \end{proof}
%
% \noindent
% Now let $\mathcal{E}_{\text{all-sources}}$ be the event that for every pixel
% $i$ and for every mixture component $(\rho_{ic},\mu_{ic},\lambda_{ic})$ in
% MM$(i)$, there exists some training intensity image whose patch at pixel~$i$
% is generated by $(\rho_{ic},\mu_{ic},\lambda_{ic})$. By the result of Lemma
% \ref{lem:every-cluster},
% $\mathbb{P}(\mathcal{E}_{\text{all-sources}})
% \ge
% 1
% -
% |\mathcal{I}|\mathcal{C}_{\max}
% \exp\big( -\frac{n\rho_{\min}}{8} \big)$.
% For the remainder of the proof, we assume that event
% $\mathcal{E}_{\text{all-sources}}$ holds.
%
% We prove the performance guarantee for pointwise nearest-neighbor segmentation
% in Section \ref{sub:ptwise-1nn-pf} and for pointwise weighted majority voting
% segmentation in Section \ref{sub:ptwise-wmv-pf}.
%
% \subsection{Pointwise Nearest-Neighbor Segmentation Guarantee}
% \label{sub:ptwise-1nn-pf}
%
%
% Fix pixel $i\in\mathcal{I}$. We want to infer $L(i) \in \{+1, -1\}$, which is
% a binary classification problem. We interchangeably use ``segment'' and
% ``classify'' to mean the same thing for predicting a binary label.
%
% Under the pointwise segmentation model, there exists some mixture component
% $c \in \{1, \dots, \mathcal{C}_{i}\}$ of $\text{MM}(i)$ that patch $Y[i]$ is
% generated from. In particular,
% \begin{equation}
% Y[i] = \mu_{ic} + W_1,
% \label{eq:nn-result-helper1}
% \end{equation}
% where noise $W_1 \in \mathbb{R}^d$ consists of i.i.d.~zero-mean sub-Gaussian
% entries with parameter $\sigma$. Moreover, the label of patch $Y[i]$ is
% $\lambda_{ic} \in \{+1, -1\}$.
%
% Using the jigsaw condition, there exists a neighbor $j^* \in N^*(i)$ such that
% $\text{MM}(j^*)$ has a mixture component that also has mean $\mu_{ic}$ and
% label $\lambda_{ic}$. Furthermore, since event
% $\mathcal{E}_{\text{all-sources}}$ holds, we know that there exists some
% training intensity image $Y_{u^*}$ for which $Y_{u^*}[j^*]$ is generated from
% a mixture component with mean $\mu_{ic}$ and label $\lambda_{ic}$. Hence,
% \begin{equation}
% Y_{u^*}[j^*] = \mu_{ic} + W_2,
% \label{eq:nn-result-helper2}
% \end{equation}
% where noise $W_2$ consists of i.i.d.~zero-mean sub-Gaussian entries with
% parameter $\sigma$. Putting together equations \eqref{eq:nn-result-helper1}
% and \eqref{eq:nn-result-helper2},
% \begin{equation}
% Y[i]
% =\mu_{ic} + W_1
% =(Y_{u^*}[j^*]-W_2)+W_1
% =Y_{u^*}[j^*] + \underbrace{(W_1 - W_2)}_{\triangleq W},
% \label{eq:nn-result-helper3}
% \end{equation}
% where $W$ consists of i.i.d.~zero-mean sub-Gaussian entries now with parameter
% $\sqrt{2}\sigma$.
%
% We next decompose the probability of misclassifying pixel $i$, i.e.,
% segmenting it to be foreground when it's background or vice versa. Let
% $\widehat{L}_{\text{NN}}(i|Y[i])$ denote the label that pointwise
% nearest-neighbor neighbor segmentation predicts for $L(i)$ given $Y[i]$. We
% have
% \begin{align}
% &\mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}})
% \nonumber \\
% &=\mathbb{P}(L(i)=+1
% \mid
% \mathcal{E}_{\text{all-sources}})
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}},
% L(i)=+1)\nonumber \\
% &\quad\;
% +
% \mathbb{P}(L(i)=-1
% \mid
% \mathcal{E}_{\text{all-sources}})
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid\mathcal{E}_{\text{all-sources}},
% L(i)=-1).
% \label{eq:nn-result-helper4}
% \end{align}
% We now upper-bound
% $\mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)$.
% (The bound for when $L(i)=-1$ is similar.) Suppose $L(i)=+1$. Then
% nearest-neighbor segmentation makes an error and declares
% $\widehat{L}_{\text{NN}}(i|Y[i])=-1$ when the closest training intensity
% patch $Y_{\widehat{u}}[\,\widehat{j}\,]$ to patch $Y[i]$ has label
% $L_{\widehat{u}}(\,\widehat{j}\,)=-1$, where
% \begin{align*}
% (\widehat{u},\widehat{j})
% &=\underset{u\in\{1,2,\dots,n\}, j\in N(i)}
% {\text{argmin}}
% \|Y_u[j] - Y[i]\|^2 \\
% &=\underset{u\in\{1,2,\dots,n\}, j\in N(i)}
% {\text{argmin}}
% \|Y_u[j] - (Y_{u^*}[j^*] + W)\|^2,
% \end{align*}
% and the last step above uses equation \eqref{eq:nn-result-helper3}.
%
% By optimality of $(\widehat{u},\widehat{j})$ for the above optimization
% problem, we have
% \[
% \|Y_u[j] - (Y_{u^*}[j^*] + W)\|^2
% \ge
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - (Y_{u^*}[j^*] + W)\|^2
% \]
% for all $u\in\{1,2,\dots,n\}$ and $j \in N(i)$. Setting $u=u^*$ and $j=j^*$
% (the latter of which we can do because $N(i)\supseteq N^*(i)$), we obtain
% \begin{align*}
% \|W\|^2
% &\ge
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - (Y_{u^*}[j^*] + W)\|^2
% = \|(Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]) - W\|^2 \\
% &= \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% - 2 \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% + \|W\|^2,
% \end{align*}
% or, equivalently,
% \begin{equation}
% 2 \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% \ge
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2.
% \label{eq:nn-result-helper5}
% \end{equation}
% Since the true label is $L(i)=+1$, this means that training intensity patch
% $Y_{u^*}[j^*]$ has the correct label $+1$, and since nearest-neighbor
% segmentation makes an error, it means that training intensity patch
% $Y_{\widehat{u}}[\,\widehat{j}\,]$ has incorrect label $-1$. Then the event
% $\widehat{L}_{\text{NN}}(i|Y[i])=-1$ happens when there exists
% $\widehat{u} \in \{1, \dots, n\}$ and $\widehat{j} \in N(i)$ with
% $L_{\widehat{u}}(\,\widehat{j}\,)=-1$ such that optimality condition
% \eqref{eq:nn-result-helper5} holds. Thus,
% \begin{align}
% &\mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])=-1
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)
% \nonumber \\
% &\overset{(i)}{\le}
% \mathbb{P}
% \Bigg(
% \bigcup_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \big\{
% 2 \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% \ge
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2.
% \big\}
% \Bigg) \nonumber \\
% &\overset{(ii)}{\le}
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \mathbb{P}
% \big(
% 2 \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% \ge
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% \big)
% \nonumber \\
% &=
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \mathbb{P}
% \big(
% \exp(
% 2 s \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% )
% \ge
% \exp(
% s \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% )
% \big)
% \nonumber \\
% &\overset{(iii)}{\le}
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \exp(
% -s \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% )
% \mathbb{E}
% \big[
% \exp(
% 2 s \langle Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*], W \rangle
% )
% \big]
% \nonumber \\
% &\overset{(iv)}{\le}
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \exp(
% -s \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% )
% \exp(
% 4 s^2 \sigma^2 \| Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*] \|^2
% )
% \nonumber \\
% &=
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \exp(
% -(s - 4 s^2 \sigma^2)
% \|Y_{\widehat{u}}[\,\widehat{j}\,] - Y_{u^*}[j^*]\|^2
% )
% \nonumber \\
% &\le
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \exp(
% -(s - 4 s^2 \sigma^2)
% \mathcal{G}
% )
% \nonumber \\
% &\overset{(v)}{\le}
% \sum_{\substack{\widehat{u} \in \{1,\dots,n\},
% \widehat{j} \in N(i) \\
% \text{s.t. }
% L_{\widehat{u}}(\,\widehat{j}\,)=-1}}
% \exp\Big( -\frac{1}{16\sigma^2} \mathcal{G} \Big)
% \nonumber \\
% &\le n |N| \exp\Big( -\frac{1}{16\sigma^2} \mathcal{G} \Big)
% \label{eq:nn-result-helper6}
% \end{align}
% where step $(i)$ uses optimality condition \eqref{eq:nn-result-helper5}, step
% $(ii)$ is a union bound, step $(iii)$ is Markov's inequality with $s>0$, step
% $(iv)$ is due to noise $W$ consisting of zero-mean sub-Gaussian entries with
% parameter $\sqrt{2}\sigma$, and step $(v)$ is by optimizing over $s$.
% Similarly,
% \begin{equation}
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])=+1
% \mid\mathcal{E}_{\text{all-sources}},
% L(i)=-1)
% \le
% n |N| \exp\Big( -\frac{1}{16\sigma^2}\mathcal{G} \Big).
% \label{eq:nn-result-helper7}
% \end{equation}
% Combining equation \eqref{eq:nn-result-helper4} with inequalities
% \eqref{eq:nn-result-helper6} and \eqref{eq:nn-result-helper7},
% \[
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid\mathcal{E}_{\text{all-sources}})
% \le
% n |N| \exp\Big( -\frac{1}{16\sigma^2}\mathcal{G} \Big).
% \]
% Therefore,
% \begin{align*}
% \mathbb{P}(\text{mislabel pixel }i)
% &=\mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)) \\
% &=\mathbb{P}(\mathcal{E}_{\text{all-sources}})
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}) \\
% &\quad
% +
% \mathbb{P}(\mathcal{E}_{\text{all-sources}}^c)
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}^c) \\
% &\le
% \mathbb{P}(\widehat{L}_{\text{NN}}(i|Y[i])\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}})
% + \mathbb{P}(\mathcal{E}_{\text{all-sources}}^c) \\
% &\le
% n |N| \exp\Big( -\frac{1}{16\sigma^2}\mathcal{G} \Big)
% +
% |\mathcal{I}|\mathcal{C}_{\max}
% \exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
% Finally, we establish the pointwise nearest-neighbor segmentation performance
% guarantee by noting that the expected pixel mislabel rate satisfies
% \begin{align*}
% &\mathbb{E}
% \Big[\frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}
% \delta(\text{mislabel pixel }i)
% \Big]
% = \frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}
% \mathbb{P}(\text{mislabel pixel }i) \\
% &\le
% n |N| \exp\Big( -\frac{1}{16\sigma^2}\mathcal{G} \Big)
% + |\mathcal{I}|\mathcal{C}_{\max}
% \exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
%
%
% \subsection{Pointwise Weighted Majority Voting Segmentation Guarantee}
% \label{sub:ptwise-wmv-pf}
%
% The proof for pointwise weighted majority voting starts the same way as for
% nearest-neighbor segmentation by writing patch $Y[i]$ using equation
% \eqref{eq:nn-result-helper3}. Let $\widehat{L}_{\text{WMV}}(i|Y[i];\theta)$
% denote the label that pointwise weighted majority voting segmentation predicts
% for $L(i)$ given $Y[i]$, using scale parameter $\theta$. We decompose the
% probability of misclassifying pixel $i$ as follows:
% \begin{align}
% &\mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}})
% \nonumber \\
% &=\mathbb{P}(L(i)=+1 \mid\mathcal{E}_{\text{all-sources}})
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1) \nonumber \\
% &\quad\;
% +
% \mathbb{P}(L(i)=-1
% \mid
% \mathcal{E}_{\text{all-sources}},
% Y[i])
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=-1).
% \label{eq:wmv-result-helper1}
% \end{align}
% We next upper-bound
% $\mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)$.
% (The bound for when $L(i)=-1$ is similar.) Suppose $L(i)=+1$. Let
% \begin{align*}
% \Gamma_{\text{WMV}}(i|Y[i];\theta)
% & \triangleq
% \frac{V_{+1}(i|Y[i];\theta)}
% {V_{-1}(i|Y[i];\theta)} \\
% &=\frac{\sum_{u=1}^n
% \sum_{j\in N(i)}
% \exp\big(
% -\theta\|Y_u[j] - Y[i]\|^2
% \big)
% \delta(L_u(j)=+1)}
% {\sum_{u=1}^n
% \sum_{j\in N(i)}
% \exp\big(
% -\theta\|Y_u[j] - Y[i]\|^2
% \big)
% \delta(L_u(j)=-1)}.
% \end{align*}
% By Markov's inequality,
% \begin{align}
% &\mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)
% \nonumber \\
% &=\mathbb{P}(\Gamma_{\text{WMV}}(i|Y[i];\theta)\le 1
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)
% \nonumber \\
% &=\mathbb{P}
% \Big(
% \frac{1}{\Gamma_{\text{WMV}}(i|Y[i];\theta)}\ge 1
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1
% \Big)
% \nonumber \\
% &\le
% \mathbb{E}
% \Big[
% \frac{1}{\Gamma_{\text{WMV}}(i|Y[i];\theta)}
% \Big|
% \mathcal{E}_{\text{all-sources}}, L(i)=+1
% \Big]
% \nonumber \\
% (\text{using equation }\eqref{eq:nn-result-helper3})
% &=\mathbb{E}
% \Big[
% \frac{1}{\Gamma_{\text{WMV}}(i|Y_{u^*}[j^*] + W;\theta)}
% \Big|
% \mathcal{E}_{\text{all-sources}}, L(i)=+1
% \Big].
% \label{eq:wmv-result-helper1b}
% \end{align}
% As a reminder, equation \eqref{eq:nn-result-helper3} relied on good event
% $\mathcal{E}_{\text{all-sources}}$ holding as well as the jigsaw condition.
%
% To proceed further, we begin by upper-bounding the term within the
% expectation, namely $1/\Gamma_{\text{WMV}}(i|Y_{u^*}[j^*] + W;\theta)$. Note
% that for any $\widetilde{u} \in \{1,\dots,n\}$ and $\widetilde{j} \in N(i)$
% such that $L_{\widetilde{u}}(\,\widetilde{j}\,)=+1$, and for any
% $r\in\mathbb{R}^d$,
% \[
% \frac{1}{\Gamma_{\text{WMV}}(i|r;\theta)}
% \le
% \frac{\sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta\|Y_u[j] - r\|^2 \big)}
% {\exp( -\theta\|Y_{\widetilde{u}}[\,\widetilde{j}\,] - r\|^2)}.
% \]
% In particular, we can choose $\widetilde{u} = u^*$ and $\widetilde{j} = j^*$,
% so
% \[
% \frac{1}{\Gamma_{\text{WMV}}(i|r;\theta)}
% \le
% \frac{\sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta\|Y_u[j] - r\|^2 \big)}
% {\exp( -\theta\|Y_{u^*}[j^*] - r\|^2)}.
% \]
% Recall that we're trying to upper-bound
% $1/\Gamma_{\text{WMV}}(i|Y_{u^*}[j^*] + W;\theta)$, corresponding to setting
% $r=Y_{u^*}[j^*] + W$. Doing this substitution, we have
% \begin{align*}
% &\frac{1}{\Gamma_{\text{WMV}}(i|Y_{u^*}[j^*] + W;\theta)} \\
% &\le
% \frac{\sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta\|Y_u[j] - (Y_{u^*}[j^*] + W)\|^2 \big)}
% {\exp( -\theta\|Y_{u^*}[j^*] - (Y_{u^*}[j^*] + W)\|^2)} \\
% &= \frac{\sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta\|(Y_u[j] - Y_{u^*}[j^*]) - W\|^2 \big)}
% {\exp( -\theta\|W\|^2)} \\
% &= \frac{\sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta
% (\|Y_u[j] - Y_{u^*}[j^*]\|^2
% - 2 \langle Y_u[j] - Y_{u^*}[j^*], W \rangle
% + \|W\|^2
% ) \big)}
% {\exp( -\theta\|W\|^2)} \\
% &= \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta
% (\|Y_u[j] - Y_{u^*}[j^*]\|^2
% - 2 \langle Y_u[j] - Y_{u^*}[j^*], W \rangle) \big) \\
% &= \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \exp\big( 2\theta \langle Y_u[j] - Y_{u^*}[j^*], W \rangle \big).
% \end{align*}
% We originally wanted to analyze is the expectation of this expression. Taking
% expectations of both sides of the above inequality,
% \begin{align}
% &\mathbb{E}
% \Big[
% \frac{1}{\Gamma_{\text{WMV}}(i|Y_{u^*}[j^*] + W;\theta)}
% \Big|
% \mathcal{E}_{\text{all-sources}}, L(i)=+1
% \Big]
% \nonumber \\
% &\le
% \mathbb{E}
% \Bigg[
% \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \exp\big( 2\theta \langle Y_u[j] - Y_{u^*}[j^*], W \rangle \big)
% \Big|
% \mathcal{E}_{\text{all-sources}}, L(i)=+1
% \Bigg]
% \nonumber \\
% &=
% \mathbb{E}_W
% \Bigg[
% \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \exp\big( 2\theta \langle Y_u[j] - Y_{u^*}[j^*], W \rangle \big)
% \Bigg]
% \nonumber \\
% &=
% \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \mathbb{E}_W
% \big[
% \exp\big( 2\theta \langle Y_u[j] - Y_{u^*}[j^*], W \rangle \big)
% \big]
% \nonumber \\
% &\overset{(i)}{\le}
% \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -\theta \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \exp\big( 4\theta^2\sigma^2 \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \nonumber \\
% &= \sum_{\substack{u \in \{1,\dots,n\}, j\in N(i) \\
% \text{s.t. }L_u(j)=-1}}
% \exp\big( -(\theta - 4\theta^2\sigma^2) \|Y_u[j] - Y_{u^*}[j^*]\|^2 \big)
% \nonumber \\
% &\le
% n |N| \exp\big( -(\theta-4\sigma^2 \theta^2)\mathcal{G} \big).
% \label{eq:wmv-result-helper2}
% \end{align}
% where step $(i)$ follows from noise $W$ consisting of zero-mean sub-Gaussian
% entries with parameter $\sqrt{2}\sigma$. Plugging this result into inequality
% \eqref{eq:wmv-result-helper1b},
% \begin{equation}
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=+1)
% \le n |N| \exp\big( -(\theta-4\sigma^2\theta^2)\mathcal{G} \big).
% \label{eq:wmv-result-helper2b}
% \end{equation}
% By a similar calculation,
% \begin{equation}
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}, L(i)=-1)
% \le n |N| \exp\big( -(\theta-4\sigma^2\theta^2)\mathcal{G} \big),
% \label{eq:wmv-result-helper3}
% \end{equation}
% and so combining \eqref{eq:wmv-result-helper1}, \eqref{eq:wmv-result-helper2b},
% and \eqref{eq:wmv-result-helper3},
% \[
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}})
% \le n |N| \exp\big( -(\theta-4\sigma^2\theta^2)\mathcal{G} \big).
% \]
% Then
% \begin{align*}
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i]\theta)\ne L(i))
% &=\mathbb{P}(\mathcal{E}_{\text{all-sources}})
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}) \\
% &\quad
% +
% \mathbb{P}(\mathcal{E}_{\text{all-sources}}^c)
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}}^c) \\
% &\le
% \mathbb{P}(\widehat{L}_{\text{WMV}}(i|Y[i];\theta)\ne L(i)
% \mid
% \mathcal{E}_{\text{all-sources}})
% + \mathbb{P}(\mathcal{E}_{\text{all-sources}}^c) \\
% &\le
% n |N| \exp\big( -(\theta-4\sigma^2\theta^2)\mathcal{G} \big)
% +
% |\mathcal{I}|\mathcal{C}_{\max}
% \exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
% Finally, the expected pixel labeling error rate satisfies
% \begin{align*}
% &\mathbb{E}\Big[
% \frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}
% \delta(\text{mislabel pixel }i)
% \Big]
% = \frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}
% \mathbb{P}(\text{mislabel pixel }i) \\
% &\le
% n |N| \exp\big( -(\theta-4\sigma^2\theta^2)\mathcal{G} \big)
% +
% |\mathcal{I}|\mathcal{C}_{\max}
% \exp\Big( -\frac{n\rho_{\min}}{8} \Big).
% \end{align*}
% Choosing $\theta = \frac{1}{8\sigma^2}$ yields the same performance guarantee
% that we derived for pointwise nearest-neighbor segmentation, finishing the
% proof.
%
%
%
% \section{Iterative Segmentation Algorithm Details}
%
% We first briefly motivate where the EPLL objective function comes from. Note
% that our full latent source model for multipoint segmentation describes, for
% every pixel $i$, the joint distribution between intensity image patch $Y[i]$
% and the full label image $L$. As with our pointwise segmentation model and
% recent work on modeling natural imagery patches \cite{epll,patch_gmm}, we do
% not specify how overlapping intensity image patches are jointly distributed.
% Since we have a joint distribution for label image $L$ and image patch $Y[i]$,
% then assuming that we know the model parameters, the MAP estimate for $L$
% given $Y[i]$ is
% \[
% \widehat{L}
% =
% \underset{L \in \{+1, -1\}^{|\mathcal{I}|}}
% {\text{argmax}}
% \bigg\{
% \log\bigg(
% \sum_{g=1}^k
% \pi_g
% \exp(-\alpha\mathbf{d}(L,\Lambda_g))
% \bigg)
% +
% \log p(Y[i]|L[i])
% \bigg\}.
% \]
% If we average the objective function above across all pixels $i$, we obtain
% the EPLL objective function. In particular, the EPLL framework proposes
% solving
% % we approximately solve
% \begin{equation*}
% \widehat{L}
% =\!\!
% \underset{L \in \{+1,-1\}^{|\mathcal{I}|}}
% {\text{argmax}}
% \bigg\{\!
% \log\bigg(
% \sum_{g=1}^k
% \pi_g
% \exp(-\alpha\mathbf{d}(L,\Lambda_g))
% \bigg)
% +
% \frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}\log p(Y[i]|L[i])\!
% \bigg\}.
% % \label{eq:EPLL}
% \end{equation*}
% % where we represent image $L$ also as a flattened vector in $|\mathcal{I}|$
% % dimensions.
% As we discussed in the main paper, since we do not know the model parameters,
% we instead solve
%
% % Then optimization \eqref{eq:EPLL-approx} can be rewritten
% % as
% % \begin{align}
% % \widehat{L}
% % % & \in\underset{L\in\{0,1\}^{|\mathcal{I}|}}{\text{argmax }}\Bigg\{\log\bigg(\sum_{g=1}^{k}\pi_{g}\exp(-\alpha\mathbf{d}(L,\lambda_{g}))\bigg)+\sum_{i\in\mathcal{I}}\log p(Y[i]|L[i])\Bigg\}\nonumber \\
% % & \in\underset{\substack{L\in\{0,1\}^{|\mathcal{I}|},(\xi_{i}\in\mathbb{R}^{m},i\in\mathcal{I})\\
% % \text{s.t. }L[i]=\xi_{i}\text{ for }i\in\mathcal{I}
% % }
% % }{\text{argmin }}\Bigg\{-\log\bigg(\sum_{g=1}^{k}\pi_{g}\exp(-\alpha\mathbf{d}(L,\lambda_{g}))\bigg)
% % \nonumber \\
% % &\qquad\qquad\qquad\qquad\qquad\;+\sum_{i\in\mathcal{I}}-\log p(Y[i]|\xi_i)+\frac{\beta}{2}\sum_{i\in\mathcal{I}}\|L[i]-\xi_{i}\|^{2}\Bigg\},\label{eq:EPLL-with-aux}
% % \end{align}
% \begin{equation}
% \widehat{L}
% % & \in\underset{L\in\{0,1\}^{|\mathcal{I}|}}{\text{argmax }}\Bigg\{\log\bigg(\sum_{g=1}^{k}\pi_{g}\exp(-\alpha\mathbf{d}(L,\lambda_{g}))\bigg)+\sum_{i\in\mathcal{I}}\log p(Y[i]|L[i])\Bigg\}\nonumber \\
% =\underset{\substack{L\in\{+1,-1\}^{|\mathcal{I}|},\\(\xi_{i}\in\mathbb{R}^{d'},i\in\mathcal{I})\\
% \text{s.t. }L[i]=\xi_{i}\text{ for }i\in\mathcal{I}
% }
% }{\text{argmin }}\Big\{-F(L;\alpha)
% % \nonumber \\
% % &
% %\qquad\qquad\qquad\qquad\qquad\;
% -
% \frac{1}{|\mathcal{I}|}
% \sum_{i\in\mathcal{I}}\log \widetilde{p}(Y[i]|\xi_i;\gamma)+\frac{\beta}{2}\sum_{i\in\mathcal{I}}\|L[i]-\xi_{i}\|^{2}\Big\}, \label{eq:EPLL-with-aux}
% \end{equation}
% where $\beta>0$ is a user-specified constant. % We numerically solve this
% % problem using the Alternating Direction Method of Multipliers
% % (ADMM) for distributed optimization \cite{boyd_admm_2011}, which iteratively
% % applies the three update rules below. Derivations for these updates are in the
% % supplementary material. We
%
% The original EPLL method \cite{epll}
% progressively increases $\beta$ and, for each choice of $\beta$, alternates
% between updating the label image $L$ and the auxiliary variables $\xi_i$,
% ignoring the constraints $L[i]=\xi_i$ for $i\in\mathcal{I}$. The idea is that
% as $\beta$ grows large, these constraints will eventually be satisfied.
% However, it is unclear how to increase $\beta$ in a principled manner. While
% heuristics could be used, an alternative approach is to fix $\beta$ and
% instead introduce a Lagrange multiplier $\eta_i$ for each constraint
% $L[i]=\xi_i$ and iteratively update these Lagrange multipliers. This can be
% achieved by the Alternating Direction Method of Multipliers (ADMM) for
% distributed optimization \cite{boyd_admm_2011}. Specifically, we form
% Lagrangian
% \begin{align*}
% \mathcal{L}_{\beta}(L,\xi,\eta)
% &=-F(L;\alpha)
% -\sum_{i\in\mathcal{I}}\log \widetilde{p}(Y[i]|\xi_{i};\gamma)
% +\frac{\beta}{2}\sum_{i\in\mathcal{I}}\|L[i]-\xi_{i}\|^{2}+\sum_{i\in\mathcal{I}}\eta_{i}^{T}(L[i]-\xi_{i}),
% \end{align*}
% where $\eta=(\eta_{i}\in\mathbb{R}^{d'}, i\in\mathcal{I})$ is the collection of
% Lagrange multipliers, and $\xi=(\xi_i, i\in\mathcal{I})$ is the collection of
% auxiliary variables. Indexing iterations with superscripts, the ADMM update
% equations are given by:
% \begin{align*}
% \xi^{t+1} & \leftarrow\underset{\xi}{\text{argmin }}\mathcal{L}_{\beta}(L^{t},\xi,\eta^{t}) & & (\text{minimize Lagrangian along direction }\xi),\\
% L^{t+1} & \leftarrow\underset{L}{\text{argmin }}\mathcal{L}_{\beta}(L,\xi^{t+1},\eta^{t}) & & (\text{minimize Lagrangian along direction }L),\\
% \eta^{t+1} & \leftarrow\eta^{t}+\beta(\xi^{t+1}-L^{t+1}) & & (\text{update Lagrange multipliers }\eta).
% \end{align*}
% By looking at what terms matter for each update equation, we can rewrite the
% above three steps as follows:
%
% \textit{1. Update label patch estimates}.
% We update estimate $\xi_i$ for label patch
% $L[i]$ given observed image patch $Y[i]$ in parallel across $i\in\mathcal{I}$:
% % $\xi_i$, which parallelizes across pixels $i\in\mathcal{I}$. Note that $\xi_i$
% % is an estimate for label patch $L[i]$ given observed intensity image patch
% % $Y[i]$:
% \begin{equation*}
% \xi_{i}^{t+1}\leftarrow\underset{\xi_{i}}{\text{argmin }}
% \Big\{
% -
% \frac{1}{|\mathcal{I}|}
% \log \widetilde{p}(Y[i]|\xi_{i};\gamma)+\frac{\beta}{2}\|L^{t}[i]-\xi_{i}\|^{2}+(\eta_{i}^{t})^{T}(L^{t}[i]-\xi_{i})\Big\}. %\label{eq:xi-i-update}
% \end{equation*}
% This minimization only considers $\xi_i$ among training label patches for
% which $\widetilde{p}$ is defined. Thus, this minimization effectively scores
% different nearest-neighbor training label patches found and chooses the one
% with the best score.
%
% \textit{2. Merge label patch estimates}.
% Fixing $\xi_i$, we update label image~$L$:
% \begin{equation*}
% L^{t+1}\leftarrow\underset{L}{\text{argmin }}\Big\{-F(L;\alpha)
% +\frac{\beta}{2}\sum_{i\in\mathcal{I}}\|L[i]-\xi_{i}^{t+1}\|^{2}+\sum_{i\in\mathcal{I}}(\eta_{i}^{t})^{T}(L[i]-\xi_{i}^{t+1})\Big\}. %,\label{eq:l-update}.
% \end{equation*}
% With the assumption that $F$ is differentiable, gradient methods could be used
% to numerically solve this subproblem.
% % For example, if $\mathbf{d}$ is the Dice overlap score, then we can
% % numerically minimize this using AdaGrad \cite{duchi_2011_adagrad}.
%
% \textit{3. Update Lagrange multipliers}.
% Set
% %\begin{equation*}
% $
% \eta_i^{t+1} \leftarrow\eta_i^{t}+\beta(\xi_i^{t+1}-L^{t+1}[i]).
% $
% This penalizes large discrepancies between $\xi_i$ and $L[i]$.
% %\end{equation*}
%
% % Numerical optimization readily parallelizes as follows. First, fix label
% % image $L$ and update the auxiliary variables $\xi_{i}$ in parallel across
% % pixels $i\in\mathcal{I}$. This corresponds to solving $|\mathcal{I}|$
% % different local problems of computing estimate $\xi_{i}$ for local label
% % $L[i]$ given observed intensity patch $Y[i]$. Then, treating the auxiliary
% % variables $\xi_{i}$ as fixed, update $L$.
%
% Parameters $\alpha$, $\beta$, and $\gamma$ are chosen using held-out data or
% cross-validation.
\end{document}
\begin{thebibliography}{10}
\bibitem{ailon_2006_ann_jl}
N~Ailon and B~Chazelle.
\newblock Approximate nearest neighbors and the fast johnson-lindenstrauss
transform.
\newblock In {\em Symposium on Theory of Computing}, 2006.
\bibitem{rueckert_patches_2013}
W~Bai et~al.
\newblock A probabilistic patch-based label fusion model for multi-atlas
segmentation with registration refinement: Application to cardiac {MR}
images.
\newblock {\em Transactions in Medical Imaging}, 2013.
\bibitem{patchmatch}
C~{Barnes et al}.
\newblock Patchmatch: a randomized correspondence algorithm for structural
image editing.
\newblock {\em Transactions on Graphics}, 2009.
\bibitem{boyd_admm_2011}
S~{Boyd et al}.
\newblock Distributed optimization and statistical learning via the alternating
direction method of multipliers.
\newblock {\em Foundations \& Trends in Machine Learning}, 2011.
\bibitem{georgehc_2013_time_series_nips}
G~H Chen, S~Nikolov, and D~Shah.
\newblock A latent source model for nonparametric time series classification.
\newblock In {\em Neural Information Processing Systems}, 2013.
\bibitem{coupe_2011}
P~{Coup{\'e} et al}.
\newblock Patch-based segmentation using expert priors: Application to
hippocampus and ventricle segmentation.
\newblock {\em NeuroImage}, 2011.
\bibitem{mrf_superresolution}
W~T Freeman and C~Liu.
\newblock Markov random fields for super-resolution and texture synthesis.
\newblock {\em Advances in Markov Random Fields for Vision and Image Proc.},
2011.
\bibitem{anatomy3}
A~Hanbury, H~M{\"u}ller, G~Langs, M~A Weber, B~H Menze, and T~Salas Fernandez.
\newblock Bringing the algorithms to the data: Cloud-based benchmarking for
medical image analysis.
\newblock In T~Catarci, P~Forner, D~Hiemstra, A~Pe{\~n}as, and G~Santucci,
editors, {\em CLEF Conference}, volume 7488 of {\em LNCS}, pages 24--29.
Springer, Heidelberg, 2012.
\bibitem{muja_flann_2009}
M~Muja and D~G Lowe.
\newblock Fast approximate nearest neighbors with automatic algorithm
configuration.
\newblock In {\em ICCVTA}, 2009.
\bibitem{rousseau_2011}
F~Rousseau, P~A Habas, and C~Studholme.
\newblock A supervised patch-based approach for human brain labeling.
\newblock {\em Transactions on Medical Imaging}, 2011.
\bibitem{label_fusion_generative_model_2010}
M~R Sabuncu et~al.
\newblock A generative model for image segmentation based on label fusion.
\newblock {\em Transactions on Medical Imaging}, 2010.
\bibitem{wachinger_2013}
C~Wachinger et~al.
\newblock On the importance of location and features for the patch-based
segmentation of parotid glands.
\newblock {\em MIDAS Journal - IGART}, 2014.
\bibitem{epll}
D~Zoran and Y~Weiss.
\newblock From learning models of natural image patches to whole image
restoration.
\newblock In {\em International Conference on Computer Vision}, 2011.
\bibitem{patch_gmm}
D~Zoran and Y~Weiss.
\newblock Natural images, gaussian mixtures and dead leaves.
\newblock In {\em Neural Information Processing Systems}, 2012.
\end{thebibliography}