Discovering Pattern Structure Using Differentiable Compositing
Extracted source text · arXiv 2010.08788
\documentclass[acmtog]{acmart}
\acmSubmissionID{304}
\usepackage{booktabs} % For formal tables
\usepackage{bbm} % For formal tables
\usepackage{mathtools}
\usepackage{xfrac}
% TOG prefers author-name bib system with square brackets
\citestyle{acmauthoryear}
%\setcitestyle{nosort,square} % nosort to allow for manual chronological ordering
\usepackage[ruled]{algorithm2e} % For algorithms
\renewcommand{\algorithmcfname}{ALGORITHM}
\SetAlFnt{\small}
\SetAlCapFnt{\small}
\SetAlCapNameFnt{\small}
\SetAlCapHSkip{0pt}
\usepackage{siunitx}
\sisetup{output-exponent-marker=\ensuremath{\mathrm{e}}}
\newcommand{\matt}[1]{\textcolor{green}{[MATT: #1]}}
\newcommand{\wil}[1]{\textcolor{brown}{[WIL: #1]}}
\newcommand{\paul}[1]{\textcolor{orange}{[PAUL: #1]}}
\newcommand{\niloy}[1]{\textcolor{violet}{[NILOY: #1]}}
\newcommand{\preddy}[1]{\textcolor{red}{[PREDDY: #1]}}
\newcommand{\changed}[1]{\textcolor{blue}{#1}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\twodots}{\mathinner {\ldotp \ldotp}}
\newcommand{\ftrad}{\ensuremath{\mathcal{F}^{\text{t}}}\xspace}
\newcommand{\fdiff}{\ensuremath{\mathcal{F}}\xspace}
\newcommand{\functTrad}{\ensuremath{f_t^{\text{t}}}\xspace}
\newcommand{\functDiff}{\ensuremath{f_t}\xspace}
\newcommand{\pixel}{\ensuremath{\mathrm{x}}\xspace}
% Metadata Information
\acmJournal{TOG}
%\acmVolume{38}
%\acmNumber{4}
%\acmArticle{39}
%\acmYear{2019}
%\acmMonth{7}
% Copyright
%\setcopyright{acmcopyright}
%\setcopyright{acmlicensed}
%\setcopyright{rightsretained}
%\setcopyright{usgov}
%\setcopyright{usgovmixed}
%\setcopyright{cagov}
%\setcopyright{cagovmixed}
% DOI
%\acmDOI{0000001.0000001_2}
% Paper history
%\received{February 2007}
%\received{March 2009}
%\received[final version]{June 2009}
%\received[accepted]{July 2009}
\setcopyright{acmcopyright}\acmJournal{TOG}
\acmYear{2020}\acmVolume{39}\acmNumber{6}\acmArticle{262}\acmMonth{12} \acmDOI{10.1145/3414685.3417830}
% Document starts
\begin{document}
% Title portion
%\title{Pattern Manipulation Using Differential Compositing}
\title{Discovering Pattern Structure Using Differentiable Compositing}
\author{Pradyumna Reddy}
\affiliation{%
\institution{University College London}
}
\author{Paul Guerrero}
\affiliation{%
\institution{Adobe Research}
}
\author{Matt Fisher}
\affiliation{%
\institution{Adobe Research}
}
\author{Wilmot Li}
\affiliation{%
\institution{Adobe Research}
}
\author{Niloy J. Mitra}
\affiliation{%
\institution{Adobe Research}
}
\affiliation{%
\institution{University College London}
}
\begin{teaserfigure}
\centering
\includegraphics[width=\linewidth]{images/teaser3.pdf}
\caption{
{Differentiable compositing.}
We present a differentiable function $\mathcal{F}$ to composite a set of discrete elements into a pattern image. This directly connects vector graphics to image-based losses (e.g., $L_2$ loss, style loss) and allows us to optimize discrete elements to minimize losses on the composited image. Minimizing an $L_2$ loss gives us a decomposition of an existing flat pattern image into a set of depth-ordered discrete elements that can be edited individually. Minimizing a style loss allows us to
make a pattern tileable or expand a pattern image into a larger pattern composed of discrete elements.}
\label{fig:teaser}
\end{teaserfigure}
\input{00_Abstract.tex}
%
% The code below should be generated by the tool at
% http://dl.acm.org/ccs.cfm
% Please copy and paste the code instead of the example below.
%
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10010147.10010257.10010258.10010260.10010270</concept_id>
<concept_desc>Computing methodologies~Motif discovery</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010371.10010382.10010383</concept_id>
<concept_desc>Computing methodologies~Image processing</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010371.10010396.10010402</concept_id>
<concept_desc>Computing methodologies~Shape analysis</concept_desc>
<concept_significance>300</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Computing methodologies~Motif discovery}
\ccsdesc[500]{Computing methodologies~Image processing}
\ccsdesc[300]{Computing methodologies~Shape analysis}
%
% End generated code
%
\keywords{patterns, unsupervised learning, pattern editing, auto correct and completion, space of patterns}
\maketitle
\input{01_Introduction.tex}
\input{02_RelatedWork.tex}
\input{03_Overview.tex}
\input{04_Method.tex}
\input{05_Evaluation.tex}
\input{06_Conclusion.tex}
%\input{07_Supplementary.tex}
\bibliographystyle{ACM-Reference-Format}
\bibliography{extra.bib}
% references.bib,
\end{document}
\begin{abstract}
Patterns, which are collections of elements arranged in regular or near-regular arrangements, are an important graphic art form and widely used due to their elegant simplicity and aesthetic appeal.
%
When a pattern is encoded as a flat image without the underlying structure, manually editing the pattern is tedious and challenging as one has to both preserve the individual element shapes and their original relative arrangements.
%
State-of-the-art deep learning frameworks that operate at the pixel level are unsuitable for manipulating such patterns. Specifically, these methods can easily disturb the shapes of the individual elements or their arrangement, and thus fail to preserve the latent structures of the input patterns.
%
We present a novel \textit{differentiable compositing} operator using pattern elements and use it to discover structures, in the form of a layered representation of graphical objects, directly from raw pattern images. This operator allows us to adapt current deep learning based image methods to effectively handle patterns.
%
We evaluate our method on a range of patterns and demonstrate superiority in the context of pattern manipulations when compared against state-of-the-art
% pixel-based
pixel- or point-based
alternatives.
% \paul{TODO: check if we compare to point-based}
\end{abstract}
\section{Introduction}
%% image/pixel-based DL and success story
Advances in deep learning, both in terms of network-based optimization as well as generative adversarial networks, have resulted in unprecedented advances in many classical image manipulation tasks. For example, deep learning is now the state-of-the-art in image denoising~\cite{pmlr-v80-lehtinen18a, Guo_2019_CVPR}, image stylization~\cite{gatys2016image, Karras_2019_CVPR, Jing2019NeuralStyleTransferReview}, image completion~\cite{Yu_2018_CVPR, Yu_2019_ICCV}, texture expansion~\cite{Zhou2018Non-StationaryFormat, Shaham_2019_ICCV} to name only a few. Generally, these methods directly operate on pixels and learn to optimize for image-space perceptual features or match kernel response statistics to produce compelling results.
%% we should explain what we mean by patterns and artworks
Complementary to photographs and paintings, a dominant form of creative content is graphic patterns or simply patterns. Such patterns are created by artists as collections of discrete numbers of elements, often structured in regular or near-regular arrangements. The aesthetics of such patterns are dictated by both the arrangement of the discrete elements as well as the shape of the individual elements.
%
Given a pattern, the user may may want to perform several manipulations: collectively edit the appearance of similar elements, replace the current set of elements with a different set, change the depth ordering of the elements, or redistribute the elements to match the style of a target pattern.
The common theme across the above applications is that they require the ability to synthesize the pattern according to some user-defined specifications while preserving the original pattern structure, i.e., the global arrangement of elements and the appearance of individual elements.
In most real-world examples, patterns are expressed as flat pixel images and such structures are not explicitly encoded.
%% problem with pixel-based approaches and patterns
Without direct access to the underlying pattern structure, limited options exist to effectively manipulate the patterns.
%
One can treat an input pattern as an image, and use state-of-the-art image manipulation methods. These methods, being oblivious of the underlying elements and their arrangements, operate directly on the pixels and can easily destroy the original element shapes and their arrangements (see Figure~\ref{fig:image-based}).
%
Alternately, one can manually select each individual element, arrange them into a layered structure, and then adjust the elements using image editing software. While such a workflow retains the shape of the respective elements, individually selecting and moving each element is both tedious and challenging especially when elements overlap, and the global arrangement of the elements must be manually preserved.
%
\if0
For example, Figure~\ref{fig:image-based} shows result of using state-of-the-art DL-based image manipulation method on patterns. A third option is to treat patterns as sets of (high-dimensional) points representing element locations and apply a point-based generative approach~\cite{xx} that directly operates on loss statistics of the point sets. As we show in Section~\ref{sec:results}, such a method, without having access to image-space statistics of the final patterns, can fail to preserve the overall aesthetic of the input.
\fi
\begin{figure}[t!]
\centering
\includegraphics{images/intro_comparison.pdf}
\caption{
{Image-based DL methods can break elements in patterns.}
Image-based pattern expansion methods like SinGAN~\cite{Shaham_2019_ICCV} or NSTS~\cite{Zhou2018Non-StationaryFormat} do not preserve element shapes. Our method is based on discrete elements and guarantees shape preservation. Here we show crops from expansions of the input pattern on the left.}
\label{fig:image-based}
\end{figure}
%% how we do it
In this paper, we develop a pattern manipulation framework that preserves \textit{both} the shape of the elements and their global arrangements. We achieve this by directly working with the individual elements, explicitly optimizing for element placements (i.e., locations, depth order, and orientations), while simultaneously assessing the quality of the arrangement of the manipulated pattern using image-domain statistics.
Specifically, we optimize explicitly over the space of element placements while assessing the quality of the resultant pattern directly in the image domain using deep image features.
Our main contribution is a novel \textit{differentiable compositor}~(DC) that
links, in a smooth and differentiable fashion, changes in the selection and placement of the the pattern elements to variations in features in the resultant pattern image.
%
For example, in Figure~\ref{fig:teaser}, starting from a flat pattern image with access to the different element types, the proposed DC can not only solve the inverse problem of disentangling the image into a layered representation of graphical objects (i.e., location, orientation, type, and depth order) along with the background image, but also be used to manipulate the patterns while keeping their original style. %
%This allows compositing discrete elements into an image to produce a desired pattern, as judged by image-space losses.
We further generalize the proposed DC to handle multiple element types and increase robustness and convergence via a multiscale framework.
%% results and comparison + list of contribution
We evaluate our method on a variety of pattern designs of different complexity. Specifically, we compare against a state-of-the-art image space method~\cite{Zhou2018Non-StationaryFormat},
a point-based synthesis method~\cite{PointSyn19},
as well as a single-image similarity maximizing GAN~\cite{Shaham_2019_ICCV}. Our result shows the advantage of operating concurrently in the pixel domain and the structured elements domain via the proposed differentiable compositor.
% \paul{check if we really use point-based baseline}
%
In summary, we introduce a novel differentiable compositing operator and use it to extend deep learning based image manipulation methods to directly operate on input patterns while preserving both the geometry of the individual elements and the image-domain similarity of the resultant patterns with respect to the original input.
%
We will make our code publicly available upon publication.
\section{Related Work}
In our experiments, we focus on two applications of our differentiable compositing method: pattern decomposition, where a pattern image is decomposed into its constituent elements that con subsequently be edited, and pattern expansion, where a pattern image is used to synthesize a layout of discrete elements on a larger canvas that mimics the style of the original pattern. In this section, we briefly describe work relevant to these applications.
% In this section, we briefly describe relevant texture synthesis methods operating on images and points.
% Note that during synthesis, image-based methods do not distinguish between individual elements, making them less useful for down-stream editing. On the other side, methods based on high-dimensional points do take into account the discrete nature of pattern elements, but have no notion of the shape or appearance of these elements, both of which may influence the element layout in real-world patterns.
\subsection{Template matching}
Given a set of distinct elements that are used in a pattern, pattern decomposition can be approached as a template matching problem, where parameters of each element instance, such as its position and orientation, are found in the pattern image. A large body of prior work exists in template matching.
%in a variety of applications, such as tracking~\cite{}, object detection~\cite{}, and character recognition~\cite{}.
Here we present a subset that is relevant to our approach. Existing approaches differ mainly in their search strategies and similarity metrics. In the absence of gradients to guide the search for matches, the main employed strategies are variants of a grid search~\cite{korman2013fast, briechle2001template, wei2008fast, cheng2010repfinder}, which can get prohibitively expensive as the number of element parameters increases, and frequency domain matching~\cite{lewis1995fast, hel2005real, mattoccia2008fast}, which can only handle a limited set of element parameters. The similarity between the template and an image patch is measured with a variety of hand-crafted metrics in earlier methods. Ouyang et al.~\shortcite{ouyang2011performance} present an overview of search strategies and early similarity metrics. PatchMatch~\cite{barnes2009patchmatch} is probably the most established of the non-learning based image matching methods that uses a random search strategy followed by iterative growing of initial matches. We include a comparison to our differentiable approach in Section~\ref{sec:results}.
More recent methods use data-driven features~\cite{dekel2015best, talmi2017template} or data-driven similarity metrics~\cite{han2015matchnet, melekhov2016image, hanif2019patch, luo2016efficient, wu2017deep, thewlis2016fully, bai2016exploiting, tang2016multi, rocco2017convolutional}
% zagoruyko2015learning, tao2016siamese, zagoruyko2015learning
that typically learn a prior from large datasets. This results in improved robustness to conditions such as illumination changes and certain deformations, but may come at the cost of reduced generalization to templates that differ significantly from the examples in the training set. A Siamese network is typically trained with pairs of matched and unmatched patches to estimate the similarity, which can further be improved by carefully selecting training pairs~\cite{simo2015discriminative} or re-weighting matches based on their uniqueness~\cite{cheng2019qatm}.
Our method is orthogonal to these approaches. For the types of images we are investigating,
%compositions of a known set of discrete elements with unknown layout and parameters,
our method can provide gradients for the matching problem. These gradients can be used with any existing (differentiable) template similarity metric to make searching in a high-dimensional space of element parameters feasible.
Inverse procedural modeling systems~\cite{talton2011metropolis, vst2010inverse} share our goal of recovering a set of parameter values for a parametric model that generate a given output, although they use parametric formulations that are not differentiable, unlike ours, and typically work in different outputs domains (3D models or vector graphics).
% \paul{TODO: template matching, instance search, particular object retrieval}
\subsection{Image-based texture synthesis}
In this section we briefly describe image-based texture synthesis methods that are relevant to our pattern expansion application. Note that during synthesis, image-based methods do not distinguish between individual elements, making them less useful for down-stream editing.
Various deep learning based strategies have been successfully developed for this problem. These methods differ in the controls that they provide and there is a trade-off between the quality and the diversity of the generated output.
%The methods, however, differ with regards to the fidelity versus diversity of the output, and also based on the supported controls.
An early line of work synthesizes texture by optimally tiling patches from an exemplar over a an image region~\cite{efros2001image, hertzmann2001image, liang2001texSynth, kwatra2003graphcutTextures, barnes2009patchmatch, rosenberger2009layered}. The results can be used to expand exemplars into larger textures, fill holes, or to stylize and manipulate images.
%
In their influential work, Gatys et al.~\shortcite{gatys2016image} proposed a style loss for the synthesis of stylized images. They observed that the statistical correlation between certain deep features can be used as a measure of texture or stylistic similarity, where the correlation can be measured by the Gram matrix of the features computed with a pre-trained VGG-19 network~\cite{Simonyan2015VERYRECOGNITION}. Using this loss, texture or style can be transferred from one image to another via direct pixel-level optimization.
%They use style loss i.e loss on gram matrix on the VGG-19 features and content loss i.e loss on VGG-19 features to iteratively optimize pixel values of a canvas to have texture of one image and content of another.
Subsequently, Ulyanov et al.~\shortcite{Ulyanov2016TextureImages} proposed a variant of this approach that does not require optimization at inference time, by training a network to generate stylized images in a single feed-forward step. % based on training a network.
%network-based synthesis that uses style loss to generate diverse set of similar looking textures.
%They use a convolution layer based architecture to synthesise a texture and use style loss to optimize their network. While their method seems to generate diverse set of texture it does not maintain the integrity of the elements in the synthesis process. The results of textures synthesized optimized solely using style loss do not have the same fidelity as GAN based methods.
Since these methods have no notion of individual pattern elements, they cannot preserve the integrity of pattern elements, when directly applied to pattern images (see Section~\ref{sec:results}).
%
%results in loss of integrity of the pattern elements
%
Zhou et al.~\shortcite{Zhou2018Non-StationaryFormat} proposed a framework to expand non-stationary textures to span larger canvases. The main idea is to combine a Gram matrix based texture similarity score, comparing random crops of the generated images to crops from the source image, and a GAN-based adversarial loss term. While the method works impressively on texture images, it also has no notion of individual elements, and can therefore not preserve the shape of the original elements. Further, the results lack any diversity in the output.
%
%
\if0
non-stationary Texture expansion using Adversarial training. They train a neural network to generate textures conditioned on an exemplar. They have three loss function in their pipeline, first a reconstruction loss on the input exemplar, second style loss between the exemplar and generated texture and third an adversarial loss on the generated texture. A weighted sum of these losses is used to optimize the network. Their training involves using patches from the input exemplar for training and then use the whole exemplar to generate the final results. While they have had decent success with expanding non-stationary Texture, but if the input exemplar contains large elements and patches during training time are unable to capture enough information, the optimization does not converge to any meaningful results. Also since the generation process is conditioned on input exemplar there is no straight forward way to generate diverse set of similar looking texture using the trained network.
\fi
%
Similar ideas~\cite{Shaham_2019_ICCV,Shocher_2019_ICCV} have been proposed to learn from a single image to generate a diverse set of similar looking images. Both of the approaches train a CNN in an adversarial setup to progressively generate images. While these techniques are able to synthesis diverse set of images, the generated images lose global structure, and can destroy characteristic arrangements in patterns. Also the generated images do not maintain the integrity of the elements in the training image.
Differentiable renderers~\cite{liu2018paparazzi,liu2019soft,loubet2019reparameterizing} have been used to find deformations of 3D shapes that produce output images with a given style. These methods also rely on a differentiable image generation approach. However, in our work, we do not aim at moving triangles with small offsets, but rather at moving large elements with complex shapes and textures over relatively long distances to reach their target positions, while also optimizing for the number and types of elements.
\subsection{Point-based pattern synthesis}~
Several methods represent pattern elements as points in a high-dimensional parameter space, like the space of positions and orientations. Unlike the image-based representation, this representation takes into account the discrete nature of pattern elements, but loses the notion of element shape and appearance, both of which may influence the element layout in real-world patterns.
% To take into account the discrete nature of pattern elements, we can instead synthesize high-dimensional points representing the parameters of each pattern element, such as its position and orientation. However, we lose the notion of shape and appearance of these elements, both of which may influence the element layout in real-world patterns.
%Several parametric and non-parametric point synthesis methods have been proposed over the years.
Some methods had success in synthesising textures with discrete elements~\cite{Landes2013ASynthesis, Hurtut2009Appearance-guidedExample, ma2011discrete, roveri2015example}, but are handcrafted for specific types of patterns and do not generalize well to other types of patterns, or require manually arranging the discrete elements into an exemplar texture.
More general methods~\cite{Zhou2012PointSpectrum, Oztireli2012AnalysisCorrelation, Heck2013BlueAliasing, Roveri2017GeneralCorrelations, Illian2008StatisticalPatterns} produce point patterns by matching spectral statistics, but cannot capture the local structure of elements due to the use of statistics. Patterns with discrete elements can also be manipulated efficiently by considering the shared geometric relationships between elements~\cite{GuerreroEtAl:PATEX:2016}, but require these relationships to be given, and cannot synthesize patterns from scratch.
More recently, deep learning methods for point cloud generation have been proposed~\cite{Li2018PointGAN, Sun2018PointGrow:Self-Attention, Achlioptas2017LearningClouds}, but these methods treat point clouds as 3D surfaces so they are not particularly suitable for point pattern synthesis out of the box. Other recent methods~\cite{leimkuehler2019DeepPointCorr, Muller2018NeuralSampling} deal with points from a sampling perspective, so the distribution of the points is taken into consideration. Recently, Tu et al.~\shortcite{tu2019point} present a method for expanding point patterns using a style loss. They represent points in the form of an image and use this exemplar as reference to optimize pixel values on larger canvas using style loss, histogram loss and correlation loss. While their results are compelling they do not account for the shape or appearance of the element. Also extending their method to deal with elements that are not interchangeable is not straight forward.
\begin{figure*}[t!]
\centering
\includegraphics[width=\linewidth]{images/compositing.pdf}
\caption{
{Overview of the image compositing process. }
Given a set of elements, each describing an instance of a small image patch at a position $c_i$, orientation $\theta_i$, and depth $z_i$, we first place each element in a separate image layer with the function $f_t$ and then combine the layers into the final image according to the element depth with function $f_c$, where layers with higher depth occlude layers with lower depth. We describe the traditional non-differentiable formulation of $f_t$ and $f_c$ in Section~\ref{sec:compositing_trad}, and our new differentiable formulation in Section~\ref{sec:compositing_diff}.}
\label{fig:pipeline}
\end{figure*}
\section{Overview}
\label{sec:overview}
The prevalent approach to create illustrations and patterns is to \emph{composite} an image from a set of discrete elements. Specifically, a set of discrete elements, with each element having its own properties (i.e., type and its placement), are arranged on a canvas and combined into a single image. %We can find examples of this approach in popular software like Illustrator, Photoshop, and Powerpoint.
Our goal is to create a \emph{differentiable} compositing function for such discrete elements. Enabling the differentiability of the function, with respect to the element properties, allows us to leverage image-space loss on the composited image to directly drive the optimization of the discrete elements, both the choice of elements as well as their placements. This directly connects vectorized patterns to a range of losses available for images, such as L2 image loss or perceptual style loss, and in turn, enables vector graphics to benefit from advances in deep image processing.
In this work, we will focus on pattern images that are composed from a small set of atomic image patches that are instanced multiple times and arranged, possibly as overlapping layers, over a background canvas to form a pattern. We call the instances of these patches pattern \emph{elements}. Each element is defined by a set of parameters, such as its position, orientation, and depth.
A compositing function that takes these elements and outputs a composited image is illustrated in Figure~\ref{fig:pipeline} and can be described in two main steps: first, a \emph{layer} is created for each element that has the same size as the output image and contains the image patch corresponding to the element, at the position and orientation given by the element parameters; second, the layers are combined according to the depth given by the element parameters, where elements in layers with higher depth occlude layers with lower depth.
In Section~\ref{sec:compositing_trad}, we present the traditional formulation $\ftrad$ of this compositing function, as it is often used in software such as Photoshop, Illustrator, and Powerpoint. Due to discrete quantities such as the number of elements and their visibility, this formulation is not differentiable. In Section~\ref{sec:compositing_diff}, we show, as our core contribution, that this formulation can be generalized as a differentiable image composition function $\fdiff$. In Section~\ref{sec:optimization}, we describe our approach to optimize image compositions w.r.t. the element parameters using our differentiable formulation (see Table~\ref{tab:notations} for notations).
%We start by formulating a traditional image compositing function $\ftrad$ such that it can then be generalized, in our core contribution, as a differentiable image composition function $\fdiff$ (see Table~\ref{tab:notations} for notations).
% \section{Our Approach}
% \label{sec:approach}
%In this work, we will focus on pattern images that are composed from a small set of atomic image patches that are instanced multiple times and arranged, possibly as overlapping layers, over a background canvas to form a pattern. We start by formulating a traditional image compositing function $\ftrad$ such that it can then be generalized, in our core contribution, as a differentiable image composition function $\fdiff$ (see Table~\ref{tab:notations} for notations).
%We will present applications such as pattern decomposition or expansion in Section~\ref{sec:results}.
%To use any kind of Image space operation we must first composite the point set features to form an image. Such a composition would take a set of elements and their respective features and outputs a composited image $\mathcal{I}_c$.
\begin{table}[b!]
\caption{Notation table. }
\label{tab:notations}
\begin{tabular}{r|p{0.6\columnwidth}}
symbol & description \\ \hline \hline
$I$ & RGB image \\
$\mathbf{x}$ & image location \\
$\mathcal{E}\coloneqq\{E_0 \twodots E_n\}$ & set of elements \\
$\fdiff$ & (differentiable) compositing function \\
$\mathcal{H}\coloneqq\{H_0 \twodots H_m\}$ & library of image patches \\
$E_i \coloneqq (\tau_i, c_i, \theta_i, z_i)$ & element properties \\
$\tau_i$ & type of element $i$ \\
$c_i$ & center location of element $i$ \\
$\theta_i$ & orientation of element $i$ \\
$z_i$ & layer depth of element $i$ \\
$f_t$ & element transformation function \\
$f_c$ & element compositing function \\
$\mathcal{J}\coloneqq\{J_0 \twodots J_n\}$ & layers \\
$T_i$ & element transformation matrix \\
$\kappa$ & interpolation kernel \\
$v_i(x)$ & visibility function of layer $i$ \\
$t_i = (t^0_i \twodots t^m_i)$ & type probability vector of element $i$
\end{tabular}
\end{table}
% \begin{figure}
% \centering
% %\includegraphics{}
% \vspace{1in}
% \caption{inverse compositing results
% show 2 examples: one simple (1 element + rotation + medium complexity background); one complex (2 elements + layers + 1 w/ rotation)
% in both cases, show some intermediate results and the final reconstruction. \paul{Do we need this figure? Seems like some early results that are similar to what we show in the teaser.}}
% \label{fig:inverseCompositeResults}
% \end{figure}
\section{Traditional Image Compositing}
\label{sec:compositing_trad}
In a forward process, we create an RGB image $I^{\text{t}}$ by compositing a given set of $n$ discrete elements $\mathcal{E}^{\text{t}} \coloneqq \{E_0^{\text{t}} \twodots E_n^{\text{t}}\}$
%on a given background image $B$
with a function $\ftrad$ to produce image value at 2D image coordinate $\pixel$ as,
\begin{equation}
I^{\text{t}}(\pixel) \coloneqq \ftrad(\pixel, \mathcal{E}^{\text{t}}).
\end{equation}
Each element $E_i^{\text{t}}$ is defined as a translated and rotated instance of an $RGBM$ image patch $H_j$ selected from a small library of image patches $\mathcal{H} = \{H_1 \twodots H_m\}$, where $M$ is a binary coverage mask that defines the region of the image patch covered by the element.
An image patch represents the appearance of an element and is typically much smaller than the image $I$. Each element consists of a property tuple $E_i^{\text{t}} \coloneqq (\tau_i, c_i, \theta_i, z_i)$, where $\tau_i \in \{1 \twodots m\}$ is an index into $\mathcal{H}$ that denotes the element type, $c_i \in \mathbb{R}^2$ denotes the coordinates of the element center, $\theta_i \in[0,2\pi)$ denotes the orientation of the element, and $z_i \in \mathbb{Z}^+$ defines the occlusion order of the elements. In regions of overlap, elements with higher $z$ occlude elements with lower $z$. In the case of patterns, we typically have a small set of element types (e.g., $m \le 5$) each of which appears multiple times in the pattern image (i.e., $m \ll n$).
%
We treat the background $H_0$ as a special element with a fixed set of parameters $E^t_0 \coloneqq (\tau_0=0, c_0=(0,0), \theta_0=0, z_0=0)$.
%In many applications, we do not know the number of elements $n$ in advance, and the existence probability $p_i$.
%We denote the set of all properties as $\mathcal{P} = \{P_1, P_2 \twodots, P_n\}$.
%
%\wil{I find it a bit confusing to refer to the binary mask as an alpha channel, since alpha is not binary most of the time. Maybe just refer to it as a mask and use $M$ for notation?}
Given this collection of ingredients, we create an image in two steps. First, the function $f_t^\text{t}$ transforms the image patch of each element according to its property vector, and then re-samples the transformed image patch at the pixels of the image $I$ to obtain one image layer $J_i^{\text{t}}$ per element. Second, the function $f_c^\text{t}$ combines the occlusion-ordered individual layers into a single image:
\begin{equation}
I^{\text{t}}(\pixel) \coloneqq \ftrad(\pixel, \mathcal{E}^{\text{t}}) = f_c^\text{t} \Big( \big\{f_t^{\text{t}}(\pixel, E_i^{\text{t}})\big\}_{i=0:n} \Big).
\end{equation}
\paragraph{Transforming elements} In the first step, we rasterize each element on its separate image layer, resulting in a set of layers $\mathcal{J}^t = \{J_0^{\text{t}} \twodots J_n^{\text{t}}\}$ as $J_i^{\text{t}}(\pixel) = f_t^\text{t}(\pixel, E_i^{\text{t}})$.
%
We initialize the $RGBM$ channels of each layer to zero and then place the image patch of the corresponding element by translating it to its center position $c_i$ and orienting it according to $\theta_i$. For layer $i$, we transform the image coordinates \pixel into the local coordinate frame of the element using the element's inverse transform, and then sample the image patch of the element with the local coordinates as:
\begin{equation}
\functTrad(\pixel, E_i^{\text{t}}) \coloneqq H_{\tau_i} \big(\lfloor R_{\theta_i}^{-1} (\pixel-c_i) \rceil\big),
\label{eqn:trad_rounding}
\end{equation}
where $R_\theta$ denotes a $2 \times 2$ matrix encoding rotation by angle $\theta$ and $\lfloor \rceil$ indicates rounding to the nearest pixel location.
%\niloy{we have to set M to 1} \paul{the mask channel $M$ is sampled in the same way as the other channels in Eq.~\ref{eqn:trad_rounding}}
%
%Traditionally, there are various choices for the interpolation kernel $\kappa$. Popular options include nearest neighbors or bilinear interpolation kernel.
\paragraph{Combining occlusion-ordered element layers}
In the second step, we combine the layers $\mathcal{J}$ to get the composited image $I^{\text{t}}(\pixel) = f_c^\text{t}(\{J_i^{\text{t}}(\pixel)\}_{i=0:n})$ by stacking layers on top of each other according to the occlusion order of their elements, where layers with a higher $z$-value occlude layers with with lower $z$.
%Below all the element layers, we additionally define a background layer $B(\pixel)$.
This gives the following definition for the composited image:
\begin{equation}
I^{\text{t}}(\pixel) = \sum_{i=0}^n J_i^{\text{t}}(\pixel) v_i(\pixel)% + B(\pixel)\prod_{i=1}^n (1-v_i(\pixel)).
\end{equation}
%
%
where $v^{\text{t}}_i(\pixel) \in \{0,1\}$ is the visibility of layer $i$ at each image location $\pixel$.
Only the layer with non-zero mask $M$ and highest $z$-value is visible at each image location, i.e.,
% \begin{equation}
% v^{\text{naive}}_i(\pixel) = \mathbbm{1}_{\argmax_j z_j J^a_j(\pixel)}(i)
% \end{equation}
\begin{equation}
v^{\text{t}}_i(\pixel) =
\left\{\begin{array}{ll}
1 & \text{if } i \in \argmax_j z_j M^{\text{t}}_j(\pixel)\\
0 & \text{otherwise},
\end{array}\right.
\label{eqn:trad_visibility}
\end{equation}
where $M^{\text{t}}_j$ is the coverage mask of the layer $J^t_j$.
%\wil{In this section, we often use the word ``traditionally'' (e.g., ``in the traditional version of visibility''). It's not really clear what this refers to or why it's relevant at this point in the text. I think it would be clearer to omit it and just state our definition of the compositing function. Once we go to the next section, it's clear that our main contribution is to define a differentiable version.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Differentiable Image Compositing}
\label{sec:compositing_diff}
Given a composited image, the inverse problem is to recover a set of elements, both their count and individual properties.
%, along with a background image.
Solving this problem requires differentiating the compositing function with respect to the (unknown) element properties. However, doing so with the traditional compositing function, introduced above, is difficult due to several factors:
(i)~the nearest neighbor rounding, see Equation~\ref{eqn:trad_rounding},
%\wil{two Equations have the same label, which makes this reference incorrect.}
prevents taking gradients with respect to element location and/or orientation; (ii)~the discrete visibility $v^\text{t}_i$, see Equation~\ref{eqn:trad_visibility}, prevents taking gradients with respect to the element occlusion order; (iii)~the count $n$ of elements and (iv)~their types $\tau_i$
%and (iv)~their visibility $v^\text{t}_i(\pixel)$ at any location $\pixel$
are discrete variables preventing us from taking their derivatives.
%\preddy{we should add ordering of the elements to the list of factors because that also a component we are handling}
We introduce a differentiable image compositing function $\fdiff$ that addresses these challenges by generalizing the traditional compositing function $\ftrad$. In particular, we replace the nearest neighbor rounding with a differentiable interpolation kernel for sampling image patches, and convert the discrete variables in $\ftrad$ into soft, continuous variables in $\fdiff$, and use a multi-resolution pyramid to smooth the composite spatially. While the basic image formation process remains similar, these generalizations require some important adjustments to the compositing formulation.
%that control the number of elements, their types, and paragraphtheir visibility
%
%\wil{I think we should add a high-level sentence here before diving into the details below. Broadly, I think we convert the discrete variables into soft, continuous variables that control the number of elements, their types, and their visibility; and we use continuous interpolation rather than rounding to sample image patches. Then we can mention the specifics in the sentence below (or just leave out these details since they are explained in the subsequent paragraphs).}
%
%Specifically, we (i)~transform individual elements using a bilinear interpolation kernel instead of nearest neighbor rounding; introduce (ii)~additional variable $p_i$ for each element to capture its existence probability, (iii)~likelihood variables $t_i^j$ for elements' type, and (iv)~a soft notion of element visibility $v_i(x)$ to make the operator differentiable with respect to existence of elements and their properties. We now detail each step.\preddy{we should say there is soft ordering to keep the consistency of the paragraph}
Similar to before, we create an RGB image $I$ by compositing a set of $n$ discrete elements $\mathcal{E} \coloneqq \{E_0 \twodots E_n\}$ with a differentiable function $\fdiff$ as,
\begin{equation}
I(\pixel) \coloneqq \fdiff(\pixel, \mathcal{E}),
\end{equation}
where \pixel are 2D image coordinates. We also assume availability of a library of $RGBM$ image patches $\mathcal{H} \coloneqq \{H_1 \twodots, H_m\}$. Each element is given a tuple of variables $E_i^{\text{t}} \coloneqq (t_i, c_i, \theta_i, z_i)$, where
$t_i = (t^1_i \twodots, t^m_i)$ softens the element type $\tau_i$ into a vector of logits $t^j_i$ of the probability that element $E_i$ is of type $j$,
%P_{\tau_i}(j)$ denotes the likelihood of the element being of type $j$,
$c_i \in \mathbb{R}^2$ denotes the coordinates of the element center, $\theta_i \in \mathbb{R}$ %\in[0,2\pi)$\preddy{$\theta_i$ belongs to R during optimization}
denotes the orientation of the element in radians, and $z_i \in \mathbb{R}$ defines the occlusion order of the elements, where elements with higher $z$ occlude elements with lower $z$. The background $H_0$ is represented with the element $E_0 \coloneqq (c_0=(0,0), \theta_0=0, z_0)$ that does not have type logits $t_0$ and is placed at a fixed depth $z_0$. Unlike in traditional compositing, the depth $z_i$ of any element can be smaller than the background depth $z_0$, making it invisible in the final composite, due to being fully occluded by the background. This provides a simple mechanism to make the number of elements differentiable without introducing additional parameters. We set $n$ to a large number (between $100$ and $1024$ in our experiments), and remove any elements in the final composite that end up hidden below the background (i.e., $z_i < z_0$).
%Note that $n$ now denotes
%where elements with low $p_i$ are less likely to exist.
Similar to the earlier formulation, we define the composited image in two steps. First, the function $f_t$ transforms the image patches of each element according to the element properties, and then re-samples the transformed image patch at the pixels of the image $I$ to obtain one image layer $J_i$ per element. Second, the function $f_c$ combines the layers according to the occlusion order into a single image:
\begin{equation}
I(\pixel) := \fdiff (\pixel, \mathcal{E}) = f_c \Big( \big\{ \functDiff(\pixel, E_i)\big\}_{i=0:n} \Big).
\end{equation}
Our goal is to define a function $\fdiff$ that is differentiable with respect to the properties $E_i$ of all elements. Later, we show how the differentiable function allows computing the gradients $\frac{\partial L(I)}{\partial E_i}$ of any differentiable image loss $L(I)$ with respect to the properties $E_i$.
\paragraph{Transforming elements} In the first step, we place each element on a separate image layer, resulting in a set of layers $\mathcal{J} := \{J_1 \twodots J_n\}$ as $J_i(\pixel) := \functDiff(\pixel, E_i)$.
%
We set ${J}_0$ with depth $z_0=\gamma$ with a (unknown) constant color $J_0(\pixel) = b$ and alpha = 1.
We initialize the $RGBM$ channels of each layer to zero and then place the image patch of the corresponding element by translating it to its center position $c_i$ and orienting it according to $\theta_i$.
For layer $i$, we transform the image coordinates \pixel into the local coordinate frame of the element using the element's inverse transform, and then sample the image patch of the element with the local coordinates as:
\begin{equation}
h_{\tau_i} \big( R_{\theta_i}^{-1} (\pixel-c_i) \big),
\label{eqn:rounding}
\end{equation}
where $R_\theta$ denotes a $2 \times 2$ matrix encoding rotation by angle $\theta$ and $h_j$ is an interpolation of image patch $H_j$ to give a continuous function:
\begin{equation}
h_j(\pixel) = \sum_k (H_j)_k\ \kappa(\pixel - \pixel_k),
\end{equation}
where $(H_j)_k$ is pixel $k$ of the image patch $H_j$, $\pixel_k$ are the coordinates of the same pixel, and $\kappa$ is a differentiable interpolation kernel.
We create a differentiable function $\functDiff(\pixel, E_i)$ using the expected value over type probabilities:
\begin{equation}
\label{eq:diff_type}
J_i(\pixel) := \functDiff(\pixel, E_i) = \frac{1}{\sum_{k=1}^m e^{t^k_i}} \sum_{j=1}^m e^{t^j_i}\ h_j \big( R_{\theta_i}^{-1} (\pixel-c_i) \big),
\end{equation}
where the softmax $e^{t^j_i} /\ \sum_{k=1}^m e^{t^k_i}$
%\niloy{index should run over j not k} \paul{This index $k$ is used to avoid confusion with the index $j$ in the numerator. Its not needed in the main equation above because there the sum in the denominator is outside the context where the other $j$ is defined. But we can also use $k$ in the main Equation if you think that is clearer.}
over type logits define the type probabilities.
In this formulation, the gradients with respect to the element type probabilities $t^j_i$ are well-defined. Note that the expected value is over all four $RGBM$ channels of the interpolated image patch $h_j$, and thus the coverage mask $M_i$ can take on non-binary values,
%. Values in $M$ are low for regions covered only by low-probability element types, and high for regions covered by high-probability element types,
effectively giving us a map of the coverage probability in layer $J_i$. The background layer is computed as $J_0(\pixel) := h_0 \big( R_{\theta_0}^{-1} (\pixel-c_0) \big)$.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/diff_position.pdf}
% \vspace{1in}
\caption{
{Differentiable element position.}
Nearest-neighbor sampling of an element results in an image $I$ that is not differentiable w.r.t. the element position $c_i$. Using bilinear sampling improves differentiability, but gradients at $I(\pixel)$ only have a small region of influence. Using an image pyramid further improves differetiability and increases the region of influence.}
\label{fig:diff_position}
\end{figure}
% \begin{equation}
% \frac{\partial f_t}{\partial t^j_i}(\pixel) = \sum_k (H_j)_k \kappa(R_{\theta_i}^{-1} (\pixel-c_i) - \pixel_k)
% \end{equation}
%
Empirically, we found that the choice of interpolation kernel $\kappa$ is important to get good gradients with respect to the location and orientation of the elements. We choose a bilinear interpolation kernel (cf., \cite{jaderberg2015spatial}):
\begin{equation}
\kappa(\pixel) = \frac{1}{r_x r_y} \max(0, r_x-|\pixel_x|) \max(0, r_y-|\pixel_y|),
\end{equation}
where $r_x$ and $r_y$ is the sample spacing along the $x$ and $y$ axes of the image patch $H_i$, which makes $\functDiff$ piecewise differentiable with respect to the position and orientation of the elements, see Figure~\ref{fig:diff_position} for an illustration.
\paragraph{Combining element layers}
In the second step, we combine the layers $\mathcal{J}$ to get the composited image $I(\pixel) = f_c(\{J_i(\pixel)\}_{i=0:n})$. Layers are stacked on top of each other according to the occlusion order of their elements, where layers with a higher $z$-value occlude layers with lower $z$. The final composited image is defined as:
%In real applications, we do not know the number of elements in advance. To handle this, we place a large constant number of elements $n$, and use the existence logits $p_i$ for each element leading to the compositing operator,
\begin{equation}
I(\pixel) := f_c(\{J_i(\pixel)\}_i) = \sum_{i=0}^n J_i(\pixel) v_i(\pixel),
\end{equation}
%where $\sigma$ is a Sigmoid function and $\sigma(p_i)$ is the existence probability of element $i$. Optimizing for the existence probabilities makes the number of elements $|\{E_i\ |\ \sigma(p_i) > 0.5\}|$ differentiable.
where the visibility $v_i(\pixel)$ determines which layer is visible at each image location $\pixel$. We make this visibility differentiable with respect to the last remaining element property: the z-value $z_i$.
We use a differentiable version of the visibility using a softmax function as:
\begin{equation}
v_i(\pixel) := {e^{z_i} M_i(\pixel) } / {\sum_{k=0}^n{e^{z_k} M_k(\pixel) }}.
\end{equation}
In this softer version of the visibility, all elements with non-zero alpha have some contribution to the image, even if they are occluded, but their contribution decreases exponentially with distance from the top-most element, see Figure~\ref{fig:diff_visibility} for an illustration. Recall from Eq.\ref{eq:diff_type} that $M_i$ is the coverage probability in layer $J_i$. By multiplying with $M_i$, regions of the layer that are likely to be occupied are placed near the depth $z_i$ of the element, while regions that are less likely to be occupied are placed at lower depth, making them less likely to occlude other elements.
%
Due to our continuous formulation, variables that are discrete in the traditional formulation, like the probabilities for element types, can take on non-integer values. This results in artifacts like ghosted semi-transparent elements. In Section~\ref{sec:optimization}, we describe how we project the continuous variables to integer values.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/diff_visibility.pdf}
% \vspace{1in}
\caption{
{Differentiable visibility.} The traditional visibility function is not differentiable due to $C^0$ discontinuities at disocclusions. In our differentiable formulation, each layer contributes to the final image with a weight that decreases exponentially with distance from the top-most layer.}
\label{fig:diff_visibility}
\end{figure}
%\wil{We should mention somewhere that our differentiable compositing function will often produce images with some artifacts due to the continuous variables we've introduced (mainly ghosted, semi-transparent elements). To address this, we perform a few simple post-processing steps to snap continuous values to the appropriate discrete values. Not sure exactly where to describe this post-processing, but even if we don't do it here, it would be worth mentioning that such post-processing is often required to get good final composites.}
\paragraph{Multi-resolution pyramid}
The bilinear interpolation kernel makes the composited image $I$ differentiable w.r.t. the element positions $c_i$ and orientations $\theta_i$. However, the gradients $\frac{\partial I}{\partial c_i}$ and $\frac{\partial I}{\partial \theta_i}$ have a small \emph{region of influence}: they are non-zero only at image locations $x$ that are close to the coverage mask of the element, at a distance smaller than the bilinear kernel's radius $r$, see Figure~\ref{fig:diff_position} for an illustration. This adversely affects convergence if the initial pose of an element is not already very close to its target region, since gradients of the target region w.r.t the element parameters are zero. Additionally, high spatial frequencies in the image $I$ result in unstable gradients. To improve convergence, we compute the gradients at each level of a multi-resolution pyramid:
\begin{align}
I_1 &= I, \text{and} \nonumber \\
I_{k+1} &= I_{k} \circledast G(0,\sigma_k),
\end{align}
where $G(0,\sigma_k)$ is a Gaussian kernel with standard deviation $\sigma_k=r 2^{k-1}$, and $r$ is the radius of the bilinear kernel. To improve performance, we successively down-sample the image to half its resolution in each iteration. At coarser levels, the gradients $\frac{\partial I_k}{\partial c_i}$ and $\frac{\partial I_k}{\partial \theta_i}$ are both more stable and have a larger region of influence, at the cost of spatial accuracy.
%
%$\{I_i\}_{i=1:k}$ of the image $I$ and compute the gradients of each pyramid level. The first pyramid level $I_1$ is the original image $I$, and successive levels are iteratively coarsened using a Gaussian filter $I_{i+1} = I_{i} \circledast G(0,\sigma_i)$, followed by a down-sampling step. The standard deviation $\sigma_i$ is set to the pixel spacing of $I_{i}$
%
In our experiments, we use gradients from four pyramid levels ($i=1\twodots4$).
\paragraph{Differentiable element colors}
It is straight-forward to add other element properties to our differentiable formulation. As an example, we can add an element color $o_i \in \mathbb{R}^3$ by simply re-defining the element layers in Eq.~\ref{eq:diff_type} as
$J'_i(x) = J_i(x) \cdot \nu(o_i)$, where $\nu$ is a leaky hard sigmoid, and $\cdot$ denotes a multiplication of each RGB channel in $J_i$ with the corresponding scalar color value in $\nu(o_i)$. We chose a leaky hard sigmoid $\nu(o_i)=\max(\min(o_i,\ 0.001 o_i + 0.99), 0.001 o_i)$ instead of the regular sigmoid to make it easer for the optimization to reach fully saturated colors. We will show several composited images with optimized color parameters $o_i$ in our results.
\section{Optimizing Image Compositions}
\label{sec:optimization}
Given our differentiable formulation of image compositing, we can optimize the element parameters to minimize any loss function $L(I)$ defined on the composited image $I$. In our experiments, we use the Adam~\cite{kingma2014adam} optimizer, with a learning rate of $\num{1e-6}$ and parameters $(\beta_1, \beta_2) = (0.9, 0.9)$. Since each parameter has a different value range, we adjust the learning rates for each type of parameter. Empirically, we multiply the learning rate for parameters $(t_i, c_i, \theta_i, z_i, o_i)$ with the factors $(1, 0.01, 2.25, 0.0016, 0.0025)$ in all our experiments.
%
A straight-forward optimization with a random initialization is difficult due to several factors: (i) as described earlier, gradients have a limited region of influence, which limits the basin of attraction for the optimal position and orientation of an element, and (ii) multiple elements may be attracted to the same optimum in position and orientation, potentially giving us duplicate elements.
To overcome these problems, we carefully initialize the element parameters, and periodically seed additional elements and remove elements that are no longer visible or identified as duplicates.
\paragraph{Initialization}
We start by choosing an estimated upper bound for the number of elements we are going to need, between $100$ and $1024$ elements in our experiments. The centers $c_i$ of these elements are initialized in a regular grid over the image canvas, with orientation $\theta_i = 0$. The element types logits are initialized to $t^j_i = 1$ for all types $j$ and the depth of the background and the other elements is initialized empirically to $z_0=3.3$ and $z_i = 9$, respectively. The grid initialization for element positions makes it less likely to miss the basin of attraction of a position optimum. A grid initialization for both positions and orientations would result in too many elements, so we handle missing orientations when seeding additional elements during the optimization.
\paragraph{Removing elements and seeding additional elements}
We alternate between removing and re-seeding elements periodically during the optimization. Elements are removed at iterations $4$k, $12$k, and every $8$k iterations thereafter, and re-seeded three times at iterations $8$k, $16$k, and $24$k.
%
We remove elements that are no longer visible because they are below the background $z_i < z_0$, and duplicate elements.
Specifically, we identify pairs of duplicates elements as having the same type $\tau_i = \argmax(t_i)$ and a distance $\|c_i - c_j\|_2 < 0.5 \min\big(l(M_i), l(M_j)\big)$ where $l(M)$ is the largest extent of an element's mask along the $x$ and $y$ axis. We remove the element with smaller depth in such a duplicate pair.
%
In the re-seeding steps, we sample the space of element orientations with a regular grid. For each existing element, we place three additional copies with the same parameters, but with orientation offsets of $90$, $180$, and $270$ degrees. Note that, after subsequent optimization, wrongly oriented elements end up below the $H_0$ layer, and hence are later removed. This allows us to escape local minima in the space of element orientations, which are especially pronounced if elements are close to rotationally symmetric.
%
Additionally, we place a new grid of elements, with the same parameters as in the initialization. These additional elements allow convergence to position optima that were missed with the previous set of elements.
\paragraph{Discretization}
After optimization, we perform a final \emph{discretization} step that removes any residual transparencies or ghosting introduced by our continuous parameters. We generate a discretized image by running our optimized element parameters through the traditional compositing pipeline, using discrete element type probabilities $\tau_i = \argmax(t_i)$.
%Since optima that are already occupied by existing elements do not attract new elements,
% \begin{itemize}
% \item multi-resolution pyramid
% \item adding and 'deleting' elements (and why this is especially relevant when working with multiple elements)
% \item anything else?
% \end{itemize}
\section{Results and Discussion}
\label{sec:results}
Pattern images are ubiquitous on the web, but are difficult to edit for several reasons. The constituent elements of a pattern are difficult to extract manually from the image, due to their number and the frequency of occlusions. Additionally, editing individual elements may compromise the style of a pattern, and restoring the style might require (manually) modifying all pattern elements.
%
Below we evaluate our differentiable compositing function on two applications that are targeted at simplifying the editing workflow for pattern images: in Section~\ref{sec:results_decomp}, we experiment with decomposing patterns images into a set of discrete elements that are editable individually, and in Section~\ref{sec:results_expansion}, we experiment with pattern expansion, where, given a pattern image, a new pattern with discrete elements and the same style as the input image is synthesized. Each of the two applications corresponds to a minimization of a different loss function. Additionally, we show the versatility of our differentiable compositing function with a few additional applications in Section~\ref{sec:results_additional}.
\subsection{Pattern Decomposition}
\label{sec:results_decomp}
In this application, our aim is to decompose a given pattern image $A$ into a set of elements $\mathcal{E}$. The image composited from these elements $I = \mathcal{F}(\mathcal{E})$ should reconstruct the input image $A$ as closely as possible. We use the $L_2$ distance to measure the decomposition error:
%We optimize the element parameters to minimize an $L2$ loss between the input image and the original image:
\begin{equation}
L_d(A, I) := \frac{1}{P} \sum_{p=1}^P \|A(x_p)-I(x_p)\|^2_2,
\end{equation}
where the sum is over pixels and $P$ is the number of pixels in the image. The optimal elements $\mathcal{E}^*$ are found by minimizing this loss:
\begin{equation}
\mathcal{E}^* := \argmin_{\mathcal{E}} L_d(A, \mathcal{F}(\mathcal{E})).
\end{equation}
Note that the compositing function $\mathcal{F}$ also depends implicitly on the set of image patches $\mathcal{H}$, corresponding to the different types of elements in the pattern. These are are selected by the user from the input image in the simple pattern decomposition workflow described below.
\begin{figure}[b!]
\centering
\includegraphics[width=\linewidth]{images/workflow.pdf}
\caption{
{Pattern decomposition workflow.} Starting from a pattern image, the user selects one example of each element type. Here, we only select a single element since we optimize over element colors. The input image and the element types form the input to our differentiable compositor, resulting in discrete elements that we can edit directly or use in down-stream tasks.}
\label{fig:workflow}
\end{figure}
\paragraph{Workflow}
A typical workflow is illustrated in Figure~\ref{fig:workflow}. The user starts by selecting one example of each element type in the pattern image. This can usually be done efficiently using existing methods~\cite{cheng2010repfinder,rother2004grabcut} or image editing software such as Photoshop or GIMP and is significantly easier than selecting all instances of each element type in the pattern image. Together with the input image, these form the input of our optimization, which finds the set of discrete elements in the pattern. The discrete elements can now be edited by the user. We show a few examples of possible edits in Figure~\ref{fig:teaser}. See the accompanying video for additional edits.
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{images/results_decomposition.pdf}
\caption{
{Inverse Pattern decompositions.} Given the flat pattern images and the elements in the top row, we optimize for elements that reconstruct the pattern image using our differentiable compositing function. The result of the continuous optimization is shown in the second row, and the discretized result (inferred layering information is not shown here) in the third row. In the last row, we show the $L_2$ error between the discretized result and the input image for colors in $[0,1]^3$. The majority of the errors are evident near the boundaries due to small position inaccuracies of the elements.}
\label{fig:results_decomposition}
\end{figure*}
\paragraph{Results}
Pattern decomposition results are shown in Figure~\ref{fig:results_decomposition}. We apply the pattern decomposition workflow to pattern images downloaded from the web and two manually created pattern images (patterns \#4 and \#7). We only optimize over necessary parameters in each example. These include all parameters in examples \#1, \#3 and \#5, and all parameters except the orientation in the other examples.
The pattern image and the element types extracted by the user are given in the first row, and in rows 2 and 3, we show both the output of our optimization before the discretization step described in Section~\ref{sec:optimization} (\emph{continous result}), and after the discretization (\emph{discretized result}).
%
Please note the slight transparency or ghosting in some elements of the continuous result. This is due to the soft approximations of the visibility and element type we use in our differentiable composting function. Running our optimized element parameters through the non-differentiable compositor removes these artifacts and gives us the discretized results.
%
A map of the $L_2$ error between the discretized result and the input image is given in the last row. Note that our composited image closely resembles the original, with errors concentrated at boundaries, due to slight inaccuracies in the optimized element parameters. Apart from these inaccuracies, errors occur only very sparsely; for example, orientation errors in cases where objects are close to rotationally symmetric (column 5), or errors in the depth or type of an element if the shapes and colors of different element types are similar. We believe that most of these errors could be avoided with a larger number of optimization steps, or an improved optimization setup, such as a learning rate schedule.
Overall, we can see that our optimization can successfully find the type, position, orientation, and layering of elements that form a given pattern image, even in the presence of severe occlusions between elements.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_decomposition_colors.pdf}
\caption{
{Additional inverse pattern decompositions}. In these decompositions, we also optimize for the color (scaling) of the elements.}
\label{fig:results_decomposition_additional}
\end{figure}
In Figure~\ref{fig:results_decomposition_additional}, we show additional decomposition results where we also optimize over the color $o_i$ of each element. All pattern images were downloaded from the web. The first pattern shows a simple element in large number of color variations that would be infeasible to handle without optimizing for colors, since we would need one element type per color. In the second pattern image, we show an example of elements with some amount of texture. Since we multiply the layer of each element with its color parameter, we can effectively also handle some amount of texture when optimizing for colors.
Finally, in the third column, we optimize for color in a pattern with complex layering and occlusions.
%
As we can see in the last row, the performance of our optimization with color is on par with the optimization without color, showing that we can successfully optimize for element color as well.
\paragraph{Baseline comparison}
We compare our method to two baselines that do not benefit from our gradients. A brute-force grid search over element parameters and PatchMatch~\cite{barnes2009patchmatch}. We describe each baseline shortly before presenting comparison results.
%To compare with a baseline that does not benefit from our gradients, we solve the same optimization problem described at the beginning of this section with
As a first baseline, a brute force grid search over element parameters is solved greedily, one element at a time. We create a grid over all element parameters that are relevant for a given input image, except for the depth, which is handled with a greedy approach. We set the resolution of this grid to $(m, 128^2, 36, 3^3)$ for the parameters $E'_k = (\tau_k, c_k, \theta_k, o_k)$, where $m$ is the number of element types.
%
As a distance measure between an element and the input image, we define an $L_2$ distance that is normalized by the occupancy masks $M$ of both the element and the input image:
\begin{equation}
L_m(A, J_k') = \frac{\sum_{p=1}^P M_A(x_p) M_{J_k'}(x_p)\ \|J_k'(x_p) - A(x_p)\|_2^2}{\sum_{p=1}^P M_A(x_p) M_{J_k'}(x_p)},
\end{equation}
where $A$ is the input image, $J_k'(\pixel) = f_t^\text{t}(\pixel, E_k')$ is a layer containing the element transformed to grid location $k$ (see Equation~\ref{eqn:trad_rounding} for details), and $E_k'$ are the element parameters at a grid location $k$. This distance effectively ignores background pixels that are not occupied by the element. The image starts with full occupancy, but we successively remove matched regions from its occupancy mask, which are then ignored in subsequent matches. We first compute $L_m$ at each grid location and greedily pick the best match. Then, we remove the matched region from the image by subtracting the occupancy mask of the matched element from the occupancy mask of the image and update $L_m$ at all affected grid locations. This procedure is iterated until $L_m$ falls below a threshold. We \textit{manually} select the best threshold for each image.
%
In practice, we compute the sum only over the non-zero pixels of the element's occupancy mask and ignore matches where the number of non-zero pixels in the product of the occupancy masks is smaller than a threshold. In our experiments, we set this threshold to $10\%$ of the total number of non-zero pixels in the element's occupancy mask.
Figure~\ref{fig:results_decomposition_comparison} compares our results to this baseline. As we can expect, in patterns without element rotations (first example), the baseline performs comparable to our approach. In the second example, we also need to optimize over element orientations. However, as we add more parameters, the dimension of the grid increases, making it infeasible to sample the new dimensions densely. Due to inaccuracies in the orientations, we see several errors in the result, including both false positive and false negative matches. This problem worsens as we increase the dimension of the parameter space. In the third example, we optimize over element colors. This adds three dimensions to the parameter space that can only be sampled sparsely to maintain a feasible number of grid points. As a result, we see an increased number of false positives and negatives.
The gradients obtained from our differentiable compositing function, on the other hand, allow for a much sparser sampling of the parameter space, and enable a non-greedy approach where all elements are optimized concurrently.
As second baseline, we compare to the well-known PatchMatch. We match regions in the pattern image $I$ to the image patches in $\mathcal{H}$. More specifically, we run PatchMatch once per image patch $H \in \mathcal{H}$, giving us a set of candidate regions in $I$ that each partially match one of the image patches in $\mathcal{H}$. PatchMatch finds these matching regions starting with a random search strategy, by picking random initial correspondences for each pixel in $I$ to the pixels in $H$. Pixel correspondences are scored based on an L2 distance between small rectangular neigborhoods centered at the corresponding pixels. Good matches are grown into regions. From the resulting set of candidate regions, we filter out regions that have an average matching score below a threshold. For the remaining regions, we obtain the element type $\tau_i$ from image patch $H$ the region is matched to, and the element location $c_i$ from the known correspondences between pixels in the region and pixels in $H$. Since PatchMatch has no notion of layering, we use the matching score of the regions as depth order (highest score is top-most).
Figure~\ref{fig:results_decomposition_comparison_patchmatch} compares our results to PatchMatch. We use patterns without element rotations or color changes, since these would be non-trivial to implement in PatchMatch. PatchMatch works reasonably well on a pattern with simple elements that do not overlap, but fails on more complex patterns with overlapping elements.
% PatchMatch
% % describe patchmatch here
% We compare random patches from element to patches in the patterns in a random manner. We propagate the match in the neighbourhood like proposed in \cite{barnes2009patchmatch} such a pipeline would return an affinity matrix between pixels in the pattern image to the element pixels. We threshold the affinity matrix to 99 percentile to create and binary image. We perform distance transformation on the binary image and find the peaks of the transformation. These peaks are used to as the centers of the elements of the elements in the reconstruction. We remove the duplicates based on proximity like mentioned in section 6. The layer ordering is decided based on the affinity matrix. We observed that the error in the location detected is due to the patch based nature of the algorithm.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_decomposition_comparison.pdf}
% \vspace{2in}
\caption{
{Comparison to a template matching baseline.} On the left, the results of the baseline are comparable to our approach. When adding more element parameters like orientations in the center, or colors on the right, a dense grid search becomes infeasible, resulting in several false positive and false negative matches. The gradients provided by our compositing function allow optimizing the element positions without requiring a dense grid.}
\label{fig:results_decomposition_comparison}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_decomposition_comparison_patchmatch.pdf}
% \vspace{2in}
\caption{
{Comparison to PatchMatch.} PatchMatch~\cite{barnes2009patchmatch} uses a random initialization of image-to-patch correspondences, followed by an iterative propagation of good matches. The random search strategy makes finding exact element locations hard, and occlusions further degrade results. Our decomposition uses gradients as guidance, instead of a random search, and explicitly models occlusions, resulting in a more stable decomposition.}
\label{fig:results_decomposition_comparison_patchmatch}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_expansion.pdf}
\caption{{Pattern expansion.} We expand the pattern image in the top row to twice its size (four times its area) using our differentiable compositor and compare to two state-of-the-art image-based expansion methods, SinGAN~\cite{Shaham_2019_ICCV} and NSTS~\cite{Zhou2018Non-StationaryFormat}. By construction, our results better preserve the integrity of the elements, resulting in a more appealing expansion. Also note that, unlike the image-based methods, the individual elements in our expansion results can be selected and edited.}
\label{fig:results_expansion}
\end{figure}
% Some errors are evident near the boundaries due to small position inaccuracies of the elements, but see very few errors apart from these inaccuracies.
\subsection{Pattern Expansion}
\label{sec:results_expansion}
In our second application, our goal is to synthesize a new discrete element pattern that has the same style as a pattern given in an input image, but is generated on a canvas of different size, usually a larger canvas. We measure the difference in style between the composited image $I=\mathcal{F}(\mathcal{E})$ and the input image using a style loss that was originally introduced by Gatys et al.~\shortcite{gatys2016image}:
\begin{equation}
L_s(A, I) := \sum_{l} \frac{w_l}{n_l m_l} \|G^l_A - G^l_I\|_F^2,
\end{equation}
where $G^l_A \in \mathbb{R}^{n_l \times m_l}$ is the Gram matrix of the features in layer $l$ of a pre-trained VGG network~\cite{Simonyan2015VERYRECOGNITION} that was applied to image $A$ and $w_l$ are per-layer weights. See the original work of Gatys et al. for details. We use layers $2$, $5$, $10$, and $15$ of a pre-trained VGG-19 network with all weights $w_l = 0.2$. Note that the Gram matrices represent feature statistics that remain comparable across different image sizes.
%
The optimal elements $\mathcal{E}^*$ are the minimum of this loss:
\begin{equation}
\mathcal{E}^* := \argmin_{\mathcal{E}} L_s(A, \mathcal{F}(\mathcal{E})).
\end{equation}
\paragraph{Workflow}
Similar to pattern decomposition, the user first selects one example of each element type from the input image. Then, a canvas size can be chosen that determines the size of the expanded pattern. After optimization, the user obtains a pattern in the chosen canvas that has a style similar to the input image and is composed of discrete elements.
\paragraph{Baselines}
To show the advantages of generating a pattern with discrete elements, we compare against two state-of-the-art image-based pattern synthesis methods.
SinGAN~\cite{Shaham_2019_ICCV} trains a GAN on a single image. Their model learns to generate images with similar patch statistics as the input image. This is enabled by a loss similar to the style loss we are using, but replaces the Gram matrix with a manually defined feature statistic with a learned discriminator.
As a second baseline, we use the Non-stationary Texture Synthesis (NSTS) method by Zhou et al.~\shortcite{Zhou2018Non-StationaryFormat}. This method also trains a generator with an adversarial loss on a single input image and additionally uses the style loss from Gatys et al.~\shortcite{gatys2016image}.
While the style losses are slightly more advanced in these baselines, we focus on comparing our use of discrete elements to the baselines' approach of directly generating an image. An interesting direction for future work lies in using our differentiable function $\mathcal{F}$ as a differentiable component in a network like SinGAN or NSTS to output a set of discrete elements instead of an image.
Additionally, we compare to a recent Point Pattern Synthesis (PPS) method by Tu et al.~\shortcite{tu2019point}. This method aims at expanding patterns of 2d points in a plane to a larger canvas. The points are rendered as small Gaussian hats to an image, where an image-based style loss is optimized to get an expanded image. The expanded image is then converted back to points. The authors use a histogram- and correlation-based loss in addition to a style loss similar to $L_s$. The baseline supports multiple point types by coloring the rendered Gaussians, but does not support orientations or layering.
%
We use the centers of our elements as points, color them by element type using well-separated colors, use a random occlusion order, and random orientations if the pattern has elements with multiple orientations.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_expansion_pps.pdf}
\caption{{Comparison to point-based pattern synthesis.} We compare a point-based pattern synthesis method to our approach. Note how the baseline preserves the layout of the input pattern less accurately than our method. Unlike the point-based method, our method can take into account properties of the pattern that depend on the shape of the elements, such as amounts of overlap or gap sizes between elements.}
\label{fig:results_expansion_pps}
\end{figure}
\paragraph{Results}
Figure~\ref{fig:results_expansion} compares our expansion results to the two image-based baselines. We show a simple example with a single element type and no overlaps to clearly illustrate typical errors of the image-based methods, and two more complex examples with multiple elements and overlaps. In all examples, we can see that the image-based methods cannot preserve the integrity of elements, since they do not have a notion of discrete elements. Separate elements are merged together and the shapes of individual elements are not maintained. Even though the style loss we are using for our experiments is less advanced than the baselines, we still obtain more appealing results, due to our use of discrete elements.
We compare with an expansion result of the PPS baseline in Figure~\ref{fig:results_expansion_pps}. We can see that the method struggles to synthesize a pattern with a similar layout as the input pattern. This baseline is designed for tightly packed patterns of points and performance drops on our larger, more spread out elements. In our patterns, knowledge about the shape of the elements is necessary to obtain cues about important properties of the pattern like amounts of overlap and typical gap sizes between neighboring elements. Additionally, this baseline does not support layering.
%
The output of our method, on the other hand, does take into account the shape of elements, giving the style loss more to work with, and allowing us to synthesize a stylistically correct layering.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/results_mosaic.pdf}
\caption{{Generating mosaics.} By decomposing a non-pattern image and adding a loss that penalizes overlaps between elements, we can create mosaics. Here we create a mosaic of the Siggraph Asia logo.}
\label{fig:results_mosaic}
\end{figure}
\subsection{Additional Applications}
\label{sec:results_additional}
Our differentiable compositing function can be used as a component in a large range of applications. Here we show a few additional examples.
\paragraph{Making patterns tileable.}
By combining the $L_2$ loss and the style loss, we can make an existing pattern image tileable, as shown in Figure~\ref{fig:teaser}. We first decompose the pattern as described in Section~\ref{sec:results_decomp}. This gives us discrete elements that can be edited manually. To make the pattern tileable horizontally, we can, for example, delete the elements that are clipped on the right border, select the elements that are clipped on left border and copy them over to the right border (i.e., translate the copies in x direction by the width of the image). A similar approach can be used to make the pattern tileable vertically. This manual operation however, might compromise the style of the pattern; it could, for example, create incorrect overlaps between elements, create holes, or introduce incorrect spacing. To restore the style of the pattern, we optimize the elements in the edited pattern to have the same style as the original pattern image using the loss $L_s$. We initialize the elements to their current positions and keep the elements near the border that ensure editability fixed during the optimization. This results in a tileable pattern with the same style as the input pattern.
\paragraph{Generating mosaics.}
While our compositing function can only obtain an exact reconstruction of images that are composed of a few element types, there is nothing stopping us from minimizing the loss $L_d$ on arbitrary images. This creates a mosaic that reconstructs the input image with the given set of element types. Since mosaics typically have non-overlapping elements, we introduce an auxiliary loss that penalizes overlaps: $L_o =\frac{1}{P} \sum_{p=1}^P \max\big(0, -1 + \sum_k M_k(x_p)\big)$, where we sum the occupancy masks $M_k$ of all layers, and any values larger than 1, indicating overlaps, are penalized. The outer sum takes the average over all pixels in the occupancy masks.
%
An example is shown in Figure~\ref{fig:results_mosaic}, where create a mosaic of the Siggraph Asia logo with small Siggraph Asia logos.
\subsection{Limitations}
One limitation of our method is that the output domain of our compositing function is restricted to patterns that are composed from multiple instances of a small set of atomic image patches. This includes many types of patterns found on the web, but also excludes some types, like patterns with continuously changing shapes, for example a pattern containing of continuous deformations between a circle and a rectangle. %A famous example is the artwork \emph{Sky and Water} by M.C. Escher~\cite{escher1938}.% Our decomposition result with two elements (one fish and one bird) obtains a reasonable reconstruction, but fails to match the more extreme element variations, as shown in Figure~\ref{fig:failure_cases}, first column.
A second limitation is that users have to mark one instance of each element type in advance. This is still far easier than marking all elements, but requires some manual work. Also optimizing over the image patches $\mathcal{H}$ would remove the need for this manual input and is an interesting avenue for future work.
Finally, our optimization is not always guaranteed to find the global optimum. A few typical failure modes of our optimizations are shown in Figure~\ref{fig:failure_cases}, columns 1 and 2. Faint elements, or elements very only few pixels are visible give small gradient magnitudes resulting in missed elements (column 1). Elements that close to rotationally symmetric (column 2) cause strong local minima in the space of orientations, resulting in elements with incorrect orientations.
\begin{figure}[h!]
\centering
% \includegraphics[width=\linewidth]{images/failure_cases.pdf}
\includegraphics[width=\linewidth]{images/failure_cases_v2.pdf}
\caption{
{Failure Cases.} %Patterns with continuously varying element shapes (left) cannot be represented accurately with our compositor.
Optimization results can be deteriorated by faint elements (column 1) or elements that are close to rotationally symmetric (column 2).}
\label{fig:failure_cases}
\end{figure}
\section{Conclusion and Future Work}
We presented a differentiable composting function for discrete element patterns.
%
We have shown that the gradients provided by this function can benefit pattern editing workflows in several applications.
%
First, our differentiable compositor can be used to decompose a pattern image into its discrete elements. Unlike existing template matching approaches, our gradients allow us to handle a higher-dimensional parameter space for elements, such as element orientations and colors.
%
Second, we can use our compositor to expand an existing pattern on a larger canvas by minimizing a style loss. Unlike current image-based approaches, our focus on discrete elements guarantees the integrity of elements in the expanded pattern and leads to more appealing expansion results.
%
In addition to these two main applications, we demonstrate the versatility of our method on two more specialized applications.
In future work we would like to incorporate our differentiable element compositor as a component in generative modeling approaches. For example, using our compositor in the generator of a GAN would give us the benefits in quality that a GAN provides, while our compositor would guarantee the integrity of individual elements, and make the resulting pattern editable by producing individual elements and their layering as output.
There are also several avenues to improve the differentiable function itself.
%
First, it would be interesting to solve for the image patches $\mathcal{H}$ in addition to element properties $\mathcal{E}$. This would remove the need to manually specify the element types and open up a wider range of new applications.
%
Second, defining a depth value per pixel of an element's occupancy mask instead of per element would allow handling non-planar element surfaces and enable local-layering among elements (cf., \cite{locallayering:09}).
%
Finally, since our gradients allow us to handle high-dimensional element parameters, the range of patterns our approach can handle could be extended by adding additional element parameters, such as scale or transparency.
%The latter would, however, require generalizing the visibility formulation to implement alpha compositing.
%future work: adding more representation power to handle more complex backgrounds% => included in learning \mathcal{H}
\begin{acks}
%\section{Acknowledgments}
We would like to thank the anonymous reviewers for their helpful suggestions and Maks Ovsjanikov for discussions in an early phase of this project.
This research was supported by an ERC Grant (SmartGeometry 335373), Google Faculty Award, and gifts from Adobe.
\end{acks}
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{64}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[\protect\citeauthoryear{Achlioptas, Diamanti, Mitliagkas, and
Guibas}{Achlioptas et~al\mbox{.}}{2018}]%
{Achlioptas2017LearningClouds}
\bibfield{author}{\bibinfo{person}{Panos Achlioptas}, \bibinfo{person}{Olga
Diamanti}, \bibinfo{person}{Ioannis Mitliagkas}, {and}
\bibinfo{person}{Leonidas Guibas}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Learning Representations and Generative Models for
3{D} Point Clouds}. In \bibinfo{booktitle}{\emph{Proceedings of the 35th
International Conference on Machine Learning}}, Vol.~\bibinfo{volume}{80}.
\bibinfo{pages}{40--49}.
\newblock
\bibitem[\protect\citeauthoryear{Bai, Luo, Kundu, and Urtasun}{Bai
et~al\mbox{.}}{2016}]%
{bai2016exploiting}
\bibfield{author}{\bibinfo{person}{Min Bai}, \bibinfo{person}{Wenjie Luo},
\bibinfo{person}{Kaustav Kundu}, {and} \bibinfo{person}{Raquel Urtasun}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Exploiting semantic information and deep matching
for optical flow}. In \bibinfo{booktitle}{\emph{European Conference on
Computer Vision}}. Springer, \bibinfo{pages}{154--170}.
\newblock
\bibitem[\protect\citeauthoryear{Barnes, Shechtman, Finkelstein, and
Goldman}{Barnes et~al\mbox{.}}{2009}]%
{barnes2009patchmatch}
\bibfield{author}{\bibinfo{person}{Connelly Barnes}, \bibinfo{person}{Eli
Shechtman}, \bibinfo{person}{Adam Finkelstein}, {and} \bibinfo{person}{Dan~B
Goldman}.} \bibinfo{year}{2009}\natexlab{}.
\newblock \showarticletitle{PatchMatch: A randomized correspondence algorithm
for structural image editing}. In \bibinfo{booktitle}{\emph{ACM Transactions
on Graphics (ToG)}}, Vol.~\bibinfo{volume}{28}. ACM, \bibinfo{pages}{24}.
\newblock
\bibitem[\protect\citeauthoryear{Briechle and Hanebeck}{Briechle and
Hanebeck}{2001}]%
{briechle2001template}
\bibfield{author}{\bibinfo{person}{Kai Briechle} {and} \bibinfo{person}{Uwe~D
Hanebeck}.} \bibinfo{year}{2001}\natexlab{}.
\newblock \showarticletitle{Template matching using fast normalized cross
correlation}. In \bibinfo{booktitle}{\emph{Optical Pattern Recognition XII}},
Vol.~\bibinfo{volume}{4387}. International Society for Optics and Photonics,
\bibinfo{pages}{95--102}.
\newblock
\bibitem[\protect\citeauthoryear{Cheng, Wu, AbdAlmageed, and Natarajan}{Cheng
et~al\mbox{.}}{2019}]%
{cheng2019qatm}
\bibfield{author}{\bibinfo{person}{Jiaxin Cheng}, \bibinfo{person}{Yue Wu},
\bibinfo{person}{Wael AbdAlmageed}, {and} \bibinfo{person}{Premkumar
Natarajan}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Qatm: quality-aware template matching for deep
learning}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition}}. \bibinfo{pages}{11553--11562}.
\newblock
\bibitem[\protect\citeauthoryear{Cheng, Zhang, Mitra, Huang, and Hu}{Cheng
et~al\mbox{.}}{2010}]%
{cheng2010repfinder}
\bibfield{author}{\bibinfo{person}{Ming-Ming Cheng}, \bibinfo{person}{Fang-Lue
Zhang}, \bibinfo{person}{Niloy~J Mitra}, \bibinfo{person}{Xiaolei Huang},
{and} \bibinfo{person}{Shi-Min Hu}.} \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{Repfinder: finding approximately repeated scene
elements for image editing}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics (TOG)}}
\bibinfo{volume}{29}, \bibinfo{number}{4} (\bibinfo{year}{2010}),
\bibinfo{pages}{1--8}.
\newblock
\bibitem[\protect\citeauthoryear{Dekel, Oron, Rubinstein, Avidan, and
Freeman}{Dekel et~al\mbox{.}}{2015}]%
{dekel2015best}
\bibfield{author}{\bibinfo{person}{Tali Dekel}, \bibinfo{person}{Shaul Oron},
\bibinfo{person}{Michael Rubinstein}, \bibinfo{person}{Shai Avidan}, {and}
\bibinfo{person}{William~T Freeman}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Best-buddies similarity for robust template
matching}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition}}. \bibinfo{pages}{2021--2029}.
\newblock
\bibitem[\protect\citeauthoryear{Efros and Freeman}{Efros and Freeman}{2001}]%
{efros2001image}
\bibfield{author}{\bibinfo{person}{Alexei~A Efros} {and}
\bibinfo{person}{William~T Freeman}.} \bibinfo{year}{2001}\natexlab{}.
\newblock \showarticletitle{Image quilting for texture synthesis and transfer}.
In \bibinfo{booktitle}{\emph{Proceedings of the 28th annual conference on
Computer graphics and interactive techniques}}. \bibinfo{pages}{341--346}.
\newblock
\bibitem[\protect\citeauthoryear{Gatys, Ecker, and Bethge}{Gatys
et~al\mbox{.}}{2016}]%
{gatys2016image}
\bibfield{author}{\bibinfo{person}{Leon~A Gatys}, \bibinfo{person}{Alexander~S
Ecker}, {and} \bibinfo{person}{Matthias Bethge}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Image style transfer using convolutional neural
networks}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE conference on
computer vision and pattern recognition}}. \bibinfo{pages}{2414--2423}.
\newblock
\bibitem[\protect\citeauthoryear{Guerrero, Bernstein, Li, and Mitra}{Guerrero
et~al\mbox{.}}{2016}]%
{GuerreroEtAl:PATEX:2016}
\bibfield{author}{\bibinfo{person}{Paul Guerrero}, \bibinfo{person}{Gilbert
Bernstein}, \bibinfo{person}{Wilmot Li}, {and} \bibinfo{person}{Niloy~J.
Mitra}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{{PATEX}: Exploring Pattern Variations}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph.}} \bibinfo{volume}{35},
\bibinfo{number}{4} (\bibinfo{year}{2016}), \bibinfo{pages}{48:1--48:13}.
\newblock
\showISSN{0730-0301}
\urldef\tempurl%
\url{https://doi.org/10.1145/2897824.2925950}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Guo, Yan, Zhang, Zuo, and Zhang}{Guo
et~al\mbox{.}}{2019}]%
{Guo_2019_CVPR}
\bibfield{author}{\bibinfo{person}{Shi Guo}, \bibinfo{person}{Zifei Yan},
\bibinfo{person}{Kai Zhang}, \bibinfo{person}{Wangmeng Zuo}, {and}
\bibinfo{person}{Lei Zhang}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Toward Convolutional Blind Denoising of Real
Photographs}. In \bibinfo{booktitle}{\emph{The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR)}}.
\newblock
\bibitem[\protect\citeauthoryear{Han, Leung, Jia, Sukthankar, and Berg}{Han
et~al\mbox{.}}{2015}]%
{han2015matchnet}
\bibfield{author}{\bibinfo{person}{Xufeng Han}, \bibinfo{person}{Thomas Leung},
\bibinfo{person}{Yangqing Jia}, \bibinfo{person}{Rahul Sukthankar}, {and}
\bibinfo{person}{Alexander~C Berg}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Matchnet: Unifying feature and metric learning for
patch-based matching}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition}}.
\bibinfo{pages}{3279--3286}.
\newblock
\bibitem[\protect\citeauthoryear{Hanif}{Hanif}{2019}]%
{hanif2019patch}
\bibfield{author}{\bibinfo{person}{Muhammad~Shehzad Hanif}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Patch match networks: Improved two-channel and
Siamese networks for image patch matching}.
\newblock \bibinfo{journal}{\emph{Pattern Recognition Letters}}
\bibinfo{volume}{120} (\bibinfo{year}{2019}), \bibinfo{pages}{54--61}.
\newblock
\bibitem[\protect\citeauthoryear{Heck, Schlomer, and Deussen}{Heck
et~al\mbox{.}}{2013}]%
{Heck2013BlueAliasing}
\bibfield{author}{\bibinfo{person}{Daniel Heck}, \bibinfo{person}{Thomas
Schlomer}, {and} \bibinfo{person}{Oliver Deussen}.}
\bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{{Blue noise sampling with controlled aliasing}}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics}}
\bibinfo{volume}{32}, \bibinfo{number}{3} (\bibinfo{date}{6}
\bibinfo{year}{2013}).
\newblock
\showISSN{07300301}
\urldef\tempurl%
\url{https://doi.org/10.1145/2487228.2487233}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Hel-Or and Hel-Or}{Hel-Or and Hel-Or}{2005}]%
{hel2005real}
\bibfield{author}{\bibinfo{person}{Yacov Hel-Or} {and} \bibinfo{person}{Hagit
Hel-Or}.} \bibinfo{year}{2005}\natexlab{}.
\newblock \showarticletitle{Real-time pattern matching using projection
kernels}.
\newblock \bibinfo{journal}{\emph{IEEE transactions on pattern analysis and
machine intelligence}} \bibinfo{volume}{27}, \bibinfo{number}{9}
(\bibinfo{year}{2005}), \bibinfo{pages}{1430--1445}.
\newblock
\bibitem[\protect\citeauthoryear{Hertzmann, Jacobs, Oliver, Curless, and
Salesin}{Hertzmann et~al\mbox{.}}{2001}]%
{hertzmann2001image}
\bibfield{author}{\bibinfo{person}{Aaron Hertzmann}, \bibinfo{person}{Charles~E
Jacobs}, \bibinfo{person}{Nuria Oliver}, \bibinfo{person}{Brian Curless},
{and} \bibinfo{person}{David~H Salesin}.} \bibinfo{year}{2001}\natexlab{}.
\newblock \showarticletitle{Image analogies}. In
\bibinfo{booktitle}{\emph{Proceedings of the 28th annual conference on
Computer graphics and interactive techniques}}. \bibinfo{pages}{327--340}.
\newblock
\bibitem[\protect\citeauthoryear{Hurtut, Landes, Thollot, Gousseau, Drouillhet,
and Coeurjolly}{Hurtut et~al\mbox{.}}{2009}]%
{Hurtut2009Appearance-guidedExample}
\bibfield{author}{\bibinfo{person}{T. Hurtut}, \bibinfo{person}{P.~E. Landes},
\bibinfo{person}{J. Thollot}, \bibinfo{person}{Y. Gousseau},
\bibinfo{person}{R. Drouillhet}, {and} \bibinfo{person}{J.~F. Coeurjolly}.}
\bibinfo{year}{2009}\natexlab{}.
\newblock \showarticletitle{{Appearance-guided synthesis of element
arrangements by example}}. In \bibinfo{booktitle}{\emph{NPAR Symposium on
Non-Photorealistic Animation and Rendering}}. \bibinfo{pages}{51--60}.
\newblock
\showISBNx{9781605586045}
\urldef\tempurl%
\url{https://doi.org/10.1145/1572614.1572623}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Illian}{Illian}{2008}]%
{Illian2008StatisticalPatterns}
\bibfield{author}{\bibinfo{person}{Janine. Illian}.}
\bibinfo{year}{2008}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{{Statistical analysis and modelling of
spatial point patterns}}}.
\newblock \bibinfo{publisher}{John Wiley}. 534 pages.
\newblock
\showISBNx{9780470014912}
\bibitem[\protect\citeauthoryear{Jaderberg, Simonyan, Zisserman,
et~al\mbox{.}}{Jaderberg et~al\mbox{.}}{2015}]%
{jaderberg2015spatial}
\bibfield{author}{\bibinfo{person}{Max Jaderberg}, \bibinfo{person}{Karen
Simonyan}, \bibinfo{person}{Andrew Zisserman}, {et~al\mbox{.}}}
\bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Spatial transformer networks}. In
\bibinfo{booktitle}{\emph{Advances in neural information processing
systems}}. \bibinfo{pages}{2017--2025}.
\newblock
\bibitem[\protect\citeauthoryear{{Jing}, {Yang}, {Feng}, {Ye}, {Yu}, and
{Song}}{{Jing} et~al\mbox{.}}{2019}]%
{Jing2019NeuralStyleTransferReview}
\bibfield{author}{\bibinfo{person}{Y. {Jing}}, \bibinfo{person}{Y. {Yang}},
\bibinfo{person}{Z. {Feng}}, \bibinfo{person}{J. {Ye}}, \bibinfo{person}{Y.
{Yu}}, {and} \bibinfo{person}{M. {Song}}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Neural Style Transfer: A Review}.
\newblock \bibinfo{journal}{\emph{IEEE Transactions on Visualization and
Computer Graphics}} (\bibinfo{year}{2019}), \bibinfo{pages}{1--1}.
\newblock
\bibitem[\protect\citeauthoryear{Karras, Laine, and Aila}{Karras
et~al\mbox{.}}{2019}]%
{Karras_2019_CVPR}
\bibfield{author}{\bibinfo{person}{Tero Karras}, \bibinfo{person}{Samuli
Laine}, {and} \bibinfo{person}{Timo Aila}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{A Style-Based Generator Architecture for Generative
Adversarial Networks}. In \bibinfo{booktitle}{\emph{The IEEE Conference on
Computer Vision and Pattern Recognition (CVPR)}}.
\newblock
\bibitem[\protect\citeauthoryear{Kingma and Ba}{Kingma and Ba}{2014}]%
{kingma2014adam}
\bibfield{author}{\bibinfo{person}{Diederik~P. Kingma} {and}
\bibinfo{person}{Jimmy Ba}.} \bibinfo{year}{2014}\natexlab{}.
\newblock \bibinfo{title}{Adam: A Method for Stochastic Optimization}.
\newblock
\newblock
\showeprint[arxiv]{cs.LG/1412.6980}
\bibitem[\protect\citeauthoryear{Korman, Reichman, Tsur, and Avidan}{Korman
et~al\mbox{.}}{2013}]%
{korman2013fast}
\bibfield{author}{\bibinfo{person}{Simon Korman}, \bibinfo{person}{Daniel
Reichman}, \bibinfo{person}{Gilad Tsur}, {and} \bibinfo{person}{Shai
Avidan}.} \bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{Fast-match: Fast affine template matching}. In
\bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition}}. \bibinfo{pages}{2331--2338}.
\newblock
\bibitem[\protect\citeauthoryear{Kwatra, Sch\"{o}dl, Essa, Turk, and
Bobick}{Kwatra et~al\mbox{.}}{2003}]%
{kwatra2003graphcutTextures}
\bibfield{author}{\bibinfo{person}{Vivek Kwatra}, \bibinfo{person}{Arno
Sch\"{o}dl}, \bibinfo{person}{Irfan Essa}, \bibinfo{person}{Greg Turk}, {and}
\bibinfo{person}{Aaron Bobick}.} \bibinfo{year}{2003}\natexlab{}.
\newblock \showarticletitle{Graphcut Textures: Image and Video Synthesis Using
Graph Cuts}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph.}} \bibinfo{volume}{22},
\bibinfo{number}{3} (\bibinfo{date}{July} \bibinfo{year}{2003}),
\bibinfo{pages}{277–286}.
\newblock
\showISSN{0730-0301}
\urldef\tempurl%
\url{https://doi.org/10.1145/882262.882264}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Landes, Galerne, and Hurtut}{Landes
et~al\mbox{.}}{2013}]%
{Landes2013ASynthesis}
\bibfield{author}{\bibinfo{person}{Pierre-Edouard Landes},
\bibinfo{person}{Bruno Galerne}, {and} \bibinfo{person}{Thomas Hurtut}.}
\bibinfo{year}{2013}\natexlab{}.
\newblock \showarticletitle{{A Shape-Aware Model for Discrete Texture
Synthesis}}.
\newblock \bibinfo{journal}{\emph{Computer Graphics Forum}}
\bibinfo{volume}{32}, \bibinfo{number}{4} (\bibinfo{date}{7}
\bibinfo{year}{2013}), \bibinfo{pages}{67--76}.
\newblock
\showISSN{01677055}
\urldef\tempurl%
\url{https://doi.org/10.1111/cgf.12152}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Lehtinen, Munkberg, Hasselgren, Laine, Karras,
Aittala, and Aila}{Lehtinen et~al\mbox{.}}{2018}]%
{pmlr-v80-lehtinen18a}
\bibfield{author}{\bibinfo{person}{Jaakko Lehtinen}, \bibinfo{person}{Jacob
Munkberg}, \bibinfo{person}{Jon Hasselgren}, \bibinfo{person}{Samuli Laine},
\bibinfo{person}{Tero Karras}, \bibinfo{person}{Miika Aittala}, {and}
\bibinfo{person}{Timo Aila}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{N}oise2{N}oise: Learning Image Restoration without
Clean Data}. In \bibinfo{booktitle}{\emph{Proceedings of the 35th
International Conference on Machine Learning}}, Vol.~\bibinfo{volume}{80}.
\bibinfo{pages}{2965--2974}.
\newblock
\bibitem[\protect\citeauthoryear{Leimk\"{u}hler, Singh, Myszkowski, Seidel, and
Ritschel}{Leimk\"{u}hler et~al\mbox{.}}{2019}]%
{leimkuehler2019DeepPointCorr}
\bibfield{author}{\bibinfo{person}{Thomas Leimk\"{u}hler},
\bibinfo{person}{Gurprit Singh}, \bibinfo{person}{Karol Myszkowski},
\bibinfo{person}{Hans-Peter Seidel}, {and} \bibinfo{person}{Tobias
Ritschel}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Deep Point Correlation Design}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph.}} \bibinfo{volume}{38},
\bibinfo{number}{6}, Article \bibinfo{articleno}{Article 226}
(\bibinfo{date}{Nov.} \bibinfo{year}{2019}), \bibinfo{numpages}{17}~pages.
\newblock
\showISSN{0730-0301}
\urldef\tempurl%
\url{https://doi.org/10.1145/3355089.3356562}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Lewis}{Lewis}{1995}]%
{lewis1995fast}
\bibfield{author}{\bibinfo{person}{J.P. Lewis}.}
\bibinfo{year}{1995}\natexlab{}.
\newblock \showarticletitle{Fast Template Matching}. In
\bibinfo{booktitle}{\emph{Proc. Vision Interface}}.
\bibinfo{pages}{120--123}.
\newblock
\bibitem[\protect\citeauthoryear{Li, Zaheer, Zhang, Poczos, and
Salakhutdinov}{Li et~al\mbox{.}}{2018}]%
{Li2018PointGAN}
\bibfield{author}{\bibinfo{person}{Chun-Liang Li}, \bibinfo{person}{Manzil
Zaheer}, \bibinfo{person}{Yang Zhang}, \bibinfo{person}{Barnabas Poczos},
{and} \bibinfo{person}{Ruslan Salakhutdinov}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{Point Cloud GAN}}.
\newblock (\bibinfo{date}{10} \bibinfo{year}{2018}).
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1810.05795}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Liang, Liu, Xu, Guo, and Shum}{Liang
et~al\mbox{.}}{2001}]%
{liang2001texSynth}
\bibfield{author}{\bibinfo{person}{Lin Liang}, \bibinfo{person}{Ce Liu},
\bibinfo{person}{Ying-Qing Xu}, \bibinfo{person}{Baining Guo}, {and}
\bibinfo{person}{Heung-Yeung Shum}.} \bibinfo{year}{2001}\natexlab{}.
\newblock \showarticletitle{Real-Time Texture Synthesis by Patch-Based
Sampling}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph.}} \bibinfo{volume}{20},
\bibinfo{number}{3} (\bibinfo{date}{July} \bibinfo{year}{2001}),
\bibinfo{pages}{127–150}.
\newblock
\showISSN{0730-0301}
\urldef\tempurl%
\url{https://doi.org/10.1145/501786.501787}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Liu, Tao, and Jacobson}{Liu
et~al\mbox{.}}{2018}]%
{liu2018paparazzi}
\bibfield{author}{\bibinfo{person}{Hsueh-Ti~Derek Liu},
\bibinfo{person}{Michael Tao}, {and} \bibinfo{person}{Alec Jacobson}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Paparazzi: surface editing by way of multi-view
image processing.}
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph.}} \bibinfo{volume}{37},
\bibinfo{number}{6} (\bibinfo{year}{2018}), \bibinfo{pages}{221--1}.
\newblock
\bibitem[\protect\citeauthoryear{Liu, Li, Chen, and Li}{Liu
et~al\mbox{.}}{2019}]%
{liu2019soft}
\bibfield{author}{\bibinfo{person}{Shichen Liu}, \bibinfo{person}{Tianye Li},
\bibinfo{person}{Weikai Chen}, {and} \bibinfo{person}{Hao Li}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Soft rasterizer: A differentiable renderer for
image-based 3d reasoning}. In \bibinfo{booktitle}{\emph{Proceedings of the
IEEE International Conference on Computer Vision}}.
\bibinfo{pages}{7708--7717}.
\newblock
\bibitem[\protect\citeauthoryear{Loubet, Holzschuch, and Jakob}{Loubet
et~al\mbox{.}}{2019}]%
{loubet2019reparameterizing}
\bibfield{author}{\bibinfo{person}{Guillaume Loubet}, \bibinfo{person}{Nicolas
Holzschuch}, {and} \bibinfo{person}{Wenzel Jakob}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Reparameterizing discontinuous integrands for
differentiable rendering}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics (TOG)}}
\bibinfo{volume}{38}, \bibinfo{number}{6} (\bibinfo{year}{2019}),
\bibinfo{pages}{1--14}.
\newblock
\bibitem[\protect\citeauthoryear{Luo, Schwing, and Urtasun}{Luo
et~al\mbox{.}}{2016}]%
{luo2016efficient}
\bibfield{author}{\bibinfo{person}{Wenjie Luo}, \bibinfo{person}{Alexander~G
Schwing}, {and} \bibinfo{person}{Raquel Urtasun}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Efficient deep learning for stereo matching}. In
\bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition}}. \bibinfo{pages}{5695--5703}.
\newblock
\bibitem[\protect\citeauthoryear{Ma, Wei, and Tong}{Ma et~al\mbox{.}}{2011}]%
{ma2011discrete}
\bibfield{author}{\bibinfo{person}{Chongyang Ma}, \bibinfo{person}{Li-Yi Wei},
{and} \bibinfo{person}{Xin Tong}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Discrete element textures}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics (TOG)}}
\bibinfo{volume}{30}, \bibinfo{number}{4} (\bibinfo{year}{2011}),
\bibinfo{pages}{1--10}.
\newblock
\bibitem[\protect\citeauthoryear{Mattoccia, Tombari, and Di~Stefano}{Mattoccia
et~al\mbox{.}}{2008}]%
{mattoccia2008fast}
\bibfield{author}{\bibinfo{person}{Stefano Mattoccia},
\bibinfo{person}{Federico Tombari}, {and} \bibinfo{person}{Luigi
Di~Stefano}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{Fast full-search equivalent template matching by
enhanced bounded correlation}.
\newblock \bibinfo{journal}{\emph{IEEE transactions on image processing}}
\bibinfo{volume}{17}, \bibinfo{number}{4} (\bibinfo{year}{2008}),
\bibinfo{pages}{528--538}.
\newblock
\bibitem[\protect\citeauthoryear{McCann and Pollard}{McCann and
Pollard}{2009}]%
{locallayering:09}
\bibfield{author}{\bibinfo{person}{James McCann} {and} \bibinfo{person}{Nancy
Pollard}.} \bibinfo{year}{2009}\natexlab{}.
\newblock \showarticletitle{Local Layering}. In \bibinfo{booktitle}{\emph{ACM
SIGGRAPH 2009 Papers}}. Article \bibinfo{articleno}{84},
\bibinfo{numpages}{7}~pages.
\newblock
\bibitem[\protect\citeauthoryear{Melekhov, Kannala, and Rahtu}{Melekhov
et~al\mbox{.}}{2016}]%
{melekhov2016image}
\bibfield{author}{\bibinfo{person}{Iaroslav Melekhov}, \bibinfo{person}{Juho
Kannala}, {and} \bibinfo{person}{Esa Rahtu}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Image patch matching using convolutional
descriptors with euclidean distance}. In \bibinfo{booktitle}{\emph{Asian
Conference on Computer Vision}}. Springer, \bibinfo{pages}{638--653}.
\newblock
\bibitem[\protect\citeauthoryear{M{\"{u}}ller, McWilliams, Rousselle, Gross,
and Nov{\'{a}}k}{M{\"{u}}ller et~al\mbox{.}}{2018}]%
{Muller2018NeuralSampling}
\bibfield{author}{\bibinfo{person}{Thomas M{\"{u}}ller}, \bibinfo{person}{Brian
McWilliams}, \bibinfo{person}{Fabrice Rousselle}, \bibinfo{person}{Markus
Gross}, {and} \bibinfo{person}{Jan Nov{\'{a}}k}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{Neural Importance Sampling}}.
\newblock (\bibinfo{date}{8} \bibinfo{year}{2018}).
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1808.03856}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Ouyang, Tombari, Mattoccia, Di~Stefano, and
Cham}{Ouyang et~al\mbox{.}}{2011}]%
{ouyang2011performance}
\bibfield{author}{\bibinfo{person}{Wanli Ouyang}, \bibinfo{person}{Federico
Tombari}, \bibinfo{person}{Stefano Mattoccia}, \bibinfo{person}{Luigi
Di~Stefano}, {and} \bibinfo{person}{Wai-Kuen Cham}.}
\bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Performance evaluation of full search equivalent
pattern matching algorithms}.
\newblock \bibinfo{journal}{\emph{IEEE transactions on pattern analysis and
machine intelligence}} \bibinfo{volume}{34}, \bibinfo{number}{1}
(\bibinfo{year}{2011}), \bibinfo{pages}{127--143}.
\newblock
\bibitem[\protect\citeauthoryear{{\"{O}}ztireli and Grossy}{{\"{O}}ztireli and
Grossy}{2012}]%
{Oztireli2012AnalysisCorrelation}
\bibfield{author}{\bibinfo{person}{A.~Cengiz {\"{O}}ztireli} {and}
\bibinfo{person}{Markus Grossy}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{{Analysis and synthesis of point distributions
based on pair correlation}}. In \bibinfo{booktitle}{\emph{ACM Transactions on
Graphics}}, Vol.~\bibinfo{volume}{31}.
\newblock
\showISSN{07300301}
\urldef\tempurl%
\url{https://doi.org/10.1145/2366145.2366189}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Rocco, Arandjelovic, and Sivic}{Rocco
et~al\mbox{.}}{2017}]%
{rocco2017convolutional}
\bibfield{author}{\bibinfo{person}{Ignacio Rocco}, \bibinfo{person}{Relja
Arandjelovic}, {and} \bibinfo{person}{Josef Sivic}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Convolutional neural network architecture for
geometric matching}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition}}.
\bibinfo{pages}{6148--6157}.
\newblock
\bibitem[\protect\citeauthoryear{Rosenberger, Cohen-Or, and
Lischinski}{Rosenberger et~al\mbox{.}}{2009}]%
{rosenberger2009layered}
\bibfield{author}{\bibinfo{person}{Amir Rosenberger}, \bibinfo{person}{Daniel
Cohen-Or}, {and} \bibinfo{person}{Dani Lischinski}.}
\bibinfo{year}{2009}\natexlab{}.
\newblock \showarticletitle{Layered shape synthesis: automatic generation of
control maps for non-stationary textures}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics (TOG)}}
\bibinfo{volume}{28}, \bibinfo{number}{5} (\bibinfo{year}{2009}),
\bibinfo{pages}{1--9}.
\newblock
\bibitem[\protect\citeauthoryear{Rother, Kolmogorov, and Blake}{Rother
et~al\mbox{.}}{2004}]%
{rother2004grabcut}
\bibfield{author}{\bibinfo{person}{Carsten Rother}, \bibinfo{person}{Vladimir
Kolmogorov}, {and} \bibinfo{person}{Andrew Blake}.}
\bibinfo{year}{2004}\natexlab{}.
\newblock \showarticletitle{" GrabCut" interactive foreground extraction using
iterated graph cuts}.
\newblock \bibinfo{journal}{\emph{ACM transactions on graphics (TOG)}}
\bibinfo{volume}{23}, \bibinfo{number}{3} (\bibinfo{year}{2004}),
\bibinfo{pages}{309--314}.
\newblock
\bibitem[\protect\citeauthoryear{Roveri, {\"{O}}ztireli, and Gross}{Roveri
et~al\mbox{.}}{2017}]%
{Roveri2017GeneralCorrelations}
\bibfield{author}{\bibinfo{person}{Riccardo Roveri}, \bibinfo{person}{A.~Cengiz
{\"{O}}ztireli}, {and} \bibinfo{person}{Markus Gross}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{{General Point Sampling with Adaptive Density and
Correlations}}.
\newblock \bibinfo{journal}{\emph{Computer Graphics Forum}}
\bibinfo{volume}{36}, \bibinfo{number}{2} (\bibinfo{date}{5}
\bibinfo{year}{2017}), \bibinfo{pages}{107--117}.
\newblock
\showISSN{01677055}
\urldef\tempurl%
\url{https://doi.org/10.1111/cgf.13111}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Roveri, {\"O}ztireli, Martin, Solenthaler, and
Gross}{Roveri et~al\mbox{.}}{2015}]%
{roveri2015example}
\bibfield{author}{\bibinfo{person}{Riccardo Roveri}, \bibinfo{person}{A~Cengiz
{\"O}ztireli}, \bibinfo{person}{Sebastian Martin}, \bibinfo{person}{Barbara
Solenthaler}, {and} \bibinfo{person}{Markus Gross}.}
\bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Example based repetitive structure synthesis}. In
\bibinfo{booktitle}{\emph{Computer Graphics Forum}},
Vol.~\bibinfo{volume}{34}. Wiley Online Library, \bibinfo{pages}{39--52}.
\newblock
\bibitem[\protect\citeauthoryear{Shaham, Dekel, and Michaeli}{Shaham
et~al\mbox{.}}{2019}]%
{Shaham_2019_ICCV}
\bibfield{author}{\bibinfo{person}{Tamar~Rott Shaham}, \bibinfo{person}{Tali
Dekel}, {and} \bibinfo{person}{Tomer Michaeli}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{SinGAN: Learning a Generative Model From a Single
Natural Image}. In \bibinfo{booktitle}{\emph{The IEEE International
Conference on Computer Vision (ICCV)}}.
\newblock
\bibitem[\protect\citeauthoryear{Shocher, Bagon, Isola, and Irani}{Shocher
et~al\mbox{.}}{2019}]%
{Shocher_2019_ICCV}
\bibfield{author}{\bibinfo{person}{Assaf Shocher}, \bibinfo{person}{Shai
Bagon}, \bibinfo{person}{Phillip Isola}, {and} \bibinfo{person}{Michal
Irani}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{InGAN: Capturing and Retargeting the "DNA" of a
Natural Image}. In \bibinfo{booktitle}{\emph{The IEEE International
Conference on Computer Vision (ICCV)}}.
\newblock
\bibitem[\protect\citeauthoryear{Simo-Serra, Trulls, Ferraz, Kokkinos, Fua, and
Moreno-Noguer}{Simo-Serra et~al\mbox{.}}{2015}]%
{simo2015discriminative}
\bibfield{author}{\bibinfo{person}{Edgar Simo-Serra}, \bibinfo{person}{Eduard
Trulls}, \bibinfo{person}{Luis Ferraz}, \bibinfo{person}{Iasonas Kokkinos},
\bibinfo{person}{Pascal Fua}, {and} \bibinfo{person}{Francesc
Moreno-Noguer}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Discriminative learning of deep convolutional
feature point descriptors}. In \bibinfo{booktitle}{\emph{Proceedings of the
IEEE International Conference on Computer Vision}}.
\bibinfo{pages}{118--126}.
\newblock
\bibitem[\protect\citeauthoryear{Simonyan and Zisserman}{Simonyan and
Zisserman}{2015}]%
{Simonyan2015VERYRECOGNITION}
\bibfield{author}{\bibinfo{person}{Karen Simonyan} {and}
\bibinfo{person}{Andrew Zisserman}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Very Deep Convolutional Networks for Large-Scale
Image Recognition}. In \bibinfo{booktitle}{\emph{International Conference on
Learning Representations}}.
\newblock
\bibitem[\protect\citeauthoryear{{\v{S}}t'ava, Bene{\v{s}}, M{\v{e}}ch, Aliaga,
and Kri{\v{s}}tof}{{\v{S}}t'ava et~al\mbox{.}}{2010}]%
{vst2010inverse}
\bibfield{author}{\bibinfo{person}{Ondrej {\v{S}}t'ava},
\bibinfo{person}{Bedrich Bene{\v{s}}}, \bibinfo{person}{Radomir M{\v{e}}ch},
\bibinfo{person}{Daniel~G Aliaga}, {and} \bibinfo{person}{Peter
Kri{\v{s}}tof}.} \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{Inverse procedural modeling by automatic generation
of L-systems}. In \bibinfo{booktitle}{\emph{Computer Graphics Forum}},
Vol.~\bibinfo{volume}{29}. Wiley Online Library, \bibinfo{pages}{665--674}.
\newblock
\bibitem[\protect\citeauthoryear{Sun, Wang, Liu, Siegel, and Sarma}{Sun
et~al\mbox{.}}{2018}]%
{Sun2018PointGrow:Self-Attention}
\bibfield{author}{\bibinfo{person}{Yongbin Sun}, \bibinfo{person}{Yue Wang},
\bibinfo{person}{Ziwei Liu}, \bibinfo{person}{Joshua~E. Siegel}, {and}
\bibinfo{person}{Sanjay~E. Sarma}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{PointGrow: Autoregressively Learned Point Cloud
Generation with Self-Attention}}.
\newblock (\bibinfo{date}{10} \bibinfo{year}{2018}).
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1810.05591}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Talmi, Mechrez, and Zelnik-Manor}{Talmi
et~al\mbox{.}}{2017}]%
{talmi2017template}
\bibfield{author}{\bibinfo{person}{Itamar Talmi}, \bibinfo{person}{Roey
Mechrez}, {and} \bibinfo{person}{Lihi Zelnik-Manor}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Template matching with deformable diversity
similarity}. In \bibinfo{booktitle}{\emph{Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition}}. \bibinfo{pages}{175--183}.
\newblock
\bibitem[\protect\citeauthoryear{Talton, Lou, Lesser, Duke, M{\v{e}}ch, and
Koltun}{Talton et~al\mbox{.}}{2011}]%
{talton2011metropolis}
\bibfield{author}{\bibinfo{person}{Jerry~O Talton}, \bibinfo{person}{Yu Lou},
\bibinfo{person}{Steve Lesser}, \bibinfo{person}{Jared Duke},
\bibinfo{person}{Radom{\'\i}r M{\v{e}}ch}, {and} \bibinfo{person}{Vladlen
Koltun}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Metropolis procedural modeling}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics (TOG)}}
\bibinfo{volume}{30}, \bibinfo{number}{2} (\bibinfo{year}{2011}),
\bibinfo{pages}{1--14}.
\newblock
\bibitem[\protect\citeauthoryear{Tang, Andres, Andriluka, and Schiele}{Tang
et~al\mbox{.}}{2016}]%
{tang2016multi}
\bibfield{author}{\bibinfo{person}{Siyu Tang}, \bibinfo{person}{Bjoern Andres},
\bibinfo{person}{Mykhaylo Andriluka}, {and} \bibinfo{person}{Bernt Schiele}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Multi-person tracking by multicut and deep
matching}. In \bibinfo{booktitle}{\emph{European Conference on Computer
Vision}}. Springer, \bibinfo{pages}{100--111}.
\newblock
\bibitem[\protect\citeauthoryear{Thewlis, Zheng, Torr, and Vedaldi}{Thewlis
et~al\mbox{.}}{2016}]%
{thewlis2016fully}
\bibfield{author}{\bibinfo{person}{James Thewlis}, \bibinfo{person}{Shuai
Zheng}, \bibinfo{person}{Philip~HS Torr}, {and} \bibinfo{person}{Andrea
Vedaldi}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Fully-trainable deep matching}. In
\bibinfo{booktitle}{\emph{British Machine Vision Conference}}.
\newblock
\bibitem[\protect\citeauthoryear{Tu, Lischinski, and Huang}{Tu
et~al\mbox{.}}{2019}]%
{tu2019point}
\bibfield{author}{\bibinfo{person}{Peihan Tu}, \bibinfo{person}{Dani
Lischinski}, {and} \bibinfo{person}{Hui Huang}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Point Pattern Synthesis via Irregular Convolution}.
In \bibinfo{booktitle}{\emph{Computer Graphics Forum}},
Vol.~\bibinfo{volume}{38}. Wiley Online Library, \bibinfo{pages}{109--122}.
\newblock
\bibitem[\protect\citeauthoryear{Ulyanov, Lebedev, Vedaldi, and
Lempitsky}{Ulyanov et~al\mbox{.}}{2016}]%
{Ulyanov2016TextureImages}
\bibfield{author}{\bibinfo{person}{Dmitry Ulyanov}, \bibinfo{person}{Vadim
Lebedev}, \bibinfo{person}{Andrea Vedaldi}, {and} \bibinfo{person}{Victor
Lempitsky}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{{Texture Networks: Feed-forward Synthesis of
Textures and Stylized Images}}.
\newblock (\bibinfo{date}{3} \bibinfo{year}{2016}).
\newblock
\urldef\tempurl%
\url{http://arxiv.org/abs/1603.03417}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Wei and Lai}{Wei and Lai}{2008}]%
{wei2008fast}
\bibfield{author}{\bibinfo{person}{Shou-Der Wei} {and}
\bibinfo{person}{Shang-Hong Lai}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{Fast template matching based on normalized cross
correlation with adaptive multilevel winner update}.
\newblock \bibinfo{journal}{\emph{IEEE Transactions on Image Processing}}
\bibinfo{volume}{17}, \bibinfo{number}{11} (\bibinfo{year}{2008}),
\bibinfo{pages}{2227--2235}.
\newblock
\bibitem[\protect\citeauthoryear{Wu, Abd-Almageed, and Natarajan}{Wu
et~al\mbox{.}}{2017}]%
{wu2017deep}
\bibfield{author}{\bibinfo{person}{Yue Wu}, \bibinfo{person}{Wael
Abd-Almageed}, {and} \bibinfo{person}{Prem Natarajan}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Deep matching and validation network: An end-to-end
solution to constrained image splicing localization and detection}. In
\bibinfo{booktitle}{\emph{Proceedings of the 25th ACM international
conference on Multimedia}}. \bibinfo{pages}{1480--1502}.
\newblock
\bibitem[\protect\citeauthoryear{Yu, Lin, Yang, Shen, Lu, and Huang}{Yu
et~al\mbox{.}}{2018}]%
{Yu_2018_CVPR}
\bibfield{author}{\bibinfo{person}{Jiahui Yu}, \bibinfo{person}{Zhe Lin},
\bibinfo{person}{Jimei Yang}, \bibinfo{person}{Xiaohui Shen},
\bibinfo{person}{Xin Lu}, {and} \bibinfo{person}{Thomas~S. Huang}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Generative Image Inpainting With Contextual
Attention}. In \bibinfo{booktitle}{\emph{The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR)}}.
\newblock
\bibitem[\protect\citeauthoryear{Yu, Lin, Yang, Shen, Lu, and Huang}{Yu
et~al\mbox{.}}{2019}]%
{Yu_2019_ICCV}
\bibfield{author}{\bibinfo{person}{Jiahui Yu}, \bibinfo{person}{Zhe Lin},
\bibinfo{person}{Jimei Yang}, \bibinfo{person}{Xiaohui Shen},
\bibinfo{person}{Xin Lu}, {and} \bibinfo{person}{Thomas~S. Huang}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Free-Form Image Inpainting With Gated Convolution}.
In \bibinfo{booktitle}{\emph{The IEEE International Conference on Computer
Vision (ICCV)}}.
\newblock
\bibitem[\protect\citeauthoryear{Zhou, Huang, Wei, and Wang}{Zhou
et~al\mbox{.}}{2012}]%
{Zhou2012PointSpectrum}
\bibfield{author}{\bibinfo{person}{Yahan Zhou}, \bibinfo{person}{Haibin Huang},
\bibinfo{person}{Li~Yi Wei}, {and} \bibinfo{person}{Rui Wang}.}
\bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{{Point sampling with general noise spectrum}}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Graphics}}
\bibinfo{volume}{31}, \bibinfo{number}{4} (\bibinfo{date}{7}
\bibinfo{year}{2012}).
\newblock
\showISSN{07300301}
\urldef\tempurl%
\url{https://doi.org/10.1145/2185520.2185572}
\showDOI{\tempurl}
\bibitem[\protect\citeauthoryear{Zhou, Huang, Zhu, Bai, Lischinski, and
Cohen-Or}{Zhou et~al\mbox{.}}{2018}]%
{Zhou2018Non-StationaryFormat}
\bibfield{author}{\bibinfo{person}{Yang Zhou}, \bibinfo{person}{Hui Huang},
\bibinfo{person}{Zhen Zhu}, \bibinfo{person}{Xiang Bai},
\bibinfo{person}{Dani Lischinski}, {and} \bibinfo{person}{Daniel Cohen-Or}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{Non-Stationary Texture Synthesis by Adversarial
Expansion Additional Key Words and Phrases: Example-based texture synthesis,
non-stationary textures, generative adversarial networks ACM Reference
Format}}.
\newblock \bibinfo{journal}{\emph{ACM Trans. Graph}} \bibinfo{volume}{37}
(\bibinfo{year}{2018}), \bibinfo{pages}{13}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3197517.3201286}
\showDOI{\tempurl}
\end{thebibliography}