Multi-Directional Subspace Editing in Style-Space
Extracted source text ยท arXiv 2211.11825
% CVPR 2023 Paper Template
% based on the CVPR template provided by Ming-Ming Cheng (https://github.com/MCG-NKU/CVPR_Template)
% modified and extended by Stefan Roth (stefan.roth@NOSPAMtu-darmstadt.de)
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage[review]{cvpr} % To produce the REVIEW version
\usepackage{cvpr} % To produce the CAMERA-READY version
% \usepackage[pagenumbers]{cvpr} % To force page numbers, e.g. for an arXiv version
% Include other packages here, before hyperref.
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
% My packages
\usepackage{epstopdf}
\usepackage{epsfig}
\usepackage{stfloats}
\usepackage{url}
\usepackage{times}
\usepackage{amsthm}
\usepackage{cite}
\usepackage{color}
\usepackage{float}
\usepackage{caption}
% \usepackage{tikz}
\usepackage{collcell}
\usepackage{xstring}
\usepackage{multirow}
\usepackage{arydshln}
\usepackage{grffile}
\usepackage{setspace}
\makeatletter
\@namedef{ver@everyshi.sty}{}
\makeatother
\usepackage{tikz}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues, e.g. with the
% file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete
% ReviewTempalte.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you
% should be clear).
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\cvprPaperID{6910} % *** Enter the CVPR Paper ID here
\def\confName{CVPR}
\def\confYear{2023}
\begin{document}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{OrthoGAN: Multifaceted Semantics for Disentangled Face Editing}
\author{Chen Naveh \quad\quad Yacov Hel-Or\\
School of Computer Science, Reichman University\\
Herzliya, Israel\\
{\tt\small chen.naveh@post.runi.ac.il}, {\tt\small toky@runi.ac.il}}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
% \and
% Yacov Hel-Or\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small toky@runi.ac.il}
% }
\newcommand*{\imwidth}{0.16}
\twocolumn[{
\maketitle
\begin{center}
\captionsetup{type=figure[htb]}
\begin{tabular}{cc|cccc}
\rotatebox{90}{\hspace{0.29in}Gender} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im9_gender_0_0_0.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im9_gender_0_3_-3_-5_5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im9_gender_0_3_3_-5_5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im9_gender_3_0_3_-5_5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im9_gender_3_-3_0_-5_5.png} \\
\rotatebox{90}{\hspace{0.29in}Age} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im8_age_0_3_0_-5_5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im8_age_-3_0_0_-5_5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im8_age_0_3_-3_-5_-5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im8_age_0_-3_0_5_-5.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im8_age_-3_3_-3_5_-5.png} \\
\rotatebox{90}{\hspace{0.29in}Race} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im3_race_0_0_0.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im3_race_0_2_2.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im3_race_0_2_-2.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im3_race_-2_-2_-2.png} &
\includegraphics[width=\imwidth\linewidth]{images/abstract/im_im3_race_4_0_2.png} \\
& Input & \multicolumn{4}{c}{Edits}
\end{tabular}
\captionof{figure}{Multifaceted human face editing. Each row indicates changes in a subspace associated with a particular attribute: (top to bottom) gender, age and race.}
\label{fig:fig1}
\end{center}
}]
%%%%%%%%% ABSTRACT
% \doublespacing
\input{abstract}
\input{intro}
\input{related_work}
\input{method}
\input{experiments}
\input{conclusions}
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{ieee_fullname}
\bibliography{references}
}
\end{document}
\begin{abstract}
This paper describes a new technique for finding disentangled semantic directions in the latent space of StyleGAN. OrthoGAN identifies meaningful orthogonal subspaces that allow editing of one human face attribute, while minimizing undesired changes in other attributes. Our model is capable of editing a single attribute in multiple directions. Resulting in a range of possible generated images. We compare our scheme with three state-of-the-art models and show that our method outperforms them in terms of face editing and disentanglement capabilities. Additionally, we suggest quantitative measures for evaluating attribute separation and disentanglement, and exhibit the superiority of our model with respect to those measures.
\end{abstract}
\section{Introduction}
Recent developments in computer vision have enabled the generation of photo-realistic high-resolution synthetic images. The most notable technique is Generative Adversarial Networks (GANs)\cite{goodfellow2014generative}, which advanced the progress of many applications such as image generation, super resolution and image editing. In particular, the generator of StyleGAN~\cite{abdal2019image2stylegan}, one of the most notable GAN models, has been explored widely by researchers.
StyleGAN samples a latent vector $\mathbf{z} \in \mathbb{R}^{512}$ from a Gaussian distribution $\mathcal{N}(\textbf{0}, \textbf{I})$, maps it to an intermediate vector $\mathbf{w} \in \mathcal{W} = \mathbb{R}^{512}$ using a mapping network, which is then used to generate a 1024x1024 RGB image. The vector $\mathbf{w}$ is inserted into 18 multi-resolution style blocks that control various characteristics of the synthesized images. Vectors at lower resolutions control high-level features such as pose and hair, intermediate resolutions affect face expression, and vectors at the higher resolutions control fine details such as colors and texture. The original model of StyleGAN uses the $\mathcal{W}$ latent space and the same $\mathbf{w}$ vector in all 18 style blocks. Later studies\cite{abdal2019image2stylegan, abdal2020image2stylegan++, richardson2021encoding, tov2021designing} uses the $\mathcal{W^+} = \mathbb{R}^{18\cdot 512}$ space that extends the $\mathcal{W}$ space and uses different $\mathbf{w}$ vectors in different style blocks, allowing better control over the resulting image. The $\mathcal{W^+}$ latent space is commonly used for inverse mapping, from an image into $\mathbf{w} \in \mathcal{W}^+$.
Recent works have explored methods for gaining control over the synthesized images by altering the latent space vectors \cite{radford2015unsupervised, bojanowski2017optimizing, shen2020interfacegan,harkonen2020ganspaceli2021dystyle,shen2021closed, abdal2021styleflow}. For example, shifting the latent vector of a generated image towards the direction associated with ``smile" will gradually increase the smile of the resulting image. Despite demonstrating impressive and realistic editing, finding the attribute directions is still challenging. One approach is to use the latent data sampled from the GAN model along with attribute ground truth, labeled by hand or using a pre-trained attribute estimator\cite{shen2020interfacegan, goetschalckx2019ganalyze, jahanian2019steerability}. These models often suffer from the entangled data problem and struggle to overcome biases in the training data. Such biases can manifest in a correlation of attributes such as glasses and age or beard and gender; thus, adding glasses to a face will probably increase its age as well. Other methods use unsupervised techniques~\cite{harkonen2020ganspaceli2021dystyle, shen2021closed}, such as PCA, to find orthogonal directions in the latent space. These directions, however, usually rely on subjective human post-annotation in order to associate directions with attributes in the resulting images. All theses models share the concept of a single direction for each attribute despite the fact that some attribute might be explained by multiple dimensions (\eg, age might affect hair color and skin wrinkles, see\cref{fig:fig1}).
In this work we present OrthoGAN, a framework to identify orthogonal semantics in the latent space of a pre-trained StyleGAN. We attempt to learn meaningful semantic orthogonal subspaces that control certain attributes in the human face domain. Multifaceted editing refers to changing a latent vector within a subpsace in multiple directions. This allows for a variety of generated images with a modification in the related attribute and minimum change in others. We study the disentanglement properties of our model and quantitatively compare its performance against the state-of-the-art image editing methods.
\section{Related Work}
\subsection{Generative Adversarial Networks (GANs)}
Generative models have enabled the generation of massive amounts of graphical content. These models can produce visually pleasing content with high fidelity and can be extremely useful for image editing applications. GAN models learn a mapping from a specific distribution (usually
Gaussian or uniform) to the target distribution. The generator $G(\cdot)$ is trained to fool an adversary $D(\cdot)$, which is trying to distinguish between the generated images and the real images. $G(\cdot)$ and $D(\cdot)$ are trained simultaneously using the min-max loss \cite{goodfellow2014generative}. GANs are very common and currently superior to other techniques such as variational autoencoders (VAEs)\cite{kingma2013auto}. Other image synthesis techniques are the denoising diffusion probabilistic models (DDPM)~\cite{ho2020denoising}. These models define a Markov process of diffusion steps that gradually add noise to an image, and learn the reverse process of image denoising. If the number of steps is large enough, the model learns to synthesize an image from a noise image drawn from an isotropic Gaussian distribution. Although considered state of the art with regard to image generation, they have not yet been studied enough for image editing and semantic interpretation. In this work, we focus on a GAN-based generator, specifically, the StytleGAN\cite{karras2019style, karras2020analyzing, karras2021alias}, which is extremely powerful in the human face domain.
\textbf{GAN Latent Space.} Given a pre-trained generator, GAN inversion is the process of inverting a given image into a latent vector that faithfully reconstructs the image using the generator. Most of the common methods either train an encoder\cite{tov2021designing, richardson2021encoding, pidhorskyi2020adversarial} from the image space into the latent space, or optimize the latent vector directly until it generates the desired image using back propagation\cite{lipton2017precise,abdal2019image2stylegan, abdal2020image2stylegan++}. Lately, new approaches that modify the generator's weight have become popular and achieve improved image reconstruction\cite{roich2021pivotal,alaluf2021hyperstyle}. Previous works show that the latent space of a pre-trained GAN model encodes image semantics in a meaningful structure, where latent codes of images with similar semantics are located close together within the latent space\cite{shen2020interfacegan}. In the context of human face generation, faces with shared semantics (\eg, young and blond guys) will have latent vectors close to each other. This observation is the fundamental idea behind image editing methods.
\subsection{Latent Space Multi-faceted Manipulation for Image Editing}
The notion of editing an image via latent space manipulation was widely studied in recent years\cite{radford2015unsupervised, bojanowski2017optimizing}. These approaches attempt to utilize the strong capabilities of the generator, hence freezing the generator network, and perform vectorial operation only in the semantic space (\ie, the latent space). Recent works primarily rely on a linear latent space assumption and demonstrate pleasing-looking image editing results by adding and subtracting vectors in the latent space\cite{shen2020interfacegan,harkonen2020ganspaceli2021dystyle,shen2021closed}. InterFaceGAN\cite{shen2020interfacegan} uses pre-trained binary classifiers to annotate images with facial attributes such as young-old, male-female, with/without glasses, etc. Then, it trains a linear classifier (support vector machine - SVM) on the corresponding latent vectors and obtains a classifying hyperplane. The normal to the separating hyperplane is used as a 1D direction for editing the corresponding attribute.
GANSpace\cite{harkonen2020ganspaceli2021dystyle} uses a data driven approach and performs PCA to find meaningful directions on a set of vectors sampled from $\mathcal{W}$. The eigenvectors corresponding to the most significant eigenvalues are used as the directions for editing. The derived directions may be entangled (for example, head rotation and gender). To overcome this problem, the edits are then limited to a carefully chosen subset of layers of the generator.
Another work, SeFa\cite{shen2021closed}, does not rely on an image sampling pre-processing step, instead it finds a closed-form factorization of the latent space by using an eigen-decomposition of the generator's weights. For StyleGAN, the style affine transformation matrices are used for the vector decomposition. The eigenvectors associated with the most variations are chosen to describe the semantic directions. This unsupervised approach, however, suffers from two problems. First, a subjective post-annotation process is applied to associate the directions with meaningful semantics. Second, the obtained directions tend to entangle multiple semantics (\eg, age and glasses) due to the biases in the training data.
A different collection of works discards the linear approach and learns non-linear functions to manipulate latent vectors. StyleFlow\cite{abdal2021styleflow} learns continuous normalizing flows conditioned on the image attribute vector. This attribute vector must be provided using pre-trained networks before every edit. Another non-linear approach is StyleRig\cite{tewari2020stylerig}, which incorporates a 3D semantic network and leverages it to perform rigid edits such as pose and light. Although non-linear approaches generally perform better, we focus our work on the more holistic linear approach.
All previous methods that manipulate the latent space of StyleGAN define a semantic as a single dimension. Both linear and non-linear approaches ultimately derive a positive and a negative direction for editing a desired attribute. We believe that this oversimplification may restrict the editing capabilities and limit the diversity of the generated images. For example, imagine we want to edit the gender of a given image. One person may argue that hairstyle is a distinguishing factor between men and women, while another might think of facial structure or the presence of facial hair as a more prominent factor. In this work, we overcome this variability by associating an attribute with an $n$-dim \textbf{subspace} where $n \geq 1$, containing more than a single direction, in the semantic space. We call this a multi-faceted subspace, as different vectors within the subspace can affect the resulting image differently, but all affect the associated attribute (see examples in \cref{fig:fig1}). We also require different attributes to be associated with orthogonal subspaces, to promote disentanglement between attributes.
We summarize our contributions as follows:
\begin{itemize}
\item We propose OrthoGAN for expanding the concept of meaningful latent directions to multi-faceted subspaces, thereby diversifying and expanding the capabilities of the editing process.
\item OrthoGAN finds orthogonal directions in StyleGAN's latent space, to promote disentanglement. We introduce an orthogonality loss in the model training and demonstrate its significance in preserving changing image attributes while editing others. We visualize the image editing qualities of the model, and show improved results in consecutive edits of multiple attributes (see \cref{fig:fig4}).
\item We propose a new metric for evaluating the disentanglement properties of image editing generative models. In addition to the conventional approaches of visual comparisons, we suggest using our evaluation metrics as a quantitative measurement of disentanglement capabilities.
\end{itemize}
\section{Method}
Given a set of attributes $B \triangleq \left \{gender, glasses, \cdots \right \}$, we wish to find a set of latent vectors for each attribute $b_i \in B$ such that editing along those vectors results in changes in the associated attribute. Different directions in each set are responsible for distinct visual semantics. For the purpose of disentanglement we would like each set to only affect its corresponding attribute.
Some attributes such as glasses and age have been found to be correlated with each other\cite{shen2020interfacegan}. Disentangling such attributes makes the editing task more challenging. To overcome this problem, our model attempts to decompose the latent space $\mathcal{W^+}$ into multiple orthogonal subspaces each of which is associated with a single attribute. This action has two main implications. To start with, editing within a subspace allows for multifaceted edits of a single attribute. Second, it allows changing a particular facial attribute without impacting others.
\subsection{Latent Space Decomposition}
We divide $\mathcal{W^+}$ into $N+1$ orthogonal subspaces $\left \{ S_i \right \}_{i=0}^N$ and define each subspace $S_i$ as
\begin{equation}
S_i \triangleq span \left \{ \textbf{p}_i^1, \textbf{p}_i^2, \cdots, \textbf{p}_i^{n_i} \right \}, n_i < dim(\mathcal{W^+})
\end{equation}
where $\textbf{p}_i^j \in \mathcal{W^+}$, $j=\{1,\cdots,n_i\}$ are linearly independent vectors. Accordingly, $\{ \textbf{p}_i^j \}_{j=1}^{n_i}$ form a basis for subspace $S_i$ with cardinality $n_i$. There are $N+1$ subspaces, where $N=|B|$ is the size of the attribute set $B$. Each of $S_i$ is associated with $b_i$ respectively, while $S_0$ is associated with all other information that is not labeled in $B$. This may include other semantics, \eg, clothing and image background. The correspondence of $S_i$ and $b_i$ will be explained later in the paper.
To ensure orthogonality we require that the following holds
\begin{equation}\label{eq4}
S_i \subset \mathcal{W^+} \;, \forall i \in \left \{ {0,\cdots,N} \right \}
\end{equation}
\begin{equation}\label{eq5}
\bigoplus_{i=0}^N S_i = \mathcal{W^+}
\end{equation}
\begin{equation}\label{eq6}
S_i\perp S_j \;, \forall i\neq j
\end{equation}
where $\bigoplus$ denotes direct sum and $\perp$ denotes orthogonal subspaces. Notice that the vector set $\{ \textbf{p}_i^j \}$ forms a basis for $\mathcal{W^+}$. Consequently, we can uniquely express every vector $\textbf{w} \in \mathbb{R}^{18 \cdot 512}$ with a set of scalars $\{ a_i^j \}$ and write $\textbf{w}$ as the linear combination
\begin{equation}\label{eq8}
\textbf{w} = \sum_{i=0}^{N} \sum_{j=1}^{n_i} a_i^j \textbf{p}_i^j
\end{equation}
This can also be written in a matrix form
\begin{equation}
\textbf{w} =
\begin{bmatrix}
\underbrace{\begin{bmatrix}
\vert & & \vert \\
\textbf{p}_0^1 & \cdots & \textbf{p}_0^{n_0} \\
\vert & & \vert
\end{bmatrix}}_{\textbf{P}_0}
\cdots
\underbrace{\begin{bmatrix}
\vert & & \vert \\
\textbf{p}_N^1 & \cdots & \textbf{p}_N^{n_N} \\
\vert & & \vert
\end{bmatrix}}_{\textbf{P}_N}
\end{bmatrix}
\begin{bmatrix}
\begin{bmatrix}
a_0^1 \\
\vdots \\
a_0^{n_0} \\
\end{bmatrix}
\\
\vdots
\\
\begin{bmatrix}
a_N^1 \\
\vdots \\
a_N^{n_N} \\
\end{bmatrix}
\end{bmatrix}
\end{equation}
\begin{equation}
\textbf{w} = \sum_{i=0}^N \mathbf{P}_i\textbf{a}_i = \\
\underbrace{\begin{bmatrix}
\mathbf{P}_0, \cdots, \mathbf{P}_N \\
\end{bmatrix}}_{\mathbf{P}}
\underbrace{\begin{bmatrix}
\textbf{a}_0 \\ \vdots \\ \textbf{a}_N \\
\end{bmatrix}}_{\mathbf{a}}
\end{equation}
where $\mathbf{P}$ is an orthogonal matrix defined by:
\begin{equation}
\mathbf{P} = \left [ \textbf{p}_0^1 \cdots \textbf{p}_0^{n_0} \cdots \textbf{p}_N^1 \cdots \textbf{p}_N^{n_N} \right ]
\end{equation}
and $\mathbf{a} = [\textbf{a}^T_0, \dots, \textbf{a}^T_N]$ is a vector of coefficients. Finding $\mathbf{P}$ that satisfies \cref{eq4}-(\ref{eq6}) is the core part of our framework.
% \begin{figure}
% \centering
% \includegraphics[width=0.6\linewidth]{images/subs.pdf}
% \caption{Decomposing of latent vector $\textbf{w}$ into two orthogonal supspaces $S_1$, $S_2$. Each subspace is associated with a different semantic.}
% \label{fig:fig2}
% \end{figure}
\subsection{Training Procedure}
\label{sec:train}
Our dataset comprised a set of 2000 vector pairs $\left \{ (\textbf{w}^{(i)}, \textbf{y}^{(i)}) \right \}_{i=1}^{2000}$. The latent vectors $\left \{ \textbf{w}^{(i)} \right \}$ were generated from random vectors sampled from a Gaussian distribution $\textbf{z}^{(i)} \sim \mathbb{N}(0, 1)$ and then mapped to $\mathcal{W^+}$ space using the StyleGAN mapping function: $\textbf{w}^{(i)}=M(\textbf{z}^{(i)})$. Each sample $\textbf{w}^{(i)} \in \mathbb{R}^{18 \cdot 512}$ is mapped onto the image space using the StyleGAN generator, $\textbf{x}^{(i)}=G(\textbf{w}^{(i)})$ and then labeled using pre-trained classifiers to determine an attribute score vector
\begin{equation}
\textbf{y}^{(i)} \triangleq (y^{(i)}_1, \cdots, y^{(i)}_N) = \left ( \mathcal{C}_1(\textbf{x}^{(i)}), \cdots, \mathcal{C}_N(\textbf{x}^{(i)}) \right )
\end{equation}
Each $y^{(i)}_k$ denotes the $b_k$ attribute score for sample $i$ and, depending on the attribute, is either a discrete or continuous number. $\mathcal{C}_k$ is the pre-trained classifier for attribute $k$. For age, smile, gender and glasses, we used the face attribute classifier from~\cite{he2018harnessing} trained on the FFHQ dataset~\cite{karras2019style}. For the pose attribute, we used the img2pose~\cite{albiero2021img2pose} face estimation. In addition, we used a race classifier~\cite{karkkainenfairface} trained on the Yahoo YFCC100M dataset~\cite{thomee2016yfcc100m}.
% \subsection{Training Method} \label{sec:train}
The main objective of the training phase is to find the matrix $\mathbf{P}$ and utilize it to reconstruct all samples from the training set. To satisfy \cref{eq8}, we jointly learn a vector $\textbf{a}^{(i)}$ for each vector $\textbf{w}^{(i)}$ such that $\textbf{w}^{(i)} = \mathbf{P}\textbf{a}^{(i)}$ and introduce the following loss
\begin{equation}
\mathcal{L}^{(i)}_{rec} = \left \| \textbf{w}^{(i)} - \mathbf{P}\textbf{a}^{(i)} \right \|_1
\end{equation}
where $\mathcal{L}_{rec}$ stands for reconstruction loss. Instead of pixel-based image reconstructions, we use a $L_1$ loss in the latent space utilizing the original $\textbf{w}^{(i)}=M(\textbf{z}^{(i)})$ as the target value.
Additionally, to force disentanglement, as referred in \cref{eq6}, we add an orthogonality loss and require that the columns of matrices $\mathbf{P}_i, \mathbf{P}_j$ be orthogonal for $i \neq j$
\begin{equation}
\mathcal{L}_{orth} = \sum_{i \neq j} \left \| \mathbf{P}_i^T \mathbf{P}_j \right \|_2^2
\end{equation}
where $\left \| \cdot \right \|_2$ is the element-wise Frobenius-norm.
Since learning disentangled representations is fundamentally impossible without a supervised inductive bias on the data\cite{locatello2019challenging}, we utilize the attribute vector $\textbf{y}^{(i)}$ and introduce a mixing loss to encourage association between $S_i$ and $b_i$. Given a vector $\textbf{w}^{(i)}$, we can import attribute $b_k$ from a randomly chosen vector $\textbf{w}^{(j)}$, to generate $\textbf{w}^{(i)}_{mix}$:
\begin{equation}
\textbf{w}^{(i)}_{mix} = \mathbf{P}_k\textbf{a}^{(j)}_k + \sum_{l\neq k} \mathbf{P}_l\textbf{a}^{(i)}_l
\end{equation}
and its corresponding image $\textbf{x}^{(i)}_{mix} \triangleq G(\textbf{w}^{(i)}_{mix})$. For the modified image we require $y^{(i)}_k$ to be similar to $y^{(j)}_k$ while keeping the other attributes unchanged. Thus a mixing loss is added to the network to force changes only in $b_k$:
\begin{equation}
\begin{split}
\mathcal{L}^{(i)}_{mixing} = L_k \left ( \mathcal{C}_k(\textbf{x}^{(i)}_{mix}), y_k^{(j)} \right ) \\
+ \sum_{l \neq k} L_l \left ( \mathcal{C}_l(\textbf{x}^{(i)}_{mix}), y_l^{(i)} \right )
\end{split}
\end{equation}
$L_i$ are loss functions that differ according to the attribute types. For categorical labels we use a softmax with cross-entropy loss, while continuous labels are optimized using the $L_1$ loss. In practice, we mix all attributes together from randomly chosen vectors. This introduces complex changes in the image and promotes the coupling of a single subspace to a single attribute.
Finally, our model is trained with an objective function consisting of the following three losses
\begin{equation}
\mathcal{L} = \lambda_{orth}\mathcal{L}_{orth} + \frac{1}{n} \sum_{i} \mathcal{L}^{(i)}_{rec} + \lambda_{mixing}\mathcal{L}^{(i)}_{mixing}
\end{equation}
where $\lambda_{orth}$, $\lambda_{mixing}$ are hyperparameters for controlling the loss weights.
% We have chosen $\lambda_{edit}=0.001$ and $\lambda_{orth}=0.001$ in our experiments.
% our total loss is
% optimize adam, epochs, learning rates...
\section{Experiments}
In this section we evaluate our model's performance compared to state-of-the-art image editing models, all based on styleGAN generator for image synthesis.
To test our theory we generated a source and target latent vector, projected them onto a subspace $S_i$ and replaced the projections of the source vector with the target one. Since our subspaces are orthogonal, we expect to see changes only in the corresponding attribute $b_i$. Example results are shown in \cref{fig:fig2}. Note, for example, that if the source face has glasses, they remain after the edit since we import only pose, smile and race from the target face.
% ############
\newcommand*{\fsize}{0.2}
\begin{figure}
\centering
\begin{tabular}{c|ccc}
\hspace{0.0in}Source \hspace{0.0in} \hspace{0.05in}\rotatebox[origin=l]{90}{\hspace{0.1in}Target} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_3_orig.png} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_4_orig.png} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_5_orig.png}\\
\hline \\
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_0_orig.png} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_0_to_3.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_0_to_4.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_0_to_5.png}\\
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_1_orig.png} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_1_to_3.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_1_to_4.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_1_to_5.png}\\
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_2_orig.png} & \includegraphics[width=\fsize\linewidth]{images/edit_train/im_2_to_3.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_2_to_4.png} &
\includegraphics[width=\fsize\linewidth]{images/edit_train/im_2_to_5.png}
\end{tabular}
\caption{Image editing capabilities using source images and target attribute images. The attributes taken from the target images are pose, smile and race.}
\label{fig:fig2}
\end{figure}
% ############
\subsection{Comparison with Previous Methods}
We compared our model with three previous image editing methods including InterFaceGAN~\cite{shen2020interfacegan}, StyleFlow~\cite{abdal2021styleflow} and SeFa~\cite{shen2021closed}. We evaluated the editing and disentanglement qualities of these models for a common set of attributes supported by all of them. To ensure fair comparison we chose age, gender, smile, pose and glasses. Additionally, since our method discovers a subspace rather than a single direction, we trained a linear SVM inside each subspace and used the normal to the hyperplane as the direction for editing.
\subsection{Qualitative Comparison}
\Cref{fig:fig3} provides a comparison of the quality of the real image editing. The images were inverted into the latent space of StyleGAN using HyperStyle\cite{alaluf2022hyperstyle}. We found that multiple edits on a given image can dramatically reduce image quality and are more susceptible to visual artifacts. Moreover, some attributes are more correlated than others in StyleGAN's latent space due to biases in the FFHQ dataset~\cite{karras2019style} (\eg, younger people are less likely to wear glasses). As can be seen in \cref{fig:fig3}, most of the models successfully alter a single attribute. Nevertheless, when all attributes are changed all at once, our model is more accurate and better preserves every single attribute edit. A demonstration of sequential edit is presented in \cref{fig:fig4}. We can see that as we move along the edit direction, images preserve former attributes without compromising image quality.
In \cref{fig:fig1} we visualize our model's capability of generating diverse results for different attributes due to its multifaceted nature. The images are generated by shifting the latent vectors to different direction inside the relevant subspace. We found that some attributes (\eg, smile, pose) behave like binary attributes, meaning they contain most of the information in a single direction. Others (\eg, gender, age), however, can be edited in multiple directions resulting in various images.
% ######### real image editing ############
\newcommand*{\imsize}{0.11}
\begin{figure*}
\centering
\setlength{\tabcolsep}{1pt}
\hspace*{-2.0cm}
\begin{tabular}{cccccccc}
& Reconstruction & Smile & Gender & Pose & Age & Glasses & All \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.26in}Sefa} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_orig_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_smile_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_gender_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_pose_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_age_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_glasses_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_all_sefa.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.0in}InterfaceGan} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_orig_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_smile_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_gender_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_pose_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_age_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_glasses_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_all_interfacegan.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.12in}StyleFlow} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_orig.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_smile.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_gender.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_pose.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_age.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_glasses.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_0_all.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.29in}Ours} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_orig_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_smile_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_gender_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_pose_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_age_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_glasses_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00000_all_baseline_2.0.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.26in}Sefa} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_orig_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_smile_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_gender_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_pose_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_age_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_glasses_sefa.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_all_sefa.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.0in}InterfaceGan} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_orig_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_smile_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_gender_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_pose_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_age_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_glasses_interfacegan.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_all_interfacegan.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.12in}StyleFlow} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_orig.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_smile.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_gender.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_pose.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_age.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_glasses.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/img_3_all.jpg} \\
\hspace{0.7in}\rotatebox{90}{\hspace{0.29in}Ours} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_orig_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_smile_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_gender_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_pose_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_age_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_glasses_baseline_2.0.jpg} &
\includegraphics[width=\imsize\linewidth]{images/quality_comp/00003_all_baseline_2.0.jpg} \\
\end{tabular}
\caption{Real image editing comparison between our method and baselines. The edit direction for each attribute was chosen using a linear SVM classifier.}
\label{fig:fig3}
\end{figure*}
% #####################
% \newcommand*{\figsize}{0.18}
% \begin{figure}
% \centering
% \setlength{\tabcolsep}{1pt}
% \begin{tabular}{cccccc}
% Input & StyleFlow & (a) & (b) & (c) \\
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_eigen_0_gender_5_3.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/img_5_gender_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_gender_3_0_0.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_gender_0_0_5.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_gender_0_5_0.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Gender} \\
% & \includegraphics[width=\figsize\linewidth]{images/subspace/img_5_age_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_age_-5_-2_-2.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_age_-5_5_6.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im5_age_-1_-1_-6.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Age} \\
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_age_0_0_0.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/img_1_gender_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_gender_-4_0_0.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_gender_0_0_-5.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_gender_0_-5_0.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Gender} \\
% & \includegraphics[width=\figsize\linewidth]{images/subspace/img_1_age_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_age_-5_-2_-2.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_age_-5_5_0.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im1_age_-1_-1_-6.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Age} \\
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_eigen_0_gender_8_3.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/img_8_gender_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_gender_3_0_0.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_gender_0_0_5.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_gender_0_5_0.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Gender} \\
% & \includegraphics[width=\figsize\linewidth]{images/subspace/img_8_age_styleflow.jpg} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_age_-5_-2_-2.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_age_-5_5_6.png} &
% \includegraphics[width=\figsize\linewidth]{images/subspace/im_im8_age_-1_-1_-6.png} &
% \rotatebox[origin=c]{-90}{\hspace{-0.8in}Age} \\
% \end{tabular}
% \caption{Multifaceted editing of real images and comparison to StyleFlow. Columns (a) to (c) represent edits in different directions inside the corresponding subspace (gender and age), allowing for reach and diverse edits (\eg, when editing gender, vectors (b) and (c) tend to better preserve hairstyle).}
% \label{fig:fig4}
% \end{figure}
% ##############
\newcommand*{\figsize}{0.16}
\begin{figure}
\centering
\setlength{\tabcolsep}{1pt}
\begin{tabular}{c|ccccc}
\multicolumn{6}{c}{$\xrightarrow[]{\text{edit direction}}$} \\
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_0.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_1.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_2.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_3.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_4.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_0_5.png} \\
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_0.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_1.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_2.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_3.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_4.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_1_5.png} \\
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_0.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_1.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_2.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_3.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_4.png} &
\includegraphics[width=\figsize\linewidth]{images/intro/im_2_5.png} \\
Input & +Smile & +Age & +Gender & +Glasses & +Pose
\end{tabular}
\caption{Sequential human face editing result with our approach.}
\label{fig:fig4}
\end{figure}
% ###############
\subsection{Quantitative Comparison}
\label{sec:quantitative}
To conduct a quantitative evaluation of our model, we chose two different methods: attribute correlation and face preservation.
% ######### FUNCTIONS ###########
%The min, mid and max values
\newcommand*{\MinNumber}{0.0}%
\newcommand*{\MidNumber}{0.18}%
\newcommand*{\MaxNumber}{1.0}%
%Apply the gradient macro
\newcommand{\ApplyGradient}[1]{%
\pgfmathsetmacro{\PercentColor}{max(min(100.0*(#1 - \MidNumber)/(\MaxNumber-\MidNumber),100.0),0.00)} %
\hspace{-0.33em}\colorbox{green!\PercentColor!white}{#1}
}
\newcolumntype{R}{>{\collectcell\ApplyGradient}c<{\endcollectcell}}
\renewcommand{\arraystretch}{0}
\setlength{\fboxsep}{2mm} % box size
\setlength{\tabcolsep}{0pt}
% ######### correlation ###########
\begin{table*}[t]
\caption{Attribute correlation matrices of edited images (in absolute values).}
\label{tab:table1}
\centering
% \resizebox{16cm}{!}{
\begin{tabular}{cc}
(a) SeFa & (b) InterFaceGan \vspace{0.04in} \\
\begin{tabular}{l*{5}{R}}
&\multicolumn{1}{c}{Pose}&\multicolumn{1}{c}{Smile}&\multicolumn{1}{c}{Age}&\multicolumn{1}{c}{Gender}&\multicolumn{1}{c}{Glasses}\\
Pose & 1.000 & 0.150 & 0.039 & 0.133 & 0.051 \\
Smile & 0.150 & 1.000 & 0.001 & 0.170 & 0.233 \\
Age & 0.039 & 0.001 & 1.000 & 0.533 & 0.367 \\
Gender & 0.133 & 0.170 & 0.533 & 1.000 & 0.339 \\
Glasses & 0.051 & 0.233 & 0.367 & 0.339 & 1.000 \\ \hdashline \\
Avg & 0.093 & 0.115 & 0.235 & 0.293 & 0.247
\end{tabular} & \hspace{0.1in}
\begin{tabular}{l*{5}{R}}
&\multicolumn{1}{c}{pose}&\multicolumn{1}{c}{Smile}&\multicolumn{1}{c}{Age}&\multicolumn{1}{c}{Gender}&\multicolumn{1}{c}{Glasses}\\
Pose & 1.000 & 0.098 & 0.090 & 0.017 & 0.079 \\
Smile & 0.098 & 1.000 & 0.154 & 0.198 & 0.084 \\
Age & 0.090 & 0.154 & 1.000 & 0.565 & 0.600 \\
Gender & 0.017 & 0.198 & 0.565 & 1.000 & 0.363 \\
Glasses & 0.079 & 0.084 & 0.600 & 0.363 & 1.000 \\ \hdashline \\
Avg & 0.071 & 0.133 & 0.352 & 0.285 & 0.281
\end{tabular} \vspace{0.1in} \\
(c) StyleFlow & (d) Ours \vspace{0.04in} \\
\begin{tabular}{l*{5}{R}}
&\multicolumn{1}{c}{Pose}&\multicolumn{1}{c}{Smile}&\multicolumn{1}{c}{Age}&\multicolumn{1}{c}{Gender}&\multicolumn{1}{c}{Glasses}\\
Pose & 1.000 & 0.113 & 0.126 & 0.103 & 0.056 \\
Smile & 0.113 & 1.000 & 0.264 & 0.011 & 0.074 \\
Age & 0.126 & 0.264 & 1.000 & 0.581 & 0.494 \\
Gender & 0.103 & 0.011 & 0.581 & 1.000 & 0.338 \\
Glasses & 0.056 & 0.074 & 0.494 & 0.338 & 1.000 \\ \hdashline \\
Avg & 0.099 & 0.115 & 0.366 & 0.258 & 0.240
\end{tabular} & \hspace{0.1in}
\begin{tabular}{l*{5}{R}}
&\multicolumn{1}{c}{Pose}&\multicolumn{1}{c}{Smile}&\multicolumn{1}{c}{Age}&\multicolumn{1}{c}{Gender}&\multicolumn{1}{c}{Glasses}\\
Pose & 1.000 & 0.018 & 0.008 & 0.029 & 0.046 \\
Smile & 0.018 & 1.000 & 0.120 & 0.050 & 0.100 \\
Age & 0.008 & 0.120 & 1.000 & 0.388 & 0.363 \\
Gender & 0.029 & 0.050 & 0.388 & 1.000 & 0.203 \\
Glasses & 0.046 & 0.100 & 0.363 & 0.203 & 1.000 \\ \hdashline \\
Avg & 0.025 & 0.072 & 0.219 & 0.167 & 0.178
\end{tabular}
\end{tabular}
% }
\end{table*}
% #############
\textbf{Correlation between edited attributes:} We can assign an image with an attribute score by utilizing $\mathcal{C}_k$ as a feature extractor, and computing the distance between the feature vector $\textbf{v}$ and the classifying hyperplane, where the hyperplane is defined by $\left \{ \textbf{v} \mid \textbf{c}_k^T\textbf{v} + \textbf{t}_k = 0\right \}$. The attribute score of an image $\textbf{x}$ is then defined as
\begin{equation}
dist_k(\textbf{x}) = \frac{\textbf{c}_k^Tl(\textbf{x})+\textbf{t}_k}{|\textbf{c}_k|}
\end{equation}
where $l(\textbf{x})$ is the activation of the last hidden layer of $\mathcal{C}_k$.
To measure the disentanglement of the our model, we first generated an evaluation set consisting of $1K$ pairs of images and desired face attributes. The images are sampled directly from the StyleGAN generator and the attributes are generated independently for each image. We then manipulated the image to achieve an edited version, $\textbf{x}_{edit}$, by modifying the latent vector $\mathbf{w}\in \mathcal{W^+}$ along the learned directions and passed it to the generator (note that we change all attributes at once).
Next, we measured the difference in attribute score between the original and the edited images. For example, for smile, we defined the perceptual distance as $\Delta_{smile} = dist_{smile}(\textbf{x}_{edit}) - dist_{smile}(\textbf{x})$.
For the final score, we calculated the Pearson correlation between the perceptual distance of every two attributes. Every cell in \cref{tab:table1} holds the absolute value of the correlation, \eg, for smile and pose we find $\Delta_{smile}$, $\Delta_{pose}$ as the perceptual distance for the entire evaluation set, and then show the absolute value of $corr(pose,smile) = \left | \frac{cov(\Delta_{pose}, \Delta_{smile})}{E[\Delta_{pose}]E[\Delta_{smile}]} \right |$.
\Cref{tab:table1} shows the correlation results in different models. As previously mentioned~\cite{shen2020interfacegan,doubinsky2021multi, harkonen2020ganspaceli2021dystyle, shen2021closed}, we can see that some attributes are more correlated than others, \ie, glasses-age correlations are higher than smile-pose correlations across all the models. The last row of the table sums the off-diagonal column and represents the average correlation score. The lower scores our method gets indicate its disentanglement superiority.
The above-mentioned correlation was tested when all attributes were edited together. To measure the robustness of other attribute when only a single attribute is edited, we repeated the experiment but this time performed only one edit. We measured the perceptual distances (using the last layers in the classifiers) conditioned on one edit. As a result, we obtain a distance metric representing the change in each attribute.
The plots in \cref{fig:fig5} show the results for the different models. The x axis shows changes in an edited attribute and the y axis shows undesired changes in the correlated attribute. This inter-attribute effect is more noticeable as we apply larger edits to the image. We compare this side effect across all models. It can be seen that our model is the least to be affected, therefore, suggesting stronger disentanglement between attributes.
\begin{figure}[h]
\centering
\begin{tabular}{ccccc}
\includegraphics[width=.2\linewidth]{images/graphs/smile_pose.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/pose_smile.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/pose_age.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/pose_gender.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/pose_glasses.pdf} \\
\includegraphics[width=.2\linewidth]{images/graphs/age_pose.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/age_smile.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/smile_age.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/smile_gender.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/smile_glasses.pdf} \\
\includegraphics[width=.2\linewidth]{images/graphs/gender_pose.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/gender_smile.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/gender_age.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/age_gender.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/age_glasses.pdf} \\
\includegraphics[width=.2\linewidth]{images/graphs/glasses_pose.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/glasses_smile.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/glasses_age.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/glasses_gender.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/gender_glasses.pdf} \\
\hdashline \\
\includegraphics[width=.2\linewidth]{images/graphs/pose.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/smile.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/age.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/gender.pdf} &
\includegraphics[width=.2\linewidth]{images/graphs/glasses.pdf}
\end{tabular}
\caption{Comparisons of the inter-attribute effect between the different models. Upper rows show changes in one attribute as a function of changes in the edited attribute; bottom row summarizes all rows above.}
\label{fig:fig5}
\end{figure}
% #############
\textbf{Face Identity Preservation:} Similarly to the face identity score~\cite{abdal2021styleflow}, we evaluate the edited images using a pre-trained face embedding network~\cite{Geitgey2021}. Images were edited in identity preserving attributes only (pose, glasses and smile) and embedded using the network into a latent space where the Euclidean distance, $E_d$ and the cosine similarity, $C_s$ were measured between the original and edited images.
\Cref{tab:table2} presents the face identity preservation results. We can see that our technique outperforms all other models across all edits. Although when single edits are applied we see similar scores from StyleFlow, when all edits are performed together, our model tends to better preserve the identity.
% ######### id preservation ###########
\setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1}
\begin{table}
\centering
\caption{Identity preservation scores by different models.}
\label{tab:table2}
\begin{tabular}{lccccc}
\hline
Edit & Metric & SeFa & InterFaceGan & StyleFlow & Ours \\ \hline
\multirow{2}{*}{Smile} & $C_s \Uparrow$ & 0.970 & 0.978 & 0.989 & \textbf{0.991}\\
& $E_d \Downarrow$ & 0.329 & 0.276 & 0.192 & \textbf{0.174}\\ \hline
\multirow{2}{*}{Pose} & $C_s \Uparrow$ & 0.978 & 0.979 & 0.981 & \textbf{0.982}\\
& $E_d \Downarrow$ & 0.283 & 0.279 & 0.2662 & \textbf{0.253}\\ \hline
\multirow{2}{*}{Glasses} & $C_s \Uparrow$ & 0.966 & 0.978 & 0.984 & \textbf{0.985}\\
& $E_d \Downarrow$ & 0.348 & 0.270 & 0.227 & \textbf{0.215}\\ \hline
\multirow{2}{*}{All} & $C_s \Uparrow$ & 0.911 & 0.933 & 0.935 & \textbf{0.948}\\
& $E_d \Downarrow$ & 0.622 & 0.537 & 0.534 & \textbf{0.467}\\ \hline
\end{tabular}
\caption*{\textit{Notation: $C_s$ - Cosine Similarity; $E_d$ - Euclidean Distance.}}
\end{table}
% #################
\subsection{Ablation Study}
To test the significance of our orthogonality loss described in \cref{sec:train}, we conducted an ablation test. We trained a variant of our model in which we set $\lambda_{orth} = 0$. All other configurations including the losses and training procedure remain the same. After training, we performed consecutive edits starting from an original image; the results appear in \cref{fig:fig6}. We found that when applying multiple edits the images look sharper and suffer less from visual artifacts. We also performed the same quantitative tests as in \cref{sec:quantitative}. The results are shown in \cref{tab:table3,tab:table4}.
It can be seen that the latent vectors, found using our orthogonality loss, are less likely to affect other attributes, indicating a more disentangled model.
% ###################
\newcommand*{\ffsize}{0.18}
\begin{figure}[H]
\centering
\setlength{\tabcolsep}{1pt}
% \hspace*{-1.5cm}
\begin{tabular}{c c}
& $\lambda_{orth}=0$ \vspace{0.05in} \\
\rotatebox[]{90}{Pose} &
\begin{tabular}{cccc}
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_orig.png} \hspace{0.05in} & \includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_3_m1.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_4_m1.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_5_m1.png}
\end{tabular} \\
\rotatebox[]{90}{Smile} &
\begin{tabular}{cccc}
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_orig.png} \hspace{0.05in} & \includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_2_m1.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_3_m1.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_4_m1.png}
\end{tabular} \\
& $\lambda_{orth}=0.001$ \\
\rotatebox[]{90}{Pose} &
\begin{tabular}{cccc}
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_orig.png} \hspace{0.05in} & \includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_3_m2.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_4_m2.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_66_pose_5_m2.png}
\end{tabular} \\
\rotatebox[]{90}{Smile} &
\begin{tabular}{cccc}
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_orig.png} \hspace{0.05in} & \includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_2_m2.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_3_m2.png} \hspace{0.05in} &
\includegraphics[width=\ffsize\linewidth]{images/orth_comp/img_73_smile_4_m2.png}
\end{tabular}
\end{tabular}
\caption{Comparison of our model with and without orthogonality loss applied in training. First and third rows show improvement in pose-glasses entanglement, second and fourth rows show better identity preservation with consistent hair and shirt color.}
\label{fig:fig6}
\end{figure}
% ###################
\begin{table}[h]
\caption{Attribute correlation matrices with and without orthogonal loss. The values represent the average correlation between an attribute and the rest.}
\label{tab:table3}
\begin{tabular}{l|c|c|c|c|c}
\hline
Model & Pose & Smile & Age & Gender & Glasses \\
\hline
$\lambda_{orth}=0$ & 0.149 & 0.242 & 0.342 & 0.283 & 0.253 \\
\hline
$\lambda_{orth}=0.001$ & \textbf{0.025} & \textbf{0.072} & \textbf{0.219} & \textbf{0.167} & \textbf{0.178} \\ \hline
\end{tabular}
\end{table}
% ###################
\setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1}
\begin{table}[h]
\centering
\caption{Identity preservation comparison with and without orthogonal loss.}
\label{tab:table4}
\begin{tabular}{lccc}
\hline
Edit & Metric & $\lambda_{orth}=0$ & $\lambda_{orth}=0.001$ \\ \hline
\multirow{2}{*}{All} & $C_s \Uparrow$ & 0.932 & \textbf{0.948}\\
& $E_d \Downarrow$ & 0.536 & \textbf{0.467}\\ \hline
\end{tabular}
\caption*{\textit{Notation: $C_s$ - Cosine Similarity; $E_d$ - Euclidean Distance.}}
\end{table}
% ###################
\section{Conclusions}
In this work we proposed OrthoGAN, a disentangling generative network for disentangled multi-attribute image editing. We introduce an orthogonality loss that helps finding disentangled latent subspaces in StyleGAN's latent space that enable refined control over the generated images and their resulting face attributes. In addition, our model supports multifaceted edits for diverse image generation. We believe that such ideas may also be useful when applied during the training of the generator and can potentially offer additional improvements to disentangled models.
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{abdal2019image2stylegan}
Rameen Abdal, Yipeng Qin, and Peter Wonka.
\newblock Image2stylegan: How to embed images into the stylegan latent space?
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 4432--4441, 2019.
\bibitem{abdal2020image2stylegan++}
Rameen Abdal, Yipeng Qin, and Peter Wonka.
\newblock Image2stylegan++: How to edit the embedded images?
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 8296--8305, 2020.
\bibitem{abdal2021styleflow}
Rameen Abdal, Peihao Zhu, Niloy~J Mitra, and Peter Wonka.
\newblock Styleflow: Attribute-conditioned exploration of stylegan-generated
images using conditional continuous normalizing flows.
\newblock {\em ACM Transactions on Graphics (TOG)}, 40(3):1--21, 2021.
\bibitem{alaluf2022hyperstyle}
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano.
\newblock Hyperstyle: Stylegan inversion with hypernetworks for real image
editing.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 18511--18521, 2022.
\bibitem{alaluf2021hyperstyle}
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit~H Bermano.
\newblock Hyperstyle: Stylegan inversion with hypernetworks for real image
editing.
\newblock {\em arXiv preprint arXiv:2111.15666}, 2021.
\bibitem{albiero2021img2pose}
Vitor Albiero, Xingyu Chen, Xi Yin, Guan Pang, and Tal Hassner.
\newblock img2pose: Face alignment and detection via 6dof, face pose
estimation.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 7617--7627, 2021.
\bibitem{bojanowski2017optimizing}
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam.
\newblock Optimizing the latent space of generative networks.
\newblock {\em arXiv preprint arXiv:1707.05776}, 2017.
\bibitem{doubinsky2021multi}
Perla Doubinsky, Nicolas Audebert, Michel Crucianu, and Herv{\'e}~Le Borgne.
\newblock Multi-attribute balanced sampling for disentangled gan controls.
\newblock {\em arXiv preprint arXiv:2111.00909}, 2021.
\bibitem{Geitgey2021}
A. Geitgey.
\newblock face\_recognition.
\newblock \url{https://github.com/ageitgey/face_recognition}, 2021.
\bibitem{goetschalckx2019ganalyze}
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola.
\newblock Ganalyze: Toward visual definitions of cognitive image properties.
\newblock In {\em Proceedings of the ieee/cvf international conference on
computer vision}, pages 5744--5753, 2019.
\bibitem{goodfellow2014generative}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative adversarial nets.
\newblock In {\em Advances in neural information processing systems}, pages
2672--2680, 2014.
\bibitem{harkonen2020ganspaceli2021dystyle}
Erik H{\"a}rk{\"o}nen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.
\newblock Ganspace: Discovering interpretable gan controls.
\newblock {\em Advances in Neural Information Processing Systems},
33:9841--9850, 2020.
\bibitem{he2018harnessing}
Keke He, Yanwei Fu, Wuhao Zhang, Chengjie Wang, Yu-Gang Jiang, Feiyue Huang,
and Xiangyang Xue.
\newblock Harnessing synthesized abstraction images to improve facial attribute
recognition.
\newblock In {\em IJCAI}, pages 733--740, 2018.
\bibitem{ho2020denoising}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock {\em Advances in Neural Information Processing Systems},
33:6840--6851, 2020.
\bibitem{jahanian2019steerability}
Ali Jahanian, Lucy Chai, and Phillip Isola.
\newblock On the" steerability" of generative adversarial networks.
\newblock {\em arXiv preprint arXiv:1907.07171}, 2019.
\bibitem{karkkainenfairface}
Kimmo Karkkainen and Jungseock Joo.
\newblock Fairface: Face attribute dataset for balanced race, gender, and age
for bias measurement and mitigation.
\newblock In {\em Proceedings of the IEEE/CVF Winter Conference on Applications
of Computer Vision}, pages 1548--1558, 2021.
\bibitem{karras2021alias}
Tero Karras, Miika Aittala, Samuli Laine, Erik H{\"a}rk{\"o}nen, Janne
Hellsten, Jaakko Lehtinen, and Timo Aila.
\newblock Alias-free generative adversarial networks.
\newblock {\em Advances in Neural Information Processing Systems}, 34, 2021.
\bibitem{karras2019style}
Tero Karras, Samuli Laine, and Timo Aila.
\newblock A style-based generator architecture for generative adversarial
networks.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 4401--4410, 2019.
\bibitem{karras2020analyzing}
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and
Timo Aila.
\newblock Analyzing and improving the image quality of stylegan.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 8110--8119, 2020.
\bibitem{kingma2013auto}
Diederik~P Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock {\em arXiv preprint arXiv:1312.6114}, 2013.
\bibitem{lipton2017precise}
Zachary~C Lipton and Subarna Tripathi.
\newblock Precise recovery of latent vectors from generative adversarial
networks.
\newblock {\em arXiv preprint arXiv:1702.04782}, 2017.
\bibitem{locatello2019challenging}
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly,
Bernhard Sch{\"o}lkopf, and Olivier Bachem.
\newblock Challenging common assumptions in the unsupervised learning of
disentangled representations.
\newblock In {\em international conference on machine learning}, pages
4114--4124. PMLR, 2019.
\bibitem{pidhorskyi2020adversarial}
Stanislav Pidhorskyi, Donald~A Adjeroh, and Gianfranco Doretto.
\newblock Adversarial latent autoencoders.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 14104--14113, 2020.
\bibitem{radford2015unsupervised}
Alec Radford, Luke Metz, and Soumith Chintala.
\newblock Unsupervised representation learning with deep convolutional
generative adversarial networks.
\newblock {\em arXiv preprint arXiv:1511.06434}, 2015.
\bibitem{richardson2021encoding}
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav
Shapiro, and Daniel Cohen-Or.
\newblock Encoding in style: a stylegan encoder for image-to-image translation.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 2287--2296, 2021.
\bibitem{roich2021pivotal}
Daniel Roich, Ron Mokady, Amit~H Bermano, and Daniel Cohen-Or.
\newblock Pivotal tuning for latent-based editing of real images.
\newblock {\em arXiv preprint arXiv:2106.05744}, 2021.
\bibitem{shen2020interfacegan}
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.
\newblock Interfacegan: Interpreting the disentangled face representation
learned by gans.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
2020.
\bibitem{shen2021closed}
Yujun Shen and Bolei Zhou.
\newblock Closed-form factorization of latent semantics in gans.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 1532--1540, 2021.
\bibitem{tewari2020stylerig}
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter
Seidel, Patrick P{\'e}rez, Michael Zollhofer, and Christian Theobalt.
\newblock Stylerig: Rigging stylegan for 3d control over portrait images.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 6142--6151, 2020.
\bibitem{thomee2016yfcc100m}
Bart Thomee, David~A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,
Douglas Poland, Damian Borth, and Li-Jia Li.
\newblock Yfcc100m: The new data in multimedia research.
\newblock {\em Communications of the ACM}, 59(2):64--73, 2016.
\bibitem{tov2021designing}
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.
\newblock Designing an encoder for stylegan image manipulation.
\newblock {\em ACM Transactions on Graphics (TOG)}, 40(4):1--14, 2021.
\end{thebibliography}