CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer
Extracted source text ยท arXiv 2311.06443
% WACV 2024 Paper Template
% based on the CVPR 2023 template (https://media.icml.cc/Conferences/CVPR2023/cvpr2023-author_kit-v1_1-1.zip) with 2-track changes from the WACV 2023 template (https://github.com/wacv-pcs/WACV-2023-Author-Kit)
% based on the CVPR template provided by Ming-Ming Cheng (https://github.com/MCG-NKU/CVPR_Template)
% modified and extended by Stefan Roth (stefan.roth@NOSPAMtu-darmstadt.de)
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage[review,algorithms]{wacv} % To produce the REVIEW version for the algorithms track
%\usepackage[review,applications]{wacv} % To produce the REVIEW version for the applications track
\usepackage{wacv} % To produce the CAMERA-READY version
%\usepackage[pagenumbers]{wacv} % To force page numbers, e.g. for an arXiv version
% Include other packages here, before hyperref.
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage[accsupp]{axessibility}
% % additional Package
% \usepackage{graphicx}
% \usepackage{subfigure}
\usepackage{multirow}
% \usepackage{xcolor}
% \usepackage{booktabs}
\usepackage{adjustbox}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues, e.g. with the
% file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete
% ReviewTempalte.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you
% should be clear).
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\wacvPaperID{216} % *** Enter the WACV Paper ID here
\def\confName{WACV}
\def\confYear{2024}
\begin{document}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer}
% \author{First Author\\
% Institution1\\
% Institution1 address\\
% {\tt\small firstauthor@i1.org}
% % For a paper whose authors are all at the same institution,
% % omit the following lines up until the closing ``}''.
% % Additional authors and addresses can be added with ``\and'',
% % just like the second author.
% % To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\tt\small secondauthor@i2.org}
% }
\author{Haoyu Ma, \hspace{1pt} Tong Zhang, \hspace{1pt} Shanlin Sun, \hspace{1pt} Xiangyi Yan, \hspace{1pt} Kun Han, \hspace{1pt} Xiaohui Xie \\
University of California, Irvine \\
\{haoyum3, tongz27, shanlins, xiangyy4, khan7, xhx\}@uci.edu}
\maketitle
%%%%%%%%% ABSTRACT
\begin{abstract}
Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities.
%
In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVTHead considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios. Codes and pre-trained model can be found at \href{https://github.com/HowieMa/CVTHead}{https://github.com/HowieMa/CVTHead}.
\end{abstract}
\section{Introduction}
% Multi-view
Personalized head avatars play a crucial role in a wide range of applications, including AR/VR, teleconferencing, and the movie industry. Over the past few decades, there has been an extensive exploration of personalized head avatars in the fields of computer graphics and computer vision.
%
Traditional solution \cite{alexander2010digital} reconstructs a personalized mesh and texture for the source actor explicitly with 3D head scans \cite{yang2020facescape,wuu2022multiface}.
To perform full face control, 3D Morphable Models (3DMM) \cite{blanz1999morphable,li2017learning} are used as a strong prior of face geometry.
3DMMs is a parametric model and uses PCA-based linear blendshapes to explicitly control face shape, expressions, texture, and head pose independently. However, 3DMM does not model the facial detail and hair region of the human face \cite{grassal2022neural}.
%
Recently, with the development of Neural Radiance Fields (NeRF) \cite{mildenhall2020nerf}, reconstructing avatars with implicit models becomes popular as it can reconstruct detailed regions \cite{gafni2021dynamic,park2021nerfies,athar2022rignerf}.
However, all these methods are subject-specific and they usually require video inputs or multi-view images of the same subject, which limits their usage in practice.
% P2: one-shot face animation
Hence, acquiring human avatars from a single image (i.e., one-shot face reenactment) becomes more and more popular \cite{wiles2018x2face,thies2020neural,zhang2023sadtalker,siarohin2019first,zakharov2020fast,wang2021one,thies2016face2face,yao2020mesh,doukas2021headgan,khakhulin2022realistic}. Given a facial image of an actor, the synthetic images can be driven by videos from other actors. A key step behind these methods is to decouple the facial appearance and motion information from the source and driven images.
As a result, mesh-guided face animation has gained significant attention, primarily due to the inherent disentanglement of identity and expression offered by 3DMM.
%
Generally, one-shot mesh-guided face animation can be roughly divided into warp-based and graphics-based.
Warp-based methods \cite{yao2020mesh,doukas2021headgan,yang2022face2face,zeng2022fnevr} employ the motion field to transfer the driving pose and expression into the source face.
These methods effectively preserve fine facial details and produce high-fidelity results but only work well for a limited range of head poses.
%
Graphic-based methods \cite{feng2021learning,khakhulin2022realistic} learn texture maps \cite{blinn1976texture} from single-image and apply computer graphics pipelines to render the animated face image. Thus, it can maintain performance under large head rotations and guarantee the 3D consistency of rendered images.
However, the rendering pipeline is usually computationally heavy \cite {kato2020differentiable}, which makes efficient rendering unachievable.
% Point-rendering
Alternatively, point-based graphics \cite{gross2011point} get rid of the surface mesh and directly use point clouds to model the 3D geometry. Later on, the point-based neural rendering techniques \cite{aliev2020neural,rakhimov2022npbg++,xu2022point} augment each RGB point with a learnable neural descriptor that is interpreted by the neural renderer. The recent SMPLpix \cite{prokudin2021smplpix} further extends these techniques from static scenes to dynamic scenes and enables the efficient rendering of human body avatars under novel subject identities and human poses. Although efficient, these methods still require multi-view images with calibrated cameras to reconstruct the accurate point cloud first.
Consequently, applying these methods to one-shot face reenactment is infeasible.
% Our contributions
In this paper, we utilize point-based neural rendering to achieve an efficient and realistic generation of head avatars from a single image.
We direct utilize the sparse vertices from the FLAME head model \cite{li2017learning} as our point set, instead of reconstructing a dense point cloud of the subject tediously.
Specifically, given the vertices from pre-trained 3D face reconstruction networks \cite{feng2021learning}, we learn a local feature descriptor aligned with each vertex.
When learning the local descriptor of a 3D point from a 2D reference image, pixel-aligned features \cite{saito2019pifu,huang2020arch,he2021arch++,corona2023structured} are a popular choice. However, these aligned features often become incorrect when the projected 2D location is occluded in the source image.
To address this challenge, we propose the \textit{Vertex-feature Transformer}. This approach treats each vertex as a query token \cite{dosovitskiy2021an} and utilizes transformers \cite{vaswani2017attention} to directly learn the canonical vertex features from the reference image. By incorporating a global attention mechanism, our model can capture long-range dependencies within the features of all vertices. Thus, the feature descriptor of invisible 3D points can still be reconstructed correctly.
Next, we project the feature descriptor and depth of each vertex into image space and employ a UNet-like neural rendering to generate the RGB image.
Since the feature descriptor is aligned with the vertices of the FLAME head model, the rendered face can be explicitly controlled by the shape, expression, and head pose coefficients. We name this end-to-end framework as CVTHead, in short for Controllable head avatar with Vertex-feature Transformer.
Our major contributions are summarized as follows:
\begin{itemize}
\item We propose CVTHead, a one-shot controllable head avatar framework using point-based neural rendering that can efficiently render novel human heads under novel expressions and camera views. To the best of our knowledge, this is the first work that performs point-based neural rendering from a monocular face image.
\item We propose Vertex-feature Transformer to learn the vertex descriptor in canonical space from a single image with transformers, and demonstrate its superiority beyond projection methods.
\item By conducting experiments on VoxCeleb1 and VoxCeleb2, we establish that our method achieves performance that is on par with the state-of-the-art approaches, while additionally improving efficiency.
\end{itemize}
\section{Related Work}
\paragraph{Mesh-guided Face Reenactment}
Extensive research has been conducted on employing 3DMM for the explicit animation of human face images \cite{tewari2020stylerig,ghosh2020gif,yao2020mesh,doukas2021headgan,wang2021safa,yang2022face2face,khakhulin2022realistic, Drobyshev22MP,zeng2022fnevr}.
Mesh-guided face reenactment can be divided into warp-based and graphic-based.
Warping-based methods \cite{yao2020mesh,doukas2021headgan,wang2021safa,yang2022face2face} warp the source image with explicit motion fields.
For example, given both source and driving meshes, Yao et al. \cite{yao2020mesh} extract the motion features with Graph Convolutional Networks. HeadGAN \cite{doukas2021headgan} learns the dense flow field with PNCC \cite{zhu2016face} and SPADE. Face2Face$^\rho$ \cite{yang2022face2face} calculates the motion with a set of pre-specified 3D keypoints.
However, when faced with significant head rotations, the quality of these approaches drops significantly.
Meanwhile, other methods \cite{feng2021learning,khakhulin2022realistic,grassal2022neural} obtain the animated face images from the head mesh with the classic graphics rendering pipeline. In detail, DECA \cite{feng2021learning} simultaneously learns both the head mesh and the linear albedo subspace of the Basel Face Model \cite{paysan20093d}.
To create realistic face photos, ROME \cite{khakhulin2022realistic} estimates a neural texture and offset for each vertex from the source image and renders the rigged mesh with deferred neural rendering technique \cite{thies2019deferred}.
Nevertheless, these methods still require the time-consuming classic differentiable rendering \cite{ravi2020accelerating}.
\begin{figure*}[!t]
\centering
\includegraphics[width=0.95 \linewidth]{CVTHead.pdf}
\caption{\small{Overview of the CVTHead framework. We employ a pretrained face reconstruction network \cite{feng2021learning} to obtain the face mesh (Section \ref{sec:mesh_recon}) and utilize the proposed verte feature transformers to obtain the feature descriptor of each vertex from the source image (Section \ref{sec:vtx_transformer}). We then consider the sparse vertex as point set and use point-based neural rendering to synthesize the image (Section \ref{sec:neural_rendering}). } }
\label{fig:overview}
\end{figure*}
\paragraph{Neural Head Avatars}
Recently, several works extend NeRF \cite{aliev2020neural} to model dynamic objects such as virtual avatars with implicit neural representations \cite{gafni2021dynamic,park2021nerfies,kania2022conerf,athar2022rignerf,hong2022headnerf,zhuang2022mofanerf}. For example, subject-dependent methods such as NerRACE \cite{gafni2021dynamic} and RigNeRF \cite{athar2022rignerf} use 3DMM-guided deformation neural fields to enable control over head pose, facial expression, and viewpoints. Typically, these approaches employ an optimization-based head tracker \cite{thies2016face2face} as a preprocessing step to extract accurate 3DMM coefficients from a monocular video of the subject. Meanwhile, subject-agnostic methods such as HeadNeRF \cite{hong2022headnerf} and MofaNeRF \cite{zhuang2022mofanerf} learn the radiance fields from large-scale multi-view images \cite{yang2020facescape}. When generating an avatar for a novel subject, these methods require time-consuming inverse rendering optimization to obtain the latent codes.
Most recently, HiDe-NeRF \cite{li2023one} and OTAvatar \cite{ma2023otavatar} employ tri-plane representations \cite{chan2022efficient} to efficiently extract multi-scale features for each query 3D point and also use volume rendering to reconstruct images. Although promising, a limitation of volumetric rendering is the necessity to sample hundreds of 3D points per ray and then feed them through the network to render a single pixel or feature patch. In contrast, our method utilizes a set of points to represent the head avatar, thereby necessitating only a single forward pass for rendering.
\paragraph{Neural Point-based Rendering}
Neural point-based rendering \cite{aliev2020neural,kopanas2021point,rakhimov2022npbg++,xu2022point,kerbl3Dgaussians} has gained significant attention in recent years for its ability to generate high-quality images by directly rendering point clouds from static scenes. Aliev et al. \cite{aliev2020neural} introduced Neural Point-Based Graphics (NPBG), which employs learnable neural descriptors to enhance each point for better rendering. Later on, NPBG++ \cite{rakhimov2022npbg++} further predicts the descriptors with a single pass to accelerate rendering.
Meanwhile, SMPLpix \cite{prokudin2021smplpix} extends point-based neural rendering to generate human avatars under the control of SMPL \cite{loper2015smpl}. Although SMPLpix is a subject-agnostic method, it requires registering the SMPL model to ground truth 3D scans to obtain the RGB color of each vertex.
The most recent PointAvatar \cite{zheng2023pointavatar} models the human head as an explicit canonical point cloud and continuous deformation to create realistic and relightable head avatars. However, it is still subject-dependent and requires a video caption of the subject to train the model. Our method also employs neural point-based rendering but simplifies the setting as we only require a single image of the novel subject.
\paragraph{Transformers in Mesh}
Over the past few years, transformers \cite{vaswani2017attention} have made significant progress in many computer vision tasks \cite{touvron2020training,dosovitskiy2021an,carion2020end, yan2022after,ma2022ppt,yang2023capturing}.
There are a few works that also apply transformers to mesh data \cite{lin2020end,lin2021mesh,cho2022FastMETRO,yoshiyasu2023deformable,zheng2023potter,dou2023tore}. In detail, METRO \cite{lin2020end} apply transformers to predict the mesh coordinates and 3D joints simultaneously of the human body \cite{loper2015smpl}.
Mesh Graphormer \cite{lin2021mesh} further utilizes the topology of the mesh with graph convolution to improve the mesh reconstruction. Both works consider each vertex as a query token and use transformers to learn the non-local relationships among vertices.
In our work, we also use transformers to learn the correspondence among vertex features.
% \vspace{-1.0em}
\section{Methodology}
\vspace{-0.2em}
\subsection{Overview}
\vspace{-0.2em}
Fig \ref{fig:overview} illustrates the overall framework of our CVTHead. Given both source image $\mathbf{I^s}$ and driven image $\mathbf{I^d}$, we utilize a pre-trained face reconstruction model \cite{feng2021learning} to obtain the source and driven vertex coordinates $\mathbf{V^s}$ and $\mathbf{V^d} \in \mathbb{R}^{N \times 3} $ of the FLAME model \cite{li2017learning} (Sec. \ref{sec:mesh_recon}), where $N$ is the number of vertices in FLAME model.
Simultaneously, we employ the proposed vertex feature transformer to learn the feature descriptor for all vertices in canonical space $\mathbf{V_F} \in \mathbb{R}^{N \times C}$ from the source image (Sec. \ref{sec:vtx_transformer}), where $C$ is the number of channels of feature descriptor.
Then we project driven vertices and their corresponding feature descriptors onto the vertex feature image $\mathbf{P_F^d} \in \mathbb{R}^{H \times W \times C}$ and the depth image $\mathbf{P_D^d} \in \mathbb{R}^ {H \times W \times 1}$, where $H$ and $W$ is the height and width of the original image. Next, we conduct neural rendering with a U-Net $\mathcal{G(\cdot)}$ to generate the synthetic image $\mathbf{\hat{I}} = \mathcal{G}(\mathbf{P_F}, \mathbf{P_D}) \in \mathbb{R}^{H \times W \times 3}$ (Sec. \ref{sec:neural_rendering}).
%
%
Our framework enables end-to-end training, allowing the entire process to be optimized jointly. During inference, our system enables the rendered image to be animated with novel shapes, expressions, head poses, and viewpoints by manipulating the FLAME parameters. This flexibility allows for the generation of diverse and customizable head avatars.
\subsection{Head Mesh Reconstruction}
\label{sec:mesh_recon}
FLAME \cite{li2017learning} is a parametric 3D head model with $N=5023$ vertices. It encompasses a mean template $V_{b} \in \mathbb{R}^{N \times 3}$, along with shape blendshapes $ \mathcal{S} \in \mathbb{R}^{N \times 3 \times L}$, and expression blendshapes $ \mathcal{E} \in \mathbb{R}^{N \times 3 \times K}$.
These blendshapes are derived from a vast collection of 4D scans of human heads, allowing FLAME to capture a wide range of facial variations.
Given parameters of facial identity $\beta \in \mathbb{R}^L$, expression $\phi \in \mathbb{R}^K$ and pose $\theta \in \mathbb{R}^{3k+3}$ (with $k=4$ joints for neck, jaw, and eyeballs), FLAME first apply $\beta$ and $\phi$ to corresponding blendshapes, resulting in modified vertex positions.
Next, the linear blend skinning (LBS) technique $W(\cdot, \cdot)$ is employed to rotate the vertices based on $\theta$. The final reconstruction of FLAME in world coordinates is calculated by:
\begin{equation}
M(\beta, \phi, \theta) = W(V_b + \mathcal{S} \beta + \mathcal{E} \phi, \theta) \in \mathbb{R}^{3n}
\end{equation}
We employ the pre-trained DECA \cite{feng2021learning} $f_D(\cdot)$ to obtain $\beta$,$\phi$,$\theta$ and camera parameters $c$ from both source images and driven images with a single forward, i.e, $\beta^s, \phi^s, \theta^s, c^s = f_D( \mathbf{I^s} )$ and $\beta^d, \phi^d, \theta^d, c^d = f_D(\mathbf{I^d})$.
We also obtain the deformation of hair and shoulder regions from the source image with the pre-trained linear deformation model $f_H(\cdot)$ \cite{khakhulin2022realistic} to refine the vertices locations.
Then we obtain the driven vertex coordinates by
\begin{equation}
\mathbf{V^d} = M(\beta^s, \phi^d, \theta^d) + f_H(\mathbf{I^s}) \in \mathbb{R}^{N \times 3}
\end{equation}
\subsection{Vertex-feature Transformer}
\label{sec:vtx_transformer}
\paragraph{Motivations}
In previous approaches that utilize pixel-aligned features \cite{saito2019pifu,corona2023structured}, the feature descriptor of a given 3D point is determined by the feature located at its corresponding 2D projection. In detail, given the 3D point $\mathbf{k}^s \in \mathbf{V^s}$, we project it into the 2D image space by $(u^s, v^s, d^s) = \Pi(\mathbf{k}^s, c_s)$, where $\Pi(\cdot)$ represents the orthographic projection function and $c_s$ is the camera parameters of the reference image obtained from pre-trained DECA. The descriptor of $\mathbf{k}^s$ is defined as $I'[u^s, v^s]$, where $I'$ is the 2D feature map of the source image. However, these methods have several limitations.
First, it requires accurate mesh reconstruction to locate the correct 2D pixels. Moreover, when the point is invisible, the feature at the 2D projection cannot represent the real features of that point. For instance, if the ear is occluded by the face, the projection may result in capturing features from the eye or nose instead. As a result, relying solely on the feature at the 2D projection can lead to incomplete or misleading feature descriptors.
\paragraph{Vertex Feature as Tokens}
To tackle the aforementioned problem, we propose a solution wherein we treat each vertex as an individual query token and leverage the attention mechanism of transformers to acquire its corresponding features from the image feature tokens. This approach avoids the need for a fixed 2D projection and allows for more flexible learning.
%
%
Specifically, we employ $N$ learnable embedding vectors $\mathbf{X_v} \in \mathbb{R}^{N \times C'}$ to represent the feature descriptors associated with each vertex in canonical space and name it as \textit{Vertex Tokens}, where $C'$ is the number of channels.
To further encode the location information of each vertex, we incorporate the sine positional encoding \cite{vaswani2017attention} to its corresponding image space coordinates $(u^s, v^s)$ and depth $d^s$, denoting as $\mathbf{E^s_{uv}}$ and $\mathbf{E^s_{dep}}$, respectively.
Finally, the vertex query token is defined as $\mathbf{\Tilde{X}_v} = \mathbf{X_v}+ \mathbf{E^s_{uv}} + \mathbf{E^s_{dep}}$.
%
%
On the other hand, we train a CNN encoder $\mathcal{E}(\cdot)$ to extract feature maps from the source image $\mathbf{I^s}$ and flatten the 2D features into a sequence of tokens $\mathbf{F^s} = \mathcal{E}(\mathbf{I^s}) \in \mathbb{R}^{hw \times C' }$. We also apply the 2D sine positional encodings \cite{dosovitskiy2021an} to encode spatial information, denoted as $\mathbf{E}$. Finally, the image token is defined as $\mathbf{X^s_F} = \mathbf{F^s} + \mathbf{E}$.
\paragraph{Transformers}
The input to the transformer is the concatenation of both image tokens $\mathbf{X^s_F}$ and vertex tokens $\mathbf{\Tilde{X}_v}$, i.e, $\mathbf{X} = [\mathbf{\Tilde{X}_v}, \mathbf{X^s_F}] \in \mathbb{R}^{(N+hw) \times C'}$.
The standard transformer encoder layer \cite{vaswani2017attention} consists of alternating layers of the multi-headed self-attention (MHSA) and multi-layer perceptron (MLP).
First, three linear projections are applied to transfer $\mathbf{X}$ into three matrices of equal size, namely the query $\mathbf{Q}$, the key $\mathbf{K}$, and the value $\mathbf{V}$.
The self-attention is calculated by:
\begin{equation}
\text{SA}(\mathbf{X}) = \text{Softmax}( \frac{ \mathbf{Q} \mathbf{K}^T }{\sqrt{D}} )\mathbf{V},
\label{eq:attention}
\end{equation}
For MHSA, $H$ self-attention modules are applied to $\mathbf{X}$ separately, and each of them produces an output sequence. We utilize the state of the vertex tokens at the output of the transformer encoder and employ a linear transformation to modify its dimensionality, thereby acquiring the vertex descriptor $\mathbf{V_F} \in \mathbb{R}^{N \times C}$.
The vertex-feature transformer has several benefits. Firstly, it eliminates the need for a fixed 2D projection to determine the corresponding feature for each vertex. Instead, it leverages attention mechanisms to identify the relevant feature, introducing a higher degree of flexibility. The transformer incorporates positional encoding to encode location information, further enhancing its adaptability and versatility.
%
Additionally, the global attention mechanism of transformers facilitates long-range correspondence among all vertex features. Even when the projection of a vertex is occluded, the vertex feature can still be obtained from neighboring regions or symmetrical vertices.
\subsection{Neural Vertex Rendering}
\label{sec:neural_rendering}
Given the learned vertex feature $\mathbf{V_F}$, we further use neural point-based rendering to generate synthetic images.
During the training, we use the driven vertex $\mathbf{V}^d$ to reconstruct the driven image. In detail, we first project the driven vertices $ \mathbf{k}^d \in \mathbf{V^d}$ into image space with the driven camera parameter $c^d$, i.e., $(u^d,v^d, d^d) = \Pi (\mathbf{k}^d, c^d)$.
Subsequently, we create the vertex projection features $ \mathbf{P_F^d} \in \mathbb{R}^{H \times W \times C}$. For each vertex $\mathbf{k}^d$, along with its corresponding descriptor $\mathbf{v_F} \in \mathbb{R}^{C}$, we assign the descriptor to location $(u^d,v^d)$ in the vertex projection features \cite{prokudin2021smplpix}:
\begin{equation}
\mathbf{P_F^d}[ \lfloor u^d \rfloor, \lfloor v^d \rfloor ] = \mathbf{v_F}
\end{equation}
We keep the features of the nearest vertex when two vertices are projected into the same pixel on $\mathbf{P_F^d}$. For all pixels without projection (i.e., the background pixel), we assign a constant value. Similarly, we also project the depth $d^d$ value into a depth image $P_D$ which satisfies $\mathbf{P_D^d}[ \lfloor u^d \rfloor, \lfloor v^d \rfloor ] = d^d$.
Finally, we concatenate $\mathbf{P_F^d}$ and $\mathbf{P_D^d}$ and employ a U-Net $\mathcal{G}(\cdot)$ to generate the synthetic image $\hat{\mathbf{I^d}}$ as well as the binary foreground mask $\hat{\mathbf{M^d}}$, i.e.,
\begin{equation}
(\hat{\mathbf{I^d}}, \hat{\mathbf{M^d}}) = \mathcal{G}([\mathbf{P_F^d}, \mathbf{P_D^d}]).
\end{equation}
\subsection{Training}
During the training time, we randomly sample $\mathbf{I^s}$ and $\mathbf{I^d}$ from the same video.
We fixed the pre-trained DECA and only update the parameters of vertex-feature transformers and neural render.
Following \cite{khakhulin2022realistic}, we use the L1 loss $L_{L1}$, VGG perceptual loss $L_{vgg}$ \cite{johnson2016perceptual}, face recognition loss $L_{id}$ \cite{cao2018vggface2}, and adversarial loss \cite{NIPS2014_5ca3e9b1, wang2018high} $L_{a}$ to measure the difference between the reconstructed driven image $\hat{\mathbf{I^d}}$ and the ground truth $\mathbf{I^d}$. We use the Dice loss to match the predicted segmentation masks. The total loss is calculated by:
\begin{equation}
L = \lambda_{L1} L_{L1} + \lambda_{vgg} L_{vgg} + \lambda_{id} L_{id} + \lambda_{seg} L_{seg} + \lambda_{a} L_{a}
\end{equation}
, where $\lambda_{L1}$, $\lambda_{vgg}$, $\lambda_{id}$, $\lambda_{seg}$ and $\lambda_{a}$ is the corresponding weights of each loss term.
\section{Experiments}
\subsection{Experimental Set up}
\paragraph{Dataset}
For a fair comparison with previous works, we conduct experiments on VoxCeleb1 \cite{nagrani2017voxceleb} and VoxCeleb2 \cite{chung2018voxceleb2}.
VoxCeleb1 contains around 20k video sequences of over 1000 actors and VoxCeleb2 contains around 150k videos of over 6000 actors.
Note that, ROME \cite{khakhulin2022realistic} carefully selects a subset of around 15k high-quality video sequences from VoxCeleb2 for training and evaluation, which is not publicly available. We directly use all VoxCeleb2 videos instead.
Following \cite{siarohin2019first}, each frame is cropped into $256 \times 256$ and normalized to $[-1, 1]$.
We follow the identity-based split thus all subjects in the validation set are unseen by the model.
Besides, we apply an off-the-shelf face parsing network \cite{yu2018bisenet} to obtain the foreground mask of each frame, which is considered as the pseudo ground truth.
\paragraph{Implementation Details}
We use the same CNN encoder $\mathcal{E}(\cdot)$ as in ROME \cite{khakhulin2022realistic}, which downsamples $16\times$ of the original image.
Naturally, our vertex-feature transformer is able to process arbitrary sizes of mesh. However, due to the quadratic computation complexity w.r.t. the sequence length of the transformer, it's hard to model all $N=5023$ tokens. Thus, we use the coarse mesh of the FLAME model with $N'=314$ tokens in our vertex-feature transformer and use the decoder of Spiralnet++ \cite{gong2019spiralnet++} to upsample the vertex features after the transformer, which serializes the neighboring vertices based on triangular meshes.
Our vertex-feature transformer has $6$ transformer encoder layers and the head of MHSA is set to $4$. The feature dimension is set to $C'=128$ and $C=32$.
%
%
Our model is implemented using PyTorch and optimized with the Adam optimizer \cite{kingma2014adam} for a duration of $200$ epochs. The learning rate is set to $1e-4$ and the batch size is set to $16$.
$\lambda_{L1}$, $\lambda_{vgg}$, and $\lambda_{seg}$ are set to $1.0$, and $\lambda_{id}$ and $\lambda_{a}$ are set to $0.1$.
% The weights of each loss are set to $\lambda_{L1}=\lambda_{vgg} = \lambda_{seg} = 1.0$ and $\lambda_{id} = \lambda_{a} = 0.1$.
\paragraph{Metrics}
Following previous works \cite{khakhulin2022realistic}, we evaluate our CVTHead on both self-reenactment and cross-identity reenactment.
In self-reenactment, the source and driving image come from the same video. In this scenario, the driving image can be viewed as the ground truth. We use the following metrics to measure the reconstruction quality between the driving image and the synthesized results: (1)L1 loss on the masked region; (2) peak signal-to-noise ratio (PSNR); (3) learned perceptual image patch similarity (LPIPS) with pre-trained AlexNet \cite{zhang2018unreasonable}, and (4) multi-scale structured similarity (MS-SSIM).
%
%
In the cross-identity reenactment, the source and driven image come from different subjects. Given the source image of one subject, We random sample a different subject in the validation set as the driving image. This evaluation requires the model to fully disentangle the identity and expression information. Since ground truth is unavailable, this task can only be evaluated by some proxy metrics. In detail, we use (1) FID \cite{heusel2017gans} to evaluate the image realism; (2) CSIM \cite{zakharov2019few}, which measures the cosine similarity of the identity embeddings from a pre-trained model between the source image and the synthesized image; and (3) image quality assessment (IQA) \cite{su2020blindly}
\begin{figure}[!ht]
\centering
\includegraphics[width=0.9 \linewidth]{self_recon2.pdf}
\caption{Qualitative comparisons of self-reenactment on VoxCeleb1. The 1st column is the source image. The 2nd column is the driving image, which can be considered as the ground truth. The 3rd column is the results from ROME, and the 4th column is the result from our CVTHead. }
\label{fig:self_reco}
\end{figure}
\subsection{Results of talking-face synthesis}
We first evaluate the performance of our method on talking-face synthesis.
To the best of our knowledge, ROME \cite{khakhulin2022realistic} is the only method that share the same setting with our method, i.e., one-shot mesh-based face reenactment based on graphics without warping field. Thus, we mainly compare our method with ROME \cite{khakhulin2022realistic}. Besides, we also compare with warping-based methods including First-Order Motion Model (FOMM) \cite{siarohin2019first} and the Bi-Layer \cite{zakharov2020fast}.
\begin{table}[!ht]
% >>>>>>>>>>>>>>>>>>>>>>>>>>> self >>>>>>>>>>>>>>>>>>>>>>>>>>>
\centering
\resizebox{0.99\linewidth}{!}{
\begin{tabular}{l|c|c|c|c}
\toprule
Dataset & \multicolumn{4}{c}{VoxCeleb1} \\
\midrule
Method & L1 $\downarrow$ & PSNR $\uparrow$ & LPIPS $\downarrow$ & MS-SSIM $\uparrow$ \\
\midrule
FOMM \cite{siarohin2019first} & 0.048 & 22.43 & 0.139 & 0.836 \\
% Bi-Layer \cite{zakharov2020fast} & 0.048 & 21.11 & 0.1077 & 0.851 \\
Bi-Layer \cite{zakharov2020fast} & 0.050 & 21.48 & 0.108 & 0.839 \\
% {'L1': tensor(0.0504), 'PSNR': tensor(20.4868), 'MS_SSIM': tensor(0.8389), 'LPIPS': tensor(0.1532), 'SSIM': tensor(0.7608)}
% hard mask {'L1': tensor(0.0478), 'PSNR': tensor(20.1186), 'MS_SSIM': tensor(0.8510), 'LPIPS': tensor(0.1077), 'SSIM': tensor(0.7652)}
ROME \cite{khakhulin2022realistic} & 0.048 & 21.13 & 0.116 & 0.838 \\
% Ours & 0.040 & 22.49 & 0.128 & 0.849 \\ w/o GAN loss
Ours & 0.041 & 22.09 & 0.111 & 0.840 \\ % +GAN loss
\midrule
% \end{tabular}
% \begin{tabular}{l|c|c|c|c}
\midrule
Dataset & \multicolumn{4}{c}{VoxCeleb2} \\
\midrule
Method & L1 $\downarrow$ & PSNR $\uparrow$ & LPIPS $\downarrow$ & MS-SSIM $\uparrow$ \\
\midrule
FOMM \cite{siarohin2019first} & 0.059 & 20.93 & 0.165 & 0.793 \\
ROME \cite{khakhulin2022realistic} & 0.050 & 20.75 & 0.117 & 0.834 \\
Ours & 0.042 & 21.37 & 0.119 & 0.841 \\
\bottomrule
\end{tabular}
}
\caption{Results of self-reenactment on the VoxCeleb1 and VoxCeleb2 ($\uparrow$ means larger is better, $\downarrow$ means smaller is better.)}
\label{tab:self_recon}
\end{table}
\paragraph{Self-reenactment}
The quantitative comparison results are summarized in Table \ref{tab:self_recon}.
It is noteworthy that our CVTHead achieves comparable performance with previous methods over all metrics.
% outperform ROME on L1, PSNR, and MS-SSIM. We only perform slightly worse than ROME on LPIPS.
%
Figure \ref{fig:self_reco} illustrates the qualitative comparisons. We also add on the predicted soft mask as in \cite{khakhulin2022realistic} to compare its quality.
The first three rows showcase scenarios with minimal head rotations and predominantly frontal source images. In such cases, both ROME and CVTHead exhibit similar performance. However, when the source images depict side views while the driving images present frontal views, ROME tends to generate images with blurry foreground masks in the occluded regions of the source image. Furthermore, ROME often renders these concealed areas in darker colors. These observations indicate that ROME struggles to effectively learn the features of occluded regions and fails to capture the correspondence between mesh vertices. Conversely, our CVTHead addresses these limitations by leveraging transformers to capture long-range dependencies among vertices. These observations suggest that ROME does not effectively learn the features of occluded regions and does not capture the correspondence between mesh vertices. In contrast, our CVTHead addresses these two issues by leveraging transformers to capture long-range dependencies among vertices.
\begin{table}[!ht]
% >>>>>>>>>>>>>>>>>>>>>>>>>>> cross + >>>>>>>>>>>>>>>>>>>>>>>>>>>
\centering
\resizebox{0.85\linewidth}{!}{
\begin{tabular}{l|c|c|c|c}
\toprule
Dataset & \multicolumn{4}{c}{VoxCeleb1} \\
\midrule
Method & FID $\downarrow$ & CSIM $\uparrow$ & IQA $\uparrow$ & FPS $\uparrow$ \\
\midrule
FOMM \cite{siarohin2019first} & 39.69 & 0.592 & 37.00 & 64.3\\ % +bg
% FOMM \cite{siarohin2019first} & 39.69 & 0.692 & 37.00 & 64.3\\ % +bg
% FOMM/ms \cite{siarohin2019first} & 36.66 & 0.701 & 36.65 & 64.3 \\ % no bg
Bi-Layer \cite{zakharov2020fast} & 43.8 & 0.697 & 41.4 & 20.1 \\
ROME \cite{khakhulin2022realistic} & 29.23 & 0.717 & 39.11 & 12.9 \\
% Ours & 31.15 & 0.676 & \textbf{42.36} & \textbf{24.3} \\ % wo GAN loss
Ours & 25.78 & 0.675 & 42.26 & 24.3 \\ % +GAN loss
\midrule
\midrule
Dataset & \multicolumn{4}{c}{VoxCeleb2} \\
\midrule
Method & FID $\downarrow$ & CSIM $\uparrow$ & IQA $\uparrow$ & FPS $\uparrow$ \\
\midrule
FOMM \cite{siarohin2019first} & 61.28 & 0.624 & 36.20 & 64.3 \\
% FOMM-raw \cite{siarohin2019first} & 72.25 & 0.625 & 38.92 & 64.3 \\
% ROME \cite{khakhulin2022realistic} & 73.52 & 0.529 & 37.34 & 12.9 \\
ROME \cite{khakhulin2022realistic} & 53.52 & 0.729 & 37.34 & 12.9 \\
Ours & 48.48 & 0.712 & 40.27 & 24.3 \\
\bottomrule
\end{tabular}
}
\caption{Results of cross-identity reenactment. }
\label{tab:cross_recon}
\end{table}
\begin{figure}[!ht]
\centering
\includegraphics[width=1.0\linewidth]{cross-recon3.pdf}
\caption{{Qualitative comparisons of cross-identity reenactment on VoxCeleb1.} }
\label{fig:cross_recon}
\end{figure}
\paragraph{Cross-identity Reenactment}
We proceed to evaluate our method in comparison to other methods for cross-identity reenactment. The quantitative comparison results are presented in Table \ref{tab:cross_recon}. Strikingly, we achieve similar performance on the assessed metrics as ROME, indicating the effectiveness of our method in cross-identity reenactment tasks.
Furthermore, we provide qualitative results in Figure \ref{fig:cross_recon}, showcasing the ability of our method to generate images with desired expressions, head poses, and other attributes. Notably, warping-based methods usually cannot maintain the identity information such as face shape from the source image. For mesh-guided methods, ROME tends to generate lower-quality images when local regions are occluded in the source image. In contrast, our method demonstrates the capability to maintain the quality of all local regions even in such challenging scenarios.
\begin{figure*}[!ht]
\centering
\includegraphics[width=0.95\linewidth]{animation_new2.pdf}
\caption{Qualitative results of face animation with novel views, novel face shapes (identity), and novel expressions. }
\label{fig:face_animation}
\end{figure*}
\paragraph{Inference time comparison}
We also evaluate the inference time of each model, considering the complete duration of 3D mesh reconstruction, the vertex deformation model, and the rendering process. To provide a comprehensive analysis, we report the average FPS (Frames per Second) based on 1000 runs performed on a single RTX 3090Ti. The results are presented in the last column of Table \ref{tab:cross_recon}. Notably, warping-based method is more efficient as they don't need the tedious rendering and mesh reconstruction.
ROME achieves a modest 12.9 FPS, while our CVTHead model achieves a significantly higher rate of 24.3 FPS. This outcome highlights the superior efficiency of the point-based neural rendering approach compared to traditional graphic-based rendering methods.
\subsection{Results of 3DMM-based Face Animation}
After obtaining the vertex descriptors using the vertex feature transformer, the resulting face can be further manipulated by adjusting the coefficients of the FLAME model \cite{li2017learning}, which control expression $\phi$, pose $\theta$, face shape $\beta$, and camera views $c$. The ability to explicitly control these coefficients enables us to generate faces of the same subject with different expressions, face shapes, and camera views, as illustrated in Figure \ref{fig:face_animation}. This result demonstrates that the learned feature descriptors exhibit a strong alignment with the vertices in the canonical space. Consequently, neural point-based rendering can serve as a viable alternative to traditional graphic-based rendering methods.
%
Moreover, we intentionally select two distinct source images of the same subject. Interestingly, the generated images, utilizing vertex features from these distinct sources, exhibit a striking resemblance. This intriguing observation further underscores the effectiveness and robustness of our method.
\subsection{Ablation Studies}
\paragraph{Vertex deformation}
We utilize the linear deformation model $f_H(\cdot)$ from ROME \cite{khakhulin2022realistic} to deform the vertices of the hair and shoulder region. In this study, we conduct an ablation experiment where we train CVTHead without this vertex deformation module, instead employing the default FLAME mesh with a bald head.
The results presented in Table \ref{tab:linear_deform} demonstrate that the removal of the vertex deformation (``D." in short) has only a minor impact on the performance. Interestingly, Figure \ref{fig:abla_deform} reveals that the synthesized images from CVTHead, both with and without vertex deformation, appear nearly identical.
Furthermore, even in cases where the subject has fluffy or long hair that extends beyond the head area, the absence of vertex deformation in CVTHead does not hinder its ability to generate the correct hairstyle.
These results indicate that the local vertex descriptor can effectively capture the necessary features.
\begin{table}[!ht]
% >>>>>>>>>>>>>>>>>>>>>>>>>>> self >>>>>>>>>>>>>>>>>>>>>>>>>>>
\centering
\resizebox{1.0\linewidth}{!}{
\begin{tabular}{l|c|c|c|c}
\toprule
Method & L1 $\downarrow$ & PSNR $\uparrow$ & LPIPS $\downarrow$ & MS-SSIM $\uparrow$ \\
\midrule
CVTHead (w/o D.) & 0.041 & 22.47 & 0.121 & 0.842 \\
% CVTHead & 0.042 & 21.37 & 0.119 & 0.841 \\
CVTHead & 0.041 & 22.09 & 0.111 & 0.840 \\
\bottomrule
\end{tabular}
}
\caption{\small{Ablation study on the vertex deformation module. We evaluate the performance of self-reenactment on the VoxCeleb1.}}
\label{tab:linear_deform}
\end{table}
%
\begin{figure}[ht]
\centering
\includegraphics[width=1.0\linewidth]{linear_deform2.pdf}
\caption{\small{Ablation study of CVTHead with and without the vertex deformation model for hair and shoulder region (D. in short) }}
\label{fig:abla_deform}
\end{figure}
\begin{table}[!ht]
% >>>>>>>>>>>>>>>>>>>>>>>>>>> self >>>>>>>>>>>>>>>>>>>>>>>>>>>
\centering
\resizebox{1.0\linewidth}{!}{
\begin{tabular}{l|c|c|c|c}
\toprule
Method & L1 $\downarrow$ & PSNR $\uparrow$ & LPIPS $\downarrow$ & MS-SSIM $\uparrow$ \\
\midrule
Pixel-aligned features & 0.045 & 21.81 & 0.107 & 0.841 \\
CVTHead & 0.041 & 22.09 & 0.111 & 0.840 \\
\bottomrule
\end{tabular}
}
\caption{{Ablation study on the pixel-aligned features}}
\label{tab:pix_align}
\end{table}
\paragraph{Pixel-aligned features}
In our work, we design the vertex feature transformers to learn the vertex feature. In this study, we consider the pixel-aligned features as the baseline, which project the 3D vertex into 2D and choose the corresponding pixel from the image. We follow the architecture design in S3F \cite{corona2023structured} and use a UNet-like feature extractor and sample features of each vertex with its corresponding 2D projection.
Table \ref{tab:pix_align} indicates that this approach yields a marginally lower PSNR, but a slightly improved LPIPS score.
As shown in Figure \ref{fig:abla_s3f}, this design can maintain more detailed local features such as hair due to the high-resolution features. Thus, a slightly better LPIPS is achieved. However, when the point is occluded in the source image, the synthesized image tends to generate blur and shadow in these areas if they are visible in the driving pose, which is the reason of the worse PSNR.
These results suggest that pixel-aligned methods cannot capture the correct features due to the ambiguity of depth. In this case, when a large head rotation happens, this method encounters the same issue as warp-based methods.
\begin{figure}[ht]
\centering
\includegraphics[width=0.9\linewidth]{pifu.pdf}
\caption{Ablation study of vertex features.}
\label{fig:abla_s3f}
\end{figure}
\vspace{-0.5em}
\section{Limitations}
\vspace{-0.5em}
While our method demonstrates effective face animation capabilities from a single image, one potential limitation is that the performance of our approach heavily relies on the accuracy of the 3D mesh reconstruction, specifically utilizing DECA \cite{feng2021learning} in our setup. In certain challenging scenarios, DECA may struggle to fully disentangle the shape and expression factors from the driving images. Consequently, CVTHead may generate images that differ in expressions or head poses from the intended outcome. This highlights the need for further advancements in the accuracy and robustness of 3D mesh reconstruction techniques to address such limitations.
\vspace{-0.5em}
\section{Conclusion}
\vspace{-0.5em}
% In this paper, we apply point-based neural rendering to explicitly controllable head avatar generation from a single reference image.
% We consider the sparse vertices of the head mesh as a point set and use the vertex-feature transformer to learn the local feature descriptor of each vertex. Our work shows that point-based rendering can serve as an alternative to graphic-based rendering methods in terms of efficiency. We believe our method is compatible with several generative tools such as GAN and diffusions to further improve its performance and we decide to leave it as future work.
In this paper, we propose a novel approach for generating explicitly controllable head avatars from a single reference image, utilizing point-based neural rendering. We treat the sparse vertices of the head mesh as a point set and leverage the vertex-feature transformer to learn the local feature descriptor for each vertex. Through our research, we demonstrate that point-based rendering can effectively replace traditional graphic-based rendering methods, offering enhanced efficiency.
Moreover, we envision that our method can be seamlessly integrated with various generative tools, such as diffusions, to further enhance the quality of generated images and we consider this as future work.
% %%%%%%%%% BODY TEXT
% \section{Introduction}
% \label{sec:intro}
% Please follow the steps outlined below when submitting your manuscript to the IEEE Computer Society Press.
% This style guide now has several important modifications (for example, you are no longer warned against the use of sticky tape to attach your artwork to the paper), so all authors should read this new version.
% %-------------------------------------------------------------------------
% \subsection{Language}
% All manuscripts must be in English.
% \subsection{Dual submission}
% Please refer to the author guidelines on the \confName\ \confYear\ web page for a
% discussion of the policy on dual submissions.
% \subsection{Paper length}
% Papers, excluding the references section, must be no longer than eight pages in length.
% The references section will not be included in the page count, and there is no limit on the length of the references section.
% For example, a paper of eight pages with two pages of references would have a total length of 10 pages.
% {\bf There will be no extra page charges for \confName\ \confYear.}
% Overlength papers will simply not be reviewed.
% This includes papers where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
% Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.
% The reason such papers will not be reviewed is that there is no provision for supervised revisions of manuscripts.
% The reviewing process cannot determine the suitability of the paper for presentation in eight pages if it is reviewed in eleven.
% %-------------------------------------------------------------------------
% \subsection{The ruler}
% The \LaTeX\ style defines a printed ruler which should be present in the version submitted for review.
% The ruler is provided in order that reviewers may comment on particular lines in the paper without circumlocution.
% If you are preparing a document using a non-\LaTeX\ document preparation system, please arrange for an equivalent ruler to appear on the final output pages.
% The presence or absence of the ruler should not change the appearance of any other content on the page.
% The camera-ready copy should not contain a ruler.
% (\LaTeX\ users may use options of wacv.sty to switch between different versions.)
% Reviewers:
% note that the ruler measurements do not align well with lines in the paper --- this turns out to be very difficult to do well when the paper contains many figures and equations, and, when done, looks ugly.
% Just use fractional references (\eg, this line is $087.5$), although in most cases one would expect that the approximate location will be adequate.
% \subsection{Paper ID}
% Make sure that the Paper ID from the submission system is visible in the version submitted for review (replacing the ``*****'' you see in this document).
% If you are using the \LaTeX\ template, \textbf{make sure to update paper ID in the appropriate place in the tex file}.
% \subsection{Mathematics}
% Please number all of your sections and displayed equations as in these examples:
% \begin{equation}
% E = m\cdot c^2
% \label{eq:important}
% \end{equation}
% and
% \begin{equation}
% v = a\cdot t.
% \label{eq:also-important}
% \end{equation}
% It is important for readers to be able to refer to any particular equation.
% Just because you did not refer to it in the text does not mean some future reader might not need to refer to it.
% It is cumbersome to have to use circumlocutions like ``the equation second from the top of page 3 column 1''.
% (Note that the ruler will not be present in the final copy, so is not an alternative to equation numbers).
% All authors will benefit from reading Mermin's description of how to write mathematics:
% \url{http://www.pamitc.org/documents/mermin.pdf}.
% \subsection{Blind review}
% Many authors misunderstand the concept of anonymizing for blind review.
% Blind review does not mean that one must remove citations to one's own work---in fact it is often impossible to review a paper unless the previous citations are known and available.
% Blind review means that you do not use the words ``my'' or ``our'' when citing previous work.
% That is all.
% (But see below for tech reports.)
% Saying ``this builds on the work of Lucy Smith [1]'' does not say that you are Lucy Smith;
% it says that you are building on her work.
% If you are Smith and Jones, do not say ``as we show in [7]'', say ``as Smith and Jones show in [7]'' and at the end of the paper, include reference 7 as you would any other cited work.
% An example of a bad paper just asking to be rejected:
% \begin{quote}
% \begin{center}
% An analysis of the frobnicatable foo filter.
% \end{center}
% In this paper we present a performance analysis of our previous paper [1], and show it to be inferior to all previously known methods.
% Why the previous paper was accepted without this analysis is beyond me.
% [1] Removed for blind review
% \end{quote}
% An example of an acceptable paper:
% \begin{quote}
% \begin{center}
% An analysis of the frobnicatable foo filter.
% \end{center}
% In this paper we present a performance analysis of the paper of Smith \etal [1], and show it to be inferior to all previously known methods.
% Why the previous paper was accepted without this analysis is beyond me.
% [1] Smith, L and Jones, C. ``The frobnicatable foo filter, a fundamental contribution to human knowledge''. Nature 381(12), 1-213.
% \end{quote}
% If you are making a submission to another conference at the same time, which covers similar or overlapping material, you may need to refer to that submission in order to explain the differences, just as you would if you had previously published related work.
% In such cases, include the anonymized parallel submission~\cite{Authors14} as supplemental material and cite it as
% \begin{quote}
% [1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324, Supplied as supplemental material {\tt fg324.pdf}.
% \end{quote}
% Finally, you may feel you need to tell the reader that more details can be found elsewhere, and refer them to a technical report.
% For conference submissions, the paper must stand on its own, and not {\em require} the reviewer to go to a tech report for further details.
% Thus, you may say in the body of the paper ``further details may be found in~\cite{Authors14b}''.
% Then submit the tech report as supplemental material.
% Again, you may not assume the reviewers will read this material.
% Sometimes your paper is about a problem which you tested using a tool that is widely known to be restricted to a single institution.
% For example, let's say it's 1969, you have solved a key problem on the Apollo lander, and you believe that the WACV70 audience would like to hear about your
% solution.
% The work is a development of your celebrated 1968 paper entitled ``Zero-g frobnication: How being the only people in the world with access to the Apollo lander source code makes us a wow at parties'', by Zeus \etal.
% You can handle this paper like any other.
% Do not write ``We show how to improve our previous work [Anonymous, 1968].
% This time we tested the algorithm on a lunar lander [name of lander removed for blind review]''.
% That would be silly, and would immediately identify the authors.
% Instead write the following:
% \begin{quotation}
% \noindent
% We describe a system for zero-g frobnication.
% This system is new because it handles the following cases:
% A, B. Previous systems [Zeus et al. 1968] did not handle case B properly.
% Ours handles it by including a foo term in the bar integral.
% ...
% The proposed system was integrated with the Apollo lunar lander, and went all the way to the moon, don't you know.
% It displayed the following behaviours, which show how well we solved cases A and B: ...
% \end{quotation}
% As you can see, the above text follows standard scientific convention, reads better than the first version, and does not explicitly name you as the authors.
% A reviewer might think it likely that the new paper was written by Zeus \etal, but cannot make any decision based on that guess.
% He or she would have to be sure that no other authors could have been contracted to solve problem B.
% \medskip
% \noindent
% FAQ\medskip\\
% {\bf Q:} Are acknowledgements OK?\\
% {\bf A:} No. Leave them for the final copy.\medskip\\
% {\bf Q:} How do I cite my results reported in open challenges?
% {\bf A:} To conform with the double-blind review policy, you can report results of other challenge participants together with your results in your paper.
% For your results, however, you should not identify yourself and should not mention your participation in the challenge.
% Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\
% \begin{figure}[t]
% \centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
% %\includegraphics[width=0.8\linewidth]{egfigure.eps}
% \caption{Example of caption.
% It is set in Roman so that mathematics (always set in Roman: $B \sin A = A \sin B$) may be included without an ugly clash.}
% \label{fig:onecol}
% \end{figure}
% \subsection{Miscellaneous}
% \noindent
% Compare the following:\\
% \begin{tabular}{ll}
% \verb'$conf_a$' & $conf_a$ \\
% \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
% \end{tabular}\\
% See The \TeX book, p165.
% The space after \eg, meaning ``for example'', should not be a sentence-ending space.
% So \eg is correct, {\em e.g.} is not.
% The provided \verb'\eg' macro takes care of this.
% When citing a multi-author paper, you may save space by using ``et alia'', shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word).
% If you use the \verb'\etal' macro provided, then you need not worry about double periods when used at the end of a sentence as in Alpher \etal.
% However, use it only when there are three or more authors.
% Thus, the following is correct:
% ``Frobnication has been trendy lately.
% It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
% Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''
% This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...'' because reference~\cite{Alpher03} has just two authors.
% % Update the wacv.cls to do the following automatically.
% % For this citation style, keep multiple citations in numerical (not
% % chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
% % \cite{Alpher02,Alpher03,Authors14}.
% \begin{figure*}
% \centering
% \begin{subfigure}{0.68\linewidth}
% \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
% \caption{An example of a subfigure.}
% \label{fig:short-a}
% \end{subfigure}
% \hfill
% \begin{subfigure}{0.28\linewidth}
% \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
% \caption{Another example of a subfigure.}
% \label{fig:short-b}
% \end{subfigure}
% \caption{Example of a short caption, which should be centered.}
% \label{fig:short}
% \end{figure*}
% %------------------------------------------------------------------------
% \section{Formatting your paper}
% \label{sec:formatting}
% All text must be in a two-column format.
% The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
% Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
% The main title (on the first page) should begin 1 inch (2.54 cm) from the top edge of the page.
% The second and following pages should begin 1 inch (2.54 cm) from the top edge.
% On all pages, the bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
% for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the
% page.
% %-------------------------------------------------------------------------
% \subsection{Margins and page numbering}
% All printed material, including text, illustrations, and charts, must be kept
% within a print area $6\frac{7}{8}$ inches (17.46 cm) wide by $8\frac{7}{8}$ inches (22.54 cm)
% high.
% %
% Page numbers should be in the footer, centered and $\frac{3}{4}$ inches from the bottom of the page.
% The review version should have page numbers, yet the final version submitted as camera ready should not show any page numbers.
% The \LaTeX\ template takes care of this when used properly.
% %-------------------------------------------------------------------------
% \subsection{Type style and fonts}
% Wherever Times is specified, Times Roman may also be used.
% If neither is available on your word processor, please use the font closest in
% appearance to Times to which you have access.
% MAIN TITLE.
% Center the title $1\frac{3}{8}$ inches (3.49 cm) from the top edge of the first page.
% The title should be in Times 14-point, boldface type.
% Capitalize the first letter of nouns, pronouns, verbs, adjectives, and adverbs;
% do not capitalize articles, coordinate conjunctions, or prepositions (unless the title begins with such a word).
% Leave two blank lines after the title.
% AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
% and printed in Times 12-point, non-boldface type.
% This information is to be followed by two blank lines.
% The ABSTRACT and MAIN TEXT are to be in a two-column format.
% MAIN TEXT.
% Type main text in 10-point Times, single-spaced.
% Do NOT use double-spacing.
% All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
% Make sure your text is fully justified---that is, flush left and flush right.
% Please do not place any additional blank lines between paragraphs.
% Figure and table captions should be 9-point Roman type as in \cref{fig:onecol,fig:short}.
% Short captions should be centred.
% \noindent Callouts should be 9-point Helvetica, non-boldface type.
% Initially capitalize only the first word of section titles and first-, second-, and third-order headings.
% FIRST-ORDER HEADINGS.
% (For example, {\large \bf 1. Introduction}) should be Times 12-point boldface, initially capitalized, flush left, with one blank line before, and one blank line after.
% SECOND-ORDER HEADINGS.
% (For example, { \bf 1.1. Database elements}) should be Times 11-point boldface, initially capitalized, flush left, with one blank line before, and one after.
% If you require a third-order heading (we discourage it), use 10-point Times, boldface, initially capitalized, flush left, preceded by one blank line, followed by a period and your text on the same line.
% %-------------------------------------------------------------------------
% \subsection{Footnotes}
% Please use footnotes\footnote{This is what a footnote looks like.
% It often distracts the reader from the main flow of the argument.} sparingly.
% Indeed, try to avoid footnotes altogether and include necessary peripheral observations in the text (within parentheses, if you prefer, as in this sentence).
% If you wish to use a footnote, place it at the bottom of the column on the page on which it is referenced.
% Use Times 8-point type, single-spaced.
% %-------------------------------------------------------------------------
% \subsection{Cross-references}
% For the benefit of author(s) and readers, please use the
% {\small\begin{verbatim}
% \cref{...}
% \end{verbatim}} command for cross-referencing to figures, tables, equations, or sections.
% This will automatically insert the appropriate label alongside the cross-reference as in this example:
% \begin{quotation}
% To see how our method outperforms previous work, please see \cref{fig:onecol} and \cref{tab:example}.
% It is also possible to refer to multiple targets as once, \eg~to \cref{fig:onecol,fig:short-a}.
% You may also return to \cref{sec:formatting} or look at \cref{eq:also-important}.
% \end{quotation}
% If you do not wish to abbreviate the label, for example at the beginning of the sentence, you can use the
% {\small\begin{verbatim}
% \Cref{...}
% \end{verbatim}}
% command. Here is an example:
% \begin{quotation}
% \Cref{fig:onecol} is also quite important.
% \end{quotation}
% %-------------------------------------------------------------------------
% \subsection{References}
% List and number all bibliographical references in 9-point Times, single-spaced, at the end of your paper.
% When referenced in the text, enclose the citation number in square brackets, for
% example~\cite{Authors14}.
% Where appropriate, include page numbers and the name(s) of editors of referenced books.
% When you cite multiple papers at once, please make sure that you cite them in numerical order like this \cite{Alpher02,Alpher03,Alpher05,Authors14b,Authors14}.
% If you use the template as advised, this will be taken care of automatically.
% \begin{table}
% \centering
% \begin{tabular}{@{}lc@{}}
% \toprule
% Method & Frobnability \\
% \midrule
% Theirs & Frumpy \\
% Yours & Frobbly \\
% Ours & Makes one's heart Frob\\
% \bottomrule
% \end{tabular}
% \caption{Results. Ours is better.}
% \label{tab:example}
% \end{table}
% %-------------------------------------------------------------------------
% \subsection{Illustrations, graphs, and photographs}
% All graphics should be centered.
% In \LaTeX, avoid using the \texttt{center} environment for this purpose, as this adds potentially unwanted whitespace.
% Instead use
% {\small\begin{verbatim}
% \centering
% \end{verbatim}}
% at the beginning of your figure.
% Please ensure that any point you wish to make is resolvable in a printed copy of the paper.
% Resize fonts in figures to match the font in the body text, and choose line widths that render effectively in print.
% Readers (and reviewers), even of an electronic copy, may choose to print your paper in order to read it.
% You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.
% When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the figure width as a multiple of the line width as in the example below
% {\small\begin{verbatim}
% \usepackage{graphicx} ...
% \includegraphics[width=0.8\linewidth]
% {myfile.pdf}
% \end{verbatim}
% }
% %-------------------------------------------------------------------------
% \subsection{Color}
% Please refer to the author guidelines on the \confName\ \confYear\ web page for a discussion of the use of color in your document.
% If you use color in your plots, please keep in mind that a significant subset of reviewers and readers may have a color vision deficiency; red-green blindness is the most frequent kind.
% Hence avoid relying only on color as the discriminative feature in plots (such as red \vs green lines), but add a second discriminative feature to ease disambiguation.
% %------------------------------------------------------------------------
% \section{Final copy}
% You must include your signed IEEE copyright release form when you submit your finished paper.
% We MUST have this form before your paper can be published in the proceedings.
% Please direct any questions to the production editor in charge of these proceedings at the IEEE Computer Society Press:
% \url{https://www.computer.org/about/contact}.
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{alexander2010digital}
Oleg Alexander, Mike Rogers, William Lambeth, Jen-Yuan Chiang, Wan-Chun Ma,
Chuan-Chang Wang, and Paul Debevec.
\newblock The digital emily project: Achieving a photorealistic digital actor.
\newblock {\em IEEE Computer Graphics and Applications}, pages 20--31, 2010.
\bibitem{aliev2020neural}
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor
Lempitsky.
\newblock Neural point-based graphics.
\newblock In {\em ECCV}, pages 696--712, 2020.
\bibitem{athar2022rignerf}
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu.
\newblock Rignerf: Fully controllable neural 3d portraits.
\newblock In {\em CVPR}, pages 20364--20373, 2022.
\bibitem{blanz1999morphable}
Volker Blanz and Thomas Vetter.
\newblock A morphable model for the synthesis of 3d faces.
\newblock In {\em Proceedings of the 26th annual conference on Computer
graphics and interactive techniques}, pages 187--194, 1999.
\bibitem{blinn1976texture}
James~F Blinn and Martin~E Newell.
\newblock Texture and reflection in computer generated images.
\newblock {\em Communications of the ACM}, pages 542--547, 1976.
\bibitem{cao2018vggface2}
Qiong Cao, Li Shen, Weidi Xie, Omkar~M Parkhi, and Andrew Zisserman.
\newblock Vggface2: A dataset for recognising faces across pose and age.
\newblock In {\em 2018 13th IEEE international conference on automatic face \&
gesture recognition (FG 2018)}, pages 67--74. IEEE, 2018.
\bibitem{carion2020end}
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander
Kirillov, and Sergey Zagoruyko.
\newblock End-to-end object detection with transformers.
\newblock In {\em ECCV}, 2020.
\bibitem{chan2022efficient}
Eric~R Chan, Connor~Z Lin, Matthew~A Chan, Koki Nagano, Boxiao Pan, Shalini
De~Mello, Orazio Gallo, Leonidas~J Guibas, Jonathan Tremblay, Sameh Khamis,
et~al.
\newblock Efficient geometry-aware 3d generative adversarial networks.
\newblock In {\em CVPR}, pages 16123--16133, 2022.
\bibitem{cho2022FastMETRO}
Junhyeong Cho, Kim Youwang, and Tae-Hyun Oh.
\newblock Cross-attention of disentangled modalities for 3d human mesh recovery
with transformers.
\newblock In {\em ECCV}, 2022.
\bibitem{chung2018voxceleb2}
Joon~Son Chung, Arsha Nagrani, and Andrew Zisserman.
\newblock Voxceleb2: Deep speaker recognition.
\newblock {\em arXiv preprint arXiv:1806.05622}, 2018.
\bibitem{corona2023structured}
Enric Corona, Mihai Zanfir, Thiemo Alldieck, Eduard~Gabriel Bazavan, Andrei
Zanfir, and Cristian Sminchisescu.
\newblock Structured 3d features for reconstructing controllable avatars.
\newblock In {\em CVPR}, pages 16954--16964, 2023.
\bibitem{dosovitskiy2021an}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock In {\em ICLR}, 2021.
\bibitem{dou2023tore}
Zhiyang Dou, Qingxuan Wu, Cheng Lin, Zeyu Cao, Qiangqiang Wu, Weilin Wan, Taku
Komura, and Wenping Wang.
\newblock Tore: Token reduction for efficient human mesh recovery with
transformer.
\newblock In {\em ICCV}, pages 15143--15155, 2023.
\bibitem{doukas2021headgan}
Michail~Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska.
\newblock Headgan: One-shot neural head synthesis and editing.
\newblock In {\em ICCV}, pages 14398--14407, 2021.
\bibitem{Drobyshev22MP}
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor
Lempitsky, and Egor Zakharov.
\newblock Megaportraits: One-shot megapixel neural head avatars.
\newblock In {\em MM}, 2022.
\bibitem{feng2021learning}
Yao Feng, Haiwen Feng, Michael~J Black, and Timo Bolkart.
\newblock Learning an animatable detailed 3d face model from in-the-wild
images.
\newblock {\em ACM Transactions on Graphics (ToG)}, pages 1--13, 2021.
\bibitem{gafni2021dynamic}
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nie{\ss}ner.
\newblock Dynamic neural radiance fields for monocular 4d facial avatar
reconstruction.
\newblock In {\em CVPR}, pages 8649--8658, 2021.
\bibitem{ghosh2020gif}
Partha Ghosh, Pravir~Singh Gupta, Roy Uziel, Anurag Ranjan, Michael~J Black,
and Timo Bolkart.
\newblock Gif: Generative interpretable faces.
\newblock In {\em 3DV}, pages 868--878, 2020.
\bibitem{gong2019spiralnet++}
Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou.
\newblock Spiralnet++: A fast and highly efficient mesh convolution operator.
\newblock In {\em ICCVW}, 2019.
\bibitem{NIPS2014_5ca3e9b1}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
\newblock Generative adversarial nets.
\newblock In {\em NIPS}, 2014.
\bibitem{grassal2022neural}
Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother,
Matthias Nie{\ss}ner, and Justus Thies.
\newblock Neural head avatars from monocular rgb videos.
\newblock In {\em CVPR}, pages 18653--18664, 2022.
\bibitem{gross2011point}
Markus Gross and Hanspeter Pfister.
\newblock {\em Point-based graphics}.
\newblock Elsevier, 2011.
\bibitem{he2021arch++}
Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung.
\newblock Arch++: Animation-ready clothed human reconstruction revisited.
\newblock In {\em ICCV}, pages 11046--11056, 2021.
\bibitem{heusel2017gans}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
Hochreiter.
\newblock Gans trained by a two time-scale update rule converge to a local nash
equilibrium.
\newblock {\em NeurIPS}, 30, 2017.
\bibitem{hong2022headnerf}
Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang.
\newblock Headnerf: A real-time nerf-based parametric head model.
\newblock In {\em CVPR}, pages 20374--20384, 2022.
\bibitem{huang2020arch}
Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.
\newblock Arch: Animatable reconstruction of clothed humans.
\newblock In {\em CVPR}, pages 3093--3102, 2020.
\bibitem{johnson2016perceptual}
Justin Johnson, Alexandre Alahi, and Li Fei-Fei.
\newblock Perceptual losses for real-time style transfer and super-resolution.
\newblock In {\em ECCV}, pages 694--711, 2016.
\bibitem{kania2022conerf}
Kacper Kania, Kwang~Moo Yi, Marek Kowalski, Tomasz Trzci{\'n}ski, and Andrea
Tagliasacchi.
\newblock Conerf: Controllable neural radiance fields.
\newblock In {\em CVPR}, pages 18623--18632, 2022.
\bibitem{kato2020differentiable}
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim
Kehl, and Adrien Gaidon.
\newblock Differentiable rendering: A survey.
\newblock {\em arXiv preprint arXiv:2006.12057}, 2020.
\bibitem{kerbl3Dgaussians}
Bernhard Kerbl, Georgios Kopanas, Thomas Leimk{\"u}hler, and George Drettakis.
\newblock 3d gaussian splatting for real-time radiance field rendering.
\newblock {\em ACM Transactions on Graphics}, 2023.
\bibitem{khakhulin2022realistic}
Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov.
\newblock Realistic one-shot mesh-based head avatars.
\newblock In {\em ECCV}, pages 345--362, 2022.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{kopanas2021point}
Georgios Kopanas, Julien Philip, Thomas Leimk{\"u}hler, and George Drettakis.
\newblock Point-based neural rendering with per-view optimization.
\newblock In {\em Computer Graphics Forum}, pages 29--43, 2021.
\bibitem{li2017learning}
Tianye Li, Timo Bolkart, Michael~J Black, Hao Li, and Javier Romero.
\newblock Learning a model of facial shape and expression from 4d scans.
\newblock {\em ACM Trans. Graph.}, pages 194--1, 2017.
\bibitem{li2023one}
Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen,
Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li.
\newblock One-shot high-fidelity talking-head synthesis with deformable neural
radiance field.
\newblock In {\em CVPR}, 2023.
\bibitem{lin2020end}
Kevin Lin, Lijuan Wang, and Zicheng Liu.
\newblock End-to-end human pose and mesh reconstruction with transformers.
\newblock {\em CVPR}, 2021.
\bibitem{lin2021mesh}
Kevin Lin, Lijuan Wang, and Zicheng Liu.
\newblock Mesh graphormer.
\newblock In {\em ICCV}, pages 12939--12948, 2021.
\bibitem{loper2015smpl}
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael~J
Black.
\newblock Smpl: A skinned multi-person linear model.
\newblock {\em ACM transactions on graphics (TOG)}, 34(6):1--16, 2015.
\bibitem{ma2022ppt}
Haoyu Ma, Zhe Wang, Yifei Chen, Deying Kong, Liangjian Chen, Xingwei Liu,
Xiangyi Yan, Hao Tang, and Xiaohui Xie.
\newblock Ppt: token-pruned pose transformer for monocular and multi-view human
pose estimation.
\newblock In {\em ECCV}, 2022.
\bibitem{ma2023otavatar}
Zhiyuan Ma, Xiangyu Zhu, Guo-Jun Qi, Zhen Lei, and Lei Zhang.
\newblock Otavatar: One-shot talking face avatar with controllable tri-plane
rendering.
\newblock In {\em CVPR}, pages 16901--16910, 2023.
\bibitem{mildenhall2020nerf}
Ben Mildenhall, Pratul~P. Srinivasan, Matthew Tancik, Jonathan~T. Barron, Ravi
Ramamoorthi, and Ren Ng.
\newblock Nerf: Representing scenes as neural radiance fields for view
synthesis.
\newblock In {\em ECCV}, 2020.
\bibitem{nagrani2017voxceleb}
Arsha Nagrani, Joon~Son Chung, and Andrew Zisserman.
\newblock Voxceleb: a large-scale speaker identification dataset.
\newblock {\em arXiv preprint arXiv:1706.08612}, 2017.
\bibitem{park2021nerfies}
Keunhong Park, Utkarsh Sinha, Jonathan~T Barron, Sofien Bouaziz, Dan~B Goldman,
Steven~M Seitz, and Ricardo Martin-Brualla.
\newblock Nerfies: Deformable neural radiance fields.
\newblock In {\em ICCV}, pages 5865--5874, 2021.
\bibitem{paysan20093d}
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.
\newblock A 3d face model for pose and illumination invariant face recognition.
\newblock In {\em 2009 sixth IEEE international conference on advanced video
and signal based surveillance}, pages 296--301, 2009.
\bibitem{prokudin2021smplpix}
Sergey Prokudin, Michael~J Black, and Javier Romero.
\newblock Smplpix: Neural avatars from 3d human models.
\newblock In {\em WACV}, pages 1810--1819, 2021.
\bibitem{rakhimov2022npbg++}
Ruslan Rakhimov, Andrei-Timotei Ardelean, Victor Lempitsky, and Evgeny Burnaev.
\newblock Npbg++: Accelerating neural point-based graphics.
\newblock In {\em CVPR}, pages 15969--15979, 2022.
\bibitem{ravi2020accelerating}
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo,
Justin Johnson, and Georgia Gkioxari.
\newblock Accelerating 3d deep learning with pytorch3d.
\newblock {\em arXiv preprint arXiv:2007.08501}, 2020.
\bibitem{saito2019pifu}
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa,
and Hao Li.
\newblock Pifu: Pixel-aligned implicit function for high-resolution clothed
human digitization.
\newblock In {\em ICCV}, pages 2304--2314, 2019.
\bibitem{siarohin2019first}
Aliaksandr Siarohin, St{\'e}phane Lathuili{\`e}re, Sergey Tulyakov, Elisa
Ricci, and Nicu Sebe.
\newblock First order motion model for image animation.
\newblock {\em NeurIPS}, 32, 2019.
\bibitem{su2020blindly}
Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning
Zhang.
\newblock Blindly assess image quality in the wild guided by a self-adaptive
hyper network.
\newblock In {\em CVPR}, pages 3667--3676, 2020.
\bibitem{tewari2020stylerig}
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter
Seidel, Patrick P{\'e}rez, Michael Zollhofer, and Christian Theobalt.
\newblock Stylerig: Rigging stylegan for 3d control over portrait images.
\newblock In {\em CVPR}, pages 6142--6151, 2020.
\bibitem{thies2020neural}
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias
Nie{\ss}ner.
\newblock Neural voice puppetry: Audio-driven facial reenactment.
\newblock In {\em ECCV}, pages 716--731, 2020.
\bibitem{thies2019deferred}
Justus Thies, Michael Zollh{\"o}fer, and Matthias Nie{\ss}ner.
\newblock Deferred neural rendering: Image synthesis using neural textures.
\newblock {\em Acm Transactions on Graphics (TOG)}, 38(4):1--12, 2019.
\bibitem{thies2016face2face}
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and
Matthias Nie{\ss}ner.
\newblock Face2face: Real-time face capture and reenactment of rgb videos.
\newblock In {\em CVPR}, pages 2387--2395, 2016.
\bibitem{touvron2020training}
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Herv{\'e} J{\'e}gou.
\newblock Training data-efficient image transformers \& distillation through
attention.
\newblock {\em ICML}, 2021.
\bibitem{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{wang2021safa}
Qiulin Wang, Lu Zhang, and Bo Li.
\newblock Safa: Structure aware face animation.
\newblock In {\em 3DV}, pages 679--688, 2021.
\bibitem{wang2018high}
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan
Catanzaro.
\newblock High-resolution image synthesis and semantic manipulation with
conditional gans.
\newblock In {\em CVPR}, pages 8798--8807, 2018.
\bibitem{wang2021one}
Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.
\newblock One-shot free-view neural talking-head synthesis for video
conferencing.
\newblock In {\em CVPR}, pages 10039--10049, 2021.
\bibitem{wiles2018x2face}
Olivia Wiles, A Koepke, and Andrew Zisserman.
\newblock X2face: A network for controlling face generation using images,
audio, and pose codes.
\newblock In {\em ECCV}, pages 670--686, 2018.
\bibitem{wuu2022multiface}
Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko,
Eric Brockmeyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Alexander Hypes,
Taylor Koska, Steven Krenn, Stephen Lombardi, Xiaomin Luo, Kevyn McPhail,
Laura Millerschoen, Michal Perdoch, Mark Pitts, Alexander Richard, Jason
Saragih, Junko Saragih, Takaaki Shiratori, Tomas Simon, Matt Stewart, Autumn
Trimble, Xinshuo Weng, David Whitewolf, Chenglei Wu, Shoou-I Yu, and Yaser
Sheikh.
\newblock Multiface: A dataset for neural face rendering.
\newblock In {\em arXiv}, 2022.
\bibitem{xu2022point}
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli,
and Ulrich Neumann.
\newblock Point-nerf: Point-based neural radiance fields.
\newblock In {\em CVPR}, pages 5438--5448, 2022.
\bibitem{yan2022after}
Xiangyi Yan, Hao Tang, Shanlin Sun, Haoyu Ma, Deying Kong, and Xiaohui Xie.
\newblock After-unet: Axial fusion transformer unet for medical image
segmentation.
\newblock In {\em WACV}, 2022.
\bibitem{yang2020facescape}
Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and
Xun Cao.
\newblock Facescape: a large-scale high quality 3d face dataset and detailed
riggable 3d face prediction.
\newblock In {\em CVPR}, pages 601--610, 2020.
\bibitem{yang2022face2face}
Kewei Yang, Kang Chen, Daoliang Guo, Song-Hai Zhang, Yuan-Chen Guo, and Weidong
Zhang.
\newblock Face2face $\rho$: Real-time high-resolution one-shot face
reenactment.
\newblock In {\em ECCVC}, pages 55--71, 2022.
\bibitem{yang2023capturing}
Sen Yang, Wen Heng, Gang Liu, GUOZHONG LUO, Wankou Yang, and Gang YU.
\newblock Capturing the motion of every joint: 3d human pose and shape
estimation with independent tokens.
\newblock In {\em ICLR}, 2023.
\bibitem{yao2020mesh}
Guangming Yao, Yi Yuan, Tianjia Shao, and Kun Zhou.
\newblock Mesh guided one-shot face reenactment using graph convolutional
networks.
\newblock In {\em Proceedings of the 28th ACM international conference on
multimedia}, pages 1773--1781, 2020.
\bibitem{yoshiyasu2023deformable}
Yusuke Yoshiyasu.
\newblock Deformable mesh transformer for 3d human mesh recovery.
\newblock In {\em CVPR}, pages 17006--17015, 2023.
\bibitem{yu2018bisenet}
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.
\newblock Bisenet: Bilateral segmentation network for real-time semantic
segmentation.
\newblock In {\em ECCV}, pages 325--341, 2018.
\bibitem{zakharov2020fast}
Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.
\newblock Fast bi-layer neural synthesis of one-shot realistic head avatars.
\newblock In {\em ECCV}, pages 524--540, 2020.
\bibitem{zakharov2019few}
Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.
\newblock Few-shot adversarial learning of realistic neural talking head
models.
\newblock In {\em ICCV}, pages 9459--9468, 2019.
\bibitem{zeng2022fnevr}
Bohan Zeng, Boyu Liu, Hong Li, Xuhui Liu, Jianzhuang Liu, Dapeng Chen, Wei
Peng, and Baochang Zhang.
\newblock {FN}e{VR}: Neural volume rendering for face animation.
\newblock In {\em NeurIPS}, 2022.
\bibitem{zhang2018unreasonable}
Richard Zhang, Phillip Isola, Alexei~A Efros, Eli Shechtman, and Oliver Wang.
\newblock The unreasonable effectiveness of deep features as a perceptual
metric.
\newblock In {\em CVPR}, pages 586--595, 2018.
\bibitem{zhang2023sadtalker}
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan,
and Fei Wang.
\newblock Sadtalker: Learning realistic 3d motion coefficients for stylized
audio-driven single image talking face animation.
\newblock In {\em CVPR}, pages 8652--8661, 2023.
\bibitem{zheng2023potter}
Ce Zheng, Xianpeng Liu, Guo-Jun Qi, and Chen Chen.
\newblock Potter: Pooling attention transformer for efficient human mesh
recovery.
\newblock In {\em CVPR}, pages 1611--1620, 2023.
\bibitem{zheng2023pointavatar}
Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael~J Black, and Otmar
Hilliges.
\newblock Pointavatar: Deformable point-based head avatars from videos.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 21057--21067, 2023.
\bibitem{zhu2016face}
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan~Z Li.
\newblock Face alignment across large poses: A 3d solution.
\newblock In {\em CVPR}, pages 146--155, 2016.
\bibitem{zhuang2022mofanerf}
Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao.
\newblock Mofanerf: Morphable facial neural radiance field.
\newblock In {\em Computer Vision--ECCV 2022: 17th European Conference, Tel
Aviv, Israel, October 23--27, 2022, Proceedings, Part III}, pages 268--285.
Springer, 2022.
\end{thebibliography}