RealDenseFace: Real-time Monocular 3D Face Reconstruction from
Dense UV-space Priors

Prior definition. The correspondence prior provides a dense 2D alignment target for each projected FLAME vertex. However, 2D correspondence alone may not solve the depth ambiguity. We therefore introduce a relative-depth prior to provide complementary geometric constraints for each vertex.

Network architecture. We instantiate $f_{\theta}$ with a DINOv3 [36] image encoder and two UV-space decoder branches. The encoder extracts multi-scale features from the input image. A learnable UV feature map serve as queries tied to the FLAME UV layout. Each decoder branch fuses these UV queries with the image features and produces a UV-aligned feature map. We use separate branches for correspondence and relative depth. Each branch is followed by a DPT head that predicts the target map and its log-variance map.

Training. Following FlowFace [37], we supervise the predictions at two levels. The dense UV loss supervises the whole UV map, while the vertex loss supervises the per-vertex values. For both levels, we use Gaussian negative log-likelihood loss [19]. Let $q\in\{\mathrm{c},\mathrm{d}\}$ denote either correspondence or relative depth. For UV pixel $p$, let $y_{p}^{q}$, $\hat{y}_{p}^{q}$, and $\hat{\ell}_{p}^{q}$ denote the ground-truth target, predicted target, and predicted log-variance. The dense loss is

where $w_{p}$ denotes different weights for different head areas in UV space. Please refer the supplementary document for details.

We solve FLAME fitting as a nonlinear least-squares problem over the dense priors predicted in Section 3.1. Previous dense-prior fitting methods [16, 37] rely on a generic optimizer such as Adam, which require hundreds of iterations for convergence. In contrast, our objective is explicitly a sum of squared residuals, making it well suited to Gauss-Newton optimization, which converges in far fewer iterations than Adam. We derive close-form Jacobians and implement the solver with custom CUDA kernels.

Parameter grouping. We group FLAME parameters into identity parameters and dynamic parameters. The identity coefficients are denoted by $\beta\in\mathbb{R}^{300}$. For frame $t$, the dynamic parameters are denoted as

where $\boldsymbol{\psi}_{t}\in\mathbb{R}^{100}$ denotes expression, $\boldsymbol{\theta}_{t}\in\mathbb{R}^{3}$ is the global rotation, $\boldsymbol{\phi}_{t}\in\mathbb{R}^{12}$ is the local articulation parameters, and $\mathbf{t}_{t}\in\mathbb{R}^{3}$ is the global translation. Given $(\beta,\mathbf{x}_{t})$, FLAME produces posed vertices and joints

During optimization, we alternate between updating the dynamic parameters with identity fixed and updating the identity parameters with the dynamic parameters fixed.

Vertex-wise targets from dense priors. For each FLAME vertex $i$, we sample the predicted correspondence, relative-depth, and log-variance maps at its pre-defined UV coordinate $\nu_{i}$, yielding$\hat{\mathbf{u}}_{t,i},\ \hat{d}_{t,i},\ \hat{\ell}^{\mathrm{c}}_{t,i},\ \hat{\ell}^{\mathrm{d}}_{t,i}.$ These sampled quantities are fixed targets during fitting. Given the current FLAME parameters and camera, we compute the projected 2D vertex position and the relative depth

where $\Pi$ denotes the camera projection and $z^{\mathrm{cam}}_{t,\mathrm{neck}}$ is the camera-space depth of the FLAME neck joint. Fitting then minimizes the difference between the targets $(\hat{\mathbf{u}}_{t,i},\hat{d}_{t,i})$ and the current FLAME estimates $(\mathbf{u}_{t,i},d_{t,i})$.

Gauss-Newton optimization. For each group update, we linearize the current residual vector $\mathbf{r}$ with respect to the active variables and solve the damped Gauss-Newton system

where $\mathbf{J}$ is the Jacobian matrix and $\lambda_{\mathrm{GN}}$ is the damping coefficient. The active variables are then updated by

For dynamic-parameter updates, $\mathbf{J}$ contains derivatives with respect to $\mathbf{x}_{t}$. For identity-parameter updates, $\mathbf{J}$ contains derivatives with respect to $\beta$. We solve the resulting normal equations with Cholesky factorization. More details are provided in the supplementary material.

We use the optimizer in Section 3.2 under three settings: single-image fitting, offline sequence reconstruction, and online tracking. These settings share the same vertex-wise targets, fitting energy, and Gauss-Newton optimization, and differ only in how the update of dynamic-parameter and identity-parameter is scheduled.

Single-image fitting. For a single image, we initialize the identity ($\beta$) with the MICA prediction ($\beta_{\mathrm{mica}}$), and first optimize only the global pose parameters $\{\boldsymbol{\theta},\mathbf{t}\}$ while keeping expression and identity fixed. This pose-only stage gives a stable initialization and reduces the risk of poor local minima in the subsequent full fitting. Starting from this initialization, we run group descent between the dynamic parameters $\mathbf{x}$ and the identity parameters $\beta$: each iteration updates $\mathbf{x}$ by minimizing $E_{\mathbf{x}}$ with $\beta$ fixed, and then updates $\beta$ by minimizing $E_{\beta}$ on the same frame with $\mathbf{x}$ fixed.

Offline sequence reconstruction. For a monocular sequence, we first apply the single-image procedure to the first frame to initialize $\beta$ and $\mathbf{x}_{0}$. We then fix $\beta$ and track the full sequence in temporal order. For each frame, $\mathbf{x}_{t}$ is initialized as the optimized $\mathbf{x}_{t-1}$ and refined by minimizing $E_{\mathbf{x}_{t}}$. Given the fitting results, we select keyframes by farthest point sampling to cover diverse facial expressions and poses, and use these keyframes to refine the shared identity. Specifically, we run group descent on the selected keyframes. With the refined identity, we run another full tracking pass over the sequence. We repeat this keyframe identity refinement and full-sequence tracking for three rounds.

Online tracking. In the online setting, the input is a video stream, so frames are processed in temporal order without access to future frames. The first frame is fitted as above. It is then inserted into a fixed-size keyframe buffer and used to initialize the shared identity. For each subsequent frame, we initialize $\mathbf{x}_{t}$ as the previous dynamic parameters $\mathbf{x}_{t-1}$, keep the current identity fixed, and update only the dynamic parameters by minimizing $E_{\mathbf{x}_{t}}$.

Dataset preprocessing. Following Pixel3DMM [15], we use three studio-captured datasets for prior network training: AVA-256 [26], NPHM [14], and FaceScape [46]. For AVA-256, each frame is associated with a high-quality tracked mesh with consistent topology. We transfer the tracked mesh topology to FLAME and obtain high-quality FLAME registrations accordingly. Since AVA-256 is a video dataset, we further estimate SMIRK [30] expression coefficients and perform farthest point sampling for each subject to select around 60 frames with diverse expressions. After processing AVA-256, we train an initial prior network. For NPHM and FaceScape, although topology-consistent tracked meshes are not available, both datasets provide high-quality scan meshes. We therefore fit FLAME to these scans using Iterative Closest Point (ICP) together with the dense alignment predicted by the model pretrained on AVA-256, as illustrated in Fig. 3. In addition, NPHM does not provide the original captured multi-view images, but it provides textured scan meshes. We therefore render training images in Blender by randomly sampling HDRI environment lighting and camera viewpoints, as shown in Fig. 3. This preprocessing pipeline yields approximately 349K, 384K, and 244K training images from AVA-256, NPHM, and FaceScape, respectively.

Focal-length search. When camera intrinsics are not available, we estimate the focal length before fitting. For single-image fitting, the search is performed on the input image; for sequence reconstruction, it is performed on the first frame. We do not include the focal length directly in the Gauss-Newton variables. Instead, we perform a one-dimensional search outside the main fitting problem: for each candidate focal length, we construct the corresponding camera model, run a short fitting steps, and evaluate the resulting fitting energy. The focal length with the lowest energy is selected using golden-section search over a field-of-view range of $5^{\circ}$ to $40^{\circ}$ with $5$ iterations.

Feed-forward methods. We compare with representative feed-forward monocular face reconstruction methods, including DECA [13], EMOCA [11], TokenFace [45], MICA [48], SMIRK [30], and SHeaP [35]. Among them, MICA estimates only the neutral facial shape. SMIRK and SHeaP are two recent strong baselines for expressive monocular reconstruction. SMIRK improves expressive reconstruction with an analysis-by-neural-synthesis training scheme, while SHeaP predicts head geometry under Gaussian-based rendering supervision.

Optimization-based methods. We also compare with optimization-based methods, including VHAP [29], FlowFace [37], and Pixel3DMM [15]. VHAP is a traditional sequence-based tracking method using photometric and landmark objectives, and we evaluate it only for offline sequence reconstruction and online tracking because the photometric constraints are not meaningful in the single-image fitting setting. FlowFace predicts dense UV-to-image correspondences for fitting, while Pixel3DMM predicts screen-space UV coordinates and surface normals for fitting.

Official benchmark protocol. The NeRSemble SVFR benchmark [20, 15] evaluates single-image FLAME reconstruction on monocular views from the NeRSemble dataset [20]. For each test scan, the benchmark reconstructs reference geometry from multi-view images using COLMAP and measures the distance from the reference point cloud to the submitted FLAME mesh after alignment. The official evaluation contains $391$ scans and reports three metrics: L1 Chamfer distance, L2 Chamfer distance, and normal similarity between the predicted mesh and the reference point cloud. It evaluates two tasks: neutral reconstruction, which measures the subject-specific neutral facial shape, and posed reconstruction, which measures the expressive posed geometry of the input frame. Since the test annotations and evaluation code are not released, methods are evaluated by submitting reconstructed FLAME parameters to the official evaluation server. Table 1 reports the official results.

Official benchmark results. Our method achieves the best overall performance on the official NeRSemble SVFR benchmark [20] in both neutral and posed reconstruction. On neutral reconstruction, our method reduces the L1 error from $1.654$ to $1.637$ compared with Pixel3DMM [15], while also achieving the best L2 error and normal similarity. On posed reconstruction, our method similarly improves over Pixel3DMM, reducing the L1 error from $1.659$ to $1.610$ and the L2 error from $1.118$ to $1.086$. These results show that the proposed dense priors and Gauss-Newton solver improve reconstruction accuracy under the official evaluation protocol.

Extended benchmark. Because the official benchmark releases no ground-truth scans or evaluation code, we cannot inspect predictions qualitatively, run ablations, or analyze reconstruction behavior under controlled views. We therefore construct an extended NeRSemble-style benchmark of $1{,}150$ scans from $20$ subjects disjoint from the $391$ official scans. Following Pixel3DMM [15], we reconstruct ground-truth pointclouds with COLMAP from the multi-view images. Our method achieves the best performance here as well. The detailed protocol and results are reported in the supplementary material.

Qualitative comparison. The qualitative comparisons in Fig. 4 further illustrate the advantage of our method on challenging posed examples. Our reconstructions remain stable under large head rotations and strong expressions, while SMIRK [30] often fails to recover the correct expression and SHeaP [35] and Pixel3DMM [15] tend to under-fit expressive regions. The difference is particularly visible in side-view mouth-opening cases, where dense 2D alignment alone is insufficient to resolve the full 3D facial structure. By combining UV-space correspondence with relative-depth constraints, our method provides stronger geometric priors to the fitting stage and better recovers the posed facial geometry. The neutral reconstruction comparisons in Fig. 5 show a similar trend: our method produces stable identity geometry while still refining the initial identity estimate through optimization.

Runtime. On a single RTX 4090 GPU, the full single-image fitting pipeline takes $64.4\,\mathrm{ms}$ with known camera intrinsics and $181.9\,\mathrm{ms}$ when focal-length search is conducted. We provide a per-stage runtime breakdown in the supplementary material.

Evaluation protocol. For sequence reconstruction, we evaluate monocular video-based neutral reconstruction on five NeRSemble sequences [20] with noticeable head motion using a frontal camera. Since feed-forward monocular reconstruction methods such as MICA [48], SMIRK [30], and SHeaP [35] do not directly recover a single canonical identity from a video sequence, we evaluate them by averaging the predicted FLAME shape coefficients over all frames and then computing the neutral reconstruction metrics from the averaged shape. We also include VHAP in this sequence setting. This protocol allows us to compare how well different methods consolidate identity information from multiple frames of the same subject under changing poses.

Offline reconstruction results. As shown in Table 3, our offline sequence reconstruction achieves the best neutral reconstruction quality among all compared methods. Compared with the strongest previous monocular optimization baseline Pixel3DMM [15], our method reduces CD from $1.76$ to $1.69$, while also improving NC from $0.930$ to $0.933$ and $R^{2.5}$ from $76.9$ to $78.5$. It also outperforms MICA [48], which obtains CD $1.85$, by a clear margin. Importantly, unlike our single-image fitting pipeline, this sequence experiment does not use MICA for identity initialization, but initializes the optimization from zero. This result indicates that with sufficiently rich priors across frames, the proposed fitting framework can recover accurate identity geometry directly from monocular video.

Online tracking results. Our online tracking results remains close to the offline reconstruction while operating in the streaming setting. Each frame is processed only once in temporal order, and the online optimizer uses only the current frame and previously stored states, without access to future frames. Despite this stronger constraint, the online version achieves CD $1.73$, NC $0.930$, and $R^{2.5}=77.8$, outperforming all previous baselines on CD and $R^{2.5}$ and remaining close to the offline result. The qualitative behavior in Fig. 6 further shows that the neutral reconstruction error gradually decreases as more keyframes are selected and incorporated, demonstrating that the online tracking method can progressively refine identity geometry in a fully streaming setting.

Runtime performance. The sequence runtime is measured on a $25.5$-second video containing $1862$ frames at $73$ FPS, using a single NVIDIA RTX 4090 GPU. As expected, feed-forward methods remain the fastest because they avoid iterative fitting, with SHeaP reaching $343.96$ FPS in this setting. Among optimization-based methods, previous baselines are substantially slower: VHAP requires $2058$ seconds and Pixel3DMM [15] requires $2994$ seconds on the evaluated sequences. In contrast, our offline sequence reconstruction finishes in $84$ seconds while achieving better reconstruction accuracy. More importantly, the online version processes the same sequence in $23$ seconds, corresponding to $83.96$ FPS, and therefore runs faster than the input frame rate. A per-stage runtime breakdown is provided in the supplementary material.

Impact of dense priors. We analyze the contribution of the relative-depth prior by removing the depth term and optimizing the model using correspondence alone. As shown in Fig. 7, dense 2D alignment already provides strong UV-to-image constraints, but remains ambiguous in depth. This is particularly evident under side views, where the depth of the chin and nose cannot be fully resolved from alignment cues alone. By introducing the relative-depth prior, our method receives additional geometric constraints along the viewing direction, which helps recover more plausible facial structure in these ambiguous cases. This effect is also reflected in the posed reconstruction results in Table 4, where removing the depth prior leads to worse geometric accuracy.

Impact of pose initialization. We evaluate the effect of the pose initialization stage by disabling the pose-only optimization before the full fitting procedure. Because pose estimation is highly non-linear, directly optimizing pose and expression from a generic initialization can trap the solver in poor local minima, especially for large head rotations. As illustrated in Fig. 8, without pose initialization the reconstructed mesh can become severely distorted under profile or near-profile views. A short pose-only initialization stage provides a better starting point for subsequent optimization and improves robustness in challenging poses. This behavior is also confirmed quantitatively in Table 4, where removing pose initialization degrades reconstruction performance.

Impact of MICA. We also study the role of MICA-based identity initialization. MICA [48] provides a strong subject-specific identity prior, which helps disentangle the identity and expression spaces of FLAME before iterative fitting. In practice, this leads to a cleaner identity estimate and reduces the tendency of expression-dependent geometry to leak into the identity component. As a result, the neutral facial shape can be reconstructed more accurately when MICA is used for initialization. This trend is supported by the quantitative results in Table 4, where removing MICA causes a clear drop on the neutral reconstruction task.

Impact of the Gauss-Newton solver. We further evaluate the impact of our Gauss-Newton optimizer by replacing it with Adam. As shown in Fig. 9, our solver reduces the fitting energy much more rapidly and converges to a lower final energy within far fewer iterations. This efficiency gap is substantial: Table 5 shows that Adam requires 800 steps and 18 seconds to optimize a single image, whereas our Gauss-Newton solver reaches a lower final energy with only 10 steps and $29.64\,\mathrm{ms}$. Therefore, the proposed solver improves optimization speed by several orders of magnitude, making real-time fitting practical. In terms of reconstruction quality, the NeRSemble results (Table 4) also favor our solver overall, indicating that this large gain in efficiency does not come at the cost of accuracy.