A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields
Extracted source text · arXiv 2310.05837
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{cvm}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{color}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\usepackage[capitalise, nameinlink]{cleveref}
\crefname{section}{Sec.}{Sec.}
\newcommand{\keywords}[1]{{\bf \emph{Keywords: #1}}}
\newcommand{\warning}[1]{{\color{black}#1}}
\cvmfinalcopy
% \cvmfinalcopy % *** Uncomment this line for the final submission
\def\cvmPaperID{****} % *** Enter the cvm Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ifcvmfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{A Real-time Method for Inserting Virtual Objects into Neural Radiance Fields}
%\title{Inserting Virtual Objects into Neural Radiance Fields}
\author{Keyang Ye\textsuperscript{1}, Hongzhi Wu\textsuperscript{1}, Xin Tong\textsuperscript{2}, Kun Zhou\textsuperscript{1}\\
\textsuperscript{1} State Key Lab of CAD\&CG, Zhejiang University, Hangzhou, China\\
\textsuperscript{2} Microsoft Research Asia, Beijing, China \\
{\tt\small kunzhou@acm.org}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
% \and
% Second Author\\
% Institution2\\
% First line of institution2 address\\
% {\small\url{http://www.author.org/~second}}
}
\maketitle
% \thispagestyle{empty}
%%%%%%%%% ABSTRACT
\begin{abstract}
We present the first real-time method for inserting a rigid virtual object into a neural radiance field, which produces realistic lighting and shadowing effects, as well as allows interactive manipulation of the object. By exploiting the rich information about lighting and geometry in a NeRF, our method overcomes several challenges of object insertion in augmented reality. For lighting estimation, we produce accurate, robust and 3D spatially-varying incident lighting that combines the near-field lighting from NeRF and an environment lighting to account for sources not covered by the NeRF. For occlusion, we blend the rendered virtual object with the background scene using an opacity map integrated from the NeRF. For shadows, with a precomputed field of spherical signed distance field, we query the visibility term for any point around the virtual object, and cast soft, detailed shadows onto 3D surfaces. Compared with state-of-the-art techniques, our approach can insert virtual object into scenes with superior fidelity, and has a great potential to be further applied to augmented reality systems.
\end{abstract}
\keywords{Neural radiance field, all-frequency rendering, shadow, augmented reality.}
%%%%%%%%% BODY TEXT
\section{Introduction}
Neural Radiance Field (NeRF) is a popular technique for novel view synthesis, by representing a scene with implicit fields of density and view-dependent color, trained end-to-end with respect to input images~\cite{original_nerf}. Substantial research efforts have been made to extend the original work to different scenarios (e.g., dynamic scenes~\cite{dynamic_nerf}, human bodies~\cite{nerf_human_act} or illumination variations~\cite{nerf_w}).
Despite that NeRF is a promising, novel representation with rich information about a scene, its application to augmented reality (AR) is still limited. Related work focuses on inserting a NeRF into another one~\cite{editable_nerf}, or a NeRF onto a background photograph~\cite{cao2023real}. To our knowledge, very few existing techniques exploit the information in a NeRF to perform virtual object insertion, a classic AR task.
\warning{While our task looks straightforward at first glance, a number of challenges arise in inserting a virtual object into a NeRF. First, the input NeRF may not completely cover the complete lighting information of a scene. How to compensate for the potentially missing light sources? Second, it is difficult to represent and/or precompute near-field lighting in an efficient manner. Moreover, complex occlusion/shadowing effects between the virtual object and the NeRF must be modeled with high fidelity and performance.
%The key to this common task is the proper estimation of lighting and geometry from the NeRF. Without accurately estimated lighting, the virtual objects would be too bright/dark, or exhibit unconvincing reflections. On the other hand, without accurate scene geometry, the scene could not correctly occlude the virtual object, and the shadow cast by the virtual object would be problematic.
To tackle the above challenges, we present in this paper the first real-time method for inserting a virtual object into a neural radiance field, which produces realistic rendering with shadowing and occlusion effects, as well as allows interactive manipulation of the virtual object. Our method takes as input an HDR NeRF and one or more virtual objects only, and requires a modest overhead for precomputation and storage. To account for light sources not covered by NeRF, we estimate a distant environment lighting with inverse rendering. To efficiently handle near-field lighting, we sample the NeRF and combine with the environment lighting as the incident lighting, expressed in spherical Gaussians (SG) for high-performance rendering. To rapidly model shadowing effects, we precompute the self-visibility of the virtual object, and store the result as a field of spherical signed distance fields (SSDF)~\cite{all_freq}, which can be queried efficiently in real time.} Our method compares favorably with state-of-the-art techniques in generating high-fidelity insertion results.
%For lighting estimation, we extrapolate the NeRF to obtain a distant environment lighting, which is combined with the local lighting ray-traced from the NeRF, and express in spherical Gaussians (SG) for subsequent efficient rendering. For occlusion, we compute an opacity map by integrating the density field of NeRF. For shadowing, we precompute the self-visibility of the virtual object and store the result as a field of spherical signed distance fields (SSDF)~\cite{all_freq}, which can be queried efficiently in real time. Our method compares favorably with state-of-the-art techniques in generating high-fidelity insertion results.
%ray-trace into the NeRF to obtain spatially-varying incident lighting expressed
%We make use of the lighting and geometry information extracted from the input NeRF and implement a real-time neural rendering pipeline with Instant-NGP~\cite{instant_ngp}.
%The Neural Radiance Fields (NeRFs)~\cite{original_nerf} represent scenes with implicit fields of density and view-dependent color, which enables to synthesize photo-realistic novel views of a scene from a sparse set of images by volume rendering. Compared to classic methods of view synthesis such as structure-from-motion~\cite{sfm} and image-based rendering~\cite{ibr}, NeRF handles complex geometry and material reflectance properties through compact multilayer-perceptrons (MLPs), exhibiting an unparalleled level of fidelity over a large camera baseline. Due to these excellent features mentioned above, NeRF has been researched and expanded widely in recent years. D-NeRF~\cite{dynamic_nerf} extends NeRF to dynamic scenes. NeRF in the wild~\cite{nerf_w} extends NeRF to scenes with variable illuminations. Animatable NeRF~\cite{nerf_human_act} extends NeRF to human bodies by representing deformation fields with linear blending skinning (LBS) models. Instant-ngp~\cite{instant_ngp} enables NeRF to be well trained in a few minutes, and supports real-time rendering. These works greatly expand the application fields of NeRF, gradually making it a mature technology.
%To the best of our knowledge, there is almost no work attempts to insert virtual objects (represented as meshes) into NeRFs. Similar works including inserting NeRF into NeRF~\cite{editable_nerf} and inserting NeRF into reality~\cite{nerf_to_reality} just simply put things together without considering fidelity compositing. However, inserting virtual objects into scenes is one of the most fundamental technique in augmented reality. Although commercial AR platforms like ARKit
%\footnote{ARKit. Version 6. Apple. 2022}
%and ARCore
%\footnote{ARCore. Version 1.37.0. Google. 2022}
%provide accurate tracking technique to support object placement, compositing virtual objects into scenes in a realistic and natural way is still a challenging task.
%The challenge comes from lighting and geometry. Without correct lighting estimation, the rendered virtual objects will be too bright, too dark, or show fake reflections, which makes it outstanding and unconvincing. Similarly, without accurate geometry estimation, objects in the scene cannot correctly occlude virtual objects. Virtual objects are either always in front of all objects when ignoring occlusion, or appear significant cracks or overlaps on occluded boundaries when using rough geometry information. In addition to the impact of the scene on virtual objects, virtual objects can also cast shadows or reflections to the scene. Lack of these effects also reduces fidelity.
%To solve problems mentioned above, we propose a real-time method for inserting virtual objects into NeRFs, which produces realistic rendering with shadow and occlusion effects, while also enables users to transform the virtual object and edit its material interactively. Our method only takes multi-view posed HDR images and virtual objects as inputs, and displays object insertion results in real time. The storage consumption and the precomputation time for launching our pipeline are little. We make the full use of the lighting and geometry information extracted from NeRF and build a real-time rendering pipeline based on instant-ngp~\cite{instant_ngp} with the following contributions.
% \begin{itemize}
% \item{For lighting estimation, we compute the incident light at the virtual object center by ray tracing. Rays intersecting with the NeRF will obtain local lighting by NeRF volume rendering. The remaining rays will obtain the environment lighting estimated by inverse rendering, which contains all lights unrecorded by NeRF. Compared to learning-based methods~\cite{IRAdobe, emlight} that only output single or 2D spatially-varying incident lighting with weak generalization ability, our method provides accurate, robust, high-frequency and 3D spatially-varying incident lighting represented by spherical Gaussian (SG) lobes for real-time rendering.}
% \item{For geometry occlusion, we use the depth map of virtual objects to update the sampling range of NeRF, which enables us to compute the opacity maps by integrating density values of samples. We achieve detailed occlusion and even translucency effects by blending the virtual object and the background scene using the opacity map instead of depth test.}
% \item{For shadows, we precompute the field of spherical signed distance field~\cite{all_freq} (SSDF field) for each virtual object like the shadow field~\cite{shadow_field} and compress it with the principal component analysis (PCA) method~\cite{PCA}. When rendering, the SSDF are queried at estimated surface points and used as the visibility term for inner product with the SG incident lighting. Compared to general AR applications using shadow maps to generate hard shadows on specified planes, our method is able to cast soft, detailed and physical shadows on any irregular surfaces. In addition, to reduce the time and space consumption of SSDF fields, we propose a simple method to approximate the SSDF at points far from the virtual object from the SSDF at points closed to the object, which makes our method more practical.}
% \end{itemize}
%Our contribution is not only designs of specific algorithms, but also the design of a complete rendering framework. Although there may be room for improvement in each step, the experimental results of our method can be treated as a baseline for subsequent related research.
Our main contribution is the first complete framework to insert a virtual object into a neural radiance field in real time, an increasingly popular representation. The rest of this paper is structured as follows. \cref{related} discusses the related work, and~\cref{methods steps} describes our method. In~\cref{experiment sec}, we present experiment results for both synthetic and real scenes. Finally, \cref{conclusion sec} concludes the paper.
\section{Related Work}
\label{related}
%Our approach lies at the intersection of multiple fields. We briefly review the related prior works below.
Below we review three categories of work most related to this paper.
%Lighting estimation is a classic challenge in computer graphics. It is critical for realistic relighting in objects insertion. We categorize the previous methods into scene-based and object-based methods.
%and the input image is considered as a partial content of the environment map, so it is similar to the task of image completion.
\textbf{Lighting estimation.} Here previous work can be divided into scene-based and object-based methods. Scene-based methods estimate the lighting from partial view(s) of the scene, which is similar to the task of image completion. The missing part is approximated by directly copying from input images~\cite{khan2006image}, or by searching a panorama database for an environment map similar to the input~\cite{karsch2014automatic}. Recently, deep learning is employed to fill in the missing information. Convolutional Neural Network (CNN) or Generative Adversarial Network (GAN) are used to predict an environment map from input images with limited fields of view~\cite{gardner2017learning, deeplight, emlight, ma2021neural}. Garon et al.~\cite{garon2019fast} take images with coordinate masks and corresponding local patches as input to a neural network to predict 2D spatially-varying lighting expressed in spherical harmonics (SH). However, learning-based methods suffer from weak generalization ability, when the input images differ considerably from the training set. For example, the models trained on indoor scene datasets are difficult to generalize to outdoor~\cite{emlight, deep_parametric}.
On the other hand, object-based methods estimate lighting from the appearance of object(s) presented in the scene. A common technique is to jointly optimize the object material and the environment lighting by inverse rendering~\cite{nerfactor, nerv, nerd, physg, zhang2022modeling}. This is a highly ill-posed problem, and its solution often requires strong priors, such as smoothness~\cite{nerfactor} or Lambertian only~\cite{azinovic2019inverse}. Recently, deep learning has also been applied. Much of related work is targeted for human faces, due to the existing prior knowledge about their shapes and appearance~\cite{tewari2017mofa, calian2018faces}.
Our method applies inverse rendering to estimate a distant environment lighting from a NeRF to compensate for potential missing lights not recorded in the NeRF. We take both the near-field lighting from NeRF and the distant environment lighting into consideration and build an inverse rendering pipeline, similar to~\cite{zhang2022modeling}.
%Therefore, different from works that focus on object-centric scenes and consider objects are only illuminated by environment lighting~\cite{physg, nerfactor}. Our method takes both the local lighting in NeRF and environment lighting from infinity into consideration and build a inverse rendering pipeline based on the method of Zhang et al~\cite{zhang2022modeling}.
\textbf{Visibility estimation.} Most contemporary AR systems estimate depths for visibility computation. Depths can be directly acquired with specialized devices~\cite{kendall2017end, bhat2021adabins, chang2018pyramid}. To reduce the hardware requirement, single-view approaches apply deep learning~\cite{alhashim2018high, godard2017unsupervised} or employ inverse rendering that adds depths as additional unknowns to a joint optimization~\cite{IRAdobe}. When a sequence of images is available, Multi-View Stereo (MVS) estimates depths by matching pixels~\cite{wang2018mvdepthnet} or features~\cite{mvs_net}. Waston et al.~\cite{watson2023virtual} sidestep the depth estimation and directly predict the occlusion mask from a feature map. However, these methods still fall short in some cases, especially on translucent objects or objects with tiny structures, such as leaves on plants. In comparison, our method exploits the implicit geometry in a NeRF, which can handle cases that are challenging for traditional explicit representations.
\textbf{Real-time shadows.} Shadow field~\cite{shadow_field} precomputes visibility maps of samples from concentric shells surrounding the virtual object and use SH as basis to compress them. However, the approach only supports soft shadows when using low order SH basis. Wang et al.~\cite{all_freq} introduce spherical signed distance fields (SSDF) for high-frequency shadows. Their method precomputes the SSDF at every vertex that receives shadows, and compresses them with PCA method, which requires both the scene and the object being static. Kei et al.~\cite{all_freq_sphere} approximate objects with a set of spheres and project them to the hemisphere integral region of the shading point as occluded patches. Dynamic objects are supported at the cost of intensive computation. We propose an SSDF field, which combines the idea of shadow field and SSDF, to efficiently render shadows cast by the virtual object.
%We take the depth of the virtual object and a NeRF as inputs and calculate the opacity map to blend the virtual object and the background, which creates detailed opaque and translucent occlusion effects.
%One of the most challenging task of augmented reality is occlusion, which means hiding virtual objects behind real things. Depth estimation is the main solution of current AR occlusion systems. Depth can be estimated directly by specialized device~\cite{kendall2017end, bhat2021adabins, chang2018pyramid}, such as binocular cameras with structured light and Lidar. To reduce the requirement of device, several works attempt to estimate depth from a single RGB image by deep learning~\cite{alhashim2018high, godard2017unsupervised} or by inverse rendering that adds depth to the joint optimization~\cite{IRAdobe}. When a sequence of images is available, Multi-View Stereo (MVS) estimates depth of a reference image using several source images by matching pixel~\cite{wang2018mvdepthnet} or deep features~\cite{mvs_net} to create a cost volume, which can then be processed and obtain the depth finally. The estimated depth map is then used for the depth test with the depth of virtual objects to create an occlusion mask. Waston et al.~\cite{watson2023virtual} sidesteps the depth estimation and directly predict the occlusion mask from a feature map. However, these methods still fall short in some cases, especially real objects with many tiny structures, such as leaves on plants. For translucent real objects, the virtual object can be seen through the occlusion, which is infeasible for these methods. Our method takes the advantage of the ability of NeRF's implicit geometry representation. We take the depth of the virtual object and a NeRF as inputs and calculate the opacity map to blend the virtual object and the background, which creates detailed opaque and translucent occlusion effects.
%In addition to the occlusion of objects in the scene, due to the insertion of the virtual object, the incident light of each point in the scene is partially occluded by the virtual object, which results in shadow effects. To cast shadows onto the scene, the visibility of the virtual object at each surface point of the scene should be calculated in real time to support movements of the virtual object. Shadow field~\cite{shadow_field} precompute visibility maps of samples from concentric shells surrounding the virtual object and use SH as basis to compress them. When calculating shadows, visibility term can be obtained by SH interpolation. However, this method only supports very soft shadows when using low order SH basis. Wang et al.~\cite{all_freq} introduce spherical signed distance fields (SSDF) for high-frequency shadows. This method precomputes SSDF of every vertex that receiving shadows and compress them using PCA method, which requires both the scene and the object to be static. Kei et al.~\cite{all_freq_sphere} approximates objects with a set of spheres and projects them to the hemisphere integral region of the shading point as occluded patches. This method supports dynamic objects at the cost of large computation. We propose SSDF field, the combination of shadow field and SSDF, to enable physical shadows cast by the moving virtual object.
% \textbf{Virtual object insertion.} Object insertion is one fundamental task in augmented reality. The classic object insertion process consists of object tracking, rendering and compositing. We do not review object tracking here, as it is relatively mature and commercial platforms like ARKit and ARCore provide APIs for rapid development.
% %Commercial AR platforms like ARCore and ARKit provide abundant APIs to support fast development of AR applications. For example, ARCore supports accurate feature points tracking and plane detection for placing virtual objects. It also provides functions of estimating environment map or main light source direction for rendering. With the multi-view stereo (MVS) technique, ARCore can estimate depth maps to support rough occlusion effects between virtual objects and the scene.
% %In the academic field, many works contribute to part of the object insertion system. For object tracking, simultaneous localization and mapping (SLAM) techniques~\cite{orb_slam3, hdr_slam} enables to track virtual objects without any marker, which gives the system high flexibility and convenience.
% For object rendering, estimating the incident light around the virtual object is an important predecessor task. Some works estimate lighting by inverse rendering. These works usually take multi-view images as inputs and jointly optimize materials, geometry and environment lighting~\cite{nerfactor, nerv, nerd, physg, zhang2022modeling}, which is a highly ill-posed problem and needs to be constrained by some priors. Other works infer lighting directly from one image by neural networks~\cite{emlight, deep_parametric, gardner2017learning, deeplight}. LeGendre et al.~\cite{deeplight} predicts the panorama lighting from limited-FOV images by lightweight mobile net. Garon et al.~\cite{garon2019fast} use images with coordinate masks and corresponding local patches as input to predict 2D spatially-varying lighting represented as spherical harmonics (SH). However, learning based methods suffer from weak generalization ability when the input images are captured from the scenes greatly differ from the training set. In contrast, our method compute local lighting and visibility from NeRF and estimate environment lighting by inverse rendering. The local and environment lighting with visibility are combined to give accurate, robust, high-frequency and 3D spatially-varying incident light for rendering.
% For occlusion effects, there are range of methods for depth estimation from single or multiple images~\cite{IRAdobe, mvs_net, fu2018deep, chen2016single}. Objects in the scene are able to occlude the virtual object by performing depth test. However, these methods, including NeRF, only provide rough depth estimation that are likely to present artifacts on the occlusion boundary. Therefore, our method replaces depth estimation and depth test with the blending strategy to avoid unnatural occlusion caused by incorrect depth.
% It should be noted that the goal of this paper is not to design a complete object insertion system, because it is a comprehensive task that involves various techniques. We mainly focus on rendering a virtual object and compositing it into the scene. Our method illustrates significant application prospect of NeRF in AR field.
% \textbf{Neural rendering.} These works try to learn novel view synthesis from images through deep neural networks~\cite{thies2019deferred, aliev2020neural, sitzmann2019deepvoxels}. In particular, NeRF~\cite{original_nerf} implicitly presents scenes as radiance fields, which enables photo-realistic rendering by integrating density and view-dependent color inferred from multilayer-perceptrons (MLPs) along camera rays. Neural rendering, especially NeRF, has been researched and expanded widely in recent years~\cite{dynamic_nerf, nerf_w, nerfactor, instant_ngp, nerf_human_act, lombardi2021mixture, control_nerf}. Some works extend NeRF to dynamic scenes or deformable objects~\cite{dynamic_nerf, nerf_human_act, nerfies}. Some works extend NeRF to unbounded or large scale scenes~\cite{mip_nerf_360, block_nerf}. Some works enable NeRF to be relighted~\cite{nerfactor, nerv}. Some works aim to accelerate training or rendering of NeRF~\cite{instant_ngp, fast_nerf}. However, to the best of our knowledge, there is no work focusing on inserting virtual objects into NeRFs. Similar work includes inserting NeRFs into NeRFs~\cite{control_nerf, editable_nerf} and inserting NeRFs into reality~\cite{nerf_to_reality}. Our methods insert virtual objects into NeRFs using the lighting and geometry information obtained from NeRF and provides high fidelity compositing results.
% %such as dynamic NeRF~\cite{dynamic_nerf}, which introduces a deformation field that enables to fit dynamic scenes, NeRF in the wild~\cite{nerf_w}, which models the scene as static and transient parts to eliminate the impact of non-static objects and NeRFactor~\cite{nerfactor}, which builds differentiable inverse neural rendering pipeline to factorize shape and material parameters. With a multiresolution hash encoding~\cite{instant_ngp}, NeRF can be well trained in a few minutes, and render in real time, greatly expanding the application fields of NeRF.
% %NeRF contains abundant lighting and geometric information of the scene. Unlike the edit NeRF~\cite{editable_nerf}, which simply pieces together NeRFs to obtain composited results without lighting effects, our methods composite virtual objects into the scene using information queried from NeRF and combines neural rendering with mesh based rendering, which provides high fidelity rendering results.
% %Lighting estimation is a classic challenge in computer graphics. It is critical for realistic relighting in objects insertion. Lots of works try to estimate lighting from one or more images, some of which optimize the lighting by inverse rendering, while others are data-driven methods using deep neural networks to directly regress lighting. For inverse-rendering based methods, more information, such as geometry, materials and light sources, needs to be provided by the user~\cite{karsch2011rendering, zhang2016emptying, emlight} or as additional parameters to be optimized~\cite{nerfactor, nerv, nerd}, which makes usage inconvenient or make the optimization underconstrained, resulting in inaccurate and unstable results. For data-driven methods, the accuracy of the prediction severely depends on the training data. For example, models trained in indoor scene datasets are difficult to generalize to outdoor scenes~\cite{emlight, deep_parametric}. Several works use Convolutional Neural Network (CNN) or Generative Adversarial Network (GAN) to predict the panorama lighting from limited-FOV images~\cite{gardner2017learning, deeplight, emlight}, which supports high frequency rendering. The above works can only predict the global illumination. However, when objects move in the scene, local lighting effects like occlusion and indirect lighting are significant. Garon et al.~\cite{garon2019fast} use images with coordinate masks and corresponding local patches as input to neural network to predict spatially-varying lighting as spherical harmonics (SH). Our method gets precise, all-frequency and spatially-varying lighting by integrating colors along rays in all directions in NeRF and also takes lighting uncovered by NeRF into consideration.
% \textbf{Precomputed Radiance Transfer.} Precomputed Radiance Transfer (PRT)~\cite{PRT} enables to relight static objects in real time. The original method use spherical harmonics (SH) as a basis for lighting, which only supports low-frequency shading effects. Several works substitute other basis functions, such as Haar wavelets~\cite{PRT_haar_wave}, spherical radial basis functions (SRBFs)~\cite{PRT_spherical_radial} and spherical Gaussian lobes (SG)~\cite{PRT_gaussian}, to allow all-frequency effects. In addition to shading, shadows are also very important for realistic rendering. AR applications typically estimate the direction of the main light source to cast shadows onto the plane by the method of shadow maps, which only supports hard shadows. Shadow field~\cite{shadow_field} precompute visibility maps of samples from concentric shells surrounding an object and use SH as basis to compress them. When calculating shadows, visibility term can be obtained by SH interpolation. However, this method only supports very soft shadows when using low order SH basis. Wang et al.~\cite{all_freq} introduce spherical signed distance fields (SSDF) for detailed shadows. This method precomputes SSDF of every vertex that receiving shadows and compress them using PCA method, which requires both the scene and the object to be static. Kei et al.~\cite{all_freq_sphere} approximates objects with a set of spheres and projects them to the hemisphere integral region of the shading point as occluded patches. This method supports dynamic objects at the cost of large computation. Our method use SG basis for both lighting and BRDF to support all-frequency rendering and interactive BRDF editing. Similar to shadow field, we efficiently precompute SSDF field of samples surrounding the virtual object, which enables dynamic and detailed shadows.
\section{Preliminaries}
\label{preliminary}
A NeRF represents a scene with a volumetric field $\mathcal{F}$ of density $\sigma$, and color $\mathbf{c}$ which varies with a view direction $\mathbf{d}$. For a ray $\mathbf{r}(t) = \mathbf{o}+\mathbf{d}t$ emitted from camera center $\mathbf{o}$ along $\mathbf{d}$, the rendered color $\mathbf{\hat{C}}(\mathbf{d}; \mathbf{o})$ of a NeRF is computed with simple volumetric rendering~\cite{original_nerf}.
The opacity $\hat{O}(\mathbf{d}; \mathbf{o})$ and depth $\hat{D}(\mathbf{d}; \mathbf{o})$ are computed in a similar fashion: $\hat{O}(\mathbf{d}; \mathbf{o}) = \sum_{k=1}^K T\left(t_k\right) \alpha\left(\sigma\left(t_k\right) \delta_k\right)$ and $\hat{D}(\mathbf{d}; \mathbf{o}) = \sum_{k=1}^K T\left(t_k\right) \alpha\left(\sigma\left(t_k\right) \delta_k\right) t_k$. Here $t_k$ is a distance of a point sample traveled along the ray, $t_\mathrm{k}\in[t_\mathrm{n}, t_\mathrm{f}]$, where $t_\mathrm{n}$ and $t_\mathrm{f}$ are the near and far distances of the intersection of the NeRF boundary and the ray. Please refer to~\cref{sample} for an illustration of $t_\mathrm{k}$. $T\left(t_k\right)$ is the cumulative transmittance, $\sigma(t_k)$ are the density at the sample point on the ray $\mathbf{r}(t)$. In addition, $\alpha(x) = 1-e^{-x}$ and $\delta_k = t_{k+1} - t_k$ is the distance between adjacent two samples along the ray.
%To render a NeRF, a simple volumetric model is employed to integrate the field of density and color along a view ray~\cite{original_nerf}.
%NeRF approximates the color of a single pixel by emitting a ray from the center of the camera $\mathbf{o}$ through the pixel on the image plane and integrating colors and densities along the ray. Let $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}, t\in[t_\mathrm{n}, t_\mathrm{f}]$ be the camera ray ($t_\mathrm{n}$ and $t_\mathrm{f}$ are the near and far distance of the intersection of the bounded space and the ray). The color $\mathbf{\hat{C}}(\mathbf{r})$ of the pixel corresponding to the ray
%can be estimated by the NeRF volume rendering~\cite{original_nerf}.
% \begin{equation}
% \mathbf{\hat{C}}(\mathbf{r}) = \sum_{k=1}^K T\left(t_k\right) \alpha\left(\sigma\left(t_k\right) \delta_k\right) \mathbf{c}\left(t_k\right),
% \label{nerf_integral}
% \end{equation}
% \begin{equation}
% \text {where\quad} T\left(t_k\right)=\exp \left(-\sum_{k^{\prime}=1}^{k-1} \sigma\left(t_{k^{\prime}}\right) \delta_{k^{\prime}}\right),
% \label{nerf_integral_ex}
% \end{equation}
% where $\sigma(t)$ and $\mathbf{c}(t)$ are the density and color at the sample point on $\mathbf{r}(t)$, $\alpha(x) = 1-\exp(-x)$ and $\delta_k = t_{k+1} - t_k$ is the distance between adjacent two sample points. $\{t_k\}_{k=1}^K$ can be sampled by stratified sampling~\cite{original_nerf} or ray marching in occupancy grid~\cite{instant_ngp}.
%$\hat{O}(\mathbf{r})$ and $\hat{D}(\mathbf{r})$ provide important geometry information that will be frequently used for subsequent calculations.
% NeRF can be trained by a set of images $\left\{\mathcal{I}_i\right\}_{i=1}^N$ with corresponding intrinsic and extrinsic camera parameters. The parameters of the radiance field are optimized by minimizing the sum of squared reconstruction error for each pixel as follow:
% \begin{equation}
% \sum_{\mathbf{r}\in\mathcal{R}(\mathbf{P})}\left\|\mathbf{C}\left(\mathbf{r}\right)-\hat{\mathbf{C}}\left(\mathbf{r}\right)\right\|_2^2,
% \label{nerf_loss}
% \end{equation}
% where $\mathcal{R}(\mathbf{P})$ is the set of camera rays corresponding to pixels $\mathbf{P}$ of all images and $\mathbf{C}\left(\mathbf{r}\right)$ is the ground-truth color.
%$\left\{\mathcal{I}_i\right\}_{i=1}^N$
NeRF can be trained from a set of images with known intrinsic and extrinsic camera parameters. While low-dynamic-range (LDR) images are usually used, they are not suitable for physically based rendering. Therefore, we use high-dynamic-range (HDR) NeRF as input to our pipeline throughout the paper. Moreover, we employ instant-ngp~\cite{instant_ngp} as the implementation, due to its excellent performance.
%\subsection{Assumptions}
%\label{assumption}
%Our target is to insert a virtual object into a NeRF in real time. To this end, we make some assumptions for the virtual object and the scene.
\textbf{Assumptions.} We assume that the virtual object is rigid and opaque. Also its size is relatively small with respect to the input NeRF, so that the incident lighting (ignoring occlusion) does not change over its surface. For the input NeRF, we assume that the geometry incorporated in it can be sampled as surface points, and its materials are Lambertian. In the whole pipeline, we only consider the direct illumination and ignore interreflection effects.
\warning{Note that the above assumptions may be loosened, at the expense of more runtime computations. For example, one may build the incident lighting at sampled locations across the object surface, and interpolate them to obtain the result for any point. For the surface assumption of the NeRF, one can remove it by switching to a more involved inverse rendering computation, which takes scattering into account. On the high level, our framework is not limited to the above assumptions and can be extended to handle more diverse cases.}
%\textbf{Scene.} We assume that the geometry of the scene is simple in most areas. Our method is not suitable for scenes fully covered by fur or other tiny structures, which severely affects the geometry estimation. We assume the material of the scene is all diffuse. Note that this assumption is only used in light estimation and shadowing, which improves the efficiency of our method and does not have a significant impact on the visual quality of rendering.
%accelerates training to convergence in a few minutes, and supports real-time rendering. Therefore, we use instant-ngp as the NeRF framework to represent scenes and apply the full loss function to train HDR radiance fields.
%NeRF usually uses low dynamic range (LDR) images as the training set, which is not suitable for physically based rendering. Therefore, we use HDR images to train a HDR NeRF, and apply tonemapped loss~\cite{nerf_in_dark}.
% as follow:
% \begin{equation}
% \ell_{\mathrm{rgb}} = \sum_{\mathbf{r}\in\mathcal{R}(\mathbf{P})}\left\| \frac{\mathbf{C}\left(\mathbf{r}\right) - \hat{\mathbf{C}}\left(\mathbf{r}\right)}{\mathrm{sg}(\hat{\mathbf{C}}\left(\mathbf{r}\right))+\epsilon}\right\|_2^2,
% \label{hdr_nerf_loss}
% \end{equation}
% where $\mathcal{R}(\mathbf{P})$ is the set of camera rays corresponding to pixels $\mathbf{P}$ of all images, $\mathbf{C}\left(\mathbf{r}\right)$ is the ground-truth color, $\mathrm{sg}(\cdot)$ indicates a stop-gradient that treats its argument as an constant, and $\epsilon=10^{-3}$ is a tiny value to ensure the denominator not zero.
%To provide better geometry estimation for shadow casting, we add the distortion loss~\cite{mip_nerf_360} $\ell_\mathrm{dist}$ and opacity loss $\ell_\mathrm{opac}$~\cite{nerfstudio} to reduce "foggy" and "muddy" floaters.
% The opacity loss that encourages opacity to be either 0 or 1 is defined as follow:
% \begin{equation}
% \ell_{\mathrm{opac}} = \sum_{\mathbf{r}\in\mathcal{R}(\mathbf{P})}\left\| -\hat{O}(\mathbf{r})\log(\hat{O}(\mathbf{r})+\epsilon)\right\|_2^2,
% \label{opac_loss}
% \end{equation}
% where $\epsilon$ is set to $10^{-10}$.
% We note that NeRF tends to produce opaque floaters in front of cameras when the camera poses are not dense. To reduce overfitting, we assume that there are no objects very close to the camera, so we add the depth loss $\ell_\mathrm{depth}$ to encourage estimation of surface depth away from the camera:
% \begin{equation}
% \ell_{\mathrm{depth}} = \sum_{\mathbf{r}\in\mathcal{R}(\mathbf{P})}\left\| -\log(\min(\hat{D}(\mathbf{r})/r_\mathrm{bound}+\epsilon), 1)\right\|_2^2.
% \label{depth_loss}
% \end{equation}
% where $\epsilon=10^{-10}$ and $r_\mathrm{bound}$ is the radius of the NeRF bound.
%When the surface depth is optimized correctly, the depth loss becomes zero, so it will not have any negative impact. Figure~\ref{fig_depthloss} illustrates how depth loss eliminate floaters near the camera.
% \begin{figure}[h]
% \centering
% \hspace{-0.1\textwidth}
% \includegraphics[width=0.35\textwidth]{imgs/第1页.pdf}
% \caption{Floaters near the camera in NeRF. Areas closer to the camera are often not observed by other cameras, so floaters tend to be generated in these areas due to overfitting. Adding depth loss encourage these floaters to be farther away from the camera, so that they can be observed by more cameras, which overcomes overfitting problem and reduces floaters during optimization.}
% \label{fig_depthloss}
% \end{figure}
% The full loss function is: $\ell_\mathrm{rgb}+\lambda_1\ell_\mathrm{dist}+\lambda_2\ell_\mathrm{opac}+\lambda_3\ell_\mathrm{depth}$. We set $\lambda_1 = 10^{-2}$, $\lambda_2 = 10^{-3}$ and $\lambda_3 = 4\times10^{-2}$, which provides best results.
%Since the color distribution in HDR images can span many orders of magnitude, the original loss function focuses on errors of bright areas, resulting in muddy dark region after tone mapping~\ref{nerf in the dark}. The tone mapping function should have the ability to compress the dynamic range and make the loss function strongly penalize errors in dark regions. However, mapping functions like sigmoid and tanh will approach 1 rapidly when the input value get larger, which leads to huge errors in bright areas. Although the error will not greatly affect the quality of NeRF rendering since human perception is not sensitive to errors of bright areas, the virtual object will be rendered badly due to the incorrect ambient light estimated from NeRF.
%The original NeRF takes several hours of training and renders a single image in a few minutes. With the multiresolution hash encoding, instant-ngp~\cite{instant_ngp} accelerates training to convergence in a few minutes, and supports real-time rendering. Therefore, we use instant-ngp as the NeRF framework to represent scenes and apply the full loss function to train HDR radiance fields.
\section{Our method}
\label{methods steps}
\subsection{Overview}
\label{overview}
\begin{figure*}[h]
\centering
\includegraphics[width=1.0\textwidth]{figs/flowchart.pdf}
\caption{The full pipeline of our method. Our method takes an HDR NeRF and a rigid virtual object as input. We estimate the environment lighting to compensate for potential lighting not covered by the NeRF. Next, we precompute the visibility field of the virtual object as a SSDF field. At runtime, we perform ray tracing from the virtual object center to obtain the near-field lighting and the visibility of NeRF, and compute the incident lighting fitted with spherical Gaussians (SG) to fast render the virtual object. The rendered virtual object will be composited into the NeRF by blending. We also use the incident light and the SSDF field to compute $\kappa$ for shadows at each pixel (\cref{shadow ratio}), and multiply the pixel color by the corresponding $\kappa$ to produce the final result.}
\label{fig_pipeline}
\end{figure*}
%We summarize our method into five steps, including high dynamic range (HDR) NeRF training, lighting estimation, virtual object rendering, virtual object compositing and shadow casting. Our method takes multi-view HDR images, an opaque virtual object (meshes) and its materials (albedo, roughness and metallic) and transformation (rotation, translation and scaling), camera poses and some precomputation data as inputs, and outputs object insertion results as HDR images. The full pipeline of our method is shown in~\cref{fig_pipeline}. We use different colors to denote the corresponding sections of the pipeline.
We take as input an HDR NeRF (denoted as $\mathcal{F}$) and a rigid virtual object with geometry (represented as a mesh) and appearance (parameters stored in texture maps). First, to compensate for potential lighting not covered by the NeRF, we extrapolate it to an additional environment lighting, and then combine both into an incident lighting expressed in SGs for fast evaluations (\cref{env light est sec}). Next, to facilitate rapid shadow computation, we precompute the visibility field around the virtual object as a field of SSDF (\cref{SSDF fields sec}). At runtime, we render the virtual object with estimated incident lighting and the visibility field, and composite with the original NeRF as the output in real time (\cref{object insertion sec}). A graphical illustration of the pipeline is shown in~\cref{fig_pipeline}.
\warning{Note that an alternative option is to precompute an field of incident lighting, expressed in SGs. However, the non-linear nature of SGs makes the interpolation difficult, which motivates our current approach for lighting estimation.}
%Inspired by shadow field~\cite{shadow_field}, we precompute a visibility field around the virtual object. Note that while a more natural and consistent choice to express precomputed visibility is via a field of SGs, its non-linear nature makes interpolation difficult. Therefore, we adopt SSDF~\cite{all_freq}, which also allows fast rendering with incident lighting expressed as SGs~\cite{all_freq}.}
%We summarize our method into precomputation stage and real-time stage. The precomputation stage includes high dynamic range (HDR) NeRF training, environment lighting estimation and SSDF field precomputation. The real-time stage includes virtual object rendering, compositing and shadow casting, which are performed in real time on every frame. Our method takes multi-view posed HDR images, an opaque mesh object with its materials (albedo, roughness and metallic) and transformation (rotation, translation and scaling) as inputs, and outputs object insertion results as HDR images. The full pipeline of our method is shown in~\cref{fig_pipeline}. We use different colors to denote the two stages.
%Our method starts from training a HDR NeRF by multi-view posed HDR images to represent the scene. The environment lighting unrecorded by NeRF is then estimated by inverse rendering. For virtual objects to be inserted, the SSDF fields are precomputed for shadow casting.
%In the real-time stage, we render the virtual object by using the incident light at the virtual object center, which is the sum of the local lighting qurried from NeRF and the result of the environment lighting multiplied by visibility. Both the incident light and the virtual object material are represented as SG lobes to support real-time rendering and material editing.
%We render out the color results and the depth map of the virtual object at the same time, and the depth map is input to NeRF to generate the blend weight map.
%The color and opacity map of the scene corresponding to current camera pose are computed by NeRF volume rendering. Then the rendered virtual object is composited into the scene by blending according to the opacity map. For shadows, we compute the surface points from the depth map and query the SSDF from the precomputed SSDF field at each surface point to update the incident light occluded by the virtual object. %precompute SSDF of the virtual object as the visibility term that enables fast queries and interpolation. The depth map of the scene estimated by NeRF and the depth map of the virtual object are merged together to compute the surface points.
%Finally, the shadow effects are added by multiplying each pixel by a shadow ratio.
%by the inner product of the SSDF and SG lighting, and these coefficients will be multiplied by the composited color result to obtain the final output.
%The following subsections will elaborate on the steps mentioned above.
\subsection{Environment lighting estimation}
\label{env light est sec}
Ideally, all incident lighting of the virtual object comes from $\mathcal{F}$. In practice, however,
$\mathcal{F}$ may not cover the complete surroundings: there might be light sources in the scene, which are not recorded by $\mathcal{F}$. To alleviate this issue, we extrapolate $\mathcal{F}$ to an additional distant environment lighting to fully cover the virtual object. Specifically, we formulate the incident lighting at a surface point $\mathbf{x}$ of the virtual object along a direction $\mathbf{d}$ as follows:
%In order to realistically render a virtual object, it is necessary to obtain the incident light at its center. A naive approach is performing ray tracing in NeRF to get incident light from other surfaces, which is equivalent to treating the whole NeRF as a light source that illuminates the virtual object. However, it is hard to ensure that NeRF contains all objects in the scene, which means there may be some light sources unrecorded by NeRF. This challenge is very common in general AR applications, and thus we need to estimate missing lighting using the known information.
%NeRF uses a set of images as the training set, so the scene represented by NeRF only contains objects captured in images. When light sources are not captured in images, but illuminate the scene, the ambient light obtained from NeRF only includes the indirect lighting reflected from objects, which will make the rendering result incorrectly darker.
%We model the missing lighting as the environment lighting that comes from infinity. When computing the incident light at some position, we uniformly shoot rays from the position to perform ray tracing. Rays intersecting with NeRF will obtain corresponding outgoing radiance by NeRF volume rendering, which is called local lighting. The remaining rays will obtain the environment lighting depending on their directions.
%We model light sources that are not included in the scene as global lighting from infinity. Light sources included in the scene and indirect lighting reflected from objects are considered as local lighting. Local lighting is 3D spatially-varying and can be queried from the NeRF while global lighting is unknown and needs to be estimated. We model the ambient light $L_\mathrm{i}$ at a point $\mathbf{x}$ as a mixture of the global lighting $L_\mathrm{g}$ and local lighting $L_\mathrm{l}$:
%Therefore, the 3D spatially-varying incident light is formulated as follow:
\begin{equation}
L_\mathrm{i}(\mathbf{d}; \mathbf{x}) = (1-\hat{O}(\mathbf{d}; \mathbf{x})) L_\mathrm{env}(\mathbf{d}) + \hat{O}(\mathbf{d}; \mathbf{x}) L_\mathrm{nerf}(\mathbf{d}; \mathbf{x}),
\label{incident_light}
\end{equation}
%$\hat{O}(\mathbf{d}; \mathbf{x})$
where $\hat{O}(\mathbf{d}; \mathbf{x})$ is the NeRF opacity according to~\cref{preliminary}, and $L_\mathrm{i}$, $L_\mathrm{env}$, $L_\mathrm{nerf}$ are the incident lighting, the distant environment lighting and the near-field lighting from $\mathcal{F}$, respectively.
%The NeRF visibility is 0 when rays intersect with NeRF, otherwise it is 1. When there are translucent objects in the scene, the visibility value may be between 0 and 1. We use the opacity in~\cref{opac_eq} to compute the NeRF visibility as $V(\mathbf{d}; \mathbf{x}) = 1-\hat{O}(\mathbf{d}; \mathbf{x})$.
%The computation model of the ambient light is illustrated in light red area of~\cref{fig_pipeline}. The local lighting, including indirect (light yellow arrow) and direct (yellow arrows) light, can be computed by emitting sample rays from the object center to all directions and integrating color results using~\cref{nerf_integral}. With the use of the local lighting, indirect lighting and local occlusion of lights can be modeled, which are often ignored by other methods~\cite{physg}.
%\begin{figure}[h]
%\centering
%\includegraphics{imgs/第6页.pdf}
%\caption{Computation of ambient light. The ambient light is computed at the center of the virtual object, and it consists of local light and global light. The local light, including local indirect and direct light, can be obtained from NeRF. Light sources outside the scene bounding box are considered as global light from infinity.}
%\label{global_local_light}
%\end{figure}
For the environment lighting, we represent it as the sum of $M=32$ spherical Gaussian (SG) lobes, which strike a good balance between the fidelity and rendering efficiency~\cite{all_freq}:
\begin{equation}
L_\mathrm{env}(\mathbf{d}) = \sum_{k=1}^M G(\mathbf{d}; \mathbf{p}_k, \lambda_k, \boldsymbol{\mu}_k),
\label{SG_light}
\end{equation}
where $\mathbf{p}_k\in\mathbb{S}^2$ is the center, $\lambda_k\in\mathbb{R}^+$ is the sharpness, $\boldsymbol{\mu}_k\in\mathbb{R}_+^n$ is the amplitude, for a particular SG.
%\warning{view sampling of NeRF. Specifically, we randomly sample camera positions on the sphere with the NeRF center as its center and half of the NeRF grid size as its radius. We let all cameras look at the NeRF center and render views. To remove views of poor quality (usually the views outside the range of NeRF training), we score the views by the no-reference image quality assessment method: BRISQUE~\cite{brisque}, and save the views with the score lower than 50 (the higher the quality, the lower the score). We repeatedly perform the sampling until there are 100 views available.}
To estimate the SG parameters of an environment lighting, we first randomly sample views that pointing towards the center of the NeRF volume, with a distance of half of the NeRF grid size. The view will be rejected if the image quality of the corresponding NeRF rendering result is not sufficient (BRISQUE~\cite{brisque} score $>$ 50). We repeat the process until we have 100 sampled views. For the rendered image at each sampled view, we randomly sample 64 pixels that pass through $\mathcal{F}$. For each such pixel, we compute a surface point sample as: $\mathbf{x}_\mathrm{surf} = \mathbf{o} + \hat{D}(\mathbf{d}; \mathbf{o})\mathbf{d}$, similar to~\cite{nerfactor}, where $\mathbf{o}$ is the camera center. The normal of this sample is estimated by the gradient of density as $\mathbf{n}(\mathbf{x}_\mathrm{surf}) = -\nabla\sigma(\mathbf{x}_\mathrm{surf})$. Finally, the environment lighting is linked with the rendered pixels ($L_\mathrm{o}(\mathbf{x})$) according to the rendering equation as follows, which allows us to perform inverse rendering to optimize the SG parameters:
\begin{equation}
L_\mathrm{o}(\mathbf{x}) = \frac{\mathbf{a}(\mathbf{x}; \mathbf{\Theta_\mathrm{a}})}{\pi}(E_\mathrm{nerf}(\mathbf{x})+E_\mathrm{env}(\mathbf{x}; \mathbf{\Theta_\mathrm{env}})).
\end{equation}
Here $\mathbf{a}$ is the albedo, represented as a 6-layer MLP~\cite{zhang2022modeling} (whose parameters are $\mathbf{\Theta_\mathrm{a}}$) and jointly optimized along with the environment lighting. $\mathbf{\Theta_\mathrm{env}}$ are the SG parameters. $E_\mathrm{nerf}$ is the irradiance integrated from $\mathcal{F}$ via Monte-Carlo sampling, which is only computed once throughout the optimization. For $E_\mathrm{env}$, we first cache the NeRF opacity at each sample point into texture maps, and then perform importance sample according to the mixture of SGs to compute the integral during the optimization, similar to the method of Zhang et al.~\cite{zhang2022modeling}.
%use the importance sampling in each SG lobe to test its visibility according to the opacity cube map, and reduce the amplitude of the SG lobe if it is occluded, which is similar to the method of Zhang et al.~\cite{zhang2022modeling} and then it can be fast computed as the SG inner product according to~\cite{physg}.
% \begin{equation}
% L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}\right)=\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i},
% \label{render_equation}
% \end{equation}
%which allows us to perform inverse rendering to optimize the SG parameters.
%The use of SG basis allows high-frequency estimation as well as fast inner product. Then, we need to build an inverse rendering pipeline to optimize these SG parameters. Specifically, we compute the surface point of the scene at which a ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ emitted from the camera center $\mathbf{o}$ along the direction $\mathbf{d}$ is expected to terminate: $\mathbf{x}_\mathrm{surf} = \mathbf{o} + \hat{D}(\mathbf{r})\mathbf{d}$. The normals of surface points are estimated by the gradient of density as $\mathbf{n}(\mathbf{x}_\mathrm{surf}) = -\nabla\sigma(\mathbf{x}_\mathrm{surf})$. Then we compute the incident light at these surface points to render out colors using the render equation~\cite{render_equation}:
%where $\mathbf{n}$ is the normal of the point $\mathbf{x}$, $\Omega^+=\left\{\boldsymbol{\omega}_\mathrm{i}: \boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}>0\right\}$ is the hemisphere integral region, $f_r$ is the BRDF of the surface material.
%According to~\cref{incident_light}, to compute the incident light $L_\mathrm{i}$, the near-field lighting and opacity from $\mathcal{F}$ should be computed by ray tracing at each surface point. However, performing ray tracing repeatedly during optimization is too time-consuming. Zhang et al.~\cite{zhang2022modeling} use ray tracing results as supervision to train MLPs at the beginning, and query results from MLPs during optimization process. Due to its sparse sampling (16 rays per point) and MLP cache strategy, the local lighting and visibility of Zhang et al. is low-frequency and inaccurate. Our target is to estimate environment lighting rather than surface materials. Therefore, for each image (used for NeRF training), we just randomly sample 64 pixels and compute their depth to obtain their corresponding 3D locations and trace $128\times256$ rays for each location. The opacity results are saved as texture maps to preserve details. We assume that the materials of the scene are all diffuse, which provides more constraints to avoid highly ill-posed inverse rendering problem. Therefore, \cite{render_equation} can be converted as the sum of the irradiance of near-field lighting $E_\mathrm{nerf}$ and environment lighting $E_\mathrm{env}$:
% \begin{equation}
% L_\mathrm{o}(\mathbf{x}) = \frac{\mathbf{a}(\mathbf{x}; \mathbf{\Theta_\mathrm{a}})}{\pi}(E_\mathrm{nerf}(\mathbf{x})+E_\mathrm{env}(\mathbf{x}; \mathbf{\Theta_\mathrm{env}})),
% \end{equation}
% where $\mathbf{a}$ is the albedo parameterized as a MLP, $\mathbf{\Theta_\mathrm{a}}$ and $\mathbf{\Theta_\mathrm{env}}$ are the parameters of MLP and SG lobes respectively, which need to be optimized. $E_\mathrm{nerf}$ can be estimated by Monte Carlo method after ray tracing and keep unchanged during the optimization. For $E_\mathrm{env}$, we use the importance sampling in each SG lobe to test its visibility according to the opacity cube map, and reduce the amplitude of the SG lobe if it is occluded, which is similar to the method of Zhang et al.~\cite{zhang2022modeling} and then it can be fast computed as the SG inner product according to~\cite{physg}.
% The ground-truth colors are the pixel values on the images. The architecture of the albedo MLP and loss function are the same as ~\cite{zhang2022modeling}.
% To estimate the global lighting, previous data-driven methods train models with HDR image datasets. These methods take limited-FOV images as input and predict main light sources or even the whole HDR environment map. However, the color distributions of HDR images are significantly varying among scenes, which results in the weak generalization ability of these methods. Therefore, we use the lighting and geometry information in NeRF to robustly estimate the global lighting by optimization approach.
% Given the normal $\mathbf{n}(\mathbf{x})$, BRDF $f_\mathrm{r}(\boldsymbol{\omega_\mathrm{o}}, \boldsymbol{\omega_\mathrm{i}}; \mathbf{x})$ and the ambient light $L_\mathrm{i}(\boldsymbol{\omega_\mathrm{i}}; \mathbf{x})$ at each surface point $\mathbf{x}$, the output radiance $L_\mathrm{o}(\boldsymbol{\omega_\mathrm{o}}; \mathbf{x})$ is an integral over the hemisphere $\Omega^+=\left\{\boldsymbol{\omega}_\mathrm{i}: \boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}>0\right\}$ according to the render equation~\cite{render_equation}:
% \begin{equation}
% L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}\right)=\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}.
% \label{render_equation}
% \end{equation}
% The target output radiance is the corresponding pixel value of HDR images and the global lighting is the parameter to be optimized. To ensure that the optimization is a well-posed problem, the introduction of other unknowns, such as material and geometry parameters, should be minimized. Our goal is not to get the detailed global lighting, but to ensure that the optimization result approximate the ground-truth at low frequency, since the local lighting is all-frequency and enables to fill in detailed shading effects. We build an differentiable inverse rendering pipeline and the following contents will introduce important parts of it.
% \textit{Surface points and normals}. Given a camera and a trained NeRF, we can compute the location at which a ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ emitted from the camera center $\mathbf{o}$ along the direction $\mathbf{d}$ is expected to terminate: $\mathbf{x}_\mathrm{surf} = \mathbf{o} + \hat{D}(\mathbf{r})\mathbf{d}$. In addition, the normal of surface points can also be estimated by the gradient of density as $\mathbf{n}(\mathbf{x}_\mathrm{surf}) = -\nabla\sigma(\mathbf{x}_\mathrm{surf})$. However, although NeRF can render realistic novel view, its estimation of scene geometry (especially surface normal) tends to be rough and noisy. Some works~\cite{nerfactor, ref_nerf} train a MLP with the supervision of raw normals to predict smoother normals since the MLP can be considered as a low-pass filter. We do not attempt to optimize the accuracy of all surface point normals, but rather ensure that the surface points participating in the optimization have accurate normals, since our goal is only to obtain the global lighting instead of extracting geometry or materials.
% We extract point cloud from NeRF by computing surface points corresponding to rays emitted from all cameras and perform downsampling. Then we extract planes composed of over 1/10 total number of points from the point cloud by RANSAC~\cite{RANSAC} method and compute their normals. Surface points belonging to planes will be used as input for the optimization model, and their normals will be updated to the plane normals.
% \textit{Albedo}. We parameterize the albedo $\boldsymbol{a}$ at each surface point $\mathbf{x}_\mathrm{surf}$ as an MLP $f_\mathrm{a}: \mathbf{x}_\mathrm{surf} \mapsto \boldsymbol{a}$. To disambiguite between, eg. the ``red-painted surface" case and the ``white-painted surface illuminated by a red light" case (albedo-lighting ambiguity) or the ``black-painted surface" case and the ``white-painted surface under shadow" case (albedo-shadow ambiguity), we assume that the albedo of a plane is spatially smooth. Therefore, we use a tiny MLP with two hidden layers and apply the smooth loss function as follow:
% \begin{equation}
% \ell_\mathrm{a}=\lambda_1 \sum_{\mathbf{x}_\mathrm{surf}} \left\|f_\mathrm{a}\left(\mathbf{x}_\mathrm{surf}\right)-f_\mathrm{a}\left(\mathbf{x}_\mathrm{surf}+\boldsymbol{\epsilon}\right)\right\|_2^2,
% \label{loss_mat}
% \end{equation}
% where $\boldsymbol{\epsilon}$ is a random displacement that is perpendicular to the normal $\mathbf{n}(\mathbf{x}_\mathrm{surf})$, which is uniformly sampled from a circular domain with a radius of $10^{-3}$, and $\lambda_1$ is a hyperparameter set to 0.2.
% \textit{BRDF}. To minimize unknowns and quicken the pace of optimization, we assume the planes are all diffuse material, so the BRDF at a surface point is $f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right) = f_\mathrm{a}\left(\mathbf{x}_\mathrm{surf}\right) / \pi$. \cref{render_equation}) can be simplified as:
% \begin{equation}
% \begin{aligned}
% L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}_\mathrm{surf}\right)&=\frac{f_\mathrm{a}\left(\mathbf{x}_\mathrm{surf}\right)}{\pi} \int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{surf}\right) \left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i} \\
% &=\frac{f_\mathrm{a}\left(\mathbf{x}_\mathrm{surf}\right)}{\pi} E(\mathbf{x}_\mathrm{surf}),
% \label{sim_render_eq}
% \end{aligned}
% \end{equation}
% where $E(\mathbf{x}_\mathrm{surf})$ is the irradiance at the surface point $\mathbf{x}_\mathrm{surf}$.
% \textit{Ambient light}. Computing the local lighting, mixing with the global lighting and integrating numerically to obtain the irradiance in each iteration is very inefficient. We express the ambient light in the SH-9 basis as follow:
% \begin{equation}
% L_\mathrm{i}(\mathbf{d}; \mathbf{x})=\sum_{\ell=1}^{9} k_{\ell}^\mathrm{i}(\mathbf{x}) Y_{\ell}(\mathbf{d}),
% \label{gl_sh}
% \end{equation}
% where $k_{\ell}^\mathrm{i}(\mathbf{x})$ is SH coefficients of the amlbient light $L_\mathrm{i}$ and $Y_{\ell}$ is SH-9 basis function. Similarly, the global lighting $L_\mathrm{g}$, the local lighting $L_\mathrm{l}$ and the complementary opacity $(1-\hat{O})$ can all be expressed in the SH-9 basis and their SH coefficients are denoted as $\{k_{\ell}^\mathrm{g}\}_{\ell=1}^9$, $\{k_{\ell}^\mathrm{l}\}_{\ell=1}^9$ and $\{k_{\ell}^\mathrm{o}\}_{\ell=1}^9$, respectively. The SH coefficients of the ambient light at position $\mathbf{x}$ can be derived from~\cref{ambient_mix} as follow (subscript $\ell$ is droped for brevity):
% \begin{equation}
% k^\mathrm{i}(\mathbf{x}) = k^\mathrm{g}\odot k^\mathrm{o}(\mathbf{x}) + k^\mathrm{l}(\mathbf{x}),
% \label{gl_SH_mix}
% \end{equation}
% where $\odot$ means SH triple product~\cite{SH_triple_product}. The irradiance $E(\mathbf{x}_\mathrm{surf})$ can now be calculated efficiently with the linear combination of the ambient light SH coefficients~\cite{SH_irradiance}. We precompute $\{k_{\ell}^\mathrm{l}\}_{\ell=1}^9$ and $\{k_{\ell}^\mathrm{o}\}_{\ell=1}^9$ for each point belonging to planes by sampling rays and numerically integrating as follows:
% \begin{equation}
% k_{\ell}^\mathrm{l}(\mathbf{x}) = \int_{\Omega^+} L_\mathrm{l}(\boldsymbol{\omega}; \mathbf{x})Y_{\ell}(\boldsymbol{\omega}) \mathrm{d}\boldsymbol{\omega},
% \label{gl_SH_g}
% \end{equation}
% \begin{equation}
% k_{\ell}^\mathrm{o}(\mathbf{x}) = \int_{\Omega^+} (1-\hat{O}(\boldsymbol{\omega}; \mathbf{x}))Y_{\ell}(\boldsymbol{\omega}) \mathrm{d}\boldsymbol{\omega}.
% \label{gl_SH_o}
% \end{equation}
% The SH coefficients of the global lighting $\{k_{\ell}^\mathrm{g}\}_{\ell=1}^9$ are parameters to be optimized.
% Finally, the rendering equation in our setup is (subscript of $\mathbf{x}_\mathrm{surf}$ is dropped for brevity):
% \begin{equation}
% L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}\right)= \frac{f_\mathrm{a}\left(\mathbf{x}; \Theta\right)}{\pi} \mathcal{E}(k^\mathrm{i}(\mathbf{x}), \mathbf{n}(\mathbf{x})),
% \label{irr_equation}
% \end{equation}
% where $\Theta$ is the parameter of the MLP and $\mathcal{E}(\cdot)$ is the function that calculates the irradiance at position $\mathbf{x}$ with its normal $\mathbf{n}(\mathbf{x})$ and the ambient light SH coefficients $k^\mathrm{i}(\mathbf{x})$ using the method of~\cite{SH_irradiance}. The final reconstruction loss $\ell_{recon}$ has the same form to~\cref{hdr_nerf_loss}. Therefore, our full loss function is: $\ell_{recon}+\ell_\mathrm{a}$ and paramters to optimized are $\{\Theta, \{k_{\ell}^\mathrm{g}\}_{\ell=1}^9\}$.
\subsection{Visibility field precomputation}
\label{SSDF fields sec}
Once the environment lighting is estimated, the next step is to precompute the visibility for any point around the virtual object, so that such information can be rapidly retrieved during runtime for shadow computation. Inspired by shadow field~\cite{shadow_field}, we precompute a field of SSDF around the virtual object, which allows fast rendering with incident lighting expressed as SGs~\cite{all_freq}.
%. Note that while a more natural and consistent choice to express precomputed visibility is via a field of SGs, its non-linear nature makes interpolation difficult. Therefore, we adopt SSDF~\cite{all_freq}, which also allows fast rendering with incident lighting expressed as SGs~\cite{all_freq}.}
First, a spherical signed distance field at a point $\mathbf{x}$ around the virtual object is defined as follows. Given a ray from $\mathbf{x}$ along the direction $\mathbf{d}$, the SSDF $S(\mathbf{d}; \mathbf{x})$ is the minimal angle between $\mathbf{d}$ and the direction from $\mathbf{x}$ to the the silhouette of the virtual object. The angle is positive when $\mathbf{d}$ does not intersect with the object, and negative otherwise. Interested readers are directed to~\cite{all_freq} for a detailed derivation.
Next, we precompute the SSDFs at sampled points around the virtual object. Similar to~\cite{shadow_field}, we use the points on a uniform 3D grid of $16\times16\times16$, whose center coincides with that of the object $\mathbf{x}_\mathrm{o}$. The length of the grid is 3 times the radius of the bounding sphere of the object. Furthermore, we perform principal component analysis (PCA) to compress all sampled SSDFs to 1.8\% of the original size.
%$r_\mathrm{obj}$ of the virtual object is defined as half of the length of the longest axis. We precompute the SSDF of ponits uniformly sampled from a 3D grid centered on the virtual object, with the size of $16\times16\times16$ and the range from $-1.5r_\mathrm{obj}$ to $1.5r_\mathrm{obj}$, in about 10 minutes with 9 MB storage.
%Shadow fields~\cite{shadow_field} precompute the source radiance field (SRF) and the object occlusion field (OOF) to enable real-time soft shadow for dynamic scenes with moving objects and lights. The SRF records incoming light from the illuminant at surrounding sample points as cube maps, and the OOF records the occlusion area of the object as viewed from sample points around it as cube maps. Shadow fields~\cite{shadow_field} compress these cube maps to SH coefficients. At any given shading point, the lighting and occlusion information that represented as SH coefficients can be interpolated from the SRF and the OOF rapidly. The SH triple product is performed to calculate the final color. Shadow fields can compute detailed shadows by using high-order SH basis at the cost of high computational complexity of SH triple products.
%To compute detailed shadows with low computational complexity and be compatible with the incident light represented by SG lobes, we use the field of spherical signed distance field (SSDF~\cite{all_freq} field) as a similar role to the OOF.
%We first give the definition of the spherical signed distance field (SSDF) at a point $\mathbf{x}$ around the virtual object. Given a ray from $\mathbf{x}$ to the direction $\mathbf{d}$. The SSDF $S(\mathbf{d}; \mathbf{x})$ is defined as the minimized angle between $\mathbf{d}$ and the direction from $\mathbf{x}$ to the the virtual object's outline. The angle is positive when $\mathbf{d}$ does not intersect with the virtual object, otherwise it is negative. We recommend readers to read~\cite{all_freq} for detailed derivation.
% \warning{move}
% With the SSDF, we can fast compute the inner product of incident light (projected to the SG basis) and the visibility of the virtual object at $\mathbf{x}$ ($\mathbf{x}$ is dropped for brevity):
% \begin{equation}
% \begin{aligned}
% \int_{\Omega}L_\mathrm{i}(\mathbf{i})V(\mathbf{i}) \mathrm{d}\mathbf{i} &= \int_{\Omega}(\sum_{k=1}^M G(\mathbf{d}; \mathbf{p}_k, \lambda_k, \boldsymbol{\mu}_k))V(\mathbf{i}) \mathrm{d}\mathbf{i} \\
% &= \sum_{k=1}^M\int_{\Omega}( G(\mathbf{d}; \mathbf{p}_k, \lambda_k, \boldsymbol{\mu}_k))V(\mathbf{i}) \mathrm{d}\mathbf{i} \\
% &\approx \sum_{k=1}^M\boldsymbol{\mu}_k f_\mathrm{h}(S(\mathbf{p}_k), \lambda_k),
% \label{SG_visi_product}
% \end{aligned}
% \end{equation}
% where $f_\mathrm{h}(\theta, \lambda)$ is a precomputed table~\cite{all_freq} related to the spherical signed distance (SSD) $\theta$ and the sharpness $\lambda$ of the SG lobe, and $V(\mathbf{d})$ is the binary visibility term. When $\mathbf{d}$ does not intersect with the virtual object, $V(\mathbf{d}) = 1$, otherwise $V(\mathbf{d}) = 0$.
% At a sample point $\mathbf{x}$ around the virtual object, if the ray from $\mathbf{x}$ to the direction $\mathbf{d}$ intersects with the object, the binary visibility function $V(\mathbf{d}; \mathbf{x})$ equals to 0, otherwise it equals to 1. The SSDF $S(\mathbf{d}; \mathbf{x})$ is given by:
% \begin{eqnarray}
% S(\mathbf{d}; \mathbf{x})=
% \left\{
% \begin{aligned}
% + \mathop{\min}_{V(\mathbf{t})=0} \arccos(\mathbf{t}\cdot\mathbf{d}),&\quad \text{if } V(\mathbf{d})=1,\\
% - \mathop{\min}_{V(\mathbf{t})=1} \arccos(\mathbf{t}\cdot\mathbf{d}),&\quad \text{if } V(\mathbf{d})=0.
% \end{aligned}
% \right.
% \label{SSDF}
% \end{eqnarray}
% where $\mathbf{t}$ is the nearest direction that is on the object's silhouette boundary to $\mathbf{d}$.
% To enable the transformation including translation, rotation and scaling of the virtual object, we precompute the SSDF field similar to the OOF at sample points around the virtual object, and use PCA to compress it for efficient storage and interpolation. For a given point, we can compute the relative position to the virtual object center, use it to interpolate the PCA coefficients from the SSDF field, and finally recover the SSDF at the point using these coefficients.
% However, the fully SSDF field precomputation is both time-consuming and space-consuming. For $64\times64\times64$ points uniformly sampled from a 3D grid, it takes about 10 hours to precompute and occupy 576 MB of storage, which is unacceptable for AR applications. Considering that the visibility function $V(\mathbf{d}; \mathbf{x})$ changes greatly when the sample point $\mathbf{x}$ is closed to the object, while it changes slightly and regularly when $\mathbf{x}$ is far away from the object, we can only precisely compute the SSDF at points near the virtual object and roughly estimate the SSDF at other points.
% Specifically, we first compute the bounding box of the virtual object. The radius $r_\mathrm{obj}$ of the virtual object is defined as half of the length of the longest axis. We precompute the SSDF of ponits uniformly sampled from a 3D grid centered on the virtual object, with the size of $16\times16\times16$ and the range from $-1.5r_\mathrm{obj}$ to $1.5r_\mathrm{obj}$, in about 10 minutes with 9 MB storage.
At runtime, the precomputed SSDF samples are processed to a continuous field for pixel-level rendering.
For a point inside the grid, the SSDF can be obtained via a trilinear interpolation of the SSDFs at related sampled points. For a point $\mathbf{x}_\mathrm{f}$ outside the grid, we snap to $\mathbf{x}_\mathrm{b}$, the nearest point in the grid. We adjust the SSDF at $\mathbf{x}_\mathrm{b}$ with simple geometric relations to approximate that of $\mathbf{x}_\mathrm{f}$.
% to approximate the SSDF at $\mathbf{x}_\mathrm{f}$ by $S(\mathbf{d}; \mathbf{x}_\mathrm{f}) = S(\mathbf{d}; \mathbf{x}_\mathrm{b})+\Delta\theta$, where $\Delta\theta$ is the correction for spherical signed distance. To compute $\Delta\theta$ in real time, we approximate the virtual object as a sphere with $\mathbf{x}_\mathrm{o}$ as its center and $r$ as its radius, where $r$ is the average distance from the virtual object surface to $\mathbf{x}_\mathrm{o}$~\cite{wang2006variational}. Therefore $\Delta\theta$ can be approximated by:
% \begin{equation}
% \Delta\theta \approx \arcsin(\frac{r}{r_\mathrm{b}}) - \arcsin(\frac{r}{r_\mathrm{f}}),
% \label{ssdf_eq}
% \end{equation}
% where $r_\mathrm{b}$ is the distance between $\mathbf{x}_\mathrm{o}$ and $\mathbf{x}_\mathrm{b}$ and $r_\mathrm{f}$ is defined similarly.
%A graphical illustration is in~\cref{fig_ssdf}.
% \begin{figure}[h]
% \centering
% \includegraphics[width=0.35\textwidth]{figs/ssdf.pdf}
% \caption{Our SSDF estimation model. We use a sphere to approximate the virtual object. $\mathbf{x}_\mathrm{o}$ is its center. $\mathbf{x}_\mathrm{f}$ is a point out of the SSDF grid and $\mathbf{x}_\mathrm{b}$ is the closest point on the SSDF field boundary to $\mathbf{x}_\mathrm{f}$. $\mathbf{t}_\mathrm{b}$ and $\mathbf{t}_\mathrm{f}$ are tangent to the sphere. $\mathbf{d}$ is an arbitrary direction.
% The angles between the tangents and the line from $\mathbf{x}_\mathrm{o}$ to $\mathbf{x}_\mathrm{f}$ are $\theta_\mathrm{b}$ and $\theta_\mathrm{f}$, respectively. We estimate the SSD value $\varphi_\mathrm{f}=S(\mathbf{d}; \mathbf{x}_\mathrm{f})$ from $\varphi_\mathrm{b}=S(\mathbf{d}; \mathbf{x}_\mathrm{b})$ by the equation $\varphi_\mathrm{b}+\theta_\mathrm{b} = \varphi_\mathrm{f}+\theta_\mathrm{f}$.}
% \label{fig_ssdf}
% \end{figure}
% %points within the grid range, their SSDF can be directly interpolated. Points $\mathbf{x}_\mathrm{f}$ out of the grid will be clamped to the grid boundary and its SSDF can be derived from the SSDF of the corresponding boundary points $\mathbf{x}_\mathrm{b}$ as shown in ~\cref{fig_ssdf}.
% We provide a brief explanation of~\cref{ssdf_eq} in~\cref{fig_ssdf}. Because $\mathbf{t}_\mathrm{b}$ and $\mathbf{t}_\mathrm{f}$ are tangents, $\theta_\mathrm{b}$ and $\theta_\mathrm{f}$ can be computed as $\theta_\mathrm{b}=\arcsin(\frac{r}{r_\mathrm{b}}), \theta_\mathrm{f}=\arcsin(\frac{r}{r_\mathrm{f}})$. $\Delta\theta = S(\mathbf{d}; \mathbf{x}_\mathrm{f}) - S(\mathbf{d}; \mathbf{x}_\mathrm{b}) = \varphi_\mathrm{f}-\varphi_\mathrm{b} = \theta_\mathrm{b} - \theta_\mathrm{f}$.}
% Let $\mathbf{d}_\mathrm{b}$ be the directions from $\mathbf{x}_\mathrm{b}$ to $\mathbf{p}_\mathrm{b}$, which is tangent to the object and satisfies $S(\mathbf{d}_\mathrm{b}, \mathbf{x}_\mathrm{b})=0$. $\mathbf{d}_\mathrm{f}$ is defined similarly. Let $r_b$ be the distance between $\mathbf{x}_\mathrm{b}$ and the object center $\mathbf{x}_\mathrm{o}$, and $r_f$ is defined similarly. Analytically deriving $S(\mathbf{d}; \mathbf{x}_\mathrm{f})$ from $S(\mathbf{d}; \mathbf{x}_\mathrm{b})$ is complex and time-consuming for real-time rendering. Therefore, we make three assumptions:
% \begin{itemize}
% \item The distance between $\mathbf{x}_\mathrm{o}$ and $\mathbf{p}_\mathrm{b}$ is the same as the distance between $\mathbf{x}_\mathrm{o}$ and $\mathbf{p}_\mathrm{f}$, both of which are equal to $r$.
% \item Vector from $\mathbf{x}_\mathrm{o}$ to $\mathbf{p}_\mathrm{b}$ is perpendicular to $\mathbf{d}_\mathrm{b}$. Similarly, Vector from $\mathbf{x}_\mathrm{o}$ to $\mathbf{p}_\mathrm{f}$ is perpendicular to $\mathbf{d}_\mathrm{f}$.
% \item Vector from $\mathbf{x}_\mathrm{o}$ to $\mathbf{x}_\mathrm{b}$, $\mathbf{d}$, $\mathbf{d}_\mathrm{b}$ and $\mathbf{d}_\mathrm{f}$ are in the same plane.
% \end{itemize}
% With the above assumptions, it can be derived that:
% \begin{equation}
% \phi_\mathrm{b}+\theta_\mathrm{b} = \phi_\mathrm{f}+\theta_\mathrm{f},
% \label{SG_visi_0}
% \end{equation}
% where $\phi_\mathrm{b}=S(\mathbf{d}; \mathbf{x}_\mathrm{b}), \phi_\mathrm{f}=S(\mathbf{d}; \mathbf{x}_\mathrm{f}), \theta_\mathrm{b}=\arcsin(\frac{r}{r_\mathrm{b}}), \theta_\mathrm{f}=\arcsin(\frac{r}{r_\mathrm{f}})$.
% Therefore $S(\mathbf{d}; \mathbf{x}_\mathrm{f})$ is approximated by:
% \begin{equation}
% S(\mathbf{d}; \mathbf{x}_\mathrm{f}) = S(\mathbf{d}; \mathbf{x}_\mathrm{b}) + \arcsin(\frac{r}{r_\mathrm{b}}) - \arcsin(\frac{r}{r_\mathrm{f}}).
% \label{SG_visi}
% \end{equation}
% We simply set $r$ to the average distance from the virtual object surface to object center, which ensures that the object contour corresponding to the zero level set of the SSDF at $\mathbf{x}_\mathrm{f}$ keeps a shape similar to the contour at $\mathbf{x}_\mathrm{b}$.
\subsection{Real-time virtual object insertion}
\label{object insertion sec}
With the precomputed SSDF field, we perform virtual object insertion as follows. We compute the incident lighting by ray-tracing at the center of the virtual object according to~\cref{env light est sec} and fit it with SGs \warning{(which we refer to it as SG updating)}, render the object, composite it into the NeRF and add shadows at each frame. Note that for SG fitting, we adopt the estimates from the previous frame as initial values, both for temporal stability and faster convergence. Below are detailed descriptions.
%The latter three steps are detailed below.
%\warning{Note that for runtime performance, the above SG optimization takes the estimates at the previous frame as initial values, which we refer to as SG updating in the remainder of the text.}
%while the amplitude of SGs are multiplied by different self-shadow ratios $\gamma$ at different surface points to yield the self occlusion effects. We will explain $\gamma$ in the shadow effects part.
%We use the incident light at the virtual object center to render the entire object, according to our assumption in~\cref{preliminary}. Specifically, we first perform ray tracing to obtain the incident light by~\cref{incident_light} and use 32 SG lobes to fit it. Note that SG lobes do not form a set of orthonormal bases, which means that we must use numerically optimization method to perform fitting. Therefore, we randomly initialize the parameters of SG lobes at the first frame. For subsequent frames, we use the optimization results of the previous frame as the initialization, which preserves temporal stability and accelerates convergence when the virtual object moves slowly. The incident light at each surface point of the virtual object is approximated by the incident light at the virtual object center,
\textbf{Virtual object rendering}. To model the object appearance, we adopt the simplified Disney BRDF model~\cite{disney_brdf} with roughness $r$, metallic $m$ and diffuse albedo $\mathbf{a}$, and assume a fixed $F_0=0.02$ in the Fresnel term. As in previous work~\cite{physg, all_freq}, the BRDF $f_r$ and the cosine term $\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right)$ can be converted to mixtures of SGs as well. Therefore, the rendering computation is approximated as a rapid inner product of SGs using a fragment shader:
\begin{equation}
L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}\right) = \sum_{k=1}^M (G(\boldsymbol{\omega}_\mathrm{i}; \mathbf{p}_k, \lambda_k, \gamma_k(\mathbf{x})\boldsymbol{\mu}_k) \otimes f_\mathbf{r}) \cdot \left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right),
\label{SG_render_eq}
\end{equation}
where $\mathbf{p}_k, \lambda_k$ and $\boldsymbol{\mu}_k$ are the SG parameters of the incident lighting at the virtual object center. $\gamma_k$ attenuates the final result to account for self-shadows, which we will detail in the description of shadow computation. $\otimes$ represents direction-wise multiplication (SG product). As the BRDF is represented with SGs, this allows direct editing of appearance in real time.
%The shape of BRDF SG is directly related to BRDF parameters, so we can edit the material in real time without any extra time consumption.
\begin{figure}
\centering
\includegraphics[width=0.37\textwidth]{figs/sample.pdf}
\caption{The illustration of sampling points for volume rendering. Rays are emitted from the camera center to the NeRF to render the view. For rays not intersecting with the virtual object, we sample points with the distance range $t_\mathrm{k}\in[t_\mathrm{n}, t_\mathrm{f}]$, where $t_\mathrm{n}$ and $t_\mathrm{f}$ are the near and far distances of the intersection of the NeRF boundary and the ray. Otherwise, we sample points with the distance range $t_\mathrm{k}\in[\min (d, t_\mathrm{n}), \min (d, t_\mathrm{f})]$, where $d$ is the depth value of the virtual object, to eliminate the part of the NeRF occluded by the virtual object. }
\label{sample}
\end{figure}
\textbf{Compositing}. Here we mix the rendering results of the input NeRF and the virtual object. First, for a pixel whose camera ray hits the virtual object, we adjust the integration range $t$ to $[\min (d, t_\mathrm{n}), \min (d, t_\mathrm{f})]$, where $d$ is the depth of the object at the current pixel. The idea is to eliminate the part of the NeRF occluded by the virtual object. Please refer to~\cref{sample} for an illustration. We then perform conventional NeRF rendering to obtain a color and an opacity map, $\mathcal{I}_\mathrm{c}$ and $\mathcal{I}_\mathrm{\alpha}$. Next, we blend in the rendering result of the virtual object (without shadows) $\mathcal{I}_{v}$ as:
\begin{equation}
\mathcal{I}_\mathrm{\alpha} \mathcal{I}_\mathrm{c} + (1 - \mathcal{I}_\mathrm{\alpha}) \mathcal{I}_\mathrm{v}.
\label{blend_result}
\end{equation}
%We first perform conventional NeRF rendering to obtain a color and an opacity map, $\mathcal{I}_\mathrm{c}$ and $\mathcal{I}_\mathrm{o}$. Note that the distance range of a point sample in~\cref{preliminary} is now changed to $t_\mathrm{k}\in[\min (d, t_\mathrm{n}), \min (d, t_\mathrm{f})]$, where $d$ is the depth value of the virtual object. We illustrate the sampling method in~\cref{sample}.
% We denote the rendering color and depth map of the virtual object as $\mathcal{I}_\mathrm{vc}$ and $\mathcal{I}_\mathrm{vd}$ respectively. The scene is rendered by emitting rays from the camera center to the image plane and performing volume rendering to obtain color $\mathcal{I}_\mathrm{nc}$ and opacity $\mathcal{I}_\mathrm{no}$ results. Note that the distance range of the sampling ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ is now changed to $t\in[\min (d(\mathbf{r}), t_\mathrm{n}), \min (d(\mathbf{r}), t_\mathrm{f})]$, where $d(\mathbf{r})$ is the depth value queried from $\mathcal{I}_\mathrm{vd}$. Finally, we blend the rendering results as follow:
% \begin{equation}
% \mathcal{I}_\mathrm{c} = \mathcal{I}_\mathrm{nc} + \mathcal{I}_\mathrm{vc} \otimes (1 - \mathcal{I}_\mathrm{no}),
% \label{blend_result}
% \end{equation}
% where $\mathcal{I}_{c}$ is the compositing result (without shadows), and $\otimes$ means element-wise multiplication. We achieve detailed opaque and translucent occlusion effects between the scene and the virtual object without using the explicit geometry of the scene, such as the depth map.
\textbf{Shadow computation}. We handle two types of shadows in our pipeline: the shadow from the virtual object to the NeRF, and the self-shadow of the virtual object.
For the first type of shadows, similar to previous work~\cite{wang2022neural}, we compute an attenuation factor $\kappa$, as the ratio of the irradiance value before and after the virtual object insertion:
\begin{equation}
\begin{aligned}
\kappa(\mathbf{x})&=\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}.
\end{aligned}
\label{shadow ratio}
\end{equation}
Here $\mathbf{x}_\mathrm{o}$ is the object center. $V$ is the binary visibility term: it is 0 if occluded by the virtual object, and 1 otherwise. For each pixel in $\mathcal{I}_\mathrm{c}$, we compute its corresponding 3D location $\mathbf{x}$ from its depth using the same method in~\cref{env light est sec}, and then attenuate its rendered result by a corresponding $\kappa$.
% \begin{equation}
% \begin{aligned}
% \kappa(\mathbf{x})&=\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}} \\
% &\approx\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}},
% \end{aligned}
% \label{shadow ratio}
% \end{equation}
%First, the virtual object might occlude the incident lighting of certain parts of the NeRF, essentially forming shadows. Second, the virtual object may cause self-shadows.
%For any point in the scene, the virtual object will occlude part of the incident light, so its color will be darker than before insertion, which we call casting shadow. In addition, the surface points of the virtual object may not receive the full incident light due to the self occlusion, which causes self-shadow effects. These two kinds of shadows are essentially the same, however, considering that the material of the virtual object is known while the material of the scene is unknown, we use different methods to handle them.
% For the casting shadow, we compute shadow ratios $\kappa$ as the ratio of the irradiance value before and after the virtual object insertion for each scene point similar to the previous work~\cite{wang2022neural}, which is given by:
% \begin{equation}
% \begin{aligned}
% \kappa(\mathbf{x})&=\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}} \\
% &\approx\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{o}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}},
% \end{aligned}
% \label{shadow ratio}
% \end{equation}
%where $\mathbf{x}_\mathrm{o}$ is the virtual object center, and $V(\mathbf{d})$ is the binary visibility term. When $\mathbf{d}$ does not intersect with the virtual object, $V(\mathbf{d}) = 1$, otherwise $V(\mathbf{d}) = 0$. For each pixel in $\mathcal{I}_\mathrm{c}$, we compute its corresponding 3D location $\mathbf{x}$ from its depth and then multiply its color by the shadow ratio computed at $\mathbf{x}$ to get the final result. Note that we use two assumptions in the derivation of~\cref{shadow ratio}. First, we still assume that the BRDF of the scene are all diffuse so that the BRDF term can be removed~\cite{wang2023neural}. Pixel values in $\mathcal{I}_\mathrm{c}$ can be regarded as being rendered with correct BRDFs. Therefore, despite the use of the diffuse assumption, our results still keep convincing visual effects. In addition, we approximate the incident light at each surface point $\mathbf{x}$ to the incident light at the virtual object center $\mathbf{x}_\mathrm{o}$ to reduce real-time computation and keep the spatial consistency of shadows~\cite{supan2006image}. Our approximation yields a good trade-off because shadows usually appear near the object in most cases.
%The shadow ratio can be fast computed by SG product, SG inner product and the SG-visibility inner product. Specifically, for the numerator, we first multiply $L_\mathrm{i}$ by the cosine term using the SG product, and then perform the SG-visibility inner product:
\cref{shadow ratio} can be rapidly computed thanks to the SG basis. For its numerator, we first multiply $L_\mathrm{i}$ with the cosine term as a product of SGs, and then compute as follows:
\begin{equation}
\begin{aligned}
\int_{\Omega}(\sum_{k=1}^M G(\boldsymbol{\omega}_\mathrm{i}; \mathbf{p}_k, \lambda_k, \boldsymbol{\mu}_k))V(\boldsymbol{\omega}_\mathrm{i}) \mathrm{d}\boldsymbol{\omega}_\mathrm{i} \\
\approx \sum_{k=1}^M\boldsymbol{\mu}_k f_\mathrm{h}(S(\mathbf{p}_k), \lambda_k),
\label{SG_visi_product}
\end{aligned}
\end{equation}
where $f_\mathrm{h}(\theta, \lambda)$ is a precomputed table~\cite{all_freq} related to the spherical signed distance (SSD) $\theta$ and the sharpness $\lambda$ of an SG. For the denominator, it can be quickly evaluated in a manner similar to~\cref{SG_render_eq}.
%we perform SG inner product similar to~\cref{SG_render_eq}.
%multiply the SG amplitude of the incident light by the ratios $\gamma$ and render the virtual object using the updated incident light. We define $\gamma$ as the ratio of the SG integral of the unobstructed region to the integral of the full hemispherical region, which is given by:
For adding the self-shadows, we first attenuate each SG for the incident lighting with a factor $\gamma_k$, and then render the virtual object. $\gamma_k$ is the ratio of the integral of the unobstructed region to that of the complete upper hemisphere:
\begin{equation}
\gamma_k(\mathbf{x})=\frac{\boldsymbol{\mu}_k f_h(S(\mathbf{p}_k; \mathbf{x}), \lambda)}{\boldsymbol{\mu}_k f_h(\frac{\pi}{2}, \lambda)}=\frac{f_h(S(\mathbf{p}_k; \mathbf{x}), \lambda)}{f_h(\frac{\pi}{2}, \lambda)}.
\label{self shadow ratio}
\end{equation}
Note that here we do not attenuate with $\kappa$ to avoid computing the shadows twice.
% \label{insertion sec}
% The ambient light at the center of the virtual object is computed using~\cref{ambient_mix} in real time and will be used to render the virtual object. To achieve all-frequency rendering effects and avoid high computational complexity of numerical integration, we use $M=32$ SG lobes to fit the ambient light. A SG lobe has the form:
% \begin{equation}
% G(\mathbf{d}; \mathbf{p}, \lambda, \boldsymbol{\mu}) = \boldsymbol{\mu} e^{\lambda(\mathbf{d}\cdot\mathbf{p} - 1)},
% \label{SG}
% \end{equation}
% where $\mathbf{p}\in\mathbb{S}^2$ is the lobe axis, $\lambda\in\mathbb{R}^+$ is the lobe sharpness, $\boldsymbol{\mu}\in\mathbb{R}_+^n$ is the lobe amplitude ($\boldsymbol{\mu}\in\mathbb{R}_+^3$ for RGB color) and the direction $\mathbf{d}\in\mathbb{S}^2$ is the function input. The ambient light is approximated by the sum of a set of SG lobes:
% \begin{equation}
% L_\mathrm{i}(\mathbf{d}) = \sum_{k=1}^M G(\mathbf{d}; \mathbf{p}_k, \lambda_k, \boldsymbol{\mu}_k).
% \label{SG_ambient}
% \end{equation}
% Unlike SH, SG lobes do not form a set of orthonormal bases, which means parameters of SGs cannot be calculated through inner product. Therefore, we fit the ambient light by numerically optimizing parameters of SGs in real time. Specifically, we uniformly sample spherical directions as the initialization of the axes of SGs. The sharpness and the amplitude of SGs are initialized to values sampled from the uniform distribution $U(0,1)$. In our experiment, if the initial value of sharpness is too large, the gradient used to update axes will be too small and the amplitude tends to zero, which leads to the ``death" of some SG lobes. To produce temporally stable results and keep real-time performance, we optimize parameters of SG lighting with 25 iterations per frame, and the initial value of the current frame is set to the result of the previous frame.
% As for the material of the virtual object, we use the microfacet Cook-Torrance BRDF~\cite{cook_torrance}:
% \begin{equation}
% f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i}\right) = k_\mathrm{d} \frac{\boldsymbol{a}}{\pi} + k_\mathrm{s} \frac{D(\mathbf{h}) F(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i}) G(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i})}{4(\boldsymbol{\omega}_\mathrm{o}\cdot\mathbf{n})(\boldsymbol{\omega}_\mathrm{i}\cdot\mathbf{n})},
% \end{equation}
% where $k_\mathrm{d}$ and $k_\mathrm{s}$ are weights of diffuse and specular components, which are related to the metallic parameter, $\boldsymbol{a}$ is the spatially-varying albedo of the virtual object, $\mathbf{h}$ is the half-vector $(\boldsymbol{\omega}_\mathrm{i}+\boldsymbol{\omega}_\mathrm{o})/\|\boldsymbol{\omega}_\mathrm{i}+\boldsymbol{\omega}_\mathrm{o}\|_2$. $D(\mathbf{h})$ is the normal distribution term whose shape is determined by the roughness parameter, $F(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i})$ is the Fresnel term and $G(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i})$ is the geometry term. As in prior works~\cite{physg, all_freq}, $F(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i}) G(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i})$ is approximated as $M(\boldsymbol{\omega}_\mathrm{o})$ that is independent of the incident direction $\boldsymbol{\omega}_\mathrm{i}$, so that~\cref{render_equation} is simplified as follow (shading position $\mathbf{x}$ dropped for brevity):
% \begin{equation}
% \begin{aligned}
% L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o}\right)&=k_\mathrm{d} \frac{\boldsymbol{a}}{\pi} \int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}\right) \left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i} \\
% &+ k_\mathrm{s} \frac{M(\boldsymbol{\omega}_\mathrm{o})}{4(\boldsymbol{\omega}_\mathrm{o} \cdot \mathbf{n})} \int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}\right) D(\mathbf{h}) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}.
% \label{render_equation_SG}
% \end{aligned}
% \end{equation}
% The normal distribution term $D(\mathbf{h})$ is defined in half-vector domain, so we warp $D(\mathbf{h})$ to the incident direction $\boldsymbol{\omega}_\mathrm{i}$ domain and approximate it with a single SG lobe~\cite{all_freq} for efficient computation:
% \begin{equation}
% \mathcal{W}(D(\mathbf{h})) \approx G(\boldsymbol{\omega}_\mathrm{i}; \mathbf{r}, \lambda(\rho), \boldsymbol{\mu}),
% \label{warp_distribution}
% \end{equation}
% where $\mathcal{W}(\cdot)$ is the warp operator, $\mathbf{r} = 2(\boldsymbol{\omega}_\mathrm{o}\cdot\mathbf{n})\mathbf{n} - \boldsymbol{\omega}_\mathrm{o}$ is the reflection direction, and $\rho$ is the surface roughness related to the sharpness of the lobe. We recommend readers to read~\cite{all_freq} for detailed derivation. We further approximate the remaining cosine term $\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}$ with a SG~\cite{hemi_SG_inte}:
% \begin{equation}
% \boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n} = G(\boldsymbol{\omega}_\mathrm{i},0.0315,\mathbf{n},32.7080)-31.7003.
% \label{cos_SG}
% \end{equation}
% Now, each term of the integrand is approximated as SG lobes. With the efficient multiplication, integration and inner product of SGs~\cite{hemi_SG_inte}, the virtual object with the spatially-varying material (albedo, roughness and metallic) can be rendered with all-frequency effects.
% \subsection{Composite the virtual object into the scene}
% \label{sec_3.4}
% We denote the rendering color and depth map of the virtual object as $\mathcal{I}_\mathrm{vc}$ and $\mathcal{I}_\mathrm{vd}$ respectively. The scene is rendered by emitting rays from the camera center to the image plane and performing volume rendering to obtain color $\mathcal{I}_\mathrm{nc}$ and opacity $\mathcal{I}_\mathrm{no}$ results using~\cref{nerf_integral} and~\cref{opac_eq}. Note that the distance range of the sampling ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ is now changed to $t\in[\min (d(\mathbf{r}), t_\mathrm{n}), \min (d(\mathbf{r}), t_\mathrm{f})]$, where $d(\mathbf{r})$ is the depth value queried from $\mathcal{I}_\mathrm{vd}$. Finally, we blend the rendering results as follow:
% \begin{equation}
% \mathcal{I}_\mathrm{c} = \mathcal{I}_\mathrm{nc} + \mathcal{I}_\mathrm{vc} \otimes (1 - \mathcal{I}_\mathrm{no}),
% \label{blend_result}
% \end{equation}
% where $\mathcal{I}_{c}$ is the compositing result (without shadows), and $\otimes$ means element-wise multiplication. We achieve detailed occlusion and even translucency effects between the scene and the virtual object without using the explicit geometry of the scene, such as depth map and mesh.
% The above process is illustrated as Figure~\ref{fig_blend}.
% \begin{figure}[h]
% \centering
% \hspace{-0.05\textwidth}
% \includegraphics[width=0.32\textwidth]{imgs/第3页.pdf}
% \caption{Composite the virtual object into NeRF. Points are sampled on the camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, where $t\in[\min (d(\mathbf{r}), t_\mathrm{n}), \min (d(\mathbf{r}), t_\mathrm{f})]$. The color intensity of the sample points represents their blend weight. }
% \label{fig_blend}
% \end{figure}
% \subsection{Cast shadows to the scene}
% \label{ssdf shadow}
% Now, we can efficiently get the visibility information from the precomputed SSDF at any surface point. The render equation with the visibility term is given by:
% \begin{equation}
% \begin{aligned}
% &L_\mathrm{o}\left(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}\right) \\
% &\quad=\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i} \\
% &\quad=\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x})} \mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}) \\
% &\quad=\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right) f_\mathrm{r}\left(\boldsymbol{\omega}_\mathrm{o}, \boldsymbol{\omega}_\mathrm{i} ; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}} \mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}) \\
% &\quad\approx\frac{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{obj}\right) V\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}}{\int_{\Omega^+} L_\mathrm{i}\left(\boldsymbol{\omega}_\mathrm{i}; \mathbf{x}_\mathrm{obj}\right)\left(\boldsymbol{\omega}_\mathrm{i} \cdot \mathbf{n}\right) \mathrm{d} \boldsymbol{\omega}_\mathrm{i}} \mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}) \\
% &\quad=\kappa(\mathbf{x})\mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x}),
% \label{render_equation_visi}
% \end{aligned}
% \end{equation}
% where $\mathbf{C}(\boldsymbol{\omega}_\mathrm{o} ; \mathbf{x})$ is the rendering result in~\cref{sec_3.4}, $\mathbf{x}_\mathrm{obj}$ is the center of the virtual object, and $\kappa(\mathbf{x})$ is the shadow coefficient. We do not attempt to estimate the material of surface points, which is a highly ill-posed problem. Therefore, we assume that the material of surface points is diffuse, so the BRDF term is split from the integrand and removed. We approximate the ambient light at each surface point $\mathbf{x}$ to the ambient light at $\mathbf{x}_\mathrm{obj}$ to reduce real-time computation and keep the consistency of shadows. Due to the use of SG basis, precomputing the ambient light of some grid points and performing interpolation in real time is not feasible, and using a MLP to cache ambient light of each point will loss high-frequency parts, which leads to totally unconvincing shadows. Our approximation yield a good trade-off. The remaining terms can be computed by the inner product between SG lightings and the visibility term using \cref{SG_visi} (for numerator), and the spherical integral of SG lobes~\cite{hemi_SG_inte} (for denominator). Finally, shadow effects is added by element-wise multiplying the composited rendering result by the shadow coefficients $\kappa(\mathbf{x})$.
% For shadows cast to the virtual object itself, i.e. self-shadows, the surface material is knwon, so we correct the amplitude of each SG lighting according to the visibility:
% \begin{equation}
% \begin{aligned}
% &G(\mathbf{d}; \mathbf{p}, \lambda, \boldsymbol{\mu})V(\mathbf{d}; \mathbf{x}_\mathrm{surf}) \\
% &\quad\approx G(\mathbf{d}; \mathbf{p}, \lambda, \frac{f_h(S(\mathbf{p}; \mathbf{x}_\mathrm{surf}), \lambda)}{f_h(\frac{\pi}{2}, \lambda)}\boldsymbol{\mu}),
% \label{SG_visi_corr}
% \end{aligned}
% \end{equation}
% where $\mathbf{x}_\mathrm{surf}$ is the surface point of the virtual object. Then we use the corrected SG lightings to render the virtual object as described in~\cref{SG renderer}, which generates self-shadows naturally. Note that we do not multiply the shadow coefficients to the virtual object surface colors to avoid double counting.
\begin{figure*}[!ht]
\centering
\includegraphics[width=1.0\textwidth]{figs/effects.pdf}
\caption{Visualization of intermediate and final object insertion results of the \textsc{Plants}, \textsc{Books} and \textsc{Dog} synthetic scenes. From left column to right, background images, virtual objects rendering results, compositing results with occlusion or translucency effects, $\kappa$ (\cref{shadow ratio}), object insertion results with all effects.}
\label{intermediate}
\end{figure*}
\begin{figure*}[!ht]
\centering
\includegraphics[width=1.0\textwidth]{figs/lightest.pdf}
\caption{A qualitative comparison of using different environment lighting in \textsc{Plants}, \textsc{Books} and \textsc{Dog} synthetic scenes. In the first column, virtual objects are rendered by only using the NeRF near-field lighting. From the second left column to the fourth column, we compensate different estimated environment lighting estimated by the method of PhySG~\cite{physg}, InvRender~\cite{zhang2022modeling} and our proposed method. The last column is the ground-truth rendered by Blender Cycles. Some details are magnified for better comparison. The environment lighting is shown by equirectangular projection under the rendered images.}
\label{woenv}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=1.0\textwidth]{figs/ray_pixels.pdf}
\caption{Estimating environment lighting by different sampling strategies. All the results are estimated in \textsc{Dog}. The result of our proposed method is shown on the second from left, which samples 64 pixels per view and traces 32768 rays at each surface point corresponding to the pixel. Results estimated by other sampling strategies are shown on the right side of the dashed line.}
\label{rays_pixels}
\end{figure*}
\begin{figure*}[ht]
\centering
\includegraphics[width=1.0\textwidth]{figs/incident.pdf}
\caption{Visualization of the lighting at the virtual object center. From left column to right, near-field lighting, environment lighting, incident lighting, SG fitted incident lighting and the ground-truth lighting rendered by Blender Cycles. The incident lighting is the blend of the near-field and environment lighting according to the NeRF opacity.}
\label{incident}
\end{figure*}
\section{Results}
\label{experiment sec}
All experiments are conducted on a workstation with an Xeon(R) Platinum 8352V CPU, 90GB memory and an NVIDIA RTX 4090 GPU. Our pipeline achieves a performance of 18 frames per second on average at a resolution of $1280\times720$. \warning{Specifically, it takes about 40ms for instant-ngp to render the NeRF. And it takes 15ms for virtual object insertion, including 4ms for incident lighting computation, 7ms for SG updating (except that 84ms is needed for the first frame), and 4ms for the rendering, compositing and shadow casting of the virtual object. The average GPU usage is 55\%, which shows room for future improvement over memory access.}
%which indicates that our method still has the potential for further optimization in implementation to break memory bottlenecks.
%We implement an editor with interactive GUI that supports to place a virtual object, edit its material and observe the scene from free views.
%The editor will pass required parameters to the rendering pipeline and present the rendering result in real time. We implement our rendering pipeline based on ngp-pl
%\footnote{ngp-pl. \href{https://github.com/kwea123/ngp_pl}{https://github.com/kwea123/ngp\_pl}}
%
%, which is an unofficial Pytorch implementation of instant-ngp. Compared to the original implementation, it runs relatively slow but is easy-developing.
%Running on a NVIDIA RTX 3070 GPU, our system can render images of $800\times800$ at an average of 14 fps.
For precomputation, it takes about 25 minutes to estimate the environment lighting from an input NeRF. For a virtual object, we spend 8 minutes to precompute the SSDF field, which takes up 9MB. Note that the SSDF field can be used with any NeRF.
%including 15 minutes for high-quality NeRF training and 30 min for environment lighting estimation. For each virtual object, it takes about with 9 MB storage consumption. Once the precomputation is completed, the virtual object can be used in any scene.
In the following, we describe the data related to our experiments, including synthetic and captured HDR NeRFs, in~\cref{datasets}. Ablation studies are demonstrated in~\cref{ablation}. Finally, we compare against state-of-the-art methods in~\cref{comparisons}.
%multi-view datasets to test the proposed method Then, we perform ablation studies on synthetic datasets to discuss our key components in~\cref{ablation}. Finally, we make qualitatively comparisons with state-of-the-art methods to show our realistic object insertion results in~\cref{comparisons}. For all HDR image results, we perform tonemapping using the method of Reinhard et al.~\cite{reinhard_tonemapping} with the same parameters to display LDR image in the following experiments.
\subsection{Data}
\label{datasets}
We build four synthetic HDR NeRFs by training using images rendered from multiple views with Blender Cycles~\cite{BlenderCycles}.
%\footnote{Blender. Version 3.4}%
%We use the most prominent object in each scene to name it.
For \textsc{Plants}, all light sources can be observed in the rendered images, and therefore the environment lighting should be totally black. For the other 3 NeRFs (\textsc{Books}, \textsc{Dog} and \textsc{Cups}), there are some light sources that do not appear in any of the rendered images, which are considered as the environment lighting.
%We use these scenes to verify the necessity of the global light estimation in the case of missing complete lighting information.
%The reason we use synthetic scenes is to conveniently obtain the ground-truth images by placing the virtual object in the scene with the same parameters from the editor and rendering it with Blender.
In addition to synthetic NeRFs, we also construct two NeRFs by training with captured HDR photographs from two real scenes (\textsc{Horse} and \textsc{Sheep}). We set the camera to the bracketing mode with the step of 1.5 EV and shoot 9 images with different exposures for each camera pose. Each HDR image is recovered from 9 LDR images using~\cite{debevec_hdr_recover}.
To provide the ground-truth for object insertion experiments, we 3D-print several models and uniform sprayed them with paints to simulate different materials. We apply Ma et al.~\cite{ma2021free} to measure the material parameters, including roughness and metallic, and use the same parameters for virtual objects rendering. The 3D printed models are placed in the real scene, and images are taken around them as the ground-truth. We use the images without the presence of the model to train NeRFs, and compare object insertion results with the images with the physical model in place. The sizes of virtual objects are adjusted manually to ensure that it is closed to the ground-truth.
%All related images are cropped to a resolution of $800\times800$.
%ground-truth images from bothsynthetic and real scenes are used for reference only. Our methods always provides biased results for some reason. For example, Blender render objects using Monte Carlo path tracing to precisely solve the render equation offline while our SG-based renderer simplifies the render equation for real-time rendering. In addition, the material of real objects also differs from the standard Cook-Torrance model. Effects like subsurface or anisotropic reflection are not considered in our rendering method. Therefore, the following experiments mainly perform qualitative comparison and analysis. All of the outputs in experiments are HDR images, so we perform tonemapping using the method of Reinhard et al.~\cite{reinhard_tonemapping} with the same parameters for each output to present results.
%due to the use of SG rendering method,
\begin{figure*}[h]
\centering
\includegraphics[width=1.0\textwidth]{figs/shadow.pdf}
\caption{Comparison of shadows generated by using the $16\times16\times16$ SSDF field that obtains SSD values at far positions by clamping to the boundary values (a), truncating to positive infinity (b) and approximating from the boundary values (c), and shadows generated by using the $64\times64\times64$ SSDF field that interpolates SSDF at any position (d). The ground-truth is rendered by Blender Cycles.}
\label{ssdf cmp}
\end{figure*}
\subsection{Ablation studies}
\label{ablation}
We first perform ablation experiments to show the intermediate results of each step described in~\cref{object insertion sec}. In~\cref{intermediate}, the virtual object shows all-frequency effects with realistic reflections.
We can see the virtual object through the glass bottle in the \textsc{Books} scene, which is a challenging task for traditional AR methods~\cite{macedo2021occlusion}. The shadows computed by our method also considerably enhanced the visual realism.
%Shadow effects convince the presence of the virtual object, which is physics-based without the limit on the number of light sources. All the above effects make it difficult to distinguish the inserted object from the surrounding environments.
%We create occlusion or translucency effects by changing the sampling range of NeRF and performing blending between the rendered object and the scene, which makes
%We also prove the necessity of the compensated environment lighting and the impact of modeling local lighting for environment lighting estimation in~\cref{woenv}.
%The main difference between the three methods is how to model the lighting.
%supposes that the environment light is the full incident light, so it does not take the NeRF near-field lighting and visibility into consideration.
Next, we show the impact of our estimated environment lighting as well as the near-field lighting represented in the NeRF in~\cref{woenv}. Without the environment lighting which estimates the sources not covered in the NeRF, the virtual object appears darker in the \textsc{Books} and \textsc{Dog} scene. Moreover, we compare our method with PhySG~\cite{physg} and InvRender~\cite{zhang2022modeling} on environment lighting estimation. These three methods have a similar estimation framework, and all represent the environment lighting as 32 SGs in our experiments. PhySG does not take into account the near-field lighting and visibility. InvRender roughly models the near-field lighting by training a MLP to cache ray-tracing results. Our method more accurately model the near-field lighting (especially the visibility, cached to texture maps), and sparsely sample the surface points to reduce memory consumption. In the \textsc{Plants} scene, all light sources are included in the NeRF, and therefore the ground-truth environment lighting is totally black. PhySG treats the light sources in NeRF as the environment lighting. The virtual object is brighter than the ground-truth in the \textsc{Plants} scene, while the self-shadow is too dark in the \textsc{Books} scene. For InvRender, caching near-filed lighting with MLP by tracing 16 rays at each point is too rough for complex scene-level NeRFs, which leaves a lot of ambiguity during optimization. Our method provides the best results, and the estimation is not affected by the light sources incorporated in the NeRF.
%In addition to validating the necessity of the local lighting,
In addition, we evaluate the impact of the number of pixels and rays over the estimated environment lighting in~\cref{rays_pixels}. If we sample less pixels, the estimation will suffer from the ambiguity of albedo and shading. If we trace less rays at each point, the near-field lighting is less accurate, resulting in a highly noisy estimate. In comparison, sampling fewer pixels has a greater impact than sampling fewer rays. Sampling more pixels and rays usually leads to slightly better results, but bears the increasing costs of time and memory.
To validate the our lighting model in~\cref{incident_light}, we show the near-field lighting (from NeRF), estimated environment lighting, full incident lighting (before and after SG fitting), and the ground-truth in~\cref{incident}. Our method generates high-quality incident lighting close to the ground-truth. After fitting with SGs, the main light sources are preserved. We notice that there are some extra floaters (\textsc{Plants} scene) and missing parts (\textsc{Books} scene) in the near-field lighting. Nevertheless, they do not notably influence the rendering as the brightness is relatively low. Also, the estimated environment lighting well compensates for the lighting not covered by NeRF.
%After adding the global light, the virtual object appears more reasonable and casts shadows to the same direction as other objects in the scene. Although our global light estimation is low-frequency, with the supplementary details of the indirect light, the insertion results are still convincing.
% replace the first row of our results, it is too bright
% In Section~\ref{NeRF train}, we propose using the depth loss to reduce floaters near cameras. As shown in Figure~\ref{fig_depthloss_cmp}, scenes reconstructed by NeRF without the depth loss are over-fitted. Floaters or tiny image planes fill in the details of scenes to better fit the target images when training, which not only leads to messy geometry, but also reduces the quality of novel view synthesis. The reasonable results are generated after adding the depth loss.
% \begin{figure}[h]
% \centering
% \includegraphics[width=0.4\textwidth]{imgs/第2页.pdf}
% \caption{Comparison of scenes reconstructed with or without the depth loss.}
% \label{fig_depthloss_cmp}
% \end{figure}
To validate our method for extrapolating SSDF (\cref{SSDF fields sec}), we compare in~\cref{ssdf cmp} the shadows generated by our parameters and using a $64\times64\times64$ SSDF field with an extended range of $-6r_\mathrm{obj}$ to $6r_\mathrm{obj}$, where $r_\mathrm{obj}$ is the radius of the virtual object bounding sphere, so that the SSDF is interpolated at any pixel in the rendered image. There are no significant differences between the results computed with two sets of parameters, while the simple clamping method generates unsatisfactory results in regions far away from the virtual object. Compared to the ground-truth rendering results, our shadows appear slightly darker, due to the approximation of the inner product of the environment lighting and the visibility term.
%In~\cref{SSDF fields sec}, we approximate the spherical signed distance (SSD) value at the position far away from the virtual object by the SSD value at the position closed to the object to reduce precomputation. We compare the shadows generated by using the $16\times16\times16$ SSDF field with the range of $-1.5r_\mathrm{obj}$ to $1.5r_\mathrm{obj}$ that obtaining SSD values at far positions by approximating from the boundary value, clamping to the boundary value and truncating to infinity, and shadows generated by using the $64\times64\times64$ SSDF field with the range of $-6r_\mathrm{obj}$ to $6r_\mathrm{obj}$ that using the interpolated SSDF at any position. The comparison results are shown in~\cref{ssdf cmp}.
\begin{figure*}[!ht]
\centering
\includegraphics[width=1.0\textwidth]{figs/compare1.pdf}
\caption{Comparison with the work of Li et el.~\cite{IRAdobe} and Zhan et al.~\cite{emlight} in both synthetic scenes and real scenes. The materials of virtual objects and the noteworthy effects are indicated under each column.}
\label{fig_cmp_res}
\end{figure*}
\subsection{Comparisons}
\label{comparisons}
We first compare our approach with two state-of-the-art methods, Li et al.~\cite{IRAdobe} and Zhan et al.~\cite{emlight}. Both take RGB images as input and employ neural networks to estimate the lighting. The former outputs 2D spatially-varying incident lighting, from a joint optimization of material parameters and depths. And the latter only outputs one incident lighting. For comparison, we first render (for synthetic scenes) or capture (for real scenes) images of a scene without the virtual object, and use them as input for both methods. The estimated lighting are represented as SGs, so that we can render the virtual object as described in~\cref{object insertion sec} to compare across different methods. Depth test is performed in~\cite{IRAdobe} to account for occlusions. Since the method of~\cite{emlight} does not provide any geometric information, we directly place the virtual object on top of the background. We do not compare our shadows with the baseline methods, as both of them generate shadows on manually specified planes. Instead, we compare our shadows with the ground-truth.
%For shadows, both methods use off-the-shelf renderers to generate shadows on manually specified planes, which is not suitable for comparison with our method. Therefore, there are no shadows in the results of the two methods. We mainly compare shadow effects with the ground-truth.
%The comparison results of synthetic and real scenes are shown in~\cref{fig_cmp_res}.
%The comparison results of synthetic scenes are shown in Figure~\ref{fig5} and real scenes are shown in Figure~\ref{fig_real}.
%To compare noteworthy details like shading, shadow and occlusion effects, we use boxes of different colors to frame them and present the corresponding enlarged image. %due to their weak generalization ability
\cref{fig_cmp_res} shows the comparisons on synthetic and real scenes. Virtual objects rendered with~\cite{IRAdobe} or~\cite{emlight} can be easily distinguished from the background. These two methods are trained with indoor image datasets and are likely to estimate results with considerable bias. In comparison, our method is able to robustly estimate the incident lighting. Our rendering results exhibit realistic reflections and shadows that closely resemble the ground-truth, regardless of the material, e.g. diffuse in \textsc{Dog} or glossy in \textsc{Books}. Even for scenes with complex lighting, such as multiple light sources in \textsc{Plants} and the environment lighting with local lights in \textsc{Dog}, our method is able to produce reliable results. The depth map computed by Li et al. is rough, leading to noticeable cracks or overlaps, e.g., in \textsc{Plants} and \textsc{Cups}. Note that our method supports the transparency and translucency represented in the input NeRF (see the plastic cup in \textsc{Horse} in~\cref{fig_cmp_res} and the glass vase in \textsc{Books} in~\cref{intermediate}).
\warning{In~\cref{DMRF}, we further compare with the concurrent work of DMRF~\cite{qiao2023dynamic}. There are 3 major differences. First, based on a path-tracing framework, their performance cannot reach real time. Second, their approach does not estimate light sources not covered the input NeRF. Finally, the visibility in the NeRF is ignored, leading to artifacts like shadow leakage, as shown in~\cref{DMRF}.}
%DMRF only attempts to detect light sources in NeRF to support points sampling on emissive surfaces. For light sources outside the NeRF, DMRF still explain it as light sources in NeRF. Therefore, as shown in~\cref{DMRF}, due to imperfect lighting model, DMRF shows hard shadows, although there are only low-frequency skylights in the scene. Besides, DMRF only considers the visibility of mesh objects but ignores the NeRF visibility, which leads to shadow leakage.}
%lighting effects.
%Note that there are some differences between our rendering and reality of the armadillo model in real scenes Figure~\ref{fig_cmp_res}(d), because the 3D printed model is hollow and not completely opaque, which forms the subsurface reflection effect. Our reflection model does not consider this effect.
%but also support to see virtual objects through transparent or translucent occlusions such as the .
% \begin{figure*}[!ht]
% \centering
% \includegraphics[width=0.9\textwidth]{imgs/第10页.pdf}
% \caption{Comparison with the work of Li et el.~\cite{IRAdobe} and Zhan et al.~\cite{emlight} in real scenes.}
% \label{fig_real}
% \end{figure*}
%For shadows, our method slightly differs from the ground-truth in terms of brightness and shape due to the approximations used in computation. Yet sufficient details are preserved. We cast shadows onto arbitrary surfaces without the common plane assumption \warning{cite}. We query SSDF at the surface points of virtual objects to yield the self-shadow, which mostly represents in the \textsc{Horse} scene (at the bend of the dragon).
%In addition, in the scenes with multiple light sources, our method can generate corresponding shadows to different directions, such as~\cref{fig_cmp_res}(a).
Moreover, we qualitatively compare against popular AR frameworks from the industry, including Google Depth Lab~\cite{depthlab}, AR Toolbox~\cite{AR_Toolbox} and Unity AR Foundation~\cite{AR_Foundation}, due to the difficulty for precisely controlled experiments. As the common types of virtual object supported by all frameworks are limited,
we only test a red ball in the \textsc{Horse} scene. As shown in~\cref{fig6}, our method produces more accurate and detailed rendering with soft shadows, while the counterparts only implement hard shadows on estimated planes.
%, which bears increasing computation for area or multiple lights, while our method is able to handle any light setting with the same computational complexity. However, the advantage of AR applications is better real-time performance and less computational complexity, which enables to be deployed on mobile terminals.
% In addition to comparing with recent researches, we also compare our method with public AR applications that contain complete AR pipeline. Google Depth Lab~\cite{depthlab}, AR Toolbox
% %\footnote{AR Toolbox. Version 1.13.0. \href{https://github.com/SimonMarquis/AR-Toolbox}{https://github.com/SimonMarquis/AR-Toolbox}}
% , Unity's AR Foundation
% %\footnote{AR Foundation. Version 5.0.6. Unity. 2023}
% are Android applications based on Google ARCore framework that use a smartphone as the input device, which supports realistic rendering by estimating environment map, occlusion effects by estimating depth map, and shadow effects by estimating the direction of the main light and casting shadows to the estimated plane using methods like the shadow map. The three sample applications mentioned above only support limited kinds of virtual object, so we only test a red ball in \textsc{Horse}. We launch AR applications, move and capture the entire scene to ensure comprehensive perception of the whole environment.
% These AR applications only allows mobile video streams with various camera information as input, and uses mobile inertial measurement unit and other sensors to provide auxiliary information. Therefore, it is difficult to ensure that the parameters of the camera and virtual objects are the same as our method inputs. We only conduct a simple and qualitative comparison of the results. As shown in~\cref{fig6}, our method provides more precise and detailed rendering with soft shadows in the correct direction. AR applications use shadow map~\cite{shadowmap} to cast hard shadows on estimated planes, which bears increasing computation for area or multiple lights, while our method is able to handle any light setting with the same computational complexity. However, the advantage of AR applications is better real-time performance and less computational complexity, which enables to be deployed on mobile terminals.
\begin{figure}
\centering
\includegraphics[width=0.47\textwidth]{figs/DMRF.pdf}
\caption{Comparison with DMRF~\cite{qiao2023dynamic}. It takes 50ms to render the frame by our methods while DMRF takes 38s to render a 4spp frame and 611s to render a 64spp frame. Leaked shadows are observed with their approach.}
\label{DMRF}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.47\textwidth]{figs/compare2.pdf}
\caption{Comparison with popular AR frameworks in industry, including Unity AR Foundation~\cite{AR_Foundation}, AR Toolbox~\cite{AR_Toolbox} and Google Depth Lab~\cite{depthlab}.}
\label{fig6}
\end{figure}
\subsection{Limitations}
First, as shown in~\cref{fig_fail}(a), certain local lighting effects may not be modeled, due to our assumption that all surface points of the virtual object shares the same incident lighting. Second, due to the inaccurate depth estimation in NeRF, there is a small amount of noise in our shadows in~\cref{fig_fail}(b). Finally, we consider direct illumination only, which may lead to artifacts. For example, in~\cref{fig_fail}(c), the virtual object rendered by our method shows red reflections from the wood table. However, this is incorrect, as this part of the wood table is actually occluded by shadows of the object.
%our method only use the incident light at the object center to render the object. When the incident light significantly changes on the virtual object surface, some local effects can not be modeled. \cref{fig_fail}(b) presents noisy shadows caused by inaccurate surface points estimated by NeRF. In~\cref{fig_fail}(c), the virtual object rendered by our method reflects red color from the wood table, but this part of the wood table is actually covered by shadows, which should be black as the ground-truth. Besides, our method only considers the direct illumination and shadows, but in fact, there are complex multi-bounce light effects, e.g. the reflection of the virtual object on the smooth surface in the NeRF. These effects are subtle but require a lot of computation. Previous work like~\cite{pan2007precomputed} attempts to achieve interreflections by huge precomputations, which is impractical for application. We look forward to better solutions in future work.
%For specular or textureless surfaces, NeRF may cause reflections below the surface or messy floaters in the space, which affects our shadows that are based on the surface estimation.
%Our shadow method depends on robust and reliable geometry information estimated from NeRF, which needs to be improved in future work.
\begin{figure}[h]
\centering
\includegraphics[width=0.47\textwidth]{figs/fail.pdf}
\caption{Failure cases: (a) shadow missing due to the assumption that all surface points of the virtual object shares the same incident lighting; (b) noisy shadow caused by inaccurate depth estimation; (c) incorrect shading due to only considering the direct illumination.}
\label{fig_fail}
\end{figure}
\section{Conclusion}
\label{conclusion sec}
We propose a real-time method for inserting virtual objects into a NeRF. Our key contribution is the first framework that fully exploits the lighting and geometry information in NeRFs to enable realistic rendering with occlusion and shadowing effects. We outperform state-of-the-art techniques in terms of quality. Compared with existing work, combining NeRF with AR has considerable benefits and may be widely deployed with the popularity of NeRF. We hope that this paper could inspire further research along this direction.
%Currently, our system takes HDR images as input, which requires postproceesing from LDR images with multiple exposures. We have tried methods using LDR images to train HDR NeRF like~\cite{hdr_nerf, nerf_in_dark}. Although they can achieve high-quality tone mapped results, the estimated HDR radiance have significant errors for the lack of supervision, which should be further researched.
%Our method still suffers from some limitations. The training of instant-ngp is still not robust enough. There are strict requirements for camera poses used for generating image training set. Casually captured images often result in failed NeRF training. In addition, the applicable scenes are also limited. NeRFs trained with scenes that contains specular or textureless objects tend to provide incorrect geometry information. For practical applications, various defects of NeRF should be resolved in future work.
%Our method currently render images of $800\times800$ resolution at average 14 fps, which is not fast enough for general AR applications. NeRF rendering takes the majority of time. We believe that the frame rate can be further increased to over 25 fps by more efficient code implementation and introducing spatial or temporal interpolation methods.
%Our method works well in most cases but some defects also exist. As shown in~\cref{fig_fail}(a), our method only use the incident light at the object center to render the object. When the incident light significantly changes on the virtual object surface, some local effects can not be modeled. \cref{fig_fail}(b) presents noisy shadows caused by inaccurate surface points estimated by NeRF. In~\cref{fig_fail}(c), the virtual object rendered by our method reflects red color from the wood table, but this part of the wood table is actually covered by shadows, which should be black as the ground-truth. Besides, our method only considers the direct illumination and shadows, but in fact, there are complex multi-bounce light effects, e.g. the reflection of the virtual object on the smooth surface in the NeRF. These effects are subtle but require a lot of computation. Previous work like~\cite{pan2007precomputed} attempts to achieve interreflections by huge precomputations, which is impractical for application. We look forward to better solutions in future work.
% \subsection{Date}
% Computational Visual Media Conference will be held on xxx to xxx.
% \subsection{Language}
% English is the official language of the conference.
% \section{About the paper submission}
% \subsection{Paper length}
% cvm papers may be between 4 pages and 14 pages. Over length papers will simply not be reviewed.
% %-------------------------------------------------------------------------
% \subsection{Draft and final copy}
% The \LaTeX\ style defines a printed ruler which should be present in the
% version submitted for review. The ruler is provided in order that
% reviewers may comment on particular lines in the paper without
% circumlocution. The camera ready copy should not contain a ruler.
% (\LaTeX\ users may uncomment the \verb'\cvmfinalcopy' command in the document preamble.)
% \subsection{Blind review}
% Many authors misunderstand the concept of anonymizing for blind
% review. Blind review does not mean that one must remove
% citations to one's own work---in fact it is often impossible to
% review a paper unless the previous citations are known and
% available.
% Blind review means that you do not use the words ``my'' or ``our''
% when citing previous work.
% \begin{figure}[t]
% \begin{center}
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
% %\includegraphics[width=0.8\linewidth]{egfigure.eps}
% \end{center}
% \caption{Example of caption. }
% \label{fig:long}
% \label{fig:onecol}
% \end{figure}
% \subsection{Miscellaneous}
% \noindent
% Compare the following:\\
% \begin{tabular}{ll}
% \verb'$conf_a$' & $conf_a$ \\
% \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
% \end{tabular}\\
% See The \TeX book, p165.
% The space after \eg, meaning ``for example'', should not be a
% sentence-ending space. So \eg is correct, {\em e.g.} is not. The provided
% \verb'\eg' macro takes care of this.
% When citing a multi-author paper, you may save space by using ``et alia'',
% shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
% However, use it only when there are three or more authors. Thus, the
% following is correct: ``
% Frobnication has been trendy lately.
% It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
% Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''
% This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
% because reference~\cite{Alpher03} has just two authors. If you use the
% \verb'\etal' macro provided, then you need not worry about double periods
% when used at the end of a sentence as in Alpher \etal.
% For this citation style, keep multiple citations in numerical (not
% chronological) order, so prefer \cite{Alpher03,Alpher02,Authors12} to
% \cite{Alpher02,Alpher03,Authors12}.
% %-------------------------------------------------------------------------
% \subsection{References}
% List and number all bibliographical references in 9-point Times,
% single-spaced, at the end of your paper. When referenced in the text,
% enclose the citation number in square brackets, for
% example~\cite{Authors12}. Where appropriate, include the name(s) of
% editors of referenced books.
% \begin{table}
% \begin{center}
% \begin{tabular}{|l|c|}
% \hline
% Name & Performance \\
% \hline\hline
% A & OK\\
% B & Bad \\
% Ours & Great\\
% \hline
% \end{tabular}
% \end{center}
% \caption{An example for using tables.}
% \end{table}
% %-------------------------------------------------------------------------
% \subsection{Illustrations, graphs, and photographs}
% All graphics should be centered. Please ensure that any point you wish to
% make is resolvable in a printed copy of the paper. Resize fonts in figures
% to match the font in the body text, and choose line widths which render
% effectively in print. Many readers (and reviewers), even of an electronic
% copy, will choose to print your paper in order to read it. You cannot
% insist that they do otherwise, and therefore must not assume that they can
% zoom in to see tiny details on a graphic.
% When placing figures in \LaTeX, it's almost always best to use
% \verb+\includegraphics+, and to specify the figure width as a multiple of
% the line width as in the example below
% {\small\begin{verbatim}
% \usepackage[dvips]{graphicx} ...
% \includegraphics[width=0.8\linewidth]
% {myfile.eps}
% \end{verbatim}
% }
% %-------------------------------------------------------------------------
% \section*{Acknowledgement}
% An acknowledgement is used to thank the person, fund, etc., that support this work.
%-------------------------------------------------------------------------
{\small
\bibliographystyle{cvm}
\bibliography{myBib}
}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{alhashim2018high}
I.~Alhashim and P.~Wonka.
\newblock High quality monocular depth estimation via transfer learning.
\newblock {\em arXiv preprint arXiv:1812.11941}, 2018.
\bibitem{AR_Foundation}
Unity {AR} {F}oundation.
\newblock Website, 2023.
\newblock \url{https://unity.com/unity/features/arfoundation}.
\bibitem{AR_Toolbox}
{AR} {T}oolbox.
\newblock Website, 2023.
\newblock
\url{https://play.google.com/store/apps/details?id=fr.smarquis.ar_toolbox}.
\bibitem{azinovic2019inverse}
D.~Azinovic, T.-M. Li, A.~Kaplanyan, and M.~Nie{\ss}ner.
\newblock Inverse path tracing for joint material and lighting estimation.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 2447--2456, 2019.
\bibitem{bhat2021adabins}
S.~F. Bhat, I.~Alhashim, and P.~Wonka.
\newblock Adabins: Depth estimation using adaptive bins.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 4009--4018, 2021.
\bibitem{nerd}
M.~Boss, R.~Braun, V.~Jampani, J.~T. Barron, C.~Liu, and H.~Lensch.
\newblock Nerd: Neural reflectance decomposition from image collections.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 12684--12694, 2021.
\bibitem{disney_brdf}
B.~Burley and W.~D.~A. Studios.
\newblock Physically-based shading at disney.
\newblock In {\em Acm Siggraph}, volume 2012, pages 1--7. vol. 2012, 2012.
\bibitem{calian2018faces}
D.~A. Calian, J.-F. Lalonde, P.~Gotardo, T.~Simon, I.~Matthews, and
K.~Mitchell.
\newblock From faces to outdoor light probes.
\newblock In {\em Computer Graphics Forum}, volume~37, pages 51--61. Wiley
Online Library, 2018.
\bibitem{cao2023real}
J.~Cao, H.~Wang, P.~Chemerys, V.~Shakhrai, J.~Hu, Y.~Fu, D.~Makoviichuk,
S.~Tulyakov, and J.~Ren.
\newblock Real-time neural light field on mobile devices.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 8328--8337, 2023.
\bibitem{chang2018pyramid}
J.-R. Chang and Y.-S. Chen.
\newblock Pyramid stereo matching network.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 5410--5418, 2018.
\bibitem{BlenderCycles}
B.~O. Community.
\newblock {\em Blender - a 3D modelling and rendering package}.
\newblock Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
\bibitem{debevec_hdr_recover}
P.~E. Debevec and J.~Malik.
\newblock Recovering high dynamic range radiance maps from photographs.
\newblock In {\em Proceedings of the 24th Annual Conference on Computer
Graphics and Interactive Techniques}, page 369–378. ACM
Press/Addison-Wesley Publishing Co., 1997.
\bibitem{depthlab}
R.~Du, E.~Turner, M.~Dzitsiuk, L.~Prasso, I.~Duarte, J.~Dourgarian, J.~Afonso,
J.~Pascoal, J.~Gladstone, N.~Cruces, et~al.
\newblock Depthlab: Real-time 3d interaction with depth maps for mobile
augmented reality.
\newblock In {\em Proceedings of the 33rd Annual ACM Symposium on User
Interface Software and Technology}, pages 829--843, 2020.
\bibitem{deep_parametric}
M.-A. Gardner, Y.~Hold-Geoffroy, K.~Sunkavalli, C.~Gagn{\'e}, and J.-F.
Lalonde.
\newblock Deep parametric indoor lighting estimation.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 7175--7183, 2019.
\bibitem{gardner2017learning}
M.-A. Gardner, K.~Sunkavalli, E.~Yumer, X.~Shen, E.~Gambaretto, C.~Gagn{\'e},
and J.-F. Lalonde.
\newblock Learning to predict indoor illumination from a single image.
\newblock {\em arXiv preprint arXiv:1704.00090}, 2017.
\bibitem{garon2019fast}
M.~Garon, K.~Sunkavalli, S.~Hadap, N.~Carr, and J.-F. Lalonde.
\newblock Fast spatially-varying indoor lighting estimation.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 6908--6917, 2019.
\bibitem{godard2017unsupervised}
C.~Godard, O.~Mac~Aodha, and G.~J. Brostow.
\newblock Unsupervised monocular depth estimation with left-right consistency.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 270--279, 2017.
\bibitem{all_freq_sphere}
K.~Iwasaki, W.~Furuya, Y.~Dobashi, and T.~Nishita.
\newblock Real-time rendering of dynamic scenes under all-frequency lighting
using integral spherical gaussian.
\newblock In {\em Computer Graphics Forum}, volume~31, pages 727--734. Wiley
Online Library, 2012.
\bibitem{karsch2014automatic}
K.~Karsch, K.~Sunkavalli, S.~Hadap, N.~Carr, H.~Jin, R.~Fonte, M.~Sittig, and
D.~Forsyth.
\newblock Automatic scene inference for 3d object compositing.
\newblock {\em ACM Transactions on Graphics (TOG)}, 33(3):1--15, 2014.
\bibitem{kendall2017end}
A.~Kendall, H.~Martirosyan, S.~Dasgupta, P.~Henry, R.~Kennedy, A.~Bachrach, and
A.~Bry.
\newblock End-to-end learning of geometry and context for deep stereo
regression.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 66--75, 2017.
\bibitem{khan2006image}
E.~A. Khan, E.~Reinhard, R.~W. Fleming, and H.~H. B{\"u}lthoff.
\newblock Image-based material editing.
\newblock {\em ACM Transactions on Graphics (TOG)}, 25(3):654--663, 2006.
\bibitem{deeplight}
C.~LeGendre, W.-C. Ma, G.~Fyffe, J.~Flynn, L.~Charbonnel, J.~Busch, and
P.~Debevec.
\newblock Deeplight: Learning illumination for unconstrained mobile mixed
reality.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5918--5928, 2019.
\bibitem{IRAdobe}
Z.~Li, M.~Shafiei, R.~Ramamoorthi, K.~Sunkavalli, and M.~Chandraker.
\newblock Inverse rendering for complex indoor scenes: Shape, spatially-varying
lighting and svbrdf from a single image.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 2475--2484, 2020.
\bibitem{ma2021neural}
S.~Ma, Q.~Shen, Q.~Hou, Z.~Ren, and K.~Zhou.
\newblock Neural compositing for real-time augmented reality rendering in
low-frequency lighting environments.
\newblock {\em Science China Information Sciences}, 64:1--15, 2021.
\bibitem{ma2021free}
X.~Ma, K.~Kang, R.~Zhu, H.~Wu, and K.~Zhou.
\newblock Free-form scanning of non-planar appearance with neural trace
photography.
\newblock {\em ACM Transactions on Graphics (TOG)}, 40(4):1--13, 2021.
\bibitem{macedo2021occlusion}
M.~C. d.~F. Macedo and A.~L. Apolinario.
\newblock Occlusion handling in augmented reality: Past, present and future.
\newblock {\em IEEE Transactions on Visualization and Computer Graphics}, 2021.
\bibitem{nerf_w}
R.~Martin-Brualla, N.~Radwan, M.~S. Sajjadi, J.~T. Barron, A.~Dosovitskiy, and
D.~Duckworth.
\newblock Nerf in the wild: Neural radiance fields for unconstrained photo
collections.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 7210--7219, 2021.
\bibitem{original_nerf}
B.~Mildenhall, P.~P. Srinivasan, M.~Tancik, J.~T. Barron, R.~Ramamoorthi, and
R.~Ng.
\newblock Nerf: Representing scenes as neural radiance fields for view
synthesis.
\newblock In {\em ECCV}, 2020.
\bibitem{brisque}
A.~Mittal, A.~K. Moorthy, and A.~C. Bovik.
\newblock No-reference image quality assessment in the spatial domain.
\newblock {\em IEEE Transactions on image processing}, 21(12):4695--4708, 2012.
\bibitem{instant_ngp}
T.~M{\"u}ller, A.~Evans, C.~Schied, and A.~Keller.
\newblock Instant neural graphics primitives with a multiresolution hash
encoding.
\newblock {\em ACM Transactions on Graphics (ToG)}, 41(4):1--15, 2022.
\bibitem{nerf_human_act}
S.~Peng, J.~Dong, Q.~Wang, S.~Zhang, Q.~Shuai, X.~Zhou, and H.~Bao.
\newblock Animatable neural radiance fields for modeling dynamic human bodies.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 14314--14323, 2021.
\bibitem{dynamic_nerf}
A.~Pumarola, E.~Corona, G.~Pons-Moll, and F.~Moreno-Noguer.
\newblock D-nerf: Neural radiance fields for dynamic scenes.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 10318--10327, 2021.
\bibitem{qiao2023dynamic}
Y.-L. Qiao, A.~Gao, Y.~Xu, Y.~Feng, J.-B. Huang, and M.~C. Lin.
\newblock Dynamic mesh-aware radiance fields.
\newblock {\em arXiv preprint arXiv:2309.04581}, 2023.
\bibitem{nerv}
P.~P. Srinivasan, B.~Deng, X.~Zhang, M.~Tancik, B.~Mildenhall, and J.~T.
Barron.
\newblock Nerv: Neural reflectance and visibility fields for relighting and
view synthesis.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 7495--7504, 2021.
\bibitem{tewari2017mofa}
A.~Tewari, M.~Zollhofer, H.~Kim, P.~Garrido, F.~Bernard, P.~Perez, and
C.~Theobalt.
\newblock Mofa: Model-based deep convolutional face autoencoder for
unsupervised monocular reconstruction.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision workshops}, pages 1274--1283, 2017.
\bibitem{all_freq}
J.~Wang, P.~Ren, M.~Gong, J.~Snyder, and B.~Guo.
\newblock All-frequency rendering of dynamic, spatially-varying reflectance.
\newblock In {\em ACM SIGGRAPH Asia 2009 Papers}, New York, NY, USA, 2009.
Association for Computing Machinery.
\bibitem{wang2018mvdepthnet}
K.~Wang and S.~Shen.
\newblock Mvdepthnet: Real-time multiview depth estimation neural network.
\newblock In {\em 2018 International conference on 3d vision (3DV)}, pages
248--257. IEEE, 2018.
\bibitem{wang2022neural}
Z.~Wang, W.~Chen, D.~Acuna, J.~Kautz, and S.~Fidler.
\newblock Neural light field estimation for street scenes with differentiable
virtual object insertion.
\newblock In {\em European Conference on Computer Vision}, pages 380--397.
Springer, 2022.
\bibitem{watson2023virtual}
J.~Watson, M.~Sayed, Z.~Qureshi, G.~J. Brostow, S.~Vicente, O.~Mac~Aodha, and
M.~Firman.
\newblock Virtual occlusions through implicit depth.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 9053--9064, 2023.
\bibitem{editable_nerf}
B.~Yang, Y.~Zhang, Y.~Xu, Y.~Li, H.~Zhou, H.~Bao, G.~Zhang, and Z.~Cui.
\newblock Learning object-compositional neural radiance field for editable
scene rendering.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 13779--13788, 2021.
\bibitem{mvs_net}
Y.~Yao, Z.~Luo, S.~Li, T.~Fang, and L.~Quan.
\newblock Mvsnet: Depth inference for unstructured multi-view stereo.
\newblock In {\em Proceedings of the European conference on computer vision
(ECCV)}, pages 767--783, 2018.
\bibitem{emlight}
F.~Zhan, C.~Zhang, Y.~Yu, Y.~Chang, S.~Lu, F.~Ma, and X.~Xie.
\newblock Emlight: Lighting estimation via spherical distribution
approximation.
\newblock In {\em Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~35, pages 3287--3295, 2021.
\bibitem{physg}
K.~Zhang, F.~Luan, Q.~Wang, K.~Bala, and N.~Snavely.
\newblock Physg: Inverse rendering with spherical gaussians for physics-based
material editing and relighting.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5453--5462, 2021.
\bibitem{nerfactor}
X.~Zhang, P.~P. Srinivasan, B.~Deng, P.~Debevec, W.~T. Freeman, and J.~T.
Barron.
\newblock Nerfactor: Neural factorization of shape and reflectance under an
unknown illumination.
\newblock {\em ACM Transactions on Graphics (TOG)}, 40(6):1--18, 2021.
\bibitem{zhang2022modeling}
Y.~Zhang, J.~Sun, X.~He, H.~Fu, R.~Jia, and X.~Zhou.
\newblock Modeling indirect illumination for inverse rendering.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 18643--18652, 2022.
\bibitem{shadow_field}
K.~Zhou, Y.~Hu, S.~Lin, B.~Guo, and H.-Y. Shum.
\newblock Precomputed shadow fields for dynamic scenes.
\newblock In {\em ACM SIGGRAPH 2005 Papers}, page 1196–1201, New York, NY,
USA, 2005. Association for Computing Machinery.
\end{thebibliography}