Decomposing NeRF for Editing via Feature Field Distillation

Extracted source text · arXiv 2205.15585

\documentclass{article}


\PassOptionsToPackage{numbers, compress}{natbib}

\usepackage[preprint]{neurips_2022}







\usepackage[utf8]{inputenc} %
\usepackage[T1]{fontenc}    %
\usepackage{hyperref}       %
\usepackage{url}            %
\usepackage{booktabs}       %
\usepackage{amsfonts}       %
\usepackage{nicefrac}       %
\usepackage{microtype}      %
\usepackage{xcolor}         %
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{lscape}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{wrapfig}
\usepackage{caption}

\usepackage{amsthm}
\theoremstyle{remark}
\newtheorem*{claim}{Claim}

\newcommand{\methodnamelongallcaptal}{Distilled Feature Field}
\newcommand{\methodnamelongcaptal}{Distilled feature field}
\newcommand{\methodnamelong}{distilled feature field}
\newcommand{\methodname}{DFF}
\newcommand{\VS}[1]{\textcolor{pink}{[VS: #1]}}

\newcommand{\SK}[1]{\textcolor{blue}{[SK: #1]}}

\newcommand{\ctext}[1]{\raise0.2ex\hbox{\textcircled{\scriptsize{#1}}}}

\hypersetup{
     colorlinks=true,
     citecolor=[rgb]{0,0,0.7},
     linkcolor=red,
     urlcolor=orange,
}

\title{Decomposing NeRF for Editing \\ via Feature Field Distillation}


\author{%
  Sosuke Kobayashi \\
  Preferred Networks, Inc. \\
  \texttt{sosk@preferred.jp} \\
   \And
   Eiichi Matsumoto \\
   Preferred Networks, Inc. \\
   \texttt{matsumoto@preferred.jp} \\
   \AND
   Vincent Sitzmann \\
   Massachusetts Institute of Technology \\
   \texttt{sitzmann@mit.edu} \\
}


\begin{document}


\maketitle

\vbox{%
         \vskip -0.2in
         \hsize\textwidth
         \linewidth\hsize
         \centering
         \normalsize
         \tt\href{https://pfnet-research.github.io/distilled-feature-fields/}{pfnet-research.github.io/distilled-feature-fields/}
         \vskip 0.2in
}



\begin{abstract}
Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations.
However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional.
In particular, it has been difficult to selectively edit specific regions or objects.
In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes.
We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field.
Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training, and enables us to semantically select and edit regions in the radiance field.
Our experiments validate that the distilled feature fields can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

\end{abstract}

\section{Introduction}

Emerging neural implicit representations or neural fields have been shown to be a promising approach for representing a variety of signals~\citep{sitzmann2019srn,mescheder2019occupancynetworks,park2019deepsdf,yariv2020idr,mildenhall2020nerf}.
In particular, they play an important role in 3D scene reconstruction and novel view synthesis from a limited number of context images.
Neural radiance fields (NeRF)~\citep{mildenhall2020nerf} enabled the recovery of a continuous volume density and radiance field from a limited number of observations, producing high-quality images from arbitrary views via volume rendering with promising applications in computer graphics.
However, editing a scene reconstructed by NeRF is non-obvious because the scene is not object-centric and is implicitly encoded in the weights of a connectionist representation such as an MLP~\cite{mildenhall2020nerf} or a voxelgrid~\cite{yu_and_fridovichkeil2021plenoxels}.
Although we can transform the scene in input or output space or via optimization-based editing~\citep{jang2021codenerf,wang2022clipnerf}, this does not enable selective \emph{object-centric} or \emph{semantic}, local edits, such as moving a single object. 
Prior work has addressed this challenge via coordinate-level, semantic decompositions which allow to selectively move, deform, paint, or optimize parts of a NeRF, but relies on costly annotation of instance segmentations and training of instance-specific networks~\citep{yang2021objectnerf}.
While this can be alleviated  with pre-trained segmentation models~\citep{fu2022panopticnerf,kundu2022panopticneuralfields}, such models require pre-defined closed label sets and domains (e.g., traffic scenes), limiting decomposition and editing.
Local editing of NeRFs ideally requires an efficient, open-set method for coordinate-level decomposition.

In this work, we present \emph{\methodnamelong{}s} (\methodname{}s), a novel approach to query-based scene decomposition for local, interactive editing of NeRFs.
We focus on 3D neural \emph{feature} fields, which map every 3D coordinate to a semantic feature descriptor of that coordinate.
Conditioned on a user query such as a text or image patch, this 3D feature field can compute a decomposition of a scene without re-training.
We train a scene-specific \methodname{} via teacher-student distillation~\citep{hinton2015distilling}, using supervision from feature encoders pre-trained in a self-supervised framework on the image domain.
Unlike the domain of 3D scenes, the image domain boasts massive high-quality datasets and abundant prior work on self-supervised training of effective feature extraction models.
Notably, recently proposed transformer-based models~\citep{vaswani2017transformer,dosovitskiy2021vit} have demonstrated impressive capabilities across various vision- and text-based tasks (e.g., CLIP~\citep{radford2021clip}, LSeg~\citep{li2022languagedriven}, DINO~\citep{caron2021emergingdino}).
Such feature spaces capture semantic properties of regions and make it possible to correspond and segment them well by text, image queries, or clustering.
We employ these models as teacher networks and distill them into 3D feature fields via volume rendering.
The trained feature field enables us to semantically select and edit specific regions in 3D NeRF scenes and render multi-view consistent images from the locally edited scenes.


In extensive experiments, we investigate the applications of neural feature fields with two different pre-trained teacher networks, (1) LSeg~\citep{li2022languagedriven}, a CLIP-inspired language-driven semantic segmentation network, and (2) DINO~\citep{caron2021emergingdino,amir2021deepvitfeatures}, a self-supervised network aware of various object boundaries and correspondences.
LSeg and DINO features allow us to select 3D regions by a simple text query or an image patch, respectively.
We first quantitatively demonstrate that LSeg-based \methodname{}s with label queries can have high 3D segmentation performance compared with an existing point-cloud based 3D segmentation baseline trained on ScanNet~\citep{dai2017scannet}, a supervised point-cloud dataset.
We then demonstrate a variety of 3D appearance and geometry edits across real-world NeRF scenes with no annotations of segmentation; and show that we may edit regions with a single query of text, image, pixel, or cluster choice.


\section{Related Work}\label{sec:related}

\paragraph{Neural Implicit Representations.}
Neural implicit representations or neural fields have recently advanced neural processing for 3D data and multi-view 2D images~\citep{sitzmann2019srn,mescheder2019occupancynetworks,park2019deepsdf,yariv2020idr,mildenhall2020nerf}.
For a review of this emerging space we point the reader to the reports by \citet{kato2020drsurvey,tewari2021neuralrenderingsurvey}, and \citet{xie2021neuralfield}.
In particular, a neural radiance field (NeRF) can be fit to a set of posed 2D images and maps a 3D point coordinate and a view direction to an RGB color and density.
When observations are limited, NeRF often overfits and fails to synthesize novel views with correct geometry and appearance.
Pre-trained vision models have been used for regularizing NeRF via flows~\citep{niemeyer2022regnerf}, multi-view consistency~\citep{jain2021dietnerf}, perceptual loss~\citep{zhang2021ners}, or depth estimation~\citep{wei2021nerfingmvs,roessle2022depthpriorsnerf}.
Some pre-trained models operate not only on the visual world, but also on other modalities such as language.
The recently proposed CLIP model~\citep{radford2021clip} has demonstrated impressive performance in image-and-text alignment, with strong generalization to various textual and visual concepts.
\citet{wang2022clipnerf} and \citet{jain2022dreamfields} use CLIP to edit or generate a single-object NeRF with a text prompt query by optimizing the NeRF parameters to generate images matched with the text.
While such methods are promising, they do not enable selective editing of specific scene regions.
For example, the prompt ``yellow flowers'' may affect unintended scene regions, such as the leaves of a plant.
Our proposed decomposition method leverages pre-trained foundation models to enable selective editing of real-world NeRF scenes.
Neural descriptor fields~\citep{simeonovdu2021ndf} use intermediate features that emerge in a 3D occupancy field network~\citep{mescheder2019occupancynetworks} for efficiently teaching robots object grasping.
Instead of a pre-trained object-centric 3D model, we use 2D vision models as teacher networks via distillation, exploiting recent progress in pre-trained foundation models~\citep{bommasani2021foundation}.



\paragraph{Geometric Decomposition of Neural Scene Representations}


\citet{kohli2020semantic} and \citet{zhi2021semanticnerf} show that neural implicit representations can be combined with supervision of semantic labels.
\citet{yang2021objectnerf} demonstrate that, given view-consistent ground-truth instance segmentation masks during training, NeRF can be trained to represent each object with different parameters, although such an annotation is expensive in practice.
Conditional~\citep{liu2021editing,jang2021codenerf,deng2020nasa,noguchi2021narf} and generative models~\citep{nguyenphuoc2020blockgan,niemeyer2021giraffe,hao2021gancraft} enable a degree of category-specific decomposition (e.g., human bodyparts) and editing on constrained domains with large datasets.
Regular structures such as voxelgrids or octrees~\citep{chabra2020deepls,liu2020nsvf,chen2021mvsnerf,takikawa2021nglod,turk2021meganerf,tancik2022blocknerf,lazova2022controlnerf,sitzmann2019deepvoxels,yu2021plenoctrees,nguyen2019hologan,nguyenphuoc2020blockgan} or unsupervised decomposition~\citep{rebain2021derf,stelzner2021obsurf,yu2022unsupervised,smith2022colf} enable editability via manipulation of localized parameters.
However, the decomposition is limited due to the inflexibly structured boundaries or strong assumptions about scenes; self-supervised object-centric learning is a difficult task.
Other studies also explored reconstruction with more structured hybrid representations via pipelines specialized to a domain (e.g., traffic scene)~\citep{ost2021neuralscenegraphs,fu2022panopticnerf,kundu2022panopticneuralfields} or situation (e.g., each object data is independently accessible)~\citep{guo2020osfs, granskog2021neural,yang2022nrinaroom}.
Note that this line of work defines and constrains domains or the types of segmentation during or before training, and thus limits the degrees of freedom for editable scenes and objects.
In contrast, our method can decomposes scene-specific NeRFs into arbitrary semantic units via text and image queries, enabling versatile scene edits without re-training.




\paragraph{Zero-shot Semantic Segmentation.}
Zero-shot semantic segmentation is a challenging task~\citep{frome2013zeroshotdevise,akata2016labelemb, bucher2019zeroshotsemseg} where a model has to predict semantic labels of pixels in images without a-priori information of the categories.
A typical solution is to use vision-and-language cross-modal encoders.
They are trained to encode images (pixels) and text labels into the same semantic space, and perform zero-shot prediction based on similarity or alignments of the two inputs.
Recent development of image encoder architectures~\citep{vaswani2017transformer,dosovitskiy2021vit,ranftl2021dpt} and large-scale training~\citep{radford2021clip,caron2021emergingdino} have improved the ability and generalization of vision models, including zero-shot models~\citep{li2022languagedriven,lueddecke22promptclipseg,wang2022cris,xu2022simplebaseclip,zhong2022regionclip,rao2022denseclip}.
On the other hand, zero-shot perception in 3D still suffers from the lack of effective architectures and large-scale datasets~\citep{michele2021generativezeroshot,he2021dl3dsegsurvey,gu2021pcsurvey,wu2021towersofbabel}.
Our method is a new approach to perform zero-shot semantic segmentation on scene-specific 3D fields by exploiting progress in the image domain, without semantic 3D supervision.
We note that the goal of this paper is not to achieve state-of-the-art performance on 3D semantic segmentation tasks.
Instead, our goal is the decomposition of neural scene representations for editing, which requires smooth segmentation results on continuous 3D space rather than segmentation of discrete point clouds or voxelgrids.

\section{Preliminaries}\label{sec:preliminaries}


\subsection{Neural Radiance Fields (NeRF)}

NeRF~\citep{mildenhall2020nerf} uses MLPs to output density $\sigma$ and color $\mathbf{c}$ given a point coordinate $\mathbf{x} = (x, y, z)$ in a 3D scene.
This simple scene representation can be rendered and optimized via volume rendering.
Given a pixel's camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, depth $t$ with bounds $[t_{\text{near}}, t_{\text{far}}]$, camera position $\mathbf{o}$, and its view direction $\mathbf{d}$, NeRF calculates the color of a ray using quadrature of $K$ sampled points $\{ \mathbf{x}_k \}_{k=1}^{K}$ with depths $\{ t_k \}_{k=1}^{K}$ as

\vspace{-5mm}
\begin{align}
    \hat{{\mathbf{C}}}(\mathbf{r}) =&
        \sum_{k=1}^{K} \hat{T}(t_k) \, \alpha \left( {\sigma(\mathbf{x}_k) \delta_{k}}\right) \mathbf{c}(\mathbf{x}_k, \mathbf{d}) \, ,
        \quad \hat{T}(t_k) =
        \exp \left(-\sum_{k'=1}^{k-1} \sigma(\mathbf{x}_{k'}) \delta_{k'} \right) \, ,\label{eq:rendering_cT}
\end{align}
where $\alpha \left(x\right) = 1-\exp(-x)$, and $\delta_k = t_{k+1} - t_k$ is the distance between adjacent point samples.
NeRFs are optimized solely on a dataset of images and their camera poses by minimizing a re-rendering loss. 

\subsection{Pre-trained Models and Zero-shot Segmentation of Images}\label{sec:pretrainedzeroshot}

Most semantic segmentation models pre-define a closed set of labels, and cannot flexibly change the segmentation categories or boundaries without supervised training.
In contrast, zero-shot semantic segmentation predicts target regions given open-set queries.
\citet{li2022languagedriven} proposes LSeg, a model to perform zero-shot semantic segmentation by aligning pixel-level features and a text query feature.
LSeg employs an image feature encoder with the DPT architecture~\citep{ranftl2021dpt} and a CLIP-based text label feature encoder~\citep{radford2021clip}, trained via large-scale language-image contrastive learning.
The probability of a text label $l$ given a pixel $r$ in an image $I$, $\mathbf{p}(l | I, r)$, is then calculated via dot product of pixel-level image feature $\mathbf{f}_\text{img}(I, r)$ and queried text feature $\mathbf{f}_\text{q}(l)$ followed by a softmax:

\vspace{-5mm}
\begin{align}
    \mathbf{p}(l|I,r) =&
        \frac{ \exp( \mathbf{f}_\text{img}(I, r) \mathbf{f}_\text{q}(l)^\text{T}) } {\sum_{l'\in \mathcal{L}} \exp( \mathbf{f}_\text{img}(I, r) \mathbf{f}_\text{q}(l')^\text{T}) }  \,\,\, ,\label{eq:probability_of_label}
\end{align}
where $\mathcal{L}$ is a set of possible labels.
If negative labels are not available, we may use other scores like thresholded cosine similarity to directly compute the probability of a label.
During training, LSeg optimizes only the image encoder $\mathbf{f}_\text{img}(I, r)$ by minimizing cross-entropy on supervised semantic segmentation datasets.
The text encoder $\mathbf{f}_\text{q}(l)$ is obtained from a pre-trained CLIP model~\citep{radford2021clip}.
Recently, pre-trained CLIP has been leveraged as backbone for a variety of tasks and has been extended with additional modules sharing the same latent space.
For example, \citet{reimers2019sentencebert,reimers2020makingmultilingual} trains a multi-lingual (more than 50+ languages) text encoder, which enables CLIP and CLIP-inspired variants to use non-English queries like Japanese.
We similarly use the latent space of a pre-trained CLIP for LSeg via distillation, enabling decomposition of NeRFs with both English and non-English queries.
Segmentation can further be performed with other modalities such as image, patch or pixel query features $\mathbf{f}_\text{q}$ using a similar dot-product similarity formulation as in Eq.~\ref{eq:probability_of_label}.
Notably, DINO~\citep{caron2021emergingdino}, a self-supervised vision model, solves video instance segmentation and tracking by calculating similarity among features in adjacent frames.
\citet{amir2021deepvitfeatures} also demonstrate that DINO features work well on co-segmentation and point correspondence by similarity and clustering.
In our experiments, we use these two publicly available models, LSeg and DINO, to obtain features of images and texts for 3D decomposition.




\section{\methodnamelongallcaptal{}s}\label{sec:proposed_method}

\begin{figure}
    \centering
    \includegraphics{figures/network__fixedfont.pdf}
    \caption{Left: A Distilled Feature Field (\methodname{}) maps a coordinate $\mathbf{x}$ and a viewing direction $\mathbf{d}$ to density $\sigma$, color $\mathbf{c}$, and feature $\mathbf{f}$. It is trained by minimizing the difference between rendered features and features as predicted by a pre-trained image feature encoder, as well as the rendered color and ground-truth pixel color. Right: At test time, we may decompose and edit 3D space via selecting and manipulating different 3D regions with a variety of queries.}
    \label{fig:network}
    \vspace{-10pt}
\end{figure}

\subsection{Distilling Foundation Modules into 3D Feature Fields via Volume Rendering}\label{sec:feature_learning_volume_rendering}


NeRF learns a neural field to compute the density and view-dependent color, $\sigma(\mathbf{x})$ and $\mathbf{c}(\mathbf{x}, \mathbf{d})$.
We may extend NeRF by adding decoders for other quantities of interest.
For example, SemanticNeRF~\citep{zhi2021semanticnerf} adds a branch outputting a probability distribution of closed-set semantic labels, trained with supervision via images with ground-truth semantic labels.
This enables prediction of pairs of RGB and semantic segmentation masks from novel views, useful for data augmentation.
However, because ground-truth annotation is costly, the method is inefficient as a means of scene editing~\citep{yang2021objectnerf}.
For specific domains like traffic scenes~\citep{fu2022panopticnerf,kundu2022panopticneuralfields}, we may instead train a closed-set segmentation model and use its prediction for training object-aware neural fields.
However, this approach is possible only if types of objects are limited and the domain-specific supervised dataset is available; limiting the application of scene editing in terms of domain and flexibility of decomposition.

We build on top of these ideas and perform 3D zero-shot segmentation of NeRFs using open-set text labels or other feature queries.
Instead of a branch performing closed-set classification, we propose to add a feature branch outputting a feature vector itself.
This branch models a 3D feature field describing semantics of each spatial point.
We supervise the feature field by a pretrained pixel-level image encoder $\textbf{f}_\text{img}$ as a teacher network.
Given a 3D coordinate $\mathbf{x}$, the feature field outputs a feature vector $\mathbf{f}(\mathbf{x})$ in addition to density $\sigma(\mathbf{x})$ and color $\mathbf{c}(\mathbf{x}, \mathbf{d})$, as shown in Figure~\ref{fig:network}.
Volume rendering of the feature field is similarly performed via
\vspace{-2mm}
\begin{align}
    \hat{{\mathbf{F}}}(\mathbf{r}) =&
        \sum_{k=1}^{K} \hat{T}(t_k) \, \alpha\!\left( {\sigma(\mathbf{x}_k) \delta_{k}}\right) \mathbf{f}(\mathbf{x}_k) \,\, . \label{eq:feature_rendering}
\end{align}
We can optimize $\mathbf{f}$ by minimizing the difference between rendered features $\hat{{\mathbf{F}}}(\mathbf{r})$ and the teacher's features $\textbf{f}_\text{img}(I, r)$. Effectively, we are distilling~\citep{hinton2015distilling} the 2D teacher network into our 3D student network via differentiable rendering, and thus dub this model a \emph{\methodnamelong{}} (\methodname{}).
We add a feature objective $\mathcal{L}_f$ penalizing the difference between rendered features $\hat{{\mathbf{F}}}(\mathbf{r})$ and the teacher's outputs $\textbf{f}_\text{img}(I, r)$ to the photometric loss of the original NeRF.
We use two networks for volume rendering with coarse-and-fine hierarchical sampling.
We thus minimize the sum of photometric loss $L_p$ and feature loss $L_f$, in total, $L$:
\begin{align}
 L &= L_p + \lambda L_f , \,\,\,\, L_p  =\sum_{\mathbf{r}\in \mathcal{R}} {\left\lVert \hat{\mathbf{C}}(\mathbf{r}) - \mathbf{C}(r) \right\rVert}_2^2,  \,\,\,\,
L_f  = \sum_{\mathbf{r}\in \mathcal{R}}
{\left\lVert \hat{\mathbf{F}}(\mathbf{r}) - \mathbf{f}_\text{img}(I, r) \right\rVert}_1
, \label{eq:losses}
\end{align}
where $\mathcal{R}$ are sampled rays, $\mathbf{C}(r)$ is the ground truth pixel color of ray $r$, $\lambda$ is the weight of the feature loss and is set to 0.04 to balance the losses~\citep{zhi2021semanticnerf}.
We apply stop-gradient to density in rendering of features $\hat{{\mathbf{F}}}(\mathbf{r})$ in Equation~\ref{eq:feature_rendering} as the teacher's features $\mathbf{f}_\text{img}(I, r)$ are not fully multi-view consistent, which could harm the quality of reconstructed geometry.


\subsection{Query-based Decomposition and Editing}\label{sec:query_decomposition_editing}

A trained \methodname{} model can perform 3D zero-shot segmentation by its feature field $\mathbf{f}$ and a query encoder $\mathbf{f}_\text{q}$.
Probability of a label $l$ of a point $\mathbf{x}$ in the 3D space, $\mathbf{p}(l | \mathbf{x})$, is calculated by dot product of the 3D feature $\mathbf{f}(\mathbf{x})$ and text label feature $\mathbf{f}_\text{q}(l)$ followed by a softmax:

\vspace{-5mm}
\begin{align}
    \mathbf{p}(l|\mathbf{x}) =&
        \frac{ \exp( \mathbf{f}(\mathbf{x}) \mathbf{f}_\text{q}(l)^\text{T}) } {\sum_{l'\in \mathcal{L}} \exp( \mathbf{f}(\mathbf{x}) \mathbf{f}_\text{q}(l')^\text{T}) }  \,\,\, .\label{eq:probability_field}
\end{align}
This query-based segmentation field is at the core of the proposed method.
It can be calculated at any 3D point without limiting resolution, naturally used in tandem with a radiance field and volume rendering.
Note that the segmentation depends on only the 3D coordinate and the query\footnote{It is an interesting extension to introduce a user's viewpoint to the function for recognizing view-dependent queries like referring expressions (e.g., ``the chair left to the table'')~\citep{chen2020scanrefer,achlioptas2020referit3d,liu2021refer,azuma2022scanqa}. We leave this to future work.}. As the original NeRF, it is thus multi-view consistent.
In addition and important for interactive editing, we can change the segmentation via queries without re-training, which cannot be realized by closed-set methods using semantic~\citep{zhi2021semanticnerf} or instance segmentation annotation~\citep{yang2021objectnerf}.
We may now use this query-conditional segmentation to identify a specific 3D region for editing.
Various edits can be generalized to the merging of two NeRF scenes $\sigma_1(\mathbf{x}), \mathbf{c}_1(\mathbf{x}, \mathbf{d})$ and $\sigma_2(\mathbf{x}), \mathbf{c}_2(\mathbf{x}, \mathbf{d})$, where we use the segmentation field $\mathbf{p}$ for blending.
In the experiments section, we simply modify Eq.~\ref{eq:rendering_cT} as a blend of two scenes based on the ratio of $\alpha$:
\vspace{-1mm}
\begin{align}
    \hat{{\mathbf{C}}}(\mathbf{r}) =&
        \sum_{k=1}^{K} \hat{T}(t_k) \left( \alpha\! \left( {\sigma_1(\mathbf{x}_k) \delta_{k}}\right) \mathbf{c}_1(\mathbf{x}_k, \mathbf{d}) \rho_k
        + \, \alpha\! \left( {\sigma_2(\mathbf{x}_k) \delta_{k}}\right) \mathbf{c}_2(\mathbf{x}_k, \mathbf{d}) (1-\rho_k) \right)
        \, ,\label{eq:mix_c} \\
    \text{where} \quad \rho_k =& \frac{\alpha(\sigma_1(\mathbf{x}_k)\delta_{k})}{ \alpha(\sigma_1(\mathbf{x}_k)\delta_{k}) + \alpha(\sigma_2(\mathbf{x}_k)\delta_{k}) } \, , \,\,
    \hat{T}(t_k) = 
         \prod_{k'=1}^{k-1} \alpha(\sigma_1(\mathbf{x}_{k'})\delta_{k'}) + \alpha(\sigma_2(\mathbf{x}_{k'}) \delta_{k'})  \,\, .\label{eq:mix_rhot}
\end{align}

For example, if we want to apply a geometric transformation $\mathbf{g}$ to a region of a query $l$ in a NeRF scene $(\sigma, \mathbf{c})$, we can render the transformed scene via Eqs.~\ref{eq:mix_c} and \ref{eq:mix_rhot} by setting $\alpha(\sigma_1(\mathbf{x}_k) \delta_{k}) = (1-\mathbf{p}(l|\mathbf{x}_k)) \alpha(\sigma(\mathbf{x}_k) \delta_{k})$, \, $\alpha(\sigma_2(\mathbf{x}_k) \delta_{k}) = \mathbf{p}(l|\mathbf{g}^{-1}(\mathbf{x}_k)) \alpha(\sigma(\mathbf{g}^{-1}(\mathbf{x}_k))\delta_{k})$, \, $\mathbf{c}_1(\mathbf{x}_k, \mathbf{d}) = (1-\mathbf{p}(l|\mathbf{x}_k)) \mathbf{c}(\mathbf{x}_k, \mathbf{d})$, and $\mathbf{c}_2(\mathbf{x}_k, \mathbf{d}) = \mathbf{p}(l|\mathbf{g}^{-1}(\mathbf{x}_k)) \mathbf{c}(\mathbf{g}^{-1}(\mathbf{x}_k), \mathbf{g}^{-1}(\mathbf{d}))$.
We can combine this with more complex edits, including optimization-based methods like CLIPNeRF~\citep{wang2022clipnerf}.
While CLIPNeRF itself cannot selectively edit specific regions in multi-object scenes, our decomposition method enables it to update only desired objects without breaking unintended areas.


\section{Experiments}\label{sec:experiments}

We first conduct a quantitative evaluation of the decomposition achieved by \methodname{}.
We demonstrate that \methodname{} enables 3D semantic segmentation in a benchmark dataset using scanned point clouds with human-annotated semantic segmentation labels.
We then investigate the capabilities of \methodname{} for editing and subsequent novel-view synthesis on real-world datasets.
We use two teacher networks, LSeg~\cite{li2022languagedriven} and DINO~\cite{caron2021emergingdino}, which are pre-trained and publicly available.
Each training image is encoded by the image encoders of the networks and used as target feature maps, $\mathbf{f}_\text{img}(I, r)$, defined in Equation~\ref{eq:losses}.
Because the feature maps are of reduced sizes due to the limitation of the networks, we first resize them to the original image size.
The implementation and settings of NeRF follow \citet{zhi2021semanticnerf}.
In the experiments, an eight-layer MLP as shown in Fig.~\ref{fig:network} is used and optimized via Adam minimizing the loss $L$ in Equation~\ref{eq:losses}.
Positional encoding of lengths 10 and 4 is used for coordinate and view direction.
See appendix~\ref{app:training_model_details} and \ref{app:feature_encoders} for further training details.

\subsection{3D Semantic Segmentation}\label{sec:replica_segmentation}
We construct a 3D semantic segmentation benchmark from four scenes in the Replica dataset~\citep{straub19replica} with data split and posed images provided by \citep{zhi2021semanticnerf}.
See appendix~\ref{app:replica_dataset_experiment} for further details of the dataset.
We train \methodname{} to reconstruct each scene with radiance and feature fields from training images and evaluate the quality of novel view synthesis and 3D segmentation of the annotated point clouds.
We use LSeg as a teacher network.
The LSeg text encoder encodes each label, and the probability of each point is calculated by Equation~\ref{eq:probability_of_label}\footnote{
While LSeg-\methodname{} can perform zero-shot inference using text labels that are not seen during training, we do not focus on thoroughly evaluating the zero-shot ability.
The evaluation has been conducted in the original paper on the teacher network, and \methodname{}'s ability is expected to follow it due to distillation.
Please refer to \citet{li2022languagedriven} for the detail of the zero-shot ability of LSeg.
}.
Note that the training uses only the photometric and feature losses (Equation~\ref{eq:losses}) and does not access any supervision via semantic labels.

\paragraph{Semantic Segmentation Results.}


\begin{figure}
\begin{minipage}{0.4\textwidth}
  \centering
  \includegraphics{figures/compare_coarse_fine_room_0__fixedfont.pdf}
  \caption{Comparison of predictions by coarse and fine MLPs.}\label{fig:compare_coarse_fine}
\end{minipage}
  \hspace{0.05\columnwidth}
\begin{minipage}{0.5\textwidth}
  \captionof{table}{Performance of 3D semantic segmentation on Replica dataset. \methodname{} outperforms a \emph{supervised} point-cloud segmentation model MinkowskiNet42.}
  \label{tab:exp_replica_compare_minko}
  \centering
  \begin{tabular}{lll}
 & mIoU & accuracy\\
\toprule
Supervised 3DCNN & 0.475 & 0.758\\
\midrule
\methodname{} (Coarse) & 0.589 & 0.855 \\
\methodname{} (Fine) & 0.583 & 0.855 \\
\bottomrule
\end{tabular}
\end{minipage}
\end{figure}
\begin{table}
\vspace{-3mm}
  \caption{Performance of novel view synthesis on Replica dataset. PSNR, SSIM, and LPIPS are metrics of image synthesis. $\delta\!<\!\!1.25$ and absrel are metrics of geometry (depth estimation).}
  \vspace{1mm}
  \label{tab:exp_replica_view_synthesis}
  \centering
  \begin{tabular}{lllllll}
 & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ & $\delta\!<\!\!1.25\!\uparrow$ & absrel$\downarrow$ \\
\toprule
basic NeRF & 32.87 & 0.934 & 0.148 & 0.993 & 0.018 \\
\midrule
\methodname{} & 32.85 & 0.932 & 0.150 & 0.993 & 0.017 \\
\methodname{} (overweighting $\lambda$) & 32.68 & 0.927 & 0.162 & 0.993 & 0.018 \\
\bottomrule
  \end{tabular}
\end{table}
First, we show evaluation metrics of 3D semantic segmentation, mean intersection-over-union (mIoU) and accuracy in Table~\ref{tab:exp_replica_compare_minko}.
For comparison, we also experiment with a sparse 3D convolution-based segmentation model, MinkowskiNet42~\citep{choy2019minkowskiconv} taking a colored point cloud as input.
It has a standard state-of-the-art architecture for point cloud segmentation and is trained on the ScanNet dataset~\citep{dai2017scannet}, the largest annotated training dataset of 3D semantic segmentation\footnote{
For a fair comparison, the label set follows the ScanNet dataset.
We also manually tune the range and scale of input point clouds for maximizing the performance of MinkowskiNet42 on the Replica dataset.}.
Results demonstrate that \methodname{}, taught by LSeg, achieves promising performance, even better than the supervised model.
This indicates that \methodname{} succeeds at distilling 3D semantic segmentation from the 2D teacher network.



\paragraph{Impact of Sampling on Semantic Segmentation.}
NeRF employs two MLPs for hierarchical sampling, where the coarse MLP performs volume rendering with fewer points (64) using stratified sampling, and the fine MLP works with importance sampling (192 in total).
So, we have two sampling options to train a feature field.
Although fine sampling is critical for training accurate radiance fields, segmentation is of significantly lower spatial frequency than texture.
We thus analyze the impact of coarse and fine training in Fig.~\ref{fig:compare_coarse_fine}.
As expected, the coarse model produces smooth segmentations, while the fine version introduces high-frequency artifacts.
This smoothness property is important for natural editable novel view synthesis and  is discussed again later.


\paragraph{Compatibility with View Synthesis.}
We also check and compare the quality of novel view synthesis with NeRF, which does not learn feature fields.
Because the feature branch partially shares the layers with the radiance field (as shown in Fig.~\ref{fig:network}), learning feature fields could possibly harm the radiance field.
Despite this concern, as shown in Tab.~\ref{tab:exp_replica_view_synthesis}, the performance of view synthesis is not degraded.
Thus, we can train and use the branch-based \methodname{} with small computational and parameter overhead compared to the original NeRF.
If we excessively increased the weight of the feature loss, $\lambda \times 10$, it hurt view synthesis while not improving segmentation performance further.
We further confirm that independent, light-weight feature-field and radiance-field MLPs achieves for semantic segmentation results competitive with the branch-based approach (see appendix Tab.~\ref{tab:exp_replica_variants} for the result of all variants).
This option is useful especially when we want to introduce \methodname{} decomposition into arbitrary 3D scene representations, including off-the-shelf NeRF models, dynamic NeRFs~\citep{gao2021dynamic,park2021hypernerf,li2021dynerf}, or meshes, without re-training of the radiance field.

\subsection{Editable Novel View Synthesis}\label{sec:editable_novel_view_synthesis}

In the previous section, we quantitatively validated the ability of \methodname{} to perform semantic decomposition.
We now discuss the capability for editable view synthesis on real-world scenes, including the LLFF dataset~\citep{mildenhall2019llff} and our own dataset.
Our method can be used even for LLFF scenes based on normalized device coordinates.
Please see the supplemental material for further results, including video.
In addition to LSeg using a text query, we also experiment with self-supervised DINO~\citep{caron2021emergingdino} as another teacher network to enable query-based decomposition using image patch queries.
Here, we use thresholded cosine similarity to directly compute the probability of a query instead of softmax with negative queries in Eq.~\ref{eq:probability_field}, and set $\mathbf{p}=1$ if the similarity exceeds the threshold~\footnote{
The results of the LLFF room scene used a label set, \{whiteboard, ceiling, light, television, wall, bin, table, cabinet, cable, chair, box, floor\}, for avoiding tuning of thresholds.
}, and $\mathbf{p}=0$ otherwise for hard decomposition.
We first train NeRFs without a feature branch for each scene for 200K iterations ($L_p$), and then finetune them with a feature branch via distillation for 5K iterations ($L_p + \lambda L_f$), since we found that the feature loss converged significantly faster than the photometric loss and short training was thus sufficient.
We use coarse sampling for training feature branches and use it for edited rendering with fine sampling.
See appendix~\ref{app:training_model_details} for the details.


\paragraph{Appearance Editing, Deletion, Extraction.}

\begin{figure}
  \centering
  \includegraphics[width=1\columnwidth]{figures/figure-3--appearance-edits.pdf}
  \caption{Appearance edits of specific objects via different query modalities: an image patch or text.}\label{fig:surface_appearance_changes}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=1\columnwidth]{figures/query_deletion_flower__fixedfont.pdf}
  \caption{Extraction and deletion of specific objects via different query modalities, an image patch or text. The edited views are 3D consistent, unlike an image inpainting baseline~\citep{suvorov2022lamaresolution}}\label{fig:query_deletion_flower}
\end{figure}

We show qualitative evaluations of novel view synthesis in Fig.~\ref{fig:surface_appearance_changes} and Fig.~\ref{fig:query_deletion_flower}.
Specific 3D regions in these scenes are identified and locally edited via decomposition depending on various query modalities.
In these experiments, we use a text query for LSeg-\methodname{} as in Section~\ref{sec:replica_segmentation} and use an image patch query for DINO-\methodname{}.
Because DINO features capture the similarity and correspondences of regions well thanks to self-supervised learning~\citep{caron2021emergingdino,amir2021deepvitfeatures}, image patch queries help select all semantically similar areas at once.
The patch feature is then calculated by averaging the features of all pixels in the patch.

\begin{figure}
\begin{minipage}{0.5\columnwidth}
  \centering
  \includegraphics{figures/nerf_feature_bad__fixedfont.pdf}
  \caption{Appearance edits of specific objects, compared with decomposition using features of a NeRF hidden layer. For reference, we also show PCA-based visualizations of the features.}\label{fig:nerf_feature_bad}
\end{minipage}
  \hspace{0.02\columnwidth}
\begin{minipage}{0.45\columnwidth}
  \centering
   \includegraphics{figures/compare_coarse_fine_edit_horns__fixedfont.pdf}
  \caption{Comparison of predictions by a branch-based feature field MLP and independent MLP with no positional encoding, each of which is trained with coarse and fine sampling.}\label{fig:compare_coarse_fine_edit_horns}
\end{minipage}
\end{figure}

In Figure~\ref{fig:surface_appearance_changes}, we demonstrate that the \methodname{} enables convincing selective appearance edits.
Because our focus is region selection via decomposition, we use simple color transformation for clarity here (e.g., flip RGB to BGR, blend colors).
One might think that the MLP of a radiance field by the original NeRF also has hidden layers, and their features possibly could be used for decomposition.
We confirm that the naive usage of NeRF features is not robust to decomposition, as shown in Fig.~\ref{fig:nerf_feature_bad}, especially in a complex multi-object scene.
We use the 8th hidden layer of the fine radiance field network (i.e., the layer just before branching in Figure~\ref{fig:network})\footnote{Other layers or the coarse MLP of the NeRF also indicated similar behaviors but a little worse qualitatively.}.
NeRF features cannot clearly decompose even objects with simple shapes and colors.
The region selections are leaked to other parts with similar colors, geometry, or positions while they do not entirely cover the targets.
For example, floor selection is leaked to mainly walls, a table, bins, or ceilings.
Chair selection is leaked to irrelevant black parts like television, cables, lighting equipment, or shadows.
This indicates that the feature space of the original NeRF does not learn semantic similarity well and is entangled with unpredictable and more low-level factors like color or spatial adjacency.


In Fig.~\ref{fig:query_deletion_flower}, we demonstrate that the \methodname{} also works well on deletion or extraction of objects, using two patch queries (query-\ctext{1} for leaves and ground, query-\ctext{2} for flowers) and a text query-\ctext{3} ``flower''.
For comparison with a baseline editing method, we show the results by a state-of-the-art image inpainting model, LaMa~\citep{suvorov2022lamaresolution}.
Because the model requires masks for inpainting regions, we manually annotate the views for the evaluation.
As shown in the figure, the image inpainting model cannot generate clear and realistic images, and the different views are not consistent.
On the other hand, \methodname{} produces multi-view consistent plausible results, especially succeeding at extracting foreground objects.
Although the performance on deleting foreground objects is high, a remaining shortcoming is the existence of floating artifacts and blurred volumes in the far distance behind the deleted object.


\paragraph{Priors for Smooth Decomposition}
\begin{wrapfigure}{R}{0.45\columnwidth}
  \vspace{-10pt}
  \centering
  \includegraphics{figures/warp_and_deform__fixedfont.pdf}
  \caption{Editing with warping, deformation, shift, and rotation.}\label{fig:warp_object_to_different_scene}
  \vspace{-10pt}
\end{wrapfigure}
We can organize the challenges of editable NeRFs into several categories: surface decomposition, volume decomposition, lighting decomposition, and estimation of less or never observed parts.
If we edit appearances only, it practically requires decomposing regions only near the surface of objects, i.e., surface decomposition, because the color of a ray is determined mostly in a condensed interval around the surface.
On the other hand, geometric transformations often require a higher level of decomposition.
As shown in the deletion examples, geometric transformation may move or remove some surfaces and expose the space behind them.
This forces models to render unknown regions less or never observed due to occlusions, including even the inside of objects.
Thus, it is desirable to decompose volumes smoothly while synthesizing its inside and back\footnote{
Note that this problem also arises when \citet{yang2021objectnerf} used \emph{ground-truth} instance segmentation masks and trained multiple networks, although the authors did not investigate this issue.
}.
Although these include the same challenges as novel view synthesis tackles, editability further highlights their importance.



Apart from lighting decomposition discussed in prior work~\citep{boss2021nerd,boss2021neuralpil,zhang2021nerfactor}, we further investigate the new challenge of smooth volume decomposition by experimenting with different \methodname{} setups.
As discussed in Section~\ref{sec:replica_segmentation}, \methodname{} has two sampling options to train feature fields.
The coarse training may introduce smoothness regularization and help cohesive decomposition and smoother in-painting of unobserved regions.
Another reasonable smoothness regularizer is to eliminate the high-frequency positional encoding (PE).
We thus train an independent MLP network for a feature field without PE.
We compare four combinations of renderings in Fig.~\ref{fig:compare_coarse_fine_edit_horns}.
To better understand their behavior, we use the DINO-\methodname{}, show k-means clusters of the rendered feature map, and delete the head of the Triceratops by a query choosing its corresponding clusters.
As expected, coarsely trained models and no-PE models succeed in smoother volume decomposition, and this combination can minimize high-frequency floating artifacts.
A side effect is the lack of high-frequency representation power, which sometimes deletes  disparate background regions and misses to represent features of complex structures (e.g., see the cluster visualization of the thin frames of the window).
Towards the best of both worlds, developing proper priors or inductive biases is an important direction for future work~\citep{ramasinghe2022regcoordmlp}.
Otherwise, surface-aware representations like IDR~\citep{yariv2020idr,wang2021neus} could avoid problems with floating artifacts.
Note that not all geometric edits suffer from these problems.
For example, it is often less problematic to move objects closer to the camera, enlarge them, or warp them to other scenes, as shown in Figure~\ref{fig:warp_object_to_different_scene}.

\paragraph{Localizing Optimization-based Editing.}
\begin{figure}[h!]
  \includegraphics[width=\columnwidth]{figures/clipnerf__fixedfont.pdf}
  \caption{Comparison of appearance editing by CLIPNeRF and our extension.}\label{fig:clipnerf}
\end{figure}
Finally, we show a combination with an optimization-based editing method.
CLIPNeRF~\cite{wang2022clipnerf} optimizes the parameters of a radiance field so that its rendered images match with a text prompt via CLIP.
While it is mainly designed for a single-object scene of specific categories, it is possible to apply to other real-world NeRFs.
However, because it cannot control the scope of editing, a prompt like ``white flower'' may change the color of unintentional targets like leaves.
Our \methodname{}-based decomposition can upgrade such an optimization-based method to render a scene via the composition of a CLIP-optimized NeRF scene and the original NeRF scene.
We show the results in Figure~\ref{fig:clipnerf}\footnote{
Because the official implementation is not available, we implemented CLIPNeRF by ourselves for reproducing the experiment in Figure 14 of the paper to the best of our abilities.
See appendix~\ref{sec:clipnerf_implementation} for the details.
}.
Although the naive CLIPNeRF edits unintentional parts, our method helps it to locally edit intentional parts only.
In addition to switching rendering, we can also use the decomposition for controlling training signals during backpropagation.
These extensions broaden the application of CLIPNeRF or other optimization-based editing methods to various real-world scenes.


\section{Discussion and Conclusions}\label{sec:discussion_and_conclusions}
In this work, we propose \methodnamelong{} (\methodname{}), a novel method of NeRF scene decomposition for selective editing.
We present quantitative evaluations of segmentation and extensive qualitative evaluations of editable novel view synthesis.
In addition to these promising results, \methodname{}-based models will benefit from future improvements to self-supervised 2D foundation models.
We also clarify future directions on editable view synthesis through our experiments, especially for smoothness priors and estimation of unobserved regions.
Furthermore, while this work focuses on editable view synthesis, it is also intriguing to transfer \methodname{} to other applications, including 3D registration of text queries~\cite{chen2020scanrefer,achlioptas2020referit3d,liu2021refer,azuma2022scanqa} or robot teaching~\cite{hatori2018interactivepick,simeonovdu2021ndf}.

As a possible negative societal impact, one might use our method for making realistic but fake content by editing NeRFs as desired.
Automatic fake detection methods may help in preventing such misus.
NeRFs are further computation-intense, leading to high electricity usage.
Recent work on efficient NeRFs~\citep{yu_and_fridovichkeil2021plenoxels,mueller2022instantngp,chen2022tensorf} may alleviate this concern.




\section*{Acknowledgements}
We thank Tsukasa Takagi, Toshiki Nakanishi, Hiroharu Kato, and Masaaki Fukuda for helpful feedback.


\bibliographystyle{plainnat}
\bibliography{reference}

























\newpage
\appendix

\section{Training and Model Architectures}\label{app:training_model_details}

In the experiments, during training of 200K iterations, the loss $L$ in Equation~\ref{eq:losses} is minimized by Adam with a linearly decaying learning rate (5e-4 to 8e-5).
During training, Gaussian noise for density is also applied.
The number of coarse and fine samplings is 64 and 128, respectively.
The MLP of neural radiance field consists of eight ReLU layers with 256 dimensions, followed by a linear layer for density, three layers for color, and three layers for feature, as shown in Figure~\ref{fig:network}.
Positional encoding of length 10 is used for input coordinate and its skip connection, and that of length 4 is for viewing direction.
If an independent MLP is prepared for the feature field, it consists of four layers (with a skip connection at the third layer, if positional encoding is used).
The size of training image is $320\times240$ for Replica dataset, and $1008\times756$ for the other datasets.
The batchsize of training rays is 1024 for Replica, 2048 for the others.
During finetuning of feature fields or radiance fields, Gaussian noise is removed and learning rate is set 1e-4.

For segmentation, unless otherwise stated, we use thresholded cosine similarity to directly compute the probability of a query instead of softmax with negative queries in Eq.~\ref{eq:probability_field}, and set $\mathbf{p}=1$ if the similarity exceeds the threshold, and $\mathbf{p}=0$ otherwise for hard decomposition.
In Figure~\ref{fig:surface_appearance_changes} and \ref{fig:nerf_feature_bad}, the results of the room scene used a label set, \{whiteboard, ceiling, light, television, wall, bin, table, cabinet, cable, chair, box, floor\}, for skipping tuning of thresholds.


\section{Feature Encoders}\label{app:feature_encoders}

We investigate two teacher networks, LSeg and DINO, which are pre-trained and publicly available\footnote{\url{https://github.com/isl-org/lang-seg} \quad \url{https://github.com/facebookresearch/dino}}.
Each training image is encoded by the encoders of the networks and used as target feature maps, $\mathbf{f}_\text{img}(I, r)$, defined in Equation~\ref{eq:losses}.
Because the feature maps are of reduced sizes, we use them after resizing to the original image size via interpolation.

For LSeg, we use the official demo model, which has the ViT-L/16 image encoder and CLIP's ViT-B/32 text encoder.
Inference follows the official script and uses multi-scale inference (scales = [0.75, 0.83, 0.92, 1.0, 1.08, 1.17, 1.25] for Replica, [0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25] for the others)\footnote{Multi-scale inference stabilizes features and alleviates artifacts due to discrete patch processing in ViT. Inference with large scale means inference with a zoomed and cropped image. Although it may increase the effective resolution of feature maps, it loses context information and may produce noisy features with wrong semantic understanding.}.
The model is trained on seven different datasets~\citep{lambert2020mseg}, including ADE20K~\citep{zhou2019semantic}, BDD~\citep{yu2020bdd100k}, Cityscapes~\citep{Cordts2016Cityscapes}, COCO-Panoptic~\citep{lin2014microsoft,caesar2018coco},  IDD~\citep{varma2019idd}, Mapillary Vistas~\citep{neuhold2017mapillary}, and SUN RGBD~\citep{song2015sun}.
Possibly because the mixed dataset is biased, especially to traffic scenes, we experimentally confirmed that LSeg does not work well in out-of-distribution regions and queries.
For example, small objects or parts are often not discriminative by the network.
More careful training will improve its zero-shot ability in various domains and that of student neural feature fields.

For DINO, we use the extended implementation by \citet{amir2021deepvitfeatures}\footnote{\url{https://github.com/ShirAmir/dino-vit-features}}.
It uses overlapping patches with stride 4 to enlarge the feature map's size.
We use the feature at the 11th layer of the dino\_vits8 model taking a 448x448-resized image as input.
We average it with the features of the horizontally flipped image.
While DINO cannot accept text queries, its feature captures more fine-grained information than LSeg's.

We visualize features of teacher networks, LSeg and DINO, in Figure~\ref{fig:visualize_pca_features}.
We also visualize features of NeRF in Figure~\ref{fig:nerf_feature_bad}.
For visualizing feature spaces, we use scikit-learn's \texttt{sklearn.decomposition.PCA}~\citep{scikitlearn2011}.
We calculate 3-dimensional PCA components using the teacher's feature map of the first frame of training view image, and visualize features by the components as RGB, normalized with min-max values with outlier removal.
Although the 3-dimensional PCA cannot perfectly visualize the feature space, we show the results for reference and understanding.

As explained, although the features of both models are sufficiently view consistent, their semantic resolution seems different.
LSeg tends to show boundaries with coarse-grained categories, possibly due to the bias in the training datasets.
The result will be changed if we train LSeg with other datasets focusing on fine-grained parts or regions.
In contrast to LSeg, DINO produces fine-grained features, where we can feel most of the original edges.
This enables a wide range of decomposition based on various queries as we confirmed in the experiments.
However, even with DINO, the finest-grained details in high resolution are still challenging to capture, e.g., the ribs of T-rex.
The improvement of feature encoders will push the boundary of the decomposition quality of complex objects.


\begin{figure}[h]
    \centering
    \includegraphics[width=12.0cm]{figures/visualize_pca_features.png}
    \caption{Visualization of features of LSeg (center) and DINO (right). Note that it is invalid to compare colors themselves between different scenes or models.}
    \label{fig:visualize_pca_features}
\end{figure}


\section{Replica Dataset Experiment}\label{app:replica_dataset_experiment}

We experiment \methodnamelong{} on 3D semantic segmentation in Section~\ref{sec:replica_segmentation} for demonstrating basic ability of segmentation of a 3D space and text queries.
Because 3D zero-shot semantic segmentation is a novel task except for some studies~\citet{michele2021generativezeroshot}, no standard benchmark datasets exist.
Furthermore, even for 3D semantic segmentation with closed label sets, there is no dataset with high-quality images, accurate point clouds\footnote{
If the input point clouds exist at geometrically wrong coordinates in reality, their annotated labels could be wrong. It is problematic, especially for evaluating point-cloud-agnostic models like our model.
}, and reasonable annotations.
For proof-of-concept of our work, we created a dataset from the existing Replica dataset~\citep{straub19replica}.
Replica is a moderate-quality room reconstruction dataset with semantic segmentation labels.
We use four scenes, room\_0, room\_1, office\_3, and office\_4 with posed images rendered by \citet{zhi2021semanticnerf}.
Because the pose trajectory was randomly generated, some images are unrealistically rendered (e.g., rendered from a camera inside a furniture).
We filtered such images from training and evaluation.
Near and far of NeRF follow their setting, (0.1, 10.0), except for office\_3 (0.1, 15.0).
Although the quality of reconstructions is better than other datasets like ScanNet~\citep{dai2017scannet}, it still suffers from collapsed geometry, less photo-realistic appearances, and defective annotations.
Some objects are difficult to predict their labels due to bad appearance, label ambiguity, or both.
Because this evaluation is not intended to measure the ability to overcome such unrealistic biases or artifacts in a dataset, we semi-automatically fix the label set of the dataset.
We first apply LSeg to predict the semantic label map of each RGB image and evaluate the accuracy against Replica's original label map.
In the evaluation, the accuracy of some labels is almost or exactly zero.
We ignore points with such labels from the evaluation.
In addition, noticeable label ambiguity is also fixed manually by ignoring the label or re-labeling (e.g., `rug' and `floor' are merged as they are intrinsically nonexclusive and often indistinguishable in the dataset).
For reference, we show a visualization of predictions by LSeg and Replica's ground truth in Figure~\ref{fig:lseg_pred_vis}, and the confusion matrix in Figure~\ref{fig:teacher_confusion_matrix_room_0}.
Note that it is not guaranteed that training images cover all the regions of the scene and its point cloud.
Hence, the evaluation also measures the ability of generalization to unobserved regions via propagation.


\begin{figure}[h]
    \centering
    \includegraphics[width=13.5cm]{figures/lseg_pred_vis.png}
    \caption{Visualization of prediction by the LSeg (teacher network) on the image semantic segmentation task on Replica's room\_0.}
    \label{fig:lseg_pred_vis}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=13.0cm]{figures/teacher_confusion_matrix_room_0.png}
    \caption{Confusion matrix of the LSeg on the image semantic segmentation task on Replica room\_0 (full label setting).}
    \label{fig:teacher_confusion_matrix_room_0}
\end{figure}

\begin{figure}[h]
  \centering
  \includegraphics[width=8cm]{figures/label_similarity_matrix_room_0.png}
  \caption{Similarity matrix of text label features by LSeg of Replica room\_0.}\label{fig:label_similarity_matrix_room_0}
\end{figure}



\section{Ablation Experiments of Variants}

We experiment with some variants of \methodname{} architectures and show the result in Table~\ref{tab:exp_replica_variants}.
While they could indicate different behavior for each scene, the average performance difference was marginal.
In total, using MLP trained with volume rendering of coarse sampling performs better than that of fine sampling based on hierarchical sampling.
The coarse sampling might implicitly regularize MLP and help to obtain smoothness property.
Even if increasing the weight of feature loss, $\lambda$, it did not improve the performance and hurt the quality of view synthesis.

\begin{table}
  \caption{Performance of zero-shot semantic segmentation and novel view synthesis on Replica. mIoU is calculated on the 3D point cloud. @2 allows the matching of labels with the second-highest probability. 2D accuracy to teacher indicates the agreement ratio with the labels predicted by the LSeg teacher network. PSNR, SSIM, and LPIPS are metrics of image synthesis. $\delta\!<\!\!1.25$ and absrel are metrics of geometry (depth estimation).}
  \label{tab:exp_replica_variants}
  \centering
  \small
  \begin{tabular}{llllllllll}
 & mIoU & -@2 & \shortstack{2D acc. to \\teacher} & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ & $\delta\!<\!\!1.25\!\uparrow$ & absrel$\downarrow$ \\
\toprule
basic NeRF & - & - & - & 32.87 & 0.934 & 0.148 & 0.993 & 0.018 \\
\midrule
Coarse MLP \\
indep. freq4 & 0.544 & 0.760 & - & - & - & - & - & - \\
branch & 0.562 & 0.779 & - & - & - & - & - & - \\
branch ($\lambda \times 10$) & 0.565 & 0.790 & - & - & - & - & - & - \\
\midrule
Fine MLP \\
indep. freq4 & 0.553 & 0.774 & 0.941 & 32.87 & 0.934 & 0.148 & 0.993 & 0.018 \\
branch & 0.553 & 0.784 & 0.942 & 32.85 & 0.932 & 0.150 & 0.993 & 0.017 \\
branch ($\lambda \times 10$) & 0.543 & 0.770 & 0.942 & 32.68 & 0.927 & 0.162 & 0.993 & 0.018 \\
\bottomrule
  \end{tabular}
\end{table}


\section{Implementation of CLIPNeRF Experiment}\label{sec:clipnerf_implementation}

Because the official implementation of CLIPNeRF~\cite{wang2022clipnerf} is not available, we implement it from the description by ourselves.
In addition, the experiment for Figure 14 in their paper (i.e., editing of the LLFF scene) unfortunately lacks significant descriptions to be required for reproduction.
Thus, there might be nuanced differences while we confirmed that our result shows similar behaviors.
We follow the description in the paper as much as possible .
For 500 iterations, we optimize the parameters of the NeRF with freezing the sub-parameters related to density.
It minimizes negative cosine similarity between a text prompt and a rendered image from a randomly sampled pose in the training dataset, calculated by CLIP ViT-B/32, with Adam of learning rate 0.0001.
Because a naive differentiable rendering of high-resolution images is intractable due to memory constraint, we use an efficient method.
We render a patch of $128 \times 128$ size, followed by bilinear interpolation for resizing, and feed it to CLIP.
For making the patch cover the view widely, rays in $128 \times 128$ patch are sampled with stride interval 5; thus, CLIP's receptive field is $640 \times 640$ of the original image.
For memory efficiency, during rendering of a patch, we reduce the number of importance sampling to 16; fine MLP uses $64+16=80$ in total.
While it is out of scope of this paper, it is possible to further increase the patch resolution if we use a recomputation technique, where NeRF first computes depth without the need of gradient, and then, render a patch with a few samples nearby the depth surface per ray in a differentiable manner.


\section{CLIP-inspired Segmentation Models}\label{app:clipmodels}

We use 2D vision(-and-language) models as teacher networks for distillation.
In particular, for processing text queries, we use LSeg~\citep{li2022languagedriven} as a teacher network.
We can use other models for teacher networks.
As introduced in Section~\ref{sec:related}, within a few months\footnote{\citet{li2022languagedriven} appeared anonymously at OpenReview on September 29th, 2021. The others appeared at arXiv on November or December, 2021.}, many studies~\citep{li2022languagedriven,lueddecke22promptclipseg,wang2022cris,xu2022simplebaseclip,rao2022denseclip} concurrently tackled image semantic segmentation tasks with similar approaches using CLIP~\citep{radford2021clip}.
They use the pretrained CLIP models as a text encoder and an image feature encoder in freezing or finetuning manners, and train zero-shot image semantic segmentation models.
Because they were concurrent, differences of various choices and performances are missing so far.
While we use LSeg by \citet{li2022languagedriven} for simplicity of implementation, it is possible to use other models, especially text-unconditional encoders, where the image encoder produces a feature map without using text labels and calculates a score map by pixel-level calculations with text labels.
Although the publicly available LSeg model can accept any text queries thanks to the CLIP's text encoder, it is not so robust to out-of-distribution data (e.g., ``fossils'', ``T-Rex'', ``Triceratops'') due to their training strategy overfitting to the bias in its training datasets (e.g., traffic scenes).
Using other concurrent models or the ongoing improvement of training and architectures~\citep{liu2022swinv2} will alleviate such issues and improve \methodnamelong{}.



  











\end{document}
\begin{thebibliography}{101}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Achlioptas et~al.(2020)Achlioptas, Abdelreheem, Xia, Elhoseiny, and
  Guibas]{achlioptas2020referit3d}
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and
  Leonidas~J. Guibas.
\newblock {ReferIt3D}: Neural listeners for fine-grained 3d object
  identification in real-world scenes.
\newblock In \emph{16th European Conference on Computer Vision (ECCV)}, 2020.

\bibitem[Akata et~al.(2016)Akata, Perronnin, Harchaoui, and
  Schmid]{akata2016labelemb}
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid.
\newblock Label-embedding for image classification.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
  Intelligence}, 38\penalty0 (7):\penalty0 1425--1438, 2016.
\newblock \doi{10.1109/TPAMI.2015.2487986}.

\bibitem[Amir et~al.(2021)Amir, Gandelsman, Bagon, and
  Dekel]{amir2021deepvitfeatures}
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.
\newblock Deep vit features as dense visual descriptors.
\newblock \emph{arXiv preprint arXiv:2112.05814}, 2021.

\bibitem[Azuma et~al.(2022)Azuma, Miyanishi, Kurita, and
  Kawanabe]{azuma2022scanqa}
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoki Kawanabe.
\newblock Scanqa: 3d question answering for spatial scene understanding.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.

\bibitem[Bommasani et~al.(2021)Bommasani, Hudson, Adeli, Altman, Arora, von
  Arx, Bernstein, Bohg, Bosselut, Brunskill, Brynjolfsson, Buch, Card,
  Castellon, Chatterji, Chen, Creel, Davis, Demszky, Donahue, Doumbouya,
  Durmus, Ermon, Etchemendy, Ethayarajh, Fei{-}Fei, Finn, Gale, Gillespie,
  Goel, Goodman, Grossman, Guha, Hashimoto, Henderson, Hewitt, Ho, Hong, Hsu,
  Huang, Icard, Jain, Jurafsky, Kalluri, Karamcheti, Keeling, Khani, Khattab,
  Koh, Krass, Krishna, Kuditipudi, and et~al.]{bommasani2021foundation}
Rishi Bommasani, Drew~A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney
  von Arx, Michael~S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma
  Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon,
  Niladri~S. Chatterji, Annie~S. Chen, Kathleen Creel, Jared~Quincy Davis,
  Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano
  Ermon, John Etchemendy, Kawin Ethayarajh, Li~Fei{-}Fei, Chelsea Finn, Trevor
  Gale, Lauren Gillespie, Karan Goel, Noah~D. Goodman, Shelby Grossman, Neel
  Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel~E. Ho, Jenny
  Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky,
  Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar
  Khattab, Pang~Wei Koh, Mark~S. Krass, Ranjay Krishna, Rohith Kuditipudi, and
  et~al.
\newblock On the opportunities and risks of foundation models.
\newblock \emph{arXiv}, 2021.
\newblock URL \url{https://arxiv.org/abs/2108.07258}.

\bibitem[Boss et~al.(2021{\natexlab{a}})Boss, Braun, Jampani, Barron, Liu, and
  Lensch]{boss2021nerd}
Mark Boss, Raphael Braun, Varun Jampani, Jonathan~T. Barron, Ce~Liu, and
  Hendrik P.~A. Lensch.
\newblock Nerd: Neural reflectance decomposition from image collections.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021{\natexlab{a}}.
\newblock URL \url{http://arxiv.org/abs/2012.03918v4}.

\bibitem[Boss et~al.(2021{\natexlab{b}})Boss, Jampani, Braun, Liu, Barron, and
  Lensch]{boss2021neuralpil}
Mark Boss, Varun Jampani, Raphael Braun, Ce~Liu, Jonathan~T. Barron, and
  Hendrik Lensch.
\newblock Neural-{PIL}: Neural pre-integrated lighting for reflectance
  decomposition.
\newblock In A.~Beygelzimer, Y.~Dauphin, P.~Liang, and J.~Wortman Vaughan,
  editors, \emph{Advances in Neural Information Processing Systems},
  2021{\natexlab{b}}.
\newblock URL \url{https://openreview.net/forum?id=fATZNtA1-V0}.

\bibitem[Bucher et~al.(2019)Bucher, VU, Cord, and
  P\'{e}rez]{bucher2019zeroshotsemseg}
Maxime Bucher, Tuan-Hung VU, Matthieu Cord, and Patrick P\'{e}rez.
\newblock Zero-shot semantic segmentation.
\newblock In H.~Wallach, H.~Larochelle, A.~Beygelzimer, F.~d\textquotesingle
  Alch\'{e}-Buc, E.~Fox, and R.~Garnett, editors, \emph{Advances in Neural
  Information Processing Systems}, volume~32, 2019.
\newblock URL
  \url{https://proceedings.neurips.cc/paper/2019/file/0266e33d3f546cb5436a10798e657d97-Paper.pdf}.

\bibitem[Caesar et~al.(2018)Caesar, Uijlings, and Ferrari]{caesar2018coco}
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.
\newblock Coco-stuff: Thing and stuff classes in context.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 1209--1218, 2018.

\bibitem[Caron et~al.(2021)Caron, Touvron, Misra, J\'egou, Mairal, Bojanowski,
  and Joulin]{caron2021emergingdino}
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv\'e J\'egou, Julien Mairal,
  Piotr Bojanowski, and Armand Joulin.
\newblock Emerging properties in self-supervised vision transformers.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
  Computer Vision (ICCV)}, pages 9650--9660, October 2021.

\bibitem[Chabra et~al.(2020)Chabra, Lenssen, Ilg, Schmidt, Straub, Lovegrove,
  and Newcombe]{chabra2020deepls}
Rohan Chabra, Jan~Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven
  Lovegrove, and Richard Newcombe.
\newblock Deep local shapes: Learning local sdf priors for detailed 3d
  reconstruction.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
  (ECCV)}, 2020.
\newblock URL \url{http://arxiv.org/abs/2003.10983v3}.

\bibitem[Chen et~al.(2021)Chen, Xu, Zhao, Zhang, Xiang, Yu, and
  Su]{chen2021mvsnerf}
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu,
  and Hao Su.
\newblock Mvsnerf: Fast generalizable radiance field reconstruction from
  multi-view stereo.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2103.15595v2}.

\bibitem[Chen et~al.(2022)Chen, Xu, Geiger, , Yu, and Su]{chen2022tensorf}
Anpei Chen, Zexiang Xu, Andreas Geiger, , Jingyi Yu, and Hao Su.
\newblock Tensorf: Tensorial radiance fields.
\newblock \emph{arXiv}, 2022.

\bibitem[Chen et~al.(2020)Chen, Chang, and Nie{\ss}ner]{chen2020scanrefer}
Dave~Zhenyu Chen, Angel~X Chang, and Matthias Nie{\ss}ner.
\newblock Scanrefer: 3d object localization in rgb-d scans using natural
  language.
\newblock \emph{16th European Conference on Computer Vision (ECCV)}, 2020.

\bibitem[Choy et~al.(2019)Choy, Gwak, and Savarese]{choy2019minkowskiconv}
Christopher Choy, JunYoung Gwak, and Silvio Savarese.
\newblock 4d spatio-temporal convnets: Minkowski convolutional neural networks.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pages 3075--3084, 2019.

\bibitem[Cordts et~al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson,
  Franke, Roth, and Schiele]{Cordts2016Cityscapes}
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler,
  Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.
\newblock The cityscapes dataset for semantic urban scene understanding.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 3213--3223, 2016.

\bibitem[Dai et~al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and
  Nie{\ss}ner]{dai2017scannet}
Angela Dai, Angel~X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser,
  and Matthias Nie{\ss}ner.
\newblock Scannet: Richly-annotated 3d reconstructions of indoor scenes.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2017.

\bibitem[Deng et~al.(2020)Deng, Lewis, Jeruzalski, Pons-Moll, Hinton, Norouzi,
  and Tagliasacchi]{deng2020nasa}
Boyang Deng, JP~Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton,
  Mohammad Norouzi, and Andrea Tagliasacchi.
\newblock Nasa: Neural articulated shape approximation.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
  (ECCV)}, 2020.
\newblock URL \url{http://arxiv.org/abs/1912.03207v4}.

\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
  Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and
  Houlsby]{dosovitskiy2021vit}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
  Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
  Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
\newblock An image is worth 16x16 words: Transformers for image recognition at
  scale.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=YicbFdNTTy}.

\bibitem[Fridovich-Keil et~al.(2022)Fridovich-Keil, Yu, Tancik, Chen, Recht,
  and Kanazawa]{yu_and_fridovichkeil2021plenoxels}
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and
  Angjoo Kanazawa.
\newblock Plenoxels: Radiance fields without neural networks.
\newblock In \emph{CVPR}, 2022.

\bibitem[Frome et~al.(2013)Frome, Corrado, Shlens, Bengio, Dean, Ranzato, and
  Mikolov]{frome2013zeroshotdevise}
Andrea Frome, Greg~S Corrado, Jon Shlens, Samy Bengio, Jeff Dean,
  Marc\textquotesingle~Aurelio Ranzato, and Tomas Mikolov.
\newblock Devise: A deep visual-semantic embedding model.
\newblock In C.~J.~C. Burges, L.~Bottou, M.~Welling, Z.~Ghahramani, and K.~Q.
  Weinberger, editors, \emph{Advances in Neural Information Processing
  Systems}, volume~26. Curran Associates, Inc., 2013.
\newblock URL
  \url{https://proceedings.neurips.cc/paper/2013/file/7cce53cf90577442771720a370c3c723-Paper.pdf}.

\bibitem[Fu et~al.(2022)Fu, Zhang, Chen, Lu, Zhu, Zhou, Geiger, and
  Liao]{fu2022panopticnerf}
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou,
  Andreas Geiger, and Yiyi Liao.
\newblock Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene
  segmentation.
\newblock \emph{arxiv}, 2022.
\newblock URL \url{https://arxiv.org/abs/2203.15224}.

\bibitem[Gao et~al.(2021)Gao, Saraf, Kopf, and Huang]{gao2021dynamic}
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.
\newblock Dynamic view synthesis from dynamic monocular video.
\newblock \emph{arXiv preprint arXiv:2105.06468}, 2021.
\newblock URL \url{http://arxiv.org/abs/2105.06468v1}.

\bibitem[Granskog et~al.(2021)Granskog, Schnabel, Rousselle, and
  Nov{\'a}k]{granskog2021neural}
Jonathan Granskog, Till~N Schnabel, Fabrice Rousselle, and Jan Nov{\'a}k.
\newblock Neural scene graph rendering.
\newblock \emph{ACM Transactions on Graphics (TOG)}, 40\penalty0 (4):\penalty0
  1--11, 2021.

\bibitem[Guo et~al.(2020)Guo, Fathi, Wu, and Funkhouser]{guo2020osfs}
Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser.
\newblock Object-centric neural scene rendering.
\newblock \emph{arXiv preprint arXiv:2012.08503}, 2020.
\newblock URL \url{http://arxiv.org/abs/2012.08503v1}.

\bibitem[Guo et~al.(2021)Guo, Wang, Hu, Liu, Liu, and
  Bennamoun]{gu2021pcsurvey}
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li~Liu, and Mohammed Bennamoun.
\newblock Deep learning for 3d point clouds: A survey.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
  Intelligence}, 43\penalty0 (12):\penalty0 4338--4364, 2021.
\newblock \doi{10.1109/TPAMI.2020.3005434}.

\bibitem[Hao et~al.(2021)Hao, Mallya, Belongie, and Liu]{hao2021gancraft}
Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.
\newblock Gancraft: Unsupervised 3d neural rendering of minecraft worlds.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2104.07659v1}.

\bibitem[Hatori et~al.(2018)Hatori, Kikuchi, Kobayashi, Takahashi, Tsuboi,
  Unno, Ko, and Tan]{hatori2018interactivepick}
Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi,
  Yuya Unno, Wilson Ko, and Jethro Tan.
\newblock Interactively picking real-world objects with unconstrained spoken
  language instructions.
\newblock In \emph{Proceedings of International Conference on Robotics and
  Automation}, 2018.

\bibitem[He et~al.(2021)He, Yu, Liu, Yang, Sun, Wang, Fu, Zou, and
  Mian]{he2021dl3dsegsurvey}
Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang, Wei Sun, Yaonan Wang, Qiang
  Fu, Yanmei Zou, and Ajmal Mian.
\newblock Deep learning based 3d segmentation: {A} survey.
\newblock \emph{arxiv}, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.05423}.

\bibitem[Hinton et~al.(2015)Hinton, Vinyals, and Dean]{hinton2015distilling}
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean.
\newblock Distilling the knowledge in a neural network.
\newblock In \emph{NIPS Deep Learning and Representation Learning Workshop},
  2015.
\newblock URL \url{http://arxiv.org/abs/1503.02531}.

\bibitem[Jain et~al.(2021)Jain, Tancik, and Abbeel]{jain2021dietnerf}
Ajay Jain, Matthew Tancik, and Pieter Abbeel.
\newblock Putting nerf on a diet: Semantically consistent few-shot view
  synthesis.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
  Computer Vision (ICCV)}, pages 5885--5894, October 2021.

\bibitem[Jain et~al.(2022)Jain, Mildenhall, Barron, Abbeel, and
  Poole]{jain2022dreamfields}
Ajay Jain, Ben Mildenhall, Jonathan~T. Barron, Pieter Abbeel, and Ben Poole.
\newblock Zero-shot text-guided object generation with dream fields.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.
\newblock URL \url{https://arxiv.org/abs/2112.01455}.

\bibitem[Jang and Agapito(2021)]{jang2021codenerf}
Wonbong Jang and Lourdes Agapito.
\newblock Codenerf: Disentangled neural radiance fields for object categories.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2109.01750v1}.

\bibitem[Kato et~al.(2020)Kato, Beker, Morariu, Ando, Matsuoka, Kehl, and
  Gaidon]{kato2020drsurvey}
Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim
  Kehl, and Adrien Gaidon.
\newblock Differentiable rendering: {A} survey.
\newblock \emph{arxiv}, 2020.
\newblock URL \url{https://arxiv.org/abs/2006.12057}.

\bibitem[Kohli et~al.(2020)Kohli, Sitzmann, and Wetzstein]{kohli2020semantic}
Amit Kohli, Vincent Sitzmann, and Gordon Wetzstein.
\newblock Semantic implicit neural scene representations with semi-supervised
  training.
\newblock In \emph{International Conference on 3D Vision (3DV)}. IEEE, 2020.
\newblock URL \url{http://arxiv.org/abs/2003.12673v2}.

\bibitem[Kundu et~al.(2022)Kundu, Genova, Yin, Fathi, Pantofaru, Guibas,
  Tagliasacchi, Dellaert, and Funkhouser]{kundu2022panopticneuralfields}
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru,
  Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.
\newblock {Panoptic Neural Fields: A Semantic Object-Aware Neural Scene
  Representation}.
\newblock In \emph{CVPR}, 2022.

\bibitem[Lambert et~al.(2020)Lambert, Liu, Sener, Hays, and
  Koltun]{lambert2020mseg}
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun.
\newblock Mseg: A composite dataset for multi-domain semantic segmentation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition}, pages 2879--2888, 2020.

\bibitem[Lazova et~al.(2022)Lazova, Guzov, Olszewski, Tulyakov, and
  Pons-Moll]{lazova2022controlnerf}
Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard
  Pons-Moll.
\newblock Control-nerf: Editable feature volumes for scene rendering and
  manipulation.
\newblock \emph{arXiv preprint arXiv:2204.10850}, 2022.

\bibitem[Li et~al.(2022)Li, Weinberger, Belongie, Koltun, and
  Ranftl]{li2022languagedriven}
Boyi Li, Kilian~Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl.
\newblock Language-driven semantic segmentation.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\newblock URL \url{https://openreview.net/forum?id=RriDjddCLN}.

\bibitem[Li et~al.(2021)Li, Slavcheva, Zollhoefer, Green, Lassner, Kim,
  Schmidt, Lovegrove, Goesele, and Lv]{li2021dynerf}
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner,
  Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang
  Lv.
\newblock Neural 3d video synthesis.
\newblock \emph{arXiv preprint arXiv:2103.02597}, 2021.
\newblock URL \url{http://arxiv.org/abs/2103.02597v1}.

\bibitem[Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan,
  Doll{\'a}r, and Zitnick]{lin2014microsoft}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
  Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In \emph{European conference on computer vision}, pages 740--755.
  Springer, 2014.

\bibitem[Liu et~al.(2021{\natexlab{a}})Liu, Lin, Han, Yang, Yu, and
  Cui]{liu2021refer}
Haolin Liu, Anran Lin, Xiaoguang Han, Lei Yang, Yizhou Yu, and Shuguang Cui.
\newblock Refer-it-in-rgbd: A bottom-up approach for 3d visual grounding in
  rgbd images.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition}, pages 6032--6041, 2021{\natexlab{a}}.

\bibitem[Liu et~al.(2020)Liu, Gu, Lin, Chua, and Theobalt]{liu2020nsvf}
Lingjie Liu, Jiatao Gu, Kyaw~Zaw Lin, Tat-Seng Chua, and Christian Theobalt.
\newblock Neural sparse voxel fields.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
  (ECCV)}, 2020.
\newblock URL \url{http://arxiv.org/abs/2007.11571v2}.

\bibitem[Liu et~al.(2021{\natexlab{b}})Liu, Zhang, Zhang, Zhang, Zhu, and
  Russell]{liu2021editing}
Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and
  Bryan Russell.
\newblock Editing conditional radiance fields.
\newblock \emph{arXiv preprint arXiv:2105.06466}, 2021{\natexlab{b}}.
\newblock URL \url{http://arxiv.org/abs/2105.06466v2}.

\bibitem[Liu et~al.(2021{\natexlab{c}})Liu, Hu, Lin, Yao, Xie, Wei, Ning, Cao,
  Zhang, Dong, Wei, and Guo]{liu2022swinv2}
Ze~Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue
  Cao, Zheng Zhang, Li~Dong, Furu Wei, and Baining Guo.
\newblock Swin transformer {V2:} scaling up capacity and resolution.
\newblock \emph{arxiv}, abs/2111.09883, 2021{\natexlab{c}}.
\newblock URL \url{https://arxiv.org/abs/2111.09883}.

\bibitem[Lüddecke and Ecker(2022)]{lueddecke22promptclipseg}
Timo Lüddecke and Alexander Ecker.
\newblock Prompt-based multi-modal image segmentation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.
\newblock URL \url{https://arxiv.org/abs/2112.10003}.

\bibitem[Mescheder et~al.(2019)Mescheder, Oechsle, Niemeyer, Nowozin, and
  Geiger]{mescheder2019occupancynetworks}
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and
  Andreas Geiger.
\newblock Occupancy networks: Learning 3d reconstruction in function space.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2019.
\newblock URL \url{http://arxiv.org/abs/1812.03828v2}.

\bibitem[Michele et~al.(2021)Michele, Boulch, Puy, Bucher, and
  Marlet]{michele2021generativezeroshot}
Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet.
\newblock Generative zero-shot learning for semantic segmentation of 3d point
  clouds.
\newblock In \emph{2021 International Conference on 3D Vision (3DV)}, pages
  992--1002, 2021.
\newblock \doi{10.1109/3DV53792.2021.00107}.

\bibitem[Mildenhall et~al.(2019)Mildenhall, Srinivasan, Ortiz-Cayon, Kalantari,
  Ramamoorthi, Ng, and Kar]{mildenhall2019llff}
Ben Mildenhall, Pratul~P. Srinivasan, Rodrigo Ortiz-Cayon, Nima~Khademi
  Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.
\newblock Local light field fusion: Practical view synthesis with prescriptive
  sampling guidelines.
\newblock \emph{ACM Transactions on Graphics (TOG)}, 2019.

\bibitem[Mildenhall et~al.(2020)Mildenhall, Srinivasan, Tancik, Barron,
  Ramamoorthi, and Ng]{mildenhall2020nerf}
Ben Mildenhall, Pratul~P. Srinivasan, Matthew Tancik, Jonathan~T. Barron, Ravi
  Ramamoorthi, and Ren Ng.
\newblock Nerf: Representing scenes as neural radiance fields for view
  synthesis.
\newblock In \emph{Proceedings of the European Conference on Computer Vision
  (ECCV)}, 2020.
\newblock URL \url{http://arxiv.org/abs/2003.08934v2}.

\bibitem[M\"uller et~al.(2022)M\"uller, Evans, Schied, and
  Keller]{mueller2022instantngp}
Thomas M\"uller, Alex Evans, Christoph Schied, and Alexander Keller.
\newblock Instant neural graphics primitives with a multiresolution hash
  encoding.
\newblock \emph{ACM Trans. Graph.}, 41\penalty0 (4):\penalty0 102:1--102:15,
  July 2022.
\newblock \doi{10.1145/3528223.3530127}.
\newblock URL \url{https://doi.org/10.1145/3528223.3530127}.

\bibitem[Neuhold et~al.(2017)Neuhold, Ollmann, Rota~Bulo, and
  Kontschieder]{neuhold2017mapillary}
Gerhard Neuhold, Tobias Ollmann, Samuel Rota~Bulo, and Peter Kontschieder.
\newblock The mapillary vistas dataset for semantic understanding of street
  scenes.
\newblock In \emph{Proceedings of the IEEE international conference on computer
  vision}, pages 4990--4999, 2017.

\bibitem[Nguyen-Phuoc et~al.(2019)Nguyen-Phuoc, Li, Theis, Richardt, and
  Yang]{nguyen2019hologan}
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang
  Yang.
\newblock Hologan: Unsupervised learning of 3d representations from natural
  images.
\newblock In \emph{Proc. ICCV}, pages 7588--7597, 2019.

\bibitem[Nguyen-Phuoc et~al.(2020)Nguyen-Phuoc, Richardt, Mai, Yang, and
  Mitra]{nguyenphuoc2020blockgan}
Thu~H Nguyen-Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy
  Mitra.
\newblock Blockgan: Learning 3d object-aware scene representations from
  unlabelled images.
\newblock In H.~Larochelle, M.~Ranzato, R.~Hadsell, M.~F. Balcan, and H.~Lin,
  editors, \emph{Advances in Neural Information Processing Systems}, volume~33,
  pages 6767--6778, 2020.
\newblock URL
  \url{https://proceedings.neurips.cc/paper/2020/file/4b29fa4efe4fb7bc667c7b301b74d52d-Paper.pdf}.

\bibitem[Niemeyer and Geiger(2021)]{niemeyer2021giraffe}
Michael Niemeyer and Andreas Geiger.
\newblock Giraffe: Representing scenes as compositional generative neural
  feature fields.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2011.12100v2}.

\bibitem[Niemeyer et~al.(2022)Niemeyer, Barron, Mildenhall, Sajjadi, Geiger,
  and Radwan]{niemeyer2022regnerf}
Michael Niemeyer, Jonathan~T. Barron, Ben Mildenhall, Mehdi S.~M. Sajjadi,
  Andreas Geiger, and Noha Radwan.
\newblock Regnerf: Regularizing neural radiance fields for view synthesis from
  sparse inputs.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.

\bibitem[Noguchi et~al.(2021)Noguchi, Sun, Lin, and Harada]{noguchi2021narf}
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.
\newblock Neural articulated radiance field.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2104.03110v2}.

\bibitem[Ost et~al.(2021)Ost, Mannan, Thuerey, Knodt, and
  Heide]{ost2021neuralscenegraphs}
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide.
\newblock Neural scene graphs for dynamic scenes.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, pages 2856--2865, June 2021.

\bibitem[Park et~al.(2019)Park, Florence, Straub, Newcombe, and
  Lovegrove]{park2019deepsdf}
Jeong~Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven
  Lovegrove.
\newblock Deepsdf: Learning continuous signed distance functions for shape
  representation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2019.
\newblock URL \url{http://arxiv.org/abs/1901.05103v1}.

\bibitem[Park et~al.(2021)Park, Sinha, Hedman, Barron, Bouaziz, Goldman,
  Martin-Brualla, and Seitz]{park2021hypernerf}
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan~T. Barron, Sofien Bouaziz,
  Dan~B Goldman, Ricardo Martin-Brualla, and Steven~M. Seitz.
\newblock Hypernerf: A higher-dimensional representation for topologically
  varying neural radiance fields.
\newblock \emph{arXiv preprint arXiv:2106.13228}, 2021.
\newblock URL \url{http://arxiv.org/abs/2106.13228v2}.

\bibitem[Pedregosa et~al.(2011)Pedregosa, Varoquaux, Gramfort, Michel, Thirion,
  Grisel, Blondel, Prettenhofer, Weiss, Dubourg, Vanderplas, Passos,
  Cournapeau, Brucher, Perrot, and Duchesnay]{scikitlearn2011}
F.~Pedregosa, G.~Varoquaux, A.~Gramfort, V.~Michel, B.~Thirion, O.~Grisel,
  M.~Blondel, P.~Prettenhofer, R.~Weiss, V.~Dubourg, J.~Vanderplas, A.~Passos,
  D.~Cournapeau, M.~Brucher, M.~Perrot, and E.~Duchesnay.
\newblock Scikit-learn: Machine learning in {P}ython.
\newblock \emph{Journal of Machine Learning Research}, 12:\penalty0 2825--2830,
  2011.

\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal,
  Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{radford2021clip}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
  Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
  Gretchen Krueger, and Ilya Sutskever.
\newblock Learning transferable visual models from natural language
  supervision.
\newblock In Marina Meila and Tong Zhang, editors, \emph{Proceedings of the
  38th International Conference on Machine Learning}, volume 139 of
  \emph{Proceedings of Machine Learning Research}, pages 8748--8763. PMLR,
  18--24 Jul 2021.
\newblock URL \url{https://proceedings.mlr.press/v139/radford21a.html}.

\bibitem[Ramasinghe et~al.(2022)Ramasinghe, MacDonald, and
  Lucey]{ramasinghe2022regcoordmlp}
Sameera Ramasinghe, Lachlan~E. MacDonald, and Simon Lucey.
\newblock On regularizing coordinate-mlps.
\newblock \emph{arxiv}, 2022.

\bibitem[Ranftl et~al.(2021)Ranftl, Bochkovskiy, and Koltun]{ranftl2021dpt}
Ren\'{e} Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.
\newblock Vision transformers for dense prediction.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.13413}.

\bibitem[Rao et~al.(2022)Rao, Zhao, Chen, Tang, Zhu, Huang, Zhou, and
  Lu]{rao2022denseclip}
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang,
  Jie Zhou, and Jiwen Lu.
\newblock Denseclip: Language-guided dense prediction with context-aware
  prompting.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.

\bibitem[Rebain et~al.(2021)Rebain, Jiang, Yazdani, Li, Yi, and
  Tagliasacchi]{rebain2021derf}
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke~Li, Kwang~Moo Yi, and Andrea
  Tagliasacchi.
\newblock Derf: Decomposed radiance fields.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2011.12490v1}.

\bibitem[Reimers and Gurevych(2019)]{reimers2019sentencebert}
Nils Reimers and Iryna Gurevych.
\newblock Sentence-{BERT}: Sentence embeddings using {S}iamese {BERT}-networks.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
  Natural Language Processing and the 9th International Joint Conference on
  Natural Language Processing (EMNLP-IJCNLP)}, pages 3982--3992, November 2019.
\newblock \doi{10.18653/v1/D19-1410}.
\newblock URL \url{https://aclanthology.org/D19-1410}.

\bibitem[Reimers and Gurevych(2020)]{reimers2020makingmultilingual}
Nils Reimers and Iryna Gurevych.
\newblock Making monolingual sentence embeddings multilingual using knowledge
  distillation.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
  Natural Language Processing (EMNLP)}, pages 4512--4525, November 2020.
\newblock \doi{10.18653/v1/2020.emnlp-main.365}.
\newblock URL \url{https://aclanthology.org/2020.emnlp-main.365}.

\bibitem[Roessle et~al.(2022)Roessle, Barron, Mildenhall, Srinivasan, and
  Nie{\ss}ner]{roessle2022depthpriorsnerf}
Barbara Roessle, Jonathan~T. Barron, Ben Mildenhall, Pratul~P. Srinivasan, and
  Matthias Nie{\ss}ner.
\newblock Dense depth priors for neural radiance fields from sparse input
  views.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, June 2022.

\bibitem[Simeonov et~al.(2021)Simeonov, Du, Tagliasacchi, Tenenbaum, Rodriguez,
  Agrawal, and Sitzmann]{simeonovdu2021ndf}
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua~B. Tenenbaum, Alberto
  Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.
\newblock Neural descriptor fields: Se(3)-equivariant object representations
  for manipulation.
\newblock \emph{arXiv preprint arXiv:2112.05124}, 2021.

\bibitem[Sitzmann et~al.(2019{\natexlab{a}})Sitzmann, Thies, Heide,
  Nie{\ss}ner, Wetzstein, and Zollh{\"o}fer]{sitzmann2019deepvoxels}
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nie{\ss}ner, Gordon
  Wetzstein, and Michael Zollh{\"o}fer.
\newblock Deepvoxels: Learning persistent 3d feature embeddings.
\newblock In \emph{Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
  2019{\natexlab{a}}.

\bibitem[Sitzmann et~al.(2019{\natexlab{b}})Sitzmann, Zollhofer, and
  Wetzstein]{sitzmann2019srn}
Vincent Sitzmann, Michael Zollhofer, and Gordon Wetzstein.
\newblock Scene representation networks: Continuous 3d-structure-aware neural
  scene representations.
\newblock In \emph{Advances in Neural Information Processing Systems
  (NeurIPS)}. Curran Associates, Inc., 2019{\natexlab{b}}.
\newblock URL \url{http://arxiv.org/abs/1906.01618v2}.

\bibitem[Smith et~al.(2022)Smith, Yu, Zakharov, Durand, Tenenbaum, Wu, and
  Sitzmann]{smith2022colf}
Cameron Smith, Hong-Xing Yu, Sergey Zakharov, Fredo Durand, Joshua~B.
  Tenenbaum, Jiajun Wu, and Vincent Sitzmann.
\newblock Unsupervised discovery and composition of object light fields.
\newblock \emph{arXiv}, 2022.

\bibitem[Song et~al.(2015)Song, Lichtenberg, and Xiao]{song2015sun}
Shuran Song, Samuel~P Lichtenberg, and Jianxiong Xiao.
\newblock Sun rgb-d: A rgb-d scene understanding benchmark suite.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 567--576, 2015.

\bibitem[Stelzner et~al.(2021)Stelzner, Kersting, and
  Kosiorek]{stelzner2021obsurf}
Karl Stelzner, Kristian Kersting, and Adam~R. Kosiorek.
\newblock Decomposing 3d scenes into objects via unsupervised volume
  segmentation.
\newblock \emph{arxiv}, 2021.
\newblock URL \url{http://arxiv.org/abs/2104.01148}.

\bibitem[Straub et~al.(2019)Straub, Whelan, Ma, Chen, Wijmans, Green, Engel,
  Mur-Artal, Ren, Verma, Clarkson, Yan, Budge, Yan, Pan, Yon, Zou, Leon,
  Carter, Briales, Gillingham, Mueggler, Pesqueira, Savva, Batra, Strasdat,
  Nardi, Goesele, Lovegrove, and Newcombe]{straub19replica}
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green,
  Jakob~J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson,
  Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou,
  Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler,
  Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke~M. Strasdat, Renzo~De
  Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.
\newblock The {R}eplica dataset: A digital replica of indoor spaces.
\newblock \emph{arxiv}, 2019.

\bibitem[Suvorov et~al.(2022)Suvorov, Logacheva, Mashikhin, Remizova, Ashukha,
  Silvestrov, Kong, Goka, Park, and Lempitsky]{suvorov2022lamaresolution}
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova,
  Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong
  Park, and Victor Lempitsky.
\newblock Resolution-robust large mask inpainting with fourier convolutions.
\newblock In \emph{Proceedings of the IEEE/CVF Winter Conference on
  Applications of Computer Vision (WACV)}, pages 2149--2159, January 2022.
\newblock URL \url{https://arxiv.org/abs/2109.07161}.

\bibitem[Takikawa et~al.(2021)Takikawa, Litalien, Yin, Kreis, Loop,
  Nowrouzezahrai, Jacobson, McGuire, and Fidler]{takikawa2021nglod}
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek
  Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.
\newblock Neural geometric level of detail: Real-time rendering with implicit
  3d shapes.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2101.10994v1}.

\bibitem[Tancik et~al.(2022)Tancik, Casser, Yan, Pradhan, Mildenhall,
  Srinivasan, Barron, and Kretzschmar]{tancik2022blocknerf}
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall,
  Pratul Srinivasan, Jonathan~T. Barron, and Henrik Kretzschmar.
\newblock {Block-NeRF}: Scalable large scene neural view synthesis.
\newblock \emph{arXiv}, 2022.

\bibitem[Tewari et~al.(2021)Tewari, Fried, Thies, Sitzmann, Lombardi, Xu,
  Simon, Nie\ss{}ner, Tretschk, Liu, Mildenhall, Srinivasan, Pandey,
  Orts-Escolano, Fanello, Guo, Wetzstein, Zhu, Theobalt, Agrawala, Goldman, and
  Zollh\"{o}fer]{tewari2021neuralrenderingsurvey}
A.~Tewari, O.~Fried, J.~Thies, V.~Sitzmann, S.~Lombardi, Z.~Xu, T.~Simon,
  M.~Nie\ss{}ner, E.~Tretschk, L.~Liu, B.~Mildenhall, P.~Srinivasan, R.~Pandey,
  S.~Orts-Escolano, S.~Fanello, M.~Guo, G.~Wetzstein, J.-Y. Zhu, C.~Theobalt,
  M.~Agrawala, D.~B Goldman, and M.~Zollh\"{o}fer.
\newblock Advances in neural rendering.
\newblock In \emph{ACM SIGGRAPH 2021 Courses}, SIGGRAPH '21, 2021.
\newblock \doi{10.1145/3450508.3464573}.
\newblock URL \url{https://arxiv.org/abs/2111.05849}.

\bibitem[Turki et~al.(2021)Turki, Ramanan, and
  Satyanarayanan]{turk2021meganerf}
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan.
\newblock Mega-nerf: Scalable construction of large-scale nerfs for virtual
  fly-throughs.
\newblock \emph{arxiv}, 2021.
\newblock URL \url{https://arxiv.org/abs/2112.10703}.

\bibitem[Varma et~al.(2019)Varma, Subramanian, Namboodiri, Chandraker, and
  Jawahar]{varma2019idd}
Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and
  CV~Jawahar.
\newblock Idd: A dataset for exploring problems of autonomous navigation in
  unconstrained environments.
\newblock In \emph{2019 IEEE Winter Conference on Applications of Computer
  Vision}, pages 1743--1751. IEEE, 2019.

\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
  Kaiser, and Polosukhin]{vaswani2017transformer}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
  Aidan~N Gomez, \L~ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock In I.~Guyon, U.~V. Luxburg, S.~Bengio, H.~Wallach, R.~Fergus,
  S.~Vishwanathan, and R.~Garnett, editors, \emph{Advances in Neural
  Information Processing Systems}, volume~30. Curran Associates, Inc., 2017.
\newblock URL
  \url{https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf}.

\bibitem[Wang et~al.(2022{\natexlab{a}})Wang, Chai, He, Chen, and
  Liao]{wang2022clipnerf}
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.
\newblock Clip-nerf: Text-and-image driven manipulation of neural radiance
  fields.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022{\natexlab{a}}.

\bibitem[Wang et~al.(2021)Wang, Liu, Liu, Theobalt, Komura, and
  Wang]{wang2021neus}
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping
  Wang.
\newblock Neus: Learning neural implicit surfaces by volume rendering for
  multi-view reconstruction.
\newblock In \emph{Proceedings of the Thirtieth International Joint Conference
  on Artificial Intelligence (IJCAI)}. International Joint Conferences on
  Artificial Intelligence Organization, 2021.
\newblock URL \url{http://arxiv.org/abs/2106.10689v1}.

\bibitem[Wang et~al.(2022{\natexlab{b}})Wang, Lu, Li, Tao, Guo, Gong, and
  Liu]{wang2022cris}
Zhaoqing Wang, Yu~Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and
  Tongliang Liu.
\newblock Cris: Clip-driven referring image segmentation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2111.15174}.

\bibitem[Wei et~al.(2021)Wei, Liu, Rao, Zhao, Lu, and Zhou]{wei2021nerfingmvs}
Yi~Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou.
\newblock Nerfingmvs: Guided optimization of neural radiance fields for indoor
  multi-view stereo.
\newblock Sep 2021.
\newblock URL \url{http://arxiv.org/abs/2109.01129v2}.

\bibitem[Wu et~al.(2021)Wu, Averbuch-Elor, Sun, and
  Snavely]{wu2021towersofbabel}
Xiaoshi Wu, Hadar Averbuch-Elor, Jin Sun, and Noah Snavely.
\newblock Towers of babel: Combining images, language, and 3d geometry for
  learning multimodal vision.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
  Computer Vision (ICCV)}, pages 428--437, October 2021.

\bibitem[Xie et~al.(2021)Xie, Takikawa, Saito, Litany, Yan, Khan, Tombari,
  Tompkin, Sitzmann, and Sridhar]{xie2021neuralfield}
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or~Litany, Shiqin Yan, Numair
  Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.
\newblock Neural fields in visual computing and beyond, 2021.
\newblock URL \url{https://neuralfields.cs.brown.edu/}.

\bibitem[Xu et~al.(2021)Xu, Zhang, Wei, Lin, Cao, Hu, and
  Bai]{xu2022simplebaseclip}
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang
  Bai.
\newblock A simple baseline for zero-shot semantic segmentation with
  pre-trained vision-language model.
\newblock \emph{arxiv}, 2021.
\newblock URL \url{https://arxiv.org/abs/2112.14757}.

\bibitem[Yang et~al.(2021)Yang, Zhang, Xu, Li, Zhou, Bao, Zhang, and
  Cui]{yang2021objectnerf}
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng
  Zhang, and Zhaopeng Cui.
\newblock Learning object-compositional neural radiance field for editable
  scene rendering.
\newblock In \emph{International Conference on Computer Vision ({ICCV})},
  October 2021.
\newblock URL \url{https://arxiv.org/abs/2109.01847}.

\bibitem[Yang et~al.(2022)Yang, Zhang, Li, Cui, Fanello, Bao, and
  Zhang]{yang2022nrinaroom}
Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao,
  and Guofeng Zhang.
\newblock Neural rendering in a room: Amodal 3d understanding and
  free-viewpoint rendering for the closed scene composed of pre-captured
  objects.
\newblock \emph{ACM Trans. Graph.}, 41\penalty0 (4):\penalty0 101:1--101:10,
  July 2022.
\newblock \doi{10.1145/3528223.3530163}.
\newblock URL \url{https://doi.org/10.1145/3528223.3530163}.

\bibitem[Yariv et~al.(2020)Yariv, Kasten, Moran, Galun, Atzmon, Basri, and
  Lipman]{yariv2020idr}
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri,
  and Yaron Lipman.
\newblock Multiview neural surface reconstruction by disentangling geometry and
  appearance.
\newblock In \emph{Advances in Neural Information Processing Systems
  (NeurIPS)}. Curran Associates, Inc., 2020.
\newblock URL \url{http://arxiv.org/abs/2003.09852v3}.

\bibitem[Yu et~al.(2021)Yu, Li, Tancik, Li, Ng, and
  Kanazawa]{yu2021plenoctrees}
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.
\newblock Plenoctrees for real-time rendering of neural radiance fields.
\newblock \emph{Proc. ICCV}, 2021.

\bibitem[Yu et~al.(2020)Yu, Chen, Wang, Xian, Chen, Liu, Madhavan, and
  Darrell]{yu2020bdd100k}
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu,
  Vashisht Madhavan, and Trevor Darrell.
\newblock Bdd100k: A diverse driving dataset for heterogeneous multitask
  learning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
  and pattern recognition}, pages 2636--2645, 2020.

\bibitem[Yu et~al.(2022)Yu, Guibas, and Wu]{yu2022unsupervised}
Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu.
\newblock Unsupervised discovery of object radiance fields.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\newblock URL \url{https://openreview.net/forum?id=rwE8SshAlxw}.

\bibitem[Zhang et~al.(2021{\natexlab{a}})Zhang, Yang, Tulsiani, and
  Ramanan]{zhang2021ners}
Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan.
\newblock Ners: Neural reflectance surfaces for sparse-view 3d reconstruction
  in the wild.
\newblock In M.~Ranzato, A.~Beygelzimer, Y.~Dauphin, P.S. Liang, and J.~Wortman
  Vaughan, editors, \emph{Advances in Neural Information Processing Systems},
  volume~34, pages 29835--29847. Curran Associates, Inc., 2021{\natexlab{a}}.
\newblock URL
  \url{https://proceedings.neurips.cc/paper/2021/file/f95ec3de395b4bce25b39ef6138da871-Paper.pdf}.

\bibitem[Zhang et~al.(2021{\natexlab{b}})Zhang, Srinivasan, Deng, Debevec,
  Freeman, and Barron]{zhang2021nerfactor}
Xiuming Zhang, Pratul~P. Srinivasan, Boyang Deng, Paul Debevec, William~T.
  Freeman, and Jonathan~T. Barron.
\newblock Nerfactor: Neural factorization of shape and reflectance under an
  unknown illumination.
\newblock \emph{arXiv preprint arXiv:2106.01970}, 2021{\natexlab{b}}.
\newblock URL \url{http://arxiv.org/abs/2106.01970v1}.

\bibitem[Zhi et~al.(2021)Zhi, Laidlow, Leutenegger, and
  Davison]{zhi2021semanticnerf}
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew~J. Davison.
\newblock In-place scene labelling and understanding with implicit scene
  representation.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer
  Vision (ICCV)}, 2021.
\newblock URL \url{http://arxiv.org/abs/2103.15875v2}.

\bibitem[Zhong et~al.(2022)Zhong, Yang, Zhang, Li, Codella, Li, Zhou, Dai,
  Yuan, Li, and Gao]{zhong2022regionclip}
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella,
  Liunian~Harold Li, Luowei Zhou, Xiyang Dai, Lu~Yuan, Yin Li, and Jianfeng
  Gao.
\newblock Regionclip: Region-based language-image pretraining.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2022.
\newblock URL \url{https://arxiv.org/abs/2112.09106}.

\bibitem[Zhou et~al.(2019)Zhou, Zhao, Puig, Xiao, Fidler, Barriuso, and
  Torralba]{zhou2019semantic}
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso,
  and Antonio Torralba.
\newblock Semantic understanding of scenes through the ade20k dataset.
\newblock \emph{International Journal of Computer Vision}, 127\penalty0
  (3):\penalty0 302--321, 2019.

\end{thebibliography}