FreePoint: Unsupervised Point Cloud Instance Segmentation
Extracted source text · arXiv 2305.06973
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{iccv}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{multirow}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
% \usepackage[breaklinks=true,bookmarks=false]{hyperref}
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\iccvfinalcopy % *** Uncomment this line for the final submission
\def\iccvPaperID{****} % *** Enter the ICCV Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
\ificcvfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
\title{FreePoint: Unsupervised Point Cloud Instance Segmentation}
\author{
Zhikai Zhang$^1$, ~~~
Jian Ding$^{1}$, ~~~
Li Jiang$^2$, ~~~
Dengxin Dai$^3$, ~~~
Gui-Song Xia$^{1}$ \\[0.25cm]
$^1$CAPTAIN, Wuhan University, China \\
~~~~
$^2$ Max Planck Institute for Informatics, Saarland Informatics Campus, Germany \\
$^3$ ETH Zurich, Switzerland
}
\maketitle
% Remove page # from the first page of camera-ready.
\ificcvfinal\thispagestyle{empty}\fi
\maketitle
% Remove page # from the first page of camera-ready.
\ificcvfinal\thispagestyle{empty}\fi
%%%%%%%%% ABSTRACT
\begin{abstract}
Instance segmentation of point clouds is a crucial task in 3D field with numerous applications that involve localizing and segmenting objects in a scene. However, achieving satisfactory results requires a large number of manual annotations, which is a time-consuming and expensive process. To alleviate dependency on annotations, we propose a method, called FreePoint, for underexplored unsupervised class-agnostic instance segmentation on point clouds. In detail, we represent the point features by combining coordinates, colors, normals, and self-supervised deep features.
Based on the point features, we perform a multicut algorithm to segment point clouds into coarse instance masks as pseudo labels, which are used to train a point cloud instance segmentation model.
To alleviate the inaccuracy of coarse masks during training, we propose a weakly-supervised training strategy and corresponding loss. Our work can also serve as an unsupervised pre-training pretext for supervised semantic instance segmentation with limited annotations.
%
For class-agnostic instance segmentation on point clouds, FreePoint largely fills the gap with its fully-supervised counterpart based on the state-of-the-art instance segmentation model Mask3D and even surpasses some previous fully-supervised methods. When serving as a pretext task and fine-tuning on S3DIS, FreePoint outperforms training from scratch by 5.8\% AP
with only 10\% mask annotations.
\end{abstract}
%%%%%%%%% BODY TEXT
\section{Introduction}
Instance segmentation on point clouds aims to segment and recognize objects in a 3D scene, serving as the foundation for a wide range of applications such as autonomous driving, virtual reality, and robot navigation. This task has received increasing attention~\cite{chen2021hierarchical,elich20193d,engelmann20203d,han2020occuseg,he2021dyco3d,jiang2020pointgroup,lahoud20193d,liang2021instance,yang2019learning,schult2022mask3d,vu2022softgroup} for the availability of large-scale point cloud datasets~\cite{armeni20163d,dai2017scannet,mo2019partnet,song2017semantic}. Most of the previous works focus on fully-supervised point cloud segmentation, which requires a large number of bounding boxes and per-point annotations to achieve satisfactory results. However, the annotations of point clouds are labor-intensive. For example, labeling an average scene in ScanNet takes about 22.3 minutes~\cite{dai2017scannet}.
\begin{figure}[!t]
\centering
\includegraphics[width=0.90859095\linewidth]
{demo.pdf}
\vspace{-0.5em}
% \textbf{Qualitative visualization results of \textcolor{blue}{OurNet}.}
\caption{We explore class-agnostic instance segmentation for point clouds without any annotations. To address this task, we propose a method that clusters points based on coordinates, normal vectors, colors, and self-supervised deep features of points. Then we use the clustered pseudo masks to perform a two-step training and improve the unsupervised segmentation quality further. }
\label{fig:demo}
\vspace{-1.0em}
\end{figure}
To relieve the annotation requirements, some weakly-supervised 3D segmentation methods~\cite{Xu20CVPR, Liu21CVPR, Zhang21AAAI, Zhang21ICCV,chibane2022box2mask} and semi-supervised 3D segmentation methods~\cite{Cheng21AAAI,Jiang2021CVPR} have been proposed. Besides, some works explore unsupervised pre-training methods for 3D point clouds~\cite{Hou21CVPR, xie2020pointcontrast,zhang2021self}, mainly focusing on data-efficient scene understanding and achieving satisfactory results when fine-tuning on downstream tasks with limited annotations.
%which require annotations of downstream tasks for fine tuning.
These works, however, still rely on considerable box, point annotations, or a certain proportion of mask annotations to achieve competitive results.
% For 2D images, several works~\cite{wang2022,wang2023cut} have been proposed to explore class-agnostic instance segmentation without any annotations and achieved good results.
% Some unsupervised instance segmentation methods~\cite{wang2022,wang2023cut} have been proposed in 2D images and achieved promising results. But the unsupervised segmentation in 2D images is extremely challenging because of
% Unsupervised instance segmentation is However, there are no works for point clouds.
To further alleviate dependency on manual annotations, we try to address point cloud instance segmentation \textit{without any labels}.
% TODO: structure of introduction:
% (1) Instance segmentation on point clouds are important. Annotation in 3D is very expensive. So we need unsupervised segmentation.
% (2) 3D point cloud is more suitable for unsupervised segmentation than 2D images. Resaons: xxx. Directly using the 2D methods can not achieve good results on 3D point cloud segmentation. However, there is no previous work on unsupervised 3D point cloud segmentation.
% (3) Brief description of our method. The current version is too long.
% (4) Our method outperformed some supervised segmentation models.
% (5) Our method can also be used as a pre-training task.
% \jian{Reorganize}\zzk{only for methods.}
%\jian{Maybe we should also say something about plane segmentation here?}\zzk{I think plane segmentation is detail. We don't have to mention it here.}
In detail, we propose FreePoint, which can be split into three parts: (1) preprocessing and point feature extraction; (2) pseudo mask label generation by point feature based graph partitioning; (3) two-step training using the pseudo labels.
We first adopt plane segmentation algorithm to split a point cloud scene into foreground points and background points. Then, for foreground points, we use a self-supervised pre-trained backbone to generate deep-learning feature embeddings for each point. To enhance our feature representation, we add traditional features, including coordinates, colors, and normal vectors, as additional point features. Our motivation for this choice is based on the observation that the geometry and color features are important and helpful in point cloud segmentation, as validated in some
%These information has been widely adopted by some
traditional point-clustering methods~\cite{rusu2010semantic, rusu20113d, achanta2012slic, papon2013voxel}.
%\jian{Maybe we should add some description of methods that use traditional features for point clustering.}
To generate pseudo mask labels, we solve a multicut~\cite{chopra1993partition} problem based on the affinities of point features and constructed point graphs.
These pseudo masks are used to train an existing instance segmentation model.
In our work, we choose Mask3D\cite{schult2022mask3d} for its efficiency and good performance.
% These coarse class-agnostic instance masks are used as pseudo labels to train a Mask3D model.
Since the pseudo masks are inaccurate and the training can be unstable, we propose a weakly-supervised training strategy and corresponding loss to alleviate this problem. The overview of FreePoint is shown in Figure~\ref{fig:demo}.
We evaluate our method on \textit{class-agnostic instance segmentation}. In this setting, our method shows surprising results without any annotations, even surpassing some existing fully-supervised instance segmentation methods.
Apart from directly acquiring the class-agnostic instance masks, our method can also be used for unsupervised pre-training on 3D point clouds. The learned parameters of the backbone can be used to initialize a supervised instance segmentation model and improve final results with limited annotations.
%\jian{Reorganize}
% TODO: do we need the following several sentences?
% Therefore, we evaluate our pre-trained model on point cloud semantic instance segmentation.
% The backbone of our model can serve as a good point cloud encoder for semantic instance segmentation and improve final results.
%\jian{we need better summary of contributions}
Our contributions in this paper are three-fold:
%\zzk{first work to explore unsupervised instance segmentation, implementing a framework ?}
\begin{itemize}
%\item As far as we know, we are the first to explore the problem of point cloud instance segmentation without any annotations. We implemented the first framework to address the problem effectively.
% \item We propose a new method to extract point features and measure their affinities by combining deep-learning methods and traditional methods. Based on the affinities, we construct
% a graph of points and solve a multicut problem on it to generate 3D pseudo instance masks in an unsupervised manner.
\item We propose a new method to cluster points into instance masks in an unsupervised manner. In detail, we extract point features and measure their affinities by combining deep-learning methods and traditional methods. Based on the affinities, we construct a graph of points and solve a multicut problem on it to generate 3D pseudo instance masks.
\item We further propose a carefully-designed training strategy and corresponding losses, taking pseudo instance masks as annotations and training an instance segmentation model to refine the coarse instance masks.
% These coarse masks are used as pseudo labels to train a Mask3D\cite{schult2022mask3d} model following a carefully designed training strategy and loss.
% For feature extraction, we combine traditional features and features encoded by backbones from unsupervised pre-training.
%\jian{For point segmenting steps, xxx}\zzk{For point segmenting step, we use multi-cut algorithm, but it is not our "contribution".}
%\item We further propose OurNet to generate coarse masks based on points' similarities. These coarse masks are used as pseudo labels to train a Mask3D\cite{schult2022mask3d} model following a novel training strategy and loss.
% \item \textcolor{blue}{OurNet} serves as a strong baseline for class-agnostic point cloud instance segmentation. It achieves more than 50\% accuracy in ScanNet AP of the baseline which uses full annotations and surpasses some existing fully-supervised methods.
% \item We also evaluate \textcolor{blue}{OurNet}'s performance as a pretext task for representation learning for semantic instance segmentation. It shows good results. For example, when fine-tuning on S3DIS dataset with 10\% labeled masks, \textcolor{blue}{OurNet} outperforms training from scratch by +4.8\% AP.
\item We evaluate FreePoint's performance on point cloud class-agnostic instance segmentation. It achieves more than 50\% accuracy in ScanNet AP of the fully-supervised counterpart and even surpasses some previous fully-supervised models. We also evaluate FreePoint's performance as a pretext task. For example, when fine-tuning on S3DIS dataset with 10\% labeled masks, FreePoint outperforms training from scratch by +5.8\% AP and CSC by 3.4\% AP.
\end{itemize}
\section{Related Work}
\paragraph{Point cloud instance segmentation}
Early works on point cloud instance segmentation focus on grouping points based on their affinities~\cite{Wang18CVPR, Wang19CVPR,elich20193d}. They use dense labels to train point feature encoders and segment point clouds by measuring the point affinities. 3D-SIS~\cite{hou20193d} and 3D-BoNet~\cite{yang2019learning} extract bounding box proposals and classify them. Recent works prefer to group points based on predicted semantics and object centers~\cite{Jiang20CVPR, Engelmann20CVPR, Chen21ICCV, Liang21CVPR, Han20CVPR}. Mask3D~\cite{schult2022mask3d} is the first Transformer-based approach to challenge this task. We choose it as our two-step training model for its high efficiency. The above works highly rely on per-point labels to achieve good results. However, acquiring such labels is labor-intensive.
Recently, some 3D instance segmentation works have been proposed to alleviate dependency on costly manual annotations.~\cite{Xu20CVPR, Hou21CVPR, Cheng21AAAI, Jiang2021CVPR, Liu21CVPR, Xu20CVPR, Zhang21AAAI, Zhang21ICCV} assume a sparse number of points is annotated and~\cite{chibane2022box2mask} use only bounding box labels. However, they still rely on considerable annotations to achieve competitive results.
\paragraph{Unsupervised segmentation and detection}
In 2D images, several works explore unsupervised object detection~\cite{cho2015unsupervised,simeoni2021localizing,wang2022tokencut,melas2022deep,shin2022unsupervised}, instance segmentation~\cite{wang2022freesolo,wang2023cut}, and semantic segmentation~\cite{Ke_2022_CVPR, cho2021picie,van2021unsupervised}. In object detection area, some works~\cite{wang2022tokencut, melas2022deep,shin2022unsupervised} use spectral methods to discover and segment main objects in a scene. They first construct an adjacency matrix using spatial features, color features, or features from pre-trained backbones. Then the matrix's eigenvectors and eigenvalues are computed to decompose the image. Recently, a few works~\cite{wang2022freesolo,wang2023cut} have explored unsupervised instance segmentation for 2D images and achieved satisfactory results.
FreeSOLO~\cite{wang2022freesolo} bilinearly downsamples an image into patches to generate queries and keys for each patch. Then the affinities between each query and key are computed and compared with a threshold to generate multiple overlapping masks. The coarse masks are used as pseudo labels to train an instance segmentation model following a weakly-supervised design, which includes a loss utilizing center and bounding box information. Inspired by it, we design a similar loss. Rather than projecting the masks on to $x$-axis and $y$-axis via a max or average operation along each axis, we simply compute \textit{one} center and bounding box for each mask and remove the pairwise term. This modification eliminates voxelization, making the loss simpler and more suitable for point clouds.
%\jian{describe the differences between our weakly-supervised loss and the weakly supervised loss}
\paragraph{3D feature representation}
Traditional methods~\cite{achanta2012slic, papon2013voxel} use features like \textit{coordinates}, \textit{colors} and \textit{normals} to describe each point in a scene. Following the tendency of unsupervised pre-training in 2D field, various works~\cite{afham2022crosspoint,xie2020pointcontrast,Hou21CVPR,pang2022masked,yu2022point,
min2022voxel, zhang2021self,zhang2022point} have been proposed recently to represent 3D features, but mostly focusing on single-object classification tasks on ShapeNet~\cite{chang2015shapenet} or ModelNet~\cite{wu20153d}. Only a few works~\cite{xie2020pointcontrast, Hou21CVPR, zhang2021self} focus on large-scale indoor point cloud datasets, which are important for multi-object segmentation tasks and contain far more than only one object.~\cite{Hou21CVPR} mainly explores how to address downstream tasks in a data-efficient semi-supervised way rather than using full annotations. As a result, many works on instance segmentation and semantic segmentation train their model from scratch and can not benefit from 3D pre-training.
%In our work, we need to segment point clouds based on the similarities of the point features. So our best choice for pre-trained backbone should be trained on dense contrastive learning pretext task as mentioned in \cite{wang2022}. However, unlike DenseCL\cite{wang2021dense} in 2D vision, such works are still empty in 3D field.
%Lack of such pre-trained model limits our work's performance.
\section{Method}
%\jian{The overview is too long.}
\begin{figure*}[t!]
\centering
\includegraphics[width=0.90859095\linewidth]
{pipeline.pdf}
\vspace{-0.5em}
\caption{\textbf{Overview.} For inputted point clouds, we first use plane segmentation to filter out backgrounds. Then we represent the features for points by combining self-supervised deep features and traditional features. After that, we construct a graph and compute the edge affinity costs between points. Based on the graph, we apply a multicut algorithm to segment point clouds into coarse instance masks. These masks are adopted as pseudo labels to train a 3D instance segmentation model with our proposed weakly-supervised loss and two-step training strategy.
}
\label{fig:pipeline}
\vspace{-1.0em}
\end{figure*}
% The goal of this paper is to take a 3D point cloud as input and predict instance segmentation results \textit{without any annotations}.
%The pipeline of our method is illustrated in Fig.~\ref{fig:pipeline}. First, we apply a plane segmentation to separate the foreground point cloud and background point cloud.
Our pipeline can be split into three parts: (1) preprocessing and point feature extraction; (2) pseudo mask label generation by point feature based graph partitioning; (3) two-step training using the pseudo labels.
Concretely, as shown in Figure~\ref{fig:pipeline}, we first apply plane segmentation to separate foreground points from background points.
Next, to represent foreground points, we leverage a combination of traditional features (\textit{i.e.,} coordinates, colors, and normal vectors) and self-supervised deep-learning embeddings. Using this representation, we construct an undirected graph ${\bf{G}} = ({\bf{V}}, {\bf{E}}, {\bf{A}})$, where the vertices ${\bf{V}}$ correspond to the points and the edges ${\bf{E}}$ represent their connections. The affinity cost vector ${\bf{A}}$ measures the affinities between the point features. We then employ a multicut algorithm to decompose ${\bf{G}}$ into coarse instance masks.
% In this part, we generate masks of two different segmenting levels, which are important for our weakly-supervised training strategy.
% We will introduce the details in Sec.~\ref{segmenting}.
Finally, we use the coarse masks to perform two-step training with our proposed weakly-supervised loss and two-step training strategy.
\subsection{Preprocessing and point feature extraction}
\label{feature_extraction}
\paragraph{Preprocessing}
Distinguishing instances from backgrounds in an unsupervised setting is challenging, as numerous inconspicuous objects are integrated into nearby backgrounds, making it difficult to distinguish them.
However, in the context of indoor point cloud datasets, backgrounds primarily consist of large, flat surfaces such as floors, walls, and ceilings, which can be easily recognized. Given this observation, we utilize plane segmentation~\cite{zhou2018open3d} to filter out major surfaces in a scene and consider them as backgrounds. After this step, the original input point cloud
${{\bf{V}}_{full}}\in{\mathbb{R}^{N \times6}}$, which contains coordinate and color information, is divided into two subsets: foreground point cloud ${{\bf{V}}_{fg}}\in{\mathbb{R}^{N_{fg} \times6}}$ and background point cloud ${{\bf{V}}_{bg}}\in{\mathbb{R}^{N_{bg} \times6}}$. Since segmenting backgrounds is not the goal of instance segmentation, we only use ${{\bf{V}}_{fg}}$ in the next feature extracting and point cloud segmenting step.
After that, we perform farthest point sampling~\cite{qi2017pointnet++} to down sample ${{\bf{V}}_{fg}}$ into ${{\bf{V}}_{sampled}}\in{\mathbb{R}^{N_{sampled} \times6}}$. This step can reduce computation cost and make the point distribution more sparse. Because of the sparsity, the sampled points are farther from each other in feature space, which is beneficial for our point cloud segmenting method described in Section~\ref{segmenting}.
\paragraph{Feature extraction} Since our segmenting method is based on the affinity between the feature representation of each point, we should find a way to make points closer in feature embedding space if they belong to the same object and farther otherwise. Hence, we use self-supervised pre-trained backbones to encode points.
% since 3D pre-training~\cite{afham2022crosspoint,xie2020pointcontrast,pang2022masked,yu2022point,min2022voxel,Hou21CVPR} has witnessed great progress by adapting 2D pretext tasks to 3D
However, we find
that deep-learning features alone are not sufficiently discriminative to ensure that points from different instances are well-separated in the feature embedding space. This can result in points belonging to different instances being incorrectly grouped together.
%it difficult to encode points discriminatively using deep-learning features alone, which means even points belonging to different instances can be close to each other in the feature embedding space.
As a supplement, we also explore some traditional methods to represent point features.
Before the era of deep learning, some methods~\cite{achanta2012slic, papon2013voxel} use traditional features to cluster points. For example, Supervoxel~\cite{papon2013voxel} uses features like \textit{coordinates}, \textit{colors} and \textit{normals} to measure the affinities between points and cluster them accordingly. Hence, we use both traditional features - including \textit{coordinates}, \textit{colors} and \textit{normals} - as well as deep-learning features to represent each sampled point and measure their affinities in our work.
% Specifically, from raw point cloud input $P_{full}$, we use least square method to compute the normal vector of each point in $P_{sampled}$.
\subsection{Point cloud segmenting}
\label{segmenting}
\paragraph{Preliminary} Minimum-cost multicut~\cite{chopra1993partition} problem aims to decompose an undirected graph ${\bf{G}} = ({\bf{V}}, {\bf{E}}, {\bf{A}})$ into a set of point subsets ${\{{\bf{V}}_{1},\ldots,{\bf{V}}_{k}\}}$ where ${\bf{V}}_1 \cup \ldots \cup {\bf{V}}_k = {\bf{V}}$ and ${\bf{V}}_i \cap {\bf{V}}_j = \varnothing$ $\forall i\neq j$. Edges that straddle distinct clusters which decomposes ${\bf{G}}$ form the \emph{cut} $\delta({\bf{V}}_1,\ldots,{\bf{V}}_k)$. ${\bf{A}} \in \mathbb{R}^{\bf{E}}$ is an affinity cost vector. Each edge $(u,v) \in {\bf{E}}$ has a cost ${\bf{A}}_{(u,v)}$. We need to find a decomposition \emph{cut} of the undirected graph ${\bf{G}}$ that agrees as much as possible with the affinity cost vector, minimizing the whole cost of \emph{cut}.
%If more edge cost values are negative, ${\bf{G}}$ will be decomposed into more clusters.
\paragraph{Segmenting}In our work, we select RAMA~\cite{abbas2022rama}, a rapid multicut algorithm on GPU, to segment point clouds. Each $v_{i} \in {\bf{V}}$ is connected to the closest $k_{1}$ points ${\{u_{i_{1}}, . . . , u_{i_{k_{1}}}\}} \in {\bf{V}}$ by edges $(v_i,u_{i_{j}}) \in {\bf{E}}$, where $j \in \{1,\ldots,k_{1}\}$.
%\jian{how to select the param k, does that influence the results?}. \zzk{Actually, the hyper-parameters used in point cloud segmenting is very hard to tune. There is another k in up-sampling step(in the next paragraph, denote it as k2). k, k2, $\sigma$ influence each other. My answer is they all influence the results, but very little. Computation effciency.}
Affinity cost vector ${\bf{A}}$ is the affinities of both deep features and traditional features.
For deep-learning feature embeddings ${{\bf{F}}}\in{\mathbb{R}^{N_{sampled} \times {dim}}}$ and normal vectors ${{\bf{N}}}\in{\mathbb{R}^{N_{sampled} \times3}}$, we calculate their cosine similarities:
\begin{equation}
\begin{aligned}
\mathcal{\bf A}_{(i,j),emb} = {\tt Cos}({\bf F}_{i}, {\bf F}_{j}), \mathcal{\bf A}_{(i,j),norm} = {\tt Cos}({\bf N}_{i}, {\bf N}_{j})
\end{aligned}
\end{equation}
We split ${{\bf{V}}_{sampled}}$ into point coordinates ${{\bf{V}}}\in{\mathbb{R}^{N_{sampled} \times3}}$ and point colors ${{\bf{C}}}\in{\mathbb{R}^{N_{sampled} \times3}}$. Then we compute L2 distance respectively in XYZ space and RGB space:
%\jian{It is very strange, that there are two "Euclidean"s.}\zzk{Maybe we can write it as in box2mask, using L2 distance style($||***||_{2} $) instead of "Euclidean". !}
\begin{equation}
\begin{aligned}
\mathcal{\bf A}_{(i,j),xyz} = -\parallel{\bf V}_{i}, {\bf V}_{j}\parallel_2, \mathcal{\bf A}_{(i,j),rgb} = -\parallel{\bf C}_{i}, {\bf C}_{j}\parallel_2
\end{aligned}
\end{equation}
These four affinities are all normalized to have a mean value of 0 and variance of 1. The total affinity can be written as:
\begin{equation}
\label{eq:loss_mask}
\begin{aligned}
\mathcal{\bf A} &= \alpha_{1}{\bf A}_{emb} + \alpha_{2}{\bf A}_{norm} + \alpha_{3}{\bf A}_{xyz} + \alpha_{4}{\bf A}_{rgb}
\end{aligned}
\end{equation}
where $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$, $\alpha_{4}$ are the weights to balance the importance of different affinities.
${\bf{G}}$ will be sent to RAMA~\cite{abbas2022rama} and the output is pseudo instance labels. After running RAMA, preliminary pseudo instance labels ${{\bf{L}}_{sampled}}\in{\mathbb{R}^{N_{sampled}}}$ are formed. To recover to original size, we use $knn$ to find the closest $k_{2}$ points and corresponding labels in ${{\bf{V}}_{sampled}}$ for each point in ${{\bf{V}}_{fg}}$. By majority voting of the $k_{2}$ points, we obtain ${{\bf{L}}_{fg}}\in{\mathbb{R}^{N_{fg}}}$.
%Thus we have a label assignment matrix ${{\bf{M}}}\in{\mathbb{R}^{N_{fg}\times k}}$. The mode along the last axis will be seen as the pseudo label for each point in ${{\bf{P}}_{fg}}$.
Then we annotate points in ${{\bf{V}}_{bg}}$ as background and concatenate it with ${{\bf{L}}_{fg}}$ to obtain final pseudo labels ${{\bf{L}}}\in{\mathbb{R}^{N}}$. The pipeline of pseudo-label generation is shown in Figure~\ref{fig:pseudo_label generation}.
\begin{figure}[t]
\begin{center}
\includegraphics[width=1.0\linewidth]{graph_construction.pdf}
\end{center}
\caption{\textbf{Pseudo-label Generation.} In this figure, we show the complete pipeline of pseudo-label generation. For simplicity, we set $k_{1} = k_{2} = 2$. }
\label{fig:pseudo_label generation}
\end{figure}
%As mentioned before,
RAMA generally segments the scene into more objects if more edge values are negative. When running RAMA, we will add different hyper-parameters $\sigma_{low}, \sigma_{high}$ to affinity:
\begin{equation}
\label{eq:loss_mask}
\begin{aligned}
\mathcal{\bf A}_{final} &= {\bf A} + \{\sigma_{low}, \sigma_{high}\}
\end{aligned}
\end{equation}
%\jian{This paragraph is not clear. Maybe we can use equations to describe sigma.} \zzk{This equation is too simple.!}
By changing $\sigma$, we generate coarse masks of two different segmenting levels. One is able to localize and identify most objects in the scene. We denote these masks as base masks. They have the best accuracy while being over-segmented to some degree (\textit{i.e.,} fail to generate complete masks for instances). To overcome base masks' defects, we generate under-segmented masks with a relatively higher $\sigma$. They will be in good use for the next step following our weakly-supervised design.
%\zzk{Maybe we should delete this paragraph.}For different datasets, $\sigma$ can be different. To a dataset without any annotations, we can find suitable $\sigma$ for either masks respectively by controlling the number of generated instances or visualizing a few scenes following figure. In ablation study, we will show that the $\sigma$ for both masks are robust, which means we can achieve satisfactory enough results without tuning $\sigma$ to a very precise value.
\subsection{Training with coarse masks}
\label{coarsemask}
To further refine the coarse masks, we aim to train a point cloud instance segmenter using these masks as pseudo labels. In our work, we choose Mask3D~\cite{schult2022mask3d}, a Transformer-based model for semantic instance segmentation, for its good performance and efficiency. Coarse masks are often inaccurate, so directly using them to train an instance segmenter in a fully-supervised way will cause unsatisfactory results. Therefore we propose two designs to solve this problem, including a new weakly-supervised loss and a two-step training strategy.
\paragraph{Loss for weakly-supervised training}
In the original implementation of Mask3D~\cite{schult2022mask3d}, both dice loss $\mathcal{L}_{dice}$ and binary cross entropy loss $\mathcal{L}_{BCE}$ are used as mask loss for training. However, our pseudo labels are inaccurate, so using such per-point loss directly may lead to sub-optimal results. So we propose to use these coarse masks as a kind of weak annotation and design a weakly-supervised loss.
%\jian{"Inspired by" is a very dangerous claim, especially in method section. We need to specify which part is inspired by previous works. What is the difference between our work and the previous work. What is our contribution and novelty?}\zzk{I am not very sure how to revise it. In 's loss design part, they also use "Inspired by" and they do not clarify what's their contribution. Maybe we can just delete "Inspired by"! write in related work.}
Inspired by ~\cite{wang2022freesolo,chibane2022box2mask,tian2021boxinst}, we believe mask centers and bounding boxes are important for weakly-supervised training. Mask centers can help to localize instances. We compute the mean value of normalized coordinates in a predicted mask ${\bf m}$ and target mask ${\bf m^{*}}$ along each axis to get prediction center ${c_{mean}}\in(x_{i}, y_{i}, z_{i})$ and target center ${t_{mean}}\in(x_{t}, y_{t}, z_{t})$. Our model is trained to minimize the Euclidean distance between $c_{mean}$ and $t_{mean}$:
\begin{equation}
\begin{aligned}
\mathcal{L}_{mean} = {\tt Euclidean}({avg}({\tt {\bf m}}), {avg}({\tt {\bf m^{*}}}))
\end{aligned}
\end{equation}
We further propose a bounding box loss. Bounding box supervision ensures that the model's predictions accurately reflect the sizes and locations of the objects in the scene. This design can further improve our work's performance. For implementation, we pick the maximum and minimum value along each axis for a predicted mask and a target mask to get two boundary point pairs $(c_{max}, t_{max})$ and $(c_{min}, t_{min})$.
The Euclidean distance of each pair is summed to be our bounding-box loss. The loss can be written as:
\begin{equation}
\begin{aligned}
\mathcal{L}_{box} = {sum}(&{\tt Euclidean}({max}({\tt {\bf m}}),
{{max}(\tt {\bf m^{*}})}),
\\
&{\tt Euclidean}({min}({\tt {\bf m}}), {{min}(\tt {\bf m^{*}})}))
\end{aligned}
\end{equation}
We compute the above losses directly on points without voxelization. Then the weighted sum of each term in weakly-supervised loss and fully-supervised loss will be our final loss, which can be written as:
\begin{equation}
\label{eq:loss_mask}
\begin{aligned}
\mathcal{L} &= \lambda_{dice}\mathcal{L}_{dice} + \lambda_{BCE}\mathcal{L}_{BCE}
\\
&+ \lambda_{mean}\mathcal{L}_{mean} + \lambda_{box}\mathcal{L}_{box}
\end{aligned}
\end{equation}
where $\lambda_{dice}$, $\lambda_{BCE}$, $\lambda_{mean}$, and $\lambda_{box}$ are the weights to balance the importance of different loss terms.
\paragraph{Two-step training strategy}
In section~\ref{segmenting}, we observe that our segmenting method can generate masks of different segmenting levels. For over-segmented masks, the scene will be generally split into object parts. More instances can be identified and localized in this situation, but they lack complete masks. For the under-segmented setting, the scene has fewer instance proposals, which means we will have more masks covering a whole object. However, instances in this setting are easy to be mistakenly connected with other instances.
We wonder which kind of masks we should use to achieve better results. Both over-segmented and under-segmented masks have insurmountable defects if adopted as pseudo labels alone. Therefore, we explore a novel training strategy so that over-segmented and under-segmented masks can compensate for each other's shortcomings and significantly improve final results. Concretely, we use over-segmented masks as pseudo labels for the first training step. At this stage, the model is trained to segment points of similar features,
regardless of whether they belong to object parts or whole objects.
%whether they are segmented as object parts or whole objects.
For the second training step, we use under-segmented masks instead. With only a few epochs, the model learns to connect mistakenly segmented object parts into a whole object. This step can improve the results of the first step by a large margin with little time cost.
The improvement in accuracy aligns with our intuition. The model is first trained to encode points and segment point clouds at a low level. Even though the pseudo labels we use in this step are over-segmented, the model can learn relatively good point feature representations and predict object parts. Then we use under-segmented masks to teach the model how to connect objects and predict complete instance masks.
\section{Experiments}
%\subsection{Experimental Settings}
%\jian{The paragraph is too wordy. Simplify this part.}\zzk{Delete}
%In this section, we will show our work's performance for class-agnostic instance segmentation. Our work is to explore unsupervised instance segmentation. Therefore training sets and validation sets have unusual meanings for us compared to supervised works. We consider two settings for our work. The first one is as regular supervised learning(i.e. training on one dataset and testing on another). The second one is training and testing on the same dataset.
% We think either setting makes sense.
%If we want immediate inference of this model, we should follow the first setting, collecting an unlabeled training set and training the model in advance. However, if we do not have a strict time limitation or the ability to collect a large number of point cloud scenes specially for training, the second setting should be considered. In this section, we will show both results for various tasks. For convenience, we denote the first setting as A and the second setting as B.
\paragraph{Implementation details}
For point downsampling in preprocessing, we downsample the whole point cloud to the half of the number of original points and set $k_1=k_2=4$. When running RAMA~\cite{abbas2022rama}, we set $\sigma_{low}$ = 0.9 and $\sigma_{high}$ = 1.2. For the first and the second training step, the epochs are set to 100 and 10 respectively. This means our work has relatively less time cost, which is about 1/9 of the time cost of the original implementation in Mask3D~\cite{schult2022mask3d} for fully-supervised semantic instance segmentation.
\paragraph{Datasets}
% \jian{give citations of datasets here}\zzk{*}
We evaluate our work on two publicly available indoor 3D instance segmentation datasets ScanNet v2~\cite{dai2017scannet} and S3DIS~\cite{armeni20163d}. The ScanNet v2 dataset altogether contains 1613 scans, divided into
training, validation and testing sets of 1201, 312, 100 scans respectively. 18 object classes are provided in this dataset. The S3DIS dataset contains 3D scans of 6 areas with 271 scenes in total. The dataset consists of 13 classes for instance segmentation evaluation. For unsupervised instance segmentation, we train on training sets and report results on validation sets.
\paragraph{Evaluation Metrics}
We use standard average precision as our evaluation metrics. AP$_{50}$ and AP$_{25}$ denote the scores with IoU thresholds of 0.5 and 0.25 respectively. AP denotes the average scores with IoU threshold from 0.5 to 0.95 with a step size of 0.05.
\subsection{Main Results}
\paragraph{Unsupervised instance segmentation}
%\jian{rewrite}
There are only few particular works~\cite{chen2021lrgnet} for class-agnostic point cloud instance segmentation. So we also adapt some existing models originally for fully-supervised semantic instance segmentation task to class-agnostic style by removing their semantic prediction branch and corresponding loss. Doing so can lead to a accuracy drop as shown in Table~\ref{tab:class-agnostic}.
% \jian{why do we need the explanations for the performance drop?}\zzk{Because the AP we reported here is much lower compared to their results of semantic instance segmentation reported in their original paper. If we do not explain why, I am afraid the reviewer may wonder or even think the baseline is fake. }
This phenomenon makes sense for two reasons: (1) Lack of semantic information leads to incorrect matches between prediction masks and targets when training the model; (2) The greedy algorithm used when computing AP generate more wrong matches since they don't have semantic constraints. Such a phenomenon becomes more severe when a lot of recent works~\cite{Jiang20CVPR,Engelmann20CVPR,Chen21ICCV,Liang21CVPR,Han20CVPR, vu2022softgroup} for instance segmentation highly depend on corresponding semantic predictions.
Compared with fully-supervised Mask3D~\cite{schult2022mask3d} on class-agnostic instance segmentation task, our unsupervised performance can achieve over 50\% of its accuracy. FreePoint also surpasses some existing fully-supervised methods as shown in Table~\ref{tab:class-agnostic}. The visualization results are shown in Figure~\ref{fig:visualization}.
\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{visualization.pdf}
\end{center}
\caption{\textbf{Qualitative results on ScanNet v2.} FreePoint shows surprisingly good performance without any annotations.}
\label{fig:visualization}
\end{figure}
\begin{table}
\begin{center}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccc}
\hline
Method & Supervision & mAP & mAP$_{50}$ & mAP$_{25}$\\
\hline
\textit{semantic:} \\
CSC~\cite{Hou21CVPR} & semi & - & 13.2 & - \\
3D-SIS~\cite{hou20193d} & full & - & 18.7 & 35.7 \\
GSPN~\cite{yi2019gspn} & full & 19.3 & 37.8 & 53.4 \\
PointGroup~\cite{jiang2020pointgroup} & full & 34.8 & 56.9 & 71.3 \\
\hline
Method & Supervision & AP & AP$_{50}$ & AP$_{25}$\\
\hline
\textit{class agnostic:} \\
LRGNet~\cite{chen2021lrgnet} & full & 7.5 & 13.3 & 21.1 \\
3D-SIS~\cite{hou20193d} & full & 6.3 & 12.0 & 22.4 \\
GSPN~\cite{yi2019gspn} & full & 8.1 & 15.7 & 28.8 \\
PointGroup~\cite{jiang2020pointgroup} & full &12.7 & 21.5 & 35.1 \\
Mask3D~\cite{schult2022mask3d} & full & 20.9 & 33.4 & 49.3 \\\hline
\textbf{FreePoint (Ours)} & unsup. & 10.7 & 21.4 & 37.8 & \\
% $val\_val$ & unsup. & 8.9 & 16.7 & 28.3 & \\
\hline
\end{tabular}
}
\end{center}
\caption{\textbf{Instance segmentation} on ScanNet v2 val split. We report average precision (AP) with different IoU thresholds. The first block shows original \textit{semantic instance segmentation} results and the second block is corresponding \textit{class-agnostic results} tested by ourselves. All works for comparison in the second block are fully-supervised under their original settings. We only remove their semantic prediction branches and corresponding losses. Our FreePoint is trained in an unsupervised manner. }
\label{tab:class-agnostic}
\end{table}
\paragraph{Fine-tuning on semantic instance segmentation}
\begin{table}
\begin{center}
\begin{tabular}{lcccccc}
\hline
{}&Pre-train & AP & AP$_{50}$ & AP$_{25}$\\
\hline
\multirow{6}{*}{\rotatebox{90}{\emph{10\% masks}}}
& train from scratch & 34.7 & 47.6 & 56.3 \\
&supervised & 36.9 & 50.1 & 55.5\\
&PointContrast~
~\cite{xie2020pointcontrast} & 36.1 & 49.4 & 56.8 \\
&DepthContrast~\cite{zhang2021self} & 36.8 & 49.0 & 57.3\\
&CSC~\cite{Hou21CVPR} & 37.1 & 50.7 & 57.1 \\
&\textbf{FreePoint (Ours)} & 40.5 & 52.3 & 58.3 \\
\hline
\multirow{6}{*}{\rotatebox{90}{\emph{20\% masks}}}
&train from scratch & 44.1 & 54.3 & 61.1 \\
&supervised & 45.7 & 55.2 & 61.4\\
&PointContrast~\cite{xie2020pointcontrast} & 44.4 & 54.8 & 61.7 \\
&DepthContrast~\cite{zhang2021self} &45.2 & 54.9 & 62.4\\
&CSC~\cite{Hou21CVPR} & 46.3 & 56.4 & 61.5 \\
&\textbf{FreePoint (Ours)} & 46.9 & 56.4 & 63.9 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Supervised semantic instance segmentation} with limited instance masks. ``supervised" means fully-supervised pre-training
on ScanNet and fine-tuning on S3DIS.}
\label{tab:limited_masks}
\end{table}
\begin{table}
\begin{center}
\begin{tabular}{cllcccccc}
\hline
{} & Pre-train & AP & AP$_{50}$ & AP$_{25}$\\
\hline
\multirow{6}{*}{\rotatebox{90}{\emph{10\% scenes}}}
&train from scratch & 30.1 & 41.2 & 52.2 \\
&supervised & 32.4 & 41.8 & 52.3\\
&PointContrast~\cite{xie2020pointcontrast} & 31.0 & 42.2 & 53.5 \\
&DepthContrast~\cite{zhang2021self} &32.2 & 41.5 & 53.7\\
&CSC~\cite{Hou21CVPR} & 32.7 & 42.7 & 54.4 \\
&\textbf{FreePoint (Ours)} & 35.5 & 46.3 & 57.1 \\
\hline
\multirow{6}{*}{\rotatebox{90}{\emph{20\% scenes}}}
&train from scratch & 42.1 & 49.5 & 58.3 \\
&supervised & 44.8 & 51.7 & 59.6\\
&PointContrast~\cite{xie2020pointcontrast} & 43.7 & 50.8 & 60.5 \\
&DepthContrast~\cite{zhang2021self} & 44.0 & 51.6 & 62.1\\
&CSC~\cite{Hou21CVPR} & 44.4 & 52.9 & 61.0 \\
&\textbf{FreePoint (Ours)} & 45.6 & 54.1 & 62.2 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Supervised semantic instance segmentation} with limited fully annotated point clouds. ``supervised" means fully-supervised pre-training
on ScanNet and fine-tuning on S3DIS.}
\label{tab:limited_scenes}
\end{table}
Since our work is unsupervised, it can also be seen as a pre-training pretext task. Apart from class-agnostic instance segmentation, we further evaluate our work's performance as an unsupervised pre-training model. As shown in Table~\ref{tab:limited_masks}, FreePoint pre-training outperforms training from scratch by 5.8$\%$ AP and 2.8$\%$ AP and when using 10$\%$ and 20$\%$ training masks respectively. We also compare our work with other pre-training methods~\cite{xie2020pointcontrast, Hou21CVPR,zhang2021self} and find that the improvements are clear.
We also compare the pre-training methods with different amounts of full-scene annotations. As shown in Table~\ref{tab:limited_scenes}, we conduct fine-tuning experiments with only limited scenes available. Our work can still achieve satisfactory results. FreePoint pre-training outperforms training from scratch by 5.4$\%$ AP and 3.5$\%$ AP when using 10$\%$ and 20$\%$ full-scene annotations respectively.
\subsection{Ablation Study}
In this part, we conduct ablation experiments to show the effectiveness of each designed component.
\paragraph{Different feature representations}
We explore results on different kinds of point feature representations. For features generated by various self-supervised pre-training encoders, we compare their performance on generating coarse masks and final instance segmentation results respectively. Then we combine the best performer with traditional features and find the accuracy can be further improved. The comparison between different feature representations is shown in Table~\ref{tab:feature_representation}.
\begin{table}
\begin{center}
\resizebox{\linewidth}{!}{
\begin{tabular}{lccc|ccc}
\hline
Method & AP & AP$_{50}$ & AP$_{25}$ & AP & AP$_{50}$ & AP$_{25}$\\
\hline
Traditional & 3.6 & 6.8 & 19.3 & 9.7 & 19.6 & 30.6 \\
Point-MAE~\cite{pang2022masked} & 3.5 & 6.9 & 18.5 & 9.3 & 18.3 & 28.7\\
PointContrast~\cite{xie2020pointcontrast} & 3.5 & 6.6 & 18.8 & 9.4 & 19.3 & 30.1\\
CSC~\cite{Hou21CVPR} & 3.7 & 6.7 & 19.1 & 9.8 & 18.9 & 30.3 \\
CrossPoint~\cite{afham2022crosspoint}& 3.9 & 7.1 & 20.5 & 10.1 & 20.3 & 32.7 \\
\textbf{FreePoint (Ours)} & 4.2 & 7.8 & 22.4 & 10.7 & 21.4 & 37.8 \\
\hline
\end{tabular}
}
\end{center}
\caption{\textbf{Different feature representation methods }for generating base masks. We report the accuracy of both base masks (left block) and final results (right block). Our strategy has the best performance.}
\label{tab:feature_representation}
\end{table}
\paragraph{Segmenting methods}
Owning relatively good feature representation, there are many existing ways to segment point clouds and generate coarse masks accordingly. We compare methods including Supervoxel~\cite{papon2013voxel}, FreeMasks, a method proposed by ~\cite{wang2022freesolo}, and spectral~\cite{melas2022deep} methods. For each method, we tune parameters to achieve good results as far as we can.
%\jian{how to directly apply the 2D methods to 3D?}\zzk{Maybe we can talk about this part in supplementary material.}
We observe that FreeMasks and spectral methods, which have proven successful in unsupervised object detection or segmentation task in 2D field, fail to transfer to point clouds as shown in Figure~\ref{fig:segmenting_methods}. These two methods have two main defects. Firstly, they can only identify and localize partial objects in a crowded and cluttered 3D scene. Secondly, it is hard for these non-distance-based segmenting methods to distinguish different objects of the same semantic information even if they are far away from each other. The above two defects do not have much impact on some 2D images since they generally contain only one or a few dominant objects. But point cloud scenes are not this case. Point clouds usually have many similar objects in each scene, leading to unsatisfactory results. RAMA's bottom-up mechanism relieves the above problems in essence. For each method, we report the accuracy of coarse masks and final predictions. Results are shown in Table~\ref{tab:segmenting_methods}.
\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{segmenting_methods.pdf}
\end{center}
\caption{\textbf{Comparison with originally for 2D unsupervised instance segmenting methods.} Recent methods~\cite{wang2022freesolo, melas2022deep} for 2D unsupervised instance segmentation fail to deal with crowded and cluttered point cloud scenes. }
\label{fig:segmenting_methods}
\end{figure}
\begin{table}
\begin{center}
\resizebox{\linewidth}{!}{
\begin{tabular}{lccc|ccc}
\hline
Method & AP & AP$_{50}$ & AP$_{25}$ & AP & AP$_{50}$ & AP$_{25}$\\
\hline
Supervoxel~\cite{papon2013voxel} & 2.4 & 3.5 & 15.1 & 3.8 & 6.9 & 18.3\\
FreeMasks\_3D~\cite{wang2022freesolo} & 2.9 & 3.2 & 13.3 & - & - & -\\
Spectral~\cite{melas2022deep} & 2.3 & 4.8 & 14.1 & - & - & -\\
RAMA(default) & 4.1 & 7.9 & 21.0 & 10.7 & 21.4 & 37.8\\
\hline
\end{tabular}
}
\end{center}
\caption{\textbf{Segmenting methods.} RAMA is adopted by default. We report the accuracy of both base masks (left block) and final results (right block). `-' means failing to converge.}
\label{tab:segmenting_methods}
\end{table}
\paragraph{Loss}
To validate the effectiveness of our weakly-supervised loss design, we first evaluate the result of using fully-supervised loss(\textit{i.e.,} Dice loss and BCE loss) alone, discovering that directly adopting such loss leads to unsatisfactory results. We also find that only using our proposed weakly-supervised loss terms is even much worse than only using fully-supervised loss terms. This may be attributed to that terms for weak supervision contain too little information, unable to match low-quality predictions with ground truth at the early training stage. Each loss term is validated in Table~\ref{tab:loss}. They all contribute to the final results.
\begin{table}
\begin{center}
\begin{tabular}{lcccccc}
\hline
Method & AP & AP$_{50}$ & AP$_{25}$ \\
\hline
combination(default) & 10.7 & 21.4 & 37.8 \\
- w/o $\mathcal{L}_{mean}$ & 8.8 & 18.3 & 27.4\\
- w/o $\mathcal{L}_{box}$ & 8.6 & 18.1 & 29.7\\
- w/o $\mathcal{L}_{mean}$ and $\mathcal{L}_{box}$& 8.0 & 14.9 & 27.1 \\
- w/o $\mathcal{L}_{dice}$ and $\mathcal{L}_{BCE}$ & 5.3 & 11.2 & 22.4\\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Loss.} Each loss term contributes to the final results.}
\label{tab:loss}
\end{table}
\paragraph{Training strategy and sensitivity to $\sigma$
}
% \jian{"sensitivity to hyper-parameters" may be a better choice}\zzk{*}
\begin{figure}[t]
\begin{center}
\includegraphics[width=0.98\linewidth]{training-strategy.pdf}
\end{center}
\caption{\textbf{Training strategy and sensitivity to $\sigma$.} We report the accuracy of base masks, predictions after the first step training, and predictions after the second step training with different strategies separately. The large gap between the accuracy of different strategies shows the effectiveness of our training strategy. The red box on the top of each figure shows the robustness when choosing $\sigma$ for base masks and under-segmented masks.}
\label{fig:strategy}
\end{figure}
As mentioned in section~\ref{segmenting}, we can generate coarse masks of different segmenting levels by changing parameters when running RAMA. Coarse masks of the best accuracy, denoted as base masks, are generally over-segmented. Therefore after training with the base masks, we further train the model with relatively under-segmented masks with only a few epochs. In Figure~\ref{fig:strategy} we report AP and AP$_{50}$ to evaluate the following conclusions:
\begin{itemize}
\item Our two-step training strategy is effective. After the first step training, the accuracy is improved by a large margin compared with the original coarse masks. Then the use of under-segmented coarse masks can further improve our results. We find that using \textit{under-segmented} masks directly to train our model, as shown in blue boxes, will make the training fail. As a result, if we further train the model with over-segmented masks, as the green line shows, the accuracy is unsatisfactory too. This phenomenon means that changing the training order of the two masks is inappropriate.
\item
We further evaluate our method's sensitivity to $\sigma$. The red boxes in Figure~\ref{fig:strategy} prove that as long as the $\sigma$ we select for base masks and under-segmented masks fall in 0.80-0.95 and 1.0-1.2 respectively, we can achieve good results. Such selection can be easily made following the visualization of coarse masks as shown in Figure~\ref{fig:masks} when applied to new datasets.
%Since our work is unsupervised, which means we can not use numerical metrics like AP to evaluate the quality of coarse masks. We must make sure that the choice of $\sigma$ is robust enough when our work is applied to new datasets. The red boxes in Figure~\ref{fig:strategy} prove that as long as the $\sigma$ we select for base masks and under-segmented masks fall in 0.80-0.95 and 1.0-1.2 respectively, we can achieve good results. Such selection can be easily made following the visualization of coarse masks as shown in Figure~\ref{fig:visualization}.
\end{itemize}
\begin{figure}
\begin{center}
\includegraphics[width=1.0\linewidth]{masks.pdf}
\end{center}
\caption{\textbf{Pseudo labels of a difficult scene with different $\sigma$.} Following the visualization results, users can easily set their $\sigma_{high}$ and $\sigma_{low}$ when applying our method to new datasets.}
\label{fig:masks}
\end{figure}
We also evaluate the regular self-training strategy adopted by recent works~\cite{wang2022freesolo, wang2023cut} for unsupervised instance segmentation in 2D vision. The results are shown in Table~\ref{tab:strategy} and have a large gap compared with our method, which proves that our design for point cloud datasets is effective.
\begin{table}
\begin{center}
\begin{tabular}{lcccccc}
\hline
iters & AP & AP$_{50}$ & AP$_{25}$\\
\hline
-1 & 4.1 & 7.9 & 21.0\\
0 & 6.2 & 13.3 & 22.5\\
1 & 6.4 & 13.4 & 21.9\\
2 & 6.0 & 12.6 & 22.3\\
\textbf{Ours} & 10.7 & 21.4 & 37.8 & \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Compared with regular self-training strategy.} ‘-1’ refers to base masks. ‘0’ means learning without self-training. }
\label{tab:strategy}
\end{table}
\section{Discussion and Conclusion}
\paragraph{Impacts}
In this work, we propose an effective framework FreePoint for unsupervised point cloud instance segmentation. FreePoint achieves satisfactory results as a pioneer in this underexplored field, which proves this task is worthy of further exploration. In our experiment, we also find that methods proposed in previous 2D unsupervised instance segmentation works fail to be directly adopted by point clouds as shown in Figure~\ref{fig:segmenting_methods}. We hope our work can provide insights for future unsupervised point cloud learning works on tasks like instance segmentation, object detection and panoptic segmentation.
\paragraph{Limitations} Though FreePoint shows surprisingly good results, it is still far from the performance of the fully-supervised counterpart. The lack of semantic labels prevents FreePoint from predicting the categories of instances. Besides, just like previous 2D works, our framework includes a user-defined parameter $\sigma$, which is important to FreePoint’s performance. Though we show that $\sigma$ is robust enough to be easily set with the help of visualization in Figure~\ref{fig:strategy},~\ref{fig:masks}, it still increases workload when adopting FreePoint on new datasets.
\
\
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\clearpage
\newpage
\appendix
\section*{Appendix}
\renewcommand{\thefigure}{S\arabic{figure}}
\setcounter{figure}{0}
\renewcommand{\thetable}{S\arabic{table}}
\setcounter{table}{0}
We provide more information here.
\section{Additional implementation details}
\subsection{Segmenting methods}
For FreeMasks\_3D, as the implementation for 2D images in FreeSOLO~\cite{wang2022freesolo}, we use different features (\textit{i.e.}, coordinates, colors, normals, and deep features) of ${{\bf{V}}_{sampled}}$ to form queries ${\bf{Q}}$. Corresponding features of ${{\bf{V}}_{full}}$ is the set of keys ${\bf{K}}$. For each query in ${\bf{Q}}$, we compute its similarity with every key in ${\bf{K}}$ as what we do for computing ${\bf{A}}$. Then the weighted sum of different features' similarities is our final similarity ${\bf{S}}$. The similarity maps are normalized as soft masks. We then compute their maskness scores and remove redundant masks via NMS as FreeSOLO.
For spectral methods~\cite{melas2022deep}, we construct an affinity matrix of ${{\bf{V}}_{sampled}}$ and threshold the affinities at 0:
\begin{equation}
\begin{aligned}
\mathcal {\bf W_{feat}} = {\bf f}{\bf f^{T}}\odot({\bf f}{\bf f^{T}} > 0)
\end{aligned}
\end{equation}
where $\bf f$ is coordinates, colors, normal vectors, or deep features. The weighted sum of different feature affinity matrices is the final affinity matrix $\bf W$. Given $\bf W$, we compute the eigenvectors of $\bf W$ and decompose it accordingly.
For the above two methods, we can't generate coarse masks of two different segmenting levels like what we do for FreePoint, so we only use them for first-step training. Mask3D fails to converge when training with these pseudo labels.
For Supervoxel~\cite{papon2013voxel}, we adopt the implementation of PCL~\cite{rusu20113d}. We can obtain over-segmented and under-segmented masks by changing voxel size and seed size. They are used following our two-step training strategy like FreePoint.
\subsection{Fine-tuning on S3DIS}
FreePoint can serve as a pre-training pretext for semantic instance segmentation with limited annotations. We pre-train on ScanNet v2 and fine-tune on S3DIS. When fine-tuning on S3DIS, we use Area 5 for validation and other areas for training. For the experiments with limited instance masks, we randomly ignore some target masks during training. For the experiments with limited fully annotated point clouds, we only make use of the annotations of partial scenes. The model is trained for 90k iterations when fine-tuning on S3DIS.
\section{Additional results}
\subsection{Comparison with unsupervised traditional methods}
We report the class-agnostic instance segmentation results of some traditional algorithms (\textit{i.e.}, DBSCAN~\cite{ram2010density}, K-Means~\cite{hartigan1979algorithm}, Birch~\cite{zhang1996birch}), which are also unsupervised. For these algorithms, we adopt the implementation of sklearn. They have a large gap compared with our base masks, not to speak of the final results of FreePoint.
\subsection{Unsupervised class-agnostic instance segmentation on S3DIS}
We also evaluate FreePoint's performance of class-agnostic instance segmentation on S3DIS under the same experimental setting. The results of FreePoint and its fully-supervised counterpart are shown in Table~\ref{tab:s3dis}.
\begin{table}
\begin{center}
\begin{tabular}{lccc}
\hline
Method & AP & AP$_{50}$ & AP$_{25}$ \\
\hline
DBSCAN~\cite{ram2010density} & 3.3 & 3.6 & 13.4 \\
K-Means~\cite{hartigan1979algorithm} & 2.7 & 3.4 & 12.2 \\
Birch~\cite{zhang1996birch} & 2.4 & 7.5 & 12.2 \\
\textbf{base masks(Ours)} & 4.1 & 7.9 & 21.0 \\
\textbf{FreePoint(Ours)} & 10.7 & 21.4 & 37.8 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Traditional methods.} For DBSCAN, we set $eps$ = 0.05. For K-Means and Birch, we set $n\_clusters$ = 10.}
\label{tab:traditional_methods}
\end{table}
\begin{table}
\begin{center}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccc}
\hline
Method & Supervision & AP & AP$_{50}$ & AP$_{25}$\\
\hline
Mask3D~\cite{schult2022mask3d} & full & 19.3 & 30.5 & 49.7 \\
\textbf{FreePoint (Ours)} & unsup. & 11.2 & 21.0 & 35.1 & \\
\hline
\end{tabular}
}
\end{center}
\caption{\textbf{Class-agnostic instance segmentation} on S3DIS Area 5.}
\label{tab:s3dis}
\end{table}
\section{Additional visualizations}
In this section, we provide additional visualizations of FreePoint in Figure~\ref{fig:more_case}.
\begin{figure*}[t!]
\centering
\includegraphics[width=0.90859095\linewidth]
{more_case.pdf}
\vspace{-0.5em}
\caption{\textbf{More results of class-agnostic instance segmentation on ScanNet v2.} FreePoint achieves good results without any annotations. The model does well in localizing and segmenting regular objects with high frequency, like chairs, sofas, and tables. However, for crowded and small objects which are integrated into nearby backgrounds, FreePoint fails to predict correct masks.
}
\label{fig:more_case}
\vspace{-1.0em}
\end{figure*}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{abbas2022rama}
Ahmed Abbas and Paul Swoboda.
\newblock Rama: A rapid multicut algorithm on gpu.
\newblock In {\em CVPR}, pages 8193--8202, 2022.
\bibitem{achanta2012slic}
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and
Sabine S{\"u}sstrunk.
\newblock Slic superpixels compared to state-of-the-art superpixel methods.
\newblock {\em IEEE TPAMI}, 34(11):2274--2282, 2012.
\bibitem{afham2022crosspoint}
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri,
Kanchana Thilakarathna, and Ranga Rodrigo.
\newblock Crosspoint: Self-supervised cross-modal contrastive learning for 3d
point cloud understanding.
\newblock In {\em CVPR}, pages 9902--9912, 2022.
\bibitem{armeni20163d}
Iro Armeni, Ozan Sener, Amir~R Zamir, Helen Jiang, Ioannis Brilakis, Martin
Fischer, and Silvio Savarese.
\newblock 3d semantic parsing of large-scale indoor spaces.
\newblock In {\em CVPR}, pages 1534--1543, 2016.
\bibitem{chang2015shapenet}
Angel~X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang,
Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et~al.
\newblock Shapenet: An information-rich 3d model repository.
\newblock {\em arXiv:1512.03012}, 2015.
\bibitem{chen2021lrgnet}
Jingdao Chen, Zsolt Kira, and Yong~K Cho.
\newblock Lrgnet: Learnable region growing for class-agnostic point cloud
segmentation.
\newblock {\em RA-L}, 6(2):2799--2806, 2021.
\bibitem{chen2021hierarchical}
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang.
\newblock Hierarchical aggregation for 3d instance segmentation.
\newblock In {\em ICCV}, pages 15467--15476, 2021.
\bibitem{Chen21ICCV}
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang.
\newblock Hierarchical aggregation for 3d instance segmentation.
\newblock In {\em ICCV}, pages 15467--15476, 2021.
\bibitem{Cheng21AAAI}
Mingmei Cheng, Le Hui, Jin Xie, and Jian Yang.
\newblock Sspc-net: Semi-supervised semantic 3d point cloud segmentation
network.
\newblock In {\em AAAI}, volume~35, pages 1140--1147, 2021.
\bibitem{chibane2022box2mask}
Julian Chibane, Francis Engelmann, Tuan Anh~Tran, and Gerard Pons-Moll.
\newblock Box2mask: Weakly supervised 3d semantic instance segmentation using
bounding boxes.
\newblock In {\em ECCV}, pages 681--699. Springer, 2022.
\bibitem{cho2021picie}
Jang~Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan.
\newblock Picie: Unsupervised semantic segmentation using invariance and
equivariance in clustering.
\newblock In {\em CVPR}, pages 16794--16804, 2021.
\bibitem{cho2015unsupervised}
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce.
\newblock Unsupervised object discovery and localization in the wild:
Part-based matching with bottom-up region proposals.
\newblock In {\em CVPR}, pages 1201--1210, 2015.
\bibitem{chopra1993partition}
Sunil Chopra and Mendu~R Rao.
\newblock The partition problem.
\newblock {\em Mathematical programming}, 59(1-3):87--115, 1993.
\bibitem{dai2017scannet}
Angela Dai, Angel~X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and
Matthias Nie{\ss}ner.
\newblock Scannet: Richly-annotated 3d reconstructions of indoor scenes.
\newblock In {\em CVPR}, pages 5828--5839, 2017.
\bibitem{elich20193d}
Cathrin Elich, Francis Engelmann, Theodora Kontogianni, and Bastian Leibe.
\newblock 3d bird’s-eye-view instance segmentation.
\newblock In {\em GCPR}, pages 48--61. Springer, 2019.
\bibitem{engelmann20203d}
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias
Nie{\ss}ner.
\newblock 3d-mpa: Multi-proposal aggregation for 3d semantic instance
segmentation.
\newblock In {\em CVPR}, pages 9031--9040, 2020.
\bibitem{Engelmann20CVPR}
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias
Nie{\ss}ner.
\newblock {3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance
Segmentation}.
\newblock In {\em CVPR}, 2020.
\bibitem{han2020occuseg}
Lei Han, Tian Zheng, Lan Xu, and Lu Fang.
\newblock Occuseg: Occupancy-aware 3d instance segmentation.
\newblock In {\em CVPR}, pages 2940--2949, 2020.
\bibitem{Han20CVPR}
Lei Han, Tian Zheng, Lan Xu, and Lu Fang.
\newblock Occuseg: Occupancy-aware 3d instance segmentation.
\newblock In {\em CVPR}, pages 2940--2949, 2020.
\bibitem{hartigan1979algorithm}
John~A Hartigan and Manchek~A Wong.
\newblock Algorithm as 136: A k-means clustering algorithm.
\newblock {\em Journal of the royal statistical society. series c (applied
statistics)}, 28(1):100--108, 1979.
\bibitem{he2021dyco3d}
Tong He, Chunhua Shen, and Anton van~den Hengel.
\newblock Dyco3d: Robust instance segmentation of 3d point clouds through
dynamic convolution.
\newblock In {\em CVPR}, pages 354--363, 2021.
\bibitem{hou20193d}
Ji Hou, Angela Dai, and Matthias Nie{\ss}ner.
\newblock 3d-sis: 3d semantic instance segmentation of rgb-d scans.
\newblock In {\em CVPR}, pages 4421--4430, 2019.
\bibitem{Hou21CVPR}
Ji Hou, Benjamin Graham, Matthias Nie{\ss}ner, and Saining Xie.
\newblock Exploring data-efficient 3d scene understanding with contrastive
scene contexts.
\newblock In {\em CVPR}, pages 15587--15597, 2021.
\bibitem{Jiang2021CVPR}
Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi-Wing Fu, and Jiaya
Jia.
\newblock Guided point contrastive learning for semi-supervised point cloud
semantic segmentation.
\newblock In {\em ICCV}, pages 6423--6432, 2021.
\bibitem{jiang2020pointgroup}
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia.
\newblock Pointgroup: Dual-set point grouping for 3d instance segmentation.
\newblock In {\em CVPR}, pages 4867--4876, 2020.
\bibitem{Jiang20CVPR}
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia.
\newblock Pointgroup: Dual-set point grouping for 3d instance segmentation.
\newblock In {\em CVPR}, pages 4867--4876, 2020.
\bibitem{Ke_2022_CVPR}
Tsung-Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella~X. Yu.
\newblock Unsupervised hierarchical semantic segmentation with multiview
cosegmentation and clustering transformers.
\newblock In {\em CVPR}, pages 2571--2581, June 2022.
\bibitem{lahoud20193d}
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, and Martin~R Oswald.
\newblock 3d instance segmentation via multi-task metric learning.
\newblock In {\em ICCV}, pages 9256--9266, 2019.
\bibitem{liang2021instance}
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia.
\newblock Instance segmentation in 3d scenes using semantic superpoint tree
networks.
\newblock In {\em ICCV}, pages 2783--2792, 2021.
\bibitem{Liang21CVPR}
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia.
\newblock Instance segmentation in 3d scenes using semantic superpoint tree
networks.
\newblock In {\em ICCV}, pages 2783--2792, 2021.
\bibitem{Liu21CVPR}
Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu.
\newblock One thing one click: A self-training approach for weakly supervised
3d semantic segmentation.
\newblock In {\em CVPR}, pages 1726--1736, 2021.
\bibitem{melas2022deep}
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.
\newblock Deep spectral methods: A surprisingly strong baseline for
unsupervised semantic segmentation and localization.
\newblock In {\em CVPR}, pages 8364--8375, 2022.
\bibitem{min2022voxel}
Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai.
\newblock Voxel-mae: Masked autoencoders for pre-training large-scale point
clouds.
\newblock {\em arXiv:2206.09900}, 2022.
\bibitem{mo2019partnet}
Kaichun Mo, Shilin Zhu, Angel~X Chang, Li Yi, Subarna Tripathi, Leonidas~J
Guibas, and Hao Su.
\newblock Partnet: A large-scale benchmark for fine-grained and hierarchical
part-level 3d object understanding.
\newblock In {\em CVPR}, pages 909--918, 2019.
\bibitem{pang2022masked}
Yatian Pang, Wenxiao Wang, Francis~EH Tay, Wei Liu, Yonghong Tian, and Li Yuan.
\newblock Masked autoencoders for point cloud self-supervised learning.
\newblock In {\em ECCV}, pages 604--621. Springer, 2022.
\bibitem{papon2013voxel}
Jeremie Papon, Alexey Abramov, Markus Schoeler, and Florentin Worgotter.
\newblock Voxel cloud connectivity segmentation-supervoxels for point clouds.
\newblock In {\em CVPR}, pages 2027--2034, 2013.
\bibitem{qi2017pointnet++}
Charles~Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas~J Guibas.
\newblock Pointnet++: Deep hierarchical feature learning on point sets in a
metric space.
\newblock {\em NeurIPS}, 30, 2017.
\bibitem{ram2010density}
Anant Ram, Sunita Jalal, Anand~S Jalal, and Manoj Kumar.
\newblock A density based algorithm for discovering density varied clusters in
large spatial databases.
\newblock {\em International Journal of Computer Applications}, 3(6):1--4,
2010.
\bibitem{rusu2010semantic}
Radu~Bogdan Rusu.
\newblock Semantic 3d object maps for everyday manipulation in human living
environments.
\newblock {\em KI-K{\"u}nstliche Intelligenz}, pages 345--348, 2010.
\bibitem{rusu20113d}
Radu~Bogdan Rusu and Steve Cousins.
\newblock 3d is here: Point cloud library (pcl).
\newblock In {\em ICRA}, pages 1--4. IEEE, 2011.
\bibitem{schult2022mask3d}
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and
Bastian Leibe.
\newblock Mask3d for 3d semantic instance segmentation.
\newblock {\em arXiv:2210.03105}, 2022.
\bibitem{shin2022unsupervised}
Gyungin Shin, Samuel Albanie, and Weidi Xie.
\newblock Unsupervised salient object detection with spectral cluster voting.
\newblock In {\em CVPR}, pages 3971--3980, 2022.
\bibitem{simeoni2021localizing}
Oriane Sim{\'e}oni, Gilles Puy, Huy~V Vo, Simon Roburin, Spyros Gidaris, Andrei
Bursuc, Patrick P{\'e}rez, Renaud Marlet, and Jean Ponce.
\newblock Localizing objects with self-supervised transformers and no labels.
\newblock {\em arXiv:2109.14279}, 2021.
\bibitem{song2017semantic}
Shuran Song, Fisher Yu, Andy Zeng, Angel~X Chang, Manolis Savva, and Thomas
Funkhouser.
\newblock Semantic scene completion from a single depth image.
\newblock In {\em CVPR}, pages 1746--1754, 2017.
\bibitem{tian2021boxinst}
Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen.
\newblock Boxinst: High-performance instance segmentation with box annotations.
\newblock In {\em CVPR}, pages 5443--5452, 2021.
\bibitem{van2021unsupervised}
Wouter Van~Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van~Gool.
\newblock Unsupervised semantic segmentation by contrasting object mask
proposals.
\newblock In {\em ICCV}, pages 10052--10062, 2021.
\bibitem{vu2022softgroup}
Thang Vu, Kookhoi Kim, Tung~M Luu, Thanh Nguyen, and Chang~D Yoo.
\newblock Softgroup for 3d instance segmentation on point clouds.
\newblock In {\em CVPR}, pages 2708--2717, 2022.
\bibitem{Wang18CVPR}
Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann.
\newblock Sgpn: Similarity group proposal network for 3d point cloud instance
segmentation.
\newblock In {\em CVPR}, pages 2569--2578, 2018.
\bibitem{wang2023cut}
Xudong Wang, Rohit Girdhar, Stella~X Yu, and Ishan Misra.
\newblock Cut and learn for unsupervised object detection and instance
segmentation.
\newblock {\em arXiv:2301.11320}, 2023.
\bibitem{Wang19CVPR}
Xinlong Wang, Shu Liu, Xiaoyong Shen, Chunhua Shen, and Jiaya Jia.
\newblock Associatively segmenting instances and semantics in point clouds.
\newblock In {\em CVPR}, pages 4096--4105, 2019.
\bibitem{wang2022freesolo}
Xinlong Wang, Zhiding Yu, Shalini De~Mello, Jan Kautz, Anima Anandkumar,
Chunhua Shen, and Jose~M Alvarez.
\newblock Freesolo: Learning to segment objects without annotations.
\newblock In {\em CVPR}, pages 14176--14186, 2022.
\bibitem{wang2022tokencut}
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell~Xu Hu, James~L
Crowley, and Dominique Vaufreydaz.
\newblock Tokencut: Segmenting objects in images and videos with
self-supervised transformer and normalized cut.
\newblock {\em arXiv:2209.00383}, 2022.
\bibitem{wu20153d}
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang,
and Jianxiong Xiao.
\newblock 3d shapenets: A deep representation for volumetric shapes.
\newblock In {\em CVPR}, pages 1912--1920, 2015.
\bibitem{xie2020pointcontrast}
Saining Xie, Jiatao Gu, Demi Guo, Charles~R Qi, Leonidas Guibas, and Or Litany.
\newblock Pointcontrast: Unsupervised pre-training for 3d point cloud
understanding.
\newblock In {\em ECCV}, pages 574--591. Springer, 2020.
\bibitem{Xu20CVPR}
Xun Xu and Gim~Hee Lee.
\newblock Weakly supervised semantic point cloud segmentation: Towards 10x
fewer labels.
\newblock In {\em CVPR}, pages 13706--13715, 2020.
\bibitem{yang2019learning}
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and
Niki Trigoni.
\newblock Learning object bounding boxes for 3d instance segmentation on point
clouds.
\newblock {\em NeurIPS}, 32, 2019.
\bibitem{yi2019gspn}
Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas~J Guibas.
\newblock Gspn: Generative shape proposal network for 3d instance segmentation
in point cloud.
\newblock In {\em CVPR}, pages 3947--3956, 2019.
\bibitem{yu2022point}
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.
\newblock Point-bert: Pre-training 3d point cloud transformers with masked
point modeling.
\newblock In {\em CVPR}, pages 19313--19322, 2022.
\bibitem{zhang2022point}
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao,
and Hongsheng Li.
\newblock Point-m2ae: multi-scale masked autoencoders for hierarchical point
cloud pre-training.
\newblock {\em arXiv:2205.14401}, 2022.
\bibitem{zhang1996birch}
Tian Zhang, Raghu Ramakrishnan, and Miron Livny.
\newblock Birch: an efficient data clustering method for very large databases.
\newblock {\em ACM sigmod record}, 25(2):103--114, 1996.
\bibitem{Zhang21AAAI}
Yachao Zhang, Zonghao Li, Yuan Xie, Yanyun Qu, Cuihua Li, and Tao Mei.
\newblock Weakly supervised semantic segmentation for large-scale point cloud.
\newblock In {\em AAAI}, volume~35, pages 3421--3429, 2021.
\bibitem{Zhang21ICCV}
Yachao Zhang, Yanyun Qu, Yuan Xie, Zonghao Li, Shanshan Zheng, and Cuihua Li.
\newblock Perturbed self-distillation: Weakly supervised large-scale point
cloud semantic segmentation.
\newblock In {\em ICCV}, pages 15520--15528, 2021.
\bibitem{zhang2021self}
Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra.
\newblock Self-supervised pretraining of 3d features on any point-cloud.
\newblock In {\em ICCV}, pages 10252--10263, 2021.
\bibitem{zhou2018open3d}
Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.
\newblock Open3d: A modern library for 3d data processing.
\newblock {\em arXiv:1801.09847}, 2018.
\end{thebibliography}