CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration

Extracted source text · arXiv 2110.14076

\pdfoutput=1
\documentclass{article}

% if you need to pass options to natbib, use, e.g.:
%     \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2021

% ready for submission
\usepackage[nonatbib, final]{neurips_2021}

% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
%     \usepackage[preprint]{neurips_2021}

% to compile a camera-ready version, add the [final] option, e.g.:
%     \usepackage[final]{neurips_2021}

% to avoid loading the natbib package, add option nonatbib:
%    \usepackage[nonatbib]{neurips_2021}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\hypersetup{colorlinks, urlcolor=blue}
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{xcolor}         % colors
\usepackage{graphicx}
\usepackage{float}
\usepackage{subfigure} 
\usepackage{caption}
\usepackage[square,numbers]{natbib}
\usepackage{amsfonts,amssymb}
\usepackage[namelimits]{amsmath} %数学公式
\usepackage{amssymb}             %数学公式
\usepackage{amsfonts}            %数学字体
\usepackage{mathrsfs} 
\usepackage{tabularx}
\usepackage{wrapfig}
\usepackage{verbatim}
\usepackage{dsfont}
\usepackage{tablefootnote}
\usepackage[stable]{footmisc}


\usepackage{color}

\newcommand\todo[1]{\textbf{TODO: }{\color{red}{#1}}}
\renewcommand{\vec}[1]{\mathbf{#1}}

\makeatletter
\newcommand*\bigcdot{\mathpalette\bigcdot@{.5}}
\newcommand*\bigcdot@[2]{\mathbin{\vcenter{\hbox{\scalebox{#2}{$\m@th#1\bullet$}}}}}
\makeatother

\bibliographystyle{unsrt}
\title{CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.

\author{%
  Hao Yu$^{1, 2}$ \quad Fu Li$^{1, 2}$ \quad Mahdi Saleh$^{1}$ \quad Benjamin Busam$^{1}$ \quad Slobodan Ilic$^{1, 3}$ \\
  \\ $^{1}$ Technical University of Munich \quad
  $^{2}$ National University of Defense Technology\\ $^{3}$ Siemens AG, München \\
  \texttt{\{hao.yu, fu.li, m.saleh, b.busam, slobodan.ilic\}@tum.de}
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
}

\begin{document}

\maketitle

\input{abstract}

\input{introduction}

\input{related_work}

\input{methodology}

\input{results}

\input{conclusion}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\input{sections/broader_impact}
%\appendix
%\input{sections/appendix}
%\appendix

\section*{Acknowledgments}
We appreciate the valuable discussion with Zheng Qin and Hao Wu. We would like to thank Dr. Kai Wang for paper revision. Hao Yu is supported by China Scholarship Council (CSC).


%\newpage

\bibliography{main}
\input{appendix}
%\input{sections/checklist}



\end{document}

\begin{abstract}
We study the problem of extracting correspondences between a pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet - \textbf{Co}arse-to-\textbf{Fi}ne \textbf{Net}work which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by a density-adaptive matching module capable to deal with varying point density. Extensive evaluation of CoFiNet on both indoor and outdoor standard benchmarks shows our superiority over existing methods. Especially on 3DLoMatch where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-art approaches by at least 5\% on \textit{Registration Recall}, with at most two-third of their parameters. [\href{https://github.com/haoyu94/Coarse-to-fine-correspondences}{Code}]
\end{abstract}
\section{Introduction}

Correspondence search is a core topic of computer vision and establishing reliable correspondences is a key to success in many fundamental vision tasks, such as tracking, reconstruction, flow estimation, and particularly, point cloud registration. Point cloud registration aims at recovering the transformation between a pair of partially overlapped point clouds. It is a fundamental task in a wide range of real applications, including scene reconstruction, autonomous driving, simultaneous localization and mapping (SLAM), etc. However, due to the unordered and irregular properties of point clouds, extracting reliable correspondences from them has been a challenging task for a long time. From early-stage hand-crafted methods \cite{johnson1999using, tombari2010unique1, tombari2010unique2, rusu2009fast, rusu2008persistent} to recently emerged deep learning-based approaches \cite{zeng20173dmatch, deng2018ppfnet, deng2018ppf, gojcic2019perfect, bai2020d3feat, saleh2020graphite, huang2020predator}, many works contributed to improving the reliability of correspondences.

We can broadly categorize recent deep learning-based point cloud registration methods into three categories. The first \cite{wang2019deep, wang2019prnet, yew2020rpm} follows the idea of ICP~\cite{besl1992method}, where they iteratively find dense correspondences and compute pose estimation. The second~\cite{aoki2019pointnetlk, huang2020feature} includes the correspondence-free methods based on the intuition that the feature distance between two well-aligned point clouds should be small. Such methods encode the whole point cloud as a single feature and iteratively optimize the relative pose between two frames by minimizing the distance of corresponding features. Though achieving reasonable results on synthetic object datasets~\cite{wu20153d}, both of them struggle on large-scale real benchmarks~\cite{geiger2013vision, zeng20173dmatch}, as the first suffers from low correspondence precision and high computational complexity, while the second lacks robustness to noise and partial overlap.

\begin{figure}
\centering 
\includegraphics[width=1.\textwidth]{figures/backbone_v6.jpg} 
\caption{
\textbf{Left}: Overview of CoFiNet. From top to bottom: 1) Dense points are down-sampled to uniformly distributed nodes, while associated features are jointly learned. 2) Correspondence Proposal Block (\textbf{Top Right}): Features are strengthened and used to calculate the similarity matrix. Coarse node correspondences are then proposed from the confidence matrix. 3) Strengthened features are decoded to dense descriptors associated with each input point. 4) Correspondence Refinement Block (\textbf{Bottom Right}): Coarse node proposals are first expanded to patches via grouping. Patch correspondences are then refined to point level by our proposed density-adaptive matching module, whose details can be found in Fig.~\ref{fig.refinement}.}
\label{fig.overview} 
\vspace{-0.55cm}
\end{figure}
Differently, the last category of methods \cite{zeng20173dmatch, deng2018ppfnet, deng2018ppf, gojcic2019perfect, bai2020d3feat, saleh2020graphite, huang2020predator} tackles point cloud registration in a two-stage manner. They firstly learn local descriptors of down-sampled sparse points (nodes) for matching, and afterward use robust pose estimators, e.g., RANSAC~\cite{fischler1981random}, for recovering the relative transformation. Their two-stage strategy makes them achieve state-of-the-art performance on large-scale real benchmarks~\cite{geiger2013vision, zeng20173dmatch}. Uniform sampling~\cite{deng2018ppfnet, deng2018ppf} and keypoint detection~\cite{li2019usip, bai2020d3feat, saleh2020graphite, huang2020predator} are two common ways to introduce sparsity. Compared to uniform sampling that samples points randomly, keypoint detection estimates the saliency of points and samples points with strong geometry features, which significantly reduces the ambiguity of matching. However, the sparsity by nature challenges repeatability, i.e., sub-sampling increases the risk where a certain point loses its corresponding point in the other frame, which constrains the performance of detection-based methods~\cite{li2019usip, bai2020d3feat, saleh2020graphite, huang2020predator}.

Recently, a coarse-to-fine mechanism is leveraged by our 2D counterparts~\cite{li2020dual, zhou2020patch2pix, sun2021loftr} to avoid direct keypoint detection, which shows superiority over the state-of-the-art detection-based method~\cite{sarlin2020superglue}. However, in 3D point cloud matching, where keypoint detectors usually perform worse, existing deep learning-based methods do not yet exploit such a coarse-to-fine strategy. To fill the gap, we focus on leveraging the coarse-to-fine mechanism to eliminate the side effects of detecting sparse keypoints.

Nevertheless, designing such a coarse-to-fine pipeline in point cloud matching is non-trivial, mainly due to the inherent unordered and irregular nature of point clouds. To this end, we propose a weighting scheme for coarse node matching and a density-adaptive matching module for correspondence refinement, which enables CoFiNet to extract coarse-to-fine correspondences from point clouds. More specifically, on a coarse scale, the weighting scheme proportional to local overlap ratios guides the model to propose correspondences of nodes whose vicinity areas share more overlap, which effectively squeezes the search space of the consecutive refinement. On a finer scale, the density-adaptive matching module refines coarse correspondence proposals to point level by solving a differentiable optimal transport problem \cite{sarlin2020superglue} with awareness to varying point density, which shows more robustness on irregular points. An overview of our proposed method can be found in Fig.~\ref{fig.overview}. Our main contributions are summarized as follows:
\vspace{-0.2cm}
\begin{itemize}
%\item[$\bullet$] \ben{I would emphasize the main contributions more which are 1. density adaptive strategy 2. refinement block. The main contribution to my understanding is to treat the problem in a coarse-to-fine manner (not the architecture which only is a vessel for it) - and would leave out the comparison to keypoint extraction here. Maybe we can replace the first point with something like this: "Treating point cloud registration as a coarse-to-fine correspondence problem within a learning framework. Our novel architecture thereby extracts reliable coarse correspondences for unordered and irregular point clouds which are consecutively refined."}
\item[$\bullet$] A detection-free learning framework that treats point cloud registration as a coarse-to-fine correspondence problem, where point correspondences are consecutively refined from coarse proposals that are extracted from unordered and irregular point clouds.
\item[$\bullet$] A weighting scheme that, on a coarse scale, guides our model to learn to match uniformly down-sampled nodes whose vicinity areas share more overlap, which significantly shrinks the search space for the refinement.
\item[$\bullet$] A differentiable density-adaptive matching module that refines coarse correspondences to point level based on solving an optimal transport problem with awareness to point density, which is more robust to the varying point density.
\end{itemize}
\vspace{-0.2cm}

To the best of our knowledge, we are the first deep learning-based work that incorporates a coarse-to-fine mechanism in correspondence search for point cloud registration. Extensive experiments are conducted on both indoor and outdoor benchmarks to show our superiority. Notably, CoFiNet surpasses the state-of-the-art with much fewer parameters. Compared to \cite{huang2020predator}, we only use around two-third and one-fourth of parameters on indoor and outdoor benchmarks, respectively.
\section{Related Work}
\paragraph{Learned local descriptors.}
%Inspired by their 2D counterparts, early stage local descriptors %\cite{tombari2010unique1, tombari2010unique2, rusu2009fast, %rusu2008persistent} are usually designed to extract rotation-invariant %hand-crafted features from local geometry. Though they can naturally %generalize to different data and have been successfully applied in many %downstream tasks, their inherent shortcomings of lacking robustness %against noises and occlusion constrain their further development. With %the recent bloom of deep learning, more and more attention has shifted to %learning-based approaches.
Early networks proposed to learn local descriptors for 3D correspondence search mainly take uniformly distributed local patches as input. As a pioneer, Zeng et al.~\cite{zeng20173dmatch} propose the 3DMatch benchmark, on which they exploit a Siamese network~\cite{bromley1993signature} that consumes voxel grids of TDFs (Truncated Distance Fields) to match local patches. PPFNet~\cite{deng2018ppfnet} directly consumes raw points augmented with point-pair features (PPF) by leveraging PointNet~\cite{qi2017pointnet} as its encoder. PPF-FoldNet~\cite{deng2018ppf} leverages only PPF, which is naturally rotation-invariant, as its input and further incorporates a FoldingNet~\cite{yang2018foldingnet} architecture to enable the unsupervised training of rotation-invariant descriptors. Gojcic et al.~\cite{gojcic2019perfect} propose a network to consume the smoothed density value (SDV) representation aligned to the local reference frame (LRF) to eliminate the rotation-variance of learned descriptors. To extract better geometrical features, Graphite \cite{saleh2020graphite} utilizes graph neural networks for local patch description. SpinNet~\cite{ao2020spinnet} utilizes LRF for patch alignment and 3D cylindrical convolution layers for feature extraction, achieving the best generalization ability to unseen datasets. However, patch-based methods usually suffer from low computational efficiency, as typically shared activations of adjacent patches are not reused. To address this, FCGF~\cite{choy2019fully} makes the first attempt by using sparse convolutions~\cite{choy20194d} to compute dense descriptors of the whole point cloud in a single pass, which leads to 600x speed-up while still being able to achieve comparable performance to patch-based methods.
\vspace{-0.3cm}
\paragraph{Learned 3D keypoint detectors.}
USIP~\cite{li2019usip} learns to regress the position of the most salient point in each local patch in a self-supervised manner. However, it suffers from degenerated cases when the number of desired keypoints is relatively small. D3Feat~\cite{bai2020d3feat} exploits a fully convolutional encoder-decoder architecture for joint dense detection and description. However, it does not consider overlap relationships and shows low robustness on low-overlap scenarios. In addition to jointly estimating salient scores and learning local descriptors, PREDATOR~\cite{huang2020predator} also predicts dense overlap scores that indicate the confidence whether points are on the overlap regions. Keypoints will be sampled under the condition of both saliency and overlap scores. Though it surpasses existing methods by a large margin on both 3DMatch\cite{zeng20173dmatch} and 3DLoMatch\cite{huang2020predator}, the precision of estimated scores and the repeatability of sampled keypoints constrain its performance.
\vspace{-0.3cm}
\paragraph{Coarse-to-fine correspondences.}
As witnessed in 2D image matching, many recent works~\cite{li2020dual, zhou2020patch2pix, sun2021loftr} leverage a coarse-to-fine mechanism to eliminate the inherent repeatability problem in keypoint detection and thus boost the performance. DRC-Net~\cite{li2020dual} utilizes 4D cost volumes to enumerate all the possible matches and establishes pixel correspondences in a coarse-to-fine manner. Concurrently with DRC-Net, Patch2Pix~\cite{zhou2020patch2pix} first establishes patch correspondences and then regresses pixel correspondences according to matched patches. In a similar coarse-to-fine manner with Patch2Pixel, LoFTR~\cite{sun2021loftr} leverages Transformers~\cite{vaswani2017attention}, together with an optimal transport matching layer~\cite{sarlin2020superglue}, to match mutual-nearest patches on the coarse level, and then refines the corresponding pixel of the patch center on the finer level.
\vspace{-0.2cm}

\section{Methodology}
\vspace{-0.2cm}
\paragraph{Problem statement.}
Given a pair of unordered point sets $\mathbf{X}$ with $n$ points ${x_i} \in \mathbb{R}^{3}$ and $\mathbf{Y}$ with $m$ points ${y_j} \in \mathbb{R}^{3}$, we aim at recovering the rigid transformation $\overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}\in SE(3)$ between them. For simpler notation, we define their coordinate matrices as $\mathbf{P}_\mathbf{X} \in \mathbb{R}^{n\times 3}$ and $\mathbf{P}_\mathbf{Y} \in \mathbb{R}^{m\times 3}$, respectively. We follow the path of extracting correspondences first and then estimating the relative pose, where we mainly focus on the former. For this purpose, we propose CoFiNet that takes a pair of point clouds as input and outputs point correspondences, which can be leveraged to estimate the rigid transformation by RANSAC~\cite{fischler1981random}.

\subsection{Coarse-scale Matching}
\paragraph{Point encoding.} On the coarse level, our target is matching uniformly down-sampled nodes whose vicinity areas share more overlap. To achieve this goal, we first adopt shared KPConv~\cite{thomas2019kpconv} encoders to down-sample raw points to uniformly distributed nodes $\mathbf{P}_\textbf{X}^{\prime} \in \mathbb{R}^{n^{\prime}\times3}$ and $\mathbf{P}^{\prime}_\mathbf{Y} \in \mathbb{R}^{m^{\prime}\times3}$, while jointly learning their associated features $\mathbf{F}^{\prime}_\mathbf{X} \in \mathbb{R}^{n^{\prime}\times b}$ and $\mathbf{F}^{\prime}_\mathbf{Y} \in \mathbb{R}^{m^{\prime}\times b}$. Demonstration of down-sampled nodes can be found in \textbf{1)} of Fig.~\ref{fig.overview}. Please refer to Appendix for more details of network architecture.%For matching overlapping patches, we propose Correspondence Proposal Block (CPB), which consumes a pair of down-sampled nodes with their associated features and outputs strengthened features together with estimated patch correspondences. Illustration can be found in Fig.\ref{fig.overview}. 
\vspace{-0.4cm}
\paragraph{Attentional feature aggregation.} As illustrated in Fig.~\ref{fig.overview}, the Correspondence Proposal Block (CPB) takes as input the down-sampled nodes and associated features. In CPB (\textbf{a}), following~\cite{sarlin2020superglue, huang2020predator}, the attention~\cite{vaswani2017attention} mechanism is leveraged to incorporate more global contexts to the learned features. Following~\cite{huang2020predator}, we adopt a sequence of self-, cross- and self-attention modules, which interactively aggregates global contexts across nodes from the same and the other frame in a pair of point clouds. Below we briefly introduce the cross-attention module. Given ($\mathbf{F}^{\prime}_\mathbf{X}$, $\mathbf{F}^{\prime}_\mathbf{Y}$), akin to database retrieval, the former is linearly projected by a learnable matrix $\mathbf{W}_\mathbf{Q}\in \mathbb{R}^{b\times b}$ to \textit{query} $\mathbf{Q}$ as $\mathbf{Q}=\mathbf{F}^{\prime}_\mathbf{X} \mathbf{W}_\mathbf{Q}$, while the latter is similarly projected to \textit{key} $\mathbf{K}$ and \textit{value} $\mathbf{V}$ by learnable matrices $\mathbf{W}_\mathbf{K}$ and $\mathbf{W}_\mathbf{V}$, respectively. The attention matrix $\mathbf{A}$ is represented as $\mathbf{A}=\mathbf{Q}\mathbf{K}^{T}/\sqrt{b}$, whose rows are then normalized by \textit{softmax}. The message $\mathbf{M}$ flows from $\mathbf{F}^{\prime}_\mathbf{Y}$ to $\mathbf{F}^{\prime}_\mathbf{X}$ is formulated as $\mathbf{M}=\mathbf{A}\cdot \mathbf{V}$, which represents the linear combination of \textit{values} weighted by the attention matrix. In the cross-attention module, contexts are aggregated bidirectionally, from $\mathbf{F}^{\prime}_\mathbf{X}$ to $\mathbf{F}^{\prime}_\mathbf{Y}$ and from $\mathbf{F}^{\prime}_\mathbf{Y}$ to $\mathbf{F}^{\prime}_\mathbf{X}$. For computational efficiency, we replace the graph-based module~\cite{wang2019dynamic} leveraged in~\cite{huang2020predator} with the self-attention module in \cite{sarlin2020superglue}, which has the same architecture as the cross-attention module but takes the features from the same point cloud, e.g., ($\mathbf{F}^{\prime}_\mathbf{X}$, $\mathbf{F}^{\prime}_\mathbf{X}$), as input.
\vspace{-0.4cm}
\paragraph{Correspondence proposal.}
As shown in CPB (\textbf{b}) of Fig.~\ref{fig.overview}, we leverage strengthened features $\tilde{\mathbf{F}}^{\prime}_\mathbf{X}$ and $\tilde{\mathbf{F}}^{\prime}_\mathbf{Y}$ to calculate the similarity matrix. Down-sampled nodes whose vicinity areas share enough overlap are matched. However, there can be two cases where a node fails to match: 1) The major portion of its vicinity areas is occluded in the other frame. 2) Though most of its vicinity areas are visible in the other frame, there does not exist a node whose vicinity areas share sufficient overlap with its. Thus, for the similarity matrix, we expand it with a slack row and column with $m^{\prime}$ and $n^{\prime}$ slack entries, respectively~\cite{busam2015stereo}. So that nodes fail to match other nodes could match their corresponding slack entries, i.e., having maximum scores there. Similar to \cite{sarlin2020superglue}, we compute the similarity matrix using an inner product, which can be presented as:

\vspace{-0.5cm}
\begin{equation}
\mathbf{S}^{\prime} = 
\begin{bmatrix}
\tilde{\mathbf{F}}^{\prime}_\mathbf{X} {\tilde{\mathbf{F}}^{\prime \; ^T}}_\mathbf{Y} & \mathbf{z} \\
\\
\mathbf{z}^T & z
\end{bmatrix}, \qquad \mathbf{S}^{\prime} \in \mathbb{R}^{(n^{\prime} + 1)\times (m^{\prime} + 1)},
\label{eq1}
\end{equation}
\vspace{-0.5cm}

where all slack entries are set to the same learnable parameter $z$. On $\mathbf{S}^{\prime}$ we run the Sinkhorn Algorithm~\cite{sinkhorn1967concerning, cuturi2013sinkhorn, peyre2019computational}, seeking an optimal solution for the optimal transport problem. After that, each entry ($i^{\prime}$, $j^{\prime}$) in the obtained matrix represents the matching confidence between the node $i^{\prime}$ and node $j^{\prime}$ from $\mathbf{P}^{\prime}_{\mathbf{X}}$ and $\mathbf{P}^{\prime}_{\mathbf{Y}}$, respectively. %dual-normalized, which means the sum of each row and column of $\mathbf{S}^{\prime}$ would equal to 1.
To guarantee a higher recall, we adopt a threshold ${\tau}_c$ for likely correspondences whose confidence scores are above ${\tau}_c$. We define the obtained coarse node correspondence set as $\mathbf{C}^{\prime}=\{(\mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime}), \mathbf{P}^{\prime}_{\mathbf{Y}}(j^{\prime}))\}$, with $\left|\mathbf{C^{\prime}} \right|=c^{\prime}$, where $\left|\cdot \right|$ denotes the set cardinality. Furthermore, we set the other threshold ${\tau}_m$ to guarantee that $c^{\prime} \geq {\tau}_m$. When $c^{\prime} < {\tau}_m$, we gradually decrease ${\tau}_c$ to extract more coarse node correspondences.
\vspace{-0.4cm}


\subsection{Point-level Refinement}
\paragraph{Node decoding.} On the finer scale, we aim at refining coarse correspondences from the preceding stage to point level. Those refined correspondences can then be used for point cloud registration. We first stack several KPConv~\cite{thomas2019kpconv} layers to recover the raw points, $\mathbf{P}_\mathbf{X}$ and $\mathbf{P}_\mathbf{Y}$, while jointly learning associated dense descriptors, $\mathbf{F}_\mathbf{X} \in \mathbb{R}^{n\times c}$ and $\mathbf{F}_\mathbf{Y} \in \mathbb{R}^{m \times c}$. We thereby assign to each point $p$ an associated feature $p \leftrightarrow f \in \mathbb{R}^{c}$, as illustrated in \textbf{3)} of Fig.~\ref{fig.overview}.  Then, as demonstrated in \textbf{4)} of Fig.~\ref{fig.overview}, obtained dense descriptors, together with raw points and coarse correspondences are fed into the Correspondence Refinement Block (CRB), where coarse proposals are expanded to patches that are then refined to point correspondences.
\begin{figure}[H] 
\centering 
\includegraphics[width=.9\textwidth]{figures/refinement_v3.jpg}
\caption{Illustration of our proposed density-adaptive matching module. The input is a pair of patches truncated by $k$. a) We use the context aggregation part from the Correspondence Proposal Block to condition on both patches and to strengthen features. b.1) The similarity matrix is computed. Slack entries are initialized with $0$ and muted entries corresponding to repeatedly sampled points are set to $-\infty$ . b.2) N iterations of the Sinkhorn Algorithm are performed. We drop the slack row and column for row and column normalization, respectively. b.3) We obtain the confidence matrix, whose first $k$ rows and $k$ columns are row- and column-normalized, respectively. For correspondences, we pick the maximum confidence score in every row and column to guarantee a higher precision.}
\label{fig.refinement}
\vspace{-0.6cm}
\end{figure}

%Then, we design Correspondence Refinement Block (CRB), which takes coarse correspondence set $\mathbf{C}^{\prime}$, together with $(\mathbf{P_X}, \mathbf{F_X})$ and $(\mathbf{P_Y}, \mathbf{F_Y})$ as input, to propose point correspondences. First, raw points and dense descriptors are associated to different nodes\slo{groups} to formulate finer representation of coarse correspondences. Then, coarse correspondences in finer representaion are refined by a local CPB. The correspondence proposal module of local CPB is replaced by our proposed density-adaptive matching module, which can deal with the varying point density. Demonstration can be found in Fig. \ref{fig.overview}. 
\paragraph{Point-to-node grouping.}
For refinement, we need to expand nodes in coarse correspondences to patches consisting of groups of points and associated descriptors. Accordingly, we use a point-to-node grouping strategy~\cite{kohonen1990self,li2018so, li2019usip} to assign points to their nearest nodes in geometry space. If a point has multiple nearest nodes, a random one will be picked. We demonstrate this procedure in CRB (\textbf{a}) of Fig.~\ref{fig.overview}. The advantages of point-to-node over k-nearest neighbor search or radius-based ball query are two-fold: 1) Every point will be assigned to exactly one node, while some points could be left out in other strategies. 2) It can automatically adapt to various scales~\cite{li2019usip}. After grouping, nodes with their associated points and descriptors form patches, upon which we can extract point correspondences. For a certain node $\mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime})$, its associated point set $\mathbf{G}^{\mathbf{P}}_{i^{\prime}}$ and feature set $\mathbf{G}^{\mathbf{F}}_{i^{\prime}}$ can be denoted as:

\vspace{-0.2cm}
\begin{equation}
\left\{
\begin{array}{l}
%\mathbf{G}(\mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime}) \big|\mathbf{P_X}) = \{\mathbf{P}_\mathbf{X}(i) \big| \lVert\mathbf{P}_{\mathbf{X}}(i)  - \mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime})\rVert_{2} \leq \lVert\mathbf{P}_{\mathbf{X}}(i)  - \mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime \prime})\rVert_{2}\ \forall i^{\prime \prime} \in \left[1, n^{\prime}\right]\},\\
%\\
%\mathbf{G}(\mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime}) \big|\mathbf{P_X}, \mathbf{F_X} ) = \{\mathbf{F}_\mathbf{X}(i) \big| \mathbf{P}_\mathbf{X}(i) \in  \mathbf{G}(  \mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime}) \big|\mathbf{P_X}) \},
\mathbf{G}^{\mathbf{P}}_{i^{\prime}} = \{p \in \mathbf{P}_\mathbf{X} \big| \lVert p - \mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime}) \rVert \leq \lVert p - \mathbf{P}^{\prime}_{\mathbf{X}}(j^{\prime})\rVert,  \forall j^{\prime} \neq i^{\prime} \},\\
\\
\mathbf{G}^{\mathbf{F}}_{i^{\prime}} = \{f \in \mathbf{F}_\mathbf{X} \big| f \leftrightarrow p\ \text{with}\ p \in \mathbf{G}^{\mathbf{P}}_{i^{\prime}} \},
\end{array}
\right.
\label{eq3}
\end{equation}
\vspace{-0.2cm}

where $\lVert\cdot \rVert = \lVert\cdot \rVert_{2}$ represents the Euclidean norm.
%\ben{For simplification we will use the notation $\mathbf{G}^{\mathbf{P}}_{i^{\prime}}$ and $\mathbf{G}^{\mathbf{F}}_{i^{\prime}}$ for the two sets.}
In the point-to-node grouping, we expand the coarse node correspondence set $\mathbf{C}^{\prime}$ to its corresponding patch correspondence set, both in geometry space $\mathbf{C}_{\mathbf{P}} = \{(\mathbf{G}^{\mathbf{P}}_{i^{\prime}}, \mathbf{G}^{\mathbf{P}}_{j^{\prime}})\}$ and feature space $\mathbf{C}_{\mathbf{F}} = \{(\mathbf{G}^{\mathbf{F}}_{i^{\prime}}, \mathbf{G}^{\mathbf{F}}_{j^{\prime}})\}$.



\paragraph{Density-adaptive matching.}
Extracting point correspondences from a pair of overlapped patches is in some way analogous to matching two smaller scale point clouds from a local perspective. Thus, directly leveraging the CPB in Fig.~\ref{fig.overview} with input $(\mathbf{G}^{\mathbf{P}}_{i^{\prime}}, \mathbf{G}^{\mathbf{F}}_{i^{\prime}})$ and $(\mathbf{G}^{\mathbf{P}}_{j^{\prime}}, \mathbf{G}^{\mathbf{F}}_{j^{\prime}})$ could theoretically tackle the problem. However, simply utilizing CPB to extract point correspondences would lead to a bias towards slack rows and columns, i.e., the model learns to predict more points as occluded. Reasons for this are two-fold: 1) For computational efficiency, similar to radius-based ball query, in a point-to-node grouping, we need to truncate the number of points to a unified number $k$ for every patch. If a patch contains less than $k$ points, like in \cite{qi2017pointnet++}, a fixed point or randomly sampled points will be repeated as a supplement. 2) On a coarse level, our model learns to propose corresponding nodes with overlapped vicinity areas. However, after expansion, proposed patches can be supplemented by some occluded points, which introduces biases in the training of refinement. To address the issue, we propose a density-adaptive matching module that refines coarse correspondences to point level by solving an optimal transport problem with awareness to point density. We denote the truncated patches as $\widetilde{\mathbf{G}}$ and demonstrate our proposed density-adaptive matching module in Fig.~\ref{fig.refinement}. Notably, both during and after normalization, the exponent projection of any muted entries is always equal to 0, which eliminates the side effects caused by the repeated sampling of points. The final point correspondence set $\widetilde{\mathbf{C}}$ is represented as the union of all the obtained correspondence sets $\widetilde{\mathbf{C}}^{(l)}$.  $\widetilde{\mathbf{C}}$ can be directly leveraged by RANSAC\cite{fischler1981random} for registration.


\subsection{Loss Functions}
Our total loss $\mathcal{L} = \mathcal{L}_c + \lambda \mathcal{L}_f$ is calculated as the weighted sum of the coarse-scale $\mathcal{L}_c$ and the fine-scale $\mathcal{L}_f$, where $\lambda$ is used to balance the terms. We detail the individual parts hereafter.
\vspace{-0.4cm}
\paragraph{Coarse scale.}
On the coarse scale, we leverage a weighting scheme proportional to the overlap ratios over patches as coarse supervision.
Given a pair of down-sampled nodes $\mathbf{P}^{\prime}_{\mathbf{X}}(i^{\prime})$ and $\mathbf{P}^{\prime}_{\mathbf{Y}}(j^{\prime})$, with their expanded patch representation in geometry space,  $\mathbf{G}^{\mathbf{P}}_{i^{\prime}}$ and $ \mathbf{G}^{\mathbf{P}}_{j^{\prime}}$, we can compute the ratio of points in $\mathbf{G}^{\mathbf{P}}_{i^{\prime}}$ that are visible in point cloud $\mathbf{P_Y}$ as:

\begin{equation}
r(i^{\prime}) = \frac{|\{\mathbf{p}\in \mathbf{G}^{\mathbf{P}}_{i^{\prime}} | \exists \mathbf{q}\in \mathbf{P}_\mathbf{Y}\;\text{s.t.}\;\lVert \overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}(\mathbf{p}) -  \mathbf{q}\rVert < \tau_{p}  \}|}{|\mathbf{G}^{\mathbf{P}}_{i^{\prime}}|},
\label{eq.ratio1}
\end{equation}
where $\tau_p$ is the distance threshold. Similarly, we can calculate the ratio of points in $\mathbf{G}^{\mathbf{P}}_{i^{\prime}}$ that have correspondences in $\mathbf{G}^{\mathbf{P}}_{j^{\prime}}$ as:

\begin{equation}
r(i^{\prime}, j^{\prime}) = \frac{|\{\mathbf{p}\in \mathbf{G}^{\mathbf{P}}_{i^{\prime}} | \exists \mathbf{q}\in \mathbf{G}^{\mathbf{P}}_{j^{\prime}}\;\text{s.t.}\;\lVert \overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}(\mathbf{p}) -  \mathbf{q}\rVert < \tau_{p}  \}|}{|\mathbf{G}^{\mathbf{P}}_{i^{\prime}}|}.
\label{eq.ratio2}
\end{equation}
Based on Eq.~\ref{eq.ratio1} and Eq.~\ref{eq.ratio2}, we define the weight matrix $\mathbf{W}^{\prime} \in \mathbb{R}^{\left(n^{\prime} + 1\right)\times \left(m^{\prime} + 1\right)}$ as:


\begin{equation}
\mathbf{W}^{\prime}(i^{\prime}, j^{\prime}) = \left\{
\begin{array}{ll}
min(r(i^{\prime}, j^{\prime}), r(j^{\prime}, i^{\prime})), & i^{\prime} \leq n^{\prime} \wedge j^{\prime} \leq m^{\prime},\\
1 - r(i^{\prime}), & i^{\prime} \leq n^{\prime} \wedge j^{\prime}=m^{\prime} + 1,\\
1 - r(j^{\prime}), & i^{\prime}=n^{\prime} + 1 \wedge j^{\prime} \leq m^{\prime},\\
0, &\text{otherwise}.
\end{array}
\right.
\label{eq.detailedW}
\end{equation}
Finally, we define the coarse scale loss as:

\vspace{-0.5cm}
\begin{equation}
\mathcal{L}_{c} = \frac{-\sum_{i^{\prime}, j^{\prime}}\mathbf{W}^{\prime}\left({i^{\prime}, j^{\prime}}\right) \log(\mathbf{S^{\prime}}\left({i^{\prime}, j^{\prime}}\right))}{\sum_{i^{\prime}, j^{\prime}}\mathbf{W}^{\prime}\left({i^{\prime}, j^{\prime}}\right)}.
\label{eq.coarseloss}
\end{equation}
\vspace{-0.7cm}

\paragraph{Finer scale.}

On the finer point level, for the $l^{th}$ truncated patch correspondence $(\widetilde{\mathbf{G}}^{\mathbf{P}}_{i^{\prime}},  \widetilde{\mathbf{G}}^{\mathbf{P}}_{j^{\prime}})$ s.t. $(\mathbf{G}^{\mathbf{P}}_{i^{\prime}},  \mathbf{G}^{\mathbf{P}}_{j^{\prime}} )\in \mathbf{C}_{\mathbf{P}}$, we define the binary matrix $\widetilde{\mathbf{B}}^{\left(l\right)}\in \mathbb{R}^{(k + 1)\times (k + 1)}$ as:

\begin{equation}
\widetilde{\mathbf{B}}^{\left(l\right)}(i, j) = \left\{
\begin{array}{ll} 
1, &\lVert\overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}(\widetilde{\mathbf{G}}^{\mathbf{P}}_{i^{\prime}}(i)) -\widetilde{\mathbf{G}}^{\mathbf{P}}_{j^{\prime}}(j) \rVert < \tau_p,\\
0, & \text{otherwise},

\end{array}
\right.
\qquad \forall i, \forall j \in \left[1, k\right],
\label{eq.fine1}
\end{equation}

and
\begin{equation}
\begin{aligned}
\widetilde{\mathbf{B}}^{(l)}(i, k + 1) &= \max(0, 1 - \sum\limits_{j=1}^{k}\widetilde{\mathbf{B}}^{(l)}(i, j)), \qquad \forall i \in \left[1, k\right],\\
\widetilde{\mathbf{B}}^{(l)}(k + 1, j) &= \max(0, 1 - \sum\limits_{i=1}^{k}\widetilde{\mathbf{B}}^{(l)}(i, j)),
\qquad \forall j \in \left[1, k\right].
\end{aligned}
\end{equation}

Additionally, we further set the rows and columns of $\widetilde{\mathbf{B}}^{\left(l\right)}$ which correspond to repeatedly sampled points to 0 to eliminate their side effects during training. $\widetilde{\mathbf{B}}^{(l)}(k + 1, k + 1)$ is also set to 0.
Therefore, by defining the confidence matrix in \textbf{b.3} of Fig.~\ref{fig.refinement} as  $\widetilde{\mathbf{S}}^{\left(l\right)}$, the loss function on the finer scale reads as:

\vspace{-0.5cm}
\begin{equation}
\mathcal{L}_f = \frac{-\sum_{l, i, j}\widetilde{\mathbf{B}}^{(l)}\left({i, j}\right) \log(\mathbf{\widetilde{S}}^{(l)}\left({i, j}\right))}{ \sum_{l, i, j}\widetilde{\mathbf{B}}^{(l)}\left({i, j}\right)},
\label{eq15}
\end{equation}
\vspace{-0.5cm}

where we define $0\cdot \log(0)=0$.

\section{Results}
We evaluate our model on three challenging public benchmarks, including both indoor\footnote{As PREDATOR fixed a \href{https://github.com/overlappredator/OverlapPredator/issues/15}{bug} after our submission, we update their latest results. We also update ours according to the rebuttal.} and outdoor scenarios. %Qualitative results can be found in Fig.~\ref{fig.quality}.
Following~\cite{huang2020predator}, for indoor scenes, we evaluate our model on both 3DMatch~\cite{zeng20173dmatch}, where point cloud pairs share > 30\% overlap, and 3DLoMatch~\cite{huang2020predator}, where point cloud pairs have 10\% \textasciitilde 30\% overlap. In line with existing works~\cite{bai2020d3feat, huang2020predator}, we evaluate for outdoor scenes on odometryKITTI~\cite{geiger2013vision}. Please refer to Appendix for more details of implementation and datasets.

\begin{figure}[H] %H为当前位置，!htb为忽略美学标准，htbp为浮动图形
\centering %图片居中
\includegraphics[width=0.98\textwidth]{figures/correspondence.jpg} 
\caption{Qualitative results on \textit{Inlier Ratio}. We compare our point correspondences (the last column) with our coarse correspondences (the third column) and correspondences from PREDATOR (the second column) on a hard case from 3DLoMatch. The first column provides the ground truth alignment, which shows that overlap is very limited. The significantly larger inlier ratio can be observed from the incorrect (red) and correct (green) correspondence connections.}
\label{fig.correspondence} 
\end{figure}
\vspace{-0.7cm}

\subsection{3DMatch and 3DLoMatch}
We compare our proposed CoFiNet to other state-of-the-art approaches including 3DSN~\cite{gojcic2019perfect}, FCGF~\cite{choy2019fully}, D3Feat~\cite{bai2020d3feat} and PREDATOR~\cite{huang2020predator} in Tab.~\ref{tab1} \footnote{As PREDATOR computes \textit{Inlier Ratio} on a correspondence set different to the one used for registration, we give more results in \ref{same} and Tab.~\ref{tabunified} for a fair comparison.\label{more}} and Fig.~\ref{fig.curve}. Comparisons to SpinImages~\cite{johnson1999using}, SHOT~\cite{tombari2010unique1}, FPFH~\cite{rusu2009fast} and 3DMatch~\cite{zeng20173dmatch} are also included in Fig.~\ref{fig.curve}. For efficiency and qualitative results, please refer to Appendix.
\vspace{-0.4cm}
\begin{table}[H]
\caption{Results\textsuperscript{\ref{more}} on both 3DMatch and 3DLoMatch datasets under different numbers of samples. We also show the number of utilized parameters of all the approaches in the last column. Best performance is highlighted in bold while the second best is marked with an underline.}
\resizebox{\textwidth}{40mm}{
\begin{tabular}{lccccc|cccccc}
\toprule
&\multicolumn{5}{c}{3DMatch} &\multicolumn{5}{c}{3DLoMatch} &\\
\# Samples &5000 &2500 &1000 &500 &250 &5000 &2500 &1000 &500 &250 &\# Params $\downarrow$\\
\midrule
&\multicolumn{10}{c}{\textit{Registration Recall}(\%) $\uparrow$} &\\
\midrule
3DSN\cite{gojcic2019perfect} &78.4 &76.2& 71.4 & 67.6 & 50.8 & 33.0 & 29.0 & 23.3 & 17.0 & 11.0 &10.2M\\
FCGF\cite{choy2019fully} &85.1 &84.7 &83.3 &81.6 & 71.4 &40.1 & 41.7 & 38.2 & 35.4 & 26.8 &8.76M\\
D3Feat\cite{bai2020d3feat} &81.6 &84.5 &83.4 &82.4 &77.9 &37.2 &42.7 &46.9 &43.8 &39.1 &27.3M\\
PREDATOR\cite{huang2020predator} &\underline{89.0} &\textbf{89.9} &\textbf{90.6} &\textbf{88.5} &\underline{86.6} &\underline{59.8} &\underline{61.2} &\underline{62.4} &\underline{60.8} &\underline{58.1} &\underline{7.43}M\\
CoFiNet(\textit{ours}) &\textbf{89.3} &\underline{88.9} &\underline{88.4} &\underline{87.4} &\textbf{87.0} &\textbf{67.5} &\textbf{66.2} &\textbf{64.2} &\textbf{63.1} &\textbf{61.0} &\textbf{5.48}M \\
\midrule
&\multicolumn{10}{c}{\textit{Feature Matching Recall}(\%) $\uparrow$} &\\
\midrule
3DSN\cite{gojcic2019perfect} &95.0 &94.3 &92.9 & 90.1 &82.9 &63.6 &61.7 &53.6 &45.2 &34.2 &10.2M\\
FCGF\cite{choy2019fully} &\underline{97.4} &\underline{97.3} &\underline{97.0} &\underline{96.7} &\underline{96.6} &76.6 &75.4 &74.2 &71.7 &67.3 &8.76M\\
D3Feat\cite{bai2020d3feat} &95.6 &95.4 &94.5 &94.1 &93.1 &67.3 &66.7 &67.0 &66.7 &66.5 &27.3M\\
PREDATOR\cite{huang2020predator} &96.6 &96.6 &96.5 &96.3 &96.5 &\underline{78.6} &\underline{77.4} &\underline{76.3} &\underline{75.7} &\underline{75.3} &\underline{7.43}M\\
CoFiNet(\textit{ours}) &\textbf{98.1} &\textbf{98.3} &\textbf{98.1} &\textbf{98.2} &\textbf{98.3} &\textbf{83.1} &\textbf{83.5} &\textbf{83.3} &\textbf{83.1} &\textbf{82.6} &\textbf{5.48M} \\
\midrule
&\multicolumn{10}{c}{\textit{Inlier Ratio}(\%) $\uparrow$} &\\
\midrule
3DSN\cite{gojcic2019perfect} &36.0 &32.5 &26.4 &21.5 &16.4 &11.4 &10.1 &8.0 &6.4 &4.8 &10.2M\\
FCGF\cite{choy2019fully} &\underline{56.8} &\underline{54.1} &48.7 &42.5 &34.1 &21.4 &20.0 &17.2 &14.8 &11.6 &8.76M\\
D3Feat\cite{bai2020d3feat} &39.0 &38.8 &40.4 &41.5 &41.8 &13.2 &13.1 &14.0 &14.6 &15.0 &27.3M\\
PREDATOR\cite{huang2020predator} &\textbf{58.0} &\textbf{58.4} &\textbf{57.1} &\textbf{54.1} &\underline{49.3} &\textbf{26.7} &\textbf{28.1} &\textbf{28.3} &\textbf{27.5} &\underline{25.8} &\underline{7.43}M\\
CoFiNet(\textit{ours}) &49.8 &51.2 &\underline{51.9} &\underline{52.2} &\textbf{52.2} &\underline{24.4} &\underline{25.9} &\underline{26.7} &\underline{26.8} &\textbf{26.9} &\textbf{5.48M} \\
\bottomrule
\end{tabular}}

\label{tab1}
\end{table}
\vspace{-0.8cm}
\paragraph{Metrics.}
We adopt three typically-used metrics, namely \textit{Registration Recall} (RR), \textit{Feature Matching Recall} (FMR) and \textit{Inlier Ratio} (IR), to show the superiority of CoFiNet over existing approaches. Specifically, 1) the \textit{Registration Recall} is the fraction of point cloud pairs whose error of transformation estimated by RANSAC is smaller than a certain threshold, e.g., RMSE < 0.2m, compared to the ground truth. 2) The \textit{Feature Matching Recall} indicates the percentage of point cloud pairs whose \textit{Inlier Ratio} is larger than a certain threshold, e.g., $\tau_2=5\%$. 3) The \textit{Inlier Ratio} is the fraction of correspondences whose residual error in geometry space is less than a threshold, e.g., $\tau_1=10$cm, under the ground truth transformation. Metric details are given in Appendix.

\paragraph{Correspondence sampling.}
We follow \cite{bai2020d3feat, huang2020predator} and report performance with different numbers of samples. However, as CoFiNet avoids keypoint detection and directly outputs point correspondences, we cannot strictly follow \cite{bai2020d3feat, huang2020predator} to sample different numbers of interest points. For a fair comparison, we instead sample correspondences in our experiments but keep the same number as them. Correspondences are sampled with probability proportional to a global confidence $c_{global}=c_{fine}\cdot c_{coarse}$. For a certain point correspondence refined from patch correspondence $(\widetilde{\mathbf{G}}^{\mathbf{P}}_{i^{\prime}}, \widetilde{\mathbf{G}}^{\mathbf{P}}_{j^{\prime}})$, we define $c_{fine}$ as its fine-level confidence score and $c_{coarse}$ as $\textbf{S}^{\prime}({i^{\prime}, j^{\prime}})$. 
% $(\mathbf{P}_{\mathbf{X}}(i), \mathbf{P}_{\mathbf{Y}}(j))$, we can find the truncated coarse patch correspondence $(\widetilde{\mathbf{G}}^{\mathbf{P}}_{i^{\prime}}, \widetilde{\mathbf{G}}^{\mathbf{P}}_{j^{\prime}})$, from which it is obtained, together with its fine-level confidence $c_{fine}$. Then global confidence $c_{global}$ can be calculated as $c_{global} = \textbf{S}^{\prime}\left({i^{\prime}, j^{\prime}}\right) \cdot c_{fine}$.
\vspace{-0.2cm}
\paragraph{\textit{Inlier Ratio}.\textsuperscript{\ref{more}}}
As the main contribution of CoFiNet is that we adopt the coarse-to-fine mechanism to avoid keypoint detection, while existing methods struggle to sample repeatable keypoints for matching, we first check the \textit{Inlier Ratio} of CoFiNet, which is directly related to the quality of extracted correspondences. We show quantitative results in Tab.~\ref{tab1} and qualitative results in Fig.~\ref{fig.correspondence}. As shown in Tab.~\ref{tab1}, on \textit{Inlier Ratio}, CoFiNet outperforms all the previous methods except PREDATOR~\cite{huang2020predator} on 3DLoMatch and only performs worse than PREDATOR~\cite{huang2020predator} and FCGF\cite{choy2019fully} on 3DMatch. Notably, when the sample number is 250, we perform the best on both datasets, since detection-based methods face a more severe repeatability problem in this case. By contrast, as our method leverages a coarse-to-fine mechanism and thus avoids keypoint detection, it is more robust to the aforementioned case. Furthermore, the fact that sampling fewer correspondences leads to a higher \textit{Inlier Ratio} indicates that our learned scores are well-calibrated, i.e., higher confidence scores indicate more reliable correspondences. 

 \vspace{-0.3cm}
 \begin{table}[H]
\caption{Registration results without RANSAC~\cite{fischler1981random}. Relative poses are directly solved based on extracted correspondences by singular value decomposition (SVD). Best performance is highlighted in bold while the second best is marked with an underline.}
\centering
\resizebox{0.8\textwidth}{14mm}{

\begin{tabular}{lccccc|ccccc}
\toprule
&\multicolumn{5}{c}{3DMatch} &\multicolumn{5}{c}{3DLoMatch}\\
\# Samples &5000 &2500 &1000 &500 &250 &5000 &2500 &1000 &500 &250\\
\midrule
&\multicolumn{10}{c}{\textit{Registration Recall} w.o. RANSAC (\%) $\uparrow$} \\
\midrule
FCGF\cite{choy2019fully} &28.5 &27.9 &25.7 &23.2 & 21.2 &2.3 & 1.7 & 1.3 & 1.1 & 1.1 \\
D3Feat\cite{bai2020d3feat} &24.3 &24.0 &23.0 &22.4 &19.1 &1.1 &1.4 &1.1 &1.0 &1.0\\
PREDATOR\cite{huang2020predator} &\underline{48.7} &\underline{51.8} &\underline{54.3} &\underline{53.5} &\underline{53.0} &\underline{6.1} &\underline{8.1} &\underline{10.1} &\underline{11.4} &\underline{11.3} \\
CoFiNet(\textit{ours}) &\textbf{63.2} &\textbf{63.4} &\textbf{63.8} &\textbf{64.9} &\textbf{64.6} &\textbf{19.0} &\textbf{20.4} &\textbf{21.0} &\textbf{20.9} &\textbf{21.6} \\
\bottomrule
\end{tabular}}

\label{tabsvd}
\end{table}
 \vspace{-0.3cm}

 \paragraph{Reliability of our correspondences.}
Though \textit{Inlier Ratio} is an important metric of correspondence quality, it is naturally affected by the distance threshold $\tau_1$. To better illustrate the reliability of correspondences extracted by CoFiNet and show our superiority over existing methods, we conduct another experiment and show related results in Tab.~\ref{tabsvd}. In this experiment, we directly solve the relative poses using singular value decomposition (SVD) based on extracted correspondences, without the assistance of the robust estimator RANSAC~\cite{fischler1981random}. As we can see, for FCGF~\cite{choy2019fully} and D3Feat~\cite{bai2020d3feat}, though they can work on 3DMatch, they fail on 3DLoMatch, where point cloud share less overlap and thus reliable correspondences are harder to obtain. Compared with PREDATOR~\cite{huang2020predator}, on both 3DMatch and 3DLoMatch, our proposed CoFiNet performs much better, which indicates that we propose more reliable correspondences on both datasets.

\begin{wrapfigure}{r}{0.6\textwidth} %H为当前位置，!htb为忽略美学标准，htbp为浮动图形
\centering
\setlength{\abovecaptionskip}{1pt}%    
\setlength{\belowcaptionskip}{0pt}%
\includegraphics[width=0.56\textwidth]{figures/curve.jpg} 
\caption{\textit{Feature Matching Recall} in relation to: 1) \textit{Inlier Ratio Threshold} ($\tau_2$) and 2) \textit{Inlier Distance Threshold} ($\tau_1$) on 3DMatch.}
\label{fig.curve}
\end{wrapfigure}
\vspace{-0.2cm}


\paragraph{\textit{Feature Matching Recall} and \textit{Registration Recall}.}
 On \textit{Feature Matching Recall}, CoFiNet significantly outperforms all the other methods on both 3DMatch and 3DLoMatch. Especially on 3DLoMatch, which is more challenging due to the low-overlap scenarios, our proposed method surpasses others with a large margin of more than 4\%. It indicates that CoFiNet is more robust to different scenes, i.e., we find at least 5\% inlier correspondences for more test cases. Additionally, we also follow \cite{bai2020d3feat, huang2020predator} to show the \textit{Feature Matching Recall} in relation to $\tau_2$ and $\tau_1$ on 3DMatch in Fig.~\ref{fig.curve}, which further shows our superiority over other methods. When referring to the most important metric \textit{Registration Recall} which better reflects the final performance on point cloud registration, though we perform slightly worse than PREDATOR~\cite{huang2020predator}, we significantly outperform others on 3DMatch. When evaluated on 3DLoMatch, our proposed approach significantly surpasses all the others, which shows the advantages of our method in scenarios with less overlap. Moreover, we also compare the number of parameters used in different methods in the last column of Tab.\ref{tab1}, which shows that CoFiNet uses the least parameters while achieving the best performance.
 \vspace{-0.2cm}
 
 \begin{table}[H]
\caption{\textit{Inlier Ratio} and \textit{Registration Recall} on the same correspondence set. For CoFiNet, coarse correspondences are extracted based on thresholds and \textit{non-mutual} selection is used on the finer scale. Best performance is highlighted in bold while the second best is marked with an underline.}
\resizebox{\textwidth}{22mm}{
\begin{tabular}{lccccc|ccccc}
\toprule
&\multicolumn{5}{c}{3DMatch} &\multicolumn{5}{c}{3DLoMatch}\\
\# Samples &5000 &2500 &1000 &500 &250 &5000 &2500 &1000 &500 &250\\
\midrule
&\multicolumn{10}{c}{\textit{Registration Recall}(\%) $\uparrow$}\\
\midrule
PREDATOR\cite{huang2020predator}(\textit{mutual}) &86.6 &86.4 &85.3 &85.6 &84.3 &\underline{61.8} &\underline{61.8} &61.6 &58.4 &56.2 \\
PREDATOR\cite{huang2020predator}(\textit{non-mutual}) &\underline{89.0} &\textbf{89.9} &\textbf{90.6} &\textbf{88.5} &\underline{86.6} &59.8 &61.2 &\underline{62.4} &\underline{60.8} &\underline{58.1}\\
CoFiNet(\textit{ours}) &\textbf{89.3} &\underline{88.9} &\underline{88.4} &\underline{87.4} &\textbf{87.0} &\textbf{67.5} &\textbf{66.2} &\textbf{64.2} &\textbf{63.1} &\textbf{61.0} \\
\midrule
%&\multicolumn{10}{c}{\textit{Feature Matching Recall}(\%) $\uparrow$}\\
%\midrule
%PREDATOR\cite{huang2020predator}(\textit{mutual}) &96.6 &96.6 &96.5 &96.3 &96.5 &\underline{78.6} &77.4 &76.3 &75.7 &75.3 \\
%PREDATOR\cite{huang2020predator}(\textit{non-mutual}) &\underline{96.7} &\underline{97.0} &\underline{96.9} &\underline{96.9} &\underline{96.8} &76.4 &\underline{77.9} &\underline{78.2} &\underline{77.7} &\underline{77.2}\\
%CoFiNet(\textit{ours}) &\textbf{98.1} &\textbf{98.3} &\textbf{98.1} &\textbf{98.2} &\textbf{98.3} &\textbf{83.1} &\textbf{83.5} &\textbf{83.3} &\textbf{83.1} &\textbf{82.6} \\
%\midrule
&\multicolumn{10}{c}{\textit{Inlier Ratio}(\%) $\uparrow$} \\
\midrule
PREDATOR\cite{huang2020predator}(\textit{mutual}) &\textbf{58.0} &\textbf{58.4} &\textbf{57.1} &\textbf{54.1} &\underline{49.3} &\textbf{26.7} &\textbf{28.1} &\textbf{28.3} &\textbf{27.5} &\underline{25.8} \\
PREDATOR\cite{huang2020predator}(\textit{non-mutual}) &46.6 &48.3 &47.2 &44.1 &38.8 &19.3 &21.6 &22.1 &21.3 &19.7\\
CoFiNet(\textit{ours}) &\underline{49.8} &\underline{51.2} &\underline{51.9} &\underline{52.2} &\textbf{52.2} &\underline{24.4} &\underline{25.9} &\underline{26.7} &\underline{26.8} &\textbf{26.9}\\
\bottomrule
\end{tabular}}

\label{tabunified}
\end{table}
 \paragraph{\textit{Inlier Ratio} and \textit{Registration Recall} on the same correspondence set.}\label{same}
In Tab.~\ref{tab1}, PREDATOR~\cite{huang2020predator} reports \textit{Inlier Ratio} on a correspondence set that is different to the one used for registration, while CoFiNet uses the same. PREDATOR uses correspondences extracted by \textit{mutual} selection to report \textit{Inlier Ratio}, while computing \textit{Registration Recall} on a correspondence set obtained by \textit{non-mutual} selection. As we target at registration, we consider it meaningless to evaluate on a correspondence set that is not used for pose estimation. Thus, to make a fair comparison, we compare CoFiNet with both PREDATOR(\textit{mutual}) and PREDATOR(\textit{non-mutual}) in Tab.~\ref{tabunified}. In \textit{mutual} selection, two points $\mathbf{x}$ and $\mathbf{y}$ are considered as a correspondence when $\mathbf{x}$ match to $\mathbf{y}$ \textbf{and} $\mathbf{y}$ match to $\mathbf{x}$, while in \textit{non-mutual} selection, the correspondence is  extracted when $\mathbf{x}$ match to $\mathbf{y}$ \textbf{or} $\mathbf{y}$ match to $\mathbf{x}$. In Tab.~\ref{tabunified}, compared to \textit{non-mutual}, \textit{mutual} selection rejects some outliers, and thus increases \textit{Inlier Ratio} of PREDATOR. However, as it meanwhile filters out some inlier correspondences, when combined with RANSAC~\cite{fischler1981random}, \textit{Registration Recall} usually drops. Since our task is registration, PREDATOR(\textit{non-mutual}) with higher \textit{Registration Recall} is preferred over itself with \textit{mutual} selection. In this case, CoFiNet achieves higher \textit{Inlier Ratio} than PREDATOR on both datasets.

 

\begin{wraptable}{r}{0.72\textwidth}
\caption{Ablation study of the number of coarse correspondences, tested with \# Samples=2500. \# Coarse indicates the average number of sampled coarse correspondences. Best performance is highlighted in bold.}
    \centering
    \resizebox{0.72\textwidth}{12mm}{
   \begin{tabular}{ll|ccc|ccc}
    \toprule
     & &\multicolumn{3}{c}{3DMatch} &\multicolumn{3}{c}{3DLoMatch}
     \\
     \hline
     $\tau_c$ &$\tau_m$ &IR (\%)$\uparrow$ &RR (\%) $\uparrow$ &\# Coarse &IR (\%)$\uparrow$ &RR (\%)$\uparrow$  &\# Coarse\\
     \hline
     \hline
    0.05 &- &49.4 &87.4 &575 &27.3 &62.8 &260\\
    0.10 &- &51.1 &88.1 &335 &29.8 &62.3 &128\\
    0.15 &- &\textbf{55.5} &85.5 &222 &\textbf{32.7} &58.9 &74\\
    0.20 &200 &51.2 &\textbf{88.9} &230 &25.9 &\textbf{66.2} &203\\
    \bottomrule
    \end{tabular}}
    \label{tab4}
\end{wraptable}
\vspace{-0.2cm}

\paragraph{Influence of the number of coarse correspondences.} 
 As illustrated in Tab.~\ref{tab4}, on both 3DMatch and 3DLoMatch, when sampled only with $\tau_c$, a higher threshold results in fewer coarse correspondences and meanwhile a higher \textit{Inlier Ratio}, which indicates the learned confidence scores are well-calibrated on the coarse level. However, \textit{Registration Recall} drops at the same time, as the number of correspondences for refinement is decreased, and thus fewer point correspondences are leveraged in RANSAC for pose estimation. The last row is the strategy used in our paper. Except for $\tau_c$, we also use $\tau_m$ to guarantee that CoFiNet samples at least $\tau_m$ coarse correspondences on each point cloud pair, as described before. This strategy slightly sacrifices \textit{Inlier Ratio} but brings significant improvements on \textit{Registration Recall}.



\paragraph{Importance of individual modules.}
  As shown in Tab.~\ref{tab2}, in the first experiment, we directly use the coarse correspondence set $\mathbf{C}^{\prime}$ for point cloud registration. Unsurprisingly, it performs worse on all the metrics, indicating that CoFiNet benefits from refinement. Then, we ablate the weighting scheme which is proportional to overlap ratios and guides the coarse matching of down-sampled nodes. We replace it with a binary mask similar to the one used on the finer level. Results show that it leads to a worse performance, which proves that coarse matching of nodes benefits from our designed weighting scheme. Finally, we do the last ablation study on the density-adaptive matching module. Results indicate that on both 3DMatch and 3DLoMatch, with the density-adaptive matching module, CoFiNet better adapts to the irregular nature of point clouds.
\vspace{-0.3cm}
\begin{table}[H]
  \caption{Ablation study of individual modules, tested with \# Samples=2500. Best performance is highlighted in bold.}
  \centering
\resizebox{0.9\textwidth}{12mm}{
    \begin{tabular}{l|ccc|ccc}
    \toprule
      &\multicolumn{3}{c}{3DMatch} &\multicolumn{3}{c}{3DLoMatch}
     \\
     \hline
      &RR (\%)$\uparrow$ &FMR (\%)$\uparrow$ &IR (\%) $\uparrow$ &RR (\%)$\uparrow$ &FMR (\%)$\uparrow$ &IR (\%)$\uparrow$\\
     \hline
     \hline
     Full CoFiNet &\textbf{88.9} &\textbf{98.3} &\textbf{51.2} &\textbf{66.2} &\textbf{83.5} &\textbf{25.9}\\
     w/o refinement &79.6 &96.5 &44.3 &41.2 &81.4 &21.3\\
     w/o weighting &87.4 &97.3 &50.0 &61.5 &80.5 &23.5\\
     w/o density-adaptive  &88.3 &97.9 &49.3 &65.1 &82.7 &24.7\\
    \bottomrule
    \end{tabular}}
    \label{tab2}
\end{table}
\vspace{-0.6cm}

%\begin{figure}[H] %H为当前位置，!htb为忽略美学标准，htbp为浮动图形
%\centering %图片居中
%\includegraphics[width=1.\textwidth]{figures/quality.pdf} %%插入图片，[]中设置图片大小，{}中是图片文件名
%\caption{Example visualization of registration results. CoFiNet provides reliable %correspondences which makes RANSAC capable of recovering the optimal transformation in all %of the listed scenarios.} %最终文档中希望显示的图片标题
%\label{fig.quality} %用于文内引用的标签
%\end{figure}




\subsection{KITTI}

\paragraph{Metrics.}
We follow \cite{huang2020predator} and use 3 metrics, namely, the \textit{Relative Rotation Error} (RRE), which is the geodesic distance between estimated and ground truth rotation matrices, the \textit{Relative Translation Error} (RTE), which is the Euclidean distance between the estimated and ground truth translation, and the \textit{Registration Recall} (RR) mentioned before. More details are provided in Appendix.

  \begin{wraptable}{r}{0.68\textwidth}
\caption{Quantitative comparisons on KITTI. Best performance is highlighted in bold.}
    \centering
    \resizebox{0.67\textwidth}{13mm}{
    \begin{tabular}{l|cccc}
    \toprule
     Method &RTE(cm)$\downarrow$ &RRE($^{\circ}$)$\downarrow$ &RR(\%)$\uparrow$ &Params$\downarrow$\\
    \hline
    \hline
    3DFeat-Net~\cite{yew20183dfeat} &25.9 &0.57 &96.0 &\textbf{0.32M}\\
    FCGF~\cite{choy2019fully} &9.5 &0.30 &96.6 &8.76M\\
    D3Feat~\cite{bai2020d3feat} &7.2 &0.30 &\textbf{99.8} &27.3M\\
    PREDATOR~\cite{huang2020predator} &\textbf{6.8} &\textbf{0.27} &\textbf{99.8} &22.8M\\
    CoFiNet(ours) &8.5 &0.41 &\textbf{99.8} &5.48M\\
    \bottomrule
    \end{tabular}}
    \label{tab3}
\end{wraptable}
\paragraph{Comparisons to existing approaches.}
  On KITTI, we compare CoFiNet to 3DFeat-net~\cite{yew20183dfeat}, FCGF~\cite{choy2019fully}, D3Feat~\cite{bai2020d3feat} and PREDATOR~\cite{huang2020predator}. Quantitative results can be found in Tab.~\ref{tab3}, while qualitative results are given in Appendix. On RTE and RRE, we stay in the middle, but for RR, together with \cite{bai2020d3feat, huang2020predator}, we perform the best. Notably, we achieve such performance by using only 5.48M parameters and training for 20 epochs compared to the best performing model~\cite{huang2020predator}, which uses over 20M parameters and is trained for 150 epochs. This experiment indicates that our model can deal with outdoor scenarios. 


\section{Conclusion}
In this paper, we present a deep neural network that leverages a coarse-to-fine strategy to extract correspondences from unordered and irregularly sampled point clouds for registration. Our proposed model is capable of directly consuming unordered point sets and proposing reliable correspondences without the assistance of keypoints. To tackle the irregularity of point clouds, on a coarse scale, we first propose a weighting scheme proportional to local overlap ratios. It guides the model to match nodes that have overlapped vicinity areas, which significantly shrinks the search space of the following refinement. On a finer level, we then adopt a density-adaptive matching module, which eliminates the side effects from repeated sampling and enables our model to deal with density varying points. Extensive experiments on both indoor and outdoor benchmarks validate the effectiveness of our proposed model. We stay on par with the state-of-the-art approaches on 3DMatch and KITTI, while surpassing them on 3DLoMatch using a model with significantly fewer parameters. Limitations and broader impact are discussed in Appendix.
\newpage
\appendix
\section{Appendix}
%Optionally include extra information (complete proofs, additional experiments and plots) in the appendix.
%This section will often be part of the supplemental material.
In this supplementary material, we first provide detailed network architectures in Sec.~\ref{sec.arch}. Then details of metrics utilized in our experiments are demonstrated in Sec.~\ref{sec.metric}. We give more implementation details in Sec.~\ref{sec.implementation}. We further introduce our utilized datasets in Sec.~\ref{sec.data} and evaluate the inference time of CoFiNet in Sec.~\ref{sec.timing}. Limitations and broader impact are then discussed in Sec.~\ref{sec.limitation} and Sec.~\ref{sec.impact}, respectively. Finally, qualitative results of registration are provided in Sec.~\ref{sec.visual}.


\subsection{Network Architectures}
\label{sec.arch}
CoFiNet mainly leverages an encoder-decoder architecture based on KPConv~\cite{thomas2019kpconv} operations,  where we also add two attention-based networks~\cite{vaswani2017attention} for context aggregation. Details of our network architecture are demonstrated in Fig.~\ref{fig.architecture}. Compared to~\cite{huang2020predator}, though we add additional local attention layers, our coarse-to-fine design enables us to use a lightweight encoder, which leads to the reduction of around 2M and over 20M parameters on 3DMatch/3DLoMatch and KITTI, respectively. Since we use the voxel size and convolution radius same to PREDATOR~\cite{huang2020predator} for our KPConv backbone, each time of point down-sampling in CoFiNet results in nodes identical to that in~\cite{huang2020predator}.

\subsection{Evaluation Metrics}
\label{sec.metric}
\paragraph{Inlier Ratio} \textit{Inlier Ratio} (IR) measures the fraction of point correspondences $(x_{i}, y_{j}) \in \mathbf{\widetilde{\mathbf{C}}}$ s.t. the Euclidean Norm of residual $\lVert \overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}(x_i) - y_j \rVert$ is within a certain threshold ${\tau}_1$=10cm, where $\overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}$ indicates the ground truth transformation between $\mathbf{X}$ and $\mathbf{Y}$. Given the  estimated correspondence set $\widetilde{\mathbf{C}}$, \textit{Inlier Ratio} of a single point cloud pair $(\mathbf{X}, \mathbf{Y})$ can be calculated by:

\begin{equation}
    \text{IR}(\mathbf{X}, \mathbf{Y}) = \frac{1}{|\widetilde{\mathbf{C}}|} \sum\limits_{(x_i, y_j)\in\widetilde{\mathbf{C}}} \mathds{1} (\lVert \overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}(x_i) - y_j\rVert < {\tau}_1),
    \label{eq.ir}
\end{equation}
where $\mathds{1}(\cdot)$ represents the indicator function and $\lVert \cdot \rVert=\lVert \cdot \rVert_2$ denotes the Euclidean Norm.

\paragraph{Feature Matching Recall}
\textit{Feature Matching Recall} (FMR) measures the fraction of point cloud pairs whose \textit{Inlier Ratio} is larger than a certain threshold $\tau_2=5\%$. It is first utilized in~\cite{deng2018ppf} and it indicates the likelihood that the optimal transformation between two point clouds can be recovered by a robust pose estimator, e.g., RANSAC~\cite{fischler1981random}, based on the predicted correspondence set $\widetilde{\mathbf{C}}$. Given a dataset $\mathcal{D}$ with $|\mathcal{D}|$ point cloud pairs, \textit{Feature Matching Recall} can be represented as:

\begin{equation}
    \text{FMR}(\mathcal{D}) = \frac{1}{|\mathcal{D}|}\sum\limits_{(\mathbf{X}, \mathbf{Y})
    \in \mathcal{D}} \mathds{1}(\text{IR}(\mathbf{X}, \mathbf{Y}) > \tau_2).
    \label{eq.fmr}
\end{equation}


\paragraph{Registration Recall} Different from the aforementioned metrics which measure the quality of extracted correspondences,  \textit{Registration Recall} (RR) directly measures the performance on our target task of point cloud registration. It measures the fraction of point cloud pairs whose Root Mean Square Error (RMSE) is within a certain threshold $\tau_3$ = 0.2m. Give a dataset $\mathcal{D}$ with $|\mathcal{D}|$ point cloud pairs, \textit{Registration Recall} is defined as:

\begin{equation}
    \text{RR}(\mathcal{D}) = \frac{1}{|\mathcal{D}|}\sum\limits_{(\mathbf{X}, \mathbf{Y})
    \in \mathcal{D}} \mathds{1}(\text{RMSE}(\mathbf{X}, \mathbf{Y}) < \tau_3),
    \label{eq.rr}
\end{equation}

where for each $(\mathbf{X}, \mathbf{Y})\in \mathcal{D}$, RMSE of the ground truth correspondence set $\overline{\mathbf{C}}$ after applying the estimated transformation $\mathbf{T}^{\mathbf{X}}_{\mathbf{Y}}$ reads as:

\begin{equation}
    \text{RMSE}(\mathbf{X}, \mathbf{Y}) = \sqrt{\frac{1}{|\overline{\mathbf{C}}|}\sum\limits_{(x_i, y_j)\in\overline{\mathbf{C}}}\lVert \mathbf{T}^{\mathbf{X}}_{\mathbf{Y}}(x_i) - y_j\rVert^{2}}.
    \label{eq.rmse}
\end{equation}

\vspace{-0.3cm}
Additionally, we follow the original evaluation protocol in 3DMatch~\cite{zeng20173dmatch}, which excludes immediately adjacent point clouds with very high overlap ratios.

\begin{figure}
\centering 
\includegraphics[width=1.0\textwidth]{figures/architecture.jpg} 
\caption{The detailed architecture of our proposed CoFiNet. In self- and cross-attention modules, we use four heads for the multi-head attention part. For instance, in self-attention modules (bottom centre), for \textit{query} $\mathbf{Q}\in \mathbb{R}^{n^{\prime}\times b}$, \textit{key} $\mathbf{K}\in \mathbb{R}^{n^{\prime}\times b}$ and \textit{value} $\mathbf{V}\in \mathbb{R}^{n^{\prime}\times b}$, we first reshape each of them into shape ($n^{\prime}$, 4, $\frac{b}{4}$) and then compute messages separately, which leads to messages in shape ($n^{\prime}, 4, \frac{b}{4}$). Finally we concatenate all the computed messages and obtain the message $\mathbf{M}\in \mathbb{R}^{n^{\prime}\times b}$. The Patch Grouping layer indicates the Grouping module in the Correspondence Refinement Block. Self- and cross-attention modules (lower right) represent the case in the Attention part (centre), while in the Local Attention part (upper right), $\mathbf{F}_{\mathbf{X}}^{\prime}$ and $\mathbf{F}_{\mathbf{Y}}^{\prime}$ are replaced with $\widetilde{\mathbf{G}}^{\mathbf{F}}_{i^{\prime}}$ and $\widetilde{\mathbf{G}}^{\mathbf{F}}_{j^{\prime}}$, respectively.}
\label{fig.architecture} 
\vspace{-0.6cm}
\end{figure}


\paragraph{Relative Translation and Rotation Errors} Given the estimated transformation $\mathbf{T}^{\mathbf{X}}_{\mathbf{Y}} \in SE(3)$ with a translation vector $\mathbf{t} \in \mathbb{R}^{3}$ and a rotation matrix $\mathbf{R}\in SO(3)$. Its Relative Translation Error (RTE) and Relative Rotation Error (RRE) from the ground truth pose $\overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}$ are computed as:


\begin{equation}
\text{RTE} = \lVert\mathbf{t} - \overline{\mathbf{t}}\rVert \qquad \text{and} \qquad
\text{RRE} = \text{arccos}(\frac{\text{trace}(\mathbf{R}^{\top}\overline{\mathbf{R}}) - 1}{2}),
\label{eq.relative}
\end{equation}
where $\overline{\mathbf{t}}$ and $\overline{\mathbf{R}}$ are the the ground truth translation and rotation in $\overline{\mathbf{T}}^{\mathbf{X}}_{\mathbf{Y}}$, respectively.


\subsection{Implementation Details}
\label{sec.implementation}
CoFiNet is implemented in PyTorch~\cite{paszke2019pytorch} and can be trained end-to-end on a single RTX 2080Ti GPU. We train 20 epochs on 3DMatch/3DLoMatch and KITTI, with $\lambda=1$, both using Adam optimizer with an initial learning rate of 3e-4, which is exponentially decayed by 0.05 after each epoch. We adopt similar encoder and decoder architectures as \cite{huang2020predator}, but with significantly fewer parameters. We use a batch size of 1 in all experiments. For training the attention-based network on a finer scale, we sample 128 coarse correspondences, with truncated patch size $k=64$ on 3DMatch (3DLoMatch). On KITTI, the numbers are 128 and 32, respectively. Moreover, due to the severely varying point density on KITTI, we only sample node correspondences with overlap ratios $> 20\%$ for training. At test time, all the extracted coarse correspondences are fed into the finer stage for refinement, with the same $k$ as in training. We use our proposed point correspondences and RANSAC~\cite{fischler1981random} for registration. 


\subsection{Data}
\label{sec.data}
\paragraph{3DMatch and 3DLoMatch} 3DMatch~\cite{zeng20173dmatch} collects 62 scenes from SUN3D~\cite{xiao2013sun3d}, 7-Scenes~\cite{shotton2013scene}, RGB-D Scenes v.2~\cite{lai2014unsupervised}, Analysis-by-Synthesis~\cite{valentin2016learning}, BundleFusion~\cite{dai2017bundlefusion} and Halbel et al.~\cite{halber2017fine}, where 46 scenes are used for training, 8 scenes for validation and 8 scenes for testing.  We utilize the training data in~\cite{huang2020predator} for training and also follow its evaluation protocols for testing. In training, input point cloud frames are generated by fusing 50 consecutive depth frames using TSDF volumetric fusion~\cite{curless1996volumetric}. Different from the original 3DMatch~\cite{zeng20173dmatch} that only consists of point cloud pairs with \textgreater 30\% overlaps, in~\cite{huang2020predator}, point cloud pairs with overlaps between 10\% and 30\% are also included. Two benchmarks are leveraged for testing, namely, 3DMatch that consists of point cloud pairs with \textgreater 30\% overlaps, and 3DLoMatch which only includes point cloud pairs whose overlaps are between 10\% and 30\%. We also follow~\cite{huang2020predator} to use voxel-grid down-sampling for preprocessing, where a random point will be picked when multiple points fall into the same voxel grid.

\paragraph{OdometryKITTI} KITTI~\cite{geiger2013vision} is published under the NonCommercial-ShareAlike 3.0 License. It consists of 11 sequences scanned by a Velodyne HDL-64 3D laser scanner in driving scenarios. We follow~\cite{bai2020d3feat} to pick point cloud pairs with at least 10m intervals from the raw data, which leads to 1,358 training pairs, 180 validation pairs, and 555 testing pairs. Moreover, as the ground truth poses provided by GPS are noisy, we follow~\cite{bai2020d3feat} to use ICP to further refine them.
\vspace{-0.3cm}
\begin{table}[H]
  \caption{Model runtime comparisons for a single inference. Time is averaged over the whole 3DMatch~\cite{zeng20173dmatch} testing set, which consists of 1,623 point cloud pairs. As our target task is registration and neural networks only provide intermediate results which are later consumed by RANSAC~\cite{fischler1981random} for pose estimation, we also include the time of writing related results to hard disks.}


\resizebox{\textwidth}{7mm}{
    \begin{tabular}{c|cccc}
    \toprule
      &\textbf{CPU} &\textbf{GPU} &\textbf{Time(s)}$\downarrow$ &\textbf{Improvement(\%)$\uparrow$}\\
     \hline
      PREDATOR~\cite{huang2020predator} &i7-9700KF @ 3.60GHZ $\times$ 8 &GeForce RTX 3070 &0.72 &-\\
      CoFiNet(\textit{ours}) &i7-9700KF @ 3.60GHZ $\times$ 8 &GeForce RTX 3070 &\textbf{0.25} &\textbf{65.3}\\
    \bottomrule
    \end{tabular}}
    \label{tab.timing}
\end{table}


\subsection{Timings}
\label{sec.timing}
We further evaluate the inference time of CoFiNet and compare it to that of PREDATOR~\cite{huang2020predator} which obtains the highest inference rate among all the state-of-the-art methods. Related results in Tab.~\ref{tab.timing} indicate the superiority of CoFiNet over PREDATOR in terms of computational efficiency. Notably, CoFiNet directly proposes point correspondences, while PREDATOR only outputs dense descriptors, and correspondences are extracted during RANSAC~\cite{fischler1981random}. We further compare CoFiNet to PREDATOR in regard to RANSAC runtime, related results are illustrated in Tab.~\ref{tab.ransac}. Benefiting from our design, we reduce the RANSAC runtime significantly, especially when more correspondences are leveraged for pose estimation.

 \begin{table}[H]
  \caption{RANSAC~\cite{fischler1981random} runtime comparisons for a single inference. Time is averaged over the whole 3DMatch~\cite{zeng20173dmatch} testing set, which consists of 1,623 point cloud pairs. Settings are the same with Tab.~\ref{tab.timing}}
    \centering
    \begin{tabular}{c|ccccc}
    \toprule
     \# Samples &5000 &2500 &1000 &500 &250\\
     \hline
      PREDATOR~\cite{huang2020predator} &2.86s &1.25s &0.45s &0.22s &0.11s\\
      CoFiNet(\textit{ours})  &\textbf{0.18s} &\textbf{0.11s} &\textbf{0.07s} &\textbf{0.05s} &\textbf{0.05s}\\
    \bottomrule
    \end{tabular}
    \label{tab.ransac}
\end{table}

 \begin{figure}
\centering 
\includegraphics[width=1.0\textwidth]{figures/inlier.jpg} 
\caption{Visualization of correspondences. Examples are from 3DLoMatch~\cite{huang2020predator} and we compare our method to PREDATOR~\cite{huang2020predator}. In column (b) and column (d), we only visualize 250 correspondences for better visibility but mark all the incorrectly matched points as red in both source and target point clouds. Correct correspondences are drawn in green.}
\label{fig.inlier} 
\vspace{-0.6cm}
\end{figure}
\subsection{Limitations}
\label{sec.limitation}
The limitations of our proposed CoFiNet are three-fold. 1) There is no explicit design for rejecting outliers from a coarse scale. False coarse correspondences can be expanded to false point correspondences which could result in lower \textit{Inlier Ratio} on a finer level. As shown in column (c) and column (d) of the first row in Fig.~\ref{fig.inlier}, after refinement, the \textit{Inlier Ratio} drops. 2) CoFiNet is challenged by those non-distinctive regions. As illustrated in column (d) of the first row in Fig.~\ref{fig.inlier}, mismatched points are located on the surface of the table, which is a flat area with little variability. 3) Point correspondences expanded from coarse correspondences are not sparse enough, which might introduce side effects to RANSAC\cite{fischler1981random} based point cloud registration. As demonstrated in column (d) of the second row in Fig.~\ref{fig.inlier}, in comparison to PREDATOR~\cite{huang2020predator}, our method produces a much better \textit{Inlier Ratio} but extracts less sparser correspondences.

\subsection{Broader Impact}
\label{sec.impact}
We present a novel deep neural network that leverages the coarse-to-fine mechanism to extract correspondences from point clouds, which can be utilized for registration. It makes a first attempt towards the detection-free matching between a pair of unordered, irregular point sets. Our work can contribute to a wide range of applications, such as scene reconstruction, autonomous driving, simultaneous localization and mapping (SLAM), or any other where point cloud registration plays a role. For instance, the reconstruction of indoor scenes from unlabeled RGB-D images could benefit from our method, as it is capable of extracting reliable correspondences that can be leveraged to recover the rigid transformation between different frames precisely. Also, in autonomous driving scenarios, our methods can help agents better sense their surroundings. As our method aims at tackling a fundamental problem in computer vision, we do not anticipate a direct negative outcome. Potential negative outcomes might occur in real applications where our method is involved.

\subsection{Qualitative Results of Registration}
\label{sec.visual}
Visualization of example registration from different datasets can be found in Fig.~\ref{fig.registration}. Relative poses are estimated by RANSAC~\cite{fischler1981random} that takes correspondences extracted by CoFiNet as input.


\begin{figure}
\centering 
\includegraphics[width=1.0\textwidth]{figures/3dmatch.jpg} 
\includegraphics[width=1.0\textwidth]{figures/3dlomatch.jpg} 
\includegraphics[width=1.0\textwidth]{figures/kitti.jpg} 
\caption{Qualitative registration results. We show two examples for each dataset. Column (a) and column (b) demonstrate the input point cloud pairs. Column (c) shows the estimated registration while column (d) provides the ground truth alignment.}
\label{fig.registration} 
\end{figure}


\begin{thebibliography}{10}

\bibitem{johnson1999using}
Andrew~E. Johnson and Martial Hebert.
\newblock Using spin images for efficient object recognition in cluttered 3d
  scenes.
\newblock {\em IEEE Transactions on pattern analysis and machine intelligence},
  21(5):433--449, 1999.

\bibitem{tombari2010unique1}
Federico Tombari, Samuele Salti, and Luigi Di~Stefano.
\newblock Unique signatures of histograms for local surface description.
\newblock In {\em European conference on computer vision}, pages 356--369.
  Springer, 2010.

\bibitem{tombari2010unique2}
Federico Tombari, Samuele Salti, and Luigi Di~Stefano.
\newblock Unique shape context for 3d data description.
\newblock In {\em Proceedings of the ACM workshop on 3D object retrieval},
  pages 57--62, 2010.

\bibitem{rusu2009fast}
Radu~Bogdan Rusu, Nico Blodow, and Michael Beetz.
\newblock Fast point feature histograms (fpfh) for 3d registration.
\newblock In {\em 2009 IEEE international conference on robotics and
  automation}, pages 3212--3217. IEEE, 2009.

\bibitem{rusu2008persistent}
Radu~Bogdan Rusu, Zoltan~Csaba Marton, Nico Blodow, and Michael Beetz.
\newblock Persistent point feature histograms for 3d point clouds.
\newblock In {\em Proc 10th Int Conf Intel Autonomous Syst (IAS-10),
  Baden-Baden, Germany}, pages 119--128, 2008.

\bibitem{zeng20173dmatch}
Andy Zeng, Shuran Song, Matthias Nie{\ss}ner, Matthew Fisher, Jianxiong Xiao,
  and Thomas Funkhouser.
\newblock 3dmatch: Learning local geometric descriptors from rgb-d
  reconstructions.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 1802--1811, 2017.

\bibitem{deng2018ppfnet}
Haowen Deng, Tolga Birdal, and Slobodan Ilic.
\newblock Ppfnet: Global context aware local features for robust 3d point
  matching.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 195--205, 2018.

\bibitem{deng2018ppf}
Haowen Deng, Tolga Birdal, and Slobodan Ilic.
\newblock Ppf-foldnet: Unsupervised learning of rotation invariant 3d local
  descriptors.
\newblock In {\em Proceedings of the European Conference on Computer Vision
  (ECCV)}, pages 602--618, 2018.

\bibitem{gojcic2019perfect}
Zan Gojcic, Caifa Zhou, Jan~D Wegner, and Andreas Wieser.
\newblock The perfect match: 3d point cloud matching with smoothed densities.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition}, pages 5545--5554, 2019.

\bibitem{bai2020d3feat}
Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai.
\newblock D3feat: Joint learning of dense detection and description of 3d local
  features.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition}, pages 6359--6367, 2020.

\bibitem{saleh2020graphite}
Mahdi Saleh, Shervin Dehghani, Benjamin Busam, Nassir Navab, and Federico
  Tombari.
\newblock Graphite: Graph-induced feature extraction for point cloud
  registration.
\newblock In {\em 2020 International Conference on 3D Vision (3DV)}, pages
  241--251, 2020.

\bibitem{huang2020predator}
Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad
  Schindler.
\newblock Predator: Registration of 3d point clouds with low overlap.
\newblock {\em arXiv preprint arXiv:2011.13005}, 2020.

\bibitem{wang2019deep}
Yue Wang and Justin~M Solomon.
\newblock Deep closest point: Learning representations for point cloud
  registration.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
  Computer Vision}, pages 3523--3532, 2019.

\bibitem{wang2019prnet}
Yue Wang and Justin~M Solomon.
\newblock Prnet: Self-supervised learning for partial-to-partial registration.
\newblock {\em arXiv preprint arXiv:1910.12240}, 2019.

\bibitem{yew2020rpm}
Zi~Jian Yew and Gim~Hee Lee.
\newblock Rpm-net: Robust point matching using learned features.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
  pattern recognition}, pages 11824--11833, 2020.

\bibitem{besl1992method}
Paul~J Besl and Neil~D McKay.
\newblock Method for registration of 3-d shapes.
\newblock In {\em Sensor fusion IV: control paradigms and data structures},
  volume 1611, pages 586--606. International Society for Optics and Photonics,
  1992.

\bibitem{aoki2019pointnetlk}
Yasuhiro Aoki, Hunter Goforth, Rangaprasad~Arun Srivatsan, and Simon Lucey.
\newblock Pointnetlk: Robust \& efficient point cloud registration using
  pointnet.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition}, pages 7163--7172, 2019.

\bibitem{huang2020feature}
Xiaoshui Huang, Guofeng Mei, and Jian Zhang.
\newblock Feature-metric registration: A fast semi-supervised approach for
  robust point cloud registration without correspondences.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition}, pages 11366--11374, 2020.

\bibitem{wu20153d}
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang,
  and Jianxiong Xiao.
\newblock 3d shapenets: A deep representation for volumetric shapes.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 1912--1920, 2015.

\bibitem{geiger2013vision}
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.
\newblock Vision meets robotics: The kitti dataset.
\newblock {\em The International Journal of Robotics Research},
  32(11):1231--1237, 2013.

\bibitem{fischler1981random}
Martin~A Fischler and Robert~C Bolles.
\newblock Random sample consensus: a paradigm for model fitting with
  applications to image analysis and automated cartography.
\newblock {\em Communications of the ACM}, 24(6):381--395, 1981.

\bibitem{li2019usip}
Jiaxin Li and Gim~Hee Lee.
\newblock Usip: Unsupervised stable interest point detection from 3d point
  clouds.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
  Computer Vision}, pages 361--370, 2019.

\bibitem{li2020dual}
Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu.
\newblock Dual-resolution correspondence networks.
\newblock {\em Advances in Neural Information Processing Systems}, 33, 2020.

\bibitem{zhou2020patch2pix}
Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe.
\newblock Patch2pix: Epipolar-guided pixel-level correspondences.
\newblock {\em arXiv preprint arXiv:2012.01909}, 2020.

\bibitem{sun2021loftr}
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.
\newblock Loftr: Detector-free local feature matching with transformers.
\newblock {\em arXiv preprint arXiv:2104.00680}, 2021.

\bibitem{sarlin2020superglue}
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.
\newblock Superglue: Learning feature matching with graph neural networks.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
  pattern recognition}, pages 4938--4947, 2020.

\bibitem{bromley1993signature}
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S{\"a}ckinger, and Roopak
  Shah.
\newblock Signature verification using a" siamese" time delay neural network.
\newblock {\em Advances in neural information processing systems}, 6:737--744,
  1993.

\bibitem{qi2017pointnet}
Charles~R Qi, Hao Su, Kaichun Mo, and Leonidas~J Guibas.
\newblock Pointnet: Deep learning on point sets for 3d classification and
  segmentation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 652--660, 2017.

\bibitem{yang2018foldingnet}
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.
\newblock Foldingnet: Point cloud auto-encoder via deep grid deformation.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pages 206--215, 2018.

\bibitem{ao2020spinnet}
Sheng Ao, Qingyong Hu, Bo~Yang, Andrew Markham, and Yulan Guo.
\newblock Spinnet: Learning a general surface descriptor for 3d point cloud
  registration.
\newblock {\em arXiv preprint arXiv:2011.12149}, 2020.

\bibitem{choy2019fully}
Christopher Choy, Jaesik Park, and Vladlen Koltun.
\newblock Fully convolutional geometric features.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
  Computer Vision}, pages 8958--8966, 2019.

\bibitem{choy20194d}
Christopher Choy, JunYoung Gwak, and Silvio Savarese.
\newblock 4d spatio-temporal convnets: Minkowski convolutional neural networks.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition}, pages 3075--3084, 2019.

\bibitem{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
  Aidan~N Gomez, Lukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock {\em arXiv preprint arXiv:1706.03762}, 2017.

\bibitem{thomas2019kpconv}
Hugues Thomas, Charles~R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui,
  Fran{\c{c}}ois Goulette, and Leonidas~J Guibas.
\newblock Kpconv: Flexible and deformable convolution for point clouds.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
  Computer Vision}, pages 6411--6420, 2019.

\bibitem{wang2019dynamic}
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay~E Sarma, Michael~M Bronstein, and
  Justin~M Solomon.
\newblock Dynamic graph cnn for learning on point clouds.
\newblock {\em Acm Transactions On Graphics (tog)}, 38(5):1--12, 2019.

\bibitem{busam2015stereo}
Benjamin Busam, Marco Esposito, Simon Che'Rose, Nassir Navab, and Benjamin
  Frisch.
\newblock A stereo vision approach for cooperative robotic movement therapy.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
  Vision Workshops}, pages 127--135, 2015.

\bibitem{sinkhorn1967concerning}
Richard Sinkhorn and Paul Knopp.
\newblock Concerning nonnegative matrices and doubly stochastic matrices.
\newblock {\em Pacific Journal of Mathematics}, 21(2):343--348, 1967.

\bibitem{cuturi2013sinkhorn}
Marco Cuturi.
\newblock Sinkhorn distances: Lightspeed computation of optimal transport.
\newblock {\em Advances in neural information processing systems},
  26:2292--2300, 2013.

\bibitem{peyre2019computational}
Gabriel Peyr{\'e}, Marco Cuturi, et~al.
\newblock Computational optimal transport: With applications to data science.
\newblock {\em Foundations and Trends{\textregistered} in Machine Learning},
  11(5-6):355--607, 2019.

\bibitem{kohonen1990self}
Teuvo Kohonen.
\newblock The self-organizing map.
\newblock {\em Proceedings of the IEEE}, 78(9):1464--1480, 1990.

\bibitem{li2018so}
Jiaxin Li, Ben~M Chen, and Gim~Hee Lee.
\newblock So-net: Self-organizing network for point cloud analysis.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 9397--9406, 2018.

\bibitem{qi2017pointnet++}
Charles~R Qi, Li~Yi, Hao Su, and Leonidas~J Guibas.
\newblock Pointnet++: Deep hierarchical feature learning on point sets in a
  metric space.
\newblock {\em arXiv preprint arXiv:1706.02413}, 2017.

\bibitem{yew20183dfeat}
Zi~Jian Yew and Gim~Hee Lee.
\newblock 3dfeat-net: Weakly supervised local 3d features for point cloud
  registration.
\newblock In {\em Proceedings of the European Conference on Computer Vision
  (ECCV)}, pages 607--623, 2018.

\bibitem{paszke2019pytorch}
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
  Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et~al.
\newblock Pytorch: An imperative style, high-performance deep learning library.
\newblock {\em arXiv preprint arXiv:1912.01703}, 2019.

\bibitem{xiao2013sun3d}
Jianxiong Xiao, Andrew Owens, and Antonio Torralba.
\newblock Sun3d: A database of big spaces reconstructed using sfm and object
  labels.
\newblock In {\em Proceedings of the IEEE international conference on computer
  vision}, pages 1625--1632, 2013.

\bibitem{shotton2013scene}
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi,
  and Andrew Fitzgibbon.
\newblock Scene coordinate regression forests for camera relocalization in
  rgb-d images.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pages 2930--2937, 2013.

\bibitem{lai2014unsupervised}
Kevin Lai, Liefeng Bo, and Dieter Fox.
\newblock Unsupervised feature learning for 3d scene labeling.
\newblock In {\em 2014 IEEE International Conference on Robotics and Automation
  (ICRA)}, pages 3050--3057. IEEE, 2014.

\bibitem{valentin2016learning}
Julien Valentin, Angela Dai, Matthias Nie{\ss}ner, Pushmeet Kohli, Philip Torr,
  Shahram Izadi, and Cem Keskin.
\newblock Learning to navigate the energy landscape.
\newblock In {\em 2016 Fourth International Conference on 3D Vision (3DV)},
  pages 323--332. IEEE, 2016.

\bibitem{dai2017bundlefusion}
Angela Dai, Matthias Nie{\ss}ner, Michael Zollh{\"o}fer, Shahram Izadi, and
  Christian Theobalt.
\newblock Bundlefusion: Real-time globally consistent 3d reconstruction using
  on-the-fly surface reintegration.
\newblock {\em ACM Transactions on Graphics (ToG)}, 36(4):1, 2017.

\bibitem{halber2017fine}
Maciej Halber and Thomas Funkhouser.
\newblock Fine-to-coarse global registration of rgb-d scans.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pages 1755--1764, 2017.

\bibitem{curless1996volumetric}
Brian Curless and Marc Levoy.
\newblock A volumetric method for building complex models from range images.
\newblock In {\em Proceedings of the 23rd annual conference on Computer
  graphics and interactive techniques}, pages 303--312, 1996.

\end{thebibliography}