Extrinsic Camera Calibration with Semantic Segmentation
Extracted source text · arXiv 2208.03949
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
% \documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
%In case you encounter the following error:
%Error 1010 The PDF file may be corrupt (unable to open PDF file) OR
%Error 1000 An error occurred while parsing a contents stream. Unable to analyze the PDF file.
%This is a known problem with pdfLaTeX conversion filter. The file cannot be opened with acrobat reader
%Please use one of the alternatives below to circumvent this error by uncommenting one or the other
%\pdfobjcompresslevel=0
%\pdfminorversion=4
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document
% The following packages can be found on http:\\www.ctan.org
\usepackage{graphicx} % for pdf, bitmapped graphics files
%\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb} % assumes amsmath package installed
\usepackage{siunitx}
\usepackage{todonotes}
\usepackage{placeins}
\usepackage{ifthen}
\usepackage{makecell}
\renewcommand{\cellalign}{lc}
%\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage{booktabs}
\usepackage{hyperref}
%\usepackage{caption}
%\usepackage{subcaption}
\newboolean{PrintVersion}
\setboolean{PrintVersion}{true}
\newcommand{\ReviewChangeRemoved}[1]{\sout{#1}}
\newcommand{\ReviewChange}[1]{\textcolor{blue}{#1}}
\ifthenelse{\boolean{PrintVersion}}{
\renewcommand{\ReviewChangeRemoved}[1]{}
\renewcommand{\ReviewChange}[1]{#1}
}{}
\title{\LARGE \bf
Extrinsic Camera Calibration with Semantic Segmentation
}
\author{Alexander Tsaregorodtsev$^{1}$, Johannes~Müller$^{1}$, Jan~Strohbeck$^{1}$, Martin~Herrmann$^{1}$, \\ Michael~Buchholz$^{1}$ and Vasileios Belagiannis$^{2}$% <-this % stops a space
%\thanks{A. T. and J. M. were supported by the research program ``Highly and Fully Automated Driving in Demanding Driving Situations'' (project LUKAS, grant number 19A20004F), funded by the Federal Ministry for Economic Affairs and Climate Action of Germany.}% <-this % stops a space
\thanks{*Part of this work was financially supported by the Federal Ministry of Economic Affairs and Energy of Germany within the program "New Vehicle and System Technologies" (project LUKAS, grant number 19A20004F). Part of this work has been conducted as part of ICT4CART project which has received funding from the European Union’s Horizon 2020 research \& innovation program under grant agreement No. 768953. Content reflects only the authors’ view and European Commission is not responsible for any use that may be made of the information it contains. }% <-this % stops a space
%\thanks{$^{1}$Alexander Tsaregorodtsev,
%Johannes M{\"u}ller, Martin Herrmann, Jan Strohbeck, Michael Buchholz, and Vasileios Belagiannis are with the Institute of Measurement, Control and Microtechnology,
%Ulm University, D-89081 Ulm, Germany
%{\tt\small \{alexander.tsaregorodtsev,johannes-christian. mueller, martin.herrmann, jan.strohbeck, michael.buchholz, vasileios.belagiannis\} @uni-ulm.de} }%
%}
\thanks{$^{1}$The authors are with the Institute of Measurement, Control and Microtechnology,
Ulm University, D-89081 Ulm, Germany \newline
\{$<$first\,\,name$>$.$<$family\,\,name$>$@uni-ulm.de\}
}
\thanks{$^{2}$Vasileios Belagiannis is with the Department of Simulation and Graphics,
University Magdeburg, D-39106 Magdeburg, Germany \newline
\{vasileios.belagiannis@ovgu.de\}. Most of this work was done during his time at Ulm University.
}}%
\begin{document}
\newcommand\copyrighttextinitial{%
\scriptsize This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.}%
\newcommand\copyrighttextfinal{%
\scriptsize\copyright\ 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.}% DOI: 10.1109/ICRA48506.2021.9561375.}%
\newcommand\copyrightnotice{%
\begin{tikzpicture}[remember picture,overlay]%
\node[anchor=south,yshift=10pt] at (current page.south) {{\parbox{\dimexpr\textwidth-\fboxsep-\fboxrule\relax}{\copyrighttextfinal}}};%
\end{tikzpicture}%
%\vspace{-10pt}%
}
\maketitle
\copyrightnotice%
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information from images and point clouds. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle
with high-precision
localization to capture a point cloud of the camera environment. Afterward, a mapping between the camera and world coordinate spaces is obtained by performing a lidar-to-camera registration of the semantically segmented sensor data. We evaluate our method on simulated and real-world data to demonstrate low error measurements in the calibration results. Our approach is suitable for infrastructure sensors as well as vehicle sensors, while it does not require motion of the camera platform.
%in order to assess calibration quality and demonstrate our approach in a real use case.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{INTRODUCTION}
Monocular cameras are a keystone to cost-efficient sensor setups \cite{rinner2008introduction} for both intelligent vehicles (IVs) and smart infrastructure, supporting automated driving. The calibration of such cameras is vital to their operation out in the field and, therefore, of utmost importance for applications such as self-localization~\cite{engel2019deeplocalization} or recognition~\cite{wiederer2020traffic}. However, while the intrinsic camera calibration can easily be accomplished once in a constrained environment, the recovery of the extrinsic calibration in the field or between vehicle sensors is still a challenging process. In particular, classical calibration methods \cite{andrew2001multiple} often require manually placed calibration targets in the scene. These approaches, therefore, cannot easily be applied to traffic cameras, observing densely populated junctions or to IVs out in the field. Thus, an automatic calibration solution, which works for single infrastructure cameras, is needed to reduce calibration effort. Such a solution can be used to calibrate cameras mounted on IVs as well.
%Lidar sensor calibration has been recently automated with the support of an intelligent vehicle that composes the calibration target~\cite{muller2019laci}. Nevertheless, this approach is not applicable to RGB cameras. In turn
Recently, sensor-to-sensor registration methods, e.g. \cite{datondji2016rotation, zhu2016robust}, have been proposed to automate the calibration process. However, these methods often require broadly overlapping fields of view (FOVs), which implies redundant sensors and hence negates the cost savings achieved by a single sensor. For IVs, specific fully automated calibration schemes, such as \cite{horn2021online}, have been developed, which rely on the motion of the sensor platform, i.e., the IV, to optimize the calibration during each time step.
\begin{figure}
%\centering
\input{Kalib.pdf_tex}
\caption{Approach Overview. On the left side, data from both domains are fed into the pipeline. For lidar data, the 3D environment is reconstructed using multiple scans. Afterward, both domains are labeled semantically. In the final step, our proposed optimization is performed to register both domains to retrieve the extrinsic calibration parameters.}
\label{fig:PipelineOverview}
\end{figure}
~A caveat of these specific calibration schemes is their dependency on sensor motion, which excludes static infrastructure from the range of possible applications. Deep learning approaches like \cite{li2021deepi2p} have recently been developed to estimate extrinsic camera parameters from camera images and lidar data. While these approaches have shown promising results, they still tend to have higher estimation errors compared to previous methods like \cite{horn2021online}, making them unsuitable for IV and infrastructure calibration. Finally, another classical method is described in \cite{ataer2014calibration}. It localizes a camera sensor inside of a 3D model of the environment taken from a SLAM with stereo cameras.
However, this method obtains its 3D environments from a mobile stereo camera setup and thus contains texture information not present in lidar data. %An additional hindrance is their use of a RGB-D-camera setup on a mobile platform which does not provide RTK data, resulting in lower localization accuracy.
~Furthermore, this approach is not transferable to stereo camera setups permanently mounted on IVs due to their limited field of view.
We propose a monocular camera calibration approach to perform cross-domain registration of a semantically segmented mono camera image and a semantically labeled 3D model of the environment the camera is located in. An overview of the approach is given in Fig. \ref{fig:PipelineOverview}. %Such 3D point cloud-based models may be available from mapping providers or obtained from a simultaneous localization and mapping (SLAM) algorithm running on an IV with lidar data.% or taken from a single shot lidar measurement when performing IV cross-sensor calibration.
By using a semantic representation of both the camera and the point cloud domain, we introduce semantic labels, which can be matched between the domains. Furthermore, the use of semantic labels instead of raw RGB and RGBD camera data makes the registration more robust and reduces the possibility of registration algorithm errors. These can occur with raw camera data due to color space and exposure differences between different camera types.%The semantic features are extracted automatically using neural networks.
%Furthermore, our approach works with a single camera and doesn't require motion of the camera platform, thus eliminating some of the constraints introduced by other methods.
Our approach consists of data pre-processing steps as well as an optimization step performing cross-domain registration. In the pre-processing steps, the 3D environment model is reconstructed and then labeled semantically using neural networks. The segmentation is also performed on images of the target camera that we aim to calibrate. Then, an optimization pass is invoked to match the visual appearance and projection of a rendered view of the 3D model with the segmented camera view. % then yields the extrinsic parameter estimates.
This optimization then yields the extrinsic camera calibration.% that is the main contribution of our paper.
~Finally, experiments with the CARLA simulator \cite{Dosovitskiy17} and the KITTI dataset \cite{geiger2013vision} show that our method leads to the accuracy level that is suitable for automated driving applications, both for static and dynamic camera platforms. The contribution of this paper is twofold. From a practical perspective, our approach allows for low-effort, highly automated calibration of monocular cameras in (geo-referenced) world coordinates without requiring motion of the target sensor platform.
From a methodological perspective to the best of our knowledge, we are the first to use semantic segmentation as a visual feature set to guide the required parameter optimization. The source code of our approach is publicly available\footnote{\url{https://github.com/Tuxacker/semantic\_calibration}}.
The related work to extrinsic camera calibration is outlined in Section II. The complete algorithm is described in Section III, while in Section IV, a theoretical and practical evaluation is provided. At last, the conclusion of this work is described in Section V.
%\ReviewChange{The rest of the article is structure as follows: the complete algorithm is described in Section II, while in Section III, a theoretical and practical evaluation is provided. Finally, the conclusion of this work is described in Section IV.}
%In recent years, a lot of progress in the area of traffic monitoring and connected road infrastructure has been made. Initially, traffic monitoring equipment was developed in order to measure traffic flow \ReviewChange{!!, register traffic offenses!!}, to redirect traffic\ReviewChange{!! and control traffic light phases!!}. Due to recent progress made in the area of automated driving, road-side infrastructure is becoming even more important. By using infrastructure, automated driving function can be assisted in a meaningful way. An example would be traffic lights sending their current and future state, which enables optimized trajectory planning and prevents unnecessary braking maneuvers!!. Another use case, which is designed to increase the safety of all traffic participants, are digital mirrors!!. By equipping light poles with camera sensors, road users can be detected to create an external environment model which can then be sent to and used by an automated vehicle to plan more efficient and safe maneuvers, especially in highly occluded environments like intersections in urban environments, where no traffic lights are installed and the view is blocked by neighboring buildings.
%As the importance and adoption of roadside infrastructure is increasing, its cost, simplicity and setup time becomes important. Especially, if the goal is to provide environment data using relatively inexpensive camera sensors, the calibration of such cameras becomes increasingly important. The calibration of infrastructure is needed!! to the extent that environment data has to be provided in a world reference frame instead of pixel coordinates that are usually provided by camera detectors. If the calibration process requires too much manual work and setup time, then it becomes very difficult to achieve widespread adoption of such systems. Current systems mostly rely on homography estimation through manual annotation!! or image registration with georeferenced sattelite imagery!!. However, as this homography approach merely maps 2D coordinate frames, it inherently limits the accuracy of the resulting data, as all data points are assumed to be in 2 dimensions, e.g. a ground plane. While sufficiently accurate for flat topologies, this approach introduces large errors, if points of interest are on elevated surfaces or hillsides, especially in the far field of the camera. On the other hand, using annotated or georeferenced source material introduces another error source. If the source data contains some deviations, its impact on the result may be immense, as cameras tend to have high error heteroscedasticity. Therefore, if the depth dimension of an image point (usually the y-axis) is incorrect by a couple of pixels, it may result in a real world deviation of over one meter. The high calibration effort and error susceptability, however, can be easily avoided to produce more accurate and meaningful results. We present a new calibration approach for infrastructure camera calibration using automated vehicles. As automated vehicles are providing high quality sensor data and localization information, this data can be used to provide reference features to perform feature matching and registration.!!Mention that we don't need camera motion!! Our main contributions consist of:
%\begin{itemize}
% \item !!(This is technically not a contribution)!!A fully automated data processing pipeline which reconstructs the 3D enviroment of a camera using lidar sensors and also performs semantic segmentation of point cloud data as well as image data
% \item Use of a differentiable renderer to convert point cloud into image data to implement cross-domian registration
% \item Two-stage optimization using coordinate descent to perform coarse calibration and loss backpropagation to achieve fine alignment
% \item Evaluation on simulated data as well as on real-world infrastructure
%\end{itemize}
%
\section{RELATED WORK}
Calibration of cameras has been extensively investigated in the past. A classical approach for intrinsic and extrinsic camera calibration is presented in \cite{andrew2001multiple}. One of the first approaches specifically for infrastructure sensors was described in \cite{datondji2016rotation}, where a vehicle target was used to calibrate a pair of fisheye cameras with a large overlapping field of view. Another approach using planar structure from motion and textured environments was presented in \cite{zhu2016robust}, which is able to calibrate multi-sensor configurations. The approach of \cite{ataer2014calibration} circumvents the field of view (FoV) overlap requirement of the other two approaches by reconstructing a 3D environment using a mobile RGB-D camera setup. Then, a matching of feature descriptors between the target camera image and the model is performed and the closest matching keyframes are found to determine the camera pose. The automatic calibration method of \cite{dubska2014fully} estimates the camera vanishing points by tracking image features of moving vehicles, while a newer method by \cite{bartl2021automatic} uses a neural network to find landmark positions on vehicles. By using the positional relations between landmarks obtained from real 3D models of the vehicles, a calibration is estimated from the landmarks.
Another class of algorithms performs direct 2D to 3D matching and registration. Feng et al. \cite{feng20192d3d} use a neural network to directly match point cloud patches to image patches, while the algorithm in \cite{yu2020monocular} performs camera localization by finding correspondences between 2D and 3D line features. Li et al. \cite{li2021deepi2p} train a classifier to detect if a point in the point cloud is in the camera frustum and then solve the inverse camera projection problem using the frustum labels.
Methods performing calibration through sensor motion can also be found in the literature (\cite{horn2021online, wei2018calibration, wise2020certifiably}). In particular, Horn et al. \cite{horn2021online} use a visual SLAM approach to estimate the camera displacement and then represent the translation and rotation using dual quaternions instead of homogeneous transformation matrices, which allows for a more efficient implementation of the optimization. The optimization consists of finding a calibration matrix that is able to close the loop between the current and previous time step, where the calibration matrices of the previous time step are known.\\%However, approaches \cite{feng20192d3d} and \cite{li2021deepi2p} result in high position errors of the resulting matching, making it unsuitable for assisted driving tasks. \cite{yu2020monocular} on the other hand, while having a low final error, relies on camera movement during multiple time steps to improve and smooth its results, making it impossible to use for static sensor environments.\\
A recent method using automated vehicles as calibration targets is shown in \cite{muller2019laci}. It uses lidar measurements in an empty scene to determine the ground plane and background and then regresses the position of the automated vehicle as it passes through the sensors FoV with localization data of the vehicle being available.
We improve on previous methods by exploring a new concept for matching cross-domain features by visual alignment, which does not require specific calibration targets or motion of the sensor itself. It, therefore, allows us to perform a single-shot calibration in diverse scenarios without requiring a lot of manual intervention.
\section{CALIBRATION METHOD}
%Before showing an overview over all steps of the calibration pipeline and describing every step in detail, a brief problem formulation is provided and all assumptions that are needed for the algorithm to function are described.
In this section, we formulate our problem and present each step of our approach required to estimate the extrinsic camera calibration.
\subsection{Problem Formulation and Constraints}
Consider the perspective camera model \cite{andrew2001multiple}
\begin{equation}
%\underset{\in \mathcal{I}}
{\begin{bmatrix}
u \\ v \\ 1
\end{bmatrix}} =
\underbrace{\boldsymbol{K}_\text{int} \;
\begin{bmatrix}
\boldsymbol{R} & \boldsymbol{t} \\
\boldsymbol{0} & 1
\end{bmatrix}}_{:= \boldsymbol{P}}
%\underset{\in \mathcal{M}^{3\text{D}}}
{\begin{bmatrix}
x \\ y \\ z \\ 1
\end{bmatrix}} \; ,
\end{equation}
where $[u, v, 1]^T$ denotes a homogeneous camera image coordinate of an image $\mathcal{I}$ with pixel color value $\mathcal{I}_{u,v} \in [0;1]^3$, and $[x, v, z, 1]^T\in\mathcal{M}^{3\text{D}}$ is a homogeneous point coordinate of a 3D model $\mathcal{M}^{3\text{D}}$ of the observed scene. $\mathcal{M}^{3\text{D}}$ is represented as a point cloud. Furthermore, it is assumed that the scene visible in the camera image is overlapping with the scene represented in the point cloud.%and $\mathcal{M}^{3\text{D}}(x,y,z)$ being the semantic label of the respective point.
Our goal is to recover the parameters of the extrinsic camera matrix $\boldsymbol{P}$, i.e., the rotation matrix $\boldsymbol{R} \in \mathbb{R}^{3 \times 3}$ and the translation vector $\boldsymbol{t} \in \mathbb{R}^{3}$. At the same time we assume that the intrinsic parameters $\boldsymbol{K}_\text{int} \in \mathbb{R}^{3 \times 4}$, which represent the mapping from 3D coordinates in the camera coordinate frame to 2D pixel coordinates, are already known, e.g.~as described in~\cite{andrew2001multiple}. Below, we propose a new calibration approach to estimate these extrinsic camera parameters.
%The goal of the extrinsic camera calibration is to recover the camera matrix $\boldsymbol{P}$, i.e. the rotation matrix $\boldsymbol{R} \in \mathbb{R}^{3 \times 3}$ and the translation vector $\boldsymbol{t} \in \mathbb{R}^{3}$, where for this work, the intrinsic parameters $\boldsymbol{K}_\text{int} \in \mathbb{R}^{3 \times 4}$ are assumed to be given. The parameters of $\boldsymbol{K}_\text{int}$ are assumed to be known or can be determined by the method described in \cite{andrew2001multiple}.
%Let $\mathcal{C}$ be a camera, which is modeled as a perspective pinhole camera with a known intrinsic parameter matrix!!maybe swap equation order
%\begin{eqnarray}
% \mathcal{K}_{\mathrm{int}} = \begin{bmatrix}
% \frac{z_n}{x_\mathrm{max}} & 0 & 0 & 0 \\
% 0 & \frac{z_n}{y_\mathrm{max}} & 0 & 0 \\
% 0 & 0 & \frac{z_f}{z_f - z_n} & -\frac{z_f z_n}{z_f - z_n} \\
% 0 & 0 & 1 & 0 \\
% \end{bmatrix},\\
% y_\mathrm{max} = z_n \cdot \tan(\frac{\alpha}{2}), x_\mathrm{max} = z_n \cdot \tan(\frac{\alpha}{2}) \cdot \frac{w}{h}
%\end{eqnarray}, where $\alpha$ is the vertical FoV of the camera, $z_n$ and $z_f$ are the near and far clipping planes of the camera and $w$ and $h$ is the horizontal and vertical resolution of the camera!!perspective projection!!. It should be noted that the clipping plane parameters are required by the rendering algorithm and are not actually present in real cameras, so they can be chosen from a reasonable range, as will be shown later. Our goal is to determine the extrinsic camera matrix
%\begin{equation}
% \mathcal{K}_{\mathrm{ext}} =\begin{bmatrix}
% \mathbf{R}^{\alpha\beta}_\mathrm{ext} & \mathbf{t}_\mathrm{ext} \\
% \mathbf{0} & 1 \\
% \end{bmatrix} \in\mathbb{R}^{4\times 4},
%\end{equation}
%which has 3 degrees of freedom in position $\mathbf{t}_\mathrm{ext}$ as well as two rotational degrees of freedom encoded in $\mathbf{R}^{\alpha\beta}_\mathrm{ext}$, while assuming the roll of the camera to be zero, as cameras are normally installed in such a way that the lower edge of the image is parallel to the ground plane. It should be noted that this approach can be extended to estimate the roll as well if needed.\\
%As our approach requires an initial guess of both position and rotation, these parameters can be easily measured manually. The x and y position (latitude and longitude) can be easily measured using a cheap GPS receiver found in smartphones, while the height can be measured with a laser rangefinder. The yaw can be obtained from a compass measurement, while the pitch can be set manually on the mounting plate if the plate is then mounted leveled with the ground. Alternatively, the pitch can be simply estimated "by eye". Furthermore, a vehicle equipped with lidar and IMU/RTK sensors is required in order to record a georeferenced point cloud. As all manual measuring methods are very coarse and imprecise, we use our approach to optimize the initial guess and remove high deviations of multiple meters and multiple degrees.
\subsection{Approach Overview}
We use visual matching of semantic information for the parameter estimation of the camera rotation $\boldsymbol{R}$ and translation $\boldsymbol{t}$.
%To be independent of gradient information, we use the simplex optimization method of Nelder and Mead \cite{nelder1965simplex} for finding an optimal parameter set. To recover $\boldsymbol{R}$ and $\boldsymbol{t}$, a reconstructed 3D model of the camera environment is rendered into an image $\hat{\mathcal{I}}$ using an estimate of the camera matrix $\hat{\boldsymbol{P}}$ and a semantic color coding of the respective model components $\mathcal{M}^{3\text{D}}(x,y,z)$. This image is then matched with a semantically segmented camera image.
Our method consists of four steps, starting with the 3D environment reconstruction $\mathcal{M}^{3\text{D}}$, represented as a point cloud (see Sec.~\ref{lbl:3drec}). It is recorded with the lidar sensor mounted on a vehicle. Next, the point cloud segmentation step (presented in Sec.~\ref{lbl:pclseg}) follows to extract semantic information for each 3D point. Then, the camera image segmentation (see Sec.~\ref{lbl:camseg}) assigns a class category to each pixel. Given the segmented 3D point cloud and image, we propose a cross-domain registration approach (explained in Sec.~\ref{lbl:cdreg}) to estimate the extrinsic camera parameters by visually matching the camera segmentation with a rendered point cloud view using linear programming~\cite{nelder1965simplex}.%A graphical overview of our approach is presented in Fig.~\ref{fig:PipelineOverview}.%Next, we discuss each of the four steps in detail.
%The processing pipeline is illustrated in Figure~\ref{fig:PipelineOverview} and consists of the following steps:
%\begin{enumerate}
% \item 3D environment reconstruction (\ref{lbl:3drec}): A 3D environment model $\mathcal{M}^{3\text{D}}$ is obtained, e.g. by environment reconstruction using a lidar SLAM of an IV.
% \item Point cloud segmentation (\ref{lbl:pclseg}): For each point in $\mathcal{M}^{3\text{D}}$, a semantic label is generated using a deep neural network Cylinder3D.
% \item Camera image segmentation (\ref{lbl:camseg}): Semantic labels are determined for each pixel of the image taken by the calibration target.
% \item Cross-domain registration (\ref{lbl:cdreg}): The extrinsic camera parameters are estimated by solving \eqref{eq:OptimizationProblem} using the simplex method \cite{nelder1965simplex}.
%\end{enumerate}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{example-image}
% \caption{Pipeline Overview}
% \label{fig:PipelineOverview}
% \end{figure}
%
%The main goal of this approach is to reconstruct a 3D environment that lies inside of the cameras FoV, then convert both the 3D and 2D data into a semantic representation in order to have the same features in both domains and finally create an optimization loop which uses an initial guess for the extrinsic camera matrix $\mathcal{K}_{\mathrm{ext}}$ to transform 3D data to 2D data and then optimizes the matching between the original and transformed image. The pipeline consists of the following steps:
%\begin{enumerate}
% \item 3D environment reconstruction using a lidar- and IMU-equipped automated vehicle
% \item Semantic segmentation of 3D point cloud and 2D image
% \item Coarse cross-domain registration using coordinate descent
% \item Fine cross-domain registration using backpropagation
%\end{enumerate}
%
%Each step will be now described in detail.
\subsection{3D Environment Reconstruction}
\label{lbl:3drec}
%\ReviewChange{This is still quite lengthy. We may shorten this subsection, if we require space...}
%In the first step, a 3D environment model $\mathcal{M}^{3\text{D}}$ of the observed scene is constructed. Ideally, this model should cover the same space that is observed in the field of view of the camera, however, as described in Sec. \ref{lbl:cdreg}, even data with partial coverage can be used for this purpose with minimal to no loss in calibration quality.\\
%In the simplest case, $\mathcal{M}^{3\text{D}}$ is already available from the digital map of the IV. Otherwise, if only a camera-to-lidar calibration is desired, as is the case with IV sensor calibration, one or multiple lidar scans with $360^\circ$ surround view can be used directly for calibration. Using multiple scans increases the point density and thus improves the resulting calibration precision.
We assume access to a vehicle equipped with a lidar sensor and high-precision localization to construct the 3D environment model $\mathcal{M}^{3\text{D}}$. Several lidar scans are recorded as point clouds and expressed in the world coordinate frame. The transformation from lidar sensor coordinates to world coordinates requires known lidar extrinsics. %the automated vehicle to be equipped with an RTK/IMU localization system and the transform from lidar coordinates to the RTK coordinates in reference to the vehicle has to be known and provided by the vehicle.
Due to road bumps and inaccurate orientation measurements, points in the sensor's far field come with increased positional errors, resulting in noisy 3D point clouds with many outliers. Thus, we filter out points that exceed the maximum distance $d_\text{max}$ from the vehicle's lidar sensor. In practice, $d_\text{max}$ is chosen to be between $50~\text{m}$ and $75~\text{m}$, which are standard cutoff values for Velodyne sensors like the VLP-32C or the HDL-64E used in the KITTI \cite{geiger2013vision} dataset. Next, we rely on the multi-way Iterative Closest Point (ICP) registration algorithm~\cite{chen1992object, choi2015robust} between all scans, which partially compensates for localization errors that accumulate over time. To accelerate the registration, instead of registering every scan against every other scan, all scans are sorted by their capture time, and groups of three neighboring scans are recursively registered and merged. The recursion ends when all groups are combined into the single point cloud $\mathcal{M}^{3\text{D}}$. At last, the point cloud is down-sampled during each recursion step in order to relax the ICP computation requirements. The reconstruction step can be skipped if the 3D environment is obtained by another mapping algorithm or from a different source.%is extracted from the resulted point cloud.
%In the more general case of global camera calibration, each lidar scan has to be registered to the world frame, i.e. to global coordinates before calculating the calibration. This is especially important for infrastructure calibration.\\
%For this purpose, $\mathcal{M}^{3\text{D}}$ can be generated with an IV, which has a $360^\circ$ surround view lidar for perception and a real-time kinematic system for localization aboard, or directly by using SLAM methods. In this case, the lidar scans are transformed into the world frame using the lidar-to-vehicle calibration and the ego localization result of the IV.
%Due to road bumps and inaccurate measurements of vehicle pitch and roll, a lot of points in the sensors' far field come with high positional errors, creating noisy clouds and ghost objects.
%To remedy this issue, two solutions are applied. \\
%The first solution consists of filtering out lidar measurements that exceed a maximum distance $d_\text{max}$ from the IV sensor.$d_\text{max}$ is chosen to be between $50~\text{m}$ and $75~\text{m}$, as points beyond this range are heavily affected by rotation errors of the world transform.\\
%As the second solution we use a multiway ICP registration \cite{chen1992object, choi2015robust} between all scans, which also partially compensates localization errors accumulating over time.
%To make the registration real-time capable, instead of registering every scan against every other scan, all scans are sorted by their capture time and groups of $3$ neighbouring scans are recursively. The recursion ends when all groups are combined into one single point cloud $\mathcal{M}^{3\text{D}}$ using the ICP registration results.
%Simultaneously, the point cloud is downsampled to keep its size small and speed up the ICP.
%After these processing steps, the resulting point cloud has significantly less outliers, which makes the following cross-domain registration more stable.
%\subsection{3D environment reconstruction and pre-processing}
%
%!!There are many methods to do this in literature already!! In the first step we want to reconstruct the neighborhood of the camera we want to register. To achieve this, multiple scans of a \ang{360} lidar sensor mounted on a vehicle are recorded in the area, where the camera is located. While one single scan could suffice in theory, the resulting scans point cloud density would not suffice to perform adequate matching. Therefore multiple scans are aggregated while the vehicle is driving. In addition to recording the scans, GPS/IMU data is recorded as well. After recording the data, each scan is transformed into world space by using GPS data as well as a sensor-to-vehicle transform, which should be saved in the automated vehicle as an internal calibration. By combining all scans in world space, a digital map of the environment is created. However, due to road bumps and inaccurate measurements of pitch an roll, a lot of points in the far field of the sensor have high positional deviations, creating noisy clouds and ghost objects. To remedy this issue, two solutions are applied. An easy way to filter points is to limit the lidar range to a value between 50m and 75m, as points in the near field are less affected of rotational offsets of the target coordinate frame. Another step that was added is a multiway registration between all scans, which also partially compensates errors of the IMU system that accumulate over time. To make the registration real-time capable, instead of registering every scan against each other, a divide-and-conquer approach is used. In our approach, we ordered all scans by their capture time and then recursively registered bundles of 3 neighboring scans to then combine the scans according to the registration result while simultaneously downsampling the point cloud in order to keep its size small and make registrations fast. The recursion ends when all bundles are combined into one single point cloud. The registered point cloud has severely less ghost objects and outliers, which makes the following cross-domain registration more stable. In order compress the point cloud size even more, a region of interest overlapping the cameras' field of view can be cropped out, as an initial guess for the camera position and orientatioon needs to be available.
\subsection{Point Cloud Segmentation}
\label{lbl:pclseg}
In the next step, semantic labels are assigned to the points of $\mathcal{M}^{3\text{D}}$ to obtain the semantic segmentation map $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$, which contains a color label for each point in $\mathcal{M}^{3\text{D}}$. For this task, we rely on a pre-trained deep neural network to perform semantic segmentation, namely Cylinder3D~\cite{zhu2021cylindrical}, \ReviewChange{which supports segmentation of classes defined in SemanticKITTI~\cite{behley2019semantickitti}}.%The advantage of \cite{zhu2021cylindrical} consists in its approach to use a cylindrical partition of the point cloud for label generation, which mimics the sensor model of a \ang{360}-lidar and thus achieves better results on such lidar scans. By using a neural network, no further human input is required, making this step fully automatic.% For point clouds data, convolutional networks for sparse voxel data that use a cylindrical representation of point clouds (e.g. \cite{zhu2021cylindrical}) can be used to fully automate the labelling process.
\\Furthermore, the segmentation is also used to filter out dynamic object classes like cars and pedestrians. %, as well as semantics that may not exist in image segmentation,
This leaves the buildings, ground points, vegetation, fences, poles, and traffic signs classes for registration.
Therefore, the calibration method does not require exact time synchronicity between the lidar scans and the camera image, as points belonging to dynamic objects can easily be filtered out by their label. %However, at least the seasons have to coincide, as the shape of vegetation changes between seasons and may introduce registration errors.
%Depending on the scene, it may be beneficial to not filter out parked cars from the scene, as they may aid in the following registration process and produce better results. In this case, however, approximate time synchronicity is required.\\
\subsection{Camera Image Segmentation}
\label{lbl:camseg}
After labeling the point cloud $\mathcal{M}^{3\text{D}}$, the semantic segmentation map $\mathcal{I}^\text{sem}$ is extracted from the camera image $\mathcal{I}$. Similar to the point cloud segmentation, we rely on a pre-trained deep neural network to extract semantic labels \ReviewChange{available in the Cityscapes dataset~\cite{cordts2016cityscapes}}, e.g.,~OCRNet~\cite{yuan2021segmentation}. \ReviewChange{We chose to use Cityscapes labels, as they share most classes available in SemanticKITTI and thus enable direct matching of class labels between domains.} Again, dynamic object class categories, e.g.,~cars or pedestrians, are removed from the segmentation map. For the remaining object categories, we perform a category alignment between the point cloud~\cite{zhu2021cylindrical} and the image~\cite{yuan2021segmentation} segmentation models by neglecting points and pixels with labels that are unique to this domain. It should be noted that the removal of dynamic objects results in holes in the resulting map, especially in crowded scenes. To minimize their impact, we introduce the normalization factor $\beta$ during the registration step in Section~\ref{lbl:cdreg}.%For instance, some classes that are not always well-defined (especially in point clouds), e.g. road, sidewalk and parking can be unified to one "road/ground" class.
The image segmentation can also be done semi-automatically to improve calibration quality. In this case, the resulting segmentation is manually edited to fix incorrectly labeled regions in order to improve data quality and hence reduce the overall calibration error.
%This manual editing is especially useful if the segmentation network is trained for cameras mounted within an IV, but the calibration target is the camera component of a smart infrastructure system observing the traffic from above. In this case, the segmentation network may not produces good result due to the large domain shift caused by the different viewpoint.
%In this step, features are extracted from both the image and the point cloud domain. An easy feature type that can be extracted from both domains and also creates big connected regions of the same feature is a semantic label feature. Texture data from the camera cannot be used as the lidar data does not contain color information, therefore making registration difficult. Using semantic labels as features also has the benefit that certain labels can be filtered out, which is especially important to filter out dynamic objects like cars and pedestrians. During the point cloud prep-rocessing, dynamic objects in the map do not get filtered out, resulting in ghost trails which create incorrect semantic 2D maps that are much harder to match. Furthermore, semantic labeling can help ignore holes that are caused by point cloud sparsity as well as help identifying the ground plane and driveable area. For segmentation, neural networks are used as they show high segmentation accuracy. For point clouds, Cylinder3D \cite{zhu2021cylindrical} using Semantic-KITTI labels \cite{behley2019semantickitti} was applied, which uses a sparse convolutional network and a cylindrical representation of point clouds. For camera image segmentation, OCRNet \cite{yuan2021segmentation} with Cityscapes labels \cite{cordts2016cityscapes} was used. For our use case, we filtered out labels that are the same between both datasets as well as removing all dynamic objects. Furthermore, some classes that are not always well-defined (especially in point clouds), e.g. road, sidewalk and parking were unified to one "road" class. During this phase, manual work may be required if the camera image segmentation is not precise enough. The lower accuracy of infrastructure camera images stems from the fact that segmentation datasets like Cityscapes are recorded from the perspective of an automated vehicle, making the camera views heavily biased toward ground level perspectives. Training on a dataset containing images from real world infrastructure can severely improve performance, however due to the immense work required to create such a dataset, the raw semantic map was manually improved to better reflect reality, which only takes a couple of minutes for one camera.\\
%After this processing, a filtered and labeled point cloud $\mathcal{P}_\mathrm{sem}$ as well as a semantic map of the camera $\mathcal{I}_\mathrm{sem}$ are available.
\subsection{Cross-domain registration}
\label{lbl:cdreg}
%The cross-domain registration is the main step in the calibration pipeline.
Next, we aim to estimate the rotation matrix $\boldsymbol{R}$ and translation vector $\boldsymbol{t}$. We formulate this estimation problem as an optimization, where a loss correlated to the error $\lVert\hat{\boldsymbol{R}} - \boldsymbol{R}\rVert$ and $\lVert\hat{\boldsymbol{t}} - \boldsymbol{t}\rVert$ is minimized using the simplex algorithm by Nelder and Mead \cite{nelder1965simplex}.
%The optimization algorithm is well-proven in practice, comes with the benefit that it does not require gradient information, and is well-suited for low-dimensional problems like ours, as we only aim to estimate 5 parameters. Hence,
As the Nelder-Mead method does not require gradient information, it allows a broad range of data transforms to be applied to $\mathcal{M}^{3\text{D}}$. This includes rendering transforms, which can be used to obtain a rasterized image $\hat{\mathcal{I}}$ of a specific perspective view of $\mathcal{M}^{3\text{D}}$. We, therefore, define the rendering function $\boldsymbol{f}$ as
\begin{equation}
\hat{\mathcal{I}} = \boldsymbol{f} \big(\mathcal{M}^{3\text{D}},\boldsymbol{P}(\hat{\boldsymbol{R}},\hat{\boldsymbol{t}}), \mathcal{M}^{3\text{D}}_{\boldsymbol{c}} \big) \; ,
\end{equation}
where $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$ is the color encoding of the segmentation and $\hat{\boldsymbol{R}}$, $\hat{\boldsymbol{t}}$ are estimates of $\boldsymbol{R}$ and $\boldsymbol{t}$. This rendering function is used to transform the point cloud $\mathcal{M}^{3\text{D}}$ with color labels $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$ into an image $\hat{\mathcal{I}}$ with the viewpoint defined by the perspective camera matrix $\boldsymbol{P}(\hat{\boldsymbol{R}},\hat{\boldsymbol{t}})$. $\hat{\mathcal{I}}$ can then be visually matched with the previously obtained image segmentation $\mathcal{I}^\text{sem}$ by calculating a distance metric between both images and interpreting the result as a loss value. By adjusting $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$ to minimize the visual difference between these images, and therefore minimizing the loss value, we actively determine the optimal camera parameters. An advantage of the Nelder-Mead method lies in its use of a simplex to define the initial search space, which then moves and shrinks towards the function minimum. By scaling the initial simplex bounds according to the expected measurement variance of the initial guess, we can easily define a sensible search space covered by the starting simplex. Note that other gradient-free approaches may be used as well.
\paragraph{Initialization}
In order to start the optimization, the Nelder-Mead method requires an initial parameter set. The convergence speed of the algorithm and its ability to find an optimal solution depend on the initial parameter set and the search space.
Therefore, an initial guess for the camera position and orientation is required.
We furthermore constrain the search space of the pitch and yaw angle to $\pm 5^\circ$, while we set the roll angle to zero. We expect our method to also work for non-vanishing roll angles, but in our use cases, no camera roll is desired and can be achieved by proper mounting. This is a reasonable simplification since potential camera roll is often not desired for automotive applications. %and thus normally prevented by the mounting mechanism.
~The yaw angle can be measured with a compass, while the pitch can be determined with a spirit level. The maximal position offsets are constrained to $\pm2.5\,\si{\meter}$, i.e., the GPS accuracy of a typical smartphone GPS receiver.
We argue that an initial guess with such accuracy can be easily obtained for most practical situations and is sufficient to obtain a good solution for the extrinsic calibration problem, as shown in Sections~\ref{sec:ep} and~\ref{sec:rd}.
However, it is possible to extend the search space at a tolerable increase in calculation effort by combining the Nelder-Mead method with a coarse grid search to estimate parameters that were not measured and are therefore not known. If an IV sensor-to-sensor calibration is desired, no initial measurements are required in most cases, as the convergence range of our approach is sufficient to bridge the translation and rotation between both sensors. Therefore, they can be set to be zero vectors in such cases. After setting the initial guess, we also have to calculate the initial simplex, which will converge to the optimal solution. To this end, we generate 6 values (as a simplex of dimension $n$ is described by $n+1$ points) by adding and subtracting the maximal constrained parameter range for each pair of simplex vertices and for each parameter.
\paragraph{Optimization Loop}
The Nelder-Mead method is invoked until the loss distance between two steps falls below $10^{-4}$ in order to estimate $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$.
The loss target we optimize is defined as
\begin{equation}
\label{eq:Loss}
%\hat{\boldsymbol{R}}, \hat{\boldsymbol{t}} =
\mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem}) = \beta \sum_{u=0}^{h-1} \sum_{v=0}^{w-1} \mathcal{V}_{u,v}\, \ell_{\mathrm{L2}}(\hat{\mathcal{I}}_{u,v},\,\mathcal{I}^\text{sem}_{u,v})
\end{equation}
with the L2 loss term $\ell$
\begin{equation}
\ell_{\mathrm{L2}}(\boldsymbol{a},\, \boldsymbol{b})=(\boldsymbol{a}- \boldsymbol{b})^T(\boldsymbol{a}-\boldsymbol{b}).
\end{equation}
Here, $\beta$ is a normalization factor, $w$ and $h$ denote the width and height of the images,% $\ell$ is a metric for measuring a distance between two pixel values,
~and $\mathcal{V}_{u,v}$ is a function checking the validity of the classes. If one of the images contains an invalid class for a pixel, $\mathcal{V}_{u,v}=0$, i.e., the pixel is ignored for loss calculations. Otherwise, $\mathcal{V}_{u,v}=1$ implies a contribution to the final loss. It should be noted that, while in theory, any distance metric may be used for $\ell$, we decided to use an L2 loss, as this loss is commonly used as an image similarity measure. Finally, the optimization performed by the Nelder-Mead method can be represented as
\begin{equation}
\label{eq:OptimizationProblem}
%\hat{\boldsymbol{R}}, \hat{\boldsymbol{t}} =
\hat{\boldsymbol{R}}_{\mathrm{opt}}, \hat{\boldsymbol{t}}_{\mathrm{opt}} = \arg \min_{\hat{\boldsymbol{R}}, \hat{\boldsymbol{t}}}\left\{ \mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem}) \right\} .
\end{equation}
%As can be seen form the loss structure, we opted to provide two possible loss types to compare the visual appearance of the color-encoded image segmentation $\mathcal{I}^\text{sem}$ target to a rendered representation of the point cloud segmentation $\hat{\mathcal{I}}$. Both losses are suitable for image comparison, however the Huber loss is more resistant to outliers and may provide better results in scenes containing more incorrect labels.
\paragraph{Appearance Matching and Masking}
Before optimizing the camera pose, we want to closely match the general appearance of the rendered image $\hat{\mathcal{I}}$ to the target $\mathcal{I}^\text{sem}$ as well as reduce rendering artifacts. For that reason, we first aim to minimize the appearance of background pixels due to the sparsity of points in the 3D model. This is achieved by calculating the distance $d_i$ of every point in $\mathcal{M}^{3\text{D}}$ to the camera position. Then, each point in the 3D model is rendered as a circle of radius $r_i=\lambda/d_i$, where $\lambda$ is a scaling factor that depends on the point cloud density and can be determined empirically by rendering a test view and increasing $\lambda$ until the appearance of the rendered view roughly matches the target segmentation image. \ReviewChange{This adaptive point size approach also adds the benefit of limiting the impact of varying line counts between different lidar models, as lidars with fewer lines can still be densely rendered by increasing $\lambda$}. Secondly, pixels, which still cannot be assigned with a semantic label corresponding to a static object, e.g., due to point cloud sparsity, finally are classified as invalid so that these pixels do not contribute to the loss when estimating $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$. This is achieved by storing this information in the mask elements $\mathcal{V}_{u,v}$ and masking the loss. In order to compensate for the potentially different amount of pixels contributing to the loss between two optimization steps, a normalization factor $\beta=\mathrm{numel}(\mathcal{V})/\sum_{u, v}\mathcal{V}_{u,v}$ is introduced, where $\mathrm{numel}$ denotes the element count of an array. Without $\beta$, a change of the camera pose could potentially reveal more surface area of a masked dynamic object. This may result in a lower loss due to fewer non-masked pixels contributing to it. At the same time, the pose may have changed away from the true pose, which should imply a higher loss, thus creating a need for loss normalization.
\paragraph{Result Verification}
\label{lbl:verification}
After performing an optimization step, we obtain the initial calibration result. However, as the loss function is highly non-convex with respect to the calibration parameters, this result may represent a local minimum. To avoid local minima, the optimization is restarted twice using the previous result as the new start value while also lowering the convergence threshold from $10^{-4}$ to $10^{-6}$ during the last optimization step. The parameter set resulting in the lowest loss is chosen.%This is a step to verify if the result is a global minimum. If the results highly differ between invocations, it suggests that the global minimum was not reached.
To check that the resulting parameters are indeed optimal, a small additive noise is added to the initial guess in order to exit a potentially found local minimum of the loss. The scale of the additive noise can be based on the measurement accuracy of the initially measured camera pose. Repeating the optimization with noisy initial values can then be used to discard sub-optimal parameter sets with high final loss values, which are caused by the non-convexity of the loss function.%\\An advantage of our optimization approach is its robustness to segmentation errors and the choice of the color encoding $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$, as . If the label colors of both domains match and a majority of the segmentation features overlap, an optimal parameter set can be determined.% in most cases, \ReviewChange{as segmentation errors only result in additive biases to the loss without creating a huge impact on the end result.}
%To further compress the point cloud size, the obtained point cloud is cropped to a region of interest.
%\ReviewChange{We have to add some details here, since this is the main step of our work. Can we visualize the simplex at work? E.g., we could fix the position and show how the simplex iterates to find the right angles.}
%This is the main step of the calibration pipeline. Our goal is to estimate $\mathcal{K}_\mathrm{ext}$, which is the extrinsic camera calibration required for infrastructure operation. This is achieved by optimizing an estimated matrix $\hat{\mathcal{K}}_\mathrm{ext}$ so that a loss function is minimized. $\hat{\mathcal{K}}_\mathrm{ext}$ is initialized from an initial measurement, which should be available, as outlined in the constraints. This initial measurement doesn't need to be very precise. The accuracy of a simple GPS receiver or smartphone is sufficient, while for the orientation a compass measurement is enough. The camera pitch could be directly set on the camera mount itself, provided the mount is leveled properly during installation.
%The goal of the loss function is to quantify the current calibration error. In order to measure this error, the semantic point cloud needs to be converted into the image domain, where two images can easily be compared per-pixel using a simple euclidean (L2) loss. As this conversion requires a pose to render the view at, this calibration loss $\mathcal{L}_\mathrm{cal}$ can be written as !!fix eq length!!
%\begin{eqnarray}
% \hat{\mathcal{I}}_\mathrm{sem}^{xyc} = f(\mathcal{P}_\mathrm{sem}, \hat{\mathcal{K}}_\mathrm{ext}, \mathcal{K}_\mathrm{int}, x, y, c),\\
% \mathcal{L}_\mathrm{cal}(\hat{\mathcal{K}}_\mathrm{ext}) = \sum_{c=0}^{2}\sum_{x=0}^{w}\sum_{y=0}^{h} (\hat{\mathcal{I}}_\mathrm{sem}^{xyc} - \mathcal{I}_\mathrm{sem}^{xyc})^2,
%\end{eqnarray}
%where $f$ is a differentiable render function and $\mathcal{I}_\mathrm{sem}^{{xyc}^2}$ is the value of color channel $c$ at coordinates $x$ and $y$. We use a differentiable render function instead of a standard renderer, as the fine registration step requires the entire loss function and its components to be differentiable, as explained in the next section. We furthermore change the approach in which we render the semantic point cloud in such a way that we minimize the appearance of "holes" between holes due to downsampling or insufficient sensor coverage, thus increasing the density of pixels labeled as any kind of foreground. This is achieved by calculating the distance of every point in the point cloud to the camera position $d_i$ and then rendering each point as a circle of radius $r_i=\lambda/d_i$, where $\lambda$ is a scaling factor that depends on point cloud density.
%Using the calibration loss, an optimization using coordinate descent and numerical differentiation !!ref?!! can be applied in order to minimize the calibration error. By using coordinate descent we avoid high-dimensional search spaces that appear when performing grid search for the 5 parameters of $\mathcal{K}_\mathrm{ext}$. The optimization is performed by taking one of the parameters, then uniformly dividing an interval around the current value of the parameter and finding the parameter value with the lowest loss. In order to converge to a solution even on highly non-convex parameter spaces !!wording?!!, the search interval for each dimension is initialized to reflect the real-world variance of the measured parameter and shrunk each iteration. After finishing coordinate descent optimization, the estimated extrinsic matrix $\hat{\mathcal{K}}_\mathrm{ext}$ should have only minor deviations from the true value $\mathcal{K}_\mathrm{ext}$, which can be further corrected by performing a fine-grained optimization using a different optimization method.
%!!result: < 15 cm!!
%
%\subsection{Fine registration}
%
%!!We will probably remove this as the fine registration is not needed!!After performing a coarse registration and estimating $\hat{\mathcal{K}}_\mathrm{ext}$, the result can be further refined. The reason the previous result may not precise enough can be caused by the interval size of the coordinate descent and the limitation to coordinate descent instead of a joint optimization of all parameters. As mentioned in the previous section, the rendering function used in the pipeline is fully differentiable and as we have a defined task and a loss function, we use backpropagation and standard gradient descent without momentum to further minimize the loss function. This is only done to $\hat{\mathcal{K}}_\mathrm{ext}$, while all other inputs to the renderer are frozen. Note that this approach is only used to finetune the result instead of being used for the entire optimization.
%This is caused by the high non-convexity of the used loss function, which causes the optimization to be stuck in local minima and even increasing the loss due to suboptimal momentum, if the optimizer uses any kind of momentum. Therefore, the initialization needs to be very close to the final value, which is achieved through the previous step. Furthermore, as the euclidean loss doesn't provide offset information to the next correct pixel and instead uses every pixel separately, the gradients obtained from such a loss are not meaningful unless the overlap between both images is very large. At the end of this step, we have estimated $\hat{\mathcal{K}}_\mathrm{ext}$, which is the extrinsic calibration of interest. As a byproduct, we can use the calibration in order to re-render the point cloud and extract the driveable area visible in the image as well as generate a depth map of the current camera perspective which can be useful for downstream tasks like object detection and localization in camera images.
\section{EVALUATION}
%In this subsection, the setup used to evaluate algorithm performance is described. Furthermore, changes to the basic algorithm are also mentioned. For evaluation, in order to provide access to the evaluation data, the CARLA simulator ~\cite{Dosovitskiy17} is used to generate ideal infrastructure sensor data, while for real data, scenes from the KITTI Odometry benchmark \cite{geiger2013vision} are used to test an IV use case.
We evaluate our approach on infrastructure scenarios by observing a virtual intersection environment using the CARLA simulator~\cite{Dosovitskiy17}, as well as on a real-world environment based on the KITTI Odometry benchmark~\cite{geiger2013vision}. Next, we describe the evaluation protocol including the scene processing used for benchmarking and finally present our results.
\subsection{Evaluation Scenes}
\begin{figure}[!t]
\centering
\subfloat[Ideal segmentation]{
\includegraphics[trim=0cm 2.5cm 0cm 0.75cm, clip, width=0.8\linewidth]{carla_example_1.png}
\label{fig:carla1}}
\hfil
\subfloat[Rendered point cloud view]{
\includegraphics[trim=0cm 2.5cm 0cm 0.75cm, clip, width=0.8\linewidth]{carla_example_2.png}
\label{fig:carla2}}
\caption{CARLA simulator \cite{Dosovitskiy17} sample result (Town10HD). (a) depicts an ideal semantic image segmentation of a camera sensor. (b) shows a rendered view of a semantically segmented point cloud.}
\label{fig:carla}
\end{figure}
%For the evaluation setup, a two-stage pipeline was used. In the first stage, the sensor data was processed in order to obtain a labeled point cloud and a target image segmentation, as described in Sec. \ref{lbl:camseg}.
Similar to related work like \cite{datondji2016rotation} and \cite{ataer2014calibration}, we use predefined scenes for evaluation. For both CARLA and KITTI, two distinct scenes were defined. For CARLA, we configured urban intersection views from an elevated static perspective to mimic real use cases for roadside infrastructure. Meanwhile, for KITTI, one time frame of two different urban driving sequences was chosen, as we only need one time frame at which the camera will be calibrated. In this case, however, we merged 250 lidar scans around the timestamp of the chosen frame to reconstruct the 3D environment. In the case of CARLA, the segmented point clouds, as well as the segmented camera images of the defined scenes, were generated using the built-in maps. Given completely correct segmentation maps, it is easier to assess the performance of our approach. \ReviewChange{The virtual lidar was configured to have 128 lines and a vertical field of view of $90^\circ$ with the center of the FoV aligned to the ground plane. The virtual segmentation camera used a resolution of 960x600 with a horizontal FoV of $90^\circ$}. For the KITTI data, the Cylinder3D~\cite{zhu2021cylindrical} model, trained with Semantic-KITTI labels \cite{behley2019semantickitti}, was used to generate the point cloud segmentation. Therefore, no ground truth data available in Semantic-KITTI was used. Moreover, the image segmentation was obtained with OCRNet~\cite{yuan2021segmentation}, trained with Cityscapes labels \cite{cordts2016cityscapes}. \ReviewChange{The sensor setup used for recording the raw data passed to the network is described in the KITTI paper~\cite{geiger2013vision}}. As the image segmentation network provided sub-optimal results in some cases, we manually corrected the segmentation to additionally evaluate for a \textit{semi-automatic} setting. In our \textit{automatic} setting, the original image segmentation was utilized without any modification. In all KITTI scenes, road and sidewalk labels were merged in both domains to circumvent noisy borders between ground points which would hinder the optimization. At the same time, the dynamic object removal mentioned in Sec.\ref{lbl:pclseg} was limited to the driveable area covered by the measurement vehicle, which helped retain parked vehicles in the point cloud. This helped to create more visual comparison areas not masked out by $\mathcal{V}_{u,v}$.
%Another tweak to improve segmentation quality consisted of merging road and sidewalk labels in KITTI scenes, as both segmentation domains had issues with distinguishing sidewalks. After labeling, the point cloud was assembled using the poses provided by the benchmark in order to prioritize the evaluation of the cross-domain registration. Outliers from moving vehicles and parts of the test vehicle were removed by filtering out points based on their label as well as cropping out points from each space the test vehicle occupied during recording, as not all points belonging to dynamic objects were labeled correctly. By cropping out bounding boxes containing ghost objects instead of filtering all vehicle points, parked cars could be used to improve registration results, as they provide additional regions that can be matched between both images.
Furthermore, during CARLA evaluation, the point cloud view was rendered with a sky background, as visible in Fig. \ref{fig:carla2}. In order to only match the sky region common between both images, sky regions in the rendered view not matching sky regions in the target were masked out, as these pixels are caused by point cloud sparsity. Another efficient measure consisted of only using the bottom half of the image for registration, as in the KITTI lidar data, the point cloud was cropped in the height dimension, resulting in a trimmed view of the scene which introduced high discrepancies in the upper half of the image. This can be clearly seen in Fig. \ref{fig:kitti3}. Finally, the point cloud was cropped to a 75m radius around the initial location to reduce the memory footprint, in the case of KITTI evaluation. The lidar segmentation view was rendered with Pytorch3D \cite{ravi2020pytorch3d}.
\subsection{Evaluation Protocol}
\label{sec:ep}
%\begin{figure}
% \centering
% \includegraphics[width=0.8\linewidth]{carla_example_new.png}
% \caption{CARLA simulator sample result, a) ideal segmentation, b) rendered point cloud view}
%\end{figure}
As the ground truth calibration for both evaluations is known, a random initialization procedure was performed to assess the convergence behavior of the optimization approach. This was implemented by initializing the optimization algorithm with noisy ground truth, where the additive noise was uniformly sampled from the interval $[-2.5\,\si{\meter};2.5\,\si{\meter}]$ for positional offsets and $[-5^\circ;5^\circ]$ for rotational offsets, as mentioned in Sec. \ref{lbl:cdreg}. By using this approach, we can also verify the usability of our approach, as such offsets could appear in field measurements. After the completion of the optimization, the remaining offsets were interpreted as the calibration quality. As all coordinate systems used are scaled to meters, the positional offsets are measured in centimeters. The algorithm is invoked 30 times on each scene and the 10 parameter sets resulting in the lowest loss $\mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem})$ are considered as acceptable results to filter out results that would be considered outliers%but with newly sampled noise added to the ground truth in order to simulate noisy measurements of the real-world position and orientation as well as reliably find the global minimum of the optimization problem
, as described in the Result Verification of Sec. \ref{lbl:cdreg}.\\ %Mean values as well as variances are reported.%A tweak was used in both evaluations to improve the final precision, which consisted of discarding the upper half of the image for loss calculation, as the KITTI lidar scans were trimmed to a specific height, resulting in large empty regions, which caused unwanted contributions to the loss function. Similarly, the virtual lidar sensor in CARLA was following a path similar to a car, resulting in huge gaps in the point cloud in spots where the lidar hadn't enough height to scan over obstacles, while the infrastructure camera view could record these areas.
%In the case of CARLA, we also specify a position and orientation of the camera which we aim to reconstruct.
We then report the mean vector norm of the displacement error vector $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert} = \overline{\lVert\hat{\boldsymbol{t}_\mathrm{ext}} - \boldsymbol{t}_\mathrm{ext}\rVert}$ as well as the mean absolute angular error $\overline{|\Delta_{\alpha}|}$. Furthermore, to evaluate our choice of the L2 loss as a distance metric, we also repeated the previously described evaluation protocol with the Huber loss \cite{huber1992robust}
\begin{align}
\ell_{\mathrm{Huber}}(\boldsymbol{a},\, \boldsymbol{b}) = \begin{cases}
\frac{1}{2}(\boldsymbol{a}- \boldsymbol{b})^T(\boldsymbol{a}-\boldsymbol{b}),&\lVert\boldsymbol{a}-\boldsymbol{b}\rVert<\delta\\
\delta(\lVert\boldsymbol{a}-\boldsymbol{b}\rVert - \frac{\delta}{2}),&\mathrm{otherwise}
\end{cases}
\end{align}
For results using the Huber loss, we set the loss threshold $\delta$ to 0.3. %To provide the scene information necessary to reproduce these experiments, we supply the CARLA map name and location as well as the KITTI sequence and camera frame IDs in Table \ref{tbl:mce}.% of the 10 results with the lowest final loss in order to quantify the final precision of the extrinsic calibration matrix.\\%Furthermore, as results that arose from local minima of the loss function heavily influence the average error, we also provided \ReviewChange{SYM}. For these values, the 4 best passes based on the final loss were chosen and averaged as well, however, any outlier detection method can be applied in a practical use case.\\
\begin{table}[h]
\caption{Evaluation results.}
\label{tbl:mce}
\centering
%. We report the average translation error in \si{\cm} and rotation error in degrees of each scene. The 10 results with the lowest final loss over a total of 30 runs were used in order to find the parameter sets that mostly likely converged to the global minimum. For KITTI, both the semi-automatic as well as the fully automated pipelines were tested.}
\begin{tabular}{lcccc}
\toprule
\textbf{Loss} & \multicolumn{2}{c}{L2} & \multicolumn{2}{c}{Huber}\\
\midrule
\textbf{Metric} & $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert}$ & $\overline{|\Delta_{\alpha}|}$ & $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert}$ & $\overline{|\Delta_{\alpha}|}$\\
\midrule
\multicolumn{5}{l}{\textbf{Fully automated CARLA-simulated infrastructure}}\\
Town10HD& $1.5\,\si{\cm}$&$0.03^\circ$ & $1.5\,\si{\cm}$&$0.02^\circ$\\
Town01& $6.0\,\si{\cm}$&$0.11^\circ$ & $6.2\,\si{\cm}$&$0.10^\circ$\\
%Town02, $\boldsymbol{t}=[40, 243, 6]$ & $10.8 \si{\cm}$&$0.11^\circ$ & $10.9 \si{\cm}$&$0.11^\circ$\\
\midrule
\multicolumn{5}{l}{\textbf{Semi-automated KITTI}}\\
Sequence 00 & $12.7\,\si{\cm}$&$0.28^\circ$ & $15.3\,\si{\cm}$&$0.26^\circ$\\
%Sequence 05 Frame 1650 & $13.0 \si{\cm}$&$0.20^\circ$ & $14.3 \si{\cm}$&$0.21^\circ$\\
Sequence 09 & $20.6\,\si{\cm}$&$0.40^\circ$ & $19.3\,\si{\cm}$&$0.42^\circ$\\
\midrule
\multicolumn{5}{l}{\textbf{Fully automated KITTI}}\\
Sequence 00 & $20.2\,\si{\cm}$&$0.34^\circ$ & $20.3\,\si{\cm}$&$0.33^\circ$\\
Sequence 09 & $18.6\,\si{\cm}$&$0.40^\circ$ & $19.6\,\si{\cm}$&$0.43^\circ$\\
\bottomrule
\end{tabular}
\end{table}
\subsection{Results Discussion}
\label{sec:rd}
Table~\ref{tbl:mce} summarizes our results for both evaluations and examined loss functions.
%As can be seen in Table \ref{tbl:mce}, the results show the feasibility of our method. %especially if we want to rule out any impact of the data pre-processing steps, which could affect final performance due to incorrectly filtered and noisy data.
For the CARLA evaluation, our translation error is below 7\,\si{\cm} for both scenes, while the rotation error is at most $0.11^\circ$.%This is especially reflected in the CARLA scene performance with translation errors below 15\si{\cm} and rotation errors below $0.15^\circ$.
~We observed that the errors mostly stem from point cloud sparsity, as can be seen in Fig. \ref{fig:carla}. It should also be noted that, due to the rasterized representation of an image, we cannot distinguish image translations below a single pixel, therefore very small changes of under 1\si{\cm} in camera translation and \ang{0.01} in camera rotation may not be visible in the rendered view. The real-world KITTI scenes also perform very well. While both the point cloud labels and image segmentation contain incorrect labels and, in the case of point clouds, incorrectly measured data points, the final calibration quality is still very high, especially if used for infrastructure sensor calibration. Furthermore, we tested fully automatically labeled KITTI scenes by not manually correcting the image segmentation.% and only combining road and sidewalk labels.
~The performance in these scenes dropped only slightly or even stayed in the same range and is therefore comparable with the semi-automatic use case. However, it should be noted that this performance highly depends on the quality of the sensor data and the dataset used for training the segmentation network. Comparing both loss types, we can see only minor differences, which can be explained by the random initialization and testing approach. Thus, both losses are equally suitable for the task. %We can notice that only the Huber loss approach manages to improve its results on segmentation-corrected KITTI sequence 09 compared to the fully automated scene.%\\ By comparing the left and right parts of Table \ref{tbl:mce}, we can see that in some cases the optimization tends to converge toward local minima, resulting in high average errors. By adding noise to the initial measurement and repeating the optimization multiple times, we can explore the \ReviewChange{neighbourhood} of the initial measurement and use the result with the smallest loss. By only considering the parameter sets with the lowest loss, the average error can be further improved.
%\begin{table}[]
% \centering
% \caption{Monte Carlo experiment results.}
% \begin{tabular}{|l|c|c||c|c|}
% \hline
% Scene & $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert}$ & $\overline{|\Delta_{\alpha}|}$&\ReviewChange{SYM}&\ReviewChange{SYM}\\
% \hline
% \hline
% CARLA 1 & $9.68 \si{\cm}$& $0.05^\circ$& $9.0 \si{\cm}$&$0.03^\circ$\\
% \hline
% CARLA 2 & $70.5 \si{\cm}$& $0.68^\circ$& $4.3 \si{\cm}$&$0.07^\circ$\\
% \hline
% CARLA 3 & $118 \si{\cm}$& $4.05^\circ$& $10.9 \si{\cm}$&$0.12^\circ$\\
% \multicolumn{5}{l}{Segmentation-corrected KITTI}\\
% \hline
% \hline
% KITTI 00 & $13.0 \si{\cm}$& $0.28^\circ$& $12.8 \si{\cm}$&$0.27^\circ$\\
% \hline
% KITTI 05 & $154 \si{\cm}$& $2.48^\circ$& $12.7 \si{\cm}$&$0.18^\circ$\\
% \hline
% KITTI 09 & $42.3 \si{\cm}$& $1.06^\circ$& $21.0 \si{\cm}$&$0.56^\circ$\\
% \multicolumn{5}{l}{Fully automated KITTI}\\
% \hline
% \hline
% \makecell{KITTI 00\\(automatic)} & $20.0 \si{\cm}$& $0.33^\circ$& $20.0 \si{\cm}$&$0.32^\circ$\\
% \hline
% \makecell{KITTI 09\\(automatic)} & $41.9 \si{\cm}$& $0.54^\circ$& $19.2 \si{\cm}$&$0.43^\circ$\\
% \hline
% \end{tabular}
% \label{tbl:mce}
%\end{table}
\subsection{Comparison with Related Approaches}
\begin{figure}[!t]
\centering
\subfloat[Rectified camera image]{
\includegraphics[width=0.8\linewidth]{KITTI_example_1.png}
\label{fig:kitti1}}
\hfil
\subfloat[Image segmentation]{
\includegraphics[width=0.8\linewidth]{KITTI_example_2.png}
\label{fig:kitti2}}
\hfil
\subfloat[Rendered point cloud view]{
\includegraphics[width=0.8\linewidth]{KITTI_example_3.png}
\label{fig:kitti3}}
\caption{KITTI \cite{geiger2013vision} dataset result (Sequence 00 fully automated). We can see a noticeable visual difference between the segmentation image and the rendered point cloud, which is caused by point cloud sparsity as well as incorrect labels. However, they still share enough common labelfeatures to be reliably matched, as shown in Table \ref{tbl:mce}.}
\label{fig:kitti}
\end{figure}
As our approach is quite different from previous works in this area and no common evaluation protocols are defined, it is difficult to directly compare performance metrics, so we only look at qualitative results. Considering Horn et al. \cite{horn2021online}, who achieve translation errors of $20.8\,\si{\cm}$ and rotation errors of $0.26^\circ$ on the KITTI dataset, but require sensor movement, we get equally good results (see Table \ref{tbl:mce}) without sensor movement. The authors of the mobile stereo camera SLAM approach in \cite{ataer2014calibration} report translation errors in the range of $6\si{\cm}-67 \si{\cm}$ and rotation errors between $1.5^\circ$ and $5.4^\circ$ on their own data. Comparably, our positional errors are on the lower end of the spectrum, while our rotation errors are smaller by a factor of 10. We, therefore, conclude that our approach is competitive with other methods and can be used for infrastructure and IV sensor calibration.
%We compare our method to other sensor calibration methods by finding an optimal calibration for each of the KITTI scans. As a direct comparison between various calibration approaches \cite{ataer2014calibration, wodtko2021globally, datondji2016rotation} is very difficult due to different scenarios, use cases, and sensor types to be calibrated, we report the maximal observed translation and rotation errors over all sequences, which is a common comparison metric between various calibration scenarios. Our calibration result is obtained by using the parameter set with the lowest achieved loss from the previous Monte Carlo experiment.%\ReviewChange{In this case, our fully automatic calibration on two scenarios even slightly outperforms our semi-automatic scenario.}
%Compared to other calibration methods, we achieve very good results, which are presented in Table \ref{tbl:comparison}. While the method presented in \cite{wodtko2021globally} has the lowest translation error, it also requires motion and therefore is not applicable to infrastructure sensors. The method described in \cite{muller2019laci} achieves a lower rotation error, however it is designed to be used with lidar sensors and is not directly applicable to camera images. We therefore achieve a good balance between maximal translation and rotation errors, while having comparable results with other methods.
%\begin{table}[]
% \centering
% \caption{Performance Comparison with other Calibration Approaches}%. We report the maximum translation and rotation errors observed over all KITTI sequences, while differentiating between our semi- and fully automated approaches. For each KITTI sequence, the parameter set resulting in the lowest loss is used as the final calibration result.}
% \begin{tabular}{|l|c|c|c|c|}
% \hline
% Method & Level & \makecell{Motion\\required} & $\lVert\Delta_{\boldsymbol{t}}\rVert_\mathrm{max}$ & $|\Delta_{\alpha}|_\mathrm{max}$\\
% \hline
% Our approach & semi & no & $ 17.8 \si{\cm}$& \ang{0.4}\\
% \hline
% Our approach & full & no & $ 21.7 \si{\cm}$& \ang{0.4}\\
% \hline
% %\makecell{M\"uller et al. \cite{muller2019laci}\\(4 laser scanners)} & full & $ 14.9 \si{\cm}$& \ang{0.13}\\
% %\makecell{Ataer et al. \cite{ataer2014calibration}\\(indoor pose)} & full & no & $ 67.6 \si{\cm}$& \ang{5.4}\\
% %\hline
% %\makecell{Datondji et al. \cite{datondji2016rotation}\\(fisheye stereo)} & semi & no & $ 15.3 \si{\cm}$& \ang{2.9}\\
% %\hline
% \makecell{Wodtko et al. \cite{wodtko2021globally}\\(sensor motion)} & full & yes & $ 1.1 \si{\cm}$& \ang{0.63}\\
% \hline
% \end{tabular}
% \label{tbl:comparison}
%\end{table}
\section{CONCLUSION}
We presented an extrinsic camera calibration approach for both infrastructure and intelligent vehicle cameras.
%In this work, a novel calibration approach for camera calibration of both infrastructure and vehicle cameras was presented.
%Our approach requires minimal to no manual interaction, works without the use of specific calibration targets and does not require camera sensor motion to work, which also makes it suitable for on-the-fly re-calibration.
~In our approach, we obtained a semantic segmentation of both IV-collected lidar data as well as camera data. This segmented data was then used to find an optimal calibration by using an optimization loop to match the appearance of the image segmentation with a rendered view of the segmented lidar data, effectively performing cross-domain registration. The algorithm has been evaluated on simulated CARLA \cite{Dosovitskiy17} scenes and real sensor data from the KITTI dataset \cite{geiger2013vision} to show its feasibility in a practical application, showing equally good or better results than existing approaches, which, however, are more constrained in their application.
\addtolength{\textheight}{-9cm} % This command serves to balance the column lengths
% on the last page of the document manually. It shortens
% the textheight of the last page by a suitable amount.
% This command does not take effect until the next page
% so it should come on the page before the last. Make
% sure that you do not shorten the textheight too much.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section*{APPENDIX}
%
%Appendixes should appear before the acknowledgment.
%
%\section*{ACKNOWLEDGMENT}
%
%The preferred spelling of the word acknowledgment in America is without an e after the g. Avoid the stilted expression, One of us (R. B. G.) thanks . . .
%Instead, try R. B. G. thanks. Put sponsor acknowledgments in the unnumbered footnote on the first page.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{ieeetran}
\bibliography{IEEEexample}
\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
% \documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
%In case you encounter the following error:
%Error 1010 The PDF file may be corrupt (unable to open PDF file) OR
%Error 1000 An error occurred while parsing a contents stream. Unable to analyze the PDF file.
%This is a known problem with pdfLaTeX conversion filter. The file cannot be opened with acrobat reader
%Please use one of the alternatives below to circumvent this error by uncommenting one or the other
%\pdfobjcompresslevel=0
%\pdfminorversion=4
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document
% The following packages can be found on http:\\www.ctan.org
\usepackage{graphicx} % for pdf, bitmapped graphics files
%\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb} % assumes amsmath package installed
\usepackage{siunitx}
\usepackage{todonotes}
\usepackage{placeins}
\usepackage{ifthen}
\usepackage{makecell}
\renewcommand{\cellalign}{lc}
%\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage{booktabs}
\usepackage{hyperref}
%\usepackage{caption}
%\usepackage{subcaption}
\newboolean{PrintVersion}
\setboolean{PrintVersion}{true}
\newcommand{\ReviewChangeRemoved}[1]{\sout{#1}}
\newcommand{\ReviewChange}[1]{\textcolor{blue}{#1}}
\ifthenelse{\boolean{PrintVersion}}{
\renewcommand{\ReviewChangeRemoved}[1]{}
\renewcommand{\ReviewChange}[1]{#1}
}{}
\title{\LARGE \bf
Extrinsic Camera Calibration with Semantic Segmentation
}
\author{Alexander Tsaregorodtsev$^{1}$, Johannes~Müller$^{1}$, Jan~Strohbeck$^{1}$, Martin~Herrmann$^{1}$, \\ Michael~Buchholz$^{1}$ and Vasileios Belagiannis$^{2}$% <-this % stops a space
\thanks{*Part of this work was financially supported by the Federal Ministry of Economic Affairs and Energy of Germany within the program "New Vehicle and System Technologies" (project LUKAS, grant number 19A20004F). Part of this work has been conducted as part of ICT4CART project which has received funding from the European Union’s Horizon 2020 research \& innovation program under grant agreement No. 768953. Content reflects only the authors’ view and European Commission is not responsible for any use that may be made of the information it contains. }% <-this % stops a space
\thanks{$^{1}$The authors are with the Institute of Measurement, Control and Microtechnology,
Ulm University, D-89081 Ulm, Germany \newline
\{$<$first\,\,name$>$.$<$family\,\,name$>$@uni-ulm.de\}
}
\thanks{$^{2}$Vasileios Belagiannis is with the Department of Simulation and Graphics,
University Magdeburg, D-39106 Magdeburg, Germany \newline
\{vasileios.belagiannis@ovgu.de\}. Most of this work was done during his time at Ulm University.
}}%
\begin{document}
\newcommand\copyrighttextinitial{%
\scriptsize This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.}%
\newcommand\copyrighttextfinal{%
\scriptsize\copyright\ 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.}% DOI: 10.1109/ICRA48506.2021.9561375.}%
\newcommand\copyrightnotice{%
\begin{tikzpicture}[remember picture,overlay]%
\node[anchor=south,yshift=10pt] at (current page.south) {{\parbox{\dimexpr\textwidth-\fboxsep-\fboxrule\relax}{\copyrighttextfinal}}};%
\end{tikzpicture}%
%\vspace{-10pt}%
}
\maketitle
\copyrightnotice%
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information from images and point clouds. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle
with high-precision
localization to capture a point cloud of the camera environment. Afterward, a mapping between the camera and world coordinate spaces is obtained by performing a lidar-to-camera registration of the semantically segmented sensor data. We evaluate our method on simulated and real-world data to demonstrate low error measurements in the calibration results. Our approach is suitable for infrastructure sensors as well as vehicle sensors, while it does not require motion of the camera platform.
%in order to assess calibration quality and demonstrate our approach in a real use case.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{INTRODUCTION}
Monocular cameras are a keystone to cost-efficient sensor setups \cite{rinner2008introduction} for both intelligent vehicles (IVs) and smart infrastructure, supporting automated driving. The calibration of such cameras is vital to their operation out in the field and, therefore, of utmost importance for applications such as self-localization~\cite{engel2019deeplocalization} or recognition~\cite{wiederer2020traffic}. However, while the intrinsic camera calibration can easily be accomplished once in a constrained environment, the recovery of the extrinsic calibration in the field or between vehicle sensors is still a challenging process. In particular, classical calibration methods \cite{andrew2001multiple} often require manually placed calibration targets in the scene. These approaches, therefore, cannot easily be applied to traffic cameras, observing densely populated junctions or to IVs out in the field. Thus, an automatic calibration solution, which works for single infrastructure cameras, is needed to reduce calibration effort. Such a solution can be used to calibrate cameras mounted on IVs as well.
Recently, sensor-to-sensor registration methods, e.g. \cite{datondji2016rotation, zhu2016robust}, have been proposed to automate the calibration process. However, these methods often require broadly overlapping fields of view (FOVs), which implies redundant sensors and hence negates the cost savings achieved by a single sensor. For IVs, specific fully automated calibration schemes, such as \cite{horn2021online}, have been developed, which rely on the motion of the sensor platform, i.e., the IV, to optimize the calibration during each time step.
\begin{figure}
%\centering
\input{Kalib.pdf_tex}
\caption{Approach Overview. On the left side, data from both domains are fed into the pipeline. For lidar data, the 3D environment is reconstructed using multiple scans. Afterward, both domains are labeled semantically. In the final step, our proposed optimization is performed to register both domains to retrieve the extrinsic calibration parameters.}
\label{fig:PipelineOverview}
\end{figure}
~A caveat of these specific calibration schemes is their dependency on sensor motion, which excludes static infrastructure from the range of possible applications. Deep learning approaches like \cite{li2021deepi2p} have recently been developed to estimate extrinsic camera parameters from camera images and lidar data. While these approaches have shown promising results, they still tend to have higher estimation errors compared to previous methods like \cite{horn2021online}, making them unsuitable for IV and infrastructure calibration. Finally, another classical method is described in \cite{ataer2014calibration}. It localizes a camera sensor inside of a 3D model of the environment taken from a SLAM with stereo cameras.
However, this method obtains its 3D environments from a mobile stereo camera setup and thus contains texture information not present in lidar data.
~Furthermore, this approach is not transferable to stereo camera setups permanently mounted on IVs due to their limited field of view.
We propose a monocular camera calibration approach to perform cross-domain registration of a semantically segmented mono camera image and a semantically labeled 3D model of the environment the camera is located in. An overview of the approach is given in Fig. \ref{fig:PipelineOverview}.
By using a semantic representation of both the camera and the point cloud domain, we introduce semantic labels, which can be matched between the domains. Furthermore, the use of semantic labels instead of raw RGB and RGBD camera data makes the registration more robust and reduces the possibility of registration algorithm errors. These can occur with raw camera data due to color space and exposure differences between different camera types.
Our approach consists of data pre-processing steps as well as an optimization step performing cross-domain registration. In the pre-processing steps, the 3D environment model is reconstructed and then labeled semantically using neural networks. The segmentation is also performed on images of the target camera that we aim to calibrate. Then, an optimization pass is invoked to match the visual appearance and projection of a rendered view of the 3D model with the segmented camera view.
This optimization then yields the extrinsic camera calibration.
~Finally, experiments with the CARLA simulator \cite{Dosovitskiy17} and the KITTI dataset \cite{geiger2013vision} show that our method leads to the accuracy level that is suitable for automated driving applications, both for static and dynamic camera platforms. The contribution of this paper is twofold. From a practical perspective, our approach allows for low-effort, highly automated calibration of monocular cameras in (geo-referenced) world coordinates without requiring motion of the target sensor platform.
From a methodological perspective to the best of our knowledge, we are the first to use semantic segmentation as a visual feature set to guide the required parameter optimization. The source code of our approach is publicly available\footnote{\url{https://github.com/Tuxacker/semantic\_calibration}}.
The related work to extrinsic camera calibration is outlined in Section II. The complete algorithm is described in Section III, while in Section IV, a theoretical and practical evaluation is provided. At last, the conclusion of this work is described in Section V.
\section{RELATED WORK}
Calibration of cameras has been extensively investigated in the past. A classical approach for intrinsic and extrinsic camera calibration is presented in \cite{andrew2001multiple}. One of the first approaches specifically for infrastructure sensors was described in \cite{datondji2016rotation}, where a vehicle target was used to calibrate a pair of fisheye cameras with a large overlapping field of view. Another approach using planar structure from motion and textured environments was presented in \cite{zhu2016robust}, which is able to calibrate multi-sensor configurations. The approach of \cite{ataer2014calibration} circumvents the field of view (FoV) overlap requirement of the other two approaches by reconstructing a 3D environment using a mobile RGB-D camera setup. Then, a matching of feature descriptors between the target camera image and the model is performed and the closest matching keyframes are found to determine the camera pose. The automatic calibration method of \cite{dubska2014fully} estimates the camera vanishing points by tracking image features of moving vehicles, while a newer method by \cite{bartl2021automatic} uses a neural network to find landmark positions on vehicles. By using the positional relations between landmarks obtained from real 3D models of the vehicles, a calibration is estimated from the landmarks.
Another class of algorithms performs direct 2D to 3D matching and registration. Feng et al. \cite{feng20192d3d} use a neural network to directly match point cloud patches to image patches, while the algorithm in \cite{yu2020monocular} performs camera localization by finding correspondences between 2D and 3D line features. Li et al. \cite{li2021deepi2p} train a classifier to detect if a point in the point cloud is in the camera frustum and then solve the inverse camera projection problem using the frustum labels.
Methods performing calibration through sensor motion can also be found in the literature (\cite{horn2021online, wei2018calibration, wise2020certifiably}). In particular, Horn et al. \cite{horn2021online} use a visual SLAM approach to estimate the camera displacement and then represent the translation and rotation using dual quaternions instead of homogeneous transformation matrices, which allows for a more efficient implementation of the optimization. The optimization consists of finding a calibration matrix that is able to close the loop between the current and previous time step, where the calibration matrices of the previous time step are known.\\
A recent method using automated vehicles as calibration targets is shown in \cite{muller2019laci}. It uses lidar measurements in an empty scene to determine the ground plane and background and then regresses the position of the automated vehicle as it passes through the sensors FoV with localization data of the vehicle being available.
We improve on previous methods by exploring a new concept for matching cross-domain features by visual alignment, which does not require specific calibration targets or motion of the sensor itself. It, therefore, allows us to perform a single-shot calibration in diverse scenarios without requiring a lot of manual intervention.
\section{CALIBRATION METHOD}
In this section, we formulate our problem and present each step of our approach required to estimate the extrinsic camera calibration.
\subsection{Problem Formulation and Constraints}
Consider the perspective camera model \cite{andrew2001multiple}
\begin{equation}
{\begin{bmatrix}
u \\ v \\ 1
\end{bmatrix}} =
\underbrace{\boldsymbol{K}_\text{int} \;
\begin{bmatrix}
\boldsymbol{R} & \boldsymbol{t} \\
\boldsymbol{0} & 1
\end{bmatrix}}_{:= \boldsymbol{P}}
{\begin{bmatrix}
x \\ y \\ z \\ 1
\end{bmatrix}} \; ,
\end{equation}
where $[u, v, 1]^T$ denotes a homogeneous camera image coordinate of an image $\mathcal{I}$ with pixel color value $\mathcal{I}_{u,v} \in [0;1]^3$, and $[x, v, z, 1]^T\in\mathcal{M}^{3\text{D}}$ is a homogeneous point coordinate of a 3D model $\mathcal{M}^{3\text{D}}$ of the observed scene. $\mathcal{M}^{3\text{D}}$ is represented as a point cloud. Furthermore, it is assumed that the scene visible in the camera image is overlapping with the scene represented in the point cloud.
Our goal is to recover the parameters of the extrinsic camera matrix $\boldsymbol{P}$, i.e., the rotation matrix $\boldsymbol{R} \in \mathbb{R}^{3 \times 3}$ and the translation vector $\boldsymbol{t} \in \mathbb{R}^{3}$. At the same time we assume that the intrinsic parameters $\boldsymbol{K}_\text{int} \in \mathbb{R}^{3 \times 4}$, which represent the mapping from 3D coordinates in the camera coordinate frame to 2D pixel coordinates, are already known, e.g.~as described in~\cite{andrew2001multiple}. Below, we propose a new calibration approach to estimate these extrinsic camera parameters.
\subsection{Approach Overview}
We use visual matching of semantic information for the parameter estimation of the camera rotation $\boldsymbol{R}$ and translation $\boldsymbol{t}$.
Our method consists of four steps, starting with the 3D environment reconstruction $\mathcal{M}^{3\text{D}}$, represented as a point cloud (see Sec.~\ref{lbl:3drec}). It is recorded with the lidar sensor mounted on a vehicle. Next, the point cloud segmentation step (presented in Sec.~\ref{lbl:pclseg}) follows to extract semantic information for each 3D point. Then, the camera image segmentation (see Sec.~\ref{lbl:camseg}) assigns a class category to each pixel. Given the segmented 3D point cloud and image, we propose a cross-domain registration approach (explained in Sec.~\ref{lbl:cdreg}) to estimate the extrinsic camera parameters by visually matching the camera segmentation with a rendered point cloud view using linear programming~\cite{nelder1965simplex}.
\subsection{3D Environment Reconstruction}
\label{lbl:3drec}
We assume access to a vehicle equipped with a lidar sensor and high-precision localization to construct the 3D environment model $\mathcal{M}^{3\text{D}}$. Several lidar scans are recorded as point clouds and expressed in the world coordinate frame. The transformation from lidar sensor coordinates to world coordinates requires known lidar extrinsics.
Due to road bumps and inaccurate orientation measurements, points in the sensor's far field come with increased positional errors, resulting in noisy 3D point clouds with many outliers. Thus, we filter out points that exceed the maximum distance $d_\text{max}$ from the vehicle's lidar sensor. In practice, $d_\text{max}$ is chosen to be between $50~\text{m}$ and $75~\text{m}$, which are standard cutoff values for Velodyne sensors like the VLP-32C or the HDL-64E used in the KITTI \cite{geiger2013vision} dataset. Next, we rely on the multi-way Iterative Closest Point (ICP) registration algorithm~\cite{chen1992object, choi2015robust} between all scans, which partially compensates for localization errors that accumulate over time. To accelerate the registration, instead of registering every scan against every other scan, all scans are sorted by their capture time, and groups of three neighboring scans are recursively registered and merged. The recursion ends when all groups are combined into the single point cloud $\mathcal{M}^{3\text{D}}$. At last, the point cloud is down-sampled during each recursion step in order to relax the ICP computation requirements. The reconstruction step can be skipped if the 3D environment is obtained by another mapping algorithm or from a different source.
\subsection{Point Cloud Segmentation}
\label{lbl:pclseg}
In the next step, semantic labels are assigned to the points of $\mathcal{M}^{3\text{D}}$ to obtain the semantic segmentation map $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$, which contains a color label for each point in $\mathcal{M}^{3\text{D}}$. For this task, we rely on a pre-trained deep neural network to perform semantic segmentation, namely Cylinder3D~\cite{zhu2021cylindrical}, \ReviewChange{which supports segmentation of classes defined in SemanticKITTI~\cite{behley2019semantickitti}}.
\\Furthermore, the segmentation is also used to filter out dynamic object classes like cars and pedestrians.
This leaves the buildings, ground points, vegetation, fences, poles, and traffic signs classes for registration.
Therefore, the calibration method does not require exact time synchronicity between the lidar scans and the camera image, as points belonging to dynamic objects can easily be filtered out by their label.
\subsection{Camera Image Segmentation}
\label{lbl:camseg}
After labeling the point cloud $\mathcal{M}^{3\text{D}}$, the semantic segmentation map $\mathcal{I}^\text{sem}$ is extracted from the camera image $\mathcal{I}$. Similar to the point cloud segmentation, we rely on a pre-trained deep neural network to extract semantic labels \ReviewChange{available in the Cityscapes dataset~\cite{cordts2016cityscapes}}, e.g.,~OCRNet~\cite{yuan2021segmentation}. \ReviewChange{We chose to use Cityscapes labels, as they share most classes available in SemanticKITTI and thus enable direct matching of class labels between domains.} Again, dynamic object class categories, e.g.,~cars or pedestrians, are removed from the segmentation map. For the remaining object categories, we perform a category alignment between the point cloud~\cite{zhu2021cylindrical} and the image~\cite{yuan2021segmentation} segmentation models by neglecting points and pixels with labels that are unique to this domain. It should be noted that the removal of dynamic objects results in holes in the resulting map, especially in crowded scenes. To minimize their impact, we introduce the normalization factor $\beta$ during the registration step in Section~\ref{lbl:cdreg}.
The image segmentation can also be done semi-automatically to improve calibration quality. In this case, the resulting segmentation is manually edited to fix incorrectly labeled regions in order to improve data quality and hence reduce the overall calibration error.
\subsection{Cross-domain registration}
\label{lbl:cdreg}
Next, we aim to estimate the rotation matrix $\boldsymbol{R}$ and translation vector $\boldsymbol{t}$. We formulate this estimation problem as an optimization, where a loss correlated to the error $\lVert\hat{\boldsymbol{R}} - \boldsymbol{R}\rVert$ and $\lVert\hat{\boldsymbol{t}} - \boldsymbol{t}\rVert$ is minimized using the simplex algorithm by Nelder and Mead \cite{nelder1965simplex}.
As the Nelder-Mead method does not require gradient information, it allows a broad range of data transforms to be applied to $\mathcal{M}^{3\text{D}}$. This includes rendering transforms, which can be used to obtain a rasterized image $\hat{\mathcal{I}}$ of a specific perspective view of $\mathcal{M}^{3\text{D}}$. We, therefore, define the rendering function $\boldsymbol{f}$ as
\begin{equation}
\hat{\mathcal{I}} = \boldsymbol{f} \big(\mathcal{M}^{3\text{D}},\boldsymbol{P}(\hat{\boldsymbol{R}},\hat{\boldsymbol{t}}), \mathcal{M}^{3\text{D}}_{\boldsymbol{c}} \big) \; ,
\end{equation}
where $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$ is the color encoding of the segmentation and $\hat{\boldsymbol{R}}$, $\hat{\boldsymbol{t}}$ are estimates of $\boldsymbol{R}$ and $\boldsymbol{t}$. This rendering function is used to transform the point cloud $\mathcal{M}^{3\text{D}}$ with color labels $\mathcal{M}^{3\text{D}}_{\boldsymbol{c}}$ into an image $\hat{\mathcal{I}}$ with the viewpoint defined by the perspective camera matrix $\boldsymbol{P}(\hat{\boldsymbol{R}},\hat{\boldsymbol{t}})$. $\hat{\mathcal{I}}$ can then be visually matched with the previously obtained image segmentation $\mathcal{I}^\text{sem}$ by calculating a distance metric between both images and interpreting the result as a loss value. By adjusting $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$ to minimize the visual difference between these images, and therefore minimizing the loss value, we actively determine the optimal camera parameters. An advantage of the Nelder-Mead method lies in its use of a simplex to define the initial search space, which then moves and shrinks towards the function minimum. By scaling the initial simplex bounds according to the expected measurement variance of the initial guess, we can easily define a sensible search space covered by the starting simplex. Note that other gradient-free approaches may be used as well.
\paragraph{Initialization}
In order to start the optimization, the Nelder-Mead method requires an initial parameter set. The convergence speed of the algorithm and its ability to find an optimal solution depend on the initial parameter set and the search space.
Therefore, an initial guess for the camera position and orientation is required.
We furthermore constrain the search space of the pitch and yaw angle to $\pm 5^\circ$, while we set the roll angle to zero. We expect our method to also work for non-vanishing roll angles, but in our use cases, no camera roll is desired and can be achieved by proper mounting. This is a reasonable simplification since potential camera roll is often not desired for automotive applications.
~The yaw angle can be measured with a compass, while the pitch can be determined with a spirit level. The maximal position offsets are constrained to $\pm2.5\,\si{\meter}$, i.e., the GPS accuracy of a typical smartphone GPS receiver.
We argue that an initial guess with such accuracy can be easily obtained for most practical situations and is sufficient to obtain a good solution for the extrinsic calibration problem, as shown in Sections~\ref{sec:ep} and~\ref{sec:rd}.
However, it is possible to extend the search space at a tolerable increase in calculation effort by combining the Nelder-Mead method with a coarse grid search to estimate parameters that were not measured and are therefore not known. If an IV sensor-to-sensor calibration is desired, no initial measurements are required in most cases, as the convergence range of our approach is sufficient to bridge the translation and rotation between both sensors. Therefore, they can be set to be zero vectors in such cases. After setting the initial guess, we also have to calculate the initial simplex, which will converge to the optimal solution. To this end, we generate 6 values (as a simplex of dimension $n$ is described by $n+1$ points) by adding and subtracting the maximal constrained parameter range for each pair of simplex vertices and for each parameter.
\paragraph{Optimization Loop}
The Nelder-Mead method is invoked until the loss distance between two steps falls below $10^{-4}$ in order to estimate $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$.
The loss target we optimize is defined as
\begin{equation}
\label{eq:Loss}
\mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem}) = \beta \sum_{u=0}^{h-1} \sum_{v=0}^{w-1} \mathcal{V}_{u,v}\, \ell_{\mathrm{L2}}(\hat{\mathcal{I}}_{u,v},\,\mathcal{I}^\text{sem}_{u,v})
\end{equation}
with the L2 loss term $\ell$
\begin{equation}
\ell_{\mathrm{L2}}(\boldsymbol{a},\, \boldsymbol{b})=(\boldsymbol{a}- \boldsymbol{b})^T(\boldsymbol{a}-\boldsymbol{b}).
\end{equation}
Here, $\beta$ is a normalization factor, $w$ and $h$ denote the width and height of the images,
~and $\mathcal{V}_{u,v}$ is a function checking the validity of the classes. If one of the images contains an invalid class for a pixel, $\mathcal{V}_{u,v}=0$, i.e., the pixel is ignored for loss calculations. Otherwise, $\mathcal{V}_{u,v}=1$ implies a contribution to the final loss. It should be noted that, while in theory, any distance metric may be used for $\ell$, we decided to use an L2 loss, as this loss is commonly used as an image similarity measure. Finally, the optimization performed by the Nelder-Mead method can be represented as
\begin{equation}
\label{eq:OptimizationProblem}
\hat{\boldsymbol{R}}_{\mathrm{opt}}, \hat{\boldsymbol{t}}_{\mathrm{opt}} = \arg \min_{\hat{\boldsymbol{R}}, \hat{\boldsymbol{t}}}\left\{ \mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem}) \right\} .
\end{equation}
\paragraph{Appearance Matching and Masking}
Before optimizing the camera pose, we want to closely match the general appearance of the rendered image $\hat{\mathcal{I}}$ to the target $\mathcal{I}^\text{sem}$ as well as reduce rendering artifacts. For that reason, we first aim to minimize the appearance of background pixels due to the sparsity of points in the 3D model. This is achieved by calculating the distance $d_i$ of every point in $\mathcal{M}^{3\text{D}}$ to the camera position. Then, each point in the 3D model is rendered as a circle of radius $r_i=\lambda/d_i$, where $\lambda$ is a scaling factor that depends on the point cloud density and can be determined empirically by rendering a test view and increasing $\lambda$ until the appearance of the rendered view roughly matches the target segmentation image. \ReviewChange{This adaptive point size approach also adds the benefit of limiting the impact of varying line counts between different lidar models, as lidars with fewer lines can still be densely rendered by increasing $\lambda$}. Secondly, pixels, which still cannot be assigned with a semantic label corresponding to a static object, e.g., due to point cloud sparsity, finally are classified as invalid so that these pixels do not contribute to the loss when estimating $\hat{\boldsymbol{R}}$ and $\hat{\boldsymbol{t}}$. This is achieved by storing this information in the mask elements $\mathcal{V}_{u,v}$ and masking the loss. In order to compensate for the potentially different amount of pixels contributing to the loss between two optimization steps, a normalization factor $\beta=\mathrm{numel}(\mathcal{V})/\sum_{u, v}\mathcal{V}_{u,v}$ is introduced, where $\mathrm{numel}$ denotes the element count of an array. Without $\beta$, a change of the camera pose could potentially reveal more surface area of a masked dynamic object. This may result in a lower loss due to fewer non-masked pixels contributing to it. At the same time, the pose may have changed away from the true pose, which should imply a higher loss, thus creating a need for loss normalization.
\paragraph{Result Verification}
\label{lbl:verification}
After performing an optimization step, we obtain the initial calibration result. However, as the loss function is highly non-convex with respect to the calibration parameters, this result may represent a local minimum. To avoid local minima, the optimization is restarted twice using the previous result as the new start value while also lowering the convergence threshold from $10^{-4}$ to $10^{-6}$ during the last optimization step. The parameter set resulting in the lowest loss is chosen.
To check that the resulting parameters are indeed optimal, a small additive noise is added to the initial guess in order to exit a potentially found local minimum of the loss. The scale of the additive noise can be based on the measurement accuracy of the initially measured camera pose. Repeating the optimization with noisy initial values can then be used to discard sub-optimal parameter sets with high final loss values, which are caused by the non-convexity of the loss function.
\section{EVALUATION}
We evaluate our approach on infrastructure scenarios by observing a virtual intersection environment using the CARLA simulator~\cite{Dosovitskiy17}, as well as on a real-world environment based on the KITTI Odometry benchmark~\cite{geiger2013vision}. Next, we describe the evaluation protocol including the scene processing used for benchmarking and finally present our results.
\subsection{Evaluation Scenes}
\begin{figure}[!t]
\centering
\subfloat[Ideal segmentation]{
\includegraphics[trim=0cm 2.5cm 0cm 0.75cm, clip, width=0.8\linewidth]{carla_example_1.png}
\label{fig:carla1}}
\hfil
\subfloat[Rendered point cloud view]{
\includegraphics[trim=0cm 2.5cm 0cm 0.75cm, clip, width=0.8\linewidth]{carla_example_2.png}
\label{fig:carla2}}
\caption{CARLA simulator \cite{Dosovitskiy17} sample result (Town10HD). (a) depicts an ideal semantic image segmentation of a camera sensor. (b) shows a rendered view of a semantically segmented point cloud.}
\label{fig:carla}
\end{figure}
Similar to related work like \cite{datondji2016rotation} and \cite{ataer2014calibration}, we use predefined scenes for evaluation. For both CARLA and KITTI, two distinct scenes were defined. For CARLA, we configured urban intersection views from an elevated static perspective to mimic real use cases for roadside infrastructure. Meanwhile, for KITTI, one time frame of two different urban driving sequences was chosen, as we only need one time frame at which the camera will be calibrated. In this case, however, we merged 250 lidar scans around the timestamp of the chosen frame to reconstruct the 3D environment. In the case of CARLA, the segmented point clouds, as well as the segmented camera images of the defined scenes, were generated using the built-in maps. Given completely correct segmentation maps, it is easier to assess the performance of our approach. \ReviewChange{The virtual lidar was configured to have 128 lines and a vertical field of view of $90^\circ$ with the center of the FoV aligned to the ground plane. The virtual segmentation camera used a resolution of 960x600 with a horizontal FoV of $90^\circ$}. For the KITTI data, the Cylinder3D~\cite{zhu2021cylindrical} model, trained with Semantic-KITTI labels \cite{behley2019semantickitti}, was used to generate the point cloud segmentation. Therefore, no ground truth data available in Semantic-KITTI was used. Moreover, the image segmentation was obtained with OCRNet~\cite{yuan2021segmentation}, trained with Cityscapes labels \cite{cordts2016cityscapes}. \ReviewChange{The sensor setup used for recording the raw data passed to the network is described in the KITTI paper~\cite{geiger2013vision}}. As the image segmentation network provided sub-optimal results in some cases, we manually corrected the segmentation to additionally evaluate for a \textit{semi-automatic} setting. In our \textit{automatic} setting, the original image segmentation was utilized without any modification. In all KITTI scenes, road and sidewalk labels were merged in both domains to circumvent noisy borders between ground points which would hinder the optimization. At the same time, the dynamic object removal mentioned in Sec.\ref{lbl:pclseg} was limited to the driveable area covered by the measurement vehicle, which helped retain parked vehicles in the point cloud. This helped to create more visual comparison areas not masked out by $\mathcal{V}_{u,v}$.
Furthermore, during CARLA evaluation, the point cloud view was rendered with a sky background, as visible in Fig. \ref{fig:carla2}. In order to only match the sky region common between both images, sky regions in the rendered view not matching sky regions in the target were masked out, as these pixels are caused by point cloud sparsity. Another efficient measure consisted of only using the bottom half of the image for registration, as in the KITTI lidar data, the point cloud was cropped in the height dimension, resulting in a trimmed view of the scene which introduced high discrepancies in the upper half of the image. This can be clearly seen in Fig. \ref{fig:kitti3}. Finally, the point cloud was cropped to a 75m radius around the initial location to reduce the memory footprint, in the case of KITTI evaluation. The lidar segmentation view was rendered with Pytorch3D \cite{ravi2020pytorch3d}.
\subsection{Evaluation Protocol}
\label{sec:ep}
As the ground truth calibration for both evaluations is known, a random initialization procedure was performed to assess the convergence behavior of the optimization approach. This was implemented by initializing the optimization algorithm with noisy ground truth, where the additive noise was uniformly sampled from the interval $[-2.5\,\si{\meter};2.5\,\si{\meter}]$ for positional offsets and $[-5^\circ;5^\circ]$ for rotational offsets, as mentioned in Sec. \ref{lbl:cdreg}. By using this approach, we can also verify the usability of our approach, as such offsets could appear in field measurements. After the completion of the optimization, the remaining offsets were interpreted as the calibration quality. As all coordinate systems used are scaled to meters, the positional offsets are measured in centimeters. The algorithm is invoked 30 times on each scene and the 10 parameter sets resulting in the lowest loss $\mathcal{L}(\hat{\mathcal{I}}, \mathcal{I}^\text{sem})$ are considered as acceptable results to filter out results that would be considered outliers
, as described in the Result Verification of Sec. \ref{lbl:cdreg}.\\
We then report the mean vector norm of the displacement error vector $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert} = \overline{\lVert\hat{\boldsymbol{t}_\mathrm{ext}} - \boldsymbol{t}_\mathrm{ext}\rVert}$ as well as the mean absolute angular error $\overline{|\Delta_{\alpha}|}$. Furthermore, to evaluate our choice of the L2 loss as a distance metric, we also repeated the previously described evaluation protocol with the Huber loss \cite{huber1992robust}
\begin{align}
\ell_{\mathrm{Huber}}(\boldsymbol{a},\, \boldsymbol{b}) = \begin{cases}
\frac{1}{2}(\boldsymbol{a}- \boldsymbol{b})^T(\boldsymbol{a}-\boldsymbol{b}),&\lVert\boldsymbol{a}-\boldsymbol{b}\rVert<\delta\\
\delta(\lVert\boldsymbol{a}-\boldsymbol{b}\rVert - \frac{\delta}{2}),&\mathrm{otherwise}
\end{cases}
\end{align}
For results using the Huber loss, we set the loss threshold $\delta$ to 0.3.
\begin{table}[h]
\caption{Evaluation results.}
\label{tbl:mce}
\centering
\begin{tabular}{lcccc}
\toprule
\textbf{Loss} & \multicolumn{2}{c}{L2} & \multicolumn{2}{c}{Huber}\\
\midrule
\textbf{Metric} & $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert}$ & $\overline{|\Delta_{\alpha}|}$ & $\overline{\lVert\Delta_{\boldsymbol{t}}\rVert}$ & $\overline{|\Delta_{\alpha}|}$\\
\midrule
\multicolumn{5}{l}{\textbf{Fully automated CARLA-simulated infrastructure}}\\
Town10HD& $1.5\,\si{\cm}$&$0.03^\circ$ & $1.5\,\si{\cm}$&$0.02^\circ$\\
Town01& $6.0\,\si{\cm}$&$0.11^\circ$ & $6.2\,\si{\cm}$&$0.10^\circ$\\
\midrule
\multicolumn{5}{l}{\textbf{Semi-automated KITTI}}\\
Sequence 00 & $12.7\,\si{\cm}$&$0.28^\circ$ & $15.3\,\si{\cm}$&$0.26^\circ$\\
Sequence 09 & $20.6\,\si{\cm}$&$0.40^\circ$ & $19.3\,\si{\cm}$&$0.42^\circ$\\
\midrule
\multicolumn{5}{l}{\textbf{Fully automated KITTI}}\\
Sequence 00 & $20.2\,\si{\cm}$&$0.34^\circ$ & $20.3\,\si{\cm}$&$0.33^\circ$\\
Sequence 09 & $18.6\,\si{\cm}$&$0.40^\circ$ & $19.6\,\si{\cm}$&$0.43^\circ$\\
\bottomrule
\end{tabular}
\end{table}
\subsection{Results Discussion}
\label{sec:rd}
Table~\ref{tbl:mce} summarizes our results for both evaluations and examined loss functions.
For the CARLA evaluation, our translation error is below 7\,\si{\cm} for both scenes, while the rotation error is at most $0.11^\circ$.
~We observed that the errors mostly stem from point cloud sparsity, as can be seen in Fig. \ref{fig:carla}. It should also be noted that, due to the rasterized representation of an image, we cannot distinguish image translations below a single pixel, therefore very small changes of under 1\si{\cm} in camera translation and \ang{0.01} in camera rotation may not be visible in the rendered view. The real-world KITTI scenes also perform very well. While both the point cloud labels and image segmentation contain incorrect labels and, in the case of point clouds, incorrectly measured data points, the final calibration quality is still very high, especially if used for infrastructure sensor calibration. Furthermore, we tested fully automatically labeled KITTI scenes by not manually correcting the image segmentation.% and only combining road and sidewalk labels.
~The performance in these scenes dropped only slightly or even stayed in the same range and is therefore comparable with the semi-automatic use case. However, it should be noted that this performance highly depends on the quality of the sensor data and the dataset used for training the segmentation network. Comparing both loss types, we can see only minor differences, which can be explained by the random initialization and testing approach. Thus, both losses are equally suitable for the task.
\subsection{Comparison with Related Approaches}
\begin{figure}[!t]
\centering
\subfloat[Rectified camera image]{
\includegraphics[width=0.8\linewidth]{KITTI_example_1.png}
\label{fig:kitti1}}
\hfil
\subfloat[Image segmentation]{
\includegraphics[width=0.8\linewidth]{KITTI_example_2.png}
\label{fig:kitti2}}
\hfil
\subfloat[Rendered point cloud view]{
\includegraphics[width=0.8\linewidth]{KITTI_example_3.png}
\label{fig:kitti3}}
\caption{KITTI \cite{geiger2013vision} dataset result (Sequence 00 fully automated). We can see a noticeable visual difference between the segmentation image and the rendered point cloud, which is caused by point cloud sparsity as well as incorrect labels. However, they still share enough common labelfeatures to be reliably matched, as shown in Table \ref{tbl:mce}.}
\label{fig:kitti}
\end{figure}
As our approach is quite different from previous works in this area and no common evaluation protocols are defined, it is difficult to directly compare performance metrics, so we only look at qualitative results. Considering Horn et al. \cite{horn2021online}, who achieve translation errors of $20.8\,\si{\cm}$ and rotation errors of $0.26^\circ$ on the KITTI dataset, but require sensor movement, we get equally good results (see Table \ref{tbl:mce}) without sensor movement. The authors of the mobile stereo camera SLAM approach in \cite{ataer2014calibration} report translation errors in the range of $6\si{\cm}-67 \si{\cm}$ and rotation errors between $1.5^\circ$ and $5.4^\circ$ on their own data. Comparably, our positional errors are on the lower end of the spectrum, while our rotation errors are smaller by a factor of 10. We, therefore, conclude that our approach is competitive with other methods and can be used for infrastructure and IV sensor calibration.
\section{CONCLUSION}
We presented an extrinsic camera calibration approach for both infrastructure and intelligent vehicle cameras.
~In our approach, we obtained a semantic segmentation of both IV-collected lidar data as well as camera data. This segmented data was then used to find an optimal calibration by using an optimization loop to match the appearance of the image segmentation with a rendered view of the segmented lidar data, effectively performing cross-domain registration. The algorithm has been evaluated on simulated CARLA \cite{Dosovitskiy17} scenes and real sensor data from the KITTI dataset \cite{geiger2013vision} to show its feasibility in a practical application, showing equally good or better results than existing approaches, which, however, are more constrained in their application.
\addtolength{\textheight}{-9cm} % This command serves to balance the column lengths
% on the last page of the document manually. It shortens
% the textheight of the last page by a suitable amount.
% This command does not take effect until the next page
% so it should come on the page before the last. Make
% sure that you do not shorten the textheight too much.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section*{APPENDIX}
%
%Appendixes should appear before the acknowledgment.
%
%\section*{ACKNOWLEDGMENT}
%
%The preferred spelling of the word acknowledgment in America is without an e after the g. Avoid the stilted expression, One of us (R. B. G.) thanks . . .
%Instead, try R. B. G. thanks. Put sponsor acknowledgments in the unnumbered footnote on the first page.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{ieeetran}
\bibliography{IEEEexample}
\end{document}
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@rmstyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\bibitem{rinner2008introduction}
B.~Rinner and W.~Wolf, ``An introduction to distributed smart cameras,''
\emph{Proceedings of the IEEE}, vol.~96, no.~10, pp. 1565--1575, 2008.
\bibitem{engel2019deeplocalization}
N.~Engel, S.~Hoermann, M.~Horn, V.~Belagiannis, and K.~Dietmayer,
``Deeplocalization: Landmark-based self-localization with deep neural
networks,'' in \emph{2019 IEEE Intelligent Transportation Systems Conference
(ITSC)}, 2019, pp. 926--933.
\bibitem{wiederer2020traffic}
J.~Wiederer, A.~Bouazizi, U.~Kressel, and V.~Belagiannis, ``Traffic control
gesture recognition for autonomous vehicles,'' in \emph{2020 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)}, 2020, pp.
10\,676--10\,683.
\bibitem{andrew2001multiple}
R.~Hartley and A.~Zisserman, ``Computation of the camera matrix {P},'' in
\emph{Multiple View Geometry in Computer Vision}.\hskip 1em plus 0.5em minus
0.4em\relax Cambridge University Press, 2004, p. 178–236.
\bibitem{datondji2016rotation}
S.~R.~E. Datondji, Y.~Dupuis, P.~Subirats, and P.~Vasseur, ``Rotation and
translation estimation for a wide baseline fisheye-stereo at crossroads based
on traffic flow analysis,'' in \emph{2016 IEEE 19th International Conference
on Intelligent Transportation Systems (ITSC)}, 2016, pp. 1534--1539.
\bibitem{zhu2016robust}
C.~Zhu, Z.~Zhou, Z.~Xing, Y.~Dong, Y.~Ma, and J.~Yu, ``Robust plane-based
calibration of multiple non-overlapping cameras,'' in \emph{2016 Fourth
International Conference on 3D Vision (3DV)}, 2016, pp. 658--666.
\bibitem{horn2021online}
M.~Horn, T.~Wodtko, M.~Buchholz, and K.~Dietmayer, ``Online extrinsic
calibration based on per-sensor ego-motion using dual quaternions,''
\emph{IEEE Robotics and Automation Letters}, vol.~6, no.~2, pp. 982--989,
2021.
\bibitem{li2021deepi2p}
J.~Li and G.~H. Lee, ``Deep{I}2{P}: {I}mage-to-point cloud registration via
deep classification,'' in \emph{Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition}, 2021, pp. 15\,960--15\,969.
\bibitem{ataer2014calibration}
E.~Ataer-Cansizoglu, Y.~Taguchi, S.~Ramalingam, and Y.~Miki, ``Calibration of
non-overlapping cameras using an external {SLAM} system,'' in \emph{2014 2nd
International Conference on 3D Vision}, vol.~1, 2014, pp. 509--516.
\bibitem{Dosovitskiy17}
A.~Dosovitskiy, G.~Ros, F.~Codevilla, A.~Lopez, and V.~Koltun, ``{CARLA}: {An}
open urban driving simulator,'' in \emph{Proceedings of the 1st Annual
Conference on Robot Learning}, 2017, pp. 1--16.
\bibitem{geiger2013vision}
A.~Geiger, P.~Lenz, C.~Stiller, and R.~Urtasun, ``Vision meets {R}obotics: The
{KITTI} {D}ataset,'' \emph{The International Journal of Robotics Research},
vol.~32, no.~11, pp. 1231--1237, 2013.
\bibitem{dubska2014fully}
M.~Dubsk{\'a}, A.~Herout, R.~Jur{\'a}nek, and J.~Sochor, ``Fully automatic
roadside camera calibration for traffic surveillance,'' \emph{IEEE
Transactions on Intelligent Transportation Systems}, vol.~16, no.~3, pp.
1162--1171, 2014.
\bibitem{bartl2021automatic}
V.~Bartl, J.~{\v{S}}pa{\v{n}}hel, P.~Dobe{\v{s}}, R.~Juranek, and A.~Herout,
``Automatic camera calibration by landmarks on rigid objects,'' \emph{Machine
Vision and Applications}, vol.~32, no.~1, pp. 1--13, 2021.
\bibitem{feng20192d3d}
M.~Feng, S.~Hu, M.~H. Ang, and G.~H. Lee, ``2{D}3{D}-{M}atchnet: Learning to
match keypoints across 2{D} image and 3{D} point cloud,'' in \emph{2019
International Conference on Robotics and Automation (ICRA)}, 2019, pp.
4790--4796.
\bibitem{yu2020monocular}
H.~Yu, W.~Zhen, W.~Yang, J.~Zhang, and S.~Scherer, ``Monocular camera
localization in prior lidar maps with 2{D}-3{D} line correspondences,'' in
\emph{2020 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS)}, 2020, pp. 4588--4594.
\bibitem{wei2018calibration}
L.~Wei, L.~Naiguang, D.~Mingli, and L.~Xiaoping, ``Calibration-free
robot-sensor calibration approach based on second-order cone programming,''
in \emph{MATEC Web of Conferences}, vol. 173.\hskip 1em plus 0.5em minus
0.4em\relax EDP Sciences, 2018, p. 02005.
\bibitem{wise2020certifiably}
E.~Wise, M.~Giamou, S.~Khoubyarian, A.~Grover, and J.~Kelly, ``Certifiably
optimal monocular hand-eye calibration,'' in \emph{2020 IEEE International
Conference on Multisensor Fusion and Integration for Intelligent Systems
(MFI)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 271--278.
\bibitem{muller2019laci}
J.~M{\"u}ller, M.~Herrmann, J.~Strohbeck, V.~Belagiannis, and M.~Buchholz,
``{LACI}: {L}ow-effort automatic calibration of infrastructure sensors,'' in
\emph{2019 IEEE Intelligent Transportation Systems Conference (ITSC)}, 2019,
pp. 3928--3933.
\bibitem{nelder1965simplex}
J.~A. Nelder and R.~Mead, ``A simplex method for function minimization,''
\emph{The computer journal}, vol.~7, no.~4, pp. 308--313, 1965.
\bibitem{chen1992object}
Y.~Chen and G.~Medioni, ``Object modelling by registration of multiple range
images,'' \emph{Image and vision computing}, vol.~10, no.~3, pp. 145--155,
1992.
\bibitem{choi2015robust}
S.~Choi, Q.-Y. Zhou, and V.~Koltun, ``Robust reconstruction of indoor scenes,''
in \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition}, 2015, pp. 5556--5565.
\bibitem{zhu2021cylindrical}
X.~Zhu, H.~Zhou, T.~Wang, F.~Hong, Y.~Ma, W.~Li, H.~Li, and D.~Lin,
``Cylindrical and asymmetrical 3{D} convolution networks for lidar
segmentation,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition}, 2021, pp. 9939--9948.
\bibitem{behley2019semantickitti}
J.~Behley, M.~Garbade, A.~Milioto, J.~Quenzel, S.~Behnke, C.~Stachniss, and
J.~Gall, ``Semantic{KITTI}: A dataset for semantic scene understanding of
lidar sequences,'' in \emph{Proceedings of the IEEE/CVF International
Conference on Computer Vision}, 2019, pp. 9297--9307.
\bibitem{cordts2016cityscapes}
M.~Cordts, M.~Omran, S.~Ramos, T.~Rehfeld, M.~Enzweiler, R.~Benenson,
U.~Franke, S.~Roth, and B.~Schiele, ``The {C}ityscapes dataset for semantic
urban scene understanding,'' in \emph{Proceedings of the IEEE conference on
computer vision and pattern recognition}, 2016, pp. 3213--3223.
\bibitem{yuan2021segmentation}
Y.~Yuan, X.~Chen, X.~Chen, and J.~Wang, ``Segmentation transformer:
{O}bject-contextual representations for semantic segmentation,'' in
\emph{European Conference on Computer Vision (ECCV)}, vol.~1, 2021.
\bibitem{ravi2020pytorch3d}
N.~Ravi, J.~Reizenstein, D.~Novotny, T.~Gordon, W.-Y. Lo, J.~Johnson, and
G.~Gkioxari, ``Accelerating 3{D} {D}eep {L}earning with {P}y{T}orch3{D},''
\emph{arXiv:2007.08501}, 2020.
\bibitem{huber1992robust}
P.~J. Huber, ``Robust estimation of a location parameter,'' in
\emph{Breakthroughs in statistics}.\hskip 1em plus 0.5em minus 0.4em\relax
Springer, 1992, pp. 492--518.
\end{thebibliography}
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@rmstyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\bibitem{rinner2008introduction}
B.~Rinner and W.~Wolf, ``An introduction to distributed smart cameras,''
\emph{Proceedings of the IEEE}, vol.~96, no.~10, pp. 1565--1575, 2008.
\bibitem{engel2019deeplocalization}
N.~Engel, S.~Hoermann, M.~Horn, V.~Belagiannis, and K.~Dietmayer,
``Deeplocalization: Landmark-based self-localization with deep neural
networks,'' in \emph{2019 IEEE Intelligent Transportation Systems Conference
(ITSC)}, 2019, pp. 926--933.
\bibitem{wiederer2020traffic}
J.~Wiederer, A.~Bouazizi, U.~Kressel, and V.~Belagiannis, ``Traffic control
gesture recognition for autonomous vehicles,'' in \emph{2020 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)}, 2020, pp.
10\,676--10\,683.
\bibitem{andrew2001multiple}
R.~Hartley and A.~Zisserman, ``Computation of the camera matrix {P},'' in
\emph{Multiple View Geometry in Computer Vision}.\hskip 1em plus 0.5em minus
0.4em\relax Cambridge University Press, 2004, p. 178–236.
\bibitem{datondji2016rotation}
S.~R.~E. Datondji, Y.~Dupuis, P.~Subirats, and P.~Vasseur, ``Rotation and
translation estimation for a wide baseline fisheye-stereo at crossroads based
on traffic flow analysis,'' in \emph{2016 IEEE 19th International Conference
on Intelligent Transportation Systems (ITSC)}, 2016, pp. 1534--1539.
\bibitem{zhu2016robust}
C.~Zhu, Z.~Zhou, Z.~Xing, Y.~Dong, Y.~Ma, and J.~Yu, ``Robust plane-based
calibration of multiple non-overlapping cameras,'' in \emph{2016 Fourth
International Conference on 3D Vision (3DV)}, 2016, pp. 658--666.
\bibitem{horn2021online}
M.~Horn, T.~Wodtko, M.~Buchholz, and K.~Dietmayer, ``Online extrinsic
calibration based on per-sensor ego-motion using dual quaternions,''
\emph{IEEE Robotics and Automation Letters}, vol.~6, no.~2, pp. 982--989,
2021.
\bibitem{li2021deepi2p}
J.~Li and G.~H. Lee, ``Deep{I}2{P}: {I}mage-to-point cloud registration via
deep classification,'' in \emph{Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition}, 2021, pp. 15\,960--15\,969.
\bibitem{ataer2014calibration}
E.~Ataer-Cansizoglu, Y.~Taguchi, S.~Ramalingam, and Y.~Miki, ``Calibration of
non-overlapping cameras using an external {SLAM} system,'' in \emph{2014 2nd
International Conference on 3D Vision}, vol.~1, 2014, pp. 509--516.
\bibitem{Dosovitskiy17}
A.~Dosovitskiy, G.~Ros, F.~Codevilla, A.~Lopez, and V.~Koltun, ``{CARLA}: {An}
open urban driving simulator,'' in \emph{Proceedings of the 1st Annual
Conference on Robot Learning}, 2017, pp. 1--16.
\bibitem{geiger2013vision}
A.~Geiger, P.~Lenz, C.~Stiller, and R.~Urtasun, ``Vision meets {R}obotics: The
{KITTI} {D}ataset,'' \emph{The International Journal of Robotics Research},
vol.~32, no.~11, pp. 1231--1237, 2013.
\bibitem{dubska2014fully}
M.~Dubsk{\'a}, A.~Herout, R.~Jur{\'a}nek, and J.~Sochor, ``Fully automatic
roadside camera calibration for traffic surveillance,'' \emph{IEEE
Transactions on Intelligent Transportation Systems}, vol.~16, no.~3, pp.
1162--1171, 2014.
\bibitem{bartl2021automatic}
V.~Bartl, J.~{\v{S}}pa{\v{n}}hel, P.~Dobe{\v{s}}, R.~Juranek, and A.~Herout,
``Automatic camera calibration by landmarks on rigid objects,'' \emph{Machine
Vision and Applications}, vol.~32, no.~1, pp. 1--13, 2021.
\bibitem{feng20192d3d}
M.~Feng, S.~Hu, M.~H. Ang, and G.~H. Lee, ``2{D}3{D}-{M}atchnet: Learning to
match keypoints across 2{D} image and 3{D} point cloud,'' in \emph{2019
International Conference on Robotics and Automation (ICRA)}, 2019, pp.
4790--4796.
\bibitem{yu2020monocular}
H.~Yu, W.~Zhen, W.~Yang, J.~Zhang, and S.~Scherer, ``Monocular camera
localization in prior lidar maps with 2{D}-3{D} line correspondences,'' in
\emph{2020 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS)}, 2020, pp. 4588--4594.
\bibitem{wei2018calibration}
L.~Wei, L.~Naiguang, D.~Mingli, and L.~Xiaoping, ``Calibration-free
robot-sensor calibration approach based on second-order cone programming,''
in \emph{MATEC Web of Conferences}, vol. 173.\hskip 1em plus 0.5em minus
0.4em\relax EDP Sciences, 2018, p. 02005.
\bibitem{wise2020certifiably}
E.~Wise, M.~Giamou, S.~Khoubyarian, A.~Grover, and J.~Kelly, ``Certifiably
optimal monocular hand-eye calibration,'' in \emph{2020 IEEE International
Conference on Multisensor Fusion and Integration for Intelligent Systems
(MFI)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 271--278.
\bibitem{muller2019laci}
J.~M{\"u}ller, M.~Herrmann, J.~Strohbeck, V.~Belagiannis, and M.~Buchholz,
``{LACI}: {L}ow-effort automatic calibration of infrastructure sensors,'' in
\emph{2019 IEEE Intelligent Transportation Systems Conference (ITSC)}, 2019,
pp. 3928--3933.
\bibitem{nelder1965simplex}
J.~A. Nelder and R.~Mead, ``A simplex method for function minimization,''
\emph{The computer journal}, vol.~7, no.~4, pp. 308--313, 1965.
\bibitem{chen1992object}
Y.~Chen and G.~Medioni, ``Object modelling by registration of multiple range
images,'' \emph{Image and vision computing}, vol.~10, no.~3, pp. 145--155,
1992.
\bibitem{choi2015robust}
S.~Choi, Q.-Y. Zhou, and V.~Koltun, ``Robust reconstruction of indoor scenes,''
in \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition}, 2015, pp. 5556--5565.
\bibitem{zhu2021cylindrical}
X.~Zhu, H.~Zhou, T.~Wang, F.~Hong, Y.~Ma, W.~Li, H.~Li, and D.~Lin,
``Cylindrical and asymmetrical 3{D} convolution networks for lidar
segmentation,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition}, 2021, pp. 9939--9948.
\bibitem{behley2019semantickitti}
J.~Behley, M.~Garbade, A.~Milioto, J.~Quenzel, S.~Behnke, C.~Stachniss, and
J.~Gall, ``Semantic{KITTI}: A dataset for semantic scene understanding of
lidar sequences,'' in \emph{Proceedings of the IEEE/CVF International
Conference on Computer Vision}, 2019, pp. 9297--9307.
\bibitem{cordts2016cityscapes}
M.~Cordts, M.~Omran, S.~Ramos, T.~Rehfeld, M.~Enzweiler, R.~Benenson,
U.~Franke, S.~Roth, and B.~Schiele, ``The {C}ityscapes dataset for semantic
urban scene understanding,'' in \emph{Proceedings of the IEEE conference on
computer vision and pattern recognition}, 2016, pp. 3213--3223.
\bibitem{yuan2021segmentation}
Y.~Yuan, X.~Chen, X.~Chen, and J.~Wang, ``Segmentation transformer:
{O}bject-contextual representations for semantic segmentation,'' in
\emph{European Conference on Computer Vision (ECCV)}, vol.~1, 2021.
\bibitem{ravi2020pytorch3d}
N.~Ravi, J.~Reizenstein, D.~Novotny, T.~Gordon, W.-Y. Lo, J.~Johnson, and
G.~Gkioxari, ``Accelerating 3{D} {D}eep {L}earning with {P}y{T}orch3{D},''
\emph{arXiv:2007.08501}, 2020.
\bibitem{huber1992robust}
P.~J. Huber, ``Robust estimation of a location parameter,'' in
\emph{Breakthroughs in statistics}.\hskip 1em plus 0.5em minus 0.4em\relax
Springer, 1992, pp. 492--518.
\end{thebibliography}
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@rmstyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\bibitem{rinner2008introduction}
B.~Rinner and W.~Wolf, ``An introduction to distributed smart cameras,''
\emph{Proceedings of the IEEE}, vol.~96, no.~10, pp. 1565--1575, 2008.
\bibitem{engel2019deeplocalization}
N.~Engel, S.~Hoermann, M.~Horn, V.~Belagiannis, and K.~Dietmayer,
``Deeplocalization: Landmark-based self-localization with deep neural
networks,'' in \emph{2019 IEEE Intelligent Transportation Systems Conference
(ITSC)}, 2019, pp. 926--933.
\bibitem{wiederer2020traffic}
J.~Wiederer, A.~Bouazizi, U.~Kressel, and V.~Belagiannis, ``Traffic control
gesture recognition for autonomous vehicles,'' in \emph{2020 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)}, 2020, pp.
10\,676--10\,683.
\bibitem{andrew2001multiple}
R.~Hartley and A.~Zisserman, ``Computation of the camera matrix {P},'' in
\emph{Multiple View Geometry in Computer Vision}.\hskip 1em plus 0.5em minus
0.4em\relax Cambridge University Press, 2004, p. 178–236.
\bibitem{datondji2016rotation}
S.~R.~E. Datondji, Y.~Dupuis, P.~Subirats, and P.~Vasseur, ``Rotation and
translation estimation for a wide baseline fisheye-stereo at crossroads based
on traffic flow analysis,'' in \emph{2016 IEEE 19th International Conference
on Intelligent Transportation Systems (ITSC)}, 2016, pp. 1534--1539.
\bibitem{zhu2016robust}
C.~Zhu, Z.~Zhou, Z.~Xing, Y.~Dong, Y.~Ma, and J.~Yu, ``Robust plane-based
calibration of multiple non-overlapping cameras,'' in \emph{2016 Fourth
International Conference on 3D Vision (3DV)}, 2016, pp. 658--666.
\bibitem{horn2021online}
M.~Horn, T.~Wodtko, M.~Buchholz, and K.~Dietmayer, ``Online extrinsic
calibration based on per-sensor ego-motion using dual quaternions,''
\emph{IEEE Robotics and Automation Letters}, vol.~6, no.~2, pp. 982--989,
2021.
\bibitem{li2021deepi2p}
J.~Li and G.~H. Lee, ``Deep{I}2{P}: {I}mage-to-point cloud registration via
deep classification,'' in \emph{Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition}, 2021, pp. 15\,960--15\,969.
\bibitem{ataer2014calibration}
E.~Ataer-Cansizoglu, Y.~Taguchi, S.~Ramalingam, and Y.~Miki, ``Calibration of
non-overlapping cameras using an external {SLAM} system,'' in \emph{2014 2nd
International Conference on 3D Vision}, vol.~1, 2014, pp. 509--516.
\bibitem{Dosovitskiy17}
A.~Dosovitskiy, G.~Ros, F.~Codevilla, A.~Lopez, and V.~Koltun, ``{CARLA}: {An}
open urban driving simulator,'' in \emph{Proceedings of the 1st Annual
Conference on Robot Learning}, 2017, pp. 1--16.
\bibitem{geiger2013vision}
A.~Geiger, P.~Lenz, C.~Stiller, and R.~Urtasun, ``Vision meets {R}obotics: The
{KITTI} {D}ataset,'' \emph{The International Journal of Robotics Research},
vol.~32, no.~11, pp. 1231--1237, 2013.
\bibitem{dubska2014fully}
M.~Dubsk{\'a}, A.~Herout, R.~Jur{\'a}nek, and J.~Sochor, ``Fully automatic
roadside camera calibration for traffic surveillance,'' \emph{IEEE
Transactions on Intelligent Transportation Systems}, vol.~16, no.~3, pp.
1162--1171, 2014.
\bibitem{bartl2021automatic}
V.~Bartl, J.~{\v{S}}pa{\v{n}}hel, P.~Dobe{\v{s}}, R.~Juranek, and A.~Herout,
``Automatic camera calibration by landmarks on rigid objects,'' \emph{Machine
Vision and Applications}, vol.~32, no.~1, pp. 1--13, 2021.
\bibitem{feng20192d3d}
M.~Feng, S.~Hu, M.~H. Ang, and G.~H. Lee, ``2{D}3{D}-{M}atchnet: Learning to
match keypoints across 2{D} image and 3{D} point cloud,'' in \emph{2019
International Conference on Robotics and Automation (ICRA)}, 2019, pp.
4790--4796.
\bibitem{yu2020monocular}
H.~Yu, W.~Zhen, W.~Yang, J.~Zhang, and S.~Scherer, ``Monocular camera
localization in prior lidar maps with 2{D}-3{D} line correspondences,'' in
\emph{2020 IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS)}, 2020, pp. 4588--4594.
\bibitem{wei2018calibration}
L.~Wei, L.~Naiguang, D.~Mingli, and L.~Xiaoping, ``Calibration-free
robot-sensor calibration approach based on second-order cone programming,''
in \emph{MATEC Web of Conferences}, vol. 173.\hskip 1em plus 0.5em minus
0.4em\relax EDP Sciences, 2018, p. 02005.
\bibitem{wise2020certifiably}
E.~Wise, M.~Giamou, S.~Khoubyarian, A.~Grover, and J.~Kelly, ``Certifiably
optimal monocular hand-eye calibration,'' in \emph{2020 IEEE International
Conference on Multisensor Fusion and Integration for Intelligent Systems
(MFI)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 271--278.
\bibitem{muller2019laci}
J.~M{\"u}ller, M.~Herrmann, J.~Strohbeck, V.~Belagiannis, and M.~Buchholz,
``{LACI}: {L}ow-effort automatic calibration of infrastructure sensors,'' in
\emph{2019 IEEE Intelligent Transportation Systems Conference (ITSC)}, 2019,
pp. 3928--3933.
\bibitem{nelder1965simplex}
J.~A. Nelder and R.~Mead, ``A simplex method for function minimization,''
\emph{The computer journal}, vol.~7, no.~4, pp. 308--313, 1965.
\bibitem{chen1992object}
Y.~Chen and G.~Medioni, ``Object modelling by registration of multiple range
images,'' \emph{Image and vision computing}, vol.~10, no.~3, pp. 145--155,
1992.
\bibitem{choi2015robust}
S.~Choi, Q.-Y. Zhou, and V.~Koltun, ``Robust reconstruction of indoor scenes,''
in \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition}, 2015, pp. 5556--5565.
\bibitem{zhu2021cylindrical}
X.~Zhu, H.~Zhou, T.~Wang, F.~Hong, Y.~Ma, W.~Li, H.~Li, and D.~Lin,
``Cylindrical and asymmetrical 3{D} convolution networks for lidar
segmentation,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition}, 2021, pp. 9939--9948.
\bibitem{behley2019semantickitti}
J.~Behley, M.~Garbade, A.~Milioto, J.~Quenzel, S.~Behnke, C.~Stachniss, and
J.~Gall, ``Semantic{KITTI}: A dataset for semantic scene understanding of
lidar sequences,'' in \emph{Proceedings of the IEEE/CVF International
Conference on Computer Vision}, 2019, pp. 9297--9307.
\bibitem{cordts2016cityscapes}
M.~Cordts, M.~Omran, S.~Ramos, T.~Rehfeld, M.~Enzweiler, R.~Benenson,
U.~Franke, S.~Roth, and B.~Schiele, ``The {C}ityscapes dataset for semantic
urban scene understanding,'' in \emph{Proceedings of the IEEE conference on
computer vision and pattern recognition}, 2016, pp. 3213--3223.
\bibitem{yuan2021segmentation}
Y.~Yuan, X.~Chen, X.~Chen, and J.~Wang, ``Segmentation transformer:
{O}bject-contextual representations for semantic segmentation,'' in
\emph{European Conference on Computer Vision (ECCV)}, vol.~1, 2021.
\bibitem{ravi2020pytorch3d}
N.~Ravi, J.~Reizenstein, D.~Novotny, T.~Gordon, W.-Y. Lo, J.~Johnson, and
G.~Gkioxari, ``Accelerating 3{D} {D}eep {L}earning with {P}y{T}orch3{D},''
\emph{arXiv:2007.08501}, 2020.
\bibitem{huber1992robust}
P.~J. Huber, ``Robust estimation of a location parameter,'' in
\emph{Breakthroughs in statistics}.\hskip 1em plus 0.5em minus 0.4em\relax
Springer, 1992, pp. 492--518.
\end{thebibliography}