Metric, inertially aligned monocular state estimation via kinetodynamic priors
Extracted source text · arXiv 2511.20496
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
% The following packages can be found on http:\\www.ctan.org
\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
\usepackage{mathptmx} % assumes new font selection scheme installed
\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath}
\usepackage{bm}
\usepackage{amssymb} % assumes amsmath package installed
\usepackage{booktabs}
\usepackage{gensymb}
\usepackage{multirow}
\usepackage{subcaption}
\usepackage{stfloats}
\usepackage{array}
\usepackage{siunitx} % For aligning numbers by decimal point
\usepackage{graphicx} % For \resizebox
\usepackage[ruled,vlined]{algorithm2e}
\usepackage[section]{placeins}
\usepackage{utfsym}
\usepackage{hyperref}
\hypersetup{hypertex=true}
\pdfminorversion=4
\title{\LARGE \bf
Metric, inertially aligned monocular state estimation via kinetodynamic priors
}
\author{ Jiaxin Liu$^{1*}$, Min Li$^{1*}$, Wanting Xu$^{1}$, Liang Li$^{2}$, Jiaqi Yang$^{1}$ and Laurent Kneip$^{1}$
\thanks{$^{*}$Two authors contribute equally in this work}% <-this % stops a space
\thanks{$^{1}$Min Li, Jiaxin Liu, Wanting Xu, Jiaqin Yang and Laurent Kneip are with School of Information Science and Technology, ShanghaiTech University, Shanghai, China. \{limin1, liujx2024, xuwt, yangjq, lkneip\}@shanghaitech.edu.cn}%
\thanks{$^{2}$Liang Li is with the Department of Mechanical Engineering, The University of Hongkong, Hongkong S.A.R., China. llihku@connect.hku.hk} %
\thanks {Corresponding author: Laurent Kneip, lkneip@shanghaitech.edu.cn}
}
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{sec/main/0_Abstract}
\input{sec/main/1_Introduction}
\input{sec/main/2_Relatedwork}
\input{sec/main/3_Methodology}
\input{sec/main/4_Experiments}
\input{sec/main/6_Ablation}
\input{sec/main/5_Conclusion}
\input{sec/appendix/appendix}
{
\small
\bibliographystyle{IEEEtran}
\bibliography{IEEEexample}
}
\end{document}
\begin{abstract}
Accurate state estimation for flexible robotic systems poses significant challenges, particular for platforms with dynamically deforming structures that invalidate rigid-body assumptions. This paper tackles this problem and allows to extend existing rigid-body pose estimation methods to non-rigid systems. Our approach hinges on two core assumptions: first, the elastic properties are captured by an injective deformation-force model, efficiently learned via a Multi-Layer Perceptron; second, we solve the platform's inherently smooth motion using continuous-time B-spline kinematic models. By continuously applying Newton's Second Law, our method establishes a physical link between visually-derived trajectory acceleration and predicted deformation-induced acceleration. We demonstrate that our approach not only enables robust and accurate pose estimation on non-rigid platforms, but that the properly modeled platform physics instigate inertial sensing properties. We demonstrate this feasibility on a simple spring-camera system, and show how it robustly resolves the typically ill-posed problem of metric scale and gravity recovery in monocular visual odometry.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
\begin{figure}[t]
\includegraphics[width=0.5\textwidth]{Figs/Nonrigid_System.pdf}
\vspace{-2 cm}
\caption{Non-rigid monocular camera system demonstrating trajectory divergence. Unlike rigid systems, the deformable connection induces camera oscillations (green) distinct from the platform's motion (blue).}
\vspace{-0.6 cm}
\label{fig:nonrigid_system}
\end{figure}
Accurate environmental perception and precise self-pose estimation are paramount for autonomous navigation, human-robot collaboration, and complex robotic tasks. Traditionally, these capabilities have relied on rigid-body assumptions, simplifying multi-sensor fusion and motion estimation. However, the burgeoning field of soft robotics and flexible systems—such as compliant manipulators and UAVs—challenges this paradigm \cite{chen2025survey}. While valued for their inherent safety, enhanced dexterity, and potential for lower manufacturing costs, these non-rigid systems leverage structural deformation for environmental adaptation, which critically introduces dynamic, time-variant relative sensor poses(Fig. \ref{fig:nonrigid_system}). This invalidates classical rigid-body algorithms, presenting significant perception and state estimation challenges \cite{thanabalan2020learning}. In this work, we demonstrate that non-rigid platforms may not necessarily complicate state estimation and sensor fusion. Rather, we show how non-rigid elements and kinetodynamic priors \cite{agudo2015sequential} can even add additional constraints to the system. The leading example we use is monocular motion estimation. While the recovery of scale and inertial alignment are severely ill-posed problems that would normally require fusion with additional sensors \cite{kim2021probabilistic,el2020monocular}, the exploitation of motion and deformation priors perhaps surprisingly renders these dimensions observable.
This work draws inspiration from the work of Li et al. \cite{li2024relative}, which demonstrated the successful embedding of a physical deformation model into a non-rigid multi-perspective camera system. Intriguingly, the deformable connections between sensors in that work acted as a ``passive IMU". Building on this insight, but targeting robust state estimation using a single exteroceptive sensor, this paper addresses a key limitation of prior models. We adopt a mechanical design similar to the Zebedee system \cite{bosse2012zebedee}, where a monocular camera is connected to a mobile platform via a spring mechanism. While this setup serves as a scientific example to explore the potential of priors in non-rigid platforms, we note that there are practically used multi-sensor heads in which non-rigidity was actively added to, for example, enlarge the perceptive field of view \cite{zlot2014three,martinez2019continuous}.
To robustify state estimation under such dynamic non-rigid motion, we explicitly incorporate kinetodynamic priors. This work introduces a novel framework that unifies kinematic and dynamic constraints for non-rigid systems, built upon two core assumptions: 1) Continuous-time Kinematic Models: We use B-Splines to model the smooth motion of at least one platform point, enabling the derivation of high-order derivatives crucial for dynamic analysis. 2) Learned Deformation-force Model: The platform's elastic properties are captured through an injective deformation-force model, efficiently learned via a Multi-Layer Perceptron \cite{li2023mlp}, thus bypassing computationally expensive Finite Element Analysis \cite{lahariya2022learning}. This approach resolves critical challenges like scale and gravity recovery in monocular visual odometry. It continuously applies Newton's Second Law, establishing a physical relationship between the camera's visually derived trajectory acceleration and the acceleration predicted by our learned deformation-force model. Minimizing the discrepancy between these accelerations allows us to determine the unknown scale factor, enabling metric pose and scale estimation from monocular vision \cite{tian2021accurate}. Effectively, any camera motion not explained by the smooth body trajectory is attributed to spring deformations, thus aligning with the support structure's physical properties through an inertial aligning transformation to achieve metric and inertially aligned monocular ego-motion estimation.
Our contributions are summarized as follows:
\begin{itemize}
\item We introduce compact neural representations for modeling elastic deformation properties of sensor support platforms, coupled with a calibration method using a motion capture device.
\item We demonstrate how the combination of a suitable body motion model and an elastic deformation model can be leveraged for passive inertial sensing and accurate monocular exteroceptive motion estimation in non-rigid scenarios.
\item We present the complete computational paradigm, encompassing numerical differentiation of the camera trajectory, variable initialization, and an optimization framework with an embedded, differentiable neural body deformation model.
\end{itemize}
As demonstrated by our results, this outlined setup develops inherent passive inertial sensing capabilities, demonstrating the feasibility of accurate real-world motion estimation from ego-centric video through a mostly mathematical model. Although tested on an exemplary setup, we believe this approach holds significant promise and is applicable to a wide range of future robotic platforms possessing specific motion models and potentially elastic actuation chains.
\section{Related Work}
\label{sec:relatedwork}
\subsection{State Estimation on Non-Rigid Platforms}
The integration of flexible elements into robotic systems, such as deformable UAV wings \cite{hutter2012high} and soft robots \cite{hinzmann2018flexible,hinzmann2019flexible,foehn2022agilicious,hopkins2009survey}, has gained significant traction. This paradigm contrasts sharply with traditional robotics, which assumes sensors follow rigid trajectories. The dynamic changes in sensor-to-body transformations pose a significant challenge to conventional state estimation algorithms that rely on rigid-body assumptions. A few exceptions exist that explicitly model non-rigid systems, such as the work by Peng et al. \cite{peng2019articulated} and Hinzmann et al. \cite{hinzmann2018flexible,hinzmann2019flexible}. However, these methods are either limited to non-elastic \cite{peng2019articulated} or static scenarios \cite{li2024relative} or depend on overlapping sensor fields of view \cite{hinzmann2019flexible}. Our work, in contrast, tackles the more challenging problem of pure monocular visual state estimation on a non-rigid platform in dynamic environments. The Zebedee system \cite{bosse2012zebedee} is a seminal example that leverages a platform’s elastic motion to extend sensor viewpoints, thereby achieving large-scale 3D coverage, but our approach focuses on using the non-rigidity itself as a source of information.
\subsection{Monocular Visual and Multi-Sensor State Estimation}
Monocular cameras are simple and cheap, but they suffer from a fundamental limitation: scale ambiguity. Classic methods like ORB-SLAM \cite{mur2015orb} and COLMAP \cite{schoenberger2016sfm} can reconstruct 3D maps and trajectories, but their scale is always relative. This means the reconstructed environment can be resized by an unknown factor, making it unsuitable for tasks requiring absolute measurements. To overcome this, many systems fuse a camera with other sensors. For example, VINS-Mono \cite{qin2018vins} and maplab \cite{schneider2018maplab} combine a camera with an IMU to resolve scale using inertial data. Similarly, other methods use LiDAR \cite{shan2020lio} or GPS \cite{cao2022gvins}. While effective, these solutions increase hardware cost and complexity. Recently, learned-based methods like Mast3r-SLAM \cite{murai2025mast3r} and VGGt-SLAM \cite{maggio2025vggt} have tried to predict scale directly from images. However, these often require significant computation and cannot be used in a real-time system.
Instead of adding new sensors, we use the non-rigid platform's own elastic properties to resolve scale ambiguity. By modeling the deformation, we achieve metrically accurate state estimation with just a single camera, providing a novel solution for flexible robotics without additional hardware.
\subsection{Passive Sensing and Kinetodynamic Modeling}
Our work is inspired by the concept of ``passive inertial sensing" from Li et al. \cite{li2024relative}, who demonstrated that a sensor's support deformation under gravity can be used to infer inertial information. However, their method was limited to a specific, static physical model. To overcome the limitations of traditional physical models, we employ a more general, data-driven neural model to characterize complex, continuous deformations. While neural networks have been applied to model soft robot control \cite{zheng2020robust} and simulate soft body deformations \cite{wang2024pinn}, these works primarily focus on feed-forward or control problems. In contrast, our approach innovatively integrates a differentiable neural representation of the non-rigid connection's physical properties directly into a monocular visual state estimation framework. This allows us to perform passive perception of continuous deformation states, offering a new perspective for flexible robotics and embodied intelligence research.
\section{System Overview}
\begin{figure*}[t]
\includegraphics[width=2.0\columnwidth]{Figs/pipeline_2.pdf}
% \vspace{-1 cm}
\caption{Block diagram of this pipeline, mainly contains two part. \textbf{a) Modeling:} We use a Deformation-force Network (DFN) to implicitly model the non-rigid connection, incorporating both linear and angular accelerations. It's important to note that our modeling is offline. \textbf{b) Solving:} We utilize odometry to recover the camera's trajectory first, as shown in the green section of the figure. An optimization objective function is then formulated based on Eq. \ref{opt-bspline} and minimized. Specifically, we optimize the control knots in B-Spline instead of optimizing the pose directly, as shown in the brown section of the figure.}
% \vspace{-0.5 cm}
\label{fig:overview_pipeline}
\end{figure*}
% \subsection{Overview}
Fig. \ref{fig:overview_pipeline} illustrates our main pipeline, which consists of a camera (denoted as $c$), a moving platform (denoted as $b$), and a non-rigid, deformable connection between them. Unlike traditional rigid camera systems, our approach models this non-rigid link to recover overall motion.
Our method involves two main steps: \textbf{a) Modeling} We introduce a Deformation-force Network to model the non-rigid connection. \textbf{b) Optimization}: The trajectory and acceleration of the camera could be estimated by odometry, which then feeds into an optimization process to restore the trajectory and scale of the moving platform.
More detailed information on these stages can be found in Section \ref{subsec:deformation_force_network} and Section \ref{sec:Scale recovery and state estimation}, respectively.
\label{sec:methodology}
\subsection{Preliminaries: B-Spline for $\mathbb{SE}3$}
For a $k$-th order B-Spline with control points $\mathbf{T}_i, \mathbf{T}_{i+1}, \dots, \mathbf{T}_N$, the spline is defined as:
\begin{equation}
\label{bspline}
\mathbf{T}(t)=\mathbf{T}_i\prod_{j=1}^{k-1}\text{Exp}(\widetilde{\mathbf{B}}_{j}(t)\cdot\text{Log}(\mathbf{T}_{i+j-1}^{-1}\cdot\mathbf{T}_{i+j})),
\end{equation}
where $\text{Exp}(\cdot)$ and $\text{Log}(\cdot)$ represent the exponential and logarithmic mappings on $\mathbb{SE}3$, respectively. The basis functions $\widetilde{\mathbf{B}}_{j}(t)$ are determined by the blending matrix $\widetilde{\mathbf{M}}$ with entries:
\begin{equation}
\begin{aligned}
m_{s,n}^{(k)}=\frac{C_{k-1}^n}{(k-1)!}\sum_{l=s}^{k-1}(-1)^{l-s}C_k^{l-s}(k-1-l)^{k-1-n}\\ s,n\in{0,..,k-1}.
\end{aligned}
\end{equation}
The acceleration can be derived by defining $\mathbf{A}_j(t)=\text{Exp}(\widetilde{\mathbf{B}}_{j}(t))\text{Log}(\mathbf{T}_{i+j-1}^{-1}\cdot\mathbf{T}_{i+j})$, leading to:
\begin{equation}
\dot{\mathbf{T}}(t)=\mathbf{T}_i\cdot\sum_{j=1}^{k-1} \left( \prod_{l=1}^{j-1}\mathbf{A}_l(t) \right) \dot{\mathbf{A}}_j(t) \left( \prod_{l=j+1}^{k-1} \mathbf{A}_l(t) \right).
\end{equation}
The aforementioned formula can give the linear and angular accelerations from the visual odometry trajectory, which are then used to construct the optimization term.
\subsection{Modeling: Deformation-force Network}
\label{subsec:deformation_force_network}
The mechanical properties of a nonrigid connection can be modeled using a nonlinear differential equation with damping characteristics \cite{friswell2010dynamics}:
\begin{equation}
\boldsymbol{f}_{s}=\kappa_1\Delta \boldsymbol{x} + \kappa_3\Delta \boldsymbol{x}^3 + c\frac{d \Delta \boldsymbol{x}}{d t},
\end{equation}
where $\boldsymbol{f}_{s}$ is the elastic force exerted by the non-rigid connection on the camera. Under typical low-speed conditions, we neglect air resistance and potential collisions. Combining the Newton’s second law, the relationship between the camera's acceleration and displacement can be implicitly modeled as:
\begin{equation}
\boldsymbol{a}_{c}^{linear} - \boldsymbol{g} = F(\Delta \boldsymbol{x}),
\end{equation}
here, $\mathbf{F}$ is the function of $\Delta \boldsymbol{x}$. We propose using a neural network \cite{hruby2022learning} to fit this mapping function. Similarly, angular acceleration can be modeled as:
\begin{equation}
\boldsymbol{a}_{c}^{angular} = G(\Delta \boldsymbol{\theta}).
\end{equation}
Combining linear and angular accelerations together:
\begin{equation}
\boldsymbol{a}_{c} - \boldsymbol{g} = H(\Delta \boldsymbol{x}, \Delta\boldsymbol{\theta}),
\end{equation}
where $\boldsymbol{a}_{c} = \begin{bmatrix} \boldsymbol{a}_{c}^{linear} \\ \boldsymbol{a}_{c}^{angular} \end{bmatrix}$ and $\boldsymbol{g} = \begin{bmatrix} \boldsymbol{g} \\ \mathbf{0} \end{bmatrix}$. The spring displacement $\Delta \boldsymbol{x}$ and twist $\Delta\boldsymbol{\theta}$ represent the relative pose between the base ($\mathbf{T}_{b}$) and the camera ($\mathbf{T}_{c}$). This essentially establishes a mapping from relative pose to camera acceleration. Therefore, we propose to model the non-rigid connection by fitting this mapping using a neural network:
\begin{equation}
\boldsymbol{a}_{c} - \boldsymbol{g} = \mathcal{N}(\mathbf{T}_{b}^{-1}\mathbf{T}_{c}).
\end{equation}
It takes 6 DoF (rotation and translation) as input, as in \cite{gonzalez2021l6dnet}. The network, a Multi-Layer Perceptron (MLP), outputs the camera's acceleration, maintaining simplicity for subsequent nonlinear optimization.
We denote the world coordinate system of the $i$-th sequence as $gt_i$. Given the base and camera poses, $\mathbf{R}_{b}^{gt_i}$ and $\mathbf{R}_{c}^{gt_i}$ respectively, we transform the B-Spline fitted acceleration to keep the reference coordinate:
\begin{equation}
\label{eq:normlization}
\boldsymbol{a}^{c} - \boldsymbol{g}^{c} = (\mathbf{R}_{c}^{gt_i})^{-1}(\boldsymbol{a}^{gt_i} - \boldsymbol{g}^{gt_i}),
\end{equation}
or
\begin{equation}
\boldsymbol{a}^{b} - \boldsymbol{g}^{b} = (\mathbf{R}_{b}^{gt_i})^{-1}(\boldsymbol{a}^{gt_i} - \boldsymbol{g}^{gt_i}).
\end{equation}
These transformations yield a consistent reference coordinate for training, independent of $gt_i$'s specific construction. We take the first transformation in the following discussion.
\subsection{Optimization: State Estimation and Scale Recovery}
\label{sec:Scale recovery and state estimation}
Visual odometry could be utilized to get the camera poses, denoted as $vo$. We relate these estimated trajectories to optimized trajectories, denoted as $opt$, using a 6-DoF transformation $\mathbf{T}_{vo}^{opt}$ and a scaling factor $\lambda$. The relationship is given by:
\begin{equation}
\lambda \cdot
\begin{bmatrix}
\mathbf{R}_{vo}^{opt}&\boldsymbol{t}^{opt}_{vo}\\
\mathbf{0}&1
\end{bmatrix}
\cdot \begin{bmatrix}
\boldsymbol{t}_{c_i}^{vo}\\
1
\end{bmatrix}
=\begin{bmatrix}
\boldsymbol{t}_{c_i}^{opt}\\
1
\end{bmatrix},
\end{equation}
where $\boldsymbol{t}_{c_i}^{vo}$ and $\boldsymbol{t}_{c_i}^{opt}$ are the translation components of the camera poses by optimization and estimated by VO, respectively. To reduce the parameters in optimization, we can assume $\boldsymbol{t}_{vo}^{opt}=[0,0,0]^T$, reducing $\mathbf{T}_{vo}^{opt}$ to a rotation matrix $\mathbf{R}_{vo}^{opt}$ \cite{neyestani2023survey}.
The acceleration constraint is derived by differentiating the translation equation with respect to time:
\begin{equation}
\label{eq:acc-}
\lambda \cdot \mathbf{R}_{vo}^{opt} \cdot \boldsymbol{a}_{c_i}^{vo}=\boldsymbol{a}_{c_i}^{opt}.
\end{equation}
$\boldsymbol{a}_{c_i}^{opt}$ could also be described in terms of its physical meaning:
\begin{equation}
\boldsymbol{a}_{c_i}^{opt} =\mathbf{T}_{c_i}^{opt}\cdot (\boldsymbol{a}_{c_i}^{c}-\boldsymbol{g}^{c}) + \boldsymbol{g}^{opt}=\mathbf{T}_{c_i}^{opt}\cdot \mathcal{N} ((\mathbf{T}^{opt}_{b_i})^{-1} \cdot \mathbf{T}_{c_i}^{opt}) + \boldsymbol{g}^{opt}.
\end{equation}
Substituting this expression for $\boldsymbol{a}_{c_i}^{opt}$ into the acceleration relationship and we define the cost term $cost_i$ as:
\begin{equation}
cost_i=\mathbf{T}_{c_i}^{opt}\cdot \mathcal{N} ((\mathbf{T}^{opt}_{b_i})^{-1} \cdot \mathbf{T}_{c_i}^{opt}) +\
\boldsymbol{g}^{opt}-\lambda \cdot \boldsymbol{R}_{vo}^{opt} \cdot \boldsymbol{a}_{c_i}^{vo},
\end{equation}
where $\mathbf{T}_{c}^{opt} =(\mathbf{T}^{opt}_{b})^{-1} \cdot \mathbf{T}^{opt}_{vo}$. The overall cost function $J$ is then:
\begin{equation}
J(\lambda,\mathbf{T}_b^{opt},\boldsymbol{R}_{vo}^{opt})=\sum_i\left\|cost_i\right\|^2.
\end{equation}
The optimization problem to solve for the trajectory and the scale:
\begin{equation}
\label{eq:optimization}
\mathop{\arg\min}\limits_{\lambda,\mathbf{T}_b^{opt},\mathbf{R}_{vo}^{opt}}
J(\lambda,\mathbf{T}_b^{opt},\mathbf{R}_{vo}^{opt}).
\end{equation}
Here, we can directly set $\boldsymbol{g}^{opt} = \boldsymbol{g}^{gt}$ and then verify $\mathbf{T}_b^{opt}$ to ensure the angle between the optimized coordinate system and the ground truth gravity direction remains small.
Optimizing a large number of poses directly is computationally challenging. To mitigate this, we represent the base trajectory implicitly using a B-Spline curve \cite{sommer2020efficient}. This allows us to optimize a smaller set of control points, $\boldsymbol{K}^{opt}_{i}$, instead of individual poses. A key advantage of using B-Splines is that it can serve as additional kinetodynamic priors, as shown in Eq. \ref{bspline}, thus ensuring that the final optimized trajectory is minimally affected by elastic forces \cite{demmel2021square}. Our reformulated optimization objective is:
\begin{equation}
\label{opt-bspline}
\mathop{\arg\min}\limits_{\lambda,\boldsymbol{K}_0^{opt},\boldsymbol{K}_1^{opt},\ldots,\boldsymbol{K}_n^{opt},\boldsymbol{R}_{vo}^{opt}}J(\lambda,\mathbf{T}_b^{opt},\boldsymbol{R}_{vo}^{opt}).
\end{equation}
% \vspace{-0.2 cm}
\section{Experiments}
% case study
\label{sec:experiments}
\subsection{Implementation details}
\subsubsection{Hardware Setup}
\begin{figure}[htpb]
% \hspace{-0.1 cm}
\includegraphics[width=1.0\columnwidth]{Figs/Hardware_Setup.pdf}
\vspace{-2.5 cm}
\caption{Hardware setup. A typical non-rigid system usually consists of a base, a camera, and a non-rigid connection (we use a spring as the non-rigid connection). The poses that need to be estimated are the pose and scale of the base.}
\vspace{-0.5 cm}
\label{fig:real_exp_setup}
\end{figure}
Fig. \ref{fig:real_exp_setup} presents a typical 3D perception system: a monocular camera mounted on a moving platform via a non-rigid passive linkage (elastic spring). A handheld holder introduces irregular camera motion from body-induced accelerations and vibrations. The camera's full 6-DoF motion results from the interplay of spring deformation recovery force and gravity.
\subsubsection{Data Preparation and Network Training}
Optical markers (Fig. \ref{fig:real_exp_setup}) on both the camera and holder enable ground truth trajectory acquisition via a motion capture system. Initial vertical coordinates for both markers are gravity-aligned, correlating with network training and optimization. Trajectory sequences (25-45 seconds) are recorded for both markers. We then construct the dataset for network training as shown in Eq. \ref{netlabel} and Eq. \ref{netinput}.
\subsubsection{Optimization} We utilize COLMAP \cite{schonberger2016structure} for its high accurate performance in 3D reconstruction and VO. As described in Eq. \ref{opt-bspline}, we construct optimization terms for base trajectory and scale from the camera's trajectory and acceleration, solved using Ceres Solver \cite{Agarwal_Ceres_Solver_2022}.
\subsection{Experiments result}
\subsubsection{Evaluation Metrics}
\label{sec:Evaluation Metrics}
\begin{figure*}[ht]
\includegraphics[width=2.0\columnwidth]{Figs/expfigure.pdf}
\caption{Optimization results for representative trajectories. For improved visualization and comparison, X-Z 2D projections are shown due to the subtle and less significant Y-axis (gravity direction) motion. Each subplot depicts the ground truth trajectory (solid line) and our optimized results (dashed line), with color intensity indicating the time scale.}
\label{fig:optimization_results}
\end{figure*}
Since visual odometry estimates the camera's trajectory, its accuracy is crucial. We evaluate VO performance by calculating the Absolute Pose Error (APE) between $\mathbf{T}_c^{vo}$ and $\mathbf{T}_c^{gt}$. If the VO trajectory is valid, we then assess our optimization by computing the APE between $\mathbf{T}_b^{opt}$ and $\mathbf{T}_b^{gt}$. We use EVO \cite{grupp2017evo} to determine the scale $\lambda_{gt}$ between $\mathbf{T}_b^{opt}$ and $\mathbf{T}_b^{gt}$, and report the relative scale error $err_\lambda=\frac{\|\lambda_{gt} - \lambda\|}{\lambda_{gt}}$. Finally, given the integration of physical gravity (Eq. \ref{netlabel}) in both training and optimization, we verify the alignment of the optimized coordinate system with the physical world's gravity vector. This is quantified by the angular error $err_G=\arccos(\frac{\boldsymbol{g}^{opt}\cdot \boldsymbol{g}^{gt}}{\|\boldsymbol{g}^{opt}\|\cdot \|\boldsymbol{g}^{gt}\|})$.
\begin{table}[hbtp]
\caption{Average performance metrics under varying noise magnitudes. }
\centering
\begin{tabular}{cccccc}
\toprule
\multirow{2}[4]{*}{Noise} & \multicolumn{3}{c}{APE} & \multirow{2}[4]{*}{$\text{err}_\lambda$} & \multirow{2}[4]{*}{$\text{err}_G$(\degree)} \\
\cmidrule{2-4} & Mean & Median & STD & & \\
\midrule
0\% & 0.036 & 0.035 & 0.014 & 0.006 & 0.447 \\
3\% & 0.040 & 0.038 & 0.043 & 0.003 & 0.638 \\
5\% & 0.067 & 0.062 & 0.034 & 0.058 & 1.328 \\
10\% & 0.096 & 0.092 & 0.041 & 0.042 & 1.841 \\
\bottomrule
\end{tabular}%
\label{tab:simulation_noise}
\end{table}%
\begin{table}[hbtp]
\centering
\caption{Average performance metrics under varying outlier proportions.}
\begin{tabular}{cccccc}
\toprule
\multirow{2}[4]{*}{Outlier} & \multicolumn{3}{c}{APE} & \multirow{2}[4]{*}{$\text{err}_\lambda$} & \multirow{2}[4]{*}{$\text{err}_G$(\degree)} \\
\cmidrule{2-4} & Mean & Median & STD & & \\
\midrule
0\% & 0.040 & 0.038 & 0.043 & 0.003 & 0.638 \\
1\% & 0.083 & 0.080 & 0.039 & 0.062 & 1.579 \\
3\% & 0.141 & 0.138 & 0.042 & 0.065 & 1.943 \\
5\% & 0.164 & 0.158 & 0.061 & 0.108 & 2.657 \\
\bottomrule
\end{tabular}%
\label{tab:simulation_outlier}%
\end{table}%
\subsubsection{Simulation Experiments}
We acquire the real motion trajectories of a non-rigid system using a tracking system. The optimized scale factor $\lambda_{opt_{sim}}$ and base trajectory $\mathbf{T}_{b}^{opt_{sim}}$ were then compared against their ground truth counterparts, $\lambda_{sim}$ and $\mathbf{T}_{b}^{gt}$. Gaussian noise with a specified amplitude and outliers at a specified ratio are then added to the ground truth camera trajectories:
\begin{equation}
\mathbf{T}_{c_i}^{sim}=\mathbf{T}_{c_i}^{gt}\oplus (S \cdot \mathbf{N}(0, 1)) \oplus \mathbf{O},
\end{equation}
where $\mathbf{T}_{c_i}^{sim}$ represents the simulated camera pose at time $i$; $\mathbf{T}_{c_i}^{gt}$ is the ground truth camera pose at time $i$; $\mathbf{N}(0, 1)$ denotes Gaussian noise with mean $0$ and specified standard deviation $1$; $S$ is the amplitude of the noise; $\mathbf{O}$ represents outliers added to the trajectory at a specified ratio.
For each noise category, we randomly generate six independent datasets. The average error is then computed for each dataset, and these statistics are subsequently presented.
As shown in Table \ref{tab:simulation_noise}, the algorithm demonstrated robustness to amplitude noise, with APE errors remaining small across 0\%, 3\%, 5\%, and 10\% noise levels. Even at 10\% noise, errors in scale ($\text{err}_\lambda$) and gravity ($\text{err}_G$) remained low. Robustness to outliers was further confirmed by adding 0\%, 1\%, 3\%, and 5\% outliers to trajectories with 3\% nominal noise (Table \ref{tab:simulation_outlier}). Errors remained within acceptable limits, even with increasing outlier percentages.
\subsubsection{Real Experiments}
In addition to simulation experiments, we employed COLMAP as the Visual Odometry to estimate the camera trajectory $\mathbf{T}_c^{vo}$ and its corresponding acceleration $\boldsymbol{a}_{c}^{vo}$. This information was then used to optimize the base trajectory $\mathbf{T}_b^{opt}$. Evaluation metrics are detailed in Section \ref{sec:Evaluation Metrics}.
Fig. \ref{fig:optimization_results} showcases optimization results for representative trajectories, illustrating the algorithm's performance across various motion types (as shown in appendix-\ref{appendix-da}). For clarity, 2D X-Z projections of the ground truth and optimized trajectories are displayed. The strong overlap observed in each subfigure demonstrates the proposed pipeline's accuracy in estimating the base trajectory.
It is important to acknowledge that large system deformations and non-rigidity can induce rapid camera motion, leading to motion blur and thus impacting VO and optimization performance. Subfigure \ref{fig:optimization_results}(d) exemplifies this; while overall effectiveness is maintained, the accuracy is noticeably reduced compared to other scenarios. This phenomenon, observed in real-world experiments, could be mitigated by using higher frame rate cameras or ensuring a denser feature point environment.
Table \ref{tab:error_analysis_1} presents a detailed error analysis of real-world experimental data, further supporting the effectiveness of our proposed method. This analysis confirms our method's ability to accurately estimate the moving platform's state in a non-rigid camera system and successfully recover the scale of a monocular camera system by integrating the Deformation-force Network and visual odometry, without additional sensors.
\begin{table*}[htbp]
\centering
\caption{Error analysis for the real experiments data. APE(Optimization) is the comparison between $\mathbf{T}_b^{opt}$ and $\mathbf{T}_b^{gt}$; APE(VO Component) is the comparison between $\mathbf{T}_c^{vo}$ and $\mathbf{T}_c^{gt}$. What we want to clarify here is that Visual Odometry serves as an integral part of our proposed method's pipeline, rather than a baseline for comparison. Currently, as far as we know, there are no other established methods that effectively address trajectory estimation specifically for non-rigid system.}
\begin{tabular}{cccccccccccc}
\toprule
\multirow{2}[4]{*}{No} & \multicolumn{3}{c}{APE(Optimization)} & & \multicolumn{3}{c}{APE(VO Component)} & \multirow{2}[4]{*}{$\text{err}_\lambda$} & \multirow{2}[4]{*}{$\lambda_{opt}$} & \multirow{2}[4]{*}{$\lambda_{gt}$} & \multirow{2}[4]{*}{Gravity Error(\degree)}\\
\cmidrule{2-4}\cmidrule{6-8} & Mean & Median & STD & & Mean & Median & STD & & & & \\
\midrule
1 & 0.120 & 0.081 & 0.088 & & 0.196 & 0.201 & 0.086 & 0.271 & 0.183 & 0.251 & 7.34 \\
2 & 0.149 & 0.138 & 0.069 & & 0.158 & 0.147 & 0.086 & 0.246 & 0.308 & 0.247 & 6.67 \\
3 & 0.170 & 0.121 & 0.039 & & 0.159 & 0.154 & 0.086 & 0.201 & 0.194 & 0.243 & 4.83 \\
4 & 0.136 & 0.101 & 0.068 & & 0.274 & 0.273 & 0.156 & 0.147 & 0.132 & 0.115 & 3.28 \\
5 & 0.181 & 0.188 & 0.093 & & 0.127 & 0.117 & 0.077 & 0.276 & 0.143 & 0.112 & 6.28 \\
6 & 0.165 & 0.172 & 0.064 & & 0.161 & 0.132 & 0.117 & 0.707 & 0.031 & 0.106 & 13.4 \\
7 & 0.158 & 0.152 & 0.081 & & 0.234 & 0.212 & 0.112 & 0.669 & 0.202 & 0.121 & 6.88 \\
8 & 0.150 & 0.156 & 0.081 & & 0.306 & 0.251 & 0.211 & 0.170 & 0.112 & 0.135 & 7.10 \\
9 & 0.283 & 0.257 & 0.148 & & 0.289 & 0.255 & 0.165 & 0.622 & 0.318 & 0.196 & 6.82 \\
10 & 0.213 & 0.154 & 0.143 & & 0.268 & 0.238 & 0.141 & 0.711 & 0.344 & 0.201 & 8.22 \\
11 & 0.218 & 0.211 & 0.094 & & 0.292 & 0.293 & 0.155 & 0.372 & 0.177 & 0.129 & 7.12 \\
12 & 0.293 & 0.254 & 0.142 & & 0.248 & 0.240 & 0.128 & 0.476 & 0.056 & 0.107 & 2.12 \\
13 & 0.715 & 0.598 & 0.406 & & 0.563 & 0.584 & 0.217 & 1.509 & 0.527 & 0.210 & 5.98 \\
14 & 0.120 & 0.081 & 0.088 & & 0.121 & 0.110 & 0.072 & 0.648 & 0.183 & 0.521 & 7.34 \\
15 & 0.461 & 0.399 & 0.307 & & 0.285 & 0.294 & 0.171 & 0.027 & 0.258 & 0.251 & 0.92 \\
16 & 0.094 & 0.084 & 0.042 & & 0.096 & 0.095 & 0.045 & 0.671 & 0.028 & 0.085 & 7.27 \\
\midrule
Median & 0.167 & 0.155 & 0.088 & & 0.241 & 0.225 & 0.122 & 0.483 & - & - & 6.854 \\
Mean & 0.226 & 0.196 & 0.122 & & 0.236 & 0.224 & 0.126 & 0.424 & - & - & 6.357 \\
\bottomrule
\end{tabular}
\label{tab:error_analysis_1}
\end{table*}
\section{Ablation Study}
\begin{table}[htbp]
\centering
\caption{Comparison of errors: Normalized(with Eq \ref{eq:normlization}) vs. Directly(without \ref{eq:normlization}). We specifically report metrics such as the mean and median errors for linear and angular accelerations across each axis.}
\begin{tabular}{rrrrr}
\toprule
& & \multicolumn{1}{c}{Mean} & \multicolumn{1}{c}{Median} & \multicolumn{1}{c}{STD} \\
\midrule
\multicolumn{1}{c|}{\multirow{6}[0]{*}{$E_{normalized}$}} & \multicolumn{1}{c}{$^{(x)}{a}_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{0.323} & \multicolumn{1}{c}{0.327} & \multicolumn{1}{c}{0.151} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(y)}a_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{0.799} & \multicolumn{1}{c}{0.653} & \multicolumn{1}{c}{0.636} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(z)}a_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{0.167} & \multicolumn{1}{c}{0.147} & \multicolumn{1}{c}{0.109} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(x)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.148} & \multicolumn{1}{c}{0.119} & \multicolumn{1}{c}{0.122} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(y)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.121} & \multicolumn{1}{c}{0.092} & \multicolumn{1}{c}{0.109} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(z)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.203} & \multicolumn{1}{c}{0.168} & \multicolumn{1}{c}{0.171} \\
\midrule
\multicolumn{1}{c|}{\multirow{6}[0]{*}{$E_{directly}$}} & \multicolumn{1}{c}{$^{(x)}a_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{1.128} & \multicolumn{1}{c}{0.91} & \multicolumn{1}{c}{0.898} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(y)}a_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{1.578} & \multicolumn{1}{c}{1.197} & \multicolumn{1}{c}{1.213} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(z)}a_{c}^{linear}$ ($m/s^2$)} & \multicolumn{1}{c}{1.251} & \multicolumn{1}{c}{1.028} & \multicolumn{1}{c}{1.089} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(x)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.456} & \multicolumn{1}{c}{0.401} & \multicolumn{1}{c}{0.338} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(y)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.353} & \multicolumn{1}{c}{0.296} & \multicolumn{1}{c}{0.283} \\
\multicolumn{1}{c|}{} & \multicolumn{1}{c}{$^{(z)}a_{c}^{angular}$ ($rad/s^2$)} & \multicolumn{1}{c}{0.405} & \multicolumn{1}{c}{0.377} & \multicolumn{1}{c}{0.261} \\
\bottomrule
\end{tabular}%
\label{tab:unnormalized-cmp}%
\end{table}%
To validate the importance of our data collection and processing pipeline for accurately modeling a non-rigid system, we designed two ablation studies. In the first study, we eliminated the step of normalizing data to the camera coordinate system. The second study demonstrated the advantage of using multi-dimensional motion patterns for robustly modeling gravity.
\subsubsection{Relative Pose Normalization (Eq. \ref{eq:normlization})}
To demonstrate the importance of normalizing data to the camera coordinate system, we reconstructed the dataset and performed system identification again, this time removing the step represented by Eq. \ref{eq:normlization}. Table \ref{tab:unnormalized-cmp} shows the resulting modeling errors, listing the acceleration and angular acceleration errors in each axis. These results highlight the significant advantage of our normalization approach.
\subsubsection{Movement Patterns on Modeling}
As previously mentioned, we deliberately curated a diverse set of motion patterns for our training data. This strategy was employed to ensure the final model could accurately handle the constant-direction gravitational vector. Table \ref{tab:patt-alb} presents the results of training with these distinct patterns. The results indicate that while individual patterns enable the network to learn dynamics specific to certain axes, a combination of all patterns is crucial for the network to generalize and robustly handle all motion scenarios.
\begin{table}[t]
\centering
\caption{Modeling performance with different motion patterns and the units are $m/s^2$ and $rad/s^2$, respectively. The best and second-best performing metrics are bolded for clarity.}
\begin{tabular}{lrrrrr}
\toprule
& \multicolumn{1}{l}{Pattern A} & \multicolumn{1}{l}{Pattern B} & \multicolumn{1}{l}{Pattern C} & \multicolumn{1}{l}{Pattern D} & \multicolumn{1}{l}{Total} \\
\midrule
$^{(x)}a_{c}^{linear}$ & 1.071 & \textbf{0.864} & 1.198 & 1.041 & \textbf{0.939} \\
$^{(y)}a_{c}^{linear}$ & 5.244 & 4.816 & 4.808 & \textbf{4.186} & \textbf{2.325} \\
$^{(z)}a_{c}^{linear}$ & \textbf{0.534} & 1.781 & 0.665 & 0.834 & \textbf{0.485} \\
$^{(x)}a_{c}^{angular}$ & \textbf{0.368} & 0.375 & \textbf{0.348} & 0.400 & 0.402 \\
$^{(y)}a_{c}^{angular}$ & 0.376 & \textbf{0.355} & 0.455 & 0.436 & \textbf{0.329} \\
$^{(z)}a_{c}^{angular}$ & \textbf{0.346} & \textbf{0.353} & 0.449 & 0.447 & 0.552 \\
\bottomrule
\end{tabular}%
\label{tab:patt-alb}%
\end{table}%
\section{Conclusion}
\label{sec:conclusion}
This paper introduces a novel perception system leveraging a monocular camera passively coupled to a moving platform via a non-rigid connection. By integrating a learned deformation-force model with our state estimation framework, we demonstrate that this non-rigid coupling, rather than complicating the system, provides crucial constraints for passive inertial sensing. This approach successfully resolves the inherent scale and inertial alignment ambiguities of monocular vision, a problem typically requiring additional sensors. Our experiments in both simulation and the real world validate the feasibility of this method. We have shown that by correctly modeling the system's kineto-dynamics, accurate motion and metric scale estimation is achievable. This work serves as a foundational example, showcasing how exploiting physical priors in a flexible system can enhance perception capabilities. We believe this approach holds significant promise for future robotic platforms with specific motion models and elastic actuation chains.
\appendix
\subsection{Data Acquisition}
\setcounter{figure}{0}
\renewcommand{\thefigure}{A\arabic{figure}}
\setcounter{table}{0}
\renewcommand{\thetable}{A\arabic{table}}
\setcounter{equation}{0}
\renewcommand{\theequation}{A\arabic{equation}}
\label{appendix-da}
We use a tracking system to acquire ground truth trajectories of both the camera and the base at 360 Hz. We collected trajectories lasting 25-45 seconds across various patterns, encompassing both smooth and vigorous motions with diverse poses. As illustrated in Fig. \ref{fig:data-fig1}, for each pose, we collected data for:
\begin{itemize}
\item \textbf{Pure translational motion}: The spring primarily undergoes tangential deformation along the direction of motion.
\item \textbf{Pure rotational motion}: The camera experiences radial stretching of the spring due to centrifugal force.
\item \textbf{Translational motion with vertical movement (gravity aligned)}: This setup amplifies the effect of gravity, enabling accurate modeling of its influence.
\item \textbf{Rotational motion with vertical movement (gravity aligned)}: This case induces the most complex spring deformation, resulting from the combined effects of centrifugal force and gravity.
\end{itemize}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth]{Figs/data_fig1.pdf}
% \vspace{-1 cm}
\caption{Trajectories for different states of motion.}
\label{fig:data-fig1}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth]{Figs/data_fig2.pdf}
% \vspace{-1 cm}
\caption{Trajectories with varying gravitational orientations and smoothness levels.}
\label{fig:data-fig2}
\end{figure}
To capture the spring's behavior under diverse gravitational influences, we collected data with the system inverted and on its side, addressing the compressive effect of gravity in a vertical downward orientation. Fig. \ref{fig:data-fig2} illustrates these varied configurations, alongside trajectories demonstrating different intensities of complex motion states.
\subsection{Training DFN}
\setcounter{figure}{0}
\renewcommand{\thefigure}{B\arabic{figure}}
\setcounter{table}{0}
\renewcommand{\thetable}{B\arabic{table}}
\setcounter{equation}{0}
\renewcommand{\theequation}{B\arabic{equation}}
We take a MLP to model the non-rigid connection. The network's computational complexity and parameter count were quantified using the \texttt{thop} library, revealing 375.392 KFLOPs and 377.222 K parameters, respectively.
We use approximately 259 K samples derived from 16 collected trajectories for training, and another 16 sequences reserved for real experiments. These samples are partitioned into training, testing, and validation sets in a 7:2:1 ratio. We use Adam as an optimizer, and the learning rate is 1e-4. We train the network for 100 epochs with the batch size 1024.
We use the method described in the main text to construct the ground truth, and the $j$-th label for the $i$-th sequence is:
\begin{equation}
\label{netlabel}
\mathcal{N}^i_{label_j} = \boldsymbol{a}_j^{c} - \boldsymbol{g}^{c} = (\mathbf{R}_{c_j}^{gt_i})^{-1}(\boldsymbol{a}_j^{gt_i} - \boldsymbol{g}^{gt_i}).
\end{equation}
The $j$-th input for the $i$-th sequence is the relative transformation between the base and camera:
\begin{equation}
\label{netinput}
\mathcal{N}^i_{input_j} = (\mathbf{T}_{b_j}^{gt_i})^{-1}\mathbf{T}_{c_j}^{gt_i}.
\end{equation}
We use L1 Loss as the training metric:
\begin{equation}
\label{loss}
Loss = \sum \| \mathcal{N}^i_{output_j}-\mathcal{N}^i_{label_j}\|.
\end{equation}