ModPack: An Extensible Teleoperation Interface for Bimanual Mobile Manipulation
Extracted source text · arXiv 2607.19479
\documentclass{article}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{amssymb}
\usepackage{xspace}
\usepackage{soul}
\usepackage{amsmath}
\usepackage{float}
\usepackage{placeins}
\usepackage{wrapfig}
\usepackage{caption}
\usepackage[normalem]{ulem}
\usepackage[export]{adjustbox}
% \usepackage{corl_2026} % Use this for the initial submission.
% \usepackage[final]{corl_2026} % Uncomment for the camera-ready ``final'' version.
\usepackage[preprint]{corl_2026} % Uncomment for pre-prints (e.g., arxiv); This is like ``final'', but will remove the CORL footnote.
\newcommand{\ours}{ModPack}
\newcommand{\shuran}[1]{\textcolor{blue}{[shuran: #1]}}
\newcommand{\maxcomment}[1]{\textcolor{blue}{[Max: #1]}}
\newcommand{\todo}[1]{\textcolor{red}{#1}}
\newcommand{\josh}[1]{\textcolor{orange}{[josh: #1]}}
\newcommand{\zeyi}[1]{\textcolor{myPink}{[zeyi: #1]}}
\newcommand{\renee}[1]{\textcolor{green}{[renee: #1]}}
\newcommand{\mypara}[1]{\vspace{0mm}\noindent\textbf{#1}}
\title{\ours: An Extensible Teleoperation Interface for Bimanual Mobile Manipulation}
% \shuran{Extensible / Modular / Adaptable? }
% zeyi: We are selling this design decision that this backpack can serve as the portable base, and you can add so many useful features on top relatively easily, whereas prior works are non-expandable, at least not designed with this intention, which causes trouble to scale up to new features.
% ARMI: An All-in-One Robot Modular Teleoperation Interface
% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.
% NOTE: authors will be visible only in the camera-ready and preprint versions (i.e., when using the option 'final' or 'preprint').
% For the initial submission the authors will be anonymized.
\author{
\vspace{-2mm}
Joshua Citron \qquad Renee Zbizika \qquad Zeyi Liu \qquad Shuran Song \\\\
\vspace{-2mm}
\mdseries{Stanford University} \\\\
\mdseries{\href{https://modpack-robotics.github.io}{\textcolor[RGB]{93,64,205}{modpack-robotics.github.io}}}
}
\begin{document}
\maketitle
%===============================================================================
\begin{abstract}
Existing teleoperation systems are often tailored to specific robot hardware and task domains, limiting their scalability and adaptability. We present {ModPack}, a modular and extensible teleoperation system designed to support diverse robot embodiments and task requirements within a unified framework. At the core of ModPack is a self-contained wearable ``backpack'' that integrates onboard computation, power, communication, and data storage. Built on top of this shared interface, the system supports plug-and-play capability modules including joint-level teleoperation with haptic feedback, mobile manipulation, and active perception. Experiments across two distinct robot platforms and real-world mobile manipulation tasks demonstrate that ModPack provides a flexible and reusable framework for data collection and policy learning. To support future research, we open-source the complete hardware design and software stack.
\end{abstract}
% Two or three meaningful keywords should be added here
\keywords{Mobile Manipulation, Teleoperation Systems, Imitation Learning}
%===============================================================================
\input{text/intro_v1}
\vspace{-2mm}
\section{Related work}
\label{sec:related}
\input{text/related_works}
%===============================================================================
\vspace{-2mm}
\section{Method}
\label{sec:method}
\input{text/methods_v1}
\vspace{-2mm}
\section{Experiments}
\label{sec:result}
\input{text/experiments}
%===============================================================================
% \maxcomment{Put limitations in conclusion sectoin maybe?}
\vspace{-2mm}
\section{Limitations}
\vspace{-2mm}
\label{sec:limitations}
\looseness-1
One limitation of the current system is the torque capacity of the Dynamixel motors, particularly when simultaneously supporting gravity compensation and haptic feedback. Higher-torque motors could mitigate this limitation, but would also increase the weight and bulk of the wearable system. Another limitation is the battery requirement for extended untethered operation: powering both the leader-arm motors and onboard compute for several hours requires a relatively large battery setup, increasing backpack weight. Future work could reduce this weight by sharing batteries across the two leader arms and replacing the current mini PC with a lighter-weight compute platform. In the current design, we prioritize longer battery life and larger onboard storage capacity as a practical tradeoff.
%Despite the extensibility achieved by ModPack in a compact design, there are still limitations to the method. One such limitation is the limited capacity to provide torque from the Dynamixel motors when it comes to providing gravity compensation and haptic feedback simultaneously. Furthermore, to provide the battery power for hours of use for motors as well as the PC that sends commands makes the backpack weight more. Future work can reduce the weight by sharing batteries across the two leader arms, as well as using a more lightweight compute option that the current mini PC. We opted for a longer battery life as well as increased storage space as a tradeoff.
%Another limitation to note is that each robot will need its own kinematically-similar leader arm. To this end, domain knowledge is necessary to construct these arms. Future work could include a URDF to CAD pipeline that simplifies this process for new arms. Lastly, with modularity comes new systems which bring their own complexity despite the effort made to create a standard pipeline for instantiation.
%\zeyi{For base tracking, we only consider mobile base with 3 DoF. Future work can support torso tracking.}
%===============================================================================
\vspace{-3mm}
\section{Conclusion}
\vspace{-2mm}
\label{sec:conclusion}
We present \ours, a modular and extensible teleoperation system for collecting demonstrations across diverse mobile manipulation platforms and task settings. Built around a self-contained wearable backpack, \ours~provides a unified interface for plug-and-play modules including joint-level leader-arm control with haptic feedback, mobile-base control, and active perception. We demonstrate the flexibility of this design by deploying the system on two distinct robot embodiments and collecting demonstrations for real-world bimanual mobile manipulation tasks. Policies trained on data collected with \ours~achieve strong deployment performance, validating its utility as a practical data collection interface for robot learning.
%===============================================================================
\clearpage
\acknowledgments{
The authors would like to thank all REALab members as well as Chen Chen for feedback and guidance during this project. In particular, we thank Yihuai Gao and Jeff Liu for their work on robologger as well as Xiaomeng Xu, Dian Wang, and Jisang Park for their help with the RB-Y1 robot. We would also like to thank Haochen Shi for his advice on hardware and gravity compensation. This work was supported in part by the
NSF Graduate Fellowship, NSF Award \#2143601, \#2037101, and \#2132519, Apple and Amazon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
}
%===============================================================================
% no \bibliographystyle is required, since the corl style is automatically used.
\bibliography{references} % .bib
\newpage
\appendix
{\Large \textbf{Appendix}}
% \zeyi{what are modified for gello code?}\josh{like skeleton is there but it is compeletely revamped, hard to condense what was done succinctly?}
\input{text/supp}
\end{document}
\section{Introduction}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/teaser_v4.pdf}
% \vspace{-5mm}
\captionsetup{font=small}
\caption{\footnotesize \textbf{\ours} is an extensible teleoperation interface for bimanual mobile manipulation, built around a wearable backpack base with plug-and-play capability modules. For example, the active perception module, shown on the left, lets operators control the robot’s head camera to search for cloth placement locations, while the leader arms module with haptic feedback, shown on the right, supports a box transfer task by preventing excessive force and facilitating bimanual coordination. Swappable leader arms further allow adaptation to different robot embodiments.
% \todo{make figure bigger, change left task figure more active preception}
}
\label{fig:teaser}
\vspace{-3mm}
%https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/Cover-Figure?node-id=29-23&t=vC3MeFdSGjeHzBnm-0
\end{figure}
Designing a teleoperation interface involves an inherent trade-off between system simplicity and functional capability. The most capable systems are often more complex and sometimes unnecessary for simpler applications. For instance, while active perception is vital for long-horizon search-and-reach tasks, it may introduce redundant overhead for tabletop manipulation. As a result, prior work has often required researchers to carefully and rigidly tailor teleoperation interfaces to match the requirements of a given robot hardware and task domain.
This task- and robot-specific specialization, while effective, leads to a fragmented design space. Systems optimized for one setting are often difficult to extend or adapt to others, limiting scalability and slowing iteration. In particular, adding new capabilities typically requires substantial redesign of both hardware and software pipelines. This lack of \textbf{modularity} presents a key bottleneck for building unified data collection and control systems that generalize across tasks and robot platforms.
In this work, we introduce \textbf{\ours} -- a teleoperation system that explicitly addresses this trade-off through an \textbf{extensible} and \textbf{modular} design. At its core, \ours~is a self-contained ``backpack" unit that provides integrated power, onboard computation, and data storage. This unit serves as a common substrate, decoupling core system infrastructure from task- or robot- specific functionalities. Built on this foundation, we propose a plug-and-play interface that allows operators to flexibly add or remove capability modules based on task requirements. We demonstrate the versatility of this design through several modular extensions, each enabling a distinct capability:
\vspace{-1mm}
\begin{itemize}[leftmargin=6mm]
\item \textbf{Cross-robot joint-level control:} Through swappable leader-arm modules, as shown in \Cref{fig:teaser}, the system can adapt to diverse robot arm kinematics, enabling precise joint-level control across different robot platforms.
\item \textbf{Mobility:} We track the operator's base motion through an iPhone mounted on \ours, enabling mobile manipulation for tasks that require large workspace coverage.
\item \textbf{Active perception:} By wirelessly connecting to a VR headset, such as the Vision Pro, \ours~streams real-time robot head-camera observations while allowing the operator to control the camera pose. This module enables demonstrations of active perception behaviors, such as object search, occlusion handling, and viewpoint selection.
\item \textbf{Haptic feedback:} By leveraging force feedback from the robot and active motors on the leader arms, our system provides real-time haptic feedback, enabling safer and more precise teleoperation of contact-rich tasks.
\end{itemize}
% While these capabilities have been explored individually, integrating them into a unified, flexible, and robust teleoperation system is non-trivial. First, simultaneous real-time control of the arms, mobile base, and head requires \ours~to \textbf{disentangle operator motions} to component-wise control commands. Second, the system synchronizes \textbf{high-rate heterogeneous data}, including proprioception, visual observations, and force/torque for haptic feedback, while maintaining low latency and precise temporal alignment.
% Third, extended operation requires an \textbf{unobtrusive and comfortable} interface, which we address with effective gravity compensation on the exoskeleton leader arms to reduce operator fatigue.
% %
% Finally, these capabilities must fit within a \textbf{compact wearable form factor}; we package power, computation, sensing, and communication hardware into a self-contained backpack while navigating additional constraints on weight, thermal management, and usability.
\looseness-1
While these capabilities have been explored individually, integrating them into a unified, flexible, and robust teleoperation system remains challenging. First, simultaneous control of the arms, mobile base, and head introduces ambiguity in operator motion; \ours~addresses this by disentangling operator movements into component-wise commands for arm manipulation, base motion, and active perception. Second, the system must process high-rate heterogeneous data, including proprioception, visual observations, and force/torque signals for haptic feedback; we collect these streams with low-latency communications through message-queue servers. Third, extended operation requires an unobtrusive and comfortable interface; we reduce operator fatigue through gravity compensation on the exoskeleton leader arms. Finally, all capabilities must be integrated into a compact wearable form factor; we package power, computation, sensing, and communication hardware into a self-contained backpack while balancing constraints on weight, thermal management, and usability.
% While each of these functionalities has been individually explored in prior work, integrating them into a \textbf{unified, flexible, and robust} system introduces significant technical challenges.
% %
% First, enabling simultaneous real-time control of multiple robot subsystems -- including the arms, mobile base, and head--- requires carefully \textbf{disentangling operator motions} to avoid ambiguous or unintended commands. For example, when head motion is captured using a VR headset, the motions of the robot neck and mobile base can become intrinsically coupled, making it unclear whether the operator intends to rotate the head or reorient the entire robot. Without proper motion decomposition and coordination, such coupling can lead to unstable behavior, over-rotation, or drift.
% %
% Second, the system must reliably synchronize \textbf{high-rate heterogeneous data}, including joint states, visual observations, and haptic feedback and active perception. This demands a unified communication and message management framework that ensures low-latency execution and precise temporal alignment.
% %
% Third to make the system \textbf{unobtrusive and comfortable} for extended operation, we implemented effective gravity compensation on the exoskeleton leader arms to reduce substantial physical strain on the operator during prolonged teleportation sessions.
% %
% Finally, these capabilities must be realized within a \textbf{compact and wearable form factor}. To make the design reusable, we the core components like power, computation, sensing, and communication hardware into a self-contained “backpack” poses additional constraints on weight, thermal management, and usability. Addressing these constraints requires careful system-level co-design.
We evaluate \ours~through real-world bimanual mobile manipulation experiments across two distinct robot platforms. Using our teleoperation device, we collect demonstrations, train behavior cloning policies, and evaluate deployment success rates. To support future research, we will open-source the full hardware design, software stack, and policy learning and deployment framework.
% Teleoperation remains the most straightforward and effective paradigm for collecting high-quality demonstration data for robot learning. Early approaches include kinesthetic teaching, however it is very tiring and slow. Then there are approaches to separate operator with the robot, including using space mouse, joystick, or vr controller to control the end effector poses of the arm, then solve for joints through IK. Or with kinesthetic replica such as GELLO \cite{wu2024gello} to directly control joints through 1-1 mapping.
% As robots acquire more advanced manipulation capabilities, mobile manipulation becomes increasingly important, as tasks extend beyond structured tabletop settings into unstructured, real-world environments. the Mobile ALOHA system \cite{fu2024mobile} adopts a leader–follower teleoperation setup, enabling the operator to simultaneously drive the mobile base while controlling a bimanual manipulator via two leader arms. Separate arm and base control, not natural.
% Tidybot++ \cite{wu2024tidybot++} offers a solution to simultaneously control base and a single robot arm through an iPhone app. Though it could be hard to control two arms and the base at once.
% To achieve natural bimanual arm teleoperation and achieve joint-level control, some works adopt an exoskeleton approach to have operator 'wear' the replica arms.
% AirExo \cite{fang2024airexo}, AirExo-2 \cite{fang2025airexo}: cross-robot adaptability, not mobile, static global camera, no base
% CHILD \cite{myers2025child}, PAPRLE \cite{kwon2025paprle}: both arm and leg control, but need to be either arm and leg at each time, cannot control together. PAPRLE proposes a modular system with pluggable puppeteers or other input devices to collet teleoperation signals.
% Supersuit \cite{chen2026supersuit}: most relevant, isomorphic exoskeleton + base / torso tracking; no haptic feedback. active demonstration.
% HOMIE \cite{ben2025homiehumanoidlocomanipulationisomorphic}: Similar table and motivation but humanoids, no live streaming / active perception, base is controlled using pedal not very natural.
% TeleMoMa \cite{Shivin_Wensi_Yuqian_Samik_Jiaheng_Ruohan_Peter_Ben_Roberto_2024}: VR controller.
% One important feature is haptic feedback. thru VR vibration \cite{ding2025bunny}, gloves \cite{zhang2025doglove}.
\mypara{Teleoperation for Robot Manipulation.} Teleoperation serves as a primary paradigm for collecting high-quality demonstration data for robot learning \cite{punamiya2026egoverse, barreiros2026careful, bjorck2025gr00t, team2025gemini, o2024open, khazatsky2024droid}. Early approaches such as kinesthetic teaching \cite{Argall2009LfD} enable direct physical guidance but are labor-intensive and difficult to scale. To improve usability, many systems decouple the operator from the robot, using input devices (e.g., spacemouse, joysticks, or VR controllers) to command end-effector poses with inverse kinematics ~\cite{Shivin_Wensi_Yuqian_Samik_Jiaheng_Ruohan_Peter_Ben_Roberto_2024, iyer2024open}. In contrast, kinematic replicas such as GELLO~\cite{wu2024gello} enable one-to-one joint mapping, improving control fidelity and granularity. As tasks move beyond static tabletop settings, teleoperation for mobile manipulation becomes critical. Mobile ALOHA~\cite{fu2024mobile} enables simultaneous base and bimanual arm control via a leader–follower setup, though base and arm control are mostly decoupled. TidyBot++~\cite{wu2024tidybot++} supports joint control of a mobile base and a single arm, but does not scale naturally to coordinated bimanual tasks.
To achieve more natural human-like control, recent works explore exoskeleton-based teleoperation with isomorphic mappings. AirExo and AirExo-2~\cite{fang2024airexo,fang2025airexo} provide high-fidelity arm control and cross-robot adaptability, but lack integrated mobile base control. Full-body systems such as CHILD~\cite{myers2025child} and PAPRLE~\cite{kwon2025paprle} support both arm and leg control, though typically not simultaneously. Most closely related, Supersuit~\cite{chen2026supersuit} enables whole-body teleoperation with base and torso tracking, but does not provide active perception or haptic feedback. In contrast, ModPack covers key capabilities for mobile teleoperation, and in addition, allows users to select different combinations of components for their target system through its modular design. Please see Appendix~\ref{sec:prior_table} for a detailed comparison with prior works.
% \josh{another work i found that we should include is \cite{Lenz_2023}, which has most of our features but relies on Panda arms and a foot pedal to steer}
% while HOMIE~\cite{ben2025homiehumanoidlocomanipulationisomorphic} explores similar ideas for humanoids.
\mypara{Teleoperation with Haptic Feedback.}
Haptic feedback is critical for collecting high quality demonstrations for contact-rich tasks. A key challenge in haptic feedback for teleoperation is balancing feedback fidelity with hardware portability. High-fidelity bilateral systems can provide rich force feedback, but often rely on stationary or off-the-shelf robot arms that are too heavy and cumbersome for wearable mobile interfaces~\cite{Lenz_2023,nimbro}. In contrast, lightweight wearable systems \cite{zhang2023wearable, purushottam2024wheeledhumanoidbilateralteleoperation, wu2025vrbased} and vibrotactile feedback \cite{ding2025bunny} improve portability, but typically sacrifice one-to-one joint-level transparency and kinesthetic resistance. In this work, we address this trade-off with a wearable teleoperation interface built from motorized GELLO arms integrated into a backpack-mounted frame. By actuating each joint directly, our system provides high-fidelity, joint-level haptic feedback while preserving the portability required for mobile teleoperation.
% Advanced wearable devices have attempted to offer joint-level feedback \cite{zhang2025doglove}, but often struggle with the interpretability of complex forces, leading some to explore task-space remapping \cite{Lenz_2023} or discrete contact categorization \cite{9991865}.
% Collaborative Teleop with Haptic Feedback \cite{9981426}: haptic feedback guidance to the path of robot to guide, human co-navigates autonomous robot
% Improving Teleoperation Through Human-Aware Haptic Feedback \cite{9991865}: creates different haptic feedback signals based on what type of contact robot is in, i.e. slipping versus contact
% Bimanual Telemanipulation with Force and Haptic Feedback \cite{Lenz_2023}: controller with haptic and force feedback, use an identical robot arm so a bit overkill, also has 6d head pose control with VR headset, mobile base controlled with 3D foot "rudder"
% Wheeled Humanoid Bilateral Teleoperation with Position-Force Control \cite{purushottam2024wheeledhumanoidbilateralteleoperation}: could also be relevant for backpack-esc design, haptic feedback through torso
% HoMMI \cite{xu2026hommilearningwholebodymobile}: Not teleop, UMI-style data collection, with active perception
% HoMeR \cite{sundaresan2025homerlearninginthewildmobile}: learns to predict relative vs absolute actions for end effector, uses TidyBot++ base, create a whole body controller
% EMMA \cite{zhu2026emma}: cotrain whole body human data with static teleoperation data
\mypara{Teleoperation with Active Perception. }
Active perception is the ability to adaptively change viewpoint to acquire task-relevant visual information. Recent works study active perception with head- or body-mounted sensing~\cite{xu2026hommilearningwholebodymobile, zeng2025activeumiroboticmanipulationactive, yu2025egomi}. In teleoperation, active perception is often achieved by tracking the operator's head pose with a VR headset and mapping it to robot camera motion while streaming the robot's egocentric view~\cite{cheng2024open, xiong2025vision}. However, existing systems primarily operate in stationary settings, where camera and base motion are decoupled by design. Mobile teleoperation introduces a key challenge: disentangling viewpoint changes caused by head motion from those caused by body or base motion. Our system tracks the operator’s head and body motion with the Vision Pro and a backpack-mounted iPhone, respectively, and compensates for base motion before sending adjusted robot head commands. This decoupling enables operators to naturally walk, look around, and control the arms during data collection.
% \textbf{Teleoperation with Active Perception.} Active perception refers to the process of moving a robot or demonstrator's viewpoint to perceive a scene from different angles and directions.
% %
% Multiple works control a target robot by tracking an operator's head pose with VR headsets while simultaneously visualizing robot point of view \cite{xiong2025vision, cheng2024tv}, though are not integrated in a mobile system. OpenTeleVision \cite{cheng2024tv} allows a demonstrator to move their head and have the robot's camera track this person's head pose, as well as viewing robot point-of-view, though is limited to stationary teleoperation.
% % \zeyi{what's the difference between their work and ours?}
% Vision-in-Action \cite{xiong2025vision} build on this work by teaching natural searching policies through human demonstrations, though not mobile. Prior work done to create mobile active perception \cite{Lenz_2023} relies on foot pedals to steer, and prevent the operator from walking around. Such works avoid the challenge of decoupling base motion with neck motion during mobile teleoperation by remaining stationary during collection. In contrast, our system must explicitly decouple head pose tracked by the Vision Pro and base movement tracked by the iPhone, which are both mounted via the backpack to the operator.
% % \josh{Last section was attempt to justify challenges, can rework}
% %
% Other works choose to collect demonstrations without teloperation while still capturing operator head pose \cite{xu2026hommilearningwholebodymobile, zeng2025activeumiroboticmanipulationactive, yu2025egomi}. ActiveUMI \cite{zeng2025activeumiroboticmanipulationactive} presents a compact “backpack” interface for VR data collection, but crucially lacks joint-level data collection abilities. Similarly, EgoMI \cite{yu2025egomi} uses VR head pose tracking, but lacks joint-level as well as mobile data collection.
% SPIN \cite{uppal2024spin}: mobile robot learns to look around and move at the same time via RL
{\setlength{\textfloatsep}{0pt}
\begin{figure}[t]
\centering
\includegraphics[width=0.98\linewidth]{figures/modpack_method_v3.pdf}
\caption{\footnotesize \textbf{\ours} consists of a backpack core that supports plug-and-play modules, such as a Vision Pro for active perception and an iPhone for base control. Separate 6-DoF and 7-DoF leader arms are configured to control a customized mobile robot (top right) and an RB-Y1m robot (bottom right), respectively.
}
\label{fig:method}
%https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/Cover-Figure?node-id=29-23&t=vC3MeFdSGjeHzBnm-0
\vspace{-4mm}
\end{figure}}
\ours~is a self-contained, 3D-printed backpack system that provides onboard power, storage, and compute for mobile teleoperation. Its primary design goal is \textbf{extensibility}: users can easily add, remove, or replace hardware and software modules to support new robot embodiments with minimal setup. We describe the core system, implemented modules---including \textsc{leader arms}, \textsc{mobile base}, and \textsc{active perception}---and integrations with two distinct mobile robot platforms.
\subsection{\ours~Core}
At the core of \ours~is a portable backpack that serves as a shared base platform for modular hardware extensions and provides the software interfaces for integration. This section describes the core hardware and software architecture that enables these extensions.
\mypara{Hardware.}
The \ours~core is a wearable backpack base unit that moves with the operator while providing a configurable mounting and housing structure for modular hardware components. The backpack is fabricated from 3D-printed parts, improving accessibility and enabling rapid modification by the research community. As shown in Fig.~\ref{fig:method}, the main backpack assembly consists of 3D-printed front and rear panels secured with toggle latches, allowing convenient access to the internal components. The front panel includes two top-mounted slots for inserting support bars used to mount the leader-arm modules.
The internal volume is organized into five removable shelves that house the onboard compute, power, and optional accessories.
% Specifically, the backpack holds a Geekom A5 Mini PC, three 145,W, 25{,}000,mAh portable power banks, and an optional Vision Pro charger. Two power banks are assigned to the left and right leader arms, respectively, while the third powers the onboard PC.
Following ergonomic observations from prior backpack-based systems~\cite{genitrini2022backpacks}, which suggest that vertically distributed loads near the thoracic region can reduce carrying strain, the PC and power banks are arranged along the operator’s spine. To further reduce weight and improve thermal ventilation, excess material is removed from both the structural panels and internal shelves. Refer to Appendix~\ref{sec:3d_printing} for 3D printing details and Appendix~\ref{sec:circuit} for the power circuit diagram.
% \maxcomment{The exact numbers of these power banks, etc may not be super important if you want to save space. The hardware can probably fit into one paragraph instead of two}
% The \ours~core is a wearable backpack serving as a base unit that goes where the user goes while also being easily retrofitted in what it carries. From this, we also inherit the benefit of a 3D printed structure, making the design more accessible to the community. Figure \ref{fig:method} b) displays an exploded view of the main ``backpack'' unit, which is composed of a 3D printed front and back panel that snap together with toggle latches to allow for easy access to internals. Two slots at the top of the front panel allow for bars to be slotted in for mounting leader arms. The inside of the backpack is occupied by five shelves that can slide in and out, which house a Geekom A5 Mini PC, a Vision Pro charger for optional active perception, and three 145 W, 25000mAh portable power banks. Two of the chargers are dedicated to each of the leader arms, respectively, while the third battery powers the PC. As noted by prior work \cite{genitrini2022backpacks}, strain induced from carrying a backpack can be reduced by distributing load vertically, with placement of the load around the thoracic region. As such, care was taken to place the PC and chargers along the spine. To further reduce strain on the operator as well as introduce ventilation for the electronics, unnecessary material is removed for both panels as well as the shelves.
\mypara{Software.} The onboard mini PC serves as a standalone compute unit for module activation, data logging, and target-robot startup and shutdown. Target robots communicate with the backpack PC through a lightweight bidirectional ``bridge'' API. The underlying communication is abstracted from the user and implemented using a lightweight message queue, following~\cite{gao2026gatedmemorypolicy}. This abstraction decouples the core backpack stack from embodiment-specific implementations, allowing the same system to support different robots and arbitrary subsets of extended modules. A diagram of the software stack can be found in Appendix \ref{sec:software_appendix}.
% To support extensibility, we define a standardized workflow for adding new modules. Each module is implemented as an independent process, registered through a common interface, and optionally managed through a shared lifecycle controller. This design allows users to easily extend \ours~with new hardware or software capabilities while preserving a consistent system interface.
\subsection{Extensible Modules}
The core system is extensible through modular components, each consisting of a hardware unit and its corresponding software interface. This design enables physical modules to be swapped across target robots while preserving a unified API and minimizing code changes. Below, we describe three modules---\textsc{leader arms}, \textsc{mobile base}, and \textsc{active perception}.
\subsubsection{Leader Arms}
\label{method:leader-arms}
% \mypara{Hardware design.}
% additional fig specifically for 1-1 config example? flow from each cad limb to each robot arm, motor to motor arrows in diagram
As shown in Fig.~\ref{fig:method}, we design and 3D-print two leader-arm variants: a pair of 6-DoF ARX5 leader arms for a customized mobile manipulation platform \cite{xiong2025vision}, and a pair of 7-DoF leader arms for the RB-Y1m robot. Details of both robot platforms are provided in Appendix~\ref{sec:robots}. Given the CAD model of a target robot, each leader arm is constructed to be kinematically equivalent to its corresponding follower arm, following GELLO~\cite{wu2024gello}, enabling direct joint-space teleoperation. The leader arms are actuated with off-the-shelf Dynamixel servo motors. Additional details on 3D printing and motor selection are provided in Appendix~\ref{sec:3d_printing} and \ref{sec:leader_arms}, respectively.
%Low-level control is handled by a modified version of the code presented in GELLO \cite{wu2024gello}.
% We adopt a capability-driven cost design. Our implementation uses Dynamixel XM-series actuators with torque control for gravity compensation and haptic feedback, but supports a continuum of configurations: we may substitute lower-cost motors based on desired torque performance. As a result, gravity compensation and haptic fidelity scale with actuator choice, allowing flexible trade-offs between cost and performance.
% Non-actuator parts (PLA, fasteners, etc.) remain negligible in cost.
% Reproducibility: The system is constructed from off-the-shelf components and standard fabrication workflows, enabling straightforward replication.
% In software, the leader arm are interfaced via a heavily modified the DYNAMIXEL Python API for low-level joint access. ModPack abstracts/wraps this with a standardized interface for the manipulation module, requiring minimal user configuration.
% \mypara{Software Interface / features.}
% The leader arm is interfaced using the DYNAMIXEL Python API, which provides low-level access to joint states and control modes. ModPack exposes a standardized software interface for the manipulation module, while requiring minimal user configuration:
% \begin{itemize}
% \item \textbf{ModPack provided:} Joint state streaming, command interfaces, and integration with the system-wide communication framework.
% \item \textbf{User-defined:} Config. files specifying motor-to-joint mappings and leader-to-follower joint correspondences.
% \end{itemize}
% This separation enables consistent integration of new embodiments while preserving flexibility in hardware design.
% The ARX5 leader arm logs joint positions and is actuated with 2 Dynamixel XM-540-W270-Ts, 2 Dynamixel XM-430-W350-Ts, 2 Dynamixel XL-430-W250-Ts, and 1 Dynamixel XL-330-M288-t for the gripper, while the RB-Y1 uses 5 Dynamixel XM-540-W270-Ts, 1 Dynxamiel XM-430-W210-T, 1 Dynamixel XM-430-W2350-T, and 1 Dynamixel XL-330-M288-T for the gripper. We use these motors to provide feedback to the operator via gravity compensation as well as haptic feedback.
\mypara{Gravity Compensation.} To alleviate operator fatigue during teleoperation, we implement active gravity compensation \cite{shi2026minimalistcompliancecontrol} on the leader arms. The gravity torque vector $\tau_g = G(q)$ is computed using the Orocos Kinematics and Dynamics Library (KDL) in conjunction with the leader arm's URDF model. To enhance stability, we optionally append a joint-space damping term to form the augmented torque vector at time $t$: $\tau'_t = \tau_{g,t} + K_d \dot{q}_t$, where $K_d$ is a diagonal damping gain matrix. To ensure smooth and stable assistance, the augmented torque for each joint $i$, denoted as $\tau'_{i,t}$, is filtered using two cascaded exponential moving average (EMA) filters:
\begin{equation}
\tau_{i,t}^{\text{inter}} = \alpha \tau_{i,t-1}^{\text{inter}} + (1-\alpha)\tau'_{i,t},
\quad
\tau_{i,t}^{\text{final}} = \alpha \tau_{i,t-1}^{\text{final}} + (1-\alpha)\tau_{i,t}^{\text{inter}},
\end{equation}
where $\alpha \in [0, 1)$ is the smoothing factor. The final torque vector is scaled by a gain factor and commanded to the leader arm motors, subject to the maximum current constraints configured.
% \maxcomment{Text is too small in Figure 3}
\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\vspace{-4mm}
\includegraphics[width=0.98\linewidth]{figures/haptic_feedback_v4.pdf}
\caption{\footnotesize \textbf{Haptic feedback.} The top plot shows the average joint torque on RB-Y1m left arm (blue), along with the corresponding haptic feedback torque norm commanded to the left leader arm (orange). The bottom panel shows operator’s response to the haptic feedback to adjust the force exerted on the box.
}
\label{fig:haptic}
\vspace{-4mm}
%https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/Cover-Figure?node-id=29-23&t=vC3MeFdSGjeHzBnm-0
\end{wrapfigure}
\mypara{Haptic Feedback} To enhance operator situational awareness, we implement haptic feedback for follower robots equipped with force/torque (F/T) sensors, in our case, the RB-Y1m robot.
When enabled, the system reads the net wrench at the end-effector, $F_{ee}$, and maps it to joint-space torques via the Jacobian transpose: $\tau_{c} = k_{c} \cdot J^TF_{ee},$
% \begin{equation}
% \begin{aligned}
% \tau_{c} &= k_{c} \cdot J^TF_{ee},\\
% \end{aligned}
% \end{equation}
where $k_{c}$ is a tuneable gain.
The final control torque $\tau_{f}$ commanded to the leader motors is the superposition of the gravity compensation torque and the haptic feedback component: $\tau_{f} = \tau_{g_i,t}^\text{final} + \tau_{c}$.
% \begin{equation}
% \tau_{f} = \tau_{g_i,t}^\text{filtered} - \tau_{c}.
% \end{equation}
To maintain hardware longevity and accommodate lower-cost actuators, we implement a current budgeting strategy. This limits peak torque commands and ensures the motors operate within conservative amperage constraints, facilitating safe operation without compromising the transparency of the force feedback.
% This formulation allowing users to perceive contact interactions while maintaining smooth and stable teleoperation, ensuring safe operation and consistent feedback across a range of actuator capabilities.
\subsubsection{Mobile Base}
% \todo{@josh take a pass, I'm a bit confused by the notation. what is p? is it the 6DoF pose, or 3D position? what is R, is it the rotation matrix?}\josh{p is position vector (x,y,z) and R is the rotation matrix yes}
We track the operator’s $SE(2)$ pose using an iPhone mounted on the backpack, which runs a modified WebXR session following \cite{wu2024tidybot++}. To ensure intuitive base control, we implement an egocentric mapping procedure that couples the robot’s base translation directly to the operator’s displacement within their local frame. For example, when the operator walks forward, the robot moves forward in its own local frame.
To recover the operator's pose in the robot frame, we first transform the raw pose from WebXR $(p_{xr}, R_{xr})$ to align with the robot's coordinate convention and obtain $(p_{r}, R_{r})$. The key challenge is then that the operator and iPhone rotate about different centers, as the iPhone is mounted to the backpack and not the operator. To account for the physical displacement between the iPhone and the operator, we apply sequential offsets to recover the operator’s center of rotation $p_o$: a device-specific offset $p_d$ to find the camera mount center $p_c$ \cite{wu2024tidybot++}, then an operator-specific offset $p_u$ to recover the user's true center of rotation $p_o$, where $R_c$ is the camera mounting orientation:
\begin{equation}
p_{c} = p_{r} + R_{r}p_{d}, \quad p_{o} = p_{c} + R_{c}p_{u}
\end{equation}
% This sequential transformation shifts the tracking reference from the physical sensor to the operator's body center. Consequently, the resulting trajectory represents the user's true translational intent rather than the arc-like swinging motion typical of a backpack-mounted sensor during body rotations.
This transformation ensures that the tracked trajectory represents the user's actual body movement rather than the motion of the backpack-mounted sensor. Details about offsets can be found in Appendix~\ref{sec:webxr_calibration}.
To synchronize the coordinate systems, we map the operator’s tracking pose $(p_o, R_o)$ from the WebXR session frame into the robot’s world frame. Given the session's initial pose $(p_s, R_s)$ and the robot's global starting pose $(p_w, R_w)$, the world-relative target position $p_t$ is computed via:
\begin{equation}
R_{\text{rel}} = R_w R_s^{-1}, \quad p_t = p_w + R_{\text{rel}}(p_o - p_s)
\end{equation}
The target orientation matrix is similarly updated as $R_t = R_{\text{rel}} R_o$. From the resulting world-relative target state $(p_t, R_t)$, we extract the planar $x, y$ coordinates and the yaw heading, which are then issued as absolute target setpoints to the robot base controller.
% It is important to note that the iPhone and the center of rotation of the operator are fundamentally different. Thus, to recover the operator's pose in the robot frame, we apply a series of transformations. Because we are interested in the operator's pose and frame and not that of the iPhone, we must apply a set of transformations. Firstly, we apply a change of basis to the robot convention. This can be expressed as
% \begin{equation}
% \begin{aligned}
% p_{r} &= A\,p_{xr}\\
% R_{r} &= A \, R_{xr} \, A^T,
% \end{aligned}
% \end{equation}
% where $A$ maps the WebXR frame to the robot basis and $p_{xr}, R_{xr}$ are the position and rotation of the iPhone in the WebXR frame, respectively. We then apply a device offset $p_{d}$ to get center of rotation as done in \cite{wu2024tidybot++}:
% \begin{equation}
% p_{c} = p_{r} + R_{r}\,p_{d}.
% \end{equation}
% Next, we apply another offset $p_u$ to account for the difference in centers between the iPhone and the operator expressed as $p_{o} = p_{c} + R_{c}\,p_{u}$.
% % \begin{equation}
% % p_{o} = p_{c} + R_{c}\,p_{u}.
% % \end{equation}
% We convert the pose to a delta in the starting position $p_w$ and rotation $R_{w}$ for the WebXR session, which may be different from the current episode's starting position $p_s$ and rotation $R_s$.
% \begin{equation}
% \begin{aligned}
% R_{rel} &= R_w R_s\\
% p_{t} &= p_w + R_{rel}\,(p_o - p_s).
% \end{aligned}
% \end{equation}
% Finally, we take the $x,y$ and yaw values and convert them to absolute commands to send them as targets in base frame.
\subsubsection{Active Perception}
We implement the active perception module with an Apple Vision Pro running a custom visionOS app, which streams the operator’s head pose as robot neck commands similar to~\cite{xiong2025vision}. The interface provides a low-latency egocentric view of the robot, rendering either monocular RGB video or 3D point clouds in real time.
To map the operator's head motion, we track the current device pose $T_c \in SE(3)$ relative to the initial session pose $T_s \in SE(3)$, yielding the relative transformation $T_{\text{rel}} = T_s^{-1}T_c$. We then convert this motion into the robot head coordinate convention using a fixed basis transformation $B \in SO(3)$, mapping the relative translation and rotation as $p_r = Bp_{\text{rel}}$ and $R_r = BR_{\text{rel}}B^\top$. Together, these define the robot-frame head command $T_r \in SE(3)$. Finally, we transform $T_r$ into the robot world frame using the initial tracking reference $T_{\text{init}}$, yielding the global target head pose $T_{\text{target}} = T_{\text{init}}T_r$.
% \maxcomment{Might be good to preempt coment about this not being a problem if there's a unified coordinate system, could be shorter note.}
\mypara{Decoupling Head and Base Motion.}
We cannot directly command $T_{\text{target}}$, as it is expressed in the robot world frame and includes motion already induced by the mobile base. To avoid redundant commands during simultaneous navigation and active perception, we compensate for the current base pose $T_{\text{base}} \in SE(3)$ and express the target in the local robot base frame:
$T_{\text{neck}} = T_{\text{base}}^{-1} T_{\text{target}}$.
This isolates the residual neck motion intended by the operator.
% \mypara{Decoupling Head and Base Motion.} When an operator yaws their whole body, the neck and base both record rotation separately. This means that the neck will over-rotate. The same issue occurs for position. To prevent this, we must account for base movement before sending target poses to the neck. We first transform the relative commanded pose $T_r$ into the neck frame using the neck start pose, yielding $T_{er} = T_a T_r$. We then compensate for the current base motion using the base transform $T_b$, giving $T_{ee} = T_b^{-1} T_{er}$.
% We transform the relative commanded pose $T_r$ from the base frame to the neck frame $T_{er}$ using the base position $T_b$ and neck start pose $T_a$. We then compensate for the base movement using $T_b$. Putting this together, we compute this as
% \begin{equation}
% \begin{aligned}
% T_{er} &= T_a\,T_r\\
% T_{ee} &= T_b^{-1}\,T_{er}.
% \end{aligned}
% \end{equation}
\subsection{Policy Learning}
\vspace{-1mm}
We train a standard transformer-based Diffusion Policy model \cite{chi2023diffusion} for each task using the collected teleoperation demonstrations. For the customized mobile robot, the policy takes as input RGB-D observations from the head camera and RGB observations from the left and right wrist cameras. For the RB-Y1m robot, the policy takes as input RGB observations from two head cameras and the left and right wrist cameras, and optionally left and right arm joint torques.
%
\looseness-1
All visual inputs are encoded using a CLIP-pretrained ViT, with depth images repeated across three channels before encoding. Torque observations are encoded with sequential CausalConv layers following~\cite{lee2019icra}. The resulting observation tokens are concatenated and cross-attended to the DiT model as conditioning inputs. Refer to Appendix~\ref{sec:policy} for additional details.
% \maxcomment{Small note: maybe use texttt or textsc instead of italic for the module names to make it stand out more}
\vspace{-1mm}
To demonstrate \ours's adaptibility to different robot embodiments, we evaluate \texttt{Cloth Placement with Active Perception} task on the customized bimanual robot with ARX5 arms from \cite{xiong2025vision} with added holonomic base \cite{wu2024tidybot++}, and the \texttt{Box Transfer with Haptic Feedback} on an RB-Y1m robot. During collection, \ours~was configured to use the appropriate \textsc{leader arms} module, the \textsc{mobile base} module, and the \textsc{active perception} module. For the RB-Y1m robot, we also utilize haptic feedback on the leader arm. We discuss the specifics of each task below, followed by a summary of the results.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth, center]{figures/task_1_v4.pdf}
\caption{\footnotesize \textbf{Summary of Cloth Placement task.} a) The robot picks up a cloth, uses active perception to locate a basket placed on either side, navigates toward it, and places the cloth inside. b) Different evaluation scenarios with varying towel colors and basket locations. c) Task success rates across policy variants over input visual modalities. d) Failure cases across different policies.
}
\label{fig:results-task1}
\vspace{-4mm}
\end{figure}
% https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/ModPack-Figure?node-id=449-16
\subsection{Cloth Placement with Active Perception}
\textbf{Task Description. } As shown in Figure \ref{fig:results-task1} a) and b): a cloth is placed in front of the robot on a counter, where a basket is either to the left or to the right of the robot. The robot must move forward to grasp the cloth and then reverse, at which point it should locate the basket on either side and traverse towards it, finishing the task once it places the cloth in the basket. We collect data with two cloths (blue, pink), two basket locations (left, right), and randomize initial positions for the cloth. The task is considered a success if \textbf{the cloth is dropped into the basket}. We collected 125 demonstrations in total for this task.
% This task highlights four distinct skills: \emph{Whole-system coordination}: The robot must control the base, neck, and arm movements simultaneously throughout the entirety of the task. \emph{Active perception}: The robot moves its neck to locate the basket. \emph{Navigation}: The robot must accurately move its base towards the basket and orient appropriately to drop the cloth. \emph{Bimanual coordination}: Both arms are actively controlled, with the right arm performing manipulation while the left arm maintains a steady position.
% For each scenario, we perform 5 rollouts, for 20 total evaluation trials. We additionally evaluate 5 challenge trials: one rollout from each scenario and one fully randomized trial varying basket side, cloth color, robot pose, and cloth pose, for 25 total trials overall.
\textbf{Results. }
We evaluate the task across scenarios with varying basket locations (left, right) and cloth colors (blue, pink). We compare three policies with different visual observation modalities: head camera only [Head Cam], wrist cameras only [Wrist Cam], and all cameras (head + both wrist cameras) [All Cam]. For each policy, we perform 25 rollouts with randomized cloth initial positions. See a breakdown of the evaluation scenarios in Appendix \ref{sec:cloth_conditions}. Quantitative results are shown in \Cref{fig:results-task1} c). [Head Cam] policy achieves the highest success rate, succeeding in 22/25 rollouts. [All Cam] policy succeeds in 20/25 rollouts, while [Wrist Cam] policy succeeds on only 3/25 rollouts. We hypothesize that the head camera performs best because the cloth admits many viable grasps, reducing the need for close-up wrist observations, while the head camera provides depth and a broader egocentric view for grasping and basket localization. Moreover, using only the head camera avoids potential out of distribution wrist camera observations caused by variations in wrist pose and cloth deformation. In contrast, [Wrist Cam] most often fails due to placement errors, as the policy lacks sufficient context of the basket location. Figure~\ref{fig:results-task1} d) shows a common failure mode for the [Wrist Cam] policy, which often dropped the cloth on the edge or out of the basket.
\vspace{-1mm}
\subsection{Box Transfer with Haptic Feedback}
\vspace{-1mm}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth, center]{figures/task_2_v4.pdf}
\caption{\footnotesize \textbf{Summary of Box Transfer task.} a) The robot picks up a box, rotates to observe the existing box on the shelf and infer the target placement location, and then places the box accordingly. b) Different task scenarios with varying shelf availability configurations. c) Task success rates across policy variants, comparing the full policy against head-camera-only and wrist-camera-only ablations. d) Failure cases across different policies.}
\label{fig:results-task2}
\vspace{-6mm}
\end{figure}
% https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/ModPack-Figure?node-id=724-2
\textbf{Task Description. } As shown in Figure \ref{fig:results-task2} a) and b), the robot is required to pick up a box from a cabinet, move to the rack, and place the box onto the unoccupied shelf, which can be either at the top or bottom. A trial is considered successful if \textbf{the robot places the box in a stable pose on the unoccupied shelf}. We collected 102 demonstrations in total, half for each of the two scenarios.
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-6mm}
\includegraphics[width=0.4 \textwidth]{figures/attn.pdf}
\caption{\textbf{Attention Analysis.} }
\vspace{-2mm}
\label{fig:attention}
\vspace{-4mm}
\end{wrapfigure}
% This task displays four capabilities: \emph{Whole-system coordination}: The robot must continue to grasp the box during transport. \emph{Bimanual coordination}: The robot must use both arms to grasp the box. \emph{Active perception}: The robot must keep the box in view for grasp adjustments. \emph{Navigation}: The robot must traverse to the placement location to place the box.
% \begin{itemize}[leftmargin=*, topsep=0pt, itemsep=0pt, parsep=0pt, partopsep=0pt]
% \item \textbf{Initial stage:} The robot approaches the table and grasps the box using both arms. The goal location is not in view.
% \item \textbf{Intermediate stage:} The robot lifts and transports the box while searching across a large workspace toward the goal location. Once the goal location is in view, it navigates there.
% \item \textbf{Goal stage:} The robot aligns with the target and places the box precisely.
% \end{itemize}
\textbf{Results. }
We evaluate the task across two scenarios where either the top or bottom shelf is occupied and the robot has to place the box on the other unoccupied shelf. We compare three policy variants: one using all RGB camera views (two head cameras and both wrist cameras) [All Cam], one using only the two head cameras [Head Cam], and one that takes in all camera views plus joint torque information for both arms [All Cam + Torque]. For each policy, we conduct 20 rollouts across two evaluation scenarios, varying the cart location where the box is initially placed and the robot’s starting positions. A detailed breakdown of the evaluation scenarios is provided in Appendix~\ref{sec:bt_conditions}.
Quantitative results are shown in Fig.~\ref{fig:results-task2} c). The policy conditioned on all camera views and joint torques [All Cam + Torque] achieves the highest success rate, succeeding in 12/20 rollouts, while [Head Cam] policy succeeds in 11/20 rollouts, and [All Cam] policy succeeds in 6/20 rollouts. As seen in Figure~\ref{fig:results-task2} d), a common failure mode of the [All Cam] policy is bad grasp poses. We hypothesize that incorporating projected joint torques improves performance by providing a coarse alignment signal for grasp timing and depth.
We provide a qualitative analysis of the model’s attention patterns. As shown in \Cref{fig:attention} a), we visualize the attention weights over image patches from the final layer of the ViT encoder for both the [All Cam] and [All Cam + Torque] policies. The torque-conditioned policy appears to attend more strongly to task-relevant regions, such as the edges of the box that are informative for grasping, whereas the vision-only policy attends to less relevant regions. In \Cref{fig:attention} b), we visualize the aggregated attention weights assigned to the torque tokens in the DiT model for the [All Cam + Torque] policy. The attention weights for both the left- and right-arm torque tokens increase as the robot approaches the box and initiates grasping, suggesting that the policy increasingly relies on torque information during contact-critical phases of the task.
% \subsection{Task Capabilities} \josh{instead, we should combine this with discussion of tasks in their respective subsections}
% This task is designed to evaluate key capabilities enabled by \ours:
% %
% \begin{itemize}[leftmargin=*, topsep=0pt, itemsep=0pt, parsep=0pt, partopsep=0pt]
% % \renee{Unsure where to put this -- not a capability that the robot must show, but important for demonstrators to use haptic feedback during high contact tasks, provides higher quality demonstration data even if we don't use it to train} \item \textbf{Haptic Feedback:} To avoid deforming or dropping the box during demonstration, haptic feedback is critical for regulating grasp force and maintaining a stable grasping pose during box transport. During placement, haptic feedback provides contact awareness that is not fully observable from vision alone.
% \item \textbf{Coordination and Mobility:} The robot must coordinate its mobile base, active head (neck), and both arms to transport the object across a large workspace. The base enables large-scale motion, the arms perform precise manipulation (including all vertical motion, as the torso remains fixed), and the head adjusts viewpoint to support perception during execution. \renee{not sure if this is worth highlihgting, active perception during occlusion plus navigation seems to be the most distinct skill compared to Task 1}
% \item \textbf{Active Perception under Occlusion:} Once grasped, the box partially occludes the robot's field of view. This requires active neck motion to maintain visibility of the environment during navigation and periodically re-acquire the goal location.
% \end{itemize}
% \subsection{Data Collection}
% We collected \todo{$X$} demonstrations with varied initial positions and end goal locations.
% \paragraph{Haptic Feedback during Demonstration.}
% During data collection, haptic feedback is provided to the human operator to improve demonstration quality in contact-rich interactions. In particular, it enables regulation of grasp force to avoid deforming or dropping the box during transport, and provides contact awareness during placement when visual cues are insufficient due to occlusion or limited depth perception. \renee{double check that we do not plan on using force for training} While haptic feedback is not used as an input modality for policy learning, it is useful for collecting stable and precise demonstrations.
% \subsection{Test Scenarios} As shown in Figure \ref{fig:task2_rollout}, we conducted \todo{$X$} rollouts in total, consisting of varied initial box positions and varied end locations.
% \begin{figure}[h]
% \centering
% \includegraphics[width=\linewidth]{figures/results_t1_v4.pdf}
% \caption{The rightmost chart displays the percent success rates of three policies on the cloth in basket task across 20 rollouts each. The policy trained on the head camera plus the two wrist cameras succeeded 18 times, the head camera only policy succeeded 20 times, and the wrist cam only policy succeeded only 3 times. ... \todo{sepertate the result plot}}
% \label{fig:task_results}
% %https://www.figma.com/design/eiAH0RQlPKBrBsUeB9fEXt/Cover-Figure?node-id=256-2&p=f&t=B4ssEAPwW31BEX3n-0
% \end{figure}
% During this experiment, we intentionally oscillates the grasp by repeatedly squeezing inward and stretching out the box to test the responsiveness of the force feedback system.
% \textbf{Haptic Feedback Comparison}
% To evaluate the effect of haptic feedback on user experience and task completion during teleoperation, we conduct a small comparison with various users using the \textbf{Box Transfer} task. This task is chosen because it requires precise alignment and contact awareness, as shown in \Cref{tab:haptic_results}.
% %
% We recruit 8 people, all first-time users with no prior experience using the system and randomly split them into two groups of four: one group completes the task with haptic feedback, and the other without. We record the following metrics for both cases, ensuring the order of conditions is randomized across participants to mitigate ordering effects:
% \begin{itemize}[leftmargin=*, topsep=0pt, itemsep=0pt, parsep=0pt, partopsep=0pt]
% \item {First attempt success:} Participants' success rate on their first attempt.
% \item {Successes in fixed time:} Number of box placements within a 3-minute window.
% \item {Peak force:} Average of the norm across the 3 force axes of the wrist force/torque sensor across the top 10 samples.
% % \todo{Average joint force applied during a success demo? lower the better? }
% \end{itemize}
% In the absence of haptic feedback, participants relied solely on visual cues, which often led to increased trial-and-error behavior, longer completion times, and reduced success rates. These findings are consistent with prior work \cite{haptic_2025} on haptic-feedback assisted teleoperation in contact-rich manipulation tasks.
% \begin{table}[t]
% \centering
% \small
% \begin{tabular}{lcccc}
% \toprule
% \textbf{Condition} & \textbf{1st Succ Avg. Time (s)} & \textbf{Total Succ in 5 min} & \textbf{Max force (N)}\\
% \midrule
% Without Haptic Feedback & $X$ & $X$ & $X$ \\
% With Haptic Feedback & $X$ & $X$ & $X$ \\
% \bottomrule
% \end{tabular}
% \vspace{2mm}
% \caption{Results for box transfer task with and without haptic feedback. We report first-time success rate, average completion time, and number of successful pick and place instances within a 5-minute window.}
% \vspace{-4mm}
% \label{tab:haptic_results}
% \end{table}
% As shown in Fig.~\ref{fig:teleop_setup}, we visualize the human operator's perspective during teleoperation. For the study, \ours~is configured with 1-to-1 leader arm modules, active perception, and optional haptic feedback. The mobility module is disabled since no mobile base is required for this tabletop task.
% \josh{@renee can continue from here, let me know if you like this framing}
% figure shows egocentric viewpoint in Apple Vision Pro, close-up of the operator’s arms interacting with the leader device, also backpack.
% \josh{we dont need to re-introduce the apple vision pro, we will have explained this in methods, same comment for description of teleoperation description}
% \begin{itemize}[leftmargin=*, topsep=0pt, itemsep=0pt, parsep=0pt, partopsep=0pt]
% \item \textbf{Visual input:} A mixed reality headset (e.g., Apple Vision Pro) providing a first-person view from the robot's perspective.
% \item \textbf{Control interface:} Our wearable \ours backpack with leader arms, which the user operates to perform the task.
% \item \textbf{Interaction conditions:}
% \begin{itemize}
% \item Visual-only (no haptic feedback)
% \item Visual + haptic feedback
% \end{itemize}
% \end{itemize}
% A central goal of our system design is to reduce the embodiment gap between the human operator and the robot, enabling high-quality demonstration data without requiring the operator to mentally compensate for differences in morphology, kinematics, or control interfaces. To this end, we abstract away low-level robot execution details, allowing the operator to focus on high-level task intent and intuitive bimanual control through the leader arms, rather than reasoning about how to "map" their actions onto the robot.
% In this setting, we compare two conditions—visual-only and visual plus haptic feedback—to evaluate the impact of haptic sensing on performance in contact-rich manipulation tasks.
% \subsubsection{Participants}
% \subsubsection{Evaluation Metrics}
% We measure the following metrics:
% \subsubsection{Experimental Procedure}
% Each participant performs the peg insertion task with and without haptic feedback enabled.
% \begin{enumerate}
% \item Visual-only
% \item Visual + haptic feedback
% \end{enumerate}
% Haptic feedback improves contact awareness, precision during alignment, and overall efficiency. \josh{we cannot conclude this based on the metrics we are planning to collect, this is speculative}
% \zeyi{A visualization of how the haptic feedback signals change during one demo can also be helpful. expect a spike when the gripper get into contact with the object}
\section{Comparison with Prior Works}
\label{sec:prior_table}
\begin{table}[h]
\centering
\scriptsize
\vspace{-2mm}
\setlength{\tabcolsep}{6pt}
\begin{tabular}{lccccc}
\toprule
& \textbf{Cross-Robot} & \textbf{Joint-Level} & \textbf{Haptic} & \textbf{Mobile} & \textbf{Active} \\
& \textbf{Adaptability} & \textbf{Control} & \textbf{Feedback} & \textbf{Manipulation} & \textbf{Perception} \\
\midrule
Open-Television~\cite{cheng2024open}
& \cmark & \xmark & \xmark & \xmark & \cmark \\
Mobile-Television~\cite{lu2025mobile}
& \cmark & \xmark & \xmark & \cmark & \cmark \\
GELLO~\cite{wu2024gello}
& \cmark & \cmark & \xmark & \xmark & \xmark \\
Mobile ALOHA \cite{fu2024mobile}
& \xmark & \cmark & \xmark & \cmark & \xmark \\
%TidyBot++ \cite{wu2024tidybot++}
%& \cmark & \xmark & \xmark & \cmark & \xmark \\
%Bunny-VisionPro \cite{ding2025bunny}
%& \cmark & \xmark & \cmark & \xmark & \xmark \\
AirExo-2 \cite{fang2025airexo}
& \cmark & \cmark & \xmark & \cmark & \xmark \\
ViA~\cite{xiong2025vision}
& \xmark & \cmark & \xmark & \xmark & \cmark \\
CHILD~\cite{myers2025child}, PAPRLE~\cite{kwon2025paprle}, SuperSuit \cite{chen2026supersuit}
& \cmark & \cmark & \xmark & \cmark & \xmark \\
DLM w/ Feedback \cite{purushottam2024wheeledhumanoidbilateralteleoperation}
& \xmark & \xmark & \cmark & \cmark & \xmark \\
VR + Haptic \cite{zhang2023wearable, wu2025vrbased,ding2025bunny}
& \cmark & \xmark & \cmark & \xmark & \xmark \\
\midrule
\textbf{\ours}
& \cmark & \cmark & \cmark & \cmark & \cmark \\
\bottomrule
\end{tabular}
\vspace{2mm}
\caption{\footnotesize \textbf{Comparison} with existing teleoperation systems across key capabilities. \ours~supports the key capabilities studied in prior work within a unified, modular interface.
}
\vspace{-4mm}
\label{tab:capability_comparison}
\end{table}
\section{Hardware Overview}
\subsection{3D Printing}
\label{sec:3d_printing}
Any 3D printed components of the backpack were printed using the Bambu Labs H2D
for larger components such as leader arms and backpack panels, as well as the
smaller Bambu X1 Carbon for smaller pieces. The choice for the backpack to be fabricated in two parts is such that it can be easily 3D printed. The print settings can be found in
Table \ref{tab:print-settings}.
\begin{table}[h]
\centering
\caption{3D Print Settings}
\label{tab:print-settings}
\begin{tabular}{ll}
\toprule
Parameter & Value \\
\midrule
Material & PLA \\
Infill & 15 \% \\
Infill Pattern & Gyroid \\
Layer Height & 0.2 mm \\
Wall Count & 2 \\
Nozzle Diameter & 0.4 mm\\
Support Strategy & Tree \\
\bottomrule
\end{tabular}
\end{table}
% \todo{Why backpack is split in two, printer used, filament used, printer settings}
% \subsection{Weight and Cost Summary}
% \todo{Put weight! Put rough estimate of cost!}
\section{Backpack Design}
\subsection{Backpack Circuit}
\label{sec:circuit}
\begin{figure}[h]
\centering
\includegraphics[width=0.7\linewidth, center]{figures/circuit_diagram_v2.pdf}
\caption{Circuit diagram for ModPack}
\label{fig:circuit}
\end{figure}
Empirically, when ModPack is run with one battery per arm and a third for the Geekom A5 mini PC, the system battery life is around 4-6 hours. Figure \ref{fig:circuit} displays the layout of the power circuit.
\FloatBarrier
\section{Leader Arms}
\label{sec:leader_arms}
\subsection{ARX (6-dof) Leader Arm}
\subsubsection{Motor Selection}
% \todo{list motors selected in table with relevant parameters like stall torque etc, and math to justify, control mode used}
Motors were selected to provide the minimum amount of torque necessary to hold the arm for mass assumptions made using OnShape materials library. All motors are Dynamixel brand, with the models and control modes enumerated below. For this leader arm, we choose to not actuate joints 5 and 6 to allow for better wrist mobility.
\begin{table}[h]
\centering
\caption*{Motor Specifications 6-DoF Leader Arm}
\label{tab:}
\begin{tabular}{clcc}
\toprule
Joint & Motor Model & Stall Torque (Nm) & Control Mode\\
\midrule
1& XM540-W270-T & 10.6 & Torque (Current)\\
2& XM540-W270-T & 10.6 & Torque (Current)\\
3& XM430-W350-T & 4.1 & Torque (Current)\\
4& XM430-W350-T & 4.1 & Torque (Current)\\
5& XL430-W250-T & 1.4 & N/A \\
6& XL430-W250-T & 1.4 & N/A\\
Gripper& XL330-M288-T& 0.52 & N/A \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Motor Parameters}
Each joint motor is clipped to an empirically determined current limit, which was chosen to provide stable gravity compensation without making the joint too stiff. Damping gains were chosen empirically to reduce jitter.
\begin{table}[h]
\centering
\caption*{Joint Control Parameters}
\begin{tabular}{ccc}
\toprule
Joint & Current Limit (mA) & Damping Gain $k_d$ \\
\midrule
1 & 100 & 0.1 \\
2 & 200 & 0.1 \\
3 & 200 & 0.1 \\
4 & 60 & 0.1 \\
5 & 500 & 0.1 \\
6 & 100 & 0.1 \\
Gripper & 100 & --- \\
\bottomrule
\end{tabular}
\end{table}
\subsection{RB-Y1m (7-dof) Leader Arm}
\subsubsection{Motor Selection}
Motors were selected to provide the minimum amount of torque necessary to hold the arm for mass assumptions made using OnShape materials library, as well as have a high enough maximum current limit to support haptic feedback. All motors are Dynamixel brand, with the models and control modes enumerated below. For this leader arm, we choose to not actuate joints 5 and 7 to allow for more stable control and better wrist mobility, respectively.
% \todo{list motors selected in table with relevant parameters like stall torque etc, and math to justify, control mode used}
\begin{table}[h]
\centering
\caption*{Motor Specifications 7-DoF Leader Arm}
\label{tab:rby1-motors}
\begin{tabular}{clcc}
\toprule
Joint & Motor Model & Stall Torque (Nm) & Control Mode \\
\midrule
1 & XM540-W270-T & 10.6 & Torque (Current) \\
2 & XM540-W270-T & 10.6 & Torque (Current) \\
3 & XM540-W270-T & 10.6 & Torque (Current) \\
4 & XM540-W270-T & 10.6 & Torque (Current) \\
5 & XM540-W270-T & 10.6 & N/A \\
6 & XM430-W350-T & 4.1 & Torque (Current) \\
7 & XM430-W350-T & 4.1 & N/A \\
Gripper & XL330-M288-T & 0.52 & N/A \\
\bottomrule
\end{tabular}
\end{table}
\subsubsection{Motor Parameters}
% \todo{list current budgets, gain parameters for both robots, rate limiter for haptic feedback, baudrate}
Each joint motor is clipped to an empirically determined current limit, which was chosen to provide stable gravity compensation without making the joint too stiff. An additional current budget is allocated to actuated motors to allow for haptic feedback torques, which are rate limited to 50 Nm/s and then scaled by 0.8 on the left arm specifically, which empirically prevented jitter.
\begin{table}[h]
\centering
\caption*{Joint Control Parameters 7-DoF Leader Arm}
\begin{tabular}{ccc}
\toprule
Joint & Gravity Current Limit (mA) & Contact Current Limit (mA) \\
\midrule
1 & 200 & 150 \\
2 & 50 & 50 \\
3 & 100 & 100 \\
4 & 100 & 100 \\
5 & --- & --- \\
6 & 100 & 80 \\
7 & --- & --- \\
Gripper & --- & --- \\
\bottomrule
\end{tabular}
\end{table}
\section{Base Motion}
\subsection{WebXR Calibration}
\label{sec:webxr_calibration}
% \todo{describe how to get constants mentioned in paper for offsets}
There are two offsets mentioned in the paper: the device offset $p_d$ and the offset from mount center to the operator's center of rotation $p_u$. $p_d$ is a fixed offset given an iPhone 17, but should be re-examined for other models. This can be done using calipers to get the offset to center of iPhone. $p_u$ can be measured from roughly the center of operator's head to the mount center, though in practice does not need to be changed much from operator to operator.
\section{Robots}
\label{sec:robots}
\subsection{Customized mobile robot \cite{xiong2025vision}}
Our customized mobile robot consists of bimanual 6-DoF ARX5 arms as well as an additional 6-DoF ARX5 for active perception as presented in \cite{xiong2025vision}. The neck is equipped with an iPhone that records depth as well as RBG images, while the other two arms have a Logitech webcam to capture wrist RBG images. Details on capture rate are left for Appendix \ref{sec:policy}. This setup sits on a Tidybot++ base, a four-wheeled base from \cite{wu2024tidybot++}.
% \todo{cite via and how this robot differs, camera configs}
\subsection{RB-Y1m}
The RB-Y1m robot we use consists of bimanual setup with 7-DoF arms, as well as a 6-DoF actuated torso on a holonomic mobile base. Using the setup from \cite{xu2026hommilearningwholebodymobile}, we make use of a 2-DoF active neck as well as fin-ray fingers as grippers. Continuing with the setup from \cite{xu2026hommilearningwholebodymobile}, we use 2 wide-angle cameras, \texttt{FLIR
BFS-PGE-23S3C-CS}, and one \texttt{FLIR BFS-PGE-50S5C-C} on each wrist of the two 7-DoF arms.
\section{Software Diagram}
Software is split into two distinct sides as seen in Figure~\ref{fig:software}, with \ours~handling lifecycle management and modules while also setting relevant configurations for downstream robots. The per-robot configurations define the bridge topics as well as what modules are turned on, meaning that all the end user must do is call the relevant bridge API for bi-directional communication with \ours~through \texttt{robot-message-queue} \cite{gao2026gatedmemorypolicy}. This means that actions flow from module commands to robot through \texttt{robot-message-queue} servers. Data is saved in a \texttt{zarr} format using \texttt{robologger}, an open source logging pipeline. This pipeline has a main logger than coordinates individual process loggers.
\label{sec:software_appendix}
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth, center]{figures/software_diagram.pdf}
\caption{Software diagram for ModPack.}
\label{fig:software}
\end{figure}
\FloatBarrier
\section{Policy}
\label{sec:policy}
During training, we synchronize all sensory inputs to the downsampled head-camera timestamps (often 10-20 Hz). Proprioceptive states are interpolated at each head-camera timestamp, with positions and gripper states interpolated linearly and orientations interpolated using SLERP. Other modalities are aligned by selecting the nearest preceding measurement to the head-camera timestamp.
For the custom mobile robot, the action space consists of a 3-DoF mobile-base command $(x, y, \theta)$, a 6-DoF head-camera end-effector pose, and bimanual arm commands. We represent the head pose using 3D translation and the 6D rotation representation, yielding a 9-dimensional head action. Each 6-DoF arm is controlled by six joint positions and a one-dimensional gripper position, resulting in 7 dimensions per arm. The full action dimension is therefore $3 + 9 + 7 + 7 = 26$. We predict the base command relative to the current base state, while the head, arm, and gripper commands are represented as absolute targets.
For the RB-Y1M robot, the action space consists of a 3-DoF mobile-base command $(x, y, \theta)$, a 2-DoF neck command corresponding to pitch and yaw, and bimanual arm commands. Each 7-DoF arm is controlled by seven joint positions and a one-dimensional gripper position, yielding 8 dimensions per arm. The full action dimension is therefore $3 + 2 + 8 + 8 = 21$. As with the custom mobile robot, we predict the base command relative to the current base state, while the neck, arm, and gripper commands are represented as absolute targets.
During inference, the model denoises an action chunk of length 16 using DDIM with 16 inference steps. We execute the first 8 actions for the cloth placement task and the full 16-action chunk for the box transfer task.
\section{Cloth Placement with Active Perception}
\subsection{Demonstration Collection and Rollout Conditions}
\label{sec:cloth_conditions}
% Approximately the same amount of demonstrations were collected for each of the four task configurations (basket left or right, cloth pink or blue) as well as for rollouts.
\begin{table}[h]
\centering
\begin{minipage}{0.45\linewidth}
\centering
\textbf{Demos}\\[4pt]
\begin{tabular}{lcc}
\toprule
& \textbf{Pink Cloth} & \textbf{Blue Cloth} \\
\midrule
Basket Left & 33 & 31 \\
Basket Right & 28 & 33 \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}{0.45\linewidth}
\centering
\textbf{Rollouts}\\[4pt]
\begin{tabular}{lcc}
\toprule
& \textbf{Pink Cloth} & \textbf{Blue Cloth} \\
\midrule
Basket Left & 6 & 6 \\
Basket Right & 6 & 7 \\
\bottomrule
\end{tabular}
\end{minipage}
\vspace{4pt}
\caption{Demonstration and rollout splits across task configurations.}
\label{tab:data_split}
\end{table}
\section{Box Transfer with Haptic Feedback}
\subsection{Demonstration Collection and Rollout Conditions}
\label{sec:bt_conditions}
\begin{table}[H]
\centering
\begin{minipage}{0.45\linewidth}
\centering
\textbf{Demos}\\[4pt]
\begin{tabular}{lc}
\toprule
\textbf{Target Shelf} & \\
\midrule
High & 54 \\
Low & 48 \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}{0.45\linewidth}
\centering
\textbf{Rollouts}\\[4pt]
\begin{tabular}{lc}
\toprule
\textbf{Target Shelf} & \\
\midrule
High & 10 \\
Low & 10 \\
\bottomrule
\end{tabular}
\end{minipage}
\vspace{4pt}
\caption{Demonstration and evaluation rollout splits across target shelf configurations.}
\label{tab:box_split}
\end{table}