Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
Extracted source text · arXiv 2505.20795
\documentclass{article}
\usepackage{amssymb}
% \usepackage{corl_2025} % Use this for the initial submission.
\usepackage[final]{corl_2025} % Uncomment for the camera-ready ``final'' version.
%\usepackage[preprint]{corl_2025} % Uncomment for pre-prints (e.g., arxiv); This is like ``final'', but will remove the CORL footnote.
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb} %数学公式
\usepackage{amsfonts} %数学字体
\usepackage{mathrsfs} %数学花体
\usepackage{graphicx}
\usepackage{float}
% \usepackage{subfig}
\usepackage{booktabs}
\usepackage{times}
\usepackage{multirow} % 用于合并单元格
\usepackage{wrapfig}
% numbers option provides compact numerical references in the text.
\usepackage{stfloats}
\usepackage{caption}
\usepackage{makecell}
\usepackage{subcaption}
\usepackage{titlesec}
\usepackage{enumitem}
\setlist[enumerate]{left=0pt}
% 缩小行间距
\titlespacing*{\section}{0pt}{1ex}{0.5ex} % section 标题与段落间距
\titlespacing*{\subsection}{0pt}{1ex}{0.5ex} % subsection 标题与段落间距
\setlength{\parskip}{0.5ex}
% 缩小公式间距
\setlength{\abovedisplayskip}{5pt}
\setlength{\belowdisplayskip}{5pt}
\setlength{\abovedisplayshortskip}{5pt}
\setlength{\belowdisplayshortskip}{5pt}
\setlength{\jot}{3pt} % 多行公式
\setlength{\textfloatsep}{5pt}
\setlength{\intextsep}{5pt}
\setlength{\itemsep}{2pt} % 控制每个元素之间的间距
\setlength{\parsep}{0pt} % 控制段落与列表项之间的间距
\setlength{\topsep}{0pt} % 控制列表和段落之间的空隙
\setlength{\partopsep}{0pt} % 控制列表前后与其他段落之间的间距
\title{Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt}
% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.
% NOTE: authors will be visible only in the camera-ready and preprint versions (i.e., when using the option 'final' or 'preprint').
% For the initial submission the authors will be anonymized.
\begin{document}
\author{
Xiang Zhu$^*$\\
Tsinghua University \\
Shanghai Qi Zhi Institute \\
\texttt{zhuxiang24@mails.tsinghua.edu.cn} \\
$^*$Equal contribution. Project Co-lead. \\
%% examples of more authors
\And
Yichen Liu$^*$ \\
Tsinghua University \\
\texttt{liu-yc22@mails.tsinghua.edu.cn} \\
$^*$Equal contribution. Project Co-lead. \\
\And
Hezhong Li \\
Tsinghua University \\
\texttt{lhz21@mails.tsinghua.edu.cn} \\
\And
Jianyu Chen$^\text{†}$ \\
Tsinghua University \\
Shanghai Qi Zhi Institute\\
RobotEra TECHNOLOGY CO., LTD.\\
\texttt{jianyuchen@mail.tsinghua.edu.cn} \\
† Corresponding Author. \\
}
\maketitle
\vspace{-8mm}
\begin{center}
\centering
\hspace*{-1mm}
\includegraphics[width=1.00\linewidth]{images/Main_Figure_V4.pdf}
\vspace*{-5mm}
\captionof{figure}{First, fine-tune a video diffusion model on diverse datasets to obtain informative representations. In the second stage, use a video generation model to extract information from human prompt videos for skill learning with both human and robot data. Finally, during inference, the model prompts on unseen human demos to perform tasks based on the human input.}
\label{fig:main_exp}
\end{center}%
% \vspace{-0.05in}
%===============================================================================
\begin{abstract}
% Learning from human demonstration videos provides a natural and efficient paradigm for teaching robots complex dexterous manipulation skills by directly leveraging the rich, high‑dimensional information inherent in human motion. In this paper, we introduce a novel two‑stage framework that transfers manipulation patterns from human videos to a robotic dexterous hand, mitigating the challenges of scarce robot data and handcrafted rewards. In the first stage, we train a cross‑prediction video generation model that, given a human demonstration video and a target hand’s initial scene, synthesizes a corresponding video of the robot executing the task. By alternating between cross‑prediction (human→robot) and normal‑prediction (human→human or robot→robot), the model learns representations that encode embodiment‑invariant task dynamics alongside modality‑specific scene context. In the second stage, we fuse these learned representations with extensive human and limited robot datasets to train a unified diffusion policy. A novel prototypical contrastive loss and a shared action space design bridge the two modalities and various skills, enabling the policy to generalize robust manipulation strategies even with minimal robot demonstrations. Empirical evaluations on real‑world dexterous tasks show a promising success rate, performance boosting, and transferability.
Recent robot learning methods commonly rely on imitation learning from massive robotic dataset collected with teleoperation. When facing a new task, such methods generally require collecting a set of new teleoperation data and finetuning the policy. Furthermore, the teleoperation data collection pipeline is also tedious and expensive. Instead, human is able to efficiently learn new tasks by just watching others do. In this paper, we introduce a novel two-stage framework that utilizes human demonstrations to learn a generalizable robot policy. Such policy can directly take human demonstration video as a prompt and perform new tasks without any new teleoperation data and model finetuning at all. In the first stage, we train video generation model that captures a joint representation for both the human and robot demonstration video data using cross-prediction. In the second stage, we fuse the learned representation with a shared action space between human and robot using a novel prototypical contrastive loss. Empirical evaluations on real‑world dexterous manipulation tasks show the effectiveness and generalization capabilities of our proposed method.
% Two or three meaningful keywords should be added here
\keywords{Learn from Human, Human Video Prompt}
\end{abstract}
\section{Introduction}
\input{introduction/intro}
\section{Related Works}
\input{related_works/related_works}
% \section{Preliminaries}
% \input{preliminaries/diffusion model}
\section{METHODS}
\input{methods/CrossPredict}
% \begin{center}
% \centering
% \hspace*{-1mm}
% \includegraphics[width=1.00\linewidth]{images/Stage1V2.pdf}
% \vspace*{-5mm}
% \captionof{figure}{Stage 1}
% \label{fig:stage1}
% \end{center}%
\input{methods/human-action}
\input{methods/PDCP}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{EXPERIMENTS}
\input{experiment/exp_intro}
\input{experiment/exp_video_generation}
\input{experiment/exp_cross_prediction}
\input{experiment/exp_stage2}
\section{CONCLUSIONS}
\input{conclusion/con}
\section{LIMITATIONS}
\input{limitation/lim}
\section{APPENDIX}
\input{appendix/appendix}
%===============================================================================
\bibliography{example}
\clearpage
% The acknowledgments are automatically included only in the final and preprint versions of the paper.
% \acknowledgments{If a paper is accepted, the final camera-ready version will (and probably should) include acknowledgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.}
%===============================================================================
% no \bibliographystyle is required, since the corl style is automatically used.
% \bibliography{example} % .bib
\end{document}
% Learning from human demonstration videos offers a promising and natural paradigm for teaching robots complex dexterous manipulation tasks. Human videos inherently capture rich, high-dimensional information, including subtle hand movements, object interactions, and contextual cues, which are often difficult to encode explicitly through traditional programming or simulation-based approaches. By treating these videos as prompts, robots can extract nuanced behavioral patterns and adaptively infer task-relevant strategies without the need for meticulously handcrafted reward functions or extensive supervision. This direct transfer of human expertise not only accelerates the learning process, but also enhances the robot’s ability to generalize across varied tasks and real-world environments. Ultimately, leveraging human demonstration videos bridges the gap between intuitive human motion and robotic precision, paving the way for more robust and adaptable robotic manipulation systems. In this work, we propose a novel two-stage framework designed to effectively transfer these human manipulation patterns to a robotic dexterous hand, thereby overcoming data scarcity issues and enhancing generalization.
Traditional robot learning methods typically train language-conditioned policies on large datasets collected through teleoperation\cite{Brohan2022RT1RT, Brohan2023RT2VM}. While effective for known tasks, this paradigm faces two fundamental limitations when encountering novel tasks. First, language instructions, though intuitive for humans, provide only categorical information and lack the rich spatial and temporal details crucial for physical manipulation. Second, adapting to new tasks usually requires collecting additional robot demonstrations, a process that is both time-consuming and expensive due to the complexity of teleoperation systems.
Humans, in contrast, can efficiently acquire new skills simply by observing others perform tasks. This observation suggests that visual demonstrations may offer a more natural and information-rich medium for teaching robots. Videos inherently capture not just what task to perform, but also how to perform it, including critical aspects like object relationships, motion trajectories, and timing. Moreover, human demonstration videos are significantly more scalable to acquire compared to robot data, whether captured in laboratory settings or obtained from existing online resources.
Current approaches have only begun to explore this direction. While works like EgoMimic\cite{kareer2024egomimic}have incorporated human demonstrations, it focuses on single-task scenarios. The field still lacks a general framework that can leverage the full potential of human videos for robot learning.
We propose a two-stage human-prompted learning framework that combines robotic datasets with human demonstration data to address challenges in task learning.
In the first stage, we use a video generation model that receives a prompt video of a human performing a task and an image of the robotic hand. The model generates a video of the robot performing the task, embedding embodiment transfer information through a cross-prediction strategy. This helps the model learn a representation that captures the task, context, and target modality effectively.
In the second stage, we fine-tune the representation using a diffusion policy, incorporating both human and robot data. A unified action space bridges the gap between the two modalities, while a cluster-based loss enhances skill separation and multi-skill imitation performance.
Experiments on real-world tasks demonstrate the framework’s effectiveness in improving human-robot interaction and versatile manipulation.
% In this work, we address these challenges by proposing a novel two-stage human-prompted learning framework that effectively leverages both robotic datasets and abundant human demonstration data.
% In the first stage, our method centers on a cross prediction strategy. Specifically, a video generation model is trained to receive a prompt video, depicting a task executed by a source embodiment (such as a human), alongside an initial scene image of a target embodiment (the robotic hand). The model then generates a video of the target embodiment performing the task. By randomly choosing between cross-prediction (using distinct source and target embodiments) and normal-prediction (using the same embodiment), we embed crucial embodiment transfer information into the representation learned by the model. This approach ensures that the representation effectively captures the target modality, intended task, and scene context.
% In the second stage, we leverage the intermediate representation obtained from the video generation model. Combined with extensive human demonstration data, we train a diffusion policy that is concurrently optimized using both human and robot data. A carefully designed unified action space further bridges the gap between the two modalities, enabling the diffusion policy to learn robust manipulation patterns even when high-quality dexterous hand data are limited. We also design a cluster-based loss into the diffusion process, which yield policies with stronger skill separation and improved performance on multi‐skill imitation.
% Through extensive experiments on real-world dexterous manipulation tasks, our framework demonstrates superior performance highlighting its potential to pave the way for more effective and versatile human-robot interaction in complex manipulation scenarios.
% The main contributions of our work are summarized as follows
% The main contributions of our work can be summarized as follows.
% \begin{enumerate}
% \item We propose a novel two-stage training framework that learns from human video demonstrations which efficiently integrates abundant human demonstration data with limited robotic dexterous hand data.
% \item A cross prediction video generation model that effectively transfers manipulation patterns across different embodiments, thereby embedding critical task and scene context information.
% \item A unified diffusion policy named \textbf{P}roto\textbf{D}iffusion \textbf{C}ontrastive \textbf{P}olicy(PDCP), trained on both human and robotic data under a common action space with addition prototypical contrastive loss, that enhances generalization and performance in dexterous hand manipulation tasks.
% \end{enumerate}
\subsection{Learn from Human Videos}
% Recent advances in robot manipulation have increasingly leveraged human video data to enhance both dexterous and gripper-based manipulation. In the dexterous manipulation domain, works such as \cite{chen2024vividex, qin2022dexmv, wang2024dexcap} have focused on achieving fine-grained control of complex, multi-fingered systems, while \cite{kannan2023deft} integrates affordance cues to further improve manipulation performance. For gripper manipulation, end-to-end video-conditioned policy methods like \cite{jain2024vid2robot, yuan2024general, ju2024robo, xu2023xskill} have demonstrated robust performance by directly translating visual cues into actionable policies.
% Approaches that use paired human–robot demonstration data, including \cite{wang2023mimicplay}, \cite{smith2019avid, xiong2021learning}, \cite{wen2023any}, \cite{gu2023rt}, \cite{kareer2024egomimic}, and \cite{ye2024latent}, address the domain gap by explicitly correlating human actions with robot trajectories.
% More recently, generative video techniques have emerged as a promising direction, with works such as \cite{liang2024dreamitate}, \cite{du2023learning}, \cite{bharadhwaj2024gen2act}, and \cite{shridhar2024generative} leveraging video synthesis and textual cues to generate visuomotor policies. Together, these studies illustrate a comprehensive trend toward utilizing human video demonstrations, paired data, and generative methods to develop more adaptable, robust, and generalizable robot manipulation policies.
Recent advances in robot manipulation have increasingly utilized human video data to enhance both dexterous and gripper-based manipulation. In dexterous manipulation, works like \cite{chen2024vividex, qin2022dexmv, wang2024dexcap} focus on fine-grained control of multi-fingered systems, while \cite{kannan2023deft} integrates affordance cues. For gripper manipulation, end-to-end video-conditioned policies such as \cite{jain2024vid2robot, yuan2024general, ju2024robo, xu2023xskill} translate visual cues into actionable policies.
Approaches using paired human–robot demonstration data, including \cite{wang2023mimicplay}, \cite{smith2019avid, xiong2021learning}, \cite{wen2023any}, \cite{gu2023rt}, \cite{kareer2024egomimic}, and \cite{ye2024latent}, address the domain gap by linking human actions with robot trajectories. More recently, generative video techniques like \cite{liang2024dreamitate}, \cite{du2023learning}, \cite{bharadhwaj2024gen2act}, and \cite{shridhar2024generative} leverage video synthesis and textual cues to generate visuomotor policies. These studies highlight the growing trend of using human video demonstrations, paired data, and generative methods to create more adaptable and robust robot manipulation policies.
\begin{figure*}[b]
\centering
\includegraphics[width=0.9\linewidth]{images/Stage1V2.pdf}
\captionof{figure}{\textbf{Stage 1}: We fine-tune a pre-trained video generation model using cross-prediction, enabling the model to retain physical knowledge while gaining the ability for embodiment modality transfer.}
\label{fig:stage1}
\end{figure*}
\subsection{Video as Prompt for Robotic Learning}
% Recent works\cite{chang2023one, xiong2023robotube} in robot learning has increasingly leveraged human demonstration videos as a rich source of guidance. For example, \cite{xiong2021learning} tackles the challenge of human–robot embodiment mismatch by converting human videos into robot-centric demonstrations via unsupervised domain adaptation and keypoint extraction, thereby generating structured representations for policy learning. Building on this, \cite{chane2023learning} conditions robot policies on pretrained video embeddings, enabling zero-shot generalization to entirely new tasks. In a similar vein, \cite{jain2024vid2robot} proposes an end-to-end framework that fuses video features with the current robot state using cross-attention transformers, directly mapping human demonstration videos to robot actions. Moreover, \cite{qian2024contrast} introduces a three-stage process—extracting task priors via contrastive learning, imitating demonstrated action trajectories, and adapting them through limited interaction—to improve sample efficiency and generalization. Finally, \cite{xu2023xskill} focuses on cross-embodiment skill discovery, learning transferable skill representations across different agents. In our work, we extend these insights by using video as a prompt and integrating video generation model to more effectively bridge the human–robot gap by embodiment transfer and to achieve robust, generalized policy learning for complex manipulation tasks.
Recent works \cite{chang2023one, xiong2023robotube} have increasingly used human demonstration videos to guide robot learning. For example, \cite{xiong2021learning} addresses human-robot embodiment mismatch by converting human videos into robot-centric demonstrations using unsupervised domain adaptation and keypoint extraction. \cite{chane2023learning} enables zero-shot generalization by conditioning robot policies on pretrained video embeddings. Similarly, \cite{jain2024vid2robot} maps human videos to robot actions through cross-attention transformers, while \cite{qian2024contrast} improves sample efficiency and generalization through contrastive learning, imitation, and limited adaptation. \cite{xu2023xskill} focuses on cross-embodiment skill discovery for transferable representations. In our work, we extend these insights by using video as a prompt and integrating video generation model to more effectively bridge the human–robot gap by embodiment transfer and to achieve robust, generalized policy learning for complex manipulation tasks.
\subsection{Diffusion Models for Robot Policy Learning}
% The remarkable success of diffusion models in generative computer vision \cite{Ho2020DenoisingDP,Song2020ScoreBasedGM} has inspired their adaptation to robotic policy learning. Pioneering works by \cite{Chi2023DiffusionPV, Pearce2023ImitatingHB, Reuss2023GoalConditionedIL} demonstrated that diffusion policies could effectively generate denoised robot actions while capturing multi-modal behavior distributions. Subsequent scaling efforts, such as \cite{Team2024OctoAO}'s transformer-based diffusion policy pretrained on the Open X-Embodiment dataset, showed these approaches could generalize across diverse robotic platforms through large-scale learning.
Diffusion models have shown great success in generative computer vision \cite{Ho2020DenoisingDP, Song2020ScoreBasedGM}, leading to their adaptation in robotic policy learning. Pioneering works \cite{Chi2023DiffusionPV, Pearce2023ImitatingHB, Reuss2023GoalConditionedIL} demonstrated their ability to generate denoised robot actions and capture multi-modal behavior distributions. Scaling efforts, such as \cite{Team2024OctoAO}, showcased generalization across diverse robotic platforms with transformer-based diffusion policies pretrained on the Open X-Embodiment dataset. Models like MDT \cite{Reuss2024MultimodalDT} and RDT-1B \cite{Liu2024RDT1BAD} use transformer-based diffusion models, replacing the traditional U-Net. RDT-1B further unifies action representations across robots and incorporates multi-robot data for bimanual manipulation.
% Prior work on diffusion‐based policies has also shown that denoising diffusion can elegantly model multimodal action distributions in high‐dimensional spaces \cite{Chi2023DiffusionPV, wang2024diffusion, jain2024sampling}. \cite{li2024learning} further augments diffusion policies with off‐the‐shelf unsupervised clustering and novelty‐based intrinsic rewards to discover and maintain multiple behavior modes, using mode‐specific Q‐learning to ensure balanced improvement across clusters. \cite{zhang2024entropy} shows adding an entropy‐style regularizer (analogous to a “soft cluster” prior) enhances policy robustness, hinting at the benefit of integrating more structured clustering losses. Inspired by these advances, we propose to inject SPCL’s\cite{li2020prototypical, mo2022siamese} prototypical contrastive loss, leveraging task category labels as supervised prototypes—into the diffusion policy training loop, thereby sharpening policy representations along known task dimensions and enhancing multi‐task imitation performance. Our method can be framed as "in-context learning", where human demonstrations serve as the contextual basis for the policy to achieve zero-shot transfer of unseen robotic skills.
Diffusion-based policies can model multimodal action distributions in high-dimensional spaces \cite{Chi2023DiffusionPV, wang2024diffusion, jain2024sampling}. \cite{li2024learning} augments these models with unsupervised clustering and intrinsic rewards to maintain multiple behavior modes, while \cite{zhang2024entropy} adds an entropy regularizer to enhance robustness. Building on these approaches, we propose injecting SPCL’s \cite{li2020prototypical, mo2022siamese} prototypical contrastive loss into diffusion policy training, leveraging task category labels as supervised prototypes. Thereby sharpening policy representations along known task dimensions and enhancing multi‐task imitation performance. Our method can be framed as "in-context learning", where human demonstrations serve as the contextual basis for the policy to achieve zero-shot transfer of unseen robotic skills.
In this section, we detail our framework by outlining its two-stage training process. Our goal is to enable the agent to learn and extract meaningful features and representations from human demonstration videos to perform specific tasks. The algorithm is divided into two stages. In the first stage, we develop a robust representation from a large collection of human data, robotic gripper data, and a small amount of dexterous hand data. This representation captures the target modality, the intended task, and the scene context. In the second phase, using this solid representation, we learn human manipulation patterns from abundant human data and a limited set of high-quality dexterous hand operation data, transferring these patterns to the dexterous hand to achieve better generalization with only a small amount of valuable dexterous hand data.
\subsection{Augment \textbf{V}ideo \textbf{G}eneration by \textbf{C}ross \textbf{P}rediction(VGCP)}
\label{sec:CrossPrediction}
Recent advancements in video generation models have leveraged extensive online video datasets containing prior knowledge about physical world dynamics. However, these models lack data relevant to robotic manipulation. To address this, we fine-tune an existing model with a custom dataset focused on robotic and object manipulation. Our goal is to enable agents to perform tasks based on human prompt videos, particularly complex dexterous hand operations. Therefore, the video generation model is trained to include details on human manipulation, object motion, scene context, and affordances. Our dataset consists mainly of human manipulation videos, supplemented by robotic gripper and self-collected dexterous hand videos. While gripper videos differ from dexterous hand operations, they provide valuable information on object movement and scene understanding, enhancing the model’s performance.
% Recent advances have focused on training general video generation models using extensive online video datasets that embed abundant prior knowledge about the dynamics of the physical world. However, these models lack sufficient training data and knowledge relevant to robotic manipulation. Therefore, we fine-tune an existing video generation model using a specially designed dataset to incorporate more information on robotic and object manipulation. Since our goal is to enable an agent to perform tasks based on human prompt videos, especially complex dexterous hand operations. We require the video generation model to include additional details on human manipulation, object motion, scene context, and object affordances. Thus, our dataset primarily consists of a large number of human manipulation videos, a moderate amount of robotic gripper videos, and a few self-collected dexterous hand videos. Although gripper videos differ from dexterous hand operations a lot, they contain rich information on object movement and scene understanding, such as object affordances, making them a valuable addition.
\textbf{Cross-Prediction:} To further leverage our data and enhance the quality of the learned representation, we propose a method called cross-prediction. As shown in Fig.\ref{fig:stage1}, our video generation model receives a video prompt showing a task performed by a \textbf{source embodiment} and an initial scene for a \textbf{target embodiment}, then generates a video of the target embodiment performing the task, effectively transferring the embodiment. For example, given a prompt video of a human grasping a cup, the model produces a video of a dexterous hand performing the same action. During training, we choose cross-prediction with probability $P$ (using different \textbf{source} and \textbf{target} embodiments) or normal-prediction with probability $1–P$ (using the same embodiment). This approach embeds embodiment transfer information into the video generation model, further improving the transferability of the learned representation. Our cross-prediction method randomly selects different source (s) and target (t) embodiments with probability $P$, and identical ones with probability $1-P$. When the two embodiments are the same, the process mirrors typical video generation models that generate subsequent frames from an initial frame. Our goal is for the model to learn the modality transfer between human and robot, while preserving existing knowledge of their manipulation. We believe that this approach enables the model to capture a video prompt representation that encompasses the skills used by the source embodiment, the manipulated objects, and some environmental context. The fine-tuned model will then be frozen during the second stage of training.
\begin{figure*}[t]
\centering
\includegraphics[width=0.85\linewidth]{images/Stage_2v3.pdf}
\captionof{figure}{\textbf{Stage 2}: We use the diffusion model trained in the first stage to combine information from human prompt videos with the target embodiment, generating an informative representation. This representation, along with a shared action between human and robot and a prototypical contrastive loss, enables the diffusion policy to learn common task and skill information for both.}
\label{fig:stage2}
\end{figure*}
\subsection{Skill Learning by Human Video-Action Pair Boosting}
\label{sec:human-action}
% Training manipulation policies using videos and data collected from dexterous hands has emerged as a prominent approach in robotic manipulation. However, acquiring such dexterous hand operation data typically relies on teleoperation, which incurs substantial time and hardware costs. To mitigate this, we aim to reduce the reliance on robotic hand data while leveraging human hand demonstrations to enhance the dexterous hand’s manipulation capabilities.The scale of available human hand data is virtually unbounded. Beyond self-collected human hand demonstrations, we can source human manipulation videos from the internet. Regardless of the collection method, human hand data acquisition requires minimal time and infrastructure investment. Thus, in addition to teleoperated robotic hand data, we propose incorporating human hand demonstrations by processing them into a compatible format. These two data sources are then jointly trained using Imitation Learning(IL). This constitutes the core methodology of our Stage 2 algorithm.
Training manipulation policies with dexterous hand data has become a key approach in robotic manipulation. However, collecting such data through teleoperation is time-consuming and costly. To address this, we reduce reliance on robotic hand data by leveraging human hand demonstrations to enhance manipulation capabilities. The availability of human hand data is virtually unlimited, with videos easily sourced from the internet or self-collected, requiring minimal time and infrastructure. We propose incorporating human hand demonstrations into a compatible format alongside teleoperated robotic hand data, which are then jointly trained using Imitation Learning (IL). This forms the core methodology of our Stage 2 algorithm.
\textbf{Human Data Preprocessing: }The objective of human data processing is to establish correspondence between human demonstration data and robot motion data. Given that we only have third-person view RGB videos of human demonstrations, while the robot data contains both third-person view RGB videos and joint state information. The robot's end-effector state comprises two components: 6D Wrist pose (wrist position $\in$ $\mathbb{R}^3$ and wrist orientation $\in$ $\mathsf{SO}(3)$) and finger joint $\theta^n$. We employ the hand-tracking method WiLoR\cite{Potamias2024WiLoRE3} for hand localization in video frames and 3D hand mesh reconstruction, and the model outputs 21 keypoints $J_{h_i}\in\mathbb{R}^3$ for each frame. We are able to obtatin wrist 6D from these 21 keypoint, with detaied methodology provide in Appendix.
\begin{wrapfigure}{r}{0.6\textwidth} % r 表示右侧，0.4\textwidth
\centering
\hspace*{-1mm}
\includegraphics[width=1.00\linewidth]{images/video_gen_v2.pdf}
\vspace*{-5mm}
\captionof{figure}{Examples of Cross-Prediction Video Generation.}
\label{fig:video gen}
\end{wrapfigure}
And for the wrist position, we can compute the 3D position of the human hand in the camera coordinate system using the camera intrinsic parameters. To align this with the robot's base coordinate system, we transform all coordinate systems to the camera coordinate. To establish joint-level correspondence between human and robotic systems, hand motion retargeting is required to map the kinematic configurations from the human demonstrator's hand to the target robot manipulator. We follow Anyteleop\cite{Qin2023AnyTeleopAG} to formulate the retargeting problem as an optimization.
\textbf{Representation Conditioned Diffusion Policy: }Our framework employs a diffusion policy that models the conditional action distribution $p(a_t|s_t,z)$, where: $s_t = f_{resnet34}(o_t)$ $\in$ $\mathbb{R}^{1000}$ is the visual observation feature extracted by a pretrained ResNet-34, $z$ $\in$ $\mathbb{R}^{4096}$ is the stage-one representation, $a_t$ $\in$ $\mathbb{R}^{19}$ is the action vector at time t. The diffusion process operates in the action space through:
\begin{equation}
a_{t}^{i} = \sqrt{\alpha_i}a_0 + \sqrt{1 - \alpha_i}\epsilon, \epsilon \sim \mathcal{N}(0, \mathbf{I})
\end{equation}
while the reverse process learns to predict and remove the noise through a transformer-based denoising network $\epsilon_{\theta}$.
For dexterous hand control tasks, we decompose the action space into three distinct components: finger joint angles $a_{t}^{finger}\in\mathbb{R}^{12}$, wrist orientation $a_t^{rot}\in\mathsf{SO}(3)$, and wrist position $a_t^{pos}\in\mathbb{R}^3$. This decomposition enables specialized handling of each action modality through separate prediction heads in the denoising network. The training loss combines weighted component losses:
\begin{equation}
\mathcal{L}_{action} = \lambda_f \mathcal{L}_{finger} + \lambda_r\mathcal{L}_{rotation}+\lambda_p\mathcal{L}_{postion}
\end{equation}
where:
\begin{equation}
\mathcal{L}_* = \mathbb{E}_{i,a_0,\epsilon}[||\epsilon_*-\epsilon_{\theta}(a_t^{i}, i, h_0)||^2]
\end{equation}
$h_0=[s_t,z]W_e+b_e$ is the concatenated input token, $W_e$ is the embedding matrix, $b_e$ is the bias. $\lambda_f,\lambda_r,\lambda_p$ are weighting coefficients determined through validation.
\subsection{\textbf{P}roto\textbf{D}iffusion \textbf{C}ontrastive \textbf{P}olicy(PDCP)}
Siamese Prototypical Contrastive Learning (SPCL)\cite{li2020prototypical, mo2022siamese} first groups feature embeddings into prototypes via K-Means and then applies a Siamese‐style metric loss to pull together embeddings within each prototype while pushing apart those from different prototypes, alongside a prototypical cross-entropy loss that treats prototype assignments as soft labels to sharpen each sample’s affinity to its cluster. By leveraging learned cluster structure rather than individual instances, SPCL mitigates false negatives and yields more stable positive sets, thereby enhancing semantic discriminability in self-supervised learning.
To adapt SPCL for diffusion policy training, we add a learnable clustering token to the input of our Diffusion Transformer (DiT) encoder and use the encoder’s output for that token—denoted $h$—as a prototype‐aware latent. During training we jointly optimize three losses:
(1) the NT-Xent contrastive loss $\mathcal{L}_{\mathrm{contra}}$, which treats tasks sharing the same skill as positives and tasks of different skills as negatives (analogous to data‐augmented positives in vision); (2) a prototypical cross-entropy loss $\mathcal{L}_{\mathrm{proto}}$, which encourages each the learned latent $\mathbf{h}$ to align with the cross-entropy distribution over its K-Means prototype label, promoting tighter intra-prototype clustering and clearer inter-prototype separation.
And (3) a Siamese-style metric loss $\mathcal{L}_{\mathrm{metric}}$ at the prototype level, which further enforces proximity among same-prototype samples in metric space while repelling different-prototype samples, thus reducing semantic confusion. Please refer to the appendix for detailed information about these three losses.
The total loss for PDCP consist of the action loss and a weighted sum of the mentioned loss with coefficient $w_p, w_p, w_m$:
\begin{equation}
\mathcal{L} = \mathcal{L}_{action} + w_c\mathcal{L}_{\mathrm{contra}} + w_p\mathcal{L}_{\mathrm{proto}} + w_m\mathcal{L}_{\mathrm{metric}}
\end{equation}
Together, these objectives guide the network to learn task-discriminative, skill-aware, and modality-agnostic representations which improving cross-modal alignment between human and robot demonstrations and ultimately enhancing the robustness and generalization of our diffusion policy.
In the next section, we conduct extensive experiments to validate our proposed method. We deploy our algorithm on the dexterous hand Xhand, performing engaging tasks such as pick-and-place, pressing, pouring water, flipping cups, push-pull operations, and even ball flicking. Our experiments aim to answer the following questions:
\begin{enumerate}[left=0pt]
\item Can cross-prediction enhance the performance, effectiveness, and representational capacity of the learned representation?
\item Is it feasible to use different video prompts in combination with a video generation model for skill extraction, and what is the resulting impact?
\item Does incorporating human action data improve the generalization ability of the diffusion policy?
\item Can PDCP boosting and outperforming the norm diffusion policy?
\item Can the final learned policy demonstrate “in-context learning” capabilities?
\end{enumerate}
% \begin{figure*}[!h]
% \centering
% \includegraphics[width=0.85\linewidth]{images/video gen v2.pdf}
% \captionof{figure}{Examples of Cross-Prediction Video Generation.}
% \label{fig:video gen}
% \end{figure*}
\subsection{Evaluation for Cross Prediction Video Generation}
Our cross-prediction mechanism extracts the task performed by the source embodiment from the prompt video and, using the scene context and target embodiment information form another given image, generates a video of the target embodiment executing that task.
As shown in Fig.\ref{fig:video gen}, we show two examples of same-embodiment prediction alongside with two examples of cross-embodiment prediction. The generated videos faithfully capture the task details from the prompt, and in the cross-embodiment cases, the generated target embodiment motion and scene context align precisely with the original task. This demonstrates that our model has learned not only the physical dynamics of the world but also learned how to transfer task and skill across different embodiments.
\subsection{Evaluation for the Learned Representations}
As shown in previous experiments, applying cross-prediction enables the video generation model to convert between human and robot embodiments while retaining its original capabilities. However, our goal is to extract the task information performed by the source embodiment from the prompt video using the cross-prediction fine-tuned SVD, and then transfer that information to the target embodiment. So, how do we evaluate and verify the performance of the representations extracted by the video generation model? We conducted the following experiments:
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\linewidth}
\centering
\includegraphics[width=\textwidth]{images/t-SNE.pdf}
\caption{t-SNE visualization of learned representations.}
\label{fig:t-SNE}
\end{subfigure}
% \hfill
\begin{subfigure}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{images/latent_validation.pdf}
\caption{Vector properties of learned representations.}
\label{fig:latent validation}
\end{subfigure}
\caption{Evaluation for the Learned Representations}
\label{fig:eval rep'}
\end{figure}
\subsubsection{\textbf{t-SNE Visualization of the Learned Representation}}
We conducted a t-SNE visualization experiment. We applied t-SNE to reduce the dimensionality of the skills extracted from the \textbf{human representation}(with a human video as the prompt and a human image as the initial frame) and the \textbf{robot representation} (with a human video as the prompt and a robot image as the initial frame), and then visualized them on a two-dimensional plane. The results, shown in Fig.\ref{fig:t-SNE}, reveal that tasks of the same category are grouped by similar color schemes, with identical hues indicating representations at different time points. Dashed lines highlight the positions of target objects in the two-dimensional space. The plot demonstrates that representations can effectively distinguish tasks, with the same task’s representations clustering closely based on embodiment, while still distinguishing target object positions. Additionally, representations from different time steps within the same video are tightly grouped yet distinguishable, reflecting the capture of continuous temporal information. These results show that the learned representations not only encode embodiment and task information but also capture object positioning and motion details, which is challenging to obtain with language prompts, showcasing the advantage of video prompts.
% As shown in Fig.\ref{fig:t-SNE}, similar color schemes are used for tasks of the same category, with identical hues representing representations at different time points. Dashed lines outline the positions on the two-dimensional visualization corresponding to target objects at various locations. The plot clearly shows that the learned representations can effectively distinguish different tasks. For the same task, representations with the same embodiment cluster more closely, yet still differentiate the positions of distinct target objects. Moreover, the representations corresponding to different time steps within the same video are tightly grouped while maintaining a degree of distinction, indicating that our representations capture continuous temporal information. Overall, these results demonstrate that the learned representations not only encompass embodiment and task-related information but also capture finer details of object positioning and motion details that is hard to obtain using language prompts, thus highlighting the advantage of video prompts.
\subsubsection{\textbf{Vector Properties of the Learned Representations}} In word embeddings, a fascinating phenomenon is observed: these embeddings exhibit an intrinsic vector structure. For example, taking the representation of “king,” subtracting that of “queen,” and then adding the representation of “nurse” yields that of “doctor.” This arithmetic property is celebrated as both elegant and intriguing. Thus, we aim to explore whether our learned representations possess a similar vectorial nature. We subtracted the human representation from the corresponding robot representation for the "push blue box" task and then added the human representation for the "grasp orange" task. Using this composite representation, we generated a video that, as shown in the Fig.\ref{fig:latent validation}, ultimately depicted a robot executing the "grasp orange" task. Moreover, we conducted tests on other tasks—such as "push button" and "pour water", and observed similarly intriguing results. This demonstrates that our learned representations indeed exhibit notable vector properties.
% \begin{center}
% \centering
% \hspace*{-1mm}
% \includegraphics[width=1.00\linewidth]{images/latent_validation.pdf}
% \vspace*{-5mm}
% \captionof{figure}{The Vector Properties of the Learned Representations.}
% \label{fig:latent validation}
% \end{center}%
\begin{table}
\centering
\caption{Success Rate (SR) and Score metrics on position, scene, and background generalization}
\label{tab:results}
\begin{tabular}{l*{3}{cc}}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{Position} & \multicolumn{2}{c}{Scene} & \multicolumn{2}{c}{Background} \\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
& SR & Score & SR & Score & SR & Score \\
\midrule
Language+R.+H. & 0.58 & 69.5 & 0.56 & 55.6 & 0.36 & 50.9 \\
\addlinespace[3pt]
Repr.+R. & 0.68 & 77.9 & 0.67 & 81.1 & 0.55 & 69.1 \\
\addlinespace[3pt]
Repr.+R.+H. & 0.74 & 81.6 & 0.67 & 84.4& 0.64 & 74.5 \\
\addlinespace[3pt]
\makecell{\textbf{Repr.+R.+H.+DPCP} \\ \textbf{(Ours)}} & \textbf{0.79} & \textbf{85.3} & \textbf{0.69} & \textbf{88.9} & \textbf{0.73} & \textbf{80.9}\\
\bottomrule
\end{tabular}
\end{table}
\subsection{Real-World Robot Experiments}
% We conduct real-world experiments using an xArm7 robotic arm equipped with a dexterous hand as the hardware platform. The system is controlled via 6D wrist pose (position + orientation) for arm movements and 12D finger joint angles for dexterous hand articulation.
Our policy relies solely on third-person RGB images from a fixed camera, and we evaluate our method on a multi-task benchmark(See Appendix). We conduct comparisons with three baselines: (1)\textbf{Representation+Robot} means Using only robotic data without human data. (2)\textbf{Language+Robot+Human }Means replacing skill representations with CLIP-encoded task labels (e.g., "pour water") as policy inputs. (3)\textbf{Representation+Robot+Human }means using both robot data and human data
\subsubsection{\textbf{Position, Scene, Background Generalization}}
To assess the generalization capabilities of our policy, we conducted comprehensive testing across three critical dimensions. \textbf{Positional Generalization} means randomly relocating target objects within the workspace. \textbf{Scene Generalization} means randomly changing or replacing inoperable objects within the workspace. \textbf{Background Generalization} means testing under different background (tablecloth, ...) To evaluate task execution performance, we introduce two metrics: Success Rate (SR) and Task Score. The binary SR metric (1 for complete success, 0 otherwise) and a subtask metric (details provided in Appendix).
As evidenced by the comparative results in Table \ref{tab:results}, our method demonstrates better performance across all three generalization scenarios when evaluated against the three baseline approaches. The comparative evaluation between our Stage-1 representation-guided policy and language-conditioned alternatives reveals fundamental advantages in the learned representations' capacity to encode richer task-relevant information.
% \begin{wrapfigure}{r}{0.45\textwidth} % r 表示右侧，0.4\textwidth
% \centering
% \hspace*{-1mm}
% \includegraphics[width=1.00\linewidth]{images/new obj.pdf}
% \vspace*{-5mm}
% \captionof{figure}{Evaluation on Obj Generalization}
% \label{fig:algorithm_performance}
% \end{wrapfigure}
Our analysis further reveals that incorporating human demonstration data consistently enhances policy performance across all three generalization scenarios. The performance improvements observed suggest that high-quality human demonstrations provide critical behavioral patterns and manipulation strategies that are often underrepresented in robot-collected data. Table \ref{tab:results} clearly demonstrates the effectiveness of the proposed PDCP algorithm in mitigating mis-grasping issues when handling closely spaced objects. We posit that while the Stage-1 representation primarily focuses on spatial variations, the PDCP module provides complementary functionality by enabling task-aware feature clustering.
\vspace{-5pt}
\subsubsection{Object Generalization}
To investigate the complementary benefits of our representation learning framework and human demonstration data, we design object-level generalization experiments to evaluate their individual and combined effects. While our representations perform well across all generalization scenarios, these experiments isolate and quantify the contribution of human data with novel object instances.
As shown in Figure \ref{fig:algorithm_performance}, we test four distinct tasks involving unseen objects. For each task, we compare two training conditions: (1) using robot-collected data with similar, non-identical objects (excluding target objects), and (2) combining robot data with a larger set of human demonstrations that include the target objects. The results clearly show that human data significantly improves manipulation success rates for novel objects, highlighting its ability to transfer knowledge on object affordances and manipulation strategies.
Additionally, our representation outperforms direct language input, thanks to its incorporation of end-effector positional information. We argue that for tasks absent from the robot training set, our representation provides essential spatial cues and partial operational knowledge, while the human data offers complementary information on novel objects and manipulation strategies.
% To further investigate the complementary benefits of our representation learning framework and human demonstration data, we specifically design object-level generalization experiments that systematically evaluate their individual and combined effects. While our representations have demonstrated effectiveness across all three generalization scenarios, these controlled experiments aim to isolate and quantify the specific contribution of human data when facing novel object instances.
% As shown in Figure \ref{fig:algorithm_performance}, we evaluate this capability through four distinct tasks(Unseen Object). For each task, we compare two training conditions: (1) using only robot-collected data containing similar but non-identical objects (strictly excluding target objects),
% versus (2) combining this robot data with a larger set of human demonstrations that include the target objects. The results clearly demonstrate that incorporating human data significantly improves manipulation success rates for these novel objects. This suggests that human demonstrations provide transferable knowledge about object affordances and manipulation strategies that can effectively compensate for the absence of specific objects in the robot training set.
% Furthermore, our representation demonstrates superior performance compared to direct language input, which we attribute crucially to its incorporation of end-effector positional information. We posit that for tasks completely absent from the robot training dataset, our representation provides essential spatial cues and partial operational knowledge, while the human data contributes complementary information about novel objects and their manipulation strategies.
% \begin{wrapfigure}{r}{0.45\textwidth} % r 表示右侧，0.4\textwidth
% \centering
% \hspace*{-1mm}
% \includegraphics[width=1.00\linewidth]{images/new skill.pdf}
% \vspace*{-5mm}
% \captionof{figure}{\textbf{Unseen Skill transfers}.}
% \label{fig:new_skill}
% \end{wrapfigure}
\begin{figure}[t]
\centering
\begin{subfigure}[b]{0.48\linewidth}
\centering
\includegraphics[width=\textwidth]{images/new_skill.pdf}
\caption{Unseen Skill transfers}
\label{fig:new_skill}
\end{subfigure}
% \hfill
\begin{subfigure}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{images/new_obj.pdf}
\caption{Evaluation on Obj Generalization}
\label{fig:algorithm_performance}
\end{subfigure}
% \caption{Evaluation for the Learned Representations}
% \label{fig:eval rep'}
\end{figure}
% \begin{center}
% \centering
% \hspace*{-1mm}
% \includegraphics[width=1.00\linewidth]{images/new obj.pdf}
% \vspace*{-5mm}
% \captionof{figure}{\textbf{Evaluation on Object Generalization: }We evaluate the generalization capability of our algorithm against three baseline methods across four novel object manipulation tasks}
% \label{fig:algorithm_performance}
% \end{center}%
\subsection{Skill Generalization}
Given that our policy can handle tasks unseen in the robot training dataset, we investigate whether it can also demonstrate novel skills absent from the original robotic data. To test this hypothesis, we trained three distinct policies on three novel tasks: grasping an orange juice bottle, pressing a button, and pushing a red block. Each policy was trained using approximately 50 robotic trajectories of diverse grasping tasks combined with extensive human demonstration data. For comparison, we implemented baseline models trained solely on either (1) the same 50 robotic trajectories or (2) language-augmented robot and human data. Crucially, these baseline models failed to generalize to new skills at the skill level, whereas our approach combining representation learning with joint human-robot data demonstrated clear potential for novel skill generalization.
As illustrated in Figure \ref{fig:new_skill}, the left panel displays human hand manipulation data, while the right panel shows the robot executing corresponding actions that were never present in its original training dataset. This comparative analysis reveals that incorporating human data significantly enhances the policy's ability to acquire and transfer novel skills to the robotic domain.
This work fully leverages human data throughout both the representation generation and diffusion policy processing stages. By utilizing human videos as prompts—which provide substantially richer information than language inputs—our approach demonstrates superior performance in dexterous hand manipulation tasks.
Our work faces inherent challenges in generalizing new skills from human videos. While we demonstrate successful cases where the robot replicates human manipulations to complete tasks (occurring when the policy successfully interprets and follows human data), many attempts fail due to misalignment between the policy's execution and the observational prerequisites for skill activation. Currently, our new skill acquisition success rate ranges under 10\%.
Although this remains a significant challenge, the human-demonstrated novel skills we introduced show measurable potential for training corresponding robotic capabilities. Importantly, we hypothesize that substantially increasing human demonstration data for specific tasks - similar to the scaling approach used with robot data - could help mitigate this issue.
\appendix
\section{Appendix}
% \begin{appendices}
\subsection{Details on SVD formulation}
More specifically, we selected the open-source stable video diffusion (SVD) model with 1.5 billion parameters as the base model. The original SVD model conditions on the initial-frame image $I_0$, we augment it by adding a CLIP image encoder $\phi(\cdot)$ to extract features from the prompt video and integrate them into the SVD UNet block via cross-attention. To enable longer video outputs, reduce training costs, and improve efficiency, we set the model to generate videos with $T=16$ frames at a resolution of 256×256. We denote our model as $G_\theta$, with inputs comprising an initial image $I^t_0$ of the target embodiment and a video $V^s=I^t_{0:T}$ of a source embodiment performing the specified task. We fine-tune the base SVD model with a diffusion objective which generates video from noised samples $x_t = \sqrt{\bar{a_t}}x_0 + \sqrt{1 - \bar{a_t}}\epsilon$, where $x_0 = V^t$ and $D$ is our dataset.
\begin{equation}
\mathcal{L}_{CP} = \mathbb{E}_{x_o \sim D,t,\epsilon }||G_\theta(x_t, \phi(V^s), I^t_0) - x_0||^2
\end{equation}
\textbf{Dataset for Training Video Generation by Cross Prediction:}
The dataset for video generation model training comprises three parts: (1) videos of human object manipulation, (2) videos of robotic gripper manipulation, and (3) videos of dexterous hand manipulation that we collected ourselves. The human manipulation videos are sourced from HOI4D, RH20T, and our own collected footage, with the aim of teaching the model how humans handle objects. The robotic gripper videos, primarily from BRIGE and RH20T, enable the model to learn the physical dynamics of object motion and acquire the ability to transfer from human to gripper embodiments. Finally, a small set of high-quality dexterous hand videos is included to capture authentic dexterous manipulation and to facilitate the transfer between human and dexterous hand operations. The video generation model training process takes 5 days on 8 NVIDIA A100 GPUs. The second stage of policy learning requires approximately 6-12 hours on four NVIDIA A100 GPUs.
\subsection{Loss Details for ProtoDiffusion Contrastive Policy(PDCP)}
\textbf{(1) The NT-Xent contrastive loss}, which treats tasks sharing the same skill as positives and tasks of different skills as negatives (analogous to data‐augmented positives in vision);
\begin{equation}
\mathcal{L}_{\mathrm{contra}} = -\frac{1}{|B|}\sum_{i\in B} \frac{1}{|P(i)|}
\sum_{p\in P(i)} \log \frac {\exp\bigl(\mathrm{sim}(z_i, z_p)/\tau\bigr)} {\sum\limits_{a\in B\setminus\{i\}}
\exp\bigl(\mathrm{sim}(z_i, z_a)/\tau\bigr)}
\end{equation}
where $P(i)=\{j\in B|y_j=y_i, j\neq i\}$ indicate samples which have the same skill with $i$. $z_i=g_c(h_i)$ where $g_c(\cdot)$ is a projection head, and $\mathrm{sim}(u,v)=\frac{u^Tv}{||u||\,||c||}$ is the similairty metric.
\textbf{(2) A prototypical cross-entropy loss}, which encourages each the learned latent $\mathbf{h}$ to align with the cross-entropy distribution over its K-Means prototype label, promoting tighter intra-prototype clustering and clearer inter-prototype separation.
\begin{equation}
\mathcal{L}_{\mathrm{proto}}
= -\frac{1}{|B|}\sum_{i \in B}
\log
\frac{
\exp\!\bigl(\mathrm{sim}(h_i,\,\mu_{c(i)})/\tau\bigr)
}{
\displaystyle\sum_{j=1}^{K}
\exp\!\bigl(\mathrm{sim}(h_i,\,\mu_j)/\tau\bigr)
}
\end{equation}
where $\mu_j$ is the $j$-th prototypes(cluster center); $c_i$ is the cluster index for sample $i$ and $\tau$ is the temperature parameter.
\textbf{(3) A Siamese-style metric loss} at the prototype level, which further enforces proximity among same-prototype samples in metric space while repelling different-prototype samples, thus reducing semantic confusion.
\begin{equation}
\mathcal{L}_{\mathrm{metric}}
= \frac{1}{|\mathcal{P}|}
\sum_{(i,j)\in\mathcal{P}}
\text{CE}\left(g_m(\|h_i - h_j\|), 1\right)
+ \frac{1}{|\mathcal{N}|}
\sum_{(i,j)\in\mathcal{N}}
\text{CE}\left(g_m(\|h_i - h_j\|), 0\right)
\end{equation}
where $\mathcal{P}$ indicates all the sample pairs in the cluster, while $\mathcal{N}$ indicates sample pairs in different cluster and $g_m(\cdot)$ is also a projection head.
\subsection{Details of Human Data Preprocessing}
To extract the 6D wrist pose (position + orientation) from 21 keypoints, we first obtain the 3D position directly from the wrist keypoint $\textbf{W}\in\mathbb{R}^3$. For the 3D orientation, we construct a local coordinate frame using the wrist keypoint $\textbf{W}$, the first metacarpophalangeal joint of the index finger $\textbf{I}$, and the first metacarpophalangeal joint of the middle finger $\textbf{I}$, The x-axis unit vector $\textbf{x}$ is defined as the normalized direction from $\textbf{W}$ to $\textbf{M}$:
\begin{equation}
\textbf{x} = \frac{\textbf{M} - \textbf{W}}{||\textbf{M}-\textbf{W}||}
\end{equation}
The z-axis unit vector $\textbf{z}$ is computed as the normalized normal vector of the plane formed by $\textbf{W}$, $\textbf{I}$ and $\textbf{M}$, oriented toward the palm:
\begin{equation}
\textbf{z} = \frac{(\textbf{I} - \textbf{W}) \times (\textbf{M} - \textbf{W})}{||(\textbf{I} - \textbf{W}) \times (\textbf{M} - \textbf{W})||}
\end{equation}
Finally, the y-axis $\textbf{y}$ is derived via the right-hand rule as the cross product of $\textbf{z}$ and $\textbf{x}$:
\begin{equation}
\textbf{y} = \textbf{z}\times\textbf{x}
\end{equation}
The resulting rotation matrix $\textbf{R}=[\textbf{x}\,\textbf{y}\,\textbf{z}]$ and wrist position $\textbf{W}$ form the complete 6D pose representation. This method ensures orthonormality and a right-handed coordinate system.
\subsection{Integrated Normalization Methodology for Robot and Human Datasets in Stage 2}
In Stage 2, we normalize the actions of \textbf{both the human hand and the robot dexterous hand} to the range $[0,1]$ using a joint normalization approach. Specifically, for the 19-dimensional action space, we first compute the \textbf{global maximum} ($max_a$) and \textbf{global minimum} ($min_a$) for each dimension across the combined dataset of human and robotic actions. Each action component $a_i$ (where $i\in\{1, 2, \dots, 19\}$) is then normalized as follows:
\begin{equation}
a_{norm, i} = \frac{a_i - min_a}{max_{a, i} - min_{a, i}}
\end{equation}
\subsection{Real-World Robot Experiments Setting}
\subsubsection{Hardware Settings}
In the physical experiment phase, we employed an xArm7 robotic manipulator coupled with a dexterous robotic hand as our hardware platform. The robotic system was controlled using two distinct control signals: (1) 6D wrist pose data for arm movement control, and (2) 12D finger articulation data for precise hand manipulation. Notably, our observation inputs were exclusively derived from third-person RGB camera images.
\subsubsection{Experiment Settings and Details}
We integrate the frameworks of Stage 1 and Stage 2 and evaluate our algorithm across various skills. The detailed dataset statistics are presented in Table \ref{tab:dataset_stats}. For each skill task, we collected paired robot and human demonstration videos (with one-to-one correspondence between robot and human videos) as Self-Collected Datasets. The table reports: (1) whether the dataset incorporate the Human/ Gripper/ Dexhand data, (2) number trajectories of dataset, (3) whether Stage1 and Stage 2 trained using the dataset.
% \begin{table}
% \centering
% \caption{Skill Task Statistics and Dataset Composition}
% \label{tab:dataset_stats}
% \begin{tabular}{lccc}
% \toprule
% Skill Category & \# Video Pairs & Robot Frames & Human Frames\\
% \midrule
% \textbf{Pick-and-Place} & 840 & 72842 & 68610 \\
% \textbf{Push Box} & 160 & 15678 & 13732 \\
% \textbf{Pour water} & 288 & 31938 & 29399 \\
% \textbf{Flip cups} & 120 & 10920 & 10312 \\
% \textbf{Press Buttons} & 153 & 11481 & 11471 \\
% \midrule
% \textbf{Total} & 1561 & 142859 & 133524\\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{table}
\centering
\caption{Dataset Composition}
\label{tab:dataset_stats}
\begin{tabular}{lcccccc}
\toprule
Datasets & Human & Gripper & Dexhand & Number Trajectories & Stage 1 & Stage 2\\
\midrule
\textbf{RH20T} & \checkmark & \checkmark & $\times$ & 87529 & \checkmark & $\times$\\
\textbf{HOI4D} & \checkmark & $\times$ & $\times$ & 2972 & \checkmark & $\checkmark$\\
\textbf{Bridge} & $\times$ & \checkmark & $\times$ & 25460 & \checkmark & $\times$\\
\textbf{Self-Collected} & \checkmark & $\times$ & \checkmark & 3679 & \checkmark & \checkmark\\
\bottomrule
\end{tabular}
\end{table}
To evaluate the performance of our tasks, we have defined two metrics: \textbf{Success Rate (SR)} and \textbf{Score}. The Success Rate (SR) is a binary indicator that signifies the completion status of a task, with a value of 1 representing successful task completion and 0 indicating failure. However, this metric alone is insufficient to fully capture the nuances of task performance. Therefore, we introduce the Score metric to assess the completion status of multi-stage tasks. For the Pick-and-Place task, approaching the object and initiating a picking action yields 30 points, successfully picking the object awards 40 points, and placing the object at the designated location results in an additional 30 points. In the Pour water task, approaching the bottle and initiating a grasping action earns 30 points, grasping the bottle gains 40 points, and accurately pouring water into the target cup adds another 30 points. For the Flip Cups task, approaching the cup and initiating a grasping action scores 30 points, performing the flipping action awards 30 points, and successfully standing the cup upright on the table grants 40 points. In the Push Box task, approaching the block and initiating a pushing action yields 60 points, while successfully moving the block forward adds 40 points. For the Press Buttons task, approaching the object and initiating a pressing action earns 30 points, and completing the task awards 70 points.
\vspace{5pt}
\textbf{Experiment Settings of Position, Scene and Background Generalization}
In the generalization experiments on position, scene, and background in Section 4.3.1, we utilized the entire dataset and tested all skills to comprehensively demonstrate the performance improvement of our policy following the introduction of human datasets and representation learning.
\vspace{5pt}
\textbf{Experiment Settings of Object Generalization}
In the novel object generalization experiments in Section 4.3.2, the following four tasks were showcased: picking up a capybara and placing it on a plate, picking up dice and placing it on a plate, pouring orange juice, and pushing a red box. For training the Pick-and-Place task, our robot dataset comprised approximately 300 grasping tasks (excluding picking capybara and dice), while the Human dataset included around 700 tasks (including picking capybara and dice). The Pour Orange Juice task was trained using roughly 150 Robot data entries (excluding pouring orange juice) and about 300 Human data entries (including pouring orange juice). For training the Push Red Block task, we employed approximately 120 Robot data entries related to pushing actions (excluding pushing the red block) and around 250 Human data entries related to pushing actions (including the red block).
\vspace{5pt}
\subsection{Skill Generalization}
In the novel skill generalization experiments in Section 4.3.3, skill generalization refers to the capability of a robotic system to acquire and execute skills that are \textbf{NOT present in the original robot training dataset} but can be learned by leveraging \textbf{human demonstration data}. To validate this capability, we construct novel skills by partitioning our self-collected dataset. For instance, we introduce a previously unseen skill P (Push)—while the robot training dataset contains no examples of P, the human dataset includes demonstrations of this skill P. Through our proposed algorithm, the robot is trained on this combined data and subsequently demonstrates the ability to successfully perform P (Push) in deployment.
For our experiments in Section 4.3.3, three new skills were introduced by controlling the selection of datasets: pushing a box, holding a bottle, and pressing a button. The Push Block task was trained using 40 Robot data entries related to pick-and-place tasks and 240 Human data entries related to pushing a box. The Hold Cup task utilized 40 Robot data entries related to grasping and 300 human data entries related to holding cups. The Press Button task was trained with 40 robot data entries related to grasping and 160 human data entries related to pressing actions.
\subsection{Training Details}
We list the hyperparameters for \textbf{Stage 1 of Our Algorithm} in Table \ref{tab:hyper_1} \textbf{Stage2 of Our Algorithm} in Table \ref{tab:hyper_2}. Stage 2 model is trained for 20 epoches with global batch size of 128 across 1 A800 gpu.
% \begin{table}
% \centering
% \caption{Hyperparameters}
% \label{tab:hyper}
% \begin{tabular}{lc}
% \toprule
% Batch Size & 128 \\
% Optimizer & adamw \\
% Learning Rate & 1e-4 \\
% Weight Decay & 1e-6 \\
% observation horizon & 2 \\
% action horizon & 8 \\
% \textbf{Resnet34} & \\
% Size of Input Image & [160, 160, 3]\\
% \textbf{DiT} & \\
% embed dim & 512 \\
% \# decoder layers & 6 \\
% \# encoder layers & 4 \\
% No. of heads & 8 \\
% goal drop & 0 \\
% embed pdrop & 0 \\
% attn pdrop & 0.3 \\
% resid pdrop & 0.1 \\
% mlp pdrop & 0.05 \\
% sigma data & 0.5 \\
% sigma min & 0.001 \\
% sigma max & 80 \\
% \textbf{PDCP} & \\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{table}[htbp]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\begin{tabular}{lc}
\toprule
Batch Size & 64 \\
Optimizer & adamw \\
Learning Rate & 5e-6 \\
Cross Prediction Prob & 0.6 \\
inference step & 50 \\
video frame & 16 \\
\bottomrule
\end{tabular}
\caption{Hyperparameters of Stage 1}
\label{tab:hyper_1}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\begin{tabular}{lc}
\toprule
Batch Size & 128 \\
Optimizer & adamw \\
Learning Rate & 1e-4 \\
Weight Decay & 1e-6 \\
observation horizon & 2 \\
action horizon & 8 \\
\midrule
\textbf{Resnet34} & \\
Size of Input Image & [160, 160, 3]\\
\midrule
\textbf{DiT} & \\
embed dim & 512 \\
\# decoder layers & 6 \\
\# encoder layers & 4 \\
No. of heads & 8 \\
goal drop & 0 \\
embed pdrop & 0 \\
attn pdrop & 0.3 \\
resid pdrop & 0.1 \\
mlp pdrop & 0.05 \\
sigma data & 0.5 \\
sigma min & 0.001 \\
sigma max & 80.0 \\
$\lambda_f$ & 12/19 \\
$\lambda_r$ & 4/19 \\
$\lambda_p$ & 3/19 \\
\textbf{PDCP} & \\
$\omega_c$ & 1.0 \\
$\omega_p$ & 1.0 \\
$\omega_m$ & 1.0 \\
\bottomrule
\end{tabular}
\caption{Hyperparameters of Stage 2}
\label{tab:hyper_2}
\end{minipage}
\end{table}
% \end{appendices}
\begin{thebibliography}{41}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Brohan et~al.(2022)Brohan, Brown, Carbajal, Chebotar, Dabis, Finn, Gopalakrishnan, Hausman, Herzog, Hsu, Ibarz, Ichter, Irpan, Jackson, Jesmonth, Joshi, Julian, Kalashnikov, Kuang, Leal, Lee, Levine, Lu, Malla, Manjunath, Mordatch, Nachum, Parada, Peralta, Perez, Pertsch, Quiambao, Rao, Ryoo, Salazar, Sanketi, Sayed, Singh, Sontakke, Stone, Tan, Tran, Vanhoucke, Vega, Vuong, Xia, Xiao, Xu, Xu, Yu, and Zitkovich]{Brohan2022RT1RT}
A.~Brohan, N.~Brown, J.~Carbajal, Y.~Chebotar, J.~Dabis, C.~Finn, K.~Gopalakrishnan, K.~Hausman, A.~Herzog, J.~Hsu, J.~Ibarz, B.~Ichter, A.~Irpan, T.~Jackson, S.~Jesmonth, N.~J. Joshi, R.~C. Julian, D.~Kalashnikov, Y.~Kuang, I.~Leal, K.-H. Lee, S.~Levine, Y.~Lu, U.~Malla, D.~Manjunath, I.~Mordatch, O.~Nachum, C.~Parada, J.~Peralta, E.~Perez, K.~Pertsch, J.~Quiambao, K.~Rao, M.~S. Ryoo, G.~Salazar, P.~R. Sanketi, K.~Sayed, J.~Singh, S.~A. Sontakke, A.~Stone, C.~Tan, H.~Tran, V.~Vanhoucke, S.~Vega, Q.~H. Vuong, F.~Xia, T.~Xiao, P.~Xu, S.~Xu, T.~Yu, and B.~Zitkovich.
\newblock Rt-1: Robotics transformer for real-world control at scale.
\newblock \emph{ArXiv}, abs/2212.06817, 2022.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:254591260}.
\bibitem[Brohan et~al.(2023)Brohan, Brown, Carbajal, Chebotar, Choromanski, Ding, Driess, Dubey, Finn, Florence, Fu, Arenas, Gopalakrishnan, Han, Hausman, Herzog, Hsu, Ichter, Irpan, Joshi, Julian, Kalashnikov, Kuang, Leal, Levine, Michalewski, Mordatch, Pertsch, Rao, Reymann, Ryoo, Salazar, Sanketi, Sermanet, Singh, Singh, Soricut, Tran, Vanhoucke, Vuong, Wahid, Welker, Wohlhart, Xiao, Yu, and Zitkovich]{Brohan2023RT2VM}
A.~Brohan, N.~Brown, J.~Carbajal, Y.~Chebotar, K.~Choromanski, T.~Ding, D.~Driess, K.~A. Dubey, C.~Finn, P.~R. Florence, C.~Fu, M.~G. Arenas, K.~Gopalakrishnan, K.~Han, K.~Hausman, A.~Herzog, J.~Hsu, B.~Ichter, A.~Irpan, N.~J. Joshi, R.~C. Julian, D.~Kalashnikov, Y.~Kuang, I.~Leal, S.~Levine, H.~Michalewski, I.~Mordatch, K.~Pertsch, K.~Rao, K.~Reymann, M.~S. Ryoo, G.~Salazar, P.~R. Sanketi, P.~Sermanet, J.~Singh, A.~Singh, R.~Soricut, H.~Tran, V.~Vanhoucke, Q.~H. Vuong, A.~Wahid, S.~Welker, P.~Wohlhart, T.~Xiao, T.~Yu, and B.~Zitkovich.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control.
\newblock \emph{ArXiv}, abs/2307.15818, 2023.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:260293142}.
\bibitem[Kareer et~al.(2024)Kareer, Patel, Punamiya, Mathur, Cheng, Wang, Hoffman, and Xu]{kareer2024egomimic}
S.~Kareer, D.~Patel, R.~Punamiya, P.~Mathur, S.~Cheng, C.~Wang, J.~Hoffman, and D.~Xu.
\newblock Egomimic: Scaling imitation learning via egocentric video.
\newblock \emph{arXiv preprint arXiv:2410.24221}, 2024.
\bibitem[Chen et~al.(2024)Chen, Chen, Arlaud, Laptev, and Schmid]{chen2024vividex}
Z.~Chen, S.~Chen, E.~Arlaud, I.~Laptev, and C.~Schmid.
\newblock Vividex: Learning vision-based dexterous manipulation from human videos.
\newblock \emph{arXiv preprint arXiv:2404.15709}, 2024.
\bibitem[Qin et~al.(2022)Qin, Wu, Liu, Jiang, Yang, Fu, and Wang]{qin2022dexmv}
Y.~Qin, Y.-H. Wu, S.~Liu, H.~Jiang, R.~Yang, Y.~Fu, and X.~Wang.
\newblock Dexmv: Imitation learning for dexterous manipulation from human videos.
\newblock In \emph{European Conference on Computer Vision}, pages 570--587. Springer, 2022.
\bibitem[Wang et~al.(2024)Wang, Shi, Wang, Zhang, Fei-Fei, and Liu]{wang2024dexcap}
C.~Wang, H.~Shi, W.~Wang, R.~Zhang, L.~Fei-Fei, and C.~K. Liu.
\newblock Dexcap: Scalable and portable mocap data collection system for dexterous manipulation.
\newblock \emph{arXiv preprint arXiv:2403.07788}, 2024.
\bibitem[Kannan et~al.(2023)Kannan, Shaw, Bahl, Mannam, and Pathak]{kannan2023deft}
A.~Kannan, K.~Shaw, S.~Bahl, P.~Mannam, and D.~Pathak.
\newblock Deft: Dexterous fine-tuning for real-world hand policies.
\newblock \emph{arXiv preprint arXiv:2310.19797}, 2023.
\bibitem[Jain et~al.(2024)Jain, Attarian, Joshi, Wahid, Driess, Vuong, Sanketi, Sermanet, Welker, Chan, et~al.]{jain2024vid2robot}
V.~Jain, M.~Attarian, N.~J. Joshi, A.~Wahid, D.~Driess, Q.~Vuong, P.~R. Sanketi, P.~Sermanet, S.~Welker, C.~Chan, et~al.
\newblock Vid2robot: End-to-end video-conditioned policy learning with cross-attention transformers.
\newblock \emph{arXiv preprint arXiv:2403.12943}, 2024.
\bibitem[Yuan et~al.(2024)Yuan, Wen, Zhang, and Gao]{yuan2024general}
C.~Yuan, C.~Wen, T.~Zhang, and Y.~Gao.
\newblock General flow as foundation affordance for scalable robot learning.
\newblock \emph{arXiv preprint arXiv:2401.11439}, 2024.
\bibitem[Ju et~al.(2024)Ju, Hu, Zhang, Zhang, Jiang, and Xu]{ju2024robo}
Y.~Ju, K.~Hu, G.~Zhang, G.~Zhang, M.~Jiang, and H.~Xu.
\newblock Robo-abc: Affordance generalization beyond categories via semantic correspondence for robot manipulation.
\newblock In \emph{European Conference on Computer Vision}, pages 222--239. Springer, 2024.
\bibitem[Xu et~al.(2023)Xu, Xu, Chi, Veloso, and Song]{xu2023xskill}
M.~Xu, Z.~Xu, C.~Chi, M.~Veloso, and S.~Song.
\newblock Xskill: Cross embodiment skill discovery.
\newblock In \emph{Conference on robot learning}, pages 3536--3555. PMLR, 2023.
\bibitem[Wang et~al.(2023)Wang, Fan, Sun, Zhang, Fei-Fei, Xu, Zhu, and Anandkumar]{wang2023mimicplay}
C.~Wang, L.~Fan, J.~Sun, R.~Zhang, L.~Fei-Fei, D.~Xu, Y.~Zhu, and A.~Anandkumar.
\newblock Mimicplay: Long-horizon imitation learning by watching human play.
\newblock \emph{arXiv preprint arXiv:2302.12422}, 2023.
\bibitem[Smith et~al.(2019)Smith, Dhawan, Zhang, Abbeel, and Levine]{smith2019avid}
L.~Smith, N.~Dhawan, M.~Zhang, P.~Abbeel, and S.~Levine.
\newblock Avid: Learning multi-stage tasks via pixel-level translation of human videos.
\newblock \emph{arXiv preprint arXiv:1912.04443}, 2019.
\bibitem[Xiong et~al.(2021)Xiong, Li, Chen, Bharadhwaj, Sinha, and Garg]{xiong2021learning}
H.~Xiong, Q.~Li, Y.-C. Chen, H.~Bharadhwaj, S.~Sinha, and A.~Garg.
\newblock Learning by watching: Physical imitation of manipulation skills from human videos.
\newblock In \emph{2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages 7827--7834. IEEE, 2021.
\bibitem[Wen et~al.(2023)Wen, Lin, So, Chen, Dou, Gao, and Abbeel]{wen2023any}
C.~Wen, X.~Lin, J.~So, K.~Chen, Q.~Dou, Y.~Gao, and P.~Abbeel.
\newblock Any-point trajectory modeling for policy learning.
\newblock \emph{arXiv preprint arXiv:2401.00025}, 2023.
\bibitem[Gu et~al.(2023)Gu, Kirmani, Wohlhart, Lu, Arenas, Rao, Yu, Fu, Gopalakrishnan, Xu, et~al.]{gu2023rt}
J.~Gu, S.~Kirmani, P.~Wohlhart, Y.~Lu, M.~G. Arenas, K.~Rao, W.~Yu, C.~Fu, K.~Gopalakrishnan, Z.~Xu, et~al.
\newblock Rt-trajectory: Robotic task generalization via hindsight trajectory sketches.
\newblock \emph{arXiv preprint arXiv:2311.01977}, 2023.
\bibitem[Ye et~al.(2024)Ye, Jang, Jeon, Joo, Yang, Peng, Mandlekar, Tan, Chao, Lin, et~al.]{ye2024latent}
S.~Ye, J.~Jang, B.~Jeon, S.~Joo, J.~Yang, B.~Peng, A.~Mandlekar, R.~Tan, Y.-W. Chao, B.~Y. Lin, et~al.
\newblock Latent action pretraining from videos.
\newblock \emph{arXiv preprint arXiv:2410.11758}, 2024.
\bibitem[Liang et~al.(2024)Liang, Liu, Ozguroglu, Sudhakar, Dave, Tokmakov, Song, and Vondrick]{liang2024dreamitate}
J.~Liang, R.~Liu, E.~Ozguroglu, S.~Sudhakar, A.~Dave, P.~Tokmakov, S.~Song, and C.~Vondrick.
\newblock Dreamitate: Real-world visuomotor policy learning via video generation.
\newblock \emph{arXiv preprint arXiv:2406.16862}, 2024.
\bibitem[Du et~al.(2023)Du, Yang, Dai, Dai, Nachum, Tenenbaum, Schuurmans, and Abbeel]{du2023learning}
Y.~Du, S.~Yang, B.~Dai, H.~Dai, O.~Nachum, J.~Tenenbaum, D.~Schuurmans, and P.~Abbeel.
\newblock Learning universal policies via text-guided video generation.
\newblock \emph{Advances in neural information processing systems}, 36:\penalty0 9156--9172, 2023.
\bibitem[Bharadhwaj et~al.(2024)Bharadhwaj, Dwibedi, Gupta, Tulsiani, Doersch, Xiao, Shah, Xia, Sadigh, and Kirmani]{bharadhwaj2024gen2act}
H.~Bharadhwaj, D.~Dwibedi, A.~Gupta, S.~Tulsiani, C.~Doersch, T.~Xiao, D.~Shah, F.~Xia, D.~Sadigh, and S.~Kirmani.
\newblock Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation.
\newblock \emph{arXiv preprint arXiv:2409.16283}, 2024.
\bibitem[Shridhar et~al.(2024)Shridhar, Lo, and James]{shridhar2024generative}
M.~Shridhar, Y.~L. Lo, and S.~James.
\newblock Generative image as action models.
\newblock \emph{arXiv preprint arXiv:2407.07875}, 2024.
\bibitem[Chang and Gupta(2023)]{chang2023one}
M.~Chang and S.~Gupta.
\newblock One-shot visual imitation via attributed waypoints and demonstration augmentation.
\newblock In \emph{2023 IEEE International Conference on Robotics and Automation (ICRA)}, pages 5055--5062. IEEE, 2023.
\bibitem[Xiong et~al.(2023)Xiong, Fu, Zhang, Bao, Zhang, Huang, Xu, Garg, and Lu]{xiong2023robotube}
H.~Xiong, H.~Fu, J.~Zhang, C.~Bao, Q.~Zhang, Y.~Huang, W.~Xu, A.~Garg, and C.~Lu.
\newblock Robotube: Learning household manipulation from human videos with simulated twin environments.
\newblock In \emph{Conference on Robot Learning}, pages 1--10. PMLR, 2023.
\bibitem[Chane-Sane et~al.(2023)Chane-Sane, Schmid, and Laptev]{chane2023learning}
E.~Chane-Sane, C.~Schmid, and I.~Laptev.
\newblock Learning video-conditioned policies for unseen manipulation tasks.
\newblock In \emph{2023 IEEE International Conference on Robotics and Automation (ICRA)}, pages 909--916. IEEE, 2023.
\bibitem[Qian et~al.(2024)Qian, You, Zhou, Xu, Fu, Xue, and He]{qian2024contrast}
Z.~Qian, M.~You, H.~Zhou, X.~Xu, H.~Fu, J.~Xue, and B.~He.
\newblock Contrast, imitate, adapt: Learning robotic skills from raw human videos.
\newblock \emph{IEEE Transactions on Automation Science and Engineering}, 2024.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{Ho2020DenoisingDP}
J.~Ho, A.~Jain, and P.~Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{ArXiv}, abs/2006.11239, 2020.
\bibitem[Song et~al.(2020)Song, Sohl-Dickstein, Kingma, Kumar, Ermon, and Poole]{Song2020ScoreBasedGM}
Y.~Song, J.~N. Sohl-Dickstein, D.~P. Kingma, A.~Kumar, S.~Ermon, and B.~Poole.
\newblock Score-based generative modeling through stochastic differential equations.
\newblock \emph{ArXiv}, abs/2011.13456, 2020.
\bibitem[Chi et~al.(2023)Chi, Feng, Du, Xu, Cousineau, Burchfiel, and Song]{Chi2023DiffusionPV}
C.~Chi, S.~Feng, Y.~Du, Z.~Xu, E.~Cousineau, B.~Burchfiel, and S.~Song.
\newblock Diffusion policy: Visuomotor policy learning via action diffusion.
\newblock \emph{ArXiv}, abs/2303.04137, 2023.
\bibitem[Pearce et~al.(2023)Pearce, Rashid, Kanervisto, Bignell, Sun, Georgescu, Macua, Tan, Momennejad, Hofmann, and Devlin]{Pearce2023ImitatingHB}
T.~Pearce, T.~Rashid, A.~Kanervisto, D.~Bignell, M.~Sun, R.~Georgescu, S.~V. Macua, S.~Z. Tan, I.~Momennejad, K.~Hofmann, and S.~Devlin.
\newblock Imitating human behaviour with diffusion models.
\newblock \emph{ArXiv}, abs/2301.10677, 2023.
\bibitem[Reuss et~al.(2023)Reuss, Li, Jia, and Lioutikov]{Reuss2023GoalConditionedIL}
M.~Reuss, M.~X. Li, X.~Jia, and R.~Lioutikov.
\newblock Goal-conditioned imitation learning using score-based diffusion policies.
\newblock \emph{ArXiv}, abs/2304.02532, 2023.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:257952177}.
\bibitem[Team et~al.(2024)Team, Ghosh, Walke, Pertsch, Black, Mees, Dasari, Hejna, Kreiman, Xu, Luo, Tan, Sanketi, Vuong, Xiao, Sadigh, Finn, and Levine]{Team2024OctoAO}
O.~M. Team, D.~Ghosh, H.~R. Walke, K.~Pertsch, K.~Black, O.~Mees, S.~Dasari, J.~Hejna, T.~Kreiman, C.~Xu, J.~Luo, Y.~L. Tan, P.~R. Sanketi, Q.~Vuong, T.~Xiao, D.~Sadigh, C.~Finn, and S.~Levine.
\newblock Octo: An open-source generalist robot policy.
\newblock \emph{ArXiv}, abs/2405.12213, 2024.
\bibitem[Reuss et~al.(2024)Reuss, Yagmurlu, Wenzel, and Lioutikov]{Reuss2024MultimodalDT}
M.~Reuss, {\"O}.~E. Yagmurlu, F.~Wenzel, and R.~Lioutikov.
\newblock Multimodal diffusion transformer: Learning versatile behavior from multimodal goals.
\newblock \emph{ArXiv}, abs/2407.05996, 2024.
\bibitem[Liu et~al.(2024)Liu, Wu, Li, Tan, Chen, Wang, Xu, Su, and Zhu]{Liu2024RDT1BAD}
S.~Liu, L.~Wu, B.~Li, H.~Tan, H.~Chen, Z.~Wang, K.~Xu, H.~Su, and J.~Zhu.
\newblock Rdt-1b: a diffusion foundation model for bimanual manipulation.
\newblock \emph{ArXiv}, abs/2410.07864, 2024.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:273233386}.
\bibitem[Wang et~al.(2024)Wang, Wang, Jiang, Zou, Liu, Song, Wang, Xiao, Wu, Duan, et~al.]{wang2024diffusion}
Y.~Wang, L.~Wang, Y.~Jiang, W.~Zou, T.~Liu, X.~Song, W.~Wang, L.~Xiao, J.~Wu, J.~Duan, et~al.
\newblock Diffusion actor-critic with entropy regulator.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 54183--54204, 2024.
\bibitem[Jain et~al.(2024)Jain, Akhound-Sadegh, and Ravanbakhsh]{jain2024sampling}
V.~Jain, T.~Akhound-Sadegh, and S.~Ravanbakhsh.
\newblock Sampling from energy-based policies using diffusion.
\newblock \emph{arXiv preprint arXiv:2410.01312}, 2024.
\bibitem[Li et~al.(2024)Li, Krohn, Chen, Ajay, Agrawal, and Chalvatzaki]{li2024learning}
S.~Li, R.~Krohn, T.~Chen, A.~Ajay, P.~Agrawal, and G.~Chalvatzaki.
\newblock Learning multimodal behaviors from scratch with diffusion policy gradient.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 38456--38479, 2024.
\bibitem[Zhang et~al.(2024)Zhang, Luo, Sj{\"o}lund, Sch{\"o}n, and Mattsson]{zhang2024entropy}
R.~Zhang, Z.~Luo, J.~Sj{\"o}lund, T.~Sch{\"o}n, and P.~Mattsson.
\newblock Entropy-regularized diffusion policy with q-ensembles for offline reinforcement learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 98871--98897, 2024.
\bibitem[Li et~al.(2020)Li, Zhou, Xiong, and Hoi]{li2020prototypical}
J.~Li, P.~Zhou, C.~Xiong, and S.~C. Hoi.
\newblock Prototypical contrastive learning of unsupervised representations.
\newblock \emph{arXiv preprint arXiv:2005.04966}, 2020.
\bibitem[Mo et~al.(2022)Mo, Sun, and Li]{mo2022siamese}
S.~Mo, Z.~Sun, and C.~Li.
\newblock Siamese prototypical contrastive learning.
\newblock \emph{arXiv preprint arXiv:2208.08819}, 2022.
\bibitem[Potamias et~al.(2024)Potamias, Zhang, Deng, and Zafeiriou]{Potamias2024WiLoRE3}
R.~A. Potamias, J.~Zhang, J.~Deng, and S.~Zafeiriou.
\newblock Wilor: End-to-end 3d hand localization and reconstruction in-the-wild.
\newblock \emph{ArXiv}, abs/2409.12259, 2024.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:272753318}.
\bibitem[Qin et~al.(2023)Qin, Yang, Huang, Wyk, Su, Wang, Chao, and Fox]{Qin2023AnyTeleopAG}
Y.~Qin, W.~Yang, B.~Huang, K.~V. Wyk, H.~Su, X.~Wang, Y.-W. Chao, and D.~Fox.
\newblock Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system.
\newblock \emph{ArXiv}, abs/2307.04577, 2023.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:259367735}.
\end{thebibliography}