Physical Autoregressive Model for Robotic Manipulation without Action Pretraining
Extracted source text · arXiv 2508.09822
\documentclass[11pt, a4paper, logo, onecolumn,copyright]{googledeepmind}
\usepackage[authoryear, sort&compress, round]{natbib}
\bibliographystyle{abbrvnat}
% Information about your document.
\title{Physical Autoregressive Model for Robotic Manipulation without Action Pretraining}
% Can leave this option out if you do not wish to add a corresponding author.
% \correspondingauthor{my-email@google.com}
% Remove these if they are not needed
% \keywords{\LaTeX, Publications process, tools}
% \paperurl{arxiv.org/abs/123}
% Use the internally issued paper ID, if there is one
% \reportnumber{001} % Leave blank if n/a
% Assign your own date to the report.
% Can comment out if not needed or leave blank if n/a.
\renewcommand{\today}{}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{multicol}
\usepackage{siunitx}
\usepackage{array}
\usepackage[nameinlink]{cleveref}
\usepackage{bbm}
\usepackage{multirow}
\usepackage{subfig}
\usepackage{soul}
\usepackage{floatrow}
\usepackage{float}
\usepackage{wrapfig}
\usepackage{blindtext}
\usepackage{tablefootnote}
\usepackage{amsfonts}
\usepackage[flushleft]{threeparttable}
\usepackage{colortbl}
\usepackage{bbding}
\usepackage{lipsum}
\input{commands}
\makeatletter
\renewcommand{\thefootnote}{}%
\renewcommand{\@makefnmark}{}%
\renewcommand{\@makefntext}[1]{#1}%
\makeatother
% Can have as many authors and as many affiliations as needed. Best to indicate joint
% first-authorship as shown below.
\author[1]{Zijian Song}
\author[1]{Sihan Qin}
\author[2]{Tianshui Chen}
\author[1]{Liang Lin}
\author[1]{Guangrun Wang}
% Affiliations *must* come after the declaration of \author[]
% \affil[1]{Sun Yat-sen University, GuangZhou, China\\
% \texttt{\{songzj8,qinsh9,wanggr3\}@mail2.sysu.edu.cn}\\
% \texttt{linliang@ieee.org}}
% \affil[2]{Guangdong University of Technology, GuangZhou, China\\
% \texttt{chentianshui@gdut.edu.cn}}
\author{\textbf{\normalsize{Zijian Song$^{1}$, Sihan Qin$^{1}$, Tianshui Chen$^{3,4}$, Liang Lin$^{1,2,3}$, Guangrun Wang$^{1,2,3*}$}} \\
$^1$Sun Yat-sen University\\
$^2$Guangdong Key Laboratory of Big Data Analysis and Processing\\
% $^3$Peng Cheng Laboratory\\
$^3$X-Era AI Lab\\
$^4$Guangdong University of Technology\\
Emails: \texttt{\{songzj8,qinsh9\}@mail2.sysu.edu.cn, wanggrun@gmail.com, linliang@ieee.org, chentianshui@gdut.edu.cn}\\
Website: \href{https://hcplab-sysu.github.io/PhysicalAutoregressiveModel/}{https://hcplab\-sysu.github.io/PhysicalAutoregressiveModel/}\footnote{$^*$corresponding author.}
}
\begin{abstract}
The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR), where physical tokens combine frames and actions to represent the joint evolution of the robot and its environment.
PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining, enabling accurate video prediction and consistent action trajectories.
It also adopts a DiT-based de-tokenizer to model frames and actions as continuous tokens, mitigating quantization errors and facilitating mutual enhancement.
Furthermore, we incorporate a causal mask with inverse kinematics, parallel training, and the KV-cache mechanism to further improve performance and efficiency.
Experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task, matches the performance of action-pretrained baselines on other tasks, and accurately predicts future videos with tightly aligned action trajectories.
These findings underscore a promising direction for robotic manipulation by transferring world knowledge from autoregressive video pretraining.
\end{abstract}
\begin{document}
\maketitle
\input{section/1-intro}
\input{section/2-relatedwork}
\input{section/3-preliminaries}
\input{section/4-method}
\input{section/5-experiment}
\input{section/6-conclusion}
\bibliography{main}
\end{document}
\usepackage{xspace}
% Our commands
\newcommand{\Ours}{Gemini Embedding}
\newcommand{\ours}{Gemini Embedding}
\newcommand{\dataset}{FRet}
\newcommand{\sect}[1]{Section \ref{#1}}
\newcommand{\conc}{\mathbin{\|}}
\newfloatcommand{capbtabbox}{table}[][\FBwidth]
\newcommand{\draftonly}[1]{#1}
\newcommand{\eat}[1]{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% % Uncomment for removing notes.
\renewcommand{\draftonly}[1]{}
\definecolor{darkgreen}{RGB}{0, 102, 0}
\newcommand{\draftcomment}[3]{\draftonly{\textcolor{#2}{{{[#1: #3]}}}}}
\newcommand{\todo}[1]{\draftcomment{TODO}{red}{#1}}
\newcommand{\jinhyuk}[1]{\draftcomment{Jinhyuk}{red}{#1}}
\newcommand{\kai}[1]{\draftcomment{Feiyang}{brown}{#1}}
\newcommand{\mw}[1]{\draftcomment{Sahil}{purple}{#1}}
\newcommand{\cer}[1]{\draftcomment{Daniel}{teal}{#1}}
\newcommand{\yi}[1]{\draftcomment{Madhuri}{orange}{#1}}
\newcommand{\inaim}[1]{\draftcomment{Iftekhar}{violet}{#1}}
\newcommand{\gustavoha}[1]{\draftcomment{Gustavo}{cyan}{#1}}
\newcommand{\mjb}[1]{\draftcomment{Michael}{darkgreen}{#1}}
% \newcommand{\eat}[1]{}
\crefformat{section}{\S#2#1#3}
\newcommand{\pairsim}{\operatorname{sim}}
\newcommand{\meanpool}{\texttt{mean\_pool}}
\newcommand{\mask}{\texttt{mask}}
% \definecolor{mygray}{gray}{0.0}
% \definecolor{lightgray}{gray}{0.925}
% \newcommand{\none}[1]{-}
% \newcommand{\macroavg}[1]{#1}
% \newcommand{\microavg}[1]{\underline{#1}}
% \newcommand{\indomain}[1]{\underline{\textcolor{mygray}{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\renewcommand{\floatpagefraction}{0.8}
\section{Introduction}
\label{sec:intro}
% 提纲：重点是连续自回归+视频预训练
% intro思路：需要video驱动的VLA（世界模型驱动的VLA）----构建物理自回归的模型来表示世界的迭代发展----并且直接从视频模型获取知识来无缝迁移----其中连续向量起重要的作用----本文搭建一个连续信号自回归而且继承视频预训练的能力
% 生成式预训练模型快速发展，在CV和NLP中取得了成功。但是action的预训练需要大规模的数据，而action的监督数据获取成本很高，让这件事非常困难。一个合适的解决方法就是利用其他的预训练模型，将他们迁移到action空间中。就比如XXX，他们在大语言模型的基础上构建action模型，通过action decoder head等方法把语言模型的输出映射到low-level robot control。
% 但是语言和action本质上差距很大，这会导致xxx问题，更合适的方法应该是借助预训练的视频模型，特别是自回归视频模型。视频自回归模型它迭代地预测下一帧，它隐式的涵盖了世界往后的迭代发展规律。这与action的预测目标非常统一，都是学会下一帧的变化。通过把action与video深度绑定，能反映物理空间的发展变化，能让action和视频双向促进，让action精确控制视频预测的发展，让视频提供action的guidence并通过逆运动学指导动作预测。自回归模型天然能把两个目标深度结合在一起。因此本文希望提出一个video-action自回归模型。
% 在video-action自回归模型中，一个不能忽视的内容是分词器的设计。传统方法普遍采用离散的分词器。这会导致某某某问题。在更新的工作中，有人尝试用连续信号来建模视频，但缺少与action的关联。在本文中，我们同时采用连续信号同时建模action和video，这将带来某某某好处。
% 综上，在本文中，我们提出一种连续自回归新框架，它直接利用预训练的video预测模型，继承其中的世界知识。
% 本文的贡献：1.我们将frame和action绑定在一起组合为physical token，进而构建了一个物理自回归模型。我们将视频生成模型中的世界知识迁移到了该模型中，让它反映物理世界的迭代运行规律。2.我们将frame和action都表示为连续空间中的token，通过DiT和Diffusion Loss实现对连续信号分布的建模。3.我们为自回归模型实现了代表隐式逆运动学的因果mask，并行训练，kv-cache机制，两种互补的测试策略。在ManiSkill Benchmark上的仿真实验证明了我们整体方案和各个设计的有效性。
% 其他一些没来得及加入的想法：1.我们要强调我们的特点是可变长度的历史序列，这允许我们维持计划同时做出动态调整。（不过之前讨论，这个属于AR的特点，我们不做主要介绍）2.我们的贡献是不是要更加强调：从视频自回归到物理自回归的transfer
The success of generative pretraining in computer vision (CV) and natural language processing (NLP) has highlighted the power of large-scale pretraining, demonstrating strong generalization ability across diverse tasks~\cite{brown2020language, kirillov2023segment, tian2024visual, dubey2024llama, guo2025deepseek, SD-MVS}.
This paradigm inspires the robotics community to pursue powerful foundational models for visuomotor control and manipulation.
Yet, unlike CV and NLP, generative action pretraining presents unique challenges, as acquiring large-scale human demonstration data is both time-consuming and labor-intensive~\cite{zitkovich2023rt}. A promising strategy is to transfer knowledge from existing pretrained models to the action domain~\cite{zitkovich2023rt}. For instance, recent approaches build action policies upon large language models (LLMs), using action decoders to map linguistic outputs to low-level robot control, namely Vision-Language-Action-Models (VLAs)\cite{kim2024openvla, li2024cogact, bu2025agibot, zhao2025cot}.
However, the intrinsic gap between text modality and action modality poses significant challenges for such approaches, often resulting in suboptimal alignment between symbolic reasoning and physical control.
A more suitable solution is to build on pretrained video generation models, particularly autoregressive video generation models~\cite{gao2024vid, xie2025progressive}. These models predict future frames iteratively based on past observations, implicitly grounded in the physical dynamics of the physical world.
This predictive capability aligns naturally with the goal of action generation, as both tasks require understanding how the current state evolves over time.
% Moreover, autoregressive video models provide a rich temporal context that can serve as a prior for reasoning about causal relationships, motion patterns, and the unfolding of future events.
Moreover, integrating actions with video representations creates a bidirectional synergy.
Actions ensure the predicted dynamics remain consistent with controllable outcomes.
Video offers valuable guidance for action generation through inverse kinematics.
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{figure/world_evolve1.pdf}
\caption{\textbf{The illustration of Physical Autoregression.} Our autoregressive process operates over a sequence of physical tokens (marked by red), each combining the visual world state (marked by orange) and the embodiment state (marked by black), providing a progressive estimate of their joint evolution.
This process runs in sync with the environment: at each step, the predicted token is decoded into an image and an action, which interact with the environment to update its state (marked by blue), while the resulting observations and proprios are encoded back into the context.}
\label{fig:world_evolve}
\end{figure}
The autoregressive nature provides an ideal foundation for representing the physical world.
Based on this insight, this work presents the \emph{Physical Autoregressive Model, PAR}. The underlying motivation is shown in~\cref{fig:world_evolve}.
Specifically, the frame tokens and the action tokens are combined into physical tokens, which effectively represent the joint evolution of robotic manipulation and environmental feedback.
To mitigate the scarcity of human demonstrations, we transfer world knowledge from video pretraining into our PAR model, facilitating a seamless transition from understanding visual dynamics to capturing physical dynamics.
A critical component for autoregressive models lies in the token representation~\cite{yu2023language, vuong2025action}.
Most existing methods rely on discrete token representations to represent visual signals and action signals~\cite{zitkovich2023rt, kim2024openvla, wang2025unified}.
This may introduce resolution errors whose accumulation propagates through long-horizon prediction and produces substantial trajectory drift\cite{zhang2025chain}.
Recent studies have explored continuous token representations for vector modeling~\cite{li2024autoregressive, deng2024autoregressive}, but their integration into a unified vision-action autoregressive framework remains limited.
In this work, we propose to represent both frame and action as continuous signals. Specifically, we leverage the Diffusion-Transformer model (DiT)~\cite{peebles2023scalable} coupled with Diffusion Loss training objective to model the arbitrary distribution of the continuous frame and action tokens. This design not only improves smoothness and coherence, but also facilitates a deeper mutual interaction between the continuous spaces of vision and action.
Besides, we further refine our Physical Autoregressive Model through two key components.
(1) During autoregression, we let the action tokens attend to the current frame tokens, which implicitly encode the prediction of the next visual state. This mechanism functions as an implicit inverse-kinematics step that improves accuracy.
To support this, we introduce a tailored causal masking strategy specifically designed for the frame–action joint autoregressive model.
(2) Similar to LLMs, we incorporate parallelized training~\cite{vaswani2017attention} to accelerate convergence and the KV-caching mechanism~\cite{NVIDIA2023LLM} for efficient inference.
% Third, we design two complementary inference strategies: direct inference and adaptive restart. The former maintains the entire history context to ensure coherent execution of long-horizon tasks. The latter selectively resets the history context to avoid being trapped in erroneous manipulation patterns and explore alternative solutions.
We conduct experiments on the ManiSkill Benchmark~\cite{taomaniskill3}, which demonstrates the effectiveness of PAR.
Our method achieves a 100\% success rate on the PushCube task and performs on par with action-pretrained baselines on other tasks. Visualizations further show that PAR accurately predicts future video frames while generating action trajectories that remain tightly aligned with the predicted dynamics.
Ablation studies confirm the necessity of each key design component.
Key contributions of this paper includes:
(1) We combine frames and actions into physical tokens to construct a Physical Autoregressive Model. The world knowledge is inherited from video generation models to capture the iterative dynamics of the physical world, which needs zero action pretraining.
(2) We represent frames and actions as continuous vectors and model their distribution through DiT model and diffusion loss, allowing for fine-grained representation and deeper interaction between two modalities.
(3) We further incorporate a causal mask with implicit inverse-kinematics, parallel training, and KV-cache mechanism. Experiments on the ManiSkill benchmark validate the effectiveness of our overall framework as well as each of its key components.
\section{Related Work}
\label{sec:related_work}
\begin{figure}[t]
\centering
\begin{minipage}[b]{0.8\linewidth}
\centering
\includegraphics[width=1.0\linewidth]{figure/phytransformer1a.pdf}
\end{minipage}
\hfill
\begin{minipage}[b]{0.18\linewidth}
\centering
\includegraphics[width=1.0\linewidth]{figure/phytransformer1b.pdf}
\end{minipage}
\caption{\textbf{The model architecture of Physical Autoregressive Model.} Starting from text tokens, PAR leverages a causal Transformer to autoregressively predict physical tokens. Notably, we apply a frame diffusion and an action diffusion to estimate the conditional distributions of visual and action signals in continuous space.
The structure of the action diffusion network is shown on the right column, where the predicted token serves as the conditional vector and is injected through cross-attention.
}
\label{fig:ar_transformer}
\end{figure}
\subsection{Vision-Language-Action Models}
The scarcity of manipulation demonstrations makes large-scale action pretraining prohibitively expensive. A recent trend in robotics involves transferring pretrained knowledge from LLMs to the action domain~\cite{zitkovich2023rt, kim2024openvla, black2024pi0, xu2025a0}.
These approaches typically append a lightweight action head to the LLM, mapping language tokens to action outputs~\cite{wen2025tinyvla, wen2025dexvla, liu2025hybridvla, bu2025univla}.
They have demonstrated strong manipulation success rates with minimal or even zero action pretraining.
In contrast, our approach builds the action model on top of a pretrained video generation model rather than an LLM.
This design enables a seamless transfer of physical knowledge that aligns actions with the world dynamics captured by the video.
\subsection{Video-Action Joint Predition}
The concept of jointly predicting video and actions has been explored in prior studies, demonstrating its effectiveness~\cite{wu2023unleashing, cheang2024gr, cen2025worldvla, yelatent, fan2025towards}.
For example, UVA~\cite{li2025unified} jointly learns a video-action representation to model the correlation between video and action. VPP~\cite{hu2024video} leverages the pretrained video diffusion model for implicit inverse dynamics.
Building on this idea, our work employs an autoregressive framework that models the coupled evolution of the world and the robot in a step-by-step manner.
A key advantage is that our framework can leverage a variable-length history context for coherent planning and adaptive adjustment.
\subsection{Continuous Signal Tokenization}
For continuous signals such as images or actions, discretizing them may disrupt their inherent structure and introduce resolution errors~\cite{zhang2025chain}.
To preserve continuity for action generation, some approaches employ linear or MLP projectors but fall short for generative modeling~\cite{wu2023unleashing, fu2024context, zhang2025chain}.
To overcome this, Li et al.~\cite{li2024autoregressive} proposed using a denoising process to compute the conditional distribution of signals, agnostic to the signal distribution. This idea has been extended to video generation~\cite{deng2024autoregressive} and action prediction~\cite{pi05vision, li2025unified, team2024octo, zhong2025dexgraspvla}.
However, its application to action autoregressive modeling remains underexplored. In this work, we leverage a denoising process as a de-tokenizer for continuous action autoregression, fostering interactions between action and frame modalities.
\section{Preliminaries}
\label{sec:preliminaries}
This section provides a review of the key concepts that form the foundation of our work. We begin with an overview of diffusion models, then describe how the diffusion process can be adapted as a loss function for autoregressive modeling in a continuous-valued space. Finally, we discuss how these principles are applied to video generation.
\subsection{Diffusion Models}
Diffusion Models\cite{ho2020denoising} are a class of generative models that produce samples by progressively adding Gaussian noise to clean data \(x_0\) and then learning to invert this noising process. Formally, the forward diffusion process is defined as:
\begin{equation}
q(x_t \mid x_{t-1})
= \mathcal{N}\bigl(x_t;\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I\bigr),
\end{equation}
where \(x_t\) is the noisy sample at timestep \(t\), \(\beta_t\) is the pre-defined noise schedule, and \(\epsilon\sim\mathcal{N}(0,I)\) is the injected noise. The reverse diffusion process is defined as:
\begin{equation}
p_\theta(x_{t-1} \mid x_t)
= \mathcal{N}\bigl(x_{t-1};\mu_\theta(x_t,t),\,\Sigma_\theta(x_t,t)\bigr),
\end{equation}
which inverts the corruption, starting from the standard normal prior \(x_T\sim\mathcal{N}(0,I)\) at the final timestep \(T\). Training then teaches the model to remove noise by minimizing
\begin{equation}
L_{\text{simple}}(\theta) = \mathbb{E}_{t,x_{0},\epsilon}\left[ \left\| \epsilon - \epsilon_{\theta}(\sqrt{\overline{\alpha}_{t}}x_{0} + \sqrt{1 - \overline{\alpha}_{t}}\epsilon, t) \right\|^{2} \right],
\end{equation}
where $\epsilon_{\theta}(\sqrt{\overline{\alpha}_{t}}x_{0} + \sqrt{1 - \overline{\alpha}_{t}}\epsilon, t)$ is the neural network's prediction of the added noise . At inference, the reverse process is applied to pure Gaussian noise to progressively remove noise and thus generate high-quality samples.
\subsection{Diffusion Loss}
Diffusion Loss~\cite{li2024autoregressive} repurposes the diffusion framework to model each continuous token’s conditional distribution \(p(x|z)\) by applying the noise–denoise cycle directly in latent space. Concretely, given a ground-truth token embedding \(x\in\mathbb{R}^{d}\) and its autoregressive context \(z\), we train a lightweight denoiser \(\epsilon_{\theta}(x_{t}|t,z)\) to predict the injected noise by minimizing the squared-error loss:
\begin{equation}
\mathcal{L}(z,x)=\mathbb{E}_{\epsilon,t}[||\epsilon-\epsilon_{\theta}(x_{t}|t,z)||^{2}].
\label{eq:diffusion_loss}
\end{equation}
Here, \(\epsilon\in\mathbb{R}^{d}\) is a noise vector sampled from \(\mathcal{N}(0,I)\). The noise-corrupted vector is \(x_{t} = \sqrt{\overline{\alpha}_{t}}x+\sqrt{1-\overline{\alpha}_{t}}\epsilon\), where \(\overline{\alpha}_{t}\) defines a noise schedule and \(t\) is a time step of that schedule. The notation \(\epsilon_{\theta}(x_{t}|t,z)\) indicates that the denoising network takes the corrupted token \(x_t\) as input and is conditioned on both the time step \(t\) and the context vector \(z\).
This loss enables us to learn \(p(x|z)\) directly in continuous space—eliminating the need for vector quantization. At inference, each token is sampled by running the learned reverse diffusion chain:
\begin{equation}
x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\epsilon_{\theta}(x_{t}|t,z)\right)+\sigma_{t}\delta.
\end{equation}
Starting with a random vector \(x_{T}\sim\mathcal{N}(0,I)\), this iterative procedure produces a final sample \(x_{0}\sim p(x|z)\), yielding high-quality continuous embeddings.
This objective teaches \(\epsilon_\theta\) to invert the forward corruption, effectively learning \(p(x_n\mid z_n)\). At inference, one samples \(x_n\) by running the learned reverse diffusion chain from \(x_n^T\sim\mathcal{N}(0,I)\) back to \(x_n^0\), yielding high-quality continuous tokens without vector quantization.
\subsection{Autoregressive Modeling for Video}
The principle of autoregressive modeling in a continuous, non-quantized space can be extended to video generation~\cite{deng2024autoregressive}.
First, the temporal autoregressive process can be formally expressed by the conditional probability:
\begin{equation}
% \begin{gathered}
p(P,m,B,S_{1},...,S_{N})=\prod_{n=1}^{N}p(S_{n}|P,m,B,S_{1},...,S_{n-1}).
% \end{gathered}
\end{equation}
In this formulation, the probability of generating the n-th frame, \(S_{n}\), is dependent on all prior frames (\(S_{1},...,S_{n-1}\)) and other conditioning contexts, ensuring coherence.
Second, the spatial autoregressive model generates each frame in a set-by-set order.
Concretely, the generation of token sets for the f-th frame is formulated as follows:
\begin{equation}
% \begin{gathered}
p(S_{n}^{\prime},S_{(n,1)},...,S_{(n,K)})=\prod_{k=1}^{K}p(S_{(n,k)}|S_{n}^{\prime},S_{(n,1)},...,S_{(n,k-1)}).
% \end{gathered}
\end{equation}
Here, the k-th set of tokens, \(S_{(n,k)}\), is predicted conditioned on indicator features \(S_{n}^{\prime}\) and all previously generated sets (\(S_{(n,1)},...,S_{(n,k-1)}\)). This allows the model to leverage bidirectional context for generating spatial details.
\section{Method}
\label{sec:cmast_method}
\subsection{Problem Formulation}
As an autoregressive model, one key characteristic of our Physical Autoregressive Model (PAR) is its ability to operate with a variable-length history context.
This design enables the model to consistently follow its planned trajectory while dynamically adjusting its outputs as the context evolves.
Formally, given the task instruction $T$, we maintain an accumulated history context consisting of a sequence of image observations $ \left\{ O_0, O_1, \cdots, O_{N-1} \right\} $ and the corresponding sequence of action chunks $ \left\{ A_1, A_2, \cdots, A_{N-1} \right\} $.
The image sequence is one step longer than the action sequence because an initial observation exists before any action is taken.
The objective of PAR is to predict the subsequent image frame $ O_N $ and its associated action chunk $ A_N $ in an autoregressive manner.
The subscript $n$ denotes the timestep.
To enhance consistency, we adopt action-chunk modeling~\cite{chi2023diffusion, zhao2023learning}, where each action chunk $A_n$ consists of $L$ consecutive actions.
Consequently, the action sampling rate is $L$ times that of the images.
% This means we take $L$ times of sample rate for action compared that of the image.
\begin{figure}[t]
\centering
\includegraphics[width=0.4\linewidth]{figure/attn_mask.pdf}
\caption{\textbf{The causal attention mask.} For illustration, we use 3 blocks to represent the task instruction tokens, 2 blocks to represent the frame tokens, 2 blocks to represent the action tokens.}
\label{fig:casual_mask}
\end{figure}
\subsection{Model Architecture}
In this work, we construct our Physical Autoregressive Model (PAR) on top of a pretrained autoregressive video generation model, i.e., NOVA~\cite{deng2024autoregressive}. By transferring the world knowledge embedded in NOVA’s pretrained representations, PAR acquires a seamless understanding of physical dynamics without action pretraining, enabling it to unify video prediction and action generation within a single autoregressive framework.
\subsubsection{Tokenizer}
To fully exploit the capabilities of NOVA, our framework adopts its pretrained language and image encoders without modification.
Specifically, the task instruction is embedded into $K_t$ tokens, each with $d$-dimension features, using a frozen Phi model~\cite{javaheripi2023phi} followed by a linear projector.
For visual input, we employ an open-source 3D variational autoencoder~\cite{zheng2024open} to encode the video frames to the latent space, which is also kept frozen.
The resulting latent images are then projected to physical embedding space and flattened to form $K_O$ tokens, each with a feature dimension of $d$.
In addition, action chunks are encoded through a lightweight multi-layer perceptron (MLP), which maps continuous action signals into the same physical embedding space as the frame tokens.
Each action step constitutes a $d$-dimensional token, resulting in a total of $K_A = L$ tokens.
\begin{table}
\centering
\setlength{\tabcolsep}{8pt}
\begin{tabular}{cc}
\toprule
Module & Parameters \\
\midrule
Action-Tokenizer & 0.6M \\
Action-DeTokenizer & 21.1M \\
Frame-DeTokenizer & 32.8M \\
Casual-Transformer & 608.7M \\
\midrule
Total & 663.2M \\
\bottomrule
\end{tabular}
\caption{\textbf{PAR parameter count.} The PAR architecture only introduces minimal parameters to transfer knowledge from video pretraining to action generation.}
\label{tab:param}
\end{table}
\subsubsection{Physical Autoregression}
We follow the standard \emph{token-by-token} autoregressive paradigm commonly used in large language models (LMMs).
The key difference is that our model operates on physical tokens rather than language tokens, which unifies frame and action signals.
This design allows us to jointly model the iterative evolution of both the robot and the external world.
Specifically, each physical token is constructed by concatenating a frame token with its corresponding action chunks in the token dimension, $P_{n} = \left[ O_n; A_n\right]$.
For the first observation frame, we concatenate it with a learnable Begin-Of-Action token, $A_0 = [BOA]$.
The autoregression process then predicts the next physical token conditioned on the sequence of preceding physical tokens.
This process can be written as:
\begin{equation}
% \begin{gathered}
p\left( T,O_0,A_0,\cdots ,O_N,A_N \right)=p\left( T,P_0,\cdots ,P_N \right)=\prod_{n=0}^N{p\left( P_n|T,P_0,\cdots ,P_{n-1} \right)}.
% \end{gathered}
\end{equation}
where the conditional distribution is modeled by a Transformer that exactly replicates the architecture of NOVA\cite{deng2024autoregressive}.
\subsubsection{De-Tokenizer}
Previous works typically rely on discrete tokenization for both images and actions~\cite{zitkovich2023rt, zheng2024open, wang2025unified}. This aligns well with language modeling, as discrete tokens can be directly integrated into the vocabulary.
However, such discretization introduces resolution errors that accumulate over the long horizon~\cite{zhang2025chain}.
Another line of work leverages linear or MLP projectors for decoding~\cite{wu2023unleashing, fu2024context, zhang2025chain}.
While these approaches avoid quantization artifacts, they primarily model deterministic values, limiting their generative capacity. Other extensions such as Gaussian Mixture Models still fail to capture arbitrary distributions~\cite{fu2024context, cheang2024gr}.
In this work, we address these limitations by adopting a continuous de-tokenizer, which maintains both a continuous and generative representation.
Moreover, our design also stimulates mutual enhancement between the image and action modalities.
Specifically, we follow the Diffusion-based de-tokenizer proposed by MAR\cite{li2024autoregressive}.
The tokens output by Transformer are taken as the conditional vector $Z_n$ for denoising process:
\begin{equation}
Z_n = Transformer(T, P_0, \cdots, P_{n-1}).
\end{equation}
A DiT network is applied to compute the probability of the predicted tokens $p(P_n|Z_n)$ through the Diffusion Loss:
\begin{equation}
\mathcal{L}(Z_n,P_n)=\mathbb{E}_{\epsilon,t}[||\epsilon-\epsilon_{\theta}(P_{n,t}|t,Z_n)||^{2}],
\end{equation}
where the $Z_n$ represents the conditioning vector.
The gradient is backpropagated to $z$ for optimizing the parameters in the Transformer backbone.
The decoding of frame tokens and action tokens is performed separately.
For frame tokens, we follow the same process used in NOVA, employing an additional autoregression process for masked patch set prediction.
For action tokens, we employ a lightweight DiT network for denoising, named Action-DiT module, where the conditioning vectors are injected into the DiT blocks via cross-attention.
\subsubsection{Causal Mask}
Our causal masking scheme differs from those commonly used in language models or video prediction models; the diagram can be found in~\cref{fig:casual_mask}.
Each physical token consists of a frame part and an action part.
For the frame part, we employ a chunk-based attention mask, which allows all patches within the same frame to attend to each other.
For the action part, we use a temporal causal mask so that earlier actions within a chunk cannot attend to later actions.
Additionally, we allow action tokens to attend to frame tokens unidirectionally.
Since frame tokens processed by the Transformer are further decoded to future frames, this unidirectional design implicitly models inverse kinematics, enabling action planning to leverage future visual states for better accuracy.
Finally, across different time steps, we adopt a temporal causal mask to preserve temporal causality.
\subsubsection{Loss Function}
The training objective is defined as the average diffusion loss over the sequence of physical tokens, which is decomposed into frame and action components that are combined with equal weighting:
\begin{equation}
% \begin{aligned}
loss=\sum_{n=1}^N\mathcal{L}(Z_n,P_n)=\sum_{n=1}^N\left(\mathcal{L}_{obs}(Z_{O,n},O_n) + \mathcal{L}_{act}(Z_{A,n},A_n)\right).
% \end{aligned}
\end{equation}
\subsubsection{Efficient Training and Inference}
For training, we adopt the standard parallelization strategy, where the entire token sequence is processed with teacher forcing and the loss for all tokens is computed in parallel. During inference, we employ a KV-cache mechanism~\cite{NVIDIA2023LLM} to store intermediate features at each layer, enabling efficient reuse when generating subsequent tokens.
% \subsubsection{Inference Strategy}
% We introduce two complementary inference strategies: direct inference and adaptive restart. Direct inference preserves the entire history context, enabling a coherent manipulation. Adaptive restart selectively resets the history context to escape erroneous manipulation patterns and explore alternative solutions. Specifically, when the context length exceeds twice the average task horizon observed during training, the accumulated history is cleared.
\begin{table}
\centering
\setlength{\tabcolsep}{2pt}
\begin{tabular}{lcccc}
\toprule
Method & PushCube & PickCube & StackCube & Avg. \\
\midrule
ACT [\citeyear{zhao2023learning}] & 76\% & 20\% & 30\% & 42\% \\
BC-T [\citeyear{mandlekar2021matters}] & 98\% & 4\% & 14\% & 39\% \\
DP [\citeyear{chi2023diffusion}] & 88\% & 40\% & \textbf{80}\% & 69\% \\
ICRT [\citeyear{vuong2025action}] & 77\% & \textbf{78}\% & 30\% & 62\% \\
RDT [\citeyear{liurdt}] & \textbf{100}\% & 77\% & 74\% & \textbf{84}\% \\
\midrule
PAR(Ours) & \textbf{100}\% & 73\% & 48\% & 74\% \\
\bottomrule
\end{tabular}
\caption{\textbf{The success rate comparison} in the ManiSkill Benchmark. The best are highlighted by \textbf{bold}. Our method, without any action pretraining, achieves performance comparable to SOTA.}
\label{tab:comparison_label}
\end{table}
\subsection{Implementation Details}
Our method requires no action pretraining, but it is finetuned on downstream manipulation tasks.
All finetuning experiments are conducted on a single NVIDIA A100-SXM-64GB GPU, with the longest training completed within 40 GPU hours.
During finetuning, we adopt full-parameter finetuning, updating all parameters of the NOVA along with the newly introduced action module.
The total number of learnable parameters is summarized in~\cref{tab:param}.
During training, we follow~\cite{li2024autoregressive} to sample $t$ by 4 times for each image and each action.
We employ the Rotary Positional Embeddings (RoPE)~\cite{su2024roformer} to preserve the temporal dependencies, with distinct frequency settings for frame and action tokens.
\section{Experiment}
\label{sec:experiment}
\begin{figure}[t]
\centering
\includegraphics[width=0.9\linewidth]{figure/vis_success2.pdf}
\caption{\textbf{Video predictions and actual task executions.} Each row shows PAR's predicted video alongside the corresponding execution video for three different tasks. The strong visual similarity between predicted and actual action videos highlights the effectiveness of our approach in transferring knowledge from video pretraining.}
\label{fig:vis_success}
\end{figure}
\subsection{Evaluation Protocol}
\label{sec:evaluation_protocol}
\subsubsection{Benchmark}
We use the ManiSkill Benchmark~\cite{taomaniskill3} and follow prior works\cite{chi2023diffusion, li2025unified, cen2025worldvla} by evaluating the single task scenarios. ManiSkill Benchmark provides diverse, high-fidelity simulation environments and standardized metrics, making it a reliable platform for robotic manipulation evaluation.
We specifically focus on three manipulation tasks: PushCube, PickCube, and StackCube.
We render 1K demonstrates for each task as training data.
\subsubsection{Metric}
Each task is evaluated with 125 rollouts, covering 5 random seeds and 25 distinct initial states. We report the average success rate across these rollouts as the primary performance metric.
\subsubsection{Baseline}
We compared with the following baseline, all methods are fine-tuned on the same data as our model and tested using the same random seed and initial states.
\begin{itemize}
\item \textbf{Action Chunk Transformer (ACT)}~\cite{zhao2023learning} leverages action chunking and encoder-decoder transformer layers to learn bimanual manipulation policies.
\item \textbf{Behavior Cloning Transformer (BC-T)}~\cite{mandlekar2021matters} directly learn the mapping from visual observations to actions through transformer architecture.
\item \textbf{Diffusion Policy (DP)}~\cite{chi2023diffusion} reformulates action generation as a conditional denoising diffusion process and achieves SOTA visuomotor performance. We use the Transformer-based design from their original implementation for all tasks.
\item \textbf{In-Context Robot Transformer (ICRT)}~\cite{fu2024context, vuong2025action} is pretrained on in-context action trajectories and uses a Lipschitz-conditioned VAE for action embedding.
\item \textbf{RobotDiffusionTransformer (RDT)}~\cite{liurdt} is an end-to-end robotic model based on the DiT, which has 1.3 billion parameters and is pretrained on a large-scale robotic dataset.
\end{itemize}
\subsection{Quantitative Comparison}
We quantitatively evaluate our method across three manipulation tasks, with results summarized in~\cref{tab:comparison_label}.
On average, our method achieves the second-best success rate, only behind RDT, which requires action pretraining.
In the PushCube task, our approach achieves a 100\% success rate, significantly outperforming the ICRT~\cite{fu2024context} baseline by 30\%.
In the PickCube task, our performance is within 7\% of the SOTA baseline, RDT~\cite{liurdt}, despite RDT relying on action pretraining while our method does not.
In the StackCube task, our method falls behind DP~\cite{chi2023diffusion} and RDT~\cite{liurdt}, but still outperforms ICRT~\cite{fu2024context} by 60\%.
These results highlight the strength of our approach in transferring world knowledge from video pretraining, with zero action pretraining and just 30M additional parameters.
Overall, these results validate the advantages of our autoregressive joint modeling framework, which facilitates cross-modal interaction and offers a promising direction for addressing data scarcity in action pretraining.
\subsection{Ablation Study}
We conduct a ablation study to examine the contribution of each component in our framework. Specifically, we compare our full method, denoted as PAR-Full against two ablated variants: (1) PAR-NoAR, which removes the autoregressive architecture. (1) PAR-Discrete, which replaces the generative de-tokenizer with a discriminative one.
These ablation models are trained and evaluated using the same evaluation protocol as our full method (see ~\cref{sec:evaluation_protocol}).
The results are summarized in~\cref{tab:ablation}.
PAR-Full consistently outperforms both baselines by a large margin, confirming the effectiveness of our designs.
\subsubsection{Autoregressive Architecture}
We remove the entire autoregressive framework inherited from NOVA to investigate its contribution, remaining only the MLP-based action encoder and the diffusion-based action decoder.
Consequently, the model directly decodes the embedded visual tokens and text tokens without any autoregressive transformer reasoning.
This modification leads to a decrease in average success rate from 73.6\% to 11.2\%, demonstrating the critical role of the autoregressive module.
This result suggests that our autoregressive video pretraining enables joint reasoning over the robot and the environment, effectively providing implicit inverse kinematics guidance to the action decoder, and thereby improving execution accuracy.
\begin{table}
\centering
\setlength{\tabcolsep}{2pt}
\begin{tabular}{lcccc}
\toprule
Method & PushCube & PickCube & StackCube & Avg. \\
\midrule
PAR-NoAR & 29.6\% & 4.0\% & 0.0\% & 11.2\% \\
PAR-Discrete & 87.2\% & 65.6\% & 7.2\% & 53.3\% \\
PAR-Full & \textbf{100.0}\% & \textbf{72.8}\% & \textbf{48.0}\% & \textbf{73.6}\% \\
\bottomrule
\end{tabular}
\caption{\textbf{The ablation study} in the ManiSkill Benchmark. The best are highlighted by \textbf{bold}.}
\label{tab:ablation}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{figure/attn_map1.pdf}
\caption{\textbf{The attention map.} The top row shows the token-level attention map, indicating how the predicted action attends to previous frame and action tokens. The bottom row shows the pixel-level attention map, indicating how the predicted action attends to different spatial regions of the frame.}
\label{fig:attn_map}
\end{figure}
\subsubsection{De-Tokenizer Type}
To assess the role of the generative de-tokenizer, we replace the diffusion-based de-tokenizer with a MLP-based discriminative de-tokenizer, aligning with designs adopted in prior works~\cite{wu2023unleashing, fu2024context, zhang2025chain}.
This modification reduces the average success rate from 73.6\% to 53.3\%, indicating that the discriminative de-tokenizer may collapse to deterministic value regression and fail to capture uncertainty in the action distribution. This loss of distributional modeling likely results in overfitting and poorer generalization.
\subsection{Visualization}
\subsubsection{Video Action Alignment}
We visualize the predicted video sequences alongside the corresponding execution videos in~\cref{fig:vis_success}. The results reveal a clear alignment between the predicted visual trajectories and actual robot actions, with highly similar motion trajectories and timing of key actions.
For example, in the second row of the PickCube task, the predicted video correctly captures \textcolor{brown}{the rotation of the arm to match the cube’s orientation}, a meticulous step for a successful grasping.
The actual execution precisely mirrors this behavior, \textcolor{brown}{rotating by the correct angle to match the cube’s orientation} and enable a successful grasping.
This consistency highlights not only the model’s fine-grained scene understanding from video pretraining, but also its effectiveness in transferring that understanding into action planning.
\subsubsection{Attention Map}
We visualize the attention maps of PAR to better understand how it attends to visual and action cues during prediction, as shown in~\cref{fig:attn_map}. At the token level, different attention heads selectively focus on frame tokens or action tokens, indicating that the model effectively captures the interplay between perception and action for joint modeling. At the pixel level, the attention is concentrated on task-critical regions, such as the cube, target area, and the robotic arm, demonstrating that PAR can attend to spatially relevant features that directly influence task execution.
\subsubsection{Failure Case Study}
We illustrate a failure case in~\cref{fig:failure_case}.
While the predicted trajectory seems valid, the robot misses the green target when moving the red cube.
Upon inspecting the last few steps of the manipulation, we observe that the robot arm performs light movements around the target area, failing to find the correct placement along the depth axis.
We hypothesis that this is because the ManiSkill Benchmark setup only provides single-view RGB observations, making it difficult to infer the depth information between the end-effector and the goal area.
Future works could incorporate depth-estimation module (e.g. \cite{ze20243d, Yuan2025b, li2025bridgevla}) to enhance spatial awareness.
% This failure case highlights the limitation of single-view observation input and suggests a potential avenue for further research.
\begin{figure}[t]
\centering
\includegraphics[width=0.7\linewidth]{figure/failure_case.pdf}
\caption{\textbf{A failure case} in the PickCube task. The top row shows the predicted video while the bottom row presents the actual execution video. The last two columns provide zoomed-in views of the final few steps of the execution. The robot arm fails to find the correct placement (marked by the green point) along the depth axis.}
\label{fig:failure_case}
\end{figure}
\section{Conclusion}
\label{sec:conclusion}
This work introduces the Physical Autoregressive Model (PAR), a unified framework that autoregressively models the coupled dynamics of visual observations and robotic actions. By transferring world knowledge from video pretraining, PAR captures physical dynamics without action pretraining, enabling accurate video prediction and coherent action generation. Our design leverages continuous token representations for both frames and actions, facilitating fine-grained modeling and cross-modal interaction.
Additionally, we incorporate a causal mask with implicit inverse kinematics, along with efficient training and inference strategies.
% , and two complementary inference strategies.
Our experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task and matches the performance of action-pretrained baselines on other tasks.
It further demonstrates strong alignment between predicted video and action trajectories, evidencing successful transfer of world knowledge from video pretraining to action generation.
Ablation studies further validate the necessity of each component in our design.
Overall, by leveraging physical autoregression from video pretraining, our method suggests a promising new direction for addressing data scarcity in action pretraining.
\noindent{\textbf{Limitations.}}
In this work, we adopt full-parameter finetuning to showcase our approach. However, as a transformer-based model, PAR should support parameter-efficient finetuning like LoRA to achieve comparable performance at lower training costs, which we leave for future exploration.
\begin{thebibliography}{51}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Black et~al.(2024)Black, Brown, Driess, Esmail, Equi, Finn, Fusai, Groom, Hausman, Ichter, Jakubczak, Jones, Ke, Levine, Li-Bell, Mothukuri, Nair, Pertsch, Shi, Tanner, Vuong, Walling, Wang, and Zhilinsky]{black2024pi0}
K.~Black, N.~Brown, D.~Driess, A.~Esmail, M.~Equi, C.~Finn, N.~Fusai, L.~Groom, K.~Hausman, B.~Ichter, S.~Jakubczak, T.~Jones, L.~Ke, S.~Levine, A.~Li-Bell, M.~Mothukuri, S.~Nair, K.~Pertsch, L.~X. Shi, J.~Tanner, Q.~Vuong, A.~Walling, H.~Wang, and U.~Zhilinsky.
\newblock $\pi_0$: A vision-language-action flow model for general robot control, 2024.
\newblock URL \url{https://arxiv.org/abs/2410.24164}.
\bibitem[Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, et~al.]{brown2020language}
T.~Brown, B.~Mann, N.~Ryder, M.~Subbiah, J.~D. Kaplan, P.~Dhariwal, A.~Neelakantan, P.~Shyam, G.~Sastry, A.~Askell, et~al.
\newblock Language models are few-shot learners.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 1877--1901, 2020.
\bibitem[Bu et~al.(2025{\natexlab{a}})Bu, Cai, Chen, Cui, Ding, Feng, Gao, He, Hu, Huang, et~al.]{bu2025agibot}
Q.~Bu, J.~Cai, L.~Chen, X.~Cui, Y.~Ding, S.~Feng, S.~Gao, X.~He, X.~Hu, X.~Huang, et~al.
\newblock Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems.
\newblock \emph{arXiv preprint arXiv:2503.06669}, 2025{\natexlab{a}}.
\bibitem[Bu et~al.(2025{\natexlab{b}})Bu, Yang, Cai, Gao, Ren, Yao, Luo, and Li]{bu2025univla}
Q.~Bu, Y.~Yang, J.~Cai, S.~Gao, G.~Ren, M.~Yao, P.~Luo, and H.~Li.
\newblock Univla: Learning to act anywhere with task-centric latent actions.
\newblock \emph{arXiv preprint arXiv:2505.06111}, 2025{\natexlab{b}}.
\bibitem[Cen et~al.(2025)Cen, Yu, Yuan, Jiang, Huang, Guo, Li, Song, Luo, Wang, et~al.]{cen2025worldvla}
J.~Cen, C.~Yu, H.~Yuan, Y.~Jiang, S.~Huang, J.~Guo, X.~Li, Y.~Song, H.~Luo, F.~Wang, et~al.
\newblock Worldvla: Towards autoregressive action world model.
\newblock \emph{arXiv preprint arXiv:2506.21539}, 2025.
\bibitem[Cheang et~al.(2024)Cheang, Chen, Jing, Kong, Li, Li, Liu, Wu, Xu, Yang, et~al.]{cheang2024gr}
C.-L. Cheang, G.~Chen, Y.~Jing, T.~Kong, H.~Li, Y.~Li, Y.~Liu, H.~Wu, J.~Xu, Y.~Yang, et~al.
\newblock Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation.
\newblock \emph{arXiv preprint arXiv:2410.06158}, 2024.
\bibitem[Chi et~al.(2023)Chi, Xu, Feng, Cousineau, Du, Burchfiel, Tedrake, and Song]{chi2023diffusion}
C.~Chi, Z.~Xu, S.~Feng, E.~Cousineau, Y.~Du, B.~Burchfiel, R.~Tedrake, and S.~Song.
\newblock Diffusion policy: Visuomotor policy learning via action diffusion.
\newblock \emph{The International Journal of Robotics Research}, page 02783649241273668, 2023.
\bibitem[Deng et~al.(2024)Deng, Pan, Diao, Luo, Cui, Lu, Shan, Qi, and Wang]{deng2024autoregressive}
H.~Deng, T.~Pan, H.~Diao, Z.~Luo, Y.~Cui, H.~Lu, S.~Shan, Y.~Qi, and X.~Wang.
\newblock Autoregressive video generation without vector quantization.
\newblock \emph{arXiv preprint arXiv:2412.14169}, 2024.
\bibitem[Dubey et~al.(2024)Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Yang, Fan, et~al.]{dubey2024llama}
A.~Dubey, A.~Jauhri, A.~Pandey, A.~Kadian, A.~Al-Dahle, A.~Letman, A.~Mathur, A.~Schelten, A.~Yang, A.~Fan, et~al.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv e-prints}, pages arXiv--2407, 2024.
\bibitem[Fan et~al.(2025)Fan, Yan, Bai, Wang, Zhang, Wang, and Li]{fan2025towards}
C.~Fan, F.~Yan, C.~Bai, J.~Wang, C.~Zhang, Z.~Wang, and X.~Li.
\newblock Towards a generalizable bimanual foundation policy via flow-based video prediction.
\newblock \emph{arXiv preprint arXiv:2505.24156}, 2025.
\bibitem[Fu et~al.(2024)Fu, Huang, Datta, Chen, Panitch, Liu, Li, and Goldberg]{fu2024context}
L.~Fu, H.~Huang, G.~Datta, L.~Y. Chen, W.~C.-H. Panitch, F.~Liu, H.~Li, and K.~Goldberg.
\newblock In-context imitation learning via next-token prediction.
\newblock \emph{arXiv preprint arXiv:2408.15980}, 2024.
\bibitem[Gao et~al.(2024)Gao, Shi, Zhang, Wang, and Xiao]{gao2024vid}
K.~Gao, J.~Shi, H.~Zhang, C.~Wang, and J.~Xiao.
\newblock Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models.
\newblock \emph{arXiv preprint arXiv:2406.10981}, 2024.
\bibitem[Guo et~al.(2025)Guo, Yang, Zhang, Song, Zhang, Xu, Zhu, Ma, Wang, Bi, et~al.]{guo2025deepseek}
D.~Guo, D.~Yang, H.~Zhang, J.~Song, R.~Zhang, R.~Xu, Q.~Zhu, S.~Ma, P.~Wang, X.~Bi, et~al.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2501.12948}, 2025.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho2020denoising}
J.~Ho, A.~Jain, and P.~Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{Advances in neural information processing systems}, 33:\penalty0 6840--6851, 2020.
\bibitem[Hu et~al.(2024)Hu, Guo, Wang, Chen, Wang, Zhang, Sreenath, Lu, and Chen]{hu2024video}
Y.~Hu, Y.~Guo, P.~Wang, X.~Chen, Y.-J. Wang, J.~Zhang, K.~Sreenath, C.~Lu, and J.~Chen.
\newblock Video prediction policy: A generalist robot policy with predictive visual representations.
\newblock \emph{arXiv preprint arXiv:2412.14803}, 2024.
\bibitem[Intelligence et~al.(2025)Intelligence, Black, Brown, Darpinian, Dhabalia, Driess, Esmail, Equi, Finn, Fusai, Galliker, Ghosh, Groom, Hausman, Ichter, Jakubczak, Jones, Ke, LeBlanc, Levine, Li-Bell, Mothukuri, Nair, Pertsch, Ren, Shi, Smith, Springenberg, Stachowicz, Tanner, Vuong, Walke, Walling, Wang, Yu, and Zhilinsky]{pi05vision}
P.~Intelligence, K.~Black, N.~Brown, J.~Darpinian, K.~Dhabalia, D.~Driess, A.~Esmail, M.~Equi, C.~Finn, N.~Fusai, M.~Y. Galliker, D.~Ghosh, L.~Groom, K.~Hausman, B.~Ichter, S.~Jakubczak, T.~Jones, L.~Ke, D.~LeBlanc, S.~Levine, A.~Li-Bell, M.~Mothukuri, S.~Nair, K.~Pertsch, A.~Z. Ren, L.~X. Shi, L.~Smith, J.~T. Springenberg, K.~Stachowicz, J.~Tanner, Q.~Vuong, H.~Walke, A.~Walling, H.~Wang, L.~Yu, and U.~Zhilinsky.
\newblock $\pi_{0.5}$: a vision-language-action model with open-world generalization, 2025.
\newblock URL \url{https://arxiv.org/abs/2504.16054}.
\bibitem[Javaheripi et~al.(2023)Javaheripi, Bubeck, Abdin, Aneja, Bubeck, Mendes, Chen, Del~Giorno, Eldan, Gopi, et~al.]{javaheripi2023phi}
M.~Javaheripi, S.~Bubeck, M.~Abdin, J.~Aneja, S.~Bubeck, C.~C.~T. Mendes, W.~Chen, A.~Del~Giorno, R.~Eldan, S.~Gopi, et~al.
\newblock Phi-2: The surprising power of small language models.
\newblock \emph{Microsoft Research Blog}, 1\penalty0 (3):\penalty0 3, 2023.
\bibitem[Kim et~al.(2024)Kim, Pertsch, Karamcheti, Xiao, Balakrishna, Nair, Rafailov, Foster, Lam, Sanketi, et~al.]{kim2024openvla}
M.~J. Kim, K.~Pertsch, S.~Karamcheti, T.~Xiao, A.~Balakrishna, S.~Nair, R.~Rafailov, E.~Foster, G.~Lam, P.~Sanketi, et~al.
\newblock Openvla: An open-source vision-language-action model.
\newblock \emph{arXiv preprint arXiv:2406.09246}, 2024.
\bibitem[Kirillov et~al.(2023)Kirillov, Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, Whitehead, Berg, Lo, et~al.]{kirillov2023segment}
A.~Kirillov, E.~Mintun, N.~Ravi, H.~Mao, C.~Rolland, L.~Gustafson, T.~Xiao, S.~Whitehead, A.~C. Berg, W.-Y. Lo, et~al.
\newblock Segment anything.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on computer vision}, pages 4015--4026, 2023.
\bibitem[Li et~al.(2025{\natexlab{a}})Li, Chen, Wu, Ma, Wu, Huang, Wang, Kong, and Tan]{li2025bridgevla}
P.~Li, Y.~Chen, H.~Wu, X.~Ma, X.~Wu, Y.~Huang, L.~Wang, T.~Kong, and T.~Tan.
\newblock Bridgevla: Input-output alignment for efficient 3d manipulation learning with vision-language models.
\newblock \emph{arXiv preprint arXiv:2506.07961}, 2025{\natexlab{a}}.
\bibitem[Li et~al.(2024{\natexlab{a}})Li, Liang, Wang, Luo, Chen, Liao, Wei, Deng, Xu, Zhang, et~al.]{li2024cogact}
Q.~Li, Y.~Liang, Z.~Wang, L.~Luo, X.~Chen, M.~Liao, F.~Wei, Y.~Deng, S.~Xu, Y.~Zhang, et~al.
\newblock Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2411.19650}, 2024{\natexlab{a}}.
\bibitem[Li et~al.(2025{\natexlab{b}})Li, Gao, Sadigh, and Song]{li2025unified}
S.~Li, Y.~Gao, D.~Sadigh, and S.~Song.
\newblock Unified video action model.
\newblock \emph{arXiv preprint arXiv:2503.00200}, 2025{\natexlab{b}}.
\bibitem[Li et~al.(2024{\natexlab{b}})Li, Tian, Li, Deng, and He]{li2024autoregressive}
T.~Li, Y.~Tian, H.~Li, M.~Deng, and K.~He.
\newblock Autoregressive image generation without vector quantization.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:\penalty0 56424--56445, 2024{\natexlab{b}}.
\bibitem[Liu et~al.(2025)Liu, Chen, An, Liu, Zhang, Gu, Li, Guo, Chen, Liu, et~al.]{liu2025hybridvla}
J.~Liu, H.~Chen, P.~An, Z.~Liu, R.~Zhang, C.~Gu, X.~Li, Z.~Guo, S.~Chen, M.~Liu, et~al.
\newblock Hybridvla: Collaborative diffusion and autoregression in a unified vision-language-action model.
\newblock \emph{arXiv preprint arXiv:2503.10631}, 2025.
\bibitem[Liu et~al.(2024)Liu, Wu, Li, Tan, Chen, Wang, Xu, Su, and Zhu]{liurdt}
S.~Liu, L.~Wu, B.~Li, H.~Tan, H.~Chen, Z.~Wang, K.~Xu, H.~Su, and J.~Zhu.
\newblock Rdt-1b: a diffusion foundation model for bimanual manipulation.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2024.
\bibitem[Mandlekar et~al.(2021)Mandlekar, Xu, Wong, Nasiriany, Wang, Kulkarni, Fei-Fei, Savarese, Zhu, and Mart{\'\i}n-Mart{\'\i}n]{mandlekar2021matters}
A.~Mandlekar, D.~Xu, J.~Wong, S.~Nasiriany, C.~Wang, R.~Kulkarni, L.~Fei-Fei, S.~Savarese, Y.~Zhu, and R.~Mart{\'\i}n-Mart{\'\i}n.
\newblock What matters in learning from offline human demonstrations for robot manipulation.
\newblock \emph{arXiv preprint arXiv:2108.03298}, 2021.
\bibitem[{NVIDIA Developer Team}(2023)]{NVIDIA2023LLM}
{NVIDIA Developer Team}.
\newblock Mastering llm techniques: Inference optimization, 11 2023.
\newblock URL \url{https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/}.
\bibitem[Peebles and Xie(2023)]{peebles2023scalable}
W.~Peebles and S.~Xie.
\newblock Scalable diffusion models with transformers.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on computer vision}, pages 4195--4205, 2023.
\bibitem[Su et~al.(2024)Su, Ahmed, Lu, Pan, Bo, and Liu]{su2024roformer}
J.~Su, M.~Ahmed, Y.~Lu, S.~Pan, W.~Bo, and Y.~Liu.
\newblock Roformer: Enhanced transformer with rotary position embedding.
\newblock \emph{Neurocomputing}, 568:\penalty0 127063, 2024.
\bibitem[Tao et~al.(2025)Tao, Xiang, Shukla, Qin, Hinrichsen, Yuan, Bao, Lin, Liu, kai Chan, Gao, Li, Mu, Xiao, Gurha, Rajesh, Choi, Chen, Huang, Calandra, Chen, Luo, and Su]{taomaniskill3}
S.~Tao, F.~Xiang, A.~Shukla, Y.~Qin, X.~Hinrichsen, X.~Yuan, C.~Bao, X.~Lin, Y.~Liu, T.~kai Chan, Y.~Gao, X.~Li, T.~Mu, N.~Xiao, A.~Gurha, V.~N. Rajesh, Y.~W. Choi, Y.-R. Chen, Z.~Huang, R.~Calandra, R.~Chen, S.~Luo, and H.~Su.
\newblock Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai.
\newblock \emph{Robotics: Science and Systems}, 2025.
\bibitem[Team et~al.(2024)Team, Ghosh, Walke, Pertsch, Black, Mees, Dasari, Hejna, Kreiman, Xu, et~al.]{team2024octo}
O.~M. Team, D.~Ghosh, H.~Walke, K.~Pertsch, K.~Black, O.~Mees, S.~Dasari, J.~Hejna, T.~Kreiman, C.~Xu, et~al.
\newblock Octo: An open-source generalist robot policy.
\newblock \emph{arXiv preprint arXiv:2405.12213}, 2024.
\bibitem[Tian et~al.(2024)Tian, Jiang, Yuan, Peng, and Wang]{tian2024visual}
K.~Tian, Y.~Jiang, Z.~Yuan, B.~Peng, and L.~Wang.
\newblock Visual autoregressive modeling: Scalable image generation via next-scale prediction.
\newblock \emph{Advances in neural information processing systems}, 37:\penalty0 84839--84865, 2024.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]{vaswani2017attention}
A.~Vaswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~N. Gomez, {\L}.~Kaiser, and I.~Polosukhin.
\newblock Attention is all you need.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Vuong et~al.(2025)Vuong, Vu, An, and Reid]{vuong2025action}
A.~D. Vuong, M.~N. Vu, D.~An, and I.~Reid.
\newblock Action tokenizer matters in in-context imitation learning.
\newblock \emph{arXiv preprint arXiv:2503.01206}, 2025.
\bibitem[Wang et~al.(2025)Wang, Li, Wang, Zhang, Li, Chen, Wang, and Zhang]{wang2025unified}
Y.~Wang, X.~Li, W.~Wang, J.~Zhang, Y.~Li, Y.~Chen, X.~Wang, and Z.~Zhang.
\newblock Unified vision-language-action model.
\newblock \emph{arXiv preprint arXiv:2506.19850}, 2025.
\bibitem[Wen et~al.(2025{\natexlab{a}})Wen, Zhu, Li, Tang, Shen, and Feng]{wen2025dexvla}
J.~Wen, Y.~Zhu, J.~Li, Z.~Tang, C.~Shen, and F.~Feng.
\newblock Dexvla: Vision-language model with plug-in diffusion expert for general robot control.
\newblock \emph{arXiv preprint arXiv:2502.05855}, 2025{\natexlab{a}}.
\bibitem[Wen et~al.(2025{\natexlab{b}})Wen, Zhu, Li, Zhu, Tang, Wu, Xu, Liu, Cheng, Shen, et~al.]{wen2025tinyvla}
J.~Wen, Y.~Zhu, J.~Li, M.~Zhu, Z.~Tang, K.~Wu, Z.~Xu, N.~Liu, R.~Cheng, C.~Shen, et~al.
\newblock Tinyvla: Towards fast, data-efficient vision-language-action models for robotic manipulation.
\newblock \emph{IEEE Robotics and Automation Letters}, 2025{\natexlab{b}}.
\bibitem[Wu et~al.(2023)Wu, Jing, Cheang, Chen, Xu, Li, Liu, Li, and Kong]{wu2023unleashing}
H.~Wu, Y.~Jing, C.~Cheang, G.~Chen, J.~Xu, X.~Li, M.~Liu, H.~Li, and T.~Kong.
\newblock Unleashing large-scale video generative pre-training for visual robot manipulation.
\newblock \emph{arXiv preprint arXiv:2312.13139}, 2023.
\bibitem[Xie et~al.(2025)Xie, Xu, Hong, Tan, Liu, Liu, Kaufman, and Zhou]{xie2025progressive}
D.~Xie, Z.~Xu, Y.~Hong, H.~Tan, D.~Liu, F.~Liu, A.~Kaufman, and Y.~Zhou.
\newblock Progressive autoregressive video diffusion models.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pages 6322--6332, 2025.
\bibitem[Xu et~al.(2025)Xu, Zhang, Guo, Wen, Yang, Lin, Huang, Li, Zhang, Wang, et~al.]{xu2025a0}
R.~Xu, J.~Zhang, M.~Guo, Y.~Wen, H.~Yang, M.~Lin, J.~Huang, Z.~Li, K.~Zhang, L.~Wang, et~al.
\newblock A0: An affordance-aware hierarchical model for general robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2504.12636}, 2025.
\bibitem[Ye et~al.(2025)Ye, Jang, Jeon, Joo, Yang, Peng, Mandlekar, Tan, Chao, Lin, et~al.]{yelatent}
S.~Ye, J.~Jang, B.~Jeon, S.~J. Joo, J.~Yang, B.~Peng, A.~Mandlekar, R.~Tan, Y.-W. Chao, B.~Y. Lin, et~al.
\newblock Latent action pretraining from videos.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\bibitem[Yu et~al.(2023)Yu, Lezama, Gundavarapu, Versari, Sohn, Minnen, Cheng, Birodkar, Gupta, Gu, et~al.]{yu2023language}
L.~Yu, J.~Lezama, N.~B. Gundavarapu, L.~Versari, K.~Sohn, D.~Minnen, Y.~Cheng, V.~Birodkar, A.~Gupta, X.~Gu, et~al.
\newblock Language model beats diffusion--tokenizer is key to visual generation.
\newblock \emph{arXiv preprint arXiv:2310.05737}, 2023.
\bibitem[Yuan et~al.(2024)Yuan, Cao, Li, Jiang, and Wang]{SD-MVS}
Z.~Yuan, J.~Cao, Z.~Li, H.~Jiang, and Z.~Wang.
\newblock {{SD-MVS}}: {{Segmentation-Driven Deformation Multi-View Stereo}} with {{Spherical Refinement}} and {{EM Optimization}}.
\newblock In \emph{Proceedings of the {{AAAI Conference}} on {{Artificial Intelligence}}}, volume~38, pages 6871--6880, 2024.
\bibitem[Yuan et~al.(2025)Yuan, Liu, Shen, Li, Luo, Mao, and Wang]{Yuan2025b}
Z.~Yuan, C.~Liu, F.~Shen, Z.~Li, J.~Luo, T.~Mao, and Z.~Wang.
\newblock {{MSP-MVS}}: {{Multi-granularity}} segmentation prior guided multi-view stereo.
\newblock In \emph{Proceedings of the {{AAAI Conference}} on {{Artificial Intelligence}}}, volume~39, pages 9753--9762, 2025.
\bibitem[Ze et~al.(2024)Ze, Zhang, Zhang, Hu, Wang, and Xu]{ze20243d}
Y.~Ze, G.~Zhang, K.~Zhang, C.~Hu, M.~Wang, and H.~Xu.
\newblock 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.
\newblock \emph{arXiv preprint arXiv:2403.03954}, 2024.
\bibitem[Zhang et~al.(2025)Zhang, Hu, Qiao, Zhang, Qin, Li, Liu, Kong, Liu, and Ma]{zhang2025chain}
W.~Zhang, T.~Hu, Y.~Qiao, H.~Zhang, Y.~Qin, Y.~Li, J.~Liu, T.~Kong, L.~Liu, and X.~Ma.
\newblock Chain-of-action: Trajectory autoregressive modeling for robotic manipulation.
\newblock \emph{arXiv preprint arXiv:2506.09990}, 2025.
\bibitem[Zhao et~al.(2025)Zhao, Lu, Kim, Fu, Zhang, Wu, Li, Ma, Han, Finn, et~al.]{zhao2025cot}
Q.~Zhao, Y.~Lu, M.~J. Kim, Z.~Fu, Z.~Zhang, Y.~Wu, Z.~Li, Q.~Ma, S.~Han, C.~Finn, et~al.
\newblock Cot-vla: Visual chain-of-thought reasoning for vision-language-action models.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pages 1702--1713, 2025.
\bibitem[Zhao et~al.(2023)Zhao, Kumar, Levine, and Finn]{zhao2023learning}
T.~Z. Zhao, V.~Kumar, S.~Levine, and C.~Finn.
\newblock Learning fine-grained bimanual manipulation with low-cost hardware.
\newblock \emph{arXiv preprint arXiv:2304.13705}, 2023.
\bibitem[Zheng et~al.(2024)Zheng, Peng, Yang, Shen, Li, Liu, Zhou, Li, and You]{zheng2024open}
Z.~Zheng, X.~Peng, T.~Yang, C.~Shen, S.~Li, H.~Liu, Y.~Zhou, T.~Li, and Y.~You.
\newblock Open-sora: Democratizing efficient video production for all.
\newblock \emph{arXiv preprint arXiv:2412.20404}, 2024.
\bibitem[Zhong et~al.(2025)Zhong, Huang, Li, Zhang, Liang, Yang, and Chen]{zhong2025dexgraspvla}
Y.~Zhong, X.~Huang, R.~Li, C.~Zhang, Y.~Liang, Y.~Yang, and Y.~Chen.
\newblock Dexgraspvla: A vision-language-action framework towards general dexterous grasping.
\newblock \emph{arXiv preprint arXiv:2502.20900}, 2025.
\bibitem[Zitkovich et~al.(2023)Zitkovich, Yu, Xu, Xu, Xiao, Xia, Wu, Wohlhart, Welker, Wahid, et~al.]{zitkovich2023rt}
B.~Zitkovich, T.~Yu, S.~Xu, P.~Xu, T.~Xiao, F.~Xia, J.~Wu, P.~Wohlhart, S.~Welker, A.~Wahid, et~al.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control.
\newblock In \emph{Conference on Robot Learning}, pages 2165--2183. PMLR, 2023.
\end{thebibliography}