DIDO: Distilling Interaction-Centric Dynamics into One-Step Denoising for World Action Models

For the manipulated object, we follow the automatic annotation pipeline of LaRA-VLA Bai et al. (2026a). The target entity is first parsed from the language instruction and localized in a reference frame using an open-vocabulary detector. A video tracker then propagates the target through the sequence, producing dense frame-level bounding boxes.

For the gripper, LIBERO and RoboTwin provide its pose through simulator state, which is projected into image space to obtain the corresponding bounding boxes. For Galbot G1 in real-world experiments, where such state information is unavailable, the gripper is detected using Grounding DINO Liu et al. (2024) and propagated through the sequence with SAM 3 Carion et al. (2026). Tracking is performed in both temporal directions, and the higher-confidence prediction is retained at each frame. Missing intermediate boxes are filled by linear interpolation between valid detections.

The manipulated-object annotations are also used to construct the targets for multi-layer visual alignment. For each frame, we crop the target-object region using its annotated bounding box, enlarge the crop by a fixed margin to retain local context, and resize it to the DINOv3 input resolution. A frozen DINOv3 encoder Siméoni et al. (2026) then extracts a sequence of object-centric visual tokens, which serve as fixed targets for the dedicated alignment tokens $\mathcal{T}_{\mathrm{align}}$. Cropping before encoding keeps the supervision object-specific and reduces interference from static background content.

We audit the automatically generated bounding-box annotations at the trajectory level. On LIBERO, the target-object boxes remain correct throughout 95 of 100 sampled trajectories, while 5 exhibit tracking drift; the gripper boxes are correct in all sampled trajectories. On RoboTwin, the target-object boxes are correct in 181 of 200 sampled trajectories, with 9 detection failures and 10 tracking drifts, while the gripper boxes are correct in all cases. On Galbot G1, all 40 sampled trajectories, with 10 per task, contain correct annotations for both the target object and gripper. These results indicate that the automatic pipeline provides reliable supervision for interaction-centric visual reasoning.

DIDO builds on the pretrained Wan2.2-TI2V-5B Wan et al. (2025) video generation model and is initialized from its publicly released four-step distilled checkpoint Li et al. (2025b). We further adapt this checkpoint to robot manipulation demonstrations, yielding a robot-domain four-step teacher used for distribution matching in Stage I. The VAE decoder is used only to visualize predicted future frames and is not required during policy inference.

We distill the fine-tuned four-step teacher into a one-step generator following the distribution-matching principle of DMD2 Yin et al. (2024a). We denote the student generator by $G_{\theta}$, the frozen teacher by $f_{\mathrm{T}}$, and the online fake-score model by $f_{\mathrm{F}}$. All three networks are initialized from the same robot-domain four-step checkpoint. During distillation, $f_{\mathrm{T}}$ remains frozen, while $G_{\theta}$ and $f_{\mathrm{F}}$ are optimized alternately.

Given conditioning information $c$ and an initial noisy latent $z$, the student predicts the future-video latent in a single network evaluation:

Let $p_{\theta}$ denote the distribution induced by the one-step student and $p_{\mathrm{T}}$ the target distribution represented by the frozen teacher. Rather than reproducing the teacher’s multi-step sampling trajectory, we match their perturbed distributions:

where $p_{\theta,t}$ and $p_{\mathrm{T},t}$ denote the corresponding distributions at noise level $t$.

For a noisy distribution $p_{t}$, its score is defined as $s(x_{t},t)=\nabla_{x_{t}}\log p_{t}(x_{t})$. The gradient of Eq. (12) with respect to the student parameters can be expressed using the difference between the fake and target scores:

where timestep-dependent positive weighting factors are omitted for clarity. The score difference therefore provides a distribution-level direction for aligning the student with the teacher without explicitly evaluating probability densities.

Because the four-step teacher is trained for a discrete denoising schedule, we evaluate its score at noise levels aligned with this schedule. Specifically, we sample a nominal timestep from

and map it to the corresponding scheduler timestep $t=w(\tau)$.

Given the student output $x_{g}$, we construct a noisy sample using the flow-matching forward process:

The same noisy sample is evaluated by the frozen teacher and the online fake-score model:

where the original Wan flow predictions are converted to clean-latent predictions before computing the distribution-matching signal.

For the corruption process in Eq. (15), an $x_{0}$ predictor induces the score

Therefore, for $0<\sigma_{t}<1$, the fake and teacher score difference becomes

Thus, the difference between their clean-latent predictions provides the same gradient direction as the theoretical score difference up to a timestep-dependent positive scaling factor.

Following the practical DMD formulation, we use the normalized gradient surrogate

which reduces variation in gradient magnitude across noise levels. We inject $g_{\mathrm{DMD}}$ into $G_{\theta}$ through a stop-gradient surrogate objective such that the resulting gradient with respect to $x_{g}$ is proportional to $g_{\mathrm{DMD}}$. This aligns the one-step student with the distribution represented by the four-step teacher without requiring the student to reproduce its sampling trajectory.

As the student distribution evolves during training, the fake-score model $f_{\mathrm{F}}$ is updated online to track the current generator distribution. Given a student sample $x_{g}$, we independently sample a score timestep, perturb $x_{g}$ according to Eq. (15), and optimize

where $\operatorname{sg}(\cdot)$ denotes the stop-gradient operator. Following DMD2, $f_{\mathrm{F}}$ is updated more frequently than $G_{\theta}$ so that it can track the evolving student distribution and provide a stable distribution-matching signal.

During training, the frozen four-step teacher provides the target score estimate, while $f_{\mathrm{F}}$ estimates the score of the current one-step student distribution. At inference time, both score models are discarded and only $G_{\theta}$ is retained, reducing four-step future prediction to a single network evaluation.

DIDO uses 64 interaction-centric tokens divided into four groups of 16: object tokens $\mathcal{T}_{\mathrm{obj}}$, interaction tokens $\mathcal{T}_{\mathrm{int}}$, gripper tokens $\mathcal{T}_{\mathrm{grip}}$, and alignment tokens $\mathcal{T}_{\mathrm{align}}$. These tokens are inserted into the video transformer and evolve jointly with its latent representations.

For trajectory prediction, $[\mathcal{T}_{\mathrm{obj}};\mathcal{T}_{\mathrm{int}}]$ and $[\mathcal{T}_{\mathrm{grip}};\mathcal{T}_{\mathrm{int}}]$ predict future object and gripper boxes, respectively. Each branch uses a two-layer MLP with GELU activations and hidden width equal to the video-transformer width, predicting normalized bounding-box coordinates for $H_{a}=32$ future steps. For visual alignment, the representations $\mathcal{T}_{\mathrm{align}}^{l}$ at selected transformer layers are mapped by lightweight projections $P_{l}$ to the DINOv3 feature space and supervised by Equation 5. DINOv3 is used only to construct training targets and is removed at inference.

The action expert has the same depth as the video model and one quarter of its hidden width. It is coupled to the video backbone through the mixture-of-transformers architecture described in Section 3.3. Its output projection matches the action dimension of each embodiment, which is 7 for LIBERO, 14 for dual-arm RoboTwin, and 14 for Galbot G1. The action expert predicts a chunk of $H_{a}=32$ actions per forward pass. During inference, the flow field is integrated with 10 Euler steps, and the first 10 actions on LIBERO and 24 actions on RoboTwin are executed before replanning.

The video model and action expert are coupled through shared attention at corresponding transformer layers, as shown in Figure 6. In Stage I, interaction-centric, current-frame, and future-frame tokens attend within the world-model stream. In Stage II, action tokens are introduced and can attend to all world-model representations and action tokens. World-model tokens remain masked from the action stream, preserving the predictive pathway learned in Stage I. This asymmetric attention gives the action expert direct access to intermediate world-model representations without decoding future observations.

Before the two-stage training of DIDO, we adapt the publicly released four-step Wan2.2-TI2V-5B checkpoint to robot manipulation videos. Interaction-centric supervision is enabled during this adaptation, with the corresponding warm-up schedules specified in Table 4. The resulting robot-domain four-step model serves as the teacher for Stage I and as the four-step fine-tuned baseline in Section 4.2.

The single-step generator and both DMD score networks are initialized from the robot-domain teacher. The real score network remains frozen, while the fake score network is updated every iteration and the generator every fifth. The generator operates at timestep $1000$, while score-network timesteps are sampled uniformly from $\{1000,750,500,250\}$, with one timestep sampled per training example and shared across its latent frames.

The fake score network uses no classifier-free guidance. The real score network is evaluated as

corresponding to a guidance scale of $7$ under the standard convention $\epsilon_{\mathrm{uncond}}+s(\epsilon_{\mathrm{cond}}-\epsilon_{\mathrm{uncond}})$. We use the normalized DMD gradient and backward simulation without an explicit adversarial discriminator or GAN objective.

In Stage II, the text encoder, VAE, and DINOv3 encoder remain frozen, while the video-transformer backbone, interaction-centric tokens, their prediction and projection heads, action expert, and proprioception encoder are jointly optimized. The DMD objective is removed and replaced by the video-generation objective following Fast-WAM Yuan et al. (2026), retaining future prediction as a training signal during policy learning.

Let $z$ denote the ground-truth future video latents encoded by the frozen VAE, and let $\mathcal{C}_{\mathrm{vid}}$ denote the video-model conditioning. We sample $\tau\sim U[0,1]$ and $\epsilon_{z}\sim\mathcal{N}(0,I)$, and construct $z_{\tau}=(1-\tau)\epsilon_{z}+\tau z$ with target velocity $u_{z}=z-\epsilon_{z}$. The video-generation objective is

where $v_{\phi}$ denotes the video model. Video generation, trajectory supervision, visual alignment, and action prediction are jointly optimized during Stage II.

For each predicted bounding box, the smooth-$L_{1}$ term uses $\beta=1.0$, while the GIoU term is weighted by $\lambda_{\mathrm{giou}}$. The resulting object and gripper box losses are further scaled by $\lambda_{\mathrm{obj}}$ and $\lambda_{\mathrm{grip}}$, respectively, to form $\mathcal{L}_{\mathrm{inter}}$. The multi-layer visual alignment loss $\mathcal{L}_{\mathrm{align}}$ is weighted by $\lambda_{\mathrm{align}}$. In Stage II, the video-generation loss $\mathcal{L}_{\mathrm{video}}$ and action loss $\mathcal{L}_{\mathrm{act}}$ are additionally weighted by $\lambda_{\mathrm{video}}$ and $\lambda_{\mathrm{act}}$, respectively. All loss weights and optimization hyperparameters are summarized in Table 4.

All experiments are trained on NVIDIA H100 GPUs with the global batch sizes reported in Table 4. For the simulation benchmarks, LIBERO requires approximately 15 hours for teacher preparation, 5 hours for Stage I, and 7 hours for Stage II. RoboTwin requires approximately 25 hours, 12 hours, and 100 hours for the three phases, respectively, primarily due to its larger demonstration set and optimization budget.

LIBERO, RoboTwin, and LIBERO-Plus follow their standard evaluation protocols. Performance is reported as task success rate averaged over evaluation episodes.

As shown in Figure 7, we evaluate DIDO on four real-world manipulation tasks: (i) Corn on Plate, placing the corn onto the plate; (ii) Bowl on Plate and Pepper in Bowl, placing the bowl on the plate and then putting the pepper in the bowl; (iii) Wipe Stain and Store Cloth, wiping the stain with a cloth and then putting the cloth in the basket; and (iv) Open Drawer and Place Duck, opening the upper drawer and putting the little duck inside. The first is a single-goal task evaluated with a three-stage rubric, while the remaining three are long-horizon tasks evaluated with four-stage rubrics.

Binary success does not distinguish between failures occurring at different stages of a task Ji et al. (2026). We therefore use a staged progress score, where each rollout receives the score associated with the furthest completion stage reached. Each task is scored on a 10-point scale: the single-goal task uses stages worth 3, 6, and 10 points, while the long-horizon tasks use stages worth 2, 5, 7, and 10 points. Figure 7 illustrates the target state of each stage. We conduct 10 rollouts per task under each evaluation protocol, resulting in 80 rollouts in total, and report the mean score as a percentage of the maximum possible score.

Since public results are unavailable for our real-robot platform, all real-world baselines are trained using the same demonstrations, observation and action spaces, and training budget as DIDO. Each baseline is trained separately for each task using its released implementation and recommended hyperparameters. $\pi_{0.5}$ Physical Intelligence et al. (2025) is initialized from its official pretrained weights, while Fast-WAM Yuan et al. (2026) is trained on our robot demonstrations. This ensures a consistent data and evaluation setting across methods.

For each token group $g$, we first compute its share of total action attribution as

where a larger $A_{g}$ indicates that the group contributes more strongly to action generation. Since groups contain different numbers of tokens, however, $A_{g}$ alone cannot distinguish token importance from group size. We therefore normalize the attribution share by the token-frequency share and define the per-token attribution enrichment as

where $N_{g}$ is the number of tokens in group $g$ and $N$ is the total number of tokens. $E_{g}=1$ indicates attribution proportional to token frequency, while $E_{g}>1$ indicates that tokens in group $g$ contribute more to action generation per token than expected from their frequency. We report means and 95% confidence intervals using task-level bootstrap with 10K resamples over 50 clips and 31 tasks.

Figure 8 shows that action attribution is highly concentrated in the interaction-centric tokens. Although they constitute only 5.74% of the cache, they account for 33.9% of total action attribution, corresponding to approximately $5.9\times$ greater attribution per token than expected from their frequency. This indicates that the action expert places disproportionate emphasis on these compact interaction representations.

This pattern is consistent across the individual interaction-centric groups. Object, Interaction, Gripper, and Alignment tokens account for 4.7%, 4.7%, 5.8%, and 18.7% of total action attribution, respectively, with per-token enrichments of $6.03\times$, $5.98\times$, $7.32\times$, and $5.60\times$. In contrast, Current Visual and Future Visual tokens achieve enrichments of only $0.61\times$ and $0.73\times$, despite comprising most of the cache. These results suggest that the interaction-centric tokens concentrate action-relevant information into a small token budget, complementing the broader scene and future context provided by the visual tokens.