LIFD: Anchored Diffusion for
3D-Aware Scene Memory in Robotic ManipulationThanks: ^†Corresponding author: Qingyao Wu.

Visual pretraining supplies transferable features for visuomotor learning [10, 11]. Point-cloud and voxel policies expose spatial structure for manipulation [1, 2, 3]. Geometry-aware image encoders such as VGGT predict camera parameters, depth, point maps, and tracks from image collections [4]. LIFD uses these geometric priors to learn a scene-token representation for RGB deployment, with multi-view supervision available during training.

Memory-based policies address observation ambiguity through different historical interfaces. TraceVLA overlays tracked point trajectories on current images [12], while MemoryVLA stores perceptual and semantic features for retrieval and fusion [6]. SAM2Act+ extends a 3D manipulation policy with memories of visual features and predicted action heatmaps [5]. BridgeVLA++ combines temporal context with spatial memory constructed from initial point clouds [7]. These approaches already connect spatial perception and observation history for control. LIFD addresses this problem through generative completion: recurrent history and current RGB features jointly condition predictions of a multi-view scene-token target.

World models learn environment representations and dynamics for planning and control [8, 13, 9]. Diffusion-based policies generate actions or trajectories [14, 15], and WorldVLA jointly models action generation and future images [16]. LIFD uses generation to infer a current scene representation for downstream control. Its multi-view targets build on set-latent aggregation [17, 18], while a rectified-flow objective [19] learns conditional completion in token space. The generated scene tokens are conditioned on observation history and visible evidence; they are not action-conditioned future predictions.

From local geometry to scene-level targets. GeoVGGT-R is obtained by LoRA-adapting a pretrained VGGT backbone [4] with camera, depth, point-map, and tracking supervision, and is frozen in subsequent stages. These geometric priors make the RGB features sensitive to spatial structure, while multi-view aggregation supplies information beyond an individual view. The supervision heads and loss definitions are given in Appendix A.

For views $\mathcal{I}=\{I_{a},I_{2},\ldots,I_{n}\}$, every training subset contains the same anchor image $I_{a}$; the number of support views increases progressively from one to $n-1$. For an anchor-containing subset $C\subseteq\mathcal{I}$, an aggregator $A_{\phi}$ uses learned queries to collect evidence into a fixed-length scene representation:

The shared anchor gives different subsets a common visual reference. To encourage agreement despite changing support views, two subsets $C_{A},C_{B}$ are aligned by a symmetric self-distillation objective [20, 21]:

The terms $\mathbf{p}_{B},\mathbf{z}_{A}$ swap $A$ and $B$. Here $E(C)$ collects the encoded views, $h_{\phi}$ is the online predictor, $\operatorname{sg}$ stops gradients, and $D$ is cosine distance between corresponding scene-token arrays. The target parameters follow an exponential moving average (EMA), $\phi^{\prime}\leftarrow\beta\phi^{\prime}+(1-\beta)\phi$ [20]. This stage defines the representation that the deployed model should infer: scene tokens learned from multiple views, rather than features of the current image alone. The trained teacher supplies completion targets $\mathbf{S}_{t}^{\star}$; its extra views are used only during learning and offline diagnostics.

Retaining history and incorporating current evidence. The completion model must infer the multi-view target from a partial observation. A gated recurrent unit (GRU) [22] retains observation history, and memory queries attend to current LST:

Thus history enters completion through features conditioned on the visible scene. This is learned feature alignment, without a known geometric transformation between views. Memory updates once per environment observation; we distinguish environment time $t$ from generative flow progress $u\in[0,1]$.

Visible-feature anchoring. Historical evidence can inform missing content, but the current view must continue to condition the generated representation. After mapping the two feature arrays to compatible dimensions, FiLM modulation [23] combines them as

With $0\leq\alpha_{\min}\leq\alpha_{\max}\leq 1$, the weight on current visual evidence is nondecreasing along the flow. Anchor-Guided Cross-Attention (AGCA) supplies this condition to every block of a DiT-style transformer [24]. For evolving hidden tokens $\mathbf{H}$,

Here MHA and LN denote multi-head attention and layer normalization. Self-attention exchanges information among scene tokens, while cross-attention repeatedly exposes them to $\mathbf{C}_{t}(u)$. This provides learned anchoring throughout generation, without imposing exact geometric consistency.

Learning to complete scene tokens. The network learns a rectified flow [19] from Gaussian noise $\boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$ to the teacher representation:

The expectation covers training observations, noise, and sampled flow progress. At inference, integrating the learned vector field from $u=0$ to $u=1$, with classifier-free guidance [25], produces $\widehat{\mathbf{S}}_{t}$. The number of integration steps $J$ controls inference cost; within these steps, memory stays fixed while its weighting against visible features changes with $u$. The generated tokens represent the current scene for control; the recurrence in Eq. (3) accumulates observed LST. Appendix A gives the attention, modulation, and guidance details.

The Focus module applies slot attention [34] to $\widehat{\mathbf{S}}_{t}$, producing $K_{s}$ compact slot features $\mathbf{Z}_{t}\in\mathbb{R}^{K_{s}\times d}$. Slots compete for scene-token evidence and iteratively aggregate it, providing a compact interface between scene completion and control. Appendix A specifies scene-conditioned initialization and slot updates. This grouping does not require each slot to correspond to one physical object. The Do module predicts an action from slots, proprioception, and language:

During Staged policy learning, perception and completion remain frozen, while slot attention and the policy head learn from demonstrations:

Here $\mathbf{A}_{t}^{\star}$ is the demonstration action and $\lambda_{\rm tv}\geq 0$ weights temporal smoothness. The first term imitates actions and the second penalizes abrupt changes. Separating this objective from representation learning allows the same scene representation to support downstream policy adaptation.

Joint versus Staged training. GeoVGGT-R remains frozen in both schedules. Staged first learns the multi-view teacher, then recurrent-memory-conditioned RF completion toward that frozen teacher, and finally slots/policy with perception/completion frozen. Joint simultaneously optimizes aggregation, RF/recurrent-memory completion, and slots/policy, combining representation, completion, and control objectives. This tests representation-first against simultaneous optimization.

LIFD (Staged) achieves 91.6% LIBERO average success and 79.8% MetaWorld success (Table I). MetaWorld offers complementary validation: LIFD is competitive but 1.8 points below $\pi_{0.5}$ (81.6%) and below the strongest listed references. Staged improves on Joint by 2.8–3.2 points across LIBERO suites, 3.1 on average, and 2.4 on MetaWorld, supporting representation-first learning.

RoboTwin uses 50 clean demonstrations/task and 100 test rollouts/task/condition across 50 tasks [37], without randomized-domain training. Hard (Clean$\rightarrow$Random) precedes Easy (Clean$\rightarrow$Clean). Staged achieves 19.4%/63.2% versus Joint’s 16.8%/59.6%, improving by 2.6/3.6 points. Hard tests direct generalization to randomized environments; its 43.8-point decrease from Easy shows remaining sensitivity to this shift.

Scene consistency. The Scene Contradiction Index (SCI, lower is better) measures generated-token agreement with a frozen multi-view teacher after Hungarian matching. It is a teacher-relative latent diagnostic, not an independent measure of geometric accuracy; task success remains primary. Table I reports LIBERO-Spatial SCI as mean $\pm$ SE over three training seeds (aggregation: Appendix B); baselines without compatible tokens have no entry. Staged reduces SCI from 0.20 to 0.14 alongside its control gains.

Completion and anchoring. Table II tests LIBERO-Spatial components against the 93.6% full-model result in Table I. FiLM-only (no RF) retains current-view and memory FiLM fusion but removes generative completion, passing the deterministic representation directly to scene readout/control. Success falls by 62.8 points and SCI rises from 0.14 to 0.62. Without anchor removes direct current-view conditioning but retains memory-conditioned AGCA ($-23.7$ points); without memory retains only current-view conditioning ($-18.8$ points). The Half-blind teacher uses the deployed policy’s same-view evidence instead of richer multi-view evidence ($-42.5$ points). Removing iterative object-centric slot grouping uses direct, non-slot global policy readout ($-23.3$ points). Without CFG retains the full model and training-time condition dropout but uses the conditional vector field alone at inference; its 28.4-point decrease supports guidance for reliable completion.

Occlusion and memory interventions. In LIBERO-Spatial, random square occluders cover 0–60% of each image in 10% increments. With identical masks covering at least 40%, LIFD retains an 8–10-point success advantage over OpenVLA-7B. This tests missing-pixel robustness, not hidden-object reconstruction. On LIBERO-Long, zeroing memory at steps 10, 20, or 40 with weights and noise schedules fixed produces the largest reported decrease at step 20. This dependence supports accumulated history’s control utility without identifying its encoded physical quantities.

Fig. 3 applies a fixed gray ROI only to cup-stacking policy RGB at steps 12–24, releasing it immediately after step 24; the first displayed re-observation is step 28. Full LIFD has the smallest masked-period contradiction increase ($\Delta\mathrm{SCI}_{\rm mask}$: 0.07 versus 0.16 without memory and 0.11 without anchor) and fastest recovery after release (four versus ten and twelve control steps). Memory removal causes the largest degradation during occlusion; anchor removal leaves a longer stale-state tail after observations return. These are teacher-relative consistency/recovery readouts, not metric hidden-geometry reconstruction.

The physical evaluation uses a UR5e with a laterally mounted Intel RealSense D435i RGB stream across twelve instances in four task families (Fig. 2): occluded-object extraction, ordered insertion/removal, hidden-target retrieval, and relocation under viewpoint changes. Ten human demonstrations per family adapt only the policy head; perception/completion remain frozen after representation learning with real images. Each method receives 50 independent trials/family spanning predefined variations in poses, distractors, lighting, and camera extrinsics (Appendix B).

LIFD achieves 56.0% macro-average success versus OpenVLA-7B’s 40.5%, a 15.5-point gain (Table III). SA has the largest gain, 30 points; HTR gains only 2 points (one additional success/50 trials). LR remains difficult: LIFD succeeds in 14/50 trials (28%) versus 8/50 (16%) for Diffusion Policy, that task’s strongest baseline.

Inference cost. We use 50 RF integration steps by default. On a single RTX 4090, reducing integration to 35 and 20 steps lowers end-to-end latency from 190 ms to 130 ms and 75 ms, respectively. The corresponding reciprocal throughputs are 5.3, 7.7, and 13.3 policy updates per second, distinct from the robot command-interface rate. Appendix B reports the full timing sweep (Table IV).