UniQuery4R: Unified 4D Scene Reconstruction from a Single Query

Classical 3D reconstruction is grounded in multi-view geometry. SfM and SLAM estimate camera motion together with scene structure, whereas MVS recovers dense geometry from calibrated views (30; 43). Learned feature matching and differentiable optimization have further improved the robustness and accuracy of these pipelines (29; 17; 34). Despite these advances, most classical reconstruction pipelines assume a static scene and retain iterative optimization, limiting their applicability to dynamic videos.

Feed-forward models instead infer geometry and camera parameters directly from images. DUSt3R showed that dense pointmaps can be recovered from unposed pairs without SfM (38), and MUSt3R/MASt3R-SfM extended this to multi-view and unconstrained settings (2; 3); VGGT unified camera, depth, point-map, and point-track prediction in one pass (36), later scaled by VGGT-$\Omega$ (37) and extended to online SLAM (22), while MonST3R generalized pointmaps to dynamic videos (46).

For dynamic scenes, Dynamic Point Maps (DPM) and V-DPM introduced temporally consistent pointmaps that jointly model static and dynamic geometry over time (32; 31), and D2USt3R folded temporal correspondence into 4D pointmaps (8). Recent feed-forward frameworks recover different combinations of geometry, motion, correspondence, and appearance for dynamic reconstruction (14; 15; 4; 41; 45; 21; 9); related foundation models such as Dens3R focus instead on static 3D geometry prediction (5). Most of these systems construct dense full-scene representations. Two conditional designs are closest to ours. D4RT answers point-level spatiotemporal queries that specify source time, target time, and camera reference, but identifies the target through a learned temporal embedding over a fixed frame horizon (45). 4RC exposes a conditional interface that selects a target time from a reconstructed 4D representation and decodes dense geometry and motion (21). In contrast, we jointly encode a variable-length clip, select source and target only at decoding time, and answer each continuous pixel query $(u,v)$ by source-to-target cross-attention, with per-frame cameras decoded separately as in VGGT (36). Related continuous decoding appears in InfiniDepth, which also uses a feature-pyramid decoder, but for single-image depth rather than multi-frame source-to-target queries (44).

Tracking establishes consistent cross-frame correspondences and underpins dynamic scene understanding. Recent feed-forward methods differ mainly in how they couple trajectories with geometry. SpatialTrackerV2 jointly estimates depth and camera motion, then refines tracks and poses by differentiable joint optimization that decomposes world motion into geometry, ego-motion, and object motion (40). St4RTrack predicts paired pointmaps in a shared world frame and chains anchored pairs for long-range correspondence (6). Trace Anything forms a dense trajectory field by predicting per-pixel B-spline control points for continuous-time 3D trajectories in one pass (19). Track4World globally encodes the video and estimates dense pairwise 2D/3D flow to obtain world-centric all-pixel trajectories (20). Reconstruction-oriented variants include TrajVG, which couples sparse camera-frame 3D trajectories with local pointmaps (23), and Uni4D, which combines pretrained depth, tracking, and segmentation in multi-stage optimization (42). These works emphasize full trajectories or scene-level reconstruction; we instead answer a continuous source-pixel query at a chosen target by jointly decoding correspondence, target-time 3D geometry, and scene flow.

Scene flow is the 3D displacement of scene points between observations and links dynamic geometry to tracking. MoVieS models time-varying motion of pixel-aligned Gaussians for view synthesis, geometry, and zero-shot scene flow (15); Any4D predicts dense metric per-pixel geometry and motion over multiple frames (14); and 4RC encodes once then conditionally decodes dense geometry and motion to arbitrary target times (21). Track4World estimates dense pairwise 2D/3D flow from a globally encoded video (20). Other representations encode motion differently: V-DPM predicts time-variant and time-synchronized dynamic pointmaps from which per-point 3D motion can be recovered (31), while OmniX parameterizes dense trajectory fields with compact dynamic tokens that separate dynamic from static geometry (9). Flow4R treats camera-space scene flow as the central quantity, predicting per-pixel point, flow, pose weight, and confidence from each image pair and processing sequences as independent anchor–frame pairs (26), so each pair needs a new two-view pass rather than reusing one jointly encoded multi-view clip.

Given a clip $\mathcal{I}=\{I_{n}\}_{n=0}^{N-1}$, UniQuery4R first jointly encodes all $N$ views into multi-view-contextualized feature pyramids. A continuous query $\mathbf{q}=(u,v,s,t)$ selects source $I_{s}$ and target $I_{t}$ only at decoding time. Multi-scale sampling from the selected source yields $\mathbf{q}_{s}$, which attends to all tokens of the selected target to form $\mathbf{q}_{c}$ for correspondence, geometry, and motion; source features provide depth, while per-frame tokens predict cameras (Figure 2). Source and target views are selected from the encoded features only at decoding time, allowing the backbone computation to be reused across different $(s,t)$ choices. The query decoder represents temporal relations through source-to-target feature interaction rather than a learned temporal embedding indexed by a predefined frame set.

The query interface preserves sub-pixel coordinates through differentiable sampling. At a fixed target resolution, decoding cost scales with the requested number of points, so sparse on-demand inference and dense reconstruction share the same model, with the latter obtained by batching a grid of queries. Because one query-conditioned representation drives correspondence, geometry, and motion, these outputs can exploit common evidence instead of being reconstructed by isolated task pipelines.

A query $\mathbf{q}=(u,v,s,t)$ specifies a pixel $(u,v)$ in source view $I_{s}$ and a target view $I_{t}$. UniQuery4R predicts

together with query-independent per-view camera parameters $\boldsymbol{\pi}$. We fix view $0$, rather than the source view, as the reference. In this frame, $\mathbf{P}\in\mathbb{R}^{3}$ is the target-time location of the queried source point and $\Delta\mathbf{P}\in\mathbb{R}^{3}$ is its $s\!\to\!t$ scene flow. Pixel coordinates are mapped to $[-1,1]^{2}$ for sampling. The correspondence $\mathbf{f}\in\mathbb{R}^{2}$ is directly regressed as the absolute target coordinate in this normalized system; its linear output is not hard-clipped to $[-1,1]^{2}$. The supervised auxiliary outputs are point confidence $c_{P}$, four correspondence confidence/precision logits $\mathbf{c}_{f}\in\mathbb{R}^{4}$, and depth confidence $c_{d}$ (Table 2). Finally, $\boldsymbol{\pi}$ contains per-view translation, quaternion rotation, and FoVs ordered as vertical then horizontal $(\mathrm{FoV}_{v},\mathrm{FoV}_{h})$.

UniQuery4R uses an internal multi-view 3D foundation model initialized from DA3-Giant (16). It jointly processes $\mathcal{I}$ so that each view’s tokens are contextualized by the other input views. Tokens from four encoder stages are linearly projected and pixel-shuffled with stage-dependent upsampling factors $s_{\ell}$:

The resulting multi-scale feature pyramid $\{\mathbf{F}_{n}^{(\ell)}\}$ combines fine localization from shallow stages with semantic and multi-view context from deeper stages. Source and target pyramids are selected from this shared representation only after encoding; further encoder details are provided in the supplementary material.

Given a query $(u,v)$, we bilinearly sample each level of the source pyramid at the continuous location, preserving sub-pixel coordinates without quantization. We order the sampled features from the shallowest/finest to the deepest as $\{\mathbf{f}^{(k)}\}_{k=1}^{4}$. Starting from the finest sampled feature, we progressively inject deeper semantics through learned gates:

Here $\mathbf{W}_{h}^{(k)}$ projects the accumulated feature to the current level, and the gate adaptively retains fine localization while incorporating progressively deeper semantic and multi-view context. The fused feature $\mathbf{q}_{s}=\mathbf{h}^{(4)}$ is the continuous source query representation.

For source–target interaction, the query indices $(s,t)$ select two feature hierarchies from the already encoded clip, and $\mathbf{q}_{s}$ attends to the entire target feature map $\mathbf{M}_{t}=\mathrm{Flatten}(\mathbf{F}_{t}^{(4)})$. We apply a stack of Pre-LN cross-attention blocks in which $\mathbf{q}_{s}$ is the query and $\mathbf{M}_{t}$ provides keys and values, followed by a linear readout to the query-conditioned representation $\mathbf{q}_{c}$. Crucially, we do not sample the target at the same $(u,v)$: correspondence emerges from attending over the full target field. The module uses neither local windows, predefined matches, nor a temporal embedding. Thus, $(s,t)$ is a decoding choice rather than an input-pair choice, and different source–target relations can be queried from the same jointly encoded clip.

Shallow task-specific MLPs decode $\mathbf{P}$, $\Delta\mathbf{P}$, $\mathbf{f}$, point confidence, and correspondence confidence/precision from $\mathbf{q}_{c}$, while $\mathbf{q}_{s}$ produces depth and its confidence. We use an inverse-log parameterization for $\mathbf{P}$ and an exponential map for depth; $\mathbf{f}$ is a linear absolute target coordinate $\mathbf{f}^{*}\in[-1,1]^{2}$ rather than a displacement from the source query. Separate supervision with shared query representations encourages the tasks to exploit common spatiotemporal evidence. Cameras are predicted per view by a VGGT-style token head (36) and supervised with reference-normalized relative $L_{1}$ on translation, quaternion (canonicalized to $w\geq 0$), and FoV, following the relative-pose supervision principle of $\pi^{3}$ (39).

Rather than directly regressing the three Cartesian components of scene flow, we factorize the displacement into an $\epsilon$-normalized direction vector and a non-negative magnitude:

Here $\tilde{\mathbf{v}}\in\mathbb{R}^{3}$ and $\tilde{m}\in\mathbb{R}$ are predicted from $\mathbf{q}_{c}$ and $\epsilon_{v}=10^{-4}$. Because of the stabilizing $\epsilon_{v}$, $\hat{\mathbf{v}}$ is not constrained to have exactly unit norm. This four-parameter representation separates motion orientation from motion magnitude, guarantees a non-negative magnitude, and lets the two factors receive complementary supervision.

For scene-flow supervision, let $\Delta\mathbf{P}^{*}$ denote the ground-truth displacement and define the stable radial log transform

where $\epsilon_{\Delta}=10^{-6}$. This transform preserves the displacement direction while compressing its dynamic range, preventing large motions from overwhelming small ones. We classify a valid query as moving when the metric ground-truth magnitude exceeds $\gamma=0.02$ m, yielding dynamic and static sets $\mathcal{D}$ and $\mathcal{S}$. The displacement term is

Here $\alpha$ controls the relative emphasis on moving points; we use $\alpha=0.85$ during training. If one set is empty, only the available term is used. We further supervise orientation on moving points and suppress spurious motion on static points:

Thus, the composed displacement is constrained globally in log space, while its direction and near-zero static magnitude are each explicitly regularized.

UniQuery4R is trained with a multi-task objective that supervises the full set of query outputs and camera parameters:

where $\mathcal{L}_{P}$ and $\mathcal{L}_{d}$ are confidence-weighted $L_{1}$ losses on the 3D point and source depth; $\mathcal{L}_{\Delta}$ balances static and dynamic regions; $\mathcal{L}_{\mathrm{dir}}$ supervises moving-point direction and suppresses the magnitude on static points; $\mathcal{L}_{f}$ is a confidence-aware 2D matching loss; and $\mathcal{L}_{\pi}$ supervises reference-normalized camera parameters. Dataset-dependent reweighting disables unavailable supervision and reduces the point-loss weight on sparse-reconstruction data. Additional implementation details are provided in the supplementary material.

UniQuery4R is trained on a mixture of synthetic and real-world dynamic datasets, including Kubric-4D (35; 7), PointOdyssey (48), Hypersim (28), DL3DV (18), CoTracker3Kubric (12), Stereo4D (10), Virtual KITTI 2 (1), Waymo (33), and an internal dynamic-scene collection. Together they supervise depth, camera pose, optical flow, scene flow, and 2D/3D trajectories. All data follow a unified coordinate convention. Training clips contain $4$–$12$ views with varied baselines and motion magnitudes; for each clip, we sample ordered source–target pairs $(s,t)$, including self-pairs, for query supervision. The main model is trained for $150$k iterations on $16$ NVIDIA H20 GPUs; optimizer settings and full hyperparameters are in the supplementary material.

We follow the WorldTrack protocol (6) on PointOdyssey (PO) (48), Panoptic Studio (PStudio) (11), Dynamic Replica (DR) (13), and Aria Digital Twin (ADT) (24), with qualitative long-sequence tracking on DAVIS (25). Quantitative runs use the first $64$ frames with global median scale alignment before metric computation; macro-averages are unweighted over the four datasets. Table captions detail source–target sampling, validity masks, and thresholds. Although trained on $4$–$12$-frame clips, UniQuery4R jointly encodes all $64$ evaluation frames in one encoder pass, and source–target queries reuse those features without sliding windows or cross-window fusion. All entries labeled D4RT use OpenD4RT (27), an unofficial reimplementation, and should not be read as results of the original model (45).

Table 3 reports $\tau@0.1\mathrm{m}$ and EPE with unweighted four-dataset macro-averages. UniQuery4R attains the best macro-average on both metrics and the best ADT scores, ranks second on DR, and trails V-DPM/4RC on individual PStudio/PO dataset–metric combinations. The direction–magnitude design in Sec. 3 balances static/dynamic supervision so large displacements do not dominate training (Table 8).

Table 4 evaluates world-coordinate 3D tracking of queried dynamic source points via Average Percentage of Points within Distance (APD) and EPE after global median scale alignment. UniQuery4R leads the four-dataset macro-average and ranks first on PStudio, PO, and ADT, while 4RC is strongest on DR. The dynamic-point and scene-flow heads decode from the shared query representation $\mathbf{q}_{c}$, providing a common representation for correspondence, geometry, and motion. Figure 3 qualitatively shows less drift and fragmentation under occlusion and non-rigid deformation.

Tables 5 and 6 summarize depth and camera results. UniQuery4R leads ADT depth and ADT AUC@5/ATE/RPE_r; 4RC is stronger on AUC@30/RPE_t. On Sintel, AbsRel is close to VGGT-$\Omega$ with stronger pose AUC but weaker trajectory RPE. On NRGBD, it leads AUC@30 and ATE but trails DA3-Giant/4RC in relative RPE; on ScanNet++, depth remains below 4RC and reconstruction-oriented backbones.

Table 7 summarizes encode/decode marginal cost: encoding grows with views and resolution and is nearly $Q$-independent, while decoding scales with query count; chunking reduces peak memory at large $Q$. Detailed sweeps are in the supplementary material.

Table 8 shows that correspondence and local depth improve dynamic-point metrics; direction–magnitude raises flow $\tau@0.1\mathrm{m}$ from $73.71$ to $76.48$ and lowers flow EPE from $0.0794$ to $0.0663$, with four multi-scale CA blocks performing best.

Figure 3 suggests improved long-horizon stability in the shown examples relative to feed-forward baselines, and direction–magnitude supervision is central (Table 8). Performance can still degrade under severe occlusion and large rotations; with fast motion or flipping articulations, left–right hand/foot associations are particularly prone to identity swaps. Encode cost also grows with clip length (Table 7), so longer videos need temporal windowing; cross-window fusion is left open.

All 3D outputs and camera translations use the view-$0$ reference frame. During training, geometry is normalized by the clip-level mean point-to-origin distance

computed over valid depth pixels $\Omega$; metric labels are restored by multiplying by $a$. At evaluation, WorldTrack applies median-based global scale alignment separately to each output type:

Camera losses average relative-pose terms over all reference views.

The encoder is an internal multi-view ViT-G ($1536$ width, $40$ blocks, $14{\times}14$ patches) with joint multi-view attention, initialized from DA3-Giant (16) and further pretrained on an internal mixture (corpus and schedule not released). Section B compares alternative initializations. During UniQuery4R training the backbone uses $0.1\times$ the learning rate of newly added modules. We extract $3072$-D tokens from blocks $\{19,27,33,39\}$, project each stage, and PixelShuffle with factors $(4,2,1,1)$ to build the four-level pyramid fed to the query decoder. Rotary positional encoding and query/key normalization start at block $13$.

The query decoder operates at width $256$. Source coordinates are mapped to $[-1,1]^{2}$ for continuous bilinear sampling; up to $100$k queries are supported per forward pass. We use four Pre-LN source–target cross-attention blocks ($8$ heads, head width $32$, FFN${\times}4$, GELU, no dropout). At training time, queries follow dataset annotations when available and otherwise use a dense integer grid; at inference, queries may be placed continuously on the source image.

Two-layer ReLU MLPs (hidden $64$) decode the outputs listed in Table A.1.

Following VGGT (36), per-view camera tokens pass through a four-block, $16$-head trunk (width $3072$) with four additive pose updates. Supervision follows $\pi^{3}$ (39) relative-pose $L_{1}$ on translation, quaternion ($w{\geq}0$), and FoV, averaged over reference views; the principal point is fixed at the image center.

The full objective combines $\mathcal{L}_{P}$, $\mathcal{L}_{d}$, $\mathcal{L}_{f}$, $\mathcal{L}_{\pi}$, scene-flow terms, and dataset-dependent reweighting. Table A.2 lists the verified weights used for the final model (outer multipliers: depth/motion $=1$, camera $=10$).

Table A.3 lists the training mixture (relative weights need not sum to one).

Table A.4 lists the optimizer, learning-rate schedule, and related hyperparameters of the final 16-GPU run.

Table B.1 lists the measured runtime points underlying the scaling analysis.

Query-dependent decoding scales linearly with $Q$, with a slope of 4.18 ms per 1k queries ($R^{2}{=}1.000$). At $504{\times}504$, pyramid construction adds approximately 12.19 ms per chunk and is nearly constant across the sparse sweep (CV $=0.34\%$). Dense decoding follows the same approximately linear trend ($R^{2}{=}0.999$). At $Q{=}60$k, 30k-query chunking reduces peak memory from 15.68 GB to 12.52 GB, while changing time per query by only $-1.87\%$.

Across the examples, the peak response consistently localizes to the physically corresponding target location, including under partial occlusion: when the matched point is not directly visible, attention remains concentrated near the correct image region instead of drifting to unrelated areas with similar appearance. This suggests that source-to-target interaction carries useful 3D correspondence cues, not just category-level semantic matching. The pattern also persists as the source–target separation grows, indicating that our pairwise design remains effective on longer sequences without a fixed-horizon temporal embedding.

We also observe a recurring failure mode that highlights a current limitation. When the query is placed on one symmetric body part, such as the right foot, the map often assigns secondary mass to the semantically and visually similar counterpart (e.g., the left foot). Similar ambiguities appear for left/right hands. The model can therefore confuse mirror-symmetric extremities over long horizons, and stable long-term identity maintenance for such parts remains an open challenge.

Table B.2 compares the internal encoder of Section A with publicly available VGGT-$\Omega$ (37) and DA3-Giant (16) as alternative initializations. All three variants use the same UniQuery4R training data and recipe, trained for $150$k iterations on $8{\times}$ NVIDIA A800 GPUs (distinct from the final $16{\times}$ H20 checkpoint reported in the main paper).

Figure C.1 covers human motion, articulated objects, and non-rigid deformation. Each color tracks one fixed source query in the view-$0$ frame; trajectories remain coherent under occlusion and viewpoint change.

The query decoder predicts target-image coordinates for any source–target pair (Figure C.2). Correspondences follow object boundaries and stay on the target object under large motion and partial occlusion.

Motion masks are obtained without a dedicated segmentation head. Given an $N$-frame clip (views $0,\ldots,N{-}1$), we encode all frames once. After encoding, we construct dense grid queries on consecutive source–target pairs $(0,1),(1,2),\ldots,(N{-}2,N{-}1)$ and the single reverse pair $(N{-}1,N{-}2)$. Decoding each pair yields a dense scene flow $\Delta\mathbf{P}$ from the same direction–magnitude head used at training time. A pixel is marked dynamic if $\|\Delta\mathbf{P}\|$ exceeds $\gamma{=}0.02$ m after metric-scale restoration and static otherwise; short forward pairs $(n,n{+}1)$ capture frame-to-frame motion, while the reverse pair $(N{-}1,N{-}2)$ stabilizes the mask at the clip end. Figure C.3 shows that this simple threshold recovers reasonable foreground motion regions despite sharing the same encoder and decoder used for sparse queries.

The shared encoder and per-frame camera head also reconstruct static scenes (Figure C.4): estimated poses and fused point clouds on indoor and outdoor clips without task-specific finetuning.