Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning
Extracted source text · arXiv 2606.00963
\documentclass{bmvc2k}
%% Enter your paper number here for the review copy
% \bmvcreviewcopy{??}
\title{Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning}
% Enter the paper's authors in order
% \addauthor{Name}{email/homepage}{INSTITUTION_CODE}
\addauthor{Jixuan He}{jh2926@cornell.edu}{1}
\addauthor{Xueting Li}{xuetingli1123@gmail.com}{2}
\addauthor{Chieh Hubert Lin}{hubert052702@gmail.com}{3}
\addauthor{Ming-Hsuan Yang}{minghsuanyang@gmail.com}{4}
% Enter the institutions
% \addinstitution{Name\\Address}
\addinstitution{
Cornell Tech, Cornell University
}
\addinstitution{
NVIDIA
}
\addinstitution{
illoca AI
}
\addinstitution{
The University of California, Merced
}
\runninghead{HE, Li, Lin and Yang}{Reasmory}
% Any macro definitions you would like to include
% These are not defined in the style file, because they don't begin
% with \bmva, so they might conflict with the user's own macros.
% The \bmvaOneDot macro adds a full stop unless there is one in the
% text already.
\def\eg{\emph{e.g}\bmvaOneDot}
\def\Eg{\emph{E.g}\bmvaOneDot}
\def\etal{\emph{et al}\bmvaOneDot}
\def\ie{\emph{i.e}\bmvaOneDot}
\def\Ie{\emph{I.e}\bmvaOneDot}
\def\cf{\emph{c.f}\bmvaOneDot}
\def\Cf{\emph{C.f}\bmvaOneDot}
\def\etc{\emph{etc}\bmvaOneDot}
\def\vs{\emph{vs}\bmvaOneDot}
\usepackage{booktabs}
\usepackage[table]{xcolor}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{makecell}
\usepackage{multirow}
%-------------------------------------------------------------------------
% Document starts here
\begin{document}
\maketitle
\input{0_abstract}
\input{1_intro}
\input{2_related}
\input{3_method}
\input{4_exp}
\input{5_conclusion}
%-------------------------------------------------------------------------
\clearpage
\newpage
\bibliography{egbib}
\end{document}
% \begin{abstract}
% Vision-Language Models (VLMs) have shown emerging spatial reasoning abilities, yet they still struggle to construct accurate spatial understanding and perform reliable spatial transformations in complex scenarios. A key challenge is that spatial evidence is often sparse and distributed across redundant observations, such as multi-view images and monocular videos, making it difficult for VLMs to organize, retain, and exploit relevant visual information. Reconstruction-based Vision Foundation Models (VFMs) offer a natural way to address this issue by converting observations into explicit spatial memory, such as point clouds. However, reliable spatial reasoning requires not only an explicit memory representation, but also a structured mechanism for interacting with it: when VFMs are exposed as free-form tools, VLMs often invoke them incorrectly, misuse intermediate outputs, or fail to execute the spatial transformations required by the query.
% %
% We propose \textbf{Reasmory}, a framework that formulates spatial reasoning as structured program execution over reconstruction-based spatial memory. Reasmory first constructs explicit 3D memory from multi-view images or videos using VFMs, then introduces a lightweight Domain-Specific Language (DSL) to standardize and constrain how VLMs query, transform, render, and reason over this memory. The generated programs are parsed and validated before execution, enabling more consistent and verifiable interactions than unconstrained tool use. We evaluate Reasmory on multi-view image and video spatial reasoning benchmarks, achieving consistent improvements of 10--18\% over strong baselines including GPT-5-mini and Gemini-3-flash. These results suggest that reliable spatial reasoning requires both explicit spatial memory and structured mechanisms for accessing and manipulating that memory.
% \end{abstract}
% !TEX root = bmvc_review.tex
\begin{abstract}
Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding, such as viewpoint reasoning, directional comparison, and distance estimation. In multi-view images and monocular videos, relevant spatial cues are often sparse and distributed across redundant observations, making them difficult to organize and exploit. Reconstruction-based Vision Foundation Models (VFMs) offer a natural way to aggregate such observations into explicit spatial memory, such as point clouds. However, simply exposing reconstruction models as free-form tools is brittle, VLMs may invoke tools incorrectly, skip required spatial transformations, or misuse intermediate results.
%
We propose \textbf{Reasmory}, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory. Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language (DSL) that constrains how VLMs query objects and cameras, transform viewpoints, and render observations during reasoning. Generated programs are parsed and validated before execution, enabling more reliable interaction with spatial memory than unconstrained tool use. Experiments on multi-view image and video spatial reasoning benchmarks show consistent gains of 6--18\% over strong baselines, including GPT-5-mini and Gemini-3-flash, indicating that explicit 3D memory is most useful when accessed through constrained, validated operations rather than free-form tool calls.
\end{abstract}
% \section{Introduction}
% \begin{figure}[t]
% \centering
% % \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
% \includegraphics[width=1.0\linewidth]{images/teaser.pdf}
% \caption{Motivation and overview of \textbf{Reasmory}. Spatial evidence in multi-view images and videos is often sparse and redundant, making organization important for VLM spatial reasoning. Reasmory addresses this by constructing explicit 3D spatial memory and constraining VLM interaction with this memory through validated DSL programs.}
% \label{fig:teaser}
% \vspace{-5mm}
% \end{figure}
% Understanding spatial relationships in the surrounding environment is a fundamental capability for intelligent agents. For Vision-Language Models (VLMs), spatial reasoning is also essential for downstream tasks such as vision-language action~\cite{zitkovich2023rt,kim2024openvla} and navigation~\cite{qi2025vln} in embodied settings. Despite recent advances in visual understanding~\cite{liu2023visual,liu2024improved,bai2025qwen3}, VLMs still struggle to construct accurate spatial understanding and perform reliable spatial transformations across multiple views and videos. Such tasks require a model to perceive relevant objects, recall observations across time or viewpoints, construct a coherent spatial representation, and perform transformations such as changing viewpoint, comparing directions, or estimating distances. This perception--recall--reasoning pipeline is fragile: errors in perception or recall can corrupt the spatial representation, while failures in transformation can lead to incorrect answers even when the relevant evidence is present.
% We argue that reliable spatial reasoning depends on two complementary factors: how spatial information is organized, and how that information is accessed and manipulated during reasoning. In multi-view image and video settings, useful spatial evidence is often sparse and distributed across many redundant observations. Simply providing more frames can therefore increase visual clutter and make recall harder, rather than giving the model a clearer spatial understanding. To examine this effect, we conduct controlled experiments on 200 samples from the spatial reasoning benchmark VSIBench~\cite{yang2025thinking}. Using a spatial reasoning model (SpaceOM~\cite{VQASynth}), we vary how input frames are selected under a fixed budget of 16 frames. As shown in Fig.~\ref{fig:teaser}, question-conditioned frame selection with CLIP~\cite{radford2021learning} improves accuracy from 29.6\% to 30.6\%, while manually selecting informative frames further improves accuracy to 36.1\%. These results suggest that VLMs need a better way to organize, retain, and retrieve relevant spatial evidence from redundant visual inputs.
% Frame selection, however, can only filter existing observations. It does not build a geometrically consistent scene representation, nor can it expose spatial information from viewpoints that are not directly present in the input. Recent approaches such as MindJourney~\cite{yang2026mindjourney} attempt to address this issue by generating additional views with video generation models, yet generated views may lack the spatial consistency required for reliable reasoning. In contrast, reconstruction-based Vision Foundation Models (VFMs) provide a natural way to aggregate multi-view images or video frames into explicit spatial memory. Representations such as point clouds preserve geometric structure, support viewpoint transformation, rendering, and spatial querying, and can increasingly be constructed efficiently at inference time using feed-forward reconstruction models~\cite{wang2025vggt,wang2026pi}.
% Our second key insight is that spatial reasoning requires not only an explicit memory representation, but also a structured mechanism for interacting with it. When reconstruction modules are exposed to VLMs as free-form tools, the model must decide when to call each tool, how to interpret intermediate outputs, and how to maintain a coherent reasoning state across multiple operations. In practice, we observe that VLMs often misuse these tools, skip necessary operations, or produce inconsistent reasoning trajectories. For instance, in a controlled experiment on camera-transition problems from MindCube~\cite{yin2025spatial}, providing GPT-5-mini with spatial memory and direct tool access does not reliably solve the task, whereas explicitly planning tool usage before execution yields substantially better performance.
% Based on these insights, we propose \textbf{Reasmory} (\textbf{Re}construction \textbf{as} Me\textbf{mory}), a framework that formulates spatial reasoning as structured program execution over reconstruction-based spatial memory. Reasmory first constructs explicit 3D memory from multi-view images or videos using reconstruction-based VFMs. It then introduces a lightweight Domain-Specific Language (DSL) that standardizes and constrains how VLMs query objects, transform viewpoints, render observations, and reason over the memory. A compiler parses and validates generated programs before execution, checking syntactic validity, function usage, variable dependencies, and state transitions. The validated program is then executed deterministically by an engine, and a reasoner uses the execution results to produce the final answer.
% This design combines informative spatial memory with controlled interaction. Instead of requiring a VLM to implicitly integrate redundant observations or invoke reconstruction tools through unconstrained function calls, Reasmory turns spatial reasoning into verifiable program execution over explicit memory. We evaluate Reasmory on multi-view image and video spatial reasoning benchmarks, achieving consistent improvements of 10--18\% over strong baselines including GPT-5-mini and Gemini-3-flash.
% Our contributions are threefold:
% \begin{itemize}
% \item We propose \textbf{Reasmory}, a framework that constructs explicit 3D spatial memory from visual observations and formulates spatial reasoning as program execution over this memory.
% \item We introduce a lightweight DSL and validation mechanism that standardize VLM interactions with spatial memory, improving robustness over free-form tool use.
% \item We empirically show that reliable spatial reasoning requires both explicit spatial memory and structured memory interaction, achieving consistent gains on multi-view image and video benchmarks.
% \end{itemize}
% \begin{table}
% \centering
% \begin{tabular}{| l | l | l | l |}
% \hline
% & \textbf{run 1} & \textbf{run 2} & \textbf{run 3} \\
% \hline
% \hline
% GPT-5-mini & 51.61 & 35.48 & 45.16 \\
% \hline
% +tool & 74.19 & 67.74 & 64.51 \\
% \hline
% +tool + plan (self) & 93.54 & 87.09 & 84.91 \\
% \hline
% \end{tabular}
% \end{table}
% !TEX root = bmvc_review.tex
\section{Introduction}
\begin{figure}[t]
\centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
\includegraphics[width=.9\linewidth]
% {images/teaser.pdf}
{images/teaser_logo.pdf} % for no logo version, use teaser.pdf
\vspace{-2mm}
\caption{Overview of \textbf{Reasmory}. Spatial evidence in multi-view images and videos is often sparse and redundant, making it important to organize evidence explicitly for VLM spatial reasoning. Reasmory addresses this by constructing explicit 3D spatial memory and constraining VLM interaction with this memory through validated DSL programs.}
\label{fig:teaser}
\vspace{-5mm}
\end{figure}
Understanding spatial relationships is a fundamental capability for intelligent agents. For Vision-Language Models (VLMs), this capability is particularly important in embodied settings, where tasks such as vision-language action~\cite{zitkovich2023rt,kim2024openvla} and navigation~\cite{qi2025vln} require reliable spatial reasoning. Despite recent advances in visual understanding~\cite{liu2023visual,liu2024improved,bai2025qwen3}, VLMs remain unreliable when reasoning across multi-view images and videos. As shown in Figure~\ref{fig:teaser}, these tasks require perceiving relevant objects, retaining observations across time or viewpoints, constructing coherent spatial representations, and performing spatial transformations such as viewpoint changes, directional comparisons, or distance estimation. This perception--recall--reasoning pipeline is inherently fragile: failures in perception or memory can corrupt the spatial representation, while errors in spatial transformation can produce incorrect answers even when sufficient evidence is available.
We show that reliable spatial reasoning depends on how spatial information is organized, retrieved, and used during reasoning. This challenge is especially evident in multi-view image and video settings, where relevant spatial cues are sparse and scattered across redundant observations. As a result, adding more frames can increase visual clutter and hinder recall rather than improve spatial understanding. To study this effect, we conduct controlled experiments on 200 samples from VSI-Bench~\cite{yang2025thinking} using SpaceOM~\cite{VQASynth} under a fixed budget of 16 frames. Question-conditioned CLIP-based frame selection~\cite{radford2021learning} yields only marginal gains, improving accuracy from 29.6\% to 30.6\%, while manually selecting informative frames raises accuracy to 36.1\%. These results suggest that VLMs need a more effective way to organize and retrieve spatial evidence from redundant visual inputs.
Frame selection only filters existing observations, it cannot construct a geometrically consistent scene representation or expose spatial information from viewpoints that are not directly observed. Recent methods such as MindJourney~\cite{yang2026mindjourney} address this limitation by generating additional views, but generated views may lack the spatial consistency needed for reliable reasoning. In contrast, reconstruction-based Vision Foundation Models (VFMs) offer a natural way to aggregate multi-view images or videos into explicit spatial memory, such as point clouds. This memory preserves geometric structure and supports viewpoint transformation, rendering, and spatial querying~\cite{wang2025vggt,wang2026pi}.
However, effective spatial reasoning also depends on how VLMs interact with this memory. Simply exposing reconstruction models as free-form tools is brittle: VLMs may invoke tools incorrectly, skip required spatial transformations, or misuse intermediate results. For example, as shown in Figure~\ref{fig:motivation}, on camera-transition tasks from MindCube~\cite{yin2025spatial}, GPT-5-mini improves by more than 20 percentage points with direct tool access, but remains unstable because it uses the provided tools in only 60\% of cases. In contrast, explicit tool-use planning raises the tool-use rate to 83.2\% and improves accuracy by an additional 20 percentage points. These observations motivate structured, verifiable interaction with spatial memory.
Based on these insights, we propose \textbf{Reasmory} (\textbf{Re}construction \textbf{as} Me\textbf{mory}), a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory. As shown in Figure~\ref{fig:teaser}, Reasmory first constructs explicit 3D memory from multi-view images or videos using VFMs and augments it with semantically grounded 3D object instances. It then introduces a lightweight Domain-Specific Language (DSL) that constrains how VLMs query objects and cameras, transform viewpoints, and render observations during reasoning. Generated programs are parsed and validated for syntactic correctness, function usage, and execution dependencies before deterministic execution, after which a reasoner produces the final answer.
%
This design turns spatial reasoning into a controlled sequence of memory operations rather than unconstrained tool calls. We evaluate Reasmory on multi-view image and video spatial reasoning benchmarks, achieving consistent gains of 6--18\% over strong baselines, including GPT-5-mini and Gemini-3-flash.
Our contributions are threefold. First, we propose \textbf{Reasmory}, a framework that constructs explicit 3D spatial memory, augments it with semantically grounded 3D object instances, and formulates spatial reasoning as program execution over reconstructed spatial memory. Second, we introduce a lightweight DSL with validation mechanisms that constrain querying, viewpoint transformation, and rendering over spatial memory, improving robustness over free-form tool use. Third, we evaluate Reasmory on multi-view image and video spatial reasoning benchmarks and show improvements over strong VLM and test-time scaling baselines.
\begin{figure}[t]
\centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
\includegraphics[width=0.9\linewidth]{images/motivation.pdf}
\vspace{-2mm}
\caption{Camera-transition results on MindCube. Explicit tool-use planning yields more reliable performance than direct tool access.}
\label{fig:motivation}
\vspace{-5mm}
\end{figure}
% \begin{table}[tp]
% \centering
% \small
% \caption{
% Camera-transition results on MindCube. Explicit tool-use planning yields more reliable performance than direct tool access.
% }
% \label{tab:tool-use-study}
% \footnotesize
% \begin{tabular}{lccc}
% \toprule
% & \textbf{Run 1} & \textbf{Run 2} & \textbf{Run 3} \\
% \midrule
% GPT-5-mini & 51.61 & 35.48 & 45.16 \\
% +tool & 74.19 & 67.74 & 64.51 \\
% +tool + plan (self) & 93.54 & 87.09 & 84.91 \\
% \bottomrule
% \end{tabular}
% \vspace{-5mm}
% \end{table}
% \section{Related Work}
% \noindent \textbf{Spatial Memory.}
% Spatial cognition studies suggest that structured memory enables flexible viewpoint transformation and navigation~\cite{tolman1948cognitive,siegel1975development,o1978hippocampus,o1987single}. Inspired by this, modern models incorporate memory mechanisms to improve spatial consistency. In streaming 3D reconstruction, memory is maintained either implicitly through hidden states~\cite{wang20253d,wang2025continuous,chen2026tttr,zhang2026loger,zhuo2026streaming} or explicitly through feature-augmented 3D points~\cite{wu2026pointr}. Similar trends appear in video generation, where models use implicit memory~\cite{po2025long,savov2026statespacediffuser,zhang2026testtime} or explicit mechanisms such as retrieval and 3D point representations~\cite{yu2025context,li2025vmem,xiao2026worldmem,zhou2025learning,liu2025dynamem}. In embodied reasoning, memory is often represented as token sequences or graphs~\cite{he2025mem4nav,liu2026msnav,zhang2025constructing,zemskova20253dgraphllm}. We instead adopt explicit 3D reconstruction as spatial memory, enabling controllable rendering and verifiable reasoning.
% \noindent \textbf{3D Reconstruction for Spatial Reasoning.}
% Advances in NeRF~\cite{mildenhall2021nerf,chen2022tensorf,muller2022instant,martin2021nerf}, Gaussian Splatting~\cite{kerbl20233d,you2025instainpaint}, and feed-forward reconstruction~\cite{zhang2026loger,wang2024dust3r,wang2025vggt,wang2026pi,lin2026depth} have made efficient 3D geometry estimation practical. Recent work integrates reconstruction into VLMs through spatial encoders~\cite{wu2026spatialmllm,fan2025vlm} or distilled 3D features~\cite{huang20253daware,chen2025think}, but these methods often require large-scale spatial supervision~\cite{wu2026spatialmllm,ouyang2025spacer,feng2026videor}. Other approaches, such as Think3D~\cite{zhang2026think3d} and MindJourney~\cite{yang2026mindjourney}, use free-form tool invocation, which remains difficult to control and verify. In contrast, we expose explicit 3D memory through structured program execution.
% \noindent \textbf{Domain-Specific Language Tools.}
% Tool-augmented reasoning frameworks such as ReAct~\cite{yao2023react} and Toolformer~\cite{schick2023toolformer} show that external tools can improve complex reasoning. However, free-form tool use is often brittle in spatial tasks, leading to inconsistent operations and unstable reasoning states. Domain-specific languages (DSLs) provide structured and verifiable intermediate representations~\cite{mernik2005and,fowler2010domain,ellis2021coder}. Building on this paradigm, we introduce a lightweight DSL that standardizes interaction between VLMs and spatial memory, enabling consistent state transitions and robust spatial reasoning.
\section{Related Work}
\noindent \textbf{Spatial Memory.}
Spatial cognition studies suggest that structured memory enables viewpoint transformation and navigation~\cite{tolman1948cognitive,siegel1975development,o1978hippocampus,o1987single}. Inspired by this, modern models incorporate memory to improve spatial consistency. In streaming 3D reconstruction, memory is maintained either implicitly through hidden states~\cite{wang20253d,wang2025continuous,chen2026tttr,zhang2026loger,zhuo2026streaming} or explicitly through feature-augmented 3D points~\cite{wu2026pointr}. Similar trends appear in video generation, where models use implicit memory~\cite{po2025long,savov2026statespacediffuser,zhang2026testtime} or explicit mechanisms such as retrieval and 3D point representations~\cite{yu2025context,li2025vmem,xiao2026worldmem,zhou2025learning,liu2025dynamem}. In embodied reasoning, memory is often represented as token sequences or graphs~\cite{he2025mem4nav,liu2026msnav,zhang2025constructing,zemskova20253dgraphllm}. In contrast, Reasmory constructs explicit 3D spatial memory at inference time, augments it with semantically grounded 3D object instances, and exposes operations that can be queried, transformed, rendered, and validated.
\noindent \textbf{3D Reconstruction and Spatial VLMs.}
Advances in NeRF~\cite{mildenhall2021nerf,chen2022tensorf,muller2022instant,martin2021nerf}, Gaussian Splatting~\cite{kerbl20233d,you2025instainpaint}, and feed-forward reconstruction~\cite{zhang2026loger,wang2024dust3r,wang2025vggt,wang2026pi,lin2026depth} have made efficient 3D geometry estimation practical. Recent spatial VLMs inject geometric structure through spatial data generation and supervision~\cite{chen2024spatialvlm,ouyang2025spacer,feng2026videor}, geometry or reconstruction encoders~\cite{wu2026spatialmllm,fan2025vlm,hu2025g2vlm,zhao2025spacemind}, or distilled 3D-aware features~\cite{huang20253daware,chen2025think}. These methods improve spatial understanding through model training or feature integration. Other approaches operate at test time by generating additional views~\cite{yang2026mindjourney}, invoking 3D tools~\cite{zhang2026think3d,luo2026pyspatial}, or encoding geometric references for an MLLM~\cite{yuan2026gr3d}. Reasmory is closest to this line of test-time methods, but differs by turning reconstruction-based memory access into validated DSL program execution rather than unconstrained tool invocation.
\noindent \textbf{Programmatic Tool Use and DSLs.}
Tool-augmented reasoning frameworks such as ReAct~\cite{yao2023react} and Toolformer~\cite{schick2023toolformer} show
that tools can improve reasoning, while program-aided approaches delegate parts of reasoning to executable
code~\cite{gao2022pal}. In visual reasoning, systems such as VISPROG~\cite{gupta2022visprog} and
ViperGPT~\cite{suris2023vipergpt} generate programs that compose vision modules at inference time. Recent spatial
agents such as MSSR~\cite{guo2026mssr} query 3D scenes with expert tools and prune redundant spatial information
before answering. These methods show the value of external computation, but executable interaction with vision and 3D
tools can still be brittle in spatial tasks, where errors in viewpoint state, tool order, or intermediate outputs can
corrupt reasoning. Domain-specific languages (DSLs) provide structured and verifiable intermediate
representations~\cite{mernik2005and,fowler2010domain,ellis2021coder}. Building on this paradigm, Reasmory introduces a
lightweight spatial DSL that validates generated programs before execution over explicit 3D memory, directly
controlling tool usage, dependencies, and viewpoint-state transitions.
% \section{Method}
% \begin{figure}[t]
% \centering
% % \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
% \includegraphics[width=1.0\linewidth, height=3in]{images/pipeline.png}
% \caption{This is the pipeline for this paper.}
% \label{fig:pipeline}
% \vspace{-5mm}
% \end{figure}
% \subsection{Preliminaries}
% \noindent \textbf{Feed-forward 3D Reconstruction.}
% Given an input sequence of images (we sample up to 64 frames for video inputs) ${V_1, V_2, \dots, V_n}$, a feed-forward 3D reconstructor $\mathcal{R}$ directly estimates dense depth maps ${D_1, D_2, \dots, D_n}$, camera poses ${T_1, T_2, \dots, T_n}$, and camera intrinsics $K$ in a single forward pass.
% Using the predicted depth and camera parameters, each pixel can be back-projected into 3D space to construct an RGB point cloud. Specifically, for a pixel $p=(u,v)$ in image $V_i$ with depth value $D_i(u,v)$, its corresponding 3D point is computed as:
% \begin{equation}
% \mathbf{x}
% =
% T_i^{-1}
% \left(
% D_i(u,v)\cdot K^{-1}
% \begin{bmatrix}
% u \\ v \\ 1
% \end{bmatrix}
% \right).
% \end{equation}
% Each 3D point is associated with its RGB value from the input image, forming a colored point cloud representation of the scene.
% % Each 3D point is associated with its RGB value from the input image, forming a colored point cloud representation of the scene.
% % move the formular to preliminary
% % \begin{equation}
% % \label{eq:recon}
% % \text{Memory} = \mathcal{R}(\{V_1, V_2, \dots, V_n\}).
% % \end{equation}
% \noindent \textbf{Domain-Specific Language.}
% A Domain-Specific Language (DSL) is a structured programming language designed for a specific task domain. A DSL typically consists of: (1) a syntax that defines valid operations and program structures; (2) an Abstract Syntax Tree (AST), which represents program structure and dependencies; and (3) a compiler, which validates programs according to predefined rules before execution. Compared to free-form interaction, DSLs provide a more controllable and verifiable interface for reasoning.
% \subsection{Pipeline Overview}
% % Split into two parts:
% % 1. construction
% % 2. constrained interaction
% % Based on our analysis for spatial reasoning questions (how spatial information is organized, and how that information is accessed and manipulated during reasoning are important), our pipeline tackles memory construction in Sec 3.3 and a constrained interaction format in Sec 3.4. Given multi-view images or video observations, our pipeline enhances spatial reasoning for VLMs using this explicit memory and structured interaction.
% % The system follows a multi-stage pipeline to construct and execute valid programs. As shown in Figure~\ref{fig:pipeline}, first, a \textbf{Decomposer} analyzes the input problem and breaks it into components following the spatial reasoning principles described in Sec.~\ref{sec:primitives}. This decomposition is recorded in the AST as high-level structure. Next, a \textbf{Planner} generates a program in the DSL conditioned on both the problem and its decomposition to build memory and the structured interaction plan. The generated program is then passed to an \textbf{AST compiler}, which checks its validity. If the program fails validation, the planner is prompted to regenerate the code based on error feedback. Once the program passes validation, an \textbf{execution engine} executes it deterministically to retrieve additional information from the spatial memory. Finally, a \textbf{reasoner} produces the answer based on the augmented context.
% Based on our analysis of spatial reasoning tasks, we identify two key challenges: (1) how spatial information is organized, and (2) how such information is accessed and manipulated during reasoning. Our framework addresses these challenges through explicit spatial memory construction and constrained program execution.
% Given multi-view images or video observations, the system follows a multi-stage pipeline to construct and execute valid spatial reasoning programs. As shown in Figure\ref{fig:pipeline}, a \textbf{Decomposer} first analyzes the input problem and decomposes it according to the spatial reasoning principles described in Sec.\ref{sec:primitives}. The decomposition is converted into structured constraints that guide program generation.
% Next, a \textbf{Planner} generates a program in the DSL conditioned on both the original problem and its decomposition. The generated program is parsed into an AST and passed to an \textbf{AST Compiler}, which verifies program legality. If validation fails, the planner regenerates the program according to compiler feedback. Once validated, the program is executed deterministically by an \textbf{Execution Engine} to retrieve additional information from spatial memory. Finally, a \textbf{Reasoner} produces the final answer based on the augmented context.
% % While 3D reconstruction provides a detailed representation of the scene, it is difficult for VLMs to directly utilize such information in a reliable manner. To address this, we propose a tool-augmented framework with a lightweight DSL, which enforces consistent state transitions and reduces brittle interactions during spatial reasoning.
% % We present \textbf{Reasmory}, a tool-augmented framework that enables VLMs to interact with an explicit 3D spatial memory for solving spatial reasoning tasks. To ensure generalization and robustness, we first analyze spatial reasoning problems and identify a minimal set of primitives in Sec.~\ref{sec:primitives}. We then construct a spatial memory based on 3D reconstruction in Sec.~\ref{sec:recon} and bring semantic to the memory in Sec.~\ref{sec:semantic}. Finally, we introduce a DSL-based interaction pipeline in Sec.~\ref{sec:dsl} to enable structured and reliable reasoning over the memory.
% \subsection{Memory Construction}
% \label{sec:recon}
% \begin{table*}[t]
% \centering
% \small
% \begin{tabular}{p{5.2cm} p{2.2cm} p{3.0cm} p{3.0cm} p{4.8cm}}
% \toprule
% \textbf{Problem} & \textbf{Initial Reference} & \textbf{Viewpoint Refinement} & \textbf{Comparison} & \textbf{Primitive Sequence} \\
% \midrule
% How many tables are in the room?
% & /
% & /
% & count(table)
% & Build $\rightarrow$ Query(table) $\rightarrow$ BEV(table)
% \\
% In which direction did I move from the first view to the second view?
% & $V_{\text{ini}} =$ camera$_1$
% & /
% & direction(ego, camera$_2$)
% & Build $\rightarrow$ Query(cam$_1$) $\rightarrow$ SetView(cam$_1$) $\rightarrow$ Query(cam$_2$) $\rightarrow$ BEV(cam$_2$)
% \\
% If I am standing at the same spot and facing the same direction as shown in image 1, then I turn right and move forward, will I get closer to the pink plush toy and headboard?
% & $V_{\text{ini}} =$ camera$_1$
% & rotate(right), move(forward)
% & direction(ego, plush toy)
% & Build $\rightarrow$ Query(cam$_1$) $\rightarrow$ SetView(cam$_1$) $\rightarrow$ Turn(right) $\rightarrow$ Step(forward) $\rightarrow$ RGB
% \\
% What is behind me if I stand at the same spot and facing direction as image 3?
% & $V_{\text{ini}} =$ camera$_3$
% & rotate(back)
% & visibility(ego, object)
% & Build $\rightarrow$ Query(cam$_3$) $\rightarrow$ SetView(cam$_3$) $\rightarrow$ Turn(back) $\rightarrow$ RGB
% \\
% \bottomrule
% \end{tabular}
% \caption{
% Examples of spatial reasoning decomposition. Most spatial reasoning problems can be formulated as viewpoint selection, viewpoint refinement, and entity comparison under the transformed viewpoint.
% }
% \label{tab:decomposition}
% \end{table*}
% \textbf{Memory Representation.}
% Spatial reasoning requires models to understand spatial arrangements and recall information under different viewpoints. However, VLMs often struggle to process and retain dense visual information, especially in long image sequences or videos.
% In contrast, 3D reconstruction naturally provides a structured representation of spatial information. As the number of observations increases, reconstruction quality improves, making it a complementary memory representation for VLMs. Moreover, explicit 3D representations enable flexible retrieval from arbitrary viewpoints and at different levels of fidelity, facilitating more effective recall during reasoning.
% Therefore, we represent the scene using an explicit colored point cloud reconstructed by a feed-forward reconstructor $\mathcal R$ (e.g., Pi3 or Depth Anything v3). Besides the point cloud representation $\mathbf X$, the memory also stores camera trajectories $\mathbf T$ and corresponding depth maps $\mathbf D$ for all input frames.
% To simplify spatial reasoning, we further align the memory with canonical axes. Specifically, we align the global up direction with the $y$-axis and perform minimum-area bounding-box alignment to align dominant wall directions with the $x$- and $z$-axes.
% \textbf{Semantic Augmentation.}
% Some problem needs to localize specific object in the memory. Therefore, we augment the reconstructed 3D scene with semantic information to enable semantic query. Unlike works like IGGT[], which use the latent semantic feature to augment point cloud, we adopt a simple yet effective strategy to associate category-specific segment proposals across views into consistent 3D object instances, enabling operations such as \texttt{query\_3d\_object\_location}.
% Given multi-view observations and reconstructed 3D geometry, we first extract category-specific segmentation proposals in each view using SAM3~\cite{carion2025sam}. Each proposal is lifted into 3D space using reconstructed geometry and represented as an occupied voxel set.
% To establish cross-view consistency, we propagate object proposals across views through voxel overlap and compute agreement scores between instances. We then construct a graph where nodes correspond to view-level proposals and edges connect geometrically consistent matches. Final object instances are obtained through greedy graph merging under a one-instance-per-view constraint.
% This process produces a semantically grounded and geometrically consistent 3D memory for object-level spatial reasoning. Additional details are provided in the supplementary material.
% \subsection{Constrained Interaction}
% \textbf{Spatial Reasoning Primitives.}
% \begin{table*}[t]
% \centering
% \begin{subtable}[t]{0.48\textwidth}
% \centering
% \footnotesize
% \begin{tabular}{p{2.8cm} p{3.5cm}}
% \toprule
% \textbf{Primitive Type} & \textbf{Operations} \\
% \midrule
% Memory Construction & \texttt{build\_static\_memory}, \texttt{build\_metric\_static\_memory}, \texttt{build\_dynamic\_memory}
% \\
% Memory Query & \texttt{query\_camera\_pose}, \texttt{query\_3d\_object\_location}
% \\
% Memory Transformation & \texttt{set\_viewpoint},
% \texttt{step\_camera}, \texttt{turn\_camera}
% \\
% Rendering & \texttt{render\_egocentric\_rgb}, \texttt{render\_rgb\_bev}, \texttt{render\_semantic\_bev}
% \\
% \bottomrule
% \end{tabular}
% \caption{Spatial reasoning primitives.}
% \label{tab:primitive_defs}
% \end{subtable}
% \hfill
% \begin{subtable}[t]{0.46\textwidth}
% \centering
% \footnotesize
% \begin{tabular}{p{2.2cm} p{3.0cm}}
% \toprule
% \textbf{Reasoning Stage} & \textbf{Allowed Primitive Types} \\
% \midrule
% Reference Viewpoint Selection & Memory Construction, Memory Query
% \\
% Viewpoint Refinement & Memory Query, Memory Transformation, Rendering
% \\
% Entity Comparison & Memory Query, Rendering
% \\
% \bottomrule
% \end{tabular}
% \caption{Allowed primitives for each reasoning stage.}
% \label{tab:primitive_stage}
% \end{subtable}
% \caption{
% Spatial reasoning primitives and their allowed usage across different reasoning stages.
% }
% \label{tab:primitives}
% \end{table*}
% \label{sec:primitives}
% % Show some cases to make it clear. (put 2/3 cases, and a big chart in supp)
% To guide the design of our framework, we analyze a wide range of spatial reasoning questions and observe that most tasks can be decomposed into three steps:
% \begin{enumerate}
% \item \textbf{Reference viewpoint selection.} Choose an initial reference viewpoint $V_{\text{ini}}$.
% \item \textbf{Viewpoint refinement.} Refine the viewpoint to obtain $V_{\text{final}}$ based on the problem to simplify reasoning (e.g., moving or rotating the viewpoint to achieve a more canonical view).
% \item \textbf{Entity transformation and comparison.} Identify the relevant entities ($E_i, E_j, \dots$), transform them into the final viewpoint ($T_{V_{\text{final}}}(E_i), T_{V_{\text{final}}}(E_j), \dots$), and perform geometric comparison using a function $f$ (e.g., direction, distance, or visibility).
% \end{enumerate}
% Conceptually this can be written as:
% \begin{center}
% \begin{minipage}{0.62\linewidth}
% \begin{verbatim}
% V_ini = select_init(problem)
% while V_final is not sufficient:
% V_final = refine_viewpoint(V_ini, problem)
% result = f(T_V_final(E_i), T_V_final(E_j), ...)
% \end{verbatim}
% \end{minipage}
% \end{center}
% Based on this formulation, we define a set of atomic operations as spatial reasoning primitives, which enable VLMs to interact with spatial memory in a compositional manner. These primitives are organized into four categories:
% % \begin{itemize}
% % \item \textbf{Memory construction.} The model can use feed-forward reconstruction models;
% % \item \textbf{Memory query.} The model can query spatial information from memory;
% % \item \textbf{Memory transformation.} The model can modify the current viewpoint through operations;
% % \item \textbf{Rendering.} The model can render observations for visual inspection, including perspective RGB views and bird’s-eye-view (BEV) images at the current viewpoint, as well as queried entities at different levels of fidelity (see Figure~3).
% % \end{itemize}
% \textbf{Memory construction.} The model can use feed-forward reconstruction models;
% \textbf{Memory query.} The model can query spatial information from memory;
% \textbf{Memory transformation.} The model can modify the current viewpoint through operations;
% \textbf{Rendering.} The model can render observations for visual inspection, including perspective RGB views and bird’s-eye-view (BEV) images at the current viewpoint, as well as queried entities at different levels of fidelity (see Figure~3).
% Different reasoning stages are associated with different allowable primitive categories. For example, viewpoint refinement permits memory transformation and rendering operations, while reference viewpoint selection only allows memory construction and querying. This constrained interaction design reduces invalid reasoning trajectories and enforces consistent state transitions during execution. Detailed primitive categories and their corresponding stage constraints are summarized in Table~\ref{tab:primitives}.
% \textbf{DSL-based Pipeline.}
% \label{sec:dsl}
% Although spatial memory provides rich geometric information and we define a set of spatial primitives for interaction, we find that VLMs struggle to reliably utilize these tools when generating free-form tool calls. To address this issue, we introduce a DSL that standardizes the interaction between VLMs and spatial memory. During inference, the model generates structured programs in the DSL to solve spatial reasoning tasks. We design the DSL as a restricted subset of Python, with a limited set of allowed operations and explicit constraints enforced through an Abstract Syntax Tree (AST).
% The generated program is parsed into an AST and verified before execution. The AST compiler enforces several constraints to ensure correctness, including: (1) syntactic validity; (2) valid function usage; (3) dependency consistency; and (4) viewpoint state consistency
% Programs that fail validation are rejected and regenerated. Only validated programs are executed by the runtime engine.
% This design transforms spatial reasoning from free-form tool usage into verifiable program execution, significantly improving robustness and consistency.
% % \include{tables/mindcube}
% % The AST enforces several constraints to ensure correctness, including: (1) syntactic validity; (2) valid function usage; (3) proper variable definition and dependency tracking; and (4) consistent state transitions. Detailed rules and implementation are provided in the supplementary material.
% % This design transforms spatial reasoning from free-form tool usage into verifiable program execution, significantly improving robustness and consistency.
% \subsection{Previously in the appendix: Multi-View Instance Aggregation via 3D Voxel Agreement}
% \label{sec:multiview_aggregation}
% Given a set of RGB views $\mathcal{V}=\{1,\dots,V\}$ of a static scene and the corresponding dense 3D maps
% $\mathbf{X}^{(v)} \in \mathbb{R}^{H \times W \times 3}$, our goal is to merge category-specific 2D segment proposals from multiple views into a consistent set of 3D object instances.
% For each queried category $c$, we first run a text-conditioned segmenter on every view and obtain a set of binary masks
% $\mathcal{M}_c^{(v)} = \{M_{c,i}^{(v)}\}_{i=1}^{N_v}$.
% We begin by suppressing redundant masks within each view.
% Instead of standard IoU, we use a containment-style overlap:
% \begin{equation}
% \mathrm{ov}(M_a, M_b) = \frac{|M_a \cap M_b|}{\min(|M_a|, |M_b|)}.
% \end{equation}
% Masks with $\mathrm{ov}(M_a, M_b) \ge \tau_{\mathrm{nms}}$ are treated as duplicates, and only the highest-scoring one is kept.
% This is preferable in our setting because same-object proposals often differ mainly by small boundary variations.
% For each surviving mask $M_{c,i}^{(v)}$, we resize it to the spatial-memory resolution and lift it to 3D by collecting all dense 3D points covered by the mask:
% \begin{equation}
% P_{c,i}^{(v)} = \{ \mathbf{X}^{(v)}(p) \mid p \in \tilde{M}_{c,i}^{(v)} \},
% \end{equation}
% where $\tilde{M}_{c,i}^{(v)}$ denotes the resized binary mask.
% Each such point set defines a \emph{view-level instance candidate}.
% To measure cross-view consistency, we discretize the global 3D space into voxels of size $\delta$ and define a voxel hash function $h(\cdot)$.
% For each candidate, we compute its occupied voxel set
% \begin{equation}
% K_{c,i}^{(v)} = \{ h(\mathbf{x}) \mid \mathbf{x} \in P_{c,i}^{(v)} \}.
% \end{equation}
% We then propagate this candidate to another view $u \neq v$ by marking all target pixels whose 3D points fall into the same occupied voxels:
% \begin{equation}
% \hat{M}_{c,i}^{(v \rightarrow u)}(p) = \mathbf{1}\!\left[h\!\left(\mathbf{X}^{(u)}(p)\right) \in K_{c,i}^{(v)}\right].
% \end{equation}
% If the propagated mask is too small relative to the image, we regard the object as not reliably visible in that view and exclude the view from further scoring.
% For every visible target view $u$, we compare the propagated mask against all masks of the same category in that view using a coverage score
% \begin{equation}
% \mathrm{cov}_{ij}^{v \rightarrow u}
% =
% \frac{
% \left|\hat{M}_{c,i}^{(v \rightarrow u)} \cap \tilde{M}_{c,j}^{(u)}\right|
% }{
% \left|\hat{M}_{c,i}^{(v \rightarrow u)}\right|
% }.
% \end{equation}
% We take the best-matching target instance
% \begin{equation}
% j^\star = \arg\max_j \mathrm{cov}_{ij}^{v \rightarrow u},
% \qquad
% \mathrm{cov}_{i}^{v \rightarrow u} = \max_j \mathrm{cov}_{ij}^{v \rightarrow u},
% \end{equation}
% and define an agreement indicator
% \begin{equation}
% a_{i}^{v \rightarrow u} = \mathbf{1}\!\left[\mathrm{cov}_{i}^{v \rightarrow u} \ge \tau_{\mathrm{cov}}\right].
% \end{equation}
% The multi-view agreement score of candidate $(v,i)$ is then
% \begin{equation}
% A_{c,i}^{(v)}
% =
% \frac{1}{|\mathcal{S}_{i}^{(v)}|}
% \sum_{u \in \mathcal{S}_{i}^{(v)}} a_{i}^{v \rightarrow u},
% \end{equation}
% where $\mathcal{S}_{i}^{(v)}$ is the set of views in which the propagated mask is sufficiently visible.
% Intuitively, $A_{c,i}^{(v)}$ measures how consistently a segmented instance can be re-identified across views via 3D occupancy.
% We next build a graph for each category.
% Each node corresponds to a view-level candidate whose agreement score exceeds a threshold:
% \begin{equation}
% \mathcal{N}_c = \left\{ n_{c,i}^{(v)} \;\middle|\; A_{c,i}^{(v)} \ge \tau_{\mathrm{agr}} \right\}.
% \end{equation}
% An edge is added between $n_{c,i}^{(v)}$ and $n_{c,j}^{(u)}$ if they are selected as best matches across views.
% We assign three edge types according to bidirectional consistency:
% \begin{equation}
% \mathrm{type}(e) =
% \begin{cases}
% \texttt{hard}, & a_{i}^{v \rightarrow u}=1 \text{ and } a_{j}^{u \rightarrow v}=1,\\
% \texttt{soft}, & a_{i}^{v \rightarrow u}+a_{j}^{u \rightarrow v}=1,\\
% \texttt{weak}, & \text{otherwise},
% \end{cases}
% \end{equation}
% with edge weights
% \begin{equation}
% w(e)=
% \begin{cases}
% 1.0, & \texttt{hard},\\
% 0.8, & \texttt{soft},\\
% \mathrm{cov}_{i}^{v \rightarrow u}\,\mathrm{cov}_{j}^{u \rightarrow v}, & \texttt{weak}.
% \end{cases}
% \end{equation}
% Finally, we greedily merge nodes into multi-view object clusters by sorting edges in descending order of confidence (\texttt{hard} $>$ \texttt{soft} $>$ \texttt{weak}, then by weight).
% A merge is allowed only if the two clusters do not already contain nodes from the same frame.
% This one-instance-per-view constraint is critical, as it prevents multiple proposals from the same image from collapsing into a single 3D object.
% For each final cluster $C$, we aggregate all its 3D points:
% \begin{equation}
% P(C) = \bigcup_{n_{c,i}^{(v)} \in C} P_{c,i}^{(v)},
% \end{equation}
% and estimate the object center by coordinate-wise median:
% \begin{equation}
% \mathbf{z}(C) = \mathrm{median}(P(C)).
% \end{equation}
% The resulting set of centers $\{\mathbf{z}(C)\}$ forms our final 3D object predictions for category $c$.
% In practice, this design is robust to imperfect single-view masks because the final decision is not based on any single segmentation.
% Instead, instances are retained only when they exhibit consistent 3D support across multiple views, and are merged under explicit cross-view mutual-exclusion constraints.
% !TEX root = bmvc_review.tex
\section{Method}
\begin{figure}[t]
\centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
\includegraphics[width=\linewidth]{images/pipeline-gpt.png}
\vspace{-5mm}
\caption{Overview of \textbf{Reasmory}. The system constructs reconstruction-based spatial memory, augments it with grounded 3D object instances, generates and validates a DSL program, and executes the program to support spatial reasoning.}
\label{fig:pipeline}
\vspace{-5mm}
\end{figure}
Reasmory addresses two key challenges: organizing spatial information into an explicit memory, and controlling how VLMs access and manipulate that memory during reasoning.
%As shown in Figure~\ref{fig:pipeline}, a key component is the execution plan interacting with memory. In the plan, Reasmory first constructs reconstruction-based 3D memory from multi-view images or video observations and augments it with grounded 3D object instances. It then answers spatial questions through structured interaction with this memory.
As shown in Figure~\ref{fig:pipeline}, Reasmory first constructs reconstruction-based 3D memory from multi-view images or video observations and augments it with grounded 3D object instances. It then answers spatial questions by generating, validating, and executing DSL programs over this memory.
%
The remainder of this section is organized as follows: Sec.~\ref{sec:prelim} introduces feed-forward 3D reconstruction and DSLs; Sec.~\ref{sec:recon} presents spatial memory construction and semantic augmentation; and Sec.~\ref{sec:interaction} explains how Reasmory decomposes each question, plans a DSL program with spatial primitives, then validates the program and executes it over spatial
memory.
\input{algorithm/algorithm1}
\input{algorithm/algorithm2}
\input{tables/table-decomp}
\input{tables/table-primitive}
\subsection{Preliminaries}
\label{sec:prelim}
We briefly introduce the two foundations of Reasmory: feed-forward 3D reconstruction and domain-specific languages.
\noindent \textbf{Feed-forward 3D Reconstruction.}
Given multi-view images or frames sampled from a video, denoted by $\{V_1, V_2, \dots, V_n\}$, a feed-forward 3D reconstructor $\mathcal{R}$ estimates dense depth maps $\{D_1, D_2, \dots, D_n\}$, camera poses $\{T_1, T_2, \dots, T_n\}$, and camera intrinsics $K$ in a single forward pass.
Using the predicted depth and camera parameters, each pixel can be back-projected into 3D. Specifically, for a pixel $p=(u,v)$ in image $V_i$ with depth value $D_i(u,v)$, the corresponding 3D point is computed as:
\begin{equation}
\mathbf{x}
=
T_i^{-1}
\left(
D_i(u,v)\cdot K^{-1}
\begin{bmatrix}
u \\ v \\ 1
\end{bmatrix}
\right).
\end{equation}
Each 3D point is associated with its RGB value from the input image, yielding a colored point cloud representation of the scene.
\noindent \textbf{Domain-Specific Language.}
A Domain-Specific Language (DSL)~\cite{mernik2005and} defines a restricted programming interface for a specific task domain. A DSL typically consists of: (1) a syntax that defines valid operations and program structures; (2) an Abstract Syntax Tree (AST), which represents program structure and dependencies; and (3) a compiler. In our setting, the DSL specifies the valid spatial-memory operations, the program structure used to compose them, and the compiler checks applied to the resulting AST before execution.
Compared with free-form tool use, this provides a more controllable and verifiable interface for reasoning over spatial memory.
% \input{algorithm/algorithm2}
% While 3D reconstruction provides a detailed representation of the scene, it is difficult for VLMs to directly utilize such information in a reliable manner. To address this, we propose a tool-augmented framework with a lightweight DSL, which enforces consistent state transitions and reduces brittle interactions during spatial reasoning.
% We present \textbf{Reasmory}, a tool-augmented framework that enables VLMs to interact with an explicit 3D spatial memory for solving spatial reasoning tasks. To ensure generalization and robustness, we first analyze spatial reasoning problems and identify a minimal set of primitives in Sec.~\ref{sec:primitives}. We then construct a spatial memory based on 3D reconstruction in Sec.~\ref{sec:recon} and bring semantic to the memory in Sec.~\ref{sec:semantic}. Finally, we introduce a DSL-based interaction pipeline in Sec.~\ref{sec:dsl} to enable structured and reliable reasoning over the memory.
\subsection{Spatial Memory Construction}
\label{sec:recon}
\textbf{Memory Representation.}
Reasmory represents each scene as an explicit 3D spatial memory in the form of a colored point cloud. Given multi-view images or video frames, we reconstruct a colored point cloud using a feed-forward reconstructor $\mathcal R$ (e.g., Pi3~\cite{wang2026pi} or Depth Anything v3~\cite{lin2026depth}). The memory stores the point cloud $\mathbf X$, camera trajectories $\mathbf T$, and depth maps $\mathbf D$ for all input frames. This representation fuses unordered, partially overlapping observations into a geometrically consistent scene structure while preserving information needed for viewpoint transformation, rendering, and spatial querying.
%
To simplify downstream spatial reasoning, we align the memory to canonical axes. Specifically, we align the global up direction with the $y$-axis and use minimum-area bounding-box alignment to align dominant wall directions with the $x$- and $z$-axes, yielding a more interpretable coordinate system and making the problems related to size estimation easier.
\noindent \textbf{Semantic Augmentation.}
Many spatial reasoning questions refer to objects by category or name, such as chairs, tables, or other named objects. To support such object-level queries, we augment the reconstructed 3D memory with semantically grounded object instances. Each instance stores a category label, an estimated 3D center, and the set of associated 3D points, enabling operations such as \texttt{query\_3d\_object\_location}.
Algorithm~\ref{alg:semantic_augmentation} summarizes the construction process. For each queried category, we extract category-specific segmentation proposals in all views using SAM3~\cite{carion2025sam}, lift each proposal into 3D using the reconstructed geometry, and represent it by occupied 3D grid cells. Proposals corresponding to the same physical object should occupy consistent 3D regions across views. We therefore score each lifted proposal by how consistently its occupied grid cells match same-category proposals in other views. Proposals with sufficient agreement are connected in a category-specific graph and greedily merged under a one-instance-per-view constraint. Scoring thresholds and implementation details are provided in the supplementary material.
\subsection{Structured Interaction with Spatial Memory}
\label{sec:interaction}
As illustrated in Figure~\ref{fig:pipeline}, the interaction pipeline consists of three stages. First, a \emph{Question Decomposer} analyzes the spatial reasoning problem and converts it into a structured representation. Second, a \emph{Planner} generates a DSL program composed of spatial reasoning primitives that describes how to interact with the spatial memory. To ensure correctness, the generated program is checked by an AST-based compiler and iteratively refined using compiler feedback when necessary. Third, a validated program is executed over the spatial memory to retrieve additional observations and spatial evidence, which are then incorporated into the context for the final \emph{Reasoner}. Together, they transform spatial reasoning from unconstrained tool usage into a verifiable interaction process with explicit spatial memory.
\noindent \textbf{Question Decomposition.}
The decomposer maps each spatial reasoning question into three stages. First, \emph{reference viewpoint selection} chooses an initial viewpoint $V_{\mathrm{ini}}$ when the question is anchored to a specific image, camera, or observer pose. Second, \emph{viewpoint refinement} updates this viewpoint through rotations or translations specified by the question. Third, \emph{entity transformation and comparison} identifies the relevant entities, expresses them under the refined viewpoint, and applies a task-specific comparison function $f$, such as direction, distance, visibility, or counting.
This decomposition provides a structured scaffold for planning.
% This decomposition separates spatial perception from spatial reasoning and provides a structured scaffold for program generation.
Algorithm~\ref{alg:decomposition} summarizes this abstraction, and Table~\ref{tab:decomposition} provides representative examples.
\noindent \textbf{Program Planning with Spatial Primitives.}
Given this decomposition, the planner generates a DSL program using atomic spatial reasoning primitives. These primitives fall into four categories: \emph{memory construction}, which builds or loads the spatial memory; \emph{memory query}, which retrieves camera or object information; \emph{memory transformation}, which updates the current viewpoint; and \emph{rendering}, which produces egocentric or bird's-eye-view observations for visual inspection.
We design the DSL as a restricted subset of Python, where the planner can compose only these allowed primitives and must follow the program structure checked by the compiler described below.
Different reasoning stages are associated with different allowable primitive categories. For example, viewpoint refinement permits memory transformation and rendering operations, while reference viewpoint selection only allows memory construction and querying. This constrained interaction design reduces invalid reasoning trajectories and enforces consistent state transitions during execution. Detailed primitive categories and their corresponding stage constraints are summarized in Table~\ref{tab:primitives}.
\noindent \textbf{Program Validation and Execution.}
\label{sec:dsl}
\input{tables/table-taxonomy}
The program generated by the planner is passed to an \emph{AST Compiler}, which verifies correctness before execution. As shown in Table~\ref{tab:ast-taxonomy-summary}, the compiler checks: (1) syntactic validity; (2) valid function usage; (3) dependency consistency; (4) viewpoint state consistency; (5) execution discipline; and (6) plan consistency. Programs that fail validation are rejected and regenerated using compiler feedback. Once validated, the \emph{Execution Engine} runs the program deterministically over the spatial memory, producing query results, transformed viewpoints, and rendered observations. Finally, the \emph{Reasoner} uses these execution outputs, together with the original question, to produce the final answer.
This converts spatial reasoning from unconstrained tool use into validated program execution, where invalid operations, missing dependencies, and inconsistent state transitions can be
detected before execution.
Compiler feedback also enables iterative self-correction, allowing the planner to repair most invalid programs with only a few regeneration attempts.
% The program generated by the planner is passed to an \emph{AST Compiler}, which verifies correctness before execution. As shown in Table~\ref{tab:ast-taxonomy-summary}, the compiler checks: (1) syntactic validity; (2) valid function usage; (3) dependency consistency; (4) viewpoint state consistency; (5) execution discipline; and (6) plan consistency. Programs that fail validation are rejected and regenerated using compiler feedback. Once validated, the \emph{Execution Engine} runs the program deterministically over the spatial memory, producing query results, transformed viewpoints, and rendered observations. Finally, the \emph{Reasoner} uses these execution outputs, together with the original question, to produce the final answer.
% This converts spatial reasoning from unconstrained tool use into validated program execution, where invalid operations, missing dependencies, and inconsistent state transitions can be detected before execution.
% \subsection{Multi-View Instance Aggregation via 3D Voxel Agreement}
% \label{sec:multiview_aggregation}
% Given a set of RGB views $\mathcal{V}=\{1,\dots,V\}$ of a static scene and the corresponding dense 3D maps
% $\mathbf{X}^{(v)} \in \mathbb{R}^{H \times W \times 3}$, our goal is to merge category-specific 2D segment proposals from multiple views into a consistent set of 3D object instances.
% For each queried category $c$, we first run a text-conditioned segmenter on every view and obtain a set of binary masks
% $\mathcal{M}_c^{(v)} = \{M_{c,i}^{(v)}\}_{i=1}^{N_v}$.
% We begin by suppressing redundant masks within each view.
% Instead of standard IoU, we use a containment-style overlap:
% \begin{equation}
% \mathrm{ov}(M_a, M_b) = \frac{|M_a \cap M_b|}{\min(|M_a|, |M_b|)}.
% \end{equation}
% Masks with $\mathrm{ov}(M_a, M_b) \ge \tau_{\mathrm{nms}}$ are treated as duplicates, and only the highest-scoring one is kept.
% This is preferable in our setting because same-object proposals often differ mainly by small boundary variations.
% For each surviving mask $M_{c,i}^{(v)}$, we resize it to the spatial-memory resolution and lift it to 3D by collecting all dense 3D points covered by the mask:
% \begin{equation}
% P_{c,i}^{(v)} = \{ \mathbf{X}^{(v)}(p) \mid p \in \tilde{M}_{c,i}^{(v)} \},
% \end{equation}
% where $\tilde{M}_{c,i}^{(v)}$ denotes the resized binary mask.
% Each such point set defines a \emph{view-level instance candidate}.
% To measure cross-view consistency, we discretize the global 3D space into voxels of size $\delta$ and define a voxel hash function $h(\cdot)$.
% For each candidate, we compute its occupied voxel set
% \begin{equation}
% K_{c,i}^{(v)} = \{ h(\mathbf{x}) \mid \mathbf{x} \in P_{c,i}^{(v)} \}.
% \end{equation}
% We then propagate this candidate to another view $u \neq v$ by marking all target pixels whose 3D points fall into the same occupied voxels:
% \begin{equation}
% \hat{M}_{c,i}^{(v \rightarrow u)}(p) = \mathbf{1}\!\left[h\!\left(\mathbf{X}^{(u)}(p)\right) \in K_{c,i}^{(v)}\right].
% \end{equation}
% If the propagated mask is too small relative to the image, we regard the object as not reliably visible in that view and exclude the view from further scoring.
% For every visible target view $u$, we compare the propagated mask against all masks of the same category in that view using a coverage score
% \begin{equation}
% \mathrm{cov}_{ij}^{v \rightarrow u}
% =
% \frac{
% \left|\hat{M}_{c,i}^{(v \rightarrow u)} \cap \tilde{M}_{c,j}^{(u)}\right|
% }{
% \left|\hat{M}_{c,i}^{(v \rightarrow u)}\right|
% }.
% \end{equation}
% We take the best-matching target instance
% \begin{equation}
% j^\star = \arg\max_j \mathrm{cov}_{ij}^{v \rightarrow u},
% \qquad
% \mathrm{cov}_{i}^{v \rightarrow u} = \max_j \mathrm{cov}_{ij}^{v \rightarrow u},
% \end{equation}
% and define an agreement indicator
% \begin{equation}
% a_{i}^{v \rightarrow u} = \mathbf{1}\!\left[\mathrm{cov}_{i}^{v \rightarrow u} \ge \tau_{\mathrm{cov}}\right].
% \end{equation}
% The multi-view agreement score of candidate $(v,i)$ is then
% \begin{equation}
% A_{c,i}^{(v)}
% =
% \frac{1}{|\mathcal{S}_{i}^{(v)}|}
% \sum_{u \in \mathcal{S}_{i}^{(v)}} a_{i}^{v \rightarrow u},
% \end{equation}
% where $\mathcal{S}_{i}^{(v)}$ is the set of views in which the propagated mask is sufficiently visible.
% Intuitively, $A_{c,i}^{(v)}$ measures how consistently a segmented instance can be re-identified across views via 3D occupancy.
% We next build a graph for each category.
% Each node corresponds to a view-level candidate whose agreement score exceeds a threshold:
% \begin{equation}
% \mathcal{N}_c = \left\{ n_{c,i}^{(v)} \;\middle|\; A_{c,i}^{(v)} \ge \tau_{\mathrm{agr}} \right\}.
% \end{equation}
% An edge is added between $n_{c,i}^{(v)}$ and $n_{c,j}^{(u)}$ if they are selected as best matches across views.
% We assign three edge types according to bidirectional consistency:
% \begin{equation}
% \mathrm{type}(e) =
% \begin{cases}
% \texttt{hard}, & a_{i}^{v \rightarrow u}=1 \text{ and } a_{j}^{u \rightarrow v}=1,\\
% \texttt{soft}, & a_{i}^{v \rightarrow u}+a_{j}^{u \rightarrow v}=1,\\
% \texttt{weak}, & \text{otherwise},
% \end{cases}
% \end{equation}
% with edge weights
% \begin{equation}
% w(e)=
% \begin{cases}
% 1.0, & \texttt{hard},\\
% 0.8, & \texttt{soft},\\
% \mathrm{cov}_{i}^{v \rightarrow u}\,\mathrm{cov}_{j}^{u \rightarrow v}, & \texttt{weak}.
% \end{cases}
% \end{equation}
% Finally, we greedily merge nodes into multi-view object clusters by sorting edges in descending order of confidence (\texttt{hard} $>$ \texttt{soft} $>$ \texttt{weak}, then by weight).
% A merge is allowed only if the two clusters do not already contain nodes from the same frame.
% This one-instance-per-view constraint is critical, as it prevents multiple proposals from the same image from collapsing into a single 3D object.
% For each final cluster $C$, we aggregate all its 3D points:
% \begin{equation}
% P(C) = \bigcup_{n_{c,i}^{(v)} \in C} P_{c,i}^{(v)},
% \end{equation}
% and estimate the object center by coordinate-wise median:
% \begin{equation}
% \mathbf{z}(C) = \mathrm{median}(P(C)).
% \end{equation}
% The resulting set of centers $\{\mathbf{z}(C)\}$ forms our final 3D object predictions for category $c$.
% In practice, this design is robust to imperfect single-view masks because the final decision is not based on any single segmentation.
% Instead, instances are retained only when they exhibit consistent 3D support across multiple views, and are merged under explicit cross-view mutual-exclusion constraints.
\begin{algorithm}[t]
%\fontsize{7pt}{11pt}\selectfont
\scriptsize
\caption{{Semantic augmentation with multi-view 3D agreement}}
\label{alg:semantic_augmentation}
\begin{algorithmic}[1]
\Require Views $\{V_i\}_{i=1}^{n}$, reconstructed 3D maps $\{\mathbf{X}^{(i)}\}_{i=1}^{n}$, query categories $\mathcal{C}$
\Statex \textbf{Output:} 3D object instances $\mathcal{O}$ with category labels, centers, and associated 3D points
\State $\mathcal{O} \gets \emptyset$
\ForAll{category $c \in \mathcal{C}$}
\State Extract 2D segmentation proposals for $c$ in all views
\State Remove duplicate proposals within each view
\ForAll{remaining proposal mask $M$ in view $i$}
\State Lift $M$ into 3D using $\mathbf{X}^{(i)}$
\State Store the occupied 3D grid cells for this proposal
\State Compare these cells with same-category proposals in other views
\State Score $M$ by its cross-view 3D agreement
\EndFor
\State Keep proposals with sufficient agreement
\State Build a graph connecting mutually consistent proposals
\State Greedily merge connected proposals, with at most one mask per view
\State Add each merged cluster as a 3D object instance in $\mathcal{O}$
\EndFor
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[t]
\caption{{Spatial reasoning decomposition. $T_V(\mathcal{E})$ are entities expressed in viewpoint $V$}}
\scriptsize
\label{alg:decomposition}
\begin{algorithmic}[1]
\Require Question $q$
\State $V_{\mathrm{ini}} \gets \textsc{SelectReference}(q)$
\State $V \gets V_{\mathrm{ini}}$
\While{$\neg\,\textsc{ReadyForComparison}(V, q)$}
\State $V \gets \textsc{RefineViewpoint}(V, q)$
\EndWhile
\State $\mathcal{E} \gets \textsc{IdentifyEntities}(q)$
\State $a \gets f\!\left(T_V(\mathcal{E})\right)$
\Return $a$
\end{algorithmic}
\end{algorithm}
% \begin{table*}[t]
% \centering
% \small
% \begin{tabular}{p{2.5cm} p{2cm} p{2cm} p{2cm} p{2cm}}
% \toprule
% \textbf{Problem} & \textbf{Initial Reference} & \textbf{Viewpoint Refinement} & \textbf{Comparison} & \textbf{Primitive Sequence} \\
% \midrule
% How many tables are in the room?
% & /
% & /
% & count(table)
% & Build
% % $\rightarrow$
% Query(table)
% % $\rightarrow$
% RenderBEV(table)
% \\
% In which direction did I move from the first view to the second view?
% & $V_{\text{ini}} =$ camera$_1$
% & /
% & direction(ego, camera$_2$)
% & Build
% % $\rightarrow$
% Query(cam$_1$)
% % $\rightarrow$
% SetView(cam$_1$)
% % $\rightarrow$
% Query(cam$_2$)
% % $\rightarrow$
% RenderBEV(cam$_2$)
% \\
% What is behind me if I stand at the same spot and facing direction as image 3?
% & $V_{\text{ini}} =$ camera$_3$
% & rotate(back)
% & visibility(ego, object)
% & Build
% % $\rightarrow$
% Query(cam$_3$)
% % $\rightarrow$
% SetView(cam$_3$)
% % $\rightarrow$
% Turn(back)
% % $\rightarrow$
% RenderRGB
% \\
% If I am standing at the same spot and facing the same direction as shown in image 1, then I turn right and move forward, will I get closer to the pink plush toy and headboard?
% & $V_{\text{ini}} =$ camera$_1$
% & rotate(right), move(forward)
% & direction(ego, plush toy)
% & Build
% % $\rightarrow$
% Query(cam$_1$)
% % $\rightarrow$
% SetView(cam$_1$)
% % $\rightarrow$
% Turn(right)
% % $\rightarrow$
% Step(forward)
% % $\rightarrow$
% RenderRGB
% \\
% \bottomrule
% \end{tabular}
% \caption{
% Examples of spatial reasoning decomposition. Most spatial reasoning problems can be formulated as viewpoint selection, viewpoint refinement, and entity comparison under the transformed viewpoint.
% }
% \label{tab:decomposition}
% \end{table*}
\begin{table*}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{4pt}
\caption{
Examples of spatial reasoning decomposition. Each case selects a reference viewpoint when needed, applies optional viewpoint refinement, and then compares entities under the transformed viewpoint.
}
\begin{tabular}{p{0.36\textwidth} p{0.58\textwidth}}
\toprule
\textbf{Problem} & \textbf{Decomposition} \\
\midrule
How many tables are in the room?
&
\textbf{Reference:} none.
\newline
\textbf{Refinement:} none.
\newline
\textbf{Comparison:} count(table).
\newline
\textbf{Primitives:} Build $\rightarrow$ Query(table) $\rightarrow$ BEV(table).
\\
\midrule
In which direction did I move from the first view to the second view?
&
\textbf{Reference:} $V_{\text{ini}} =$ camera$_1$.
\newline
\textbf{Refinement:} none.
\newline
\textbf{Comparison:} direction(ego, camera$_2$).
\newline
\textbf{Primitives:} Build $\rightarrow$ Query(cam$_1$) $\rightarrow$ SetView(cam$_1$) $\rightarrow$ Query(cam$_2$) $\rightarrow$ BEV(cam$_2$).
\\
\midrule
What is behind me if I stand at the same spot and facing direction as image 3?
&
\textbf{Reference:} $V_{\text{ini}} =$ camera$_3$.
\newline
\textbf{Refinement:} rotate(back).
\newline
\textbf{Comparison:} visibility(ego, object).
\newline
\textbf{Primitives:} Build $\rightarrow$ Query(cam$_3$) $\rightarrow$ SetView(cam$_3$) $\rightarrow$ Turn(back) $\rightarrow$ RGB.
\\
\midrule
If I am standing at the same spot and facing the same direction as shown in image 1, then I turn right and move forward, will I get closer to the pink plush toy and headboard?
&
\textbf{Reference:} $V_{\text{ini}} =$ camera$_1$.
\newline
\textbf{Refinement:} rotate(right), move(forward).
\newline
\textbf{Comparison:} direction(ego, plush toy).
\newline
\textbf{Primitives:} Build $\rightarrow$ Query(cam$_1$) $\rightarrow$ SetView(cam$_1$) $\rightarrow$ Turn(right) $\rightarrow$ Step(forward) $\rightarrow$ RGB.
\\
\bottomrule
\end{tabular}
\label{tab:decomposition}
\vspace{-3mm}
\end{table*}
\begin{table*}[t]
\centering
\begin{minipage}[t]{\textwidth}
\centering
\scriptsize
\caption{
Spatial reasoning primitives and stage-specific constraints that together define valid interactions between the planner and explicit spatial memory.
}
\begin{tabular}{p{3cm} p{8.7cm}}
\toprule
\textbf{Primitive Type} & \textbf{Operations} \\
\midrule
\textbf{Memory Construction} & \texttt{build\_static\_memory},
\texttt{build\_dynamic\_memory}
\\
\textbf{Memory Query} & \texttt{query\_camera\_pose}, \texttt{query\_3d\_object\_location}
\\
\textbf{Memory Transformation} & \texttt{set\_viewpoint}, \texttt{step\_camera}, \texttt{turn\_camera}
\\
\textbf{Rendering} & \texttt{render\_egocentric}, \texttt{render\_semantic\_bev}, \texttt{render\_rgb\_bev}
\\
\bottomrule
\end{tabular}
\vspace{1mm}
{\footnotesize (a) Spatial reasoning primitives.}
\vspace{2mm}
\end{minipage}
\hfill
\begin{minipage}[t]{\textwidth}
\centering
\scriptsize
\begin{tabular}{p{3.7cm} p{6.0cm}}
\toprule
\textbf{Reasoning Stage} & \textbf{Allowed Primitive Types} \\
\midrule
Reference Viewpoint Selection & Memory Construction, Memory Query
\\
Viewpoint Refinement & Memory Query, Memory Transformation, Rendering
\\
Entity Comparison & Memory Query, Rendering
\\
\bottomrule
\end{tabular}
\vspace{1mm}
{\footnotesize (b) Allowed primitives for each reasoning stage.}
\end{minipage}
\label{tab:primitives}
\end{table*}
\begin{table}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{4pt}
\caption{Representative AST validation rules in the main text, organized by taxonomy category. The full taxonomy is deferred to the appendix.}
\begin{tabular}{p{0.28\linewidth} p{0.28\linewidth} p{0.34\linewidth}}
\toprule
\textbf{Category} & \textbf{Representative rule} & \textbf{Description} \\
\midrule
Program syntax
& Restricted Python subset
& Decorators, default arguments, variadic arguments, keyword-only arguments, and return annotations are disallowed. \\
\midrule
Tool usage
& Whitelisted primitives
& Only predefined spatial memory primitives in the DSL may be called. \\
\midrule
Dependency consistency
& Define-before-use
& Variables must be defined before they are referenced. \\
\midrule
Viewpoint-state consistency
& Stale-query invalidation
& Query results derived under a previous viewpoint become invalid after viewpoint-changing operations and must be recomputed. \\
\midrule
Execution discipline
& No dangling motion
& Each camera-motion operation must be followed by at least one subsequent render call. \\
\midrule
Plan consistency
& Motion-direction alignment
& Camera motions must exactly match the directions specified by the decomposition. \\
\bottomrule
\end{tabular}
\label{tab:ast-taxonomy-summary}
% \vspace{-3mm}
\end{table}
% \section{Experiments}
% \subsection{Setup}
% \noindent \textbf{Visual Auxiliary Models.} Our pipeline augments VLMs with access to visual auxiliary models to retrieve information from spatial memory. These models are used both for memory construction and semantic querying. We use Pi3~\cite{wang2026pi} to reconstruct the input image sequence. In addition, the metric-depth variant Pi3X is used when the agent needs to estimate object sizes at real-world scale. All images used for reconstruction are downsampled to a resolution with the short edge set to 378 pixels. To better utilize video inputs, we sample denser frame sequences for Pi3 reconstruction (i.e., 64 frames), while using sparser inputs for VLM reasoning (i.e., 16 frames) to reduce visual redundancy and accelerate reasoning. For the \texttt{query\_3d\_object\_position} operation, we use SAM3 to perform semantic segmentation. For video inputs, SAM3 processes all 64 frames to support cross-view instance merging. To ensure higher precision, we set the confidence threshold to 0.65.
% \noindent \textbf{Benchmarks.} We evaluate our pipeline on three spatial reasoning benchmarks covering multi-view images, static-scene videos, and dynamic-scene videos. We use MindCube~\cite{yin2025spatial} to evaluate reasoning under multi-view observations. We uniformly sample 50 problems from each camera-motion category (rotation, motion, and transition), resulting in a 150-sample evaluation set MindCube-Tiny. Notably, we observe that the original MindCube benchmark contains strong textual cues for spatial reasoning, such as:
% \emph{``Based on these four images (image 1, 2, 3, and 4) showing the blue table from different viewpoints (front, left, back, and right), ...''}
% These cues may simplify the perception process and provide shortcuts for reasoning.
% To study this effect, we conduct controlled experiments on MindCube-Tiny.
% We remove textual hints and shuffle both the image order and answer options while evaluating GPT-5-mini and Gemini-3-flash.
% As shown in Table~\ref{tab:debias}, the accuracy of Gemini-3-flash drops by more than 5\% after removing textual hints, and decreases by an additional 6\% when image orders are shuffled.
% Similar trends are observed for GPT-5-mini. This suggests that textual hints and fixed input ordering introduce undesirable biases into evaluation. Therefore, we remove all textual hints and randomize the ordering of input images and answer options, constructing a debiased version of MindCube-Tiny for subsequent evaluation.
% For video-based static scene, we sampled 50 problems for each problem category from VSI-Bench~\cite{yang2025thinking}, resulting 400 questions for final evaluation.
% For dynamic scenes, we sample 50 examples from both the ego-centric and exo-centric splits of VLM4D~\cite{zhou2025vlm4d} to evaluate spatial reasoning under changing environments.
% \noindent\textbf{Models.} As a test-time scaling framework, we primarily focus on improving strong scalable baselines. Therefore, we evaluate Reasmory on GPT-5-mini and Gemini-3-flash. We access these models through API calls using the same default temperature and reasoning settings across all benchmarks. For comparison, we consider two categories of baselines: (1) spatial QA fine-tuned models, including SpatialMLLM~\cite{wu2026spatialmllm}, SpatialLadder~\cite{li2026spatialladder}, and SpaceOM~\cite{VQASynth}; and (2) test-time scaling approaches. MindJourney~\cite{yang2026mindjourney} uses video generation models to simulate camera motion, while Think3D~\cite{zhang2026think3d} also leverages 3D reconstruction for reasoning, but without explicit primitive design or constrained interaction.
% \input{tables/table-debias}
% \input{tables/table-mindcube}
% \subsection{Main Results}
% %\subsubsection{MindCube}
% \noindent \textbf{Multi-view Images.} Table~\ref{tab:eval-mindcube} presents results on the de-biased MindCube-Tiny benchmark, where each sample contains 2--4 input images. We first observe that existing spatially fine-tuned models are not competitive with frontier closed-source VLMs. In particular, Spatial-MLLM achieves only 38.3\% overall accuracy. We hypothesize that these relatively small-scale models (3B/4B) struggle to generalize to the diverse spatial configurations in MindCube, which differ substantially from their training distributions. Among test-time scaling approaches, we find that existing methods typically improve only specific subsets of problems. For example, MindJourney improves GPT-5-mini by 10\% on \textit{Around} problems, while Think3D improves GPT-5-mini by 22\% on \textit{Among} problems. However, these gains do not generalize consistently across all categories, often resulting in limited or even negative overall improvements. In contrast, Reasmory consistently improves performance across all problem categories and both frontier models. Specifically, Reasmory improves GPT-5-mini from 58.0\% to 76.0\% (+18.0\%) and Gemini-3-flash from 68.7\% to 86.0\% (+17.3\%). Notably, our method achieves balanced improvements across \textit{Among}, \textit{Rotation}, and \textit{Around} tasks, suggesting that explicit spatial memory and constrained program execution provide a more general and robust mechanism for spatial reasoning than existing test-time scaling approaches.
% %\subsubsection{VSI-Bench}
% \noindent \textbf{Video-based Static Scenes.} Table~\ref{tab:eval-vsi} presents results on VSI-Bench-Tiny, a video-based spatial reasoning benchmark for static scenes. Compared with multi-view image settings, all models exhibit substantially lower performance, with frontier VLMs achieving only around 50\% overall accuracy. This suggests that long video inputs introduce severe visual redundancy and significantly increase the difficulty of spatial reasoning. Compared with MindCube-Tiny, the performance gap between spatially fine-tuned models and frontier VLMs becomes smaller. We hypothesize that this is because VSI-Bench is closer to the training distribution of models such as SpatialMLLM and SpatialLadder. In contrast, SpaceOM, which is primarily trained in single-image settings, fails to generalize to video-based reasoning tasks. Among test-time scaling approaches, MindJourney performs poorly in video settings. Since the original method is designed for two-image inputs (an anchor image with a helper image), we adapt it by using one frame as an anchor image and stitching the remaining 16 frames into a $4 \times 4$ grid as the helper input. The resulting performance degradation suggests that video generation-based imagination discards a large amount of visual information from the input video, limiting its effectiveness for long-context spatial reasoning.
% For Think3D, although the method provides access to 3D reconstruction tools, performance on several sub-tasks becomes worse than direct free-form reasoning. We attribute this behavior to unstable or inconsistent interaction with the reconstruction tools, indicating that unconstrained tool usage may confuse the model during reasoning.
% In contrast, Reasmory consistently improves or maintains performance across nearly all sub-tasks and both frontier models. Specifically, Reasmory improves GPT-5-mini from 49.6\% to 55.8\% and Gemini-3-flash from 53.7\% to 65.0\%. Notably, our method achieves strong gains on geometry-intensive tasks such as \textit{Absolute Distance}, \textit{Relative Distance}, and \textit{Room Size}, demonstrating that explicit spatial memory and constrained program execution are particularly effective for long-context spatial reasoning in video settings.
% \input{tables/table-vsi}
% \noindent \textbf{Video-based Dynamic Scene.} Table~\ref{tab:eval-vlm4d} presents results on VLM4D-Real, which evaluates spatial reasoning in dynamic environments containing both camera motion and object motion. Compared with static-scene settings, dynamic scenes introduce additional temporal complexity, requiring models to jointly reason about scene geometry and motion dynamics. All the spatially fine-tuned models struggle with the dynamic input. Among test-time scaling approaches, MindJourney fails to improve performance in dynamic scenes and often causes performance degradation. While generated videos may help hallucinate plausible novel viewpoints in static scenes, they struggle to preserve accurate object dynamics and temporal consistency under motion-heavy settings, leading to unreliable reasoning trajectories. Think3D achieves moderate improvements over direct reasoning baselines, indicating that explicit 3D reasoning is beneficial even in dynamic environments. However, the gains remain limited because the method mainly relies on static reconstruction and unconstrained interaction with tools, making it difficult to consistently capture and utilize temporal dynamics during reasoning. In contrast, Reasmory achieves the best overall performance across both ego-centric and exo-centric settings. Specifically, Reasmory improves GPT-5-mini from 65.3\% to 72.7\% and Gemini-3-flash from 76.0\% to 82.0\%. Unlike static-memory approaches, our method incorporates Flow3R as dynamic spatial memory, enabling the model to explicitly track object motion and maintain temporally consistent spatial representations. These results suggest that combining explicit dynamic memory with constrained program execution is particularly effective for spatial reasoning in dynamic environments.
% \input{tables/table-vlm4D}
% %\subsubsection{}
% %\subsubsection{Ablation}
% \noindent \textbf{Ablation.}
% \input{tables/table-ablation}
% Table~\ref{tab:ablation} presents ablation results under three settings: vanilla VLMs, VLMs augmented with spatial primitives without verification, and VLMs augmented with both spatial primitives and the DSL verifier.
% We observe that directly exposing spatial primitives to VLMs without constrained interaction does not consistently improve performance and can even degrade reasoning accuracy on several benchmarks. In particular, both GPT and Gemini experience noticeable drops on MindCube and VSI-Bench when using primitives without verification. This suggests that unconstrained tool interaction introduces unstable reasoning trajectories and inconsistent state transitions, which may confuse the model during reasoning.
% In contrast, adding the DSL verifier consistently improves performance across all benchmarks and both frontier models. The gains are especially significant on MindCube and VSI-Bench, where structured spatial reasoning and viewpoint manipulation are more critical. These results demonstrate that the effectiveness of Reasmory does not solely come from providing additional tools or memory, but from constraining and verifying how models interact with spatial memory.
% \subsection {Component Analysis}
% \noindent \textbf{3D Grounding.}
% \input{tables/table-grounding}
% To ensure correct semantic injection for the memory, we compare our semantic augmentation pipeline with MaskClustering, a representative mask-merging based approach. Table~\ref{tab:grounding} presents results on open-vocabulary 3D grounding using ScanNet scenes. We randomly sample 20 scenes and evaluate grounding performance on 10 object categories.
% We observe that MaskClustering achieves reasonable performance only under dense-view settings (231 frames per scene), while performance drops substantially under sparse observations. In contrast, our method significantly improves grounding quality in sparse-view settings, achieving much higher mAP50 and mAP25 scores.
% We attribute this improvement to two factors. First, SAM3 provides stronger open-vocabulary segmentation capability. In particular, for several categories such as \emph{door}, MaskClustering fails completely with AP scores of 0.0, while our method is still able to recover valid object instances. Second, our geometry-guided cross-view merging is more robust under sparse multi-view observations, which better matches the practical input setting of VLM reasoning pipelines.
% These results suggest that our semantic augmentation strategy produces more reliable object-level spatial memory under limited observations.
% \noindent \textbf{Planner's Performance.}
% Table~\ref{tab:pass-ratio} reports the planner’s program validation pass rate on the de-biased MindCube benchmark. We evaluate pass@k, where a sample is considered successful if at least one valid program is generated within $k$ attempts.
% We observe that both GPT-5-mini and Gemini-3-flash achieve relatively high pass@1 scores, indicating that frontier VLMs already possess strong capabilities for generating structured spatial reasoning programs. More importantly, the pass rate increases rapidly with iterative regeneration guided by compiler feedback. GPT-5-mini improves from 82.6\% at pass@1 to 95.3\% at pass@3, while Gemini-3-flash reaches 100\% validity within three attempts.
% These results demonstrate the effectiveness of our constrained interaction pipeline. Although VLMs may initially produce invalid programs due to local structural mistakes, the DSL verifier and compiler feedback mechanism enable efficient self-correction, allowing nearly all planning trajectories to eventually become executable. This significantly improves the stability and reliability of spatial reasoning compared with unconstrained free-form tool interaction.
% \input{tables/table-pass-ratio}
% \noindent \textbf {Reasoning Trajectory.}
% I will show a real reasoning trajectory here using a figure and write some analysis for the model's output. So skip auditing this section for now.
\section{Experiments}
We evaluate Reasmory on three input modalities that stress complementary forms of evidence: multi-view images from MindCube~\cite{yin2025spatial} with a small set of observed views, static-scene videos from VSI-Bench~\cite{yang2025thinking} with cues distributed across redundant frames, and dynamic-scene videos from VLM4D~\cite{zhou2025vlm4d} with both camera and object motion. We compare against direct inference with GPT-5-mini and Gemini-3-flash, spatial QA fine-tuned models~\cite{wu2026spatialmllm,li2026spatialladder,VQASynth}, and test-time scaling methods including MindJourney~\cite{yang2026mindjourney} and Think3D~\cite{zhang2026think3d}. Sec.~\ref{sec:exp_setup} details the setup, Sec.~\ref{sec:exp_main} presents benchmark comparisons, Sec.~\ref{sec:exp_ablation} ablates DSL verification, Sec.~\ref{sec:exp_reliability} analyzes semantic object grounding and planner validity, and Sec.~\ref{sec:exp_example} provides an end-to-end Reasmory reasoning example.
\subsection{Experimental Setup}
\label{sec:exp_setup}
\noindent \textbf{Vision Modules for Spatial Memory.} Reasmory uses external vision modules to construct spatial memory and support semantic object queries. Specifically, we use Pi3~\cite{wang2026pi} to reconstruct point clouds, camera poses, and depth maps from input image sequences. In addition, the metric-depth variant Pi3X is used when the agent needs to estimate object sizes at real-world scale. For experiments on dynamic-scene videos, we use Flow3r~\cite{cong2026flow3r} to estimate geometry and camera poses across video frames. All images used for reconstruction are downsampled to a resolution with the short edge set to 378 pixels. To better utilize video inputs, we sample denser frame sequences for Pi3 reconstruction (\eg, 64 frames), while using sparser inputs for VLM reasoning (\eg, 16 frames) to reduce visual redundancy and accelerate reasoning. For the \texttt{query\_3d\_object\_location} operation, we use SAM3~\cite{carion2025sam} to perform semantic segmentation. For video inputs, SAM3 processes all 64 frames to support cross-view merging. To ensure higher precision, we set the confidence threshold to 0.65.
\noindent \textbf{Benchmarks.} We evaluate our pipeline on three spatial reasoning benchmarks covering multi-view images, static-scene videos, and dynamic-scene videos. For multi-view reasoning, we use MindCube~\cite{yin2025spatial} and uniformly sample 50 problems from each of the \textit{Among}, \textit{Rotation}, and \textit{Around} categories, resulting in a 150-sample evaluation set, MindCube-Tiny. We observe that the original MindCube benchmark contains strong textual cues for spatial reasoning, such as:
\emph{``Based on these four images (image 1, 2, 3, and 4) showing the blue table from different viewpoints (front, left, back, and right), ...''}
Such descriptions leak viewpoint information by explicitly annotating the camera pose of each image. This substantially reduces the need for cross-view correspondence reasoning and introduces a shortcut that may overestimate a model's true spatial reasoning capability.
To study this effect, we remove textual hints and shuffle both image order and answer options when evaluating GPT-5-mini and Gemini-3-flash. As shown in Table~\ref{tab:debias}, Gemini-3-flash drops by more than 5\% after removing textual hints and by an additional 6\% after shuffling image order; GPT-5-mini follows the same trend. We therefore use this debiased MindCube-Tiny split for subsequent experiments. For static-scene videos, we sample 50 problems from each VSI-Bench~\cite{yang2025thinking} category, yielding 400 questions. For dynamic scenes, we sample 50 examples from both the ego-centric and exo-centric splits of VLM4D~\cite{zhou2025vlm4d}.
\noindent \textbf{Models.} Reasmory is a test-time framework that can be applied to different backbone VLMs. We evaluate it with two strong frontier models, GPT-5-mini and Gemini-3-flash, to test whether structured spatial memory can improve already capable multimodal reasoners. We access both models through APIs using the same default temperature and reasoning settings across all benchmarks. For comparison, we include two groups of baselines. The first group consists of spatial QA fine-tuned models, including SpatialMLLM~\cite{wu2026spatialmllm}, SpatialLadder~\cite{li2026spatialladder}, and SpaceOM~\cite{VQASynth}. The second group consists of test-time scaling methods: MindJourney~\cite{yang2026mindjourney}, which uses video generation models to simulate camera motion, and Think3D~\cite{zhang2026think3d}, which also leverages 3D reconstruction for reasoning but without explicit primitive design or constrained interaction. For different agents in our pipeline, we instantiate multiple copies of the same backbone model, assigning them specialized roles using different prompts.
% including question decomposition, program planning, verification, and final answer generation.
\input{tables/table-debias}
\input{tables/table-mindcube}
\subsection{Benchmark Results}
\label{sec:exp_main}
%\subsubsection{MindCube}
\noindent \textbf{Multi-view Images.} Table~\ref{tab:eval-mindcube} presents results on the debiased MindCube-Tiny benchmark, where each sample contains 2--4 input images. Existing spatially fine-tuned models~\cite{wu2026spatialmllm,li2026spatialladder,VQASynth} are not competitive with frontier closed-source VLMs. For example, Spatial-MLLM achieves only 38.3\% accuracy. We hypothesize that these relatively small-scale models (3B/4B) struggle to generalize to the diverse spatial configurations in MindCube, which may differ from their training distributions. Among test-time scaling approaches, existing methods improve only specific subsets of problems. For example, MindJourney improves GPT-5-mini by 10 points on \textit{Around} problems, while Think3D improves GPT-5-mini by 22 points on \textit{Among} problems. However, these gains do not generalize across categories, leading to limited or even negative overall improvements. In contrast, Reasmory substantially improves performance for both frontier models. GPT-5-mini improves from 58.0\% to 76.0\% (+18.0), and Gemini-3-flash improves from 68.7\% to 86.0\% (+17.3). Reasmory also achieves balanced performance across \textit{Among}, \textit{Rotation}, and \textit{Around} tasks, suggesting that explicit spatial memory and constrained program execution provide a more general mechanism for multi-view spatial reasoning than existing test-time scaling approaches.
%\subsubsection{VSI-Bench}
\noindent \textbf{Video-based Static Scenes.} Table~\ref{tab:eval-vsi} presents results on VSI-Bench-Tiny, a video-based spatial reasoning benchmark for static scenes. Compared with de-biased MindCube-Tiny, all methods achieve lower overall accuracy, and direct frontier VLM inference remains around 50\%. This suggests that long video inputs introduce substantial visual redundancy and make spatial evidence harder to retrieve. The gap between spatially fine-tuned models and frontier VLMs is also smaller than on MindCube-Tiny. This may be because VSI-Bench is closer to the training distributions of spatial models, such as Spatial-MLLM-120K~\cite{wu2026spatialmllm} and SpatialLadder-26K~\cite{li2026spatialladder}. In contrast, SpaceOM~\cite{VQASynth}, which is primarily trained in single-image dataset SpaceThinker, fails to generalize to video-based reasoning tasks.
Among test-time scaling approaches, MindJourney~\cite{yang2026mindjourney} performs poorly in video settings. Since the original method is designed for two-image inputs, we adapt it by using one frame as the anchor image and stitching the remaining 16 frames into a $4 \times 4$ grid as the helper input. The resulting performance degradation suggests that generated-view reasoning discards substantial information from long video inputs, limiting its effectiveness for long-context spatial reasoning.
For Think3D~\cite{zhang2026think3d}, although the method provides access to 3D reconstruction tools, performance on several sub-tasks becomes worse than direct free-form reasoning. We attribute this to unstable interaction with reconstruction tools, indicating that unconstrained tool usage may confuse the model during reasoning.
In contrast, Reasmory achieves the best overall performance for both frontier backbones. It improves GPT-5-mini from 49.6\% to 55.8\% (+6.2) and Gemini-3-flash from 51.6\% to 65.0\% (+13.4). The gains are especially strong on geometry-intensive tasks such as \textit{Absolute Distance}, \textit{Relative Distance}, and \textit{Room Size}, demonstrating that explicit spatial memory and constrained program execution are particularly effective for long-context spatial reasoning in video settings.
\input{tables/table-vsi}
\noindent \textbf{Video-based Dynamic Scenes.} Table~\ref{tab:eval-vlm4d} presents results on VLM4D-Real~\cite{zhou2025vlm4d}, which evaluates spatial reasoning in dynamic environments with both camera motion and object motion. For this setting, Reasmory uses Flow3r~\cite{cong2026flow3r} to estimate geometry and camera poses across frames, as described in Sec.~\ref{sec:exp_setup}. Compared with static-scene settings, dynamic scenes introduce additional temporal complexity because models must reason jointly about scene geometry and motion. Spatially fine-tuned models~\cite{wu2026spatialmllm,li2026spatialladder,VQASynth} perform poorly in this setting, suggesting that dynamic video reasoning remains difficult for models trained primarily on static or simpler spatial QA data. Among test-time scaling approaches, MindJourney~\cite{yang2026mindjourney} fails to improve performance and often degrades accuracy. Generated videos may help simulate plausible novel viewpoints in static scenes, but they struggle to preserve object dynamics and temporal consistency in videos with camera and object motion. Think3D~\cite{zhang2026think3d} achieves moderate gains over direct GPT-5-mini inference, indicating that explicit 3D reasoning can still help in dynamic environments. However, its gains remain limited, likely because it relies on unconstrained interaction with reconstruction tools. In contrast, Reasmory achieves the best overall performance for both frontier backbones. It improves GPT-5-mini from 65.3\% to 72.7\% (+7.4) and Gemini-3-flash from 76.0\% to 82.0\% (+6.0). These results suggest that constrained program execution over spatial memory remains useful even when camera and object motion introduce additional temporal ambiguity.
\input{tables/table-vlm4D}
\subsection{Ablation Study}
\label{sec:exp_ablation}
\input{tables/table-ablation}
Table~\ref{tab:ablation} evaluates the role of DSL verification in Reasmory. We compare direct VLM inference, VLMs augmented with spatial primitives but without DSL verification, and the full Reasmory pipeline with both spatial primitives and DSL verification.
Directly exposing spatial primitives to VLMs without constrained interaction leads to inconsistent results. It degrades GPT-5-mini on MindCube and VSI-Bench, from 58.0\% to 53.2\% and from 49.6\% to 45.3\%, respectively. For Gemini-3-flash, spatial primitives without verification also reduce performance relative to direct inference on all three benchmarks. Although these primitives can help in some cases, such as GPT-5-mini on VLM4D, the gains remain smaller than those of the full pipeline. Adding the DSL verifier consistently gives the best performance across all benchmarks and both frontier models. These results show that Reasmory's gains do not come simply from exposing additional spatial primitives, but from constraining and verifying how models interact with spatial memory.
\subsection{Reliability Analysis}
\label{sec:exp_reliability}
We further analyze the reliability of two key parts of Reasmory: semantic 3D grounding for object-level memory, and planner reliability for generating executable DSL programs.
\noindent \textbf{3D Grounding Reliability.}
To evaluate semantic grounding in spatial memory, we compare our semantic augmentation pipeline with MaskClustering~\cite{yan2024maskclustering}, a representative mask-merging approach. Table~\ref{tab:grounding} reports open-vocabulary 3D grounding results on ScanNet~\cite{dai2017scannet}. We randomly sample 20 scenes and evaluate 10 object categories.
MaskClustering performs reasonably under dense-view settings (231 frames per scene), but degrades substantially under sparse observations. In the sparse-view setting, our method improves mAP50 from 16.7 to 38.4 and mAP25 from 35.7 to 68.4.
We attribute this improvement to two factors. First, SAM3~\cite{carion2025sam} provides stronger open-vocabulary segmentation. For example, MaskClustering fails completely on categories such as \emph{door}, while our method still recovers valid object instances. Second, our geometry-guided cross-view merging is more robust under sparse multi-view observations, matching the practical input setting of VLM reasoning pipelines. These results suggest that our semantic augmentation strategy produces more reliable object-level spatial memory under limited observations.
\noindent \textbf{Planner Reliability.}
Table~\ref{tab:pass-ratio} reports the planner's program validation pass rate on debiased MindCube. We evaluate pass@k, where a sample is considered successful if at least one valid program is generated within $k$ attempts.
Both GPT-5-mini and Gemini-3-flash achieve high pass@1 scores, indicating that frontier VLMs can often generate valid structured spatial reasoning programs. The pass rate further improves with compiler feedback: GPT-5-mini increases from 82.6\% at pass@1 to 95.3\% at pass@3, while Gemini-3-flash reaches 100\% validity within three attempts.
These results show that the DSL verifier and compiler feedback mechanism make planning reliable in practice, enabling most generated programs to become executable after only a few attempts.
% \input{tables/table-pass-ratio}
\input{tables/table-reliability}
\vspace{-5mm}
\subsection{End-to-End Reasmory Reasoning Example}
\label{sec:exp_example}
\begin{figure}[t]
\centering
% \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}
\includegraphics[width=\linewidth]{images/examples.png}
\vspace{-5mm}
\caption{\textbf{An end-to-end reasoning example.}
The trajectory illustrates how verification and repair refine a DSL plan before spatial-memory execution; see Sec.~\ref{sec:exp_example} for details.}
%The planner first generates an invalid program that is inconsistent with the decomposed spatial intent. The verifier detects the error and provides feedback for regeneration and automatic repair. The validated program is then executed over spatial memory to obtain enriched observations, allowing the reasoner to produce the correct answer.}
\label{fig:example}
\vspace{-5mm}
\end{figure}
Figure~\ref{fig:example} illustrates one complete Reasmory trajectory, from decomposition to verified execution and final answering.
Given a spatial reasoning question, the decomposer first extracts a structured representation of the task, including the reference viewpoint, required viewpoint transformations, and relevant entities. In this example, the decomposition identifies the initial viewpoint (Image~1), the required camera motion (turn right followed by reasoning about what is behind the observer), and the candidate objects involved in the final answer.
Conditioned on both the original question and the decomposition, the planner generates a DSL program for querying the spatial memory. The first generated plan builds the memory and performs a viewpoint transformation, but it omits the directional constraint encoded by \emph{behind}. As a result, the generated camera trajectory does not match the decomposition. Instead of executing an incorrect plan, the compiler detects the mismatch against the decomposition and rejects the program with explicit feedback.
After receiving the error message, the planner generates a revised program that correctly implements the required viewpoint transformation. The revised program still contains a minor implementation error, a missing return statement, which the compiler repairs automatically before validation. After verification, the execution engine deterministically executes the program and renders both egocentric and bird's-eye-view observations corresponding to the transformed viewpoint.
The resulting observations show that the final viewpoint faces the grey-green decorative wall. Based on this enriched context, the reasoner infers the spatial relationship and produces the correct answer. This example highlights two reliability benefits of Reasmory: decomposition-guided verification catches plans that deviate from the intended spatial reasoning, and compiler-assisted correction turns local program errors into recoverable steps before execution.
\begin{table}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{4pt}
\caption{Debiasing analysis on MindCube-Tiny. Removing textual viewpoint hints and shuffling image order both reduce performance, suggesting that the original benchmark leaks camera-pose information and partially bypasses cross-view correspondence reasoning.}
\begin{tabular}{lccc}
\toprule
& \textbf{Mindcube-Tiny} & \textbf{+Remove Hint} & \textbf{+Shuffle Image Order} \\
\midrule
\textbf{GPT-5-mini} & \textbf{59.4} & 58.6 (-0.8) & 58.0 (-1.4)\\
\textbf{Gemini-3-flash} & \textbf{81.9} & 74.6 (-7.3) & 68.7 (-13.2) \\
\bottomrule
\end{tabular}
\label{tab:debias}
\vspace{-1em}
\end{table}
\begin{table}[t]
\centering
\scriptsize
\caption{\textbf{Evaluation results on de-biased MindCube-Tiny.} We compare methods under each base model separately.
Bold numbers indicate the best result within the same base model family, and underlined numbers indicate the best result overall.
}
\label{tab:eval-mindcube}
\vspace{1mm}
\begin{tabular}{lcccc}
\toprule
\textbf{Method} & \textbf{Among $\uparrow$}& \textbf{Rotation $\uparrow$}& \textbf{Around $\uparrow$} & \textbf{Overall $\uparrow$} \\
\midrule
GPT-5-mini & 32.0 & 78.0 & 64.0 & 58.0\\
Gemini-3-flash & 56.0 & 70.0 & 74.0 & 68.7\\
% Claude-sonnet-4.6 \\
\midrule
\multicolumn{2}{l}{\textit{\textbf{Spatial Models}}}
\\
\midrule
SpaceOM-3B~\cite{VQASynth} & 38.0 & 32.0 & 62.0 & 44.0\\
Spatial-MLLM-4B~\cite{wu2026spatialmllm} & 47.5 & 32.5 & 35.0 & 38.3 \\
SpatialLadder-3B~\cite{li2026spatialladder} & 50.0 & 40.0 & 56.0 & 48.7\\
\midrule
\multicolumn{2}{l}{\textit{\textbf{Test-time Scaling Methods}}}\\
\midrule
Mindjourney (GPT-5-mini)~\cite{yang2026mindjourney} & 36.0 & 52.0 & 74.0& 54.0 \textcolor{red}{\textbf{(-4.0)}} \\
Mindjourney (Gemini-3-flash)~\cite{yang2026mindjourney} & 50.0 & 74.0& 74.0& 66.0 \textcolor{red}{(-2.7)}\\
Think3D (GPT-5-mini)~\cite{zhang2026think3d} & 54.0 & 56.0 & 62.0 & 56.7 \textcolor{red}{(-1.3)}\\
Think3D (Gemini-3-flash)~\cite{zhang2026think3d} & 54.0 & 66.0 & 82.0 & 67.3 \textcolor{red}{(-1.4)}\\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (GPT-5-mini)} & \textbf{78.0} & 76.0 & \textbf{74.0}& \textbf{76.0 (+18.00)}\\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (Gemini-3-flash)} & \underline{\textbf{82.0}} & \underline{\textbf{92.0}} & \underline{\textbf{84.0}} & \underline{\textbf{86.0 (+17.3)}}\\
\bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}
\begin{table}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1.1}
\caption{
\textbf{Evaluation results on video-input VSI-Bench-Tiny.} We compare methods under each base model separately.
Bold numbers indicate the best result within the same base model family, and underlined numbers indicate the best result overall.
}
\label{tab:eval-vsi}
\vspace{1mm}
\begin{tabular}{lccccccccc}
\toprule
\textbf{Method}
& \rotatebox{50}{\textbf{Obj. Count}}
& \rotatebox{50}{\textbf{Abs. Dist.}}
& \rotatebox{50}{\textbf{Obj. Size}}
& \rotatebox{50}{\textbf{Room Size}}
& \rotatebox{50}{\textbf{Rel. Dist.}}
& \rotatebox{50}{\textbf{Rel. Dir.}}
& \rotatebox{50}{\textbf{Route Plan}}
& \rotatebox{50}{\textbf{Appr. Order}}
& \rotatebox{50}{\textbf{Overall}}
\\
\midrule
GPT-5-mini
& 49.0 & 31.6 & 73.0 & 41.2 & 42.0 & 48.0 &\textbf{ 46.0} & 66.0 & 49.6 \\
Gemini-3-flash (16 frames)
& 46.4 & 15.2 & 68.6 & 48.4 & 56.0 & 58.0 & 48.0 & 72.0 & 51.6 \\
Gemini-3-flash (video)
& 45.9 & 21.8 & \underline{\textbf{75.0}} & 39.2 & 58.0 & 62.0 & 40.0 & \underline{\textbf{88.0}} & 53.7 \\
\midrule
\multicolumn{10}{l}{\textit{\textbf{Spatial Models}}}
\\
\midrule
SpaceOM-3B~\cite{VQASynth}
& 27.3 & 10.2 & 17.1 & 21.7 & 10.2 & 25.0 & 22.4 & 31.9 & 20.7 \\
Spatial-MLLM-4B~\cite{wu2026spatialmllm}
& \textbf{67.2} & 31.0 & 56.6 & 43.6 & 32.0 & 52.0 & 36.0 & 44.0 & 45.2 \\
SpatialLadder-3B~\cite{li2026spatialladder}
& 60.0 & 31.0 & 54.2 & 47.6 & 30.0 & 44.0 & 30.0 & 40.0 & 42.1 \\
\midrule
\multicolumn{10}{l}{\textit{\textbf{Other Test-time Scaling Methods}}}
\\
\midrule
Mindjourney (GPT-5-mini)~\cite{yang2026mindjourney}
& 47.0 & 25.0 & 64.5 & 23.0 & 20.0 & 50.0 & 30.0 & 50.0 & 38.7 \textcolor{red}{\textbf{(-10.9)}} \\
Mindjourney (Gemini-3-flash)~\cite{yang2026mindjourney}
& 25.5 & 36.0 & 59.5 & 56.5 & 35.0 & 50.0 & \underline{\textbf{65.0}} & 80.0 & 50.9 (-0.7) \\
Think3D (GPT-5-mini)~\cite{zhang2026think3d}
& 42.4 & 25.2 & 63.8 & 41.2 & 34.0 & 48.0 & 31.0 & 71.7 & 44.7 \textcolor{red}{(-4.9)} \\
Think3D (Gemini-3-flash)~\cite{zhang2026think3d}
& 45.8 & 10.2 & 62.8 & 46.4 & 54.0 & 52.0 & 58.0 & 52.0 & 47.3 \textcolor{red}{(-4.4)} \\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (GPT-5-mini)}
& 60.2 & \textbf{41.9} & 73.1 & \textbf{50.2} & \textbf{52.0} & \textbf{55.6} & 42.0 & 71.4 & \textbf{55.8 (+6.2)} \\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (Gemini-3-flash)}
& \underline{\textbf{69.0}}
& \underline{\textbf{50.4}}
& 68.4
& \underline{\textbf{62.0}}
& \underline{\textbf{71.4}}
& \underline{\textbf{65.0}}
& 62.0
& 72.0
& \underline{\textbf{65.0 (+13.4)}} \\
\bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}
\begin{table}[t]
\centering
\scriptsize
\caption{
\textbf{Evaluation results on video-input VLM4D-Real.} We compare methods under each base model separately.
Bold numbers indicate the best result within the same base model family, and underlined numbers indicate the best result overall.
}
\label{tab:eval-vlm4d}
% \vspace{1mm}
\begin{tabular}{lccc}
\toprule
\textbf{Method} & \textbf{Ego-centric $\uparrow$}& \textbf{Exo-centric $\uparrow$}& \textbf{Overall $\uparrow$} \\
\midrule
GPT-5-mini & 61.4 & 68.8 & 65.3\\
Gemini-3-flash & 84.0 & 68.0 & 76.0\\
% Claude-sonnet-4.6 \\
\midrule
\multicolumn{2}{l}{\textit{\textbf{Spatial Models}}}
\\
\midrule
SpaceOM-3B~\cite{VQASynth} & 34.0 & 42.9 & 38.5\\
Spatial-MLLM-4B~\cite{wu2026spatialmllm} & 42.0 & 18.0 & 30.0 \\
SpatialLadder-3B~\cite{li2026spatialladder} & 38.0 & 40.0 & 39.0\\
\midrule
\multicolumn{2}{l}{\textit{\textbf{Test-time Scaling Methods}}}\\
\midrule
Mindjourney (GPT-5-mini)~\cite{yang2026mindjourney} &46.0 & 54.0 & 50.0 \textcolor{red}{\textbf{(-15.3)}} \\
Mindjourney (Gemini-3-flash)~\cite{yang2026mindjourney} & 66.0 & 60.0 & 63.0 \textcolor{red}{(-13.0)}\\
Think3D (GPT-5-mini)~\cite{zhang2026think3d} & 65.3 & 70.0 & 67.6 (+2.3)\\
Think3D (Gemini-3-flash)~\cite{zhang2026think3d} & 72.0 & 66.0 & 69.0 \textcolor{red}{(-7.0)}\\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (GPT-5-mini)} & \textbf{71.4} & \textbf{74.0} & \textbf{72.7 (+7.4)} \\
%\rowcolor{gray!15}
\rowcolor{cyan!15}
\textbf{Reasmory (Gemini-3-flash)} & \textbf{88.0} & \textbf{76.0 }& \textbf{82.0 (+6.0)}\\
\bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}
\begin{table}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{6pt}
\vspace{2mm}
\caption{Ablations in three settings: vanilla VLMs, VLMs augmented with spatial primitives without verification, and VLMs augmented with spatial primitives and DSL verifier.}
\begin{tabular}{llccc}
\toprule
\textbf{Setting} & \textbf{Model} & \textbf{MindCube} & \textbf{VSI-Bench} & \textbf{VLM4D} \\
\midrule
\multirow{2}{*}{Vanilla VLM}
& GPT & 58.0 & 49.6 & 65.3 \\
& Gemini & 68.7 & 51.6 & 76.0 \\
% \hline
\multirow{2}{*}{+ Primitives (no verifier)}
& GPT & 53.2 & 45.3 & 70.4 \\
& Gemini & 63.5 & 50.4 & 73.0\\
% \hline
\multirow{2}{*}{+ Primitives + DSL verifier}
& GPT & \textbf{76.0} & \textbf{55.8} & \textbf{72.7} \\
& Gemini & \underline{\textbf{86.0}} & \underline{\textbf{65.0}} & \underline{\textbf{82.0}} \\
\bottomrule
\end{tabular}
\label{tab:ablation}
\end{table}
\begin{table*}[t]
\centering
\caption{
Reliability analysis for 3D semantic grounding and planner's behavior in Reasmory. Our SAM-3 based instance merging achieves high mAP under sparse input settings. The feedback loop for plan generation guarantees most of the plans are valid after three attempts.
}
\vspace{1mm}
\begin{minipage}[t]{0.61\textwidth}
\centering
\scriptsize
\begin{tabular}{lcccc}
\toprule
& \textbf{Frames} & \textbf{mAP} & \textbf{mAP50} & \textbf{mAP25} \\
\midrule
MaskClustering (dense)~\cite{yan2024maskclustering} & 231 & \textbf{19.0} & 33.9 & 51.0\\
MaskClustering (sparse)~\cite{yan2024maskclustering} & \textbf{32} & 5.9 & 16.7 & 35.7 \\
\textbf{Ours} & \textbf{32} & \underline{12.1} & \textbf{38.4} & \textbf{68.4} \\
\bottomrule
\label{tab:grounding}
\end{tabular}
% \vspace{1mm}
{\footnotesize (a) 2D mask merging result on ScanNet subset.}
\end{minipage}
\hfill
\begin{minipage}[t]{0.38\textwidth}
\centering
\scriptsize
\begin{tabular}{lcc}
\toprule
& \textbf{GPT-5-mini} & \textbf{Gemini-3-flash} \\
\midrule
\textbf{pass@1} & 82.6 & 78.0 \\
\textbf{pass@2} & 93.3 & 98.0 \\
\textbf{pass@3} & 95.3 & 100.0 \\
\bottomrule
\label{tab:pass-ratio}
\end{tabular}
% \vspace{1mm}
{\footnotesize (b) Planner's pass ratio on MindCube-Tiny debiased version.}
\end{minipage}
\label{tab:reliability}
\end{table*}
\vspace{-4mm}
\section{Conclusion}
%We introduced \textbf{Reasmory}, a framework that treats 3D reconstruction as explicit spatial memory for VLM spatial reasoning. Reasmory constructs reconstruction-based memory from multi-view images or videos, augments it with semantically grounded object instances, and constrains model interaction through validated DSL programs. This design addresses two sources of brittleness in existing VLM reasoning pipelines: redundant visual evidence that is difficult to organize, and unconstrained tool use that can lead to invalid or inconsistent spatial operations.
%
%Across multi-view image, static-scene video, and dynamic-scene video benchmarks, Reasmory improves strong frontier VLM backbones and outperforms spatially fine-tuned models and existing test-time scaling baselines. Ablations show that these gains do not come simply from exposing spatial primitives, but from verifying how models query, transform, and render spatial memory. Reliability analyses further show that semantic grounding remains effective under sparse observations and that compiler feedback enables high planner validity within a few attempts.
%Reasmory still depends on the quality of the underlying reconstruction and grounding modules, and can fail when object grounding is ambiguous, geometry is heavily occluded, or dynamic scenes contain complex interactions beyond the recovered spatial memory. Future work may improve the fidelity of dynamic memory, integrate stronger uncertainty estimates, and extend the DSL to support richer spatial and temporal reasoning operations.
We introduce \textbf{Reasmory}, a framework that uses 3D reconstruction as explicit spatial memory for VLM spatial
reasoning. Reasmory builds memory from multi-view images or videos, augments it with semantically grounded object
instances, and constrains model interaction through validated DSL programs. This reduces brittleness from redundant visual evidence and unconstrained tool use, which can cause invalid or inconsistent spatial operations.
%
Across multi-view image, static-scene video, and dynamic-scene video benchmarks, Reasmory improves frontier VLMs and
outperforms spatially fine-tuned models and test-time scaling baselines. Ablations show that the gains come not from spatial primitives alone, but from verifying how models query, transform, and render memory. Reliability analyses show effective grounding under sparse observations and high planner validity after compiler feedback.
%
Reasmory still depends on reconstruction and grounding quality, and may fail with ambiguous grounding, heavy
occlusion, or complex dynamic interactions beyond the recovered memory. Future work may improve dynamic-memory
fidelity, add uncertainty estimates, and extend the DSL for richer spatial and temporal reasoning.
\begin{thebibliography}{70}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bai et~al.(2025)Bai, Cai, Chen, Chen, Chen, Cheng, Deng, Ding, Gao, Ge, et~al.]{bai2025qwen3}
Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et~al.
\newblock Qwen3-vl technical report.
\newblock \emph{arXiv preprint arXiv:2511.21631}, 2025.
\bibitem[Carion et~al.(2025)Carion, Gustafson, Hu, Debnath, Hu, Suris, Ryali, Alwala, Khedr, Huang, et~al.]{carion2025sam}
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan~Vasudev Alwala, Haitham Khedr, Andrew Huang, et~al.
\newblock Sam 3: Segment anything with concepts.
\newblock \emph{arXiv preprint arXiv:2511.16719}, 2025.
\bibitem[Chen et~al.(2022)Chen, Xu, Geiger, Yu, and Su]{chen2022tensorf}
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.
\newblock Tensorf: Tensorial radiance fields.
\newblock In \emph{European conference on computer vision}, pages 333--350. Springer, 2022.
\bibitem[Chen et~al.(2024)Chen, Xu, Kirmani, Driess, Florence, Ichter, Sadigh, Guibas, and Xia]{chen2024spatialvlm}
Boyuan Chen, Zhuo Xu, Sean Kirmani, Danny Driess, Pete Florence, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia.
\newblock Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.
\newblock \emph{arXiv preprint arXiv:2401.12168}, 2024.
\bibitem[Chen et~al.(2026)Chen, Chen, Xiu, Geiger, and Chen]{chen2026tttr}
Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen.
\newblock {TTT}3r: 3d reconstruction as test-time training.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=aMs6FtNaY5}.
\bibitem[Chen et~al.(2025)Chen, Zhang, Yu, Luo, Sun, Pan, Feng, Pei, Cai, and Huang]{chen2025think}
Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xufang Luo, Mingze Sun, Zihao Pan, Yan Feng, Peng Pei, Xunliang Cai, and Ruqi Huang.
\newblock Think with 3d: Geometric imagination grounded spatial reasoning from limited views.
\newblock \emph{arXiv preprint arXiv:2510.18632}, 2025.
\bibitem[Cong et~al.(2026)Cong, Zhao, Jeon, and Tulsiani]{cong2026flow3r}
Zhongxiao Cong, Qitao Zhao, Minsik Jeon, and Shubham Tulsiani.
\newblock Flow3r: Factored flow prediction for scalable visual geometry learning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2026.
\newblock URL \url{https://openaccess.thecvf.com/content/CVPR2026/html/Cong_Flow3r_Factored_Flow_Prediction_for_Scalable_Visual_Geometry_Learning_CVPR_2026_paper.html}.
\bibitem[Dai et~al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nie{\ss}ner]{dai2017scannet}
Angela Dai, Angel~X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nie{\ss}ner.
\newblock Scannet: Richly-annotated 3d reconstructions of indoor scenes.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, 2017.
\bibitem[Ellis et~al.(2021)Ellis, Wong, Nye, Sabl\'{e}-Meyer, Morales, Hewitt, Cary, Solar-Lezama, and Tenenbaum]{ellis2021coder}
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sabl\'{e}-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua~B. Tenenbaum.
\newblock Dreamcoder: bootstrapping inductive program synthesis with wake-sleep library learning.
\newblock In \emph{Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation}, PLDI 2021, page 835–850, New York, NY, USA, 2021. Association for Computing Machinery.
\newblock ISBN 9781450383912.
\newblock \doi{10.1145/3453483.3454080}.
\newblock URL \url{https://doi.org/10.1145/3453483.3454080}.
\bibitem[Fan et~al.(2025)Fan, Zhang, Li, Zhang, Chen, Hu, Wang, Qu, Zhou, Wang, et~al.]{fan2025vlm}
Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, et~al.
\newblock Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction.
\newblock \emph{arXiv preprint arXiv:2505.20279}, 2025.
\bibitem[Feng et~al.(2026)Feng, Gong, Li, Guo, Wang, Peng, Wu, Zhang, Wang, and Yue]{feng2026videor}
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
\newblock Video-r1: Reinforcing video reasoning in {MLLM}s.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026.
\newblock URL \url{https://openreview.net/forum?id=a2JTVVvcEl}.
\bibitem[Fowler(2010)]{fowler2010domain}
Martin Fowler.
\newblock \emph{Domain-Specific Languages, Portable Documents}.
\newblock Pearson Education, 2010.
\bibitem[Gao et~al.(2022)Gao, Madaan, Zhou, Alon, Liu, Yang, Callan, and Neubig]{gao2022pal}
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
\newblock Pal: Program-aided language models.
\newblock \emph{arXiv preprint arXiv:2211.10435}, 2022.
\bibitem[Guo et~al.(2026)Guo, Hou, Ma, Tang, and Yang]{guo2026mssr}
Yejie Guo, Yunzhong Hou, Wufei Ma, Meng Tang, and Ming-Hsuan Yang.
\newblock Pursuing minimal sufficiency in spatial reasoning.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=bZAKJwyn1n}.
\bibitem[Gupta and Kembhavi(2023)]{gupta2022visprog}
Tanmay Gupta and Aniruddha Kembhavi.
\newblock Visual programming: Compositional visual reasoning without training.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 14953--14962, 2023.
\bibitem[He et~al.(2025)He, Dong, Chen, Yu, Feng, and Li]{he2025mem4nav}
Lixuan He, Haoyu Dong, Zhenxing Chen, Yangcheng Yu, Jie Feng, and Yong Li.
\newblock Mem4nav: Boosting vision-and-language navigation in urban environments with a hierarchical spatial-cognition long-short memory system.
\newblock \emph{arXiv preprint arXiv:2506.19433}, 2025.
\bibitem[Hu et~al.(2025)Hu, Lin, Long, Ran, Jiang, Wang, Zhu, Xu, Wang, and Pang]{hu2025g2vlm}
Wenbo Hu, Jingli Lin, Yilin Long, Yunlong Ran, Lihan Jiang, Yifan Wang, Chenming Zhu, Runsen Xu, Tai Wang, and Jiangmiao Pang.
\newblock G$^2$vlm: Geometry grounded vision language model with unified 3d reconstruction and spatial reasoning.
\newblock \emph{arXiv preprint arXiv:2511.21688}, 2025.
\bibitem[Huang et~al.(2025)Huang, Wu, Xie, and Han]{huang20253daware}
Xiaohu Huang, Jingjing Wu, Qunyi Xie, and Kai Han.
\newblock Mllms need 3d-aware representation supervision for scene understanding.
\newblock \emph{CoRR}, abs/2506.01946, June 2025.
\newblock URL \url{https://doi.org/10.48550/arXiv.2506.01946}.
\bibitem[Kerbl et~al.(2023)Kerbl, Kopanas, Leimk{\"u}hler, and Drettakis]{kerbl20233d}
Bernhard Kerbl, Georgios Kopanas, Thomas Leimk{\"u}hler, and George Drettakis.
\newblock 3d gaussian splatting for real-time radiance field rendering.
\newblock \emph{ACM Trans. Graph.}, 42\penalty0 (4):\penalty0 139--1, 2023.
\bibitem[Kim et~al.(2024)Kim, Pertsch, Karamcheti, Xiao, Balakrishna, Nair, Rafailov, Foster, Lam, Sanketi, et~al.]{kim2024openvla}
Moo~Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et~al.
\newblock Openvla: An open-source vision-language-action model.
\newblock \emph{arXiv preprint arXiv:2406.09246}, 2024.
\bibitem[Li et~al.(2026)Li, Li, Wang, Yan, Wu, Zhang, Shen, Lu, Xiao, and Zhuang]{li2026spatialladder}
Hongxing Li, Dingming Li, Zixuan Wang, Yuchen Yan, Hang Wu, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang.
\newblock Spatialladder: Progressive training for spatial reasoning in vision-language models.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=KtrFXlvgrK}.
\bibitem[Li et~al.(2025)Li, Torr, Vedaldi, and Jakab]{li2025vmem}
Runjia Li, Philip Torr, Andrea Vedaldi, and Tomas Jakab.
\newblock Vmem: Consistent interactive video scene generation with surfel-indexed view memory.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 25690--25699, 2025.
\bibitem[Lin et~al.(2026)Lin, Chen, Liew, Chen, Li, Zhao, Peng, Guo, Zhou, Shi, Feng, and Kang]{lin2026depth}
Haotong Lin, Sili Chen, Jun~Hao Liew, Donny~Y. Chen, Zhenyu Li, Yang Zhao, Sida Peng, Hengkai Guo, Xiaowei Zhou, Guang Shi, Jiashi Feng, and Bingyi Kang.
\newblock Depth anything 3: Recovering the visual space from any views.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=yirunib8l8}.
\bibitem[Liu et~al.(2026)Liu, Zhou, Zhang, Zhang, Huang, and Duan]{liu2026msnav}
Chenghao Liu, Zhimu Zhou, Jiachen Zhang, Minghao Zhang, Songfang Huang, and Huiling Duan.
\newblock Msnav: Zero-shot vision-and-language navigation with dynamic memory and llm spatial reasoning.
\newblock In \emph{ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages 20112--20116. IEEE, 2026.
\bibitem[Liu et~al.(2023)Liu, Li, Wu, and Lee]{liu2023visual}
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee.
\newblock Visual instruction tuning.
\newblock \emph{Advances in neural information processing systems}, 36:\penalty0 34892--34916, 2023.
\bibitem[Liu et~al.(2024)Liu, Li, Li, and Lee]{liu2024improved}
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong~Jae Lee.
\newblock Improved baselines with visual instruction tuning.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 26296--26306, 2024.
\bibitem[Liu et~al.(2025)Liu, Guo, Warke, Chintala, Paxton, Shafiullah, and Pinto]{liu2025dynamem}
Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad~Mahi Shafiullah, and Lerrel Pinto.
\newblock Dynamem: Online dynamic spatio-semantic memory for open world mobile manipulation.
\newblock In \emph{2025 IEEE International Conference on Robotics and Automation (ICRA)}, pages 13346--13355. IEEE, 2025.
\bibitem[Luo et~al.(2026)Luo, Zhang, Yong, Dai, Wang, Ran, Shi, Sycara, and Xie]{luo2026pyspatial}
Zhanpeng Luo, Ce~Zhang, Silong Yong, Cunxi Dai, Qianwei Wang, Haoxi Ran, Guanya Shi, Katia Sycara, and Yaqi Xie.
\newblock pyspatial: Generating 3d visual programs for zero-shot spatial reasoning.
\newblock \emph{arXiv preprint arXiv:2603.00905}, 2026.
\bibitem[Martin-Brualla et~al.(2021)Martin-Brualla, Radwan, Sajjadi, Barron, Dosovitskiy, and Duckworth]{martin2021nerf}
Ricardo Martin-Brualla, Noha Radwan, Mehdi~SM Sajjadi, Jonathan~T Barron, Alexey Dosovitskiy, and Daniel Duckworth.
\newblock Nerf in the wild: Neural radiance fields for unconstrained photo collections.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 7210--7219, 2021.
\bibitem[Mernik et~al.(2005)Mernik, Heering, and Sloane]{mernik2005and}
Marjan Mernik, Jan Heering, and Anthony~M Sloane.
\newblock When and how to develop domain-specific languages.
\newblock \emph{ACM computing surveys (CSUR)}, 37\penalty0 (4):\penalty0 316--344, 2005.
\bibitem[Mildenhall et~al.(2021)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng]{mildenhall2021nerf}
Ben Mildenhall, Pratul~P Srinivasan, Matthew Tancik, Jonathan~T Barron, Ravi Ramamoorthi, and Ren Ng.
\newblock Nerf: Representing scenes as neural radiance fields for view synthesis.
\newblock \emph{Communications of the ACM}, 65\penalty0 (1):\penalty0 99--106, 2021.
\bibitem[M{\"u}ller et~al.(2022)M{\"u}ller, Evans, Schied, and Keller]{muller2022instant}
Thomas M{\"u}ller, Alex Evans, Christoph Schied, and Alexander Keller.
\newblock Instant neural graphics primitives with a multiresolution hash encoding.
\newblock \emph{ACM transactions on graphics (TOG)}, 41\penalty0 (4):\penalty0 1--15, 2022.
\bibitem[O'keefe and Nadel(1978)]{o1978hippocampus}
John O'keefe and Lynn Nadel.
\newblock \emph{The hippocampus as a cognitive map}.
\newblock Oxford university press, 1978.
\bibitem[O'Keefe and Speakman(1987)]{o1987single}
John O'Keefe and Andrew Speakman.
\newblock Single unit activity in the rat hippocampus during a spatial memory task.
\newblock \emph{Experimental brain research}, 68\penalty0 (1):\penalty0 1--27, 1987.
\bibitem[Ouyang et~al.(2025)Ouyang, Liu, Wu, Liu, Zhou, Zhou, Meng, and Sun]{ouyang2025spacer}
Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi~Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu~Sun.
\newblock Spacer: Reinforcing mllms in video spatial reasoning.
\newblock \emph{arXiv preprint arXiv:2504.01805}, 2025.
\bibitem[Po et~al.(2025)Po, Nitzan, Zhang, Chen, Dao, Shechtman, Wetzstein, and Huang]{po2025long}
Ryan Po, Yotam Nitzan, Richard Zhang, Berlin Chen, Tri Dao, Eli Shechtman, Gordon Wetzstein, and Xun Huang.
\newblock Long-context state-space video world models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 8733--8744, 2025.
\bibitem[Qi et~al.(2025)Qi, Zhang, Yu, Wang, and Zhao]{qi2025vln}
Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao.
\newblock Vln-r1: Vision-language navigation via reinforcement fine-tuning.
\newblock \emph{arXiv preprint arXiv:2506.17221}, 2025.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et~al.]{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et~al.
\newblock Learning transferable visual models from natural language supervision.
\newblock In \emph{International conference on machine learning}, pages 8748--8763. PmLR, 2021.
\bibitem[remyxai(2024)]{VQASynth}
remyxai.
\newblock Vqasynth, 2024.
\newblock URL \url{https://github.com/remyxai/VQASynth/tree/main}.
\newblock GitHub repository.
\bibitem[Savov et~al.(2026)Savov, Kazemi, Zhang, Paudel, Wang, and Gool]{savov2026statespacediffuser}
Nedko Savov, Naser Kazemi, Deheng Zhang, Danda~Pani Paudel, Xi~Wang, and Luc~Van Gool.
\newblock Statespacediffuser: Bringing long context to diffusion world models.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026.
\newblock URL \url{https://openreview.net/forum?id=g52NwTQj0Q}.
\bibitem[Schick et~al.(2023)Schick, Dwivedi-Yu, Dess{\`\i}, Raileanu, Lomeli, Hambro, Zettlemoyer, Cancedda, and Scialom]{schick2023toolformer}
Timo Schick, Jane Dwivedi-Yu, Roberto Dess{\`\i}, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
\newblock Toolformer: Language models can teach themselves to use tools.
\newblock \emph{Advances in neural information processing systems}, 36:\penalty0 68539--68551, 2023.
\bibitem[Siegel and White(1975)]{siegel1975development}
Alexander~W Siegel and Sheldon~H White.
\newblock The development of spatial representations of large-scale environments.
\newblock \emph{Advances in child development and behavior}, 10:\penalty0 9--55, 1975.
\bibitem[Sur{\'i}s et~al.(2023)Sur{\'i}s, Menon, and Vondrick]{suris2023vipergpt}
D{\'i}dac Sur{\'i}s, Sachit Menon, and Carl Vondrick.
\newblock Vipergpt: Visual inference via python execution for reasoning.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 11888--11898, 2023.
\bibitem[Tolman(1948)]{tolman1948cognitive}
Edward~C Tolman.
\newblock Cognitive maps in rats and men.
\newblock \emph{Psychological review}, 55\penalty0 (4):\penalty0 189, 1948.
\bibitem[Wang and Agapito(2025)]{wang20253d}
Hengyi Wang and Lourdes Agapito.
\newblock 3d reconstruction with spatial memory.
\newblock In \emph{2025 International Conference on 3D Vision (3DV)}, pages 78--89. IEEE, 2025.
\bibitem[Wang et~al.(2025{\natexlab{a}})Wang, Chen, Karaev, Vedaldi, Rupprecht, and Novotny]{wang2025vggt}
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny.
\newblock Vggt: Visual geometry grounded transformer.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pages 5294--5306, 2025{\natexlab{a}}.
\bibitem[Wang et~al.(2025{\natexlab{b}})Wang, Zhang, Holynski, Efros, and Kanazawa]{wang2025continuous}
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei~A Efros, and Angjoo Kanazawa.
\newblock Continuous 3d perception model with persistent state.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pages 10510--10522, 2025{\natexlab{b}}.
\bibitem[Wang et~al.(2024)Wang, Leroy, Cabon, Chidlovskii, and Revaud]{wang2024dust3r}
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud.
\newblock Dust3r: Geometric 3d vision made easy.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pages 20697--20709, 2024.
\bibitem[Wang et~al.(2026)Wang, Zhou, Zhu, Chang, Zhou, Li, Chen, Pang, Shen, and He]{wang2026pi}
Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He.
\newblock \${\textbackslash}pi{\textasciicircum}3\$: Permutation-equivariant visual geometry learning.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=DTQIjngDta}.
\bibitem[Wu et~al.(2026{\natexlab{a}})Wu, Liu, Hung, and Duan]{wu2026spatialmllm}
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan.
\newblock Spatial-{MLLM}: Boosting {MLLM} capabilities in visual-based spatial intelligence.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026{\natexlab{a}}.
\newblock URL \url{https://openreview.net/forum?id=RnXS7aK4rK}.
\bibitem[Wu et~al.(2026{\natexlab{b}})Wu, Zheng, Zhou, and Lu]{wu2026pointr}
Yuqi Wu, Wenzhao Zheng, Jie Zhou, and Jiwen Lu.
\newblock Point3r: Streaming 3d reconstruction with explicit spatial pointer memory.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026{\natexlab{b}}.
\newblock URL \url{https://openreview.net/forum?id=yk1iqV9Etr}.
\bibitem[Xiao et~al.(2026)Xiao, LAN, Zhou, Ouyang, Yang, Zeng, and Pan]{xiao2026worldmem}
Zeqi Xiao, Yushi LAN, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xingang Pan.
\newblock Worldmem: Long-term consistent world simulation with memory.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026.
\newblock URL \url{https://openreview.net/forum?id=c6CAVKlKmU}.
\bibitem[Yan et~al.(2024)Yan, Zhang, Zhu, and Wang]{yan2024maskclustering}
Mi~Yan, Jiazhao Zhang, Yan Zhu, and He~Wang.
\newblock Maskclustering: View consensus based mask graph clustering for open-vocabulary 3d instance segmentation.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2024.
\bibitem[Yang et~al.(2025)Yang, Yang, Gupta, Han, Fei-Fei, and Xie]{yang2025thinking}
Jihan Yang, Shusheng Yang, Anjali~W Gupta, Rilyn Han, Li~Fei-Fei, and Saining Xie.
\newblock Thinking in space: How multimodal large language models see, remember, and recall spaces.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pages 10632--10643, 2025.
\bibitem[Yang et~al.(2026)Yang, Liu, Zhang, Zhou, Tan, Yang, Du, and Gan]{yang2026mindjourney}
Yuncong Yang, Jiageng Liu, Zheyuan Zhang, Siyuan Zhou, Reuben Tan, Jianwei Yang, Yilun Du, and Chuang Gan.
\newblock Mindjourney: Test-time scaling with world models for spatial reasoning.
\newblock In \emph{The Thirty-ninth Annual Conference on Neural Information Processing Systems}, 2026.
\newblock URL \url{https://openreview.net/forum?id=L2W4wQsNkY}.
\bibitem[Yao et~al.(2023)Yao, Zhao, Yu, Du, Shafran, Narasimhan, and Cao]{yao2023react}
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
\newblock {ReAct}: Synergizing reasoning and acting in language models.
\newblock In \emph{International Conference on Learning Representations (ICLR)}, 2023.
\bibitem[Yin et~al.(2025)Yin, Wang, Zhang, Zhang, Wang, Wang, Zhang, Chandrasegaran, Liu, Krishna, et~al.]{yin2025spatial}
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et~al.
\newblock Spatial mental modeling from limited views.
\newblock In \emph{Structural Priors for Vision Workshop at ICCV'25}, 2025.
\bibitem[You et~al.(2025)You, Lin, Lyu, Zhang, and Yang]{you2025instainpaint}
Junqi You, Chieh~Hubert Lin, Weijie Lyu, Zhengbo Zhang, and Ming-Hsuan Yang.
\newblock Instainpaint: Instant 3d-scene inpainting with masked large reconstruction model.
\newblock In \emph{Adv. Neural Inform. Process. Syst.}, 2025.
\bibitem[Yu et~al.(2025)Yu, Bai, Qin, Liu, Wang, Wan, Zhang, and Liu]{yu2025context}
Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di~Zhang, and Xihui Liu.
\newblock Context as memory: Scene-consistent interactive long video generation with memory retrieval.
\newblock In \emph{Proceedings of the SIGGRAPH Asia 2025 Conference Papers}, pages 1--11, 2025.
\bibitem[Yuan et~al.(2026)Yuan, Kumar, and Wang]{yuan2026gr3d}
Jiangye Yuan, Gowri Kumar, and Baoyuan Wang.
\newblock Boosting mllm spatial reasoning with geometrically referenced 3d scene representations.
\newblock \emph{arXiv preprint arXiv:2603.08592}, 2026.
\bibitem[Zemskova and Yudin(2025)]{zemskova20253dgraphllm}
Tatiana Zemskova and Dmitry Yudin.
\newblock 3dgraphllm: Combining semantic graphs and large language models for 3d scene understanding.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 8885--8895, 2025.
\bibitem[Zhang et~al.(2026{\natexlab{a}})Zhang, Herrmann, Hur, Sun, Yang, Cole, Darrell, and Sun]{zhang2026loger}
Junyi Zhang, Charles Herrmann, Junhwa Hur, Chen Sun, Ming-Hsuan Yang, Forrester Cole, Trevor Darrell, and Deqing Sun.
\newblock Loger: Long-context geometric reconstruction with hybrid memory.
\newblock \emph{arXiv preprint arXiv:2603.03269}, 2026{\natexlab{a}}.
\bibitem[Zhang et~al.(2025)Zhang, Chen, Feng, Jiang, and Meng]{zhang2025constructing}
Puzhen Zhang, Xuyang Chen, Yu~Feng, Yuhan Jiang, and Liqiu Meng.
\newblock Constructing coherent spatial memory in llm agents through graph rectification.
\newblock \emph{arXiv preprint arXiv:2510.04195}, 2025.
\bibitem[Zhang et~al.(2026{\natexlab{b}})Zhang, Bi, Hong, Zhang, Luan, Yang, Sunkavalli, Freeman, and Tan]{zhang2026testtime}
Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William~T. Freeman, and Hao Tan.
\newblock Test-time training done right.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026{\natexlab{b}}.
\newblock URL \url{https://openreview.net/forum?id=Tb9qAxT3xv}.
\bibitem[Zhang et~al.(2026{\natexlab{c}})Zhang, Wu, Jia, Wang, Zhang, Li, Ran, Zhang, Sun, Yin, et~al.]{zhang2026think3d}
Zaibin Zhang, Yuhan Wu, Lianjie Jia, Yifan Wang, Zhongbo Zhang, Yijiang Li, Binghao Ran, Fuxi Zhang, Zhuohan Sun, Zhenfei Yin, et~al.
\newblock Think3d: Thinking with space for spatial reasoning.
\newblock \emph{arXiv preprint arXiv:2601.13029}, 2026{\natexlab{c}}.
\bibitem[Zhao et~al.(2025)Zhao, Zhang, Xu, Chang, Chen, Li, Sun, and Wei]{zhao2025spacemind}
Ruosen Zhao, Zhikang Zhang, Jialei Xu, Jiahao Chang, Dong Chen, Lingyun Li, Weijian Sun, and Zizhuang Wei.
\newblock Spacemind: Camera-guided modality fusion for spatial reasoning in vision-language models.
\newblock \emph{arXiv preprint arXiv:2511.23075}, 2025.
\bibitem[Zhou et~al.(2025{\natexlab{a}})Zhou, Vilesov, He, Wan, Zhang, Nagachandra, Chang, Chen, Wang, and Kadambi]{zhou2025vlm4d}
Shijie Zhou, Alexander Vilesov, Xuehai He, Ziyu Wan, Shuwang Zhang, Aditya Nagachandra, Di~Chang, Dongdong Chen, Xin~Eric Wang, and Achuta Kadambi.
\newblock Vlm4d: Towards spatiotemporal awareness in vision language models.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on computer vision}, pages 8600--8612, 2025{\natexlab{a}}.
\bibitem[Zhou et~al.(2025{\natexlab{b}})Zhou, Du, Yang, Han, Chen, Yeung, and Gan]{zhou2025learning}
Siyuan Zhou, Yilun Du, Yuncong Yang, Lei Han, Peihao Chen, Dit-Yan Yeung, and Chuang Gan.
\newblock Learning 3d persistent embodied world models.
\newblock \emph{arXiv preprint arXiv:2505.05495}, 2025{\natexlab{b}}.
\bibitem[Zhuo et~al.(2026)Zhuo, Zheng, Guo, Wu, Zhou, and Lu]{zhuo2026streaming}
Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu.
\newblock Streaming visual geometry transformer.
\newblock In \emph{The Fourteenth International Conference on Learning Representations}, 2026.
\newblock URL \url{https://openreview.net/forum?id=5APgTKsnx8}.
\bibitem[Zitkovich et~al.(2023)Zitkovich, Yu, Xu, Xu, Xiao, Xia, Wu, Wohlhart, Welker, Wahid, et~al.]{zitkovich2023rt}
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et~al.
\newblock Rt-2: Vision-language-action models transfer web knowledge to robotic control.
\newblock In \emph{Conference on Robot Learning}, pages 2165--2183. PMLR, 2023.
\end{thebibliography}