DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding
Extracted source text · arXiv 2605.29879
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4 paper
\IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks command
\overrideIEEEmargins % Needed to meet printer requirements.
%In case you encounter the following error:
%Error 1010 The PDF file may be corrupt (unable to open PDF file) OR
%Error 1000 An error occurred while parsing a contents stream. Unable to analyze the PDF file.
%This is a known problem with pdfLaTeX conversion filter. The file cannot be opened with acrobat reader
%Please use one of the alternatives below to circumvent this error by uncommenting one or the other
%\pdfobjcompresslevel=0
%\pdfminorversion=4
% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document
\usepackage{graphics}
\usepackage[pdftex]{graphicx}
\usepackage[tight,footnotesize]{subfigure}
\usepackage{multirow}
\usepackage[cmex10]{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage{booktabs}
\usepackage{arydshln}
\usepackage{textcomp}
\usepackage{url}
\usepackage{color}
\usepackage[table]{xcolor}
% \colorlet{firstcolor}{green!52}
\definecolor{firstcolor}{RGB}{120, 220, 115}
\colorlet{secondcolor}{green!25}
\colorlet{thirdcolor}{yellow!35}
\usepackage{siunitx}
\usepackage{soul}
\usepackage{xcolor}
\usepackage{colortbl}
\usepackage{wrapfig}
\usepackage{babel}
\usepackage[small]{caption}
\usepackage{capt-of}
\makeatletter
\newcommand{\cdashmidrule}[1]{%
\noalign{\vskip\aboverulesep}
\cdashline{#1}
\noalign{\vskip\belowrulesep}}
\makeatother
% use for Hyperlink
\usepackage[bookmarks=true,colorlinks]{hyperref}
\urlstyle{same}
% DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-term Task Planning and Mobile Manipulation
\title{\LARGE \bf
DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term \\ Scene Understanding and Grounding
}
% \vspace{-1.5em}
\author{
Luzhou Ge$^{1}$, Xiangyu Zhu$^{1}$, Jinyan Liu$^{1}$ and Xuesong Li$^{1\ast}$% <-this % stops a space
\thanks{$^{\ast}$ The corresponding author:{ \tt\small lixuesong@bit.edu.cn}}
\thanks{$^{1}$ School of Computer Science, Beijing Institute of Technology, China.}%
}
\makeatletter
\let\@oldmaketitle\@maketitle
\renewcommand{\@maketitle}{%
\@oldmaketitle
% \vspace{-1em}
\begin{center}
\includegraphics[width=\textwidth]{figures/1_introduction/Teaser_1.pdf}
\captionof{figure}{Compared with previous state-of-the-art systems, \textbf{\textit{DGSG-Mind}} offers a more complete functional framework for long-term embodied scene understanding in dynamic environments. It jointly maintains a hybrid instance-aware 3D Gaussian representation and a hierarchical scene graph, supporting high-quality reconstruction, semantic mapping, accurate visual relocalization, and instance-level dynamic updates. Moreover, it leverages RoI Gaussian-rendered visual cues together with semantic and spatial relations for multimodal reasoning. These capabilities allow \textbf{\textit{DGSG-Mind}} to support long-term robotic tasks in dynamic real-world environments. \textit{Abbreviations:} SeeG (SeeGround~\cite{li2025seeground}), CG (ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}), OmniM (OmniMap~\cite{deng2025omnimap}), DovSG~\cite{yan2025dynamicopenvocabulary3dscene}, and DyGSG (DynamicGSG~\cite{ge2025dynamicgsg}).}%
\label{fig:teaser}%
\end{center}
\setcounter{figure}{1}%
% \vspace{-1em}
}
\makeatother
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}
% Integrating open-vocabulary semantic information into dynamic 3D scene representations is essential for embodied spatial perception. However, existing methods often suffer from fragile instance association due to incomplete cross-view association, and their limited ability to handle topological changes restricts long-term robotic task execution. In addition, current 3D scene understanding methods either rely on naive feature matching without explicit spatial reasoning or assume unrealistic offline ground-truth geometry. To address these challenges, we present \textbf{\textit{DGSG-Mind}}, a dynamic, instance-aware 3D Gaussian scene graph mapping system with an embodied reasoning agent. The proposed system jointly leverages explicit 3D Gaussians and a probabilistic voxel grid to support robust cross-modal instance association and incremental semantic mapping. To handle topological scene changes, we further introduce a localized masked refinement strategy guided by joint geometric-semantic consistency. Based on the Gaussian map, we construct a multi-level scene graph and develop an embodied reasoning agent that integrates hierarchical structural relations, spatial-semantic information, and visually annotated 3DGS renderings. Extensive experiments show that our system achieves state-of-the-art zero-shot 3D visual grounding on self-reconstructed maps, while also delivering strong performance in open-vocabulary semantic segmentation and photometric reconstruction. We further deploy \textbf{\textit{DGSG-Mind}} on a real-world robot to validate its long-term dynamic update and target-oriented reasoning capabilities. The project page of DGSG-Mind is available at https://dgsg-mind.github.io/.
Integrating open-vocabulary semantic information into dynamic 3D scene representations is essential for long-term embodied scene understanding. However, existing methods often suffer from fragile instance association due to incomplete cross-view cues, while their limited ability to handle object-level topological changes restricts long-term robotic task execution. Moreover, current 3D scene understanding methods either rely on simple feature matching without explicit spatial reasoning or assume offline ground-truth 3D geometry. To address these challenges, we present \textbf{\textit{DGSG-Mind}}, a hybrid instance-aware 3D Gaussian dynamic scene graph system with an embodied reasoning agent. Our system couples a probabilistic voxel grid with explicit 3D Gaussians to enable robust cross-modal instance fusion and incremental semantic mapping. It handles dynamic changes through Gaussian-based visual relocalization and localized masked refinement guided by geometric-semantic consistency. Built on the instance Gaussian map, DGSG-Mind further constructs a hierarchical scene graph and develops the \textbf{\textit{3D Gaussian Mind}}, which integrates structural relations, spatial-semantic information, and visually annotated RoI Gaussian renderings for multimodal reasoning. Extensive experiments show that DGSG-Mind achieves the best zero-shot 3DVG performance among methods operating on self-reconstructed maps, while also delivering strong performance in 3D open-vocabulary semantic segmentation and scene reconstruction. We further deploy \textbf{\textit{DGSG-Mind}} on real-world robots to demonstrate its target-oriented reasoning and dynamic update capabilities. The project page of DGSG-Mind is available at ~\href{https://icr-lab.github.io/DGSG-Mind}{https://icr-lab.github.io/DGSG-Mind}.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bstctlcite{IEEEexample:BSTcontrol}
\input{src/1_introduction}
\input{src/2_related_work}
\input{src/3_method}
\input{src/4_evaluation}
\input{src/5_conclusion}
\bibliographystyle{IEEEtran}
\bibliography{references}
\end{document}
\section{Introduction}
\label{sec:introduction}
% Dynamic 3D semantic mapping and scene understanding are fundamental prerequisites for embodied AI to execute long-term tasks. Traditional 3D scene representations are typically built on dense point clouds, meshes, or Truncated Signed Distance Fields (TSDF). Recently, 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d} has emerged as an effective paradigm for real-time dense mapping. In parallel, Vision-Language Models (VLMs) have equipped robotic systems with open-vocabulary reasoning capabilities, enabling language-guided 3D scene understanding and embodied reasoning.
% However, integrating unconstrained 3DGS with open-vocabulary semantics in dynamic environments remains challenging. Although recent works have explored instance-level representations, most methods still rely on incomplete cross-view association strategies, such as 3D point cloud overlap (e.g., ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}) or simple 2D mask intersections (e.g., OpenGS-SLAM~\cite{yang2025opengs-slam}, DynamicGSG~\cite{ge2025dynamicgsg}). As a result, they are vulnerable to multi-view inconsistencies and occlusions, often leading to inaccurate object merging. Moreover, most 3DGS-based semantic mapping frameworks ~\cite{yang2025opengs-slam, zhou2024feature, qin2024langsplat3dlanguagegaussian, yang2025opengs, deng2025omnimap} assume static scenes, which limits their ability to handle topological changes and restricts long-term scene understanding in dynamic environments. This limitation also reduces their applicability to long-term robotic tasks. Furthermore, existing 3D scene understanding methods often rely on simple feature similarity, such as \cite{zhou2024feature, qin2024langsplat3dlanguagegaussian, zhu2025objectgs}, without fully exploiting spatial and topological structure. In contrast, methods with stronger spatial reasoning capability are typically developed offline and depend on unrealistic ground-truth 3D point clouds~\cite{yuan2024visual, li2025seeground, jin2025spazer}.
% Traditional 3D maps are typically represented by dense point clouds, meshes, or Truncated Signed Distance Fields (TSDFs), which provide explicit geometry but limited semantic and appearance-level information.
% In parallel, Vision-Language Models (VLMs) have endowed robotic systems with open-vocabulary reasoning capabilities, enabling language-guided 3D scene understanding and embodied decision-making.
% Dynamic 3D semantic mapping and scene understanding are fundamental to embodied AI systems that execute long-term tasks in changing environments. Recently, 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d} has been increasingly adopted for real-time dense mapping due to its photorealistic rendering quality and efficient optimization. However, integrating unconstrained 3DGS with open-vocabulary semantics in dynamic environments remains challenging. First, existing instance-level mapping methods often rely on incomplete cross-view association strategies, such as 3D point-cloud overlap (e.g., ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}) or simple 2D mask intersections (e.g., OpenGS-SLAM~\cite{yang2025opengs-slam}, DynamicGSG~\cite{ge2025dynamicgsg}). These heuristics are sensitive to occlusions and multi-view inconsistencies, resulting in inaccurate instance merging. Second, most 3DGS-based semantic mapping frameworks~\cite{deng2025omnimap, yang2025opengs-slam, ye2024gaussiangroupingsegmentedit, qin2024langsplat3dlanguagegaussian, yang2025opengs} assume static scenes, limiting their ability to handle object-level topological changes required for long-term robotic tasks. Finally, existing 3D scene understanding methods either rely mainly on simple feature similarity~\cite{ye2024gaussiangroupingsegmentedit, qin2024langsplat3dlanguagegaussian, zhu2025objectgs} without explicit spatial-topological reasoning, or require offline ground-truth 3D point clouds for stronger spatial reasoning~\cite{li2025seeground, yuan2024visual,jin2025spazer}.
Dynamic 3D semantic mapping and scene understanding are fundamental to embodied AI systems that execute long-term tasks in changing environments. Recently, 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d} has been widely adopted for real-time dense mapping, while Vision-Language Models (VLMs) have enabled visual understanding and language-guided reasoning for embodied decision-making. However, integrating open-vocabulary semantics and spatial reasoning into dynamic scene representations remains challenging. Existing instance-level mapping methods rely on limited cross-view cues, such as 3D point-cloud overlap~\cite{gu2023conceptgraphsopenvocabulary3dscene, werby23hovsg} or simple 2D mask intersections~\cite{ge2025dynamicgsg, yang2025opengs-slam}, making them vulnerable to occlusions, detector instability, and multi-view inconsistencies. Moreover, most 3DGS-based semantic mapping frameworks~\cite{deng2025omnimap, yang2025opengs-slam, qin2024langsplat3dlanguagegaussian, yang2025opengs} assume static scenes, limiting their ability to handle object-level topological changes. Existing 3D scene understanding methods further rely either on simple feature similarity~\cite{qin2024langsplat3dlanguagegaussian, ye2024gaussiangroupingsegmentedit, zhu2025objectgs} without explicit spatial-topological reasoning, or offline ground-truth 3D point clouds for stronger spatial reasoning~\cite{li2025seeground, yuan2024visual,jin2025spazer}.
% Dynamic 3D semantic mapping and scene understanding are fundamental to embodied AI systems that execute long-term tasks in changing environments. Recently, 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d} has been increasingly adopted for real-time dense mapping due to its photorealistic rendering quality and efficient optimization. However, integrating unconstrained 3DGS with open-vocabulary semantics in dynamic environments remains challenging. First, existing instance-level mapping methods often rely on incomplete cross-view association strategies, such as 3D point-cloud overlap \cite{gu2023conceptgraphsopenvocabulary3dscene, werby23hovsg} or simple 2D mask intersections \cite{yang2025opengs-slam, ge2025dynamicgsg}. These heuristics are sensitive to occlusions and multi-view inconsistencies, resulting in inaccurate instance merging. Second, most 3DGS-based semantic mapping frameworks \cite{deng2025omnimap, yang2025opengs-slam, ye2024gaussiangroupingsegmentedit, qin2024langsplat3dlanguagegaussian, yang2025opengs} assume static scenes, limiting their ability to handle object-level topological changes required for long-term robotic tasks. Finally, existing 3D scene understanding methods either rely mainly on simple feature similarity \cite{ye2024gaussiangroupingsegmentedit, qin2024langsplat3dlanguagegaussian, zhu2025objectgs} without explicit spatial-topological reasoning, or require offline ground-truth 3D point clouds for stronger spatial reasoning \cite{li2025seeground, yuan2024visual,jin2025spazer}.
% 分栏，传统的谁不能干嘛，谁能干嘛，我们的能干嘛。 分别是动态，场景理解方式，需要不要 gt 点云，场景结构信息
To address these limitations, we propose \textbf{\textit{DGSG-Mind}}, a hybrid 3D Gaussian dynamic scene graph system for long-term scene understanding and grounding. The proposed system couples explicit 3D Gaussians with a probabilistic voxel grid to build a hybrid instance-aware representation, enabling robust cross-modal instance fusion and incremental semantic mapping. To handle dynamic environments, we combine Gaussian-based visual relocalization with localized refinement guided by geometric-semantic consistency to update object-level topological changes without re-optimizing the static background. Based on the Gaussian map, we construct a hierarchical scene graph and develop the \textit{3D Gaussian Mind} for multimodal reasoning, which integrates RoI Gaussian renderings with structured scene graph context to perform zero-shot 3D visual grounding on incrementally self-reconstructed maps.
In summary, the main contributions are as follows:
% 得写的更大，比如 混合实例场景表示
\begin{itemize}
\item \textbf{Hybrid Instance-Aware 3DGS Representation:} We propose a hybrid scene representation that couples explicit 3D Gaussians with a probabilistic voxel grid for instance-aware mapping. By jointly leveraging cross-modal similarity and multi-term Gaussian optimization, our method achieves high-quality reconstruction and accurate instance fusion.
\item \textbf{Geometric-Semantic Dynamic Update:} We introduce a localized masked refinement strategy constrained by joint geometric-semantic consistency to improve the accuracy of dynamic scene updates without re-optimizing the static background.
\item \textbf{3D Gaussian Mind:} We construct a hierarchical scene graph and develop the \textit{3D Gaussian Mind} for multimodal reasoning. This module jointly leverages scene structure, node-level spatial-semantic information, and annotated RoI Gaussian renderings to support structured scene understanding and target reasoning in complex 3D scenes.
\end{itemize}
\section{Related Work} \label{sec:related_work}
\subsection{Gaussian Dense Mapping}
\label{subsec:related_gaussian_mapping}
The development of 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d} has significantly advanced real-time dense RGB-D SLAM (e.g., SplaTAM \cite{keetha2024splatam}, GS-ICP-SLAM \cite{ha2024rgbdicp}). To mitigate the geometric artifacts inherent in unconstrained 3DGS, hybrid representations such as GSFusion \cite{wei2024gsfusion} and SEGS-SLAM \cite{wen2025segs} successfully incorporate voxel grids to enforce structural regularizations. Recently, dense mapping works have been extended to embedding open-vocabulary semantics. By lifting semantic features from foundation models (e.g., SBERT \cite{reimers2019sentence}, CLIP \cite{zhai2023sigmoidlosslanguageimage}), methods like OmniMap \cite{deng2025omnimap}, DynamicGSG \cite{ge2025dynamicgsg}, and OpenGS-Fusion \cite{yang2025opengs} incrementally integrate semantic embeddings into a global map for open-vocabulary semantic mapping and achieve photorealistic reconstruction. However, their instance association strategies often remain fragile due to incomplete cross-view merging heuristics. By contrast, our method integrates a probabilistic voxel grid with Gaussian-rendered 2D instance masks for cross-modal instance association, improving robustness and accuracy to object occlusions and instance fusion.
% \subsection{3D Dynamic Scene Graphs}
% \label{subsec:related_scene_graph}
% 3D Scene Graphs~\cite{rosinol20203ddynamicscenegraphs, hughes2024foundations} abstract spatial geometry and semantic relationships into hierarchical topological graphs. With the development of Vision-Language Models (VLMs), frameworks such as ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}, and BBQ~\cite{linok2024barequeriesopenvocabularyobject} project open-set semantic features onto 3D geometric nodes for downstream robotic tasks. To handle environmental changes, recent works such as RoboExp~\cite{jiang2024roboexpactionconditionedscenegraph}, DovSG~\cite{yan2025dynamicopenvocabulary3dscene}, and DynamicGSG~\cite{ge2025dynamicgsg} support dynamic scene updates by capturing the effects of robot actions and comparing geometric changes. However, these methods mainly rely on geometric similarity for change detection, which may not accurately capture complex object-level changes. For 3DGS-based methods, DynamicGSG~\cite{ge2025dynamicgsg} also re-optimizes unchanged background regions during dynamic updates. By contrast, our method adopts a localized masked refinement strategy guided by joint geometric-semantic consistency, enabling more accurate scene updates without re-optimizing the static background.
% \subsection{3D Dynamic Scene Graphs}
% \label{subsec:related_scene_graph}
% 3D scene graphs~\cite{rosinol20203ddynamicscenegraphs,hughes2024foundations} represent spatial geometry and semantic relationships as hierarchical topological graphs. With the development of vision-language models, methods such as ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}, and BBQ~\cite{linok2024barequeriesopenvocabularyobject} lift open-vocabulary semantics into 3D object nodes for downstream robotic tasks. To handle environmental changes, RoboExp~\cite{jiang2024roboexpactionconditionedscenegraph}, DovSG~\cite{yan2025dynamicopenvocabulary3dscene}, and DynamicGSG~\cite{ge2025dynamicgsg} update scene graphs by modeling robot actions or detecting geometric changes. However, relying mainly on geometric similarity might be insufficient for object-level change detection, especially under appearance changes, occlusion, or ambiguous geometry. In contrast, our method performs dynamic updates with joint geometric-semantic consistency and localized masked refinement, enabling more accurate scene updates without re-optimizing the static background.
% \subsection{3D Dynamic Scene Graphs}
% \label{subsec:related_scene_graph}
% 3D scene graphs~\cite{rosinol20203ddynamicscenegraphs,hughes2024foundations} represent spatial geometry and semantic relationships as hierarchical topological graphs. With the development of vision-language models, methods such as ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}, and BBQ~\cite{linok2024barequeriesopenvocabularyobject} lift open-vocabulary semantics into object nodes for downstream robotic tasks. To handle environmental changes, RoboExp~\cite{jiang2024roboexpactionconditionedscenegraph}, DovSG~\cite{yan2025dynamicopenvocabulary3dscene}, and DynamicGSG~\cite{ge2025dynamicgsg} update scene graphs by modeling robot actions or detecting geometric changes. However, these methods still rely heavily on external localization pipelines: DovSG~\cite{yan2025dynamicopenvocabulary3dscene} uses ACE~\cite{brachmann2023accelerated}, LightGlue~\cite{lindenberger2023lightglue}, and ICP for visual relocalization, while DynamicGSG~\cite{ge2025dynamicgsg} requires an online SLAM system to continuously provide coarse camera poses. Moreover, relying mainly on geometric similarity can be insufficient for object-level change detection, especially under appearance changes, occlusion, or ambiguous geometry. In contrast, our method performs dynamic updates with Gaussian-based relocalization, joint geometric-semantic consistency, and localized masked refinement, enabling more accurate scene updates without re-optimizing the static background.
\subsection{3D Dynamic Scene Graphs}
\label{subsec:related_scene_graph}
3D scene graphs~\cite{hughes2024foundations} represent spatial geometry and semantic relationships as hierarchical topological graphs. With the development of vision-language models, methods such as ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, HOVSG~\cite{werby23hovsg}, and BBQ~\cite{linok2024barequeriesopenvocabularyobject} lift open-vocabulary semantics into 3D object nodes for downstream robotic tasks. To handle environmental changes, RoboExp~\cite{jiang2024roboexpactionconditionedscenegraph}, DovSG~\cite{yan2025dynamicopenvocabulary3dscene}, and DynamicGSG~\cite{ge2025dynamicgsg} update scene graphs by modeling robot actions or detecting scene changes. However, RoboExp~\cite{jiang2024roboexpactionconditionedscenegraph} mainly focuses on action-conditioned interactive exploration rather than long-term scene reconstruction. DovSG~\cite{yan2025dynamicopenvocabulary3dscene} relies on a multi-stage relocalization pipeline with ACE \cite{brachmann2023accelerated}, feature matching, and point-cloud ICP, making its accuracy dependent on geometric registration. DynamicGSG~\cite{ge2025dynamicgsg} requires external VIO poses and still optimizes unaffected regions during scene updates. Moreover, these systems rely on geometric or photometric consistency for change detection, which can be insufficient under appearance changes, occlusion, or ambiguous geometry. In contrast, our method performs dynamic updates with Gaussian-based relocalization, joint geometric-semantic consistency, and localized masked refinement, enabling accurate scene updates without re-optimizing the static background.
\subsection{3D Scene Understanding}
\label{subsec:scene_understanding}
3D Visual Grounding (3DVG), which localizes target objects from free-form language descriptions, is an important benchmark for embodied scene understanding. Recent methods increasingly use LLMs and MLLMs as reasoning agents for this task. Methods such as LLM-Grounder~\cite{yang2024llm} and ZSVG3D~\cite{yuan2024visual} translate language queries into spatial heuristics for object search, while more advanced methods (e.g., SeeGround~\cite{li2025seeground}, SPAZER~\cite{jin2025spazer}) further incorporate structural guidance and spatial-semantic reasoning. However, existing methods still face two limitations. Open-vocabulary 3D representations such as LangSplat~\cite{qin2024langsplat3dlanguagegaussian}, GaussianGrouping~\cite{ye2024gaussiangroupingsegmentedit}, and ObjectGS~\cite{zhu2025objectgs} mainly rely on simple feature matching and lack explicit spatial-topological structure. In contrast, spatial reasoning agents~\cite{li2025seeground,yuan2024visual,jin2025spazer} are designed for offline settings with pre-scanned geometry. Our method performs zero-shot 3DVG directly on self-reconstructed Gaussian maps, using rendered RoI views and structured scene-graph contexts for visual-spatial reasoning.
% \subsection{3D Scene Understanding}
% \label{subsec:scene_understanding}
% 3D visual grounding (3DVG), which localizes target objects from free-form language descriptions, is a key benchmark for embodied scene understanding. Recent methods increasingly use LLMs and MLLMs as reasoning agents for this task. LLM-Grounder~\cite{yang2024llm} and ZSVG3D~\cite{yuan2024visual} translate language queries into spatial heuristics for object search, while SeeGround~\cite{li2025seeground} and SPAZER~\cite{jin2025spazer} further incorporate structural guidance and spatial-semantic reasoning. However, existing methods still face two limitations. Open-vocabulary 3D representations such as LERF~\cite{kerr2023lerf}, Feature-3DGS~\cite{zhou2024feature}, LangSplat~\cite{qin2024langsplat3dlanguagegaussian}, GaussianGrouping~\cite{ye2024gaussiangroupingsegmentedit}, and ObjectGS~\cite{zhu2025objectgs} mainly rely on feature matching and lack explicit spatial-topological structure. In contrast, spatial reasoning agents~\cite{yuan2024visual,li2025seeground,jin2025spazer} are often designed for offline settings with ideal pre-scanned geometry. Our \textit{3D Gaussian Mind} performs zero-shot 3DVG directly on an incrementally self-reconstructed Gaussian map, using rendered RoI views and structured scene-graph context for joint visual-spatial reasoning.
\begin{figure*}
\centering
\includegraphics[width=1.0\textwidth]{figures/1_introduction/Pipline_2.pdf}
\caption{\textbf{\textit{System Overview of DGSG-Mind.}} Given a posed RGB-D sequence, \textit{DGSG-Mind} extracts open-vocabulary instance masks and semantic features, and integrates them into a hybrid 3D Gaussian instance representation. Cross-modal association leverages a sparse probabilistic voxel grid, rendered Gaussian masks, and instance-level features to link 2D observations with persistent 3D Gaussian instances. The sparse probabilistic voxel grid also guides Gaussian initialization, while the Gaussian field is optimized across multiple views with photometric, depth, scale, and normal regularization. The instance-aware Gaussian map is further abstracted into a hierarchical 3D scene graph, enabling the \textit{3D Gaussian Mind} to perform zero-shot 3D visual grounding and spatial reasoning from rendered RoI views and structured 3D relations. For dynamic scenarios, localized masked refinement updates newly appeared or removed objects while keeping the unchanged background fixed.
}
\label{fig:pipeline}
\vspace{-1em}
\end{figure*}
\section{Methodology}
\label{sec:methodology}
An overview of \textbf{\textit{DGSG-Mind}} is shown in Fig.~\ref{fig:pipeline}. Our system builds a hybrid instance-aware 3D Gaussian representation by coupling a sparse probabilistic voxel grid with explicit Gaussians. First, we perform voxel-based instance-aware Gaussian mapping and optimization in Secs.~\ref{instance_aware_3DGS} and~\ref{subsec:gaussian_optim}. The optimized map is then abstracted into a hierarchical scene graph in Sec.~\ref{subsec:scene_graph}. For dynamic environments, Sec.~\ref{subsec:dynamic_update} introduces Gaussian-based camera relocalization, localized scene updates, and graph synchronization. Finally, Sec.~\ref{subsec:gs_mind} presents the \textit{3D Gaussian Mind} for zero-shot 3D visual grounding and spatial reasoning.
\subsection{3D Gaussian Preliminaries}
\label{para:gaussian_preliminary}
A scene is represented as a set of anisotropic 3D Gaussians,
\(
\mathcal{G}=\{G_i\}_{i=1}^{N}
\).
Each Gaussian \(G_i\) is parameterized by its center
\(\mathbf{u}_i \in \mathbb{R}^3\),
color attribute
\(\mathbf{c}_i \in \mathbb{R}^3\),
scale
\(\mathbf{s}_i \in \mathbb{R}^3\),
rotation
\(\mathbf{R}_i \in SO(3)\),
covariance
\(\mathbf{\Sigma}_i \in \mathbb{R}^{3\times 3}\),
and opacity
\(o_i \in [0,1]\).
The covariance is constructed from the scale and rotation as
\(
\mathbf{\Sigma}_i=\mathbf{R}_i \operatorname{diag}(\mathbf{s}_i^2)\mathbf{R}_i^\top.
\)
The unnormalized spatial density of \(G_i\) is defined as
\begin{equation}
\mathcal{N}_i(\mathbf{x})
=
\exp\!\left(
-\frac{1}{2}
(\mathbf{x}-\mathbf{u}_i)^\top
\mathbf{\Sigma}_i^{-1}
(\mathbf{x}-\mathbf{u}_i)
\right).
\end{equation}
Given a camera pose \(\mathbf{T} \in SE(3)\), each Gaussian is projected onto the image plane with projected covariance
\(\mathbf{\Sigma}'_i=\mathbf{J}_i \mathbf{T}^{-1}\mathbf{\Sigma}_i \mathbf{T}^{-T}\mathbf{J}_i^\top\),
where \(\mathbf{J}_i\) denotes the Jacobian of the projection function. The rendered color and depth at pixel \(p\) are computed by alpha blending:
\begin{equation}
C(p)=\sum_{i=1}^{N} T_i(p)\,\alpha_i(p)\,\mathbf{c}_i,
D(p)=\sum_{i=1}^{N} T_i(p)\,\alpha_i(p)\,d_i,
\end{equation}
where \(\alpha_i(p)\) denotes the alpha contribution of the \(i\)-th Gaussian at pixel \(p\), \(T_i(p)=\prod_{j=1}^{i-1}(1-\alpha_j(p))\) denotes the accumulated transmittance along the ray, and \(d_i\) represents the depth of the Gaussian center in the camera coordinate.
\subsection{Voxel-based Instance-Aware 3DGS Mapping}
\label{instance_aware_3DGS}
Given a posed RGB-D stream \(\mathcal{I}_t=\langle C_t, D_t, \mathbf{T}_t \rangle \), our system incrementally constructs a hybrid instance-aware map consisting of explicit 3D Gaussians and a sparse probabilistic voxel grid. For each frame \(\mathcal{I}_t\), we first identify the active voxel blocks within the current view frustum and integrate the RGB-D observation into the sparse voxel map. Valid depth pixels are then back-projected into 3D and associated with their corresponding voxel blocks and local voxel indices. Each voxel maintains candidate instance IDs and accumulated hit statistics, from which voxel-wise assignment probabilities are derived. This hybrid representation serves as an efficient geometric scaffold for incrementally linking 2D observations with persistent 3D Gaussian instances.
\paragraph{Instance Recognition}
For each observation \(\mathcal{I}_t\), we employ the open-vocabulary object detector YOLO-World~\cite{cheng2024yoloworldrealtimeopenvocabularyobject} to extract detection bounding boxes \(\{b_{t,i}\}_{i=1}^{M}\). To obtain precise instance boundaries, these detections are fed into the Segment Anything Model ~\cite{kirillov2023segment}, which produces high-quality 2D instance masks \(\{m_{t,i}\}_{i=1}^{M}\). For each mask, CLIP~\cite{zhai2023sigmoidlosslanguageimage} is used to extract a normalized semantic feature \(\mathbf{f}_i^t\). Accordingly, the 2D observation of the \(i\)-th instance at timestep \(t\) is represented as \(O_i^t = (m_i^t, \mathbf{f}_i^t)\).
\paragraph{Cross-modal Instance Association}
For each 2D instance observation \(O_i^t\), we back-project the mask \(m_i^t\) into 3D using \(D_t\) and map the corresponding points into the voxel grid with camera pose \(\mathbf{T}_t\). Let \(\mathcal{V}_i^t\) denote the set of valid voxels occupied by the observation \(O_i^t\). Each voxel stores up to three candidate instance IDs with accumulated hit counts. For voxel \(v\), we define the assignment probability of candidate instance \(I_k\) as \(P_k(v)=\frac{c_k(v)}{c(v)}\) when \(I_k\) is stored in \(v\), and \(P_k(v)=0\) otherwise, where \(c_k(v)\) and \(c(v)\) denote the hit count of \(I_k\) and the total hit count in \(v\), respectively. Querying the stored historical instance IDs within \(\mathcal{V}_i^t\) yields the candidate map instance set \(\mathcal{C}_i^t=\{I_k\}\). We compute three similarity terms to associate \(O_i^t\) with each candidate \(I_k\in\mathcal{C}_i^t\):
\begin{itemize}
\item \textbf{Geometric Similarity (\(S_{\mathrm{geo}}(i,k)\)):}
We measure the probabilistic spatial overlap in the voxel grid as \(S_{\mathrm{geo}}(i,k)=\frac{1}{|\mathcal{V}_i^t|}\sum_{v\in\mathcal{V}_i^t} P_k(v),\) where \(|\mathcal{V}_i^t|\) denotes the number of voxels occupied by \(O_i^t\) at time \(t\).
\item \textbf{Rendered Mask IoU (\(S_{\mathrm{iou}}(i,k)\)):}
We render the 3D Gaussians associated with \(I_k\) into the current view \(t\) to obtain a projected mask \(m_{k\rightarrow t}\), and compute \(S_{\mathrm{iou}}(i,k)=\frac{|m_{k\rightarrow t}\cap m_i^t|}{|m_{k\rightarrow t}\cup m_i^t|}\).
\item \textbf{Semantic Similarity (\(S_{\mathrm{sem}}(i,k)\)):}
We compute the cosine similarity between the current semantic feature \(\mathbf{f}_i^t\) and the maintained global feature \(\mathbf{F}_k\) of instance \(I_k\): \(S_{\mathrm{sem}}(i,k)=\frac{\mathbf{f}_i^t\cdot\mathbf{F}_k}{\|\mathbf{f}_i^t\|\,\|\mathbf{F}_k\|}\).
\end{itemize}
These three terms are combined into a joint similarity:
\begin{equation}
S(i,k)=\lambda_{\mathrm{1}}S_{\mathrm{geo}}(i,k)+\lambda_{\mathrm{2}}S_{\mathrm{iou}}(i,k)+\lambda_{\mathrm{3}}S_{\mathrm{sem}}(i,k),
\end{equation}
where \(\lambda_{\mathrm{1}} = 0.4\), \(\lambda_{\mathrm{2}} = 0.4\), and \(\lambda_{\mathrm{3}} = 0.2\). The observation \(O_i^t\) is associated with the instance \(k^*=\arg\max_k S(i,k)\) if the maximum score exceeds a threshold \(\tau\). Otherwise, \(O_i^t\) is initialized as a new instance.
\paragraph{Instance Fusion}
After associating \(O_i^t\) with a map instance \(I_{k^*}\), we update its semantic feature by:
\begin{equation}
\mathbf{F}_{k^*} =
\frac{\mathbf{F}_{k^*}W_{k^*}+\mathbf{f}_i^t w_i^t}{W_{k^*}+w_i^t},
\end{equation}
where \(w_i^t=(|\mathcal{V}_i^t|/{\hat{V}_{k^*})} \cdot S(i,k^*)\) is a visibility weight, \(\hat{V}_{k^*}\) denotes the total number of valid voxels currently assigned to \(I_{k^*}\) in the global voxel map, and \(W_{k^*}\) is the accumulated feature weight. We then update the voxel statistics: if \(k^*\) is already stored in voxel \(v\), its hit count \(c_{k^*}(v)\) is incremented; otherwise, \(k^*\) is inserted into an empty slot when available and updated accordingly. The total voxel count \(c(v)\) is incremented for all \(v\in\mathcal{V}_i^t\).
\subsection{Multi-dimensional Gaussian Optimization}
\label{subsec:gaussian_optim}
\paragraph{Voxel-based Gaussian Densification}
\label{para:densification}
Instead of initializing Gaussians from sparse Structure-from-Motion (SfM) points \cite{kerbl20233d,qin2024langsplat3dlanguagegaussian,zhu2025objectgs} or opacity-based initialization \cite{ge2025dynamicgsg, keetha2024splatam}, we incrementally densify the map from the probabilistic voxel grid. For each observation \(\mathcal{I}_t\), we examine valid voxels within the current camera frustum and instantiate new Gaussians only for newly observed voxels. Each voxel center \(\mathbf{x}\in\mathbb{R}^3\) initializes a Gaussian with center \(\mathbf{u}=\mathbf{x}\), identity rotation, and initial log-scale \(s_0\). The color is initialized from the zero-order SH coefficient of the observed RGB value, and the opacity is set to \(0.5\) via inverse-sigmoid mapping. We further maintain an explicit voxel-to-Gaussian index table to efficiently retrieve voxel-associated Gaussian subsets for local rendering and accelerated mask-IoU computation during instance association.
\paragraph{Scale \& Normal Regularization}
\label{para:regularization}
In unconstrained 3DGS optimization, Gaussians may become overly elongated or excessively large in textureless regions, causing geometric distortion and floating artifacts. To alleviate this issue, we employ two complementary regularizers on Gaussian scale and local surface geometry. Specifically, we introduce a scale anisotropy regularization that constrains the ratio between the maximum and minimum scales of each Gaussian:
\begin{equation}
\mathcal{L}_{scale} = \frac{1}{|\mathcal{G}^*|} \sum_{G \in \mathcal{G}^*} \max \left(0, \log\left(\frac{\max(\mathbf{s})}{\min(\mathbf{s})}\right) - \log(r_{allow}) \right),
\end{equation}
where $r_{allow} = 10$ is the maximum allowed aspect ratio, and $\mathcal{G}^*$ denotes the subset of violating Gaussians. We further apply a normal consistency regularization to stabilize local geometry by penalizing the angular discrepancy between the pseudo-ground-truth normals $\mathbf{N}_t$, computed from the spatial gradients of the input depth map, and the normals $\hat{\mathbf{N}}_t$ reconstructed from rendered depth at viewpoint $t$:
\begin{equation}
\mathcal{L}_{normal} = 1 - \langle \hat{\mathbf{N}}_t, \mathbf{N}_t \rangle,
\end{equation}
where $\langle \cdot, \cdot \rangle$ denotes the inner product between normalized normal vectors.
\paragraph{Joint Optimization}
\label{para:map_optimization}
The proposed joint optimization minimizes a multi-term loss that combines photometric and geometric regularization. The photometric term compares the rendered image $\hat{\mathbf{C}}_t$ with the RGB observation $\mathbf{C}_t$ at viewpoint $t$ using a combination of L1 loss and SSIM. For geometric supervision, we compute an L1 depth loss between the rendered depth map $\hat{\mathbf{D}}_t$ and the sensor depth map $\mathbf{D}_t$:
\begin{equation}
\mathcal{L}_{rgb} = (1 - \lambda_4) \| \hat{\mathbf{C}}_t - \mathbf{C}_t \|_1 + \lambda_4 (1 - \text{SSIM}(\hat{\mathbf{C}}_t, \mathbf{C}_t)),
\end{equation}
\begin{equation}
\mathcal{L}_{depth} = \| \hat{\mathbf{D}}_t - \mathbf{D}_t \|_1.
\end{equation}
The joint optimization loss is defined as:
\begin{equation}
\mathcal{L}_{total} = \mathcal{L}_{rgb} + \lambda_{5} \mathcal{L}_{depth} + \lambda_{6} \mathcal{L}_{normal} + \lambda_{7} \mathcal{L}_{scale},
\end{equation}
where $\lambda_{4} = 0.2$, $\lambda_{5} = \lambda_{6} = 0.1$, and $ \lambda_{7} = 1.0$. During optimization, we prune redundant Gaussians with extreme transparency or excessive spatial extent, and perform multi-view scene optimization over a keyframe set. Keyframes are selected every \(\delta_n\) frame or whenever the camera translation or rotation exceeds a preset threshold, and \(\delta_m\) historical keyframes are chosen for joint multi-view optimization.
\subsection{3D Scene Graph Construction}
\label{subsec:scene_graph}
After optimizing the instance-aware 3D Gaussian map, we construct a hierarchical 3D scene graph \(\mathcal{S}=(\mathcal{O},\mathcal{E})\), where \(\mathcal{O}=\{o_j\}_{j=1}^{M}\) is the set of object nodes and \(\mathcal{E}\) is the set of edges. Each node \(o_j=(\mathbf{p}_j,\mathbf{b}_j,\ell_j,\kappa_j,\nu_j,\delta_j)\) is associated with a 3D center \(\mathbf{p}_j\), a 3D bbox \(\mathbf{b}_j\), an object category \(\ell_j\), a caption \(\kappa_j\), a best view \(\nu_j\), and a functional role \(\delta_j\in\{\textit{Asset}, \textit{Ordinary}, \textit{Standalone}\}\). For each instance \(j\), we first select a best observation from its historical views. Given all camera positions that observe instance \(j\), we compute their centroid and retrieve the top-\(K=5\) nearest views as candidates. The best view \(\nu_j^*\) is selected as the one with the highest view quality score \(q_{u,j}=|m_{u,j}|/|\Omega_u|\cdot |\mathcal{H}_{u,j}|/|\mathcal{H}_j|\), where \(|\Omega_u|\) is the number of pixels in view \(u\), \(|m_{u,j}|\) is the number of mask pixels of instance \(j\), \(\mathcal{H}_j\) is the set of Gaussians associated with instance \(j\), and \(\mathcal{H}_{u,j}\subseteq\mathcal{H}_j\) is the visible subset in view \(u\). Following \cite{ge2025dynamicgsg}, each instance is assigned one functional role from \textit{Asset}, \textit{Ordinary}, and \textit{Standalone}. Based on the masked image from the selected best view, Describe Anything \cite{lian2025describe} first generates a fine-grained physical description, and Qwen2.5-VL \cite{bai2025qwen25vltechnicalreport} further parses it into the node category \(\ell_j\), caption \(\kappa_j\), and functional role \(\delta_j\). We then organize all nodes into a hierarchical topology according to their functional roles and relative geometric relationships estimated from node centers and 3D bbox. If two nodes satisfy the semantic-spatial constraint, such as support or containment, we add a directed edge \((o_a,o_b)\in\mathcal{E}\). Nodes labeled as \textit{Asset} or \textit{Standalone} are directly connected to the root node, while \textit{Ordinary} nodes are attached to their geometrically compatible \textit{Asset} nodes.
\subsection{Dynamic Scene Update}
\label{subsec:dynamic_update}
\paragraph{Gaussian-based Camera Relocalization}
Accurate camera relocalization is essential for long-term scene graph maintenance and task execution. Unlike DynamicGSG~\cite{ge2025dynamicgsg}, which requires continuous online SLAM, our method combines ACE-based \cite{brachmann2023accelerated} visual relocalization with pose optimization on the 3D Gaussian map. Specifically, we first train a scene-specific
ACE model~\cite{brachmann2023accelerated} using posed RGB images from the current scene. Given a new observation, ACE predicts a coarse pose \(\mathbf{T}_{\mathrm{coarse}}\in SE(3)\), which is further refined by freezing the Gaussian map and optimizing only the camera pose. To improve robustness, we use a silhouette mask to retain sufficiently visible pixels and minimize the masked RGB-D alignment loss:
\begin{equation}
\mathcal{M}_{\mathrm{reloc}}
=
\mathbb{I}\!\left(
\hat{\mathbf{D}}>0,\,
\mathbf{D}>0,\,
\sum_{i=1}^{N}\mathbf{T}_i\odot\boldsymbol{\alpha}_i>\lambda_8
\right),
\end{equation}
\begin{equation}
\mathcal{L}_{\mathrm{reloc}}
=
\mathcal{M}_{\mathrm{reloc}}
\odot
\left(
\|\hat{\mathbf{C}}-\mathbf{C}\|_1
+
\lambda_{9}\|\hat{\mathbf{D}}-\mathbf{D}\|_1
\right),
\end{equation}
where \(\lambda_8=0.98\) is the silhouette threshold and \(\lambda_9=0.5\). The pose with the minimum relocalization loss is selected as the refined pose \(\mathbf{T}_{\mathrm{refined}}\).
\paragraph{Dynamic Detection and Refinement}
Unlike DovSG \cite{yan2025dynamicopenvocabulary3dscene} and DynamicGSG \cite{ge2025dynamicgsg}, which rely primarily on geometric similarity for change detection, our method jointly evaluates geometric, appearance, and semantic consistency under the refined camera pose \(\mathbf{T}_{\mathrm{refined}}\). We consider only instances with more than \(50\%\) visible associated Gaussians in the current frustum. For each visible instance \(I_j\), let \(m_{t,j}\) be its rendered mask, \(\Omega_{t,j}=\{p\mid p\in m_{t,j},\,\hat{\mathbf{D}}_t(p)>0,\,\mathbf{D}_t(p)>0\}\) be the valid depth region, and \(\hat{\mathbf{g}}_j,\mathbf{g}_j\) be the CLIP features extracted from the rendered and observed RGB crops. We compute geometric, appearance, and semantic consistency as \(S_{\mathrm{geo
}}(j)=\sum_{p\in\Omega_{t,j}}\mathbb{I}(|\hat{\mathbf{D}}_t(p)-\mathbf{D}_t(p)|<\tau_d) / |\Omega_{t,j}|\), \(S_{\mathrm{app}}(j)=\mathrm{SSIM}(\hat{\mathbf{C}}_t(m_{t,j}),\mathbf{C}_t(m_{t,j}))\), and \(S_{\mathrm{sem}}(j)=\hat{\mathbf{g}}_j^\top\mathbf{g}_j/(\|\hat{\mathbf{g}}_j\|\,\|\mathbf{g}_j\|)\). The final change similarity is:
\begin{equation}
S_{\mathrm{change}}(j)=\lambda_{10}S_{\mathrm{geo}}(j)+\lambda_{11}S_{\mathrm{app}}(j)+\lambda_{12}S_{\mathrm{sem}}(j),
\end{equation}
where \(\tau_d = 0.05m, \lambda_{10} = 0.2, \lambda_{11}=0.4, \lambda_{12}=0.4\).
If \(S_{\mathrm{change}}(j)<\delta_{\mathrm{change}}\), we regard \(I_j\) as inconsistent with the current observation and add it to the removal set \(\mathcal{O}_{\mathrm{remove}}\). Its associated Gaussian primitives are then pruned from the map. After removing inconsistent instances, we perform residual instance detection on the current RGB-D observation following Sec.~\ref{instance_aware_3DGS} to identify newly appeared objects. The newly detected regions are back-projected into 3D and initialized as new Gaussian instances \(\mathcal{O}_{\mathrm{new}}\). Let \(\mathcal{M}_{\mathrm{new}}\) and \(\mathcal{M}_{\mathrm{remove}}\) denote the union masks of newly added and removed objects. We refine the updated Gaussian map only within the local update region \(\mathcal{M}_{\mathrm{update}}=\mathcal{M}_{\mathrm{new}}\cup\mathcal{M}_{\mathrm{remove}}\), while keeping the static background fixed following Sec.~\ref{subsec:gaussian_optim}. This localized masked refinement enables efficient adaptation to dynamic updates without re-optimizing the entire scene.
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/3_method/Dynamic_3.pdf}
\caption{\textbf{Dynamic Scene Update:} Given the current camera view, we first estimate a coarse camera pose using a scene-specific ACE model and refine the pose against the 3D Gaussian map. With the refined pose, visible instances are evaluated using joint geometric-semantic consistency to detect removed objects, while residual detection identifies newly appeared ones. The Gaussian map is then optimized by localized masked refinement, and the scene graph is synchronized with the resulting object additions and removals.}
\label{fig:Association}
\vspace{-1.5em}
\end{figure}
\paragraph{Scene Graph Update}
Let \(\mathcal{S}^{t-1}=(\mathcal{O}^{t-1},\mathcal{E}^{t-1})\) and \(\mathcal{S}^{t}=(\mathcal{O}^{t},\mathcal{E}^{t})\) denote the scene graph before and after the dynamic update. We update \(\mathcal{S}^{t-1}\) using the detected additions \(\mathcal{O}_{\mathrm{new}}\) and removals \(\mathcal{O}_{\mathrm{remove}}\). For each removed instance \(I_j\in\mathcal{O}_{\mathrm{remove}}\), the corresponding Standalone or Ordinary node \(o_j\) and its incident edges are deleted, while an Asset node is removed together with its associated subgraph. For each new instance \(I_j\in\mathcal{O}_{\mathrm{new}}\), we create a node \(o_j=(\mathbf{p}_j,\mathbf{b}_j,\ell_j,\kappa_j,\nu_j,\delta_j)\) and establish edges according to Sec.~\ref{subsec:scene_graph}. This keeps the Gaussian scene graph consistent with the updated scene state.
\subsection{3D Gaussian Mind}
\label{subsec:gs_mind}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/3_method/GS_mind_4.pdf}
\caption{\textbf{3D Gaussian Mind:} By integrating natural language queries, structured 3D scene graphs, and generated annotated Gaussian views (RoI images), this framework leverages a Vision-Language Model for joint spatial reasoning and object localization.}
\label{fig:GS_mind}
\vspace{-1.5em}
\end{figure}
\paragraph{Instruction Parsing}
To support zero-shot 3D visual grounding and spatial reasoning from free-form language instructions, we introduce the \textit{3D Gaussian Mind}, illustrated in Fig.~\ref{fig:GS_mind}. Given a natural language query, we use a VLM-based parser to parse the target object (\textit{Target}) and its reference object(s) (\textit{Anchor}). The 3D scene graph is serialized in JSON format and provided to the parser as structured context. Based on the parsed Target and Anchor object nodes, we retrieve the final target and anchor candidate sets using top-k cosine similarity.
\paragraph{RoI-based Gaussian View Generation}
A critical challenge in applying 2D VLMs to 3D environments is the lack of explicit spatial grounding and object disambiguation. To address this issue, we generate annotated Gaussian views for the candidate target and anchor instances identified by the instruction parser. As shown in Fig.~\ref{fig:GS_mind}, we construct a relation-aware rendering view that enlarges the field of view and includes both the target and its anchor in a shared image, thereby preserving their spatial relationship. Instead of directly using the target object's best view, we take the camera position from its best-view pose and re-orient the virtual camera toward the target-anchor region through a look-at transformation. We then project the 3D bounding boxes of the candidate objects onto the rendered image and annotate them with their instance IDs. Following SeeGround~\cite{li2025seeground}, these IDs are overlaid at the visual centers of the projected boxes, yielding an RoI image with explicit visual references. The resulting annotated Gaussian view provides unified visual cues for the VLM to jointly reason about object appearance and spatial relations.
\paragraph{Mind-guided Visual Grounding}
Finally, the 3D visual grounding task is formulated as a joint multimodal reasoning problem. The VLM receives two complementary inputs: 1) Gaussian-rendered views annotated with numerical instance IDs, and 2) the JSON-serialized scene graph containing structured 3D information. By jointly reasoning over visual appearance, annotated object IDs, and scene-level semantic-geometric context, the VLM identifies the target instance specified by the query. This enables grounding to be guided by both image evidence and explicit structured scene information.
\section{Experiments}
\label{sec:experiment}
\subsection{Experimental Setup}
\label{subsec:experiment_setup}
\textbf{Evaluation Tasks:} To comprehensively evaluate \textit{DGSG-Mind}, we conduct experiments across four key tasks: 1) \textit{3D open-vocabulary semantic segmentation}, which assesses instance-level, zero-shot 3D scene understanding; 2) \textit{3D visual grounding}, which evaluates the system's ability to localize target objects given free-form natural language descriptions; 3) \textit{scene reconstruction}, which measures the photometric fidelity of the optimized 3D Gaussian representation; 4) \textit{real-world deployment}, which validates its capability to support continuous robot operation, target-oriented navigation, and dynamic scene updates in real-world environments.
\textbf{Datasets:} For open-vocabulary semantic segmentation and scene reconstruction, we use all eight synthetic scenes from Replica~\cite{replica19arxiv} and eight selected ScanNet~\cite{dai2017scannet} scenes, following the scene selection protocol of BBQ~\cite{linok2024barequeriesopenvocabularyobject}. The selected ScanNet scenes include \texttt{scene0011\_00}, \texttt{scene0030\_00}, \texttt{scene0046\_00}, \texttt{scene0086\_00}, \texttt{scene0222\_00}, \texttt{scene0378\_00}, \texttt{scene0389\_00}, and \texttt{scene0435\_00}. For 3DVG, we evaluate ScanRefer~\cite{chen2020scanrefer} and Nr3D~\cite{achlioptas2020referit3d} queries from the selected ScanNet scenes.
\textbf{Implementation Details:} All experiments are conducted on a workstation with an NVIDIA RTX 4090D GPU, with thresholds set to \(\delta_n=5, \delta_m=10\) and \(\delta_{\mathrm{change}}=0.35\). For fair comparison, all agent-based methods requiring a Vision-Language Model (VLM) use \texttt{Qwen2.5-VL-72B}~\cite{bai2025qwen25vltechnicalreport} API.
\subsection{3D Open-Vocabulary Semantic Segmentation}
\label{subsec:semantic_segmentation}
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{1.7pt}
\begin{table}[!t]
\centering
\begin{tabular}{lcccccc}\toprule
\multirow{2.5}{*}{\textbf{Methods}} & \multicolumn{3}{c}{\textbf{Replica}} & \multicolumn{3}{c}{\textbf{ScanNet}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& mAcc$\uparrow$ & mIoU$\uparrow$ & F-mIoU$\uparrow$ & mAcc$\uparrow$ & mIoU$\uparrow$ & F-mIoU$\uparrow$ \\ \midrule
ConceptGraphs \cite{gu2023conceptgraphsopenvocabulary3dscene} & 40.78 & 26.28 & 42.49 & 41.31 & 22.02 & 30.90 \\
OpenGS-Fusion \cite{yang2025opengs} & 46.79 & 35.20 & 48.56 & 55.54 & 33.53 & 42.96 \\
DynamicGSG \cite{ge2025dynamicgsg} & \cellcolor{secondcolor}53.65 & 31.92 & 47.13 & \cellcolor{thirdcolor}57.83 & \cellcolor{secondcolor}42.23 & \textbf{\cellcolor{firstcolor}57.61} \\
OmniMap \cite{deng2025omnimap} & \cellcolor{thirdcolor}54.44 & \cellcolor{secondcolor}39.93 & \cellcolor{secondcolor}66.84 & \cellcolor{secondcolor}58.16 & \cellcolor{thirdcolor}42.08 & \cellcolor{thirdcolor}51.84 \\
\textbf{Ours w/o mask IoU} & 51.43 &
\cellcolor{thirdcolor}36.26 &
\cellcolor{thirdcolor}54.51 &
53.70 &
41.58 &
50.17 \\
\textbf{Ours} & \textbf{\cellcolor{firstcolor}55.48} & \textbf{\cellcolor{firstcolor}40.53} & \textbf{\cellcolor{firstcolor}67.94} & \textbf{\cellcolor{firstcolor}62.26} & \textbf{\cellcolor{firstcolor}45.51} & \cellcolor{secondcolor}56.86 \\
\bottomrule
\end{tabular}
\caption{Quantitative results of 3D open-vocabulary semantic segmentation on Replica and ScanNet.}
\label{tab:semseg}
\vspace{-1em}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{figures/4_evaluation/3dvg_5.pdf}
\caption{Qualitative results of 3DVG on the ScanRefer and Nr3D.}
\label{fig:Association}
\vspace{-1.5em}
\end{figure}
\begin{table*}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{2.8pt}
\renewcommand{\arraystretch}{1.12}
\resizebox{\textwidth}{!}{%
\begin{tabular}{l c c c c c c c c c c c c c c c c c c}
\toprule
\multirow{4.2}{*}{\textbf{Methods}}
& \multirow{4.2}{*}{\textbf{Venue}}
& \multirow{4.2}{*}{\textbf{Agent}}
& \multicolumn{6}{c}{\textbf{ScanRefer}}
& \multicolumn{10}{c}{\textbf{Nr3D}} \\
\cmidrule(lr){4-9} \cmidrule(l){10-19}
& &
& \multicolumn{2}{c}{\textbf{Unique}}
& \multicolumn{2}{c}{\textbf{Multiple}}
& \multicolumn{2}{c}{\textbf{Overall}}
& \multicolumn{2}{c}{\textbf{Easy}}
& \multicolumn{2}{c}{\textbf{Hard}}
& \multicolumn{2}{c}{\textbf{View-Dep.}}
& \multicolumn{2}{c}{\textbf{View-Indep.}}
& \multicolumn{2}{c}{\textbf{Overall}} \\
\cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9}
\cmidrule(lr){10-11} \cmidrule(lr){12-13} \cmidrule(lr){14-15} \cmidrule(lr){16-17} \cmidrule(l){18-19}
& &
& AP$_{0.25}$ & AP$_{0.5}$
& AP$_{0.25}$ & AP$_{0.5}$
& AP$_{0.25}$ & AP$_{0.5}$
& AP$_{0.1}$ & AP$_{0.25}$
& AP$_{0.1}$ & AP$_{0.25}$
& AP$_{0.1}$ & AP$_{0.25}$
& AP$_{0.1}$ & AP$_{0.25}$
& AP$_{0.1}$ & AP$_{0.25}$ \\
\midrule
\multicolumn{19}{l}{\textbf{Detections:} Zero-Shot \quad / \quad \textbf{PC or Gaussians:} Self-Reconstructed} \\
\cdashmidrule{1-19}
ConceptGraphs \cite{gu2023conceptgraphsopenvocabulary3dscene}
& ICRA24 & CLIP
& 16.18 & 12.27 & 17.18 & 7.46 & 16.94 & 8.72
& 20.28 & 9.74 & 10.70 & 3.21 & 21.69 & 6.02 & 16.34 & 8.56 & 17.65 & 7.94 \\
DynamicGSG \cite{ge2025dynamicgsg}
& IROS25 & CLIP
& 30.06 & 5.20 & 21.76 & 11.70 & 23.75 & 10.14
& \cellcolor{thirdcolor}31.24 & 19.68 & 16.58 & 10.70 & 24.10 & 15.06 & 28.21 & 17.90 & 27.21 & 17.21 \\
OmniMap \cite{deng2025omnimap}
& T-RO25 & SBERT
& \cellcolor{thirdcolor}49.71 & 11.56 & 21.57 & 9.14 & \cellcolor{thirdcolor}28.33 & 9.72
& 31.03 & 16.84 & 12.30 & 6.95 & 19.28 & 9.64 & 28.02 & 15.56 & 25.88 & 14.12 \\
BBQ \cite{linok2024barequeriesopenvocabularyobject}
& ICRA25 & Qwen2.5-VL
& 29.50 & \textbf{\cellcolor{secondcolor}19.70} & 17.70 & 10.20 & 19.40 & 11.70
& 30.80 & 20.10 & \cellcolor{thirdcolor}23.50 & \cellcolor{thirdcolor}14.80 & 21.30 & 13.40 & \cellcolor{thirdcolor}31.00 & 20.20 & \cellcolor{thirdcolor}29.90 & 20.50 \\
\textbf{Ours w/o RoI-views}
& - & Qwen2.5-VL
& 34.68 & 16.76 & \cellcolor{thirdcolor}26.14 & \cellcolor{thirdcolor}13.89 & 28.19 & \cellcolor{thirdcolor}14.58
& \cellcolor{thirdcolor}31.24 & \cellcolor{thirdcolor}25.15 & 22.99 & 14.44 & \cellcolor{thirdcolor}27.71 & \cellcolor{thirdcolor}18.67 & 29.38 & \cellcolor{thirdcolor}23.35 & 28.97 & \cellcolor{thirdcolor}22.21 \\
\textbf{Ours}
& - & Qwen2.5-VL
& \textbf{\cellcolor{secondcolor}53.18} & \cellcolor{thirdcolor}18.50 & \textbf{\cellcolor{secondcolor}37.92} & \textbf{\cellcolor{secondcolor}24.68} & \textbf{\cellcolor{secondcolor}42.65} & \textbf{\cellcolor{secondcolor}23.19}
& \textbf{\cellcolor{secondcolor}48.07} & \textbf{\cellcolor{secondcolor}36.31} & \textbf{\cellcolor{firstcolor}32.09} & \textbf{\cellcolor{secondcolor}24.06} & \textbf{\cellcolor{firstcolor}38.55} & \textbf{\cellcolor{secondcolor}28.92} & \textbf{\cellcolor{secondcolor}45.33} & \textbf{\cellcolor{secondcolor}34.24} & \textbf{\cellcolor{secondcolor}43.68} & \textbf{\cellcolor{secondcolor}32.94} \\
\midrule
\multicolumn{19}{l}{\textbf{Detections:} Preprocessed Mask3D \quad / \quad \textbf{PC:} ScanNet GT} \\
\cdashmidrule{1-19}
SeeGround \cite{li2025seeground}
& CVPR25 & Qwen2.5-VL
& \cellcolor{firstcolor}75.72 & \cellcolor{firstcolor}66.47 & \cellcolor{firstcolor}42.60 & \cellcolor{firstcolor}37.84 & \cellcolor{firstcolor}50.56 & \cellcolor{firstcolor}44.72
& \cellcolor{firstcolor}51.93 & \cellcolor{firstcolor}48.88 & \cellcolor{secondcolor}30.48 & \cellcolor{firstcolor}24.60 & \cellcolor{firstcolor}38.55 & \cellcolor{firstcolor}33.73 & \cellcolor{firstcolor}48.44 & \cellcolor{firstcolor}44.94 & \cellcolor{firstcolor}46.03 & \cellcolor{firstcolor}42.21 \\
\bottomrule
\end{tabular}%
}
\caption{Quantitative results of 3D visual grounding on the ScanRefer~\cite{chen2020scanrefer} and Nr3D~\cite{achlioptas2020referit3d} selected sets. The first group uses zero-shot detections and self-reconstructed point clouds or Gaussian maps, while SeeGround~\cite{li2025seeground} uses preprocessed Mask3D detections and ScanNet ground-truth point clouds as an upper-bound reference.}
\label{tab:joint_grounding}
\vspace{-1.5em}
\end{table*}
% Since the HSG retrieval module of DynamicGSG~\cite{ge2025dynamicgsg} is designed for simple rule-based spatial expressions, we evaluate it using its semantic matching pipeline for the complex free-form queries.
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{2.4pt}
\begin{table}[!t]
\centering
\begin{tabular}{lcccccc}\toprule
\multirow{2.5}{*}{\textbf{Methods}} & \multicolumn{3}{c}{\textbf{Replica}} & \multicolumn{3}{c}{\textbf{ScanNet}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ \\ \midrule
GS-ICP-SLAM \cite{ha2024rgbdicp} & 36.92 & 0.968 & 0.042 & 24.11 & 0.803 & 0.363 \\
DynamicGSG \cite{ge2025dynamicgsg} & 35.62 & \cellcolor{thirdcolor}0.979 & 0.068 & 23.56 & \cellcolor{thirdcolor}0.815 & \cellcolor{secondcolor}0.285 \\
OpenGS-Fusion \cite{yang2025opengs} & \cellcolor{thirdcolor}39.28 & 0.976 & \cellcolor{thirdcolor}0.039 & \cellcolor{secondcolor}26.46 & \cellcolor{secondcolor}0.822 & \textbf{\cellcolor{firstcolor}0.278} \\
OmniMap \cite{deng2025omnimap} & \cellcolor{secondcolor}39.47 & \cellcolor{secondcolor}0.980 & \cellcolor{secondcolor}0.021 & \cellcolor{thirdcolor}26.14 & 0.814 & 0.295 \\
\textbf{Ours} & \textbf{\cellcolor{firstcolor}40.31} & \textbf{\cellcolor{firstcolor}0.984} & \textbf{\cellcolor{firstcolor}0.017} & \textbf{\cellcolor{firstcolor}26.78} & \textbf{\cellcolor{firstcolor}0.829} & \cellcolor{thirdcolor}0.293 \\
\bottomrule
\end{tabular}
\caption{Quantitative comparison of photometric reconstruction quality on the Replica \cite{replica19arxiv} and ScanNet \cite{dai2017scannet} datasets.}
\label{tab:reconstruction}
\vspace{-2em}
\end{table}
We evaluate the open-vocabulary semantic quality of \textit{DGSG-Mind} by comparing it with representative state-of-the-art methods. For each foreground 3D primitive (e.g., point or Gaussian), we compute its cosine similarity with the text features of all object classes in the scene. The class with the highest similarity is assigned as the predicted label and compared against the ground-truth object class. We report mean Accuracy (mAcc), mean intersection over union (mIoU), and frequency-weighted mIoU (F-mIoU).
As shown in Table~\ref{tab:semseg}, our method achieves the best mAcc and mIoU on both Replica \cite{replica19arxiv} and ScanNet \cite{dai2017scannet}, while remaining competitive in F-mIoU on ScanNet. These results validate the effectiveness of the proposed hybrid instance-aware representation and cross-modal instance fusion strategy. The ablation without mask IoU further shows that rendered mask IoU improves instance fusion accuracy and cross-view semantic consistency. In contrast, methods based only on 3D overlap (e.g., ConceptGraphs \cite{gu2023conceptgraphsopenvocabulary3dscene}) tend to merge small objects into nearby larger ones, methods relying mainly on 2D mask IoU (e.g., DynamicGSG \cite{ge2025dynamicgsg}) are more sensitive to detector instability, and methods based mainly on voxel statistics (e.g., OpenGS-Fusion \cite{yang2025opengs}, OmniMap \cite{deng2025omnimap}) lack sufficient cross-view geometric constraints.
\subsection{3D Visual Grounding}
\label{subsec:object_retrieval}
To evaluate the ability of \textit{DGSG-Mind} in complex scene reasoning and target localization, we conduct 3D visual grounding experiments on ScanRefer~\cite{chen2020scanrefer} and Nr3D~\cite{achlioptas2020referit3d}. For fair comparison, semantic-matching baselines, including ConceptGraphs~\cite{gu2023conceptgraphsopenvocabulary3dscene}, DynamicGSG~\cite{ge2025dynamicgsg}, and OmniMap~\cite{deng2025omnimap}, use their original vision-language encoders, while agent-based methods are evaluated with \texttt{Qwen2.5-VL-72B}~\cite{bai2025qwen25vltechnicalreport}. Most compared methods, including ours, operate in a zero-shot setting on self-reconstructed geometries. In contrast, SeeGround~\cite{li2025seeground} uses ScanNet ground-truth point clouds and preprocessed Mask3D masks, and is therefore included as an upper-bound reference rather than a directly comparable zero-shot baseline. We report Average Precision (AP) at different 3D IoU thresholds. Following standard protocols, ScanRefer is evaluated on the \emph{Unique} and \emph{Multiple} splits, while Nr3D is evaluated on the \emph{Easy}, \emph{Hard}, \emph{View-Dependent}, and \emph{View-Independent} subsets. As shown in Table~\ref{tab:joint_grounding}, our method outperforms all zero-shot baselines on most metrics across ScanRefer and Nr3D. It substantially surpasses the strongest self-reconstructed VLM baseline, BBQ~\cite{linok2024barequeriesopenvocabularyobject}, which relies only on the textual structure of the scene graph during reasoning. The ablation of RoI-views confirms that RoI-based Gaussian renderings provide critical visual cues for spatial relation reasoning and object disambiguation. Compared with methods relying on simple semantic feature matching or text-only VLM reasoning, \textit{DGSG-Mind} achieves stronger visual-spatial reasoning by combining ID-annotated Gaussian renderings with structured scene graph context.
\subsection{Photometric Reconstruction Quality}
\label{subsec:reconstruction_quality}
\label{subsec:real_world}
\begin{figure}[tb]
\centering
\includegraphics[width=1.0\linewidth]{figures/4_evaluation/Real_6.pdf}
\caption{Real-World Deployment}
\label{fig:real}
\vspace{-1.8em}
\end{figure}
We evaluate the photometric reconstruction on Replica \cite{replica19arxiv} and ScanNet~\cite{dai2017scannet} using standard metrics: PSNR, SSIM, and LPIPS. For fair comparison, all methods use the dataset-provided ground-truth camera poses. As shown in Table~\ref{tab:reconstruction}, \textit{DGSG-Mind} achieves the best results on all Replica metrics and the highest PSNR and SSIM on ScanNet, while remaining competitive in LPIPS. These results demonstrate the effectiveness of voxel-guided Gaussian densification and joint optimization for accurate, photorealistic reconstruction.
\subsection{Real-World Deployment}
\begin{table}[!t]
\centering
\begin{tabular}{lcccc}\toprule
\textbf{Methods} & Removal & Movement & Addition & \textbf{All (\%)} \\ \midrule
DynamicGSG~\cite{ge2025dynamicgsg} & 4 / 20 & 0 / 10 & 7 / 20 & 22.0 \\
\textbf{Ours} & 19 / 20 & 7 / 10 & 17 / 20 & \textbf{86.0} \\
\bottomrule
\end{tabular}
\caption{\textbf{Quantitative results of dynamic update success rates:} Following the experimental protocol of DynamicGSG~\cite{ge2025dynamicgsg}, we evaluate three types of object-level changes across manipulation and navigation scenarios: 20 removal trials, 10 movement trials, and 20 addition trials. For object movement, an update is successful only if the moved object is removed from its original location and correctly reconstructed at its new location.}
\label{tab:changes}
\vspace{-2em}
\end{table}
As shown in Fig.~\ref{fig:real}, we deploy \textit{DGSG-Mind} on a wheeled mobile robot and a tabletop robotic arm to validate its real-world semantic mapping, scene understanding, and dynamic update capabilities. RGB-D streams are collected in real indoor scenes, where NVIDIA cuVSLAM provides camera poses for initial map construction, and posed RGB images are used to train a scene-specific ACE model for Gaussian-based relocalization. The reconstructed Gaussian maps provide high-quality geometry and semantically consistent instance representations, supporting tabletop manipulation and target-oriented navigation. As shown in Tab.~\ref{tab:changes}, \textit{DGSG-Mind} achieves a substantially higher overall success rate of dynamic updates than DynamicGSG \cite{ge2025dynamicgsg} during manipulation and navigation. Since \cite{ge2025dynamicgsg} relies on continuous VIO to provide initial poses, its tracking often drifts during arm manipulation or mobile robot motion, allowing it to complete only a small portion of dynamic updates at the beginning of the task. In contrast, \textit{DGSG-Mind} relocalizes the camera against the optimized Gaussian map from diverse viewpoints without requiring continuous online SLAM and detects object-level changes. It then updates the affected Gaussian instances and corresponding scene graph nodes while keeping the static background fixed, thereby maintaining an up-to-date scene representation for long-term embodied tasks.
\section{Conclusion}
\label{sec:conclusion}
\textit{DGSG-Mind} advances long-term embodied scene understanding in dynamic environments through hybrid instance-aware reconstruction, localized dynamic updates, and multimodal reasoning. Extensive experiments on public benchmarks and real-world platforms validate its effectiveness for scene understanding and task execution. Nevertheless, the system still relies on SLAM pose accuracy for initial reconstruction and ACE training, and its scalability to large-scale outdoor scenes is limited by the storage and GPU memory costs of 3D Gaussians. Future work will focus on developing an integrated tracking module and more scalable Gaussian map representations.