Constructing a 3D Scene from a Single Image
Extracted source text · arXiv 2505.15765
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
% \usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{xspace}
\usepackage[ruled,vlined]{algorithm2e}
\usepackage{amsmath}
\usepackage{graphicx}
% \usepackage{enumitem}
\newcommand{\modelname}{3DTown\xspace}
\newcommand{\kz}[1]{{\color{yellow}[Kaizhi: #1]}}
\newcommand{\xin}[1]{{\color{red}[Xin: #1]}}
\title{Constructing a 3D Town from a Single Image}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
% \author{%
% David S.~Hippocampus\thanks{Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies.} \\
% Department of Computer Science\\
% Cranberry-Lemon University\\
% Pittsburgh, PA 15213 \\
% \texttt{hippo@cs.cranberry-lemon.edu} \\
% % examples of more authors
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \AND
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% }
% \author{%
% Kaizhi Zheng%\thanks{Project Website: \url{https://sites.google.com/ucsc.edu/vlmbench}}
% \\
% % Department of Computer Science and Engineering\\
% University of California, Santa Cruz\\
% % Pittsburgh, PA 15213 \\
% \texttt{kzheng31@ucsc.edu} \\
% % examples of more authors
% \And
% Ruijian Zhang \\
% Columbia University \\
% % Address \\
% \texttt{xxx} \\
% % \And
% % Jing Gu \\
% % University of California, Santa Cruz \\
% % % Address \\
% % \texttt{jgu110@ucsc.edu} \\
% \AND
% Jie Yang \\
% Cybever AI \\
% % Address \\
% \texttt{jie@cybever.ai} \\
% \And
% Xin Eric Wang\\
% University of California, Santa Cruz \\
% % Address \\
% \texttt{xwang366@ucsc.edu} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% }
% \author{%
% Kaizhi Zheng\\
% University of California, Santa Cruz\\
% \texttt{kzheng31@ucsc.edu} \\
% % examples of more authors
% \And
% Ruijian Zhang \\
% Columbia University \\
% % Address \\
% \texttt{xxx} \\
% \And
% Jing Gu \\
% University of California, Santa Cruz \\
% % Address \\
% \texttt{jgu110@ucsc.edu} \\
% \AND
% Jie Yang \\
% Cybever AI \\
% % Address \\
% \texttt{jie@cybever.ai} \\
% \And
% Xin Eric Wang\\
% University of California, Santa Cruz \\
% % Address \\
% \texttt{xwang366@ucsc.edu} \\
% }
\author{Kaizhi Zheng$^{1}$ \quad Ruijian Zhang$^{2}$ \quad Jing Gu$^{1}$ \quad Jie Yang$^{3}$ \quad Xin Eric Wang$^{1}$
\\
$^1$UC Santa Cruz \quad $^2$Columbia University \quad $^3$Cybever AI \\
\texttt{\{kzheng31,xwang366\}@ucsc.edu}
}
\begin{document}
\maketitle
% \begin{figure}[h]
% \centering
% \includegraphics[width=\textwidth]{figures/3dtown-teaser.pdf}
% \caption{\textbf{Compositional Scene Generation with \modelname.} Given a single top-down image, \modelname synthesizes complex 3D scenes by composing locally generated regions into a globally coherent mesh. The scene is first initialized with a coarse geometry, then refined through region-wise generation using pretrained 3D object models. This compositional strategy enables high-quality reconstruction without requiring any 3D training or scene-specific supervision.\xin{this figure does not seem very novel. The input should be just one single image, while in the figure there are two inputs, the image and the spatial prior. Region-based generation also seems trivial from the figure. This is supposed to be a teaser figure to highlight the most impressive results or conceptualize the high-level idea. For this paper, maybe we can focus on showing some impressive results and compare ours with some baselines.}}
% \label{fig:teaser}
% \end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/3D_Town_teaser_update1.pdf}
\vspace{-4ex}
% \caption{\textbf{3D Scene Generation from a Single Image.} Given a single top-down reference image (left), \modelname generates a realistic and globally coherent 3D scene (top-middle) that better preserves geometry, texture, and layout compared to state-of-the-art baselines including Trellis, Hunyuan3D-2, and TripoSG. Unlike these baselines, which often suffer from broken structures, texture artifacts, or layout distortions, \modelname leverages a modular generation strategy to produce consistent, high-quality scene assets without any 3D training.}
\caption{\textbf{3D Scene Generation from a Single Image.} Given a top-down reference image (center), \modelname generates coherent and realistic 3D scenes that preserves geometry, texture, and layout compared to other state-of-the-art image-to-3D generation models. Our method also generalizes across diverse styles (right), producing high-quality outputs without any 3D training.
}
\label{fig:teaser}
\end{figure}
\begin{abstract}
% Generating high-quality 3D scenes from a single image remains a challenging problem due to the complexity of object interactions, spatial layout, and geometric consistency. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to resolution bottlenecks, layout hallucinations, and limited generalization. In this work, we introduce \modelname, a training-free framework for single-image-to-3D scene generation. Our approach builds on off-the-shelf image-to-3D object generation models and adopts a divide-and-conquer strategy to decompose scenes into manageable regions, which are generated independently to overcome resolution limitations. To ensure spatial coherence, we apply a point cloud-based 3D inpainting method that fills missing regions guided by geometric priors estimated from the input image. Without requiring any 3D training or domain-specific fine-tuning, 3DTown produces diverse, realistic, and geometrically consistent 3D scenes. Extensive qualitative evaluations demonstrate that our method significantly outperforms object-centric baselines, offering a practical solution for scalable and modular 3D scene synthesis.
Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes.
In this work, we introduce \textbf{\modelname}, a training-free framework designed to synthesize realistic and coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning.
Extensive experiments across diverse scenes show that \modelname outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, and TripoSG, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality 3D town generation is achievable from a single image using a principled, training-free approach. Project website: \url{https://eric-ai-lab.github.io/3dtown.github.io/}
\end{abstract}
\input{new_intro}
\input{related_work}
\input{method}
\section{Experiments}
\subsection{Experimental Setup}
\paragraph{Benchmark}
To the best of our knowledge, there is no established benchmark for 3D outdoor scene mesh generation from single images. Therefore, we construct a custom test set by prompting GPT-4o~\cite{hurst2024gpt} to generate 100 diverse top-down scene images in a variety of styles, such as “snow village,” “desert town,” and more. The generation details can be found in Appendix~\ref{appendix:gpt}.
% \noindent\textbf{Metrics}
% As there are no ground truth 3D meshes for these generated scenes, we evaluate model performance by comparing with each other, with both human preference and GPT-based assessments. Given the input reference image, two generated scenes from different models are evaluated along three axes: geometry quality, layout coherence, and texture coherence. \textit{Geometry quality} assesses which mesh contains more detailed and fine-grained structures closer to the image evidence. \textit{Layout coherence} evaluates whether the generated mesh has the same layout as the reference image. \textit{Texture coherence} measures how well the generated mesh texture aligns with the corresponding parts of the reference image. For each scene comparison pair, two independent human annotators from Amazon Mechanical Turk~\cite{crowston2012amazon} conduct the evaluations for \textbf{human preference win rate}. In parallel, we use GPT-4o to assess the same scenes using the same three criteria in separate evaluation passes. Since GPT-4o can provide probabilities of tokens, we use those to calculate the \textbf{GPT-4o-based weighted win rates} ($P(win)=P(\text{`A'}), P(lose)=P(\text{`B'})~\text{if `A' is our model}$), where $WeightedWinRate=P(win)*\mathbb{I}(win)+(1-P(lose))*\mathbb{I}(lose)$. All scenes are rendered as images by Blender~\cite{blender2025}. More details can be found in Appendix~\ref{appendix:gpt}.
\noindent\textbf{Metrics~}
In the absence of ground-truth meshes, we measure performance through pairwise comparisons between models. For every reference image, two generated scenes are evaluated on three criteria:
geometry quality, layout coherence, and texture coherence. \textit{Geometry quality} assesses which mesh contains more detailed and fine-grained structures closer to the image evidence. \textit{Layout coherence} evaluates whether the generated mesh has the same layout as the reference image. \textit{Texture coherence} measures how well the generated mesh texture aligns with the corresponding parts of the reference image.
Each pair is scored by two independent Amazon Mechanical Turk annotators, yielding a \textbf{human-preference win rate}.
We additionally obtain a \textbf{GPT-4o-based weighted win rate}: GPT-4o is shown the same pair, returns token probabilities for alternatives “A” and “B,” and we treat \(P(\text{A})\) as a soft vote when our method is option A (and analogously for B).
Weighted win rate is computed as
\(P(\text{win})\) if our model wins, and \(1-P(\text{lose})\) otherwise, then averaged across all pairs.
All scenes are rendered to RGB images with identical Blender settings; further GPT prompt details appear in Appendix~\ref{appendix:gpt}.
% \noindent\textbf{Baselines}
% We compare \modelname against three state-of-the-art image-to-3D generation models:
% \begin{itemize}
% \item \textbf{Trellis~\cite{xiang2024structured}}: Trellis is a large-scale 3D asset generation model that utilizes a unified Structured LATent (SLAT) representation and Rectified Flow Transformers to generate high-quality 3D assets from text or image prompts. It supports multiple output formats, including Radiance Fields, 3D Gaussians, and meshes.
% \item \textbf{Hunyuan3D-2~\cite{hunyuan3d22025tencent}}: Hunyuan3D-2 employs a two-stage generation pipeline, starting with the creation of a bare mesh using a scalable flow-based diffusion transformer, followed by the synthesis of a texture map. This approach decouples the complexities of shape and texture generation, allowing for high-resolution 3D asset creation.
% \item \textbf{TripoSG~\cite{li2025triposg}}: TripoSG is a compositional 3D scene generation method that constructs scenes by sequentially placing object instances from a learned shape library. It leverages a large-scale rectified flow transformer for 3D shape generation, achieving high-fidelity 3D meshes with precise correspondence to input images.
% \end{itemize}
% All models are evaluated in a zero-shot setting using their official pretrained models. All automatic image background removers have been deprecated to make sure the entire image is encoded, ensuring fair comparisons across methods.
\noindent\textbf{Baselines~}
We compare \modelname with three state-of-the-art image-to-3D generation methods:
(1) \textit{Trellis}~\cite{xiang2024structured} is a structured latent model that uses Rectified Flow Transformers to generate 3D assets from text or image prompts, supporting outputs such as radiance fields, 3D Gaussians, and meshes.
(2) \textit{Hunyuan3D-2}~\cite{hunyuan3d22025tencent} adopts a two-stage pipeline that separately synthesizes geometry and texture using a flow-based diffusion transformer, enabling high-resolution asset generation.
(3) \textit{TripoSG}~\cite{li2025triposg} is a high-fidelity 3D shape synthesis model that employs large-scale rectified flow transformers and hybrid geometric supervision to generate detailed 3D meshes from single images, demonstrating strong generalization across diverse styles. They use MV-Adapter~\cite{huang2024mvadapter} for texture generation.
All models are evaluated in a zero-shot setting using official pretrained checkpoints. To ensure fair comparison, background removal is disabled and the full input image is encoded for all methods.
\subsection{Main Results}
\begin{table}[t]
\centering
% \caption{\textbf{Comparisions between \modelname and baselines for 3D scene generation.}}
\caption{\textbf{Quantitative comparisons of \modelname and baselines.} We report human preference win rates and GPT-4o-based weighted win rates (\%) across geometry, layout, and texture quality. \modelname consistently outperforms all baselines by large margins in both evaluations.}
\resizebox{\columnwidth}{!}{
\begin{tabular}{lcccccc}
\toprule
& \multicolumn{3}{c}{Human Preference Win Rate (Percentage)} & \multicolumn{3}{c}{GPT-4o-based Weighted Win Rate (Percentage)} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
Models & Geometry Quality & Layout Coherence & Texture Coherence & Geometry Quality & Layout Coherence & Texture Coherence\\
\midrule
Trellis~\cite{xiang2024structured} & 31.50\% & 30.00\% & 33.00\% & 17.58\% & 19.04\% & 14.75\%\\
\modelname (Ours) & \textbf{68.50\%} & \textbf{70.00\%} & \textbf{67.00\%} & \textbf{82.42\%} & \textbf{80.96}\% & \textbf{85.25\%}\\
\midrule
Hunyuan3D-2~\cite{hunyuan3d22025tencent} & 33.50\% & 33.50\% & 31.50\% & 11.66\% & 12.13\% & ~~7.67\%\\
\modelname (Ours) & \textbf{66.50\%} & \textbf{66.50\%} & \textbf{88.34\%} & \textbf{88.34\%} & \textbf{87.87}\% & \textbf{92.33\%}\\
\midrule
TripoSG~\cite{li2025triposg} & 22.50\% & 23.00\% & 26.00\% & 24.60\% & 22.34\% & 22.79\% \\
\modelname (Ours) & \textbf{77.50\%} & \textbf{77.00\%} & \textbf{74.00\%} & \textbf{75.40\%} & \textbf{77.66\%} & \textbf{77.21\%}\\
\bottomrule
\end{tabular}
}
\label{table:main_result}
\end{table}
\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{figures/3D_Town_comparision.pdf}
\vspace{-4ex}
\caption{\textbf{Qualitative comparisons between \modelname and baselines.} Given a single top-down image (left column), we compare 3D scene outputs generated by \modelname, Trellis~\cite{xiang2024structured}, Hunyuan3D-2~\cite{hunyuan3d22025tencent}, and TripoSG~\cite{li2025triposg}. \modelname consistently produces globally coherent scenes with fine-grained geometry, accurate object layouts, and realistic textures across a variety of styles and environments. In contrast, Trellis often produces oversimplified geometry, Hunyuan3D-2 suffers from structural inconsistencies and domain mismatch, and TripoSG exhibits repetition artifacts and layout drift.}
\label{fig:qualitative_comparisons}
\end{figure}
% Since we have the region-based strategy to align region latents with specific image evidences and spatial-aware generation to improve geometry and layout coherence, we can find that \modelname outperforms Trellis, Hunyuan3D-2, and TripoSG across all criteria, under both human and GPT-4o evaluations, shown in Table~\ref{table:main_result}. Additionally, qualitative comparisons in Figure~\ref{fig:qualitative_comparisons} further demonstrate the advantages of our method.
% \noindent\textbf{Human Preference Evaluation.}
% \modelname achieves substantial win rate advantages over all baselines. Compared to Trellis, \modelname leads by 40 percentage points in layout coherence (70.0\% vs. 30.0\%), and over 35 points in both geometry (68.5\% vs. 31.5\%) and texture quality (67.0\% vs. 33.0\%). The trend is similar against Hunyuan3D-2, with a 33-point lead in both geometry and layout, and a 35.5-point advantage in texture quality. Against TripoSG, \modelname shows the largest margins—over 50 percentage points in geometry and layout, and nearly 48 points in texture. These results indicate that human annotators strongly prefer \modelname’s outputs in terms of structural realism, spatial layout, and surface quality.
% \noindent\textbf{GPT-4o-Based Evaluation.}
% The preference gap becomes even more pronounced under GPT-4o-based evaluations. \modelname outperforms Trellis by 64.8 points in geometry (82.42\% vs. 17.58\%), 61.9 points in layout (80.96\% vs. 19.04\%), and 70.5 points in texture (85.25\% vs. 14.75\%). When compared to Hunyuan3D-2, \modelname achieves an 76.7-point advantage in geometry, 75.7 in layout, and 84.7 in texture—highlighting a major quality gap. Even against TripoSG, which has a higher 3D resolution, \modelname maintains a win rate lead of roughly 50 points across all metrics. These results validate the effectiveness of our spatially guided, modular generation pipeline in producing high-fidelity and coherent 3D scenes.
% \noindent\textbf{Qualitative Comparisons.}
% Figure~\ref{fig:qualitative_comparisons} illustrates representative examples from each method. \modelname produces scene assets with clear structure, consistent layout, and realistic surface details that closely match the reference top-down image. In contrast, Trellis often generates overly centralized, low-resolution structures and lacks peripheral detail. Hunyuan3D-2 exhibits notable issues with layout distortion and geometry hallucinations despite acceptable textures in isolated parts. TripoSG maintains some compositional structure but frequently introduces repeated objects and ignores the layout evidence within the reference image. \modelname’s region-wise generation and spatial inpainting pipeline helps it avoid these artifacts while maintaining both global coherence and local fidelity.
% These findings demonstrate that our training-free, structured latent approach substantially improves 3D scene generation quality and robustness, enabling high-quality reconstruction from minimal 2D supervision. Additional qualitative results are provided in Figure~\ref{fig:comparision_appendix} in Appendix~\ref{appendix:more_results}.
Table~\ref{table:main_result} and Figure~\ref{fig:qualitative_comparisons} present the main quantitative and qualitative comparisons between \modelname and baseline methods. The results clearly demonstrate that \modelname consistently outperforms Trellis~\cite{xiang2024structured}, Hunyuan3D-2~\cite{hunyuan3d22025tencent}, and TripoSG~\cite{li2025triposg} across geometry, layout, and texture quality, as evaluated by both human annotators and GPT-4o. These improvements can be attributed to our region-based design and spatially guided generation strategy, which together promote better alignment between image features and 3D content, while preserving scene-wide consistency.
% \modelname’s region-wise decomposition enables each latent block to align with a localized image crop, improving geometric fidelity and texture detail by narrowing the domain gap between single-object training and scene-level inference. This leads to sharper object boundaries and more plausible textures in individual regions. Meanwhile, the use of spatial priors and masked 3D inpainting stabilizes inter-region transitions and enforces global layout constraints, significantly reducing spatial artifacts and disjoint structures.
% These design choices are reflected in the human evaluation results, where annotators consistently preferred \modelname for all criteria. Notably, layout coherence shows large gains over the baselines, suggesting that spatial guidance and region fusion are crucial for retaining scene structure. GPT-4o-based evaluations further amplify this trend, highlighting \modelname’s ability to produce geometry and textures that better match the image evidence from the visual features perspective.
\paragraph{Quantitative Analysis}
\modelname’s region-wise decomposition enables each latent block to align with a localized image crop, narrowing the domain gap between object-centric training and scene-level inference.
This design results in significant improvements in texture fidelity, with GPT-4o assigning \modelname a texture win rate of 92.3\% versus 7.7\% for Hunyuan3D-2. Furthermore, the upscale of resolution allows for more fine-grained structure representation, which reflects on the improvements in geometry quality, where human preferences increase by 37 percentage points over Trellis (68.5\% vs. 31.5\%) and 55 points over TripoSG (77.5\% vs. 22.5\%).
Meanwhile, the use of spatial priors and masked 3D inpainting stabilizes region layout and inter-region transitions, and enforces global layout constraints, significantly reducing spatial artifacts and disjoint structures. This is reflected in layout coherence scores: \modelname achieves 70.0\% in human preference compared to 30.0\% for Trellis, and reaches 87.9\% in GPT-4o evaluation against only 12.1\% for Hunyuan3D-2.
\paragraph{Qualitative Analysis}
Qualitatively, \modelname produces scene assets with clear structure, consistent layout, and realistic surface details that closely match the reference top-down image. In contrast, Trellis often generates overly centralized, low-resolution structures and lacks peripheral detail. Hunyuan3D-2 exhibits notable issues with layout distortion and geometry hallucinations despite acceptable textures in isolated parts. TripoSG maintains some compositional structure but frequently introduces repeated objects and ignores the layout evidence within the reference image. \modelname’s region-wise generation and spatial inpainting pipeline helps it avoid these artifacts while maintaining both global coherence and local fidelity.
These findings confirm that spatial decomposition and prior-guided inpainting are effective principles for lifting single-view image inputs into coherent, high-quality 3D scenes. Additional qualitative comparisons are available in Figure~\ref{fig:comparision_appendix} in Appendix~\ref{appendix:more_results}.
\subsection{Ablation Study}
\begin{table}[t]
\centering
% \caption{\textbf{Comparisions between \modelname and ablation methods.}}
\caption{\textbf{Ablation Study Results.} Win rates for geometry, layout, and texture show that removing region-based generation or landmark conditioning degrades performance, highlighting the importance of both components in \modelname.}
\resizebox{\columnwidth}{!}{
\begin{tabular}{lcccccc}
\toprule
& \multicolumn{3}{c}{Human Preference Win Rate (Percentage)} & \multicolumn{3}{c}{GPT-4o-based Weighted Win Rate (Percentage)} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
Models & Geometry Quality & Layout Coherence & Texture Coherence & Geometry Quality & Layout Coherence & Texture Coherence\\
\midrule
\modelname(w/o regions) & 20.00\% & 17.00\%& 13.50\% & 6.11\% & 8.48\% & 6.08\%\\
\modelname & \textbf{80.00\%} & \textbf{83.00\%} & \textbf{86.50\%} & \textbf{92.89\%} & \textbf{91.52\%} & \textbf{92.92\%}\\
\midrule
\modelname(w/o landmarks) & 41.50\% & 46.00\% & 42.00\% & 36.90\% & 44.21\% & 38.26\% \\
\modelname & \textbf{59.50\%} & \textbf{54.00\%} & \textbf{58.00\%} & \textbf{64.10\%} & \textbf{56.79\%} & \textbf{61.74\%}\\
\bottomrule
\end{tabular}
}
\label{table:ablation}
\end{table}
\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{figures/3D_Town_ablation.pdf}
\vspace{-4ex}
\caption{\textbf{Qualitative Ablation Results.} Left: Reference image. Middle: \modelname without landmark conditioning. Right: \modelname without region-based generation. Landmark conditioning ensures consistency for foreground objects, especially for objects across regions, while region-based generation preserves overall detail and coherence.}
\label{fig:ablation_figure}
\end{figure}
We conduct ablation studies to evaluate the contributions of key components in \modelname: the region-based generation strategy and the use of pre-generated landmarks. Both ablation studies still apply the spatial-aware 3D completion. Quantitative results are shown in Table~\ref{table:ablation}, and qualitative comparisons are illustrated in Figure~\ref{fig:ablation_figure}.
% \noindent\textbf{Without Region-Based Generation.}
% In this setting, we apply the pretrained 3D generator directly to the entire scene latent without dividing it into overlapping regions. As shown in Table~\ref{table:ablation}, the performance drops drastically across all evaluation axes, with win rates of only 6.11\% for geometry quality, 8.48\% for layout coherence, and 6.08\% for texture coherence. This confirms that holistic generation underutilizes the pretrained generator’s capacity and results in degraded geometry and broken spatial structure. Qualitatively (Figure~\ref{fig:ablation_figure}, right), the generated buildings are blurry, blocky, and spatially misaligned, with large gaps in the layout and a obviously loss of structural fidelity.
% \noindent\textbf{Without Landmark Conditioning.}
% To assess the effect of landmark-aware initialization, we remove the pre-generated landmark meshes and rely solely on monocular depth for spatial priors. This leads to noticeable drops in geometry and layout quality: 27.2 percentage points in geometry and 12.6 in layout coherence. The degradation is especially evident in landmark-rich scenes (e.g., bridges, towers, gates), where patch-wise generation may split the structure across regions. Without landmark anchoring, these large objects are inconsistently completed or misaligned across region boundaries. As shown in the middle example of Figure~\ref{fig:ablation_figure}, the right building area becomes fragmented and less recognizable compared to the full \modelname result.
\paragraph{Without Region-Based Generation~}
In this setting, the entire scene latent is directly passed into the pretrained 3D generator, without being split into localized regions. This leads to severe performance drops across all metrics. The results suggest that holistic generation fails to make full use of the pretrained model’s capacity, which was originally trained on single-object inputs. Without localized conditioning, the model struggles to resolve spatial context and image-to-3D correspondence, producing low-resolution and spatially incoherent outputs. As illustrated in Figure~\ref{fig:ablation_figure} (right), buildings lose structural sharpness and alignment, and the overall layout becomes underspecified.
\paragraph{Without Landmark Conditioning~}
Here, we disable the landmark-aware initialization and rely only on monocular depth to construct the spatial prior. This results in a notable decrease in geometry and layout quality, particularly in regions containing large foreground structures like gates or towers. Because region-based generation processes the scene in spatial chunks, landmarks act as anchors to maintain object continuity across patch boundaries. Without them, the model is more likely to hallucinate disjoint or mismatched content between regions. Figure~\ref{fig:ablation_figure} (middle) shows a representative failure: the landmark structure in the right-side building is incorrectly completed and misaligned with its neighboring parts.
These findings demonstrate that both region-wise decomposition and landmark conditioning are essential for producing coherent, high-fidelity 3D scenes. Regions allow pretrained object generators to operate within their effective receptive fields, while landmarks preserve global anchors and prevent semantic drift during inpainting.
\section{Conclusion}
To address the challenge of generating high-quality, coherent 3D scenes from a single image, we proposed \modelname, a training-free framework that synthesizes 3D scenes by decomposing them into overlapping regions and guiding generation with spatial priors. Central to our approach is the use of spatial-aware 3D completion, which enables localized object fidelity while maintaining global scene coherence through masked rectified flow. Empirical results show that \modelname outperforms existing methods across geometry, texture, and layout quality. These findings highlight the potential of modular, spatially grounded generation for scalable 3D scene synthesis from minimal input.
% \clearpage
\bibliographystyle{splncs04}
\bibliography{egbib}
\clearpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
\input{appendix}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \clearpage
% \input{checklist}
\end{document}
\section{Introduction}
Realistic 3D towns and structured environments serve as essential assets in simulation, robotics, digital content creation, and virtual world building. They enable scalable training for autonomous agents, immersive game environments, and rapid digital twin construction. However, constructing detailed and coherent 3D scenes typically requires either expensive 3D scanning equipment, multi-view data collection, or labor-intensive modeling. In contrast, generating 3D scenes from a single top-down image offers a lightweight and accessible alternative, making it possible to bootstrap realistic environments from minimal input.
Despite its appeal, generating a coherent and realistic 3D scene from a single image presents several fundamental challenges. First, the synthesized scene must exhibit consistent geometry across novel views, which is difficult for pure volumetric rendering methods such as Neural Radiance Fields (NeRF)~\cite{mildenhall2021nerf} or 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d}. While these techniques excel at photorealistic appearance modeling, they often suffer from geometry artifacts, especially in occluded or sparsely visible regions, leading to multiview inconsistencies and structural implausibility~\cite{li2022infinitenature,chung2023luciddreamer,zhang2024text2nerf,yu2024wonderjourney}. Second, the global layout of the generated scene must remain faithful to the input image. This is particularly difficult when considering the whole scene as one asset and using image-to-3D asset generators~\cite{xiang2024structured,hunyuan3d22025tencent,li2025triposg}, which fail to preserve spatial relationships between elements, resulting in distorted or semantically misaligned arrangements. Third, the local fidelity of individual objects should align closely with the visual evidence in the input. Due to the resolution constraints of 3D representations and the domain shift from object-level training to scene-level inference, previous image-to-3D generators are prone to creating low-quality meshes and misaligning the texture.
To address these challenges, we propose \textbf{\modelname}, a training-free framework for generating complex 3D scenes from a single top-down image, by enhancing the capability of image-to-3D object generators. Our method combines two core components: region-based generation and spatial-aware 3D inpainting. Each targets specific challenges in existing pipelines. We divide the scene into overlapping regions and synthesize each independently. This modular approach enables spatial upscaling and improves local alignment by grounding generation on localized image crops. To maintain global coherence and object continuity across regions, we estimate a coarse 3D structure from monocular depth and landmark detection, forming a spatial prior. A masked rectified flow mechanism then completes missing parts while preserving known content, enhancing structural consistency and object-level fidelity throughout the scene.
Through extensive experiments, we demonstrate that \modelname generates realistic, diverse, and geometrically consistent 3D scenes from a single top-down image. Our method significantly outperforms strong baselines, including Trellis~\cite{xiang2024structured}, Hunyuan3D-2~\cite{hunyuan3d22025tencent}, and TripoSG~\cite{li2025triposg}, across both human preference and GPT-based evaluations. Quantitative results show notable gains in geometry quality, layout coherence, and texture fidelity, while qualitative comparisons highlight \modelname's ability to preserve spatial structure and fine-grained detail. These results underscore the effectiveness of our modular, training-free approach to 3D scene synthesis.
Our main contributions are summarized as follows:
% \begin{itemize}
% \setlength\itemsep{-0.1em}
% \item We introduce \textbf{\modelname}, a training-free framework for 3D scene generation from a single top-down image, requiring no 3D supervision or fine-tuning.
% \item We design a modular generation strategy that combines region-wise synthesis with spatial-aware 3D inpainting, enabling high-quality 3D scene construction, robust image-to-3D scene alignment, and globally coherent layout without sacrificing local fidelity.
% \item We conduct comprehensive experiments across diverse visual domains and demonstrate that \modelname significantly outperforms strong baselines—including Trellis, Hunyuan3D-2, and TripoSG—in geometry quality, layout coherence, and texture realism.
% \end{itemize}
\begin{itemize}
\setlength\itemsep{-0.1em}
\item We propose \textbf{\modelname}, a training-free framework for generating structured 3D scenes from a single top-down image, leveraging pretrained object-centric generators for zero-shot scene asset synthesis.
\item We develop a modular generation strategy that combines region-wise latent synthesis with spatial-aware 3D inpainting, effectively addressing resolution bottlenecks, image-geometry misalignment, and inter-region inconsistency.
\item We conduct comprehensive evaluations on diverse scenes and show that \modelname outperforms state-of-the-art baselines (Trellis, Hunyuan3D-2, and TripoSG) in geometry quality, layout coherence, and texture realism under both human and GPT-4o-based assessments.
\end{itemize}
\section{Related Work}
% Existing image-to-3D scene generation approaches generally fall into three categories, each with distinct limitations. The first line of work leverages the expressive rendering capabilities of representations like Radiance Fields and 3D Gaussians to expand 2D images into 3D scenes. Although these methods excel at producing photorealistic appearances, they often fail to ensure geometric consistency due to occlusions and inconsistencies in multiview generation. A second class of methods, such as BlockFusion~\cite{wu2024blockfusion} and LT3SD~\cite{meng2024lt3sd}, employs block-wise generation using 3D latent spaces to preserve structural coherence. However, these models typically lack high-fidelity texture generation and are constrained to narrow domains, often relying on scene-specific datasets like 3D-FRONT for indoor environments. The third direction utilizes image-to-3D object generation models to synthesize an entire scene as a single asset. While these models can generate plausible geometry and texture at the object level, applying them directly to full scenes introduces new challenges: limited 3D resolution becomes a bottleneck when encoding entire environments, and domain shift from single-object to scene-level images often leads to severe layout hallucinations.
\paragraph{3D Scene Generation with 2D Generative Models}
Recent progress in 2D generative modeling~\cite{rombach2022high,ho2020denoising,chang2024flux,ramesh2021zero} has inspired a growing body of work that combines image synthesis with external 3D reconstruction techniques to generate 3D scenes. A common strategy is to use pretrained image generation models to outpaint a given input view according to known or sampled camera trajectories, followed by reconstruction via depth fusion or volumetric representations such as Neural Radiance Fields (NeRF)~\cite{mildenhall2021nerf} and 3D Gaussian Splatting (3DGS)~\cite{kerbl20233d}. Early approaches primarily targeted indoor scenes~\cite{wiles2020synsin,koh2021pathdreamer,hollein2023text2room,koh2023simple}, while more recent efforts have extended to arbitrary natural scenes~\cite{li2022infinitenature,cai2023diffdreamer,fridman2023scenescape,chung2023luciddreamer}, achieving improved reconstruction quality through advanced depth fusion pipelines~\cite{yu2024wonderjourney,zhang20243d,zhang2024text2nerf,yang2024scene123,shriram2024realmdreamer,engstler2024invisible,yu2024wonderworld}. Despite producing visually compelling results, these methods often suffer from geometric inconsistencies caused by hallucinations during image generation, particularly in occluded or ambiguous regions.
% To address such inconsistencies, subsequent works have explored panoramic~\cite{stan2023ldm3d,li2024scenedreamer360,wu2023panodiffusion,schult2024controlroom3d,liang2024luciddreamer} and multiview image generation~\cite{liu2024panofree,tang2023emergent,gao2024cat3d} to improve spatial coverage and structural coherence. However, these approaches typically restrict the freedom of movement or require the camera motion for close views, and still struggle with accurate 3D geometry synthesis. A recent concurrent work, Syncity~\cite{engstler2025syncity}, takes a step further by generating regional block images from text prompts and assembling them via a 3D generator. While promising, this method is currently limited to compact scene layouts and relies on language conditioning rather than visual input. In this work, we aim to create geometry geometry-consistent 3D scene asset with arbitrary layout from a single top-down image.
% Recent advances in 2D generative models~\cite{rombach2022high,ho2020denoising,chang2024flux,ramesh2021zero} have spurred interest in synthesizing 3D scenes by combining image generation with reconstruction techniques. A common pipeline involves outpainting input views using pretrained image models along known or sampled camera paths, followed by reconstruction using NeRF~\cite{mildenhall2021nerf}, 3D Gaussian Splatting~\cite{kerbl20233d}, or depth fusion. Early methods focused on indoor settings~\cite{wiles2020synsin,koh2021pathdreamer,hollein2023text2room,koh2023simple}, while recent works extend to outdoor and natural scenes~\cite{li2022infinitenature,cai2023diffdreamer,fridman2023scenescape,chung2023luciddreamer}, aided by improved fusion pipelines~\cite{yu2024wonderjourney,zhang20243d,zhang2024text2nerf,yang2024scene123,shriram2024realmdreamer,engstler2024invisible,yu2024wonderworld}. However, these methods often exhibit geometric inconsistencies due to hallucinated content, especially in occluded regions.
To mitigate this, recent approaches explore panoramic~\cite{stan2023ldm3d,li2024scenedreamer360,wu2023panodiffusion,schult2024controlroom3d,liang2024luciddreamer} and multiview generation~\cite{liu2024panofree,tang2023emergent,gao2024cat3d} to improve spatial coverage and structure. However, these approaches typically restrict the freedom of movement or require the camera motion for close views, and still struggle with accurate 3D geometry synthesis. The concurrent work Syncity~\cite{engstler2025syncity} proposes block-wise generation via a 3D generator and assembles them to scenes, but all generated scenes have compact layouts and cannot handle the entire scene images. In contrast, we aim to generate geometry-consistent 3D assets with arbitrary layout directly from a single top-down image.
\paragraph{3D Scene Generative Model}
Beyond 2D-to-3D approaches, recent research has explored the direct generation of 3D scenes using native 3D representations. One prominent line of work leverages large language models~\cite{feng2023layoutgpt,zhou2024gala3d,ccelen2024design,hu2024scenecraft,sun20233d} or trained diffusion models~\cite{tang2024diffuscene,lin2024instructscene,zhai2024echoscene,vilesov2023cg3d,maillard2024debara} to synthesize scene layouts, which are then populated by retrieving or generating 3D assets according to the predicted configuration. While these methods enable semantically plausible arrangements, they are often constrained to predefined object categories and struggle to produce coherent background geometry. Furthermore, they frequently encounter inter-object collisions.
Meanwhile, a separate class of methods directly models the 3D scene generation process in latent 3D spaces~\cite{wu2024blockfusion,Meng2024LT3SDLT,Ren2023XCubeL3,Liu2023PyramidDF,Lee2024SemCitySS,Chai2023PersistentNA}. For instance, BlockFusion~\cite{wu2024blockfusion} trains a diffusion model to generate triplane representations for regional scene blocks, while LT3SD~\cite{Meng2024LT3SDLT} and XCube~\cite{Ren2023XCubeL3} adopt truncated unsigned distance fields (TUDFs) or voxels to represent full scenes. Despite their architectural innovations, these models require large-scale, domain-specific datasets for training, such as indoor room datasets or urban layouts, which limits their generalizability to unseen scene types or out-of-distribution inputs. In contrast, our method avoids any training or fine-tuning and instead employs a modular, training-free pipeline that generates diverse 3D scenes from top-down images.
% Instead of constructing scenes from 2D images, some recent works propose to directly generate scenes with 3D representations. One line of the work is to generate scene configurations through large language models~\cite{feng2023layoutgpt,zhou2024gala3d,ccelen2024design,hu2024scenecraft,sun20233d} or trained diffusion models~\cite{tang2024diffuscene,lin2024instructscene,zhai2024echoscene,vilesov2023cg3d,maillard2024debara}, and then put the retrieved or generated objects according to the configurations. However, those works are limited to object categories and are hard to create backgrounds. Also, they may suffer from collision issues. Instead, other works design the models to generate latents in 3D formats~\cite{wu2024blockfusion,Meng2024LT3SDLT,Ren2023XCubeL3,Liu2023PyramidDF,Lee2024SemCitySS,Chai2023PersistentNA}. BlockFusion~\cite{wu2024blockfusion} train a diffusion model to generate triplanes of region blocks conditioning on 2D layout, while LT3SD~\cite{Meng2024LT3SDLT} uses truncated unsigned distance field (TUDF) and Xcube~\cite{Ren2023XCubeL3}. However, these methods all require large-scale scene datasets for training, which further limits them to generalizing to broad scene types. In contrast, our method can generate diverse 3D scene assets without any training or fine-tuning.
\paragraph{Image-to-3D Asset Generation}
%Trellis,Huyuan3D-2,TripoSG,DreamFusion,Zero-1-to-3
Another line of research focuses on generating individual 3D assets from single images, aiming to reconstruct both geometry and appearance of the depicted objects. Early works~\cite{Poole2022DreamFusionTU,Lin2022Magic3DHT,Tang2023MakeIt3DH3,Liu2023Zero1to3ZO,Tang2023DreamGaussianGG} leverage 2D generative models in combination with NeRF or 3D Gaussian Splatting optimization to lift 2D images into 3D. However, these approaches often suffer from low geometry fidelity and slow optimization. To address these limitations, more recent methods~\cite{Gupta20233DGenTL,Xiong2024OctFusionOD,Li2024CraftsManHM,vahdat2022lion,Zhang2024CLAYAC} propose to directly generate 3D latent representations via diffusion models, while others~\cite{xiang2024structured,hunyuan3d22025tencent,li2025triposg} adopt rectified flow to improve both quality and efficiency.
These methods typically rely on large-scale 3D datasets such as Objaverse-XL~\cite{Deitke2023ObjaverseXLAU}, which mainly consist of object-centric assets. As a result, applying them directly to full-scene synthesis introduces several challenges: limited 3D resolution becomes a bottleneck for encoding complex environments, and domain shift from object-level training to scene-level inference often leads to spatial inconsistencies and layout hallucinations.
In this work, we build upon pretrained 3D generation models~\cite{xiang2024structured} and address these limitations through a region-based generation strategy combined with spatial-aware 3D inpainting. This design enables us to produce 3D scene assets with both high geometric fidelity and global layout coherence from a single top-down image.
% \section{Preliminaries: Rectified Flow Model for 3D Object Generation}
% In this work, we adopt the rectified flow model Trellis~\cite{xiang2024structured} as the foundation for image-to-3D generation. Trellis introduces a structured latent representation $z$ that encodes both geometric and appearance information of a 3D asset $O$:
% \begin{equation}
% z = \{(p_i, f_i)\}^{L}_{i=1},\quad p_i \in \{0, 1, \ldots, N-1\}^3,\quad f_i \in \mathbb{R}^C,
% \end{equation}
% where $p_i$ denotes the positional index of an active voxel in the 3D grid, and $f_i$ represents the associated latent feature vector of dimension $C$. Here, $N$ is the resolution of the voxel grid, and $L$ is the total number of active voxels. In general, $p_i$ captures the coarse structural layout of the object, while $f_i$ encodes fine-grained local appearance and shape information. In the original setup, $N$ is set to 64 and $C$ to 8.
% Trellis employs two rectified flow transformers: a sparse structure generator $\mathcal{G}_s$ and a structured latent generator $\mathcal{G}_L$. At inference time, $\mathcal{G}_s$ first generates the active voxel positions $\{p_i\}^L$ from the noisy grids $V_T$ with Gaussian noise. These positions are then used by $\mathcal{G}_L$ to generate the corresponding latent features $\{f_i\}^L$ from noisy features $F_T$. Both generators are conditioned on an input image condition $C_I$, encoded by DINOv2~\cite{oquab2023dinov2}. Finally, the structured latent representation $z$ is decoded into a 3D object $O$ using different sparse 3D VAE decoders, which can output formats such as 3D Gaussians, Radiance Fields, or meshes. The overall process is described as follows:
% \begin{align}
% \{p_i\}^L &= \mathcal{G}_s(V_T\,|\,C_I),\quad V_T \sim \mathcal{N}(0, I)^{[N,N,N]} \\
% \{f_i\}^L &= \mathcal{G}_L(F_T\,|\,C_I, \{p_i\}^L),\quad F_T \sim \mathcal{N}(0, I)^{[L, C]} \\
% O &= \text{Decoder}(z),\quad z = \{(p_i, f_i)\}_{i=1}^L
% \end{align}
% By decoupling structural generation from feature generation, Trellis facilitates spatially-aware 3D inpainting, a capability we further explore in the subsequent sections.
\section{Method}
\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{figures/3D_Town_method_new.pdf}
\vspace{-4ex}
\caption{\textbf{Overview of the \modelname\ Pipeline.}
Given a single top-down image, we first estimate a coarse scene structure via monocular depth and landmark extraction to initialize the scene latent (\textit{Spatial Prior Initialization}). The scene is divided into overlapping regions for localized synthesis and progressively fused into a coherent global latent (\textit{Region-based Generation \& Fusion}). Each region is completed using a two-stage masked rectified flow pipeline with a sparse structure generator \(\mathcal{G}_s\) and a structured latent generator \(\mathcal{G}_L\) (\textit{Spatial-aware 3D Completion}). The final 3D scene is decoded from the completed structured latent. }
\label{fig:method_pipeline}
\end{figure}
Given a single top-down image of an unknown scene, our objective is to synthesize a high-quality 3D scene that is geometrically and visually consistent with the input view. Figure~\ref{fig:method_pipeline} illustrates an overview of our pipeline.
From the scene image, we first construct the scene latents with structured latent representations (Sec.~\ref{sec:latent}) from spatial priors (Sec.~\ref{sec:spatial_init}). Then, we divide the scene-level latent into region-level latents for sequential processing. For each region, we extract the latents from the latest scene-level structured latents and take those as priors for spatial-aware 3D completion (Sec.~\ref{sec:region_generation}) by using the base 3D generator, Trellis~\cite{xiang2024structured}. Later, we fuse the updated region latents to the scene latents for cross-region consistency (Sec.~\ref{sec:region_fusion}). After finishing all regions, we leverage pretrained object decoders on the complete scene structured latents to obtain a 3D scene asset. Details are explained below.
% To leverage the pretrained capabilities of the off-the-shelf 3D generator Trellis~\cite{xiang2024structured}, we represent the entire scene using a structured latent representation $z$ defined on a 3D grid of resolution $M$, where $M > N$ enables upscaling (with $M = 2N$ by default). Each subgrid of size $N^3$ can then be decoded into a 3D asset using Trellis’s pretrained decoders. To construct a coherent scene-level latent representation, we first initialize the structured latent $z_0$ using spatial priors derived from the input image (Sec.~\ref{sec:spatial_init}). Next, we divide the scene-level latent into smaller region-level latents for sequential processing (Sec.~\ref{sec:region_generation}). Finally, we fuse the generated regions to form a complete scene-level structured latent (Sec.~\ref{sec:region_fusion}).
\subsection{Structured Latent Representation}
\label{sec:latent}
To leverage the pretrained knowledge of the base 3D generator, we construct the scene with a structured latent representation $z$~\cite{xiang2024structured}:
\begin{equation}
z = \{(p_i, f_i)\}^{L}_{i=1},\quad p_i \in \{0, 1, \ldots, K-1\}^3,\quad f_i \in \mathbb{R}^C,
\end{equation}
where $p_i$ denotes the positional index of an active voxel in the 3D grid, and $f_i$ represents the associated latent feature vector of dimension $C$. Here, $K$ is the resolution of the voxel grid, and $L$ is the total number of active voxels. In general, $p_i$ captures the coarse structural layout of the object, while $f_i$ encodes fine-grained local appearance and shape information. For the pretrained models, the resolution $K$ should be equal to $N=64$. To upscale the resolution for the scene, we construct scene structured latents with resolution $M$, where $M>N$ ($M=2N$ by default).
For the general image-to-3D asset generation, there are two pretrained rectified flow transformers: a sparse structure generator $\mathcal{G}_s$ and a structured latent generator $\mathcal{G}_L$. At inference time, $\mathcal{G}_s$ first generates the active voxel positions $\{p_i\}^L$ from the noisy grids $V_T$ with Gaussian noise. These positions are then used by $\mathcal{G}_L$ to generate the corresponding latent features $\{f_i\}^L$ from noisy features $F_T$. Both generators are conditioned on an input image condition $C_I$, encoded by DINOv2~\cite{oquab2023dinov2}. Finally, the structured latent representation $z$ is decoded into a 3D object $O$ using object decoders, which include different sparse 3D VAE decoders for 3D Gaussians, Radiance Fields, and mesh generation. The overall process is described as follows:
\begin{align}
\{p_i\}^L &= \mathcal{G}_s(V_T\,|\,C_I),\quad V_T \sim \mathcal{N}(0, I)^{[N,N,N]} \\
\{f_i\}^L &= \mathcal{G}_L(F_T\,|\,C_I, \{p_i\}^L),\quad F_T \sim \mathcal{N}(0, I)^{[L, C]} \\
O &= \text{ObjectDecoder}(z),\quad z = \{(p_i, f_i)\}_{i=1}^L
\end{align}
\subsection{Spatial Prior Initialization}
\label{sec:spatial_init}
Given a top-down image, the image-to-3D generator may produce outputs in arbitrary orientations. To resolve this ambiguity and provide a consistent structural prior, we initialize the scene using point clouds. Specifically, we employ a monocular depth estimator~\cite{wang2025vggt} to predict a depth image and infer camera parameters, from which we construct pixel-wise point clouds. Due to occlusions, these point clouds contain missing regions, which will later be filled by the image-to-3D generator.
However, occluded areas of complex objects (e.g., buildings) may have multiple plausible completions. To enforce cross-region consistency in such cases, we propose first generating landmark objects independently and conditioning subsequent generations on their geometry. To achieve this, we use Florence2~\cite{xiao2024florence} to propose landmark bounding boxes and SAM2~\cite{ravi2024sam} to extract instance masks. Each detected landmark is then processed individually using the 3D generator to obtain instance-level meshes. These meshes are aligned with the raw point clouds using ICP~\cite{rusinkiewicz2001efficient}, replacing the original landmark regions with mesh-derived point clouds.
After normalization, we voxelize the aggregated scene point clouds at resolution $M$ to obtain the initial voxelized scene, including foreground voxels $V_0^f$, background voxels $V_0^b$, and the full scene voxel set $V_0$:
\begin{equation}
V_0 = \{V_0^b, V_0^f\} = \{p_i\}^L,\quad p_i \in \{0, 1, \ldots, M-1\}^3
\end{equation}
Finally, we initialize the corresponding voxel features $F_0$ as zeros and construct the initial structured latent representation for the scene as $z_0 = \{(V_0, F_0)\}$, shown in the top-left of Figure~\ref{fig:method_pipeline}.
\subsection{Region-based Generation}
\label{sec:region_generation}
% We adopt a region-based strategy for structured latent generation to address the limitations of applying pretrained object-centric models directly to full-scene synthesis. This design is motivated by two key factors: First, the pretrained rectified flow models are trained on single-object datasets, where each object occupies the entire latent space at a fixed resolution $N$. When this limited representation capacity is stretched across an entire scene, it leads to severely degraded geometry, including low-resolution meshes and missing fine structures. Constructing a scene-level latent at this resolution is thus insufficient for high-fidelity scene synthesis. Second, applying the pretrained model directly to a full scene from a single top-down view often leads to semantic hallucinations or geometric distortions. The model may struggle to maintain meaningful spatial relationships between objects or to align 3D content with image features. By splitting the scene into smaller, overlapping regions and conditioning each region on its corresponding image crop, we introduce localized grounding that mitigates hallucination and preserves coherence between geometry and appearance.
We adopt a region-based strategy to overcome the limitations of applying pretrained object-centric 3D generators directly to full scenes. These models are trained on single-object data, where each object occupies the full latent space at a fixed resolution \(N\). When extended to an entire scene, this limited capacity results in low-resolution geometry and missing details. Moreover, direct scene-level generation from a single top-down image often leads to layout distortions and semantic hallucinations, as the model struggles to maintain spatial relationships or align 3D content with image cues. To address this, we divide the scene into overlapping regions and condition each on its corresponding image crop, enabling locally grounded and high-fidelity generation.
To implement this, we divide the initial scene voxel grid $V_0$ (of resolution $M$) into a set of overlapping region-level subgrids $\{V_0^{(r)}\}_{r=1}^R$, each with shape $N^3$, where $N$ is the resolution used by the pretrained 3D generator. For each region $r$, we also extract the corresponding image crop to obtain a localized image conditioning input $C_I^{(r)}$. This ensures that the generation in each region is locally grounded in the image evidence.
\paragraph{Spatial-aware 3D Completion}
While region-based generation improves local fidelity, it introduces a new challenge: how to maintain global consistency, especially across overlapping regions. Furthermore, since the 3D generator may create assets from any orientation, we need to alleviate the misalignment between image conditions and region latents to enable further region fusion.
To address these challenges, we draw inspiration from training-free inpainting methods in 2D diffusion models, such as RePaint~\cite{lugmayr2022repaint}, and adapt a similar approach for 3D generation. We propose to use a masked rectified flow pipeline for 3D completion, which treats the partially completed global scene latent as a constraint and performs conditional generation over the current region by completing only the unknown parts.
Given a region-level subgrids $V_0^{(r)}$, we designate known active voxels as positions $\{p_{i,0}^{(r)}\}^{L_{r,0}}$ with corresponding latent features $\{f_{i,0}^{(r)}\}^{L_{r,0}}$.
For the coarse structure generation, we define a binary mask $m_s^{(r)}$ where inactive voxels are marked for regeneration. We use the sparse structure generator $\mathcal{G}_s$ to complete the region structure and obtain active voxel positions $\{p_{i}^{(r)}\}^{L_r}$. Next, for the fine-grained local features generation, we retain original features for positions overlapping with known voxels; otherwise, features are initialized with Gaussian noise. A second binary mask $m_L^{(r)}$ identifies unknown features for regeneration. We then use the structured latent generator $\mathcal{G}_L$ to obtain the inpainted local features $\{f_{i}^{(r)}\}^{L_r}$. Finally, we can construct the region structured latent $z^{(r)}=\{(p_i^{(r)},f_i^{(r)})\}^{L_r}$. These two completions both leverage the masked rectified flow pipeline (Details in the next paragraph). The whole process is shown at the bottom of Figure~\ref{fig:method_pipeline} and can be formally written as follows:
\begin{align}
\{p_i^{(r)}\}^{L_r} &= \mathcal{G}_s(V_T\,|C_I^{(r)},\{p_{i,0}^{(r)}\}^{L_{r,0}},m_s^{(r)}),\quad V_T \sim \mathcal{N}(0, I)^{[N,N,N]} \\
\{f_i^{(r)}\}^{L_r} &= \mathcal{G}_L(F_T\,|C_I^{(r)}, \{p_i^{(r)}\}^{L_r}, \{f_{i,0}^{(r)}\}^{L_{r,0}}, m_L^{(r)}),\quad F_T \sim \mathcal{N}(0, I)^{[L_r, C]} \\
z^{(r)} &= \{(p_i^{(r)},f_i^{(r)})\}^{L_r}
\end{align}
\paragraph{Masked Rectified Flow for Completion}
We adopt a masked generation strategy based on rectified flow to complete the unknown regions of a structured 3D latent. Let \(x_{\text{known}}\) denote the known latent values to preserve, and let \(m \in \{0,1\}\) be a binary mask that indicates which parts of the latent should be regenerated (\(m=1\)) and which should remain fixed (\(m=0\)).
We initialize the latent variable \(x_T \sim \mathcal{N}(0, I)\) with Gaussian noise, representing the unknown region at the final time step. For each timestep \(t = T, T{-}1, \dots, 1\), we perform \(U\) resampling steps to improve stability and smoothness~\cite{lugmayr2022repaint}. At each resampling iteration \(u\), we first compute the flow field \(v_\theta(x_t, t)\) using the rectified flow model and apply an Euler update to obtain the intermediate latent:
\begin{equation}
x_{t_{\text{prev}}} = x_t - \Delta t \cdot v_\theta(x_t, t),
\quad \text{where } \Delta t = 1.
\end{equation}
We then re-noise the known region using a forward noise operator:
\begin{equation}
\texttt{forward\_step}(x, t) = (1 - t)\cdot x + \left[\sigma_{\min} + (1 - \sigma_{\min})\cdot t\right]\cdot \epsilon,
\quad \epsilon \sim \mathcal{N}(0, I),
\end{equation}
and merge it back into the latent using the mask \(m\):
\begin{equation}
x_{t_{\text{prev}}} \leftarrow m \odot x_{t_{\text{prev}}} + (1 - m) \odot \texttt{forward\_step}(x_{\text{known}}, t_{\text{prev}}).
\end{equation}
\(\sigma_{\min}\) denotes the minimum noise scale used by the pretrained rectified model.
If \(t > 1\) and the latent needs to be resampled, we apply additional forward noise to the merged latent: $x_t \leftarrow \texttt{forward\_step}(x_{t_{\text{prev}}}, \Delta t)$.
% \begin{equation}
% x_t \leftarrow \texttt{forward\_step}(x_{t_{\text{prev}}}, \Delta t).
% \end{equation}
Otherwise, we simply continue with \(x_t \leftarrow x_{t_{\text{prev}}}\). This masked rectified flow process iterates until \(t = 0\), at which point the completed latent \(x_0\) is returned. The full procedure is outlined in Algorithm~\ref{alg:flow-euler-repaint} in Appendix~\ref{appendix:method}.
\subsection{Region Fusion}
\label{sec:region_fusion}
For each generated region, we update the scene-level structured latent $z_0$ by replacing the corresponding part with the region-level latent $z^{(r)}$. Because regions are extracted using a patchification strategy, some may contain only partial observations of foreground landmarks. To preserve landmark integrity, we discard those structured latents corresponding to partial foregrounds during fusion.
Each region is extracted from the latest version of the scene-level latent, ensuring consistency across regions. If a region overlaps with previously generated ones, its overlapping voxels are constrained to match the existing content during generation. This enforces continuity and avoids inconsistencies in overlapping areas, leading to smooth transitions between adjacent regions while preserving already synthesized content.
Once all regions have been processed, the final scene-level latent is decoded using object decoders to produce scene-level meshes and 3D Gaussians. The complete textured scene is then rendered using a combination of physically based rendering (PBR) baking and Gaussian Splatting. Additional implementation details, including the patchification strategy, are provided in Appendix~\ref{appendix:method}.
\begin{figure}[!t]
\centering
\includegraphics[width=\textwidth]{figures/3D_Town_comparision_appendix.pdf}
% \vspace{-1\baselineskip}
\caption{\textbf{More qualitative comparisons between \modelname and baselines.} From the image, we can find that \modelname can generate more coherent scenes from diverse scene images.}
\label{fig:comparision_appendix}
% \vspace{-1\baselineskip}
\end{figure}
\begin{algorithm}[t]
\caption{Masked Rectified Flow for Completion Pipeline}
\label{alg:flow-euler-repaint}
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{
\(v_\theta(x,t)\) — learned flow field;\quad
\(x_{known}\) — known latent to preserve;\\
\(m\in\{0,1\}\) — mask for regeneration (1=regenerate, 0=preserve);\\
% \(x_T\) — initial unknown latent at \(t=T\);\\
\(T\) — total steps;\quad
\(U\) — Resample times per step;\quad
\(\sigma_{\min}\) — minimum noise scale
}
\Output{\(x_0\) — regenerated latent}
\BlankLine
\tcc{Forward‐noise operator}
\(\epsilon \sim \mathcal{N}(0,I)\)\\
\(\displaystyle \text{forward\_step}(x, t)
\;=\; (1 - t)\,x \;+\; \bigl[\sigma_{\min} + (1-\sigma_{\min})\,t\bigr]\,\epsilon\)
\BlankLine
%——— Initialization ———%
\tcc{Initialization}
\(x_T \sim \mathcal{N}(0,I)\)\;
%——— Sampling Loop ———%
\For{\(t = T, T-1, \dots, 1\)}{
\(t_{\text{prev}} \leftarrow t - 1\)\;
\(\Delta t \leftarrow t - t_{\text{prev}}\)\;
\For{\(u = 1, \dots, U\)}{
\(\;v \leftarrow v_\theta(x_t, t)\)\ \tcc*{predict flow field}
\(x_{t_{\text{prev}}} \leftarrow x_t \;-\; \Delta t \, v\)\ \tcc*{Euler update on unknown}
\(\hat{x}_{t_{\text{prev}}} \leftarrow \text{forward\_step}(x_{\text{known}},\,t_{\text{prev}})\)\ \tcc*{re-noise known}
\(x_{t_{\text{prev}}} \leftarrow m \odot x_{t_{\text{prev}}}
\;+\; (1-m)\odot \hat{x}_{t_{\text{prev}}}\)\;
\If{\(u < U\) \textbf{and} \(t>1\)}{
\(x_t \leftarrow \text{forward\_step}(x_{t_{\text{prev}}},\,\Delta t)\)\;
}
\Else{
\(x_t \leftarrow x_{t_{\text{prev}}}\)\;
}
}
}
\Return \(x_0 = x_{0}\)\;
\end{algorithm}
\section{More Results}
\label{appendix:more_results}
Figure~\ref{fig:comparision_appendix} presents additional qualitative results comparing \modelname with Trellis~\cite{xiang2024structured}, Hunyuan3D-2~\cite{hunyuan3d22025tencent}, and TripoSG~\cite{li2025triposg} across a diverse set of visual scenes. These examples further demonstrate the robustness and generality of our approach across different architectural styles, spatial layouts, and artistic domains.
\modelname consistently produces scene assets that are geometrically detailed, visually coherent, and well-aligned with the input reference images. In contrast, baseline models frequently suffer from artifacts such as repeated structures, layout collapse, or low-resolution textures. These comparisons further highlight the benefits of our region-based generation and spatial-aware inpainting pipeline in generating diverse 3D scenes from a single image without training.
\section{Method Details}
\label{appendix:method}
\paragraph{Region Extraction Strategy.}
To generate overlapping regions with balanced seams, we first compute the tight bounding box of all occupied voxels and tile it with a base grid of non-overlapping patches of size \((p_x,p_y,p_z)\). For the vertical (\(z\)) axis, start positions are evenly interpolated so that the entire height is covered with the minimum number of full patches.
Next, we insert \emph{seam patches} to equalise overlap: for every pair of neighbouring base-grid origins we create an additional patch whose origin is the midpoint between them—first along the \(x\)-axis (keeping \(y,z\) fixed), then along the \(y\)-axis (keeping \(x,z\) fixed).
The union of base and seam origins is deduplicated and sorted, after which the corresponding voxel sub-volumes are extracted.
The procedure returns the list of patch origins and their binary masks, and is invoked once per scene to define the region schedule used in our region-wise generation and fusion pipeline.
\paragraph{Masked Rectified Flow Algorithm }
For completeness, we provide the full algorithmic details of the masked rectified flow completion process used in our spatial-aware 3D inpainting pipeline. While the core formulation is introduced in Section~\ref{sec:region_generation}, this pseudocode (Algorithm~\ref{alg:flow-euler-repaint}) clarifies the iterative update, re-noising, and resampling procedures that enable conditional generation of unknown regions while preserving the known latent structure.
\paragraph{Implementation Details} For all rectified flow generators, we step the sampling time step $T=50$. The classifier-free guidance scales are 7.5 and 5 for the spare structure generator and structured latent generator. Resampling time $U$ during the masked rectified flow is set to 2. The whole pipeline can be loaded with an NVIDIA RTX A5000 GPU with 24G VRAM.
\section{Experiment Settings}
\label{appendix:gpt}
\paragraph{Test Set Generation}
To evaluate models' performance under stylistically diverse conditions, we curated a human-verified synthesized image test set with 100 top-down views. Given an example image, we ask ChatGPT-o3~\cite{openai2025o3} to generate image prompts for top-down scene views with this requirments: \textit{1280 × 720 resolution, quasi-orthographic three-quarter (“isometric”) camera, one hero landmark at the image centre, 10–20 surrounding buildings, daylight illumination, and “no far-away object”}. Then, we used GPT-4o~\cite{hurst2024gpt} to generate scene images according to the corresponding prompts.
\paragraph{Human Evaluation}
Given a reference image and observations of two generated scenes, we ask the human annotator to answer these three questions:
\begin{itemize}
\item Which scene, A or B, has geometry that is more detailed, precise, and closer to the reference image?
\item Which scene, A or B, demonstrates a spatial layout and arrangement of objects that is more coherent and closely aligned with the layout in the reference image?
\item Which scene, A or B, exhibits textures that are significantly more coherent and consistent with the reference image?
\end{itemize}
\paragraph{GPT-4o-based Evaluation} For GPT-based automatic evaluation, we ask the same questions as the human evaluation and prompt the model to directly return the answer with top-5 token log probability. Then, we extract the token probability of `A' and `B` (0 if not included) as the answer weights.
\section{Limitation}
\label{appendix:limitation}
Although \modelname delivers strong scene‐level results, several challenges remain. The pretrained 3D generator we adopt is trained on single‐object imagery; even after region decomposition, the underlying distribution mismatch can lead to patch-level hallucinations—for example, duplicated façades or unrealistic roof shapes. Future work could mitigate this via scene-level fine-tuning or domain adaptation.
Our coarse spatial prior contains many holes where occlusions obscure geometry. Regions dominated by such voids sometimes inherit empty or oversmoothed surfaces from the generator. Integrating uncertainty-aware depth completion, multi-view cues, or semantic priors may yield denser scaffolds and more reliable inpainting in future work.
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{cai2023diffdreamer}
Cai, S., Chan, E.R., Peng, S., Shahbazi, M., Obukhov, A., Van~Gool, L., Wetzstein, G.: Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2139--2150 (2023)
\bibitem{ccelen2024design}
{\c{C}}elen, A., Han, G., Schindler, K., Van~Gool, L., Armeni, I., Obukhov, A., Wang, X.: I-design: Personalized llm interior designer. arXiv preprint arXiv:2404.02838 (2024)
\bibitem{Chai2023PersistentNA}
Chai, L., Tucker, R., Li, Z., Isola, P., Snavely, N.: Persistent nature: A generative model of unbounded 3d worlds. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 20863--20874 (2023), \url{https://api.semanticscholar.org/CorpusID:257687856}
\bibitem{chang2024flux}
Chang, L.W., Bao, W., Hou, Q., Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, Z., Yao, C., Jiang, Z., et~al.: Flux: fast software-based communication overlap on gpus through kernel fusion. arXiv preprint arXiv:2406.06858 (2024)
\bibitem{chung2023luciddreamer}
Chung, J., Lee, S., Nam, H., Lee, J., Lee, K.M.: Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 (2023)
\bibitem{Deitke2023ObjaverseXLAU}
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V.S., Gadre, S.Y., VanderBilt, E., Kembhavi, A., Vondrick, C., Gkioxari, G., Ehsani, K., Schmidt, L., Farhadi, A.: Objaverse-xl: A universe of 10m+ 3d objects. ArXiv \textbf{abs/2307.05663} (2023), \url{https://api.semanticscholar.org/CorpusID:259836993}
\bibitem{engstler2025syncity}
Engstler, P., Shtedritski, A., Laina, I., Rupprecht, C., Vedaldi, A.: Syncity: Training-free generation of 3d worlds. arXiv preprint arXiv:2503.16420 (2025)
\bibitem{engstler2024invisible}
Engstler, P., Vedaldi, A., Laina, I., Rupprecht, C.: Invisible stitch: Generating smooth 3d scenes with depth inpainting. arXiv preprint arXiv:2404.19758 (2024)
\bibitem{feng2023layoutgpt}
Feng, W., Zhu, W., Fu, T.j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems \textbf{36}, 18225--18250 (2023)
\bibitem{fridman2023scenescape}
Fridman, R., Abecasis, A., Kasten, Y., Dekel, T.: Scenescape: Text-driven consistent scene generation. Advances in Neural Information Processing Systems \textbf{36}, 39897--39914 (2023)
\bibitem{gao2024cat3d}
Gao, R., Holynski, A., Henzler, P., Brussee, A., Martin-Brualla, R., Srinivasan, P., Barron, J.T., Poole, B.: Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314 (2024)
\bibitem{Gupta20233DGenTL}
Gupta, A., Gupta, A.: 3dgen: Triplane latent diffusion for textured mesh generation. ArXiv \textbf{abs/2303.05371} (2023), \url{https://api.semanticscholar.org/CorpusID:257427345}
\bibitem{ho2020denoising}
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems \textbf{33}, 6840--6851 (2020)
\bibitem{hollein2023text2room}
H{\"o}llein, L., Cao, A., Owens, A., Johnson, J., Nie{\ss}ner, M.: Text2room: Extracting textured 3d meshes from 2d text-to-image models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7909--7920 (2023)
\bibitem{hu2024scenecraft}
Hu, Z., Iscen, A., Jain, A., Kipf, T., Yue, Y., Ross, D.A., Schmid, C., Fathi, A.: Scenecraft: An llm agent for synthesizing 3d scenes as blender code. In: Forty-first International Conference on Machine Learning (2024)
\bibitem{huang2024mvadapter}
Huang, Z., Guo, Y., Wang, H., Yi, R., Ma, L., Cao, Y.P., Sheng, L.: Mv-adapter: Multi-view consistent image generation made easy. arXiv preprint arXiv:2412.03632 (2024)
\bibitem{hurst2024gpt}
Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et~al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)
\bibitem{kerbl20233d}
Kerbl, B., Kopanas, G., Leimk{\"u}hler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. \textbf{42}(4), 139--1 (2023)
\bibitem{koh2023simple}
Koh, J.Y., Agrawal, H., Batra, D., Tucker, R., Waters, A., Lee, H., Yang, Y., Baldridge, J., Anderson, P.: Simple and effective synthesis of indoor 3d scenes. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol.~37, pp. 1169--1178 (2023)
\bibitem{koh2021pathdreamer}
Koh, J.Y., Lee, H., Yang, Y., Baldridge, J., Anderson, P.: Pathdreamer: A world model for indoor navigation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14738--14748 (2021)
\bibitem{Lee2024SemCitySS}
Lee, J., Lee, S., Jo, C., Im, W., Seon, J., Yoon, S.E.: Semcity: Semantic scene generation with triplane diffusion. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 28337--28347 (2024), \url{https://api.semanticscholar.org/CorpusID:268363839}
\bibitem{Li2024CraftsManHM}
Li, W., Liu, J., Chen, R., Liang, Y., Chen, X., Tan, P., Long, X.: Craftsman: High-fidelity mesh generation with 3d native generation and interactive geometry refiner. ArXiv \textbf{abs/2405.14979} (2024), \url{https://api.semanticscholar.org/CorpusID:270045086}
\bibitem{li2024scenedreamer360}
Li, W., Cai, F., Mi, Y., Yang, Z., Zuo, W., Wang, X., Fan, X.: Scenedreamer360: Text-driven 3d-consistent scene generation with panoramic gaussian splatting. arXiv preprint arXiv:2408.13711 (2024)
\bibitem{li2025triposg}
Li, Y., Zou, Z.X., Liu, Z., Wang, D., Liang, Y., Yu, Z., Liu, X., Guo, Y.C., Liang, D., Ouyang, W., et~al.: Triposg: High-fidelity 3d shape synthesis using large-scale rectified flow models. arXiv preprint arXiv:2502.06608 (2025)
\bibitem{li2022infinitenature}
Li, Z., Wang, Q., Snavely, N., Kanazawa, A.: Infinitenature-zero: Learning perpetual view generation of natural scenes from single images. In: European Conference on Computer Vision. pp. 515--534. Springer (2022)
\bibitem{liang2024luciddreamer}
Liang, Y., Yang, X., Lin, J., Li, H., Xu, X., Chen, Y.: Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6517--6526 (2024)
\bibitem{Lin2022Magic3DHT}
Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 300--309 (2022), \url{https://api.semanticscholar.org/CorpusID:253708074}
\bibitem{lin2024instructscene}
Lin, C., Mu, Y.: Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior. arXiv preprint arXiv:2402.04717 (2024)
\bibitem{liu2024panofree}
Liu, A., Li, Z., Chen, Z., Li, N., Xu, Y., Plummer, B.A.: Panofree: Tuning-free holistic multi-view image generation with cross-view self-guidance. In: European Conference on Computer Vision. pp. 146--164. Springer (2024)
\bibitem{Liu2023Zero1to3ZO}
Liu, R., Wu, R., Hoorick, B.V., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 9264--9275 (2023), \url{https://api.semanticscholar.org/CorpusID:257631738}
\bibitem{Liu2023PyramidDF}
Liu, Y., Li, X., Li, X., Qi, L., Li, C., Yang, M.H.: Pyramid diffusion for fine 3d large scene generation. ArXiv \textbf{abs/2311.12085} (2023), \url{https://api.semanticscholar.org/CorpusID:265308971}
\bibitem{lugmayr2022repaint}
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van~Gool, L.: Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11461--11471 (2022)
\bibitem{maillard2024debara}
Maillard, L., Sereyjol-Garros, N., Durand, T., Ovsjanikov, M.: Debara: Denoising-based 3d room arrangement generation. Advances in Neural Information Processing Systems \textbf{37}, 109202--109232 (2024)
\bibitem{Meng2024LT3SDLT}
Meng, Q., Li, L., Nie{\ss}ner, M., Dai, A.: Lt3sd: Latent trees for 3d scene diffusion. ArXiv \textbf{abs/2409.08215} (2024), \url{https://api.semanticscholar.org/CorpusID:272600456}
\bibitem{mildenhall2021nerf}
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM \textbf{65}(1), 99--106 (2021)
\bibitem{openai2025o3}
OpenAI: Introducing openai o3 and o4-mini. \url{https://openai.com/index/introducing-o3-and-o4-mini/} (2025)
\bibitem{oquab2023dinov2}
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et~al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
\bibitem{Poole2022DreamFusionTU}
Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. ArXiv \textbf{abs/2209.14988} (2022), \url{https://api.semanticscholar.org/CorpusID:252596091}
\bibitem{ramesh2021zero}
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International conference on machine learning. pp. 8821--8831. Pmlr (2021)
\bibitem{ravi2024sam}
Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., R{\"a}dle, R., Rolland, C., Gustafson, L., et~al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)
\bibitem{Ren2023XCubeL3}
Ren, X., Huang, J., Zeng, X., Museth, K., Fidler, S., Williams, F.: Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4209--4219 (2023), \url{https://api.semanticscholar.org/CorpusID:273025441}
\bibitem{rombach2022high}
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684--10695 (2022)
\bibitem{rusinkiewicz2001efficient}
Rusinkiewicz, S., Levoy, M.: Efficient variants of the icp algorithm. In: Proceedings third international conference on 3-D digital imaging and modeling. pp. 145--152. IEEE (2001)
\bibitem{schult2024controlroom3d}
Schult, J., Tsai, S., H{\"o}llein, L., Wu, B., Wang, J., Ma, C.Y., Li, K., Wang, X., Wimbauer, F., He, Z., et~al.: Controlroom3d: Room generation using semantic proxy rooms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6201--6210 (2024)
\bibitem{shriram2024realmdreamer}
Shriram, J., Trevithick, A., Liu, L., Ramamoorthi, R.: Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199 (2024)
\bibitem{stan2023ldm3d}
Stan, G.B.M., Wofk, D., Fox, S., Redden, A., Saxton, W., Yu, J., Aflalo, E., Tseng, S.Y., Nonato, F., Muller, M., et~al.: Ldm3d: Latent diffusion model for 3d. arXiv preprint arXiv:2305.10853 (2023)
\bibitem{sun20233d}
Sun, C., Han, J., Deng, W., Wang, X., Qin, Z., Gould, S.: 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint arXiv:2310.12945 (2023)
\bibitem{tang2024diffuscene}
Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., Nie{\ss}ner, M.: Diffuscene: Denoising diffusion models for generative indoor scene synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 20507--20518 (2024)
\bibitem{Tang2023DreamGaussianGG}
Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ArXiv \textbf{abs/2309.16653} (2023), \url{https://api.semanticscholar.org/CorpusID:263131552}
\bibitem{Tang2023MakeIt3DH3}
Tang, J., Wang, T., Zhang, B., Zhang, T., Yi, R., Ma, L., Chen, D.: Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 22762--22772 (2023), \url{https://api.semanticscholar.org/CorpusID:257757320}
\bibitem{tang2023emergent}
Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems \textbf{36}, 1363--1389 (2023)
\bibitem{hunyuan3d22025tencent}
Team, T.H.: Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation (2025)
\bibitem{vahdat2022lion}
Vahdat, A., Williams, F., Gojcic, Z., Litany, O., Fidler, S., Kreis, K., et~al.: Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems \textbf{35}, 10021--10039 (2022)
\bibitem{vilesov2023cg3d}
Vilesov, A., Chari, P., Kadambi, A.: Cg3d: Compositional generation for text-to-3d via gaussian splatting. arXiv preprint arXiv:2311.17907 (2023)
\bibitem{wang2025vggt}
Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geometry grounded transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2025)
\bibitem{wiles2020synsin}
Wiles, O., Gkioxari, G., Szeliski, R., Johnson, J.: Synsin: End-to-end view synthesis from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7467--7477 (2020)
\bibitem{wu2023panodiffusion}
Wu, T., Zheng, C., Cham, T.J.: Panodiffusion: 360-degree panorama outpainting via diffusion. arXiv preprint arXiv:2307.03177 (2023)
\bibitem{wu2024blockfusion}
Wu, Z., Li, Y., Yan, H., Shang, T., Sun, W., Wang, S., Cui, R., Liu, W., Sato, H., Li, H., et~al.: Blockfusion: Expandable 3d scene generation using latent tri-plane extrapolation. ACM Transactions on Graphics (TOG) \textbf{43}(4), 1--17 (2024)
\bibitem{xiang2024structured}
Xiang, J., Lv, Z., Xu, S., Deng, Y., Wang, R., Zhang, B., Chen, D., Tong, X., Yang, J.: Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01506 (2024)
\bibitem{xiao2024florence}
Xiao, B., Wu, H., Xu, W., Dai, X., Hu, H., Lu, Y., Zeng, M., Liu, C., Yuan, L.: Florence-2: Advancing a unified representation for a variety of vision tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4818--4829 (2024)
\bibitem{Xiong2024OctFusionOD}
Xiong, B., Wei, S.T., Zheng, X.Y., Cao, Y.P., Lian, Z., Wang, P.S.: Octfusion: Octree-based diffusion models for 3d shape generation. ArXiv \textbf{abs/2408.14732} (2024), \url{https://api.semanticscholar.org/CorpusID:271962988}
\bibitem{yang2024scene123}
Yang, Y., Yin, F., Fan, J., Chen, X., Li, W., Yu, G.: Scene123: One prompt to 3d scene generation via video-assisted and consistency-enhanced mae. arXiv preprint arXiv:2408.05477 (2024)
\bibitem{yu2024wonderworld}
Yu, H.X., Duan, H., Herrmann, C., Freeman, W.T., Wu, J.: Wonderworld: Interactive 3d scene generation from a single image. arXiv preprint arXiv:2406.09394 (2024)
\bibitem{yu2024wonderjourney}
Yu, H.X., Duan, H., Hur, J., Sargent, K., Rubinstein, M., Freeman, W.T., Cole, F., Sun, D., Snavely, N., Wu, J., et~al.: Wonderjourney: Going from anywhere to everywhere. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6658--6667 (2024)
\bibitem{zhai2024echoscene}
Zhai, G., {\"O}rnek, E.P., Chen, D.Z., Liao, R., Di, Y., Navab, N., Tombari, F., Busam, B.: Echoscene: Indoor scene generation via information echo over scene graph diffusion. In: European Conference on Computer Vision. pp. 167--184. Springer (2024)
\bibitem{zhang2024text2nerf}
Zhang, J., Li, X., Wan, Z., Wang, C., Liao, J.: Text2nerf: Text-driven 3d scene generation with neural radiance fields. IEEE Transactions on Visualization and Computer Graphics \textbf{30}(12), 7749--7762 (2024)
\bibitem{Zhang2024CLAYAC}
Zhang, L., Wang, Z., Zhang, Q., Qiu, Q., Pang, A., Jiang, H., Yang, W., Xu, L., Yu, J.: Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics (TOG) \textbf{43}, 1 -- 20 (2024), \url{https://api.semanticscholar.org/CorpusID:270619933}
\bibitem{zhang20243d}
Zhang, S., Zhang, Y., Zheng, Q., Ma, R., Hua, W., Bao, H., Xu, W., Zou, C.: 3d-scenedreamer: Text-driven 3d-consistent scene generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10170--10180 (2024)
\bibitem{zhou2024gala3d}
Zhou, X., Ran, X., Xiong, Y., He, J., Lin, Z., Wang, Y., Sun, D., Yang, M.H.: Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.07207 (2024)
\end{thebibliography}