MaGIC: Multi-modality Guided Image Completion

Extracted source text · arXiv 2305.11818

\PassOptionsToPackage{dvipsnames}{xcolor}
\documentclass{article} % For LaTeX2e
\usepackage{magic_arxiv,times}

% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}

\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{xcolor, colortbl}         % colors
\usepackage{amstext}
\usepackage{wrapfig}
\usepackage{graphicx}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{enumitem}
\usepackage{anyfontsize}
\usepackage{yfonts}
% \usepackage{euler}    
\usepackage{bbding}
\usepackage{tabularx}
\usepackage{makecell}
\usepackage{overpic}
\usepackage{txfonts}
\usepackage{subcaption}

% Define a color with lower opacity
\definecolor{Gray}{gray}{0.7} 
\colorlet{lowopacitygray}{Gray!15} 

\def\etal{{\em et al.}}
\def\ie{{\em i.e.}}
\def\eg{{\em e.g.}}

\definecolor{newyellow}{rgb}{1 , 0.7 , 0}
\definecolor{newblue}{rgb}{0. , 0.5 , 0.9}
\definecolor{newpink}{rgb}{0.9 , 0.3 , 0.4}
\definecolor{newpurple}{rgb}{0.8 , 0.4 , 0.9}

\newcommand{\HF}[1]{\textcolor[rgb]{1, 0, 0}{#1}}
\newcommand{\YYS}[1]{\textcolor{newpurple}{#1}}
\newcommand{\WH}[1]{\textcolor{cyan}{#1}}
\newcommand{\CP}[1]{\textcolor[rgb]{0.5882, 0.4471, 0.6549}{#1}}
\newcommand{\CG}[1]{\textcolor[rgb]{0.5059, 0.7059, 0.3961}{#1}}

\def\canny{\textcolor{newblue}{$\spadesuit$}}
\def\depth{\textcolor{newyellow}{$\varheartsuit$}}
\def\seg{\textcolor{newpink}{$\clubsuit$}}
\def\multimodal{\textcolor{newpurple}{$\diamond$}}

\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}
\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}}
\newlength\savewidth\newcommand\shline{\noalign{\global\savewidth\arrayrulewidth
  \global\arrayrulewidth 1pt}\hline\noalign{\global\arrayrulewidth\savewidth}}
\newcommand\hshline{\noalign{\global\savewidth\arrayrulewidth
  \global\arrayrulewidth 0.6pt}\hline\noalign{\global\arrayrulewidth\savewidth}}
\newcommand{\tablestyle}[2]{\setlength{\tabcolsep}{#1}\renewcommand{\arraystretch}{#2}\centering\footnotesize}

\title{MaGIC: Multi-modality Guided Image Completion}


\author{
    Yongsheng Yu$^{1}$\thanks{Yongsheng Yu and Hao Wang contributed equally to this work. $^\dagger$Corresponding author.} \hspace{12pt} Hao Wang$^{3}$\footnotemark[1] \hspace{12pt} Tiejian Luo$^{3}$ \hspace{12pt} Heng Fan$^{4}$ \hspace{12pt} Libo Zhang$^{2}$$^\dagger$\\
    $^{1}$Department of Computer Science, University of Rochester\\
    $^{2}$Institute of Software, Chinese Academy of Sciences\\
    $^{3}$School of Computer Science and Technology, University of Chinese Academy of Sciences\\
    $^{4}$Department of Computer Science and Engineering, University of North Texas\\
    \texttt{yongsheng.yu@rochester.edu; wanghao184@mails.ucas.ac.cn; tjluo@ucas.ac.cn}\\
    \texttt{heng.fan@unt.edu; libo@iscas.ac.cn}
}


\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}


\maketitle


\begin{abstract}

Vanilla image completion approaches exhibit sensitivity to large missing regions, attributed to the limited availability of reference information for plausible generation. To mitigate this, existing methods incorporate the extra cue as a guidance for image completion. Despite improvements, these approaches are often restricted to employing a \emph{single modality} (\eg, \emph{segmentation} or \emph{sketch} maps), which lacks scalability in leveraging multi-modality for more plausible completion. In this paper, we propose a novel, simple yet effective method for \textbf{M}ulti-mod\textbf{a}l \textbf{G}uided \textbf{I}mage \textbf{C}ompletion, dubbed \textbf{\emph{MaGIC}}, which not only supports a wide range of single modality as the guidance (\eg, \emph{text}, \emph{canny edge}, \emph{sketch}, \emph{segmentation}, \emph{depth}, and \emph{pose}), but also adapts to arbitrarily customized combination of these modalities (\ie, \emph{arbitrary multi-modality}) for image completion. For building MaGIC, we first introduce a modality-specific conditional U-Net (MCU-Net) that injects single-modal signal into a U-Net denoiser for single-modal guided image completion. Then, we devise a consistent modality blending (CMB) method to leverage modality signals encoded in multiple learned MCU-Nets through gradient guidance in  latent space. Our CMB is \emph{training-free}, thereby avoids the cumbersome joint re-training of different modalities, which is the secret of MaGIC to achieve exceptional flexibility in accommodating new modalities for completion. Experiments show the superiority of MaGIC over state-of-the-art methods and its generalization to various completion tasks. Our project with code and models is available at {\url{yeates.github.io/MaGIC-Page/}}.
\end{abstract}
\vspace{-4mm}

\begin{figure*}[!h]  
  \centering \includegraphics[width=0.97\textwidth]{fig/teaser.pdf}
  \caption{Illustration of our MaGIC for image completion tasks including \emph{outpainting} (first row) and \emph{real user-input editing} (second row) under \emph{multi-modality} guidance.
  }
  % \vspace{-5mm}%
  \label{fig:teaser}
\end{figure*}

\section{Introduction}
\input{sections/intro}

\section{Related Work}
\input{sections/related}



\section{MaGIC: Multi-modality Guided Image Completion}
\input{sections/method}


\section{Experiments}
\input{sections/exper}

\section{Conclusion and Limitation}

In this paper we propose a novel, simple yet effective method, named MaGIC, for multi-modality image completion. Specifically, we first introduce the MCU-Net that is used to achieve single-modality image completion by injecting the modality signal. Then, we devise a novel CMB algorithm that integrates multi-modality for more plausible image completion. On extensive experiments, we show that MaGIC shows superior performance. Moreover, it is generally applicable to various image completion tasks such as in/out-painting and local editing, and even the image generation task. 

MaGIC is proposed to facilitate image completion with multi-modality. Yet, there exist two limitations. First, the ability  to generate high-frequency details is tied to the backbone completion model, which means even with ample detailed guidance, achieving desired fidelity may not be guaranteed. This can be improved by adopting more powerful backbones if necessary. In addition, our MaGIC is less efficient than current single-step completion models, with inference time increasing in line with guidance modalities. This is a common issue for diffusion models, and we leave it for future research.


\bibliography{magic_arxiv}
\bibliographystyle{magic_arxiv}

\clearpage

\appendix
\begin{center}
{\LARGE \textbf{MaGIC: Multi-modality Guided Image Completion\\ $\;$ \\ ————Appendix————}}
\end{center}

{
  \hypersetup{hidelinks}
  \tableofcontents
  % \parttoc
  \noindent\hrulefill
}

\newpage
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}} 
\input{sections/supp}

\end{document}

%%%%% NEW MATH DEFINITIONS %%%%%

\usepackage{amsmath,amsfonts,bm}

% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}

% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}


% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}

\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}

\def\eps{{\epsilon}}


% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}

% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}

% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}

% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}

% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}

% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}

% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}

% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}

% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}


% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}

% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}

% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}

% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}

% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}

\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution

\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}

\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak



Image completion  \citep{criminisi03object,li22mat,lugmayr22repaint}, involving the concealment of a portion of an image and prompting a model to imaginatively restore it, has long been a subject of extensive research with many applications, such as object removal \citep{Suvorov22lama,criminisi03object}, image compositing \citep{DBLP:conf/eccv/LevinZPW04}, photo restoration \citep{DBLP:conf/cvpr/Wan0CZC0W20}, etc. Typical image completion approaches \citep{li22mat,Suvorov22lama} are prone to struggle with complex or large masking regions due to inadequate reference information. This limitation causes ambiguity to completion model over restoration or elimination and leads to noticeable artifacts in completed images, degrading the quality.

An intuitive solution to overcome the above limitation is to incorporate user-input \citep{Horita22struct,yu19free,zheng22vq} or prediction-based \citep{Nazeri19ec,yu22unbiased,guo21ctsdg,dong22zits} guidance, \eg, text \citep{avrahami22blend,xie22smartbrush,nichol22glide,wang22imagen}, edge \citep{Nazeri19ec,guo21ctsdg,yu22unbiased}, or segmentation \citep{yu22unbiased,liao20sge,zheng22struct}, into image completion. 
However, these approaches are limited to performing image completion under only single-modality guidance, which is \emph{inflexible} in employing the multi-modality, especially more than two modalities simultaneously, for plausible generation and leads to limited application scenarios. 

Recently, denoising diffusion probabilistic model \citep{ho20ddpm} has been widely employed and demonstrated superior performances in text-to-image synthesis  \citep{Rombach22ldm, DBLP:conf/cvpr/GuCBWZCYG22, DBLP:journals/corr/abs-2204-06125} and text-driven image manipulation fields \citep{DBLP:conf/cvpr/KimKY22a, avarahami22blended, DBLP:journals/corr/abs-2210-09276}. In addition to \emph{text}, many approaches \citep{bansal23universal,yu23freedom, chen23layout, avarahami22blended} have explored the integration of extra guidance modality, such as \emph{segmentation}, \emph{sketch}, \emph{pose}, and even \emph{position} of generated object, into diffusion models in a \emph{training-free} way. These methods involve designing energy loss associated with the input guidance and guiding its gradient on the latent codes during inference, yet they tend to fail in maintaining fine-grained structural information, resulting in 
insufficient control over the generated results. Meanwhile, several \emph{training-required} approaches \citep{mou23t2i,zhang23controlnet} have further enhanced the control of input modality over diffusion models by introducing an auxiliary conditional network to encode modality and directly add the encoded features to the intermediate features of frozen diffusion models. These methods bring in fresh insights and pave the way for incorporating guidance signals into image completion. Nevertheless, simply transferring these ideas to multi-modality image completion is \emph{not trivial}, as the introduction of each new modality necessitates the joint training of all auxiliary conditional networks. How to effectively integrate multi-modality guidance for image completion in a \emph{scalable} and \emph{flexible} manner  remains an open problem. 

In this paper, we propose \textbf{\emph{MaGIC}}, 
a \emph{novel}, \emph{simple} yet \emph{effective} framework for \textbf{M}ulti-mod\textbf{a}lity \textbf{G}uided  \textbf{I}mage \textbf{C}ompletion, especially when there are more than two modalities at the same time. MaGIC is designed to be \emph{scalable} and \emph{flexible}, allowing it to merge various modalities, including but not limited to \emph{text}, \emph{canny edge}, \emph{sketch}, \emph{segmentation}, \emph{depth}, and \emph{pose}, in an arbitrary combination as guidance for image completion (see Fig.~\ref{fig:teaser} and Fig.~\ref{fig:mm_edit}). To build MaGIC, there are two core ingredients, including a \emph{modality-specific conditional U-Net} (MCU-Net) and a \emph{consistent modality blending} (CMB) method, performed in two stages. 

Specifically, the proposed MCU-Net, composed of a standard U-Net denoiser from the pre-trained stable diffusion \citep{Rombach22ldm} and a simple encoding network, which injects a single modality guidance signal
into the U-Net denoiser to attain single-modal guided completion. The MCU-Net will be individually finetuned under each single modality, in the first stage. 
Then, to achieve multi-modality guidance, the CMB algorithm is proposed in the second stage to flexibly aggregate guidance signals from any combination of previously learned MCU-Nets. The CMB leverages guidance loss to gradually narrow the distances between the intermediate features from the original pretrained U-Net denoiser and multiple MCU-Nets during denoising sample stage, which ensures that the former features do not deviate too much from the original feature distribution during multi-modality guidance. 
Compared with the naive approach of achieving multi-modality guided completion by jointly re-training a unified model, our CMB is \emph{training-free} and allows for the flexible addition or removal of guidance modalities, avoiding cumbersome re-training and preserving the feature distribution of the original U-Net denoiser.

To verify the proposed MaGIC, we conduct extensive experiments on various tasks including image inpainting, outpainting, and real user-input editing, using the COCO \citep{lin14coco}, Places2 \citep{places}, and in-the-wild data. 
Our results demonstrate the superiority of MaGIC over image completion and controllable generation baselines in terms of image quality. 
In addition, we find that, surprisingly, the CMB of our MaGIC is also well applicable for multi-modality guided image generation, showing its generality and potential for generative tasks. Fig.~\ref{fig:framework} illustrates the architecture of our approach.

\begin{figure}[t]  
  \centering \includegraphics[width=0.97\textwidth]{fig/multi_modal_guidance_edit.pdf}
  % \vspace{-3mm}
  \caption{Illustration of our MaGIC for real user-input editing task using various combination of \emph{multi-modality} as guidance.
  }
  \label{fig:mm_edit}
\end{figure}

In summary, our \textbf{contributions} are four-fold: {\bf (i)} we propose a novel approach of MaGIC for flexible and scalable multi-modality guided image completion. To the best of our knowledge, MaGIC is the first to widely support arbitrary multi-modality guided image completion; {\bf (ii)} we present a simple yet effective MCU-Net to effectively and adaptively inject a modality as guidance for image completion; {\bf (iii)} we introduce a novel CMB algorithm that combines arbitrary multiple modalities for image completion without the need for additional training and {\bf (iv)} using MaGIC, we achieve performance superior to that of other state-of-the-art approaches.


% \begin{wrapfigure}[16]{r}{0.47\textwidth}
%     \centering
%     \vspace{-6mm}%
%     % \hspace{2mm}%
%     \includegraphics[width=0.47\textwidth]{fig/illustrate.pdf} 
%     \vspace{-4mm}%
%     \caption{Comparison between previous single-modality (image (a)) and the proposed MaGIC (image (b)) for image completion. Our MaGIC exhibits versatility in multi-modality guidance, enabling the seamless integration of new modalities without requiring joint re-training.}
%     \label{fig:magic_illustrate}
% \end{wrapfigure}
\textbf{Auxiliary-based image completion.} The auxiliary-based image completion methods aim to enhance the structure and texture of completed images by incorporating predicted or human-provided prior information. Early approaches primarily focus on using a single modality (\eg, edge \citep{Nazeri19ec,guo21ctsdg,dong22zits,zheng22vq} or segmentation \citep{zheng22struct,liao20sge}) as the auxiliary guidance for image completion. Recently, inspired by the superior-performing diffusion models \citep{ho20ddpm,Dhariwal21beat,Rombach22ldm}, text-based auxiliary solutions have been proposed for image completion \citep{wang22imagen,avrahami22blend,nichol22glide,avarahami22blended}, providing more user-friendly image editing applications. 

However, prompt text alone is not sufficient.
Due to the above methods are constrained by the training requirements of auxiliary guidance, making them difficult to flexibly add more types of modalities as guidance for completion. 
Our MaGIC can incorporate random combination of multiple  modalities for more plausible completion result (see Fig.~\ref{fig:teaser} again). It is versatile, requiring only the optimization of single-modality conditional networks, and allows for plug-and-play integration into the conditional image completion process without the need for additional cumbersome joint re-training.

\textbf{Controllable image generation with diffusion models}. Diffusion models \citep{ho20ddpm,Dhariwal21beat,Rombach22ldm,song23consistency} have drawn extensive attention in image generation owing to their remarkable results and stable training. These methods can be broadly categorized into \emph{train-required} and \emph{train-free} approaches. The former achieve powerful generation control by training on large-scale data or fine-tuning a conditional control sub-network on pre-trained diffusion models (\eg, \citep{Rombach22ldm}). Recent research \citep{zhang23controlnet,mou23t2i} has introduced various modalities (\eg, keypose point maps, sketch maps, etc) for generation. However, it \emph{fails} to simultaneously use multi-modality as guidance. Differently, train-free solutions \citep{yu23freedom,chen23layout,bansal23universal,jeong23hsapce} leverage the multi-step nature of diffusion models, explicitly introducing guidance signals during the iterative denoising process and achieving style \citep{jeong23hsapce}, layout \citep{chen23layout,bansal23universal}, face identity \citep{bansal23universal,yu23freedom}, segmentation map \citep{bansal23universal,yu23freedom} guidance without fine-tuning. Yet, they struggle to leverage fine-grained structural guidance (\eg, canny edge) as conditions, potentially resulting in degraded guidance \citep{yu23freedom}.

Our MaGIC is inspired by the above image generation approaches, but different in two aspects. First, MaGIC achieves multi-modality guidance without joint re-training while improving the effectiveness of fine-grained structure guidance. In addition, MaGIC goes beyond controllable generation and can be applied to guided completion and real-world editing tasks.






Masked images $x_m=x\odot m$ are obtained by corrupting images $x$ with binary masks $m \in \{0,1\}^{H\times W\times 1}$, where $x \in \mathbb{R}^{H\times W\times 3}$ are original RGB images with width $W$ and height $H$.
Given a known region $x_{\overline{m}}=x\odot (1-m)$, the goal of image completion is to learn a function $p(x_m|x_{\overline{m}})$ that completes the missing mask area with visually realistic and structurally coherent content.
To mitigate the inherent ambiguity of completion model, the direction of restoration or elimination is controlled through the auxiliary guidance $\mathcal{C}$.
In the following sections, we start by outlining necessary diffusion steps in~\ref{sec:preliminaries}) for formulating our method, then elaborate on MaGIC, addressing auxiliary guidance via our proposed MCU-Net in~\ref{sec:MCU} and multi-modality integration by our CMB algorithm in~\ref{sec:CMB}. 

\subsection{Preliminaries}
\label{sec:preliminaries}

\textbf{Diffusion models}. Denoising diffusion probabilistic models (DDPMs) \citep{ho20ddpm} are generative models that learn the true distribution $p(x_T)$ by iteratively denoising a randomly sampled noise image $x_T$. In each denoising step, a U-Net model is trained to predict the noise $\epsilon$ based on the objective function,  
\begin{equation}\label{eqn:obj}
    \Phi(x_t, t, \theta) = \text{min} (\mathbb{E}_{x_0,t,\epsilon \sim \mathcal{N}(0,I)} \|\epsilon - \epsilon_\theta^t(x_t) \|_2^2),
\end{equation} 
where $x_t = \sqrt{\alpha_t} x_0 + \sqrt{1 - \alpha_t} \epsilon$ represents the intermediate noised image obtained after applying noise $t$ times to the clean image $x_0$, and $\alpha_t=\prod_{s=1}^{t} (1-\beta_s)$ is a series of fixed hyperparameters based on the variance schedule $\beta_s, s \in [1, T]$. 
The model can be further generalized to conditional generation \citep{Dhariwal21beat,ho22free}, with predicted noise becoming $\epsilon_\theta(x_t, t, \mathcal{C})$.%, where $\mathcal{C}$ denotes the auxiliary guidance. 

\textbf{Stable diffusion}. We consider stable diffusion (SD) inpainting model \citep{Rombach22ldm} as the main backbone in the subsequent method sections. 
Instead of beginning with isotropic Gaussian noise samples in pixel space, the SD model first maps clean images to their corresponding latent space $\mathcal{Z}$ through $E(\cdot)$. Here, $E(\cdot)$ is an autoencoder with a left inverse $D$, ensuring $x=D \circ E(x)$.
Owing to the lower inference overhead of U-Net in the latent space, SD has emerged as an important class of recent image generators based on diffusion \citep{Rombach22ldm,saharia22,zhang23controlnet,avrahami22blend}. 
Specifically, the initial latent codes of iterative denoising process employ random $z_T \sim \mathcal{Z} \in \mathbb{R}^{\frac{H}{s}\times \frac{W}{s}\times 3}$, where $s$ signifies $s$-fold reduction in spatial dimensions. 
The mask and encoding masked image serve as conditions for the U-Net, modifying the objective function in Eq.~\ref{eqn:obj} to
\begin{equation}
    \Phi(z_t, t, m_{\downarrow}, x_{m\downarrow}, \theta) = \text{min} (\mathbb{E}_{z_0,t,\epsilon \sim \mathcal{N}(0,I)} \|\epsilon - \epsilon_\theta^t(z_t, m_\downarrow, x_{m\downarrow}) \|_2^2),
\end{equation}
where $m_\downarrow \in \mathbb{R}^{\frac{H}{s}\times \frac{W}{s}\times 1}$ denotes the $s$-fold nearest-neighbor downsampling of the input mask $m$, and $x_{m\downarrow}=E(x_m)$ indicates embedded masked image in latent space. 
Denoising diffusion implicit model (DDIM) \citep{song21ddim} defines the each step of denoising as a non-Markovian process while retaining the same training objective as DDPM. Accordingly, the sampling process is formulated as,
\begin{equation}\label{eqn:ddim}
    z_{t-1} = \sqrt{\alpha_{t-1}} ( \frac{z_t - \sqrt{1 - \alpha_t} \epsilon_\theta^t (z_t, m_\downarrow, x_{m\downarrow})}{\sqrt{\alpha_t}}) + \sqrt{1 - \alpha_{t-1} - \sigma^2_t} \cdot \epsilon_\theta^t (z_t, m_\downarrow, x_{m\downarrow}) + \sigma_t \epsilon_t,
\end{equation}
where the noise $\epsilon_t$ follows the standard normal distribution $\mathcal{N}(0, \mathbf{I})$ and is independent of $x_t$, and $\sigma_t = \eta \sqrt{(1-\alpha_{t-1}) / (1 - \alpha_t)} \sqrt{1 - \alpha_t / \alpha_{t-1}}$. 
By gradually denoising over $T$ timesteps, the content of missing region is hallucinated in the latent space, producing a conditional sample $z_0 \sim p(z_T| m_\downarrow, x_{m\downarrow})$. $z_0$ is then transformed into the pixel space as $\hat{x}=D(z_0)$ via the left-inverse decoder network $D$ corresponding to the autoencoder $E(\cdot)$, finally resulting in the completion outcome $\hat{x} \in \mathbb{R}^{H\times W\times 3}$.

\begin{figure}[t]
\centering
\includegraphics[width=0.96\textwidth]{fig/framework.pdf} 
\caption{Illustration of our method. 
We initiate the inference process with a randomly initialized latent $z_T$. This latent is denoised $T$ times, with the concatenation of the masked image and mask acting as conditioning for both \emph{MCU-Net} and frozen U-Net denoiser. Through \emph{CMB}, we fuse diverse modality guidance signals, aiding the frozen original U-Net $\theta_*$ to iteratively produce the desired content. The content is finally transformed into pixel space via a decoder network, resulting in the completed RGB output.
}
% \vspace{-5mm}
\label{fig:framework}
\end{figure}
\subsection{MCU-Net: Modality-specific Conditional U-Net} \label{sec:MCU}


\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-4mm}
  \centering
  \includegraphics[width=0.95\linewidth]{fig/mcunet.pdf}
  \caption{Illustration of MCU-Net.}
  \label{fig:mcunet}
\vspace{-6mm}
\end{wrapfigure}
The first stage in MaGIC is to learn image completion under single-modality guidance. For this purpose, we propose a simple yet effective modality-specific conditional U-Net (MCU-Net). Particularly, for the auxiliary guidance $c^i \in \mathcal{C}$ ($\mathcal{C}=\{c^i\}_{i=1}^{N}$ denotes the set of $N$ auxiliary guidance), MCU-Net consists of a standard U-Net denoiser $\theta_{c^i}$ \citep{Rombach22ldm} and an encoding network $\tau_{c^i}$. For simplicity, we will omit $i$ in the following sections.

The encoding network $\tau_{c}$ is employed to extract multi-scale guidance signals, represented as $F_{c}^{l}$, where $l\in \{0,\cdots,L\}$ and $L$ denotes the number of times the feature map scale is reduced within the U-Net denoiser. Afterwards, $F_{c}^{l}$ is injected to the latent in MCU-Net to obtain modality-guided feature. In specific, we denote the latent in MCU-Net as $w_{t, c}$ ($c \in \mathcal{C}$) to distinguish it from the original diffusion model's $z_t$. As illustrated in Fig.~\ref{fig:mcunet}, to inject guidance signals into the latent $w_{t, c}$, we add $F_{c}^l$ to intermediate feature maps $F_{enc}^l$ of the encoder of MCU-Net, resulting in guided feature map $\hat{F}_{c}^l=F_{enc}^l + F_{c}^l, l \in [0,L]$. And we incorporate the text modality in a manner consistent with SD, which integrates its information into intermediate features via a cross-attention mechanism.

To utilize generative capability of pre-trained SD, we freeze the original U-Net denoiser when training MCU-Net, allowing the unlocked encoding network $\tau_{c}$ to learn guidance signal extraction and fit the pre-trained denoiser.



\subsection{CMB: Consistent Modality Blending}\label{sec:CMB}
Despite achieving image completion under single-modality with MCU-Net, it is \emph{not trivial} to integrate multiple MCU-Nets for multi-modality image completion. A naive way is to jointly re-train these learned MCU-Nets, which is cumbersome and inflexible for multi-modality image completion. To deal with this, we propose the novel consistent modality blending (CMB), a \emph{training-free} algorithm to integrate guidance signals from different auxiliary modalities \emph{without} requiring additional joint re-training. A great benefit of CMB is that, the multi-modality guidance latent code in MCU-Net remains aligned with the internal knowledge of SD model, without affecting its original ability. As shown in Fig.~\ref{fig:framework}, the guidance signals from arbitrary combination of independent single-modality models (\ie, MCU-Nets) in gradient aspect gradually control the image completion process with input modalities. 



Specifically, given a series of MCU-Nets trained independently on multiple modalities $\mathcal{C}$, we can extract the guidance signals $F_{c}$. A simple way for integrating different modalities is to directly update intermediate feature maps $F_{enc}$ by adding accumulated guidance signals as $\hat{F}_{\mathcal{C}} \leftarrow F_{enc} + \sum_{c \in \mathcal{C}} F_{c}$. However, we argue that this simple manner we called \emph{feature-level addition} is impractical, as the denoiser is trained solely on the distribution of $\hat{F}_{c} = F_{enc} + F_{c}$. Drawing inspiration from recent advancements in classifier-guidance diffusion \citep{Dhariwal21beat}, we introduce a converse amplification strategy. This technique enables the intermediate feature maps $F_*$ of a original U-Net to more closely approximate each guided feature map $\hat{F}_c$.



\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-8mm}
    \begin{minipage}{0.5\textwidth}
      \begin{algorithm}[H]
    \small
    \caption{Usage of CMB in MaGIC}\label{algo:cmd}
    \noindent\textbf{Require:} Given the input masked image $x_m$, mask $m$, a series of MCU-Net parameters $\theta_{c}$, the number of times of converse amplification $P$, and the number of iteration steps of back-propagation $Q$.

    \begin{algorithmic}[1]
    \State $m_\downarrow = \text{\tt downsample}(m)$
    \State $x_{m\downarrow} = E(x_m)$
    \State $z_T \sim \mathcal{N}(0, \mathbf{I})$
    \State $w_{T, c} \sim \mathcal{N}(0, \mathbf{I}) , \forall c\in \mathcal{C}$
    \For{$t = T, \cdots, 1$}
        \If{$t \leq T - P$}
            \State $\epsilon_{\theta^*}, F_* \leftarrow \theta_* (z_t, t, m_{\downarrow}, x_{m\downarrow})$
            \State $z_{t-1} = \text{\tt sampler} (z_t, \epsilon_{\theta^*})$ ~\hfill(Eq. \ref{eqn:ddim})
            \State \text{\tt continue}
        \EndIf
        \For{$1,\cdots,Q$}
            \State $\epsilon_{\theta}, \hat{F}_{\mathcal{C}} \leftarrow \theta_{\mathcal{C}}(w_{t,\mathcal{C}}, t, m_{\downarrow}, x_{m\downarrow})$
            \State $w_{t-1, \mathcal{C}} = \text{\tt sampler} (w_{t, \mathcal{C}}, \epsilon_{\theta} )$ ~\hfill(Eq. \ref{eqn:ddim})
            \State $\epsilon_{\theta^*}, F_* \leftarrow \theta_* (z_t, t, m_{\downarrow}, x_{m\downarrow})$
            \State $z_{t-1}^\prime = \text{\tt sampler} (z_t, \epsilon_{\theta^*})$ ~\hfill(Eq. \ref{eqn:ddim})
            \State $z_{t-1} = z_{t-1}^\prime - \sigma_t \gamma \nabla_{z_t} \ell(\hat{F}_{\mathcal{C}}, F_*)$ ~\hfill(Eq. \ref{eqn:backward})
        \EndFor
    \EndFor
    \State \textbf{return} $D(z_0)$
    \end{algorithmic}
    \end{algorithm}
    \end{minipage}
    \vspace{-8mm}
\end{wrapfigure}
\textbf{Converse Amplification}. We use $F_{*}$ to denote the intermediate features from the original U-Net $\theta_*$ which is not equipped with an guidance encoding network, while $\hat{F}_{c}$ to denote guided features from MCU-Net $\theta_{c}$ of modality $c$. Notably, U-Net $\theta_*$ and MCU-Net $\theta_c$ undergo a parallel denoising process. At each step $t$, every latent is denoised using the DDIM sampler \citep{song21ddim}. In the original U-Net $\theta_*$, we denote the denoised latent as the intermediate latent $z_{t-1}^\prime$.



We bias $F_{*}$ towards $\hat{F}_{c}$ by calculating their Euclidean distance in each scale $l$:
\begin{equation}\label{eqn:loss}
    % D(F_{enc}, \hat{F}_{enc_*}) = \sum_{i=1}^4(w_1 \times \| F_{enc}^i - \hat{F}_{enc_1}^i\|_2^2 + w_2 \times \| F_{enc}^i - \hat{F}_{enc_2}^i\|_2^2)
    \ell(\hat{F}_{\mathcal{C}}, F_*) = \frac{1}{L} \sum_{c \in \mathcal{C}}  \delta_{c} \| \hat{F}_{c}^l - F_{*}^l\|_2^2
\end{equation}
where $\delta_{c}$ are scale factors to weight the strength leads to either improved alignment to guidance modality $c$ or greater diversity in the outputs. $N=| \mathcal{C}|$ indicates the modality number of auxiliary guidance set. Then we use distance to adjust latent code of the original SD model. Specifically, at each denoising step, we obtain $\hat{F}_{\mathcal{C}}$ and $F_{*}$ firstly, then the gradient of their distance is calculated through back-propagation to update the denoised latent $z_{t-1}^\prime$:
\begin{equation}\label{eqn:backward}
    z_{t-1} = z_{t-1}^\prime - \sigma_t\gamma\nabla_{z_t}\ell(\hat{F}_{\mathcal{C}}, F_*)
\end{equation}
Owing to CMB, it is \emph{not necessary} to jointly re-train the learned MCU-Nets, making MaGIC flexible in merging arbitrary multi-modality for completion. Alg.~\ref{algo:cmd} shows the procedure of CMB for MaGIC.

In this work, we study three research questions, \textbf{RQ1}, \textbf{RQ2} and \textbf{RQ3}: 

\textbf{RQ1:} \emph{Can our MCU-Net effectively perform image completion guided by various modalities?}\\
\textbf{RQ2:} \emph{Can our MaGIC with CMB seamlessly integrate guidance from multiple modalities to produce credible completion results?}\\
\textbf{RQ3:} \emph{How do different module designs (}\eg\emph{, adjustments in hyperparameters and inference processes) impact the overall effectiveness?}

\begin{figure}[t]  
  \centering \includegraphics[width=.97\textwidth]{fig/single_modal_quali.pdf}
  % \vspace{-4mm}
  \caption{Qualitative comparison for image completion using \emph{single} modality as guidance. $^*$ indicates the use of latent-level blending \citep{avrahami22blend} to preserve pixels in unmasked regions.}
  % \vspace{-2mm}
  \label{fig:single_modal_quali}
\end{figure}

\subsection{Experimental settings}
In our experiments, we select several edge-based image completion methods, including EC \citep{Nazeri19ec}, CTSDG \citep{guo21ctsdg}, ZITS \citep{dong22zits}, and state-of-the-art (SOTA) techniques such as LAMA \citep{Suvorov22lama}, LDM \citep{Rombach22ldm}, and MAT \citep{li22mat}. 
We also include controllable image generation baselines such as ControlNet \citep{zhang23controlnet} and T2I-Adapter \citep{mou23t2i} in our qualitative comparison, as they can be easily adapted to the image completion task with the concept of Blended Diffusion \citep{avarahami22blended, avrahami22blend}.
For fair comparison, we apply the same set of image mask pairs across all tests, and, for comparisons involving auxiliary guidance, we ensure that each method receives identical guidance map instructions. The masks used in testing are designed to uniformly span a masking ratio range from $0$ to $100$\%. The evaluation adopts both image metrics (\ie, FID and P/U-IDS \citep{zhao21comod}) and text-to-image metric (\ie, PickScore \citep{kirstain23pickscore}) which gauges the fidelity of generated content based on learned human preferences. Acknowledging the pluralistic outcomes of our method, we conduct tests on a total of five images to determine mean scores and standard deviations. For all diffusion-based methods, the denoising step $T$ is set to $50$. For further details on the experimental configuration, please see the supplementary material.



\subsection{Image Completion with Single-Modality Guidance using MCU-Net}
To answer \textbf{RQ1}, 
we compare our approach with state-of-the-art (SOTA) inpainting methods \citep{Suvorov22lama,li22mat} and SOTA single modality guidance image generation methods. We employ latent-level blending \citep{avrahami22blend} to preserve pixels in unmasked regions for image generation methods such as ControlNet \citep{zhang23controlnet} and T2I-Adapter \citep{mou23t2i}. As depicted in the Fig.~\ref{fig:single_modal_quali}, our method generates content without noticeable artifacts, maintaining stronger spatial context consistency. Conversely, T2I-Adapter generates a stone house on the road (1st row in Fig.~\ref{fig:single_modal_quali}) and ControlNet puts a dancer on the soccer field (2nd row in Fig.~\ref{fig:single_modal_quali}). 
% Additional illustrations for other modalities can be found in \textbf{supplementary material} due to limited space. 
%Conversely, prevailing state-of-the-art techniques, when lacking comprehensive control, struggle with ambiguity in the completion model, leading to the generation of artifacts. 

Quantitatively, 
the scores of edge-based methods on COCO and Places2 are displayed in Tab.~\ref{tab:single_modal_quantatitive}. Across all metrics, our method demonstrates significant improvements, indicating that our MCU-Net can effectively generate content under the guidance of various single modalities.

\subsection{Image Completion with Multi-Modality Guidance using MaGIC}

We further decompose \textbf{RQ2} into two smaller questions, \textbf{RQ2.1} (\emph{Is CMB effective?}) and \textbf{RQ2.2} (\emph{How does MaGIC perform?})

\textbf{Answering \textbf{RQ2.1}.} CMB aims to integrate different modalities as guidance for image completion in a \emph{training-free} fashion. Compared with CMB, a simple way is to aggregate feature maps $F_{c}$ ($c \in \mathcal{C}$) by addition (\ie, \emph{feature-level addition} or \emph{FLA} for short) to produce $\hat{F}_{\mathcal{C}}$ as $\hat{F}_{\mathcal{C}} \leftarrow F_{enc} + \sum_{c \in \mathcal{C}} F_{c}$. To show the effectiveness of CMB, we compare it with FLA on COCO as in Tab.~\ref{tab:compare_to_addition}. Note that, we experiment FLA with 30 and 50 steps, respectively. To guarantee an equitable assessment across all auxiliary modalities, we opt for a wide-ranging set of modalities. Given that specific modalities (\eg, pose) may not be applicable to all test images (\eg, certain landscape images), we ensure that our test suite incorporates a diverse range of modalities. This includes segmentation map, depth map, Canny edge map, sketch map, and a prompt text. As displayed in Tab.~\ref{tab:compare_to_addition}, the proposed CMB significantly surpasses FLA with naive addition, evidencing the effectiveness of CMB in merging multi-modality for completion. Interestingly, the performance of FLA with $50$ steps is counter-intuitively lower than that with $35$ steps, suggesting that this simple method may overly manipulate the latent code. This indicates that direct addition of different MCU-Net feature maps for multi-modality guidance is \emph{impractical}. By contrast, our CMB efficaciously integrates the signals from multi-modal guidance.



\begin{table}[t]\small
\centering
% \resizebox{\textwidth}{!}{
\begin{tabular}{lccccc} \toprule
      & \multicolumn{2}{c}{COCO}    & \multicolumn{3}{c}{Places2}                     \\ \cmidrule(r){2-3} \cmidrule(r){4-6}
Method      & FID$\downarrow$ & PickScore$\uparrow$ / \% & FID$\downarrow$           & U-IDS$\uparrow$ / \%         & P-IDS$\uparrow$ / \%         \\ \midrule
EC \citep{Nazeri19ec}~\canny{}    & 76.64           & 23.14     & 25.08         & 12.89          & 2.86           \\
CTSDG \citep{guo21ctsdg}~\canny{} & 97.05           & 24.03     & 42.81         & 0              & 0              \\
ZITS \citep{dong22zits}~\canny{}  & 61.27           & 28.09     & 18.96         & 18.75          & 7.20            \\ \midrule
Our MCU-Net$\dagger$  & 47.70$\pm$0.29  & 30.79$\pm$0.10  & 10.74$\pm$0.07  & 23.83$\pm$0.30 & 10.18$\pm$0.48 \\
Our MCU-Net~\depth{}  & \textbf{39.43$\pm$0.26}  & \textbf{37.12$\pm$0.11}     & 9.09$\pm$0.04 & 25.34$\pm$0.29 & 10.64$\pm$0.46 \\
Our MCU-Net~\seg{}  & 41.91$\pm$0.20  & 34.96$\pm$0.17     & 10.27$\pm$0.06 & 24.21$\pm$0.24 & 9.93$\pm$0.38 \\
Our MCU-Net~\canny{}  & 41.15$\pm$0.27  & 34.94$\pm$0.06     & \textbf{8.32$\pm$0.02} & \textbf{26.23$\pm$0.07} & \textbf{10.96$\pm$0.33}  \\\bottomrule
\end{tabular}%}
\vspace{0.6em}
\caption{
Comparison of using single auxiliary modality as guidance for image completion. \canny{}: ground truth edge map as guidance, \depth{}: estimated depth map as guidance, \seg{}: segmentation map as guidance, $\uparrow$: the higher the better, $\downarrow$: the lower the better, $\dagger$: completion without any guidance.}
\label{tab:single_modal_quantatitive}
\end{table}

\input{sections/tables/mmguidance}



\textbf{Answering RQ2.2.} To validate the effectiveness of our MaGIC, we compare it with state-of-the-art image completion methods, including LAMA \citep{Suvorov22lama} and MAT \citep{li22mat}, on COCO. As in Tab.~\ref{tab:multi_modal_quantatitive}, our guidance-free inpainting model denoted as MaGIC$\dagger$ is comparable to SOTA inpainting baselines MAT and LAMA. When employing multi-modality, (segmentation, canny edge, sketch, depth, and text) as guidance, our MaGIC gains significant improvements. In comparison to MaGIC$\dagger$, we obtain gains of $21$\% in FID and $61$\% in PickScore. Notably, the PickScore implies that, from the perspective of learned human preference, our completed images have a $49.57$\% chance of being more faithful to the ground truth image caption than the original images.

\subsection{Ablation Study}

\input{sections/tables/ablation_study}

To answer \textbf{RQ3}, we conduct rich ablations on COCO as follows.

\textbf{Impact of modalities.} To delve into auxiliary modalities, we investigate their individual contributions. We distinguish among five modalities used in our experiments: edge and sketch for fine-grained structural control,  segmentation and depth for coarse-grained spatial-semantic control, and text for content-specific cue. As in Tab.~\ref{tab:modality_analysis}, the guidance from text significantly enhances image quality (FID) and generated content (PickScore). Interestingly, excluding text, the performance of combined modalities appears balanced, suggesting optimal generation quality when modalities provide complementary information. When using all modalities, the performance is the best.

\textbf{Joint multi-modality re-training.} 
Our method allows multi-modality guidance without the need for additional joint training. However, exploring the joint re-training of all modality-specific conditional U-Nets with classifier-free guidance style can help identify the upper bound performance.

Building such a model necessitates a fuser mechanism to blend diverse input modalities. To ensure effectiveness, we integrated CoAdapterFuser \citep{mou23t2i}, aligning with our design goals. Addressing the lack of a paired dataset with extensive labels across various modalities was also essential. We extracted 650,000 images from the Laion dataset and generated four modalities (canny edge map, depth map, sketch map, and semantic map) using open-source tools. During joint re-training, we randomly dropped out each modality at a 0.5 probability. This training process is memory-intensive, necessitating a reduction in batch size to a quarter of single-modality training. The model underwent 180,000 iterations. As evidenced by its lower FID, shown in the first row of Tab.~\ref{tab:modality_analysis}, the unified model achieves higher fidelity than our training-free method. However, it encounters issues such as the need for paired training data, difficulty in adding new modality, and substantial computational requirements for joint training.


\textbf{Guidance in iteration}. The proposed CMB algorithm involves two important hyperparameters, \ie, the number $P$ of denoising steps incorporating CMB and the iteration times $Q$ of gradient descent performed in each CMB operation. We study the impact of different $P$ and $Q$ on the multimodal conditioning completion task as in Tab.~\ref{tab:cmb_ablation}. From Tab.~\ref{tab:cmb_ablation}, we can observe that with $Q$ fixed, the performance is almost consistently improved as increasing the number $P$ of denoising steps (from 10 to 50) equipped for CMB. Interestingly, given the fact in Tab.~\ref{tab:compare_to_addition} that incorporating guidance through simple FLA could impair the performance of completion, the results further  demonstrate the effectiveness of CMB. In addition, we can also observe from Tab.~\ref{tab:cmb_ablation} that, different $Q$ (\eg, 1 to 5 to 10) leads to different performance. We argue that, increasing the iteration times to 5 in a reasonable manner based on 1 should yield better metrics as more guidance information is introduced. Yet, the subsequent decline when further increasing $Q$ to 10 in performance can be attributed to the presence of more noise in hidden space during early stages of the denoising process. For the trade-off between inference time and image completion performance, we set the values of $P$ and $Q$ to 30 and 5, respectively.  

\begin{table*}[t!]\centering
%%%%%%%%%% subfloat a
%
\subfloat[Comparison of CMB with simple FLA.]{\tablestyle{4pt}{1.1}
\def\w{12pt} 
\def\bw{15pt}
\def\wmod{18pt} 
\def\wmethod{45pt} 
 \hspace{-5mm}
\resizebox{!}{45pt}{
\begin{tabular}{L{60pt}C{\wmethod}C{\wmethod}C{60pt}}   
        \shline
         & & \multicolumn{2}{c}{COCO} \\ 
        \cmidrule(r){3-4} Method & \makecell[c]{MMG} & FID $\downarrow$ & PickScore $\uparrow$ / \% \\ 
        \hline
        MaGIC w/ FLA (35 steps) & \Checkmark & 37.78$\pm$0.32 & 44.19$\pm$0.23 \\
        MaGIC w/ FLA (50 steps) & \Checkmark & 41.53$\pm$0.19 & 35.85$\pm$0.08 \\ 
        \hline
        MaGIC$\dagger$ & \XSolidBrush & 47.70$\pm$0.29 & 30.79$\pm$0.10 \\
        MaGIC w/ CMB & \Checkmark & \textbf{37.65$\pm$0.22} & \textbf{49.57$\pm$0.17} \\ 
        \shline
\end{tabular}
}
\label{tab:compare_to_addition}
}
\subfloat[Comparison of MaGIC with SOTA methods.]{\tablestyle{4pt}{1.1}
\def\w{12pt} 
\def\wmod{18pt} 
\def\wmethod{45pt} 
 \hspace{-5mm}
\resizebox{!}{45pt}{
\begin{tabular}{L{\wmethod}C{\wmethod}C{\wmethod}C{60pt}} 
        \shline
         & & \multicolumn{2}{c}{COCO} \\ 
        \cmidrule(r){3-4} Method & \makecell[c]{MMG}  & FID $\downarrow$ & PickScore $\uparrow$ / \% \\ 
        \hline
        CoMod & \XSolidBrush & 68.01 & 25.12 \\
        TFill & \XSolidBrush & 58.55  & 24.63 \\
        FcF & \XSolidBrush & 48.92 & 26.43 \\
        LAMA & \XSolidBrush & 48.63 & 29.06 \\
        MAT & \XSolidBrush & 45.51 & 27.10 \\ \hline
        MaGIC$\dagger$ & \XSolidBrush & 47.70$\pm$0.29 & 30.79$\pm$0.10 \\
        MaGIC & \Checkmark & \textbf{37.65$\pm$0.22} &  \textbf{49.57$\pm$0.17} \\ \shline
\end{tabular}
}
\label{tab:multi_modal_quantatitive}
}
\label{lab:main_exp}
\caption{Comparisons of CMB and FLA and MaGIC with others. MMG: multi-modality guidance.}
% \vspace{-4mm}
\end{table*}
\begin{table*}[t!]\centering
%%%%%%%%%% subfloat a
%
\subfloat[Ablation on different modalities.]{\tablestyle{4pt}{1.1}
\def\w{12pt} 
\def\bw{15pt}
\def\wmod{18pt} 
\def\wmethod{60pt} 
 \hspace{-5mm}
\resizebox{!}{45pt}{
\begin{tabular}{C{\w}C{\w}C{\bw}C{\bw}C{\bw}C{\wmethod}C{\wmethod}}
        \shline
        text & seg & depth & canny & sketch & FID $\downarrow$ & PickScore $\uparrow$ / \% \\ \hline
             % \rowcolor{gray!20}
             \rowcolor{lowopacitygray} \textcolor{Gray}{\Checkmark} & \textcolor{Gray}{\Checkmark} & \textcolor{Gray}{\Checkmark} & \textcolor{Gray}{\Checkmark} & \textcolor{Gray}{\Checkmark} & \textcolor{Gray}{35.42$\pm$0.10} & \textcolor{Gray}{47.81$\pm$0.34} \\ \hline
             & \Checkmark & & & & 41.91$\pm$0.20 & 34.96$\pm$0.17 \\
             & & \Checkmark & & & 39.43$\pm$0.26 & 37.12$\pm$0.11 \\
             & & & \Checkmark & & 41.15$\pm$0.27 & 34.94$\pm$0.06 \\
             & \Checkmark & \Checkmark & & & 41.38$\pm$0.22 & 35.99$\pm$0.13 \\
             & & & \Checkmark & \Checkmark & 42.59$\pm$0.19 & 34.98$\pm$0.16 \\
             \Checkmark & & & & & 39.96$\pm$0.12 & 48.58$\pm$0.25 \\
             \Checkmark & \Checkmark & \Checkmark & \Checkmark & \Checkmark & 37.65$\pm$0.22 & 49.57$\pm$0.17 \\ 
             \shline
             % \bottomrule
             % \multicolumn{3}{c}{} \\ % space holder
             % \multicolumn{3}{c}{} \\ % space holder
             % \multicolumn{3}{c}{} \\ % space holder
             \end{tabular}
}
\label{tab:modality_analysis}
}
\subfloat[Hyperparameter analysis of CMB.]{\tablestyle{4pt}{1.1}
\def\w{12pt} 
\def\wmod{18pt} 
\def\wmethod{60pt} 
 \hspace{-5mm}
\resizebox{!}{45pt}{
\begin{tabular}{C{\w}C{\w}C{\wmethod}C{\wmethod}}
        \shline
        $P$  &  $Q$  & FID$\downarrow$ & PickScore$\uparrow$ / \% \\ 
        \hline
        30 &  1  &  38.50$\pm$0.23  &  48.11$\pm$0.21  \\
        30 &  10  &  38.20$\pm$0.28  &  48.60$\pm$0.12  \\ 
        \shline
        10 &  5  &  40.40$\pm$0.25  &  45.98$\pm$0.12  \\
        20 &  5  &  38.37$\pm$0.34  &  48.72$\pm$0.15  \\
        30 &  5  &  37.65$\pm$0.22  &  49.57$\pm$0.17  \\
        40 &  5  &  36.78$\pm$0.22  &  50.64$\pm$0.20  \\
        50 &  5  &  36.60$\pm$0.12  &  50.52$\pm$0.07  \\ 
        \shline
        % \multicolumn{3}{c}{} \\ % space holder
        \end{tabular}
}
\label{tab:cmb_ablation}
}
\label{lab:ablation_study}
\caption{Ablation studies on the multi-modality complementary and the hyper-parameters of CMB.}
\vspace{-4mm}
\end{table*}
\section{Implementation Details}
\subsection{Different Image-based Conditions And Hyper-parameters}
Our experiments include 6 types of image-based conditions:
\begin{itemize}
    \item \textit{Canny edge} \& \textit{Sketch.} We utilize the training set of COCO \citep{lin14coco}, which contains $123K$ images, as the training data to train MCU-Net separately under canny and sketch guidance. The corresponding canny edge and sketch are generated by Canny algorithm \citep{canny} with default thresholds, and PiDiNet \citep{pidinet} with a threshold of $0.5$, respectively.
    \item \textit{Segmentation.} We utilize training set of COCO-Stuff \citep{caesar2018cvpr} as training data, which includes $123K$ images and corresponding semantic segmentation annotations. It covers 80 thing classes, 91 stuff classes and 1 "unlabeled" class, providing a comprehensive range of semantic information for MCU-Net training.
    \item \textit{Depth.} In order to obtain sufficient volume of data to train MCU-Net under this conditions with abstract representation, we select $650K$ images from LAION-AESTHETICS dataset \citep{schuhmann2022laion}. And we adopt MiDaS \citep{midas} on them to generate depth maps.
    \item \textit{Pose.} We also pick images from LAION-AESTHETICS \citep{schuhmann2022laion} to construct training data for MCU-Net under pose guidance. The key distinction from building training dataset for depth guidance is that the selected images must contain at least one person for pose generation. To achieve this, we employ MM-Pose \citep{mmpose}, an open-source toolbox for pose estimation, to filter out images that do not meet the requirement, and generate pose for the retained images. In the end, we gather a total of $600k$ image-pose pairs to train MCU-Net under this condition.
    \item  \textit{Text.} Within our default backbone, the \emph{SD-2.1 Inpainting}, the prompt text is conditioned as the key and value of the cross-attention mechanism in the U-Net denoiser. It's noteworthy that this backbone is pretrained with the prompt text in the classifier-free way \citep{ho22free}. Consequently, in this work, we opt to use the backbone directly, thus bypassing the necessity to fine-tune an MCU-Net for text guidance. 
\end{itemize}
All our experiments are conducted using 8 NVIDIA A100-40G GPUs. We set the batch size to 64 and employed the Adam optimizer \citep{adam} with the learning rate of 1e-5 for training 10 epochs. These settings remain consistent across all conditions. 

\subsection{Acquisition of Conditions}
To facilitate a reliable and convenient comparison of model performance, we employed the conditions provided by the dataset directly or leveraged existing tools \citep{yang2022cross, mmpose, midas} to estimate them. We then evaluated the model performances using quantitative metrics on completing the corresponding masked RGB images. It is important to note that our method also supports the input of manually designed guidance conditions (as shown in Fig. \ref{fig:teaser}, Fig. \ref{fig:edit_supp_1}, Fig. \ref{fig:outpaint_supp_1} and Fig. \ref{fig:outpaint_supp_2}). However, when manually design dense guidance conditions like segmentation and depth maps, it's crucial to ensure their consistency with the information retained in the unmasked regions, particularly in the case of depth maps where values represent the distance between pixels and the camera. Fortunately, sparse conditions like sketch or pose maps can offer sufficient guidance information. We intend to release our code for condition generation, enabling users to obtain modalities including sketch, pose and segmentation maps effortlessly for image editing purposes.

\subsection{Architecture of Encoding Network $\tau_{c_i}$}
The condition encoding network is designed to be simple and lightweight, and serves the purpose of extracting the multi-scale guidance signals from the input condition image. These guidance signals are aligned in size with the intermediate feature maps of the MCU-Net's encoder. As this is not the main focus of our work, we have referred to the design of T2I-Adapter \citep{mou23t2i}. Specifically, it consists of four feature extraction blocks with a downsample module placed between each pair of adjacent blocks, and each feature extraction block is composed of one convolution layer and two residual blocks. 

\begin{wrapfigure}{r}{0.4\textwidth}
\centering
\vspace{-15mm}
\includegraphics[width=0.95\linewidth]{imgs/supp/supp_mask_ratio.pdf} 
\vspace{-3mm}
\caption{Illustration of mask ratio.}
\vspace{-5mm}
\label{fig:supp_maskratio}
\end{wrapfigure}
\subsection{Experimental Setting}
In order to evaluate all baselines and our proposed method in a fair manner, the same image-mask pairs are used in quantitative experiments. Additionally, testing mask samples are obtained based on a uniform distribution ranging from 0\% to 100\% to encompass the majority of mask ratios encountered in real-world scenarios. The testing mask is randomly generated based on the algorithm from \citep{li22mat}, with the histogram of the testing mask ratio of COCO dataset visualized in Fig.~\ref{fig:supp_maskratio}.

All quantitative experiments are conducted on the COCO \citep{lin14coco} and Places \citep{places} datasets. Evaluation of the methods involves using the first 1000 images in the COCO validation set and the first 5000 images in the Places validation set. Masks from COCO are replicated five times for the Places dataset. For auxiliary guided completion, the Canny algorithm \citep{canny} and PiDiNet \citep{pidinet} are employed to obtain the canny edge map and sketch map, respectively. MiDaS \citep{midas} is adopted to acquire depth maps for both datasets. COCO serves as a dataset with semantic segmentation and prompt text annotations. As the Places dataset lacks ground-truth labels, the semantic segmentation map is estimated using CIRKD \citep{yang2022cross}.


\section{Application Results}
\subsection{Real User-input Image Editing}

We highlight the adaptability of our method in handling \emph{user-input image editing} tasks designed to manipulate real-world images based on user intention, as demonstrated in Fig.~\ref{fig:edit_supp_1}. This figure emphasizes our method's capacity to modify the structure or semantics of local regions using user-input guidance such as scribble, pose map and prompt text, while fully maintaining the integrity of the unmasked region. 

\subsection{Image Outpainting}

Our method can also be used to extend an image, like generating a panorama from a small part of the image content. As demonstrated in Fig.~\ref{fig:outpaint_supp_1} and Fig.~\ref{fig:outpaint_supp_2}, our method showcases its capability to \emph{outpaint} a photograph or a painting guided by text and sketch map. Remarkably, our method exhibit the ability to generate suitable content that is harmonious even with the broader context of a panoramic image.

\begin{table}[t]\small
\centering
% \resizebox{\textwidth}{!}{
\begin{tabular}{lccccc} \toprule
      & \multicolumn{2}{c}{COCO}    & \multicolumn{3}{c}{Places2}                     \\ \cmidrule(r){2-3} \cmidrule(r){4-6}
Method      & FID$\downarrow$ & PickScore$\uparrow$ / \% & FID$\downarrow$           & U-IDS$\uparrow$ / \%         & P-IDS$\uparrow$ / \%         \\ \midrule
ZITS~\canny{}  & 61.27           & 28.09     & 18.96         & 18.75          & 7.20 \\
T2I-Adapter~\canny{}    & 48.23 & 30.10 & 10.39 & 19.44 & 5.66             \\
ControlNet~\canny{} & \textbf{37.17} & \textbf{37.30} & 10.35 & 18.45 & 4.58             \\
Ours~\canny{}  & 41.15  & 34.94     & \textbf{8.32} & \textbf{26.23} & \textbf{10.96} \\ \midrule

T2I-Adapter~\depth{}  & 50.92 & 30.22 & 18.10 & 14.91  & 4.56 \\
ControlNet~\depth{}  & 46.13 & 32.52 & 15.96 & 14.46 & 3.18 \\
Ours~\depth{} & \textbf{39.43}  & \textbf{37.12}     & \textbf{9.09} & \textbf{25.34} & \textbf{10.64} \\ \midrule

T2I-Adapter~\seg{}  & 50.65 & 28.10  & 15.36 & 15.99 & 4.30  \\
ControlNet~\seg{}  & 58.27 & 26.11     & 18.13 & 13.68 & 3.24 \\
Ours~\seg{}  & \textbf{41.91}  & \textbf{34.96}     & \textbf{10.27} & \textbf{24.21} & \textbf{9.93} \\ \midrule

T2I-Adapter~\multimodal{}  & 39.08 &  34.26    & 14.27 & 14.76 & 3.30 \\
% ControlNet~\multimodal{}  & - & -     & - & - & - \\
Ours~\multimodal{}  & \textbf{37.65} & \textbf{49.57}     & \textbf{8.98} & \textbf{25.30} & \textbf{10.90} 
\\\bottomrule
\end{tabular}%}
\vspace{0.6em}
\caption{
{Quantitative comparisons with conditional image completion and text-to-image methods. \canny{}: ground truth edge map as guidance, \depth{}: estimated depth map as guidance, \seg{}: segmentation map as guidance, \multimodal{}: using segmentation, depth, canny, sketch, and text (on COCO) for guidance simultaneously.}
%$\uparrow$: the higher the better, $\downarrow$: the lower the better, $\dagger$: completion without any guidance.
}
\label{tab:quan_t2i}
\end{table}


\begin{table}[t]\small
\centering
% \resizebox{\textwidth}{!}{
\begin{tabular}{lccccccc} \toprule
      & \multicolumn{4}{c}{COCO}    & \multicolumn{3}{c}{Places2}                     \\ \cmidrule(r){2-5} \cmidrule(r){6-8}
Method      & CLIP$\uparrow$ / \% & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$ \\ \midrule
ZITS~\canny{} & 28.33 & 14.31 & 0.2767 & 0.5382 & \textbf{21.07}  & \textbf{0.6888}  & \textbf{0.2614}        \\
T2I-Adapter~\canny{}    & 28.59 & 18.26 & 0.6272  & 0.3409 & 18.34 & 0.6537 & 0.3208         \\
ControlNet~\canny{} & 28.97 & \textbf{19.22} & \textbf{0.6871} & \textbf{0.3183} & 18.59 & 0.6647 & 0.3220           \\
Ours~\canny{}  & \textbf{29.37}  & 18.13  & 0.6188  & 0.3467 & 19.00 & 0.6569 & 0.3111          \\ \midrule

T2I-Adapter~\depth{}  & 28.05 & 17.84 & 0.5894 & 0.3729 & 17.57 & 0.5765 & 0.3805  \\
ControlNet~\depth{}  & 28.22 & \textbf{18.16} & \textbf{0.6275} & \textbf{0.3583} & 17.49 & 0.5967 & 0.3703 \\
Ours~\depth{}  & \textbf{29.11}  & 17.47 & 0.5960  & 0.3628 & \textbf{17.91} & \textbf{0.6109} & \textbf{0.3432} \\ \midrule

T2I-Adapter~\seg{}  & 28.10 & \textbf{17.55} & 0.5635 & 0.3830  & 17.33 & 0.5529 & 0.3923 \\
ControlNet~\seg{}  & 26.48 & 16.98 & 0.5587 & 0.4023 & 17.22 & 0.5568 & 0.3948 \\
Ours~\seg{}  & \textbf{28.87}  & 17.01 & \textbf{0.5681} & \textbf{0.3799} & \textbf{17.44} & \textbf{0.5860} & \textbf{0.3591} \\ \midrule

T2I-Adapter~\multimodal{}  & 30.23 &  \textbf{19.45} & \textbf{0.6748} & \textbf{0.3217} & \textbf{19.17} & \textbf{0.6626} & \textbf{0.3255} \\
% ControlNet~\multimodal{}  & - & -     & - & - & - & - \\
Ours~\multimodal{}  & \textbf{31.29} & 17.49  & 0.5921 & 0.3717 & 17.85 & 0.6085 & 0.3439
\\\bottomrule
\end{tabular}%}
\vspace{0.6em}
\caption{
{ Additional quantitative comparison results in terms of CLIP score and traditional reconstruction metrics. \canny{}: ground truth edge map as guidance, \depth{}: estimated depth map as guidance, \seg{}: segmentation map as guidance, \multimodal{}: using segmentation, depth, canny, sketch, and text (on COCO) for guidance simultaneously. }
% $\uparrow$: the higher the better, $\downarrow$: the lower the better, $\dagger$: completion without any guidance.
}
\label{tab:extra_score}
\end{table}



\section{More Experimental Results And Studies}
\subsection{Quantitative Comparisons with Conditional Text-to-Image Methods}

Contemporary methods like ControlNet and T2I-Adapter have demonstrated remarkable achievements in controllable image generation. For a direct comparison, we employ latent-level blending to utilize these methods for image completion, maintaining the experimental settings of earlier Experiments. As Table~\ref{tab:quan_t2i} reveals, our MaGIC significantly surpasses baseline models in FID, U-IDS, P-IDS, and PickScore for most guidance types. In multi-modality guidance, we enhance T2I-Adapter with multi-adapter controlling \cite{mou23t2i} (feature-level addition), resulting in T2I-Adapter\multimodal{}. For the COCO dataset, we employ five modalities: \textit{canny edge, depth, segmentation, sketch map, and text}. For the Places dataset, we utilize \textit{canny edge, depth, segmentation, and sketch map}, as it lacks manually-crafted captions.
Our MaGIC outperforms T2I-Adapter\multimodal{} by 44.68\% in PickScore on COCO, and shows improvements of 37.07\%, 71.40\%, and 230.30\% in FID, U-IDS, and P-IDS, respectively, on Places, as detailed in the last two rows of  Table~\ref{tab:quan_t2i}.

While traditional reconstruction metrics, such as PSNR, SSIM, and LPIPS, rely on pixel-wise similarity to the ground truth and tend to favor blurry outputs, as noted by ~\cite{zhao21comod} and ~\cite{li22mat}, they are not optimal for quantitatively assessing image completion. Nevertheless, we include these traditional metrics for reference. Additionally, we provide the CLIP Score as an extra measure for a more comprehensive evaluation in the Table~\ref{tab:extra_score}.

\subsection{Qualitative Comparisons in Multimodal Conditioning}
As illustrated in Figure~\ref{fig:supp_multi_modal_places}, we perform qualitative, side-by-side comparisons with T2I-Adapter\multimodal{}. For our MaGIC\multimodal{}, we produce five diverse results. Under the guidance of four modalities, MaGIC demonstrates strong controllability and high-fidelity outputs, aligning with our quantitative findings. In contrast, while T2I-Adapter\multimodal{} effectively adheres the layout or shape to guidance, it fails to generate images of above-average quality with realistic details. This shortfall is attributed to the feature-level addition approach, leading to an out-of-distribution effect in the SD U-Net.

\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\vspace{-15mm}
\includegraphics[width=0.95\linewidth]{imgs/supp/supp_tsne.jpeg} 
\vspace{-3mm}
\caption{t-SNE visualization of features output from U-Net encoder.}
\vspace{-5mm}
\label{fig:supp_t-sne}
\end{wrapfigure}
\subsection{Study in Feature-level Addition}
% T-SNE
Although we claim that FLA (feature-level addition) is a simple yet imperfect method for combining multiple modalities, and demonstrate the effectiveness of our CMB by comparative experiments, a comprehensive understanding of these two methods remains elusive. To this end, we opt to visualize the feature distributions stemming from T2I-Adapter with single-modality training and multi-modality utilization strategies, including FLA or our proposed CMB. 

In specific, we choose the feature from the middle denoising step (\ie, the 25th step of DDIM sampler) and output from U-Net encoder. The t-SNE visualization result is shown in Figure~\ref{fig:supp_t-sne}, and different colors represents features from different sources, while the associated numbers indicate the index and cluster center of each feature type. Numbers ranging from small to large represent features obtained from T2I-Adapter-Canny (\textcolor{BrickRed}{0}), T2I-Adapter-Depth (\textcolor{YellowGreen}{1}), T2I-Adapter-Segmentation (\textcolor{Green}{2}), T2I-Adapter-Sketch (\textcolor{CornflowerBlue}{3}), T2I-Adapter-CMB (\textcolor{blue}{4}) and T2I-Adapter-FLA (\textcolor{Orchid}{5}), respectively, where the first four indicate the trained single-modality while the last two are two methods of combining these four modalities.
 
 We can draw two conclusions from Figure~\ref{fig:supp_t-sne}:
 \begin{enumerate}
 	\item Features derived from different single-modality models (\textcolor{BrickRed}{0}, \textcolor{YellowGreen}{1}, \textcolor{Green}{2} and \textcolor{CornflowerBlue}{3}) show significant distribution disparities, and FLA (\textcolor{Orchid}{5}) directly adds modality features resulting in the distribution deviation of obtained feature from all others. This observation aligns with our assertion in the main manuscript that ``we called feature-level addition is impractical, as the denoiser is trained solely on the distribution of $\hat{F}_c=F_{enc}+F_c$".
 	\item In contrast to FLA, the distribution of features obtained through CMB (\textcolor{blue}{4}) is surrounded by other single-modality distributions. This phenomenon is coherent with Equation ~\ref{eqn:loss} and ~\ref{eqn:backward}, where the distribution of obtained features is ``pulled" by the distributions of the other four single-modalities.       
 \end{enumerate} 


 
\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{imgs/supp/supp_failed_cases.pdf}
    \caption{Failed cases when adopting CMB on T2I-Adapter.}
    \label{fig:supp_failed_cases}
\end{figure}
\subsection{Failure Cases}
Figure~\ref{fig:supp_failed_cases} shows two failed instances of applying CMB on T2I-Adapter for multi-modalities guidance. Here we present a more complex testing scenario involving non-overlapping information between two modalities. In the first case, we use Anything-4.0 as the backbone and there are two mistakes in the generated image: the misshapen cat and the incorrectly positioned bench under the girl.  The former discrepancy possibly arises from the pose adapter contributing stronger features compared to those from the depth adapter, consequently affecting the representation of the latter information, which is not accurately reflected in the generated image. The issue might be alleviated by training depth adapter more stronger or increasing $\delta_{depth}$ while decreasing $\delta_{pose}$ (see Equation~\ref{eqn:loss} for details). While the latter mistake is the inherent challenge in SD, and many related works \cite{chen23layout, DBLP:journals/tog/CheferAVWC23} could be referenced for potential mitigation steategies. In the second example, the generated image depicts a wall as the background, leading to the complete loss of depth information from the depth map. 

These instances underscore that the inherent problems of SD persist despite employing CMB. Furthermore, in scenarios where information correlation between different modalities is low, evident issues such as information loss and errors in generated images become more pronounced.
Through these failed cases, it can be seen that the inherent challenges of SD cannot be eliminated by CMB. Furthermore, in scenarios where information correlation between different modalities is low, evident issues such as information loss and errors in generated images become more pronounced.

\subsection{Study in Adaptability and Image Generation Application}
Our proposed MaGIC has the ability to adapt to a variety of backbone diffusion models, including but not limited to, the image generation model Anything-4.0, Stable Diffusion-1.5 (also employed by T2I-adapter and ControlNet), and the image completion model Stable Diffusion Inpainting-2.1 (the default in MaGIC). 
In order to elucidate the differences among these backbone diffusion models, a qualitative experiment was carried out, focusing primarily on the anime-style image generation model Anything-4.0, image generation model Stable Diffusion (SD), the mask-aware T2I-adapter \citep{mou23t2i, avrahami22blend}, and our own MaGIC.

As portrayed in Fig.~\ref{fig:ablation_backbone}(a) and (b), our MaGIC method exhibits exceptional generalizability to image generation backbones. These backbones can produce convincing results guided by factors such as sketch, depth, segmentation, and the canny edge map. T2I-Adapter \citep{mou23t2i} is a conditional image generation framework based on Stable Diffusion-1.5. To equip T2I-Adapter with CMB for the completion of a masked image, we implemented a technique known as latent-level blending \citep{avrahami22blend}. As evidenced in Fig.~\ref{fig:ablation_backbone}(c) and (d), incorporating blending into T2I-Adapter can preserve the unmasked region remains while the generated masked region does not perceive the unmasked region, given the fact that there are two sheep heads in a single sheep. 

We further adapt CMB to T2I-Adapter and ControlNet \citep{zhang23controlnet} to verify its effectiveness on multi-modality guided image generation. The results are shown in Fig. ~\ref{fig:controllable_generation} and Fig. ~\ref{fig:supp_controlnet}. 


\newcommand{\mm}{0.17}
\renewcommand\arraystretch{1.1}
\begin{figure*}[!th]
	\centering
	\begin{tabular}{c@{\hspace{2.0mm}}c@{\hspace{2.0mm}}c@{\hspace{2.0mm}}c@{\hspace{2.0mm}}c@{\hspace{2.0mm}}c}

    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/gt/Places365_val_00000066.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/lama/Places365_val_00000066.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/mat/Places365_val_00000066.png}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/t2i-adapter-fla/Places365_val_00000066.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000066_0.jpg}} & \\
    \small{Masked} & \small{LAMA} & \small{MAT} & \small{T2I-Adapter\multimodal{}} & \small{Ours\multimodal{}-Result1} \\
    \frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000066_1.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000066_2.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000066_3.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000066_4.jpg}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/guidance/Places365_val_00000066.jpg}} \\
    \small{Ours\multimodal{}-Result2} & \small{Ours\multimodal{}-Result3} & \small{Ours\multimodal{}-Result4} & \small{Ours\multimodal{}-Result5} & \small{Guidance} \\

    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/gt/Places365_val_00000020.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/lama/Places365_val_00000020.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/mat/Places365_val_00000020.png}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/t2i-adapter-fla/Places365_val_00000020.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000020_0.jpg}} & \\
    \small{Masked} & \small{LAMA} & \small{MAT} & \small{T2I-Adapter\multimodal{}} & \small{Ours\multimodal{}-Result1} \\
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000020_1.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000020_2.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000020_3.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000020_4.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/guidance/Places365_val_00000020.jpg}} \\
	\small{Ours\multimodal{}-Result2} & \small{Ours\multimodal{}-Result3} & \small{Ours\multimodal{}-Result4} & \small{Ours\multimodal{}-Result5} & \small{Guidance} \\

    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/gt/Places365_val_00000015.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/lama/Places365_val_00000015.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/mat/Places365_val_00000015.png}} &
    \frame{\includegraphics[width=\mm\linewidth]{supp/baselines/t2i-adapter-fla/Places365_val_00000015.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000015_0.jpg}} & \\
    \small{Masked} & \small{LAMA} & \small{MAT} & \small{T2I-Adapter\multimodal{}} & \small{Ours\multimodal{}-Result1} \\
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000015_1.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000015_2.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000015_3.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/mm_places/Places365_val_00000015_4.jpg}} &
	\frame{\includegraphics[width=\mm\linewidth]{supp/baselines/guidance/Places365_val_00000015.jpg}} \\
	\small{Ours\multimodal{}-Result2} & \small{Ours\multimodal{}-Result3} & \small{Ours\multimodal{}-Result4} & \small{Ours\multimodal{}-Result5} & \small{Guidance} \\

\end{tabular}
\caption{
 Qualitative results of MaGIC with four guidance compared to baselines}
\label{fig:supp_multi_modal_places}
\end{figure*}

\clearpage


\begin{figure}[h]
    \centering
   \begin{overpic}[width=\textwidth]{imgs/supp/ablation_backbone_model} \small
   \put(5,-2){Masked}
   \put(24,-2){(a)}
   \put(40,-2){(b)}
   \put(57,-2){(c)}
   \put(73,-2){(d)}
   \put(87,-2){Guidance}
   \end{overpic}
   \vspace{1mm}
    \caption{Results from different backbone models. (a) Anything-4.0, (b) Stable Diffusion-1.5, (c) T2I-adapter with blending, (d) Stable Diffusion Inpainting-2.1 (default in MaGIC).}
    \label{fig:ablation_backbone}
\end{figure}


\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{imgs/supp/controllable_generation.pdf}
    \caption{Adapting the proposed CMB to T2I-Adapter \citep{mou23t2i} with Anything-4.0 backbone for multi-modality guided image generation.}
    \label{fig:controllable_generation}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{imgs/supp/supp_controlnet.pdf}
    \caption{Adapting the proposed CMB to ControlNet \citep{zhang23controlnet} for multi-modality guided image generation.}
    \label{fig:supp_controlnet}
\end{figure}





\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{imgs/supp/edit_supp_1.pdf}
    \caption{Application examples: local editing.}
    \label{fig:edit_supp_1}
\end{figure}



\begin{figure}
    \centering
    \includegraphics[width=0.86\linewidth]{imgs/supp/outpaint_supp_1.pdf}
    \caption{Application examples: sketch and text guided image outpainting.}
    \label{fig:outpaint_supp_1}
\end{figure}


\begin{figure}
    \centering
    \includegraphics[width=0.92\linewidth]{imgs/supp/outpaint_supp_2.pdf}
    \caption{Application examples: sketch and text guided image outpainting.}
    \label{fig:outpaint_supp_2}
\end{figure}
\begin{thebibliography}{50}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Avrahami et~al.(2022)Avrahami, Lischinski, and Fried]{avarahami22blended}
Omri Avrahami, Dani Lischinski, and Ohad Fried.
\newblock Blended diffusion for text-driven editing of natural images.
\newblock In \emph{CVPR}, 2022.

\bibitem[Avrahami et~al.(2023)Avrahami, Fried, and Lischinski]{avrahami22blend}
Omri Avrahami, Ohad Fried, and Dani Lischinski.
\newblock Blended latent diffusion.
\newblock In \emph{SIGGRAPH}, 2023.

\bibitem[Bansal et~al.(2023)Bansal, Chu, Schwarzschild, Sengupta, Goldblum, Geiping, and Goldstein]{bansal23universal}
Arpit Bansal, Hong{-}Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
\newblock Universal guidance for diffusion models.
\newblock \emph{arXiv:2302.07121}, 2023.

\bibitem[Caesar et~al.(2018)Caesar, Uijlings, and Ferrari]{caesar2018cvpr}
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.
\newblock Coco-stuff: Thing and stuff classes in context.
\newblock In \emph{CVPR}, 2018.

\bibitem[Canny(1986)]{canny}
John~F. Canny.
\newblock A computational approach to edge detection.
\newblock \emph{IEEE TPAMI}, 8\penalty0 (6):\penalty0 679--698, 1986.

\bibitem[Chefer et~al.(2023)Chefer, Alaluf, Vinker, Wolf, and Cohen{-}Or]{DBLP:journals/tog/CheferAVWC23}
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen{-}Or.
\newblock Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.
\newblock \emph{{ACM} Trans. Graph.}, 42\penalty0 (4):\penalty0 148:1--148:10, 2023.

\bibitem[Chen et~al.(2023)Chen, Laina, and Vedaldi]{chen23layout}
Minghao Chen, Iro Laina, and Andrea Vedaldi.
\newblock Training-free layout control with cross-attention guidance.
\newblock \emph{arXiv:2304.03373}, 2023.

\bibitem[Contributors(2020)]{mmpose}
MMPose Contributors.
\newblock Openmmlab pose estimation toolbox and benchmark.
\newblock \url{https://github.com/open-mmlab/mmpose}, 2020.

\bibitem[Criminisi et~al.(2003)Criminisi, P{\'{e}}rez, and Toyama]{criminisi03object}
Antonio Criminisi, Patrick P{\'{e}}rez, and Kentaro Toyama.
\newblock Object removal by exemplar-based inpainting.
\newblock In \emph{CVPR}, 2003.

\bibitem[Dhariwal \& Nichol(2021)Dhariwal and Nichol]{Dhariwal21beat}
Prafulla Dhariwal and Alexander~Quinn Nichol.
\newblock Diffusion models beat gans on image synthesis.
\newblock In \emph{NeurIPS}, 2021.

\bibitem[Dong et~al.(2022)Dong, Cao, and Fu]{dong22zits}
Qiaole Dong, Chenjie Cao, and Yanwei Fu.
\newblock Incremental transformer structure enhanced image inpainting with masking positional encoding.
\newblock In \emph{CVPR}, 2022.

\bibitem[Gu et~al.(2022)Gu, Chen, Bao, Wen, Zhang, Chen, Yuan, and Guo]{DBLP:conf/cvpr/GuCBWZCYG22}
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo~Zhang, Dongdong Chen, Lu~Yuan, and Baining Guo.
\newblock Vector quantized diffusion model for text-to-image synthesis.
\newblock In \emph{CVPR}, 2022.

\bibitem[Guo et~al.(2021)Guo, Yang, and Huang]{guo21ctsdg}
Xiefan Guo, Hongyu Yang, and Di~Huang.
\newblock Image inpainting via conditional texture and structure dual generation.
\newblock In \emph{ICCV}, 2021.

\bibitem[Ho \& Salimans(2021)Ho and Salimans]{ho22free}
Jonathan Ho and Tim Salimans.
\newblock Classifier-free diffusion guidance.
\newblock In \emph{NeurIPS Workshop}, 2021.

\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho20ddpm}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock In \emph{NeurIPS}, 2020.

\bibitem[Horita et~al.(2022)Horita, Yang, Chen, Koyama, and Aizawa]{Horita22struct}
Daichi Horita, Jiaolong Yang, Dong Chen, Yuki Koyama, and Kiyoharu Aizawa.
\newblock A structure-guided diffusion model for large-hole diverse image completion.
\newblock \emph{arXiv:2211.10437}, 2022.

\bibitem[Jeong et~al.(2023)Jeong, Kwon, and Uh]{jeong23hsapce}
Jaeseok Jeong, Mingi Kwon, and Youngjung Uh.
\newblock Training-free style transfer emerges from h-space in diffusion models.
\newblock In \emph{ICLR}, 2023.

\bibitem[Kawar et~al.(2023)Kawar, Zada, Lang, Tov, Chang, Dekel, Mosseri, and Irani]{DBLP:journals/corr/abs-2210-09276}
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.
\newblock Imagic: Text-based real image editing with diffusion models.
\newblock In \emph{CVPR}, 2023.

\bibitem[Kim et~al.(2022)Kim, Kwon, and Ye]{DBLP:conf/cvpr/KimKY22a}
Gwanghyun Kim, Taesung Kwon, and Jong~Chul Ye.
\newblock Diffusionclip: Text-guided diffusion models for robust image manipulation.
\newblock In \emph{CVPR}, 2022.

\bibitem[Kingma \& Ba(2015)Kingma and Ba]{adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: {A} method for stochastic optimization.
\newblock In \emph{ICLR}, 2015.

\bibitem[Kirstain et~al.(2023)Kirstain, Polyak, Singer, Matiana, Penna, and Levy]{kirstain23pickscore}
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.
\newblock Pick-a-pic: An open dataset of user preferences for text-to-image generation.
\newblock \emph{arXiv:2305.01569}, 2023.

\bibitem[Levin et~al.(2004)Levin, Zomet, Peleg, and Weiss]{DBLP:conf/eccv/LevinZPW04}
Anat Levin, Assaf Zomet, Shmuel Peleg, and Yair Weiss.
\newblock Seamless image stitching in the gradient domain.
\newblock In \emph{ECCV}, 2004.

\bibitem[Li et~al.(2022)Li, Lin, Zhou, Qi, Wang, and Jia]{li22mat}
Wenbo Li, Zhe Lin, Kun Zhou, Lu~Qi, Yi~Wang, and Jiaya Jia.
\newblock {MAT:} mask-aware transformer for large hole image inpainting.
\newblock In \emph{CVPR}, 2022.

\bibitem[Liao et~al.(2020)Liao, Xiao, Wang, Lin, and Satoh]{liao20sge}
Liang Liao, Jing Xiao, Zheng Wang, Chia{-}Wen Lin, and Shin'ichi Satoh.
\newblock Guidance and evaluation: Semantic-aware image inpainting for mixed scenes.
\newblock In \emph{ECCV}, 2020.

\bibitem[Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Doll{\'{a}}r, and Zitnick]{lin14coco}
Tsung{-}Yi Lin, Michael Maire, Serge~J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll{\'{a}}r, and C.~Lawrence Zitnick.
\newblock Microsoft {COCO:} common objects in context.
\newblock In \emph{ECCV}, 2014.

\bibitem[Lugmayr et~al.(2022)Lugmayr, Danelljan, Romero, Yu, Timofte, and Gool]{lugmayr22repaint}
Andreas Lugmayr, Martin Danelljan, Andr{\'{e}}s Romero, Fisher Yu, Radu Timofte, and Luc~Van Gool.
\newblock Repaint: Inpainting using denoising diffusion probabilistic models.
\newblock In \emph{CVPR}, 2022.

\bibitem[Mou et~al.(2023)Mou, Wang, Xie, Zhang, Qi, Shan, and Qie]{mou23t2i}
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.
\newblock T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.
\newblock \emph{arXiv:2302.08453}, 2023.

\bibitem[Nazeri et~al.(2019)Nazeri, Ng, Joseph, Qureshi, and Ebrahimi]{Nazeri19ec}
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal~Z. Qureshi, and Mehran Ebrahimi.
\newblock Edgeconnect: Structure guided image inpainting using edge prediction.
\newblock In \emph{ICCVW}, 2019.

\bibitem[Nichol et~al.(2022)Nichol, Dhariwal, Ramesh, Shyam, Mishkin, McGrew, Sutskever, and Chen]{nichol22glide}
Alexander~Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.
\newblock {GLIDE:} towards photorealistic image generation and editing with text-guided diffusion models.
\newblock In \emph{ICML}, 2022.

\bibitem[Ramesh et~al.(2022)Ramesh, Dhariwal, Nichol, Chu, and Chen]{DBLP:journals/corr/abs-2204-06125}
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
\newblock Hierarchical text-conditional image generation with {CLIP} latents.
\newblock \emph{arXiv:2204.06125}, 2022.

\bibitem[Ranftl et~al.(2022)Ranftl, Lasinger, Hafner, Schindler, and Koltun]{midas}
Ren{\'{e}} Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.
\newblock Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.
\newblock \emph{IEEE TPAMI}, 44\penalty0 (3):\penalty0 1623--1637, 2022.

\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer]{Rombach22ldm}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"{o}}rn Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{CVPR}, 2022.

\bibitem[Saharia et~al.(2022)Saharia, Chan, Saxena, Li, Whang, Denton, Ghasemipour, Ayan, Mahdavi, Lopes, Salimans, Ho, Fleet, and Norouzi]{saharia22}
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar~Seyed Ghasemipour, Burcu~Karagol Ayan, S.~Sara Mahdavi, Rapha~Gontijo Lopes, Tim Salimans, Jonathan Ho, David~J. Fleet, and Mohammad Norouzi.
\newblock Photorealistic text-to-image diffusion models with deep language understanding.
\newblock In \emph{NeurIPS}, 2022.

\bibitem[Schuhmann et~al.(2022)Schuhmann, Beaumont, Vencu, Gordon, Wightman, Cherti, Coombes, Katta, Mullis, Wortsman, et~al.]{schuhmann2022laion}
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et~al.
\newblock Laion-5b: An open large-scale dataset for training next generation image-text models.
\newblock \emph{arXiv preprint:2210.08402}, 2022.

\bibitem[Song et~al.(2021)Song, Meng, and Ermon]{song21ddim}
Jiaming Song, Chenlin Meng, and Stefano Ermon.
\newblock Denoising diffusion implicit models.
\newblock In \emph{ICLR}, 2021.

\bibitem[Song et~al.(2023)Song, Dhariwal, Chen, and Sutskever]{song23consistency}
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.
\newblock Consistency models.
\newblock In \emph{ICML}, 2023.

\bibitem[Su et~al.(2021)Su, Liu, Yu, Hu, Liao, Tian, Pietik{\"{a}}inen, and Liu]{pidinet}
Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi~Tian, Matti Pietik{\"{a}}inen, and Li~Liu.
\newblock Pixel difference networks for efficient edge detection.
\newblock In \emph{{ICCV}}, 2021.

\bibitem[Suvorov et~al.(2022)Suvorov, Logacheva, Mashikhin, Remizova, Ashukha, Silvestrov, Kong, Goka, Park, and Lempitsky]{Suvorov22lama}
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.
\newblock Resolution-robust large mask inpainting with fourier convolutions.
\newblock In \emph{WACV}, 2022.

\bibitem[Wan et~al.(2020)Wan, Zhang, Chen, Zhang, Chen, Liao, and Wen]{DBLP:conf/cvpr/Wan0CZC0W20}
Ziyu Wan, Bo~Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen.
\newblock Bringing old photos back to life.
\newblock In \emph{CVPR}, 2020.

\bibitem[Wang et~al.(2023)Wang, Saharia, Montgomery, Pont{-}Tuset, Noy, Pellegrini, Onoe, Laszlo, Fleet, Soricut, Baldridge, Norouzi, Anderson, and Chan]{wang22imagen}
Su~Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont{-}Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David~J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan.
\newblock Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.
\newblock In \emph{CVPR}, 2023.

\bibitem[Xie et~al.(2022)Xie, Zhang, Lin, Hinz, and Zhang]{xie22smartbrush}
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang.
\newblock Smartbrush: Text and shape guided object inpainting with diffusion model.
\newblock In \emph{CVPR}, 2022.

\bibitem[Yang et~al.(2022)Yang, Zhou, An, Jiang, Xu, and Zhang]{yang2022cross}
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang.
\newblock Cross-image relational knowledge distillation for semantic segmentation.
\newblock In \emph{CVPR}, 2022.

\bibitem[Yu et~al.(2019)Yu, Lin, Yang, Shen, Lu, and Huang]{yu19free}
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas~S. Huang.
\newblock Free-form image inpainting with gated convolution.
\newblock In \emph{ICCV}, 2019.

\bibitem[Yu et~al.(2023)Yu, Wang, Zhao, Ghanem, and Zhang]{yu23freedom}
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.
\newblock Freedom: Training-free energy-guided conditional diffusion model.
\newblock \emph{arXiv:2303.09833}, 2023.

\bibitem[Yu et~al.(2022)Yu, Du, Zhang, and Luo]{yu22unbiased}
Yongsheng Yu, Dawei Du, Libo Zhang, and Tiejian Luo.
\newblock Unbiased multi-modality guidance for image inpainting.
\newblock In \emph{ECCV}, 2022.

\bibitem[Zhang \& Agrawala(2023)Zhang and Agrawala]{zhang23controlnet}
Lvmin Zhang and Maneesh Agrawala.
\newblock Adding conditional control to text-to-image diffusion models.
\newblock \emph{arXiv:2302.05543}, 2023.

\bibitem[Zhao et~al.(2021)Zhao, Cui, Sheng, Dong, Liang, Chang, and Xu]{zhao21comod}
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric~I{-}Chao Chang, and Yan Xu.
\newblock Large scale image completion via co-modulated generative adversarial networks.
\newblock In \emph{ICLR}, 2021.

\bibitem[Zheng et~al.(2022{\natexlab{a}})Zheng, Song, Cham, Cai, Phung, and Luo]{zheng22vq}
Chuanxia Zheng, Guoxian Song, Tat{-}Jen Cham, Jianfei Cai, Dinh~Q. Phung, and Linjie Luo.
\newblock High-quality pluralistic image completion via code shared {VQGAN}.
\newblock \emph{arXiv:2204.01931}, 2022{\natexlab{a}}.

\bibitem[Zheng et~al.(2022{\natexlab{b}})Zheng, Lin, Lu, Cohen, Shechtman, Barnes, Zhang, Liu, Zhou, Amirghodsi, and Luo]{zheng22struct}
Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Connelly Barnes, Jianming Zhang, Qing Liu, Yuqian Zhou, Sohrab Amirghodsi, and Jiebo Luo.
\newblock Structure-guided image completion with image-level and object-level semantic discriminators.
\newblock \emph{arXiv:2212.06310}, 2022{\natexlab{b}}.

\bibitem[Zhou et~al.(2018)Zhou, Lapedriza, Khosla, Oliva, and Torralba]{places}
Bolei Zhou, {\`{A}}gata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.
\newblock Places: {A} 10 million image database for scene recognition.
\newblock \emph{IEEE TPAMI}, 40\penalty0 (6):\penalty0 1452--1464, 2018.

\end{thebibliography}