TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
Extracted source text ยท arXiv 2406.05814
% TODO:
% * read carefully call-for paper
% * read carefully latex template
\documentclass{article} % For LaTeX2e
\usepackage{iclr2025_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
% \usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{enumitem} % itemize
\usepackage{wrapfig,lipsum} % warptable
\usepackage{subfig}
\usepackage{graphicx} % resizebox
\usepackage{multirow}
\usepackage{colortbl} % for colorize table
\usepackage{bbding} % for \Checkmark
\usepackage{subfloat} % for subfig
% \usepackage{caption} % for captionof
% \usepackage{arydshln} % dash line
\usepackage{xr} % for appendix
\usepackage{makecell} % for Line breaks in text in tables
\usepackage{amsmath}
\def\eg{\emph{e.g.}}
\def\ie{\emph{i.e.}}
\newcommand{\mysubsubsec}[1]{\textbf{#1}}
\newcommand{\aka}{\emph{a.k.a. }}
\definecolor{mygray}{gray}{0.5}
\definecolor{myblue}{RGB}{24, 141, 228}
\definecolor{gray_bg}{gray}{0.9}
\newcommand{\wangtan}[1]{\textcolor{cyan}{\small{\bf [Tan: #1 ]}}}
\newcommand{\leigang}[1]{\textcolor{magenta}{\small{\bf [Leigang: #1 ]}}}
\newcommand{\mlogo}{\raisebox{-7pt}{\includegraphics[width=1.3em]{fig/tiger_logo.pdf}}}
\newcommand{\rebutt}[1]{\textcolor{blue}{{#1}}}
% \title{TIGeR: Unified Text-to-Image Generation and Retrieval in Large Multimodal Models}
\title{\mlogo TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
% \author{xxx}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\author{Leigang Qu$^{1}$ ~~ Haochuan Li$^{1}$ ~~ Tan Wang$^{2}$\thanks{Corresponding Authors: Tan Wang and Wenjie Wang} ~~ Wenjie Wang$^{3*}$ ~~ Yongqi Li$^{4}$ \\
\textbf{Liqiang Nie$^{5}$} ~~ \textbf{Tat-Seng Chua$^{1}$} \\
\small $^{1}$National University of Singapore~~~~~~$^2$Nanyang Technological University\\
\small $^3$University of Science and Technology of China~~~~~~$^4$Hong Kong Polytechnic University \\
\small $^5$Harbin Institute of Technology (Shenzhen) \\
\tt\small \{leigangqu, wenjiewang96, nieliqiang, liyongqi0\}@gmail.com\\
\tt\small haochuan@u.nus.edu \quad tan317@ntu.edu.sg \quad dcscts@nus.edu.sg
}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
How humans can effectively and efficiently acquire images has always been a perennial question.
A classic solution is \emph{text-to-image retrieval} from an existing database;
% given the text query;
however, the limited database typically lacks creativity.
By contrast, recent breakthroughs in \emph{text-to-image generation} have made it possible to produce attractive and counterfactual visual content, but it faces challenges in synthesizing knowledge-intensive images.
% In this work, we rethink the relationship between text-to-image generation and retrieval and propose a \textit{unified} framework in the context of Large Multimodal Models (LMMs).
In this work, we rethink the relationship between text-to-image generation and retrieval, proposing a \textit{unified} framework for both tasks with one single Large Multimodal Model (LMM).
% Specifically, we first explore the intrinsic discriminative abilities of MLLMs and introduce a generative retrieval method to perform retrieval in a training-free manner.
Specifically, we first explore the intrinsic discriminative abilities of LMMs and introduce an efficient generative retrieval method for text-to-image retrieval in a training-free manner.
Subsequently, we unify generation and retrieval autoregressively and propose an autonomous decision mechanism to choose the best-matched one between generated and retrieved images as the response to the text prompt.
% Additionally, we construct a benchmark called TIGeR-Bench, including creative and knowledge-intensive domains, to standardize the evaluation of unified text-to-image generation and retrieval.
To standardize the evaluation of unified text-to-image generation and retrieval, we construct TIGeR-Bench, a benchmark spanning both creative and knowledge-intensive domains.
Extensive experiments on TIGeR-Bench and two retrieval benchmarks, \ie, Flickr30K and MS-COCO, demonstrate the superiority of our proposed framework.
The code, models, and benchmark are available at \url{https://tiger-t2i.github.io}.
\end{abstract}
\input{sec/1_intro}
\input{sec/2_related_work}
\input{sec/3_method}
\input{sec/4_benchmark}
\input{sec/5_exp}
\input{sec/6_conc}
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
% You may include other additional sections here.
\clearpage
\appendix
\input{sec/appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\section{Introduction}\label{sec:intro}
\begin{figure}[h]
\centering
% \vspace{-7pt}
\includegraphics[width=1.0\textwidth]{fig/key_idea.pdf}
% \caption{We consider both \textcolor[RGB]{70, 114, 196}{creative} domains and \textcolor[RGB]{237, 125, 49}{knowledge-intensive} domains for general image acquisition towards unified Text-to-Image Generation and Retrieval (TIGeR), and build the TIGeR-Bench including 8 domains for the unified comprehensive evaluation.
\caption{\textbf{TIGeR-ONE} unifies T2I-G and T2I-R through one single LMM in a training-free autoregressive way, with a decision mechanism to adaptively select between generated and retrieved images based on user prompts. Besides, we construct TIGeR-Bench, encompassing eight \textcolor[RGB]{70, 114, 196}{creative} and \textcolor[RGB]{237, 125, 49}{knowledge-intensive} domains in total to facilitate a comprehensive evaluation of TIGeR.
%Image-text pairwise data in 8 domains is collected and constructed into our TIGeR-Bench (left). A MLLMs-based framework is proposed to unify the two tasks in a training-free manner (right).
}
% \vspace{-2pt}
\label{fig:key_idea}
% \vspace{-7pt}
\end{figure}
The explosion of visual information on the Web significantly challenges human information access.
Text-to-Image Retrieval (T2I-R)~\citep{radford2021learning, yu2022coca, li2023blip} is one of the main channels to obtain visual information given a text prompt.
However, T2I-R is limited to retrieving existing images in the database, lacking flexibility and creativity. Furthermore, as the database expands, retrieval costs increase significantly.
Recent years have witnessed thrilling progress in Text-to-Image Generation (T2I-G)~\citep{ramesh2022hierarchical, rombach2022high, podell2023sdxl},
which directly generates new images to meet human visual information needs.
% However, T2I-G falls short of handling knowledge-intensive concepts, \eg, landmarks and natural species (see Fig.~\ref{fig:key_idea}), leading to the hallucination problem~\citep{kim2024tackling, huang2024visual}.
However, T2I-G struggles with knowledge-intensive concepts such as landmarks and natural species (see the right part of Fig.~\ref{fig:key_idea}), often resulting in hallucination issues~\citep{kim2024tackling, huang2024visual}.
In this light, a single T2I-R or T2I-G model may not satisfy the diverse and evolving human information needs.
It is pivotal to unify both T2I-R and T2I-G within a framework for visual information delivery.
To this end, a straightforward solution is to empower discriminative models with the generation ability.
However, the early-stage trial (\eg, JEM~\citep{grathwohl2019your}) requires extra generative training and may compromise the original discriminative power.
% Another recent attempt~\citep{li2023your, clark2023text} is to adapt generative diffusion models~\citep{podell2023sdxl} to achieve discriminative tasks.
Another solution adapts generative models such as diffusion models~\citep{podell2023sdxl} to achieve the discriminative tasks~\citep{li2023your, clark2023text}.
Despite the significance, these methods are limited to diffusion models and inevitably suffer from the notorious inefficiency problem caused by iterative denoising~\citep{ho2020denoising}.
Moreover, diffusion models are usually tailored for simple discriminative tasks such as image classification~\citep{li2023your, he2023discriminative} and
% it is difficult for diffusion models to process complex human intentions for large-scale retrieval in practical scenarios~\citep{schuhmann2022laion}.
making them less suitable for processing complex human prompts for large-scale retrieval in practical scenarios~\citep{schuhmann2022laion}.
Unlike diffusion models, Large Multimodal Models (LMMs) offer another form of generative paradigm to solve broad vision-language problems, garnering significant attention for their powerful language understanding and instruction following abilities~\citep{ouyang2022training, touvron2023llama2}.
Recently, notable efforts in LMMs~\citep{sun2023emu2, ge2023making, dong2023dreamllm} integrate Large Language Models (LLMs) with external T2I-G models~\citep{rombach2022high} for image synthesis.
% Nevertheless, these works only consider T2I-G without T2I-R.
However, most studies focus solely on T2I-G, neglecting T2I-R.
% Even though GILL~\citep{koh2023generating} fuses an LLM and the image encoder \& decoder to achieve the unified ability of generation and retrieval, it highly depends on an external retriever (\ie, CLIP~\citep{radford2021learning}) for the discriminative ranking in the dedicated embedding space.
Even though GILL~\citep{koh2023generating} fuses an LLM and the image encoder \& decoder to enable both generation and retrieval in an ensemble strategy, it essentially incorporates an external retriever (\ie, CLIP~\citep{radford2021learning}) for dense retrieval in a dedicated embedding space.
% discriminative ranking in a dedicated embedding space and performs dense retrieval.
% Not to mention the inefficiency and inconsistency of the dense retrieval paradigm based on the embedding space in the context of MLLMs.
% As such, the inefficiency problem~\citep{tay2022transformer, zhai2024actions} of the dense retrieval paradigm still hinders large-scale image retrieval of GILL.
As such, GILL brings extra alignment costs and still suffers from the inefficiency problem (discussed in Sec.~\ref{sec:related_work} and Sec.~\ref{sec:in_depth_ana}) of the dense retrieval paradigm, particularly in large-scale image retrieval
% scenarios~\citep{tay2022transformer, zhai2024actions}.
scenarios~\citep{tay2022transformer}.
% In this work, we propose to elegantly unify \underline{T}ext-to-\underline{I}mage \underline{Ge}neration and \underline{R}etrieval (\textbf{TIGeR}) within LMMs, a promising task to achieve more flexible image acquisition, as shown in Fig.~\ref{fig:key_idea}.
% In this work, we propose the task of elegantly unifying \underline{T}ext-to-\underline{I}mage \underline{Ge}neration and \underline{R}etrieval (\textbf{TIGeR}), along with a framework named \textbf{TIGeR-ONE}, which achieves this unification within \underline{one} single LMM, enabling flexible and efficient image acquisition, as shown in Fig.~\ref{fig:key_idea}.
To this end, we propose to unify \underline{T}ext-to-\underline{I}mage \underline{Ge}neration and \underline{R}etrieval (\textbf{TIGeR}) in this work, and present a model-agnostic framework named \textbf{TIGeR-ONE} that achieves this unification within \underline{one} single LMM, enabling flexible and efficient image acquisition as shown in Fig.~\ref{fig:key_idea}.
We first delve into the intrinsic bidirectional (\ie, text-to-image and image-to-text) discriminative abilities of LMMs in Sec.~\ref{sec:intr_dist}.
Specifically, we investigate three likelihood-based proxies to estimate cross-modal semantic similarities. % between textual prompts and images.
% The similarities can be utilized to perform image-text matching, and the resulting performance reflects the discriminative abilities.
% Specifically, we investigate three conditional likelihoods as proxies for estimating semantic similarities. The similarities can be utilized to perform image-text matching, and the resulting performance reflects the discriminative abilities.
% Specifically, we study the potential of bidirectional conditional likelihoods as the proxies of semantic similarity by assessing the performance in image-text matching.
% Based on the discriminative abilities, we present an efficient generative retrieval method with forward beam search and reverse re-ranking as in Sec.~\ref{sec:generative_retr}, unifying both T2I-R and T2I-G in an autoregressive generation manner.
% Based on the discriminative performance of proxy likelihoods,
Based on these proxies,
TIGeR-ONE adopts an efficient generative retrieval method with forward beam search and reverse re-ranking as in Sec.~\ref{sec:generative_retr}, unifying both T2I-R and T2I-G in an autoregressive generation manner.
% Besides, we propose an autonomous decision mechanism to adaptively choose one from retrieved and generated images according to user prompts.
Moreover, TIGeR-ONE presents an autonomous decision mechanism to adaptively select between retrieved and generated images based on user prompts.
% To comprehensively evaluate the performance of LMMs on TIGeR, we build a benchmark (\textbf{TIGeR-Bench}) in Sec.~\ref{sec:bench} from two aspects, one for creative images from the counterfactual world or users' imaginative prompts~\citep{kirstain2024pick}, and the other is derived from six knowledge-intensive domains (\eg, news, landmark, and natural species).
% To comprehensively evaluate the performance of LMMs on the TIGeR task, we build a benchmark (\textbf{TIGeR-Bench}) in Sec.~\ref{sec:bench} spanning creative and knowledge-intensive domains, which covers creative images from the counterfactual world or imaginative scenarios~\citep{kirstain2024pick}, and knowledge-intensive images from six diverse domains (\eg, logo, landmark, and natural species).
Existing benchmarks~\citep{saharia2022photorealistic, huang2023t2i, weyand2020google} assess generation and retrieval separately, with limited domain coverage.
To comprehensively evaluate the performance of LMMs on TIGeR, we build a benchmark called \textbf{TIGeR-Bench} (Sec.~\ref{sec:bench}). It encompasses creative images from the counterfactual world and imaginative scenarios~\citep{kirstain2024pick}, and knowledge-intensive images from six diverse domains (\eg, logo, landmark, and natural species).
% Additionally, we carry out extensive experiments to assess the TIGeR performance of representative LMMs, verifying the effectiveness of the proposed method.
We carry out extensive experiments to assess the TIGeR performance of representative LMMs on TIGeR-Bench and two T2I-R benchmarks, validating the effectiveness and efficiency of TIGeR-ONE.
Overall, we summarize the contributions into three points.
\begin{itemize}[leftmargin=7.5mm]
\setlength{\itemsep}{2pt}
% \item Driven by diverse and complex practical scenarios for visual information access, we propose to unify text-to-image generation and retrieval in the context of LMMs.
\item Driven by the complementary roles of text-to-image generation and retrieval in visual information access, we propose unifying both tasks to meet complex human information needs.
%through LMMs with strong generative and discriminative capabilities.
\item We comprehensively inspect the intrinsic cross-modal discriminative abilities of LMMs and propose TIGeR-ONE, a model-agnostic framework for the TIGeR task. TIGeR-ONE performs text-to-image generation and retrieval in a training-free autoregressive manner, selecting the best-matched result autonomously and efficiently.
% propose a training-free generative retrieval approach based on bidirectional discrimination, thus unifying text-to-image generation and retrieval in an autoregressive generation manner.
% Besides, we present an autonomous decision mechanism to adaptively select retrieved between generated images.
\item We construct a comprehensive image acquisition benchmark, TIGeR-Bench, to evaluate the performance of TIGeR on LMMs in creative and knowledge-intensive domains. Extensive experiments on TIGeR-Bench and two T2I-G benchmarks including Flickr30K~\citep{young2014image} and MS-COCO~\citep{lin2014microsoft} verify the effectiveness of TIGeR-ONE.
\end{itemize}
\section{Related Work}\label{sec:related_work}
\textbf{Text-to-Image Generation. }
T2I-G has aroused wide attention in both academia and industry over the past decade, with advancements ranging from Generative Adversarial Networks~\citep{reed2016generative} to Auto-regression Models~\citep{ding2021cogview} and Diffusion Probabilistic Models (DPMs)~\citep{ho2020denoising}. Recent breakthroughs in DPMs, guided by the scaling law~\citep{kaplan2020scaling, li2024scalability}, have propelled T2I-G to new heights, \eg, models like DALL-E 2~\citep{ramesh2022hierarchical} and DALL-E 3~\citep{betker2023improving}, the Imagen series~\citep{saharia2022photorealistic}, and the Stable Diffusion (SD) series~\citep{rombach2022high, podell2023sdxl, esser2024scaling}.
%, have demonstrated remarkable capabilities in synthesizing both fanciful and photorealistic images, closely aligning with human aesthetics.
Concurrently, efforts have been made to enhance the composed text-image alignment~\citep{feng2022training, chefer2023attend, qu2023layoutllm, qu2024discriminative, yang2024mastering} and cater to human preference~\citep{lee2023aligning, xu2024imagereward}.
Some studies recognize the importance of knowledge-intensive image acquisition, employing RAG~\citep{chen2022re} or constructing benchmark~\citep{huang2024kitten}. However, they focus solely on generation and overlook the potential for unifying generation and retrieval.
% However, most prior works predominantly focus on creative domains for image generation, neglecting the imperative of knowledge-intensive image acquisition tailored to human needs.
\textbf{Text-to-Image Retrieval. }
Early studies on multimodal information retrieval focused on feature representation~\citep{faghri2017vse++, li2019visual} and modality interaction~\citep{lee2018stacked, qu2021dynamic} for precise cross-modal similarity estimation. Recent advancements, propelled by large-scale pre-training, have led to improved retrieval performance and generalization in vision-language models~\citep{radford2021learning, li2021align, dou2022empirical}. More recently, researchers have explored more challenging scenarios, such as fine-grained interaction~\citep{lin2024preflmr}, equivariant similarity~\citep{wang2023equivariant}, multimodal instruction following~\citep{wei2023uniir}, and chat-based IR~\citep{levy2024chatting}. Despite thrilling progress, retrieval systems are inherently limited by database size, and incapable of creating new visual content.
\textbf{Large Multimodal Models. }
Empowered by the versatility of LLMs, pioneering works on LMMs have shown impressive understanding capabilities~\citep{liu2023visual, zhu2023minigpt}.
Recent research explores image generation through two categories: 1) \textit{Continuous Visual Representation} methods aim to align visual representations from LLMs with condition embeddings of SD through regression~\citep{koh2023generating, sun2023emu2, wu2023next, dong2023dreamllm, zhu2023vl} or score distillation~\citep{dong2023dreamllm} objectives.
GILL~\citep{koh2023generating} combines an external dense retriever, \ie, CLIP, and a diffusion decoder with an LLM to achieve retrieval and generation, respectively.
However, such an ensemble approach brings extra alignment costs and may encounter an alignment gap~\citep{zhao2024aligngpt}. Moreover, the dense retriever suffers from inefficiency~\citep{li2024generative}, as it requires extensive similarity comparisons between the query and all items in the database.
2) \textit{Discrete Visual Tokenization} methods~\citep{yu2023scaling, lu2023unified, ge2023making} first encode an image into a sequence of discrete codes~\citep{esser2021taming, van2017neural, ge2023making, jin2023unified}, and then employ next-token prediction to train LMMs. To synthesize images, the discrete codes are decoded into the pixel space via VQ-GAN or SD. In this work, we resort to the discrete paradigm to be consistent with the inherent discreteness of language.
Compared with existing work, TIGeR-ONE achieves comprehensive image acquisition, encompassing both content creation and knowledge retrieval within a single framework.
\section{Methodology}
% In this section, we first formulate the unified problem in Sec.~\ref{sec:problem_form}, and then explore the intrinsic cross-model discriminative ability of LMMs for semantic similarity measurement in Sec.~\ref{sec:intr_dist}. Based on the ability, we propose a unified framework, as shown in Fig.~\ref{fig:framework}, including generative retrieval in Sec.~\ref{sec:generative_retr}, synchronous generation and retrieval, and decision-making in Sec.~\ref{sec:unify}.
We first formulate the task of unifying T2I-G and T2I-R through LMMs in Sec.~\ref{sec:problem_form}. We then explore the intrinsic cross-model discriminative ability of LMMs in Sec.~\ref{sec:intr_dist}. Based on the discriminative ability, we propose the TIGeR-ONE framework, as shown in Fig.~\ref{fig:framework}, including generative retrieval in Sec.~\ref{sec:generative_retr}, synchronous generation and retrieval, and decision-making in Sec.~\ref{sec:unify}.
\subsection{Task Formulation}\label{sec:problem_form}
% The goal of this work is to satisfy complex human visual information needs by acquiring images from diverse domains in a unified LMM framework.
TIGeR aims to satisfy complex human visual information needs by unifying T2I-G and T2I-R in a unified LMM framework.
We formulate this problem
in an autoregressive generation manner:
% autoregressively:
\begin{equation}\label{eqn:problem_form}
p(Y|X) = \prod_{i=1}^{N} p(y_i | Y_{<i}, X),
\end{equation}
where $X$ denotes a textual prompt provided by humans, tokenized into a sequence $X = [x_1, ..., x_M]$;
and $Y = [y_1, ..., y_N]$ denotes the sequence of visual tokens~\citep{ge2023making} that can be decoded into an image, with $Y_{<i}$ referring to the tokens before step $i$.
By sampling from the conditional distribution, we obtain a sequence instance, \ie, $Y^* \sim p(Y|X)$, where $Y^* \in \mathbb{V}^N$ and $\mathbb{V}$ denotes the visual token space defined by a visual vocabulary, and $|\mathbb{V}| = V$ is the vocabulary size. $\mathbb{V}^N$ denotes the Cartesian product of $N$ token spaces, \ie, the whole discrete visual space.
To achieve unified T2I-G and T2I-R, an LMM is required to possess the following three capabilities: 1) \textbf{creativity} to generate a novel photorealistic image $\hat{Y}$ based on the visual tokens sampled from $p(Y|X)$; 2) \textbf{discrimination} to measure semantic similarity between a prompt $X$ and each image $Y$, and then retrieve the relevant image $\tilde{Y}$ from a database $\mathcal{G}$, formally, $\tilde{Y} = \argmax_{Y \in \mathcal{G}} p(Y|X)$; and 3) \textbf{decision} to automatically determine the superior option by comparing $p(\hat{Y}|X)$ and $p(\tilde{Y}|X)$ for generation and retrieval, respectively, ultimately yielding the optimal result $Y^*$.
% \\$\bullet$ \textbf{Creativity} to generate novel photorealistic images by sampling from $p(Y|X)$;
% \\$\bullet$ \textbf{Discrimination} to match vision and language and retrieve the relevant images from an image database or gallery $\mathcal{G}$;
% \\$\bullet$ \textbf{Discrimination} to measure text-image semantic similarity and retrieve the relevant images from an image database $\mathcal{G}$;
% \\$\bullet$ \textbf{Decision} to automatically determine the superior option between generation and retrieval.
Considering the proficiency of recent LMMs~\citep{dong2023dreamllm, zheng2023minigpt, ge2023making} in creative T2I-G, we shed more light on exploring the discriminative ability for T2I-R and the potential of unifying generation and retrieval with decision-making in the remaining of this section.
\subsection{Intrinsic Discriminative Ability of LMMs}\label{sec:intr_dist}
For effective T2I-R, we first probe the discriminative capability of LLMs and present three training-free proxies to estimate the semantic similarity between the prompt and images in the database.
% For effective text-to-image retrieval, LLMs should possess a strong discriminative ability which is fundamental for accurate semantic similarity estimation between the prompt and images in the database. To probe the intrinsic capability, we present three training-free proxies for semantic similarity estimation.
\begin{wraptable}{r}{0.40\textwidth}
\centering
\vspace{-7pt}
\caption{Text-to-image ranking performance of three similarity (Sim.) proxies for SEED-LLaMA~\citep{ge2023making} and LaVIT~\citep{jin2023unified} on MS-COCO~\citep{lin2014microsoft}. }
% \\ [0.2cm]
\resizebox{.40\columnwidth}{!}{
\setlength\tabcolsep{5pt}
\renewcommand\arraystretch{1.1}
\begin{tabular}{l|cccc}
\toprule
\multirow{2}{*}{Sim. Proxy} & \multicolumn{2}{c}{SEED-LLaMA} & \multicolumn{2}{c}{LaVIT} \\
& \multicolumn{1}{c}{R@1} & \multicolumn{1}{c}{R@5} & \multicolumn{1}{c}{R@1} & \multicolumn{1}{c}{R@5} \\
\midrule
Random & 0.02 & 0.10 & 0.02 & 0.10 \\
$\log p(Y | X)$ & 3.50 & 8.79 & 0.02 & 0.16 \\
$\log \frac{p(Y | X)}{p(Y)}$ & 26.25 & 54.57 & 23.43 & 48.52 \\
$\log p(X | Y)$ & 49.34 & 75.45 & 50.35 & 70.87 \\
\bottomrule
\end{tabular}
}
\vspace{-10pt}
\label{tab:sim_proxy}
\end{wraptable}
\mysubsubsec{Proxy 1: Conditional Likelihood}.
To estimate the semantic similarity $s(X, Y)$ between a given text prompt $X$ and an image $Y \in \mathcal{G}$, a straightforward approach is to employ the conditional likelihood based on autoregressive factorization as the proxy:
\begin{equation}\label{eqn:x_to_y}
s(X, Y) = \log p(Y | X) = \sum_{i=1}^N \log p(y_i | Y_{<i}, X),
\end{equation}
% where $p(Y | X)$ denotes the likelihood of an image $Y \in \mathcal{G}$ from the database conditioned on the given prompt $X$.
where $p(Y | X)$ denotes the likelihood of autoregressively generating $Y$ conditioned on the given $X$.
In practice, we can attain it by computing the cross entropy between the predicted logits and the image tokens.
However, as shown in Tab.~\ref{tab:sim_proxy}, we observed that this proxy performs poorly.
We attribute this issue to \textit{visual bias}, caused by the interference of the visual prior $p(Y)$. Although similar phenomena have been noted in recent studies~\citep{krojer2023diffusion, lin2024revisiting} on diffusion and captioning models, the visual bias problem in LMMs has yet to receive adequate research attention.
% We attribute this problem to the \textit{modality bias}, including the visual bias~\citep{krojer2023diffusion} for T2I-G models and the linguistic bias~\citep{lin2024revisiting} for image captioning models, respectively.
\mysubsubsec{Proxy 2: Debiasing Pointwise Mutual Information}.
In this study, the visual bias largely stems from the unbalanced distribution $p(Y)$, and thus we introduce an alternative proxy based on Pointwise Mutual Information (PMI)~\citep{role2011handling, li2015diversity, lin2024revisiting} as,
\begin{equation}\label{eqn:x_to_y_debias}
s(X, Y) = \log \frac{p(Y | X)}{p(Y)^\eta} = \sum_{i=1}^N \log p(y_i | Y_{<i}, X)-\eta \sum_{i=1}^N \log p(y_i | Y_{<i}),
\end{equation}
% where we use the visual prior $p(Y)$ to help debiasing with a strength factor $\eta$, which can be estimated by $p(Y | \bar{X})$. $\bar{X}$ refers to a special prompt without any descriptive content, \eg, a null character or ``Can you give me an image?''.
where we use the visual prior $p(Y)$ to help debiasing with a strength factor $\eta$. $p(Y)$ can be approximately estimated by $p(Y | \bar{X})$, where $\bar{X}$ refers to a special prompt without any descriptive content, \eg, a null character or ``Can you give me an image?''.
The results in Tab.~\ref{tab:sim_proxy} demonstrate that this proxy could significantly alleviate the visual bias issue.
\mysubsubsec{Proxy 3: Reverse Conditional Likelihood}.
In addition to the debiasing strategy, we propose another option to circumvent the unbalanced prior distribution. Different from the generation process without the image $Y$, the retrieval task allows the model to access all images in the database.
% which means we can estimate the semantic similarity in a reverse way by predicting the conditional likelihood of another modality, \ie, $X$, with:
This means we can estimate the semantic similarity in a reverse way by predicting the conditional likelihood of $X$ given $Y$ to alleviate the visual bias of T2I-G models. Formally,
\begin{equation}\label{eqn:y_to_x}
s(X, Y) = \log p(X | Y) = \sum_{i=1}^M \log p(x_i | X_{<i}, Y),
\end{equation}
% To perform this reverse calculation, we view LMMs as image captioners.
where the LMMs work as image captioners to estimate the semantic similarity. As shown in Tab.~\ref{tab:sim_proxy}, this reverse proxy outperforms others, effectively revealing the discriminative abilities of LMMs.
%The above three training-free proxies of semantic similarities could be calculated in parallel because all tokens have been accessible beforehand.
Now we could estimate similarities between a given prompt and all $|\mathcal{G}|$ images in the database by traversing each image and calculating a proxy. Afterward, we could sort them to attain a ranking list, enabling T2I-R.
Compared with the prior work GILL~\citep{koh2023generating}, any of the three proxies can be seamlessly integrated with next-token prediction, the mainstream paradigm of generative pre-training. This proxy-based approach eliminates the need for additional discriminative training, such as contrastive learning, and benefits from fine-grained cross-modal interaction within LMMs.
\begin{figure}
\centering
\includegraphics[width=0.99\textwidth]{fig/framework.pdf}
\caption{Overview of the TIGeR-ONE framework to unify text-to-image generation and retrieval.
Images from the database are first tokenized into discrete codes and a lookup table is maintained for the correspondence between discrete codes and images. The given prompt $X$ is first fed into an LMM and Forward Beam Search is performed to retrieve and generate images in parallel. The prompt and obtained images are then fed into the same LLM for Reverse Re-Ranking and Decision-making. }
% \vspace{-2pt}
\label{fig:framework}
\vspace{-10pt}
\end{figure}
\subsection{LMMs-based Generative Retrieval}\label{sec:generative_retr}
% The above proxies make it possible to calculate cross-modal similarities and carry out ranking based on which.
The above proxies make it possible to calculate cross-modal similarities for T2I-R.
However, it is inefficient due to $|\mathcal{G}|$ times of forward propagation, each time with the extensive parameterization and the heavy internal attention interaction.
To tackle this issue and achieve an optimal balance between efficiency and recall, we introduce forward beam search and reverse re-ranking, as shown in Fig.~\ref{fig:framework}.
\mysubsubsec{Forward Beam Search (FBS)}.
Inspired by the advancement of generative retrieval~\citep{tay2022transformer, li2024generative}, we adopt constraint generation via autoregressive token decoding and beam search~\citep{freitag2017beam} with the beam size $B$ to recall $B$ images.
Specifically, we compress all images in the database into discrete tokens and store them in a Trie structure. This Trie structure constrains the sampling space and ensures that the generated prefix at any timestep corresponds to at least one image in the database. Once the beam search is finished, we could obtain a ranking list of $B$ sequences of visual tokens, each of which corresponds to an image.
This process aims to obtain a list of images given the prompt, thus the direction has to be $X \rightarrow Y$, which means we can only adopt the two forward proxies illustrated in Eqn.~\ref{eqn:x_to_y} or Eqn.~\ref{eqn:x_to_y_debias}. However, FBS method could significantly improve the efficiency since it only requires $N (N \ll |\mathcal{G}|)$ times of forward propagation of LMMs, where $N$ denotes the length of the visual token sequence for an image.
\mysubsubsec{Reverse Re-Ranking (RRR)}.
% Despite the improved efficiency, the semantic matching accuracy of the two forward proxies is obviously weaker than that of the reverse one in Eqn.~\ref{eqn:y_to_x}, as shown in Tab.~\ref{tab:sim_proxy}.
Despite the improved efficiency, the semantic matching ability of the two forward proxies is noticeably weaker than that of the reverse proxy in Eqn.~\ref{eqn:y_to_x}, as shown in Tab.~\ref{tab:sim_proxy}.
Given the ranking list $\mathcal{R} = [Y_1, ..., Y_B]$ obtained by forward beam search, we resort to the reverse proxy in Eqn.~\ref{eqn:y_to_x} for re-ranking and attain the final ranking list $\mathcal{R}^*$.
\subsection{TIGeR-ONE: Unifying Generation and Retrieval with One LMM}\label{sec:unify}
The discrete visual tokenization strategy enables LMMs to generate both language and visual content in an autoregressive generation manner. The proposed forward beam search in Sec.~\ref{sec:generative_retr} performs retrieval in the same autoregressive manner to generate visual tokens (which have been tokenized and saved in a database). Naturally, we can unify generation and retrieval with one LMM under the autoregression framework and make a decision between the generated and retrieved images. % to meet complex real-world requirements.
\mysubsubsec{Synchronous Generation and Retrieval}.
In TIGeR-ONE, an LMM can synchronously conduct unconstrained and constrained token decoding processes for image generation and retrieval, respectively.
As shown in Fig.~\ref{fig:framework}, these two tasks can be performed in parallel by maintaining respective search paths, which only requires $N$ forward propagations, ensuring efficiency.
Each path corresponds to a sequence of discrete visual tokens. We can generate a new image $Y^{G}$ by a diffusion decoder conditioned on the sequence, and meanwhile, immediately find the retrieved top-1\footnote{This work only considers one generated image and the top-1 retrieved image, but the proposed framework can also acquire more than one images batch-wise and choose the best one.} image $Y^{R}$. % via the lookup table.
\mysubsubsec{Decision Making}.
Given the generated $Y^{G}$ and the retrieved $Y^{R}$, we choose the better one based on the discriminative ability of LMMs, as discussed in Sec.~\ref{sec:generative_retr}. Specifically, we calculate two similarities $s(X, Y^G)$ and $s(X, Y^R)$ using one of the three proxies and choose the image with the higher similarity. Since we have computed similarities between the given prompt and shortlisted images through forward beam search and reverse re-ranking using the debiasing and reverse proxies, the decision process incurs no additional computational cost and can be performed efficiently.
% Different from the retrieval process performing semantic matching within the database, this decision-making module may require stronger discriminative power from LMMs due to the potential distribution gap between generated and retrieved images.
\section{TIGeR-Bench}\label{sec:bench}
% data domains and source
% pipeline: 1) collection, 2) filtering, 3) human quality check, 4) sampling and split
% extension: long text (gemini, gpt-4o), chat scenarios (gemini, gpt-4o)
To evaluate the performance of our method on TIGeR, we build a comprehensive benchmark (TIGeR-Bench), as shown in Fig.~\ref{fig:key_idea}.
It covers creative and knowledge-intensive image acquisition domains.
\mysubsubsec{Creative Domains}.
Creative images emphasize the intricate visual content that is challenging to capture in the real world. It includes unusual and \textit{counterfactual compositions} of concepts (\eg, ``A steamed train bellows rainbow-hued smoke'') and imaginary scenes aligning with \textit{real users' preference}.
To meet the two aspects, we collect prompt-image pairs from the well-designed WHOOPS!~\citep{bitton2023breaking} dataset and a large-scale open dataset named Pick-a-Pic~\citep{kirstain2024pick} which stems from a web platform collecting real users' creative intention.
\mysubsubsec{Knowledge-intensive Domains}.
Acquiring knowledge-intensive images requires models with extensive world knowledge and the ability to align such knowledge with visual objects or concepts.
% We focus on six knowledge domains including \textit{logo}~\citep{wang2020logo}, \textit{history and news}~\citep{liu2020visual}, \textit{landmark}~\citep{weyand2020google}, \textit{food}~\citep{min2023large}, \textit{nature species}~\citep{van2018inaturalist}, \textit{Wiki miscellaneous}~\citep{srinivasan2021wit}, and collect text-image pairs from six high-quality datasets.
We focus on six knowledge domains including \textit{logo}, \textit{history and news}, \textit{landmark}, \textit{food}~\citep{min2023large}, \textit{nature species}, \textit{Wiki miscellaneous}, and collect text-image pairs from six high-quality datasets.
Different from previous content-oriented data~\citep{lin2014microsoft} covering common objects in daily life, the collected data requires stronger cross-modal knowledge alignment, as the texts often consist solely of concept names without any descriptions in appearance.
We collect pairwise image-text data from eight domains, constructing the evaluation benchmark containing 6k data samples, with 3k for creative domains and 3k for knowledge domains.
See Appendix~\ref{app:tiger} for more details.
% Based on the requirements of the two domains, we collect pairwise image-text data from eight domains, constructing the evaluation benchmark containing 6,000 data samples, with 3,000 for creative domains and 3,000 for knowledge domains.
% See Appendix~\ref{app:tiger} for more details of data sources, construction pipeline, and extension to long prompts and chat scenarios.
\begin{table}[t!]
\centering
% \vspace{-7pt}
\caption{Performance comparison on TIGeR-Bench. ``Token'' refers to visual tokenization during image synthesis, including continuous (Cont.) and discrete (Dist.) approaches. Entries by {\color{mygray}gray} are expert models for T2I retrieval or generation, and those with a\colorbox{cyan!8}{cyan}background denote that an image query is first generated and then used for image-to-image retrieval. Entries with a\colorbox{gray_bg}{gray}background denote our methods. Methods with ``+'' denote agents, where models with ``*'' are decision models.}
\resizebox{.92\columnwidth}{!}{
\setlength\tabcolsep{15pt}
\begin{tabular}{l|c|c|c|cc}
\toprule
Method & Size & LLM & Token & CLIP-T~$\uparrow$ & CLIP-I~$\uparrow$ \\
\midrule
\multicolumn{6}{c}{\textit{Text-to-Image Generation}} \\
\midrule
\color{mygray}{SDXL~\citep{podell2023sdxl}} & \color{mygray}{2.6B} & \color{mygray}{-} & \color{mygray}{Cont.} & \color{mygray}{26.79} & \color{mygray}{46.71} \\
GILL~\citep{koh2023generating} & 8B & OPT-6.7B & Cont. & 14.16 & 13.72 \\
Emu~\citep{sun2023emu} & 14B & LLaMA-13B & Cont. & 22.26 & 40.78 \\
Emu 2~\citep{sun2023emu2} & 37B & LLaMA-33B & Cont. & 24.25 & 44.24 \\
DreamLLM~\citep{dong2023dreamllm} & 8B & Vicuna-7B & Cont. & 24.34 & 42.77 \\
SEED-LLaMA~\citep{ge2023making} & 8B & LLaMA-7B & Dist. & 22.00 & 43.02 \\
LaVIT~\citep{jin2023unified} & 11B & LLaMA-7B & Dist. & 27.07 & 48.75 \\
\midrule
\multicolumn{6}{c}{\textit{Text-to-Image Retrieval}} \\
\midrule
\color{mygray}{CLIP (ViT-B/32)~\citep{radford2021learning}} & \color{mygray}{151M} & \color{mygray}{-} & \color{mygray}{Cont.} & \color{mygray}{25.22} & \color{mygray}{53.95} \\
\rowcolor{cyan!8}
SDXL~\citep{podell2023sdxl} & 2.6B & - & Cont. & 15.41 & 35.96 \\
\rowcolor{cyan!8}
Emu~\citep{sun2023emu} & 14B & LLaMA-13B & Cont. & 14.44 & 34.46 \\
\rowcolor{cyan!8}
Emu 2~\citep{sun2023emu2} & 37B & LLaMA-33B & Cont. & 14.69 & 36.38 \\
\rowcolor{cyan!8}
DreamLLM~\citep{dong2023dreamllm} & 8B & Vicuna-7B & Cont. & 15.41 & 37.18 \\
\rowcolor{cyan!8}
SEED-LLaMA~\citep{ge2023making} & 8B & LLaMA-7B & Dist. & 14.78 & 36.93 \\
\rowcolor{cyan!8}
LaVIT~\citep{jin2023unified} & 11B & LLaMA-7B & Dist. & 16.34 & 39.25 \\
GILL~\citep{koh2023generating} & 8B & OPT-6.7B & Cont. & 10.96 & 16.30 \\
% \rowcolor{cyan!10}
\rowcolor{gray_bg}
Ours (SEED-LLaMA) & 8B & LLaMA-7B & Dist. & 16.95 & 40.30 \\
% \rowcolor{cyan!10}
\rowcolor{gray_bg}
Ours (LaVIT) & 11B & LLaMA-7B & Dist. & 21.30 & 50.03 \\
\midrule
\multicolumn{6}{c}{\textit{Unified Text-to-Image Generation and Retrieval}} \\
\midrule
GILL~\citep{koh2023generating} & 8B & OPT-6.7B & Cont. & 12.12 & 15.25 \\
SDXL + CLIP + SEED-LLaMA* & 11B & LLaMA-7B & Cont. & 26.91 & 47.51 \\
SDXL + CLIP + Qwen2-VL*~\citep{wang2024qwen2} & 11B & Qwen2-7B & Cont. & 27.91 & 60.65 \\
% \rowcolor{cyan!10}
\rowcolor{gray_bg}
Ours (SEED-LLaMA) & 8B & LLaMA-7B & Dist. & 23.98 & 50.52 \\
% \rowcolor{cyan!10}
\rowcolor{gray_bg}
Ours (LaVIT) & 11B & LLaMA-7B & Dist. & \textbf{28.45} & \textbf{61.37} \\
\bottomrule
\end{tabular}}
\label{tab:perfm_tiger}
\vspace{-5pt}
\end{table}
\section{Experiments} \label{sec:exp}
\subsection{Datasets, Baselines, and Evaluation Metrics}
We TIGeR-Bench to evaluate the unified performance, and the two widely-used benchmark datasets, \ie, Flickr30K~\citep{young2014image} and MS-COCO~\citep{lin2014microsoft}, to assess the text-to-image retrieval performance.
The compared baselines mainly include recent LMMs~\citep{koh2023generating, sun2023emu, sun2023emu2, ge2023making, jin2023unified} which can generate images, as well as generation and retrieval expert models~\citep{podell2023sdxl, radford2021learning}.
Following T2I-G~\citep{podell2023sdxl, esser2024scaling, saharia2022photorealistic}, the unified performance is measured by the CLIP score~\citep{hessel2021clipscore} including CLIP-T for text-image alignment and CLIP-I for the alignment between the predicted image and the ground-truth image.
As for T2I-R~\citep{hessel2021clipscore, faghri2017vse++}, we adopt the standard metric Recall at K, R@K (K=1, 5, and 10) for short.
Details on datasets, baselines, and evaluation metrics are provided in Appendix~\ref{app:tiger}.
\begin{table}[t!]
\centering
\caption{Text-to-image retrieval performance comparison on Flickr30K and MS-COCO. Entries by {\color{mygray}gray} denote dense retrieval methods and others are generative retrieval methods. Entries with a\colorbox{gray_bg}{gray}background denote our methods. }
\resizebox{.90\columnwidth}{!}{
\setlength\tabcolsep{15pt}
\begin{tabular}{l|ccc|ccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{3}{c|}{Flickr30K (1K)} & \multicolumn{3}{c}{MS-COCO (5K)} \\
% \midrule
& R@1 & R@5 & R@10 & R@1 & R@5 & R@10 \\
\midrule
\color{mygray}{CLIP (ViT-B/32)~\citep{radford2021learning}} & \color{mygray}{68.70} & \color{mygray}{90.60} & \color{mygray}{95.20} & \color{mygray}{37.80} & \color{mygray}{62.40} & \color{mygray}{72.20} \\
GRACE (Structured ID)~\citep{li2024generative} & 37.40 & 59.50 & 66.20 & 16.70 & 39.20 & 50.30 \\
IRGen~\citep{zhang2023irgen} & 49.00 & 68.90 & 72.50 & 29.60 & 50.70 & 56.30 \\
\rowcolor{gray_bg}
Ours (LaVIT) & 68.84 & 82.92 & 86.44 & 44.81 & 62.61 & 68.28 \\
\rowcolor{gray_bg}
Ours (SEED-LLaMA) & \textbf{71.70} & \textbf{91.82} & \textbf{95.44} & \textbf{46.11} & \textbf{69.02} & \textbf{76.13} \\
\bottomrule
\end{tabular}}
\label{tab:perfm_retr}
\vspace{-5pt}
\end{table}
\subsection{Performance Comparison}
\mysubsubsec{Unified Performance on TIGeR-Bench.}
We compare LMM baselines and our method, reporting results on TIGeR-Bench in Tab.~\ref{tab:perfm_tiger}, including separate and unified tasks.
The base models (\ie, SEED-LLaMA and LaVIT) and Ours share the same architectures and parameters. The key differences lie in the integration of FBS, RRR, and decision-making mechanisms, which are training-free and non-parametric.
The results demonstrate our method outperforms expert generation~\citep{podell2023sdxl} or retrieval models~\citep{radford2021learning}, the state-of-the-art LMMs~\citep{koh2023generating}, and even agent methods with expert models for generation and retrieval and the strong understanding LMM, \ie, Qwen2-VL~\cite{wang2024qwen2}, as the decision model.
Due to half of the data being sourced from knowledge domains, current generation models, \eg, SDXL~\citep{podell2023sdxl}, and LMMs could not handle the unified problem well.
Moreover, the proposed method could achieve impressive retrieval results, especially compared with other LMMs or SDXL. Compared with the vanilla SEED-LLaMA and LaVIT, our method significantly improves retrieval performance by dealing with the unified problem.
\mysubsubsec{Text-to-Image Retrieval Performance on Flickr30K~\citep{young2014image} and MS-COCO~\citep{lin2014microsoft}. }
As shown in Tab.~\ref{tab:perfm_retr}, we compare the proposed method with the representative dense retrieval model CLIP~\citep{radford2021learning} and two generative retrieval baselines~\citep{zhang2023irgen, li2024generative} which have been specially trained on the two datasets. In contrast, the proposed method is training-free but achieves the best performance across all baselines. It verifies the effectiveness of the proposed generative retrieval method and demonstrates that LMMs are capable of retrieval despite the sole optimization objective of next-token prediction.
\mysubsubsec{Chat-to-Image Acquisition Performance on VisDial~\citep{das2017visual}. }
Following GILL~\citep{koh2023generating}, we conducted experiments on VisDial to evaluate TIGeR-One based on LaVIT~\citep{jin2023unified} and SEED-LLaMA~\citep{ge2023making} for image acquisition in multi-turn chat scenarios.
\begin{wraptable}{r}{0.60\textwidth}
\centering
\vspace{-4pt}
\caption{Chat-to-image acquisition performance on VisDial~\citep{das2017visual}. The CLIP-I score is used as the evaluation metric. Unlike previous approaches limited to image generation (Gen.), our method autonomously selects between generation and retrieval (Retr.).
}
% \\ [0.2cm]
\resizebox{.55\columnwidth}{!}{
\setlength\tabcolsep{2pt}
\renewcommand\arraystretch{1.1}
\begin{tabular}{l|ccccc}
\toprule
Method & Gen. & Retr. & 1 round & 5 rounds & 10 rounds \\
\midrule
\multicolumn{6}{c}{\textit{Chat-to-Image Generation}} \\
\midrule
GLIDE~\citep{nichol2021glide} & \Checkmark & \XSolidBrush & 56.2 & 59.5 & 58.7 \\
SD-v1.5~\citep{rombach2022high} & \Checkmark & \XSolidBrush & 55.2 & 62.9 & 62.2 \\
GILL~\citep{koh2023generating}& \Checkmark & \XSolidBrush & 52.8 & 62.1 & 64.5 \\
LaVIT~\citep{jin2023unified} & \Checkmark & \XSolidBrush & 46.6 & 52.3 & 59.3 \\
SEED-LLaMA~\citep{ge2023making} & \Checkmark & \XSolidBrush & \textbf{57.0} & 64.4 & 67.9 \\
\midrule
\multicolumn{6}{c}{\textit{Unified Chat-to-Image Generation and Retrieval}} \\
\midrule
\rowcolor{gray_bg}
Ours (LaVIT) & \Checkmark & \Checkmark & 51.7 & 60.0 & 65.8 \\
& & & {\color{myblue}+5.1} & {\color{myblue}+7.7} & {\color{myblue}+6.5} \\
\rowcolor{gray_bg}
Ours (SEED-LLaMA) & \Checkmark & \Checkmark & \textbf{57.0} & \textbf{65.4} & \textbf{70.9} \\
& & & {\color{myblue}+0.0} & {\color{myblue}+1.0} & {\color{myblue}+3.0} \\
\bottomrule
\end{tabular}
}
\label{tab:chat-to-image}
\end{wraptable}
We compared our method with GLIDE~\citep{nichol2021glide}, SD v1.5~\citep{rombach2022high}, GILL, LaVIT and SEED-LLaMA, as shown in Tab.~\ref{tab:chat-to-image}.
The results demonstrate: 1) The original SEED-LLaMA exhibits the strongest multi-turn chat-to-image generation abilities, achieving the highest CLIP-I scores. 2) Our proposed unified framework significantly improves the performance of both LaVIT and SEED-LLaMA. 3) The improvement of our method becomes more pronounced with more rounds. For example, it achieves the most significant improvement in 10 rounds for SEED-LLaMA, which verifies the effectiveness of our method on complex multi-turn chat scenarios.
\begin{table}[t]
\centering
\caption{Ablation study on TIGeR-Bench investigating Reverse Re-Ranking (RRR) and two decision-making strategies, \ie, Forward with Eqn.~\ref{eqn:x_to_y_debias} and Reverse with Eqn.~\ref{eqn:y_to_x}. \%Retr. denotes the percentage of retrieved images selected as results. }
\resizebox{.83\columnwidth}{!}{
\setlength\tabcolsep{7pt}
\begin{tabular}{cc|cccc|cccc}
\toprule
\multirow{2}{*}{RRR} & \multirow{2}{*}{Decision} & \multicolumn{4}{c|}{Ours (SEED-LLaMA)} & \multicolumn{4}{c}{Ours (LaVIT)} \\
% \midrule
& & CLIP-T~$\uparrow$ & CLIP-I~$\uparrow$ & R@1~$\uparrow$ & \%Retr. & CLIP-T~$\uparrow$ & CLIP-I~$\uparrow$ & R@1~$\uparrow$ & \%Retr. \\
\midrule
% & $\log \frac{p(I|T)}{p(I)}$ & \textbf{22.63} & \textbf{49.71} & & & & & & \\
\XSolidBrush & Forward & 22.63 & 49.71 & 26.80 & 42.10 & 27.19 & 49.59 & 28.13 & 4.38 \\
\Checkmark & Forward & 23.72 & 48.86 & 29.23 & 12.72 & 27.28 & 49.62 & 49.37 & 1.40 \\
\XSolidBrush & Reverse & \textbf{23.89} & \textbf{50.52} & 26.80 & 25.60 & 28.23 & 56.51 & 28.13 & 30.47 \\
\Checkmark & Reverse & 22.84 & 49.54 & 29.23 & 61.47 & \textbf{28.45} & \textbf{61.37} & 49.37 & 56.25 \\
\bottomrule
\end{tabular}}
\vspace{-10pt}
\label{tab:ablation_rrr_dec}
\end{table}
\subsection{In-depth Analysis}\label{sec:in_depth_ana}
\mysubsubsec{Ablation Study on Reverse Re-Ranking (RRR) and Decision-Making for TIGeR}.
We evaluate TIGeR-One based on SEED-LLaMA and LaVIT by different RRR and decision settings, and report the unified and retrieval performance as well as the retrieval percentage in Tab.~\ref{tab:ablation_rrr_dec}.
We have the following discussions:
1) RRR could consistently improve the retrieval performance for SEED-LLaMA and LaVIT, but may not help in unified performance for SEED-LLaMA, because unified performance is also influenced by decision strategies.
2) Compared with the forward decision with Eqn.~\ref{eqn:x_to_y_debias}, the reverse decision with Eqn~\ref{eqn:y_to_x} could enhance the unified performance in most cases for both models, which reflects the reverse decision may have stronger discriminative power across more domains.
3) Intuitively, we expect the most correctly retrieved images can be selected and the left wrong ones can be remedied by generation.
However, we find that the two LMMs may suffer from a generation preference problem. Especially, LaVIT always prefers to choose generated images even though the retrieved ones are correct, as shown by the low \%Retr. in the first two settings. One of the reasons may be that significant gaps exist between the pre-trained and fine-tuned image generation data and TIGeR-Bench.
In all, besides the modality bias discussed in Tab.~\ref{tab:sim_proxy}, the difference between the two directional ranking and decision may be attributed to the unbalance between captioning (image-to-text) and text-to-image data at the training phase of LMMs.
\mysubsubsec{Visual Modality Debiasing for Discriminative Power}.
In Sec.~\ref{sec:generative_retr}, we discussed the visual modality bias problem with the forward $\log p(Y|X)$ in Eqn.~\ref{eqn:x_to_y} as the similarity proxy, and adopt a debiasing proxy $\log \frac{p(Y|X)}{p(Y)^\eta}$ by considering the unconditional likelihood.
To explore the influence of the debiasing strength, we set different values of the factor $\eta$ in Eqn.~\ref{eqn:x_to_y_debias} and a series of results are shown in Fig.~\ref{fig:debias_fac}.
They show that the ranking performance is sensitive to the debiasing strength and reaches the highest point around $\eta = 1$, verifying the effectiveness of the unconditional debiasing strategy.
\begin{table}
\begin{minipage}[t]{.31\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{fig/debiasing_factor.pdf}
\vspace{-5mm}
\captionof{figure}{The influence of the debiasing factor $\eta$ in Eqn.~\ref{eqn:x_to_y_debias} on the forward ranking performance of SEED-LLaMA and LaVIT on the MS-COCO dataset. The best performance is achieved around $\eta = 1$. }
\label{fig:debias_fac}
\end{minipage}\hfill
\begin{minipage}[t]{.31\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{fig/beam.pdf}
\vspace{-4mm}
\captionof{figure}{Retrieval performance on MS-COCO with different beam sizes and re-ranking strategies. Light and dark dash lines denote the forward and reverse ranking performance, respectively. S: SEED-LLaMA. L: LaVIT. }
\label{fig:beam}
\end{minipage}\hfill
\begin{minipage}[t]{.305\linewidth}
\centering
\includegraphics[width=1.0\textwidth]{fig/efficiency.pdf}
\vspace{-4mm}
\captionof{figure}{Comparison of retrieval efficiency quantified by the number of processed prompts per second among CLIP (ViT-B/32), GILL, and the proposed generative retrieval method based on SEED-LLaMA. }
\label{fig:effi}
\end{minipage}
\vspace{-0.7cm}
\end{table}
\mysubsubsec{Forward Beam Search (FBS) and Reverse Re-Ranking for Retrieval}.
Considering the trade-off of retrieval efficiency and recall, we present FBS and RRR, respectively. As shown in Fig.~\ref{fig:beam}, we compare the ranking (dotted lines) and retrieval (solid lines) performance and explore the impact of beam size and RRR. In the ranking experiments, we adopt the proxies in Sec.~\ref{sec:generative_retr} to calculate similarities, and then rank the whole database. The comparison indicates that ranking with the debiasing proxy ($\log \frac{p(Y|X)}{p(Y)^\eta}$) seems the upper bound of FBS since FBS may miss the target image with the limited beam size.
Benefiting from the reverse proxy ($\log p(X|Y)$), RRR could help FBS break through the ceiling and significantly improve recall.
Additionally, regardless of similarity proxies or base LMMs employed, increasing the beam size can reduce the recall gap between retrieval and ranking.
\mysubsubsec{Efficiency of Generative Retrieval}.
We analyze the efficiency of the proposed generative retrieval method for T2I-R and compare it with two dense retrieval methods, \ie GILL~\cite{koh2023generating} and CLIP~\citep{radford2021learning} in Fig.~\ref{fig:effi}.
The efficiency of dense retrieval gets worse with the increase in the database size due to more matching in the common feature space. In contrast, the proposed method keeps almost constant efficiency regardless of the database size. %We relegate more efficiency analysis to Appendix.~\ref{app:imp_details}.
\begin{wraptable}{r}{0.47\textwidth}
\centering
\vspace{-13pt}
\caption{Performance on TIGeR-Bench (Knowledge) in \textit{chat} scenarios, across various chat generation methods. }
% \\ [0.2cm]
% \vspace{-7pt}
\resizebox{.47\columnwidth}{!}{
\setlength\tabcolsep{6pt}
\renewcommand\arraystretch{1.1}
\begin{tabular}{l|cc|ccc}
\toprule
\multirow{2}{*}{\makecell[c]{Expansion\\Method}} & \multicolumn{2}{c|}{T2I Generation} & \multicolumn{3}{c}{T2I Retrieval} \\
& CLIP-T & CLIP-I & R@1 & R@5 & R@10 \\
\midrule
Raw Prompt & \textbf{19.50} & 36.11 & 22.57 & 36.80 & 43.23 \\
Gemini-Pro & 17.71 & 34.58 & 17.83 & 31.70 & 36.77 \\
GPT-4o & 19.40 & \textbf{38.17} & \textbf{24.03} & \textbf{40.73} & \textbf{47.83} \\
\bottomrule
\end{tabular}
}
\vspace{-8pt}
\label{tab:query_expand_chat}
\end{wraptable}
\mysubsubsec{Prompt Expansion in Chat Scenarios. }
We guide Gemini-Pro~\citep{reid2024gemini} and GPT-4o~\citep{4oapi} to imagine a scenario where a user intends to know a concept and seeks an image.
We provide them with detailed instructions and in-context examples, leveraging their expert language knowledge to process raw prompts. The prompts are then expanded into multi-round chat contexts, serving as input for T2I generation and retrieval. Results in Tab.~\ref{tab:query_expand_chat} indicate that unifying them could utilize the abundant knowledge within LMMs to improve knowledge-intensive image acquisition.
% Considering that LaVIT was not fine-tuned by chat
% instructions, we only carried out experiments based on SEED-LLaMA. We perform generative retrieval with 200 beams.
\subsection{Qualitative Analysis}
In Fig.~\ref{fig:qua_unify}, we compare our methods with SDXL on TIGeR-Bench, covering both creative and knowledge domains.
% SDXL, despite its strong generative abilities, struggles in knowledge domains. In contrast, our method excavates the intrinsic discriminative power of LMMs and endows them with the retrieval ability, enabling more effective handling for knowledge-intensive image acquisition.
Besides, we explore multi-turn chat scenarios with multimodal context and both image retrieval and generation or editing requirements in Fig.~\ref{fig:qua_chat}.
Owing to its training-free nature, our model-agnostic framework fully inherits the interleaved capabilities of the base model, facilitating accurate interleaved image generation with identity preservation.
Additionally, compared to the base model, our method can proactively retrieve knowledge-intensive images (\eg, the Eiffel Tower) and maintain their key characteristics throughout the interaction process.
% the consistency with the user-provided image can be well preserved, which stems from the interleaved image generation capabilities of the base model, which ensures loyalty to the multimodal context and preserves identity. Our model-agnostic framework fully inherits these abilities, enabling accurate interleaved image generation with identity preservation.
% Seamlessly integrated into LMMs without training and benefiting from multimodal instruction tuning, the proposed model inherits the basic multimodal understanding, generation, and user intention capturing capabilities of LMMs and meanwhile shows potential for knowledge-intensive tasks in vertical domains.
\begin{figure}[t]
\centering
\vspace{-5mm}
\includegraphics[width=0.99\textwidth]{fig/TIGeR_Showcase.pdf}
\vspace{-6mm}
\caption{Qualitative results on TIGeR-Bench. The prefix prompt "Give me an image of" is omitted here. Green ticks and red crosses highlight \textcolor[RGB]{19, 138, 7}{correct} and \textcolor{red}{wrong} retrieval results. }
\vspace{-2mm}
\label{fig:qua_unify}
% \vspace{-1pt}
\end{figure}
\begin{figure}[t]
\centering
\vspace{-5mm}
\includegraphics[width=0.97\textwidth]{fig/R2G_Chat.pdf}
\caption{Example of multi-turn chat based on SEED-LLaMA with unified generation and retrieval.}
\vspace{-6mm}
\label{fig:qua_chat}
\end{figure}
\section{Conclusion}
In this work, we start with the practical requirements for image acquisition, analyze the weaknesses of single generation and retrieval, and propose to unify these two tasks within MLLMs. Toward this end, we first delve into the intrinsic discriminative abilities of MLLMs for semantic matching and propose a generative retrieval method to perform text-to-image retrieval in an auto-regressive manner. Besides, under the same auto-regressive framework, we unify generation and retrieval synchronously and present an autonomous decision strategy to select the best image. The proposed framework exhibited effectiveness and versatility across the constructed TIGeR-Bench and two retrieval benchmarks.
% \clearpage
\appendix
% \section{Appendix}
\section{TIGeR-Bench Details}\label{app:tiger}
\subsection{Data Collection}
\mysubsubsec{Data Source. }
To comprehensively evaluate unified text-to-image generation and retrieval, we build a benchmark called TIGeR-Bench, encompassing both creative and knowledge-intensive domains.
For the creative domains, the data is derived from authentic user prompts that reflect real-world needs, requiring high levels of novelty and creativity.
We collect the data from the WHOOPS!~\cite{bitton2023breaking} and Pick-a-Pic~\cite{kirstain2024pick} datasets:
\begin{itemize}
\item \textbf{WHOOPS!}~\cite{bitton2023breaking}: The WHOOPS! dataset consists of 500 commonsense-defying prompt-image pairs created by designers. First, the designers think of counterfactual prompts by combining two elements or concepts that violate commonsense, \eg, ``Albert Einstein holding a smartphone''. Next, they are guided to use text-to-image generation tools (\eg, Midjourney, DALL-E~\cite{ramesh2021zero}, and Stable Diffusion~\cite{rombach2022high}) to synthesize images using these counterfactual prompts. Finally, the designers verify the `weirdness' of generated images to guarantee the data quality.
\item \textbf{Pick-a-Pic}~\cite{bitton2023breaking}: The Pick-a-Pic dataset consists of real-world user prompts and corresponding generated images, annotated with user preference, gathered from the Pick-a-Pic web application. In detail, we collect our data from Pick-a-Pic v2~\footnote{\url{https://huggingface.co/datasets/yuvalkirstain/pickapic_v2}}.
\end{itemize}
For knowledge-intensive domains, we collect data encompassing a wide range of categories to fulfill users' needs for visual knowledge, including Logo-2K+~\cite{wang2020logo}, Visual News~\cite{liu2020visual}, Google Landmark v2~\cite{weyand2020google}, Food2k~\cite{min2023large}, iNaturalist~\cite{van2018inaturalist}, WIT~\cite{kirstain2024pick}.
\begin{itemize}
\item \textbf{Logo-2K+}~\cite{wang2020logo}: Logo-2K+ is a large-scale real-world logo dataset, containing 167,140 images with 2,341 categories and 10 root categories, \eg, food, clothes, and institution.
\item \textbf{Visual News}~\cite{liu2020visual}: Visual News is a large-scale dataset comprising over one million news images along with associated news articles, image captions, author information, and additional metadata. Distinguished from other image captioning datasets, this dataset prioritizes factual contexts, including individuals, locations, and events, sourced from prominent news outlets such as The Guardian, BBC, USA Today, and The Washington Post.
\item \textbf{Google Landmark v2}~\cite{liu2020visual}: Google Landmark v2 includes approximately 5M images annotated with 200k distinct instance labels representing human-made and natural landmarks. It is collected from Wikimedia Commons.
\item \textbf{Food2K}~\cite{min2023large}: Food2K is a food recognition dataset with 2,000 categories and more than 1 million images, covering cereal products, vegetables, bread, snack, soup and porridge, barbecue, egg products, dessert, beam products, seafood, fried food, and meat.
\item \textbf{iNaturalist}~\cite{van2018inaturalist}: The iNaturalist dataset is constructed to reflect the diversity of the natural world, featuring an unbalanced distribution of species. It encompasses a total of 5,000 species of plants and animals, accompanied by 859,000 images.
\item \textbf{WIT}~\cite{kirstain2024pick}: Wikipedia-based Image Text (WIT) is a large multimodal multilingual dataset, comprising 37.6 million image-text pairs representing real-world entities. It encompasses 11.5 million unique images across 108 Wikipedia languages. The texts are sourced from 3 primary channels: reference descriptions, attribution descriptions, and alt-text descriptions.
\end{itemize}
\mysubsubsec{Prompts. }
The WHOOPS! and Pick-a-Pic datasets contain prompts, while Visual News also provides natural language descriptions, serving as user prompts or queries. For the remaining five datasets, only category names or concepts represented by single words or phrases are available. To address this, we utilize a template to formulate them into complete prompt sentences, \ie, ``Give me an image of [concept]''.
These datasets, originally designed for different purposes, are effectively repurposed as the creative domain and knowledge-intensive domain candidates within the TIGeR-Bench.
\begin{table}[t]
\centering
\caption{The statistics of TIGeR-Bench. We keep the ratio of $1:1$ for creative and knowledge domains and collect 6,000 high-quality text-image pairs in total. }
\vspace{4pt}
\resizebox{.75\columnwidth}{!}{
\begin{tabular}{llc}
\toprule
Domain & Data Source & \#Text-Image Pairs \\
\midrule
\multirow{2}[2]{*}{Creative} & WHOOPS!~\cite{bitton2023breaking} & 500 \\
& Pick-a-Pic~\cite{kirstain2024pick} & 2500 \\
\midrule
\multirow{6}[2]{*}{Knowledge-intensive} & Logo-2K+~\cite{wang2020logo} & 500 \\
& Visual News~\cite{liu2020visual} & 500 \\
& Google Landmark v2~\cite{weyand2020google} & 500 \\
& Food2K~\cite{min2023large} & 500 \\
& iNaturalist~\cite{van2018inaturalist} & 500 \\
& WIT~\cite{kirstain2024pick} & 500 \\
\bottomrule
\end{tabular}}
\label{tab:data_statistics}%
\end{table}%
\subsection{Automatic Data Filtration}
\mysubsubsec{Data Split}.
To evaluate text-to-image generation and retrieval, we prioritize selecting the original test split of each dataset to construct TIGeR-Bench. In cases where only a validation set is provided, we default to utilizing the validation set.
\mysubsubsec{Filtration Pipeline}
Given that all 8 datasets have undergone individual single-modality quality assessments during their construction, our emphasis now lies on cross-modal relevance and generation challenge properties. We proceed with the following three steps for data filtration.
\begin{enumerate}
\item [1)] To ensure a strong alignment between the positive text and image pairs for both generation and retrieval, we employ a filtering process to remove weakly relevant text-image pairs (\eg, outliers or noisy pairs) across 7 datasets except for WHOOPS! due to its limited scale. Specifically, we calculate the CLIP-T scores ($S_{gt}$) between the ground-truth images and texts, and remove pairs with CLIP-T scores lower than 30.0. Considering the large scale of Pick-a-Pic, we then randomly sample 7,500 pairs as candidates for the following human quality validation phase.
\item [2)] As discussed in Sec.~\ref{sec:intro}, T2I-G models may struggle with synthesizing knowledge-intensive images. To identify challenging concepts in the above six knowledge-intensive datasets, which pose difficulties for current state-of-the-art T2I-G models, we first employ open-sourced models including the SD series~\cite{rombach2022high, podell2023sdxl} to generate images by feeding the textual prompts in candidates as conditional input. Subsequently, we calculate the average CLIP-T scores ($S_{gen}$) over images generated by multiple models for each prompt. We then calculate the difference between the scores of the ground-truth pair and the generated pair for each prompt, \ie, $\Delta = S_{gt} - S_{gen}$.
\item [3)] Finally, we select the top 1,000 unique instance pairs -- comprising 1,000 different prompts and 1,000 different images -- with the highest values of $\Delta$ for each knowledge dataset. The remaining examples form a new candidate set with 500 WHOOPS! instances, 7,500 Pick-a-Pic instances, and 1,000 instances for the six knowledge datasets.
\end{enumerate}
\subsection{Human Annotator Filtration}
To further improve the data quality of TIGeR-Bench, human annotators were employed to mark evaluate each text-image pair across three aspects: text, image, and pair. Specifically, as for each text-image pair, considerations include the conciseness and unambiguity of the text, the clarity and usefulness of the image, and the relevance of the text-image pair. Annotators assigned a score of 0 (not satisfied) or 1 (satisfied) for each aspect. Finally, only text-image pairs meeting satisfaction across all three aspects were retained.
\subsection{Data Sampling}
To strike a balance between adequacy and efficiency in evaluation, we retain all 500 samples in WHOOPS!, and further randomly sample 2,500 data instances from Pick-a-Pic, along with 500 instances from each knowledge-intensive dataset. The statistics of TIGeR-Bench are presented in Fig.~\ref{tab:data_statistics}. Maintaining a balanced ratio of $1:1$ between creative and knowledge domains, we finally obtain a total of 6,000 high-quality text-image pairs.
\section{Method Details}\label{app:model_details}
\subsection{Beam Search}
Beam search~\cite{graves2012sequence, boulanger2013audio} was originally proposed for decoding tokens in sequence-to-sequence models and widely used in neural machine translation~\cite{sutskever2014sequence}. Based on breadth-first search (BFS), it explores a search tree from the root to the leaves. At each level, beam search generates all possible child nodes based on the current prefixes, then sorts and selects the top-$B$ paths by their conditional likelihood. Unlike BFS, which considers all paths, beam search maintains only $B$ valid paths at each level and prunes others. As a result, it produces $B$ ranked sequences.
\subsection{Base Models}
In this work, we introduce and implement our approach for unified text-to-image generation and retrieval, based on two foundation MLLMs: SEED-LLaMA~\cite{ge2023making} and LaVIT~\cite{jin2023unified}. The details of these two models are as follows.
\textbf{SEED-LLaMA} produces 32 discrete visual codes for each image via the SEED tokenizer. This tokenizer is composed of a Causal Q-Former, a learnable codebook, and an MLP (only for training), and is trained with contrastive learning and reconstruction objectives. SEED-LLaMA takes discrete visual codes as input for multimodal understanding tasks such as image captioning and VQA, and outputs discrete visual codes. The output codes are then fed into the unCLIP-SD model~\cite{rombach2022high, ramesh2022hierarchical} to generate images.
\mysubsubsec{LaVIT} obtains discrete visual codes with variable lengths using a meticulously designed dynamic visual tokenizer, which comprises a token selector, a token merger, and a reconstruction decoder (used solely for training).
This tokenizer is trained with a reconstruction objective.
During tokenization, LaVIT samples a binary decision mask from a Gumbel distribution to select visual patches and quantize them into discrete tokens.
To ensure reproducibility and stability in tokenization, we depart from LaVIT and employ a deterministic selection method, where a patch is selected if its importance score exceeds a threshold of 0.5; otherwise, it is discarded.
With this discriminative tokenization strategy, we pre-tokenize the 6 knowledge-intensive datasets of TIGeR-Bench, resulting in average, maximum, and minimum lengths of discrete tokens at 88, 130, and 37, respectively.
During image generation, LaVIT first autoregressively produces a sequence of discrete visual tokens and then decodes them into an image using a diffusion model initialized with SD-v1.5~\cite{rombach2022high} or SDXL~\cite{podell2023sdxl}.
In contrast to SEED-LLaMA, which utilizes discrete visual tokens as input for multimodal understanding and generation, LaVIT takes continuous visual features from the token merger as input.
\section{Experimental Details}\label{app:exp_details}
\subsection{Baselines}
As shown in Tab.~\ref{tab:perfm_tiger}, we compare the proposed method with several baselines on TIGeR-Bench across three aspects, \ie, text-to-image generation, text-to-image retrieval, and unified generation and retrieval. We introduce these baselines in the following.
\begin{itemize}
\item \textbf{Text-to-Image Generation Baselines}: There include the expert model SDXL~\cite{podell2023sdxl} and recent MLLMs with image generation abilities. The MLLMs in this category are GILL~\cite{koh2023generating}, Emu~\cite{sun2023emu}, Emu 2~\cite{sun2023emu2}, DreamLLM~\cite{dong2023dreamllm}, SEED-LLaMA~\cite{ge2023making}, and LaVIT~\cite{jin2023unified}.
\item \textbf{Text-to-Image Retrieval Baselines}: These include the expert model CLIP (ViT-B/32)~\cite{radford2021learning} and recent MLLMs. Currently, GILL~\cite{koh2023generating} is the only MLLM with retrieval ability, which maps the embeddings of special visual tokens into the CLIP feature space. Although other MLLMs~\cite{sun2023emu, sun2023emu2, dong2023dreamllm, ge2023making, jin2023unified} do not directly support text-to-image retrieval, we evaluate them through a two-step process: 1) generating an image query conditioned on the text prompt, and 2) performing nearest neighbor search for image-to-image retrieval using the CLIP (ViT-B/32) image encoder as the feature extractor and cosine similarity as the metric.
\item \textbf{Unified Text-to-Image Generation and Retrieval}: GILL~\cite{koh2023generating} is the only baseline capable of performing both text-to-image generation and retrieval. It incorporates and trains a binary classifier to decide between generation and retrieval tasks.
\end{itemize}
\subsection{Implementation Details}\label{app:imp_details}
The proposed method is training-free and based on SEED-LLaMA~\cite{ge2023making} and LaVIT~\cite{jin2023unified}.
We utilize the 8B version of SEED-LLaMA and load the parameters of supervised fine-tuning. For LaVIT, we employ the 11B model with SDXL as the pixel decoder.
We combine all images in the 6 knowledge-intensive datasets and tokenize them into discrete tokens. Subsequently, we build the mapping between images and tokens. Based on these discrete tokens, we construct a Trie for efficient storage and constrained generation.
The beam size for retrieval is set to 800, and the timestep for generation is 25.
\section{Additional Experiments}
In this section, we carry out extensive experiments and obtain quantitative and qualitative results to explore the unified text-to-image generation and retrieval problem and the proposed MLLMs-based method.
\subsection{Additional Quantitative Results}
\subsubsection{Unified Performance Comparison}
To broadly compare the performance of baselines and our method for unified text-to-image generation and retrieval, we report the results with the CLIP-T score and the CLIP-I score as the evaluation metrics across 8 domains in TIGeR-Bench, in Tab.~\ref{tab:app_unified_clipt} and Tab.~\ref{tab:app_unified_clipi}, respective. In addition, we show the retrieval percentage of our method on 8 domains in TIGeR-Bench to understand the automatic decision in Tab.~\ref{tab:app_unified_retr_percentage}.
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Unified performance comparison on the \textit{CLIP-T score }across 8 domains in TIGeR-Bench. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{l|cc|cccccc|c}
\toprule
% \midrule
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
\color{mygray}{SDXL~\cite{podell2023sdxl}} & \color{mygray}{36.90} & \color{mygray}{30.09} & \color{mygray}{16.36} & \color{mygray}{24.96} & \color{mygray}{21.51} & \color{mygray}{26.00} & \color{mygray}{24.25} & \color{mygray}{20.99} & \color{mygray}{26.79} \\
\color{mygray}{CLIP (ViT-B/32)~\cite{radford2021learning}} & \color{mygray}{16.61} & \color{mygray}{15.86} & \color{mygray}{36.17} & \color{mygray}{34.95} & \color{mygray}{32.51} & \color{mygray}{30.56} & \color{mygray}{29.44} & \color{mygray}{31.00} & \color{mygray}{24.21} \\
GILL~\cite{koh2023generating} & 10.80 & 11.22 & 14.65 & 9.49 & 14.31 & 12.92 & 13.64 & 13.55 & 12.12 \\
Emu~\cite{sun2023emu} & 23.61 & 23.98 & 17.24 & 19.21 & 21.54 & 23.43 & 21.25 & 21.00 & 22.26 \\
Emu 2~\cite{sun2023emu2} & 29.49 & 26.21 & 23.67 & 20.85 & 19.56 & 26.09 & 20.53 & 19.81 & 24.25 \\
DreamLLM~\cite{sun2023emu2} & 27.16 & 23.47 & 25.57 & 25.13 & 24.27 & 23.31 & 20.78 & 24.20 & 23.98 \\
SEED-LLaMA~\cite{ge2023making} & 27.18 & 23.97 & 16.73 & 19.66 & 19.03 & 22.66 & 19.63 & 19.29 & 22.00 \\
LaVIT~\cite{jin2023unified} & 34.60 & 29.07 & 16.70 & 25.17 & 24.14 & 29.59 & 25.07 & 24.26 & 27.07 \\
\rowcolor{gray_bg}
SEED-LLaMA (Ours) & 27.16 & 23.47 & 25.57 & 25.13 & 24.27 & 23.31 & 20.78 & 24.20 & 23.98 \\
\rowcolor{gray_bg}
LaVIT (Ours) & 32.05 & 25.39 & 35.87 & 24.30 & 32.38 & 31.28 & 27.59 & 31.01 & 28.45 \\
\bottomrule
\end{tabular}}
\label{tab:app_unified_clipt}
\end{table}
\begin{table}[t!]
\centering
\caption{Unified performance comparison on the \textit{CLIP-I score} across 8 domains in TIGeR-Bench. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{l|cc|cccccc|c}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
\color{mygray}{SDXL~\cite{podell2023sdxl}} & \color{mygray}{65.91} & \color{mygray}{55.38} & \color{mygray}{14.21} & \color{mygray}{42.09} & \color{mygray}{35.43} & \color{mygray}{44.94} & \color{mygray}{45.60} & \color{mygray}{35.40} & \color{mygray}{46.71} \\
\color{mygray}{CLIP (ViT-B/32)~\cite{radford2021learning}} & \color{mygray}{31.33} & \color{mygray}{26.68} & \color{mygray}{93.55} & \color{mygray}{91.04} & \color{mygray}{71.54} & \color{mygray}{75.38} & \color{mygray}{71.39} & \color{mygray}{75.59} & \color{mygray}{53.60} \\
GILL~\cite{koh2023generating} & 15.93 & 13.61 & 20.38 & 15.96 & 14.34 & 18.25 & 15.49 & 14.52 & 15.25 \\
Emu~\cite{sun2023emu} & 43.17 & 43.95 & 22.23 & 34.25 & 38.53 & 46.97 & 48.56 & 35.97 & 40.78 \\
Emu 2~\cite{sun2023emu2} & 59.07 & 49.17 & 32.27 & 36.27 & 33.44 & 49.77 & 40.72 & 33.51 & 44.24 \\
DreamLLM~\cite{sun2023emu2} & 53.93 & 46.22 & 33.16 & 32.09 & 37.87 & 46.06 & 43.82 & 35.13 & 42.77 \\
SEED-LLaMA~\cite{ge2023making} & 52.76 & 49.37 & 18.50 & 37.89 & 33.78 & 45.46 & 46.55 & 34.50 & 43.02 \\
LaVIT~\cite{jin2023unified} & 65.79 & 53.64 & 20.87 & 42.20 & 39.77 & 53.66 & 52.48 & 42.05 & 48.75 \\
\rowcolor{gray_bg}
SEED-LLaMA (Ours) & 52.67 & 47.81 & 51.94 & 56.86 & 48.76 & 51.99 & 52.53 & 52.52 & 50.52 \\
\rowcolor{gray_bg}
LaVIT (Ours) & 60.80 & 46.56 & 92.68 & 57.32 & 72.88 & 74.70 & 69.79 & 75.44 & 61.37 \\
\bottomrule
\end{tabular}}
\label{tab:app_unified_clipi}
\end{table}
% \begin{table}[t!]
% \centering
% \caption{Authenticity evaluation by Gemini-Pro }
% \vspace{4pt}
% \resizebox{.30\columnwidth}{!}{
% \setlength\tabcolsep{8pt}
% \begin{tabular}{l|cc}
% \toprule
% & Food & Wiki \\
% \midrule
% SEED-LLaMA (Ours) & 1.0\% & 9.6\% \\
% SEED-LLaMA (Ours) & 1.0\% & 9.6\% \\
% \bottomrule
% \end{tabular}}
% \label{tab:authenticity_gemini}
% \end{table}
\begin{table}[t!]
\centering
\caption{Retrieval percentage of our method based on SEED-LLaMA and LaVIT on 8 domains in TIGeR-Bench. }
\vspace{4pt}
\resizebox{.92\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{l|cc|cccccc|c}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
SEED-LLaMA (Ours) & 1.0\% & 9.6\% & 44.2\% & 44.2\% & 53.6\% & 26.2\% & 40.2\% & 49.8\% & 25.6\% \\
LaVIT (Ours) & 15.0\% & 27.5\% & 99.8\% & 82.0\% & 88.6\% & 80.6\% & 87.6\% & 84.0\% & 56.3\% \\
\bottomrule
\end{tabular}}
\label{tab:app_unified_retr_percentage}
\end{table}
\subsubsection{Prompt/Query Extension}
In this section, we mainly study the influence of prompts on the retrieval and generation performance in knowledge-intensive scenarios. Toward this end, we first let SEED-LLaMA, Gemini Pro, and GPT-4o to explain the raw query with knowledge concepts from 6 domains. Subsequently, we concatenate the raw prompt and expanded ones to form new long text prompts and feed them to SEED-LLaMA. The results for text-to-image generation and retrieval are listed in Tab.~\ref{tab:app_query_expand_gen} and Tab.~\ref{tab:app_query_expand_retr}, respectively. We can see that prompt/query expansion with strong LLMs could promote both generation and retrieval performance. Meanwhile, weak LLMs may introduce false explanations and do harm to the generation and retrieval performance.
% Besides, we also investigate the generation and retrieval performance separately in multi-round chat scenarios. To construct such scenarios, we use the APIs of powerful Gemini Pro and GPT-4o to imagine chat contexts and then feed the context into SEED-LLaMA. Experimental results on these two tasks are reported in Tab.~\ref{tab:app_query_expand_chat}.
We also carry out experiments in multimodal chat scenarios to explore the interplay between generation and retrieval, as shown in \ref{tab:app_query_expand_mmchat}. Specifically, we concatenate the retrieved top-1 images behind the chat context and then evaluate the generation performance. Similarly, we concatenate the generated images behind the chat context and then evaluate the retrieval performance.
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Text-to-image \textit{generation} performance on TIGeR-Bench (Knowledge) in \textit{long text} scenarios, with CLIP-T and CLIP-I scores as evaluation metrics across various prompt/query expansion methods including Self-Expansion, Gemini Pro, and GPT-4o. For expansion, we guide LLMs to explain the appearance characteristics in detail with their expert language knowledge for given raw queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries as input for text-to-image generation. We perform generative retrieval with 200 beams. }
\vspace{4pt}
\resizebox{.85\columnwidth}{!}{
\setlength\tabcolsep{20pt}
\begin{tabular}{l|cc|cc}
\toprule
\multirow{2}{*}{Expansion Method} & \multicolumn{2}{c|}{Ours (SEED-LLaMA)} & \multicolumn{2}{c}{Ours (LaVIT)} \\
& CLIP-T & CLIP-I & CLIP-T & CLIP-I \\
\midrule
Raw Query & 19.50 & 36.11 & \textbf{24.16} & 41.84 \\
Self-Expansion & 18.07 & 35.12 & - & - \\
Gemini-Pro & 17.48 & 34.74 & 22.38 & 39.28 \\
GPT-4o & \textbf{20.36} & \textbf{38.95} & 23.91 & \textbf{42.59} \\
\bottomrule
\end{tabular}}
\label{tab:app_query_expand_gen}
\end{table}
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Text-to-image \textit{retrieval} performance on TIGeR-Bench (Knowledge) in \textit{long text} scenarios, with recall as the evaluation metric across various prompt/query expansion methods including Self-Expansion, Gemini Pro, and GPT-4o. For expansion, we guide LLMs to explain the appearance characteristics in detail with their expert language knowledge for given raw queries by giving them detailed instructions. After that, the queries can be expanded into longer texts and are combined with raw queries as input for text-to-image retrieval. We perform generative retrieval with 200 beams. }
\vspace{4pt}
\resizebox{.85\columnwidth}{!}{
\setlength\tabcolsep{15pt}
\begin{tabular}{l|ccc|ccc}
\toprule
\multirow{2}{*}{Expansion Method} & \multicolumn{3}{c|}{Ours (SEED-LLaMA)} & \multicolumn{3}{c}{Ours (LaVIT)} \\
& R@1 & R@5 & R@10 & R@1 & R@5 & R@10 \\
\midrule
Raw Query & 22.57 & 36.80 & 43.23 & \textbf{25.63} & 43.63 & 49.40 \\
Self-Expansion & 17.20 & 30.10 & 36.77 & - & - & - \\
Gemini-Pro & 18.57 & 34.27 & 40.30 & 19.07 & 36.80 & 43.10 \\
GPT-4o & \textbf{25.00} & \textbf{42.50} & \textbf{48.90} & 25.20 & \textbf{46.03} & \textbf{52.17} \\
\bottomrule
\end{tabular}}
\label{tab:app_query_expand_retr}
\end{table}
% \begin{table}[t!]
% \centering
% \caption{Text-to-image generation and retrieval performance on TIGeR-Bench (Knowledge) in \textit{chat} scenarios, across various chat generation methods including Gemini Pro and GPT-4o. We guide LLMs to imagine a scenario where a user intends to know a concept and asks for an image with their expert language knowledge for given raw queries by giving them detailed instructions and in-context examples. After that, the queries can be expanded into multi-round chat contexts as input for text-to-image generation and retrieval. Considering that LaVIT was not fine-tuned by chat instructions, we only carry out experiments based on SEED-LLaMA. We perform generative retrieval with 200 beams. }
% \vspace{4pt}
% \resizebox{.85\columnwidth}{!}{
% \setlength\tabcolsep{15pt}
% \begin{tabular}{l|cc|ccc}
% \toprule
% \multirow{2}{*}{Expansion Method} & \multicolumn{2}{c|}{Text-to-Image Generation} & \multicolumn{3}{c}{Text-to-Image Retrieval} \\
% & CLIP-T & CLIP-I & R@1 & R@5 & R@10 \\
% \midrule
% Raw Query & \textbf{19.50} & 36.11 & 22.57 & 36.80 & 43.23 \\
% Gemini-Pro & 17.71 & 34.58 & 17.83 & 31.70 & 36.77 \\
% GPT-4o & 19.40 & \textbf{38.17} & \textbf{24.03} & \textbf{40.73} & \textbf{47.83} \\
% \bottomrule
% \end{tabular}}
% \label{tab:app_query_expand_chat}
% \end{table}
\begin{table}[t!]
\centering
\caption{Text-to-image generation and retrieval performance on TIGeR-Bench (Knowledge) in \textit{multimodal chat} scenarios. Based on the chat contexts with pure text generated by GPT-4o, we can perform generation and retrieval. Afterwards, we concatenate the generated or retrieved top-1 images with the chat contexts and form the multimodal context, to explore the influence on retrieval and generation, respectively. Considering that LaVIT was not fine-tuned by chat instructions, we only carry out experiments based on SEED-LLaMA. We perform generative retrieval with 200 beams. }
\vspace{4pt}
\resizebox{.95\columnwidth}{!}{
\setlength\tabcolsep{12pt}
\begin{tabular}{ll|cc|ccc}
\toprule
\multirow{2}{*}{Expansion Method} & \multirow{2}{*}{Image Context} & \multicolumn{2}{c|}{Text-to-Image Generation} & \multicolumn{3}{c}{Text-to-Image Retrieval} \\
& & CLIP-T & CLIP-I & R@1 & R@5 & R@10 \\
\midrule
Raw Query & - & \textbf{19.50} & 36.11 & \textbf{22.57} & \textbf{36.80} & \textbf{43.23} \\
GPT-4o & Retrieved & 18.62 & \textbf{38.20} & - & - & - \\
GPT-4o & Generated & - & - & 15.87 & 29.13 & 35.60 \\
\bottomrule
\end{tabular}}
\label{tab:app_query_expand_mmchat}
\end{table}
\subsubsection{Ablation Study}
In this part, we conduct comprehensive ablation studies on the components of the proposed method to study the effectiveness.
First, we compare the alignment performance between separate generation, retrieval, and unified variants across all 8 domains. The results are listed in Tab.~\ref{tab:app_unfied_vs_single_8domains_clipt} and Tab.~\ref{tab:app_unfied_vs_single_8domains_clipi} with the CLIP-T and CLIP-I score as the evaluation protocols, respectively. Besides, although Flickr30K and MS-COCO are the general datasets describing daily common scenes, we also investigate the three variants on them, as shown in Tab.~\ref{tab:app_unfied_vs_single_f30k_coco}
Second, we further study the effects of the directions of re-ranking and decision-making, across 8 domains in Tab.~\ref{tab:app_abl_rrr_decision_8domains}, and 2 general datasets, \ie, Flickr30K and MS-COCO, in Tab.~\ref{tab:app_abl_rrr_decision_f30k_coco}.
In addition, we delve into the discriminative abilities by forward and reverse ranking methods, as well as forward beam search and reverse re-ranking in Tab.~\ref{tab:app_abl_rank_retr_6domains} on 6 knowledge-intensive domains.
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Comparison of \textit{CLIP-T score} between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on 8 domains of TIGeR. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{l|cc|cccccc|c}
\toprule
% \midrule
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
\multicolumn{10}{c}{\textit{Ours (SEED-LLaMA)}} \\
Generation & \textbf{27.18 } & \textbf{23.97 } & 16.73 & 19.66 & 19.03 & 22.66 & 19.63 & 19.29 & 22.00 \\
Retrieval & 11.04 & 10.36 & \textbf{27.87 } & \textbf{26.21 } & 23.73 & 20.25 & 19.56 & 22.99 & 16.95 \\
\rowcolor{gray_bg}
Unified & 27.16 & 23.47 & 25.57 & 25.13 & \textbf{24.27 } & \textbf{23.31 } & \textbf{20.78 } & \textbf{24.20 } & \textbf{23.98 } \\
\midrule
\multicolumn{10}{c}{\textit{Ours (LaVIT)}} \\
Generation & \textbf{34.60 } & \textbf{29.07 } & 16.70 & \textbf{25.17 } & 24.14 & 29.59 & 25.07 & 24.26 & 27.07 \\
Retrieval & 13.21 & 12.17 & 35.84 & 23.64 & \textbf{32.58 } & 31.08 & \textbf{27.61 } & 30.76 & 21.30 \\
\rowcolor{gray_bg}
Unified & 32.05 & 25.39 & \textbf{35.87 } & 24.30 & 32.38 & \textbf{31.28 } & 27.59 & \textbf{31.01 } & \textbf{28.45 } \\
\bottomrule
\end{tabular}}
\label{tab:app_unfied_vs_single_8domains_clipt}
\end{table}
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Comparison of \textit{CLIP-I score} between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on 8 domains of TIGeR-Bench. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{l|cc|cccccc|c}
\toprule
% \midrule
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
\multicolumn{10}{c}{\textit{Ours (SEED-LLaMA)}} \\
Generation & \textbf{27.18 } & \textbf{23.97 } & 16.73 & 19.66 & 19.03 & 22.66 & 19.63 & 19.29 & 22.00 \\
Retrieval & 11.04 & 10.36 & \textbf{27.87 } & \textbf{26.21 } & 23.73 & 20.25 & 19.56 & 22.99 & 16.95 \\
\rowcolor{gray_bg}
Unified & 27.16 & 23.47 & 25.57 & 25.13 & \textbf{24.27 } & \textbf{23.31 } & \textbf{20.78 } & \textbf{24.20 } & \textbf{23.98 } \\
\midrule
\multicolumn{10}{c}{\textit{Ours (LaVIT)}} \\
Generation & \textbf{34.60 } & \textbf{29.07 } & 16.70 & \textbf{25.17 } & 24.14 & 29.59 & 25.07 & 24.26 & 27.07 \\
Retrieval & 13.21 & 12.17 & 35.84 & 23.64 & \textbf{32.58 } & 31.08 & \textbf{27.61 } & 30.76 & 21.30 \\
\rowcolor{gray_bg}
Unified & 32.05 & 25.39 & \textbf{35.87 } & 24.30 & 32.38 & \textbf{31.28 } & 27.59 & \textbf{31.01 } & \textbf{28.45 } \\
\bottomrule
\end{tabular}}
\label{tab:app_unfied_vs_single_8domains_clipi}
\end{table}
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Comparison between the unified method and single generation and retrieval variants based on SEED-LLaMA and LaVIT on the Flickr30K and MS-COCO datasets. Performance is evaluated by the CLIP-T score. }
\vspace{4pt}
\resizebox{.45\columnwidth}{!}{
\setlength\tabcolsep{10pt}
\begin{tabular}{l|cc}
\toprule
Method & Flickr30K & MS-COCO \\
\midrule
\multicolumn{3}{c}{\textit{Ours (SEED-LLaMA)}} \\
% \midrule
Generation & 28.65 & 27.74 \\
Retrieval & 29.86 & 28.73 \\
Unified & 30.01 & 29.09 \\
\midrule
% \midrule
\multicolumn{3}{c}{\textit{Ours (LaVIT)}} \\
% \midrule
Generation & 37.05 & 35.59 \\
Retrieval & 27.54 & 27.13 \\
Unified & 33.69 & 32.24 \\
\bottomrule
\end{tabular}}
\label{tab:app_unfied_vs_single_f30k_coco}
\end{table}
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Ablation study on 8 domains of TIGeR-Bench investigating Reverse Re-Ranking (RRR) and two decision-making strategies, \ie, Forward with Eqn.~\ref{eqn:x_to_y_debias} and Reverse with Eqn.~\ref{eqn:y_to_x}. Performance is evaluated by the \textit{CLIP-T score}. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{8pt}
\begin{tabular}{cc|cc|cccccc|c}
\toprule
% \midrule
\multirow{2}{*}{RRR} & \multirow{2}{*}{Decision} & \multicolumn{2}{c|}{Creative Domains} & \multicolumn{6}{c|}{Knowledge Domains} & \multirow{2}{*}{All} \\
& & Counterfactual & Preference & Logo & News & Landmark & Nature & Food & Wiki & \\
\midrule
\multicolumn{10}{c}{\textit{Ours (SEED-LLaMA)}} \\
& Forward & 26.64 & 22.41 & 24.02 & 23.64 & 22.85 & 20.34 & 19.63 & 22.40 & 22.63 \\
\Checkmark & Forward & 27.14 & \textbf{23.95 } & 23.57 & 23.18 & 22.82 & 24.57 & 20.62 & 22.93 & 23.72 \\
& Reverse & \textbf{27.16 } & 23.47 & 25.57 & \textbf{25.13 } & \textbf{24.27 } & 23.31 & 20.78 & \textbf{24.20 } & \textbf{23.98 } \\
\Checkmark & Reverse & 26.81 & 20.19 & \textbf{27.29 } & 23.22 & 24.16 & \textbf{26.04 } & \textbf{21.79 } & 23.84 & 22.84 \\
\midrule
\multicolumn{10}{c}{\textit{Ours (LaVIT)}} \\
& Forward & \textbf{34.59 } & 29.00 & 16.84 & 25.53 & 24.74 & 29.56 & 24.93 & 25.05 & 27.19 \\
\Checkmark & Forward & 32.05 & \textbf{29.07 } & 17.15 & 25.57 & 24.76 & 29.60 & 25.17 & 25.20 & 27.28 \\
& Reverse & 33.83 & 28.17 & 24.67 & \textbf{28.88 } & 28.06 & 29.02 & 25.65 & 27.84 & 28.23 \\
\Checkmark & Reverse & 32.05 & 25.39 & \textbf{35.87 } & 24.30 & \textbf{32.38 } & \textbf{31.28 } & \textbf{27.59 } & \textbf{31.01 } & \textbf{28.45 } \\
\bottomrule
\end{tabular}}
\label{tab:app_abl_rrr_decision_8domains}
\end{table}
\begin{table}[t!]
\centering
\caption{Ablation study on Flickr30K and MSCOCO investigating Reverse Re-Ranking (RRR) and two decision-making strategies, \ie, Forward with Eqn.~\ref{eqn:x_to_y_debias} and Reverse with Eqn.~\ref{eqn:y_to_x}. \%Retr. denotes the percentage of retrieved images selected as results. }
\resizebox{.89\columnwidth}{!}{
\setlength\tabcolsep{12pt}
\begin{tabular}{cc|ccc|ccc}
\toprule
\multirow{2}{*}{RRR} & \multirow{2}{*}{Decision} & \multicolumn{3}{c|}{Flickr30K} & \multicolumn{3}{c}{MS-COCO} \\
& & CLIP-T~$\uparrow$ & R@1~$\uparrow$ & \%Retr. & CLIP-T~$\uparrow$ & R@1~$\uparrow$ & \%Retr. \\
\midrule
\multicolumn{8}{c}{\textit{Ours (SEED-LLaMA)}} \\
& Forward & 28.89 & 58.50 & 39.52 & 25.95 & 26.17 & 67.61 \\
\Checkmark & Forward & 29.68 & 71.70 & 26.92 & 28.71 & 46.11 & 33.91 \\
& Reverse & 30.01 & 58.50 & 35.98 & 28.61 & 26.17 & 26.23 \\
\Checkmark & Reverse & \textbf{30.02} & 71.70 & 51.88 & \textbf{29.09} & 46.11 & 60.69 \\
\midrule
\multicolumn{8}{c}{\textit{Ours (LaVIT)}} \\
& Forward & 37.03 & 47.86 & 0.20 & 35.34 & 23.20 & 3.44 \\
\Checkmark & Forward & \textbf{37.04} & 68.84 & 0.10 & \textbf{35.58} & 44.81 & 0.23 \\
& Reverse & 36.18 & 47.86 & 24.34 & 34.67 & 23.20 & 20.60 \\
\Checkmark & Reverse & 33.69 & 68.84 & 41.84 & 32.24 & 44.81 & 49.11 \\
\bottomrule
\end{tabular}}
\label{tab:app_abl_rrr_decision_f30k_coco}
% \end{wraptable}
\end{table}
\begin{table}[t!]
\centering
% \vspace{-20pt}
\caption{Recall@1 performance comparison of Forward Ranking, Reverse Ranking, Forward Beam Search (FBS) with different beam sizes, and BFS + Reverse Re-Ranking (RRR). Experiments are conducted based on SEED-LLaMA and LaVIT on 6 knowledge-intensive domains of TIGeR-Bench. }
\vspace{4pt}
\resizebox{.98\columnwidth}{!}{
\setlength\tabcolsep{11pt}
\begin{tabular}{l|cccccc|c}
\toprule
% \midrule
Method & Logo & News & Landmark & Nature & Food & Wiki & ALL \\
\midrule
\multicolumn{8}{c}{\textit{Ours (SEED-LLaMA)}} \\
Forward Ranking & 56.00 & 45.60 & 22.60 & 2.60 & 4.40 & 30.40 & 26.93 \\
Reverse Ranking & 61.80 & 40.40 & 26.60 & 15.00 & 7.00 & 32.40 & 30.53 \\
FBS (\#Beam=100) & 39.40 & 29.60 & 10.40 & 8.80 & 4.60 & 19.40 & 18.70 \\
FBS (\#Beam=800) & 56.00 & 46.80 & 20.20 & 3.60 & 4.60 & 29.60 & 26.80 \\
FBS (\#Beam=100) + RRR & 37.40 & 25.80 & 10.00 & 10.20 & 6.00 & 18.60 & 18.00 \\
FBS (\#Beam=800) + RRR & 61.20 & 39.60 & 22.60 & 15.40 & 6.80 & 29.80 & 29.23 \\
\midrule
\multicolumn{8}{c}{\textit{Ours (LaVIT)}} \\
Forward Ranking & 37.80 & 52.20 & 25.00 & 10.00 & 11.40 & 33.20 & 28.27 \\
Reverse Ranking & 92.20 & 41.00 & 56.00 & 36.40 & 23.60 & 63.80 & 52.17 \\
FBS (\#Beam=100) & 16.60 & 46.00 & 20.20 & 9.40 & 10.00 & 28.40 & 21.77 \\
FBS (\#Beam=800) & 37.00 & 53.40 & 24.40 & 10.80 & 11.40 & 31.80 & 28.13 \\
FBS (\#Beam=100) + RRR & 27.00 & 30.80 & 30.60 & 16.80 & 15.40 & 41.00 & 26.93 \\
FBS (\#Beam=800) + RRR & 87.20 & 40.40 & 51.60 & 34.60 & 23.00 & 59.40 & 49.37 \\
\bottomrule
\end{tabular}}
\label{tab:app_abl_rank_retr_6domains}
\end{table}
\newpage
\subsection{Additional Qualitative Results}
We showcase more examples of our SEED-LLaMA and LaVIT in both creative and knowledge-intensive domains in Fig.~\ref{fig:creative} and Fig.~\ref{fig:knowledge}.
In the creative domain, the CLIP model, limited to retrieving images from the database, shows significant discrepancies when compared to the ground truth images. Our SEED-LLaMA and LaVIT, capable of both generation and retrieval, tend to favor image generation in the creative domain. However, our models also exhibit decision errors. For instance, as demonstrated in the last two rows of Fig.~\ref{fig:creative}, the models incorrectly selected misleading retrieved images.
As shown in Fig.~\ref{fig:creative}, our models has the advantages over SDXL in the knowledge-intensive domain, accurately retrieving the correct results. However, decision errors still occur. We leave further exploration of the decision strategy for future work.
In Fig.~\ref{fig:mllm}, we compare our models with current Text-to-Image baseline models such as Emu2, DreamLLM, and GILL, which can autonomously decide between retrieval and generation. Our models are consistently retrieving the correct images in the knowledge-intensive domain. In this domain, Emu2, DreamLLM, and GILL fail to generate closely matching images, highlighting the limitations of current MLLMs.
We further explored two scenarios: Augmented Generation for Better Retrieval and Augmented Retrieval for Better Generation. In the Augmented Generation for Better Retrieval scenario, we first use the MLLM's capability to generate an image before performing image retrieval. The generated image, along with the retrieval prompt, is then used as input for the retrieval process. As shown in Fig.~\ref{fig:Gen2Ret}, generating an image beforehand improves the model's retrieval performance.
In the Augmented Retrieval for Better Generation scenario, we leverage our model's generative retrieval capabilities to perform an image retrieval before generating an image. The retrieved image, along with the generation prompt, is then used as input for the generation process, similar to Retrieval-Augmented Generation (RAG). As shown in Fig.~\ref{fig:Ret2Gen}, performing image retrieval beforehand improves the stability and quality of the generated images.
\begin{figure}
\centering
\includegraphics[width=\textwidth]{fig/Creative_Domain.pdf}
\caption{Qualitative results in TIGeR-Bench creative domain.
We use ticks or crosses to highlight the selected results from generation or retrieval.
\textcolor{green}{Green} ticks indicate the correct generated images and \textcolor{red}{red} crosses indicate the wrong retrieved images. Black ticks refer to the correct retrieved images despite the creative domain. }
\label{fig:creative}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{fig/Knowledge_Domain.pdf}
\caption{Qualitative results in TIGeR-Bench knowledge domain.
We use ticks or crosses to highlight the selected results from generation or retrieval.
\textcolor{green}{Green} ticks indicate the correct retrieved images and \textcolor{red}{red} crosses indicate the wrong generated images.}
\label{fig:knowledge}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{fig/MLLM_Showcase.pdf}
\caption{Qualitative results in TIGeR-Bench knowledge domain compared to Text-to-Image generation baselines. \textcolor{green}{Green} tick indicates the final output by our model is correct and \textcolor{red}{Red} cross indicates the final output by our model is incorrect.}
\label{fig:mllm}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{fig/Gen2Ret.pdf}
\caption{Augmented generation for better retrieval. \textcolor{green}{Green} box is the additional generation step. \textcolor{green}{Green} tick indicates the final output by our model is correct and \textcolor{red}{Red} cross indicates the fina3l output by our model is incorrect.}
\label{fig:Gen2Ret}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{fig/Ret2Gen.pdf}
\caption{Augmented retrieval for better generation. \textcolor{green}{Green} box is the additional retrieval step. \textcolor{green}{Green} tick indicates the final output by our model is consistent and \textcolor{red}{Red} cross indicates the final output by our model is inconsistent.}
\label{fig:Ret2Gen}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{fig/TIGeR_long_context_v1.pdf}
\caption{Short prompt and long prompt retrieval comparison on TIGeR-Bench knowledge domain. \textcolor{red}{Red} box is the retrieve result of short prompt. \textcolor{green}{Green} box is the retrieve result of long prompt. \textcolor{green}{Green} tick indicates the final output by our model is correct and \textcolor{red}{Red} cross indicates the final output by our model is incorrect.}
\label{fig:long_prompt}
\end{figure}
One major limitation of the CLIP model for retrieval is its limited context length. Our model leverages the advantage of the LLM's long context length, making retrieve with longer prompts possible. As shown in Fig.~\ref{fig:long_prompt}, extending the prompt further enhances the retrieval performance of our model.
\section{Future Work}
In the future, we plan to investigate the root causes of modality biases from various perspectives, including data distribution, model architecture, and optimization objectives. We will also examine the potential impacts of these biases on generative and discriminative tasks~\cite{li2023transformer, li2022invariant, qu2020context}. Additionally, we aim to study more complex contexts involving interleaved multimodal content to advance comprehensive unified generation and retrieval tasks. Finally, it would be valuable to explore the deeper relationships and possible interactions between generation and retrieval (e.g., retrieval-augmented generation and generation-augmented retrieval) within the TIGeR framework.
\begin{thebibliography}{89}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Betker et~al.(2023)Betker, Goh, Jing, Brooks, Wang, Li, Ouyang,
Zhuang, Lee, Guo, et~al.]{betker2023improving}
James Betker, Gabriel Goh, Li~Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long
Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et~al.
\newblock Improving image generation with better captions.
\newblock \emph{Computer Science. https://cdn. openai. com/papers/dall-e-3.
pdf}, 2\penalty0 (3):\penalty0 8, 2023.
\bibitem[Bitton-Guetta et~al.(2023)Bitton-Guetta, Bitton, Hessel, Schmidt,
Elovici, Stanovsky, and Schwartz]{bitton2023breaking}
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval
Elovici, Gabriel Stanovsky, and Roy Schwartz.
\newblock Breaking common sense: Whoops! a vision-and-language benchmark of
synthetic and compositional images.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pp.\ 2616--2627, 2023.
\bibitem[Boulanger-Lewandowski et~al.(2013)Boulanger-Lewandowski, Bengio, and
Vincent]{boulanger2013audio}
Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.
\newblock Audio chord recognition with recurrent neural networks.
\newblock In \emph{ISMIR}, pp.\ 335--340. Curitiba, 2013.
\bibitem[Chefer et~al.(2023)Chefer, Alaluf, Vinker, Wolf, and
Cohen-Or]{chefer2023attend}
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.
\newblock Attend-and-excite: Attention-based semantic guidance for
text-to-image diffusion models.
\newblock \emph{ACM Transactions on Graphics (TOG)}, 42\penalty0 (4):\penalty0
1--10, 2023.
\bibitem[Chen et~al.(2022)Chen, Hu, Saharia, and Cohen]{chen2022re}
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William~W Cohen.
\newblock Re-imagen: Retrieval-augmented text-to-image generator.
\newblock \emph{arXiv preprint arXiv:2209.14491}, 2022.
\bibitem[Clark \& Jaini(2023)Clark and Jaini]{clark2023text}
Kevin Clark and Priyank Jaini.
\newblock Text-to-image diffusion models are zero-shot classifiers.
\newblock \emph{arXiv preprint arXiv:2303.15233}, 2023.
\bibitem[Das et~al.(2017)Das, Kottur, Gupta, Singh, Yadav, Moura, Parikh, and
Batra]{das2017visual}
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav,
Jos{\'e}~MF Moura, Devi Parikh, and Dhruv Batra.
\newblock Visual dialog.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 326--335, 2017.
\bibitem[Ding et~al.(2021)Ding, Yang, Hong, Zheng, Zhou, Yin, Lin, Zou, Shao,
Yang, et~al.]{ding2021cogview}
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da~Yin, Junyang
Lin, Xu~Zou, Zhou Shao, Hongxia Yang, et~al.
\newblock Cogview: Mastering text-to-image generation via transformers.
\newblock \emph{Advances in Neural Information Processing Systems},
34:\penalty0 19822--19835, 2021.
\bibitem[Dong et~al.(2023)Dong, Han, Peng, Qi, Ge, Yang, Zhao, Sun, Zhou, Wei,
et~al.]{dong2023dreamllm}
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang
Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et~al.
\newblock Dreamllm: Synergistic multimodal comprehension and creation.
\newblock \emph{arXiv preprint arXiv:2309.11499}, 2023.
\bibitem[Dou et~al.(2022)Dou, Xu, Gan, Wang, Wang, Wang, Zhu, Zhang, Yuan,
Peng, et~al.]{dou2022empirical}
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang,
Chenguang Zhu, Pengchuan Zhang, Lu~Yuan, Nanyun Peng, et~al.
\newblock An empirical study of training end-to-end vision-and-language
transformers.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 18166--18176, 2022.
\bibitem[Esser et~al.(2021)Esser, Rombach, and Ommer]{esser2021taming}
Patrick Esser, Robin Rombach, and Bjorn Ommer.
\newblock Taming transformers for high-resolution image synthesis.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pp.\ 12873--12883, 2021.
\bibitem[Esser et~al.(2024)Esser, Kulal, Blattmann, Entezari, M{\"u}ller,
Saini, Levi, Lorenz, Sauer, Boesel, et~al.]{esser2024scaling}
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas
M{\"u}ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic
Boesel, et~al.
\newblock Scaling rectified flow transformers for high-resolution image
synthesis.
\newblock \emph{arXiv preprint arXiv:2403.03206}, 2024.
\bibitem[Faghri et~al.(2017)Faghri, Fleet, Kiros, and Fidler]{faghri2017vse++}
Fartash Faghri, David~J Fleet, Jamie~Ryan Kiros, and Sanja Fidler.
\newblock Vse++: Improving visual-semantic embeddings with hard negatives.
\newblock \emph{arXiv preprint arXiv:1707.05612}, 2017.
\bibitem[Feng et~al.(2022)Feng, He, Fu, Jampani, Akula, Narayana, Basu, Wang,
and Wang]{feng2022training}
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna
Narayana, Sugato Basu, Xin~Eric Wang, and William~Yang Wang.
\newblock Training-free structured diffusion guidance for compositional
text-to-image synthesis.
\newblock \emph{arXiv preprint arXiv:2212.05032}, 2022.
\bibitem[Freitag \& Al-Onaizan(2017)Freitag and Al-Onaizan]{freitag2017beam}
Markus Freitag and Yaser Al-Onaizan.
\newblock Beam search strategies for neural machine translation.
\newblock \emph{arXiv preprint arXiv:1702.01806}, 2017.
\bibitem[Ge et~al.(2023)Ge, Zhao, Zeng, Ge, Li, Wang, and Shan]{ge2023making}
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying
Shan.
\newblock Making llama see and draw with seed tokenizer.
\newblock \emph{arXiv preprint arXiv:2310.01218}, 2023.
\bibitem[Grathwohl et~al.(2019)Grathwohl, Wang, Jacobsen, Duvenaud, Norouzi,
and Swersky]{grathwohl2019your}
Will Grathwohl, Kuan-Chieh Wang, J{\"o}rn-Henrik Jacobsen, David Duvenaud,
Mohammad Norouzi, and Kevin Swersky.
\newblock Your classifier is secretly an energy based model and you should
treat it like one.
\newblock \emph{arXiv preprint arXiv:1912.03263}, 2019.
\bibitem[Graves(2012)]{graves2012sequence}
Alex Graves.
\newblock Sequence transduction with recurrent neural networks.
\newblock \emph{arXiv preprint arXiv:1211.3711}, 2012.
\bibitem[He et~al.(2023)He, Feng, Fu, Jampani, Akula, Narayana, Basu, Wang, and
Wang]{he2023discriminative}
Xuehai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna
Narayana, Sugato Basu, William~Yang Wang, and Xin~Eric Wang.
\newblock Discriminative diffusion models as few-shot vision and language
learners.
\newblock \emph{arXiv preprint arXiv:2305.10722}, 2023.
\bibitem[Hessel et~al.(2021)Hessel, Holtzman, Forbes, Bras, and
Choi]{hessel2021clipscore}
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan~Le Bras, and Yejin Choi.
\newblock Clipscore: A reference-free evaluation metric for image captioning.
\newblock \emph{arXiv preprint arXiv:2104.08718}, 2021.
\bibitem[Ho et~al.(2020)Ho, Jain, and Abbeel]{ho2020denoising}
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
\newblock Denoising diffusion probabilistic models.
\newblock \emph{Advances in neural information processing systems},
33:\penalty0 6840--6851, 2020.
\bibitem[Huang et~al.(2024{\natexlab{a}})Huang, Wang, Bitton, Taitelbaum,
Tomar, Chang, Jia, Chan, Hu, Su, et~al.]{huang2024kitten}
Hsin-Ping Huang, Xinyi Wang, Yonatan Bitton, Hagai Taitelbaum, Gaurav~Singh
Tomar, Ming-Wei Chang, Xuhui Jia, Kelvin~CK Chan, Hexiang Hu, Yu-Chuan Su,
et~al.
\newblock Kitten: A knowledge-intensive evaluation of image generation on
visual entities.
\newblock \emph{arXiv preprint arXiv:2410.11824}, 2024{\natexlab{a}}.
\bibitem[Huang et~al.(2023)Huang, Sun, Xie, Li, and Liu]{huang2023t2i}
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.
\newblock T2i-compbench: A comprehensive benchmark for open-world compositional
text-to-image generation.
\newblock \emph{Advances in Neural Information Processing Systems},
36:\penalty0 78723--78747, 2023.
\bibitem[Huang et~al.(2024{\natexlab{b}})Huang, Liu, Guo, and
Gong]{huang2024visual}
Wen Huang, Hongbin Liu, Minxin Guo, and Neil~Zhenqiang Gong.
\newblock Visual hallucinations of multi-modal large language models.
\newblock \emph{arXiv preprint arXiv:2402.14683}, 2024{\natexlab{b}}.
\bibitem[Jin et~al.(2023)Jin, Xu, Xu, Chen, Liao, Tan, Huang, Chen, Lei, Liu,
et~al.]{jin2023unified}
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin
Chen, Chenyi Lei, An~Liu, et~al.
\newblock Unified language-vision pretraining in llm with dynamic discrete
visual tokenization.
\newblock \emph{arXiv e-prints}, pp.\ arXiv--2309, 2023.
\bibitem[Kaplan et~al.(2020)Kaplan, McCandlish, Henighan, Brown, Chess, Child,
Gray, Radford, Wu, and Amodei]{kaplan2020scaling}
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom~B Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
\newblock Scaling laws for neural language models.
\newblock \emph{arXiv preprint arXiv:2001.08361}, 2020.
\bibitem[Kim et~al.(2024)Kim, Jin, Diethe, Figini, Tregidgo, Mullokandov,
Teare, and Alexander]{kim2024tackling}
Seunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry~FJ Tregidgo, Asher
Mullokandov, Philip Teare, and Daniel~C Alexander.
\newblock Tackling structural hallucination in image translation with local
diffusion.
\newblock \emph{arXiv preprint arXiv:2404.05980}, 2024.
\bibitem[Kirstain et~al.(2024)Kirstain, Polyak, Singer, Matiana, Penna, and
Levy]{kirstain2024pick}
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and
Omer Levy.
\newblock Pick-a-pic: An open dataset of user preferences for text-to-image
generation.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Koh et~al.(2023)Koh, Fried, and Salakhutdinov]{koh2023generating}
Jing~Yu Koh, Daniel Fried, and Ruslan Salakhutdinov.
\newblock Generating images with multimodal language models.
\newblock \emph{arXiv preprint arXiv:2305.17216}, 2023.
\bibitem[Krojer et~al.(2023)Krojer, Poole-Dayan, Voleti, Pal, and
Reddy]{krojer2023diffusion}
Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, and Siva
Reddy.
\newblock Are diffusion models vision-and-language reasoners?
\newblock In \emph{Thirty-seventh Conference on Neural Information Processing
Systems}, 2023.
\bibitem[Lee et~al.(2023)Lee, Liu, Ryu, Watkins, Du, Boutilier, Abbeel,
Ghavamzadeh, and Gu]{lee2023aligning}
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier,
Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang~Shane Gu.
\newblock Aligning text-to-image models using human feedback.
\newblock \emph{arXiv preprint arXiv:2302.12192}, 2023.
\bibitem[Lee et~al.(2018)Lee, Chen, Hua, Hu, and He]{lee2018stacked}
Kuang-Huei Lee, Xi~Chen, Gang Hua, Houdong Hu, and Xiaodong He.
\newblock Stacked cross attention for image-text matching.
\newblock In \emph{Proceedings of the European conference on computer vision
(ECCV)}, pp.\ 201--216, 2018.
\bibitem[Levy et~al.(2024)Levy, Ben-Ari, Darshan, and
Lischinski]{levy2024chatting}
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski.
\newblock Chatting makes perfect: Chat-based image retrieval.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Li et~al.(2023{\natexlab{a}})Li, Prabhudesai, Duggal, Brown, and
Pathak]{li2023your}
Alexander~C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak
Pathak.
\newblock Your diffusion model is secretly a zero-shot classifier.
\newblock \emph{arXiv preprint arXiv:2303.16203}, 2023{\natexlab{a}}.
\bibitem[Li et~al.(2024{\natexlab{a}})Li, Zou, Wang, Majumder, Xie, Manmatha,
Swaminathan, Tu, Ermon, and Soatto]{li2024scalability}
Hao Li, Yang Zou, Ying Wang, Orchid Majumder, Yusheng Xie, R~Manmatha, Ashwin
Swaminathan, Zhuowen Tu, Stefano Ermon, and Stefano Soatto.
\newblock On the scalability of diffusion-based text-to-image generation.
\newblock \emph{arXiv preprint arXiv:2404.02883}, 2024{\natexlab{a}}.
\bibitem[Li et~al.(2015)Li, Galley, Brockett, Gao, and Dolan]{li2015diversity}
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.
\newblock A diversity-promoting objective function for neural conversation
models.
\newblock \emph{arXiv preprint arXiv:1510.03055}, 2015.
\bibitem[Li et~al.(2021)Li, Selvaraju, Gotmare, Joty, Xiong, and
Hoi]{li2021align}
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong,
and Steven Chu~Hong Hoi.
\newblock Align before fuse: Vision and language representation learning with
momentum distillation.
\newblock \emph{Advances in neural information processing systems},
34:\penalty0 9694--9705, 2021.
\bibitem[Li et~al.(2023{\natexlab{b}})Li, Li, Savarese, and Hoi]{li2023blip}
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
\newblock Blip-2: Bootstrapping language-image pre-training with frozen image
encoders and large language models.
\newblock \emph{arXiv preprint arXiv:2301.12597}, 2023{\natexlab{b}}.
\bibitem[Li et~al.(2019)Li, Zhang, Li, Li, and Fu]{li2019visual}
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu.
\newblock Visual semantic reasoning for image-text matching.
\newblock In \emph{Proceedings of the IEEE/CVF international conference on
computer vision}, pp.\ 4654--4662, 2019.
\bibitem[Li et~al.(2022)Li, Wang, Xiao, Ji, and Chua]{li2022invariant}
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua.
\newblock Invariant grounding for video question answering.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 2928--2937, 2022.
\bibitem[Li et~al.(2023{\natexlab{c}})Li, Wang, Xiao, Ji, and
Chua]{li2023transformer}
Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua.
\newblock Transformer-empowered invariant grounding for video question
answering.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
Intelligence}, 2023{\natexlab{c}}.
\bibitem[Li et~al.(2024{\natexlab{b}})Li, Wang, Qu, Nie, Li, and
Chua]{li2024generative}
Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua.
\newblock Generative cross-modal retrieval: Memorizing images in multimodal
language models for retrieval and beyond.
\newblock \emph{arXiv preprint arXiv:2402.10805}, 2024{\natexlab{b}}.
\bibitem[Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan,
Doll{\'a}r, and Zitnick]{lin2014microsoft}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In \emph{Computer Vision--ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13}, pp.\
740--755. Springer, 2014.
\bibitem[Lin et~al.(2024{\natexlab{a}})Lin, Mei, Chen, and
Byrne]{lin2024preflmr}
Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byrne.
\newblock Preflmr: Scaling up fine-grained late-interaction multi-modal
retrievers.
\newblock \emph{arXiv preprint arXiv:2402.08327}, 2024{\natexlab{a}}.
\bibitem[Lin et~al.(2024{\natexlab{b}})Lin, Chen, Pathak, Zhang, and
Ramanan]{lin2024revisiting}
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan.
\newblock Revisiting the role of language priors in vision-language models.
\newblock In \emph{International Conference on Machine Learning},
2024{\natexlab{b}}.
\bibitem[Liu et~al.(2020)Liu, Wang, Wang, and Ordonez]{liu2020visual}
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.
\newblock Visual news: Benchmark and challenges in news image captioning.
\newblock \emph{arXiv preprint arXiv:2010.03743}, 2020.
\bibitem[Liu et~al.(2023)Liu, Li, Wu, and Lee]{liu2023visual}
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee.
\newblock Visual instruction tuning.
\newblock \emph{arXiv preprint arXiv:2304.08485}, 2023.
\bibitem[Lu et~al.(2023)Lu, Clark, Lee, Zhang, Khosla, Marten, Hoiem, and
Kembhavi]{lu2023unified}
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan
Marten, Derek Hoiem, and Aniruddha Kembhavi.
\newblock Unified-io 2: Scaling autoregressive multimodal models with vision,
language, audio, and action.
\newblock \emph{arXiv preprint arXiv:2312.17172}, 2023.
\bibitem[Min et~al.(2023)Min, Wang, Liu, Luo, Kang, Wei, Wei, and
Jiang]{min2023large}
Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei,
Xiaolin Wei, and Shuqiang Jiang.
\newblock Large scale visual food recognition.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
Intelligence}, 2023.
\bibitem[Nichol et~al.(2021)Nichol, Dhariwal, Ramesh, Shyam, Mishkin, McGrew,
Sutskever, and Chen]{nichol2021glide}
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin,
Bob McGrew, Ilya Sutskever, and Mark Chen.
\newblock Glide: Towards photorealistic image generation and editing with
text-guided diffusion models.
\newblock \emph{arXiv preprint arXiv:2112.10741}, 2021.
\bibitem[OpenAI(2024)]{4oapi}
OpenAI.
\newblock Introducing gpt-4o: our fastest and most affordable flagship model.
\newblock \url{https://platform.openai.com/docs/guides/vision}, 2024.
\newblock Accessed: 2024-05-26.
\bibitem[Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin,
Zhang, Agarwal, Slama, Ray, et~al.]{ouyang2022training}
Long Ouyang, Jeffrey Wu, Xu~Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et~al.
\newblock Training language models to follow instructions with human feedback.
\newblock \emph{Advances in neural information processing systems},
35:\penalty0 27730--27744, 2022.
\bibitem[Podell et~al.(2023)Podell, English, Lacey, Blattmann, Dockhorn,
M{\"u}ller, Penna, and Rombach]{podell2023sdxl}
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas
M{\"u}ller, Joe Penna, and Robin Rombach.
\newblock Sdxl: Improving latent diffusion models for high-resolution image
synthesis.
\newblock \emph{arXiv preprint arXiv:2307.01952}, 2023.
\bibitem[Qu et~al.(2020)Qu, Liu, Cao, Nie, and Tian]{qu2020context}
Leigang Qu, Meng Liu, Da~Cao, Liqiang Nie, and Qi~Tian.
\newblock Context-aware multi-view summarization network for image-text
matching.
\newblock In \emph{Proceedings of the 28th ACM international conference on
multimedia}, pp.\ 1047--1055, 2020.
\bibitem[Qu et~al.(2021)Qu, Liu, Wu, Gao, and Nie]{qu2021dynamic}
Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie.
\newblock Dynamic modality interaction modeling for image-text retrieval.
\newblock In \emph{Proceedings of the 44th International ACM SIGIR Conference
on Research and Development in Information Retrieval}, pp.\ 1104--1113,
2021.
\bibitem[Qu et~al.(2023)Qu, Wu, Fei, Nie, and Chua]{qu2023layoutllm}
Leigang Qu, Shengqiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua.
\newblock Layoutllm-t2i: Eliciting layout guidance from llm for text-to-image
generation.
\newblock In \emph{Proceedings of the 31st ACM International Conference on
Multimedia}, pp.\ 643--654, 2023.
\bibitem[Qu et~al.(2024)Qu, Wang, Li, Zhang, Nie, and
Chua]{qu2024discriminative}
Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, and Tat-Seng
Chua.
\newblock Discriminative probing and tuning for text-to-image generation.
\newblock \emph{arXiv preprint arXiv:2403.04321}, 2024.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal,
Sastry, Askell, Mishkin, Clark, et~al.]{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et~al.
\newblock Learning transferable visual models from natural language
supervision.
\newblock In \emph{International conference on machine learning}, pp.\
8748--8763. PMLR, 2021.
\bibitem[Ramesh et~al.(2021)Ramesh, Pavlov, Goh, Gray, Voss, Radford, Chen, and
Sutskever]{ramesh2021zero}
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec
Radford, Mark Chen, and Ilya Sutskever.
\newblock Zero-shot text-to-image generation.
\newblock In \emph{International Conference on Machine Learning}, pp.\
8821--8831. PMLR, 2021.
\bibitem[Ramesh et~al.(2022)Ramesh, Dhariwal, Nichol, Chu, and
Chen]{ramesh2022hierarchical}
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
\newblock Hierarchical text-conditional image generation with clip latents.
\newblock \emph{arXiv preprint arXiv:2204.06125}, 1\penalty0 (2):\penalty0 3,
2022.
\bibitem[Reed et~al.(2016)Reed, Akata, Yan, Logeswaran, Schiele, and
Lee]{reed2016generative}
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and
Honglak Lee.
\newblock Generative adversarial text to image synthesis.
\newblock In \emph{International conference on machine learning}, pp.\
1060--1069. PMLR, 2016.
\bibitem[Reid et~al.(2024)Reid, Savinov, Teplyashin, Lepikhin, Lillicrap,
Alayrac, Soricut, Lazaridou, Firat, Schrittwieser, et~al.]{reid2024gemini}
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy
Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan
Firat, Julian Schrittwieser, et~al.
\newblock Gemini 1.5: Unlocking multimodal understanding across millions of
tokens of context.
\newblock \emph{arXiv preprint arXiv:2403.05530}, 2024.
\bibitem[Role \& Nadif(2011)Role and Nadif]{role2011handling}
Fran{\c{c}}ois Role and Mohamed Nadif.
\newblock Handling the impact of low frequency events on co-occurrence based
measures of word similarity-a case study of pointwise mutual information.
\newblock In \emph{International Conference on Knowledge Discovery and
Information Retrieval}, volume~2, pp.\ 218--223. Scitepress, 2011.
\bibitem[Rombach et~al.(2022)Rombach, Blattmann, Lorenz, Esser, and
Ommer]{rombach2022high}
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj{\"o}rn
Ommer.
\newblock High-resolution image synthesis with latent diffusion models.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pp.\ 10684--10695, 2022.
\bibitem[Saharia et~al.(2022)Saharia, Chan, Saxena, Li, Whang, Denton,
Ghasemipour, Gontijo~Lopes, Karagol~Ayan, Salimans,
et~al.]{saharia2022photorealistic}
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily~L
Denton, Kamyar Ghasemipour, Raphael Gontijo~Lopes, Burcu Karagol~Ayan, Tim
Salimans, et~al.
\newblock Photorealistic text-to-image diffusion models with deep language
understanding.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 36479--36494, 2022.
\bibitem[Schuhmann et~al.(2022)Schuhmann, Beaumont, Vencu, Gordon, Wightman,
Cherti, Coombes, Katta, Mullis, Wortsman, et~al.]{schuhmann2022laion}
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et~al.
\newblock Laion-5b: An open large-scale dataset for training next generation
image-text models.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 25278--25294, 2022.
\bibitem[Sun et~al.(2023{\natexlab{a}})Sun, Cui, Zhang, Zhang, Yu, Luo, Wang,
Rao, Liu, Huang, et~al.]{sun2023emu2}
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo,
Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et~al.
\newblock Generative multimodal models are in-context learners.
\newblock \emph{arXiv preprint arXiv:2312.13286}, 2023{\natexlab{a}}.
\bibitem[Sun et~al.(2023{\natexlab{b}})Sun, Yu, Cui, Zhang, Zhang, Wang, Gao,
Liu, Huang, and Wang]{sun2023emu}
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang,
Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
\newblock Generative pretraining in multimodality.
\newblock \emph{arXiv preprint arXiv:2307.05222}, 2023{\natexlab{b}}.
\bibitem[Sutskever(2014)]{sutskever2014sequence}
I~Sutskever.
\newblock Sequence to sequence learning with neural networks.
\newblock \emph{arXiv preprint arXiv:1409.3215}, 2014.
\bibitem[Tay et~al.(2022)Tay, Tran, Dehghani, Ni, Bahri, Mehta, Qin, Hui, Zhao,
Gupta, et~al.]{tay2022transformer}
Yi~Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen
Qin, Kai Hui, Zhe Zhao, Jai Gupta, et~al.
\newblock Transformer memory as a differentiable search index.
\newblock \emph{Advances in Neural Information Processing Systems},
35:\penalty0 21831--21843, 2022.
\bibitem[Touvron et~al.(2023)Touvron, Martin, Stone, Albert, Almahairi, Babaei,
Bashlykov, Batra, Bhargava, Bhosale, et~al.]{touvron2023llama2}
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et~al.
\newblock Llama 2: Open foundation and fine-tuned chat models.
\newblock \emph{arXiv preprint arXiv:2307.09288}, 2023.
\bibitem[Van Den~Oord et~al.(2017)Van Den~Oord, Vinyals, et~al.]{van2017neural}
Aaron Van Den~Oord, Oriol Vinyals, et~al.
\newblock Neural discrete representation learning.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Van~Horn et~al.(2018)Van~Horn, Mac~Aodha, Song, Cui, Sun, Shepard,
Adam, Perona, and Belongie]{van2018inaturalist}
Grant Van~Horn, Oisin Mac~Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard,
Hartwig Adam, Pietro Perona, and Serge Belongie.
\newblock The inaturalist species classification and detection dataset.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 8769--8778, 2018.
\bibitem[Wang et~al.(2020)Wang, Min, Hou, Ma, Zheng, Wang, and
Jiang]{wang2020logo}
Jing Wang, Weiqing Min, Sujuan Hou, Shengnan Ma, Yuanjie Zheng, Haishuai Wang,
and Shuqiang Jiang.
\newblock Logo-2k+: A large-scale logo dataset for scalable logo
classification.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~34, pp.\ 6194--6201, 2020.
\bibitem[Wang et~al.(2024)Wang, Bai, Tan, Wang, Fan, Bai, Chen, Liu, Wang, Ge,
et~al.]{wang2024qwen2}
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin
Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et~al.
\newblock Qwen2-vl: Enhancing vision-language model's perception of the world
at any resolution.
\newblock \emph{arXiv preprint arXiv:2409.12191}, 2024.
\bibitem[Wang et~al.(2023)Wang, Lin, Li, Lin, Yang, Zhang, Liu, and
Wang]{wang2023equivariant}
Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang,
Zicheng Liu, and Lijuan Wang.
\newblock Equivariant similarity for vision-language foundation models.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pp.\ 11998--12008, 2023.
\bibitem[Wei et~al.(2023)Wei, Chen, Chen, Hu, Zhang, Fu, Ritter, and
Chen]{wei2023uniir}
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge~Zhang, Jie Fu, Alan Ritter,
and Wenhu Chen.
\newblock Uniir: Training and benchmarking universal multimodal information
retrievers.
\newblock \emph{arXiv preprint arXiv:2311.17136}, 2023.
\bibitem[Weyand et~al.(2020)Weyand, Araujo, Cao, and Sim]{weyand2020google}
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim.
\newblock Google landmarks dataset v2-a large-scale benchmark for
instance-level recognition and retrieval.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition}, pp.\ 2575--2584, 2020.
\bibitem[Wu et~al.(2023)Wu, Fei, Qu, Ji, and Chua]{wu2023next}
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.
\newblock Next-gpt: Any-to-any multimodal llm.
\newblock \emph{arXiv preprint arXiv:2309.05519}, 2023.
\bibitem[Xu et~al.(2024)Xu, Liu, Wu, Tong, Li, Ding, Tang, and
Dong]{xu2024imagereward}
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang,
and Yuxiao Dong.
\newblock Imagereward: Learning and evaluating human preferences for
text-to-image generation.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Yang et~al.(2024)Yang, Yu, Meng, Xu, Ermon, and
Cui]{yang2024mastering}
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui.
\newblock Mastering text-to-image diffusion: Recaptioning, planning, and
generating with multimodal llms.
\newblock \emph{arXiv preprint arXiv:2401.11708}, 2024.
\bibitem[Young et~al.(2014)Young, Lai, Hodosh, and Hockenmaier]{young2014image}
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.
\newblock From image descriptions to visual denotations: New similarity metrics
for semantic inference over event descriptions.
\newblock \emph{Transactions of the Association for Computational Linguistics},
2:\penalty0 67--78, 2014.
\bibitem[Yu et~al.(2022)Yu, Wang, Vasudevan, Yeung, Seyedhosseini, and
Wu]{yu2022coca}
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and
Yonghui Wu.
\newblock Coca: Contrastive captioners are image-text foundation models.
\newblock \emph{arXiv preprint arXiv:2205.01917}, 2022.
\bibitem[Yu et~al.(2023)Yu, Shi, Pasunuru, Muller, Golovneva, Wang, Babu, Tang,
Karrer, Sheynin, et~al.]{yu2023scaling}
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu
Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et~al.
\newblock Scaling autoregressive multi-modal models: Pretraining and
instruction tuning.
\newblock \emph{arXiv preprint arXiv:2309.02591}, 2\penalty0 (3), 2023.
\bibitem[Zhang et~al.(2023)Zhang, Zhang, Chen, Wang, Chen, Xie, Sun, Deng,
Zhang, Yang, et~al.]{zhang2023irgen}
Yidan Zhang, Ting Zhang, Dong Chen, Yujing Wang, Qi~Chen, Xing Xie, Hao Sun,
Weiwei Deng, Qi~Zhang, Fan Yang, et~al.
\newblock Irgen: Generative modeling for image retrieval.
\newblock \emph{arXiv preprint arXiv:2303.10126}, 2023.
\bibitem[Zhao et~al.(2024)Zhao, Pang, Li, Wu, Guo, Xing, and
Dai]{zhao2024aligngpt}
Fei Zhao, Taotian Pang, Chunhui Li, Zhen Wu, Junjie Guo, Shangyu Xing, and
Xinyu Dai.
\newblock Aligngpt: Multi-modal large language models with adaptive alignment
capability.
\newblock \emph{arXiv preprint arXiv:2405.14129}, 2024.
\bibitem[Zheng et~al.(2023)Zheng, He, and Wang]{zheng2023minigpt}
Kaizhi Zheng, Xuehai He, and Xin~Eric Wang.
\newblock Minigpt-5: Interleaved vision-and-language generation via generative
vokens.
\newblock \emph{arXiv preprint arXiv:2310.02239}, 2023.
\bibitem[Zhu et~al.(2023{\natexlab{a}})Zhu, Chen, Shen, Li, and
Elhoseiny]{zhu2023minigpt}
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
\newblock Minigpt-4: Enhancing vision-language understanding with advanced
large language models.
\newblock \emph{arXiv preprint arXiv:2304.10592}, 2023{\natexlab{a}}.
\bibitem[Zhu et~al.(2023{\natexlab{b}})Zhu, Ding, Ge, Ge, Zhao, Zhao, Wang, and
Shan]{zhu2023vl}
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao,
Xiaohua Wang, and Ying Shan.
\newblock Vl-gpt: A generative pre-trained transformer for vision and language
understanding and generation.
\newblock \emph{arXiv preprint arXiv:2312.09251}, 2023{\natexlab{b}}.
\end{thebibliography}