FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Extracted source text · arXiv 2305.17718
\begin{abstract}
% Image captioning, a central task in computer vision, has experienced substantial progress following the advent of vision-language pre-training techniques.
The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning.
% {\color{red} The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning}{\color{blue} enchanced is already overused in the paper}
However, these models frequently produce generic captions and may omit semantically important image details.
This limitation can be traced back to the image-text datasets; while their captions typically offer a general description of image content, they frequently omit salient details.
Considering the magnitude of these datasets, manual reannotation is impractical, emphasizing the need for an automated approach.
To address this challenge, we leverage existing captions and explore augmenting them with visual details using ``frozen'' vision experts including an object detector, an attribute recognizer, and an Optical Character Recognizer (OCR).
Our proposed method, \fusecap{}, fuses the outputs of such vision experts with the original captions using a large language model (LLM), yielding comprehensive image descriptions.
We automatically curate a training set of 12M image-enriched caption pairs.
These pairs undergo extensive evaluation through both quantitative and qualitative analyses.
% {\color{red}DB: Extensive quantitative and qualitative evaluations showcase that the novel captions better describe image content.}
Subsequently, this data is utilized to train a captioning generation BLIP-based model.
This model outperforms current state-of-the-art approaches, producing more precise and detailed descriptions, demonstrating the effectiveness of the proposed data-centric approach.
We release this large-scale dataset of enriched image-caption pairs for the community.
\end{abstract}
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=0.78\linewidth]{figures/first_page_prismerfix.pdf}
% \vspace{-3pt}
% \caption{\textbf{Image Captioning Results.} While top-performing captioning models tend to provide concise and oversimplified captions, our model outputs rich captions that better describe the images.}
% \label{fig:generatior_example}
% \vspace{-6pt}
% \end{figure*}
% \begingroup
% \let\thefootnote\relax
% \footnotetext{\hspace{0pt}\vspace{-\baselineskip}\parbox{\textwidth}{Our project page: \url{https://rotsteinnoam.github.io/FuseCap}}}
% \endgroup
% \let\thefootnote\relax\footnotetext{\hspace{0pt}\vspace{-\baselineskip}\parbox{\textwidth}{Our project page: \url{https://rotsteinnoam.github.io/FuseCap}}}
% \addtocounter{footnote}{-1}
\begin{figure}
% \vspace{-0.4cm}
\centering
% \includegraphics[width=1\linewidth]{figures/fig3.png}
\includegraphics[width=\linewidth]{figures/fusecap_enrich5.pdf}
\caption{\textbf{\fusecap{} captions.} An illustration comparing our \fusecap{} enriched captions with the original ground-truth captions before the fusing process.
The examples are from COCO, SBU, CC, and CC12 datasets, displayed from top to bottom.
% The examples are from COCO, CC, CC12, SBU, top to bottom.
% \shaked{What is the source of the images? i.e. what dataset}
}
\label{fig:generatior_example2}
\vspace{-0.46cm}
% \end{figure}
% \vspace{-1cm}
\end{figure}
% \begin{figure*}
% % \vspace{-0.4cm}
% \centering
% % \includegraphics[width=1\linewidth]{figures/fig3.png}
% % \includegraphics[width=\linewidth]{figures/fusecap_enrich3.pdf}
% \includegraphics[width=\linewidth]{figures/fusecap_enrich3_horizontal.pdf}
% \caption{\textbf{\fusecap{} captions.} An illustration comparing our \fusecap{} enriched captions with the original ground-truth captions before the fusing process. \shaked{What is the source of the images? i.e. what dataset}}
% \label{fig:generatior_example2}
% \vspace{-0.46cm}
% % \end{figure}
% % \vspace{-1cm}
% \end{figure*}
\section{Introduction}\label{Se:intro}
%The generation of image captions capturing key descriptive elements has been a long-standing goal in computer vision. Image captioning notably generalizes image classification and enhances image retrieval, two central tasks in the field. Despite the remarkable advancements in the generalization abilities of captioning models, they frequently generate generic captions that omit crucial semantic elements.
The generation of image captions that effectively capture essential descriptive elements has been a longstanding goal in computer vision~\cite{longstanding0,longstanding1,longstanding2,longstanding3,longstanding4,longstanding5}.
In recent years, image captioning tasks~\cite{lin2014microsoft,agrawal2019nocaps} have gained significant research attention and interest due to the success of Vision Language (VL) models.
This achievement mainly stems from the ability to efficiently harness the massive amount of image-caption pairs accessible online, using Vision Language Pre-training (VLP)~\cite{align_before_fuse,uniter,radford2021learning}, followed by task-specific fine-tuning.
However, despite remarkable advancements in image captioning, current state-of-the-art models ~\cite{vinvl,simvlm,li2022blip,li2023blip,hu2022scaling, wang2022git,pali,wang2022ofa} produce captions that often overlook key semantic elements.
As images are rich sources of information containing intricate and complex content, providing precise descriptions requires highly detailed textual captions.
% In this work, our objective is to develop a model capable of producing rich captions that more accurately capture the information conveyed in images.\shaked{Our goal is to present new caption generation model or present a way of creating image-pair dataset? (which we prove to be value by training a generation model using it)}
We hypothesize that the current unsatisfactory captioning results are attributed to the image-caption datasets used for training.
% rather than the models themselves.
% This perspective resonates with the recently surveyed data-centric artificial intelligence (AI) paradigm \cite{zha2023data, zha2023data2}, which underscores the significance of improving both the quality and quantity of data, rather than merely concentrating on the advancement of model design.
Captions in these datasets frequently fail to capture essential elements within images and often omit fine details.
For example, consider the original caption of the top image in \Cref{fig:generatior_example2}.
The caption is missing details such as the laptop and cat.
% {\color{red} I would have removed this sentence and incorporated your point in the next one }{\color{blue} I see why this sentence fits better after, but I am not sure the next sentence fits better before (after data centric)}
% {\color{red} illustrating that a apart from omitting fine details, captions in actual datasets frequently fail to capture essential elements within images.}
% The vast number of image-caption pairs in these datasets makes manual re-annotation an unfeasible task.
Since these datasets contain a massive number of image-caption pairs, manual re-annotation is unfeasible.
In this work, our primary objective is to develop a framework that produces richer, more accurate captions for images.
To achieve this, we introduce \fusecap{}, a novel approach designed to automatically augment captions in existing datasets, thereby enhancing training data of the model.
This contrasts with methods that primarily focus on improving the caption generator model architecture and resonates with the recently surveyed data-centric artificial intelligence (AI) paradigm \cite{zha2023data, zha2023data2}, which underscores the significance of improving both the quality and quantity of data, rather than merely concentrating on the advancement of model design.
We leverage the capabilities of \textit{vision experts} such as object detectors \cite{anderson2018bottom}, attribute recognizers \cite{zhang2021vinvl}, and Optical Character Recognition (OCR) models \cite{bautista2022parseq,aberdam2022multimodal,aberdam2023clipter}.
The visual information extracted by these models is intended to provide complementary details to the original simplistic captions.
By harnessing the reasoning capabilities of a dedicated LLM, the outputs from the vision experts are fused with the original image caption.
This results in a coherent, meaningful natural description of images that is more comprehensive and detailed than the original caption, as illustrated in \Cref{fig:generatior_example2}.
Specifically, we leverage ChatGPT\cite{brown2020language} to generate ``fusing'' examples which are then used to fine-tune a pre-trained Flan-T5 model \cite{raffel2020exploring}.
We apply this method to enrich captions of a human-annotated dataset (COCO~\cite{lin2014microsoft}) and large-scale datasets collected from the web (CC12~\cite{changpinyo2021conceptual}, CC~\cite{sharma2018conceptual}, and SBU~\cite{vicente2016large}).
This process produces an enriched collection consisting of 12M image-text pairs.
To confirm the quality of our generated dataset, we first show that humans favor the fused captions, perceiving them as more descriptive and accurate than the original captions.
Furthermore, we demonstrate that the fused captions score
higher on CLIPScore~\cite{hessel2022clipscore} -- a reference-free metric that evaluates text-image alignment without using reference captions — compared to the original ones.
To further emphasize their effectiveness, we also assess the captions through image-to-text and text-to-image retrieval tasks.
To illustrate the benefits of the proposed data-centric approach we capitalize on these fused captions to train a caption generator.
We use the augmented dataset both for pre-training and for the fine-tuning of an image captioning BLIP model \cite{li2022blip}.
Despite having fewer parameters and using less training data, our model surpasses existing state-of-the-art methods \cite{li2022blip,li2023blip,wang2022ofa,git, liu2023prismer} in generating comprehensive captions.
This superiority is evident both in its higher CLIPScore and its improved performance in the retrieval tasks.
The performance advantage is further illustrated by numerous examples.
\textbf{Our contributions are as follows:}
% In Section \ref{sec:pretrain_significance}, we emphasize the importance of the large-scale caption enrichment dataset provided by \fusecap{}.
% Our results indicate that using the original large-scale captions for pretraining yields suboptimal performance compared to pretraining with the corresponding enriched caption dataset. {\color{red}DB: I would have removed the two unclear sentences."}
\begin{itemize}%[leftmargin=*]
\item Introducing \fusecap{}- a novel approach to automatically enrich existing image-captions datasets by fusing outputs from visual experts using an LLM.
\item Providing a large dataset of 12M caption-enriched text-image pairs for future research.
\item Showcasing that an enriched dataset leads to models capable of generating detailed captions that effectively incorporate previously overlooked key semantic elements.
% \item Demonstrating enriched captions' effectiveness in image-text retrieval, outperforming models that use the original captions.
\end{itemize}
\vspace{-0.5cm}
\begin{figure*}[!t]
\centering
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.87\linewidth]{figures/method3c.pdf}
\caption{Fusing Enriched Captions}
\label{method_stage1}
\end{subfigure}
\begin{subfigure}{\textwidth}
\centering
\includegraphics[width=0.85\linewidth]{figures/method3b.pdf}
\caption{Training a Captioning Model}
\label{method_stage2}
\end{subfigure}
\caption{
\textbf{Our Approach Illustration.}
\Cref{method_stage1} illustrates the automated process of enriching existing image captions using the proposed \fusecap{} approach.
Visual experts extract meaningful information from images, which is then fused with the original captions by an LLM Fuser, producing rich captions.
Following this, \Cref{method_stage2} illustrates the utilization of image datasets, paired with these augmented captions, in both the pre-training and fine-tuning phases of a comprehensive image-captioning model.
}
\vspace{-0.21cm}
\label{fig:test}
\end{figure*}
\section{Related Efforts}\label{Se:related}
\paragraph{Image-Caption Generation.}
Image captioning has been a widely researched topic at the intersection of computer vision and natural language processing.
Early strategies for caption generation made use of retrieval-based methods \cite{farhadi2010every, hodosh2013framing} and template-based methods \cite{kulkarni2013babytalk}, which were limited in their expressiveness \cite{bai2018survey}.
% The advent of deep learning marked a shift in this field, as multimodal neural networks that learned inter-modality correspondences enabled the generation of higher quality captions \cite{karpathy2015deep}.
The advent of deep learning marked a shift in this field, as multimodal neural networks enabled the generation of higher quality captions \cite{karpathy2015deep}.
Subsequently, the encoder-decoder framework, which essentially translates an image into a sentence, became one of the favored approaches \cite{vinyals2015show, donahue2015long}.
% This framework has seen further advancements by integrating attention mechanisms to focus on salient aspects of an image \cite{xu2015show, yang2016review}.
Later advancements incorporated attention mechanisms for focusing on key image aspects \cite{xu2015show, yang2016review}.
Current best captioning techniques are transformer-based architectures \cite{vaswani2017attention}, which combine vision \cite{dosovitskiy2020image} and language \cite{vaswani2017attention} transformers.
The recent advancements in the effectiveness of image captioning can largely be credited to the introduction of {\it visual language pre-training} (VLP).
VLP uses large-scale image-text pairs to pre-train vision-text models, which are later fine-tuned for downstream tasks, with image captioning as one of the central tasks \cite{li2022blip,li2023blip,ganz2023towards}.
VLP can be categorized \cite{chen2023vlp} into single-stream \cite{li2019visualbert, chen2020uniter} and dual-stream architectures \cite{dou2022empirical, radford2021learning, jia2021scaling,ganz2023clipag}, aiming to merge visual and textual modalities into a shared embedding space.
Efforts exploring integrating text generative tasks for pre-training \cite{wang2021simvlm, cho2021unifying} have fueled the development of models like BLIP \cite{li2022blip} and recent BLIP-2 \cite{li2023blip}.
OFA \cite{wang2022ofa} proposed unifying multiple unimodal and multimodal pre-training tasks, which led to a significant performance improvement.
Dense captioning can be seen as a task related to ours in generating comprehensive text for images \cite{johnson2016densecap, yang2017dense}.
However, it produces multiple captions for various regions within an image, as opposed to a single descriptive caption for the whole image.
Previous approaches to generating more comprehensive captions have consistently identified a common challenge -- the unsatisfactory capability of existing image captioners of providing detailed and accurate textual descriptions of images~\cite{liu2018show, luo2018discriminability, liu2019generating}.
These approaches focus on improving captions discriminability, whereas our method seeks to enrich captions with additional, meaningful information extracted from the images.
\vspace{-6pt}
\paragraph{Image-Caption Datasets}
Current datasets for image-caption pairs fall into two primary categories: specifically human-annotated datasets like COCO \cite{lin2014microsoft}, and web-crawled datasets such as CC, CC12, and SBU Captions \cite{sharma2018conceptual, changpinyo2021conceptual, ordonez2011im2text}.
The conjunction of the datasets serves as the foundation for the VLP, subsequently followed by downstream tasks fine-tuning.
In general, the first category of datasets is smaller in size yet exhibits substantially lower noise levels compared to the second category.
Both categories, however, are characterized by relatively short and concise captions.
Specific examples of these characteristics can be seen in \Cref{fig:generatior_example2}.
Data-centric AI underscores the importance of refining data quality to boost model performance rather than fine-tuning model designs.
It prioritizes curating, labeling, and cleaning data for superior training datasets \cite{sun2017revisiting} as well as automating its processing \cite{zha2023data, zha2023data2}.
This new emphasis suggests that with high-quality data, even basic algorithms can deliver impressive results.
\cite{shi2021enhancing} identified the need to enrich image-text datasets.
To this end, they harnessed natural language inference (NLI) to fuse multiple existing ground-truth captions into a single one.
However, this approach can operate only on datasets with multiple ground-truth captions and cannot be applied to large-scale ones with a single caption (\textit{e.g.,} CC, CC12, and SBU).
In contrast, our method can be applied to any image-caption dataset.
\vspace{-6pt}
% Moreover, we show that using our fused enriched datasets for training is highly effective in caption generation and image-text retrieval.
% Moreover, we show that using our fused enriched datasets for training is highly effective in caption generation and image-text retrieval.
% \textcolor{blue}{no more retrieval.}
% % ֿ\noam{if we want to talk about it lacking details, there is a commented part here}
% In the context of VLP, these larger, noisy web datasets, in conjunction with the noise-free human-annotated sets, serve as the foundation for pretraining.
% The latter, due to their "clean" characteristics, are reserved for fine-tuning and evaluation.
% Following this paradigm, our work utilizes enriched COCO, CC, CC12, and SBU for pretraining, and then resorts to COCO and Flickr for fine-tuning and evaluation (\Cref{Se:experiments}).
% % noam{
% % hypothesize that they lack detail. also, combine all this into one section.
% % }
\paragraph{Large Language Models}
Large language models (LLMs) have been shown to be effective in a wide range of tasks, including natural language inference, question answering, and code generation \cite{chung2022scaling, wei2021finetuned, chowdhery2022palm}.
Further, LLMs such as GPT-3 \cite{brown2020language} exhibit impressive zero and few-shot performance on a variety of tasks, including translation, text summarization, and common sense reasoning, without further fine-tuning.
Few-shot abilities allow researchers to use LLMs as a tool for data generation.
\cite{brooks2022instructpix2pix} used GPT-3 to generate instructions and edited captions dataset, which is then used to train model for image editing.
\cite{schick2023toolformer} presented an approach for training LLMs to use external API calls. For example, when the LLM is been asked to solve a mathematical problem, it can use an API of a calculator, instead of generating the output by itself. \cite{schick2023toolformer} used GPT-3 few-shot ability to curate a dataset of external API calls which is then been used to fine-tune another LLM.
\cite{peng2023instruction} used GPT-4 outputs to create an instruction-following dataset that can be used later to fine-tune other LLMs in a supervised learning fashion.
To fuse together the original caption and the visual expert outputs, in this work, we harness the impressive zero-shot capabilities of OpenAI's ChatGPT \cite{brown2020language}.
We employ it to generate a small ``fusing'' dataset. To establish an open-source framework that scales cost-effectively, we fine-tune Flan-T5 \cite{chung2022scaling}, a widely recognized open-source LLM, using this data.
\vspace{-6pt}
\paragraph{Vision Experts in VLP}
Several works attempted to improve VLP by incorporating object detectors or other experts \cite{liu2023prismer} as part of their initialization \cite{li2019visualbert, cho2021unifying}, architecture \cite{liu2023prismer}, pre-training data \cite{wang2022position} or pre-training objectives \cite{tan2019lxmert, su2019vl, zhang2021vinvl, li2020oscar}.
In image captioning, such models demonstrate limited capabilities in generating rich captions and have not fully capitalized on the information provided by vision experts. We hypothesize that this limitation stems directly from the succinct captions in existing image-text datasets.
In our data-centric approach, we focus on improving such datasets and overcoming this limitation.
An example that highlights our approach is the contrast with Prismer \cite{liu2023prismer}. Unlike our method, Prismer augments its model architecture with object detectors and OCR for caption generation but relies on traditional captioning datasets.
% A recent concurrent paper \cite{wang2023caption} proposed a model based on an LLM to merge dense captions into text. In contrast, we adopt a data-centric perspective to enhance existing caption datasets.
A recent concurrent paper \cite{wang2023caption} proposed an LLM-based model that merges captions of image segments into text.
In contrast, we adopt the LLM from a data-centric perspective to enhance existing caption datasets, an approach that is followed by extensive evaluations for quality and consistency.
\section{Fusing Enriched Captions}\label{Se:method}
% \subsection{Overview}
% Our proposed \textcolor{Magenta}{framework} operates in two distinct \textcolor{Magenta}{stage}s.
In this section, we introduce \fusecap{}, which is illustrated in \Cref{method_stage1}.
This novel strategy is designed to automatically augment existing image captions by integrating important details and inter-object relations within the image.
These details are often disregarded in traditional image captioning datasets.
First, we elaborate on our use of pre-trained vision models, referred to as \textit{vision experts}, for extracting relevant visual information from images.
We then detail how the information gathered from these expert models is subsequently fused with the original caption through a fine-tuned LLM, resulting in the enhanced captions.
\subsection{Vision Expert Models}\label{Se:vision-experts}
To enrich the information found in the original caption, we employ the following vision experts:
% The supplementary information used to enhance the initial image captions is derived from pretrained visual expert models.
% Specifically, we employ the following expert models,
\vspace{-8pt}
% \hspace{2em}\textbf{Object detection.}
\paragraph{Object Detection}
A key visual expert we rely on is an object detector model.
Following the approach proposed in
% VinVL
\cite{zhang2021vinvl}, we utilize a Faster-RCNN \cite{ren2015faster} with a ResNeXt152 \cite{xie2017aggregated} backbone.
This model is initially pre-trained on several detection datasets and then fine-tuned on the Visual Genome (VG) dataset \cite{krishna2017visual}.
The data used comprises more than 100K images with 1.6K classes, enabling strong generalization abilities.
%robust performance on out-of-dataset images.
We regard all objects along with their corresponding bounding boxes as valid detections, provided they exceed a detection confidence threshold.
% These objects are denoted as $\{o_i\}_i$, and their corresponding bounding boxes as $\{b_i\}_i$.
The presence and position of objects in the scene provide essential details to complement the caption.
\vspace{-8pt}
% \hspace{2em}\textbf{Attributes prediction.}
\paragraph{Attributes Prediction}
Beyond class identification, we derive a variety of attributes for each object in the image using features generated by our Faster-RCNN within each bounding box.
A classification model proposed in \cite{zhang2021vinvl} was trained on annotations from the VG dataset, which covers a broad array of 400 distinct attributes.
These attributes encapsulate various aspects of the objects, including size, condition, and color.
For each object, we only consider attributes predicted with a confidence level above the threshold.
\vspace{-8pt}
\paragraph{Text Detection and Recognition}
Text within images often contains critical contextual information, leading to a line of different text-image tasks \cite{biten2019scene, ganz2023towards}.
To incorporate textual information, we utilize robust pre-trained OCR models to detect and extract characters.
We first identify text within a scene with CRAFT \cite{baek2019character}, a robust scene-text detector.
We then apply Parseq \cite{bautista2022parseq}, a state-of-the-art scene text recognizer, to decode the text within the bounding boxes generated by the text detector.
To avoid contamination, we do not apply OCR methods on datasets presenting watermarks, such as the CC \cite{sharma2018conceptual} and CC12 \cite{changpinyo2021conceptual}.
\begin{figure*}[!htb]
\centering
\includegraphics[width=1\linewidth]{figures/first_page_prismerfix_2.pdf}
\vspace{-12pt}
\caption{\textbf{Image Captioning Results.} While top-performing captioning models tend to provide concise and oversimplified captions, our model outputs rich captions that better describe the images.}
\label{fig:generatior_example}
\vspace{-3pt}
\end{figure*}
% \vspace{-4pt}
\subsection{LLM Fuser}\label{Se:llm-fuser}
To generate natural and coherent captions, which are essential for the caption generation task, we utilize a large language model (LLM).
We leverage a specifically fine-tuned LLM to fuse insights from various vision experts into the original caption, creating a single coherent and fluent description.
The LLM's advanced reasoning capabilities allow us to articulate the semantic relationships among objects and to seamlessly integrate the diverse knowledge provided by these experts.
Consequently, the output captures the essence of the visual content while the impressive generative capacity of LLMs ensures these captions remain coherent and natural.
To train the LLM, we create a small ``fusing'' dataset with ChatGPT \cite{brown2020language}, and then use it to fine-tune the open-source FlanT5-XL model \cite{chung2022scaling}.
% \begin{figure}
% % \vspace{-0.4cm}
% \centering
% % \includegraphics[width=1\linewidth]{figures/fig3.png}
% \includegraphics[width=\linewidth]{figures/fusecap_enrich.pdf}
% \caption{\textbf{\fusecap{} captions.} An illustration comparing our \fusecap{} enriched captions with the original ground-truth captions before the fusing process.}
% \label{fig:generatior_example2}
% % \vspace{-0.6cm}
% % \end{figure}
% % \vspace{-1cm}
% \end{figure}
% \hspace{2em}\textbf{ChatGPT Annotation.}\label{Se:chatgpt-annotation}
\paragraph{ChatGPT Annotation}\label{Se:chatgpt-annotation}
We leverage the zero-shot capabilities of ChatGPT to generate a ``fusing'' dataset of 20K examples, which is then used to fine-tune an open-source LLM.
By employing this approach, we establish a framework based on open-source models that can generate a large-scale dataset at a reasonable cost.
To produce enriched captions with ChatGPT, we first extract information from the visual experts presented in \Cref{Se:vision-experts}.
The objects and their corresponding attributes and detected texts are then ordered from left to right based on their bounding boxes, providing basic spatial context.
The exact prompt can be found in the appendix.
We generated 20K examples of such enriched captions from CC, CC12, SBU, and COCO datasets,
which served as training data for the FlanT5-XL model.
\vspace{-6pt}
% \vspace{-1cm}
% \hspace{2em}\textbf{LLM Fine-tuning.}
\paragraph{LLM Fine-tuning}
% The process of generating our enriched caption datasets involved using the LLM fine-tuning technique with
Using the ``fusing'' dataset we created, we fine-tuned Flan-T5 \cite{chung2022scaling}, a variant of the T5 encoder-decoder model \cite{raffel2020exploring}, that has been extensively fine-tuned on numerous tasks to achieve exceptional performance on instruction-based tasks.
Specifically, we utilized the Flan-T5-XL checkpoint and fine-tuned it to our curated fuse dataset.
During the fine-tuning process, the original caption was concatenated with the output of the visual experts and served as the model input, while the enriched caption was designated as the target. The hyperparameters we used for this section can be found in the appendix.
After this phase, the fine-tuned model can be used to generate an enriched captions dataset.
\vspace{-3pt}
% \begin{figure}[!ht]
% \begin{wrapfigure}{r}{0.5\textwidth}
% \vspace{-0.4cm}
% \centering
% % \includegraphics[width=1\linewidth]{figures/fig3.png}
% \includegraphics[width=\linewidth]{figures/fusecap_enrich.pdf}
% \caption{\textbf{\fusecap{} captions.} Illustration of our \fusecap{} enriched-captions (first \textcolor{Magenta.}{stage}), compared to the original ground-truth ones.}
% \label{fig:generatior_example2}
% % \vspace{-0.6cm}
% % \end{figure}
% \vspace{-1cm}
% \end{wrapfigure}
\section{Training a Captioning Model}
\label{subsec:stage2}
We apply \fusecap{}, presented in \Cref{Se:method}, to the COCO, SBU, CC, and CC12 datasets, yielding 12 million augmented image-caption pairs.
Following our data-centric approach, and to demonstrate the effectiveness of this augmented dataset, we use it to train a captioning model based on the BLIP architecture \cite{li2022blip} (\Cref{method_stage2}).
Adopting the training strategy from the original BLIP paper, we first perform vision-language pre-training for $20$ epochs , optimizing three objectives:
\begin{inparaenum}[(a)]
\item \textbf{Image-Text Contrastive Loss (ITC)} that aligns visual and textual features by contrasting between matching and non-matching image-text pairs,
\item \textbf{Image-Text Matching Loss (ITM)} that classifies image-text pairs as matched or unmatched, and
\item \textbf{Generative Language Modeling Loss (LM)} that generates textual captions from images, which are compared to their ground truth captions.
% \end{itemize}
\end{inparaenum}
Following the pre-training phase, adhering to common practice in VLP \cite{li2022blip, li2023blip, vinvl}, we fine-tune our model for a supplementary $5$ epochs on the \emph{enriched} COCO dataset, utilizing solely LM loss.
To allow the generation of more comprehensive captions, we increase the context length over that used in the original BLIP model from $30$ to $60$ tokens.
The hyperparameters for this section are detailed in the appendix.
\section{Experiments}\label{Se:experiments}
We evaluate two sets of image captions.
The first set consists of the fused captions from the \fusecap{} dataset, described in \Cref{Se:method}. The second set includes captions generated by the trained captioning model discussed in \Cref{subsec:stage2}.
% \shaked{It's not clear - the paper ``talks'' about 3 sets of captions - the original lame captions, the FlanT5 generated captions and the our BLIP generated captions. So for which sets do you refer here?}.
The objective for both sets is to produce captions that are descriptive and accurate.
Traditionally, image captioning methods are assessed with n-gram-based metrics like BLEU \cite{papineni2002bleu}, CIDEr \cite{vedantam2015cider}, and ROUGE \cite{lin2004rouge}.
These metrics compare the tested captions to reference captions, assuming the latter represents the ideal image descriptions that the tested captions aim for.
However, as highlighted earlier, captions in existing datasets often fail to provide a comprehensive description of images.
This implies that given our emphasis on enhancing descriptiveness, relying on broad and non-specific reference captions for evaluation would be inappropriate.
Consequently, n-gram-based metrics do not effectively measure or promote this quality \cite{zhu2023chatgpt}. %{\color{green} I think this cite is wrong}
We, therefore adopt CLIPScore~\cite{hessel2022clipscore}, a reference-free metric that measures the alignment between textual and visual embeddings generated by a pre-trained CLIP model \cite{radford2021learning}.
Since CLIPScore is not dependent on a reference caption, its score is not restricted by the descriptiveness of the original captions.
Moreover, in terms of accuracy, it has been shown to exhibit a higher correlation with human judgments than n-gram-based metrics \cite{hessel2022clipscore}.
% and is commonly used as a replacement for classical metrics \cite{hessel2022clipscore, barraco2022unreasonable}.
Besides evaluating enriched captions with CLIPScore, we consider the ability to perform image-text retrieval as a pertinent metric for caption evaluation.
Comprehensive captions should inherently serve as distinct image descriptors, thereby improving retrieval precision.
To further assess the quality of captions in the \fusecap{} dataset, we conducted a human evaluation study.
\subsection{FuseCap Dataset}
\label{Se:data-validation}
In this subsection, we evaluate the enriched captions produced by the caption augmentation approach introduced in the \Cref{Se:method}.
In particular, we carry out a qualitative human-study alongside quantitative evaluations.
Examples of enriched captions via FuseCap are showcased in \Cref{fig:generatior_example2}.
\vspace{-20pt}
% \begin{table*}[htb]
% \centering
% \begin{tabular}{lcc|cc|cc}
% \toprule
% & \multicolumn{2}{c}{COCO} & \multicolumn{2}{c}{SBU} & \multicolumn{2}{c}{CC} \\
% % \midrule
% \cmidrule(r){2-3} \cmidrule(r){4-5} \cmidrule(r){6-7}
% & Mean & Voting & Mean & Voting & Mean & Voting \\
% \midrule
% Original captions & 76.7 & 31.7\% & 71.9 & 32.1\% & 72.6 & 34.7\%\\
% % \midrule
% \fusecap{} captions & \textbf{80.3} & \textbf{67.6\%} & \textbf{75.5} & \textbf{60.2\%} & \textbf{75.4} & \textbf{59.7\%} \\
% \bottomrule
% \end{tabular}
% \vspace{7pt}
% \caption{\textbf{\fusecap{} quantitative evaluation.} CLIPScore-based comparison between the original captions of common image-text datasets with our enriched ones. ``Mean'' indicated the mean CLIPScore and ``Voting'' expresses CLIP's preference in a one-vs-one setting. As can be seen, \fusecap{} obtains significantly improved results in both metrics.}
% \label{tab:caption_clips}
% \vspace{-0.5cm}
% \end{table*}
\begin{table}[b]
\centering
\vspace{-9pt}
\begin{tabular}{clccccc}
\toprule
Dataset & Captions & Mean & Voting \\
\midrule
\multirow{2}{*}{COCO} & Original & 76.7 & 31.7\%\\
% \midrule
& \fusecap{} & \textbf{80.3} & \textbf{67.6\%} \\
\midrule
\multirow{2}{*}{SBU} & Original & 71.9 & 32.1\%\\
% \midrule
& \fusecap{} & \textbf{75.5} & \textbf{60.2\%} \\
\midrule
\multirow{2}{*}{CC} & Original & 72.6 & 34.7\%\\
% \midrule
& \fusecap{} & \textbf{75.4} & \textbf{59.7\%} \\
\bottomrule
\end{tabular}
% \vspace{2pt}
\caption{\textbf{\fusecap{} data quantitative evaluation.} CLIPScore-based comparison between the original captions of common image-text datasets with our enriched ones. ``Mean'' indicated the mean CLIPScore and ``Voting'' expresses CLIP's preference in a one-vs-one setting. As can be seen, \fusecap{} obtains significantly improved results in both metrics.}
\label{tab:caption_clips}
\end{table}
% \begin{table}[htb]
% \centering
% \begin{tabular}{lcc|cc|cc|cc}
% \toprule
% Captions & Dataset & Mean & Voting \\
% \midrule
% Original & \multirow{2}{*}{COCO} & 76.7 & 31.7\%\\
% % \midrule
% \fusecap{} & & \textbf{80.3} & \textbf{67.6\%} \\
% \midrule
% Original & \multirow{2}{*}{SBU} & 71.9 & 32.1\%\\
% % \midrule
% \fusecap{} & & \textbf{75.5} & \textbf{60.2\%} \\
% \midrule
% Original & \multirow{2}{*}{CC} & 72.6 & 34.7\%\\
% % \midrule
% \fusecap{} & & \textbf{75.4} & \textbf{59.7\%} \\
% \bottomrule
% \end{tabular}
% \vspace{7pt}
% \caption{\textbf{\fusecap{} quantitative evaluation.} CLIPScore-based comparison between the original captions of common image-text datasets with our enriched ones. ``Mean'' indicated the mean CLIPScore and ``Voting'' expresses CLIP's preference in a one-vs-one setting. As can be seen, \fusecap{} obtains significantly improved results in both metrics.}
% \label{tab:caption_clips}
% \vspace{-0.5cm}
% \end{table}
\paragraph{Qualitative Evaluation}
We conducted a thorough human evaluation study to assess the ability of our enriched captions to be both descriptive and relevant to the images. Specifically, we randomly sampled 400 pairs from the COCO dataset
% \shaked{which is considered to have relatively high quality captions?}
and provided 40 participants with (1) the original caption and (2) the enriched caption.
Our study engaged a pool of random internet users as participants. To minimize biases and ensure an impartial evaluation, they completed the survey unaware of the specific research objectives or goals.
The evaluators were asked the following:
\textit{``Does caption 2 provide an additional meaningful and truthful description of the image compared to caption 1?''}.
As the enriched captions are much more detailed, as can be seen in \Cref{fig:generatior_example2}, the question focuses on whether they are accurate.
Our study results indicate that participants find the enriched captions at least as good as the original ones in $72.9\%$ of the images. This finding highlights the proposed method's effectiveness in enhancing the captions' descriptiveness while preserving alignment and relevancy to the images.
\vspace{-10pt}
\paragraph{Quantitative Evaluation}
We evaluate the enriched captions in comparison to the original captions for each dataset under consideration. To this end, we randomly selected 5000 images from each dataset and report the CLIPScore obtained with both types of captions.
As depicted in \Cref{tab:caption_clips}, the enriched captions generated by our proposed method consistently achieve a higher CLIPScore (Mean) on average by $4.6\%$.
In addition, given an image and two captions (original and enriched), we utilize CLIP to measure which caption is preferred. We evaluate this on the different datasets and summarize it under ``Voting'', which our captions outperform current ones on average by $29.7\%$.
These results demonstrate the effectiveness of our approach in generating enriched captions that better reflect the content of the images.
\vspace{-20pt}
\paragraph{Image-Text Retrieval}\label{Se:image-text-retrieval}
To further demonstrate the descriptiveness and accuracy of the \fusecap{} data, we assess its performance in the image-text retrieval task.
This task involves matching images to text queries and vice versa.
If the enriched data is descriptive and accurate, retrieval performance should improve since the additional details can serve as a discriminative factor in establishing these correspondences.
% BLIP$_\fusecap{}_P$ denotes the BLIP model pretrained using \fusecap{} data curated as described in \Cref{subsec:stage2}, while BLIP$_\fusecap{}$ denotes this model after being fine-tuned on the enriched COCO captions.
% For this evaluation the model is fine-tuned for image-text retrieval.
Our training methodology aligns with the original BLIP model.
After pre-training the model on a large-scale dataset as in \Cref{subsec:stage2}, the model is fine-tuned for image-text retrieval on the COCO training set using both ITC and ITM losses.
For the inference step, we employ the method proposed by \cite{li2021align}, previously integrated into BLIP.
This method involves the selection of $K$ candidates based on feature similarity, followed by their re-ranking using respective ITM values.
We report $R@N$ that corresponds to the accuracy of retrieving the true text/image among the top $N$ retrieved results.
The details of the competing models are provided in \Cref{Se:caption-generation}, with the exception that the versions discussed here have been fine-tuned for the retrieval tasks.
% \shaked{Makes more sense to present the baselines here and not on the next section.}
As illustrated in \Cref{coco_retrieval}, the use of fused captions contributes significantly to the enhancement of retrieval performance on the COCO test set\footnote{While we used enriched captions for training and testing BLIP$_\fusecap{}$, for the other baselines we used the original dataset without enrichment.}.
For example, compared to the model trained on corresponding non-enriched data, the $R@1$ score for image-to-text retrieval increased by $22.1\%$, and for text-to-image retrieval, it increased by $34.8\%$.
\begin{table}
\centering
\begin{tabular}{l *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}}}
\toprule
& \multicolumn{6}{c}{COCO Retrieval} \\
& \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
Model &
R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 \\
\midrule
BLIP$\dag$ & 75.1 & 92.7 & 96.4 & 58.2 & 82.4 & 89.2 \\
BLIP-L & 82.4 & 95.4 & 97.9 & 65.2 & 86.3 & 91.8 \\
BLIP2 & 85.4 & 97.0 & 98.5 & 68.3 & 87.7 & 92.6 \\
\midrule
BLIP$^*_\fusecap{}$ & \textbf{97.2} & \textbf{99.5} & \textbf{99.9} & \textbf{93.0} & \textbf{97.4} & \textbf{98.3} \\
\bottomrule
\end{tabular}
\vspace{7pt}
\caption{\textbf{Image-text retrieval results.} Performance on COCO retrieval (test sets). The ``*'' symbol indicates that the model was trained and tested on our enriched dataset. These results attest that given rich captions, BLIP$_\fusecap{}$ significantly outperforms existing methods, which utilize standard captions.
% BLIP$\dag$ \cite{li2022blip} is a model trained on the same image set as BLIP$_\fusecap{}$ but uses the original captions, while BLIP-L and BLIP2 are larger models trained on more data.
}
\label{coco_retrieval}
\vspace{-0.3cm}
\end{table}
% In addition to the previous experiments, we assess our retrieval model's ability to generalize beyond its training domain, utilizing the fused captions of the Flickr dataset to evaluate the zero-shot retrieval performance of the model.
% As shown in Table \ref{coco_retrieval}, it is evident that our model maintains its significant advantage also under zero-shot configurations.
% These findings underscore the strength of BLIP$_\fusecap{}$ as a potent retrieval model capable of harnessing rich information for retrieval.
% Furthermore, it demonstrates the enriched semantic meaning fused into the captions, compared to their original counterparts.
% \begin{table}
% \centering
% \begin{tabular}{l *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}}}
% \toprule
% & \multicolumn{6}{c}{COCO Retrieval} & \multicolumn{6}{c}{Flickr-ZS Retrieval} \\
% & \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} & \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
% Model &
% R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 \\
% \midrule
% BLIP$\dag$ & 75.1 & 92.7 & 96.4 & 58.2 & 82.4 & 89.2 & 92.4 & 99.1 & 99.9 & 79.7 & 95.2 & 97.5 \\
% BLIP-L & 82.4 & 95.4 & 97.9 & 65.2 & 86.3 & 91.8 & 96.7 & 100.0 & 100.0 & 86.7 & 97.3 & 98.7 \\
% BLIP2 & 85.4 & 97.0 & 98.5 & 68.3 & 87.7 & 92.6 & 97.6 & 100.0 & 100.0 & 89.7 & 98.1 & 98.9 \\
% \midrule
% BLIP$^*_\fusecap{}$ & \textbf{97.2} & \textbf{99.5} & \textbf{99.9} & \textbf{93.0} & \textbf{97.4} & \textbf{98.3} & \textbf{99.5} & \textbf{100} & \textbf{100} & \textbf{97.3} & \textbf{99.5} & \textbf{99.7} \\
% \bottomrule
% \end{tabular}
% \vspace{7pt}
% \caption{\textbf{Image-text retrieval results.} Performance on COCO and Flickr image-text retrieval (test sets) in fine-tuning and zero-shot settings, respectively. The "*" symbol indicates that the model was trained and tested on our enriched dataset. These results attest that given rich captions, BLIP$_\fusecap{}$ significantly outperforms existing methods, which utilize standard captions. \textcolor{blue}{moved away zero shot and changed location}}
% \label{coco_retrieval}
% \vspace{-0.7cm}
% \end{table}
\subsection{Caption Generation}\label{Se:caption-generation}
We fine-tune the BLIP model that was pre-trained on the complete \fusecap{} dataset for captioning, using the enriched COCO dataset.
We refer to this fine-tuned version as BLIP$_\fusecap{}$.
To assess the effectiveness of the captioner, and, by extension, the \fusecap{} data it is trained on, we thoroughly compare our results to various state-of-the-art captioning models using CLIPScore.
In particular, we consider the following baselines:
\begin{itemize}
\item \textbf{BLIP$\dag$}~\cite{li2022blip}: An original BLIP model, pre-trained and fine-tuned on the same image set as BLIP$_\fusecap{}$, which uses \textbf{original} captions, in contrast to BLIP$_\fusecap{}$ which uses the enriched captions. To guarantee a fair comparison, we set configuration parameters identical to those implemented in BLIP$_\fusecap{}$.
\item \textbf{BLIP-L}: A large version of BLIP, which was pre-trained on a dataset comprising 129M images prior to the captioning fine-tuning.
Among all models presented in the original BLIP paper, this model achieves superior captioning results.
\item \textbf{BLIP2-G-OPT$_{2.7}$}~\cite{li2023blip}: A state-of-the-art captioning model that integrates a large frozen vision backbone (ViT G) along with a frozen LLM (OPT \cite{zhang2022opt}), trained on the datasets considered for BLIP-L.
\item \textbf{OFA}~\cite{wang2022ofa}: A state-of-the-art image captioning method trained on unimodal and multimodal pre-training tasks and fine-tuned for captioning.
% \item \textbf{MiniGPT-4:} A recent general-purpose multi-modal method that, at the difference with the other models considered, was not specifically fine-tuned for image captioning tasks.
\item \textbf{GIT}~\cite{git}: A state-of-the-art vision-language model, with an image encoder and text decoder architecture. It is scaled up in terms of both pre-training data and model size, and it is fine-tuned for image captioning.
\item \textbf{Prismer}~\cite{liu2023prismer}: A Vision-Language Model with Multi-Modal Experts such as an object detector and an OCR model. Prismer adopts a strategy that could be seen as dual to ours, integrating vision experts into the model architecture, as opposed to leveraging them to enrich the training data.
% A presented feature of the model is detailed image description generation.
\end{itemize}
% \begin{wraptable}{R}{7.5cm}
\begin{table}
% \vspace{-0.4cm}
\centering
\begin{tabular}{lcc|cc}
\toprule
Model & Images & Parameters & Val & Test \\
\midrule
BLIP$\dag$ & 12M & 247M & 75.2 & 75.3 \\
BLIP-L & 129M & 470M & 76.1 & 76.0 \\
OFA & 20M & 470M & 76.6 & 76.4 \\
GIT & 800M & 700M & 77.1 & 77.0 \\
BLIP2-G-OPT$_{2.7}$ & 129M & 3.8B & 77.8 & 77.5 \\
Prismer & 13M & 1.6B & 76.7 & 76.7 \\ % before fix: 76.3, 76.1
\midrule
BLIP$_\fusecap{}$ & 12M & 247M & \textbf{78.3} & \textbf{78.5} \\
\bottomrule
\end{tabular}
% \vspace{7pt}
\caption{
\textbf{Image captioning results.} CLIPScore of leading models and our approach on the COCO captions dataset. Our model outperforms much larger models that have pre-trained with significantly more training data.
% \shaked{same here - explain \dag{}, -L, and G-OPT (Unless it's some known baseline).}
}
% \vspace{-0.5cm}
\label{caption_clips}
% \vspace{-0.4cm}
\end{table}
% \end{wraptable}
Building on the methodology outlined in \Cref{Se:experiments}, we evaluate the performance of our models using the mean CLIPScore metric.
As shown in \Cref{caption_clips}, our model not only outperforms the BLIP$\dag$ model by $4.3\%$ but also surpasses the best performing among other models by $1.3\%$.
This improvement over models with a considerably larger parameter count that have been trained on significantly more image-caption pairs underscores the effectiveness of our data centric approach.
Furthermore, as illustrated in \Cref{fig:generatior_example}, our approach exhibits clear superiority in the generation of captions, producing descriptions with greater semantic detail compared to those generated by competing models.
Our results surpassing Prismer indicate that for generating comprehensive image captions, leveraging vision experts to enrich data can be more beneficial than incorporating them directly into the model architecture.
Additional examples showcasing the performance and capabilities of our model can be found in the supplementary material.
\textbf{Image-Text Retrieval}
To further assess the quality of the generated captions, we once again utilize the image-text retrieval task, following the method detailed in \Cref{Se:image-text-retrieval}.
This time, however, we consider captions not from the ground truth COCO or the ``ground truth'' enriched COCO datasets.
Instead, they are produced by the BLIP$\dag$ and BLIP$_\fusecap{}$ caption generators.
Captions from BLIP$\dag$ and BLIP$_\fusecap{}$ are assessed using retrieval models fine-tuned on original and enriched captions, respectively,
both of these retrieval models are the ones discussed in \Cref{Se:image-text-retrieval}.
\Cref{generators_coco_retrieval} shows that by using the captions from the captioners instead of the ground truth captions, BLIP$_\fusecap{}$ retains its superiority over BLIP$\dag$ in both image-to-text and text-to-image retrievals. Impressively, using the BLIP$_\fusecap{}$ captions, the retrieval performance is on par with results achieved using the ground truth enriched captions in \Cref{coco_retrieval}.
In contrast, BLIP$\dag$ exhibits a significant performance drop when compared to its results with the corresponding ground truth captions.
\begin{table}
\centering
\begin{tabular}{l *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}}}
\toprule
& \multicolumn{6}{c}{COCO Retrieval} \\
& \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
Model &
R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 \\
\midrule
\vspace{-0.3em}
\multirow{1.5}{*}{BLIP$\dag$} & 56.3 & 83.0 & 90.3 & 54.5 & 81.2 & 88.7 \\
\vspace{-0.3em}
& \footnotesize{\color{OrangeRed}-18.8\%} & \footnotesize{\color{OrangeRed}-9.7\%} & \footnotesize{\color{OrangeRed}-6.1\%} & \footnotesize{\color{OrangeRed}-3.7\%} & \footnotesize{\color{OrangeRed}-1.2\%} & \footnotesize{\color{OrangeRed}-0.5\%} \\
%\cmidrule(lr){2-7}
\midrule
\vspace{-0.3em}
\multirow{1.5}{*}{BLIP$^*_\fusecap{}$} & \textbf{95.0} & \textbf{98.8} & \textbf{99.2} & \textbf{94.5} & \textbf{98.7} & \textbf{99.3} \\
\vspace{-0.3em}
& \footnotesize{\color{OrangeRed}-2.2\%} & \footnotesize{\color{OrangeRed}-0.7\%} & \footnotesize{\color{OrangeRed}-0.7\%} & \footnotesize{\color{OliveGreen}+1.5\%} & \footnotesize{\color{OliveGreen}+1.3\%} & \footnotesize{\color{OliveGreen}+1\%} \\
\bottomrule
\end{tabular}
% \vspace{7pt}
\caption{\textbf{Image-text retrieval results using generated captions.}
% Performance on the COCO retrieval test set is evaluated using captions generated by captioning models presented in \Cref{subsec:stage2}, rather than the ground truth captions.
Performance on COCO retrieval (test sets) using captions generated by the models presented in \Cref{subsec:stage2} rather than ground truth captions.
The models used for evaluation are the ones referenced in \Cref{Se:image-text-retrieval}.
The highlighted percentages indicate the performance difference when compared to the use of corresponding ground truth captions in \Cref{coco_retrieval}.
% {\color{red} Highlited percentages denote the accuracy discrepancies when using model-generated captions, in contrast to the results obtained with datasets ground-truth showcased in \Cref{coco_retrieval}.}
Notably, BLIP$_\fusecap{}$ captions excel over BLIP$\dag$, a trend consistent with ground truth captions.
Moreover, BLIP$_\fusecap{}$ maintains a performance parity with the ground truth datasets, while BLIP$\dag$ shows a noticeable decline in performance.
}
\label{generators_coco_retrieval}
\vspace{-0.4cm}
\end{table}
% \Cref{generators_coco_retrieval} demonstrates that the captions generated by BLIP$_\fusecap{}$ not only maintain the corresponding model's retrieval advantage over BLIP$\dag$, as observed with ground truth captions in \Cref{coco_retrieval}, but also further amplifies this advantage.
% As opposed to BLIP$_\fusecap{}$, BLIP$\dag$ shows a significant performance drop relative to their corresponding ground truth captions.
% In contrast, retrievals using captions from BLIP$_\fusecap{}$ consistently align with the results of the ground truth captions.
% This distinction is evident in both image-to-text and text-to-image retrievals.
% \noam{explain R1 ...?}
% \begin{table}
% \centering
% \begin{tabular}{l *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}} | *{3}{c@{\hspace{5pt}}}}
% \toprule
% & \multicolumn{6}{c}{COCO Retrieval} & \multicolumn{6}{c}{Flickr-ZS Retrieval} \\
% & \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} & \multicolumn{3}{c}{img $\rightarrow$ text} & \multicolumn{3}{c}{text $\rightarrow$ img} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10} \cmidrule(lr){11-13}
% Model &
% R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 & R\kern-0.1em@\kern-0.1em1 & R\kern-0.1em@\kern-0.1em5 & R\kern-0.1em@\kern-0.1em10 \\
% \midrule
% BLIP$\dag$ & 75.1 & 92.7 & 96.4 & 58.2 & 82.4 & 89.2 & 92.4 & 99.1 & 99.9 & 79.7 & 95.2 & 97.5 \\
% BLIP-L & 82.4 & 95.4 & 97.9 & 65.2 & 86.3 & 91.8 & 96.7 & 100.0 & 100.0 & 86.7 & 97.3 & 98.7 \\
% BLIP2 & 85.4 & 97.0 & 98.5 & 68.3 & 87.7 & 92.6 & 97.6 & 100.0 & 100.0 & 89.7 & 98.1 & 98.9 \\
% \midrule
% BLIP$^*_\fusecap{}$ & \textbf{97.2} & \textbf{99.5} & \textbf{99.9} & \textbf{93.0} & \textbf{97.4} & \textbf{98.3} & \textbf{99.5} & \textbf{100} & \textbf{100} & \textbf{97.3} & \textbf{99.5} & \textbf{99.7} \\
% \bottomrule
% \end{tabular}
% \vspace{7pt}
% \caption{\textbf{Image-text retrieval results.} Performance on COCO and Flickr image-text retrieval (test sets) in fine-tuning and zero-shot settings, respectively. The "*" symbol indicates that the model was trained and tested on our enriched dataset. These results attest that given rich captions, BLIP$_\fusecap{}$ significantly outperforms existing methods, which utilize standard captions.}
% \label{coco_retrieval}
% \vspace{-0.7cm}
% \end{table}
% \noam{
% How can I clarify that we are tested using the enriched captions and the normal one with normal captions?
% }
% \noam{
% add image examples?
% }
% \subsubsection{Zero-Shot Image-Text Retrieval}
\subsection{Large-Scale Data Influence}\label{sec:pretrain_significance}
Typical VLP frameworks involve pre-training vision-language models on image-caption pairs before fine-tuning them on downstream tasks.
Given this structure, one might argue that the generation of enriched captions could serve as an additional downstream task.
This would suggest using the standard captions dataset for pre-training, followed by fine-tuning on a smaller enriched dataset.
Yet, as mentioned in \Cref{Se:caption-generation}, our approach with the BLIP$_\fusecap{}$ model deviates from this by pre-training on enriched datasets instead of the conventional ones. In this section, we evaluate the impact of this deviation.
% {\color{red} Under this paradigm, generating enriched captions can be conceived as a specific downstream task which could be addressed by fine-tuning a model—pre-trained on \textbf{standard} caption datasets — on a dataset encapsulating detailed captions. \textcolor{blue}{$<-$ really messy sentence} Yet, as highlighted in \Cref{Se:caption-generation}, we pursue an alternative strategy, and perform pre-training on enriched datasets. In this subsection, we provide an ablation study underscoring the advantages of VLP with enriched captions.}
We compare the performance of models that are pre-trained on both the standard and enriched datasets.
Both models are subsequently fine-tuned using the enriched COCO dataset.
Unlike the evaluation method detailed in Section \ref{Se:experiments}, which is reference-free, this section employs conventional, reference-based evaluation metrics, as both models are fine-tuned and tested using the same dataset.
\Cref{ablation_table} reveals that the model pre-trained on the enriched captions outperforms its counterpart, which was pre-trained on the original captions, across all metrics when both are fine-tuned on the augmented COCO dataset.
This outcome underscores the importance of using large-scale enriched datasets for pre-training and validates our automated dataset creation approach.
Interestingly, the model that is pre-trained on the enriched captions and fine-tuned using the standard COCO dataset performs comparably to the model pre-trained on the original captions and fine-tuned on the same dataset.
% {\color{red} Finally, \Cref{ablation_table} highlights that when BLIP$_\fusecap{}_P$ is fine-tuned on the standard COCO, its performance parallels that of the BLIP$\dag_P$ model fine-tuned on the same dataset. This underscores that fine-tuning on our proposed enriched dataset does not compromise the generality of the pre-trained model, nor its suitability for conventional tasks.}
% {
% \color{blue} I am not sure this is an additional claim we want to make.
% }
% \begin{table}
% \label{ablation_table}
% \centering
% \begin{tabular}{cc|c|ccccc}
% \toprule
% \makecell{Pre-training\\Data} && \makecell{Fine-tune+Test\\Data} && B\kern-0.1em@\kern-0.1em4 & CIDEr & SPICE \\
% \midrule
% N && N & & 37.8 & 126.5 & 22.9 \\
% F && N & & 38.4 & 128.7 & 23.0 \\
% \midrule
% N && F & & 35.4 & 111.4 & 25.0 \\
% F && F & & 37.3 & 123.1 & 26.8 \\
% \bottomrule
% \end{tabular}
% \vspace{7pt}
% \caption{\noam{explain that N stands for the normal coco dataset and F for the fusing dataset, maybe not write clipscore?} \textcolor{red}{we have a problem here with the clip score!!!!}}
% \end{table}
% \noam{add table!}
% \noam{
% write missing:
% in table write about:
% - change annotations according to table
% }
\section{Limitations}
% We found that creating enriched captions is more prone to image description errors.
% \textcolor{red}{
% Qualitatively, there are instances where the \fusecap{} data contains relationships between objects in the image that may not align with the actual depicted relationships.
% We believe this could be aliviated by inserting more accurate spatial information to the LLM Fuser.
% However, as qualitatively shown in example ???, the LLM failed to use more accurate positional inputs.
% }
% \textcolor{orange}{
% In our qualitative assessments of the \fusecap{} dataset, we observed relationship errors between depicted objects in certain images as can be seen in Figure \ref{????}.
% One potential mitigation could involve enhancing the LLM Fuser with finer-grained spatial information.
% However, preliminary trials, as evidenced in Figure \ref{????} indicate that current LLMs might not benefit from improved positional inputs.}
% This is reflected in the human evaluation study (\Cref{Se:data-validation}),
% where people preferred the original caption for $27.06\%$ of the images.
% We also found that our method sometimes has difficulty expressing the true relationships and dependencies between different elements in the image. For example, in the lower image of \Cref{fig:generatior_example2} our approach could not describe that the cat is sitting between the man and the laptop..
In the human evaluation study (\Cref{Se:data-validation}), a subset of participants favored the original caption.
This may be due to our fusing process occasionally missing inter-element dependencies, as seen in \Cref{fig:generatior_example2}, where the cat's position between the man and the laptop was not captured.
Future research might integrate finer visual details, like segmentation, and refine the LLM fuser accordingly.
\vspace{-8pt}
\paragraph{Ethical considerations} \fusecap{} dataset was constructed using FlanT5-XL model \cite{chung2022scaling}, which was trained on unfiltered data potentially laden with explicit content or inherent biases. Consequently, the proposed dataset may replicate biases from the original model.
\section{Conclusions}
\begin{table}
% \vspace{-0.5cm}
\centering
\begin{tabular}{c|c|ccc}
\toprule
\makecell{Pre-training\\Data} & \makecell{Fine-tune\\+Test Data} & B\kern-0.1em@\kern-0.1em4 & CIDEr & SPICE \\
\midrule
Standard & Standard & 37.8 & 126.5 & 22.9 \\
\fusecap{} & Standard & \textbf{38.4} & \textbf{128.7} & \textbf{23.0} \\
\midrule
Standard & \fusecap{} & 35.4 & 111.4 & 25.0 \\
\fusecap{}& \fusecap{} & \textbf{37.3} & \textbf{123.1} & \textbf{26.8} \\
\bottomrule
\end{tabular}
% \vspace{4pt}
\caption{\textbf{Influence of Large-Scale Data.}
Pre-training with the \fusecap{} dataset significantly outperforms pre-training on standard data when fine-tuned for enriched caption generation.
This underscores the value of the large-scale \fusecap{} dataset and its automated creation approach.
Additionally, when fine-tuned for standard captioning, the \fusecap{} pre-trained model is comparable to, if not slightly better than, the standard data pre-trained model.
}
\label{ablation_table}
\vspace{-0.3cm}
\end{table}
In this paper, we address the problem of generating highly descriptive and detailed image captions. We observe that existing state-of-the-art methods produce short and often oversimplified captions that fail to capture the intricate details in images. We hypothesize that this is due to datasets limitation, \textit{i.e.,} existing training data composed of concise captions. Thus, the tendency to provide such captions is distilled into the trained models, regardless of their architecture or training method. Therefore, we introduce a novel and generic data-centric automated strategy to enrich existing image captions, termed \fusecap{}.
Specifically, this approach harnesses visual experts to extract meaningful information from images and an LLM to fuse such data into the existing captions, yielding enriched ones. We apply \fusecap{} to different widespread datasets and generate 12M image-enriched caption pairs.
The augmented image captions quality is evaluated qualitatively using a human evaluation survey, and quantitatively, using different evaluation methods.
Finally, we demonstrate the effectiveness of the enriched data by utilizing it to train an image captioning model, which outperforms significantly larger state-of-the-art methods.
We posit that our research highlights the marked potential of LLMs in enhancing powerful data-centric approaches in computer vision.
% In this paper, we address the problem of image captioning, focusing in particular on image captioning datasets. We found that the image captions in the commonly used datasets are not rich enough and do not capture semantically interesting elements of the image.
% To address this limitation, we propose \fusecap{} -- a novel caption enrichment method that fuses visual expert with the original caption to create an enriched caption.
% We apply our method to different datasets and generate dataset with 12M image-enriched caption pairs. Then, we validate the generated data and show that it obtains a higher CLIPScore~\cite{hessel2022clipscore} and is preferred by humans.
% Finally, we demonstrate the effectiveness of the enriched data in caption generation and in the image-to-text and text-to-image retrieval tasks.
% % There are some limitations in our method that can be addressed in future work. We found that creating long captions is more prone to image description errors. Qualitatively, we found that for some images, our method makes untrue statements describing objects that are not present in the actual image. We have also found that our method sometimes has difficulty expressing the true relationships and dependencies between different elements in the image. For example, the inability to connect a person to the item he holds in his hand.
% We believe that our work can further be extended in the several directions such as
% % In addition, we believe that our work can be extended in several directions such as
% \begin{inparaenum}[(a)]
% \item investigation of visual experts other than those we have used in this work,
% \item Finding methods to mitigate the untrue output of the LLM fuser, making it more reliable,
% \item Exploring new ways to better convey the different relations between elements in the image to the enriched caption, and
% \item trying out other LLMs for the fusing part.
% \end{inparaenum}
% To this end, we make all code and datasets publicly available.
% \noam{talk about the Prismer experiment, it is a great bonus for our concept}\shaked{??}
\pagebreak
\appendix
% \begin{center}
% \LARGE\textbf{Supplementary Material}\\
% \end{center}
\section{\fusecap{} Fused Captions}
In this section, we present supplementary details about the newly proposed fused dataset presented in Section \ref{Se:method}.
\Cref{fig:supp_fusecap_examples} presents examples of this enriched data, adding to the ones shown in Figure \ref{fig:generatior_example2}.
% Does the training dataset consist of ?,?,?,? images from the COCO, SBU, CC and CC12 datasets respectively.
% \subsection{Caption Datasets}
% We consider four well-known image-caption datasets,
% \begin{itemize}
% \item \textbf{COCO (Common Objects in Context)}: COCO is dataset published in 2014 comprising 80K image-caption pairs annotated by humans. A validation and a test set are also provided.
% \item \textbf{SBU (Stony Brook University)}: SBU is a large-scale dataset comprising one million images with associated captions harvested from the Internet released in 2011. A validation set is also provided.
% \item \textbf{Conceptual Captions (CC)}: The Conceptual Captions dataset is a large-scale dataset designed for image captioning tasks proposed by Google in 2018. It contains over 3M images paired with caption web-sourced, images coupled with captions generated from alt-texts, providing a real-world scenario for developing and evaluating image captioning models.
% \item \textbf{CC12M}: An extension of the original Conceptual Captions dataset released in 2021, the CC12M includes a staggering 12 million image-caption pairs.
% \end{itemize}
% Table \ref{} details the exact number of image-caption pairs successfully retrieved from the original repositories of each dataset.
\subsection{Visual Experts Implementation}
We utilize the Visual Experts discussed in Section
\ref{Se:vision-experts}, in the following manner,
\begin{itemize}
\item \textbf{Object detection} We consider objects as valid detections if they surpass a pre-determined detection confidence threshold of 0.7.
\item \textbf{Attribute Detection} We incorporate attributes to each valid predicted object if the attribute confidence surpasses a 0.2 threshold.
\item \textbf{OCR}: We use CRAFT and Parseq with default inference parameters \cite{baek2019character, bautista2022parseq}. The text recognized is attributed to the object that has the smallest bounding box encompassing it.
\end{itemize}
\subsection{Training Set Generation for LLM Fuser with ChatGPT.}
As outlined in Section \ref{Se:llm-fuser}, we harness the zero-shot capabilities of ChatGPT to generate a compact dataset encompassing 20,000 examples. This data is subsequently used to fine-tune an open-source Large Language Model (LLM).
% We denote $\{o_i\}_{i=1}^{N}$ as the set of objects detected, $\{a^{j}_{i}\}_{i=1}^{N}$ as the attributes related to each object, and $\{t_i\}_{i=1}^{N}$ as texts related to each object.
The prompt provided to ChatGPT is as follows:
% \begin{figure}[!htb]
% \centering
% \begin{tabular}{p{0.95\textwidth}}
\noindent\makebox[\linewidth]{\rule{0.48\textwidth}{0.4pt}}
"A caption of an image is given: \textit{original caption}. \\
The following objects are detected in the image from left to right: \\
\quad A $a^1_1$, ..., $a^{k-1}_1$ and $a^{k}_1$ $o_1$ [with the following text: $t_1$]. \\
\quad \vdots \\
\quad A $a^1_n$, ..., $a^{k_{n}-1}_n$ and $a^{k_{n}}_n$ $o_n$ [with the following text: $t_n$]. \\
Write a comprehensive and concise caption of the scene using the objects detected."\\
\noindent\makebox[\linewidth]{\rule{0.48\textwidth}{0.4pt}}
\vfill
% Insert the figure
\begin{figure}[b]
\centering
\includegraphics[width=2.1\columnwidth]{figures/supp_fusecap_examples/1.pdf}
% \caption{Your caption here}
\label{fig:supp_fusecap_examples}
\vspace{1cm}
\end{figure}
\newpage
\noindent We denote $\{o_i\}_{i=1}^{N}$ as the set of objects detected, $\{a^{j}_{i}\}_{i=1}^{N}$ as the attributes related to each object, and $\{t_i\}_{i=1}^{N}$ as texts related to each object.
% \end{tabular}
% \caption{Prompt given to ChatGPT.}
% \label{fig:prompt}
% \end{figure}
\section{Training Settings}
\subsection{LLM Fuser}
Our LLM Fuser is a fine-tuned FlanT5-XL \cite{chung2022scaling} model. We used the huggingface library to fine-tune it on a single NVIDIA A40 GPU. We trained the model for $4,000$ optimization steps, with batch size of $32$, a learning rate of $5\cdot10^{-5}$, and a linear scheduling strategy. We limit the source and target length to 100 and 200 tokens, respectively.
\subsection{Caption Generator}
\Cref{fig:supp_captioner_examples} offers further examples, supplementing those found in \Cref{fig:generatior_example} and providing additional instances of the model's outputs.
These examples further emphasize the ability of the captioning model to generate semantically rich captions.
\textbf{Training.}
The pre-training and subsequent fine-tuning of the captioning model, described in Section \ref{Se:experiments}, was performed on eight NVIDIA A100 GPUs.
For setting the pre-training hyperparameters, we followed the approach outlined in the original BLIP implementation \cite{li2022blip}, except for maximum caption length, batch size, and initial learning rate.
We deviated from the original model's batch size to accommodate the increased token length used in our implementation.
To maintain stability during pre-training with a smaller batch size, we reduced the initial learning rate.
The batch size used for pre-training was $400$ ($50$ per GPU).
The initial learning rate used was $6 \cdot 10 ^{-5}$.
\textbf{Maximum Caption Length.}
The original BLIP model imposes a training and inference limit of $30$ tokens per caption. With our enrichment process, however, caption lengths tend to be longer and beyond this original threshold.
Accordingly, we have increased the maximum caption length to 60 tokens.
This limit is maintained through both the pre-training and fine-tuning stages of the caption generator, as well as during the actual generation of captions.
% This new limit is longer than $88.6\%$ of the captions in the fused captions dataset.
% \shaked{What about putting the table of the length distribution?}
\onecolumn
\renewcommand{\LTcaptype}{figure}
\begin{longtable}{c}
\captionsetup{type=figure} % specify that the following is a figure
% \label{fig:supp_fusecap_examples}\\
% \includegraphics[width=0.9\linewidth]{figures/COCO_supp.png}\\
% \includegraphics[width=0.9\linewidth]{figures/sbu_supp.png}\\
% \includegraphics[width=0.9\linewidth]{figures/cc_supp.png}\\
% \includegraphics[width=0.9\linewidth]{figures/cc12m_supp.png}\\
% \includegraphics[width=1\linewidth]{figures/supp_fusecap_examples/1.pdf}\\
\includegraphics[width=1\linewidth]{figures/supp_fusecap_examples/2.pdf}\\
\includegraphics[width=1\linewidth]{figures/supp_fusecap_examples/3.pdf}\\
% \includegraphics[width=1\linewidth]{figures/supp_fusecap_examples/4.pdf}\\
\includegraphics[width=1\linewidth]{figures/supp_fusecap_examples/5.pdf}\\
\caption{Examples of the proposed \fusecap{} enriched-captions dataset.}
\end{longtable}
\renewcommand{\LTcaptype}{figure}
\begin{longtable}{c}
\includegraphics[width=0.9\linewidth]{figures/supp_examples/1-couple.pdf}\\
% \includegraphics[width=0.9\linewidth]{figures/supp_examples/2-snowboard.pdf}\\
% \includegraphics[width=0.9\linewidth]{figures/supp_examples/3-calm_water.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/4-cow.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/5-mini-mot.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/6-horse.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/7-frisbee-woman.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/8-elephant.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/9-bird.pdf}\\
\includegraphics[width=0.9\linewidth]{figures/supp_examples/10-helicopter.pdf}\\
\includegraphics[width=0.75\linewidth]{figures/supp_examples/11-vegi-blend.pdf}\\
\includegraphics[width=0.75\linewidth]{figures/supp_examples/12-umbrella-woman.pdf}\\
\caption{Comparative illustration of captions generated by our BLIP$_\fusecap{}$ and other top-performing models.
\label{fig:supp_captioner_examples}}\\
\end{longtable}
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage[rebuttal,applications]{wacv} % use this for an Applications Track rebuttal
%\usepackage[rebuttal,algorithms]{wacv} % use this for an Algorithms Track rebuttal
% Include other packages here, before hyperref.
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref,breaklinks,colorlinks,bookmarks=false]{hyperref}
% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
% If you wish to avoid re-using figure, table, and equation numbers from
% the main paper, please uncomment the following and change the numbers
% appropriately.
%\setcounter{figure}{2}
%\setcounter{table}{1}
%\setcounter{equation}{2}
% If you wish to avoid re-using reference numbers from the main paper,
% please uncomment the following and change the counter for `enumiv' to
% the number of references you have in the main paper (here, 6).
%\let\oldthebibliography=\thebibliography
%\let\oldendthebibliography=\endthebibliography
%\renewenvironment{thebibliography}[1]{%
% \oldthebibliography{#1}%
% \setcounter{enumiv}{6}%
%}{\oldendthebibliography}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\wacvPaperID{*****} % *** Enter the CVPR Paper ID here
\def\confName{WACV}
\def\confYear{2024}
\begin{document}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{\LaTeX\ Guidelines for Author Response} % **** Enter the paper title here
\maketitle
\thispagestyle{empty}
\appendix
%%%%%%%%% BODY TEXT - ENTER YOUR RESPONSE BELOW
\section{Introduction}
After receiving paper reviews for first round submissions, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file (second round submissions do not have a rebuttal step).
Please follow the steps and style guidelines outlined below for submitting your author response.
The author rebuttal is optional and, following similar guidelines to previous WACV conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.
Reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.
Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).
%-------------------------------------------------------------------------
\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.
%------------------------------------------------------------------------
\section{Formatting your Response}
{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}
All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.
Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.
Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.
List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.
\begin{figure}[t]
\centering
\fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}
%\includegraphics[width=0.8\linewidth]{egfigure.eps}
\caption{Example of caption. It is set in Roman so that mathematics
(always set in Roman: $B \sin A = A \sin B$) may be included without an
ugly clash.}
\label{fig:onecol}
\end{figure}
To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.
%-------------------------------------------------------------------------
\subsection{Illustrations, graphs, and photographs}
All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.
When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
\usepackage{graphicx} ...
\includegraphics[width=0.8\linewidth]
{myfile.pdf}
\end{verbatim}
}
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\end{document}
% WACV 2024 Paper Template
% based on the CVPR 2023 template (https://media.icml.cc/Conferences/CVPR2023/cvpr2023-author_kit-v1_1-1.zip) with 2-track changes from the WACV 2023 template (https://github.com/wacv-pcs/WACV-2023-Author-Kit)
% based on the CVPR template provided by Ming-Ming Cheng (https://github.com/MCG-NKU/CVPR_Template)
% modified and extended by Stefan Roth (stefan.roth@NOSPAMtu-darmstadt.de)
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
% \usepackage[review,algorithms]{wacv} % To produce the REVIEW version for the algorithms track
%\usepackage[review,applications]{wacv} % To produce the REVIEW version for the applications track
% \usepackage{wacv} % To produce the CAMERA-READY version
\usepackage[pagenumbers]{wacv} % To force page numbers, e.g. for an arXiv version
\usepackage[dvipsnames]{xcolor}
% Include other packages here, before hyperref.
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{multirow}
\usepackage{xhfill}
% \usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
% \usepackage{color}
\usepackage{changepage}
\usepackage{subcaption}
\usepackage{scrextend}
\usepackage{makecell}
\usepackage{paralist}
% \usepackage{wrapfig}
\usepackage{float}
\usepackage{indentfirst}
\usepackage{longtable}
\usepackage{caption}
\usepackage{enumitem}
\usepackage{refcount}
% \usepackage[absolute]{textpos}
\usepackage{afterpage}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues, e.g. with the
% file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete
% ReviewTempalte.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you
% should be clear).
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \usepackage{cleveref}
\usepackage{indentfirst}
\usepackage{longtable}
\usepackage{caption}
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% \newcommand{\ganz}[1]{\textcolor{LimeGreen}{Ganz: #1}}
% \newcommand{\shaked}[1]{\textcolor{Cyan}{Brody: #1}}
\newcommand{\fusecap}{\textsc{FuseCap}}
% Uncomment before submission
% \newcommand{\ganz}[1]{}
% \newcommand{\shaked}[1]{}
% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\wacvPaperID{1604} % *** Enter the WACV Paper ID here
\def\confName{WACV}
\def\confYear{2024}
\begin{document}
%%%%%%%%% TITLE - PLEASE UPDATE
% \title{\fusecap{}: Leveraging Large Language Models to Fuse \\ Visual Data into Enriched Image Captions}
\title{\fusecap{}: Leveraging Large Language Models \\ for Enriched Fused Image Captions}
\author{
Noam Rotstein* \qquad David Bensa\"id* \qquad Shaked Brody \qquad Roy Ganz \qquad Ron Kimmel
\\
% \tt\small\{snoamr, dben-said, shakedbr, ganz, ron\}@cs.technion.ac.il
% \\
Technion - Israel Institute of Technology
\\
\tt\small *Indicates equal contribution.
}
\maketitle
\input{00-abstract}
\input{01-introv2}
\input{02-related}
\input{03-method}
\input{04-experiments-a.tex}
\input{04-experiments-b.tex}
\input{04-experiments-c.tex}
\input{04-experiments-d.tex}
\input{05-Limitations}
\input{06-Conclusions}
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{ieee_fullname}
\bibliography{egbib}
}
\clearpage
\input{07-supplementary}
\end{document}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{aberdam2023clipter}
Aviad Aberdam, David Bensa{\"\i}d, Alona Golts, Roy Ganz, Oren Nuriel, Royee
Tichauer, Shai Mazor, and Ron Litman.
\newblock Clipter: Looking at the bigger picture in scene text recognition.
\newblock {\em arXiv preprint arXiv:2301.07464}, 2023.
\bibitem{aberdam2022multimodal}
Aviad Aberdam, Roy Ganz, Shai Mazor, and Ron Litman.
\newblock Multimodal semi-supervised learning for text recognition.
\newblock {\em arXiv preprint arXiv:2205.03873}, 2022.
\bibitem{agrawal2019nocaps}
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark
Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.
\newblock Nocaps: Novel object captioning at scale.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 8948--8957, 2019.
\bibitem{anderson2018bottom}
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen
Gould, and Lei Zhang.
\newblock Bottom-up and top-down attention for image captioning and visual
question answering.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 6077--6086, 2018.
\bibitem{baek2019character}
Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee.
\newblock Character region awareness for text detection.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 9365--9374, 2019.
\bibitem{bai2018survey}
Shuang Bai and Shan An.
\newblock A survey on automatic image caption generation.
\newblock {\em Neurocomputing}, 311:291--304, 2018.
\bibitem{longstanding0}
K. Barnard and D. Forsyth.
\newblock Learning the semantics of words and pictures.
\newblock In {\em Proceedings Eighth IEEE International Conference on Computer
Vision. ICCV 2001}, volume~2, pages 408--415 vol.2, 2001.
\bibitem{bautista2022parseq}
Darwin Bautista and Rowel Atienza.
\newblock Scene text recognition with permuted autoregressive sequence models.
\newblock In {\em European Conference on Computer Vision}, pages 178--196,
Cham, 10 2022. Springer Nature Switzerland.
\bibitem{biten2019scene}
Ali~Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar{\c{c}}al Rusinol,
Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas.
\newblock Scene text visual question answering.
\newblock In {\em Proceedings of the IEEE/CVF international conference on
computer vision}, pages 4291--4301, 2019.
\bibitem{brooks2022instructpix2pix}
Tim Brooks, Aleksander Holynski, and Alexei~A Efros.
\newblock Instructpix2pix: Learning to follow image editing instructions.
\newblock {\em arXiv preprint arXiv:2211.09800}, 2022.
\bibitem{brown2020language}
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared~D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et~al.
\newblock Language models are few-shot learners.
\newblock {\em Advances in neural information processing systems},
33:1877--1901, 2020.
\bibitem{changpinyo2021conceptual}
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.
\newblock Conceptual 12m: Pushing web-scale image-text pre-training to
recognize long-tail visual concepts.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 3558--3568, 2021.
\bibitem{chen2023vlp}
Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu,
and Bo Xu.
\newblock Vlp: A survey on vision-language pre-training.
\newblock {\em Machine Intelligence Research}, 20(1):38--56, 2023.
\bibitem{pali}
Xi Chen, Xiao Wang, Soravit Changpinyo, A.~J. Piergiovanni, Piotr Padlewski,
Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer,
Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari,
Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo,
Mojtaba Seyedhosseini, Chao Jia, Burcu~Karagol Ayan, Carlos Riquelme, Andreas
Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.
\newblock Pali: {A} jointly-scaled multilingual language-image model.
\newblock {\em CoRR}, 2022.
\bibitem{uniter}
Yen{-}Chun Chen, Linjie Li, Licheng Yu, Ahmed~El Kholy, Faisal Ahmed, Zhe Gan,
Yu Cheng, and Jingjing Liu.
\newblock {UNITER:} universal image-text representation learning.
\newblock In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan{-}Michael
Frahm, editors, {\em Computer Vision - {ECCV} 2020 - 16th European
Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part {XXX}}.
Springer, 2020.
\bibitem{chen2020uniter}
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El~Kholy, Faisal Ahmed, Zhe Gan, Yu
Cheng, and Jingjing Liu.
\newblock Uniter: Universal image-text representation learning.
\newblock In {\em Computer Vision--ECCV 2020: 16th European Conference,
Glasgow, UK, August 23--28, 2020, Proceedings, Part XXX}, pages 104--120.
Springer, 2020.
\bibitem{cho2021unifying}
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.
\newblock Unifying vision-and-language tasks via text generation.
\newblock In {\em International Conference on Machine Learning}, pages
1931--1942. PMLR, 2021.
\bibitem{chowdhery2022palm}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian
Gehrmann, et~al.
\newblock Palm: Scaling language modeling with pathways.
\newblock {\em arXiv preprint arXiv:2204.02311}, 2022.
\bibitem{chung2022scaling}
Hyung~Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus,
Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et~al.
\newblock Scaling instruction-finetuned language models.
\newblock {\em arXiv preprint arXiv:2210.11416}, 2022.
\bibitem{donahue2015long}
Jeffrey Donahue, Lisa Anne~Hendricks, Sergio Guadarrama, Marcus Rohrbach,
Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.
\newblock Long-term recurrent convolutional networks for visual recognition and
description.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 2625--2634, 2015.
\bibitem{dosovitskiy2020image}
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et~al.
\newblock An image is worth 16x16 words: Transformers for image recognition at
scale.
\newblock {\em arXiv preprint arXiv:2010.11929}, 2020.
\bibitem{dou2022empirical}
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang,
Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et~al.
\newblock An empirical study of training end-to-end vision-and-language
transformers.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 18166--18176, 2022.
\bibitem{farhadi2010every}
Ali Farhadi, Mohsen Hejrati, Mohammad~Amin Sadeghi, Peter Young, Cyrus
Rashtchian, Julia Hockenmaier, and David Forsyth.
\newblock Every picture tells a story: Generating sentences from images.
\newblock In {\em Computer Vision--ECCV 2010: 11th European Conference on
Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings,
Part IV 11}, pages 15--29. Springer, 2010.
\bibitem{ganz2023clipag}
Roy Ganz and Michael Elad.
\newblock Clipag: Towards generator-free text-to-image generation.
\newblock {\em arXiv preprint arXiv:2306.16805}, 2023.
\bibitem{ganz2023towards}
Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, and Ron
Litman.
\newblock Towards models that can see and read.
\newblock {\em arXiv preprint arXiv:2301.07389}, 2023.
\bibitem{hessel2022clipscore}
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan~Le Bras, and Yejin Choi.
\newblock Clipscore: A reference-free evaluation metric for image captioning,
2022.
\bibitem{hodosh2013framing}
Micah Hodosh, Peter Young, and Julia Hockenmaier.
\newblock Framing image description as a ranking task: Data, models and
evaluation metrics.
\newblock {\em Journal of Artificial Intelligence Research}, 47:853--899, 2013.
\bibitem{hu2022scaling}
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and
Lijuan Wang.
\newblock Scaling up vision-language pre-training for image captioning.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 17980--17989, 2022.
\bibitem{jia2021scaling}
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le,
Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
\newblock Scaling up visual and vision-language representation learning with
noisy text supervision.
\newblock In {\em International Conference on Machine Learning}, pages
4904--4916. PMLR, 2021.
\bibitem{johnson2016densecap}
Justin Johnson, Andrej Karpathy, and Li Fei-Fei.
\newblock Densecap: Fully convolutional localization networks for dense
captioning.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4565--4574, 2016.
\bibitem{karpathy2015deep}
Andrej Karpathy and Li Fei-Fei.
\newblock Deep visual-semantic alignments for generating image descriptions.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 3128--3137, 2015.
\bibitem{longstanding3}
Andrej Karpathy, Armand Joulin, and Li Fei{-}Fei.
\newblock Deep fragment embeddings for bidirectional image sentence mapping.
\newblock In {\em Advances in Neural Information Processing Systems 27: Annual
Conference on Neural Information Processing Systems 2014, December 8-13 2014,
Montreal, Quebec, Canada}, 2014.
\bibitem{krishna2017visual}
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua
Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David~A Shamma, et~al.
\newblock Visual genome: Connecting language and vision using crowdsourced
dense image annotations.
\newblock {\em International journal of computer vision}, 123:32--73, 2017.
\bibitem{longstanding1}
Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi,
Alexander~C. Berg, and Tamara~L. Berg.
\newblock Baby talk: Understanding and generating simple image descriptions.
\newblock In {\em The 24th {IEEE} Conference on Computer Vision and Pattern
Recognition, {CVPR} 2011, Colorado Springs, CO, USA, 20-25 June 2011}, 2011.
\bibitem{kulkarni2013babytalk}
Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li,
Yejin Choi, Alexander~C Berg, and Tamara~L Berg.
\newblock Babytalk: Understanding and generating simple image descriptions.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
35(12):2891--2903, 2013.
\bibitem{longstanding2}
Polina Kuznetsova, Vicente Ordonez, Alexander~C. Berg, Tamara~L. Berg, and
Yejin Choi.
\newblock Collective generation of natural image descriptions.
\newblock In {\em The 50th Annual Meeting of the Association for Computational
Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island,
Korea - Volume 1: Long Papers}, 2012.
\bibitem{li2023blip}
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
\newblock Blip-2: Bootstrapping language-image pre-training with frozen image
encoders and large language models.
\newblock {\em arXiv preprint arXiv:2301.12597}, 2023.
\bibitem{li2022blip}
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
\newblock Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation.
\newblock In {\em International Conference on Machine Learning}, pages
12888--12900. PMLR, 2022.
\bibitem{li2021align}
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong,
and Steven Chu~Hong Hoi.
\newblock Align before fuse: Vision and language representation learning with
momentum distillation.
\newblock {\em Advances in neural information processing systems},
34:9694--9705, 2021.
\bibitem{align_before_fuse}
Junnan Li, Ramprasaath~R. Selvaraju, Akhilesh Gotmare, Shafiq~R. Joty, Caiming
Xiong, and Steven~Chu{-}Hong Hoi.
\newblock Align before fuse: Vision and language representation learning with
momentum distillation.
\newblock In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann~N. Dauphin, Percy
Liang, and Jennifer~Wortman Vaughan, editors, {\em Advances in Neural
Information Processing Systems 34: Annual Conference on Neural Information
Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual}, 2021.
\bibitem{li2019visualbert}
Liunian~Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
\newblock Visualbert: A simple and performant baseline for vision and language.
\newblock {\em arXiv preprint arXiv:1908.03557}, 2019.
\bibitem{li2020oscar}
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan
Wang, Houdong Hu, Li Dong, Furu Wei, et~al.
\newblock Oscar: Object-semantics aligned pre-training for vision-language
tasks.
\newblock In {\em Computer Vision--ECCV 2020: 16th European Conference,
Glasgow, UK, August 23--28, 2020, Proceedings, Part XXX 16}, pages 121--137.
Springer, 2020.
\bibitem{lin2004rouge}
Chin-Yew Lin.
\newblock Rouge: A package for automatic evaluation of summaries.
\newblock In {\em Text summarization branches out}, pages 74--81, 2004.
\bibitem{lin2014microsoft}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In {\em Computer Vision--ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part V 13}, pages 740--755.
Springer, 2014.
\bibitem{liu2019generating}
Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo.
\newblock Generating diverse and descriptive image captions using visual
paraphrases.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 4240--4249, 2019.
\bibitem{liu2023prismer}
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima
Anandkumar.
\newblock Prismer: A vision-language model with an ensemble of experts.
\newblock {\em arXiv preprint arXiv:2303.02506}, 2023.
\bibitem{liu2018show}
Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang.
\newblock Show, tell and discriminate: Image captioning by self-retrieval with
partially labeled data.
\newblock In {\em Proceedings of the European conference on computer vision
(ECCV)}, pages 338--354, 2018.
\bibitem{luo2018discriminability}
Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich.
\newblock Discriminability objective for training descriptive captions.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 6964--6974, 2018.
\bibitem{longstanding4}
Rebecca Mason and Eugene Charniak.
\newblock Nonparametric method for data-driven image captioning.
\newblock In {\em Proceedings of the 52nd Annual Meeting of the Association for
Computational Linguistics, {ACL} 2014, June 22-27, 2014, Baltimore, MD, USA,
Volume 2: Short Papers}, 2014.
\bibitem{ordonez2011im2text}
Vicente Ordonez, Girish Kulkarni, and Tamara Berg.
\newblock Im2text: Describing images using 1 million captioned photographs.
\newblock {\em Advances in neural information processing systems}, 24, 2011.
\bibitem{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In {\em Proceedings of the 40th annual meeting of the Association for
Computational Linguistics}, pages 311--318, 2002.
\bibitem{peng2023instruction}
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.
\newblock Instruction tuning with gpt-4.
\newblock {\em arXiv preprint arXiv:2304.03277}, 2023.
\bibitem{radford2021learning}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et~al.
\newblock Learning transferable visual models from natural language
supervision.
\newblock In {\em International conference on machine learning}, pages
8748--8763. PMLR, 2021.
\bibitem{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter~J Liu.
\newblock Exploring the limits of transfer learning with a unified text-to-text
transformer.
\newblock {\em The Journal of Machine Learning Research}, 21(1):5485--5551,
2020.
\bibitem{longstanding5}
Krishnan Ramnath, Simon Baker, Lucy Vanderwende, Motaz~Ahmad El{-}Saban,
Sudipta~N. Sinha, Anitha Kannan, Noran Hassan, Michel Galley, Yi Yang, Deva
Ramanan, Alessandro Bergamo, and Lorenzo Torresani.
\newblock Autocaption: Automatic caption generation for personal photos.
\newblock In {\em {IEEE} Winter Conference on Applications of Computer Vision,
Steamboat Springs, CO, USA, March 24-26, 2014}, 2014.
\bibitem{ren2015faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal
networks.
\newblock {\em Advances in neural information processing systems}, 28, 2015.
\bibitem{schick2023toolformer}
Timo Schick, Jane Dwivedi-Yu, Roberto Dess{\`\i}, Roberta Raileanu, Maria
Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
\newblock Toolformer: Language models can teach themselves to use tools.
\newblock {\em arXiv preprint arXiv:2302.04761}, 2023.
\bibitem{sharma2018conceptual}
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
\newblock Conceptual captions: A cleaned, hypernymed, image alt-text dataset
for automatic image captioning.
\newblock In {\em Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers)}, pages 2556--2565, 2018.
\bibitem{shi2021enhancing}
Zhan Shi, Hui Liu, and Xiaodan Zhu.
\newblock Enhancing descriptive image captioning with natural language
inference.
\newblock In {\em Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 2: Short Papers)}, pages 269--277, 2021.
\bibitem{su2019vl}
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.
\newblock Vl-bert: Pre-training of generic visual-linguistic representations.
\newblock {\em arXiv preprint arXiv:1908.08530}, 2019.
\bibitem{sun2017revisiting}
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.
\newblock Revisiting unreasonable effectiveness of data in deep learning era.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision}, pages 843--852, 2017.
\bibitem{tan2019lxmert}
Hao Tan and Mohit Bansal.
\newblock Lxmert: Learning cross-modality encoder representations from
transformers.
\newblock {\em arXiv preprint arXiv:1908.07490}, 2019.
\bibitem{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{vedantam2015cider}
Ramakrishna Vedantam, C Lawrence~Zitnick, and Devi Parikh.
\newblock Cider: Consensus-based image description evaluation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4566--4575, 2015.
\bibitem{vicente2016large}
Tom{\'a}s F~Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris
Samaras.
\newblock Large-scale training of shadow detectors with noisily-annotated
shadow examples.
\newblock In {\em Computer Vision--ECCV 2016: 14th European Conference,
Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14},
pages 816--832. Springer, 2016.
\bibitem{vinyals2015show}
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
\newblock Show and tell: A neural image caption generator.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 3156--3164, 2015.
\bibitem{wang2022position}
Alex~Jinpeng Wang, Pan Zhou, Mike~Zheng Shou, and Shuicheng Yan.
\newblock Position-guided text prompt for vision-language pre-training.
\newblock {\em arXiv preprint arXiv:2212.09737}, 2022.
\bibitem{wang2022git}
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan,
Zicheng Liu, Ce Liu, and Lijuan Wang.
\newblock Git: A generative image-to-text transformer for vision and language.
\newblock {\em arXiv preprint arXiv:2205.14100}, 2022.
\bibitem{git}
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan,
Zicheng Liu, Ce Liu, and Lijuan Wang.
\newblock {GIT:} {A} generative image-to-text transformer for vision and
language.
\newblock {\em CoRR}, 2022.
\bibitem{wang2022ofa}
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma,
Chang Zhou, Jingren Zhou, and Hongxia Yang.
\newblock Ofa: Unifying architectures, tasks, and modalities through a simple
sequence-to-sequence learning framework, 2022.
\bibitem{wang2023caption}
Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe
Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et~al.
\newblock Caption anything: Interactive image description with diverse
multimodal controls.
\newblock {\em arXiv preprint arXiv:2305.02677}, 2023.
\bibitem{wang2021simvlm}
Zirui Wang, Jiahui Yu, Adams~Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.
\newblock Simvlm: Simple visual language model pretraining with weak
supervision.
\newblock {\em arXiv preprint arXiv:2108.10904}, 2021.
\bibitem{simvlm}
Zirui Wang, Jiahui Yu, Adams~Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.
\newblock Simvlm: Simple visual language model pretraining with weak
supervision.
\newblock In {\em The Tenth International Conference on Learning
Representations, {ICLR} 2022, Virtual Event, April 25-29, 2022}, 2022.
\bibitem{wei2021finetuned}
Jason Wei, Maarten Bosma, Vincent~Y Zhao, Kelvin Guu, Adams~Wei Yu, Brian
Lester, Nan Du, Andrew~M Dai, and Quoc~V Le.
\newblock Finetuned language models are zero-shot learners.
\newblock {\em arXiv preprint arXiv:2109.01652}, 2021.
\bibitem{xie2017aggregated}
Saining Xie, Ross Girshick, Piotr Doll{\'a}r, Zhuowen Tu, and Kaiming He.
\newblock Aggregated residual transformations for deep neural networks.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 1492--1500, 2017.
\bibitem{xu2015show}
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan
Salakhudinov, Rich Zemel, and Yoshua Bengio.
\newblock Show, attend and tell: Neural image caption generation with visual
attention.
\newblock In {\em International conference on machine learning}, pages
2048--2057. PMLR, 2015.
\bibitem{yang2017dense}
Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li.
\newblock Dense captioning with joint inference and visual context.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 2193--2202, 2017.
\bibitem{yang2016review}
Zhilin Yang, Ye Yuan, Yuexin Wu, William~W Cohen, and Russ~R Salakhutdinov.
\newblock Review networks for caption generation.
\newblock {\em Advances in neural information processing systems}, 29, 2016.
\bibitem{zha2023data2}
Daochen Zha, Zaid~Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, and Xia Hu.
\newblock Data-centric ai: Perspectives and challenges.
\newblock In {\em Proceedings of the 2023 SIAM International Conference on Data
Mining (SDM)}, pages 945--948. SIAM, 2023.
\bibitem{zha2023data}
Daochen Zha, Zaid~Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang,
Shaochen Zhong, and Xia Hu.
\newblock Data-centric artificial intelligence: A survey.
\newblock {\em arXiv preprint arXiv:2303.10158}, 2023.
\bibitem{vinvl}
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang,
Yejin Choi, and Jianfeng Gao.
\newblock Vinvl: Revisiting visual representations in vision-language models.
\newblock In {\em {IEEE} Conference on Computer Vision and Pattern Recognition,
{CVPR} 2021, virtual, June 19-25, 2021}, 2021.
\bibitem{zhang2021vinvl}
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang,
Yejin Choi, and Jianfeng Gao.
\newblock Vinvl: Revisiting visual representations in vision-language models.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5579--5588, 2021.
\bibitem{zhang2022opt}
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui
Chen, Christopher Dewan, Mona Diab, Xian Li, Xi~Victoria Lin, et~al.
\newblock Opt: Open pre-trained transformer language models.
\newblock {\em arXiv preprint arXiv:2205.01068}, 2022.
\bibitem{zhu2023chatgpt}
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and
Mohamed Elhoseiny.
\newblock Chatgpt asks, blip-2 answers: Automatic questioning towards enriched
visual descriptions.
\newblock {\em arXiv preprint arXiv:2303.06594}, 2023.
\end{thebibliography}