MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering
Extracted source text ยท arXiv 2408.08521
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage[final]{EMNLP2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
% Custom packages
\usepackage[colorinlistoftodos,textsize=tiny,disable=true]{todonotes}
% \usepackage[colorinlistoftodos,textsize=tiny]{todonotes}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{amsmath} % for \ensuremath
\usepackage{xspace}
\usepackage{multirow}
\usepackage{booktabs}
\title{MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering}
\author{
\textbf{Zhengyuan Zhu}\textsuperscript{1}, Daniel Lee\textsuperscript{2}, Hong Zhang\textsuperscript{2},\\
\textbf{Sai Sree Harsha}\textsuperscript{2}, \textbf{Loic Feujio}\textsuperscript{2}, \textbf{Akash Maharaj}\textsuperscript{2}, \textbf{Yunyao Li}\textsuperscript{2}\\
\textsuperscript{1}University of Texas at Arlington, \textsuperscript{2}Adobe\\
\texttt{zhengyuan.zhu@mavs.uta.edu, dlee1@adobe.com, yunyaol@adobe.com }
}
\begin{document}
\input{macro}
\maketitle
\input{01_abstract}
\input{02_introduction}
\input{03_design}
\input{04_user_interface}
\input{05_dataset}
\input{06_evaluation}
\input{07_conclusion}
\newpage
% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\newpage
\appendix
\input{08_appendix}
\end{document}
\newcommand{\system}[1]{{\small \ensuremath {\mathsf{#1}}}\xspace}
\begin{abstract}
Recent advancements in retrieval-augmented generation (RAG) have demonstrated impressive performance in the question-answering (QA) task. However, most previous works predominantly focus on text-based answers. While some studies address multimodal data, they still fall short in generating comprehensive multimodal answers, particularly for explaining concepts or providing step-by-step tutorials on how to accomplish specific goals. This capability is especially valuable for applications such as enterprise chatbots and settings such as customer service and educational systems, where the answers are sourced from multimodal data.
In this paper, we introduce a simple and effective framework named MuRAR (Multimodal Retrieval and Answer Refinement). MuRAR enhances text-based answers by retrieving relevant multimodal data and refining the responses to create coherent multimodal answers. This framework can be easily extended to support multimodal answers in enterprise chatbots with minimal modifications.
Human evaluation results indicate that multimodal answers generated by MuRAR are more useful and readable compared to plain text answers.
A video demo of MuRAR is available at \href{https://youtu.be/ykGRtyVVQpU}{https://youtu.be/ykGRtyVVQpU}.
\todo[color=orange]{Record the video once the paper is finished}
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{sec:introduction}
% background
The emergence of large language models (LLMs) such as GPT models~\cite{DBLP:conf/nips/BrownMRSKDNSSAA20, DBLP:journals/corr/abs-2303-08774}, Gemini~\cite{DBLP:journals/corr/abs-2312-11805}, and LLaMA~\cite{DBLP:journals/corr/abs-2302-13971}, along with retrieval-augmented generation (RAG) techniques~\cite{DBLP:conf/nips/LewisPPPKGKLYR020, DBLP:journals/corr/abs-2312-10997}, has significantly transformed the field of question answering (QA) and enhanced the quality of responses provided by AI assistants.
% problem
However, the current generation of AI assistants is limited in their ability to provide comprehensive multimodal answers to user questions. This capability is especially important in an enterprise scenario, where answers are sourced from product documentation that is generally multimodal. Here, the presence of images, tables, and videos are often \textit{crucial} to understanding complex, domain-specific topics. Enhancing AI assistants with multimodal information can therefore greatly improve user understanding and engagement~\cite{DBLP:conf/naacl/SinghNMALS21}, offering significant benefits such as increased productivity, reduced barriers to entry, higher product adoption rates, amplified creativity, and improved user experiences.
% How do previous works define their problem? And what distinguishes our work from theirs in terms of problem definition?
Previous work~\cite{DBLP:conf/iclr/TalmorYCLWAIHB21, DBLP:journals/corr/abs-2004-12238, joshi2024robust}, has primarily focused on leveraging various techniques to better understand multimodal data to generate plain text answers for a given query. In another scenario~\cite{DBLP:conf/naacl/SinghNMALS21, DBLP:journals/corr/abs-2309-05519}, the output can either be a text answer or a text answer with a retrieved image or video attached at the end. However, the existing solutions fail to adequately address the challenges posed by complex questions that require the answer to illustrate multiple steps to achieve a goal or include multiple multimodal contents.
In summary, the main challenges are: \textbf{a)} How to retrieve the relevant multimodal data that are related and helpful to answer the user questions, and \textbf{b)} How to generate a coherent multimodal answer that integrates the retrieved multimodal data and the text answer snippets.
To address these challenges, we present MuRAR (\textbf{Mu}ltimodal \textbf{R}etrieval and \textbf{A}nswer \textbf{R}efinement), a simple and effective multimodal question-answering framework that can generate a coherent multimodal answer containing the retrieved multimodal data, including images, videos, and tables.
Our framework consists of three main components: text answer generation, source-based multimodal retrieval, and multimodal answer refinement. The text answer generation component retrieves relevant text documents based on the user's query and generates the initial text answer using an LLM. The source-based multimodal retrieval component aims to retrieve multimodal data that are relevant to the text answer snippets in the initial text answer. Finally, the multimodal answer refinement component prompt LLM to generates the final answer by incorporating the retrieved multimodal data and text answer snippets.
To showcase the effectiveness of the framework, we applied this framework on a collection of multimodal data with a question-answering agent. The modalities include text, images, tables, and videos.
We evaluated the quality of the multimodal answers on a human-annotated dataset of 300 questions and answers. The results demonstrate that the multimodal answers are superior to plain text answers in terms of naturalness and relevance.
Additionally, our framework can be adapted to other enterprise-level AI assistants by collecting topic-specific multimodal data and fine-tuning a topic-specific text retrieval model.
To the best of our knowledge, this is the first work that addresses the problem of generating coherent multimodal answers to user questions.
Our contribution can be summarized as follows:
\begin{itemize}
\item We propose a multimodal QA framework that can retrieve multimodal data relevant to the user's question and interests, and generate a coherent multimodal answer containing the retrieved multimodal data. Notably, this framework can be extended to any other enterprise-level AI assistants.
\item Our evaluation on an enterprise-level dataset demonstrates that the quality of the multimodal answers produced by our framework surpasses the plain text answers.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Design of MuRAR}
\label{sec:methodology}
\begin{figure*}[htbp]
\centering
\includegraphics[width=1\textwidth]{Figures/architecture4.png}
\caption{The architecture of the MuRAR framework.}
\label{fig:architecture}
\end{figure*}
\todo[color=orange]{Yunyao: The figure is not completely accurate. It seems that Step E leads to nothing.}
\begin{figure*}[h]
\centering
\includegraphics[width=1\textwidth]{Figures/UI_9.png}
\caption{The Interface of AI Assistant demonstrating multimodal answers are constructed by combining multimodal data retrieval and answer refinement.}
\label{fig:UI}
\end{figure*}
Formally, given a user question $q$ as input and a set of multimodal data $\mathcal{D}=\{D_{S}, D_{I}, D_{T}, D_{V}\}$, where $\{D_{S}, D_{I}, D_{T}, D_{V}\}$ denote collections of text document snippet, images, tables, and videos, respectively. The objective is to generate a multimodal answer $A_{mm}=F(S, I, T, V)$, where $F$ represents a function that organizes the retrieved multimodal data $(S, I, T, V)$ into a coherent and informative answer.
\todo{we need to clarity how to define S, I, T, V, i.e., a set of retrieved image, etc}
To achieve high quality $A_{mm}$, we propose MuRAR, as illustrated in Figure~\ref{fig:architecture}.
The MuRAR framework consists of three main components: text answer generation (Section~\ref{sec:text_answer_generation}), source-based multimodal retrieval (Section~\ref{sec:multimodal_retrieval}), and multimodal answer refinement (Section~\ref{sec:multimodal_answer_refinement}).
The text answer generation component uses a RAG approach~\cite{DBLP:conf/nips/LewisPPPKGKLYR020, DBLP:journals/corr/abs-2312-10997}, first retrieving relevant text document snippets $S=\{s_{1}, s_{2}, ..., s_{n}\} \in D_{S}$ based on user query $q$ and then generating the text answer $A_{t}$ by prompting an LLM.
\todo{briefly explain the source attribution}
Then, we apply source attribution on $A_{t}$ to identify the text answer snippets $a_{[i, j]} \in A_{t}$, where a text answer snippet is a continuous text span in $A_{t}$.
Next we retrieve multimodal data, namely, $I=\{i_{1}, i_{2}, ..., i_{m}\} \in D_{I}$, $T=\{t_{1}, t_{2}, ..., t_{k}\} \in D_{T}$, and $V=\{v_{1}, v_{2}, ..., v_{l}\} \in D_{V}$ that are relevant to the text answer snippets $a_{[i, j]}$.
Finally, the multimodal answer refinement component generates the final multimodal answer $A_{mm}$ by incorporating the retrieved multimodal data and text response snippets.
Directly prompting or using techniques such as chain-of-thought~\cite{DBLP:conf/nips/Wei0SBIXCLZ22} with LLMs for tasks involving both text and multimodal data is ineffective due to two main reasons. First, the complexity of the task overwhelms the LLM as it needs to determine which data to reference, decide whether to display multimodal data, and figure out where to place it. Additionally, this complexity results in low-quality multimodal answers. To address this, we propose a step-by-step strategy that first generates a pure text answer and then refines it with multimodal data.
\subsection{Text Answer Generation}
\label{sec:text_answer_generation}
Our text answer generation follows the RAG~\cite{DBLP:conf/nips/LewisPPPKGKLYR020} style method.
Given user queries, we fine-tune a pre-trained text embedding model~\cite{reimers-2019-sentence-bert} on an annotated dataset, and apply this model to embed text snippets which are stored in a vector index.
%The fine-tuned embedding model is then applied to the text snippets range from 11 to 1,500 tokens extracted from a collection of web page documents to create a search index as FAISS embeddings~\cite{johnson2019billion}.
The index is used to retrieve relevant text document snippets $S$ based on the user query $q$, for which the top $5$ text document snippets are selected.
The LLM is then prompted with the user queries and the retrieved text snippets to generate answer $A_{t}$, the text prompt can be found in Appendix~\ref{appendix:text_answer_prompt}.
\subsection{Source-Based Multimodal Retrieval}
\label{sec:multimodal_retrieval}
The source-based multimodal retrieval component comprises two steps: source attribution and section-level multimodal data retrieval.
\paragraph{Source Attribution.} In the source attribution step, the text answer snippet $a_{[i,j]} \in A_{t}$ is matched to the corresponding text document snippet $s_{i} \in S$. Specifically, the text answer $A_{t}$ is divided into sentences, and each sentence is compared with text document snippets $S$ using cosine similarity between their embeddings. For each text answer snippet $a_{[i, j]}$, the text document snippet $s_{i}$ with the highest cosine similarity score is identified as the source of $a_{[i, j]}$. We also discard snippets with scores below a predefined threshold.
This step is crucial because the identified text answer snippet $a_{[i, j]}$ serves as the basis for retrieving relevant multimodal data, which will be integrated around the text answer in the final multimodal answer.
\paragraph{Section-Level Multimodal Data Retrieval.}
In the section-level multimodal data retrieval step, we first identify multimodal data located in the same section as the retrieved text document snippet $s_{i}$, \textit{i.e.}, the multimodal data is located at the same hierarchy level in the original document $D_{S}$ as the text document snippet $s_{i}$. This step aims to reduce the search space and ensure the precision of the retrieval results.
Next, we apply the same fine-tuned embedding model used in text document retrieval for multimodal retrieval. To better represent the multimodal data, we include the context surrounding the multimodal data and additional text data as the input for the embedding model. Specifically, for images, we include image captions generated by an LLM; for tables the table content, and for videos the LLM generated video transcript summaries.
The final multimodal responses are selected based on the cosine similarity between the text document snippet $s_{i}$ and the multimodal data embeddings. Only the top multimodal data is selected to ensure relevance. It is worth noting that the same multimodal data could be selected for multiple text answer snippets. To avoid including the same multimodal data in the final answer multiple times, we keep only the one with the highest similarity score.
\subsection{Multimodal Answer Refinement}
\label{sec:multimodal_answer_refinement}
Once the relevant multimodal responses are retrieved, the LLM is prompted to generate the final answer.
This process involves providing the LLM with the user question, initial text answer, and initial multimodal response along with their context information.
For each multimodal response, placeholders are added to the initial text answer at the corresponding source locations. Each placeholder includes the multimodal data and its context information, ensuring that the LLM accurately integrates relevant content without generating irrelevant details. Additionally, a few example responses are provided to guide the model in producing higher quality and more consistent outputs.
The final prompt is formulated by concatenating the user question, modified initial text answer, multimodal data, their context information, and example responses. The prompt example can be found in Appendix~\ref{appendix:multimodal_prompt}.
\section{User Interface}
We implemented the MuRAR framework and integrated it into a prototype version AI Assistant in Adobe Experience Platform\footnote{https://experienceleague.adobe.com/en/docs/experience-platform/ai-assistant/home}.
As shown in Figure~\ref{fig:UI}, when a user queries AI assistant, for instance, asking ``What is a good tutorial for creating a schema?'', the process begins with the user's query (A). The MuRAR first generates a text answer based on the text snippet retrieval (B), and recognizes text answer snippets by source attribution (C).
Next, MuRAR retrieves the most relevant multimodal data (D1 and D2). In this case, a screenshot of the Schemas Workspace and a video tutorial detailing the process of schema creation are retrieved. Finally, MuRAR integrates the relevant multimodal data into the text answer through multimodal answer refinement (E). This multimodal presentation helps users understand complex instructions more effectively.
\section{Data Collection}
\label{sec:dataset}
We collected two datasets: a multimodal dataset as the source data for implementing the MuRAR framework within the AI assistant application, and a multimodal question-answering dataset for human evaluation. %\footnote{The datasets will be released upon the completion of the Adobe legal process.}
\subsection{Multimodal Dataset}
The multimodal dataset was curated from 2,173 documentation pages from Adobe Experience League~\footnote{\href{https://experienceleague.adobe.com/}{https://experienceleague.adobe.com/}}.
The dataset, as shown in Table~\ref{tab:dataset_statistics}, encompasses four primary modalities: text, images, tables, and videos, along with their associated metadata such as contextual text and URLs. The textual data includes both pure text and tabular content, while the visual data consists of images and videos. See the design details in ~\ref{appendix:scraper} for more information on data collection.
The text content from the documentation pages was initially extracted and tokenized using the GPT-2 tokenizer, segmenting the document into smaller snippets ranging from 11 to 1,500 tokens, resulting in 18,071 text snippets.
For image data, we extracted image URLs along with the surrounding textual context, defined as the text from the start of the section to the image (pre-image context) and from the image to the end of the section (post-image context). Additionally, GPT-4 was used to generate captions for the images, enhancing the data for multimodal retrieval, which resulted in 6,320 image data pieces.
The tables were extracted in JSON format along with their contextual text, similar to the image data. This included the text before and after the table, leading to a total collection of 2,646 table data pieces. For video content, we extracted video URLs, contextual text, and transcripts. When transcripts were not available, we downloaded the videos and used Whisper~\cite{DBLP:conf/icml/RadfordKXBMS23} to generate transcripts from the audio. GPT-4 was then used to summarize these transcripts, enhancing the data for multimodal retrieval. This process produced 253 videos. For all data types, we also gathered the titles of the pages and the headings of the sections where the data was located, providing a comprehensive context for each multimodal data piece.
\begin{table}[t]
\centering
\small
\begin{tabular}{l l r}
\toprule
\textbf{Modality} & \textbf{Metric} & \textbf{Value} \\ \midrule
\multirow{2}{*}{\textbf{Text}}
& Count & 18,071 \\ \cline{2-3}
& \rule{0pt}{2.5ex} Avg content tokens & 192 \\ \midrule
\multirow{3}{*}{\textbf{Image}}
& Count & 6,320 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg context tokens & 238 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg caption tokens & 94 \\ \midrule
\multirow{3}{*}{\textbf{Video}}
& Count & 253 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg context tokens & 91 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg summary tokens & 33 \\ \midrule
\multirow{3}{*}{\textbf{Table}}
& Count & 2,644 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg context tokens & 160 \\ \cline{2-3}
& \rule{0pt}{2.5ex}Avg table tokens & 223 \\ \bottomrule
\end{tabular}
\caption{Multimodal dataset statistics.}
\label{tab:dataset_statistics}
\end{table}
\subsection{Multimodal Question Answering Dataset}
In constructing the multimodal question and answer dataset, we collected 764 questions and applied MuRAR to generate answers. Among these, 306 questions had multimodal answers. To evaluate the quality of these answers, we randomly sampled 150 of the 306 questions for human evaluation.
To evaluate the impact of different backbone LLMs on the performance of multimodal answers, we conducted a comparative study using GPT-3.5 and GPT-4. For each model, we generated 150 text and multimodal answers. This analysis allowed us to assess how the choice of LLM influences the effectiveness and coherence of multimodal responses.
\section{Human Evaluation}
\label{sec:evaluation}
To assess the effectiveness of our multi-modal question answering system, we conducted a comprehensive human evaluation study with two phases: (1) single-model evaluation and (2) pairwise comparison. The single-model psychometric evaluation was designed to measure three key metrics: usefulness, readability, and relevance of the multi-modal outputs. As a pairwise comparison, we included a preference-based ranking to determine the overall user preference between text-only and multi-modal responses. Notice that we do not assess the quality of the text content itself, but rather the core contribution brought by attributing multi-modal outputs to the retrieved answer.
\subsection{Study Setup}
We collected a dataset of 300 question-answer pairs, equally split between outputs from GPT-3.5 and GPT-4 models. For each pair, we generated both text-only and multi-modal responses. Eight expert annotators, all with advanced degrees in computer science and experience in NLP, were recruited to evaluate these outputs.
\subsection{Evaluation Schema}
The annotators were asked to rate each multi-modal output on a 5-point Likert scale (1 being lowest, 5 being highest) for the following metrics, which were adapted from \cite{pradeep2024convkgyarnspinningconfigurablescalable}:
\vspace{-2mm}
\begin{itemize}\setlength{\itemsep}{0pt}
\item \textbf{Usefulness:} This metric measures the extent to which multi-modal elements contribute to the user's comprehension of the text content. High scores indicate that the multi-modal output provides valuable additional information, clarifies complex concepts, or illustrates key points in ways that significantly aid understanding.
\item \textbf{Readability:} This assesses how well the multi-modal elements are integrated with the text, considering factors such as placement, size, and formatting. High scores indicate seamless integration that enhances the overall reading experience.
\item \textbf{Relevance:} This measures how closely the multi-modal elements relate to the content of the text. High scores indicate that the multi-modal output directly supports or illustrates the textual content.
\end{itemize}
\begin{table}[ht]
\centering
\small
\begin{tabular}{l c c c}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{}} & \multicolumn{3}{c}{\textbf{Model}} \\ \cmidrule{2-4}
\textbf{Metric} & \textbf{GPT-3.5} & \textbf{GPT-4} & \textbf{Both GPTs} \\ \midrule
Usefulness & 3.34 & 3.60 & 3.47 \\ \midrule
Readability & 3.49 & 3.76 & 3.63 \\ \midrule
Relevance & 3.66 & 3.90 & 3.78 \\ \midrule
Preference Rate & 0.82 & 0.90 & 0.86 \\ \bottomrule
\end{tabular}
\caption{Evaluation results for multimodal answers generated by GPT-3.5 and GPT-4.}
\vspace{-5mm}
\label{tab:evaluation_result}
\end{table}
\begin{figure*}[htbp]
\centering
\includegraphics[width=1\textwidth]{Figures/human_evaluation_bar_chart.png}
\caption{Distribution of score counts for evaluation metrics across GPT-3.5, GPT-4, and Both-GPTs.}
\vspace{-2mm}
\label{fig:evaluation_distribution}
\end{figure*}
After rating the multi-modal outputs, annotators were asked to indicate their overall preference between the text-only version, the multi-modal version, or if they found them equally effective (``Same'').
\subsection{Results}
\paragraph{Psychometric Evaluation Results.}
The psychometric evaluation focused on three key aspects: usefulness, readability, and relevance. As shown in Table~\ref{tab:evaluation_result}, when examine the results generated by both GPT-3.5 and GPT-4, the usefulness metric achieved an average score of 3.47, while readability and relevance scored 3.63 and 3.78, respectively. These scores, all above the midpoint of the scale as reflected in Figure \ref{fig:evaluation_distribution}, suggest that our approach performs well in producing useful, readable, and relevant output. Qualitative feedback from annotators further supports this conclusion, indicating that the multi-modal generation provides informative additions to the text, is understandable through its placement, and remains relevant to the associated content. In addition, the average preference rating of 0.86 demonstrates a strong overall preference for our method compared to the text-only alternative. When comparing GPT-3.5 with GPT-4, we found that using GPT-4 as the backbone LLM increased all the metrics, showcasing its superior performance in generating high-quality, multi-modal content within our framework.
\paragraph{Inter-Annotator Agreement.}
%\todo[color=green]{Yunyao: I would suggest that we simplify the writing and only share Cohen's kappa, as the main focus of work is not the user study. }
%To assess annotation reliability, we calculated two inter-annotator agreement (IAA) measures (Table \ref{tab:iaa_table}). Krippendorff's alpha was 0.4179 overall, indicating moderate agreement across all annotators. However, Cohen's kappa between the top two annotators was 0.71, suggesting substantial agreement when excluding outlier data points.
%To investigate this discrepancy, we conducted further analyses. The standard deviation ranged from 0.8519 to 0.9496, indicating a relatively tight distribution of scores (Table \ref{tab:standard_deviation_metrics}). An annotator-specific analysis (Table \ref{tab:per_annotator}) revealed that Annotators 2 and 4, accounting for 26\% and 5.8\% of annotations respectively, had lower average scores compared to others. Annotator 2 averaged 3.0619 (SD = 0.5648), while Annotator 4 averaged 3.3836 (SD = 0.253).
%These findings suggest that the lower Krippendorff's alpha may be attributed to systematic differences in scoring patterns among a subset of annotators, rather than widespread disagreement. It is worth noting that while annotators were provided with in-depth instructions, they did not undergo formal training.
We calculated two inter-annotator agreement measures: Krippendorff's alpha (0.4179, moderate agreement) and Cohen's kappa between top annotators (0.71, substantial agreement). The lower Krippendorff's alphas can be explained through the annotator-specific analysis (Table~\ref{tab:iaa_table}), which revealed lower average scores for Annotators 2 (3.0619, SD = 0.5648, 26\% of annotations) and 4 (3.3836, SD = 0.253, 5.8\% of annotations) compared to others.
% In conclusion, despite some variability in IAA results, the high Cohen's kappa for the top annotators, combined with strong psychometric evaluation scores and preference ratings, supports the overall reliability and effectiveness of our approach. The multi-modal generation approach demonstrates clear benefits over text-only alternatives, providing valuable enhancements to textual content.
\subsection{Findings}
We analyzed the errors and mistakes made by the MuRAR framework during our human evaluation. We identified some issues in the multimodal retrieval component. Although precision is ensured thanks to source attribution, it may cause low recall, \textit{i.e.}, relevant multimodal data to the text answer snippet may not be in the same section as the corresponding text document snippet on a web page. Additionally, during the multimodal answer refinement component, readability can be affected. For instance, multimodal data may contain duplicated information already explained in plain text. This repetition can negatively impact the readability and clarity of the multimodal answer, making it less effective for users.
\section{Conclusion}
We introduced MuRAR, a framework that enhances text-based responses with images, tables, and videos. Human evaluations showed that MuRAR's multimodal answers are more useful, readable, and relevant than text-only responses. The system uses text answer generation, source-based multimodal retrieval, and answer refinement to create coherent multimodal answers. Identified areas for improvement include enhancing recall of relevant data and reducing redundancy. Future work will refine these processes and expand the system's capabilities for more complex queries.
\section{Appendix}
\label{sec:appendix}
\begin{table*}[ht]
\small
\centering
\begin{tabular}{lcccccc}
\toprule
\textbf{Annotator} & \textbf{Answers} & \textbf{Usefulness} & \textbf{Readability} & \textbf{Relevance} & \textbf{Preference (Multi-Modal / Text Only / Same)} \\
\midrule
No.1 & 294 & 3.6463 & 3.8367 & 4.0170 & 207 (70.41\%) / 41 (13.95\%) / 46 (15.65\%) \\
\midrule
No.2 & 237 & 2.8861 & 3.0295 & 3.2700 & 115 (48.52\%) / 81 (34.18\%) / 41 (17.30\%) \\
\midrule
No.3 & 259 & 3.7452 & 3.8340 & 3.8764 & 205 (79.15\%) / 28 (10.81\%) / 26 (10.04\%) \\
\midrule
No.4 & 53 & 3.0566 & 3.5094 & 3.5849 & 30 (56.60\%) / 16 (30.19\%) / 7 (13.21\%) \\
\midrule
No.5 & 9 & 4.1111 & 4.4444 & 4.4444 & 9 (100.0\%) / 0 (0.0\%) / 0 (0.0\%) \\
\midrule
No.6 & 22 & 4.0000 & 4.0909 & 4.1818 & 19 (86.36\%) / 2 (9.09\%) / 1 (4.55\%) \\
\midrule
No.7 & 24 & 4.2500 & 4.2083 & 4.6250 & 23 (95.83\%) / 0 (0.0\%) / 0 (0.0\%) \\
\midrule
No.8 & 2 & 4.0000 & 4.5000 & 4.5000 & 2 (100.0\%) / 0 (0.0\%) / 0 (0.0\%) \\
\bottomrule
\end{tabular}
\caption{Per-annotator average scores and preference.}
\vspace{-5mm}
\label{tab:per_annotator}
\end{table*}
\begin{table}[ht]
\centering
\small
\begin{tabular}{l c c c}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{}} & \multicolumn{3}{c}{\textbf{Model}} \\ \cmidrule{2-4}
\textbf{Metric} & \textbf{GPT-3.5} & \textbf{GPT-4} & \textbf{Both GPTs} \\ \midrule
\textbf{Usefulness} & 0.9741 & 0.9059 & 0.9496 \\ \midrule
\textbf{Readability} & 0.7686 & 0.7059 & 0.7500 \\ \midrule
\textbf{Relevance} & 0.8576 & 0.8301 & 0.8519 \\ \bottomrule
\end{tabular}
\caption{Standard deviation of evaluation metrics for multimodal answers generated by GPT-3.5 and GPT-4.}
\label{tab:standard_deviation_metrics}
\end{table}
\begin{table}[ht]
\small
\centering
\begin{tabular}{lccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{}} & \multicolumn{3}{c}{\textbf{Agreement Metric}} \\ \cmidrule{2-4}
\textbf{Metric\&Model} & \textbf{K-$\alpha$}\textsubscript{normal} & \textbf{K-$\alpha$}\textsubscript{combined} & \textbf{C-$\kappa$} \\
\midrule
Overall & 0.4179 & 0.3437 & 0.7100 \\
\midrule
Usefulness\textsubscript{GPT-3.5} & 0.4150 & 0.3468 & 0.6879 \\
Usefulness\textsubscript{GPT-4} & 0.5424 & 0.4993 & 0.7900 \\
Usefulness\textsubscript{all} & 0.4758 & 0.4164 & 0.7383 \\
\midrule
Readability\textsubscript{GPT-3.5} & 0.3418 & 0.2147 & 0.6852 \\
Readability\textsubscript{GPT-4} & 0.3187 & 0.2502 & 0.7048 \\
Readability\textsubscript{all} & 0.3424 & 0.2374 & 0.6968 \\
\midrule
Relevance\textsubscript{GPT-3.5} & 0.3369 & 0.2958 & 0.6459 \\
Relevance\textsubscript{GPT-4} & 0.4465 & 0.3664 & 0.7291 \\
Relevance\textsubscript{all} & 0.3925 & 0.3323 & 0.6872 \\
\bottomrule
\end{tabular}
\caption{Inter-annotator agreement.}
\label{tab:iaa_table}
\end{table}
\subsection{Multimodal Scraper Design}
\label{appendix:scraper}
Our multimodal scraper design collects various fields and metadata from Adobe Experience League for images, videos, and tables. For images, we gather the link to the image and the surrounding context, specifically the text between the previous and current image and between the current and next image. The metadata collected includes the title of the document, the header of each section containing the image, and the URL of the document. For videos, the fields collected include the URL of the video, the context text before the video, and the video transcript. The metadata gathered is similar, including the document title, section headers, and document URL. For tables, we collect the table content in the form of a JSON string, the context text before the table, and the document URL. Additional metadata includes the document title and the header of each section containing the table.
\subsection{Human Evaluation Metrics}
\label{appendix:evaluation_metrics}
\paragraph{Usefulness}
Usefulness measures how much the multi-modal elements contribute to the user's comprehension of the text content.
\begin{itemize}\setlength{\itemsep}{0pt}
\item 1 - Not at all useful: Multi-modal elements provide no additional understanding or actively confuse the user.
\item 2 - Slightly useful: Multi-modal elements offer minimal enhancement to understanding.
\item 3 - Moderately useful: Multi-modal elements provide some additional clarity or information.
\item 4 - Very useful: Multi-modal elements significantly enhance understanding of the text.
\item 5 - Extremely useful: Multi-modal elements are crucial for full comprehension of the text.
\end{itemize}
\paragraph{Readability}
Readability assesses how well the multi-modal elements are integrated with the text.
\begin{itemize}\setlength{\itemsep}{0pt}
\item 1 - Severely impairs readability: Multi-modal elements are poorly placed, causing significant disruption to reading flow.
\item 2 - Somewhat impairs readability: Multi-modal elements are not well-integrated, causing minor disruptions.
\item 3 - Neutral impact on readability: Multi-modal elements neither enhance nor impair the reading experience.
\item 4 - Enhances readability: Multi-modal elements are well-placed, supporting smooth reading flow.
\item 5 - Significantly enhances readability: Multi-modal elements are perfectly integrated, greatly improving the reading experience.
\end{itemize}
\paragraph{Relevance}
Relevance measures how closely the multi-modal elements relate to the text content.
\begin{itemize}\setlength{\itemsep}{0pt}
\item 1 - Completely irrelevant: Multi-modal elements have no apparent connection to the text.
\item 2 - Mostly irrelevant: Multi-modal elements have only a tenuous connection to the text.
\item 3 - Somewhat relevant: Multi-modal elements relate to the text but not be entirely on-point.
\item 4 - Highly relevant: Multi-modal elements clearly support and illustrate the text content.
\item 5 - Perfectly relevant: Multi-modal elements are essential to the text, providing crucial illustrations or data.
\end{itemize}
\paragraph{Preference Grading}
Annotators also indicate their overall preference between the text answer and the multimodal answer:
\begin{itemize}
\item \textbf{Text Only}: Choose this if you believe the text alone would be more effective without the multi-modal elements.
\item \textbf{Multi-Modal}: Select this if you think the combination of text and multi-modal elements provides the best experience.
\item \textbf{Same}: Choose this if you feel text-only and multi-modal versions are equally effective.
\end{itemize}
\subsection{Additional Human Evaluation Results}
\label{appendix:}
Due to space constraints, we include additional human evaluation results in the Appendix. The average score and preference per annotator are presented in Table~\ref{tab:per_annotator}. The standard deviation of evaluation metrics for multimodal answers generated by GPT-3.5 and GPT-4 is detailed in Table~\ref{tab:standard_deviation_metrics}. Overall inter-annotator agreement, along with specific agreements for different models and metrics, can be found in Table~\ref{tab:iaa_table}.
\subsection{Prompts}
Please note that the actual prompts used in the system development differ from the prompts shown below. These are simplified versions that capture the essence of the prompt design.
\paragraph{Prompt for Text Answer Generation}
\label{appendix:text_answer_prompt}
The prompt for text answer generation can be found in Figure~\ref{fig:text_answer_prompt}.
\begin{figure}[ht]
\centering
\includegraphics[width=0.48\textwidth]{Figures/text_answer_prompt.png}
\caption{Prompt for text answer generation.}
\vspace{-6mm}
\label{fig:text_answer_prompt}
\end{figure}
\paragraph{Prompt for Multimodal Answer Refinement}
\label{appendix:multimodal_prompt}
The prompt for multimodal answer refinement can be found in Figure~\ref{fig:multimodal_answer_prompt}.
\begin{figure}[ht]
\centering
\includegraphics[width=0.48\textwidth]{Figures/multimodal_prompt.png}
\caption{Prompt for multimodal answer refinement.}
\label{fig:multimodal_answer_prompt}
\end{figure}
\begin{thebibliography}{15}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Anil et~al.(2023)Anil, Borgeaud, Wu, Alayrac, Yu, Soricut, Schalkwyk, Dai, Hauth, Millican, Silver, Petrov, Johnson, Antonoglou, Schrittwieser, Glaese, Chen, Pitler, Lillicrap, Lazaridou, Firat, Molloy, Isard, Barham, Hennigan, Lee, Viola, Reynolds, Xu, Doherty, Collins, Meyer, Rutherford, Moreira, Ayoub, Goel, Tucker, Piqueras, Krikun, Barr, Savinov, Danihelka, Roelofs, White, Andreassen, von Glehn, Yagati, Kazemi, Gonzalez, Khalman, Sygnowski, and et~al.}]{DBLP:journals/corr/abs-2312-11805}
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean{-}Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew~M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy~P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul~Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Ana{\"{\i}}s White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et~al. 2023.
\newblock \href {https://doi.org/10.48550/ARXIV.2312.11805} {Gemini: {A} family of highly capable multimodal models}.
\newblock \emph{CoRR}, abs/2312.11805.
\bibitem[{Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert{-}Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei}]{DBLP:conf/nips/BrownMRSKDNSSAA20}
Tom~B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert{-}Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel~M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.
\newblock \href {https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html} {Language models are few-shot learners}.
\newblock In \emph{Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual}.
\bibitem[{Gao et~al.(2023)Gao, Xiong, Gao, Jia, Pan, Bi, Dai, Sun, Guo, Wang, and Wang}]{DBLP:journals/corr/abs-2312-10997}
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi~Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023.
\newblock \href {https://doi.org/10.48550/ARXIV.2312.10997} {Retrieval-augmented generation for large language models: {A} survey}.
\newblock \emph{CoRR}, abs/2312.10997.
\bibitem[{Joshi et~al.(2024)Joshi, Gupta, Kumar, and Sisodia}]{joshi2024robust}
Pankaj Joshi, Aditya Gupta, Pankaj Kumar, and Manas Sisodia. 2024.
\newblock Robust multi model rag pipeline for documents containing text, table \& images.
\newblock In \emph{2024 3rd International Conference on Applied Artificial Intelligence and Computing (ICAAIC)}, pages 993--999. IEEE.
\bibitem[{Kumar et~al.(2020)Kumar, Mittal, and Manocha}]{DBLP:journals/corr/abs-2004-12238}
Abhishek Kumar, Trisha Mittal, and Dinesh Manocha. 2020.
\newblock \href {http://arxiv.org/abs/2004.12238} {{MCQA:} multimodal co-attention based network for question answering}.
\newblock \emph{CoRR}, abs/2004.12238.
\bibitem[{Lewis et~al.(2020)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, K{\"{u}}ttler, Lewis, Yih, Rockt{\"{a}}schel, Riedel, and Kiela}]{DBLP:conf/nips/LewisPPPKGKLYR020}
Patrick S.~H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K{\"{u}}ttler, Mike Lewis, Wen{-}tau Yih, Tim Rockt{\"{a}}schel, Sebastian Riedel, and Douwe Kiela. 2020.
\newblock \href {https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html} {Retrieval-augmented generation for knowledge-intensive {NLP} tasks}.
\newblock In \emph{Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual}.
\bibitem[{OpenAI(2023)}]{DBLP:journals/corr/abs-2303-08774}
OpenAI. 2023.
\newblock \href {https://doi.org/10.48550/ARXIV.2303.08774} {{GPT-4} technical report}.
\newblock \emph{CoRR}, abs/2303.08774.
\bibitem[{Pradeep et~al.(2024)Pradeep, Lee, Mousavi, Pound, Sang, Lin, Ilyas, Potdar, Arefiyan, and Li}]{pradeep2024convkgyarnspinningconfigurablescalable}
Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeff Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, and Yunyao Li. 2024.
\newblock \href {http://arxiv.org/abs/2408.05948} {Convkgyarn: Spinning configurable and scalable conversational knowledge graph qa datasets with large language models}.
\bibitem[{Radford et~al.(2023)Radford, Kim, Xu, Brockman, McLeavey, and Sutskever}]{DBLP:conf/icml/RadfordKXBMS23}
Alec Radford, Jong~Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.
\newblock \href {https://proceedings.mlr.press/v202/radford23a.html} {Robust speech recognition via large-scale weak supervision}.
\newblock In \emph{International Conference on Machine Learning, {ICML} 2023, 23-29 July 2023, Honolulu, Hawaii, {USA}}, volume 202 of \emph{Proceedings of Machine Learning Research}, pages 28492--28518. {PMLR}.
\bibitem[{Reimers and Gurevych(2019)}]{reimers-2019-sentence-bert}
Nils Reimers and Iryna Gurevych. 2019.
\newblock \href {https://arxiv.org/abs/1908.10084} {Sentence-bert: Sentence embeddings using siamese bert-networks}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing}. Association for Computational Linguistics.
\bibitem[{Singh et~al.(2021)Singh, Nasery, Mehta, Agarwal, Lamba, and Srinivasan}]{DBLP:conf/naacl/SinghNMALS21}
Hrituraj Singh, Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba, and Balaji~Vasan Srinivasan. 2021.
\newblock \href {https://doi.org/10.18653/V1/2021.NAACL-MAIN.418} {{MIMOQA:} multimodal input multimodal output question answering}.
\newblock In \emph{Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, {NAACL-HLT} 2021, Online, June 6-11, 2021}, pages 5317--5332. Association for Computational Linguistics.
\bibitem[{Talmor et~al.(2021)Talmor, Yoran, Catav, Lahav, Wang, Asai, Ilharco, Hajishirzi, and Berant}]{DBLP:conf/iclr/TalmorYCLWAIHB21}
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021.
\newblock \href {https://openreview.net/forum?id=ee6W5UgQLa} {Multimodalqa: complex question answering over text, tables and images}.
\newblock In \emph{9th International Conference on Learning Representations, {ICLR} 2021, Virtual Event, Austria, May 3-7, 2021}. OpenReview.net.
\bibitem[{Touvron et~al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozi{\`{e}}re, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave, and Lample}]{DBLP:journals/corr/abs-2302-13971}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie{-}Anne Lachaux, Timoth{\'{e}}e Lacroix, Baptiste Rozi{\`{e}}re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur{\'{e}}lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.
\newblock \href {https://doi.org/10.48550/ARXIV.2302.13971} {Llama: Open and efficient foundation language models}.
\newblock \emph{CoRR}, abs/2302.13971.
\bibitem[{Wei et~al.(2022)Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, and Zhou}]{DBLP:conf/nips/Wei0SBIXCLZ22}
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed~H. Chi, Quoc~V. Le, and Denny Zhou. 2022.
\newblock \href {http://papers.nips.cc/paper\_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html} {Chain-of-thought prompting elicits reasoning in large language models}.
\newblock In \emph{Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022}.
\bibitem[{Wu et~al.(2023)Wu, Fei, Qu, Ji, and Chua}]{DBLP:journals/corr/abs-2309-05519}
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat{-}Seng Chua. 2023.
\newblock \href {https://doi.org/10.48550/ARXIV.2309.05519} {Next-gpt: Any-to-any multimodal {LLM}}.
\newblock \emph{CoRR}, abs/2309.05519.
\end{thebibliography}