Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence
Extracted source text · arXiv 2502.09927
% \documentclass[12pt]{article}
\documentclass{article}
\usepackage{iclr2025_conference,times}
\usepackage{tabularx}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx} % Required for inserting images
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{rotating}
\usepackage{colortbl}
\usepackage{xcolor}
%\usepackage{csvsimple}
\usepackage{amsmath}
\usepackage{wrapfig}
\usepackage{multicol}
\usepackage{amssymb}
\usepackage{tcolorbox}
\usepackage{float}
\usepackage{tablefootnote}
\usepackage{longtable}
\usepackage{caption}
\usepackage{url}
\usepackage{geometry}
\geometry{margin=1in}
\title{Granite Vision: a lightweight, open-source \\ multimodal model for enterprise Intelligence }
% \author{}
\author{Granite Vision Team, IBM Research \\
See Contributions and Acknowledgments section for full authors list \\
\texttt{granite-inquiries@ibm.com}
}
% \And
% Ji Q. Ren \& Yevgeny LeNet \\
% Department of Computational Neuroscience \\
% University of the Witwatersrand \\
% Joburg, South Africa \\
% \texttt{\{robot,net\}@wits.ac.za} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\date{September 2024}
\newcommand{\squarebullet}[1]{\textcolor{#1}{\textbf{$\blacksquare$}}} % Colored square bullet
\definecolor{SkyBlue}{RGB}{173, 216, 230}
\definecolor{LightGreen}{RGB}{144, 238, 144}
\definecolor{Peach}{RGB}{255, 178, 127}
\definecolor{Cyan40}{RGB}{15,98,254}
\definecolor{Cyan40}{RGB}{51,177,255}
\definecolor{Purple50}{RGB}{165,110,255}
\definecolor{Teal20}{RGB}{8,189,186}
\definecolor{Green40}{RGB}{56,183,98}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\definecolor{green}{RGB}{0. 205, 0}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
We introduce Granite Vision, a lightweight large language model with vision capabilities, specifically designed to excel in enterprise use cases, particularly in visual document understanding. Our model is trained on a comprehensive instruction-following dataset, including document-related tasks, such as content extraction from tables, charts, diagrams, sketches, and infographics, as well as general image tasks. The architecture of Granite Vision is centered around visual modality alignment with a decoder-only, 2 billion-parameter Granite large language model. Additionally, we introduce a dedicated safety classification approach in test-time that leverages a sparse set of attention vectors to identify potential harmful inputs. Despite its lightweight architecture, Granite Vision achieves strong results in standard benchmarks related to visual document understanding, as well as on the LiveXiv benchmark, which is designed to avoid test set contamination by using a constantly updated corpus of recently published Arxiv papers. We are releasing the model under the Apache-2 license, allowing for both research and commercial use, while offering complete visibility into the training data and other relevant details. See \url{https://huggingface.co/ibm-granite/} for model weights.
\end{abstract}
\section{Introduction}
The confluence of computer vision and natural language processing has led to significant advances in multimodal learning, enabling large language models to effectively integrate and reason about visual content and linguistic data. Both proprietary and open-source multimodal models \citep{grattafiori2024llama3,geminiteam2024gemini15,OpenAI2023GPT4TR,abdin2024phi3technicalreporthighly,dai2024nvlm,zhang2024mm15,wang2024qwen2,agrawal2024pixtral,Li2024LLaVAOneVisionEV,chen2024internvl,molmo} have achieved remarkable success on a range of standard benchmarks, including visual question-answering, as well as more complex tasks such as multi-image reasoning and video understanding.
While this progress is exciting, a notable disparity remains: smaller models, typically with 1-4 billion parameters, have consistently fallen short of achieving performance comparable to their larger counterparts containing tens of billions of parameters. In addition, existing models are predominantly trained on natural images, which can limit their performance in other domains, such as visual document understanding, where the unique visual characteristics, such as layouts, fonts, and graphics, significantly differ from natural images and require a more fine-grained comprehension of the visual content.
\begin{figure}[htpb]
\centering
\includegraphics[width=0.92\textwidth]{Figures/qualitiative_examples.pdf}
\caption{Qualitative examples generated by Granite Vision, showcasing its diverse capabilities including (a) document understanding such as receipt calculation, (b) form understanding with human handwritten text, (c) knowledge-grounded image description, (d) table understanding, etc. }
\label{fig:qualitative}
\end{figure}
In this work, we introduce Granite Vision, a compact vision-language model with approximately 3 billion parameters\footnote{We denote our model as Granite-Vision-3.1-2B, where the version (3.1) and size (2B) of the base large language model are explicitly indicated. However, when considering the integrated vision encoder and projector, the total parameter count of our model increases to 3 billion parameters.}, tailored to excel in enterprise use cases. Although our model can process general images, the first version of Granite Vision is particularly focused on visual document understanding, enabling automated content extraction from tables, charts, infographics, plots, diagrams, sketches, and more. Figure \ref{fig:qualitative} shows qualitative examples of our model’s output for visual document understanding tasks and general image description. Granite Vision extends the Granite family of large language models (~\cite{granite2024granite}), which have been trained on more than 12 trillion tokens, achieving state-of-the-art performance for their size, while being designed for enterprise usage, with full visibility into the training data.
As a key contribution of our work, we meticulously curate a comprehensive instruction-following dataset for visual document understanding, comprising around 13 million images and 80 million instructions, which span a diverse set of of tasks, including document question-answering , scene text understanding, key-value extraction, text grounding, layout parsing, captioning, UI understanding, and code (see Figure \ref{fig:doc_data}). We constructed this dataset by (i) mining documents from the web and synthesizing instructions as described in Section \ref{sec:docfm}, and (ii) unifying multiple public datasets into a single format. In addition to documents, our training data is further enriched by the inclusion of instruction-following data for general images from public datasets \citep{tong2024cambrian1,Li2024LLaVAOneVisionEV,laurencon2024matters,liu2023improvedllava}.
Similar to LLaVA \citep{liu2023llava}, our approach uses a projector to establish a connection between the visual encoder and a Granite large language model. Our training protocol consists of multiple stages, progressing from training the projector in isolation to jointly fine-tuning both the projector and the large language model, using denser image resolution grids at the latest stages. We also extract multi-layer features from the encoder, allowing us to better capture fine-grained details that are important for visual document understanding. Finally, we propose a novel method for safety classification as a separate module to identify harmful inputs based on a sparse set of attention vectors, which are selectively chosen to maximize safety classification performance.
Compared to other models of similar parameter size, Granite Vision achieves state-of-the-art results on established benchmarks for visual document understanding, conversion of tables and charts to structured formats, and the LiveXiv benchmark \citep{shabtay2024livexiv}, which evaluates the model on recently published Arxiv papers, helping to mitigate the risk of test set contamination that can arise when models are trained on web-scraped data. In addition to its strong performance in enterprise settings, our model demonstrates competitive results on standard vision-language benchmarks. To promote openness and collaboration, we make the model publicly available under the Apache-2 license, allowing visibility into the training data and procedures.
\section{Related Work}
\subsection{Multimodal Large Language Models}
Recent advances in multimodal large language models (MLLMs) have demonstrated significant progress in understanding and generating content across different data modalities. Comprehensive surveys by \cite{Yin_2024_survey} and \cite{wadekar2024evolutionmultimodalmodelarchitectures} closely outline the various approaches to architecture design, training strategies, and evaluation methodologies in the field.
The introduction of Flamingo \citep{alayrac2022flamingo} highlighted a remarkable performance of the transformer cross-attention architecture in vision-language tasks, serving as a catalyst for further advancements in the multimodal domain. The launch of GPT-4V \citep{openai_gpt4v} has sparked a competitive race in the development of powerful commercial-use MLLMs, leading to multimodal models like GPT-4o \citep{openai_gpt4o}, Claude 3.5 Sonnet \citep{anthropic_claude3_5}, and Gemini Pro 1.5 \citep{geminiteam2024gemini15}. In parallel, the open-source community has made significant contributions, with models like BLIP2 \citep{li2023blip2}, Phi-3.5-vision \citep{abdin2024phi}, LLaVA-OneVision \citep{li2024llavaonevision}, and Llama 3.2-vision \citep{grattafiori2024llama3} showing competitive performance on benchmarks while maintaining transparency and accessibility.
To mitigate the high costs associated with training large models in an end-to-end manner, a common approach involves employing modular architectures. These architectures typically combine a pre-trained modality encoder and a pre-trained large language model (LLM) with a learnable modality connector, with the latter comprising only a small portion of the total parameters \citep{bai2023qwenvl, tong2024cambrian1, chen2024internvl}.
More recently, \cite{zhang2024mm15} demonstrated that even relatively small models can achieve strong performance with careful data curation and optimized training strategies.
Building on these insights, our model uniquely achieves state-of-the-art results on standard document and other benchmarks, all while operating at a significantly reduced scale (around 3 billion parameters).
\subsection{Visual Document Understanding}
Visual document understanding, particularly the ability to comprehend charts, diagrams, tables, and document images, represents a crucial application area for MLLMs.
Two primary technical challenges emerge in enabling MLLMs to process documents and associated images effectively: adequately encoding high-resolution images, and accurately interpreting visually-situated text within the documents. Recent approaches to addressing these challenges are often broadly categorized into two groups based on their text recognition methodology.
The first category, including models like DocFormer \citep{appalaraju2021docformer}, LayoutLMv3 \citep{huang2022layoutlmv3}, and, more recently, DocLLM \citep{wang2023docllm}, relies on external optical character recognition (OCR) systems to process text within images. The second category consists of ``OCR-free" models, e.g. Donut \citep{kim2022ocrfreedocumentunderstandingtransformer}, UReader \citep{ye2023ureader}, mPLUG-DocOwl 1.5 \citep{hu2024mplugdocowl15}, and DocPedia \citep{feng2024docpedia}.
Besides the specialized multimodal document understanding models \citep{liao2024doclayllm, liu2024hrvda}, strong performance is also achieved by general MLLMs instruction-tuned on document datasets \citep{li2024llavaonevision, dai2024nvlm}.
While recent efforts \citep{mathew2021docvqa, huggingface_docmatix, rodriguez2024bigdocs} have produced several open-source document understanding datasets to advance model performance in this domain, large comprehensive datasets without restrictive licensing remain relatively limited.
Our approach builds upon this foundation by leveraging a comprehensive instruction-following dataset for visual document understanding, incorporating both synthetic data and public datasets.
%\section{Preliminaries: Granite Large Language Model}
% I added a few parts of this section to the model architecture section
%Granite 3.1 represents IBM’s third generation of large language models~\citep{granite31github}, released in the open under an Apache 2.0 license and matching top performance on general, enterprise, and safety benchmarks. Granite 3.1 language models are lightweight, state-of-the-art, open foundation models that natively support multilinguality, coding, reasoning, and tool usage, including the potential to be run on constrained compute resources. The models' data curation and training procedure were designed for enterprise usage and customization, with a process that evaluates datasets for governance, risk and compliance (GRC) criteria, in addition to IBM's standard data clearance process and document quality checks.
%Granite 3.1 language models extend the context length of Granite 3.0 language models~\citep{granite2024granite} from 4K to 128K using a progressive training strategy by increasing the supported context length in increments while adjusting RoPE theta until the models successfully adapt to the desired length of 128K. This long-context pre-training stage was performed using approximately 500B tokens. Moreover, Granite 3.1 instruction models provide an improved developer experience for function-calling and RAG generation tasks. These instruction models are finetuned from Granite-3.1 base models using a combination of open source instruction datasets with permissive license and internally collected synthetic datasets tailored for solving long context problems.
%The Granite 3.1 language model family includes: (1) Dense Models - 2B and 8B parameter models, trained on 12 trillion tokens in total; and, (2) Mixture-of-Expert (MoE) Models - Sparse 1B and 3B MoE models, with 400M and 800M activated parameters respectively, trained on 10 trillion tokens in total. These options provide a range of models with different compute requirements to choose from, with appropriate trade-offs with their performance on downstream tasks. The Granite Dense are powerful “workhorse” models designed for enterprise AI, whereas Granite MoE are Inference-efficient Mixture of Experts (MoE) models ideal for on device applications or CPU-based deployments. Evaluation results show that Granite-3.1-8B-Instruct outperforms models of similar parameter sizes in Hugging Face's OpenLLM Leaderboard.
%{\bf(Verify this para with Granite-3.1 report)} The Granite-Vision is built on top of a dense model, Granite-3.1-2B-Instruct, which share a similar architecture as popular language models like Llama to ensure compatibility with open-source inference and fine-tuning pipelines. This dense model leveraged Grouped Query Attention (GQA)~\citep{ainslie2023gqa} with 8 key-value heads to get a good balance between memory cost and model performance, and Rotary Position Embedding (RoPE)~\citep{su2024roformer} to model the relative position between tokens. The activation function for the MLP layers is SwiGLU. Before each MLP and attention layer, RMSNorm is utilized to normalize the layer’s input. To help with reducing the size of the model, parameters are shared between the input embedding and the output linear transform. This operation of sharing embeddings have zero, or sometimes positive, impact on model performance. In Section~\ref{sec:model:arch}, we discuss in details how the Granite-3.1-2B-Instruct architecture is adapted to design Granite-Vision.
\section{Data}\label{sec:data}
%In order to support a wide range of document and general image understanding tasks, we created a comprehensive instruction-following dataset, comprising several large and diverse public datasets as well as multiple synthetic datasets. A detailed view of the entirety of our data is shown in Figs 1 and 2. In the following sections, we will dive deep into the self-curated and pre-curated datasets that we use to train Granite-Vision.
%- pitch the sunburst chart (one for documents, another for general images), comprising both IBM sysnthetic data and public datasets, motivating the subsections below
%Our comprehensive instruction-following dataset comprises several large and diverse public datasets as well as multiple synthetic datasets.
Granite Vision has been trained on a comprehensive instruction-following dataset, which covers a wide variety of visual categories ranging from document images to general images. The training data consists of a combination of pre-curated public vision datasets, common-crawl PDFs, and data that is synthesized in-house.
%and is sourced from multiple public data resources as well as synthetic data generated in house.
A detailed view of our data is presented in Figure \ref{fig:doc_data}, which depicts document understanding datasets, and Figure \ref{fig:general_image_data}, which represents general image datasets.
Our document understanding data (Figure \ref{fig:doc_data}) covers a variety of document classes such as general document images, charts, flowcharts, diagrams and several more
%We curated
%created a collection of pre-curated high-quality, publicly available document-related datasets which
%Our document understanding datasets
encompassing a diverse set of visual Q\&A tasks.
%such as Document QA, Table QA, Chart QA, Diagram QA, OCR and scene-text related tasks, reasoning and grounding, UI screen/code and structure understanding.
See Table \ref{tab:doc_datasets} (in Appendix) for a comprehensive overview of the datasets. Such diverse data is chosen to ensure the model’s ability to generalize across a wide array of document-centric applications. %We provide the full list of the document datasets in Figure \ref{fig:doc_data}.
In addition to document datasets, we incorporated training data from multiple publicly available generic image datasets (Figure \ref{fig:general_image_data}). In the following sections, we dive deep into different data categories, data collection processes, synthetic data generation techniques, and data pre-processing pipelines. %performed. %before we train the Granite Vision model.
% 4.1 DocFM (~1.5 or 2 pages)
% - large-scale data collection/crawling
% - Synthetic data generation
% 4.2 Public Datasets (~1 page)
% - Clearance, data sources, ...
% - Instruction generation
% 4.3 Data preprocessing (~0.5 pages)
%\vspace{-15mm}
\begin{figure}[htpb]
\centering
% \hspace{-20pt}
% \begin{minipage}{0.85\textwidth} % Left side: Image
% \centering
% \includegraphics[width=1\textwidth]{document_data_sunburst_graph_v5}
% \end{minipage}%
% \begin{minipage}[c]{0.2\textwidth} % Left side: Image
% \centering
% Qualitative sample:
% \includegraphics[width=1\textwidth]{Figures/conv_fig1.png}
% \end{minipage}
\hspace{-20pt}
\begin{minipage}{0.65\textwidth} % Left side: Image
\centering
\includegraphics[width=\textwidth]{Figures/document_data_sunburst_graph_v10.pdf}
\end{minipage}
\vspace{6mm}
\begin{minipage}{0.99\textwidth} % Right side: Table
\centering
\scriptsize
\setlength{\columnsep}{0pt}
\begin{multicols}{3}
\begin{itemize}
\setlength{\itemsep}{0pt}
\item[] \textbf{\fbox{Document QA}}
\item[\squarebullet{Cyan40}] DocFM-VQA (2.4 M) % Please list associated jsons
\item[\squarebullet{Purple50}] Docmatix (565.0 K) %\citep{laurençon2024building}
\item[\squarebullet{Purple50}] synthdog-en (500.0 K) % \citep{kim2022ocr}
\item[\squarebullet{Purple50}] pixmo-docs (255.4 K) % \citep{deitke2024molmo}
\item[\squarebullet{Purple50}] HME100K (74.5 K) %\citep{yuan2022syntax}
\item[\squarebullet{Purple50}] Ureader QA (72.7 K) % \citep{ye2023ureader}
\item[\squarebullet{Purple50}] ArxivQA (54.4 K) %\citep{li2024multimodal}
\item[\squarebullet{Purple50}] Ureader KG (37.6 K) %\citep{ye2023ureader}
\item[\squarebullet{Cyan40}] Tech Book QA (10.4K)% rulebasedqa_redhatbooks/qas.json
\item[\squarebullet{Purple50}] DocVQA (10.2 K) %\citep{mathew2021docvqa}
\item[\squarebullet{Purple50}] IAM (5.7 K) % \citep{marti2002iam}
\item[\squarebullet{Cyan40}] Business Document QA (3.3K)% rulebasedqa_annual_reports/qas.json
%\item[\squarebullet{Purple50}] infographic VQA (2.12 K) %\citep{mathew2022infographicvqa}
\item[\squarebullet{Green40}] Visualmrc (3.0 K) %\citep{tanaka2021visualmrc} % Also in docstruct4m
% \item[\squarebullet{Cyan40}] llava-gen-qa (100k) % Need to formalize the dataset name
% \item[\squarebullet{Cyan40}] qa-contents-31k-225k-verified (31k) % Need to formalize the dataset name
% \item[\squarebullet{Cyan40}] qa-metadata-31k-184k-verified (31k) % Need to formalize the dataset name
% \item[\squarebullet{Cyan40}] DocLayNet-v2-kv-instruct-0.3.2 (10k) % Need to formalize the dataset name
% \item[\squarebullet{Cyan40}] rulebasedqa-doclaynetpt-qas% rulebasedqa_doclaynetpt/qas.json, qa-llm-doclaynetpt-visual-cues/qas.json
%\item[\squarebullet{Cyan40}] Scientific Document QA (TODO)% including qa-cvpr-rulebased-0.0.3 (15.5k), plos-14-11-2024 (11k), plos-21-11-2024 (18k), plos-llm-26-11-2024 (12k), qa-llm-aaai-visual-cues/qas.json, qa-llm-neurips-visual-cues/qas.json, qa-llm-opencvf-visual-cues/qas.json, qa-llm-wiki-visual-cues/qas.json, qa-llm-plos-visual-cues/qas.jsonulebasedqa_aaai-qas.json
\item[] \textbf{\fbox{Chart QA}}
\item[\squarebullet{Purple50}] Unichart (611.0 K) %\citep{masry2023unichart}
\item[\squarebullet{Purple50}] Tinychart (606.0 K) %\citep{zhang2024tinychart}
\item[\squarebullet{Cyan40}] DocFM-VQA-Charts (548.7 K) % Please list associated jsons
\item[\squarebullet{Purple50}] Dvqa (200.0 K) %\citep{kafle2018dvqa}
\item[\squarebullet{Green40}] PlotQA (157.0 K) %\citep{methani2020plotqa} % Also in docstruct4m
\item[\squarebullet{Green40}] FigureQA (100.0 K)
%\citep{kahou2017figureqa} % Also in docstruct4m
\item[\squarebullet{Cyan40}] DocFM-VQA-AugChartQA \hspace{-0.905mm}(35.1 K) % % Please list associated jsons
\item[\squarebullet{Purple50}] Chart2Text (27.0 K) %\citep{obeid2020chart}
\item[\squarebullet{Green40}] ChartQA (18.3 K) %\citep{masry2022chartqa} % Also in docstruct4m
\item[\squarebullet{Purple50}] Vistext (9.9 K) % \citep{tang2023vistext}
% \item[\squarebullet{Purple50}] LRV Chart \citep{liu2023aligning}
\vspace{1mm}
\item[]\textbf{\fbox{Infographics QA}}
\item[\squarebullet{Purple50}] InfographicVQA (8.5 K) %\citep{mathew2022infographicvqa}
\item[] \textbf{\fbox{Table QA}}
\item[\squarebullet{Purple50}] RoBUT-wikiSQL (75.0 K) %\citep{zhao2023robut}
\item[\squarebullet{Purple50}] RoBUT-WTQ (38.0 K) %\citep{zhao2023robut}
\item[\squarebullet{Purple50}] tabmwp (22.7 K) %\citep{lu2022dynamic}
\item[\squarebullet{Purple50}] RoBUT-SQA (8.5 K) %\citep{zhao2023robut}
\item[\squarebullet{Purple50}] MultiHiertt (7.6 K)%\citep{zhao2022multihiertt}
\item[\squarebullet{Purple50}] Finqa (5.3 K) %\citep{chen2021finqa}
\item[\squarebullet{Purple50}] Hitab (2.5 K) %\citep{cheng2022hitab}
%\item[\squarebullet{Purple50}] TAT-QA \citep{zhu2021tat}
\item[] \textbf{\fbox{Diagram QA}}
\item[\squarebullet{Purple50}] TQA (27.3 K) % \citep{kembhavi2017you}
\item[\squarebullet{Cyan40}] DocFM-VQA-Flowcharts (16.6 K) % synthetic_flowcharts
\item[\squarebullet{Purple50}] AI2D GPT4V (4.9 K)%\footnote{\url{https://huggingface.co/datasets/Kamizuru00/diagram_image_to_text}}
\item[\squarebullet{Purple50}] AI2D InternVL (3.2 K)
\item[\squarebullet{Purple50}] AI2D (2.4 K) %\citep{kembhavi2016diagram}
\item[\squarebullet{Purple50}] Diagram Image2Text (0.3 K)
\item[] \textbf{\fbox{Reasoning / Captioning}}
\item[\squarebullet{Cyan40}] DocFM Visual Cue \& Captioning \\(1.2M)
% qa-llm-aaai-visual-cues_qas, qa-llm-doclaynetpt-visual-cues_qas, qa-llm-neurips-visual-cues_qas, qa-llm-opencvf-visual-cues_qas, qa-llm-plos-visual-cues_qas, qa-llm-wiki-visual-cues_qas
\item[\squarebullet{Purple50}] Textcaps (22.0 K) %\citep{sidorov2020textcaps}
\item[\squarebullet{Purple50}] DocReason (8.7 K) %\footnote{\url{https://huggingface.co/datasets/mPLUG/DocReason25K}}
\item[] \textbf{\fbox{Grounding / Text Localization}}
\item[\squarebullet{Green40}] Multi-grained Text Localization \\(1.0 M)
\item[\squarebullet{Cyan40}] DocFM Rule Based Grounding \\(190 K) % Please list associated jsons
% rulebasedqa*, plos-*, qa-*, dln*, DocLayNet_v2*
\item[\squarebullet{Cyan40}] KVP10K (8.0 K) %\citep{naparstek2024kvp10k}
\vspace{1mm}
\item[] \textbf{\fbox{Scene Text}}
\item[\squarebullet{Purple50}] K12 Printing (256.6 K) %\citep{li2024llava}
\item[\squarebullet{Purple50}] OCR-VQA (165.7 K) %\citep{mishra2019ocr}
\item[\squarebullet{Purple50}] TextOCR (25.1 K) %\citep{singh2021textocr}
\item[\squarebullet{Purple50}] Ureader CAP (22.0 K) %\citep{ye2023ureader}
\item[\squarebullet{Purple50}] TextVQA (22.0 K) %\citep{singh2019towards}
\item[\squarebullet{Purple50}] Llavar (19.8 K) %\citep{zhang2023llavar}
\item[\squarebullet{Purple50}] ST-VQA (16.4 K) %\citep{biten2019scene}
\item[\squarebullet{Purple50}] RenderedText (10.0 K) %\footnote{\url{https://huggingface.co/datasets/wendlerc/RenderedText}}
\item[\squarebullet{Purple50}] IIIT5K (2.0 K) %\citep{mishra2012scene}
\item[] \textbf{\fbox{UI Screen / Code}}
% \item[\squarebullet{Purple50}] GUI2Summary \citep{cheng2024seeclick} % I dont remember we use this one
\item[\squarebullet{Purple50}] Datikz-v2 (94.4 K) %\footnote{\url{https://huggingface.co/datasets/nllg/datikz-v2}}
\item[\squarebullet{Purple50}] Datikz (47.9 K) %\citep{belouadi2023automatikz}
%\item[\squarebullet{Purple50}] GUI2Box
%\item[\squarebullet{Purple50}] Image2Flow % I dont remember we use this one
%\item[\squarebullet{Purple50}] Table2Latex % I dont remember we use this one
\item[\squarebullet{Purple50}] screenqa (33.2 K) %\citep{hsiao2022screenqa}
\item[\squarebullet{Purple50}] screen2words (15.7 K) % \citep{wang2021screen2words}
\item[\squarebullet{Purple50}] websight (10.0 K) %\citep{laurenccon2024unlocking}
%\item[\squarebullet{Purple50}] svg-stack-labeled\footnote{\url{https://huggingface.co/datasets/MrOvkill/svg-stack-labeled}} % I dont remember we use this one
% \item[\squarebullet{Cyan40}] websrc-train-data-BV (5k) % Need to formalize the dataset name
\item[\squarebullet{Purple50}] Omniact (4.9 K) %\citep{kapoor2024omniact} % Need to formalize the dataset name
% \item[\squarebullet{Cyan40}] llava-datikz-v2 % Need to formalize the dataset name
\item[] \textbf{\fbox{Structure Parsing}}
\item[\squarebullet{Green40}] OCR-CC (1.0 M) %\citep{yang2021tap}
\item[\squarebullet{Green40}] CCpdf (937.8 K)
%\citep{turski2023ccpdf}
\item[\squarebullet{Purple50}] PubTables (569.3 K) %\cite{smock2022pubtables}
\item[\squarebullet{Cyan40}] DocFM-ChartExtraction (550.4 K)
% Please list associated jsons
\item[\squarebullet{Green40}] TURL (200.0 K) %\citep{deng2022turl}
\item[\squarebullet{Green40}] PubTabNet (199.8 K) %\citep{zhong2020image} % This need to be reconsidered. We use pubtabnet from docstruct4m and the number is 200K
\item[\squarebullet{Green40}] RVL-CDIP (159.4 K) %\citep{harley2015evaluation}
\item[\squarebullet{Green40}] DUE (104.5 K) %\citep{borchmann2021due}
\item[\squarebullet{Cyan40}] FinTabNet (98.9 K) %\citep{zheng2020global}
\end{itemize}
\end{multicols}
\end{minipage}
\vspace{0.2in}
\caption{Overview of our comprehensive collection of document understanding datasets used for Granite Vision training.}
\label{fig:doc_data}
\end{figure}
\begin{figure}[htbp]
\centering
\begin{minipage}{0.39\textwidth} % Left side: Image
\centering
\includegraphics[width=\textwidth]{Figures/document_data_sunburst_graph_gendata_3.pdf}
\end{minipage}
\hfill
\begin{minipage}{0.60\textwidth} % Right side: Table
\centering
\scriptsize
\textbf{Image Count per Category Across Datasets}
\vspace{0.3cm}
\begin{tabular}{l|ccc}
\toprule
& Cauldron & LLaVa-OneVision & Cambrian-7m \\
\midrule
General & 276.5 K & 881.3 K & 1.8 M \\
Language/Captioning & 202.1 K & N/A & N/A\\
Math/Science/Reasoning & 178.4 K & 318.0 K & 354.5 K\\
Image Comparison & 188.9 K & N/A & N/A\\
\bottomrule
\end{tabular}
\textbf{QA Count per Category Across Datasets}
\vspace{0.3cm}
\begin{tabular}{l|ccc}
\toprule
& Cauldron & LLaVa-OneVision & Cambrian-7m \\
\midrule
General & 812.7 K & 2.0 M & 7.9 M \\
Language/Captioning & 203.3 K & 1.2 M & 1.8 M\\
Math/Science/Reasoning & 765.1 K & 464.8 K & 802.0 K\\
Image Comparison & 237.9 K & N/A & N/A\\
\bottomrule
\end{tabular}
\end{minipage}
\caption{Overview of general image datasets used for Granite Vision training.}
\label{fig:general_image_data}
\end{figure}
\subsection{IBM Curated Datasets} \label{sec:docfm}
In this section, we will describe several large synthetic document Q\&A datasets created at IBM and focused on a diverse set of document and Q\&A tasks. %These synthetic datasets enable a wide range of downstream applications.
In order to seed this effort, we leveraged DocFM, a large-scale comprehensive dataset effort at IBM consisting of 85 million document pages extracted from unique PDF documents sourced from Common Crawl, Wikipedia, and ESG (Environmental, Social, and Governance) reports. DocFM served as a foundation for generating our synthetic datasets, which capitalized on its enriched nature for granular filtering and balancing.
DocFM was created using Docling \citep{DoclingTechReport}, an open-source document conversion toolkit developed by IBM. Docling facilitated the extraction and enrichment of each page with a rich set of attributes, resulting in a highly detailed representation of document pages.
%TODO ahn might need to collapse below to save space.
Specifically, this enabled:
\begin{itemize}
\item \textbf{Text and Positional Metadata Extraction:} Parsing each PDF page to retrieve textual content along with its positional metadata.
\item \textbf{Layout Understanding:} Identification of structural components such as headers, footers, paragraphs, lists, captions, code, formulas, form elements, pictures, footnotes, and document indices.
\item \textbf{Reading Order Detection:} Establishing the reading sequence of extracted content based on layout and parsed output.
\item \textbf{Language Detection:} Identifying the primary language present on each page.
\item \textbf{Figure Classification:} Labeling visual elements, such as charts, images, and diagrams.
\end{itemize}
To ensure high-quality textual representation, we focused exclusively on PDFs with extractable content, ensuring reliable and efficient text processing while minimizing noise.
%Dhiraj - moving these sentences to the beginning of IBM datasets section
%This dataset served as a foundation for generating our synthetic datasets, which capitalized on its enriched nature for granular filtering and balancing. These synthetic datasets enable a wide range of downstream applications which will be detailed in the following sections enabling a wide range of downstream applications, which will be detailed in the following sections.
%TODO ahn: revisit above sentence after SDG section is finished.
\subsubsection{DocFM - Data Collection \& Filtering}
A large portion of the DocFM dataset was obtained from the Common Crawl corpus which is an open repository of web data.
%, %encompassing over 250 billion pages collected over 17 years.
It is freely accessible and distributed as monthly archive files in various formats (``segments"). Among these segments, the URL index contains a list of crawled URLs accompanied by metadata, such as the media type specified by the HTTP header and the media type detected by Common Crawl from the actual URL content. By adhering to the Robots Exclusion Protocol~\citep{koster2022rfc} (commonly known as ``robots.txt"), Common Crawl ensures basic operational guarantees.
% To process a subset of the Common Crawl dataset, the following steps were undertaken:
We obtained the Common Crawl dataset partition through the following process:
\begin{itemize}
\item A set of trusted web domains likely to contain relevant PDF files was manually curated. Additionally, websites of affiliations from public GitHub \footnote{https://github.com/} repositories, primarily universities and educational institutions, were identified. In total, we cataloged more than 45,000 domains to increase the relevance of the dataset and minimize potential issues related to web crawl data (e.g., toxic language, hate speech, abusive language, or profanity).
\item For each domain in the curated list, a search was conducted across the Common Crawl index segments of 13 archives (spanning May 2021 to January/February 2023) to identify URLs with the media type “application/pdf,” as either declared or detected.
\item For each discovered URL, we made an attempt to download the PDF file from the source. If successful, the file and its metadata were stored in a database.
\end{itemize}
Through this process, 4.96 million URLs were identified, resulting in 3.98 million PDF files after accounting for exact (binary) duplicates and invalid links. Additionally, we included 838,237 PDF documents retrieved from CCpdf~\citep{CCpdf}.
After deduplicating, 4.6 million unique PDF files (75.5 million pages) remained in the Common Crawl partition. Following the conversion process, we successfully converted 65 million pages.
In addition, a total of 7,696 ESG reports were all sourced directly from \texttt{Responsibility Reports}\footnote{www.responsibilityreports.com}. To complement this, an automated pipeline using a headless Chrome browser component was employed to convert 20M pages of Wikipedia articles into PDF format. The list of articles was derived from the \texttt{wikipedia} dataset~\citep{wikidump}.
%We have created DocFM as a very large dataset which we plan to use for a variety of downstream visual learning applications in the future. For the current release of Granite Vision, we focused on a subset of DocFM for creation of a family of synthetic visual QA datasets (DocFM-VQA, DocFM-VQA-charts, and DocFM-VQA-flowcharts) which we next describe.
DocFM is a very large dataset which we plan to use for a variety of downstream visual learning applications in the future. For the current release of Granite Vision, we focused on a subset of DocFM for creation of a family of synthetic visual QA datasets (LLM and rule-based document and chart VQA) which will be described next.
%In the following section we describe how synthetic visual QA data was generated using a subset of DocFM as seed and was eventually used for Granite Model training.
% A lot of unannotated data -> Need to generate instructions and QA -> automatic -> The QA/Instructions need to be related to visual compnents in the document.
%\paragraph{Synthetic Document VQA generation}
%%Pipeline:
%% 1. Document parsing (Deepspeed) + image %classification (chart/table/figure)
%% 2. Enrichment: Textual representation of all the %visual componenets, multiple models
%% 3. ICL for QA types (LAB??)
%% 4. Filtering (???)
%% 5. Numbers: 2.4 Mil
%There’s a lack of large-scale instruction-following datasets focused on visually-rich document understanding.
%The largely used dataset, DocVQA \citep{mathew2021docvqa}, contains only 10K images and 40K QA pairs.
%We used an LLM (Mixtral 8x22B) to synthetically generate a large-scale VQA dataset to train a model with strong performance on visually reach documents.
%As a context for the QA generation by the LLM, we used a ``verbalized'' version of the documents. It contains the text extracted from the programmatic PDFs and, to encourage questions referring to visual elements, we also verbalized charts, images, and tables.
%We converted charts to table format using DePlot \citep{liu-2022-deplot};
%We replaced images with their captions generated by BLIP2 \citep{li2023blip2};
%We converted tables to a markdown format using TableFormer \citep{yang-etal-2022-tableformer}.
%A total of 26M QA pairs were generated using this pipeline on 2.4M random pages from the DocFM dataset.
%We experimented with using frontier VLMs to filter the generated QAs to remove hallucinations. We found the filtering somewhat helps with the model performance but considering the small benefit vs. the cost we have decided not to use filtering for the current version of the model.
%Docmatix \citep{huggingface_docmatix} was also recently suggested as a large-scale alternative to the DocVQA dataset.
%Docmatix was created using a small language model, Phi-3 \citep{abdin2024phi}, to generate 9.5M QA pairs based on OCR extracted from 2.4M images and then employed regex filtering to reduce hallucination.
%In comparison, we used a larger language model (Mixtral 8x22B vs. Phi-3 3.8B), verbalized visual elements rather than generating QAs based only on textual elements, and found filtering is not crucial.
% ahn's version rewritten to include both efforts
\subsubsection{IBM Synthetic Visual Q\&A Datasets}
%~\label{synth-data}
\paragraph{DocFM-VQA (Synthetic Document VQA):}
There’s a lack of large-scale instruction-following datasets focused on visually-rich document understanding. The most commonly used dataset, DocVQA \citep{mathew2021docvqa}, is limited in size as it contains only 10K images and 40K QA pairs.
Docmatix \citep{huggingface_docmatix} has been recently introduced as a potential large-scale alternative to DocVQA, but it uses only the textual parts of documents for QA pairs generation. %Docmatix employed a small language model, Phi-3 \citep{abdin2024phi3technicalreporthighly}, to generate 9.5 million QA pairs from OCR-extracted text of 2.4 million images.
In our work, we leverage a large language model (Mixtral 8x22B), to synthetically generate a large-scale visual question answering (VQA) dataset using a subset of DocFM as seed. We call this dataset DocFM verbalization-based VQA or DocFM-VQA in short.
We employed a ``verbalized" representation of the documents as context for the LLM's (Mixtral 8x22B) QA generation. This involved extracting text from programmatic PDFs and augmenting it with verbal descriptions of visual elements such as charts, images, and tables. This approach encouraged the generation of questions that specifically target these elements. Our verbalization process included: 1) Converting charts to tables using DePlot \citep{liu-2022-deplot}, 2) Replacing images with captions generated by BLIP2 \citep{li2023blip2}. 3) Converting tables to markdown format using TableFormer \citep{tableformer}.
The DocFM-VQA contains a total of 20 million QA pairs from 2.4 million randomly selected pages from the DocFM dataset. This dataset facilitates the training of models capable of robust performance on visually rich documents.
While we explored the use of leading Visual Language Models (VLMs) to filter out potential hallucinations in the generated QAs, the observed performance benefits did not outweigh the computational cost. Therefore, filtering was not incorporated into the current model version.
%\textcolor{red}{Docmatix \citep{huggingface_docmatix} has been recently introduced as a potential large-scale alternative to DocVQA. Docmatix employed a small language model, Phi-3 \citep{abdin2024phi3technicalreporthighly}, to generate 9.5 million QA pairs from OCR-extracted text of 2.4 million images. It then applied regex filtering to mitigate hallucinations. In contrast, for DocFM-VQA we utilize a significantly larger LLM (Mixtral 8x22B vs. Phi-3 3.8B), incorporate explicit verbalization of visual elements beyond just textual content, and we found that filtering is not essential for achieving strong performance.}
% This paragraph reads redundant as the previous paragraph talked about dodcFM-vqa clearly.
% Eli - I think it's good to emphasize the difference from existing datasets
% Dhiraj - Eli I will rearrange this to put Docmatix and DocVQA discussion together
\paragraph{DocFM-VQA-charts (Synthetic Chart VQA):}
% Synthetic chart generation:
% the inverse oproach for generating QAs for a given chart, is generating the chart for known data. Pros- accurate data, possiblity to generate a diverese set of charts
A general framework for synthetic QA generation on charts is to use tabular data in textual form, using an LLM to generate QAs on it, and using code to generate the appropriate visual chart.
However, we observed two main issues with this approach.
First, the generated QAs are unable to refer to visual properties like the used markers or plot colors.
Second, LLMs tend to generate simple questions, such as a single data point extraction.
As a result, large-scale synthetic chart QA datasets tend to dramatically improve the performance on synthetically generated benchmarks but have a smaller effect on human questions or human-annotated benchmarks (e.g. ChartQA's augmented vs. human splits).
In our work, we addressed the two issues mentioned above by augmenting the tabular form of the data, used as input to the LLM for generating QAs, with additional information.
To solve the first issue of the LLM lacking knowledge of the visual appearance of the chart, we first choose a random marker and color for each data sequence. This information is added to the table consumed by the LLM and provided as input to the code used for rendering the chart.
To solve the second issue (of very simplistic QAs), we added to each table additional rows and columns. These rows and columns contain additional information calculated based on the original table using code.
It includes max, min, sum, average, and a random operation (addition or subtraction) between two randomly chosen columns/rows, e.g. ``Column 1 $+$ Column 3".
This pre-calculated information encourages the model to ask questions requiring higher-order reasoning.
It also allows the model to produce more accurate ground-truth answers for cases that require arithmetic capabilities that LLMs struggle with \citep{schwartz-etal-2024-numerologic}.
We used the augmented table approach as input to the DocFM-VQA pipeline to generate the DocFM-VQA-Charts dataset. It contains 5.5M QAs on about 500K synthetic charts rendered based on random numeric tables extracted from the DocFM dataset.
We additionally augmented the ChartQA training set with the pre-calculated columns and rows information (without markers and colors) and generated additional QAs for the dataset's existing images.
%The new VQA dataset for the ChartQA images are referred to as DocFM-VQA-AugmentedChartQA.
\paragraph{DocFM-VQA-flowcharts (Synthetic Flowchart VQA):}
In order for the model to better understand flowcharts we generated synthetic flowcharts with QA pairs. We used an LLM (Mixtral 8x7B) for generating the flowchart nodes and edges. We then use those nodes and edges to query an LLM for QA pairs and to render a visual representation of the flowchart. We used the pipeline described below to generate 17K JSON representation and rendered flowcharts. The JSONs are fed to the DocFM-VQA pipeline to generate 76K QA pairs.
%This dataset is referred to as DocFM-VQA-Flowcharts.
The flowchart synthesis pipeline:
\begin{itemize}
\item {\it Identify Business Domains and Industries}: Utilize the LLM to extract various business domains and identify industries associated with typical business processes.
\item {\it List Business Process Types}: For each business domain and industry, use the LLM to generate a comprehensive list of business process types.
\item {\it Generate Business Processes}: For each business process type, we employ the LLM to create multiple business processes constructed in a JSON format. Each process should have nodes and edges, with the number of nodes ranging from 5 to 20.
\item {\it Create Flowchart Images}: For each generated business process, use the ‘graphviz’ package to create a flowchart image. Incorporate random shapes, arrows, and colors to enhance visual diversity.
% \item Create QA pairs: For each generated business process, use LLM to generate QA pairs based on a JSON representation of the process.
\end{itemize}
\paragraph{DocFM-ChartExtraction:}
\label{sec:chart_extraction}
We created a training dataset of 550K chart images to ensure that the model can perform structured data extraction from chart visualizations. The dataset was generated by converting tables extracted from DocFM and public stock market data into diverse chart types using matplotlib, including line, scatter, bar, pie, candlestick, and OHLC charts.
For each visualization, training examples included extraction instruction with randomized output format requirements (Markdown, HTML, or JSON) and varied natural language phrasings. To enhance document-level understanding capabilities, 100K DocFM-derived charts were embedded into their source document contexts by covering the original table with the chart image. The financial subset comprises 40K candlestick and OHLC charts generated from NASDAQ, S\&P500, and NYSE stock data.
This synthetic dataset enables the training of vision-language models to extract structured information from diverse visualization types while handling multiple output formats and document contexts.
\paragraph{DocFM Visual Cue and Captioning:}
%Another set of LLM based tasks are ``Visual Cue Caption-Guided Question Generation".
In order to create this dataset, we masked captions on images and then used the captions to generate questions regarding images. We implemented a multi-step process as follows: 1) Using Docling \citep{DoclingTechReport} we filtered to get figures associated with their corresponding captions; 2) We utilized Granite 3.1 8B model to analyze the captions for the presence of visual cues such as ``black arrow," ``on the left," ``red circle," and similar indicators of figure elements or regions of interest; 3) If the caption contained an explicit identifier (e.g., ``Figure 5"), we extracted it using the parsed text from the PDF. The caption body text was then masked on the image, leaving only the identifier visible. If no explicit identifier was present, we used the figure's bounding box as a fallback identifier. This identifier can then be used for the question, \textit{``What is the black arrow pointing at in Figure 5.?"}. 4) Leveraging the LLM, we generated context-specific questions about the figure, using the caption to inform the question formulation and answer while hiding the textual context in the image.
% This dataset is referred to as ``DocFM Visual Cue \& Captioning".
% TODO ahn, uncomment the following or just leave it at the bottom This pipeline generated a total of 26 million QA pairs from 2.4 million randomly selected pages from the DocFM dataset.
\paragraph{DocFM Rule Based Grounding:}
We further complemented our data by implementing rule-based approaches to generate synthetic tasks focused on specific document elements like text localization, figures, and tables. Leveraging Docling's layout detector \citep{doclaynet}, we generated questions that query the location of text passages and other page elements. To encourage a deeper understanding of visual components, we masked captions associated with figures and tables, keeping only their identifiers. Subsequently, we employed templates to generate questions prompting the model to provide captions for these figures and tables. Furthermore, using defined templates we generated more operations such as selecting row, columns by index, or by value or comparison using only images. For the latter, we leverage the table understanding methods from Docling \citep{tableformer, otsl}.
% This dataset is referred to as ``DocFM Rule Based Grounding ".
\subsection{Public Datasets}
% Document related datasets
Our Granite Vision model is primarily focused on visual document understanding tasks. Therefore, we further enriched our data using a number of pre-curated high-quality, publicly available document-related datasets, which encompass several diverse tasks such as Document QA, Table QA, Chart QA, Diagram QA, OCR and scene-text related tasks, reasoning and grounding, UI screen/code and structure understanding. A detailed view of public document centric datasets is shown in Figure \ref{fig:doc_data}. For details readers are referred to Table \ref{tab:doc_datasets} (in Appendix).
%encompassing a diverse set of visual Q\&A tasks.
%such as Document QA, Table QA, Chart QA, Diagram QA, OCR and scene-text related tasks, reasoning and grounding, UI screen/code and structure understanding.
%See Table \ref{tab:doc_datasets} (in Appendix) for a comprehensive overview of the datasets.
%Such diverse data is chosen to ensure the model’s ability to generalize across a wide array of document-centric applications.
%: 1) Document QA, 2) Chart QA, 3) Infograpic QA, 4) Table QA, 5) Diagram QA, 6) Reasoning and captioning, 7) Grounding and text localization, 8) Scene text, 9) UI Screen/Code and 10) Structure parsing. Such diversity ensures the model’s ability to generalize across a wide array of document-centric applications. We provide the full list of the datasets in Figure \ref{fig:doc_data}.
% General image datasets
Beyond document-centric content, we also included high quality general image Q\&As sourced from public datasets %(which have a mix of both documents and general images)
to ensure robust performance in general visual tasks as well. Specifically, we acquired data from three high-quality collections: %of general image datasets:
Cauldron\footnote{\url{https://huggingface.co/datasets/HuggingFaceM4/the_cauldron}}, Cambrian-7M\footnote{\url{https://huggingface.co/datasets/nyu-visionx/Cambrian-10M}} and LLaVa-OneVision-Data\footnote{\url{https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data}}.
These datasets cover a wide range of domains and tasks, including image captioning, visual question answering (VQA), and object recognition, providing the foundational knowledge for general-purpose vision-language models. More details and statistics about general images data are shown in Figure ~\ref{fig:general_image_data}.
\subsection{Data Preprocessing}
Data pre-processing consisted of multiple annotation and filtering steps to conform to regulatory and safety requirements and improve overall data quality as described below.
\begin{itemize}
\item \textit{Restricted Data removal:}
All public datasets underwent a rigorous legal vetting process to identify data that could violate any legal or PII related requirements. All such data was removed, including datasets originating outside of the US and those with unclear license agreements.
%While reviewing the datasets, we identified that certain subsets are restricted for use due to licensing agreements and legal constraints. To ensure compliance, we exclude these subsets and use only the permissible data.
%\textbf{LLaVa format conversion:}
\item \textit{Sexual Abuse Material (CSAM) removal:} All CSAM was removed from the datasets using state-of-the-art NSFW (not safe for work) detection methods. %combined with detection of children in pictures.
\item \textit{Face blurring:} Face blurring was performed to obfuscate all visual PII from the data.
\item \textit{Deduplication:}
Since we combined multiple dataset collections, there could be duplication of text as well as visual data. %some samples may be duplicated, which can lead to memorization and performance degradation.
%Our pipeline identifies duplicate samples by matching identical conversations \textbf{and} images. We intentionally retain cases where the same image appears with different conversations. We detect identical images through two methods: exact pixel-wise match, or perceptual hash matching \citep{zauner2010implementation} for robustness against minor transformations.
Similar images were detected
through two methods: exact pixel-wise match, or perceptual hash matching \citep{zauner2010implementation}, for robustness against minor transformations.
% In addition, text based de-duplication was also performed.
Records with identical texts \textit{and} similar images were removed.
\end{itemize}
%\textbf{Numbers after data filtering placeholder}
\section{Granite Vision Model}
\label{sec:model}
In this section, we describe our model, including the architecture (Section~\ref{sec:model:arch}), the training procedure (Section~\ref{sec:model:training}), and the use of Sparse Attention Vectors (SAVs) for safety classification (Section~\ref{sec:model:sav}). An overview of our architecture is shown in Figure~\ref{fig:architecture}.
\subsection{Architecture}
\label{sec:model:arch}
% Anyres, MLP projector, Siglip, Granite3.1
% 5.1 Architecture (~1 page)
% - Alignment Pipeline Overview
% - Projector — include multi-layer features, dense HR views
% - LLM granite, …
% \FloatBarrier % Ensures previous floats are placed before this one
Vision-and-language models are designed to process two data modalities simultaneously, such as images and corresponding text instructions. Each modality is encoded into a shared embedding space, which is then utilized for reasoning by a language model $f$ parameterized by $\theta$. Specifically, an image is encoded using a pre-trained visual encoder, denoted as $v$ parameterized by $\phi$, and a projector $M$ parameterized by $\lambda$. A corresponding textual prompt is tokenized and encoded using a fixed language encoder $l$ parameterized by $\gamma$. Given an input image $I$ and a text instruction $P$, the language model generates a text response $R$ as follows: $R = f_\theta(M_\lambda(v_\phi(I)), l_\gamma(P))$. This section provides a detailed description of each of these components.
\begin{wrapfigure}{r}{0.5\textwidth} % Reduce width slightly
\vspace{-0.5cm}
\centering
\includegraphics[width=0.5\textwidth]{Figures/highlevel_model5.pdf}
\vspace{-0.2cm}
\caption{\textbf{Architecture of Granite Vision.}}
\label{fig:architecture}
\vspace{-0.2cm}
\end{wrapfigure}
\paragraph{Vision Encoder $v_\phi$.} For an input image $I$, we use a vision encoder $v_\phi$ to provide visual features $X=v_\phi(I)$. In our implementation, $X$ is a concatenation of outputs from multiple layers, allowing us to combine these different levels of representation, providing rich representations that are beneficial for visual document understanding. We use SigLIP~\citep{zhai2023siglip} as our vision encoder with $384 \times 384$ resolution. Additionally, we use the ``AnyRes'' technique in our training as in~\cite{li2024llavaonevision}, which is designed to accommodate images of various high resolutions. The ability to process high-resolution images is of particular importance for document understanding, especially when dealing with documents containing small text fonts, as it enables accurate text recognition.
% This is especially critical for effective document understanding when dealing with documents containing small text fonts.
Specifically, we use a global image, and a grid configuration that divides the image into multiple patches (or views), resulting in an additional multi-patch setting. We select the closest resolution from a predefined set that allows the original image size to be tiled into patches of size $384 \times 384$.
% $(384,768)$, $(384,1152)$, $(384,1536)$, $(384,1920)$, $(384,2304)$, $(384,2688)$, $(384,3072)$, $(384,3456)$, $(384,3840)$, $(768,384)$, $(768,768)$, $(768,1152)$, $(768,1536)$, $(768,1920)$, $(1152,384)$, $(1152,768)$, $(1152,1152)$, $(1536,384)$, $(1536,768)$, $(1920,384)$, $(1920,768)$, $(2304,384)$, $(2688,384)$, $(3072,384)$, $(3456,384)$, and $(3840,384)$.
We tile the image using up to ten patches, leading to a wide variety of aspect ratios ranging from $1:10$ through $1:1$ to $10:1$ at different scales resulting in a total of 27 different tiling options. Next, the visual features $X$ are forwarded to the projector $M$.
% \begin{figure}
% \centering
% \includegraphics[width=0.8\linewidth]{Figures/supported_resolutions.png}
% \caption{Supported resolutions}
% \label{fig:enter-label}
% \end{figure}
% For better results to support high resolution multi doc, we perform 1 global (384x384)+ up to 10 views using the set from below: [[384,768],[384,1152],[384,1536],[384,1920],[384,2304],[384,2688],[384,3072],[384,3456],[384,3840],[768,384],[768,768],[768,1152],[768,1536],[768,1920],[1152,384],[1152,768],[1152,1152],[1536,384],[1536,768],[1920,384],[1920,768],[2304,384],[2688,384],[3072,384],[3456,384],[3840,384]].
% 11 images=== 1 global (384x384)+ up to 10 views using the set from above
\paragraph{Projector $M_\lambda$.} Here we apply $M_\lambda$, a two-layer MLP with GELU activation function as the projection layer to convert the visual features into a sequence of visual tokens: $T_{vis} = M_\lambda(X)$. This operation connects the image features into the word embedding space. We ensure the $T_{vis}$ tokens have the same dimensionality as the word embedding space
in the language model.
% mm_vision_select_layer 4,8,16,27
\paragraph{Language Encoder $l_\gamma$.} For a text instruction $P$, we extract the text tokens $T_{lang}$ as the instruction undergoes tokenization using a language encoder $T_{lang} = l_\gamma(P)$.
\paragraph{Language Model $f_\theta$.} The visual tokens $T_{vis}$ and language tokens $T_{lang}$ are then concatenated and fed into a language model $f_\theta$. We use Granite 3.1-2B-Instruct~\citep{granite2024granite} as our language model $f_\theta$, which is trained for next token prediction. Granite 3.1 represents IBM’s third generation of large language models, released to the open-source community under an Apache 2.0 license, as well as matching top performance on general, enterprise, and safety language benchmarks. It supports up to 128K context length and shares a similar architecture as popular LLMs like Llama \citep{grattafiori2024llama3} to ensure compatibility with open-source inference and fine-tuning pipelines.
%We next describe our training procedure.
\subsection{Training Procedure}
\label{sec:model:training}
% 5.2 Training (~2 pages)
% - 3 stages (pre-training just projector, pretraining with LLM unfrozen and extract projector, instruction tuning)
The training procedure of our Granite Vision model consists of three stages: (i) Pre-training for projector $M$, (ii) Pre-training with language model $f_\theta$ and the Projector $M$, and (iii) Instruction tuning. Each image $I$ is accompanied by a text instruction $P$, and the predictions consist of the answer $A$ produced by the model. Specifically, for a response $R$, we compute the probability of the target words by the following equation:
\nolinebreak
\begin{equation}
p(A \mid I, P) = \prod_{i=1}^{|R|} p_{\psi}(x_i \mid I, P)
\end{equation}
\nolinebreak
where $\psi$ represents the trainable parameters. We note that $\psi$ contains the following trainable parameters mentioned above ($\theta, \phi, \lambda$), with different subsets are trained at various training stages. In addition, $x_i$ is the current prediction token. To calculate loss, we use the standard cross-entropy function with these probabilities. Next, we describe our three-step training process.
\paragraph{Stage 1: Pre-training for Projector $M_\lambda$.}
%Here, we keep the entire network frozen, besides the projector $M$ in order to align correctly the visual and language tokens. %In this way, the visual tokens can be aligned with the pre-trained language encoder.
In this step, the entire network (other than the projector $M_\lambda$) is frozen to correctly align visual and language tokens. For this stage, we use 558k image-text pairs with captions taken from LLaVA-Pretrain\footnote{\hyperlink{https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain}{https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain}.}. Additionally, we use only the following scales in this stage: (384, 768), (768, 384), (768, 768), (1152, 384), and (384, 1152).
% https://github.ibm.com/ai-models-architectures/mm-align/blob/leonid/scripts/leo/bluevela/pretrain_granite_3.0_8b_dolomite_multilayer_siglip_bluevela.sh
% --image_grid_pinpoints "[[384,768],[768,384],[768,768],[1152,384],[384,1152]]" \
% batch size =4*16*8
% --learning_rate 1e-4 \
% conv_granite_v3 = Conversation(
% system="<|system|>\nA chat between a curious user and an artificial intelligence assistant. "
% "The assistant gives helpful, detailed, and polite answers to the user's questions.\n",
% roles=("<|user|>", "<|assistant|>"),
% version="v1",
% messages=(),
% offset=0,
% sep_style=SeparatorStyle.GRANITE3,
% sep="\n",
% sep2="<|end_of_text|>",
% )
Finally, we perform a hyperparameter search randomly on 5\% of the data to select a batch size of $512$, warm up ratio of $0.03$, and learning rate of $1e-4$ with a cosine scheduler. We also used the following multi-turn conversation template:
\begin{table*}[htp]\centering
\begin{minipage}{0.99\columnwidth}\vspace{0mm} \centering
%\begin{tcolorbox}
\label{multi-conversation}
\raggedright
\small
% ${\texttt{\color{green} SYSTEM}}$ \\
% "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions." \\
% $\texttt{\color{green} USER}: P^{1} \quad \texttt{\color{green} ASSISTANT}: A^{1}$ \\
% $\texttt{\color{green} USER}: P^{2} \quad \texttt{\color{green} ASSISTANT}: A^{2}$
${\texttt{\color{green} <|system|>}}$ \\
"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions." \\
$\texttt{\color{green} <|user|>} P^{1} \quad \texttt{\color{green} <|assistant|>} A^{1} \texttt{\color{green}<|end\_of\_text|>}$\\
$\texttt{\color{green} <|user|>} P^{2} \quad \texttt{\color{green} <|assistant|>} A^{2} \texttt{\color{green}<|end\_of\_text|>}$
%\end{tcolorbox}
% \vspace{-0.2cm}
\end{minipage}
\end{table*}
\noindent where ($P^i$, $A^i$) is the $i$-th conversation pair iteration.
\paragraph{Stage 2: Pre-training with Language Model $f_\theta$ for Projector $M$.} In order to align the LLM with the updated visual features that have been processed by the projector, we use a second pre-training stage, in which we keep the vision encoder frozen and fine-tune the LLM $f_\theta$ and the Projector $M_\lambda$. We use the same image-text pairs data and same multi-conversation template as in stage 1.
% Finally, we perform a hyper-parameter search randomly on 5\% of the data for f
% conv_granite_v3 = Conversation(
% system="<|system|>\nA chat between a curious user and an artificial intelligence assistant. "
% "The assistant gives helpful, detailed, and polite answers to the user's questions.\n",
% roles=("<|user|>", "<|assistant|>"),
% version="v1",
% messages=(),
% offset=0,
% sep_style=SeparatorStyle.GRANITE3,
% sep="\n",
% sep2="<|end_of_text|>",
% )
% (ii) BS, LR,
% (iii) position embedding (RoPE)
It is important to note that at the end of this stage, we retain only Projector $M_\lambda$ weights and discard the fine-tuned LLM, reverting the language model to its original weights for the next stage. We found that this procedure leads to a better starting point for the following instruction tuning stage.
% It is important to note that for this stage, we are using only the learned weights from the Projector $M_\lambda$, discarding the weights from the LLM, thus retaining the original weights for the language model for the next stage. We found that this procedure led to better improvements towards the instruction tuning stage.
%Lastly, in this part, we use positional embeddings from RoPE~\citep{rope-paper}.
In stage 2, for the LLM and Projector $M_\lambda$, we use different learning rates of $8e-05$ and $32e-05$, respectively, and kept the same other stage 1 hyperparameters.
%We also perform a hyperparameter search randomly on 5\% of the data to select a batch size of $512$, warm up ratio of $0.03$, and a cosine scheduler.
% https://github.ibm.com/ai-models-architectures/mm-align/blob/leonid/scripts/leo/bluevela/pretrain_granite_3.0_8b_step2_dolomite_multilayer_siglip_bluevela.sh
% (i) instruction template,
% conv_granite_v3 = Conversation(
% system="<|system|>\nA chat between a curious user and an artificial intelligence assistant. "
% "The assistant gives helpful, detailed, and polite answers to the user's questions.\n",
% roles=("<|user|>", "<|assistant|>"),
% version="v1",
% messages=(),
% offset=0,
% sep_style=SeparatorStyle.GRANITE3,
% sep="\n",
% sep2="<|end_of_text|>",
% )
% (ii) BS, LR,
% (iii) position embedding (RoPE)
% --per_device_train_batch_size 1*16*8*gradient_accumulation_steps \
% --learning_rate 3e-05 \
% --mm_projector_lr 0.00016 \
% --per_device_train_batch_size 1*16*8*gradient_accumulation_steps \
% --learning_rate 3e-05 \
% --mm_projector_lr 0.00016 \
\paragraph{Stage 3: Instruction Tuning.} In the last stage, we perform supervised fine-tuning with all the instruction-following data. Here, we update both the pre-trained weights of the projector $M$ and the language model $f_\theta$, while keeping the visual encoder weights frozen. We use approximately 20M image-text pairs from our collected datasets, as described in Section~\ref{sec:data}. We retained the same hyperparameters from stage 2, with the exception of the batch size, which was set to $1024$.
% We used the same hyper-parameters as in stage 2, besides the LLM learning rate $3e-5$ and batch-size of $64$.
% https://github.ibm.com/ai-models-architectures/mm-align/blob/leonid/scripts/leo/bluevela/scaled16_hpo_blaunch_finetune_anyres_granite_3.0_8b_siglip_multilayer_hr_views_mminstruct_cambrian_docmatix_docfm2_onevision_bluevela.sh
% \paragraph{Alignment}
\subsection{Sparse Attention Vectors for Safety}
\label{sec:model:sav}
% Here we outline our approach for using sparse attention vectors (as proposed in~\cite{mitra2024sparse} and~\cite{Huang2024MultimodalTV}) derived from the activation space of a Granite Vision model as features for a safety task.
Here we outline how we use sparse attention vectors (as proposed in ~\cite{mitra2024sparse} and~\cite{Huang2024MultimodalTV}), derived from the activation space of a Granite Vision model, as features for a safety classification setup. Our key insight is that within Granite Vision's many attention heads and transformer layers, there is a sparse subset of features that are useful for identifying safety if we formalize a range of safety tasks as classification problems. We present the following three-step method to identify and utilize these safety features.
% The method consists of three main steps: (i) extracting the attention vectors from all attention heads in the model, (ii) identifying a sparse set of vectors based on their ability to consistently return the correct label for some support set of examples, and (iii) using these sparse features to classify new queries.
\paragraph{Step 1: Extracting General Attention Vectors.}
Given our model and few-shot samples of label pairs
$\{(x_1, y_1), (x_2, y_2), \dots, (x_N, y_N)\}$,
we first extract the attention vectors for each sequence $x_i$. Here, the labels in the few-shot demonstrations indicate specific safety characteristics of the query, like 'harmful' and 'non-harmful' classes. Next, for every $x_i$, we compute the attention vector $\mathbf{h}_l^m$ for head $m$ from layer $l$ for the final token $x_i^T$. This yields a set of attention vectors $\{\mathbf{h}_l^m(x_i^T) \mid i = 1, \dots, N\}$ for each head $m$ and layer $l$.
\paragraph{Step 2: Finding Safety Vectors.}
In order to identify which attention vectors are naturally suited to a safety task, we evaluate each vector's discriminative ability by computing its performance under a nearest class centroid classifier.
Specifically, for each class $c \in \mathcal{C}$, we compute its mean attention vector across few shot examples:
$$\mu_c^{l,m} = \frac{1}{|N_c|}\sum_{c} \mathbf{h}_l^m(x_j^T)$$
where $N_c$ is the set of indices of examples with label $c$. For each input $x_i$, we compute its cosine similarity to each class centroid head:
$$s_{l,m}(x_i, c) = \frac{\mathbf{h}_l^m(x_i^T) \cdot \mu_c^{l,m}}{\|\mathbf{h}_l^m(x_i^T)\| \|\mu_c^{l,m}\|}, \quad \forall c \in \mathcal{C}$$
Next, we measure the discriminative ability of each head by its performance as follows:
$$
\text{score}(l, m) = \sum_{i=1}^{N} \mathbf{1}[\hat{y} = y_i] $$
\noindent where the nearest class centroid label is given as $\hat{y}$, and $\mathbf{1}[\cdot]$ is the indicator function that evaluates to 1 when the condition is true. We denote the set of $k$ top-scoring heads as $\mathcal{H}_\text{SAV}$:
$$
\mathcal{H}_\text{SAV} = \{(l,m) \mid \text{score} (l,m) \hspace{0.5em} \text{is} \hspace{0.5em} \text{among} \hspace{0.5em} \text{top K} \}
$$
\paragraph{Step 3: Classification with Safety Vectors.}
Given a sample sequence input $Q$ to classify, we leverage our sparse set of heads $\mathcal{H}_\text{SAV}$ for safety prediction. For each head $(l,m) \in \mathcal{H}_\text{SAV}$, we compute the class centroid $\mu_c^{l,m}$ closest to the sequence input as follows:
$$
\hat{y}_{l,m} = \underset{c \in \mathcal{C}}{\arg\max}\ s_{l,m}(Q^T, c)
$$
where $s_{l,m}$ is defined as in Step 2.
The final prediction counts the majority across all heads in $\mathcal{H}_\text{SAV}$:
$$
\underset{y \in \mathcal{C}} {\arg\max} \sum_{(l,m) \in \mathcal{H}_\text{SAV}} \mathbf{1}[\hat{y}_{l,m} = y]
$$
Using this approach, we are able to determine whether a new input sequence is safe. We note that this approach can be used more generally, as it has been used in~\cite{mitra2024sparse} to extract multimodal features from large multimodal models for general discriminative vision-language tasks.
% This approach reveals a surprising capability of LMMs: with just a few carefully selected attention heads ($|\mathcal{H}_\text{SAV}| \ll LH$), we can transform a generative language model into a lightweight vision-language classifier. This finding suggests that classification-relevant features naturally emerge within specific attention heads during model pretraining.
\section{Evaluation}
\subsection{Standard benchmarks}
We evaluated our models on a set of popular public benchmarks. Since the current Granite Vision release is mainly geared towards document understanding, our focus is on document-related benchmarks. However, we also report results for key natural-image benchmarks.
We used the standardized \textit{lmms-eval} package \citep{zhang2024lmmsevalrealitycheckevaluation,lmms_eval2024} to run the evaluations. We do not use any test time optimizations such as prompt tuning or chain-of-thought.
The results for most other models were also produced by running the standard \textit{lmms-eval} benchmarks when possible.
For the remaining models, we used the results reported by the original authors or from the publicly reported benchmarks.
\paragraph{Document related benchmarks}
\begin{itemize}
\item \textbf{DocVQA}: A benchmark designed to evaluate the models' ability to understand and extract textual information from documents \citep{mathew2021docvqa}.
\item \textbf{ChartQA}: A benchmark which focuses on question answering about charts, requiring both visual and logical reasoning to interpret data presented in various chart formats \citep{masry2022chartqa}.
\item \textbf{TextVQA}: Challenges models to read and reason about text present within images to answer questions accurately \citep{yang2021tap}.
\item \textbf{AI2D}: Contains grade school science diagrams, aimed at evaluating diagram understanding and question answering capabilities \citep{kembhavi2016diagram}.
\item \textbf{InfoVQA}: Contains infographics - documents that combine textual, graphical, and visual elements to communicate information effectively. The questions require models to perform joint reasoning across document layout, text, and data visualizations \citep{mathew2022infographicvqa}.
\item \textbf{OCRBench}: A benchmark designed to evaluate Optical Character Recognition (OCR) capabilities within various contexts, assessing the accuracy and robustness of OCR in documents and in-the-wild \citep{liu2024ocrbench}.
\item \textbf{WebSRC}: A Web-based Structural Reading Comprehension benchmark. It consists of screenshots and question-answer pairs created based on the corresponding HTML source code and metadata. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no.
\item \textbf{LiveXiv}: A benchmark which specifically focuses on testing multi-modal models' ability to understand domain-specific visual content like graphs, charts, and tables from scientific manuscripts. It is a live benchmark, containing recently published papers on Arxiv, helping prevent test set contamination that can occur with static benchmarks when models are trained on web-scraped data. It contains questions dealing with figures (VQA) and with tables (TQA) \citep{shabtay2024livexiv}.
\end{itemize}
\paragraph{Natural image benchmarks}
\begin{itemize}
\item \textbf{MMMU}: A comprehensive benchmark designed to evaluate Multimodal Large Language Models on both perception and cognition abilities across various subtasks, ensuring robust and diverse testing of these models \citep{yue2024mmmu}.
\item \textbf{VQAv2}: A benchmark for Visual Question Answering containing open-ended questions about images, designed to test a model's ability to understand and reason about visual content \citep{goyal2017making}.
\item \textbf{RealWorldQA}: Focuses on question answering in real-world scenarios, requiring models to interpret and reason over complex visual and textual information present in everyday images.
\item \textbf{VizWiz}: Contains over 31,000 visual questions originating from blind individuals, aiming to help answer visual questions posed by this community, with each question accompanied by crowdsourced answers \citep{gurari2018vizwiz}.
\item \textbf{OK VQA}: A benchmark designed for visual question answering tasks that require external knowledge beyond what is visible in the image, featuring over 14,000 questions to evaluate the reasoning abilities of AI models \citep{marino2019ok}.
\end{itemize}
The results are presented in Table \ref{tab:model_comparison}. Granite Vision demonstrates impressive performance across document-related and natural image benchmarks. In the small model category (1B-4B), it achieves leading scores on most key benchmarks, e.g. DocVQA (88\%) and ChartQA (86\%), outperforming other models. Particularly notable is that its performance on these tasks is competitive with much larger proprietary models, such as Gemini-1.5-Pro, GPT-4o, and GPT-4V, which have significantly more parameters. The results highlight that Granite-Vision's data curation and training approach are particularly effective for document understanding tasks, achieving strong results despite its compact size.
\begin{table}[t]
\centering
% \small
% \hspace{-5pt}
\setlength{\tabcolsep}{3pt} % Default is 6pt
\begin{tabular}{lc|ccccccccc|cccccc}
\toprule
& & \multicolumn{9}{c}{Document benchmarks} & \multicolumn{5}{c}{Other benchmarks} \\
Model & Size & \rotatebox{90}{DocVQA} & \rotatebox{90}{ChartQA} & \rotatebox{90}{TextVQA} & \rotatebox{90}{AI2D} &\rotatebox{90}{InfoVQA} & \rotatebox{90}{OCRBench} & \rotatebox{90}{WebSRC} &\rotatebox{90}{LiveXiv VQA} &\rotatebox{90}{LiveXiv TQA}& \rotatebox{90}{MMMU}& \rotatebox{90}{VQAv2}& \rotatebox{90}{RealWorldQA}&
% \rotatebox{90}{Math Vista}&
\rotatebox{90}{VizWiz VQA}& \rotatebox{90}{OK VQA} \\
\midrule
\rowcolor{gray!20} \multicolumn{15}{l}{Small models 1B-4B} &\\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{small}}}
% ]{results.csv}{}%
% {\model & \params & \docvqaXvalXanls & \chartqaXrelaxedXoverall & \textvqaXvalXexactXmatch & \aiTwodXfixedXexactXmatchXflexibleXextract & \infovqaXvalXanls & \ocrbenchXocrbenchXaccuracy & \websrcXvalXwebsrcXsquadXfOne & \livexivXvqaXmmXliveXbench & \livexivXtqaXmmXliveXbench & \mmmuXvalXmmmuXacc & \vqavTwoXvalXexactXmatch & \realworldqaXexactXmatchXflexibleXextract & \vizwizXvqaXvalXexactXmatch & \okXvqaXvalTwoZeroOneFourXexactXmatch \\}%
% \vspace{-10pt}
% \\
Molmo-E & 1B & 0.66 & 0.60 & 0.62 & 0.63 & 0.44 & \underline{0.65} & 0.68 & 0.47 & 0.36 & 0.32 & 0.57 & 0.55 & 0.49 & 0.40 \\
MM1.5* & 1B & 0.81 & 0.67 & \underline{0.72} & 0.59 & 0.50 & - & - & - & - & 0.36 & - & 0.53 & - & - \\
SmolVLM* & 2.2B & 0.80 & {0.72} & \underline{0.72} & \textbf{0.84} & - & \underline{0.65} & - & - & - & 0.38 & - & - & - & - \\
MM1.5* & 3B & \underline{0.87} & 0.74 & \textbf{0.76} & 0.66 & 0.58 & - & - & - & - & 0.37 & - & 0.57 & - & - \\
Phi3v & 4B & \underline{0.87} & 0.81 & 0.69 & \underline{0.79} & 0.55 & 0.64 & \textbf{0.91} & \underline{0.61} & 0.48 & \underline{0.42} & 0.76 & \underline{0.60} & \underline{0.57} & 0.51 \\
Phi3.5v & 4B & \textbf{0.88} & \underline{0.82} & 0.70 & \underline{0.79} &\underline{ 0.61} & 0.64 & \textbf{0.91} & \textbf{0.63} & \underline{0.51} & \textbf{0.44} & \underline{0.77} & 0.58 & \underline{0.57} & \underline{0.53} \\
\rowcolor{green!20} Granite Vision & 3B & \textbf{0.88} & \textbf{0.86} &\textbf{ 0.76} & 0.78 & \textbf{0.63} & \textbf{0.75} & \underline{0.90} & \underline{0.61} & \textbf{0.55} & 0.35 & \textbf{0.81} & \textbf{0.65} & \textbf{0.64} & \textbf{0.57} \\
% \midrule
% \rowcolor{gray!20} \multicolumn{14}{l}{Open models 1B-4B} &\\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{small}} and test{\ifcsvstrequal{\open}{yes}}}
% ]{results.csv}{}%
% {\model & \params & \docvqaXvalXanls & \chartqaXrelaxedXoverall & \textvqaXvalXexactXmatch & \aiTwodXfixedXexactXmatchXflexibleXextract & \infovqaXvalXanls & \ocrbenchXocrbenchXaccuracy & \livexivXvqaXmmXliveXbench & \livexivXtqaXmmXliveXbench & \mmmuXvalXmmmuXacc & \vqavTwoXvalXexactXmatch & \realworldqaXexactXmatchXflexibleXextract & \vizwizXvqaXvalXexactXmatch & \okXvqaXvalTwoZeroOneFourXexactXmatch \\}%
% \vspace{-10pt}
% \\
\midrule
\rowcolor{gray!20} \multicolumn{15}{l}{Mid-size models 7B-13B} &\\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{mid}}}
% ]{results.csv}{}%
% {\model & \params & \docvqaXvalXanls & \chartqaXrelaxedXoverall & \textvqaXvalXexactXmatch & \aiTwodXfixedXexactXmatchXflexibleXextract & \infovqaXvalXanls & \ocrbenchXocrbenchXaccuracy & \websrcXvalXwebsrcXsquadXfOne & \livexivXvqaXmmXliveXbench & \livexivXtqaXmmXliveXbench & \mmmuXvalXmmmuXacc & \vqavTwoXvalXexactXmatch & \realworldqaXexactXmatchXflexibleXextract & \vizwizXvqaXvalXexactXmatch & \okXvqaXvalTwoZeroOneFourXexactXmatch \\}%
% \vspace{-10pt}
% \\
Molmo-D & 7B & 0.73 & 0.58 & 0.70 & 0.79 & 0.52 & 0.70 & 0.76 & 0.61 & 0.51 & - & 0.68 & 0.51 & 0.45 & 0.51 \\
Molmo-O & 7B & 0.70 & 0.59 & 0.59 & 0.77 & 0.50 & 0.66 & 0.77 & 0.60 & 0.46 & - & 0.38 & 0.67 & 0.56 & 0.39 \\
MM1.5* & 7B & 0.88 & 0.78 & 0.76 & 0.72 & 0.59 & - & - & - & - & 0.42 & - & 0.62 & - & - \\
Cambrian-1* & 8B & 0.78 & 0.73 & 0.72 & 0.73 & - & 0.62 & - & - & - & 0.43 & - & 0.64 & - & - \\
Llama3.2* & 11B & 0.88 & 0.83 & - & 0.91 & - & - & - & - & - & 0.51 & 0.75 & - & - & - \\
Pixtral & 12B & 0.91 & 0.83 & 0.76 & 0.80 & 0.56 & 0.66 & 0.90 & 0.75 & 0.61 & 0.50 & 0.79 & 0.63 & 0.58 & 0.59 \\
Cambrian-1* & 13B & 0.77 & 0.74 & 0.73 & 0.74 & - & 0.62 & - & - & - & 0.40 & - & 0.63 & - & - \\
\midrule
\rowcolor{gray!20} \multicolumn{15}{l}{Larger models} &\\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{large}}}
% ]{results.csv}{}%
% {\model & \params & \docvqaXvalXanls & \chartqaXrelaxedXoverall & \textvqaXvalXexactXmatch & \aiTwodXfixedXexactXmatchXflexibleXextract & \infovqaXvalXanls & \ocrbenchXocrbenchXaccuracy & \websrcXvalXwebsrcXsquadXfOne & \livexivXvqaXmmXliveXbench & \livexivXtqaXmmXliveXbench & \mmmuXvalXmmmuXacc & \vqavTwoXvalXexactXmatch & \realworldqaXexactXmatchXflexibleXextract & \vizwizXvqaXvalXexactXmatch & \okXvqaXvalTwoZeroOneFourXexactXmatch \\}%
% \vspace{-10pt}
% \\
% \hline
MM1.5* & 30B & 0.91 & 0.83 & 0.79 & 0.77 & 0.67 & - & - & - & - & 0.47 & - & 0.69 & - & - \\
Cambrian-1* & 34B & 0.76 & 0.76 & 0.77 & 0.80 & - & 0.60 & - & - & - & 0.50 & - & 0.68 & - & - \\
NNLM-D* & 72B & 0.92 & 0.86 & 0.82 & 0.94 & - & - & - & - & - & 0.60 & 0.85 & 0.70 & - & - \\
Llava-OV* & 72B & 0.93 & 0.84 & - & 0.86 & 0.79 & - & - & - & - & 0.57 & - & 0.72 & - & - \\
Llama3.2* & 90B & 0.90 & 0.85 & - & 0.92 & - & - & - & - & - & 0.60 & 0.78 & - & - & - \\
Claude-Sonnet* & - & 0.95 & 0.91 & - & 0.95 & - & - & - & 0.75 & 0.83 & 0.68 & - & - & - & - \\
Gemini-1.5-Pro* & - & 0.93 & 0.87 & 0.79 & 0.94 & 0.81 & - & - & - & - & 0.62 & 0.80 & 0.70 & - & - \\
GPT-4V* & - & 0.88 & 0.78 & - & 0.76 & - & - & - & - & - & 0.53 & - & 0.56 & - & - \\
GPT-4o* & - & 0.93 & 0.85 & - & 0.84 & - & - & - & 0.60 & 0.54 & 0.69 & - & 0.75 & - & - \\
% \midrule
% \rowcolor{gray!20} \multicolumn{14}{l}{Closed larger models} &\\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{large}} and test{\ifcsvstrequal{\open}{no}}}
% ]{results.csv}{}%
% {\model & \params & \docvqaXvalXanls & \chartqaXrelaxedXoverall & \textvqaXvalXexactXmatch & \aiTwodXfixedXexactXmatchXflexibleXextract & \infovqaXvalXanls & \ocrbenchXocrbenchXaccuracy & \livexivXvqaXmmXliveXbench & \livexivXtqaXmmXliveXbench & \mmmuXvalXmmmuXacc & \vqavTwoXvalXexactXmatch & \realworldqaXexactXmatchXflexibleXextract & \vizwizXvqaXvalXexactXmatch & \okXvqaXvalTwoZeroOneFourXexactXmatch \\}%
% \vspace{-10pt}
% \\
\bottomrule
\end{tabular}
\caption{Performance comparison across different models and benchmarks. For models marked with * we report only the available numbers from the original publication or the published benchmark and did not run the evaluations in the controlled \textit{lmms-eval} setup.}
\label{tab:model_comparison}
\end{table}
\subsection{Additional evaluations}
Converting tables and charts to a structured format are important tasks for enterprise use cases. We added to lmms-eval the benchmarks described below and compared our model with other open models.
\paragraph{Table extraction}
To evaluate the ability of models to turn a table image into a structured format, we used the test sets of
PubTables \citep{smock2022pubtables} and
FinTabNet \citep{chen2021finqa}.
We instruct the model to extract the data in HTML table format
and evaluate performance according to Tree-Edit-Distance-based Similarity (TEDS) from \cite{zhong2020image}. It measures both the table structure similarity and each cell string edit distance.
\paragraph{Chart extraction}
Similar to table extraction evaluation, we also evaluate the ability of models to turn a chart image into a structured format. We used the test set of ChartQA \citep{masry2022chartqa} for this evaluation.
% and our randomly rendered synthetic charts based on tables from DocFM and stock market data (see \ref{sec:chart_extraction}).
% For the DocFM data, we also test a harder task where the chart is \textit{planted} or embedded in the original page, with the chart image covering the original table in the page image.
We instruct the model to extract the data in Markdown or HTML format, the prompt includes an example of the expected format.
Model performance is evaluated according to a Modified TEDS (mTEDS) metric. The original TEDS measures string edit distance for each cell, which is not optimal for numerical values where the distance should take into account the scale of chart values. We define mTEDS as follows, let $T$ be a table with numeric values $V=\{v_{ij}\}$ excluding headers. Define the normalized function:
$$
N(v_{ij}) = round\left(20 \cdot \frac{v_{ij}}{\max_{i,j}(abs(V_{gt}))}\right),$$
where $V_{gt}$ represents the ground truth table values. The metric applies standard TEDS computation on the normalized values, comparing structural and content similarity between predicted and ground truth tables. This normalization accounts for scale-dependent accuracy in chart data extraction while maintaining TEDS's ability to evaluate structural correctness.
The results are reported in Table \ref{tab:model_comparison_chart_extraction}.
Granite Vision significantly outperforms other models in its size category.
It shows an advantage of $+12\%$-$+39\%$ over the second-best models in tables and charts extraction to markdown and HTML.
Notably, Granite Vision's performance is comparable to the 4 times larger Pixtral model.
\begin{table}[t]
\centering
% \small
\begin{tabular}{lccccc}
\toprule
& & \multicolumn{2}{c}{Table Extraction} & \multicolumn{2}{c}{Chart Extraction} \\
% Model & Size & \rotatebox{90}{FinTabNet} & \rotatebox{90}{PubTables} &\rotatebox{90}{ChartQA} & \rotatebox{90}{Chart2Table} &\rotatebox{90}{Chart2Table Planted} &\rotatebox{90}{Stocks}\\
Model & Size & {PubTables} & FinTabNet & {MD} & HTML \\
\midrule
\rowcolor{gray!20} \multicolumn{6}{l}{Small models 1B-4B} \\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{small}} and test{\ifcsvstrequal{\open}{yes}}}
% ]{extraction_results.csv}{}%
% {\model & \params & \pubtablesXOneZeroZeroZeroXteds & \fintabnetXteds & \chartqaXextractionXtedsXmd & \chartqaXextractionXtedsXhtml \\}%
% % {\model & \params & \fintabnetXteds & \pubtablesXOneZeroZeroZeroXteds & \chartqaXextractionXteds& \chartTwotableXteds & \chartTwotableXplantedXteds &\stocksXteds \\}%
% \vspace{-10pt}
% \\
% \hline
Molmo-E & 1B & 0.28 & 0.28 & 0.57 & 0.54 \\
SmolVLM & 2.2B & 0.32 & 0.18 & 0.12 & 0.02 \\
Phi3.5v & 4B & 0.58 & 0.28 & 0.77 & 0.40 \\
\rowcolor{green!20}
Granite Vision & 3B & \textbf{0.70} & \textbf{0.54} & \textbf{0.93} & \textbf{0.95} \\
\midrule
\rowcolor{gray!20} \multicolumn{6}{l}{Larger models 7B-12B} \\
% \csvreader[head to column names,
% filter expr={ test{\ifcsvstrequal{\sizegroup}{mid}} and test{\ifcsvstrequal{\open}{yes}} }
% ]{extraction_results.csv}{}%
% {\model & \params & \pubtablesXOneZeroZeroZeroXteds & \fintabnetXteds & \chartqaXextractionXtedsXmd & \chartqaXextractionXtedsXhtml \\}%
% \vspace{-10pt}
% \\
% \hline
Molmo-D & 7B & 0.34 & 0.25 & 0.62 & 0.62 \\
Molmo-O & 7B & 0.25 & 0.15 & 0.60 & 0.60 \\
Pixtral & 12B & 0.73 & 0.48 & 0.93 & 0.92 \\
\bottomrule
\end{tabular}
\caption{Table and chart extraction performance comparison across different models. Tables from PubTables and FinTabNet are extracted into HTML format and evaluated with the TEDS metric. Charts, from the ChartQA dataset, are extracted to either MD or HTML based on the prompt and evaluated with the mTEDS metric.}
\label{tab:model_comparison_chart_extraction}
\end{table}
\subsection{Safety Benchmarks}
To evaluate the safety capabilities of our Granite Vision model, we employ two different setups. First, we utilize the standard VLM-as-a-Judge setup as described in \citep{Li2024RedTV}. Second, we also introduce a new safety classification setup, where we formalize a range of safety tasks as classification problems. This new setup is aimed at building safer and more reliable AI models by leveraging strong classification capabilities. While existing generative MLLMs typically do not excel at classification tasks~\citep{mitra2024sparse,Huang2024MultimodalTV}, we believe that enhancing their discriminative capabilities is essential for enabling diverse applications and more sophisticated reasoning.
% Although existing generative MLLMs do not usually excel at classification tasks, we believe that strong MLLM classification abilities are essential for building safer, more reliable AI models, which enable diverse applications and more sophisticated reasoning capabilities.
For the new safety classification setup, we also apply our Safety Vectors (SVs) approach (See Section~\ref{sec:model:sav}) to our Granite Vision model. Our method is compared with other strong baselines on three public benchmarks: VLGuard~\citep{zong2024safety}, RTVLM~\citep{Li2024RedTV}, and LMM-Halucination~\citep{chen2024unified}.
% a variety of public safety benchmarks, such as VLGuard~\citep{zong2024safety}, RTVLM~\citep{Li2024RedTV}, and LMM-Halucination~\citep{chen2024unified}.
% Next, we provide more details for the evaluation process.
\paragraph{Benchmarks.} (i) \textbf{VLGuard}~\citep{zong2024safety} focuses on vision-language safety and identifies four main categories of harmful content: Privacy, Risky Behavior, Deception and Hateful Speech. The dataset consists of images from diverse sources and the instructions are generated by GPT-4V~\citep{OpenAI2023GPT4TR} with each safe image having both safe and unsafe instructions, and each unsafe image having a single instruction. (ii) \textbf{RTVLM}~\citep{Li2024RedTV} is the first red teaming dataset to benchmark current MLLMs in terms of several different aspects: faithfulness, privacy, safety, and fairness. In this dataset, 5,200 samples have been annotated by humans, or generated by GPT-4 accompanied by examples provided by humans. To ensure that the test data is unique and has not been seen by any of the evaluated VLMs, the authors produced new question-image pairs from publicly available images or generated from diffusion. (iii) \textbf{LMM-Halucination}~\citep{chen2024unified} is a dataset that evaluates the hallucinations of the models when answering multi-modal tasks. We use the default evaluation method provided in the dataset to identify whether this scenario is ``hallucinating'' or ``not hallucinating'', and compute the accuracy rate on correctly identified scenarios.
% \paragraph{Baselines.} To evaluate the safety of Granite Vision, we use two different setups. In the VLM-as-a-Judge setup, we compare Granite Vision against the following MLLMs: (i) Phi-3.5-vision~\citep{abdin2024phi} is part of a family of small language models, which are designed for high capability and cost-effectiveness, performing well on tasks like language understanding, reasoning, coding, and math. Their smaller size and greater efficiency compared to larger language models make them suitable for deployment on devices with limited computing power, such as mobile phones; (ii) InternVL2~\citep{chen2024internvl} is a new family of open-source MLLM, from a 1B model for edge devices to a powerful 108B model. It uses a progressive alignment training strategy, efficiently scaling model size and refining data for strong performance with limited resources. Supporting diverse input modalities and output formats, InternVL2 generalizes across hundreds of vision-language tasks, matching specialized models' performance; (iii) DeepSeek-VL2~\citep{wu2024deepseekvl2} is a new series of large Mixture-of-Experts (MoE) Vision-Language Models. This model excels in various tasks, including visual question answering, OCR, document understanding, and visual grounding.
% For the safety classification setup, we compared the performance of Granite Vision against the following MLLMs: (i) Qwen2-VL~\citep{wang2024qwen2} achieves state-of-the-art performance on a variety of image-text and video-text tasks by dynamically processing varying-resolution images into visual tokens and employing novel positional embeddings that effectively fuse positional information across modalities. In our implementation we extract the SVs from the train set and evaluate on the test set. We note that the image-instruction pairs can be categorized into three categories: safe image and safe instruction, unsafe image and unsafe instruction, unsafe image and safe instructions. We merged the last two categories into one, as they are harmful in general; (ii) LLaVA-OneVision~\citep{Li2024LLaVAOneVisionEV} is an open source MLLM that performs well in single-image, multi-image and video tasks; (iii) MOLMO~\citep{molmo} is a recent MLLM based on PixMo, a collection of new datasets for MLLM training, including highly detailed image captions, free-form image QA, and an innovative 2D pointing dataset, all collected without external VLMs; (4) InstructBLIP~\citep{dai2023instructblip} is based on the pretrained BLIP-2 models, which are large-scale vision-language models trained on 400M image-text pairs.
\paragraph{Baselines.} We compared Granite Vision's safety performance against the following MLLMs: (i) Phi-3.5-vision~\citep{abdin2024phi} is part of a family of small language models, which are designed for high capability and cost-effectiveness, performing well on tasks like language understanding, reasoning, coding, and math; (ii) LLaVA-v1.5-7B~\citep{liu2023llava} is a state-of-the-art MLLM that maps CLIP visual features to the LLM's embedding space and uses instruction tuning on the diverse LLaVA-Instruct-158k dataset for visual alignment. This dataset combines images with various response types, including conversational, descriptive, and reasoning, to enhance alignment; (iii) SmolVLM~\citep{smolVLM} is a new family of 2B small vision-language models capable of answering questions about images, describing visual content, creating stories based on multiple images, or functioning solely as a language model. Its lightweight design makes it ideal for on-device applications while ensuring strong performance.
% (ii) InternVL2~\citep{chen2024internvl} is a new family of open-source MLLMs, from a 1B model for edge devices to a powerful 108B model. It uses a progressive alignment training strategy, efficiently scaling model size and refining data for strong performance with limited resources; (iii) DeepSeek-VL2~\citep{wu2024deepseekvl2} is a new series of large Mixture-of-Experts (MoE) Vision-Language Models. This model excels in various tasks, including visual question answering, OCR, document understanding, and visual grounding.
% For the safety classification setup, we compared the performance of Granite Vision against the following MLLMs: (i) Qwen2-VL~\citep{wang2024qwen2} achieves state-of-the-art performance on a variety of image-text and video-text tasks by dynamically processing varying-resolution images into visual tokens and employing novel positional embeddings that effectively fuse positional information across modalities. (ii) LLaVA-OneVision~\citep{Li2024LLaVAOneVisionEV} is an open source MLLM that performs well in single-image, multi-image and video tasks; (iii) MOLMO~\citep{molmo} is a recent MLLM based on PixMo, a collection of new datasets for MLLM training, including highly detailed image captions, free-form image QA, and an innovative 2D pointing dataset, all collected without external VLMs; (iv) InstructBLIP~\citep{dai2023instructblip} is based on the pretrained BLIP-2 models, which are large-scale vision-language models trained on 400M image-text pairs.
% Phi3.5-vision-ins & 24.0 & 26.9 & - & - & - & - \\
% InstructBLIP & 24.0 & 26.9 & - & - & - & - \\
% MOLMO & 24.0 & 26.9 & - & - & - & - \\
% \begin{wraptable}{r}{0\textwidth}
% % \vspace{-0.9cm}
% \begin{tabular}{lccc}
% \toprule
% & Unsafe Img. & Safe Img.-Unsafe Inst. \\
% \midrule
% Phi-3.5-vision & 91.2 & 90.7 \\
% InternVL2-4B & 92.3 & 91.0 \\
% DeepSeek-VL2 & 92.7 & 91.3 \\
% Granite Vision & \textbf{93.5} & \textbf{91.4} \\
% \bottomrule
% \end{tabular}
% \caption{\textbf{Results} for VLM-as-a-Judge}
% \label{tab:vlm-judge}
% \end{wraptable}
We next discuss the two distinct safety evaluation setups.
\input{Tables/safety-table2}
\paragraph{VLM-as-a-Judge Setup.} Here, we perform a standard evaluation using the approach of VLM-as-a-Judge in the same manner as described in the RTVLM evaluation procedure (see Table 8 in~\cite{Li2024RedTV}). Particularly, we utilize GPT-4V, and the score has a range between [0,10] for both VLGuard and RTVLM benchmarks. In the evaluation of VLGuard, we distinguish between two categories of image-instruction pairs as provided by the original categories: ``Unsafe'', where both the images and instructions are unsafe, and ``Safe-Unsafe'', where the images are unsafe but the instructions are safe. For RTVLM, we evaluate using the same setup as in~\citep{Li2024RedTV}, focusing on four safety categories: `Mislead', `Politic', `Racial', and `Jailbreak'. Last, we compare with other baselines, like Phi-3.5-vision~\citep{abdin2024phi}, LLaVA-v1.5-7B~\citep{liu2023llava}, SmolVLM~\citep{smolVLM}.
% InternVL2~\citep{chen2024internvl}, and DeepSeek-VL2~\citep{wu2024deepseekvl2}.
% \begin{wraptable}{r}{0\textwidth}
% % \vspace{-10pt}
% \begin{tabular}{lcccc}
% \toprule
% & \multicolumn{2}{c}{VLGuard} & RTVLM & \\
% \cmidrule(lr){2-3}
% & Unsafe & Safe-Unsafe \\
% \midrule
% Phi-3.5-vision & \textbf{8.7} & \textbf{9.3} & 8.4 \\
% InternVL2-4B & 7.7 & 8.9 & - \\
% DeepSeek-VL2 & 6.4 & 8.7 & - \\
% Granite Vision & 6.5 & 8.6 & 7.2 \\
% \bottomrule
% \end{tabular}
% % \vspace{-10pt}
% \caption{\textbf{Results} for \textit{VLM-as-a-Judge} setup.}
% \label{tab:vlm-judge}
% \vspace{-15pt}
% \end{wraptable}
%Results are shown in Table~\ref{tab:vlm-judge}. As can be seen, Granite Vision performs comparably to other baselines on the standard safety evaluation setup. While this is encouraging, we believe there is much more to be done to improve the model's safety, and we leave this for future work. In the next section, we describe our new efforts to formalize a wide range of safety tasks as classification problems.
%Dhiraj - somehow table labels are mixed up so correcting the reference
Results are shown in Table~\ref{tbl:safety}. %Table~\ref{tab:vlm-judge}.
As can be seen, Granite Vision performs comparably to other baselines on the standard safety evaluation setup. Readers must note that Granite Vision is the most lightweight model compared to baselines. We believe that there's definitely scope for further improvement and we discuss this in our future work. In the next section, we describe our new efforts to formalize a wide range of safety tasks as classification problems.
%Dhiraj Joshi made some changes above - please review Roi, Shafiq, Rogerio
\paragraph{Safety Classification Setup.} In this setup, we reformat the tasks in VLGuard to be a classification task and we report the raw classification accuracy of our method. Specifically, we prompt the model with labels, and ask it to select the right answer, comparing it to the ground truth. The motivation behind reformulating safety tasks as classification problems is that it encourages the development of MLLMs with robust discriminative abilities, crucial for safety-critical applications, and provides a clear, quantifiable metric for assessing safety performance. Last, we also apply our Safety Vectors (SVs) approach (See Section~\ref{sec:model:sav}) to our Granite Vision model.
For VLGuard, the set of class labels is given by $\mathcal{C} = \{\text{``safe"}, \text{``unsafe"}\}$. We use the metric VLGuard proposed Attack Success Rate (ASR) for evaluating unsafe inputs and Helpfulness for evaluating safe inputs. Note that accuracy is the complement of the ASR and can be calculated as 1 - ASR. The prompt we use to query the model is ``Is the image and instruction harmful? Answer the question with Yes or No." In our implementation, we extract the SVs from the train set and evaluate on the test set. In this experiment, we focus solely on the ``Unsafe" category, as it is the most relevant aspect for safety evaluation. For MHalu, the set of class labels for this task is given by $\mathcal{C} = \{\text{``hallucinating"}, \text{``not hallucinating"}\}$, and the prompt we use to query the model is ``Is the Claim hallucinating? Answer the question with Yes or No."
% For RTVLM in the classification setup, we focused specifically on four categories: `Mislead', `Politic', `Racial', and `Jailbreak', using images from existing datasets or generated by diffusion. The set of class labels is given by $\mathcal{C} = \{\text{``proceed with caution"}, \text{``safe to answer"}, \text{``refuse to answer"}\}$, and the prompt we use to query the model is "How would you classify this query? A. Proceed with Caution B. Safe to Answer C. Refuse to Answer?".
% \vspace{-15pt}
\begin{wraptable}{r}{0.4\textwidth}
\vspace{-10pt}
\begin{tabular}{lccc}
\toprule
& \multicolumn{1}{c}{MHalu} & \multicolumn{1}{c}{VLGuard} & \\
% \cmidrule(lr){2-3}
% & Unsafe \\
\midrule
% Phi-3.5-vision & & 85.0 \\
% InternVL2-4B & & - \\
% DeepSeek-VL2 & - & \\
% Qwen2-VL-7B-ZS & & 26.9 \\
% LLaVA-OV-7B-ZS & & 31.4 \\
Granite Vision & 78.0 & 86.0 \\
\hline
\textbf{+Safety Vectors} & {80.7} & {96.2} \\
\bottomrule
\end{tabular}
\vspace{-3pt}
\caption{\textit{Safety classification} {Results}.}
\label{tab:vlm-judge}
% \vspace{-15pt}
\end{wraptable}
% Results are shown in Table~\ref{tbl:safety}. It can be seen from the table that most MLLMs do not perform well for safety when the task is reformulated as a classification task, potentially because they are designed primarily for generative tasks (e.g., image captioning or open-ended question answering). We note that these models often struggle with general discriminative vision-and-language tasks, and thus reformulating safety tasks as classification tasks poses a major challenge for these models, which may result in a low performance. On the other hand, our approach for finding SVs seems to be more effective, probably due to the fact that this approach can utilize multimodal features for downstream discriminative tasks. Overall, we believe that framing safety evaluation as a classification task may offer a valuable framework for improving the safety of AI models, and we anticipate that future research on this aspect will provide further advancements in safety research.
%Dhiraj - table references are mixed up
Results are shown in Table~\ref{tab:vlm-judge}. %Table~\ref{tbl:safety}.
It can be observed from the table that Granite Vision performs relatively well in terms of safety when the task is reformulated as a classification task. This is somewhat surprising, as MLLMs typically struggle with discriminative vision-and-language tasks~\citep{mitra2024sparse}, and safety classification poses a significant challenge. Additionally, our approach for finding SVs seems to be more effective, probably due to the fact that this approach can utilize multimodal features for downstream discriminative tasks. Overall, we believe that framing safety evaluation as a classification task may offer a valuable framework for improving the safety of AI models, and we anticipate that future research on this aspect will provide further advancements in safety research.
% We note that MLLMs often struggle with general discriminative vision-and-language tasks, and thus reformulating safety tasks as classification tasks poses a major challenge for these models, which may result in a low performance.
% We note that while both Granite Vision and its combination with SVs perform well for several safety classification benchmarks, VLGuard contains a set of words and phrases that do not always have malicious intent. As a result, to further test our model, we have provided an additional experiment on VLGuard that uses the approach of VLM-as-a-Judge. In particular, we utilize GPT-4V as a judge in the same manner as described in the RTVLM evaluation procedure~\citep{Li2024RedTV}. We note that we evaluate the image-instruction pairs with their original categories: unsafe image and unsafe instruction, unsafe image and safe instructions.
% We highlight that evaluation can
%\paragraph{Future Directions.} In this section, we review the safety of our Granite Vision model, as ensuring the safety of generative MLLMs is crucial in order to prevent harm, build trust, address ethical concerns, and enable their responsible deployment in real-world applications. Despite its generally strong performance, Granite Vision has limitations in high-level reasoning and can sometimes produce ungrounded outputs, making it potentially unreliable in sensitive tasks, which requires nuanced classification. To address these issues, we will incorporate more reasoning-focused and structured-related data into the training process in the future.
%In addition, we believe that finding SVs in Granite Vision's attention heads could lead to significant improvements when reformulated safety tasks as classification problems. However, the current reliance on few-shot samples presents a limitation as these demonstrations, while informative, are inherently limited in their scope and may not cover the full range of potential safety issues. This could lead to gaps in the model's ability to identify and address safety concerns. To address this limitation, we plan to investigate scaling up SVs using more training data in future research.
%Dhiraj Joshi rewrote the above 2 paragraphs below - original version above - please review Roi, Shafiq, and Rogerio
%\paragraph{Future Directions.}
\paragraph{Summary and Future Directions.} Ensuring the safety of generative MLLMs is absolutely crucial in order to prevent harm, build trust, address ethical concerns, and enable their responsible deployment in real-world applications. Our results demonstrate that Granite Vision performs almost at par with baselines (despite being the lightest MLLM in the comparison pool) for {\it VLM-as-a-Judge} task. Notably, the addition of \textbf{Safety Vectors} to Granite Vision leads to a significant improvement in safety classification performance. We do acknowledge that further work needs to be done to improve high-level reasoning and correct occasional incorrect outputs to improve reliability in sensitive tasks, which require nuanced classification. To address these, we will incorporate more reasoning-focused and structure-related data into the training process in the future.
In addition, we showed in this paper that finding safety vectors (SVs) in Granite Vision's attention heads led to significant improvements when safety tasks were reformulated as classification problems. Current reliance for SVs is on few-shot samples which are informative but may have limited scope in terms of capturing the range of possible safety issues that can be encountered. To further improve the model's ability to identify and address all safety concerns, we plan to investigate scaling up SVs using more training data in future research.
%Finally, our Granite Vision occasionally fails ...
% \input{Tables/safety-table}
\section{Conclusion}
In this paper, we presented Granite Vision, a compact large language model with integrated vision capabilities, tailored to meet the unique demands of enterprise use cases, in particular visual document understanding. Our model achieves strong results in standard benchmarks and is publicly available under a permissive license for both research and commercial use. Our future work includes enabling multi-page document processing through context compression, learning with structured inputs and outputs, multi-hop reasoning, steering model activations for enhanced reasoning capabilities, and exploring techniques to preserve Granite Vision language-only capabilities.
\bibliography{references}
\bibliographystyle{iclr2025_conference}
\clearpage
%\vspace{4cm}
%\appendix
\section{Appendix}
%You may include other additional sections here.
%\begin{center}
% \hline
% \begin{tabular}{l@{\hspace{6cm}} l@{\hspace{1.8cm}} l@{\hspace{1cm}}}
% {\bf Datasets} & {\bf \# Images} & {\bf \# QA Pairs}\\
% \end{tabular}
% \hline
% \hline
% %\textbf{\fbox{Document QA}}
% %\begin{center}
% {\textbf{Document QA}}
% \vspace{1mm}
% %\end{center}
% \hline
% \begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \textcolor{Cyan40}{\bf DocFM-V$^2$} & 2.4 M & 19.9 M \\
% \textcolor{Purple50}{\bf Docmatix} \citep{laurençon2024building} & 565.0 K & 3.9 M\\
% \textcolor{Purple50}{\bf synthdog-en} \citep{kim2022ocr} & 500.0 K & 500.0 K \\
% \textcolor{Purple50}{\bf pixmo-docs} \citep{deitke2024molmo} & 255.4 K & 2.3 M \\
% \textcolor{Purple50}{\bf HME100K} \citep{yuan2022syntax} & 74.5 K & 74.5 K \\
% \textcolor{Purple50}{\bf Ureader QA} \citep{ye2023ureader} & 72.7 K & 370 K \\
% \textcolor{Purple50}{\bf ArxivQA} \citep{li2024multimodal} & 54.4 K & 100 K \\
% \textcolor{Purple50}{\bf Ureader KG} \citep{ye2023ureader} & 37.6 K & 37.6 K \\
% \textcolor{Cyan40}{\bf Tech Book QA} & 10.4 K & 17.8 K \\
% \textcolor{Purple50}{\bf DocVQA} \citep{mathew2021docvqa} & 10.2 K & 39.5 K \\
% \textcolor{Purple50}{\bf IAM} \citep{marti2002iam} & 5.7 K & 5.7 K \\
% \textcolor{Cyan40}{\bf Business Document QA} & 3.3 K & 7.8 K \\
% \textcolor{Green40}{\bf Visualmrc} \citep{tanaka2021visualmrc} & 3.0 K & 12 K \\
% %\textcolor{Cyan40}{\bf Scientific Document QA} & TBD & TBD\\
% \end{tabular}
% \hline
% \vspace{0.1 cm}
% %\begin{center}
% {\textbf{Chart QA}}
% \vspace{1mm}
% %\end{center}
% \hline
% \begin{tabular}{l@{\hspace{2cm}} l@{\hspace{2cm}} l}
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf Unichart} \citep{masry2023unichart} & 611.0 K & 6.9 M\\
% \textcolor{Purple50}{\bf Tinychart} \citep{zhang2024tinychart} & 606.0 K & 4.5 M\\
% \textcolor{Cyan40}{\bf DocFM-V$^2$-Charts} & 548.7 K & 5.5 M \\
% \textcolor{Purple50}{\bf Dvqa} \citep{kafle2018dvqa} & 200.0 K & 2.3 M \\
% \textcolor{Green40}{\bf PlotQA} \citep{methani2020plotqa} & 157.0 K & 20.2 M\\
% \textcolor{Green40}{\bf FigureQA} \citep{kahou2017figureqa} & 100.0 K & 1.3 M\\
% \textcolor{Cyan40}{\bf DocFM-V$^2$-AugChartQA} & 35.1 K & 294.2 K \\
% \textcolor{Purple50}{\bf Chart2Text} \citep{obeid2020chart} & 27.0 K & 30.0 K\\
% \textcolor{Green40}{\bf ChartQA} \citep{masry2022chartqa} & 18.3 K & 28.3 K \\
% \textcolor{Purple50}{\bf Vistext} \citep{tang2023vistext} & 9.9 K & 9.9 K \\
% \end{tabular}
% \hline
% \vspace{0.1 cm}
% %\textbf{\fbox{Document QA}}
% %\begin{center}
% {\textbf{Infographics QA}}
% \vspace{1mm}
% %\end{center}
% \hline
% \begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf Infographic VQA} \citep{mathew2022infographicvqa} & 8.5 K & 35.7 K \\
% \end{tabular}
% \hline
% \vspace{0.1 cm}
% %\begin{center}
% {\textbf{Table QA}}
% \vspace{1mm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf RoBUT-wikiSQL} \citep{zhao2023robut} & 75.0 K & 86.2 K \\
% \textcolor{Purple50}{\bf RoBUT-WTQ} \citep{zhao2023robut} & 38.0 K & 44.1 K \\
% \textcolor{Purple50}{\bf tabmwp} \citep{lu2022dynamic} & 22.7 K & 23 K \\
% \textcolor{Purple50}{\bf RoBUT-SQA} \citep{zhao2023robut} & 8.5 K & 34.1 K \\
% \textcolor{Purple50}{\bf MultiHiertt} \citep{zhao2022multihiertt} & 7.6 K & 7.8 K \\
% \textcolor{Purple50}{\bf Finqa} \citep{chen2021finqa} & 5.3 K & 6.25 K \\
% \textcolor{Purple50}{\bf Hitab} \citep{cheng2022hitab} & 2.5 K & 7.8 K \\
% \end{tabular}
% \vspace{0.1 cm}
% \hline
% %\begin{center}
% {\textbf{Diagram QA}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{3cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf TQA} \citep{kembhavi2017you} & 27.3 K & 29.8 K \\
% \textcolor{Cyan40}{\bf DocFM-V$^2$-Flowcharts} & 16.6 K & 74.6 K \\
% \textcolor{Purple50}{\bf AI2D GPT4V} \citep{Li2024LLaVAOneVisionEV} & 4.9 K & 4.9 K \\
% \textcolor{Purple50}{\bf AI2D InternVL} \citep{Li2024LLaVAOneVisionEV} & 3.2 K & 12.4 K \\
% \textcolor{Purple50}{\bf AI2D} \citep{kembhavi2016diagram} & 2.4 K & 7.5 K \\
% \textcolor{Purple50}{\bf Diagram Image2Text}\footnotemark[7] & 0.3 K & 0.3 K \\
% \end{tabular}
% \footnotetext[7]{\url{https://huggingface.co/datasets/Kamizuru00/diagram_image_to_text}}
% \clearpage
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \hline
% \begin{tabular}{l@{\hspace{6cm}} l@{\hspace{1.8cm}} l}
% {\bf Datasets} & {\bf \# Images} & {\bf \# QA Pairs}\\
% \end{tabular}
% \hline
% \hline
% \vspace{0.1 cm}
% %\begin{center}
% {\textbf{Reasoning / Captioning}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2.2cm}} l@{\hspace{2cm}} l}
% \textcolor{Cyan40}{\bf DocFM Visual Cue \& Captioning} & 1.2 M & 2.0 M \\
% \textcolor{Purple50}{\bf Textcaps} \citep{sidorov2020textcaps} & 22.0 K & 22.0 K \\
% \textcolor{Purple50}{\bf DocReason}\footnotemark[8] & 8.7 K & 25.8 K \\
% \end{tabular}
% \footnotetext[8]{\url{https://huggingface.co/datasets/mPLUG/DocReason25K}}
% \vspace{0.1 cm}
% \hline
% %\begin{center}
% {\textbf{Grounding / Text Localization}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2.4cm}} l@{\hspace{2cm}} l}
% \textcolor{Green40}{\bf Multi-grained Text Localization} \citep{hu2024mplug} & 1.0 M & 1.0 M \\
% \textcolor{Cyan40}{\bf DocFM Rule Based Grounding} & 190 K & TBD \\
% \textcolor{Cyan40}{\bf KVP10K} \citep{naparstek2024kvp10k} & 8.0 K & 8.0 K \\
% \end{tabular}
% \hline
% \vspace{0.1 cm}
% %\begin{center}
% {\textbf{Scene Text}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2.5cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf K12 Printing} \citep{li2024llava} & 256.6 K & 256.6 K \\
% \textcolor{Purple50}{\bf OCR-VQA} \citep{mishra2019ocr} & 165.7 K & 801.6 K \\
% \textcolor{Purple50}{\bf TextOCR} \citep{singh2021textocr} & 25.1 K & 25.1 K \\
% \textcolor{Purple50}{\bf Ureader CAP} \citep{ye2023ureader} & 22.0 K & 91.4 K \\
% \textcolor{Purple50}{\bf TextVQA} \citep{singh2019towards} & 22.0 K & 34.6 K \\
% \textcolor{Purple50}{\bf Llavar} \citep{zhang2023llavar} & 19.8 K & 43.2 K \\
% \textcolor{Purple50}{\bf ST-VQA} \citep{biten2019scene} & 16.4 K & 22 K \\
% \textcolor{Purple50}{\bf RenderedText}\footnotemark[9] & 10.0 K & 10.0 K \\
% \textcolor{Purple50}{\bf IIIT5K} \citep{mishra2012scene} & 2.0 K & 2.0 K \\
% \end{tabular}
% \footnotetext[9]{\url{https://huggingface.co/datasets/wendlerc/RenderedText}}
% \hline
% \vspace{0.1 cm}
% %\begin{center}
% {\textbf{UI Screen / Code}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2.4cm}} l@{\hspace{2cm}} l}
% \textcolor{Purple50}{\bf Datikz-v2}\footnotemark[10] & 94.4 K & 94.4 K \\
% \textcolor{Purple50}{\bf Datikz} \citep{belouadi2023automatikz} & 47.9 K & 48.2 K \\
% \textcolor{Purple50}{\bf screenqa} \citep{hsiao2022screenqa} & 33.2 K & 80.8 K \\
% \textcolor{Purple50}{\bf screen2words} \citep{wang2021screen2words} & 15.7 K & 15.7 K \\
% \textcolor{Purple50}{\bf websight} \citep{laurenccon2024unlocking} & 10.0 K & 10.0 K \\
% \textcolor{Purple50}{\bf Omniact} \citep{kapoor2024omniact} & 4.9 K & 0.13 K \\
% \end{tabular}
% \footnotetext[10]{\url{https://huggingface.co/datasets/nllg/datikz-v2}}
% \vspace{0.1 cm}
% \hline
% %\begin{center}
% {\textbf{Structure Parsing}}
% \vspace{0.1 cm}
% %\end{center}
% \hline
% %\begin{tabular}{l@{\hspace{1.5cm}} l@{\hspace{2cm}} l}
% \begin{tabular}{l@{\hspace{2 cm}} l@{\hspace{2cm}} l}
% \textcolor{Green40}{\bf OCR-CC} \citep{yang2021tap} & 1.0 M & 1.0 M\\
% \textcolor{Green40}{\bf CCpdf} \citep{turski2023ccpdf} & 937.8 K & 937.8 K\\
% \textcolor{Purple50}{\bf PubTables} \cite{smock2022pubtables} & 569.3 K & 569.3 K \\
% \textcolor{Cyan40}{\bf Chart2Table} & 550.4 K & 550.4 K \\
% \textcolor{Green40}{\bf TURL} \citep{deng2022turl} & 200.0 K & 200.0 K\\
% %\textcolor{Teal20}{\parbox{5 cm}{\bf Text Localization} \citep{hu2024mplug}} & DocStruct 4M & TBD & TBD\\
% \textcolor{Green40}{\bf PubTabNet} \citep{zhong2020image} & 199.8 K & 199.8 K \\
% \textcolor{Green40}{\bf RVL-CDIP} \citep{harley2015evaluation} & 159.4 K & 159.4 K\\
% \textcolor{Green40}{\bf DUE} \citep{borchmann2021due} & 104.5 K & 1.6 M\\
% \textcolor{Cyan40}{\bf FinTabNet} \citep{zheng2020global} & 98.9 K & 98.9 K \\
% \end{tabular}
% \hline
%%%%%
\begin{longtable}{@{} l r r @{}}
\caption{A comprehensive overview of document understanding datasets used in Granite Vision.} \label{tab:doc_datasets}\\
\toprule
\textbf{Datasets} & \textbf{\# Images} & \textbf{\# QA Pairs} \\
\midrule
\endfirsthead
\multicolumn{3}{c}{\tablename\ \thetable\ -- \textit{Continued from previous page}} \\
\toprule
\textbf{Datasets} & \textbf{\# Images} & \textbf{\# QA Pairs} \\
\midrule
\endhead
\midrule \multicolumn{3}{r}{\textit{Continued on next page}} \\
\endfoot
\bottomrule
\endlastfoot
\multicolumn{3}{@{}l}{\textbf{Document QA}} \\
\midrule
\textcolor{Cyan40}{DocFM-VQA} & 2.4 M & 19.9 M \\
\textcolor{Purple50}{Docmatix} \citep{laurençon2024building} & 565.0 K & 3.9 M \\
\textcolor{Purple50}{synthdog-en} \citep{kim2022ocr} & 500.0 K & 500.0 K \\
\textcolor{Purple50}{pixmo-docs} \citep{deitke2024molmo} & 255.4 K & 2.3 M \\
\textcolor{Purple50}{HME100K} \citep{yuan2022syntax} & 74.5 K & 74.5 K \\
\textcolor{Purple50}{Ureader QA} \citep{ye2023ureader} & 72.7 K & 370 K \\
\textcolor{Purple50}{ArxivQA} \citep{li2024multimodal} & 54.4 K & 100 K \\
\textcolor{Purple50}{Ureader KG} \citep{ye2023ureader} & 37.6 K & 37.6 K \\
\textcolor{Cyan40}{Tech Book QA} & 10.4 K & 17.8 K \\
\textcolor{Purple50}{DocVQA} \citep{mathew2021docvqa} & 10.2 K & 39.5 K \\
\textcolor{Purple50}{IAM} \citep{marti2002iam} & 5.7 K & 5.7 K \\
\textcolor{Cyan40}{Business Document QA} & 3.3 K & 7.8 K \\
\textcolor{Green40}{Visualmrc} \citep{tanaka2021visualmrc} & 3.0 K & 12 K \\
%\textcolor{Cyan40}{Scientific Document QA} & TBD & TBD \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Chart QA}} \\
\midrule
\textcolor{Purple50}{Unichart} \citep{masry2023unichart} & 611.0 K & 6.9 M \\
\textcolor{Purple50}{Tinychart} \citep{zhang2024tinychart} & 606.0 K & 4.5 M \\
\textcolor{Cyan40}{DocFM-VQA-Charts} & 548.7 K & 5.5 M \\
\textcolor{Purple50}{Dvqa} \citep{kafle2018dvqa} & 200.0 K & 2.3 M \\
\textcolor{Green40}{PlotQA} \citep{methani2020plotqa} & 157.0 K & 20.2 M \\
\textcolor{Green40}{FigureQA} \citep{kahou2017figureqa} & 100.0 K & 1.3 M \\
\textcolor{Cyan40}{DocFM-VQA-AugChartQA} & 35.1 K & 294.2 K \\
\textcolor{Purple50}{Chart2Text} \citep{obeid2020chart} & 27.0 K & 30.0 K \\
\textcolor{Green40}{ChartQA} \citep{masry2022chartqa} & 18.3 K & 28.3 K \\
\textcolor{Purple50}{Vistext} \citep{tang2023vistext} & 9.9 K & 9.9 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Infographics QA}} \\
\midrule
\textcolor{Purple50}{Infographic VQA} \citep{mathew2022infographicvqa} & 8.5 K & 35.7 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Table QA}} \\
\midrule
\textcolor{Purple50}{RoBUT-wikiSQL} \citep{zhao2023robut} & 75.0 K & 86.2 K \\
\textcolor{Purple50}{RoBUT-WTQ} \citep{zhao2023robut} & 38.0 K & 44.1 K \\
\textcolor{Purple50}{tabmwp} \citep{lu2022dynamic} & 22.7 K & 23 K \\
\textcolor{Purple50}{RoBUT-SQA} \citep{zhao2023robut} & 8.5 K & 34.1 K \\
\textcolor{Purple50}{MultiHiertt} \citep{zhao2022multihiertt} & 7.6 K & 7.8 K \\
\textcolor{Purple50}{Finqa} \citep{chen2021finqa} & 5.3 K & 6.25 K \\
\textcolor{Purple50}{Hitab} \citep{cheng2022hitab} & 2.5 K & 7.8 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Diagram QA}} \\
\midrule
\textcolor{Purple50}{TQA} \citep{kembhavi2017you} & 27.3 K & 29.8 K \\
\textcolor{Cyan40}{DocFM-VQA-Flowcharts} & 16.6 K & 74.6 K \\
\textcolor{Purple50}{AI2D GPT4V} \citep{Li2024LLaVAOneVisionEV} & 4.9 K & 4.9 K \\
\textcolor{Purple50}{AI2D InternVL} \citep{Li2024LLaVAOneVisionEV} & 3.2 K & 12.4 K \\
\textcolor{Purple50}{AI2D} \citep{kembhavi2016diagram} & 2.4 K & 7.5 K \\
\textcolor{Purple50}{Diagram Image2Text}\footnotemark[7] & 0.3 K & 0.3 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Reasoning / Captioning}} \\
\midrule
\textcolor{Cyan40}{DocFM Visual Cue \& Captioning} & 1.2 M & 2.1 M \\
\textcolor{Purple50}{Textcaps} \citep{sidorov2020textcaps} & 22.0 K & 22.0 K \\
\textcolor{Purple50}{DocReason}\footnotemark[8] & 8.7 K & 25.8 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Grounding / Text Localization}} \\
\midrule
\textcolor{Green40}{Multi-grained Text Localization} \citep{hu2024mplug} & 1.0 M & 1.0 M \\
\textcolor{Cyan40}{DocFM Rule Based Grounding} & 190 K & 534.4 K \\
\textcolor{Cyan40}{KVP10K} \citep{naparstek2024kvp10k} & 8.0 K & 8.0 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Scene Text}} \\
\midrule
\textcolor{Purple50}{K12 Printing} \citep{li2024llava} & 256.6 K & 256.6 K \\
\textcolor{Purple50}{OCR-VQA} \citep{mishra2019ocr} & 165.7 K & 801.6 K \\
\textcolor{Purple50}{TextOCR} \citep{singh2021textocr} & 25.1 K & 25.1 K \\
\textcolor{Purple50}{Ureader CAP} \citep{ye2023ureader} & 22.0 K & 91.4 K \\
\textcolor{Purple50}{TextVQA} \citep{singh2019towards} & 22.0 K & 34.6 K \\
\textcolor{Purple50}{Llavar} \citep{zhang2023llavar} & 19.8 K & 43.2 K \\
\textcolor{Purple50}{ST-VQA} \citep{biten2019scene} & 16.4 K & 22 K \\
\textcolor{Purple50}{RenderedText}\footnotemark[9] & 10.0 K & 10.0 K \\
\textcolor{Purple50}{IIIT5K} \citep{mishra2012scene} & 2.0 K & 2.0 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{UI Screen / Code}} \\
\midrule
\textcolor{Purple50}{Datikz-v2}\footnotemark[10] & 94.4 K & 94.4 K \\
\textcolor{Purple50}{Datikz} \citep{belouadi2023automatikz} & 47.9 K & 48.2 K \\
\textcolor{Purple50}{screenqa} \citep{hsiao2022screenqa} & 33.2 K & 80.8 K \\
\textcolor{Purple50}{screen2words} \citep{wang2021screen2words} & 15.7 K & 15.7 K \\
\textcolor{Purple50}{websight} \citep{laurenccon2024unlocking} & 10.0 K & 10.0 K \\
\textcolor{Purple50}{Omniact} \citep{kapoor2024omniact} & 4.9 K & 0.13 K \\
\midrule
\multicolumn{3}{@{}l}{\textbf{Structure Parsing}} \\
\midrule
\textcolor{Green40}{OCR-CC} \citep{yang2021tap} & 1.0 M & 1.0 M \\
\textcolor{Green40}{CCpdf} \citep{turski2023ccpdf} & 937.8 K & 937.8 K \\
\textcolor{Purple50}{PubTables} \citep{smock2022pubtables} & 569.3 K & 569.3 K \\
\textcolor{Cyan40}{Chart2Table} & 550.4 K & 550.4 K \\
\textcolor{Green40}{TURL} \citep{deng2022turl} & 200.0 K & 200.0 K \\
\textcolor{Cyan40}{PubTabNet} \citep{zhong2020image} & 199.8 K & 199.8 K \\
\textcolor{Green40}{RVL-CDIP} \citep{harley2015evaluation} & 159.4 K & 159.4 K \\
\textcolor{Green40}{DUE} \citep{borchmann2021due} & 104.5 K & 1.6 M \\
\textcolor{Cyan40}{FinTabNet} \citep{zheng2020global} & 98.9 K & 98.9 K \\
\end{longtable}
% Footnotes for the table entries:
\footnotetext[7]{\url{https://huggingface.co/datasets/Kamizuru00/diagram_image_to_text}}
\footnotetext[8]{\url{https://huggingface.co/datasets/mPLUG/DocReason25K}}
\footnotetext[9]{\url{https://huggingface.co/datasets/wendlerc/RenderedText}}
\footnotetext[10]{\url{https://huggingface.co/datasets/nllg/datikz-v2}}
\newpage
\section{Contributions And Acknowledgments}
Authors (alphabetical order):
\vspace{0.15in}
{\bf Granite Vision Technical Leadership:}
Assaf Arbelle,
Leonid Karlinsky,
Peter Staar,
Rogerio Feris,
Tal Drory
\vspace{0.15in}
{\bf Project Management:}
Abraham Daniels
\vspace{0.15in}
{\bf Core Contributors:}
Ahmed Nassar,
Amit Alfassi,
Bo Wu,
Eli Schwartz,
Dhiraj Joshi,
Jovana Kondic,
Nimrod Shabtay,
Pengyuan Li,
Roei Herzig,
Shafiq Abedin,
Shaked Perek,
Sivan Harary,
Udi Barzelay
\vspace{0.15in}
{\bf Contributors:}
Adi Raz Goldfarb,
Aude Oliva,
Ben Wieles,
Bishwaranjan Bhattacharjee,
Brandon Huang,
Christoph Auer,
Dan Gutfreund,
David Beymer,
David Wood,
Hilde Kuehne,
Jacob Hansen,
Joseph Shtok,
Ken Wong,
Luis Angel Bathen,
Mayank Mishra,
Maksym Lysak,
Michele Dolfi,
Mikhail Yurochkin,
Nikolaos Livathinos,
Nimrod Harel,
Ophir Azulai,
Oshri Naparstek,
Rafael Teixeira de Lima,
Rameswar Panda,
Sivan Doveh,
Shubham Gupta,
Subhro Das,
Syed Zawad,
Yusik Kim,
Zexue He
\vspace{0.15in}
{\bf Platform: }
Alexander Brooks,
Gabe Goodhart
\vspace{0.15in}
{\bf Product Management:}
Anita Govindjee,
Derek Leist,
Ibrahim Ibrahim
\vspace{0.15in}
{\bf IBM Research Leadership:}
Aya Soffer,
David Cox,
Kate Soule,
Luis Lastras,
Nirmit Desai,
Shila Ofek-koifman,
Sriram Raghavan,
Tanveer Syeda-Mahmood
\vspace{0.15in}
{\bf Acknowledgments:}
We would like to acknowledge Amy Angelini, Bob Calio, Brian Belgodere, Dakshi Agrawal, Heiko Ludwig, Hendrik Strobelt, John Smith, Kim Martineau, Kush Varshney, Mauro Martino, Petros Zerfos, Ray Rose, and Sandeep Gopisetty. We would also like to acknowledge the support from IBM Research AI and the MIT-IBM Watson AI lab.
\end{document}
\newcolumntype{N}{>{\centering\arraybackslash}p{1cm}} % Narrow centered column (1cm width)
\newcolumntype{C}{>{\centering\arraybackslash}X} % Flexible centered column
\renewcommand{\arraystretch}{1.0}
\setlength{\tabcolsep}{3pt}
\begin{table*}[t]
\centering
\small
\begin{tabularx}{\textwidth}{
>{\raggedright\arraybackslash}p{2.9cm}
*{9}{>{\centering\arraybackslash}X}
>{\centering\arraybackslash}p{1.5cm}
}
\toprule
% \multicolumn{1}{c}{} &
% \multicolumn{2}{c}{\textbf{Safety}} &
% \multicolumn{4}{c}{\textbf{VQA}} &
% \multicolumn{2}{c}{\textbf{Image Cls.}} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-7} \cmidrule(lr){8-9}
\textbf{Model} &
% \multicolumn{1}{c}{\textbf{MHalu}} &
\multicolumn{2}{c}{\textbf{VLGuard}} &
% \multicolumn{1}{c}{\textbf{BLINK}} &
\multicolumn{4}{c}{\textbf{RTVLM}}
% \textbf{EuroSAT} &
% \textbf{Pets}
\\
\cmidrule(lr){2-3}
\cmidrule(lr){4-7}
& \multicolumn{1}{c}{Unsafe} & \multicolumn{1}{c}{Safe-Unsafe} &
\multicolumn{1}{c}{Mislead} & \multicolumn{1}{c}{Politics} & \multicolumn{1}{c}{Racial} & \multicolumn{1}{c}{Jailbreak}
\\
\toprule
% % \rowcolor{gray!20}CLIP & - & - & - & - & - & - & 64.0 & 88.1 \\
% % \rowcolor{gray!20}SigLip & - & - & - & - & - & - & 63.9 & 98.3 \\
% % \rowcolor{gray!20}LLaVA-1.5-ZS & 44.3 & 36.7 & 38.0 & 37.7 & 43.8 & 12.7 & 26.3 & 43.0 \\
LLaVA-v1.5-7B & {5.3} & {7.4} & {8.6} & {7.3} & {7.2} & {4.4} \\
Phi3.5-vision & {8.7} & {9.3} & {8.5} & {8.2} & {8.2} & {9.3} \\
SmolVLM & {4.8} & {7.7} & {5.6} & {6.0} & {4.6} & {7.3} \\
% InternVL2-4B & 7.7 & 8.9 & 8.5 & 7.8 & 7.9 & 8.2 \\
% DeepSeek-VL2 & 6.4 & {8.7} & 8.4 & {7.3} & {7.7} & 3.4 \\
% \hline
% Qwen2-VL-7B-ZS & 24.0 & 26.9 & \underline{55.7} & 5.6 & 36.3 & 29.3 \\
% \textbf{+SVs} & \textbf{85.1} & \textbf{96.0} & \textbf{-} & \textbf{-} & \textbf{-} & \textbf{-} \\
% LLaVA-OV-7B-ZS & 34.7 & 31.4 & 34.9 & 3.1 & 9.4 & 38.8 \\
% +LoRA & 44.9 & 43.8 & \underline{47.0} & \underline{58.6} & \underline{60.9} & \underline{32.4} & \underline{85.0} & \underline{96.8} \\
% \textbf{+SVs} & \textbf{80.8} & \textbf{94.3} & - & - & - & - \\
\hline
% Granite Vision+VLM-Judge & - & - & - & - & - & - \\
Granite Vision (Ours) & 6.5 & 8.6 & {7.7} & {7.2} & {7.7} & {4.5} \\
% +LoRA & \underline{84.8} & 87.7 & \underline{46.3} & \underline{55.3} & \underline{57.4} & 28.8 & \underline{72.9} & \textbf{98.4} \\
% \textbf{+Safety Vectors} & \textbf{80.7} & \textbf{96.7} & \textbf{98.1} & \textbf{100.0} & \textbf{100.0} & \textbf{100} \\
% GV+VLM-Judge & - & 91.4 & - & - & - & - \\
\bottomrule
\end{tabularx}
\caption{\textbf{Results} for the \textit{VLM-as-a-Judge} setup. We evaluate on the VLGuard and RTVLM benchmarks. The score has a range between [0,10], and higher is better.
% We \textcolor{gray}{gray out} discriminative VLM model results, which cannot be evaluating directly on tasks with interleaved image-text queries.
}
\label{tbl:safety}
\end{table*}
\begin{thebibliography}{109}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abdin et~al.(2024{\natexlab{a}})Abdin, Aneja, Awadalla, Awadallah, Awan, Bach, Bahree, Bakhtiari, Bao, Behl, Benhaim, Bilenko, Bjorck, Bubeck, Cai, Cai, Chaudhary, Chen, Chen, Chen, Chen, Chen, Cheng, Chopra, Dai, Dixon, Eldan, Fragoso, Gao, Gao, Gao, Garg, Giorno, Goswami, Gunasekar, Haider, Hao, Hewett, Hu, Huynh, Iter, Jacobs, Javaheripi, Jin, Karampatziakis, Kauffmann, Khademi, Kim, Kim, Kurilenko, Lee, Lee, Li, Li, Liang, Liden, Lin, Lin, Liu, Liu, Liu, Liu, Liu, Luo, Madan, Mahmoudzadeh, Majercak, Mazzola, Mendes, Mitra, Modi, Nguyen, Norick, Patra, Perez-Becker, Portet, Pryzant, Qin, Radmilac, Ren, de~Rosa, Rosset, Roy, Ruwase, Saarikivi, Saied, Salim, Santacroce, Shah, Shang, Sharma, Shen, Shukla, Song, Tanaka, Tupini, Vaddamanu, Wang, Wang, Wang, Wang, Wang, Wang, Ward, Wen, Witte, Wu, Wu, Wyatt, Xiao, Xu, Xu, Xu, Xue, Yadav, Yang, Yang, Yang, Yang, Yu, Yuan, Zhang, Zhang, Zhang, Zhang, Zhang, Zhang, Zhang, and Zhou]{abdin2024phi3technicalreporthighly}
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar~Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie~Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell~J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam~Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young~Jin Kim, Lev Kurilenko, James~R. Lee, Yin~Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce~Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César~Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen,
Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de~Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu~Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang, Donghan Yu, Lu~Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li~Lyna Zhang, Yi~Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou.
\newblock Phi-3 technical report: A highly capable language model locally on your phone, 2024{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2404.14219}.
\bibitem[Abdin et~al.(2024{\natexlab{b}})Abdin, Aneja, Awadalla, Awadallah, Awan, Bach, Bahree, Bakhtiari, Bao, Behl, et~al.]{abdin2024phi}
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar~Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et~al.
\newblock Phi-3 technical report: A highly capable language model locally on your phone.
\newblock \emph{arXiv preprint arXiv:2404.14219}, 2024{\natexlab{b}}.
\bibitem[Agrawal et~al.(2024)Agrawal, Antoniak, Hanna, Bout, Chaplot, Chudnovsky, Costa, Monicault, Garg, Gervet, Ghosh, Héliou, Jacob, Jiang, Khandelwal, Lacroix, Lample, Casas, Lavril, Scao, Lo, Marshall, Martin, Mensch, Muddireddy, Nemychnikova, Pellat, Platen, Raghuraman, Rozière, Sablayrolles, Saulnier, Sauvestre, Shang, Soletskyi, Stewart, Stock, Studnia, Subramanian, Vaze, Wang, and Yang]{agrawal2024pixtral}
Pravesh Agrawal, Szymon Antoniak, Emma~Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin~De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert~Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego~Las Casas, Thibaut Lavril, Teven~Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick~Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang.
\newblock Pixtral 12b, 2024.
\newblock URL \url{https://arxiv.org/abs/2410.07073}.
\bibitem[Alayrac et~al.(2022)Alayrac, Donahue, Luc, Miech, Barr, Hasson, Lenc, Mensch, Millican, Reynolds, Ring, Rutherford, Cabi, Han, Gong, Samangooei, Monteiro, Menick, Borgeaud, Brock, Nematzadeh, Sharifzadeh, Binkowski, Barreira, Vinyals, Zisserman, and Simonyan]{alayrac2022flamingo}
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
\newblock Flamingo: a visual language model for few-shot learning, 2022.
\newblock URL \url{https://arxiv.org/abs/2204.14198}.
\bibitem[Anthropic(2024)]{anthropic_claude3_5}
Anthropic.
\newblock Claude 3.5 sonnet, 2024.
\newblock URL \url{https://www.anthropic.com/news/claude-3-5-sonnet}.
\newblock Accessed: 2025-01-15.
\bibitem[Appalaraju et~al.(2021)Appalaraju, Jasani, Kota, Xie, and Manmatha]{appalaraju2021docformer}
Srikar Appalaraju, Bhavan Jasani, Bhargava~Urala Kota, Yusheng Xie, and R.~Manmatha.
\newblock Docformer: End-to-end transformer for document understanding, 2021.
\newblock URL \url{https://arxiv.org/abs/2106.11539}.
\bibitem[Bai et~al.(2023)Bai, Bai, Yang, Wang, Tan, Wang, Lin, Zhou, and Zhou]{bai2023qwenvl}
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
\newblock Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
\newblock URL \url{https://arxiv.org/abs/2308.12966}.
\bibitem[Belouadi et~al.(2023)Belouadi, Lauscher, and Eger]{belouadi2023automatikz}
Jonas Belouadi, Anne Lauscher, and Steffen Eger.
\newblock Automatikz: Text-guided synthesis of scientific vector graphics with tikz, 2023.
\bibitem[Biten et~al.(2019)Biten, Tito, Mafla, Gomez, Rusinol, Valveny, Jawahar, and Karatzas]{biten2019scene}
Ali~Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mar{\c{c}}al Rusinol, Ernest Valveny, CV~Jawahar, and Dimosthenis Karatzas.
\newblock Scene text visual question answering, 2019.
\bibitem[Borchmann et~al.(2021)Borchmann, Pietruszka, Stanislawek, Jurkiewicz, Turski, Szyndler, and Grali{\'n}ski]{borchmann2021due}
{\L}ukasz Borchmann, Micha{\l} Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Micha{\l} Turski, Karolina Szyndler, and Filip Grali{\'n}ski.
\newblock Due: End-to-end document understanding benchmark, 2021.
\bibitem[Chen et~al.(2024{\natexlab{a}})Chen, Wang, Xue, Zhang, Yang, Li, Shen, Gu, and Chen]{chen2024unified}
Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Jinjie Gu, and Huajun Chen.
\newblock Unified hallucination detection for multimodal large language models, 2024{\natexlab{a}}.
\bibitem[Chen et~al.(2024{\natexlab{b}})Chen, Wu, Wang, Su, Chen, Xing, Zhong, Zhang, Zhu, Lu, et~al.]{chen2024internvl}
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et~al.
\newblock Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 24185--24198, 2024{\natexlab{b}}.
\bibitem[Chen et~al.(2021)Chen, Chen, Smiley, Shah, Borova, Langdon, Moussa, Beane, Huang, Routledge, et~al.]{chen2021finqa}
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et~al.
\newblock Finqa: A dataset of numerical reasoning over financial data, 2021.
\bibitem[Cheng et~al.(2022)Cheng, Dong, Wang, Jia, Guo, Gao, Han, Lou, and Zhang]{cheng2022hitab}
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang.
\newblock Hitab: A hierarchical table dataset for question answering and natural language generation, 2022.
\bibitem[Dai et~al.(2024)Dai, Lee, Wang, Yang, Liu, Barker, Rintamaki, Shoeybi, Catanzaro, and Ping]{dai2024nvlm}
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
\newblock Nvlm: Open frontier-class multimodal llms, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.11402}.
\bibitem[Deitke et~al.(2024{\natexlab{a}})Deitke, Clark, Lee, Tripathi, Yang, Park, Salehi, Muennighoff, Lo, Soldaini, Lu, Anderson, Bransom, Ehsani, Ngo, Chen, Patel, Yatskar, Callison-Burch, Head, Hendrix, Bastani, VanderBilt, Lambert, Chou, Chheda, Sparks, Skjonsberg, Schmitz, Sarnat, Bischoff, Walsh, Newell, Wolters, Gupta, Zeng, Borchardt, Groeneveld, Dumas, Nam, Lebrecht, Wittlif, Schoenick, Michel, Krishna, Weihs, Smith, Hajishirzi, Girshick, Farhadi, and Kembhavi]{molmo}
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae~Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah~A. Smith, Hannaneh Hajishirzi, Ross~B. Girshick, Ali Farhadi, and Aniruddha Kembhavi.
\newblock Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.
\newblock \emph{CoRR}, abs/2409.17146, 2024{\natexlab{a}}.
\newblock URL \url{https://doi.org/10.48550/arXiv.2409.17146}.
\bibitem[Deitke et~al.(2024{\natexlab{b}})Deitke, Clark, Lee, Tripathi, Yang, Park, Salehi, Muennighoff, Lo, Soldaini, et~al.]{deitke2024molmo}
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae~Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et~al.
\newblock Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models, 2024{\natexlab{b}}.
\bibitem[Deng et~al.(2022)Deng, Sun, Lees, Wu, and Yu]{deng2022turl}
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu.
\newblock Turl: Table understanding through representation learning, 2022.
\bibitem[Feng et~al.(2024)Feng, Liu, Liu, Tang, Zhou, Li, and Huang]{feng2024docpedia}
Hao Feng, Qi~Liu, Hao Liu, Jingqun Tang, Wengang Zhou, Houqiang Li, and Can Huang.
\newblock Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding, 2024.
\newblock URL \url{https://arxiv.org/abs/2311.11810}.
\bibitem[Foundation()]{wikidump}
Wikimedia Foundation.
\newblock Wikimedia downloads.
\newblock URL \url{https://dumps.wikimedia.org}.
\bibitem[Goyal et~al.(2017)Goyal, Khot, Summers-Stay, Batra, and Parikh]{goyal2017making}
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
\newblock Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017.
\bibitem[Granite~Team(2024)]{granite2024granite}
IBM Granite~Team.
\newblock Granite 3.0 language models, 2024.
\bibitem[Grattafiori et~al.(2024)Grattafiori, Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, and ...]{grattafiori2024llama3}
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and ...
\newblock The llama 3 herd of models, 2024.
\newblock URL \url{https://arxiv.org/abs/2407.21783}.
\bibitem[Gurari et~al.(2018)Gurari, Li, Stangl, Guo, Lin, Grauman, Luo, and Bigham]{gurari2018vizwiz}
Danna Gurari, Qing Li, Abigale~J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey~P Bigham.
\newblock Vizwiz grand challenge: Answering visual questions from blind people, 2018.
\bibitem[Harley et~al.(2015)Harley, Ufkes, and Derpanis]{harley2015evaluation}
Adam~W Harley, Alex Ufkes, and Konstantinos~G Derpanis.
\newblock Evaluation of deep convolutional nets for document image classification and retrieval, 2015.
\bibitem[Hsiao et~al.(2022)Hsiao, Zubach, Baechler, Carbune, Lin, Wang, Sunkara, Zhu, and Chen]{hsiao2022screenqa}
Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen.
\newblock Screenqa: Large-scale question-answer pairs over mobile app screenshots, 2022.
\bibitem[Hu et~al.(2024{\natexlab{a}})Hu, Xu, Ye, Yan, Zhang, Zhang, Li, Zhang, Jin, Huang, and Zhou]{hu2024mplugdocowl15}
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo~Zhang, Chen Li, Ji~Zhang, Qin Jin, Fei Huang, and Jingren Zhou.
\newblock mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2403.12895}.
\bibitem[Hu et~al.(2024{\natexlab{b}})Hu, Xu, Ye, Yan, Zhang, Zhang, Li, Zhang, Jin, Huang, et~al.]{hu2024mplug}
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo~Zhang, Chen Li, Ji~Zhang, Qin Jin, Fei Huang, et~al.
\newblock mplug-docowl 1.5: Unified structure learning for ocr-free document understanding, 2024{\natexlab{b}}.
\bibitem[Huang et~al.(2024)Huang, Mitra, Arbelle, Karlinsky, Darrell, and Herzig]{Huang2024MultimodalTV}
Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig.
\newblock Multimodal task vectors enable many-shot multimodal in-context learning, 2024.
\bibitem[Huang et~al.(2022)Huang, Lv, Cui, Lu, and Wei]{huang2022layoutlmv3}
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.
\newblock Layoutlmv3: Pre-training for document ai with unified text and image masking, 2022.
\newblock URL \url{https://arxiv.org/abs/2204.08387}.
\bibitem[Hugging-Face(2024)]{smolVLM}
Hugging-Face.
\newblock Smolvlm, 2024.
\newblock URL \url{https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct}.
\bibitem[HuggingFace(2025)]{huggingface_docmatix}
HuggingFace.
\newblock Docmatix: A new approach to document understanding, 2025.
\newblock URL \url{https://huggingface.co/blog/docmatix}.
\newblock Accessed: 2025-01-14.
\bibitem[Kafle et~al.(2018)Kafle, Price, Cohen, and Kanan]{kafle2018dvqa}
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.
\newblock Dvqa: Understanding data visualizations via question answering, 2018.
\bibitem[Kahou et~al.(2017)Kahou, Michalski, Atkinson, K{\'a}d{\'a}r, Trischler, and Bengio]{kahou2017figureqa}
Samira~Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, {\'A}kos K{\'a}d{\'a}r, Adam Trischler, and Yoshua Bengio.
\newblock Figureqa: An annotated figure dataset for visual reasoning, 2017.
\bibitem[Kapoor et~al.(2024)Kapoor, Butala, Russak, Koh, Kamble, AlShikh, and Salakhutdinov]{kapoor2024omniact}
Raghav Kapoor, Yash~Parag Butala, Melisa Russak, Jing~Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov.
\newblock Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web.
\newblock In \emph{European Conference on Computer Vision}, pp.\ 161--178. Springer, 2024.
\bibitem[Kembhavi et~al.(2016)Kembhavi, Salvato, Kolve, Seo, Hajishirzi, and Farhadi]{kembhavi2016diagram}
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.
\newblock A diagram is worth a dozen images, 2016.
\bibitem[Kembhavi et~al.(2017)Kembhavi, Seo, Schwenk, Choi, Farhadi, and Hajishirzi]{kembhavi2017you}
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi.
\newblock Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension, 2017.
\bibitem[Kim et~al.(2022{\natexlab{a}})Kim, Hong, Yim, Nam, Park, Yim, Hwang, Yun, Han, and Park]{kim2022ocr}
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.
\newblock Ocr-free document understanding transformer, 2022{\natexlab{a}}.
\bibitem[Kim et~al.(2022{\natexlab{b}})Kim, Hong, Yim, Nam, Park, Yim, Hwang, Yun, Han, and Park]{kim2022ocrfreedocumentunderstandingtransformer}
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.
\newblock Ocr-free document understanding transformer, 2022{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2111.15664}.
\bibitem[Koster et~al.(2022)Koster, Illyes, Zeller, and Sassman]{koster2022rfc}
M~Koster, G~Illyes, H~Zeller, and L~Sassman.
\newblock Rfc 9309 robots exclusion protocol, 2022.
\bibitem[Lauren{\c{c}}on et~al.(2024)Lauren{\c{c}}on, Tronchon, and Sanh]{laurenccon2024unlocking}
Hugo Lauren{\c{c}}on, L{\'e}o Tronchon, and Victor Sanh.
\newblock Unlocking the conversion of web screenshots into html code with the websight dataset, 2024.
\bibitem[Laurençon et~al.(2024{\natexlab{a}})Laurençon, Marafioti, Sanh, and Tronchon]{laurençon2024building}
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon.
\newblock Building and better understanding vision-language models: insights and future directions., 2024{\natexlab{a}}.
\bibitem[Laurençon et~al.(2024{\natexlab{b}})Laurençon, Tronchon, Cord, and Sanh]{laurencon2024matters}
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.
\newblock What matters when building vision-language models?, 2024{\natexlab{b}}.
\bibitem[Li et~al.(2024{\natexlab{a}})Li, Zhang, Zhang, Pu, Du, Dong, Liu, Zhang, Zhang, Li, and Liu]{lmms_eval2024}
Bo~Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge~Zhang, Chunyuan Li, and Ziwei Liu.
\newblock Lmms-eval: Accelerating the development of large multimodal models, March 2024{\natexlab{a}}.
\newblock URL \url{https://github.com/EvolvingLMMs-Lab/lmms-eval}.
\bibitem[Li et~al.(2024{\natexlab{b}})Li, Zhang, Guo, Zhang, Li, Zhang, Zhang, Li, Liu, and Li]{Li2024LLaVAOneVisionEV}
Bo~Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
\newblock Llava-onevision: Easy visual task transfer, 2024{\natexlab{b}}.
\bibitem[Li et~al.(2024{\natexlab{c}})Li, Zhang, Guo, Zhang, Li, Zhang, Zhang, Zhang, Li, Liu, and Li]{li2024llavaonevision}
Bo~Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li.
\newblock Llava-onevision: Easy visual task transfer, 2024{\natexlab{c}}.
\newblock URL \url{https://arxiv.org/abs/2408.03326}.
\bibitem[Li et~al.(2024{\natexlab{d}})Li, Zhang, Guo, Zhang, Li, Zhang, Zhang, Zhang, Li, Liu, et~al.]{li2024llava}
Bo~Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et~al.
\newblock Llava-onevision: Easy visual task transfer, 2024{\natexlab{d}}.
\bibitem[Li et~al.(2023)Li, Li, Savarese, and Hoi]{li2023blip2}
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
\newblock {BLIP-2:} bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
\bibitem[Li et~al.(2024{\natexlab{e}})Li, Wang, Xu, Wang, Feng, Kong, and Liu]{li2024multimodal}
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi~Liu.
\newblock Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024{\natexlab{e}}.
\bibitem[Li et~al.(2024{\natexlab{f}})Li, Li, Yin, Ahmed, Liu, and Liu]{Li2024RedTV}
Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi~Liu.
\newblock Red teaming visual language models, 2024{\natexlab{f}}.
\newblock URL \url{https://api.semanticscholar.org/CorpusID:267094801}.
\bibitem[Liao et~al.(2024)Liao, Wang, Li, Wang, Huang, and Jin]{liao2024doclayllm}
Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin.
\newblock Doclayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding, 2024.
\newblock URL \url{https://arxiv.org/abs/2408.15045}.
\bibitem[Liu et~al.(2024{\natexlab{a}})Liu, Yin, Cao, Jiang, Li, Liu, Jiang, Sun, and Xu]{liu2024hrvda}
Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu.
\newblock Hrvda: High-resolution visual document assistant, 2024{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2404.06918}.
\bibitem[Liu et~al.(2023{\natexlab{a}})Liu, Eisenschlos, Piccinno, Krichene, Pang, Lee, Joshi, Chen, Collier, and Altun]{liu-2022-deplot}
Fangyu Liu, Julian~Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun.
\newblock Deplot: One-shot visual language reasoning by plot-to-table translation, 2023{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2212.10505}.
\bibitem[Liu et~al.(2023{\natexlab{b}})Liu, Li, Li, and Lee]{liu2023improvedllava}
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong~Jae Lee.
\newblock Improved baselines with visual instruction tuning, 2023{\natexlab{b}}.
\bibitem[Liu et~al.(2023{\natexlab{c}})Liu, Li, Wu, and Lee]{liu2023llava}
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee.
\newblock Visual instruction tuning.
\newblock In \emph{NeurIPS}, 2023{\natexlab{c}}.
\bibitem[Liu et~al.(2024{\natexlab{b}})Liu, Li, Huang, Yang, Yu, Li, Yin, Liu, Jin, and Bai]{liu2024ocrbench}
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai.
\newblock Ocrbench: on the hidden mystery of ocr in large multimodal models, 2024{\natexlab{b}}.
\bibitem[Livathinos et~al.(2025)Livathinos, Auer, Lysak, Nassar, Dolfi, Vagenas, Ramis, Omenetti, Dinkla, Kim, Gupta, de~Lima, Weber, Morin, Meijer, Kuropiatnyk, and Staar]{DoclingTechReport}
Nikolaos Livathinos, Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Panos Vagenas, Cesar~Berrospi Ramis, Matteo Omenetti, Kasper Dinkla, Yusik Kim, Shubham Gupta, Rafael~Teixeira de~Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, and Peter W.~J. Staar.
\newblock Docling: An efficient open-source toolkit for ai-driven document conversion, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.17887}.
\bibitem[Lu et~al.(2022)Lu, Qiu, Chang, Wu, Zhu, Rajpurohit, Clark, and Kalyan]{lu2022dynamic}
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying~Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.
\newblock Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2022.
\bibitem[Lysak et~al.(2023)Lysak, Nassar, Livathinos, Auer, and Staar]{otsl}
Maksym Lysak, Ahmed Nassar, Nikolaos Livathinos, Christoph Auer, and Peter Staar.
\newblock Optimized table tokenization for table structure recognition.
\newblock In \emph{Document Analysis and Recognition - ICDAR 2023: 17th International Conference, San Jos\'{e}, CA, USA, August 21–26, 2023, Proceedings, Part II}, pp.\ 37–50, Berlin, Heidelberg, 2023. Springer-Verlag.
\newblock ISBN 978-3-031-41678-1.
\newblock \doi{10.1007/978-3-031-41679-8_3}.
\newblock URL \url{https://doi.org/10.1007/978-3-031-41679-8_3}.
\bibitem[Marino et~al.(2019)Marino, Rastegari, Farhadi, and Mottaghi]{marino2019ok}
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
\newblock Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019.
\bibitem[Marti \& Bunke(2002)Marti and Bunke]{marti2002iam}
U-V Marti and Horst Bunke.
\newblock The iam-database: an english sentence database for offline handwriting recognition, 2002.
\bibitem[Masry et~al.(2022)Masry, Long, Tan, Joty, and Hoque]{masry2022chartqa}
Ahmed Masry, Do~Xuan Long, Jia~Qing Tan, Shafiq Joty, and Enamul Hoque.
\newblock Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022.
\bibitem[Masry et~al.(2023)Masry, Kavehzadeh, Do, Hoque, and Joty]{masry2023unichart}
Ahmed Masry, Parsa Kavehzadeh, Xuan~Long Do, Enamul Hoque, and Shafiq Joty.
\newblock Unichart: A universal vision-language pretrained model for chart comprehension and reasoning, 2023.
\bibitem[Mathew et~al.(2021)Mathew, Karatzas, and Jawahar]{mathew2021docvqa}
Minesh Mathew, Dimosthenis Karatzas, and CV~Jawahar.
\newblock Docvqa: A dataset for vqa on document images.
\newblock In \emph{Proceedings of the IEEE/CVF winter conference on applications of computer vision}, pp.\ 2200--2209, 2021.
\bibitem[Mathew et~al.(2022)Mathew, Bagal, Tito, Karatzas, Valveny, and Jawahar]{mathew2022infographicvqa}
Minesh Mathew, Viraj Bagal, Rub{\`e}n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV~Jawahar.
\newblock Infographicvqa, 2022.
\bibitem[Methani et~al.(2020)Methani, Ganguly, Khapra, and Kumar]{methani2020plotqa}
Nitesh Methani, Pritha Ganguly, Mitesh~M Khapra, and Pratyush Kumar.
\newblock Plotqa: Reasoning over scientific plots, 2020.
\bibitem[Mishra et~al.(2012)Mishra, Alahari, and Jawahar]{mishra2012scene}
Anand Mishra, Karteek Alahari, and CV~Jawahar.
\newblock Scene text recognition using higher order language priors, 2012.
\bibitem[Mishra et~al.(2019)Mishra, Shekhar, Singh, and Chakraborty]{mishra2019ocr}
Anand Mishra, Shashank Shekhar, Ajeet~Kumar Singh, and Anirban Chakraborty.
\newblock Ocr-vqa: Visual question answering by reading text in images, 2019.
\bibitem[Mitra et~al.(2024)Mitra, Huang, Chai, Lin, Arbelle, Feris, Karlinsky, Darrell, Ramanan, and Herzig]{mitra2024sparse}
Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, and Roei Herzig.
\newblock Sparse attention vectors: Generative multimodal model features are discriminative vision-language classifiers, 2024.
\bibitem[Naparstek et~al.(2024)Naparstek, Azulai, Shapira, Amrani, Yaroker, Burshtein, Pony, Rubinstein, Dahood, Prince, et~al.]{naparstek2024kvp10k}
Oshri Naparstek, Ophir Azulai, Inbar Shapira, Elad Amrani, Yevgeny Yaroker, Yevgeny Burshtein, Roi Pony, Nadav Rubinstein, Foad~Abo Dahood, Orit Prince, et~al.
\newblock Kvp10k: A comprehensive dataset for key-value pair extraction in business documents.
\newblock In \emph{International Conference on Document Analysis and Recognition}, pp.\ 97--116. Springer, 2024.
\bibitem[Nassar et~al.(2022)Nassar, Livathinos, Lysak, and Staar]{tableformer}
Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar.
\newblock Tableformer: Table structure understanding with transformers.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, pp.\ 4614--4623, June 2022.
\bibitem[Obeid \& Hoque(2020)Obeid and Hoque]{obeid2020chart}
Jason Obeid and Enamul Hoque.
\newblock Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model, 2020.
\bibitem[OpenAI(2023{\natexlab{a}})]{OpenAI2023GPT4TR}
OpenAI.
\newblock Gpt-4 technical report.
\newblock \emph{ArXiv}, abs/2303.08774, 2023{\natexlab{a}}.
\bibitem[OpenAI(2023{\natexlab{b}})]{openai_gpt4v}
OpenAI.
\newblock Gpt-4v(ision) technical work and authors, 2023{\natexlab{b}}.
\newblock URL \url{https://openai.com/contributions/gpt-4v/}.
\newblock Accessed: 2025-01-15.
\bibitem[OpenAI(2024)]{openai_gpt4o}
OpenAI.
\newblock Hello gpt-4o, 2024.
\newblock URL \url{https://openai.com/index/hello-gpt-4o/}.
\newblock Accessed: 2025-01-15.
\bibitem[Pfitzmann et~al.(2022)Pfitzmann, Auer, Dolfi, Nassar, and Staar]{doclaynet}
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed~S. Nassar, and Peter Staar.
\newblock Doclaynet: A large human-annotated dataset for document-layout segmentation.
\newblock In \emph{Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}, KDD '22, pp.\ 3743–3751, New York, NY, USA, 2022. Association for Computing Machinery.
\newblock ISBN 9781450393850.
\newblock \doi{10.1145/3534678.3539043}.
\newblock URL \url{https://doi.org/10.1145/3534678.3539043}.
\bibitem[Rodriguez et~al.(2024)Rodriguez, Jian, Panigrahi, Zhang, Feizi, Puri, Kalkunte, Savard, Masry, Nayak, Awal, Massoud, Abaskohi, Li, Wang, Noël, Richter, Vadacchino, Agarwal, Biswas, Shanian, Zhang, Bolger, MacDonald, Fauvel, Tejaswi, Sunkara, Monteiro, Dvijotham, Scholak, Chapados, Kharagani, Hughes, Özsu, Reddy, Pedersoli, Bengio, Pal, Laradji, Gella, Taslakian, Vazquez, and Rajeswar]{rodriguez2024bigdocs}
Juan Rodriguez, Xiangru Jian, Siba~Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats~Leon Richter, Saverio Vadacchino, Shubbam Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang, Noah Bolger, Kurt MacDonald, Simon Fauvel, Sathwik Tejaswi, Srinivas Sunkara, Joao Monteiro, Krishnamurthy~DJ Dvijotham, Torsten Scholak, Nicolas Chapados, Sepideh Kharagani, Sean Hughes, M.~Özsu, Siva Reddy, Marco Pedersoli, Yoshua Bengio, Christopher Pal, Issam Laradji, Spandanna Gella, Perouz Taslakian, David Vazquez, and Sai Rajeswar.
\newblock Bigdocs: An open and permissively-licensed dataset for training multimodal models on document and code tasks, 2024.
\newblock URL \url{https://arxiv.org/abs/2412.04626}.
\bibitem[Schwartz et~al.(2024)Schwartz, Choshen, Shtok, Doveh, Karlinsky, and Arbelle]{schwartz-etal-2024-numerologic}
Eli Schwartz, Leshem Choshen, Joseph Shtok, Sivan Doveh, Leonid Karlinsky, and Assaf Arbelle.
\newblock {N}umero{L}ogic: Number encoding for enhanced {LLM}s' numerical reasoning.
\newblock In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 206--212, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.12}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.12/}.
\bibitem[Shabtay et~al.(2024)Shabtay, Polo, Doveh, Lin, Mirza, Chosen, Yurochkin, Sun, Arbelle, Karlinsky, et~al.]{shabtay2024livexiv}
Nimrod Shabtay, Felipe~Maia Polo, Sivan Doveh, Wei Lin, M~Jehanzeb Mirza, Leshem Chosen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, et~al.
\newblock Livexiv--a multi-modal live benchmark based on arxiv papers content, 2024.
\bibitem[Sidorov et~al.(2020)Sidorov, Hu, Rohrbach, and Singh]{sidorov2020textcaps}
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.
\newblock Textcaps: a dataset for image captioning with reading comprehension, 2020.
\bibitem[Singh et~al.(2019)Singh, Natarajan, Shah, Jiang, Chen, Batra, Parikh, and Rohrbach]{singh2019towards}
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu~Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.
\newblock Towards vqa models that can read, 2019.
\bibitem[Singh et~al.(2021)Singh, Pang, Toh, Huang, Galuba, and Hassner]{singh2021textocr}
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner.
\newblock Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text, 2021.
\bibitem[Smock et~al.(2022)Smock, Pesala, and Abraham]{smock2022pubtables}
Brandon Smock, Rohith Pesala, and Robin Abraham.
\newblock Pubtables-1m: Towards comprehensive table extraction from unstructured documents.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 4634--4642, 2022.
\bibitem[Tanaka et~al.(2021)Tanaka, Nishida, and Yoshida]{tanaka2021visualmrc}
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida.
\newblock Visualmrc: Machine reading comprehension on document images, 2021.
\bibitem[Tang et~al.(2023)Tang, Boggust, and Satyanarayan]{tang2023vistext}
Benny~J Tang, Angie Boggust, and Arvind Satyanarayan.
\newblock Vistext: A benchmark for semantically rich chart captioning, 2023.
\bibitem[Team et~al.(2024)Team, Georgiev, Lei, Burnell, Bai, Gulati, Tanzer, Vincent, Pan, Wang, et~al.]{geminiteam2024gemini15}
Gemini Team, Petko Georgiev, Ving~Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et~al.
\newblock Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
\newblock URL \url{https://arxiv.org/abs/2403.05530}.
\bibitem[Tong et~al.(2024)Tong, Brown, Wu, Woo, Middepogu, Akula, Yang, Yang, Iyer, Pan, Wang, Fergus, LeCun, and Xie]{tong2024cambrian1}
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai~Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie.
\newblock Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.16860}.
\bibitem[Turski et~al.(2023{\natexlab{a}})Turski, Stanis{\l}awek, Kaczmarek, Dyda, and Grali{\'{n}}ski]{CCpdf}
Micha{\l} Turski, Tomasz Stanis{\l}awek, Karol Kaczmarek, Pawe{\l} Dyda, and Filip Grali{\'{n}}ski.
\newblock Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023{\natexlab{a}}.
\bibitem[Turski et~al.(2023{\natexlab{b}})Turski, Stanis{\l}awek, Kaczmarek, Dyda, and Grali{\'n}ski]{turski2023ccpdf}
Micha{\l} Turski, Tomasz Stanis{\l}awek, Karol Kaczmarek, Pawe{\l} Dyda, and Filip Grali{\'n}ski.
\newblock Ccpdf: Building a high quality corpus for visually rich documents from web crawl data, 2023{\natexlab{b}}.
\bibitem[Wadekar et~al.(2024)Wadekar, Chaurasia, Chadha, and Culurciello]{wadekar2024evolutionmultimodalmodelarchitectures}
Shakti~N. Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello.
\newblock The evolution of multimodal model architectures, 2024.
\newblock URL \url{https://arxiv.org/abs/2405.17927}.
\bibitem[Wang et~al.(2021)Wang, Li, Zhou, Chen, Grossman, and Li]{wang2021screen2words}
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li.
\newblock Screen2words: Automatic mobile ui summarization with multimodal learning, 2021.
\bibitem[Wang et~al.(2023)Wang, Raman, Sibue, Ma, Babkin, Kaur, Pei, Nourbakhsh, and Liu]{wang2023docllm}
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu.
\newblock Docllm: A layout-aware generative language model for multimodal document understanding, 2023.
\newblock URL \url{https://arxiv.org/abs/2401.00908}.
\bibitem[Wang et~al.(2024)Wang, Bai, Tan, Wang, Fan, Bai, Chen, Liu, Wang, Ge, et~al.]{wang2024qwen2}
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et~al.
\newblock Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024.
\bibitem[Yang et~al.(2021)Yang, Lu, Wang, Yin, Florencio, Wang, Zhang, Zhang, and Luo]{yang2021tap}
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi~Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo.
\newblock Tap: Text-aware pre-training for text-vqa and text-caption, 2021.
\bibitem[Ye et~al.(2023)Ye, Hu, Xu, Ye, Yan, Xu, Li, Tian, Qian, Zhang, Jin, He, Lin, and Huang]{ye2023ureader}
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi~Qian, Ji~Zhang, Qin Jin, Liang He, Xin~Alex Lin, and Fei Huang.
\newblock Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model, 2023.
\newblock URL \url{https://arxiv.org/abs/2310.05126}.
\bibitem[Yin et~al.(2024)Yin, Fu, Zhao, Li, Sun, Xu, and Chen]{Yin_2024_survey}
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke~Li, Xing Sun, Tong Xu, and Enhong Chen.
\newblock A survey on multimodal large language models.
\newblock \emph{National Science Review}, 11\penalty0 (12), November 2024.
\newblock ISSN 2053-714X.
\newblock \doi{10.1093/nsr/nwae403}.
\newblock URL \url{http://dx.doi.org/10.1093/nsr/nwae403}.
\bibitem[Yuan et~al.(2022)Yuan, Liu, Dikubab, Liu, Ji, Wu, and Bai]{yuan2022syntax}
Ye~Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai.
\newblock Syntax-aware network for handwritten mathematical expression recognition, 2022.
\bibitem[Yue et~al.(2024)Yue, Ni, Zhang, Zheng, Liu, Zhang, Stevens, Jiang, Ren, Sun, et~al.]{yue2024mmmu}
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge~Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et~al.
\newblock Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024.
\bibitem[Zauner(2010)]{zauner2010implementation}
Christoph Zauner.
\newblock Implementation and benchmarking of perceptual image hash functions, 2010.
\bibitem[Zhai et~al.(2023)Zhai, Mustafa, Kolesnikov, and Beyer]{zhai2023siglip}
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
\newblock Sigmoid loss for language image pre-training, 2023.
\bibitem[Zhang et~al.(2024{\natexlab{a}})Zhang, Gao, Gan, Dufter, Wenzel, Huang, Shah, Du, Zhang, Li, Dodge, You, Yang, Timofeev, Xu, Chen, Fauconnier, Lai, You, Wang, Dehghan, Grasch, and Yang]{zhang2024mm15}
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, and Yinfei Yang.
\newblock Mm1.5: Methods, analysis \& insights from multimodal llm fine-tuning, 2024{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2409.20566}.
\bibitem[Zhang et~al.(2024{\natexlab{b}})Zhang, Li, Zhang, Pu, Cahyono, Hu, Liu, Zhang, Yang, Li, and Liu]{zhang2024lmmsevalrealitycheckevaluation}
Kaichen Zhang, Bo~Li, Peiyuan Zhang, Fanyi Pu, Joshua~Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu.
\newblock Lmms-eval: Reality check on the evaluation of large multimodal models, 2024{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2407.12772}.
\bibitem[Zhang et~al.(2024{\natexlab{c}})Zhang, Hu, Xu, Yan, Xu, Jin, Zhang, and Huang]{zhang2024tinychart}
Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji~Zhang, and Fei Huang.
\newblock Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning, 2024{\natexlab{c}}.
\bibitem[Zhang et~al.(2023)Zhang, Zhang, Gu, Zhou, Lipka, Yang, and Sun]{zhang2023llavar}
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.
\newblock Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023.
\bibitem[Zhao et~al.(2022)Zhao, Li, Li, and Zhang]{zhao2022multihiertt}
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang.
\newblock Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data, 2022.
\bibitem[Zhao et~al.(2023)Zhao, Zhao, Nan, Qi, Zhang, Tang, Mi, and Radev]{zhao2023robut}
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev.
\newblock Robut: A systematic study of table qa robustness against human-annotated adversarial perturbations, 2023.
\bibitem[Zheng et~al.(2021)Zheng, Burdick, Popa, Zhong, and Wang]{zheng2020global}
Xinyi Zheng, Doug Burdick, Lucian Popa, Peter Zhong, and Nancy Xin~Ru Wang.
\newblock Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context.
\newblock \emph{Winter Conference for Applications in Computer Vision (WACV)}, 2021.
\bibitem[Zhong et~al.(2020)Zhong, ShafieiBavani, and Jimeno~Yepes]{zhong2020image}
Xu~Zhong, Elaheh ShafieiBavani, and Antonio Jimeno~Yepes.
\newblock Image-based table recognition: data, model, and evaluation, 2020.
\bibitem[Zong et~al.(2024)Zong, Bohdal, Yu, Yang, and Hospedales]{zong2024safety}
Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales.
\newblock Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024.
\end{thebibliography}