PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions
Extracted source text · arXiv 2510.19060
\documentclass{article}
\usepackage{iclr2026_conference,times}
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{soul}
\usepackage{colortbl}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{tcolorbox}
\usepackage{listings}
\usepackage{xspace}
\usepackage{slantsc}
\usepackage[capitalise]{cleveref}
\tcbset{
llmpromptbase/.style={
colback=gray!10,
colframe=gray!50,
fonttitle=\bfseries,
}
}
\usepackage[locale=US]{siunitx}
\sisetup{input-ignore={.},
group-separator = {.},
input-decimal-markers={,},
output-decimal-marker = {,},
group-minimum-digits=4
}
\newcommand{\methodshort}{\textsc{PoSh}\xspace}
\newcommand{\benchmark}{\textsc{DOCENT}\xspace}
\title{\methodshort: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions}
\author{Amith Ananthram$^\diamond$, Elias Stengel-Eskin$^\star$, Lorena A. Bradford$^\clubsuit$, \\ \bf Julia Demarest$^\clubsuit$, Adam Purvis$^\clubsuit$, Keith Krut$^\clubsuit$, Robert Stein$^\clubsuit$, \\ \bf Rina Elster Pantalony$^\ddagger$, Mohit Bansal$^\dagger$, Kathleen McKeown$^\diamond$
\\[0.5em]
$^\diamond$Columbia University
\quad
$^\star$The University of Texas at Austin
\\
$^\clubsuit$The National Gallery of Art
\quad
$^\ddagger$UCLA
\quad
$^\dagger$UNC Chapel Hill
\\
{\tt{amith@cs.columbia.edu}}}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy
\begin{document}
\maketitle
\begin{abstract}
While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge. Standard metrics (e.g. CIDEr, SPICE) were designed for short texts and tuned to recognize errors that are now uncommon, such as object misidentification. In contrast,
long texts require sensitivity to attribute and relation attachments and scores that localize errors to particular text spans. In this work, we introduce \methodshort, a metric for detailed image description that uses scene graphs as \textit{structured rubrics} to guide LLMs-as-a-Judge, producing aggregate scores grounded in fine-grained errors (e.g. mistakes in compositional understanding). \methodshort is replicable, interpretable and a better proxy for human raters than existing metrics (including GPT4o-as-a-Judge). To validate \methodshort, we introduce a challenging new dataset, \benchmark. This novel benchmark contains artwork, paired
with expert-written references, and model-generated descriptions, augmented with \textit{granular} and \textit{coarse} judgments of their quality from art history students. Thus, \benchmark enables evaluating both detailed image description metrics and detailed image description itself in a challenging new domain. We show that \methodshort achieves stronger correlations (+$0.05$ Spearman $\rho$) with the human judgments in \benchmark than the best open-weight alternatives, is robust to image type (using CapArena, an existing dataset of web imagery) and is a capable reward function, outperforming standard supervised fine-tuning. Then, using \methodshort, we characterize the performance of open and closed models in describing the paintings, sketches and statues in \benchmark and find that foundation models struggle to achieve full, error-free coverage of images with rich scene dynamics, establishing a demanding new task to gauge VLM progress. Through both
\methodshort and \benchmark, we hope to enable advances in important areas
such as assistive text generation. We make our metric and our benchmark available at \url{https://github.com/amith-ananthram/posh}.
\end{abstract}
\section{Introduction}
A picture is worth a thousand words -- can vision-language models (VLMs) capture all of them? VLMs have saturated traditional image understanding benchmarks from short captioning to question answering \citep{li2025benchmark}. New, more challenging tasks are needed to measure VLM progress. Detailed image description is of particular interest as it requires \textit{comprehensive} understanding -- e.g., in \cref{fig:teaser}, a VLM must correctly specify \textit{who} is pouring the water. This deep perception is a better proxy for the demands of the real world, where diverse user queries may not be reflected in VQA benchmarks \citep{chenwe}. Moreover, it enables meaningful applications such as image assistive (``alt") text generation that could greatly expand accessibility online \citep{mack2021we}.
However, making progress on detailed description requires cheap, reliable methods for scoring models. Human evaluation is costly, involving the painstaking comparison of long texts. Even so, there is often no substitute as most metrics were designed for short texts and older models \citep{berger2024surveying}. Moreover, while metrics that produce a single coarse score of overall quality allow for the ranking of models, they offer little insight into the granular issues driving performance. Granular issues include \textit{mistakes} in each generation, like the positions of the people in \cref{fig:teaser}, and \textit{omissions} in each reference, like the details of the bird's beak in \cref{fig:dataset}. Automatically localizing such errors is critical as long generations with similar coarse scores may differ in multiple dimensions of interest (e.g., facial features, body orientations, etc.). Otherwise, prompt and/or model iteration necessitates expensive manual inspection to understand which description aspects need improvement.
\begin{figure}[t]
\includegraphics[width=\linewidth]{figures/teaser.png}
\vspace{-20pt}
\caption{Failures in attribute/relation
attachment are common in detailed image description, especially in dynamic scenes. Here, the \textit{man pouring water} is not \textit{central}. \methodshort catches such errors.
}
\label{fig:teaser}
\end{figure}
In this work, we propose \methodshort\footnote{\methodshort (PrOofing Scene grapHs) can judge if your detailed descriptions are what you (really really) want.}, a metric for evaluating detailed descriptions that addresses these challenges. \methodshort extracts scene graphs from a generated description and its reference to use as \textit{structured rubrics} for an LLM to granularly identify mistakes and omissions (see \cref{fig:posh}), pinpointing the textual spans containing errors like attribute/relation mis-attachment. Then, it aggregates these localized errors into coarse scores for mistakes, omissions and overall quality. Thus, \methodshort weds the strengths of structured methods like scene graphs \citep{anderson2016spice}, which reduce descriptions to their consequential visual components, with the strengths of LLMs/VLMs-as-a-Judge \citep{zheng2023judging}, which flexibly compare these visual components against diverse surface realizations.
As \methodshort's coarse scores are grounded in its granular scores, it is interpretable, providing clear insights into the errors driving model performance. Moreover, because \methodshort is entirely open-weight, it is inexpensive to use and replicable, an important pre-requisite for both adoption by researchers and deployment by practitioners that is not afforded by closed models.
Efforts to introduce metrics for longer generations have been constrained by a lack of human judgments, especially at a granular scale and for diverse imagery (see \cref{tab:benchmark_comp}). To address this, we introduce \benchmark, a novel benchmark whose focus is visual art. \benchmark contains paintings, sketches and sculptures with expert-written assistive text that exhaustively describes features like clothing, physical orientation, relative positioning and gaze, drawn from the U.S. National Gallery of Art (see \cref{fig:posh,fig:dataset}). It includes generations from current VLMs with judgments from art history students of their mistakes, omissions and overall quality at two resolutions: granular and coarse. Thus, \benchmark enables evaluating description metrics and descriptions themselves\footnote{AI research often uses \textit{caption} and \textit{alt-text} interchangeably. However, according to \href{https://www.w3.org/WAI/WCAG21/Understanding/non-text-content.html}{Web Content Accessibility Guidelines}, \textit{captions} are related to an image while \textit{alt-text} conveys the information in an image. As our focus is evaluating generations that could serve as \textit{alt-text}, we use the term \textit{description}.}.
We validate \methodshort against the human judgments in \benchmark. We show that \methodshort recovers human description rankings more often (+$3$ percentage points) and achieves stronger correlations with human-derived scores (+$0.05$ Spearman $\rho$) than existing overlap and open-weight alternatives (e.g. SPICE, CAPTURE, LLaVa-Critic), even surpassing GPT4o-as-a-Judge. Moreover, using judgments in CapArena \citep{cheng2025caparena}, we show this strength is robust to image type. Then, given its calibration, we experiment with using \methodshort as a reward function for describing the images in \benchmark and find that this yields meaningfully better descriptions than supervised fine-tuning (SFT).
Finally, using \methodshort, we characterize the performance of open and closed models in describing the artwork in \benchmark, establishing a difficult new task. In so doing, we extend detailed description to a technically challenging and socially impactful domain: assistive text generation for artwork, whose visual complexity and diversity stress VLMs \citep{bengamra2024comprehensive} (see \cref{fig:teaser}).
In summary, our contributions are:
\begin{enumerate}
\item We propose \methodshort, a new metric for detailed description evaluation. \methodshort is interpretable, producing \textit{coarse} scores grounded in \textit{granular} scores that are localized to text spans.
\item We present \benchmark, a new detailed description benchmark with \num{1,750} expert-written art descriptions and \num{900} \textit{granular} \& \textit{coarse} judgments of generations from informed raters.
\item We show \methodshort correlates more with \benchmark's judgments than existing metrics and GPT4o while being replicable. On CapArena, we confirm \methodshort is robust to image type.
\item We demonstrate that using \methodshort as a reward function outperforms SFT on \benchmark.
\item Using \methodshort and \benchmark, we evaluate both open and closed models on detailed description of artwork, establishing a socially impactful new task to gauge VLM progress.
\end{enumerate}
\section{Related Work}
\label{related_work}
Image description is under-specified -- the correct way to describe an image is often task-specific. This is especially true for assistive text which has context-dependent requirements \citep{kreiss2022context}. Moreover, in such sensitive applications, correlated failures between reference-free metrics and VLMs relying on similar components could prove dangerous to end users \citep{deutsch2022limitations}. Thus, our focus is reference-based evaluation. Traditional metrics were not designed to evaluate long text and can involve truncation due to limited context length (e.g. CLIPScore) \citep{papineni2002bleu,lin2004rouge,banerjee2005meteor,vedantam2015cider,see2017get,hessel2021clipscore,sarto2023positive}. Recent work has explored LLMs/VLMs-as-Judges though this requires potentially expensive API calls and offers limited replicability \citep{chan2023clair, cheng2025caparena}. Even when replicable, they do not provide interpretable, grounded granular scores \citep{xiong2024llavacritic}.
While prior metrics like SPICE and CAPTURE leverage scene graphs, they forgo their rich structure by ignoring object attachment \citep{anderson2016spice,dong2024benchmarking}. This favors generations with misattributed details (as in \cref{fig:teaser}). In summarization, \cite{scialom2021questeval} use question generation and answering (QA) to compare a summary and its source. In text-to-image generation, \cite{chodavidsonian} use GPT4 to extract and verify a scene graph from a visual prompt. \methodshort extends these approaches to detailed description evaluation that is replicable and interpretable. With small models, it extracts scene graphs to use as structured rubrics for guiding an open-weight LLM-as-a-Judge.
\begin{table*}[ht]
\setlength{\tabcolsep}{4pt}
\centering
\caption{Detailed image description benchmarks with summaries of their images, reference descriptions (where detail is average \# of entities + attributes + relations) and judgments (where source is the type of annotator used and time is the average time per judgment). Most benchmarks release no human judgments. In contrast, \benchmark contains both granular and coarse judgments of long descriptions of visually complex artwork elicited from annotators knowledgeable in art.
}
\begin{tabular}{|c|c|ccc|cccc|}
\hline
\multirow{2}{*}{\textbf{Name}} &
\multicolumn{1}{c|}{\textbf{Images}} &
\multicolumn{3}{c|}{\textbf{Reference Descriptions}} &
\multicolumn{4}{c|}{\textbf{Judgments}}
\\
&
\textbf{Source} &
\textbf{Source} &
\textbf{Words} &
\textbf{Detail} &
\textbf{Source} &
\textbf{Type} &
\textbf{Time (min)} &
\textbf{\#} \\
\hline
DCI & web &
crowd & $133$ & $71$ & \multicolumn{4}{c|}{\multirow{4}{*}{\textit{no judgments}}} \\
\cline{1-5}
DOCCI & web &
crowd & $122$ & $66$ & & & &\\
\cline{1-5}
\cline{1-5}
CompreCap & web &
crowd & - & - & & & & \\
\cline{1-5}
DeCapBench & \multicolumn{4}{c|}{uses ImageInWords} & & & & \\
\hline
ImageInWords & web &
crowd+ & $193$ & $113$ & \multicolumn{4}{c|}{\textit{no judgments with references\footnotemark}} \\
\hline
DetailCaps & web &
model & $154$ & $95$ & model & coarse & - & $14.4\text{K}$ \\
\hline
CapArena &
\multicolumn{4}{c|}{uses DOCCI} & skilled & coarse & $2.4$ & $5.6\text{K}$ \\
\hline
\hline
\benchmark & \multirow{2}{*}{art} &
\multirow{2}{*}{expert} & \multirow{2}{*}{$251$} & \multirow{2}{*}{$161$} & \multirow{2}{*}{skilled} & granular & $18$ & $300$ \\
(ours) & & & & & & coarse & $5$ & $600$ \\
\hline
\end{tabular}
\label{tab:benchmark_comp}
\end{table*}
\footnotetext{The judgments in IIW compare 1) paired references and 2) paired generations for images with no references. As such, they cannot be used to evaluate a reference-based metric.}
Evaluating such a metric requires human judgments of model generations. Though there are many detailed image description benchmarks \citep{urbanek2024picture,onoe2024docci,garg-etal-2024-imageinwords,lu2025benchmarking,ye2025painting}, most release no such judgments. One notable exception is CapArena which contains coarse rankings of descriptions for web imagery \citep{cheng2025caparena}. In contrast, our new dataset, \benchmark contains both \textit{granular} and \textit{coarse} judgments, enabling the evaluation of fine-grained metrics like \methodshort. Moreover, it expands detailed description to artwork whose scene dynamics and expert-written references are considerably more complex (see \cref{tab:benchmark_comp}).
\section{\methodshort: A New Metric for Detailed Image Description}
\label{sec:posh}
\methodshort is a reference-based metric for detailed image description evaluation that takes two descriptions, a generation and its reference, and then extracts scene graphs from each to use as \textit{structured rubrics} for granular and coarse evaluation of mistakes (i.e. precision) and omissions (i.e. recall).
It does so in three steps (\cref{fig:posh}): \textbf{Step 1)} It extracts scene graphs from a generation and its reference that preserve object attachments. \textbf{Step 2)} It evaluates the presence of generation scene graph components in the reference (and reference scene graph components in the generation) through question answering with an LLM to identify granular mistakes (and omissions). \textbf{Step 3)} It produces coarse scores for mistakes and omissions
grounded in these granular scores. We discuss each step below.
\paragraph{Scene Graph Extraction} As in SPICE \citep{anderson2016spice}, given a description $d$, a scene graph $G(d)$ is a structured representation of $d$. Specifically, $G(d) = \left< O(d), E(d), K(d) \right>$ where $O(d) \subseteq C$ is a set of objects, $E(d) \subseteq O(d) \times A$ is a set of attributes associated with each object and $K(d) \subseteq O(d) \times R \times O(d)$ are a set of relation edges between objects. $C$, $A$ and $R$ are open-world sets of all possible object, attribute and relation classes.
Given a generation $\textit{gen}$ with its reference $\textit{ref}$ we extract sentence-level scene graphs $G_i(\textit{gen}), G_j(\textit{ref})$ for each using off-the-shelf dependency parsing and combine them via coreference resolution \citep{spacy, martinelli-etal-2024-maverick}. This produces scene graphs with full coverage of each text where each component is localized to text spans, allowing for grounded, interpretable scoring. We provide pseudocode for this extraction in \cref{sec:spicy_merging}.
\begin{figure*}
\includegraphics[width=\textwidth]{figures/posh.png}
\vspace{-20pt}
\caption{
\methodshort, a metric for detailed description evaluation, that produces granular and coarse scores. \textbf{Step 1:} Given a generated description and its reference, \methodshort extracts scene graphs that reduce each text's surface diversity to its objects, attributes and relations. \textbf{Step 2:} Using each scene graph as a \textit{structured rubric}, \methodshort produces granular scores for the presence of its components in the other text through QA. \textbf{Step 3:} \methodshort aggregates these granular scores for each scene graph to produce interpretable coarse scores for mistakes and omissions.}
\label{fig:posh}
\end{figure*}
\paragraph{Granular Scoring} Given a description $d$, its scene graph $G(d)$ and a different description $d^\prime$, we apply the function $\Psi$ to every component $c \in G(d)$ to produce a score reflecting its presence in $d^\prime$.
We implement this function via question answering. We produce templated questions for each scene graph component (object, attribute and relation) $c \in G(d)$ and prompt an open-weight LLM to quantify the degree to which $c$ is described in $d^\prime$. This avoids forcing an alignment between the components of $G(d)$ and $G(d^\prime)$. For example, in \cref{fig:posh}, the reference describes the figures in the image as a ``trio." Question answering ensures that a generation that refers to all three individually is not penalized for failing to include such collectives.
As objects with the same class may appear many times in a scene graph (e.g., a description of multiple men), questions require the use of unique identifiers (e.g., ``woman in white" in \cref{fig:posh}) to disambiguate such instances in $d^\prime$. As the identifier used in $d^\prime$ (if any) is not known \textit{a priori}, we test candidate identifiers in three passes, first considering only objects not part of any other objects in $G(d)$ (e.g., ``man" but not ``face of the man"), then objects that are part of other objects in $G(d)$ (e.g., ``face of the man") and finally attributes and relations of objects identified as present in $d^\prime$.
When collecting unique candidate identifiers for an object $o \in O(d)$, we consider its class name (e.g. ``man"), its surface form (e.g. ``musician"), its attributes (e.g. ``tall man"), its relations (e.g. ``man on horse") and if part of a previously identified object, its ``part-of" relation (e.g. ``face of tall man"). We re-write these identifiers using our LLM to improve their fluency and then test each one in bulk for their presence in $d^\prime$. We use the simplest identifier confirmed present by our LLM (if any) to evaluate $o$'s attributes and relations. We provide pseudocode for this templating in \cref{sec:spicy_templating}.
We produce granular mistake scores $\pi$ for every component of $G(\textit{gen})$ and granular omission scores $\rho$ for every component of $G(\textit{ref})$:
\begin{equation*}
\begin{minipage}{0.45\textwidth}
\centering
$\pi(c_{\textit{gen}}) = \Psi(c_{\textit{gen}}, \textit{ref}), \forall c_{\textit{gen}} \in G(\textit{gen})$
\end{minipage}
\hfill
\begin{minipage}{0.45\textwidth}
\centering
$\rho(c_{\textit{ref}}) = \Psi(c_{\textit{ref}}, \textit{gen}), \forall c_{\textit{ref}} \in G(\textit{ref})$
\end{minipage}
\end{equation*}
\paragraph{Coarse Scoring} To maintain interpretability, we calculate coarse scores for mistakes (i.e. precision) and omissions (i.e. recall) by averaging over our granular scores directly:
\begin{equation*}
\begin{minipage}{0.45\textwidth}
\centering
$\text{Mistakes} = \text{mean}_{c\in O(\textit{gen})}(\pi(c))$
\end{minipage}
\hfill
\begin{minipage}{0.45\textwidth}
\centering
$\text{Omissions} = \text{mean}_{c\in O(\textit{ref})}(\rho(c))$
\end{minipage}
\end{equation*}
We note this is a natural place to introduce tunable weights (as in \cite{dong2024benchmarking}) to adapt \methodshort to particular datasets. As we aim to demonstrate robustness, we leave these terms unweighted.
\begin{figure*}
\includegraphics[width=\textwidth]{figures/docent.png}
\vspace{-20pt}
\caption{\benchmark, our newly introduced benchmark, is the first to contain both \textit{granular} (top) and \textit{coarse} (bottom) judgments from informed raters of detailed descriptions of artwork.
}
\label{fig:dataset}
\end{figure*}
\section{\benchmark: A New Benchmark for Detailed Description of Art}
\label{sec:docent}
\benchmark is a benchmark for evaluating detailed description metrics and detailed descriptions themselves. It consists of \num{1,750} works of art with expert-written references from the Open Data Program at the U.S. National Gallery of Art (NGA)\footnote{\url{https://www.nga.gov/open-access-images/open-data.html}}. For $100$ of these images, we produce four generations from current small and frontier VLMs and collect $300$ granular (for $75$ images) and $600$ coarse judgments from annotators knowledgeable in art of \textit{mistakes} and \textit{omissions}\footnote{We forgo fluency as recommended by \citet{kasai-etal-2022-transparent}}. On average, coarse judgments took $5$ minutes and granular judgments took $18$ minutes (six annotation days). This highlights both the cost of manual evaluation and the need for metrics that are reliable proxies.
We include summary statistics in \cref{tab:benchmark_comp} and example judgments in \cref{fig:dataset}.
\paragraph{Image / Reference Selection} While the majority of these works are paintings, they include sketches, statues and lithographs (e.g., the bird in \cref{fig:dataset}), all in the public domain. These images span a diverse set of styles (e.g., Baroque, Renaissance, Impressionism, Post-Impressionism), themes (e.g., war, courtship, still life, religion) and topics (e.g., fishing, drinking, animals, boating).
The accompanying references are detailed descriptions whose purpose is accessibility -- as such, they follow guidelines that include tips for describing color (e.g., ``color can be likened to temperature") and handling ambiguity (e.g, ``describe what makes something ambiguous"). These context informed requirements highlight the need for reference based metrics \citep{kreiss2022context}\footnote{\url{www.nga.gov/visit/accessibility/collection-image-descriptions}}.
Compared to existing detailed image description benchmarks, \benchmark contains considerably more visual complexity (see \cref{tab:benchmark_comp}). On average, its images contain $16\%$ more objects and nearly twice as many people\footnote{As measured by OneFormer \cite{jain2023oneformer}} who require description of their orientation, features, clothing, etc. Consequently, the average length and scene graph size of its reference descriptions are nearly double.
\paragraph{Model Selection} We generate detailed descriptions for $100$ images in \benchmark from four current VLMs that span transparency and model size (from open data/open weight to frontier models): \texttt{LLaVA-1.6-7B} \citep{liu2024improved}, \texttt{Molmo-D-7B} \citep{deitke2024molmo}, \texttt{GPT4o} and \texttt{Claude Sonnet 3.5}. A metric that discriminates among these generations similarly to their human judgments could gauge progress in detailed image description in small and large VLMs over time. Additional details (prompts, date of API access) can be found in \cref{sec:docent_generations}.
\paragraph{Annotators} Given the complexity of our images and the detail of their expert descriptions, we recruit $24$ art history undergraduate majors, masters students and PhD students with domain familiarity to provide high quality judgments of generations. All annotators were sighted with full color vision and native speakers of English. They were compensated at a rate of \$$22$/hour for their time.\footnote{This study was conducted under an approved IRB which will be specified upon publication.}
\paragraph{Granular Judgments} Half of our annotators identify \textit{mistakes} and \textit{omissions} in our model generations. For each image, an annotator is shown its reference and then its four model generations in random order. First, they look at the image, read the reference and then the current generation. Next, by selecting narrow text spans, annotators first identify \textit{mistakes} in the generation (i.e. precision errors) and then \textit{omissions} in the reference that are not in the generation (i.e. recall errors). When identifying omissions, as in \citet{kasai-etal-2022-transparent}, we ask annotators to mentally correct narrow mistakes in the generation first to avoid double-penalizing a model for both incorrect specificity and lack of specificity. For example, a generation that describes a \textit{woman} as a \textit{man} is an error in precision but not in recall. We include our task instructions and interface\footnotemark{} screenshots in \cref{fig:granular_annotation_instructions,fig:granular_annotation_interface}.
\paragraph{Coarse Judgments} The other half of our annotators provide coarse judgments of our model generations. For a given image, an annotator is shown its reference and two generations (\texttt{\#1} and \texttt{\#2}) in random order and asked to rank the generations in terms of \texttt{mistakes} (i.e. precision), \texttt{omissions} (i.e. recall) and \texttt{overall quality}. These pairwise judgments avoid some of the inter-annotator inconsistency of Likert ratings, especially for long text \citep{novikova2018rankme}.
Annotators select among five choices for each dimension: \texttt{\#1 much better}, \texttt{\#1 slightly better}, \texttt{equal}, \texttt{\#2 slightly better} and \texttt{\#2 much better}. As with our granular judgments, we ask annotators to mentally correct narrow mistakes (i.e. precision errors) in each generation before judging omissions. To avoid favoring previously seen generations, we ensure no annotator sees the same generation more than once. We include our task instructions and screenshots of our annotation interface\footnotemark[\value{footnote}] in \cref{fig:coarse_annotation_instructions,fig:coarse_annotation_interface}. \footnotetext{Hosted on Label Studio (\url{https://labelstud.io})}
\paragraph{Agreement} For a given image, each generation / pair of generations receives at least one granular and one coarse judgment respectively. For $15\%$ of our tasks, we collect additional judgments from our annotators ($2$ for coarse, $1$ for granular). Additionally, for $20$ granular tasks and $30$ coarse tasks, we collect expert judgments from a PhD in art history who authors assistive text at an art museum. We use these extra judgments to calculate agreement in two ways (among our annotators and between our annotators and our expert). We report agreement in \cref{tab:docent_granular,tab:docent_coarse} of the Appendix.
For our granular judgments, as recommended by \citet{hripcsak2005agreement} for span annotation tasks where the boundaries of negative examples (i.e. non-errors) are ill-defined, we measure agreement using the relaxed F1 (matching spans that contain 50\% overlapping tokens). Under this measure, our student annotators exhibit strong agreement among themselves and with our expert.
Our coarse judgments exhibit moderate inter-annotator agreement, with Krippendorf's $\alpha=0.509$, $0.409$ and $0.459$ for mistakes, omissions and overall quality \citep{landis1977measurement}. This level of agreement is unsurprising for coarse detailed description evaluation -- judgment requires weighing the relative importance of each text's granular errors and is consequently more subjective. Nevertheless, our student annotators exhibit moderate to strong correlations with our expert, with significant Pearson $\rho$ values of $0.727$, $0.501$ and $0.492$ for mistakes, omissions and overall quality respectively.
\paragraph{How well do these VLMs describe art?}
When considering the performance of the four models included in \benchmark, we observe expected trends, adding to our confidence in the quality of our judgments: the smaller models make more mistakes and have more omissions than the larger models (see \cref{tab:docent_granular,tab:docent_coarse}). Though most models make few mistakes, they all struggle with omissions. The best model, \texttt{gpt4o} covers only $44.0\%$ of the visual information conveyed in \benchmark's references. Raising this requires continued prompt iteration, highlighting the need for an automated metric that can reliably measure both granular and coarse differences in mistakes and omissions.
\section{Experiments}
\paragraph{\methodshort} We extract sentence-level scene graphs using \texttt{en\_core\_web\_trf} from \citet{spacy}, a transformer trained to perform dependency parsing. To merge objects across these scene graphs while preserving attribute and relation attachments, we use \texttt{maverick-mes-ontonotes} from \citet{martinelli-etal-2024-maverick} to perform co-reference resolution. Our QA scorer $\Psi$ is \texttt{qwen-3-14b} \citep{yang2025qwen3}. We template evaluation questions for each scene graph component (as in \cref{fig:posh}), re-write candidate identifiers using $\Psi$ to improve fluency and then prompt $\Psi$ to answer each templated presence question by predicting a number between $1$ and $5$. We extract scores by taking the weighted average over the token logits for each number as in \cite{liu2023g}. When determining object presence, we use a threshold of $2$, determined through tuning on a small hand-annotated validation set. We provide further implementation details and all prompts used in \cref{sec:posh_appendix}.
\paragraph{Benchmarks}
We evaluate \methodshort against the judgments in \benchmark and CapArena.
\textsc{\it{DOCENT}} is our new detailed description benchmark containing judgments from knowledgeable human annotators: granular mistake and omission spans for $300$ individual generations and coarse scaled rankings of mistakes, omissions and overall quality of $600$ paired generations. We evaluate granular metrics on this benchmark using macro F1 where we credit/penalize a model for predicting each annotated/unannotated token. Our coarse judgments are in the form $(text_1, text_2, score)$ where score indicates how much better or worse $text_1$ is than $text_2$. We evaluate each coarse metric $m$ by calculating its 1) pairwise accuracy (whether it picks the better text or a tie, using a tie threshold inferred from the gold tie rate) and 2) Spearman rank $\rho$ and Kendall's $\tau$ correlations between $m(text_1) - m(text_2)$ and $score$, a common practice in machine translation metric evaluation \citep{kocmi2021ship}. More details can be found in \cref{sec:eval_metrics}.
\textsc{\it{CapArena}} \citep{cheng2025caparena} contains \num{3,361} images and \num{10,348} detailed descriptions generated from $14$ current VLMs. \num{5,599} pairs of these generations receive coarse judgments from human raters of the better generation (or ``tie"). We include CapArena, which contains diverse images drawn from the web, to validate metric robustness. However, we note the dramatic simplicity of its images and references compared to those in \benchmark (see \cref{tab:benchmark_comp}). $64\%$ of its images\footnote{As measured by OneFormer \citep{jain2023oneformer}} contain fewer than two objects and $95\%$ depict fewer than two people (compared to $27\%$\ and $52\%$ in \benchmark). A metric is evaluated on CapArena at the caption-level (whether it picks the better text or a tie, using a tie threshold inferred from the gold tie rate) and at the model-level (Spearman's rank and Kendall's $\tau$ correlation between ELO rankings derived from metric predictions and gold judgments).
\begin{table*}[!ht]\centering
\caption{
Granular metrics evaluated on \benchmark. Reported numbers are the maximum F1 when identifying mistakes and omissions across all alerting thresholds. \methodshort is best at predicting both mistakes (which are relatively rare) and omissions (which are relatively common). As \methodshort's coarse scores are aggregated from its granular scores, this demonstrates its interpretability.
}
\begin{tabular}{l|c|c|c|c}\toprule
& Random & 4GramEmbed & SGEmbed & \textbf{\methodshort}\\\midrule
Mistakes F1 & $0.386$ & $0.513$ & $0.513$ & $\mathbf{0.559}$ \\\midrule
Omissions F1 & $0.491$ & $0.505$ & $0.634$ & $\mathbf{0.680}$ \\
\bottomrule
\end{tabular}
\label{tab:docent_granular_results}
\end{table*}
\paragraph{Granular Baselines}
Our work is the first to introduce both a metric and a benchmark for granular evaluation of detailed descriptions. As such, this limits our baselines to those able to predict \textit{localized} mistakes and omissions (i.e., the spans where errors occur). We consider two embedding-based approaches, using \texttt{Qwen/Qwen3-Embedding-8B} from \citet{yang2025qwen3}: \textbf{4GramEmbed}, which embeds and compares $4$-grams from a generation and its reference, and \textbf{SGEmbed}, which embeds and compares components from the scene graphs of a generation and its reference. As these approaches (and \methodshort) produce span scores, we report the maximum F1 scores for mistakes and omissions across all alerting thresholds. More details can be found in \cref{sec:granular_eval_baselines}.
\paragraph{Coarse Baselines}
Though \methodshort is a text-only reference-based metric, we select a representative set of reference-free (requiring only an image) and reference-based (requiring a gold standard) pointwise metrics (i.e. produce numerical scores) as our baselines. These include n-gram overlap metrics like BLEU \citep{papineni2002bleu}, ROUGE-L-Sum \citep{see2017get}, METEOR \citep{banerjee2005meteor} and CIDER \citep{vedantam2015cider} and model-based metrics like SPICE \citep{anderson2016spice}, CLIPScore \citep{hessel2021clipscore} and CAPTURE \citep{dong2024benchmarking}. Additionally, we consider several LLMs/VLMs-as-a-Judge\footnote{CLAIR/Faithscore were not included due to complications with their codebases \citep{chan2023clair, jing2024faithscore}. Due to cost (estimated at \$$1,000$), we only evaluate DCScore \citep{ye2025painting} on \benchmark.}: FLEUR \citep{lee2024fleur}, Prometheus \citep{kim2023prometheus}, LLaVA-Critic \citep{xiong2024llavacritic}, DCScore \cite{ye2025painting}, Qwen-3 \citep{yang2025qwen3} and GPT4o/GPT5 in three settings (reference-free with image, reference-based without image and reference-based with image). More details can be found in \cref{sec:coarse_eval_baselines}.
\paragraph{Reward Function} Finally, given the potential of a well-calibrated metric as a verifier in reinforcement learning (RL), we evaluate \methodshort as a reward function. We train \texttt{Qwen2.5-VL-7B} on the \num{1,000} images in \benchmark's training set in two settings: 1) supervised fine-tuning (SFT), and 2) RL with DAPO \citep{yu2025dapo} using \methodshort. We collect coarse judgments (as in \cref{sec:docent}) for $40$ generation pairs from graduate students in NLP. More details can be found in \cref{sec:reinforcement_learning}.
\begin{table*}[!ht]\centering
\setlength{\tabcolsep}{5pt}
\caption{
Selected coarse metrics evaluated on \benchmark and CapArena, identified with $\Theta$ (parameter count, in billions), \includegraphics[height=1em]{figures/page_with_curl.png} (requires a reference), \includegraphics[height=1em]{figures/frame_with_picture.png} (requires an image) and \includegraphics[height=1em]{figures/repeat.png} (replicable). ``acc" indicates accuracy at predicting the better generation (or ``tie") in each judged pair. For \benchmark, $\rho$ / $\tau$ indicate the Spearman rank / Kendall's $\tau$ correlations between differences in the metric and differences in the rank of the generations in each pair. For CapArena, $\rho$ / $\tau$ indicate the Spearman rank / Kendall's $\tau$ correlations between model ELO rankings derived from metric scores and human judgments. \textbf{Bold} indicates the best replicable metric while \underline{underlining} indicates the best metric overall.
\sethlcolor{gray!25}\hl{Gray cells} indicate correlations that are \textit{not} statistically significant at $\alpha = 0.05$.
\methodshort beats nearly all baselines, including GPT4o, across both benchmarks in all settings (caption ranking of mistakes, omissions and overall quality \& model ranking) while remaining replicable.
}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|c|ccc|ccc|ccc|ccc|ccc}\toprule
& & & & & \multicolumn{9}{c|}{\textbf{\benchmark}} & \multicolumn{3}{c}{\textbf{CapArena}} \\\midrule
& & & & & \multicolumn{3}{c}{Mistakes} &\multicolumn{3}{c}{Omissions} &\multicolumn{3}{c|}{Overall Quality} & Desc & \multicolumn{2}{c}{Model} \\
& \includegraphics[height=1em]{figures/theta.png} & \includegraphics[height=1em]{figures/page_with_curl.png} & \includegraphics[height=1em]{figures/frame_with_picture.png} & \includegraphics[height=1em]{figures/repeat.png} & acc & $\rho$ & \multicolumn{1}{c}{$\tau$} & acc &$\rho$ & \multicolumn{1}{c}{$\tau$} &acc &$\rho$ & \multicolumn{1}{c|}{$\tau$} &acc &$\rho$ & $\tau$ \\
\midrule
length & & & &$\checkmark$ & 30.5 &-0.270 &-0.206 &37.8 & \sethlcolor{gray!25}\hl{-0.002} & \sethlcolor{gray!25}\hl{-0.001} & 38.0 &-0.160 &-0.121 & 58.7 & 0.710 & 0.582 \\
\midrule
SPICE & & $\checkmark$ & &$\checkmark$ & 41.3 & 0.308 & 0.234 & 55.0 & 0.464 & 0.360 & 58.5 & 0.458 & 0.349 & 41.7 & 0.275 & 0.231 \\
CAPTURE & & $\checkmark$ & &$\checkmark$ & 43.3 & 0.259 & 0.194 & 53.8 & 0.447 & 0.340 & 56.0 &0.453 &0.347 & 52.5 & 0.613 & 0.538 \\
Qwen3 & 32 & $\checkmark$ & & $\checkmark$ & 57.7 & 0.282 & 0.235 & 53.5 & 0.286 & 0.253 & 61.2 & 0.289 & 0.257 & 56.2 & 0.899 & 0.714 \\
LLaVa Critic & 72 & $\checkmark$ & $\checkmark$ & $\checkmark$ & \textbf{\underline{62.8}} & 0.412 & 0.351 & 57.0 & 0.509 & 0.430 & 66.8 & 0.546 & 0.461 & \textbf{\underline{64.0}} & \textbf{\underline{0.987}} & \textbf{\underline{0.934}}\\
\midrule
DCScore & & $\checkmark$ & $\checkmark$ & & \underline{62.8} & \underline{0.541} & 0.422 & 54.0 & 0.395 & 0.298 & 62.8 & 0.471 & 0.362 & - & - & - \\
GPT4o & & $\checkmark$ &$\checkmark$ & & 58.5 & 0.484 & 0.396 & 56.0 & 0.380 & 0.303 & 67.3 & 0.510 & 0.402 & 55.4 & 0.890 & 0.802 \\
GPT5 & & $\checkmark$ & & & 62.5 & 0.511 & \underline{0.423} & 53.2 & 0.421 & 0.332 & 68.0 & 0.540 & 0.440 & 59.1 & 0.956 & 0.846 \\
\midrule
\methodshort & 14 & $\checkmark$ & & $\checkmark$ & 60.3 & \textbf{0.518} & \textbf{0.405} & \textbf{\underline{61.8}} & \textbf{\underline{0.578}} & \textbf{\underline{0.449}} & \textbf{\underline{70.3}} & \textbf{\underline{0.600}} & \textbf{\underline{0.466}} & 59.2 & 0.921 & 0.780 \\
\bottomrule
\end{tabular}
}
\label{tab:coarse_results}
\end{table*}
\section{Results \& Discussion}
\paragraph{PoSh as a Granular Metric}
\cref{tab:docent_granular_results} presents the performance of \methodshort and our selected metrics on identifying the mistakes and omissions in \benchmark. Given the imbalanced nature of our data (where mistakes are infrequent and omissions are common), we report macro averages for each subtask, measuring how well each approach localizes errors within a generation and its reference respectively. First, we note that this task is difficult. The considerable room for improvement highlights the value of a benchmark like \benchmark that contains granular judgments of textual spans. Even so, \textbf{\methodshort achieves the highest F1 in mistake ($0.564$) and omission ($0.675$) localization}. As its coarse scores are aggregated from these granular scores, this demonstrates its interpretability.
\paragraph{PoSh as a Coarse Metric}
\cref{tab:coarse_results} presents the performance of \methodshort and the best baselines on predicting the coarse judgments in \benchmark and CapArena (full results in \cref{sec:full_coarse_results}).
On \benchmark, across all three dimensions, \textbf{\methodshort outperforms every existing replicable metric} (i.e., metrics not reliant on an API), yielding a $0.11$ increase in Spearman $\rho$ for mistakes ($26\%\uparrow$), a $0.07$ increase for omissions ($14\%\uparrow$) and a $0.05$ increase for overall quality ($10\%\uparrow$) over the next best. It even outperforms GPT4o (in all settings) and text-only GPT5 (on omissions and overall quality). Among all metrics, DCScore \citep{ye2025painting} proves best at predicting mistakes. However, its reliance on GPT4o to extract and verify factoids fails to achieve full coverage of reference detail, underperforming in predicting omissions and overall quality. Despite employing a smaller LLM, \methodshort's use of dependency parsing and coreference resolution to extract scene graphs avoids this.
On CapArena, \methodshort achieves higher caption-level accuracies and model-ranking correlations than nearly every existing open-weight metric and GPT4o. The sole exception is LLaVa Critic, a much larger VLM-as-a-Judge \citep{xiong2024llavacritic}. This is driven in part by the simplicity of CapArena (see \cref{tab:benchmark_comp}). On the subset of CapArena depicting three or more people ($167$ judgments), each of whom requires careful description, \textbf{\methodshort outperforms LLaVa Critic with model ranking correlations of $\mathbf{\rho=0.776, \tau=0.648}$ compared to $\mathbf{\rho=0.686, \tau=0.550}$}. Thus, \methodshort is robust to image type, excelling in visually complex cases that are of particular interest in detailed image description.
\paragraph{PoSh as a Reward Function} In \cref{tab:rl_results} of the Appendix, we report annotator agreement and aggregate preferences between SFT and DAPO with \methodshort. In each dimension of interest, a \methodshort generation earns a score between $-2$ and $2$ based on how much worse or better it is than its SFT counterpart. While \methodshort-tuned generations had more mistakes (an average score of $-0.243$),
these were incurred in service of \textbf{much fewer missing details ($\mathbf{+0.432}$), resulting in higher overall quality ($\mathbf{+0.135}$)}. This speaks to the strength of \methodshort when optimized directly.
\paragraph{\benchmark Leaderboard} Finally, in \cref{fig:leaderboard}, we plot the \methodshort scores of VLMs in describing the art in \benchmark. While closed models like Gemini 2.5 Pro lead, open models remain competitive. Improvements will require continued iteration, informed in part by \methodshort's granular scores.
\section{Conclusion}
We present \methodshort, a novel metric for detailed image description that extracts scene graphs to use as structured rubrics for guiding LLMs-as-a-Judge, providing interpretable, replicable scores. To validate \methodshort, we introduce \benchmark, a new benchmark with expert-written descriptions of visually complex artwork along with granular and coarse judgments of generations from knowledgeable raters. We show that \methodshort correlates better than other metrics with these judgments, is robust to image type and is a capable reward function. Through \methodshort and \benchmark, we introduce a leaderboard for a new challenging task, detailed image description of artwork. It is our hope that this work will drive progress in meaningful areas such as assistive text generation for artwork and beyond.
\section{Ethics Statement}
The judgments in \benchmark were collected under an approved IRB protocol with all annotator data anonymized and participants receiving fair compensation (at $\$22$/hour) for their time and expertise.
All of the \num{1,750} artwork images in \benchmark are in the public domain, and the expert-written reference descriptions were published by the U.S. National Gallery of Art under their Open Data Program\footnote{\url{https://github.com/NationalGalleryOfArt/opendata}} specifically for research purposes, ensuring appropriate use and attribution.
While this work aims to benefit accessibility applications for blind and low-vision users, we acknowledge that direct community involvement in the development process would strengthen future iterations. However, we note that the expert reference descriptions were written according to the National Gallery of Art's accessibility guidelines\footnote{\url{https://www.nga.gov/visit/accessibility/collection-image-descriptions}} which lay out best practices for assistive text.
Finally, as with other computer vision systems, this work could theoretically be applied to surveillance contexts, but our focus on detailed description does not introduce novel privacy risks beyond those inherent to existing image analysis technologies. The primary intended application—improving accessibility—aligns with beneficial societal outcomes.
\section{Reproducibility Statement}
A core motivation behind \methodshort is improving replicability in detailed image description evaluation through the introduction of a performant open-weight metric. In that spirit, we ensure full reproducibilty of our findings by:
\begin{enumerate}
\item including comprehensive technical details in the Appendix
\item publishing the code for both our metric and our metric evaluations at \url{https://github.com/amith-ananthram/posh}
\item publishing our benchmark at \url{https://github.com/amith-ananthram/posh/docent}
\item making our models and our benchmark available to the broader research community on HuggingFace
\end{enumerate}
\subsubsection*{Acknowledgments}
This research is being developed in part with funding from the National Science Foundation under Cooperative Agreement PHY-2229929 (the NSF AI Institute for Artificial and Natural Intelligence) and DRL-2112635 (the NSF AI Engage Institute), the Columbia Center for Artificial Intelligence and Technology (CAIT) and ONR Grant N00014-23-1-2356. Data and data science collaboration were provided by the National Gallery of Art in Washington, DC. We gratefully acknowledge use of the research computing resources of the Empire AI Consortium, Inc, with support from Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the State of New York, the National Gallery of Art, the National Science Foundation or the U.S. Government.
\bibliography{iclr2026_conference}
\bibliographystyle{iclr2026_conference}
\appendix
\section{Appendix}
\label{sec:appendix}
\subsection{\methodshort}
\label{sec:posh_appendix}
\subsubsection{Scene Graph Extraction}
\label{sec:spicy_merging}
While we provide the complete implementation for our scene graph extraction in our codebase, we include simplified pseudocode below:
\begin{verbatim}
def GetGraph(text):
doc = ParseTextWithSpacy(text)
components = ExtractComponents(doc)
corefs = GetCorefWithMaverick(doc)
entities, relations = [], []
for each component:
if IsNoun(component):
if HasEarlierMention(component):
UpdateExistingEntity(
entities, component
)
else:
CreateNewEntity(
entities, component
)
for each component:
if IsAdjective(component):
UpdateAtributes(
entities, component
)
elif IsVerb(component):
UpdateVerbRelations(
relations, component
)
elif IsPrep(component):
UpdatePrepRelations(
relations, component
)
return (entities, relations)
\end{verbatim}
\subsubsection{Granular QA Templating}
\label{sec:spicy_templating}
While we provide the complete implementation for our question templating in our codebase, we include simplified pseudocode below:
\begin{verbatim}
def TemplateEntityQuestions(
text, entities
):
colls = GetCollisions(
entities
)
questions = []
for e in entities:
identifiers = []
if IsEmpty(colls):
identifiers.add(e.text)
for each attr in e:
if IsUnique(attr, colls):
identifiers.add(
attr + e.text
)
if len(identifiers) > 0:
AddToQuestions(identifiers)
continue
for each rel in e:
if IsUnique(rel, colls)
identifiers.add(
rel.head +
rel.text +
rel.tail
)
AddToQuestions(identifiers)
ReWriteIdentifiers(questions)
def TemplateAttrRelQuestions(
text, entities
):
questions = []
for e in entities:
for attr in e:
AddToQuestions(
attr, e.identifier
)
for rel in e:
AddToQuestions(
rel, e.identifier
)
\end{verbatim}
\subsubsection{Prompts}
\begin{tcolorbox}[llmpromptbase, title=Entity Identifier Rewrite Prompt (for attributes)]
\begin{lstlisting}
Rewrite ``{entity_identifier}" into a grammatically correct
noun phrase, keeping all details. For example, ``dog small"
should be rewritten as ``the small dog". Output ONLY the phrase.
\end{lstlisting}
\end{tcolorbox}
\begin{tcolorbox}[llmpromptbase, title=Entity Identifier Rewrite Prompt (for relations)]
\begin{lstlisting}
Rewrite ``{entity_identifier}" into a grammatically correct
noun phrase, keeping all details. ``cat jumps on window"
should be rewritten as ``the cat jumping on the window".
Output ONLY the phrase.
\end{lstlisting}
\end{tcolorbox}
\begin{tcolorbox}[llmpromptbase, title=Verification Prompt]
\begin{lstlisting}
if {precision}
DESCRIPTION1: {target_text}
DESCRIPTION2: {source_text}
{else}
DESCRIPTION: {target_text}
{if entity}
Q: Is an entity matching ``{entity_identifier}"
(from DESCRIPTION2) mentioned in (the) DESCRIPTION(1)?
{elif attribute}
Q: Is ``{entity_identifier}" (from DESCRIPTION2)
described as ``{attribute}" in (the) DESCRIPTION(1)?
{else}
Q: Is the relation between ``{entity1_identifier}"
and ``{entity2_identifier}" (in DESCRIPTION2)
described as ``{relation}" in (the) DESCRIPTION(1)?
Consider paraphrases but do NOT infer unstated details.
Scoring guide -> 1: absent; 2: weak hint; 3: partial;
4: clear; 5: explicit & unambiguous.
Respond ONLY with an integer 1-5.
\end{lstlisting}
\end{tcolorbox}
\subsection{\benchmark}
\subsubsection{Generations}
\label{sec:docent_generations}
We produce generations from the following models:
\begin{enumerate}
\item \texttt{llava-v1.6-mistral-7b-hf} on HuggingFace \citep{liu2024improved}
\item \texttt{Molmo-7B-D-0924} on HuggingFace \citep{deitke2024molmo}
\item \texttt{gpt-4o-2024-08-06}, accessed on 1/31/25
\item \texttt{claude-3-5-sonnet-20241022}, accessed on 1/31/25
\end{enumerate}
We use the same prompt (included below). For \texttt{LLaVA-1.5-7B} and \texttt{Molmo-D-7B}, we use nucleus sampling \cite{holtzmancurious} with $p=0.9$ and a temperature of $0.7$.
\begin{tcolorbox}[llmpromptbase, title=Detailed Description Prompt]
\begin{lstlisting}
[IMAGE]
Generate a detailed description of this painting, avoiding
interpretation and focusing on only its visual elements.
\end{lstlisting}
\end{tcolorbox}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/granular_instructions.png}
\caption{The beginning of our granular annotation instructions. The full instructions can be found on our GitHub: \url{https://anonymous.4open.science/r/posh/docent/annotation_instructions/granular.pdf}.}
\label{fig:granular_annotation_instructions}
\end{figure*}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/coarse_instructions.png}
\caption{The beginning of our coarse annotation instructions. The full instructions can be found on our GitHub: \url{https://anonymous.4open.science/r/posh/docent/annotation_instructions/coarse.pdf}.}
\label{fig:coarse_annotation_instructions}
\end{figure*}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/granular_annotation_interface.png}
\caption{Our granular annotation interface, hosted on Label Studio (\url{https://labelstud.io}).}
\label{fig:granular_annotation_interface}
\end{figure*}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/coarse_annotation_interface.png}
\caption{Our coarse annotation interface, hosted on Label Studio (\url{https://labelstud.io}).}
\label{fig:coarse_annotation_interface}
\end{figure*}
\subsubsection{Avoiding Double Penalties}
\label{sec:precision_correction}
In \citet{kasai-etal-2022-transparent}, after identifying an error in precision, the authors correct the error before annotating recall. This avoids doubly penalizing a description for errors in specificity which would unfairly favor more generic descriptions (which are only penalized once, for recall). We instruct our annotators to do the same. Below, we have manually identified a few cases of its application from our granular annotations though we note that annotators found our models to be reasonably precise (\cref{tab:docent_granular}) so this rule was infrequently required.
Due to the length of the generations and descriptions in \benchmark, please consult our codebase for example judgments: \url{https://anonymous.4open.science/r/posh/docent}
\subsubsection{\benchmark: Agreement and Judgment Summary}
\begin{table*}[!ht]\centering
\caption{
Granular judgments of mistakes and omissions in \benchmark. Left: inter-annotator agreement (relaxed F1, using overlap thresholds of $\geq 1$ token and $\geq 50\%$ of tokens); additionally, the recall $R$ of our expert annotations. Our student judgments exhibit strong inter-annotator agreement and good coverage of our more sparing (see \cref{fig:student_vs_expert}) error annotations.
Right: the average percentage of tokens \textit{not} marked as mistakes/omissions for each model (higher is better).
}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|cc|cccc|ccccc}\toprule
& \multicolumn{2}{c|}{\textbf{student}} & \multicolumn{4}{c|}{\textbf{expert}} & \multirow{2}{*}{\textbf{llava}} & \multirow{2}{*}{\textbf{molmo}} & \multirow{2}{*}{\textbf{claude}} & \multirow{2}{*}{\textbf{gpt4o}} \\
& F1 & F1@50 & R & F1 & R@50 & F1@50 & & & & \\\midrule
\textbf{mistakes} & $0.980$ & $0.604$ & $1.000$ & $0.890$ & $0.652$ & $0.250$ & $0.863$ & $0.898$ & $0.952$ & $0.947$ \\
\textbf{omissions} & $1.000$ & $0.754$ & $1.000$ & $1.000$ & $0.927$ & $0.475$ & $0.300$ & $0.416$ & $0.399$ & $0.437$ \\
\bottomrule
\end{tabular}
}
\label{tab:docent_granular}
\caption{
Coarse judgments of precision, recall and overall quality in \benchmark. Top: inter-annotator agreement (Krippendorff $\alpha$ for student, Pearson $\rho$ and average difference for expert). Bottom: the average relative score of each row model compared to each column model ($1$ indicates the row is much better, $5$, the column is much better).
}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|ccc|ccc|ccc}\toprule
\textbf{} &\multicolumn{3}{c|}{\textbf{mistakes}} &\multicolumn{3}{c|}{\textbf{omissions}} &\multicolumn{3}{c}{\textbf{overall quality}} \\\midrule
\textbf{student $(\alpha)$} &\multicolumn{3}{c|}{$0.509$} &\multicolumn{3}{c|}{$0.409$} &\multicolumn{3}{c}{$0.459$} \\
\textbf{expert $(\rho, \Delta)$} &\multicolumn{3}{c|}{$0.727$, $0.633$} &\multicolumn{3}{c|}{$0.501$, $0.644$} &\multicolumn{3}{c}{$0.492$, $0.788$} \\
\midrule
&\textbf{llava} &\textbf{molmo} &\textbf{claude} &\textbf{llava} &\textbf{molmo} &\textbf{claude} &\textbf{llava} &\textbf{molmo} &\textbf{claude} \\
\midrule
\textbf{molmo} & $2.42$ & & & $2.32$ & & & $2.14$ & & \\
\textbf{claude} & $1.92$ & $2.37$ & & $2.16$ & $2.74$ & & $1.87$ & $2.5$ & \\
\textbf{gpt4o} & $1.86$ & $2.3$ & $3.0$ & $2.01$ & $2.54$ & $2.61$ & $1.68$ & $2.21$ & $2.63$ \\
\bottomrule
\end{tabular}
}
\label{tab:docent_coarse}
\end{table*}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/student_vs_expert.png}
\caption{A comparison of our student and expert judgments of omissions for the same generation. Most differences are due to 1) students preferring the specificity of terms like ``women" over ``figures" and 2) students annotating all the attributes and relations of entities marked as missing, e.g., ``skin", ``halos", ``noses", and the span beginning ``painted with a scene...". Generally, expert judgments are a subset of our student judgments for these reasons.}
\label{fig:student_vs_expert}
\end{figure*}
\subsubsection{\benchmark: Granular Agreement Details}
We additionally calculate granular agreement using a more conservative threshold ($\geq 50\%$ token overlap). Here, relaxed F1 remains strong among our art history student annotators ($0.612$ for mistakes, $0.773$ for omissions). Though we observe drops in relaxed F1 when compared to our expert, it is driven by two factors: annotation style, with our expert favoring sparsity, and a relative strictness on the part of our student annotators. This is reflected in the expert annotation recall values in \cref{tab:docent_granular} where a majority of the spans identified by our expert were also marked by our student annotations for both mistakes ($0.652$) and omissions ($0.927$). Thus, our expert annotations are a subset of our stricter student annotations. We provide a side-by-side example of a student annotation and an expert annotation in \cref{fig:student_vs_expert}.
\subsection{Evaluation}
\subsubsection{Experimental Setting}
Numbers for \methodshort reported in \cref{tab:docent_granular_results,tab:docent_coarse,tab:full_coarse_results} were produced on a single H100 GPU. Our codebase specifies the necessary versions of CUDA and dependent libraries like vLLM for replication (relying on vLLM's implementation of batch invariance).
\subsubsection{Correlation Metrics}
\paragraph{Spearman's Rank Correlation Coefficient ($\rho$)} assesses the monotonic relationship by calculating Pearson's correlation on the ranks of two continuous variables rather than their raw values. It ranges from $-1$ to $+1$, with $+1$ indicating perfect monotonic increasing relationship and $-1$ indicating perfect monotonic decreasing relationship. It's less sensitive to outliers than Pearson's and can detect monotonic non-linear relationships. As the coarse annotations in \benchmark specify the rank of two generated image descriptions, Spearman is well suited for evaluating
\paragraph{Kendall's $\tau$} measures the ordinal association between two variables based on the ranks of the data. It ranges from $-1$ to $+1$, where $+1$ indicates perfect agreement between the two rankings, $0$ indicates no association, and $-1$ indicates perfect disagreement. Unlike Pearson's, Kendall's tau is non-parametric and robust to outliers, making it appropriate for non-linear relationships and non-normally distributed data.
\subsubsection{Coarse Metric Evaluation}
\label{sec:eval_metrics}
We convert each coarse judgment of a generation pair ($\text{text}_1$, $\text{text}_2$, $\text{label}$) in \benchmark to a numerical score $s$ that reflects the relative rank of $\text{text}_1$ and $\text{text}_2$. If $\text{text}_1$ was marked as \texttt{much better} than $\text{text}_2$, $s = 2$; \texttt{slightly better} than $\text{text}_2$, $s = 1$ and \texttt{equal} to $\text{text}_2$, $s = 0$. Similarly, if $\text{text}_2$ was marked as \texttt{slightly better} than $\text{text}_1$, $s = -1$ and $s = -2$ if \texttt{much better} than $\text{text}_1$. These numerical scores reflect the relative rank of $\text{text}_1$ and $\text{text}_2$ and allow us to evaluate the correlation of different metrics $m$ with the coarse judgments in \benchmark by comparing $s$ to $m(\text{text}_1) - m(\text{text}_2)$ with appropriate measures of monotonicity like Spearman's rank correlation $\rho$.
\subsubsection{Granular Baselines}
\label{sec:granular_eval_baselines}
\paragraph{4GramEmbed} We extract all of the $4$-grams from each sentence of a generation and its reference, embed them using \texttt{Qwen/Qwen3-Embedding-8B} \citep{reimers-2019-sentence-bert,yang2025qwen3} and then calculate the maximum pairwise similarities between generation $4$-grams and reference $4$-grams. Generation text spans and reference text spans with maximum pairwise similarity scores lower than $0.7$ were predicted as mistakes and omissions respectively, a threshold chosen to maximize the macro F1 scores reported for \textbf{4GramEmbed} in \cref{tab:docent_granular_results}.
\paragraph{SGEmbed} We extract all of the components (objects, attribute-object pairs, and object-relation-object triples) from the scene graphs of a generation and its reference extracted for \methodshort in \cref{sec:posh}, embed them using \texttt{Qwen/Qwen3-Embedding-8B} \citep{reimers-2019-sentence-bert,yang2025qwen3} and then calculate the maximum pairwise similarities between the generation components and the reference components. Generation components and reference components with maximum pairwise similarity scores lower than $0.8$ were predicted as mistakes and omissions respectively, a threshold chosen to maximize the macro F1 scores reported for \textbf{SGEmbed} in \cref{tab:docent_granular_results}.
\subsubsection{Coarse Baselines}
\label{sec:coarse_eval_baselines}
When prompting GPT4o and GPT5\footnote{\texttt{gpt-4o-2024-08-06} and \texttt{gpt-5-2025-08-07} (with minimal reasoning) accessed on 9/17/2025} to evaluate our generated detailed image descriptions, we use three different prompts depending on whether we are including the image (reference free) or including the reference. Additionally, we experiment with a more complicated prompt that includes a detailed scoring rubric for each score type (mistakes, omissions and overall quality) though we find that this setting underperforms the simpler prompts below.
\begin{tcolorbox}[llmpromptbase, title=Image Only]
\begin{lstlisting}
[IMAGE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 to 5) for the precision
(e.g. mistakes in the generated description), recall (e.g.
missing details from the image), and overall quality of the
generated detailed description compared to the image. Output your answer as a JSON dictionary with the keys
`precision', `recall', and `overall_quality'.
\end{lstlisting}
\end{tcolorbox}
\begin{tcolorbox}[llmpromptbase, title=Reference Only]
\begin{lstlisting}
Reference Detailed Description: [REFERENCE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 to 5) for the precision
(e.g. mistakes in the generated description), recall (e.g.
missing details from the reference description), and overall
quality of the generated description compared to the reference
description. Output your answer as a JSON dictionary with the
keys `precision', `recall', and `overall_quality'.
\end{lstlisting}
\end{tcolorbox}
\begin{tcolorbox}[llmpromptbase, title=Image \& Reference]
\begin{lstlisting}
[IMAGE]
Reference Detailed Description: [REFERENCE]
Generated Detailed Description: [GENERATION]
Please provide numerical scores (from 0 to 5) for the precision
(e.g. mistakes in the generated description), recall (e.g.
missing details from the reference description), and overall
quality of the generated detailed description compared to the
image and the reference description. Output your answer as a
JSON dictionary with the keys `precision', `recall', and
`overall_quality'.
\end{lstlisting}
\end{tcolorbox}
\subsubsection{Reinforcement Learning}
\label{sec:reinforcement_learning}
We train \texttt{Qwen2.5-VL-7B} on the $1,000$ images in \benchmark's training set in two settings:
\begin{enumerate}
\item supervised fine-tuning (SFT) with full parameter updates using a learning rate of $1e-5$, a linear warmup ratio of $0.1$, and an effective batch size of $64$ for $5$ epochs, choosing the checkpoint with the lowest loss on \benchmark's validation set
\item DAPO \citep{yu2025dapo} with full parameter updates, implemented with TRL \citep{vonwerra2022trl}, using a learning rate of $1e-6$, $20$ warmup steps, $8$ generations per sample (with a temperature of $1.0$ and $top_p=0.7$), $\epsilon=0.28$, $\beta=0$, and an effective batch size of $64$ for a single epoch, choosing the final checkpoint
\end{enumerate}
For both variants, we train using 8 H100 GPUs \citep{Bloom2025EmpireAI}.
We ask seven graduate students in NLP to compare and evaluate our SFT and DAPO generations (greedily sampled) for $40$ images from \benchmark's test set. Additionally, we collect three annotations for five of these images to calculate agreement.
\subsection{Results}
\subsubsection{Coarse}
\label{sec:full_coarse_results}
\begin{table*}[!ht]\centering
\setlength{\tabcolsep}{5pt}
\caption{
All coarse metrics evaluated on \benchmark and CapArena, identified with $\Theta$ (parameter count, in billions), \includegraphics[height=1em]{figures/page_with_curl.png} (requires a reference), \includegraphics[height=1em]{figures/frame_with_picture.png} (requires an image) and \includegraphics[height=1em]{figures/repeat.png} (replicable). ``acc" indicates accuracy at predicting the better generation (or ``tie") in each judged pair. For \benchmark, $\rho$ / $\tau$ indicate the Spearman rank / Kendall's $\tau$ correlations between differences in the metric and differences in the rank of the generations in each pair. For CapArena, $\rho$ / $\tau$ indicate the Spearman rank / Kendall's $\tau$ correlations between model ELO rankings derived from metric scores and human judgments. \textbf{Bold} indicates the best replicable metric while \underline{underlining} indicates the best metric overall.
\sethlcolor{gray!25}\hl{Gray cells} indicate correlations that are \textit{not} statistically significant at $\alpha = 0.05$.
\methodshort beats all replicable baselines and GPT4o on \benchmark in all settings (mistakes, omissions and overall quality) while remaining replicable. Moreover, \methodshort is robust, achieving the second best score among replicable metrics on CapArena.
}
\resizebox{\textwidth}{!}{
\begin{tabular}{l|c|ccc|ccc|ccc|ccc|ccc}\toprule
& & & & & \multicolumn{9}{c|}{\textbf{\benchmark}} & \multicolumn{3}{c}{\textbf{CapArena}} \\\midrule
& & & & & \multicolumn{3}{c}{Mistakes} &\multicolumn{3}{c}{Omissions} &\multicolumn{3}{c|}{Overall Quality} & Desc & \multicolumn{2}{c}{Model} \\
& \includegraphics[height=1em]{figures/theta.png} & \includegraphics[height=1em]{figures/page_with_curl.png} & \includegraphics[height=1em]{figures/frame_with_picture.png} & \includegraphics[height=1em]{figures/repeat.png} & acc & $\rho$ & \multicolumn{1}{c}{$\tau$} & acc &$\rho$ & \multicolumn{1}{c}{$\tau$} &acc &$\rho$ & \multicolumn{1}{c|}{$\tau$} &acc &$\rho$ & $\tau$ \\
\midrule
length & & & &$\checkmark$ & 30.5 &-0.270 &-0.206 &37.8 & -0.002 & -0.001 & 38.0 &-0.160 &-0.121 & 58.7 & 0.710 & 0.582 \\
\midrule
BLEU-4 & & $\checkmark$ & & $\checkmark$ & 34.2 & -0.070 & -0.053 & 42.5 & 0.118 & 0.087 & 42.8 & 0.051 & 0.038 & 47.4 & 0.424 & 0.319 \\
CIDER & & $\checkmark$ & & $\checkmark$ & 32.0 & -0.118 & -0.089 & 37.5 & -0.009 & -0.007 & 37.8 & -0.106 & -0.079 & 38.4 & -0.279 & -0.209 \\
METEOR & & $\checkmark$ & & $\checkmark$ & 36.0 & -0.103 & -0.078 & 46.2 & 0.260 & 0.197 & 44.8 & 0.113 & 0.084 & 57.6 & 0.785 & 0.582 \\
ROUGE-LS & & $\checkmark$ & & $\checkmark$ & 37.5 & 0.251 & 0.190 & 44.0 & 0.214 & 0.161 & 47.3 & 0.210 & 0.158 & 45.8 & 0.180 & 0.199 \\
\midrule
SPICE & & $\checkmark$ & &$\checkmark$ & 41.3 & 0.308 & 0.234 & 55.0 & 0.464 & 0.360 & 58.5 & 0.458 & 0.349 & 41.7 & 0.275 & 0.231 \\
CAPTURE & & $\checkmark$ & &$\checkmark$ & 43.3 & 0.259 & 0.194 & 53.8 & 0.447 & 0.340 & 56.0 &0.453 &0.347 & 52.5 & 0.613 & 0.538 \\
CLIPScore & & & $\checkmark$ & $\checkmark$ & 45.3 & 0.145 & 0.108 & 47.0 & 0.176 & 0.133 & 53.5 & 0.181 & 0.136 & 32.5 & -0.574 & -0.451 \\
FLEUR & 13 & & $\checkmark$ & $\checkmark$ & 35.2 & -0.053 & -0.040 & 38.5 & 0.029 & 0.020 & 41.2 & -0.040 & -0.031 & 45.8 & 0.393 & 0.297 \\
Prometheus & 8x7 &$\checkmark$ & & $\checkmark$ & 51.2 & 0.014 & 0.011 & 49.8 & 0.136 & 0.116 & 58.5 & -0.007 &-0.007 & 53.9 & 0.859 & 0.648 \\
Qwen3 & 32 & $\checkmark$ & & $\checkmark$ & 57.7 & 0.282 & 0.235 & 53.5 & 0.286 & 0.253 & 61.2 & 0.289 & 0.257 & 56.2 & 0.899 & 0.714 \\
LLaVa Critic & 72 & $\checkmark$ & $\checkmark$ & $\checkmark$ & \textbf{62.8} & 0.412 & 0.351 & 57.0 & 0.509 & 0.430 & 66.8 & 0.546 & 0.461 & \textbf{\underline{64.0}} & \textbf{\underline{0.987}} & \textbf{\underline{0.934}}\\
\midrule
DCScore & & $\checkmark$ & $\checkmark$ & & 62.8 & 0.541 & 0.422 & 54.0 & 0.395 & 0.298 & 62.8 & 0.471 & 0.362 & - & - & - \\
GPT4o & & &$\checkmark$ & & 63.2 & 0.469 & 0.400 & 55.5 & 0.338 & 0.274 & 66.7 & 0.477 & 0.393 & 53.6 & 0.868 & 0.692 \\
GPT4o & & $\checkmark$ & & & 53.3 & 0.324 & 0.261 & 50.0 & 0.277 & 0.215 & 60.8 & 0.388 & 0.297 & 56.7 & 0.867 & 0.685 \\
GPT4o & & $\checkmark$ & $\checkmark$ & & 58.5 & 0.484 & 0.396 & 56.0 & 0.380 & 0.303 & 67.3 & 0.510 & 0.402 & 55.4 & 0.890 & 0.802 \\
GPT5 & & & $\checkmark$ & & 66.0 & 0.584 & 0.476 & 55.5 & 0.454 & 0.345 & 69.2 & 0.593 & 0.466 & 56.9 & 0.916 & 0.802 \\
GPT5 & & $\checkmark$ & & & 62.5 & 0.511 & 0.423 & 53.2 & 0.421 & 0.332 & 68.0 & 0.540 & 0.440 & 59.1 & 0.956 & 0.846 \\
GPT5 & & $\checkmark$ & $\checkmark$ & & \underline{68.2} & \underline{0.604} & \underline{0.494} & 56.3 & 0.477 & 0.366 & 67.2 & \underline{0.602} & \underline{0.475} & 62.1 & 0.934 & 0.846 \\
\midrule
\methodshort & 14 & $\checkmark$ & & $\checkmark$ & 60.3 & \textbf{0.518} & \textbf{0.405} & \textbf{\underline{61.8}} & \textbf{\underline{0.578}} & \textbf{\underline{0.449}} & \textbf{\underline{70.3}} & \textbf{\underline{0.600}} & \textbf{\underline{0.466}} & 59.2 & 0.921 & 0.780 \\
\bottomrule
\end{tabular}
}
\label{tab:full_coarse_results}
\end{table*}
\begin{table*}[!ht]\centering
\setlength{\tabcolsep}{5pt}
\caption{
Annotator agreement (Krippendorff's $\alpha$) and aggregate preferences between SFT and DAPO with \methodshort. A \methodshort generation earns a score between $-2$ and $2$ based on how much worse or better it is than its SFT counterpart. Reported numbers are averages over these scores.
}
\begin{tabular}{l|c|c|c}\toprule
& Mistakes & Omissions & Overall Quality \\\midrule
$\alpha$ & $0.235$ & $0.464$ & $0.184$ \\\midrule
\methodshort vs SFT & $-0.243$ & $0.432$ & $0.135$ \\
\bottomrule
\end{tabular}
\label{tab:rl_results}
\end{table*}
\begin{figure*}
\includegraphics[width=\textwidth]{figures/docent_leaderboard.png}
\caption{Performance of open and closed VLMs on \benchmark, as measured by \methodshort. While open models are competitive when it comes to mistakes in their detailed descriptions, they lag behind in their omissions, covering less of \benchmark's reference descriptions than closed models.}
\label{fig:leaderboard}
\end{figure*}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{47}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Anderson et~al.(2016)Anderson, Fernando, Johnson, and Gould]{anderson2016spice}
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.
\newblock Spice: Semantic propositional image caption evaluation.
\newblock In \emph{Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14}, pp.\ 382--398. Springer, 2016.
\bibitem[Banerjee \& Lavie(2005)Banerjee and Lavie]{banerjee2005meteor}
Satanjeev Banerjee and Alon Lavie.
\newblock Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.
\newblock In \emph{Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization}, pp.\ 65--72, 2005.
\bibitem[Bengamra et~al.(2024)Bengamra, Mzoughi, Bigand, and Zagrouba]{bengamra2024comprehensive}
Siwar Bengamra, Olfa Mzoughi, Andr{\'e} Bigand, and Ezzeddine Zagrouba.
\newblock A comprehensive survey on object detection in visual art: taxonomy and challenge.
\newblock \emph{Multimedia Tools and Applications}, 83\penalty0 (5):\penalty0 14637--14670, 2024.
\bibitem[Berger et~al.(2024)Berger, Stanovsky, Abend, and Frermann]{berger2024surveying}
Uri Berger, Gabriel Stanovsky, Omri Abend, and Lea Frermann.
\newblock Surveying the landscape of image captioning evaluation: A comprehensive taxonomy and novel ensemble method.
\newblock \emph{arXiv preprint arXiv:2408.04909}, 2024.
\bibitem[Bloom et~al.(2025)Bloom, Brumberg, Fisk, Harrison, Hull, Ramasubramanian, Vliet, and Wing]{Bloom2025EmpireAI}
Stacie Bloom, Joshua~C. Brumberg, Ian Fisk, Robert~J. Harrison, Robert Hull, Melur Ramasubramanian, Krystyn~Van Vliet, and Jeannette Wing.
\newblock Empire {AI}: A new model for provisioning {AI} and {HPC} for academic research in the public good.
\newblock In \emph{Practice and Experience in Advanced Research Computing ({PEARC} '25)}, pp.\ ~4, Columbus, OH, USA, July 2025. ACM.
\newblock \doi{10.1145/3708035.3736070}.
\newblock URL \url{https://doi.org/10.1145/3708035.3736070}.
\bibitem[Chan et~al.(2023)Chan, Petryk, Gonzalez, Darrell, and Canny]{chan2023clair}
David Chan, Suzanne Petryk, Joseph Gonzalez, Trevor Darrell, and John Canny.
\newblock Clair: Evaluating image captions with large language models.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 13638--13646, 2023.
\bibitem[Chen et~al.(2024)Chen, Li, Dong, Zhang, Zang, Chen, Duan, Wang, Qiao, Lin, et~al.]{chenwe}
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu~Qiao, Dahua Lin, et~al.
\newblock Are we on the right way for evaluating large vision-language models?
\newblock In \emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}, 2024.
\bibitem[Cheng et~al.(2025)Cheng, Song, Fan, Ma, Sun, Xu, Yan, Chen, Zhang, and Chen]{cheng2025caparena}
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen.
\newblock Caparena: Benchmarking and analyzing detailed image captioning in the llm era.
\newblock \emph{arXiv preprint arXiv:2503.12329}, 2025.
\bibitem[Cho et~al.(2024)Cho, Hu, Baldridge, Garg, Anderson, Krishna, Bansal, Pont-Tuset, and Wang]{chodavidsonian}
Jaemin Cho, Yushi Hu, Jason~Michael Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su~Wang.
\newblock Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Deitke et~al.(2024)Deitke, Clark, Lee, Tripathi, Yang, Park, Salehi, Muennighoff, Lo, Soldaini, et~al.]{deitke2024molmo}
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae~Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et~al.
\newblock Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.
\newblock \emph{CoRR}, 2024.
\bibitem[Deutsch et~al.(2022)Deutsch, Dror, and Roth]{deutsch2022limitations}
Daniel Deutsch, Rotem Dror, and Dan Roth.
\newblock On the limitations of reference-free evaluations of generated text.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing}, pp.\ 10960--10977, 2022.
\bibitem[Dong et~al.(2024)Dong, Li, Wu, Wang, Zhang, and Guo]{dong2024benchmarking}
Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo.
\newblock Benchmarking and improving detail image caption.
\newblock \emph{arXiv preprint arXiv:2405.19092}, 2024.
\bibitem[Garg et~al.(2024)Garg, Burns, Karagol~Ayan, Bitton, Montgomery, Onoe, Bunner, Krishna, Baldridge, and Soricut]{garg-etal-2024-imageinwords}
Roopal Garg, Andrea Burns, Burcu Karagol~Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason~Michael Baldridge, and Radu Soricut.
\newblock {I}mage{I}n{W}ords: Unlocking hyper-detailed image descriptions.
\newblock In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pp.\ 93--127, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.emnlp-main.6}.
\newblock URL \url{https://aclanthology.org/2024.emnlp-main.6/}.
\bibitem[Hessel et~al.(2021)Hessel, Holtzman, Forbes, Le~Bras, and Choi]{hessel2021clipscore}
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le~Bras, and Yejin Choi.
\newblock Clipscore: A reference-free evaluation metric for image captioning.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pp.\ 7514--7528, 2021.
\bibitem[Holtzman et~al.(2019)Holtzman, Buys, Du, Forbes, and Choi]{holtzmancurious}
Ari Holtzman, Jan Buys, Li~Du, Maxwell Forbes, and Yejin Choi.
\newblock The curious case of neural text degeneration.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[Honnibal et~al.(2020)Honnibal, Montani, Van~Landeghem, and Boyd]{spacy}
Matthew Honnibal, Ines Montani, Sofie Van~Landeghem, and Adriane Boyd.
\newblock spacy: Industrial-strength natural language processing in python, 2020.
\bibitem[Hripcsak \& Rothschild(2005)Hripcsak and Rothschild]{hripcsak2005agreement}
George Hripcsak and Adam~S Rothschild.
\newblock Agreement, the f-measure, and reliability in information retrieval.
\newblock \emph{Journal of the American medical informatics association}, 12\penalty0 (3):\penalty0 296--298, 2005.
\bibitem[Jain et~al.(2023)Jain, Li, Chiu, Hassani, Orlov, and Shi]{jain2023oneformer}
Jitesh Jain, Jiachen Li, Mang~Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi.
\newblock Oneformer: One transformer to rule universal image segmentation.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 2989--2998, 2023.
\bibitem[Jing et~al.(2024)Jing, Li, Chen, and Du]{jing2024faithscore}
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du.
\newblock Faithscore: Fine-grained evaluations of hallucinations in large vision-language models.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2024}, pp.\ 5042--5063, 2024.
\bibitem[Kasai et~al.(2022)Kasai, Sakaguchi, Dunagan, Morrison, Le~Bras, Choi, and Smith]{kasai-etal-2022-transparent}
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le~Bras, Yejin Choi, and Noah~A. Smith.
\newblock Transparent human evaluation for image captioning.
\newblock In Marine Carpuat, Marie-Catherine de~Marneffe, and Ivan~Vladimir Meza~Ruiz (eds.), \emph{Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, pp.\ 3464--3478, Seattle, United States, July 2022. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2022.naacl-main.254}.
\newblock URL \url{https://aclanthology.org/2022.naacl-main.254/}.
\bibitem[Kim et~al.(2023)Kim, Shin, Cho, Jang, Longpre, Lee, Yun, Shin, Kim, Thorne, et~al.]{kim2023prometheus}
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et~al.
\newblock Prometheus: Inducing fine-grained evaluation capability in language models.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2023.
\bibitem[Kocmi et~al.(2021)Kocmi, Federmann, Grundkiewicz, Junczys-Dowmunt, Matsushita, and Menezes]{kocmi2021ship}
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes.
\newblock To ship or not to ship: An extensive evaluation of automatic metrics for machine translation.
\newblock In \emph{Proceedings of the Sixth Conference on Machine Translation}, pp.\ 478--494, 2021.
\bibitem[Kreiss et~al.(2022)Kreiss, Bennett, Hooshmand, Zelikman, Morris, and Potts]{kreiss2022context}
Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith~Ringel Morris, and Christopher Potts.
\newblock Context matters for image descriptions for accessibility: Challenges for referenceless evaluation metrics.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing}, pp.\ 4685--4697, 2022.
\bibitem[Landis \& Koch(1977)Landis and Koch]{landis1977measurement}
J~Richard Landis and Gary~G Koch.
\newblock The measurement of observer agreement for categorical data.
\newblock \emph{biometrics}, pp.\ 159--174, 1977.
\bibitem[Lee et~al.(2024)Lee, Park, and Kang]{lee2024fleur}
Yebin Lee, Imseong Park, and Myungjoo Kang.
\newblock Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 3732--3746, 2024.
\bibitem[Li et~al.(2025)Li, Wu, Du, Nghiem, and Shi]{li2025benchmark}
Zongxia Li, Xiyang Wu, Hongyang Du, Huy Nghiem, and Guangyao Shi.
\newblock Benchmark evaluations, applications, and challenges of large vision language models: A survey.
\newblock \emph{arXiv preprint arXiv:2501.02189}, 2025.
\bibitem[Lin(2004)]{lin2004rouge}
Chin-Yew Lin.
\newblock Rouge: A package for automatic evaluation of summaries.
\newblock In \emph{Text summarization branches out}, pp.\ 74--81, 2004.
\bibitem[Liu et~al.(2024)Liu, Li, Li, and Lee]{liu2024improved}
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong~Jae Lee.
\newblock Improved baselines with visual instruction tuning.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 26296--26306, 2024.
\bibitem[Liu et~al.(2023)Liu, Iter, Xu, Wang, Xu, and Zhu]{liu2023g}
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
\newblock G-eval: Nlg evaluation using gpt-4 with better human alignment.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pp.\ 2511--2522, 2023.
\bibitem[Lu et~al.(2025)Lu, Wu, Zheng, Ma, Gong, Liu, Zhai, Cao, Shen, and Zha]{lu2025benchmarking}
Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, and Zheng-Jun Zha.
\newblock Benchmarking large vision-language models via directed scene graph for comprehensive image captioning.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pp.\ 19618--19627, 2025.
\bibitem[Mack et~al.(2021)Mack, McDonnell, Jain, Lu~Wang, E.~Froehlich, and Findlater]{mack2021we}
Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu~Wang, Jon E.~Froehlich, and Leah Findlater.
\newblock What do we mean by “accessibility research”? a literature survey of accessibility papers in chi and assets from 1994 to 2019.
\newblock In \emph{Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems}, pp.\ 1--18, 2021.
\bibitem[Martinelli et~al.(2024)Martinelli, Barba, and Navigli]{martinelli-etal-2024-maverick}
Giuliano Martinelli, Edoardo Barba, and Roberto Navigli.
\newblock Maverick: Efficient and accurate coreference resolution defying recent trends.
\newblock In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 13380--13394, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2024.acl-long.722}.
\newblock URL \url{https://aclanthology.org/2024.acl-long.722/}.
\bibitem[Novikova et~al.(2018)Novikova, Du{\v{s}}ek, and Rieser]{novikova2018rankme}
Jekaterina Novikova, Ond{\v{r}}ej Du{\v{s}}ek, and Verena Rieser.
\newblock Rankme: Reliable human ratings for natural language generation.
\newblock In \emph{Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)}, pp.\ 72--78, 2018.
\bibitem[Onoe et~al.(2024)Onoe, Rane, Berger, Bitton, Cho, Garg, Ku, Parekh, Pont-Tuset, Tanzer, et~al.]{onoe2024docci}
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et~al.
\newblock Docci: Descriptions of connected and contrasting images.
\newblock In \emph{European Conference on Computer Vision}, pp.\ 291--309. Springer, 2024.
\bibitem[Papineni et~al.(2002)Papineni, Roukos, Ward, and Zhu]{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In \emph{Proceedings of the 40th annual meeting of the Association for Computational Linguistics}, pp.\ 311--318, 2002.
\bibitem[Reimers \& Gurevych(2019)Reimers and Gurevych]{reimers-2019-sentence-bert}
Nils Reimers and Iryna Gurevych.
\newblock Sentence-bert: Sentence embeddings using siamese bert-networks.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing}. Association for Computational Linguistics, 11 2019.
\newblock URL \url{http://arxiv.org/abs/1908.10084}.
\bibitem[Sarto et~al.(2023)Sarto, Barraco, Cornia, Baraldi, and Cucchiara]{sarto2023positive}
Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.
\newblock Positive-augmented contrastive learning for image and video captioning evaluation.
\newblock In \emph{Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, pp.\ 6914--6924, 2023.
\bibitem[Scialom et~al.(2021)Scialom, Dray, Lamprier, Piwowarski, Staiano, Wang, and Gallinari]{scialom2021questeval}
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari.
\newblock Questeval: Summarization asks for fact-based evaluation.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pp.\ 6594--6604, 2021.
\bibitem[See et~al.(2017)See, Liu, and Manning]{see2017get}
Abigail See, Peter~J Liu, and Christopher~D Manning.
\newblock Get to the point: Summarization with pointer-generator networks.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 1073--1083, 2017.
\bibitem[Urbanek et~al.(2024)Urbanek, Bordes, Astolfi, Williamson, Sharma, and Romero-Soriano]{urbanek2024picture}
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano.
\newblock A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 26700--26709, 2024.
\bibitem[Vedantam et~al.(2015)Vedantam, Lawrence~Zitnick, and Parikh]{vedantam2015cider}
Ramakrishna Vedantam, C~Lawrence~Zitnick, and Devi Parikh.
\newblock Cider: Consensus-based image description evaluation.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 4566--4575, 2015.
\bibitem[von Werra et~al.(2020)von Werra, Belkada, Tunstall, Beeching, Thrush, Lambert, Huang, Rasul, and Gallouédec]{vonwerra2022trl}
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec.
\newblock Trl: Transformer reinforcement learning.
\newblock \url{https://github.com/huggingface/trl}, 2020.
\bibitem[Xiong et~al.(2025)Xiong, Wang, Guo, Ye, Fan, Gu, Huang, and Li]{xiong2024llavacritic}
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li.
\newblock Llava-critic: Learning to evaluate multimodal models.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pp.\ 13618--13628, 2025.
\bibitem[Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, et~al.]{yang2025qwen3}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et~al.
\newblock Qwen3 technical report.
\newblock \emph{arXiv preprint arXiv:2505.09388}, 2025.
\bibitem[Ye et~al.(2025)Ye, Zeng, Li, Li, and Fan]{ye2025painting}
Qinghao Ye, Xianhan Zeng, Fu~Li, Chunyuan Li, and Haoqi Fan.
\newblock Painting with words: Elevating detailed image captioning with benchmark and alignment learning.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2025.
\bibitem[Yu et~al.(2025)Yu, Zhang, Zhu, Yuan, Zuo, Yue, Dai, Fan, Liu, Liu, et~al.]{yu2025dapo}
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu~Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et~al.
\newblock Dapo: An open-source llm reinforcement learning system at scale.
\newblock \emph{arXiv preprint arXiv:2503.14476}, 2025.
\bibitem[Zheng et~al.(2023)Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, et~al.]{zheng2023judging}
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi~Lin, Zhuohan Li, Dacheng Li, Eric Xing, et~al.
\newblock Judging llm-as-a-judge with mt-bench and chatbot arena.
\newblock \emph{Advances in Neural Information Processing Systems}, 36:\penalty0 46595--46623, 2023.
\end{thebibliography}