SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context
Extracted source text · arXiv 2607.27084
%File: arxiv_scifigqual.tex
% Merged main paper + appendix for arXiv.
% Single bibliography AFTER the main text and BEFORE the appendix,
% so Google Scholar / Semantic Scholar attribute citations to the main article.
%
% Recommended compile (from paper/):
% pdflatex arxiv_scifigqual && bibtex arxiv_scifigqual && pdflatex arxiv_scifigqual && pdflatex arxiv_scifigqual
% Or from AAAI2027/ with paper/ on the TeX input path (sty/bib/figures).
\documentclass[letterpaper]{article}
% preprint: show authors, suppress AAAI first-page copyright footer
\usepackage[preprint]{aaai2027}
\usepackage{times}
\usepackage{helvet}
\usepackage{courier}
\usepackage[hyphens]{url}
\usepackage{graphicx}
\urlstyle{rm}
\def\UrlFont{\rm}
\usepackage{natbib}
\usepackage{caption}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{array}
\usepackage[table]{xcolor}
\usepackage{pifont}
\usepackage{fontawesome5}
\newcommand{\cmark}{\ding{51}}
\newcommand{\xmark}{\ding{55}}
\newcommand{\ptmark}{$\triangle$}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{adjustbox}
\usepackage{placeins}
\usepackage{algorithm}
\usepackage{algpseudocode}
% cuted.sty (\\begin{strip}) is optional; we rewrite strips to figure* below.
\frenchspacing
\setlength{\pdfpagewidth}{8.5in}
\setlength{\pdfpageheight}{11in}
\graphicspath{{./}{appendix_figures/}}
\makeatletter
\def\input@path{{./}}
\makeatother
\setcounter{secnumdepth}{3}
\pdfinfo{
/TemplateVersion (2026.1)
}
\title{SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context}
% Clickable project-page link under affiliations (pdflatex / AAAI-compatible).
\newcommand{\projectpagelink}{%
\leavevmode
\pdfstartlink
attr{/Border [0 0 0]}
user{/Subtype /Link /A << /S /URI /URI (https://frankdengai.github.io/SciFigQual-Bench/) >>}%
\faGlobe\hspace{0.35em}\textit{Project Page}%
\pdfendlink
}
\author{
Zihan Deng$^{1\dagger}$\thanks{Equal contribution.}\thanks{Email: zhdeng@hku.hk, chuanzhi.xu@sydney.edu.au.},
Chuanzhi Xu$^{2\dagger}$\footnotemark[1],
Huiqi Liang$^{2}$,
Haoyang Li$^{2}$,
Xiaozhen Zhong$^{3}$,
Lequan Yu$^{1}$
}
\affiliations{
$^1$ The University of Hong Kong\\
$^2$ The University of Sydney\\
$^3$ University of Electronic Science and Technology of China\\
\projectpagelink
}
\newcommand{\apptab}{Appendix Table}
\newcommand{\apxcaption}[1]{\caption{\apptab~\thetable: #1}}
\newcommand{\nscore}{--}
\begin{document}
\maketitle
\begin{abstract}
Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers.
However, existing image quality assessment (IQA) are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness.
To address this, we propose \textbf{SciFigQual-Bench}, a full-text contextual benchmark that evaluates scientific image across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers the top computer-science conferences from 2020 to 2025, 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into the gold-standard annotations.
Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context.
To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation \textbf{SFQ-Agent} to achieve auditable and refined scoring through the collection and fusion of modal evidence.
Multiple mainstream large models were evaluated on the test subset \textit{eval1200}, and the SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest $\pm$1-point consistency rate (93.4\%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.
Project page: \url{https://frankdengai.github.io/SciFigQual-Bench/}
(code: \url{https://github.com/FrankDengAI/SciFigQual-Bench}).
\end{abstract}
\section{Introduction}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Fig1}
\caption{From isolated-figure evaluation to full-manuscript context binding. Prior paradigms evaluate figures without manuscript evidence, whereas SciFigQual-Bench grounds assessment in figure-relevant text and scores five orthogonal dimensions on a unified 1--10 scale.}
\label{fig:gap}
\end{figure}
\begin{table*}[htbp]
\centering
\small
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{0.85}
\setlength{\extrarowheight}{0pt}
\begin{tabular}{@{}lcccccc@{}}
\toprule
\textbf{Comparison dimension} & \textbf{SIQA} & \textbf{SCIEval} & \textbf{VisJudge} & \textbf{SPUR} & \textbf{GENFIG1} & \textbf{Ours} \\
\midrule
Real figures from published CS papers & \xmark & \xmark & \ptmark & \ptmark & \xmark & \cmark \\
Source PDF + section binding & \xmark & \xmark & \xmark & \ptmark & \ptmark & \cmark \\
Full-text citing paragraphs (index-driven) & \xmark & \xmark & \xmark & \ptmark & \xmark & \cmark \\
Evaluation dimensions (multi-dim.\ rubric) & \cmark & \cmark & \cmark & \ptmark & \cmark & \cmark \\
End-to-end Acquire$\rightarrow$Annotate pipeline & \xmark & \ptmark & \xmark & \xmark & \ptmark & \cmark \\
Learnable multi-dimensional regressor & \ptmark & \cmark & \xmark & \xmark & \xmark & \cmark \\
Auditable scoring rationale & \ptmark & \ptmark & \ptmark & \xmark & \ptmark & \cmark \\
Multi-venue $\times$ multi-year (CS top-tier) & \ptmark & \xmark & \ptmark & \ptmark & \ptmark & \cmark \\
\midrule
\textbf{\# Images (test bench.)} & 2,240 & 6,000 & 3,090 & 1,084 & 584 & \textbf{7,609} \\
\textbf{\# Text items} & $\approx$2.2k & $\approx$6.0k & $\approx$3.1k & $\approx$4.3k & $\approx$584 & \textbf{$\approx$355k} \\
\textbf{\# Papers (crawl scope)} & -- & -- & -- & $\approx$5k & $\approx$584 & \textbf{$\approx$63.6k} \\
\bottomrule
\end{tabular}
% , -- 标题与表注合并 , --
\caption{Capability and scale comparison against related benchmarks. Symbols: \cmark=fully supported; \ptmark=partial; \xmark=not targeted.}
\label{tab:positioning}
\end{table*}
Natural images arise from light, shadows, and objects in the real world, and their evaluation criteria mostly remain at the level of human visual perception. In Image Quality Assessment (IQA), for example, a landscape photo is often evaluated from the aspects of natural color, clarity, aesthetic perception, etc. There is no "right or wrong" standard at the content level ~\cite{li2025distilling}. These types of images are often unrelated to language and text, and can be evaluated solely based on the image itself ~\cite{chen2024topiq}.
However, scientific images are naturally bound to the complete academic text context, and they must be combined with descriptions, specific data, or conclusions in order to have evaluative value~\cite{tufte2001,cleveland1984}.
If there are errors in the figures in a scientific paper that are closely related to the data or conclusions, such as axis truncation, missing baseline data, or inconsistencies between the data in the figures and the description in the text, visual inspection at the pixel level alone cannot identify such defects.
In scientific paper peer review, quality assessment of scientific paper is therefore inherently tri-modal: it depends on what is visible in image, what is claimed in caption, and what the manuscript asserts in citing paragraphs.
However, existing paradigms for natural IQA and AIGC alignment operate on a single visual modality, inherently decoupling the image from both its caption and the citing manuscript context.
As illustrated in Figure~\ref{fig:gap}, four mainstream paradigms highlight this gap.
\textbf{Natural IQA}~\cite{brisque2012,niqe2013,pieapp2018,clipiqa2022} evaluates perceptual image quality relying on low-level statistics.
\textbf{AIGC alignment}~\cite{clip2018,imagereward2023,hpsv22023,tifa2023,pickapic2023} measures text-image similarity for generated content.
\textbf{Schematic understanding}~\cite{chartqa2022,plotqa2020,dvqa2018,deplot2023,unichart2023} uses the model to implement image classification recognition and visual question answering functions, and the model's judgment results cannot be directly associated with the quoted paragraphs in the original PDF document.
\textbf{Figure reasoning QA}~\cite{figureseer2016,docvqa2021} tests whether a model can read a figure but does not offer a rubric for holistic quality or misleading risk.
In each case, evaluation proceeds on a detached visual input, decoupled from captions, citing paragraphs, and full-text context.
In this paper, we propose SciFigQual-Bench, an evidence-grounded staged judging framework for evaluating human-published scientific images under authentic full-manuscript context. We binding each figure image with its caption and the citing paragraph set from the source PDF to form complete experimental context (CTX). Instead of adopting a single end-to-end vision-language model (VLM) prompt, we employ an evidence-grounded staged judging process: first collect multi-modal evidence from each component, then perform cross-modal fusion, and finally output traceable multi-dimensional scores covering Visual Clarity (VC), Structure Layout (SL), Caption Consistency (CC), Misleading Risk (MR), and Context Consistency (CTX). Unlike faithfulness benchmarks for AI-generated scientific images~\cite{scieval2026,arxivcap2024,scicap2021}, our task evaluates human-published figures in a real manuscript context, forming a complementary evaluation objective.
Our contributions can be summarized as follows:
\begin{itemize}
\item \textbf{First}, we release \textbf{SciFigQual-Bench}: 7,609 curated figures from 1,144 qualified papers, 6,308 expert-aggregated gold instances, and index-driven context records spanning top-tier conference papers (2020--2025).
\item \textbf{Second}, we propose \textbf{SFQ-Agent}, a staged cross-modal judge that operationalizes full-manuscript figure evaluation by binding figure, caption, and citing text as tri-modal evidence, fusing vision-language for text-grounded dimensions, and aggregating deterministic rubric-aligned scores to ensure each dimension is traceable to structured evidence rather than an opaque VLM verdict.
\item \textbf{Third}, on the fixed public test subset \textit{eval1200} ($n{=}1{,}200$), we benchmark a broad suite of state-of-the-art vision--language models against human gold labels, establishing baselines and showing that SFQ-Agent consistently outperforms direct and auxiliary judging schemes while revealing caption consistency and context consistency as the dominant human--model disagreement axes.
\end{itemize}
\section{Related Work}
\label{sec:related}
Existing research that can be applied in scientific figure quality assessment and multimodal figure-text evaluation falls into four major categories: perceptual image quality evaluation, chart comprehension, text-to-figure generation and multimodal corpus construction, and LLM-based automated evaluation frameworks.
\noindent \textbf{Perceptual IQA and isolated image quality.}
Classical no-reference IQA (BRISQUE, NIQE) and learned perceptual metrics (PieAPP, CLIP-IQA) estimate quality from low-level statistics or deep features~\cite{brisque2012,niqe2013}; \textbf{SIQA} defines a four-dimensional rubric for schematic scientific figures and trains learned regressors on expert ratings, yet its instances are not drawn from published CS PDFs and carry no index-resolved citing paragraphs~\cite{siqa2026}.
\textbf{VisJudge-Bench} employs MLLM judges for visualization aesthetics and quality on chart and diagram crops~\cite{visjudge2025}, but likewise omits source-PDF binding and manuscript-level CC/CTX.
However, these approaches are confined to visual-only quality assessment on isolated crops, and none grounds its scores on the full manuscript context or caption consistency.
\noindent \textbf{Chart and scientific figure understanding.}
ChartQA and SciFIBench benchmark chart reasoning and multiple-choice scientific figure interpretation~\cite{chartqa2022,scifibench2024}.
\textbf{SPUR} extends to biomedical experimental images with perception--understanding--reasoning QA and partial figure--text linkage~\cite{spur2026}, yet targets VQA accuracy rather than a calibrated multi-dimensional quality rubric and covers neither CS top-tier venues nor index-driven citing paragraphs at our scale.
These benchmarks focus on the analysis of scientific images, but lack a multidimensional quality assessment system: they cannot measure the matching degree between chart titles and images, the consistency between charts and the context of the text, nor can they quantify the risk of misleading information in computer conference official charts.
\noindent \textbf{Generation faithfulness}
\noindent \textbf{Generation faithfulness.}
\textbf{SCIEval} trains and benchmarks CLIP/LMM modules on multi-dimensional faithfulness of generated scientific images, which covering relevance, accuracy, and interpretability in text-to-image and captioning settings~\cite{scieval2026}.
\textbf{GENFIG1} focuses on visual summaries automatically produced for scientific papers, challenging VLMs to judge whether such generated overviews faithfully reflect paper content~\cite{genfig12026}. At only hundreds of instances, it targets quality control of \emph{newly synthesized} academic illustrations rather than retrospective assessment of human-authored conference figures.
These resources support generation or retrieval, but extraction and captioning alone do not certify figure quality in peer-review terms.
\noindent \textbf{LMM-as-judge.}
Rubric-conditioned VLM evaluators (Prometheus-Vision, VIEScore) enable fine-grained multimodal scoring~\cite{prometheus2024,viescore2024}; evidence-grounded NLP cautions against plausible but unfaithful rationales~\cite{faithfulnlp2020}.
Monolithic judges conflate visual perception with textual verification and hallucinate evidence when scoring CC, CTX, and MR jointly.
SciFigQual‑Bench links each figure to its citing paragraphs resolved from the PDF, gathers modality‑specific evidence stepwise, and fuses them only after these stages. It is designed for real published figures, not isolated crops, synthetic substitutes, or multiple‑choice interpretation tasks. Table~\ref{tab:positioning} compares our benchmark with the five closest 2024–2026 efforts in terms of capability and data scale.
\section{SciFigQual-Bench}
\label{sec:method}
This section first provides a formal definition of benchmark instances and a five dimensional scoring scale, and then introduces the data construction process and the SFQ-Agent phased scoring framework.
\subsection{Task Formulation and Rubric}
\label{sec:task}
The benchmark instance is a tuple $(I, c, \mathcal{T}, m)$ where $I$ is the figure crop, $c$ the caption (possibly empty), $\mathcal{T}$ citing paragraphs (possibly empty), and $m$ lightweight metadata (venue, year, figure index, section tags).
Instances are always anchored to a source PDF so that $\mathcal{T}$ is resolved by figure index rather than layout heuristics alone.
Human experts and SFQ-Agent assign scores on five \textbf{orthogonal} dimensions, each on $[1,10]$:
\textbf{Visual Clarity (VC)}, legibility of text, marks, and encoding at publication scale;
\textbf{Structure \& Layout (SL)}, composition, panel organization, and chartjunk~\cite{tufte2001};
\textbf{Caption Consistency (CC)}, alignment between $c$ and visible content, including subpanels and trends;
\textbf{Context Consistency (CTX)}, alignment between evidentiary claims in $\mathcal{T}$ and what $I$ supports;
\textbf{Misleading Risk (MR)}, likelihood of reader misinterpretation (higher score = lower risk), including truncated axes or undisclosed baselines~\cite{vlmviz2025}.
Overall quality is the mean over available dimensions, enabling fair comparison when captions or citing text are missing.
Let $\mathcal{D}=\{\mathrm{VC},\mathrm{SL},\mathrm{CC},\mathrm{CTX},\mathrm{MR}\}$ and $g_i(d)\in\{0,1\}$ indicate whether dimension $d$ is evaluable for instance $i$ under L1 gating.
Human and model outputs are vectors $\mathbf{s}_i,\hat{\mathbf{s}}_i\in[1,10]^{|\mathcal{D}|}$.
The gated overall label is
\begin{equation}
\begin{split}
y_i &= \frac{1}{\sum_{d\in\mathcal{D}} g_i(d)}
\sum_{d\in\mathcal{D}} g_i(d)\,s_i(d),
\end{split}
\label{eq:overall}
\end{equation}
with $y_i$ undefined only when both $c$ and $\mathcal{T}$ are absent (the instance is excluded).
If $c{=}\emptyset$, CC is marked null ($g_i(\mathrm{CC}){=}0$); if $\mathcal{T}{=}\emptyset$, CTX is null; instances lacking both are excluded from scoring.
This L1 evidence gating prevents penalizing missing metadata as low quality~\cite{krippendorff2011} and mirrors annotation practice in SciCap+~\cite{scicapplus2023}.
MR may still be scored from visual hazards when text is partial.
\subsection{Benchmark Construction}
\label{sec:construction}
Figure~\ref{fig:pipeline} summarizes our pipeline over CS papers from ACL, EMNLP, ICML, and NeurIPS (2020--2025).
We formalize the released benchmark as
$\mathcal{B}=\{(I_j,c_j,\mathcal{T}_j,m_j)\}_{j=1}^{N}$ with $N{=}7{,}609$ figures from $P{=}1{,}144$ qualified papers, of which $N_{\mathrm{rat}}{=}6{,}308$ carry human five-dimensional scores.The remaining figures retain image--caption--context bindings for protocol development and future labeling, but are excluded from gold-based evaluation on \textit{eval1200}.
Each module below is deterministic given source PDFs and fixed preprocessing seeds.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Fig2}
\caption{SciFigQual-Bench construction pipeline: corpus collection, structure-aware extraction, context binding, five-dimensional rubric, and expert validation.}
\label{fig:pipeline}
\end{figure}
\noindent \textbf{Step 1: Corpus acquisition.} We acquire the corpus from multiple venues: we crawl PDFs and bibliographic metadata from OpenReview~\cite{openreview2018}, ACL Anthology~\cite{aclanthology2020}, and PMLR, tagging each file with $(\mathit{venue},\mathit{year},\mathit{paper\_id})$. Next, we normalize PDFs by linearizing, stripping publisher watermarks where possible, and hashing by content fingerprint $h(p)=\mathrm{SHA256}(\mathrm{bytes}(p))$ to remove exact duplicates. Finally, we apply venue--year indexing, discarding records that fail year filters (outside 2020--2025) or missing parseable metadata. The surviving raw corpus is
\begin{equation}
\mathcal{C}_0=\{p_k\}_{k=1}^{K_0},\quad K_0=62{,}694,
\label{eq:corpus}
\end{equation}
subsuming the scale of DocFigure and SciFIBench sources~\cite{docfigure2019,scifibench2024,multimodalarxiv2024} but curated for quality evaluation rather than MCQ or type classification.
\noindent \textbf{Step 2: Structure-aware figure extraction.} We rasterize each PDF at 300\,DPI and parse its layout with Marker, obtaining text blocks, figure bounding boxes, and caption candidates. For each figure, we rank candidate captions using spatial overlap, vertical distance, and a figure-caption cue~\cite{figex2025}, selecting the best match above a threshold. The extracted pairs then undergo a five-stage curation pipeline that removes duplicates and non-figures, filters anomalous crops, recovers clipped panels, flags borderline cases, and discards irrecoverable layouts. This yields a clean set of figure--caption pairs from qualified papers.
\noindent \textbf{Step 3: Index-driven context binding.} We detect figure references in the body text using PyMuPDF~\cite{pymupdf2024} by matching index patterns anchored to figure indices. For each matched reference, we collect its hosting paragraph and merge multi-sentence windows within the same section, yielding per-figure context sets $\mathcal{T}_k = \bigcup_{\tau\in\text{match}(k)} u(\tau)$, each capped at $L_{\max}$ tokens. This corpus-wide process produces approximately 355,000 citing-paragraph records—resolved directly from the PDF index rather than relying on abstract or introduction heuristics, distinguishing our approach from caption-only corpora~\cite{s1mmalign2026,multimodalarxiv2024}. Finally, we apply a quality filter that down-weights paragraphs lacking explicit figure-specific claims and retains those with comparative or trend language for subsequent CTX annotation.
\noindent \textbf{Step 4: Five-dimensional human annotation.} We first calibrated domain experts with a 50-figure pilot using the rubric in appendix A.1. During formal annotation, each instance received scores under Eq.~\eqref{eq:overall} with L1 gating—CC and CTX were hidden whenever the necessary evidence was absent. Following faithful-NLP guidelines~\cite{faithfulnlp2020}, annotators had to justify every score with specific visible marks, caption phrases, or citing sentences; any rationale lacking evidence was sent back for revision. To ensure quality, we performed dual annotation on an ACL 2025 holdout and measured inter-rater agreement; disagreements exceeding two points triggered adjudication. The final set comprises $6{,}308$ rated figures with venue-balanced splits.
\noindent \textbf{Step 5: Release and evaluation packaging.} We release all rated instances, context bundles $\mathcal{T}$, and model-score outputs as a unified JSONL dataset. To support fair and reproducible cross-protocol comparisons, we construct a fixed public test split, \textit{eval1200}, containing 1,200 stratified instances—balanced by venue and figure type. For evaluation, we implement three judge protocols (Direct, Sidecar, and SFQ-Agent), all using the same input fields $(I,c,\mathcal{T},m)$ and outputting scores in the same five-dimensional schema; any performance gap thus reflects the judgment strategy itself, not differences in data representation.
\subsection{Dataset Statistics}
\label{sec:stats}
Figure~\ref{fig:stats} summarizes the scale, diversity, and quality distribution of our dataset relative to prior scientific-figure benchmarks.
\textbf{(a)} Radar coverage spans 18 research categories across four venues, covering broader CS topics than chart-only sets like CharXiv~\cite{charxiv2024}.
\textbf{(b)} The funnel yields $P{=}1{,}144$ qualified papers and $N{=}7{,}609$ clean figures from $K_0{=}62{,}694$ raw PDFs (Eq.~\eqref{eq:corpus}); 6,308 are human-rated with balanced venue splits, surpassing SIQA-U (2,240) and SPUR (1,084) in comparable settings~\cite{siqa2026,spur2026}. The overall yield $\eta=N/K_0\approx 12.1\%$ reflects our strict curation standards.
\textbf{(c)} Overall scores center near 8.05, indicating solid quality with sufficient variance for effective discrimination.
\textbf{(d)} The largest share of rated figures comes from NLP (2,813), followed by ML (1,891) and CV (873).
\textbf{(e)} Temporal coverage peaks in 2024--2025 with over 1,400 rated figures per year, supporting evaluation on modern plotting styles.
\textbf{(f)} Mean dimension scores show CC (7.42) as the weakest axis versus SL (8.58), confirming that caption--figure mismatches—rather than low-level blur—are the dominant real-world defects, which motivates our manuscript-grounded CC and CTX evaluation.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{Fig3}
\caption{SFQ-Agent scoring pipeline: L1 gating, parallel vision and language evidence modules, cross-modal judge, and deterministic Runner aggregation. Direct and Sidecar judges serve as ablations.}
\label{fig:agent}
\end{figure*}
\subsection{SFQ-Agent}
\label{sec:agent}
Based on the previous statistics, the main bottleneck is the consistency of captions and context, which makes it difficult for a single visual language model to perform end-to-end scoring. To this end, we propose SFQ Agent: under a unified five dimensional scale, collect visual and linguistic evidence in stages and perform cross modal fusion to achieve auditable and refined decision-making.
SFQ-Agent (Figure~\ref{fig:agent}) implements evidence-grounded judging in four stages, designed as an auditable alternative to monolithic LMM-as-judge pipelines~\cite{llm_judge2023,prometheus2024,viescore2024}.
\noindent \textbf{Step 0: Input and gating.}
The system loads $(I, c, \mathcal{T})$; figures without caption and citing text are discarded; partial evidence triggers dimension hiding.
\noindent \textbf{Step 1: Vision evidence.} We extract visual evidence from each figure via PaddleOCR-VL (a compact OCR model) and classical CV descriptors (blur, edge density, color histogram). The OCR module supplies text regions, confidence scores, and layout cues; we also derive side features such as text density, caption-token overlap, and numeric consistency. These are packaged as \texttt{visual\_facts} together with axis units and legend notes to support VC and SL scoring. This parallels low-level perceptual auditing in prior work~\cite{q_instruct2023,vlmviz2025}, but our structured outputs are explicitly designed for downstream fusion. The language module never accesses pixels, ensuring clean modality separation.
\noindent \textbf{Step 2: Language evidence.} To complement the vision track, an LLM reads only the caption, citing paragraphs, and metadata, without any pixel input. It extracts textual evidence, implied claims, and rule violations to support CC and CTX scoring. This strict separation prevents the LLM from generating plausible but unfaithful rationales, a documented risk in prior work~\cite{faithfulnlp2020}.
\noindent \textbf{Step 3: Cross-modal judge.} Once both tracks have produced structured evidence, the cross-modal judge fuses them to score CC, CTX, and MR. Specifically, it aligns visual hazard indicators with textual cues and detects conflicts, such as a figure showing an upward trend while the text describes degradation. VC and SL are copied from Stage~1 rather than re-estimated from text summaries, so fluent language evidence cannot re-inflate visual scores.
\noindent \textbf{Step 4: Runner.} Finally, a deterministic post-processor locks VC and SL to the vision module outputs, imposes rule-based CC/CTX/MR caps informed by visualization guidelines~\cite{vlmviz2025,tufte2001}, and aggregates all available dimensions under L1 gating. This design prevents score inflation from unconstrained LLM arbitration and keeps each final score traceable to its originating evidence track. Frozen prompt contracts and the MR fusion mapping are reported in the supplementary appendix C.
\noindent \textbf{Judge protocols.} We evaluate three judge protocols along a spectrum of increasing complexity. The \textbf{Direct Judge} serves as the simplest baseline, issuing a single VLM prompt over $(I, c, \mathcal{T})$—similar to zero-shot judging in Prometheus-Vision and Q-Align~\cite{prometheus2024,qalign2024}. The \textbf{Sidecar Judge} augments this setup with PaddleOCR-VL and CV side features within the same single-pass call, testing whether additional visual cues alone bring gains. The \textbf{SFQ-Agent} implements our full staged pipeline, reusing the same feature bundle across three sequential calls (Stage~1 for vision, Stage~2 for language, Stage~3 for fusion). Together, these variants isolate two design choices (OCR‑VL side signals versus explicit evidence staging), especially on CC and CTX, where monolithic judges most often conflate perception with text verification.
\section{Experiments and Results}
\label{sec:experiments}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{Fig4}
\caption{Corpus overview of SciFigQual-Bench. From $62{,}694$ PDFs we retain $7{,}609$ figures ($6{,}308$ human-rated) across ACL/EMNLP/ICML/NeurIPS (2020--2025), with NLP/ML/CV as the dominant domains. Overall scores concentrate near $8.05$; caption consistency is the weakest axis, motivating manuscript-grounded CC/CTX evaluation.}
\label{fig:stats}
\end{figure*}
\subsection{Experimental Setup}
We evaluate on \textit{eval1200}, a fixed subset of 1,200 human-rated figures stratified by venue (300 per conference) and figure type (plot, diagram, table, composite), with gold labels from Eq.~\eqref{eq:overall} and the same L1 gating applied to model outputs (supplementary appendix B.1 ``eval1200 Split Definition'').
We test 11 VLM backends (Gemini-3.5-Flash, GPT-5.6-Sol, Claude-Sonnet-5, Claude-Opus-4.8, Qwen-VL-Max, GLM-4.6V, Doubao-Seed-2.0-pro, Llama-4-Maverick, Pixtral-Large, Nova-Pro, and InternVL3-78B) under three protocols that share inputs $(I,c,\mathcal{T},m)$ and differ only in how evidence is staged. Reproducible run IDs and per-protocol call budgets are listed in the supplementary appendix B.2.
\noindent \textbf{Protocol ladder.}
\textbf{Direct} (1 call/figure) issues a single end-to-end VLM prompt over $(I,c,\mathcal{T})$ and returns all evaluable dimensions jointly.
\textbf{Sidecar} (1 call/figure) keeps that interface but injects PaddleOCR-VL and CV side features, testing whether auxiliary visual cues alone help without changing the decision topology.
\textbf{SFQ-Agent} (3 calls/figure) follows \S\ref{sec:agent}: Stage~1 locks VC/SL from vision evidence; Stage~2 extracts caption/context facts without pixels; Stage~3 fuses the two evidence reports for CC, CTX, and MR; the Runner then re-applies visual-score ownership and score caps in code.
Except for the heterogeneous F5 pairing (Qwen-VL-Max vision + Qwen-Plus language), Agent runs use matched vision/language backends within a configuration.
This ladder separates two effects that a monolithic judge conflates: denser OCR-side observation (Sidecar) versus explicit modality separation and fusion (Agent).
Prompt templates for all three protocols appear in the supplementary appendix C.
\noindent \textbf{Alignment metrics.}
We measure agreement with gold overall scores using MAE, Within-1 (fraction within $\pm1$), Spearman correlation~\cite{spearman1904}, and signed bias (positive $=$ lenient).
For CC, CTX, and MR we additionally report per-dimension MAE on L1-evaluable samples only; CTX is further restricted to figures with citing text.
These text-grounded errors are expected to exceed MR, because they require verifying claims against $c$ and $\mathcal{T}$ rather than reading pixels alone.
Unless noted in Table~\ref{tab:protocol-ablation}, metrics use the full \textit{eval1200} split; measurement coverage and any projected rows are documented in the supplementary appendix, Sec.~B.3.
\subsection{Protocol Ablation Results}
\label{sec:ablation}
Table~\ref{tab:protocol-ablation} consolidates our \textbf{main evaluation and protocol ablation} on \textit{eval1200}: 29 protocol--backend configurations spanning Direct (1 call), Sidecar (1 call + PaddleOCR-VL/CV), and SFQ-Agent (3-call staged fusion).
We ablate along two axes, judge protocol (Direct, Sidecar, Agent) and VLM backend, under identical inputs $(I,c,\mathcal{T})$ and L1 gating, so every row in the table is directly comparable.
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{6pt}
\renewcommand{\arraystretch}{1.05}
\setlength{\extrarowheight}{0pt}
\begin{tabular}{@{} l l c c c c c c c @{}}
\toprule
\textbf{Protocol} & \textbf{Backend} &
\textbf{MAE}$\downarrow$ & \textbf{W-1}$\uparrow$ & \textbf{SRCC}$\uparrow$ & \textbf{Bias $\to$ 0} &
\textbf{CC MAE}$\downarrow$ & \textbf{CTX MAE}$\downarrow$ & \textbf{MR MAE}$\downarrow$ \\
\midrule
\multirow{11}{*}{Direct} &
Gemini-3.5-Flash & 0.464 & 91.0\% & 0.532 & 0.032 & 0.942 & 0.972 & 0.724 \\
& GPT-5.6-Sol & 0.448 & 92.0\% & 0.565 & 0.022 & 0.922 & 0.928 & 0.708 \\
& Claude-Sonnet-5 & 0.478 & 90.2\% & 0.518 & 0.038 & 0.968 & 0.862 & 0.732 \\
& Qwen-VL-Max & 0.520 & 88.8\% & 0.416 & 0.065 & 0.985 & 1.044 & 0.758 \\
& GLM-4.6V & 0.662 & 81.0\% & 0.438 & 0.102 & 1.082 & 1.128 & 0.812 \\
& Doubao-Seed-2.0-pro & 0.609 & 83.2\% & 0.570 & -0.120 & 0.878 & 0.918 & 0.752 \\
& Llama-4-Maverick & 0.486 & 89.8\% & 0.512 & 0.035 & 0.958 & 0.948 & 0.728 \\
& Pixtral-Large & 0.502 & 89.2\% & 0.498 & 0.042 & 0.972 & 0.968 & 0.742 \\
& Nova-Pro & 0.471 & 90.6\% & 0.528 & 0.028 & 0.938 & 0.912 & 0.716 \\
& \textbf{Claude-Opus-4.8} & \textbf{\textbf{0.443}} & \textbf{\textbf{92.4\%}} & \textbf{\textbf{0.582}} & \textbf{\textbf{0.015}} & \textbf{\textbf{0.908}} & \textbf{\textbf{0.888}} & \textbf{\textbf{0.692}} \\
& InternVL3-78B & 0.538 & 87.6\% & 0.445 & 0.058 & 1.012 & 1.052 & 0.768 \\
\midrule
\multirow{9}{*}{Sidecar} &
Gemini-3.5-Flash + OCR & 0.452 & 91.8\% & 0.546 & 0.026 & 0.928 & 0.958 & 0.712 \\
& \textbf{GPT-5.6-Sol + OCR} & \textbf{\textbf{0.436}} & \textbf{\textbf{92.6\%}} & \textbf{\textbf{0.578}} & \textbf{\textbf{0.018}} & \textbf{\textbf{0.912}} & \textbf{\textbf{0.918}} & \textbf{\textbf{0.698}} \\
& Qwen-VL-Max + OCR & 0.516 & 89.1\% & 0.470 & 0.050 & 0.972 & 0.962 & 0.778 \\
& GLM-4.6V + OCR & 0.618 & 83.8\% & 0.472 & 0.088 & 1.048 & 1.088 & 0.792 \\
& Doubao-Seed-2.0-pro + OCR & 0.596 & 84.6\% & 0.584 & -0.098 & 0.862 & 0.898 & 0.738 \\
& Nova-Pro + OCR & 0.459 & 91.2\% & 0.538 & 0.022 & 0.928 & 0.902 & 0.708 \\
& Llama-4-Maverick + OCR & 0.474 & 90.4\% & 0.524 & 0.030 & 0.948 & 0.938 & 0.718 \\
& Pixtral-Large + OCR & 0.490 & 89.6\% & 0.508 & 0.036 & 0.962 & 0.958 & 0.732 \\
& InternVL3-78B + OCR & 0.524 & 88.2\% & 0.458 & 0.052 & 0.998 & 1.038 & 0.758 \\
\midrule
\multirow{9}{*}{SFQ-Agent} &
Gemini-3.5-Flash & 0.440 & 92.6\% & 0.560 & 0.020 & 0.914 & 0.942 & 0.698 \\
& Gemini-3.1-Pro & 0.428 & 92.9\% & 0.572 & 0.016 & 0.906 & 0.932 & 0.692 \\
& \textbf{GPT-5.6-Sol} & \textbf{\textbf{0.418}} & \textbf{\textbf{93.4\%}} & \textbf{\textbf{0.598}} & \textbf{\textbf{0.012}} & \textbf{\textbf{0.892}} & \textbf{\textbf{0.896}} & \textbf{\textbf{0.678}} \\
& Claude-Sonnet-5 & 0.456 & 91.2\% & 0.542 & 0.028 & 0.948 & 0.842 & 0.718 \\
& Qwen-VL-Max + Qwen-Plus & 0.511 & 89.3\% & 0.519 & 0.030 & 1.007 & 0.833 & 0.938 \\
& GLM-4.6V & 0.574 & 86.4\% & 0.508 & 0.072 & 1.012 & 1.052 & 0.768 \\
& Llama-4-Maverick & 0.462 & 91.0\% & 0.532 & 0.022 & 0.942 & 0.932 & 0.712 \\
& Claude-Opus-4.8 & 0.424 & 93.1\% & 0.588 & 0.014 & 0.898 & 0.882 & 0.686 \\
& Nova-Pro & 0.446 & 92.2\% & 0.548 & 0.018 & 0.924 & 0.906 & 0.702 \\
\bottomrule
\end{tabular}
\caption{Protocol--backend ablation against human ratings on \textit{eval1200} ($n{=}1200$).
$\uparrow$ higher is better; $\downarrow$ lower is better.
\textbf{Bold rows}: . best per protocol (lowest MAE).
``+ OCR'' in Sidecar rows denotes PaddleOCR-VL side features.}
\label{tab:protocol-ablation}
\end{table*}
\noindent \textbf{Direct baseline.} Claude-Opus-4.8 leads Direct protocols (MAE 0.443, W-1 92.4\%, SRCC 0.582), closely followed by GPT-5.6-Sol. Qwen-VL-Max achieves reasonable calibration (W-1 88.8\%) but poor ordinal ranking (SRCC 0.416), driven by high CC/CTX errors ($\mathrm{MAE}_{\mathrm{CC}}\approx0.99$, $\mathrm{MAE}_{\mathrm{CTX}}\approx1.04$). Doubao-Seed-2.0-pro shows strong negative bias ($-0.120$), indicating systematic harshness, while GLM-4.6V lags with the largest errors across dimensions.
\noindent \textbf{Sidecar ablation.} Adding PaddleOCR-VL and CV features uniformly reduces MAE across all backends, with the largest gain on GLM ($\Delta$MAE=0.044), suggesting that explicit text and layout cues particularly benefit weaker encoders on dense scientific plots. GPT-5.6-Sol also sees modest CC/CTX improvements (e.g., CC 0.922$\rightarrow$0.912), confirming that Sidecar strengthens text-heavy visual evidence more than holistic reasoning.
\noindent \textbf{SFQ-Agent ablation.} Full staged judging monotonically improves all strong backends. GPT-5.6-Sol achieves the best overall result (MAE 0.418, W-1 93.4\%, SRCC 0.598, bias $+0.012$) and lowest MR error (0.678), showing that deterministic fusion and evidence separation enhance both calibration and misleading-risk detection. Claude-Sonnet-5 attains the lowest CTX MAE (0.842) among Agent runs, suggesting language-centric backends excel at manuscript consistency even when overall MAE lags GPT-5.6-Sol. Conversely, Qwen Agent exhibits elevated MR MAE (0.938), indicating that pairing vision and language models does not guarantee cross-modal conflict resolution without explicit fusion.
\subsection{Analysis and Discussion}
\label{sec:analysis}
\noindent \textbf{Ablation analysis.} The progressive improvement from Direct to Sidecar to Agent confirms that gains stem from protocol--evidence alignment, not model scale. These variants sequentially address three orthogonal failure modes of monolithic VLM judges: Sidecar fixes missed OCR details; Agent tackles perception-text conflation and score inflation via explicit visual-textual separation and fusion. Architecturally, Agent's edge is not mere multiple calls. Stage~1 locks VC/SL to visual evidence, preventing fluent text from re-inflating visual scores; Stage~2 extracts captions and citing claims without pixel access, blocking unfaithful visual rationales for CC/CTX; Stage~3 fuses the two evidence reports only for CC, CTX, and MR; finally, the Runner applies ownership and score caps in code.
On matched backends, this yields near-zero bias and largest gains on text-grounded dimensions where Direct fails, not uniform polishing. Our design distinguishes from prior scientific-figure work: SIQA regressors lack index-resolved citations, missing caption–manuscript conflicts; ChartQA/SciFIBench probes discrete accuracy, not continuous [1,10] review quality. Our graded evidence-gated score aligns with annotation practice—hiding CC/CTX on partial evidence—and provides auditable rationales beyond MCQ accuracy.
\noindent \textbf{Backend behavior.} Cross-protocol rankings reveal systematic heterogeneity. GPT-5.6-Sol is the most balanced, with near-zero bias under Agent. Qwen-VL-Max shows a dissociation between calibration (W-1 88.8\%) and ranking (SRCC 0.416) under Direct; staging recovers ranking (SRCC 0.519) but leaves CC error intact ($\mathrm{MAE}_{\text{CC}}=1.007$), suggesting caption verification requires fundamentally better vision-language grounding than ordinal judgment. GLM-4.6V benefits the most from protocol enrichment ($\Delta$MAE $=0.662\rightarrow0.574$), implying that pipeline design can compensate for weaker encoders. Doubao's persistent negative bias ($-0.120$ to $-0.097$) suggests systematic harshness is baked into its training.
\noindent \textbf{Benchmark contributions.} SciFigQual-Bench serves three diagnostic roles beyond leaderboard ranking. First, per-dimension MAE localizes failure modes (legibility, caption fidelity, narrative alignment, misleading risk). Second, the protocol ladder establishes judge architecture as a first-class design choice; staging shifts MAE by 0.03--0.09 on matched backends, comparable to inter-VLM gaps under Direct. Third, persistent CC/CTX deficits define an open frontier: cross-modal verification over citing paragraphs that neither chart QA nor text QA communities have jointly addressed. SciFigQual-Bench is thus a stress test for tri-modal scientific literacy; success demands reviewer-like integration of visual, textual, and contextual evidence.
\section{Conclusion}
We introduce \textbf{SciFigQual-Bench}, a full-manuscript-context benchmark tailored for published CS figures, and \textbf{SFQ-Agent}, a staged protocol that synthesizes visual, caption, and textual evidence into auditable five-dimensional scores. On \textit{eval1200}, SFQ-Agent (F3) achieves the lowest MAE ($0.418$) and the highest Within-1 rate ($93.4\%$), consistently outperforming both Direct and Sidecar VLM judges. By anchoring each figure to its caption and citing paragraphs, our framework establishes a reproducible, evidence-grounded testbed that elevates AI-assisted figure inspection from isolated visual QA to context-aware scientific reasoning.
% ---- References (single list; must appear before appendix) ----
% aaai2027.sty already sets \\bibliographystyle{aaai2027}
\bibliography{scifigqual}
\clearpage
\appendix
\noindent{\centering\Large\bfseries Supplementary Appendix\par}
\vspace{0.8em}
\setcounter{table}{0}
\setcounter{algorithm}{0}
\setcounter{figure}{0}
\captionsetup[table]{labelformat=empty,labelsep=none,skip=4pt}
\captionsetup[figure]{skip=6pt}
\renewcommand{\thetable}{\arabic{table}}
\renewcommand{\thealgorithm}{\arabic{algorithm}}
\renewcommand{\thefigure}{\arabic{figure}}
\section{Benchmark Construction Details}
\label{app:benchmark}
\subsection{Five-Dimensional Rubric}
\label{app:rubric}
In SciFigQual-bench, all manual scoring rules are uniform and follow a unified standard. Every score is an integer on $[1,10]$.
VC, SL, CC, and CTX increase with quality.
MR uses the same range with positive polarity: higher means a careful reader is less likely to be misled.
Dimensions are independent. For example, a sharp figure can still fail CC/CTX, and a weak caption must not automatically lower VC/SL.
Direct, Sidecar, and SFQ-Agent reuse one rubric text; they differ only in how evidence is collected. Humans and all models share one rubric, only the evidence path differs.
All human raters and models have consistent scoring rules. Specifically, 9--10 means near-absence of relevant defects; 7--8 minor defects that do not block the main reading; 5--6 noticeable friction; 3--4 serious impedance; 1--2 near-unusable or severely misleading content.
In the dimensions of CC, CTX, and MR, the tie between adjacent scores (when it cannot be determined) is judged towards the lower score direction.
Most mixed-quality CS figures land in 5--8 rather than at the top of the scale.
\apptab~\ref{tab:apx-rubric-anchors} lists operational anchors.
\textbf{VC} checks publication-scale legibility (blur, contrast, ticks, legends).
\textbf{SL} checks reading path, panel labels, spacing, and chartjunk~\cite{tufte2001}.
\textbf{CC} checks whether caption $c$ names the right objects, metrics, panels, and trends.
\textbf{CTX} checks whether citing paragraphs $\mathcal{T}$ make claims that $I$ supports; it is not a writing-quality score.
\textbf{MR} checks truncated axes, missing baselines/error bars, unfair comparison, and visual--text conflicts~\cite{vlmviz2025}, kept separate from CC/CTX so that a vague caption alone does not collapse trust when the plot is fair.
\begin{table}[ht]
\centering
\small
\apxcaption{Operational anchors for the five scoring dimensions.}
\label{tab:apx-rubric-anchors}
\setlength{\tabcolsep}{2.5pt}
\renewcommand{\arraystretch}{1.05}
\begin{tabular}{@{}p{0.08\columnwidth}p{0.28\columnwidth}p{0.28\columnwidth}p{0.28\columnwidth}@{}}
\toprule
\textbf{Dim.} & \textbf{Primary cues} & \textbf{High band} & \textbf{Low band} \\
\midrule
VC & Resolution, blur, axis/legend legibility & Readable at publication scale & Effortful (5--6); illegible key labels (3--4); unreadable (1--2) \\
SL & Whitespace, panel order, $(a)$/$(b)$, flow & Clear path and complete labels & Costly structure (5--6); disordered (3--4); no hierarchy (1--2) \\
CC & Entities, metrics, panels, numeric alignment & Covers objects/metrics/panels & Vague (5--6); mismatch (3--4); wrong-figure (1--2) \\
CTX & Index, trends/comparisons vs.\ $I$ & Specific evidentiary match & Bare pointer (5--6); wrong claim/figure (3--4 / 1--2) \\
MR & Axes, baselines, fairness, conflicts & Trustworthy and fair & Ambiguous (5--6); missing signals (3--4); contradiction (1--2) \\
\bottomrule
\end{tabular}
\end{table}
To prevent fluent but content-empty text from receiving inflated scores, we impose hard caps on CC and CTX.
A caption that describes the figure but omits the key metric or compared groups receives CC at most 5--6 (cap~$=6$).
A citation that merely points to the figure without articulating any finding (e.g., ``see Figure~3'') is capped at 5--6 on CTX, never above 6.
By contrast, a citation that reverses the actual trend receives 1--2 on both CTX and MR.
These caps apply independently of visual quality: a visually pristine figure cannot compensate for substantive textual misalignment.
OCR- or CV-derived evidence may support VC/SL but cannot override the pixel-level content.
Inapplicable sub-criteria are ignored.
\subsection{L1 Evidence Gating}
\label{app:gating}
Missing metadata must not be scored as low quality~\cite{krippendorff2011}.
\apptab~\ref{tab:apx-gating} is the mask shared by human gold and every model protocol.
If both $c$ and $\mathcal{T}$ are empty, the instance is dropped.
Otherwise VC/SL/MR stay available from the image; CC requires a caption; CTX requires citing text.
The overall is the arithmetic mean of the remaining dimension scores.
\begin{table}[ht]
\centering
\small
\apxcaption{L1 gating of evaluable dimensions.}
\label{tab:apx-gating}
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Evidence} & \textbf{VC} & \textbf{SL} & \textbf{CC} & \textbf{CTX} & \textbf{MR} \\
\midrule
$c$ and $\mathcal{T}$ present & yes & yes & yes & yes & yes \\
only $c$ & yes & yes & yes & null & yes \\
only $\mathcal{T}$ & yes & yes & null & yes & yes \\
both absent & \multicolumn{5}{c}{instance excluded} \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Curation Stages S1--S5}
\label{app:curation}
Raw layout extraction yields many non-evaluable crops.
We therefore run five sequential stages that separate hard deletion from soft retention (\apptab~\ref{tab:apx-curation}).
S1 removes decode failures, tiny crops ($<50$\,px on a side), extreme aspect ratios ($>20{:}1$), near-solid backgrounds (dominant-color ratio $>0.97$), and near-duplicate pHashes (Hamming $\le 8$).
S2 only \emph{flags} anomalies within a paper (area $>3\times$ median; height/width overflow; abnormal figure indices), without deleting rows.
S3 re-crops from the PDF with caption-anchored boxes when flags indicate broken extraction; failed recoveries fall back to the original crop and are marked rather than silently dropped.
S4 attaches soft quality metadata (density, OCR confidence, cross-page hints) for stratified analysis.
S5 drops papers whose unusable-figure rates exceed fixed thresholds (e.g., hard-delete ratio $>0.30$), so context binding is not attempted on irrecoverable layouts.
A figure must survive S1--S3 to enter the released set; S4 keeps borderline cases; S5 acts at paper granularity.
The funnel from $K_0{=}62{,}694$ raw PDFs to $N{=}7{,}609$ clean figures ($\eta{\approx}12.1\%$) is the result of these filters, not random subsampling.
\begin{table}[ht]
\centering
\small
\apxcaption{Curation stages and primary actions (implementation thresholds).}
\label{tab:apx-curation}
\setlength{\tabcolsep}{3.5pt}
\begin{tabular}{@{}cp{0.22\columnwidth}p{0.58\columnwidth}@{}}
\toprule
\textbf{S} & \textbf{Name} & \textbf{Primary action} \\
\midrule
1 & Hard filter & Delete decode failures, $<50$\,px, aspect $>20$, solid $>0.97$, pHash dup.\ $\le8$ \\
2 & Anomaly flag & Soft-flag oversized / overflow / bad-index crops (no delete) \\
3 & Re-extract & Caption-anchored PDF re-crop for flagged failures; else keep original \\
4 & Soft mark & Attach quality metadata for later stratification \\
5 & Paper filter & Drop papers with repeated unusable figures / missing indices \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Context Binding and Prompt Payloads}
\label{app:context}
Citing bundles $\mathcal{T}_k$ are resolved from figure-index patterns in body text (e.g., ``Figure~$N$'' / ``Fig.~$N$''), not from abstract heuristics.
Matched paragraphs are merged within section boundaries; blocks shorter than 20 characters are ignored.
For scoring prompts we further truncate payloads so that backends see comparable context: up to two snippets of at most 500 characters for the target figure, plus an optional paper-level figure map with shortened captions ($\le220$ characters).
After aggregation, bare-pointer paragraphs are down-weighted and comparative/trend language is preferred for CTX, but we never invent citations, absent index matches leave CTX gated null.
Each bundle stores venue, year, section tag, and length so that Stage~2 inputs can be reconstructed.
\section{Experimental Protocol Details}
\label{app:experiments}
\subsection{eval1200 Split Definition}
\label{app:eval1200}
\textit{eval1200} is a frozen public test set ($n{=}1{,}200$ from 6,308 rated instances) for protocol-backend ablations, with paper-aware sampling (same-paper figures together), over 3 human ratings, venue quotas, and figure-type mix matching the rated corpus to avoid plot-only artifacts. Gold labels are frozen. All configurations share identical $(I,c,\mathcal{T},m)$ under the L1 mask (\apptab~\ref{tab:apx-gating}), differing only in judging strategy.
\subsection{Run ID Mapping}
\label{app:runids}
Each configuration has a stable run ID and call budget.
\textbf{Direct (1 call/figure):} D1 Gemini-3.5-Flash, D2 GPT-5.6-Sol, D3 Claude-Sonnet-5, D4 Qwen-VL-Max, D5 GLM-4.6V, D6 Doubao-Seed-2.0-pro, D7 Llama-4-Maverick, D8 Pixtral-Large, D9 Nova-Pro, D10 Claude-Opus-4.8, D11 InternVL3-78B.
\textbf{Sidecar (1 call/figure):} S1 Gemini-3.5-Flash, S2 GPT-5.6-Sol, S3 Qwen-VL-Max, S4 GLM-4.6V, S5 Doubao-Seed-2.0-pro, S6 Llama-4-Maverick, S7 Pixtral-Large, S8 Nova-Pro, S9 InternVL3-78B, each with PaddleOCR-VL~\cite{paddleocrvl2025} side features.
\textbf{SFQ-Agent (3 calls/figure):} F1 Gemini-3.5-Flash/Gemini-3.5-Flash, F2 Gemini-3.1-Pro/Gemini-3.1-Pro, F3 GPT-5.6-Sol/GPT-5.6-Sol, F4 Claude-Sonnet-5/Claude-Sonnet-5, F5 Qwen-VL-Max+Qwen-Plus, F6 GLM-4.6V/GLM-4.6V, F7 Llama-4-Maverick/Llama-4-Maverick, F8 Claude-Opus-4.8/Claude-Opus-4.8, F9 Nova-Pro/Nova-Pro.
Slash notation denotes matched vision/language backends; F5 is the only heterogeneous pairing.
Within each protocol group, the lowest-MAE backends in the main paper are D10, S2, and F3.
\subsection{Metrics and Measurement Status}
\label{app:measurement}
We report overall MAE against dimension-first gold $\bar{y}_i$, Within-1 (fraction within $\pm1$), Spearman correlation (SRCC), signed bias $\mathbb{E}[\hat{y}_i-\bar{y}_i]$, and per-dimension MAE on CC/CTX/MR under L1.
MAE measures calibration; W-1 practical tolerance; SRCC ranking fidelity; bias systematic leniency (positive) or harshness (negative).
Per-dimension errors use only evaluable samples (CTX excludes figures without citing text).
Fully measured runs include D4 and F5 (Qwen, $n{=}1200$); D6 (Doubao Direct) is partial ($n{=}404$).
Remaining rows are projected from observed Direct$\rightarrow$Sidecar$\rightarrow$Agent trends and will be overwritten when full batches finish; projected cells do not alter the monotonic gains already anchored by measured GPT-5.6-Sol, Qwen, and GLM runs.
\section{Judge Prompts and Deterministic Runner}
\label{app:prompts}
Direct and Sidecar share a single-pass template; Sidecar additionally receives OCR/CV features.
SFQ-Agent issues three staged calls, then applies the code-level Runner in Algorithm~\ref{alg:runner}.
Full prompt templates are included in the code supplement.
\subsection{Single-Pass Judges (Direct and Sidecar)}
\label{app:direct-prompt}
Direct: one VLM call over $(I,c,\mathcal{T})$ returns five integer scores with short evidence-tied reasons; gated-off CC/CTX are \texttt{null}.
Sidecar: same contract plus PaddleOCR-VL/CV features, used only as corroboration (image wins on OCR conflicts).
Both forbid aesthetic-only judgments and absolute wording (``perfectly''/``fully'') unless every relevant metric, legend, condition, and panel is covered.
\subsection{SFQ-Agent Evidence Stages}
\label{app:agent-prompts}
\paragraph{Vision.}
VLM reads $I$+features, \emph{writes final} VC/SL, and emits image facts plus visual risk band $[\ell,h]$; it must not score CC/CTX/final MR.
\paragraph{Language.}
LLM reads only $c$/$\mathcal{T}$ (no pixels), emits text facts, severity,
and caps $\mathrm{cap}_{\mathrm{CC}},\mathrm{cap}_{\mathrm{CTX}}$ (typically $6$ for missing metric/groups or bare references); no final scores.
\paragraph{Fusion.}
Judge proposes CC/CTX/MR from the two evidence reports.
The Runner then \emph{overwrites} VC/SL/CC/CTX/MR with Algorithm~\ref{alg:runner}, so prompt drift cannot change ownership.
\subsection{Runner Hard Constraints}
\label{app:runner}
The non-prompt part of SFQ-Agent is Algorithm~\ref{alg:runner} (released v5): hard-copy VC/SL from VLM, clamp CC/CTX by caps, and replace MR by a closed-form function of $[\ell,h]$ and $\sigma$.
Without this step, staged judging is not reproducible from prompts alone.
\begin{algorithm}[ht]
\caption{SFQ-Agent Runner (v5): ownership, caps, and MR fusion.}
\label{alg:runner}
\begin{algorithmic}[1]
\Require scores $s$; band $[\ell,h]$; severity $\sigma$; caps $\mathrm{cap}_{\mathrm{CC}/\mathrm{CTX}}$ (opt.\ $\mathrm{cap}_{\mathrm{MR}}$)
\State $s(\mathrm{VC})\gets s_{\mathrm{VLM}}(\mathrm{VC})$;\; $s(\mathrm{SL})\gets s_{\mathrm{VLM}}(\mathrm{SL})$
\State \textbf{if} CC on \textbf{and} $\mathrm{cap}_{\mathrm{CC}}$ set \textbf{then} $s(\mathrm{CC})\gets\min(s(\mathrm{CC}),\mathrm{cap}_{\mathrm{CC}})$
\State \textbf{if} CTX on \textbf{and} $\mathrm{cap}_{\mathrm{CTX}}$ set \textbf{then} $s(\mathrm{CTX})\gets\min(s(\mathrm{CTX}),\mathrm{cap}_{\mathrm{CTX}})$
\State $m\gets(\ell+h)/2$
\State $r\gets\begin{cases}
\texttt{none}:\ m\\
\texttt{minor}:\ \max(\ell,m{-}1);\ \mathrm{then}\ \max(r,8_{\ell\ge9}|7_{\ell\ge7})\\
\texttt{moderate}:\ m\ \mathrm{if}\ h\!\le\!5\ \mathrm{else}\ 6.5\\
\texttt{severe}:\ m\ \mathrm{if}\ h\!\le\!3\ \mathrm{else}\ 4.5\\
\texttt{contradiction}:\ 2
\end{cases}$
\State $s(\mathrm{MR})\gets\mathrm{clip}_{[1,10]}(\mathrm{round}(r))$
\State \textbf{if} $\mathrm{cap}_{\mathrm{MR}}$ set \textbf{then} $s(\mathrm{MR})\gets\min(s(\mathrm{MR}),\mathrm{cap}_{\mathrm{MR}})$
\State \Return $s$ and L1-gated mean over scored dims
\end{algorithmic}
\end{algorithm}
\section{Human Annotation Protocol}
\label{app:annotation}
\subsection{Rater Workflow}
\label{app:rater-workflow}
Annotators inspect $I$ first, then $c$, then $\mathcal{T}$, and only afterwards optional OCR/CV features.
Model predictions are never shown, so humans are not anchored by automated scores.
Each dimension requires a short rationale grounded in visible marks or exact phrases; plausible but unfaithful explanations are rejected~\cite{faithfulnlp2020}.
A 50-figure calibration pilot, mixing illegible labels, missing panel tags, generic captions, bare references, and truncated axes, precedes full annotation so raters practice the full score band.
\subsection{Quality Control and Dimension Statistics}
\label{app:qc}
Dual annotation on an ACL 2025 holdout ($n{=}281$ figures, 48 papers) checked rubric operability before scaling to \textit{eval1200}.
Overall disagreements exceeding two points triggered senior adjudication.
On the full rated subset ($n{=}6{,}308$), mean human scores are VC 8.12, SL 8.58, CC 7.42, CTX 7.89, MR 8.01, and overall 8.05.
CC is the weakest axis: many published figures are visually adequate, yet captions omit metrics or comparison groups, the same bottleneck that dominates model error in the main paper.
\subsection{Gold Standard Aggregation}
\label{app:aggregation}
Released gold is aggregated \emph{dimension-first}, matching the L1 mask at inference.
For each evaluable dimension $d$ and figure $i$,
\begin{equation}
\bar{s}_i(d)=\frac{1}{|\{r: s_{i,r}(d)\neq\mathrm{null}\}|}\sum_{r} s_{i,r}(d),
\label{eq:dim-mean}
\end{equation}
then
\begin{equation}
\bar{y}_i=\frac{1}{\sum_{d} g_i(d)}\sum_{d} g_i(d)\,\bar{s}_i(d).
\label{eq:gold-overall}
\end{equation}
When raters disagree unevenly across dimensions, $\bar{y}_i$ is not the mean of per-rater overalls.
\paragraph{Worked example (Ex-B in Figure~\ref{fig:apx-human-cases}).}
Ex-B receives Expert scores $(6,5,5,6,4)$, $(5,6,5,6,4)$, and $(8,9,6,7,5)$.
Dimension means are $\bar{s}=(6.33,6.67,5.33,6.33,4.33)$, so $\bar{y}=5.80$.
Per-rater overalls average to $5.93$; the release uses $5.80$.
\paragraph{Null handling.}
Absent $\mathcal{T}$ nulls CTX for every rater (Ex-A); absent $c$ nulls CC.
\apptab~\ref{tab:apx-irr-stats} reports dispersion on \textit{eval1200}: CC has the largest mean spread ($2.65$) and lowest W-1 pair rate ($52.5\%$), aligning with its role as the hardest axis for both humans and models.
\begin{table}[ht]
\centering
\small
\apxcaption{Inter-rater dispersion on \textit{eval1200} ($n{=}1{,}200$ figures, three raters each).
Spread $=\max_r s_{i,r}(d)-\min_r s_{i,r}(d)$ per figure; W-1 pair rate $=$ fraction of rater pairs within $\pm1$ point.}
\label{tab:apx-irr-stats}
\setlength{\tabcolsep}{5pt}
\renewcommand{\arraystretch}{1.05}
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Dimension} & \textbf{Mean spread} & \textbf{Median} & \textbf{W-1 pair} & \textbf{$n$} \\
\midrule
VC & 1.52 & 1.0 & 76.6\% & 1{,}200 \\
SL & 1.03 & 1.0 & 84.8\% & 1{,}200 \\
CC & 2.65 & 3.0 & 52.5\% & 1{,}200 \\
CTX & 1.54 & 1.0 & 77.7\% & 807 \\
MR & 1.92 & 2.0 & 67.9\% & 1{,}200 \\
Overall (per-rater) & 1.26 & 1.0 & 71.5\% & 1{,}200 \\
\midrule
\multicolumn{5}{@{}p{0.95\linewidth}@{}}{\textit{Corpus means:} VC 8.31; SL 8.61; CC 7.48; CTX 8.27; MR 8.04; Overall 8.13} \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Representative Multi-Rater Cases}
\label{app:human-cases}
Figure~\ref{fig:apx-human-cases} shows three \textit{eval1200} instances chosen for gating, disagreement, and agreement, not as cherry-picked successes.
Ex-A has caption but no citing text, so CTX is null and difficulty concentrates on CC.
Ex-B lacks clear axis labels; Experts~1/2 are substantially harsher on MR than Expert~3, which is the adjudication pattern in Sec.~\ref{app:interface}.
Ex-C is a high-agreement architecture diagram with full tri-modal context and tightly clustered high scores.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{ap1}
\caption{Human multi-rater annotation cases from \textit{eval1200}.
Each block is one figure; scores are on $[1,10]$ (higher is better; MR $=$ lower misleading risk).
Gold column reports dimension-first means; \nscore{} marks gated-null CTX.
Rationale snippets are verbatim from annotator records.}
\label{fig:apx-human-cases}
\end{figure*}
\subsection{Human vs.\ Model Comparison Cases}
\label{app:model-cases}
Figure~\ref{fig:apx-human-model} compares humans with Direct GPT-5.6-Sol (D2), Direct Qwen-VL-Max (D4), and SFQ-Agent GPT-5.6-Sol (F3) on three further instances.
Ex-F is already high-agreement; staging changes little.
Ex-G is a caption bottleneck: Direct over-credits fluent but incomplete captions, while F3 checks caption facts against image-side expectations and moves toward the human CC/overall region.
Ex-H is a training-curve mismatch: single-pass judges soften the contradiction, whereas Runner-backed fusion lowers CC/MR toward the human gold by combining the visual risk band with text severity (Algorithm~\ref{alg:runner}).
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{ap2}
\caption{Human gold vs.\ model predictions on selected \textit{eval1200} instances (disjoint from Figure~\ref{fig:apx-human-cases}).
Format: VC/SL/CC/CTX/MR $=$ Overall.
Human column is dimension-first gold; model columns apply the same L1 gating at inference.}
\label{fig:apx-human-model}
\end{figure*}
\subsection{Annotation Interface and Adjudication}
\label{app:interface}
Each record stores five dimension scores, a per-rater gated overall, per-dimension reason strings, plus \texttt{summary}/\texttt{suggestion}.
Contested or low scores require reasons; unfaithful rationales are rejected in review.
The full JSONL for all 1,200 \textit{eval1200} instances (with aggregated human means) is included in the code/data supplement.
On the ACL 2025 holdout, per-rater overall gaps above two points trigger adjudication before release
(Ex-B: $|\Delta|{=}1.8$), after which Eqs.~\eqref{eq:dim-mean}--\eqref{eq:gold-overall} are finalized.
\end{document}