From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations
Extracted source text · arXiv 2606.11913
%%%%%%%% ICML 2026 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2026} with \usepackage[nohyperref]{icml2026} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
\newcommand{\mychange}[1]{\textcolor{red}{#1}}
% Use the following line for the initial blind version submitted for review:
% \usepackage{icml2026}
% For preprint, use
% \usepackage[preprint]{icml2026}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage[accepted]{icml2026}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{colortbl}
\usepackage[most]{tcolorbox}
% \usepackage{algorithm}
% \usepackage{algpseudocode}
\usepackage[labelsep=period]{caption}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
\usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations}
\begin{document}
\twocolumn[
\icmltitle{From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations}
% It is OKAY to include author information, even for blind submissions: the
% style file will automatically remove it for you unless you've provided
% the [accepted] option to the icml2026 package.
% List of affiliations: The first argument should be a (short) identifier you
% will use later to specify author affiliations Academic affiliations
% should list Department, University, City, Region, Country Industry
% affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order. Ideally, you
% should not use this facility. Affiliations will be numbered in order of
% appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Yuchen Guan}{thu}
\icmlauthor{Xiao Li}{msra}
\icmlauthor{Zongyu Guo}{msra}
\icmlauthor{Xiaoyi Zhang}{msra}
\icmlauthor{Xiulian Peng}{msra}
\icmlauthor{Chun Yuan}{thu}
\icmlauthor{Yan Lu}{msra}
\end{icmlauthorlist}
\icmlaffiliation{thu}{Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China}
\icmlaffiliation{msra}{Microsoft Research Asia}
\icmlcorrespondingauthor{Xiao Li}{xili11@microsoft.com}
\icmlcorrespondingauthor{Chun Yuan}{yuanc@sz.tsinghua.edu.cn}
% \icmlcorrespondingauthor{Xiulian Peng}{xipe@microsoft.com}
% \icmlcorrespondingauthor{Yan Lu}{yanlu@microsoft.com}
% You may provide any keywords that you find helpful for describing your
% paper; these are used to populate the "keywords" metadata in the PDF but
% will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column listing the
% affiliations and the copyright notice. The command takes one argument, which
% is text to display at the start of the footnote. The \icmlEqualContribution
% command is standard text for equal contribution. Remove it (just {}) if you
% do not need this facility.
% Use ONE of the following lines. DO NOT remove the command.
% If you have no special notice, KEEP empty braces:
\printAffiliationsAndNotice{} % no special notice (required even if empty)
% Or, if applicable, use the standard equal contribution text:
% \printAffiliationsAndNotice{\icmlEqualContribution}
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_related}
\input{sec/3_method}
\input{sec/4_experiments}
\input{sec/5_conclusion}
% In the unusual situation where you want a paper to appear in the
% references without citing it in the main text, use \nocite
\nocite{langley00}
\bibliography{example_paper}
\bibliographystyle{icml2026}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
\input{sec/X_suppl}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
% This document was modified from the file originally made available by
% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
% by Iain Murray in 2018, and modified by Alexandre Bouchard in
% 2019 and 2021 and by Csaba Szepesvari, Gang Niu and Sivan Sabato in 2022.
% Modified again in 2023 and 2024 by Sivan Sabato and Jonathan Scarlett.
% Previous contributors include Dan Roy, Lise Getoor and Tobias
% Scheffer, which was slightly modified from the 2010 version by
% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
% slightly modified from Prasad Tadepalli's 2007 version which is a
% lightly changed version of the previous year's version by Andrew
% Moore, which was in turn edited from those of Kristian Kersting and
% Codrina Lauth. Alex Smola contributed to the algorithmic style files.
\begin{abstract}
We propose a new paradigm for long video understanding by treating a long video as a Neural Knowledge Representation (NKR).
NKR represents video contents neither as a stream of tokens nor pre-organized databases, but as an individual small portion of network weights attached to the VLM backbone. The NKR weights are optimized to encapsulate the video's semantic content via a novel Agentic Knowledge Distillation (AKD) process, where an agent automatically synthesizes dense descriptions and question-answer pairs to distill the video's knowledge into the NKR. While AKD serves as a comprehensive, one-time encoding phase, the resulting NKR transforms the video into a portable, reusable asset.
% At inference, the lightweight NKR is mounted onto a frozen Vision-Language Model (VLM), enabling direct, query-based understanding without accessing the original video.
At inference, the lightweight NKR is mounted onto a frozen Vision-Language Model (VLM), enabling direct, query-based understanding without reloading or re-encoding the original video.
% This approach yields two key advantages: lightning-fast responses and zero extra memory cost at runtime, regardless of video duration. Experiments on the LVBench benchmark show our method achieves performance comparable to state-of-the-art approaches while reducing end-to-end latency by over two orders of magnitude, opening new possibilities for interactive long-video understanding.
This approach decouples video length from inference cost, offering high amortized efficiency for multi-turn video understanding. Experiments on the LVBench benchmark show our method achieves performance comparable to state-of-the-art approaches while reducing end-to-end latency by over two orders of magnitude, opening new possibilities for interactive long-video understanding.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
Multi-modal Foundation models are increasingly deployed as agents that solve open-ended tasks by understanding and interacting with human users, e.g., as chatbots, personal assistants, or information analyzers~\citep{wang2025interaction, li2024personal}.
Among the capabilities such agent systems must have, long video understanding is both indispensable and uniquely challenging.
% Long videos distribute content sparsely from minutes to hours, making precise information extraction difficult, while real-world interactive queries demand low-latency feedback rather than slow, multi-stage responses.
On one hand, long videos distribute content sparsely from minutes to hours, making precise information extraction difficult. On the other hand, real-world scenarios often involve repeated querying and diverse downstream tasks on the same video, which necessitates low-latency responses and persistent, stable access to the video's underlying knowledge.
\input{figures/fig0_teaser}
Currently, there are two dominant paradigms for video understanding using foundation models.
The first directly tokenizes a video into a sequence of visual tokens fed to a Vision-Language Model (VLM) alongside the user query.
Although recent VLMs perform well on short-video tasks using this strategy~\citep{bai2025qwen2, wang2025adaretake}, they still struggle with long video inputs~\citep{wang2024lvbench}.
Their key bottleneck arises from treating long videos as extremely long token sequences, leading to prohibitive computation and extra memory cost at runtime. In practice, limited context length and compute budgets often force aggressive frame sub-sampling or token pruning, thus discarding information crucial for visual understanding.
A second line of work adopts agentic pipelines that first organize a long video into a static database (\textit{e.g.}, a RAG system), then iteratively plan, browse, retrieve, and reason over segments.
While such systems have shown their better understanding performance on long videos~\cite{zhang2025deep, chen2025lvagent}, their explicit plan–act–observe loops introduce substantial latency that can be up to minutes per query, hindering the efficiency of multi-round interactions.
In essence, existing methods fail to achieve the synergy of rapid responsiveness and long-term persistence, leaving the potential for truly efficient, multi-turn video interaction largely untapped.
\input{figures/fig1_overview}
In this paper, we jump out of existing paradigms and take a different stance of the video understanding problem.
Our central observation is that efficient and reusable long-video understanding hinges on \emph{what} knowledge is extracted and \emph{how} it is represented.
Here, \emph{knowledge} refers to a higher level, comprehensive representation of the video content, which enables a model to recognize facts expressed in the video such as entities, events, attributes, relations, etc, and then apply logical and causal reasoning to infer new information that can answer user queries~\cite{allen2309physics}.
We argue that enabling effective and efficient video understanding requires materializing a video’s knowledge into an AI-native representation directly consumable by the model, instead of relying on dense sequence processing or external retrieval at runtime. Existing paradigms
% fall short: tokenization approaches force full-sequence attention on long video inputs; agentic approaches build external databases and invoke retrieval iteratively conditioned on user queries. Both still
treat the video as a static, external “content container” that must be repeatedly opened and processed to extract query-aligned knowledge.
To this end, we propose a new paradigm for long-video understanding: the \textbf{Neural Knowledge Representation (NKR)}, which optimizes a video's entire semantic content into a compact, implicit neural representation.
Our NKR materializes a video's knowledge into a small, swappable set of network parameters, analogous to how Neural Radiance Fields (NeRFs)~\cite{mildenhall2020nerf} compress a 3D scene into a network's weights.
Instead of treating the video as an external "content container" that requires repeated access, our NKR internalizes the video's knowledge directly within the model's parameters. These parameters, which can be dynamically mounted onto a Vision-Language Model (VLM), serve as the AI-native representation of the video.
The NKR representation yields several fundamental advantages.
First, the size of NKR is independent of video duration, avoiding progressive information attenuation as length grows.
Second, inference now only requires loading the lightweight NKR adapter, bypassing the costly processing of video tokens or iterative database retrieval.
Finally, the extra memory cost for video during inference is zero once the NKR is consumed into the VLM.
Together, these properties enable highly efficient, lightning-fast responses for long-video understanding tasks.
Akin to other neural implicit representations, our paradigm performs offline optimization to compress the knowledge before online use. The key challenge is to construct an effective optimization objective that distills the rich knowledge of a video without human annotation.
We design an \emph{Agentic Knowledge Distillation} (AKD) process to address this challenge. Given an input video, AKD first conducts a holistic description extraction leveraging multiple foundation models and vision tools; it then synthesizes multi-level question–answer pairs covering entities, events, relations, timelines, as well as compositional reasoning questions requiring temporal localization and evidence aggregation. Finally, we optimize the NKR with a mixed-training strategy on both the descriptive data and the question answer data to distill the video into a constant-size implicit weight.
We evaluate our method on challenging long-video benchmarks including LVBench~\cite{wang2024lvbench} and LongVideoBench~\cite{wu2024longvideobench}. Under comparable foundation model size, replacing direct video token input with our neural knowledge field yields a comparable overall performance and better long-video performance, while reducing response time by an order of magnitude.
Our contributions can be summarized as:
\begin{itemize}
\item A new paradigm for long-video understanding that encodes a single video's knowledge into a \textbf{Neural Knowledge Representation (NKR)}.
\item An \textbf{Agentic Knowledge Distillation} process that synthesizes supervision signals to distill video knowledge without human annotation.
\item We demonstrated \textbf{lightning-fast responses with nearly zero memory overhead during inference} for long video understanding tasks.
\end{itemize}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/images/teaser}
\caption{Time-memory-accuracy comparison on hour-level video understanding benchmark LVBench.
The size of the circles indicates the additional memory overhead for video data during inference,
measured in terms of the KV cache size for Qwen series, AdaReTaKe and GPT-4o, and external database size for VideoRAG.
Compared to other methods, our proposed neural representation enables lightning fast response speed with nearly zero additional memory overhead during inference, while achieves similar accuracy.}
\label{fig:teaser}
\vspace{-2em}
\end{figure}
\begin{figure*}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{figures/images/overview}
\caption{Different Paradigms for Long Video Understanding. (a) The commonly used approach for VLM treats video as token sequences; it is limited by the context window size for long video processing. (b) The agentic approach pre-computes the video into a static database that supports tool calling and retrieval; it requires iteratively query and execute tools, which severely limits its interactive response speed with users. (c) Our NKR approach optimizes a neural representation for high-level knowledge in the video; At run time, it is directly plugged into the foundation model, enabling lightning-fast responses to user query.}
\label{fig:overview}
\vspace{-3mm}
\end{figure*}
\section{Related Works}
\label{sec:related}
\paragraph{Video Understanding.}
Recent advances in foundation models~\citep{achiam2023gpt, maaz2023video, zhang2023video, wang2024internvideo2, bai2025qwen2,cheng2025mixture,cheng2026memory} have enabled video understanding by directly tokenizing video frames as long token sequences. While their capabilities have been adapted to video inputs with tens of frames, their performance degrades significantly on longer videos due to limited context window size~\cite{wang2024lvbench,wu2024longvideobench}.
Follow-up works have improved the context window by system/model co-design~\citep{chen2024longvila, ding2024longrope} as well as redundancy-aware token pooling/pruning methods~\citep{wei2024visual, wang2024retake, wang2025adaretake}. Yet, these methods still have the fundamental limitation of long-context processing efficiency, making it struggle to process hours-long videos.
The rapid advancement of reasoning models~\citep{openaio3} has also facilitated the development of agent systems~\citep{yao2023react, yang2023mm, schick2023toolformer, zhang2023responsible}.
A series of agent frameworks have focused on long-video understanding~\citep{pang2025mr, wang2025videotree, wang2024videoagent, fan2024videoagent, ren2025videorag, yan2025empowering, zhang2025deep, chen2025lvagent}. Typically, these methods first decompose a long video into multiple short clips and pre-organize them as external databases. At runtime, the agent system iteratively thinks and conducts tool calls to retrieve relevant clips and generate answers. However, these approaches lead to extreme high latency, making them less suitable for interactive applications.
\vspace{-1mm}
\paragraph{Implicit Neural Representations.}
Implicit neural representations (INRs), also known as neural fields, have emerged as a powerful paradigm for representing complex signals such as videos~\citep{chen2021nerv,he2026compressionadaptationimplicitvisual}, 3D scenes~\citep{wang2021neus,mildenhall2020nerf}, light and reflection fields~\cite{li2022estimating,wu2023efficient,xiong2024multi,cheng2025whoever}, etc. We refer to~\citep{xie2022neural} for a comprehensive survey of INRs in the field of visual computing.
Traditionally, INRs reconstruct signals by mapping continuous spatial coordinates to signal values using neural networks.
The key advantage of INRs lies in their ability to compress high-dimensional signals with a compact set of parameters, enabling high-quality reconstruction and dynamic generation.
Instead of using INR to reconstruct signal itself, we employ it to compactly represent the knowledge within long videos by mapping queries to relevant semantics, enabling efficient understanding and real-time query response with foundation models.
\vspace{-1mm}
\paragraph{Knowledge Distillation.}
Knowledge distillation (KD) was originally proposed to transfer knowledge from a teacher model to a student model by matching softened output distributions~\citep{hinton2015distilling} or intermediate-layer representations~\citep{romero2015fitnetshintsdeepnets, zagoruyko2016paying, yim2017gift, sun2019patient, wang2020minilm, xu2020bert,guan2024enhancing}.
In the foundation model era, KD has also been used to improve a student model’s instruction following via supervised fine-tuning on data generated by a larger teacher~\citep{wang2022self, taori2023alpaca, xu2023wizardlm}, and to distill reasoning capability into smaller models by mimicking the teacher’s intermediate reasoning or policy~\citep{mukherjee2023orca, shi2024enhancing, guo2025deepseek, wei2025learning,xiong2025hs}. Our approach also involves training a small part of model weights on data generated by a larger agent system but with a distinct goal: we aim to distill the internal knowledge of a media content itself (i.e., long videos) into a compact neural representation.
\section{Method}
\label{sec:method}
\input{figures/fig2_pipeline}
In this section, we present our proposed new paradigm for long video understanding.
We begin by briefly formulating the task and providing an overview of current approaches (\cref{sec:method:overview}).
Next, we introduce our Neural Knowledge Representation (NKR) in (\cref{sec:method:nkr}).
Finally, we detail the Agentic Knowledge Distillation (AKD) process which optimizes a NKR module to effectively encode knowledge from a given long video input (\cref{sec:method:akd}).
\vspace{-1mm}
\subsection{Overview}
\label{sec:method:overview}
Given a foundation model $\mathcal{M}$, a long video $V$ consisting of $T$ frames $\{f\}=\{f_1, f_2, \ldots, f_T\}$, and a user query $Q$ related to the video content,
The goal of video understanding is to generate an accurate and contextually relevant response $A$ that addresses the query based on the information present in the video, i.e.,
\begin{equation}
A = \mathcal{M}(V, Q)
\vspace{-2mm}
\end{equation}
\paragraph{Tokenized Approaches.}
To handle video input, tokenized approaches~\cite{bai2025qwen2,lin2024video,hurst2024gpt} work by first converting the video $V$ into a sequence of visual tokens $\{v\}=\{v_1, v_2, \ldots, v_N\}$ using a visual tokenizer $\mathcal{E}_v$,
where $N$ is the number of visual tokens, and then feeding these tokens into the foundation model $\mathcal{M}$ along with the query $Q$:
\begin{equation}
A = \mathcal{M}(\{v\}, Q) = \mathcal{M}(\mathcal{E}_v(V), Q).
\vspace{-1mm}
\end{equation}
While effective for image and short videos, tokenized approaches face significant challenges when dealing with long videos due to the limited long-context capability of foundation models. For example, GPT-4o~\cite{hurst2024gpt} and Qwen-2.5VL~\cite{bai2025qwen2} has a maximum limit of 60 and 768 frames per query respectively, which is obviously insufficient for long videos.
\vspace{-1mm}
\paragraph{Agentic Approaches.}
Agentic approaches~\cite{ren2025videorag, zhang2025deep} improve long video understanding by pre-organizing information from the video $V$ into a static database $\mathcal{K}(V)$, such as a RAG system. They then retrieve and process relevant information by iteratively calling a set of tools $\mathcal{T}$. The process can be described as a ReAct~\cite{yao2023react}-style loop:
\begin{enumerate}
\vspace{-3mm}
\item \textbf{Plan:} Based on the query $Q$ and current context $S_i$, the reasoning model decides which tool $t \in \mathcal{T}$ to call and with which arguments.
\vspace{-1mm}
\item \textbf{Act:} The tool $t$ is executed on the knowledge base $\mathcal{K}(V)$, producing a result $r_i$.
\vspace{-1mm}
\item \textbf{Observe:} The result is added to the context: $S_{i+1} = S_i \cup \{r_i\}$.
\vspace{-3mm}
\end{enumerate}
This loop continues until a final answer $A$ can be synthesized from the accumulated context $S_{\text{final}}$.
While agentic approaches substantially advance long video understanding, their gains require intricate planning and multi-round tool calls, sharply increasing time and computation costs.
For instance, state-of-the-art approaches, Deep Video Discovery~\cite{zhang2025deep} repeatedly invokes the o3 reasoning model, usually requires 2–4 minutes to answer a single user query.
\vspace{-1mm}
\subsection{Neural Knowledge Representation}
\label{sec:method:nkr}
The limitations of both tokenized and agentic approaches stem from a shared, fundamental premise: they treat the video as an \emph{external source of data} that must be processed or retrieved at inference time. Whether as a long sequence of tokens or a structured database, the video's content remains static and separated from the model's internal knowledge representation.
We argue that true efficient and deep understanding require a more integrated, AI-native representation of visual contents.
Instead of (repeatedly) processing raw data during inference, we propose to \emph{optimize} the entire knowledge of a given video into a compact, implicit neural representation that the foundation model can directly consume.
Simply speaking, the input to the model is no longer a stream of visual tokens or an external database, but a set of learned parameters $\theta_V$ that \textbf{embody} the video's semantics.
\vspace{-1mm}
\paragraph{Neural Knowledge Representation.} Analogous to how Neural Radiance Fields (NeRFs)~\cite{mildenhall2020nerf} compress a 3D scene into the weights of a neural network for novel view synthesis, we represent the video as an implicit neural knowledge representation (NKR) which condenses a long video into a compact, parameterized weights representation.
Formally, for a given video $V$, we aim to optimize a set of paramters $\theta_V$ that can be mounted into the given foundation model $\mathcal{M}$ at runtime.
The resulting adapted model, denoted as $\mathcal{M}_{\theta_V}$, can then directly generate an answer $A$ for a query $Q$ without accessing the original video:
\begin{equation}
A = \mathcal{M}_{\theta_V}(Q).
\vspace{-1mm}
\end{equation}
Hence, the parameters $\theta_V$ serve as the materialized, AI-native representation of the knowledge contained within the input video $V$.
\vspace{-1mm}
\paragraph{Video Understanding with NKR.}
The inference process with our NKR is highly efficient and consists of three simple steps: (1) \textit{Load}: Instead of loading the video file, the system loads the lightweight neural weights $\theta_V$. (2) \textit{Mount}: The weights are dynamically attached and merged to the frozen VLM backbone. (3) \textit{Query}: The model directly processes the user's textual query $Q$ to generate the answer $A$.
This approach offers two significant practical advantages.
First, it enables \textbf{lightning-fast inference}, as it bypasses costly video token processing and iterative retrieval steps that are required by agentic methods.
Second, the NKR has a \textbf{nearly zero memory overhead} once it has been merged with the VLM, effectively resolving the scalability issues of tokenized approaches.
\vspace{-1mm}
\paragraph{Parameter-Efficient Adaptation.}
We implement our NKR using parameter-efficient fine-tuning (PEFT) techniques. Specifically, we employ Low-Rank Adapter (LoRA)~\cite{hu2022lora} weights as our AI-native representation for the input video. The video-specific parameters $\theta_V$ are instantiated as a small set of LoRA weights attached to the attention layers of the VLM backbone.
This design choice means that the core capabilities of the powerful VLM are preserved, while a small amount of parameters (i.e., the LoRA weights $\theta_V$) are optimized to specialize the model's behavior for the specific video input $V$. In our setup, changing video inputs simply involves swapping out the LoRA weights without modifying the foundation model $\mathcal{M}$ itself.
\vspace{-1mm}
\subsection{Agentic Knowledge Distillation}
\label{sec:method:akd}
The core challenge in creating an effective NKR is to design an optimization strategy that can distill a long video into a compact set of parameters.
An ideal NKR must satisfy two criteria: (1) it must comprehensively \textbf{memorize} the video's content, and (2) it must have the ability to \textbf{integrate} this memorized information to reason and respond to user queries when mounted on a VLM.
To achieve this, we introduce \textbf{Agentic Knowledge Distillation (AKD)}, an optimization process that optimizes the NKR parameters $\theta_V$ to a given input video $V$.
\cref{fig:pipeline} provides an overview of the AKD process. It consists of two key components: (1) an automated data synthesis agent that generates high-quality training data from the input video, and (2) a mixed-training strategy that combines self-supervised memorization with supervised fine-tuning to effectively distill knowledge into the NKR.
\vspace{-1mm}
\paragraph{Data Synthesis.}
We design an automated agent pipeline to generate a comprehensive fitting dataset for a given video $V$.
We generate two types of data for different purposes: a dense set of textual descriptions $\mathcal{D}_{\text{desc}}$ to facilitate memorization, and a diverse set of question-answer pairs $\mathcal{D}_{\text{qa}}$ to enable query-aware understanding.
First, we generate rich textual descriptions from the video to construct the dataset $\mathcal{D}_{\text{desc}}$.
We adopt text-based strategy as most open-source VLMs suitable for training can only support text-based outputs.
To construct the data, we first perform dense, multi-scale segmentation of the video $V$ into a large set of clips of different duration and sample rates.
For each clip, we use a powerful VLM (GPT 4.1) with a carefully designed prompt to generate: (a) descriptions at low-level (visual details), mid-level (actions and events), and high-level (narrative summary) granularities; and (b) a summary of all salient entities (e.g., people, objects) present. For each generated text, we further augment it by rephrasing it multiple times using a faster LLM (GPT 4.1-mini).
This diverse set of textual descriptions serves as a comprehensive proxy for the video's content. Optimization on this dataset forces the NKR to memorize and reconstruct the video's key information.
Second, to generate the question-answer dataset $\mathcal{D}_{\text{qa}}$, we repurpose a state-of-the-art video understanding agent \textit{Deep Video Discovery (DVD)}~\cite{zhang2025deep} into a data generation agent.
This agent iteratively performs in-depth analysis of the video to synthesize a wide variety of question-answer pairs at two levels:
\begin{itemize}
\vspace{-3mm}
\item \textbf{Clip-level QAs}: To ensure broad coverage, we generate a large number of questions for each video clip. These are generated in a single pass by prompting a small reasoning model (o4-mini) to cover aspects like content summarization, entity retrieval, and event description.
\vspace{-1mm}
\item \textbf{Video-level QAs}: To ensure complexity and handle cross-clip questions, we generate a smaller set of challenging questions for the whole video. We conduct this by building a ReAct-style agent~\cite{yao2023react} using the most powerful reasoning model (o3). We build a RAG database over the video clips and iteratively uses tools for deep retrieval and visual content analysis, similar to Deep Video Discovery~\cite{zhang2025deep}. This process yields complex questions involving holistic understanding and long-term temporal correlation across the video.
\vspace{-3mm}
\end{itemize}
Optimization on this dataset enables the NKR to effectively utilize its memorized knowledge to answer diverse queries.
We provide detailed prompts, tool designs and implementation details for the data synthesis agent in the supplementary material.
\vspace{-1mm}
\paragraph{Mixed-training Strategy.}
Inspired by recent findings on LLM training~\cite{allen2309physics}, we employ a mixed-fitting strategy that combines a self-supervised objective for memorization with a supervised objective for query-aware understanding.
The full loss for optimizing the NKR is a combination of the next-token prediction loss $\mathcal{L}_{\text{NTP}}$ and a supervised fine-tuning loss $\mathcal{L}_{\text{SFT}}$:
\begin{equation}
\mathcal{L}_{\text{AKD}} = \mathcal{L}_{\text{NTP}} + \mathcal{L}_{\text{SFT}},
\label{eq:akd_loss}
\end{equation}
The next-token prediction loss $\mathcal{L}_{\text{NTP}}$ is employed on the dense description data $D_{desc}$ to force the NKR to internalize the video's factual content, while the supervised fine-tuning loss $\mathcal{L}_{\text{SFT}}$ is employed on the QA data $D_{qa}$ to endow the NKR with the ability to transform its memorized, fragmented content into structured knowledge that directly addresses user queries.
\begin{table}[t]
\centering
\caption{Examples of synthesized training data for a video clip.}
\raggedright
{\small Example Video (clip frames):}
\includegraphics[width=\linewidth]{figures/images/video_clip.pdf}
\vspace{-1em}
\label{tab:data_examples}
\resizebox{\linewidth}{!}{%
\begin{tabular}{l|p{8cm}}
\toprule
\textbf{Data Type} & \textbf{Example} \\
\midrule
\multicolumn{2}{c}{\textit{Data for Next-Token Prediction ($\mathcal{D}_{\text{desc}}$)}} \\
\midrule
Low-level Desc. & From 00:00 to 00:15, In the opening moments, a female skater ... After dramatic lighting and overlays, the scene shifts to a blue Olympic stadium where the ice dance duo begins another performance ... The background is a vibrant Olympic venue in Vancouver 2010 ... Overlays present text such as 'COMPULSORY DANCE VANCOUVER 2010' ... Their choreography progresses through hold transitions, sweeping twizzles ... \\
Mid-level Desc. & From 00:00 to 00:30, A man and a woman, both wearing ice skates, are seen performing ... The woman wears ... The man ... then ... Their routine includes ... with visible text including ... \\
High-level Desc. & From 00:00 to 02:00, A male and female figure skating duo perform on ice at the Olympic Games. \\
Entity Summary & \textbf{Tessa Virtue}: female, athletic/slender build, fair/pale skin, Canadian ice dancer ... \textbf{Scott Moir}: male, athletic/medium build, fair/light skin, Canadian ice dancer ... \textbf{Audience}: large crowd in stadium, Olympic Games event spectators ... \\
\midrule
\multicolumn{2}{c}{\textit{Data for Supervised Fine-Tuning ($\mathcal{D}_{\text{qa}}$)}} \\
\midrule
Clip-level Q\&A & \textbf{Q}: What exact phrase appears on the video’s opening title card, setting the stage for the compilation? \newline \textbf{A}: TESSA VIRTUE AND SCOTT MOIR EVERY PERFORMANCE AT THE OLYMPIC GAMES. \\
\midrule
Video-level Q\&A & \textbf{Q}: What is the primary purpose of the brief scene that opens the video at 00:00:00–00:00:10, showing Virtue in a pink dress and Moir in a grey shirt beneath an “Olympic Channel” overlay? \newline \textbf{A}: It serves as an introductory montage announcing the compilation of all their Olympic performances. \\
% Video-level Q\&A & \textbf{Q}: Which description best summarizes how Virtue and Moir’s musical choices evolve over the three Olympic segments shown in the video? \newline \textbf{A}: They move from elegant classical pieces in Vancouver, to lyrical romantic scores in Sochi, and finish with bold, contemporary rhythms in PyeongChang. \\
\bottomrule
\end{tabular}%
}
\vspace{-1.7em}
\end{table}
\vspace{-1mm}
\paragraph{Implementation Details.}
For a given video, we generate clips with log-linear durations from 5 seconds up to the video length to ensure multi-scale coverage. To augment the data, we rephrase each generated text 5 times using an auxiliary LLM. For QA generation, we generate 10 distinct QA pairs at the clip level, while our ReAct agent generates approximately 1000 complex questions at the video level.
On average, a 1-hour video yields 30k--40k text description entries and 15k--20k QA pairs, with each ranging from 20 to 300 tokens. The data generation process for a 1-hour video takes 2--3 hours.
The NKR is trained by alternately optimizing the next-token prediction and supervised fine-tuning on this synthesized data with a 2:8 ratio. Unless otherwise specified, we train a LoRA adapter with a rank of 256 on a Qwen3-14B model for 5 epochs. Training a 1-hour video takes approximately 2 hours on four 40GB A100 GPUs.
\begin{figure*}[htbp]
\centering
\includegraphics[width=\textwidth]{./figures/images/data_pipeline}
\caption{
The agentic knowledge distillation (AKD) process for optimizing a neural knowledge representation (NKR).
Left-up: given a video, we first segment it to multiple clips with different length and sample rates.
Each clip is processed by a vision-language model (VLM) to extract multi-granularity textual descriptions and entity information,
forming the dense description data $D_{desc}$.
Left-bottom: we designed a ReAct-style agent to generate complex question-answer pairs $D_{qa}$, based on video content and dense descriptions.
Right: the NKR is instantiated as a LoRA adapter for the given video, and optimized with a mixed-training strategy on both $D_{desc}$ and $D_{qa}$.
}
\label{fig:pipeline}
\vspace{-3mm}
\end{figure*}
\section{Experiments}
\label{sec:expr}
\subsection{Experiments Setup}
\vspace{-1mm}
\paragraph{Datasets.}
We evaluate the proposed NKR method on a long video understanding benchmark, LVBench~\cite{wang2024lvbench}.
The LVBench benchmark stands as one of the most comprehensive and challenging benchmarks for extreme long-form video understanding.
It has a total of 103 videos with a total duration of 117 hours and 1549 manually annotated multiple-choice questions that require deep comprehension of the video's content, including temporal reasoning, event understanding and entity recognition.
\vspace{-3mm}
\paragraph{Baselines.}
\input{figures/tab2_lvbench_acc}
We compare the proposed NKR method with two existing dominant paradigms for long video understanding.
For tokenization-based approach, we compare with the currently widely-used proprietary model GPT-4o and GPT 4.1~\cite{hurst2024gpt}, open-sourced VLM Qwen2.5-VL~\cite{bai2025qwen2} and VideoLLaMA3~\cite{zhang2025videollama}. We also compared with AdaReTaKe~\cite{wang2025adaretake} which greatly improves the input length of tokenization-based approach by KV-cache compression.
For agentic approach, we compare with RAG-based agent VideoRAG~\cite{ren2025videorag}, Tree-search based agent VCA~\cite{yang2025vca}, and ReAct-based agents Deep Video Discovery~\cite{zhang2025deep}.
\vspace{-1mm}
\subsection{Main Results}
Our main results on the LVBench benchmark is reported in \cref{tab:main_results}, Overall, it demonstrates that the Neural Knowledge Representation (NKR) paradigm successfully redefines the trade-off between performance and efficiency in long video understanding. As shown our NKR approach achieves comparable overall accuracy (48.8\%) while speed-up the inference time by 100$\times$ than tokenization-based approaches with commercial GPT series (31.7s to 43.0s) open-sourced VideoLLaMA3 (64.5s) and Qwen2.5VL series (30.0s to 45.6s) as well as agentic methods such as VideoRAG (90.0s).
%Tokenization based approaches require additional memory to store the KV-cache during inference while agentic approaches need to store an external database built upon the video. NKR weights are merged into the VLM at runtime; thus it is the only method that (1) achieves both near instant inference speed and zero extra memory overhead, and (2) maintains a competitive overall accuracy.
The efficiency advantage of NKR stems from a fundamental design choice: it decouples resource consumption from video duration compared to other approaches.
% Unlike tokenization-based methods where computation and memory scale with video length, or agentic approaches where retrieval slows with larger databases, NKR compresses the video's knowledge into a fixed-size representation and the weights are merged into the VLM during inference.
\cref{fig:teaser} visualizes the accuracy, speed, and memory cost of different methods.
Tokenization-based methods handle longer videos by expanding visual tokens. The quadratic attention complexity leads to prohibitive increases in both computation time and memory usage (KV-cache).
Agentic approaches also suffer from scalability issues, as longer videos enlarge the external databases, leading to slower retrieval and increased storage requirements.
In contrast, our NKR paradigm eliminates the need for processing a perpetually growing size of video content during inference. Thus, NKR is the only method that (1) achieves both nearly instant inference speed and zero extra memory overhead regardless of video duration, and (2) maintains a competitive overall accuracy.
\input{figures/tab6_analysis_duration}
\cref{tab:analysis_duration} further validates this scalability by analyzing video understanding performance w.r.t. duration. We partition the LVBench benchmark into two subsets of roughly equal size: videos shorter than one hour and those longer than one hour.
While existing methods show significant performance degradation on videos longer than one hour (e.g., a 6.7\% drop for GPT-4.1 and a 2.9\% drop for VideoRAG), our NKR's accuracy on long videos remains remarkably stable, decreasing by a mere 0.3\%. This confirms NKR's superior scalability and its unique suitability for truly long-form video analysis.
Compared to agentic methods like VideoRAG, our NKR demonstrates a different but compelling capability performance. While VideoRAG has improved the accuracy in retrieval-heavy tasks (KIR: 57.1\% vs. 53.0\%), our NKR excels in reasoning-intensive questions (REA: 47.1\% vs. 43.9\%).
This suggests that by internalizing knowledge, NKR fosters a more holistic understanding conducive to reasoning, rather than relying on multi-round, discrete retrieval. Meanwhile, our NKR achieved this comparable performance while being ~300x faster and requiring no external database during inference.
Finally, Deep Video Discovery (DVD) achieves the best accuracy on LVBench in our experiments. However, its design philosophy and target scenarios differ fundamentally from ours. DVD's performance stems from a powerful external reasoning model with iterative planning, which incurs even more inference latency (avg. 3 minute for a single question), making it more suitable for offline analysis rather than interactive applications.
The goal of our work is not to pursue maximum accuracy at all costs. Instead, we aim to explore a novel paradigm that guarantees minimal and constant inference cost, while pushing the boundaries of understanding within this efficiency-first constraint. Consequently, our NKR framework should be seen as pioneering a different path: one that prioritizes fast response time and scalability, offering a compelling alternative for interactive applications where low latency is a critical requirement.
\vspace{-1mm}
\subsection{Discussions}
\vspace{-1mm}
\paragraph{Ablation Studies.}
\input{figures/tab4_ablation_model}
\input{figures/tab5_ablation_data}
To understand the key to NKR's performance, we conducted ablation studies on its core components: the NKR representation capacity and the Agentic Knowledge Distillation (AKD) process.
\cref{tab:ablation} reveals that the performance of NKR is more determined by the capacity of the VLM backbone rather than the size of the LoRA adapter. Upgrading the VLM from 8B to 14B yields a substantial 9.3\% accuracy with only minor speed overhead (0.33s vs. 0.26s), whereas quadrupling the LoRA rank from 64 to 256 provides a more modest 4.6\% improvement without increasing inference time.
This result suggests that NKR acts as an efficient "knowledge adapter" whose effectiveness hinges on a powerful VLM to reason upon the distilled knowledge.
The choice of a 14B model with r=256 thus offers an optimal balance of high performance and minimal speed impact.
The multiple type of data generated from our AKD process is also indispensable.
\cref{tab:ablation_data} shows that training on dense descriptions or QA pairs alone leads to significant performance drops. Without descriptions, the model only memorizes QA formats, failing to generalize. Without all QA pairs, it possesses facts but lacks the ability to integrate and reason over them. Without video-level QA pairs, the NKR failed to learn the complicated correlation of the video knowledge at a holistic level, leading to a noticeable performance drop as well.
This demonstrates that AKD's strategy of combining holistic descriptions with structured reasoning examples is the key to successfully distilling complex video content into a functional neural representation.
\vspace{-2mm}
\paragraph{Image Understanding.}
\input{figures/tab7_image_und}
To further validate the effectiveness of our implicit representation, we evaluate NKR on image understanding tasks. We construct a diverse subset from MME-RealWorld~\cite{mme-realworld} with 38 images and 526 QA pairs (258 perception, 268 reasoning). The subset covering all original sub-tasks with at least two images for each question type.
Each image's NKR is optimized using dense descriptions and 350 synthesized QA pairs via the same AKD procedure used for videos. As shown in \cref{tab:img_und}, NKR improves overall accuracy by 3.3\% over the Qwen2.5-VL-7B baseline. The advantage is particularly pronounced in reasoning tasks, where NKR achieves a 7.9\% gain, highlighting its strength in complex cognitive understanding.
% \paragraph{Limitations.}
% As same to any current LLMs and VLMs, the NKR cannot be guarenteed to output without any hallucinations.
% Building a NKR from video input currently still relies on upfront optimization for each video.
% A major limitation is the reliance on textual-only descriptions $D_{desc}$ for the AKD process. This design choice is due to the fact that current VLMs does not have the ability to \emph{output} visual token as a supervision signal.
% Consequently, our NKR may not fully capture fine-grained visual details that are difficult to articulate in text.
% In our early experiments, we have attempted to add visual tokens only as \emph{input} during the optimization process, (i.e., includes a video captioning task during training), which significantly increased the optimization time but did not observe significant improvements to the text-only counterpart.
% Overcoming this "textual bottleneck" and enabling direct distillation of visual knowledge is a significant challenge, not just for NKR but for the broader field of implicit knowledge representation. We believe that future advances in generative VLMs will unlock the potential for a truly multi-modal knowledge distillation, paving the way for even richer and more capable NKRs.
%A major limitation of our current NKR is that we still rely on textual information to build the dense description $D_{desc}$ for the optimization process. This is mainly due to the fact that the current VLM models does not have the ability to output visual tokens, making it infeasible to optimize the NKR with visual supervision.
%As a result, the current NKR has limited ability to capture fine-grained visual details that are not easily described in text.
%We believe this is mainly due to two reasons: 1) The training-inference mismatch issue, where the NKR is optimized with visual tokens as input but evaluated without them during inference; 2) The VLM backbone is often already proficient at generating high-quality captions; consequently, the loss from the captioning task is minimal, providing a weak gradient signal to effectively optimize the NKR parameters.
\begin{table*}[t]
\centering
\caption{Accuracy, model size, input size, and inference speed comparison on LVBench benchmark. The input size refer to either max number of input frames or input video fps (whichever larger). Memory overhead is measured in GBytes; we measure the KV-cache size for tokenized approaches during inference and the size of the external database built upon the video for agentic approaches. our NKR weights are merged into the VLM at runtime, thus it has zero extra memory overhead.
Inference speed is measured on a single NVIDIA A100 GPU with 80G memory for open-sourced models, while Qwen2.5-VL-32B uses 2$\times$A100 due to its memory requirement. The question set of LVBench is categorized as follows: ER: Entity Recognition, EU: Event Understanding, KIR: Key Information Retrieval, TG: Temporal Grounding, Rea: Reasoning, Sum: Summarization. The inference time and memory overhead for VCA is not available because it does not have a public implementation.
}
\label{tab:main_results}
\resizebox{\textwidth}{!}{%
\begin{tabular}{l|ccc|ccccccc}
\toprule
\multicolumn{1}{c|}{\multirow{3}{*}{\textbf{Methods}}} & \multicolumn{3}{c|}{\textbf{Input Statistics}} & \multicolumn{7}{c}{\textbf{Accuracy}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-11}
& \makecell{\textbf{Input Size} \\ \small{\textbf{(fps or max frames)}}} & \makecell{\textbf{Inference Time} \\ \small{\textbf{(s/query)}}} & \makecell{\textbf{Memory Overhead} \\ \small{\textbf{(GB, at Runtime)}}} & \textbf{Overall} & \textbf{ER} & \textbf{EU} & \textbf{KIR} & \textbf{TG} & \textbf{Rea} & \textbf{Sum} \\
\midrule
\multicolumn{11}{l}{\textit{Tokenized approaches: Commercial VLMs}~\cite{hurst2024gpt,team2023gemini}} \\
\midrule
\small GPT-4o-20241120 & \small 60 frames & \small 43.0 & \multirow{2}{*}{\small \makecell{Unknown\\(Large)}} & \small 48.9 & \small 48.9 & \small 49.5 & \small 48.1 & \small 40.9 & \small 50.3 & \small 50.0 \\
\small GPT-4.1-20250414 & \small 50 frames & \small 31.7 & & \small 45.8 & \small 48.0 & \small 43.6 & \small 41.2 & \small 36.8 & \small 47.3 & \small 39.7 \\
\midrule
\multicolumn{11}{l}{\textit{Tokenized approaches: Open-Source VLMs}~\cite{bai2025qwen2, wang2025adaretake,zhang2025videollama}} \\
\midrule
\small VideoLLaMA3-7B & \multirow{3}{*}{\small \makecell{2fps \\ ($<$768 frames)}} & \small 64.5 & \small 1.2 & \small 45.3 & \small 45.8 & \small 42.4 & \small 47.8 & \small 35.9 & \small 45.8 & \small 36.2 \\
\small Qwen2.5-VL-7B & & \small 30.0 & \small 2.5 & \small 43.8 & \small 43.0 & \small 41.9 & \small 49.8 & \small 40.5 & \small 43.8 & \small 32.8 \\
\small Qwen2.5-VL-32B & & \small 45.6\textsuperscript{*} & \small 11.3 & \small 47.6 & \small 47.1 & \small 47.8 & \small 55.0 & \small 40.5 & \small 47.3 & \small 41.4 \\
\midrule
\small AdaReTaKe-7B & \small \makecell{2fps \\ ($<$2048 frames)} & \small 45.0 & \small 0.9 & \small 51.2 & \small 51.1 & \small 47.6 & \small 62.2 & \small 43.2 & \small 50.2 & \small 27.6 \\
\midrule
\multicolumn{11}{l}{\textit{Agentic approaches}~\cite{ren2025videorag,zhang2025deep,yang2025vca}} \\
\midrule
\small VideoRAG & \small N/A & \small 90.0 & \small 0.5 & \small 49.2 & \small 47.4 & \small 49.3 & \small 57.1 & \small 36.5 & \small 43.9 & \small 39.7 \\
\small VCA & \small N/A & \small --- & \small --- & \small 41.3 & \small 43.7 & \small 40.7 & \small 37.8 & \small 38.0 & \small 46.2 & \small 27.3 \\
\small Deep Video Discovery-o3 & \small N/A & \small 180.0 & \small 0.3 & \small 74.2 & \small 73.4 & \small 73.3 & \small 80.4 & \small 72.3 & \small 70.7 & \small 74.1 \\
\midrule
\multicolumn{11}{l}{\textit{Ours}} \\
\midrule
\small Qwen3-NKR-14B-r256 & \small N/A & \small 0.33 & \small 0 & \small 48.8 & \small 54.2 & \small 48.1 & \small 53.0 & \small 45.6 & \small 47.1 & \small 35.3 \\
\bottomrule
\end{tabular}%
}
\vspace{-4mm}
\end{table*}
\begin{table}[t]
\centering
\caption{
Performance analysis w.r.t video durations on LVBench benchmark.
Our NKR method demonstrates remarkable stability for both short and long videos.
}
\label{tab:analysis_duration}
\resizebox{\linewidth}{!}{%
\begin{tabular}{l|ccc}
\toprule
\textbf{Method} & \textbf{$<$ 1 hour} & \textbf{$>$ 1 hour} & \textbf{Performance Drop} \\
\midrule
GPT4.1 & 48.7 & 42.0 & -6.7 \\
Qwen2.5VL-7B & 45.9 & 41.1 & -4.8 \\
Qwen2.5VL-32B & 51.6 & 45.3 & -6.3 \\
AdaReTaKe-7B & 51.4 & 48.4 & -3.0 \\
VideoRAG & 50.7 & 47.8 & -2.9 \\
\midrule
Qwen3-NKR-14B-r256 & 48.9 & \textbf{48.6} & \textbf{-0.3} \\
\bottomrule
\end{tabular}%
}
\vspace{-2.2em}
\end{table}
\begin{table}[t]
\centering
\caption{Ablation study on the impact of VLM backbone size and LoRA rank on NKR performance and speed. The main model used in other experiments is highlighted. Speed is measured on a single NVIDIA A100 GPU with 80G memory.}
\label{tab:ablation}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|cccc}
\toprule
\textbf{VLM Backbone} & \textbf{LoRA Rank} & \textbf{\makecell{Overall Acc. \\ (\%)}} & \textbf{\makecell{Speed \\ (s/query)}} \\
\midrule
Qwen3-8B & 256 & 39.5 & 0.26 \\
\midrule
\multirow{2}{*}{\textbf{Qwen3-14B}} & 64 & 44.2 & \multirow{2}{*}{0.33} \\
& \textbf{256} & \textbf{48.8} & \\
\bottomrule
\end{tabular}
}
\vspace{-5mm}
\end{table}
\begin{table}[t]
\centering
\caption{
Ablation study on data components generated via Agentic Knowledge Distillation. The experiment is conducted on Qwen3-NKR-14B-r256.
}
\label{tab:ablation_data}
\resizebox{\linewidth}{!}{%
\begin{tabular}{ccc|c}
\toprule
\multicolumn{3}{c|}{Data Components} & \multirow{2}{*}{Accuracy (\%)} \\
\cmidrule(r){1-3}
Dense Desc. & Clip-level QA & Video-level QA & \\
\midrule
\checkmark & & & 14.3 \\
\checkmark & \checkmark & & 45.0 \\
& \checkmark & \checkmark & 38.8 \\
\midrule
\checkmark & \checkmark & \checkmark & 48.8 \\
\bottomrule
\end{tabular}%
}
\vspace{-1.5em}
\end{table}
\begin{table}[t]
\centering
% \tiny
\caption{Comparison of performance on subset of MME-RealWorld image understanding benchmark.}
\label{tab:img_und}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|ccc}
\toprule
\textbf{Method} & \textbf{Perception} & \textbf{Reasoning} & \textbf{Overall} \\
\midrule
Qwen2.5-VL-7B & 39.9 & 24.6 & 32.1 \\
Qwen2.5-NKR-7B-r32 & 38.4 & 32.5 & 35.4 \\
\bottomrule
\end{tabular}
}
\vspace{-1.5em}
\end{table}
\section{Conclusion}
\label{sec:conclusion}
We have proposed representing a video as a Neural Knowledge Representation (NKR) for video understanding.
Instead of treating videos as external data sources, our approach encodes the knowledge of a long video into a compact set of network weights. This is achieved through our proposed Agentic Knowledge Distillation (AKD) process, which automatically synthesizes a comprehensive dataset of textual descriptions and question-answer pairs from the video.
By mounting the learned, video-specific NKR onto a frozen Vision-Language Model, we enable long video understanding with lightning-fast response and zero memory overhead at runtime while achieving comparable performance to current approaches. We believe our proposed NKR paradigm opens up new possibilities for how we interact with and process large-scale multimedia content with foundation models.
\section*{Acknowledgements}
This work is supported by the SSTIC Grant (KJZD20230\\923115106012, KJZD20230923114916032, GJHZ20240\\218113604008).
\section*{Impact Statement}
This paper presents work whose goal is to advance the field of Multimodal Understanding. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
\setcounter{page}{1}
\renewcommand{\thetable}{S\arabic{table}} % Rename Fig. 1 to Fig. S1, Table 1 to Table S1
\renewcommand{\thefigure}{S\arabic{figure}}
% \maketitlesupplementary
\appendix
% The supplemental material contains additional implementation details for our training and evaluation as well as more results and dicussions.
\section{Limitations.}
As same to any current LLMs and VLMs, the NKR cannot be guaranteed to output without any hallucinations.
Building a NKR from video input currently still relies on upfront optimization for each video.
A major limitation is the reliance on textual-only descriptions $D_{desc}$ for the AKD process. This design choice is due to the fact that current VLMs do not have the ability to \emph{output} visual tokens as a supervision signal.
Consequently, our NKR may not fully capture fine-grained visual details that are difficult to articulate in text.
In our early experiments, we have attempted to add visual tokens only as \emph{input} during the optimization process, (i.e., includes a video captioning task during training), which significantly increased the optimization time but yielded no significant improvements to the text-only counterpart.
Overcoming this "textual bottleneck" and enabling direct distillation of visual knowledge is a significant challenge, not just for NKR but for the broader field of implicit knowledge representation. We believe that future advances in generative VLMs will unlock the potential for a truly multi-modal knowledge distillation, paving the way for even richer and more capable NKRs.
\section{Future works.}
Future work includes extending this paradigm to other modalities, and investigating interactive scenarios with user feedback to further refine and adapt the knowledge representation.
\section{Implementation Details}
\label{sec:supp:implementation}
\subsection{Dense Description}
\label{sec:supp:dense_description}
We prompted the OpenAI GPT-4.1 model to generate dense descriptions. The detailed prompts used in our dense description generation are illustrated in \cref{fig:supp:prompt_caption} and \cref{fig:supp:prompt_entity}.
\subsection{Clip-level Q\&A for Video NKR}
To generated clip-level Q\&A data, we prompted the OpenAI o4-mini model to generate question-answer pairs in a single pass, based on the dense description of each video clip as input. \cref{fig:supp:prompt_qa} demonstrates the detailed prompts used in our clip-level Q\&A generation.
\subsection{Q\&A for Image NKR}
The prompt for image Q\&A generation is similar to the clip-level video Q\&A generation. \cref{fig:supp:prompt_image_qa} demonstrates the detailed prompts used in our image Q\&A generation.
\subsection{Video-level Q\&A for Video NKR}
Our video-level Q\&A generation agent follows a ReAct-style agentic pipeline, similar to Deep Video Discovery (DVD)~\cite{zhang2025deep}.
We roughly follow the overall pipeline design of DVD, with specific modifications on prompts as well as improvements on database constructions and toolsets design to better fit our Q\&A generation task.
\paragraph{Overview.}
\input{figures/taba5_agent_pipeline}
Specifically, given a long video $V$, our agent consists of a multi-grained video database $\mathcal{D}$ with timestamp indexing, a set of toolset $\mathcal{T}$, and a strong reasoning foundation model $M$ (in our case, OpenAI o3~\cite{openaio3}).
The overall pipeline is illustrated in \cref{alg:supp:video-qa}.
We categorize the generated video questions into the same six category defined in LVBench~\cite{wang2024lvbench}.
\cref{fig:supp:prompt_video_qa_agent} demonstrates the general instruction of our video-level Q\&A generation agent; the specific instructions for each question categories are illustrated from \cref{fig:supp:prompt_video_qa_specific} to \cref{fig:supp:prompt_video_qa_specific_2}, respectively.
\paragraph{Multi-grained Video Database.}
The multi-grained video database $\mathcal{D}$ consists of three components:
(1) The raw video clips split from the long video $V$ with timestamps;
(2) Dense descriptions generated in \cref{sec:supp:dense_description} for each video clip;
(3) An entity list extracted from the dense descriptions, which contains the main characters, objects, locations, and other key entities appearing in the video. We follow the same procedure as in~\cite{zhang2025deep} to extract and deduplicate the entity list.
We associate each video clips (along with their corresponding dense descriptions) and the entity list with both semantic-based and timestamp-based indexing schemes. The semantic embedding for each clip is generated from the OpenAI's text-embedding-3-large model~\cite{openaiembeddings}, which allows the agent to retrieve relevant video clips and entities based on semantics.
The timestamp-based indexing is maintained with an interval tree data structure~\cite{Halbert2025chaimleib}; it enables the agent to access video clips and entities directly based on their temporal occurrence in the video.
\paragraph{Toolsets.}
We provide the agent with four tools: \textit{Global Browse}, \textit{Clip Search}, \textit{Temporal Search} and \textit{Frame Inspect}.
The \textit{Global Browse}, \textit{Clip Search}, and \textit{Frame Inspect} tools are adapted from the original design of DVD~\cite{zhang2025deep} and we refer to their paper for more details.
The \textit{Temporal Search} tool is designed to help the agent directly retrieve video clips and entities based on timestamps, implemented by calling the intervaltree object with \verb|tree.overlap(begin_t, end_t)|.
\cref{tab:supp:action_space} summarizes the action space of our agent.
\input{figures/taba6_agent_tools}
\subsection{Training Details}
\input{figures/taba1_hparam_video}
% \input{figures/taba2_hparam_image}
We use the Azure OpenAI service to access API of the GPT-series models for data generation and evaluation.
For training, we utilize the LLAMA-Factory framework~\cite{zheng2024unified} for training both Video NKR and Image NKR.
The training hyperparameters for video and image NKR are summarized in \cref{tab:supp:hyperparams_video}.
\section{Additional Experiments}
\label{sec:supp:experiments}
\subsection{Results on LongVideoBench}
\input{figures/taba4_longvideo_acc}
To provide a more comprehensive comparison with existing methods on long video understanding, we further evaluate Video NKR on the LongVideoBench~\cite{wu2024longvideobench} benchmark. The LongVideoBench benchmark contains 6678 questions from 3763 videos, covering a wide range of video durations from a few seconds to an hour. In particular, we focus on the longest subset of the benchmark, which includes videos with durations ranging from 900 seconds to 3600 seconds (denoted as the LongVideoBench-Long). This subset has 564 questions from 188 videos. All evaluations are conducted only using video content (i.e., without using audio narrations or subtitles).
\cref{tab:supp:longvideo_acc} summarizes the results of Video NKR and other baseline methods on the LongVideoBench-Long set.
The results demonstrate similar findings to experiments on LVBench in the main paper: Video NKR achieves similar or even superior performance on long video understanding tasks while being significantly more efficient in terms of computation and memory usage.
\subsection{More Analysis on NKR}
\paragraph{Knowledge Validation.}
To validate that our NKR framework indeed encodes content into a generalizable knowledge representation that LLMs can effectively leverage for complex understanding tasks, we conduct two additional analyses on image understanding task.
First, we perform a comparative experiment on image understanding to demonstrate that NKR serves as a more effective input format for LLMs than raw content.
We establish a strong textual baseline where a complete, detailed description of an image is directly fed into an LLM. As our NKR is optimized from textual descriptions, this comparison is intentionally designed to be fair: by using the same source information (textual descriptions and text-based Q\&A pairs), we isolate the impact of the representation format. Specifically, we design the textual baseline by concatenating all dense descriptions and the generated Q\&A pairs of an image into a single long text as in-context examples. Then we prompt the LLM to answer the MME-RealWorld test questions based on this long text input.
We refer to this baseline as the "text-based LLM".
The text-based LLM baseline achieves an accuracy of 19.6\% on the MME-RealWorld test set described in the main text. In contrast, our Image-NKR, optimized from the very same textual descriptions, reaches a significantly higher accuracy of 35.4\%. Notably, the performance gain is most pronounced on the reasoning subset of questions (17.2\% vs 32.5\%). The results strongly indicate that an implicit neural representation is a more potent and suitable format for LLMs to perform complex understanding tasks.
\input{figures/figa5_knn_search}
Second, we investigate whether NKR’s success stems from mere memorization of answers to highly similar questions in the synthesized Q\&A pairs used during optimization. To test this, we perform a k-nearest neighbor search, using the test questions as queries over the entire generated Q\&A dataset ($\mathcal{D}_{\text{qa}}$). \cref{fig:supp:knn_samples} demonstrates that the retrieved results are semantically related but not identical to the test questions. This suggests that the performance gains arise from NKR’s ability to generalize knowledge, i.e., to activate relevant internal knowledge to answer a given query, rather than simply recalling previously seen pairs.
\paragraph{Ablation of Different Teachers.}
To validate the robustness of the proposed NKR against variations in the QA synthesis process, we employ three models with varying capabilities (GPT-4o mini, GPT-4.1, and GPT-5.4) to generate 150 textual samples each. We then train our model on the image dataset mentioned in the main paper using these synthesized texts. As presented in \cref{tab:iablation_teacher}, the results demonstrate that our approach maintains strong robustness across different QA synthesis models.
\input{figures/taba7_ablation_teacher}
\paragraph{Comparison with In-Context Baseline.}
To validate the effectiveness of the NKR paradigm, we further compare its performance against an in-context baseline that is provided with the exact same textual data (150 textual samples), either injected into the model's context window or absorbed via NKR. As shown in \cref{tab:incontext}, NKR substantially outperforms the in-context baseline across both models, with a particularly pronounced gap on the Reasoning subset (e.g., +15.3 points for Qwen2.5-7B and +37.1 points for Qwen3-8B). This suggests that NKR learns a more generalizable internal representation rather than merely recalling matched text.
\input{figures/taba8_incontext}
\input{figures/figa1_prompts_dense_desc}
\input{figures/figa2_prompts_shallow_qa}
\input{figures/figa3_prompts_video_qa}
\input{figures/figa4_prompts_video_qa_category}
\begin{algorithm}
\caption{Agentic Video-level Q\&A Generation.}
\label{alg:supp:video-qa}
\begin{algorithmic}
\STATE \textbf{Input:} Video database $D$, LLM $M$, tool set $\mathcal{T}$, stop sign \texttt{STOP}, question category $C$, max steps $N$.
\STATE \textbf{Output:} a QA pair $\{Q, A\}$ for video $V$.
\STATE Initialize history $H_0 \gets \{C\}$
\FOR{$i \gets 1$ to $N$}
\STATE $R_i \gets M.\textsc{reason}(H_{i-1})$
\STATE $T_i \gets M.\textsc{call}(R_i, H_{i-1})$ where $T_i \in \mathcal{T} \cup \texttt{STOP}$
\IF{$T_i$ is \texttt{STOP}}
\STATE \textbf{break}
\ENDIF
\STATE $O_i \gets T_i(\mathcal{D})$
\STATE $H_i \gets H_{i-1} \cup \{(R_i, A_i, O_i)\}$
\ENDFOR
\STATE \textbf{return} $\{Q, A\} = M.\textsc{format}(H_i)$
\end{algorithmic}
\end{algorithm}
\begin{table}[htbp]
\centering
\caption{Action space overview of our QA generation agent. The first four actions are from our toolset and the final STOP action is designed as a stop criterion.}
\label{tab:supp:action_space}
\resizebox{0.5\linewidth}{!}{%
\begin{tabular}{ll}
\toprule
\textbf{Action} & \textbf{Parameter} \\
\midrule
\textsc{Global Browse} & video database $\mathcal{D}$ \\
& user query $Q$ \\
\midrule
\textsc{Clip Search} & video database $\mathcal{D}$ \\
& agent synthesized query $\hat{Q}$ \\
& return top-$k$ captions \\
\midrule
\textsc{Frame Inspect} & raw video $\mathcal{V}$ \\
& agent synthesized query $\hat{Q}$ \\
& temporal range $[t_s, t_e]$ \\
\midrule
\textsc{Temporal Search} & video database $\mathcal{D}$ \\
& temporal range $[t_s, t_e]$ \\
\midrule
\textsc{STOP} & Stop the process and return \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[htbp]
\centering
\caption{Hyperparameters for training the Neural Knowledge Representation (NKR).}
\label{tab:supp:hyperparams_video}
\resizebox{0.65\linewidth}{!}{%
\begin{tabular}{l|c|c}
\toprule
\textbf{Hyperparameter} & \textbf{Video} & \textbf{Image} \\
\midrule
\multicolumn{3}{c}{\textit{Model Configuration}} \\
\midrule
VLM Backbone & Qwen3-14B & Qwen2.5-VL-7B \\
LoRA Rank ($r$) & 256 & 32\\
LoRA Alpha ($\alpha$) & 512 & 64\\
\midrule
\multicolumn{3}{c}{\textit{Training Configuration}} \\
\midrule
Optimizer & AdamW & AdamW \\
Learning Rate & 2e-4 & 1e-4 \\
LR Scheduler & Cosine Annealing & Cosine Annealing \\
Warmup Ratio & 0.05 epoch & 0.1 epoch \\
Global Batch Size & 32 & 2 \\
Per-device Batch Size & 4 & 1 \\
Gradient Accumulation Steps & 2 & 1 \\
GPU Specification & NVIDIA A100-40G & NVIDIA A100-40G \\
Number of GPUs & 4 & 2 \\
Epochs & 5 & 3 \\
Weight Decay & 0.1& 0.0 \\
Mixed Precision & bf16& bf16 \\
\bottomrule
\end{tabular}%
}
\end{table}
\begin{table}[t]
\centering
\caption{Accuracy, model size, input size, and inference speed comparison on LongVideoBench-Long benchmark. The input size refer to either max number of input frames or input video fps (whichever larger). Memory overhead is measured in GBytes; we measure the KV-cache size for tokenized apporaches during inference. our NKR weights are merged into the VLM at runtime, thus it has zero extra memory overhead.
Inference speed is measured on a single NVIDIA A100 GPU with 80G memory for open-sourced models, while Qwen2.5-VL-32B uses 2$\times$A100 due to its memory requirement.
}
\label{tab:supp:longvideo_acc}
\resizebox{0.8\linewidth}{!}{%
\begin{tabular}{c|cccc}
\toprule
\textbf{Methods} & \makecell{\textbf{Input Size} \\ \textbf{(fps or max frames)}} & \makecell{\textbf{Time} \\ \textbf{(s/query)}} & \makecell{\textbf{Memory Overhead} \\ \textbf{(GB, at Runtime)}} & \textbf{Accuracy} \\
\midrule
\small GPT-4o-20241120 & \small 60 frames & \small 43.0 & \small \makecell{Unknown\\(Large)} & \small 60.9 \\
\midrule
\small Qwen2.5-VL-7B & \multirow{2}{*}{\small \makecell{2fps \\ ($<$768 frames)}} & \small 30.0 & \small 2.5 & \small 45.2 \\
\small Qwen2.5-VL-32B & & \small 45.6\textsuperscript{*} & \small 11.3 & \small 48.4 \\
\midrule
\small AdaReTaKe-7B & \small \makecell{2fps \\ ($<$2048 frames)} & \small 45.0 & \small 0.9 & \small 55.5 \\
\midrule
\small \makecell{Qwen3-NKR-14B-r256 \\ (Ours)} & \small N/A & \small 0.33 & \small 0 & \small 52.9\\
\bottomrule
\end{tabular}%
}
\end{table}
% [Style A] 极浅青色 (参考 Category 5) - 用于 Inference Question
\definecolor{colorA_bg}{RGB}{235, 250, 250} % 极浅青背景
\definecolor{colorA_frame}{RGB}{120, 180, 185} % 边框色
\definecolor{colorA_title}{RGB}{0, 80, 90}
% [Style B] 极浅米色 (参考 Category 6) - 用于 KNN Search
\definecolor{colorB_bg}{RGB}{253, 248, 235} % 极浅米黄背景
\definecolor{colorB_frame}{RGB}{210, 180, 140} % 边框色
\definecolor{colorB_title}{RGB}{140, 90, 40}
% [Style C] 极浅绿色 (风格类似但用于正确预测) - 用于 Prediction
\definecolor{colorC_bg}{RGB}{240, 252, 240} % 极浅绿背景
\definecolor{colorC_frame}{RGB}{130, 200, 130} % 边框色
\definecolor{colorC_title}{RGB}{40, 100, 40}
% 通用白色大背景
\definecolor{main_bg}{RGB}{255, 255, 255}
\definecolor{main_frame}{RGB}{230, 230, 230}
\begin{figure*}[t]
\centering
% 外层容器
\begin{tcolorbox}[
enhanced,
width=\textwidth,
colback=main_bg,
colframe=main_frame,
boxrule=0.8pt,
arc=2pt,
left=2mm, right=2mm, top=2mm, bottom=2mm
]
% =================== 布局开始 ===================
% 使用 minipage 分左右两栏
% --- [左栏] Question + KNN Results (占宽 58%) ---
\begin{minipage}[t]{0.58\textwidth}
\vspace{0pt} % 用于对齐
% 1. Inference Question Box
\begin{tcolorbox}[
title={\textbf{Inference Question}},
colback=colorA_bg,
colframe=colorA_frame,
coltitle=colorA_title,
fonttitle=\bfseries\small,
boxrule=0.8pt, arc=2pt,
left=3pt, right=3pt, top=3pt, bottom=3pt
]
\small
\textbf{Q:} What is the number of people in the image? (If a human maintains standing pose or walking, please classify it as pedestrian, otherwise, it is classified as a people.)
\par\vspace{3pt}
\textbf{Options:} \\
(A) 7 \quad (B) 8 \quad (C) 10 \quad \textbf{(D) 1} \quad \\
(E) The image does not feature the people
\par\vspace{3pt}
\textbf{Type:} Perception
\end{tcolorbox}
\vspace{-1mm}
% 2. KNN Search Results (大框套小框)
\begin{tcolorbox}[
title={\textbf{KNN Search Results (Top-3 Retrieved Samples)}},
colback=colorB_bg,
colframe=colorB_frame,
coltitle=colorB_title,
fonttitle=\bfseries\small,
boxrule=0.8pt, arc=2pt,
left=2pt, right=2pt, top=2pt, bottom=2pt
]
% --- Inner Box 1 ---
\begin{tcolorbox}[
colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt,
left=2pt, right=2pt, top=1pt, bottom=1pt%, margin=0pt
]
\scriptsize
\textbf{[Top-1] Sim: 0.513} \\
\textit{Q:} Are there more than five people visible in the image? \\
\textit{A:} No, there is only one person visible in the entire image.
\end{tcolorbox}
\vspace{-3mm}
% --- Inner Box 2 ---
\begin{tcolorbox}[
colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt,
left=2pt, right=2pt, top=1pt, bottom=1pt
]
\scriptsize
\textbf{[Top-2] Sim: 0.462} \\
\textit{Q:} How many total vehicles (including moving and parked) are visible...? \\
\textit{A:} There are seven vehicles visible in the image, including both parked and moving.
\end{tcolorbox}
\vspace{-3mm}
% --- Inner Box 3 ---
\begin{tcolorbox}[
colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt,
left=2pt, right=2pt, top=1pt, bottom=1pt
]
\scriptsize
\textbf{[Top-3] Sim: 0.457} \\
\textit{Q:} Is there a pedestrian standing on the street or sidewalk? \\
\textit{A:} Yes, there is one pedestrian visible on the sidewalk on the right side of the street.
\end{tcolorbox}
\end{tcolorbox}
\end{minipage}
\hfill % 左右栏之间的间距
% --- [右栏] Image + Prediction (占宽 40%) ---
\begin{minipage}[t]{0.40\textwidth}
\vspace{0pt} % 用于对齐
% 1. Image
\centering
\includegraphics[width=\linewidth]{figures/images/knn_1.pdf}
\vspace{6mm}
% 2. Model Prediction
\begin{tcolorbox}[
title={\textbf{Model Prediction}},
colback=colorC_bg,
colframe=colorC_frame,
coltitle=colorC_title,
fonttitle=\bfseries\small,
boxrule=0.8pt, arc=2pt,
left=3pt, right=3pt, top=3pt, bottom=3pt
]
\small
\textbf{Final Answer:} (D) 1 \hfill \textcolor{green!60!black}{\Large\boldmath\checkmark}
\end{tcolorbox}
\end{minipage}
\end{tcolorbox} % End Outer Box
% % --- Example 2 ---
% \vspace{5mm}
% \begin{tcolorbox}[
% enhanced,
% width=\textwidth,
% colback=main_bg,
% colframe=main_frame,
% boxrule=0.8pt,
% arc=2pt,
% left=2mm, right=2mm, top=2mm, bottom=2mm
% ]
% % Example 2 内容 (结构同上)
% \begin{minipage}[t]{0.58\textwidth}
% \vspace{0pt}
% % Question
% \begin{tcolorbox}[title={\textbf{Inference Question}}, colback=colorA_bg, colframe=colorA_frame, coltitle=colorA_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=3pt, right=3pt, top=3pt, bottom=3pt]
% \small
% \textbf{Q:} What are the characters preparing to do?
% \par\vspace{3pt}
% \textbf{Options:} \\
% (A) They are preparing to watch another movie. \\
% \textbf{(B) They are preparing to enjoy desserts.} \\
% (C) They are preparing to say goodbye. \\
% (D) They are preparing to attend a wedding. \\
% (E) The image does not feature the related information.
% \par\vspace{3pt}
% \textbf{Type:} Reasoning
% \end{tcolorbox}
% \vspace{-1mm}
% % KNN Search
% \begin{tcolorbox}[title={\textbf{KNN Search Results (Top-3 Retrieved Samples)}}, colback=colorB_bg, colframe=colorB_frame, coltitle=colorB_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=2pt, right=2pt, top=2pt, bottom=2pt]
% \begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
% \scriptsize \textbf{[Top-1] Sim: 0.490} \\
% \textit{Q:} Which character is seen handing out desserts to the group during the dessert scene? \\
% \textit{A:} Betty is handing out desserts to the group.
% \end{tcolorbox}
% \vspace{-3mm}
% \begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
% \scriptsize \textbf{[Top-2] Sim: 0.483} \\
% \textit{Q:} What kind of desserts is Betty bringing into the room?\\
% \textit{A:} Betty is bringing in a plate of cookies.
% \end{tcolorbox}
% \vspace{-3mm}
% \begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
% \scriptsize \textbf{[Top-3] Sim: 0.479} \\
% \textit{Q:} What action does Betty take after the home movie viewing?\\
% \textit{A:} She brings dessert as a reward for sitting through the home movies.
% \end{tcolorbox}
% \end{tcolorbox}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.40\textwidth}
% \vspace{0pt}
% \centering
% \includegraphics[width=0.65\linewidth]{figures/images/knn_2.pdf}
% \vspace{0mm}
% \begin{tcolorbox}[title={\textbf{Model Prediction}}, colback=colorC_bg, colframe=colorC_frame, coltitle=colorC_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=3pt, right=3pt, top=3pt, bottom=3pt]
% \small
% \textbf{Final Answer:} (B) They are preparing to enjoy desserts. \hfill \textcolor{green!60!black}{\Large\boldmath\checkmark}
% \end{tcolorbox}
% \end{minipage}
% \end{tcolorbox}
% --- Example 3 ---
\vspace{5mm}
\begin{tcolorbox}[
enhanced,
width=\textwidth,
colback=main_bg,
colframe=main_frame,
boxrule=0.8pt,
arc=2pt,
left=2mm, right=2mm, top=2mm, bottom=2mm
]
% Example 3 内容 (结构同上)
\begin{minipage}[t]{0.58\textwidth}
\vspace{0pt}
% Question
\begin{tcolorbox}[title={\textbf{Inference Question}}, colback=colorA_bg, colframe=colorA_frame, coltitle=colorA_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=3pt, right=3pt, top=3pt, bottom=3pt]
\small
\textbf{Q:} What material is the garbage bin far away from green plants made of?
\par\vspace{3pt}
\textbf{Options:} \\
\textbf{ (A) plastic} \quad (B) iron \quad (C) aluminium \quad (D) concrete \\
(E) The image does not feature the object
\par\vspace{3pt}
\textbf{Type:} Reasoning
\end{tcolorbox}
\vspace{-1mm}
% KNN Search
\begin{tcolorbox}[title={\textbf{KNN Search Results (Top-3 Retrieved Samples)}}, colback=colorB_bg, colframe=colorB_frame, coltitle=colorB_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=2pt, right=2pt, top=2pt, bottom=2pt]
\begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
\scriptsize \textbf{[Top-1] Sim: 0.627} \\
\textit{Q:} What is the main color of the waste bins near the trash pile? \\
\textit{A:} The waste bins near the trash pile are primarily black.
\end{tcolorbox}
\vspace{-3mm}
\begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
\scriptsize \textbf{[Top-2] Sim: 0.595} \\
\textit{Q:} What is the color of the trash bin directly to the right of the central trash pile?\\
\textit{A:} Blue.
\end{tcolorbox}
\vspace{-3mm}
\begin{tcolorbox}[colback=white, colframe=colorB_frame!50, boxrule=0.5pt, arc=1pt, left=2pt, right=2pt, top=1pt, bottom=1pt]
\scriptsize \textbf{[Top-3] Sim: 0.579} \\
\textit{Q:}What type of bin is located close to the trash pile? \\
\textit{A:} A small blue plastic trash bin is located close to the trash pile.
\end{tcolorbox}
\end{tcolorbox}
\end{minipage}
\hfill
\begin{minipage}[t]{0.40\textwidth}
\vspace{0pt}
\centering
\includegraphics[width=\linewidth]{figures/images/knn_3.pdf}
\vspace{1mm}
\begin{tcolorbox}[title={\textbf{Model Prediction}}, colback=colorC_bg, colframe=colorC_frame, coltitle=colorC_title, fonttitle=\bfseries\small, boxrule=0.8pt, arc=2pt, left=3pt, right=3pt, top=3pt, bottom=3pt]
\small
\textbf{Final Answer:} (A) plastic \hfill \textcolor{green!60!black}{\Large\boldmath\checkmark}
\end{tcolorbox}
\end{minipage}
\end{tcolorbox}
\caption{KNN retrieval results for the inference question. We display the top-3 most similar QA pairs retrieved from the training set based on the query embedding.}
\label{fig:supp:knn_samples}
\end{figure*}
\begin{table}[htbp]
\centering
\tiny
\caption{Ablation study on teacher models generated via Agentic Knowledge Distillation. The experiment is conducted on Qwen3-NKR-8B-r32.}
\label{tab:iablation_teacher}
\resizebox{0.5\linewidth}{!}{
\begin{tabular}{l|ccc}
\toprule
\textbf{Method} & \textbf{Perception} & \textbf{Reasoning} & \textbf{Overall} \\
\midrule
GPT-4o mini & 37.0 & 44.3 & 40.3 \\
GPT-4.1 & 31.9 & 46.4 & 38.4 \\
GPT-5.4 & 34.9 & 45.9 & 40.5 \\
\bottomrule
\end{tabular}
}
\vspace{-1.5em}
\end{table}
\begin{table}[htbp]
\centering
% \tiny
\caption{Performance comparison between NKR and the in-context baseline on Qwen2.5-7B and Qwen3-8B.}
\label{tab:incontext}
\resizebox{0.5\linewidth}{!}{
\begin{tabular}{l|ccc}
\toprule
\textbf{Method} & \textbf{Perception} & \textbf{Reasoning} & \textbf{Overall} \\
\midrule
Qwen2.5-InContext-7B & 22.1 & 17.2 & 19.6 \\
Qwen2.5-NKR-7B-r32 & 38.4 & 32.5 & 35.4 \\
Qwen3-InContext-8B & 24.4 & 9.3 & 17.6 \\
Qwen3-NKR-8B-r32 & 31.9 & 46.4 & 38.4 \\
\bottomrule
\end{tabular}
}
\vspace{-1.5em}
\end{table}
%% filepath:
% =================================================================
% FIGURE FOR VIDEO AND IMAGE CAPTIONING PROMPTS
% =================================================================
\begin{figure*}[htbp]
\centering
% --- Video Captioning Prompt ---
\begin{tcolorbox}[
width=\linewidth, colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Multi-level Video Captioning},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=4mm, right=4mm, top=2mm, bottom=2mm
]
\scriptsize % Use scriptsize for maximum compression
\textbf{\textcolor{blue!60!black}{System Role \& Instructions:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
You are a professional video annotator. Based on the input video clip, generate three levels of natural, fluent, and descriptive captions in English.
\textbf{Output Format:} Please output a JSON with fields: \texttt{"caption\_high\_level"}, \texttt{"caption\_mid\_level"}, \texttt{"caption\_low\_level"}.
\end{tcolorbox}
\textbf{\textcolor{purple!80!black}{Guidelines:}}
\begin{tcolorbox}[enhanced, colback=orange!5!white, arc=1mm, boxrule=0.5pt, colframe=orange!20!white, left=2mm, right=2mm, top=2mm, bottom=2mm]
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{3pt} \setlength{\parsep}{3pt} \setlength{\leftmargin}{1em}
\item Capture temporal dynamics (movements, actions, transitions).
\item Describe scene evolution and remain factual.
\item Handle coarse captions by translating factual entities to English and ignoring irrelevant expressions.
\end{itemize}
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, left=2mm, right=2mm, top=2mm, bottom=2mm]
\textbf{Caption Levels:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{3pt} \setlength{\parsep}{3pt} \setlength{\leftmargin}{1em}
\item \textbf{Level 1 ($<$32 tokens):} A very short summary of the main subject.
\item \textbf{Level 2 (32-128 tokens):} A richer description covering entities, actions, and context.
\item \textbf{Level 3 (128-256 tokens):} A comprehensive narrative detailing attributes, spatial relations, and temporal evolution.
\end{itemize}
\end{tcolorbox}
\textbf{\textcolor{purple!80!black}{Few-shot Example:}}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\textbf{\textcolor{green!60!black}{User Input (Example):}} \texttt{[Example Video Frames...]}
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Assistant Output (Example):}} \texttt{\{"caption\_high\_level": "...", ...\}}
\end{tcolorbox}
\end{tcolorbox}
\vspace{1mm} % Space between the two figures
% --- Image Captioning Prompt ---
\begin{tcolorbox}[
width=\linewidth, colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Triple-Level Image Captioning},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=4mm, right=4mm, top=2mm, bottom=2mm
]
\scriptsize % Use scriptsize for maximum compression
\textbf{\textcolor{blue!60!black}{System Role:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, left=2mm, right=2mm, top=2mm, bottom=2mm]
You are a professional image annotator. Based on the input image (and optionally a coarse caption), generate three levels of natural, fluent, and descriptive captions in English.
\end{tcolorbox}
\textbf{\textcolor{purple!80!black}{Guidelines:}}
\begin{tcolorbox}[enhanced, colback=orange!5!white, arc=1mm, boxrule=0.5pt, colframe=orange!20!white, left=2mm, right=2mm, top=2mm, bottom=2mm]
\textbf{Handling the Coarse Caption}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{3pt} \setlength{\parsep}{3pt} \setlength{\leftmargin}{1em}
\item The coarse caption may be in \textbf{any language}. Always \textbf{translate or normalize} it into English concepts.
\item Ignore any \textbf{emotional, vague, or irrelevant expressions}. Do \textbf{not} copy non-English text unless it is \textbf{visually present}.
\end{itemize}
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, left=2mm, right=2mm, top=2mm, bottom=2mm]
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{3pt} \setlength{\parsep}{3pt} \setlength{\leftmargin}{1em}
\item \textbf{Level 1 Caption ($<$32 tokens):} Very short and concise, focusing only on the \textbf{main subject or scene}.
\item \textbf{Level 2 Caption (32-128 tokens):} An \textbf{extended but still compact} description. Add \textbf{object attributes}, \textbf{basic spatial layout}, and \textbf{environmental context}.
\item \textbf{Level 3 Caption (128-256 tokens):} A \textbf{fully detailed narrative}. Include: \textbf{fine-grained attributes}, \textbf{spatial relationships}, \textbf{object interactions}, and \textbf{environmental details}.
\end{itemize}
\end{tcolorbox}
\end{tcolorbox}
\caption{Prompt used for multi-level video and image captioning.}
\label{fig:supp:prompt_caption}
\end{figure*}
% =================================================================
% FIGURE FOR ENTITY REGISTRY PROMPT
% =================================================================
\begin{figure*}[htbp]
\centering
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Two-Stage Prompt for Entity Registry Generation},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=4mm, right=4mm, top=1mm, bottom=2mm % Reduced top/bottom padding
]
\footnotesize % Use smaller font for the entire box content
% --- STEP 1: ENTITY EXTRACTION ---
\begin{tcolorbox}[
width=\linewidth, enhanced, colback=white, colframe=blue!70!black,
title=\textbf{Step 1: Entity Extraction per Clip},
fonttitle=\footnotesize\bfseries, colbacktitle=blue!10, coltitle=black,
top=1mm, bottom=1mm % Reduced padding
]
\textbf{\textcolor{blue!60!black}{System Role \& Instructions:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, top=1mm, bottom=1mm]
You are a professional video entity extractor. Analyze the input video clip and detect all entities. For each entity, provide its name, appearance, identity, and the first seen timestamp within this clip. Avoid duplicates and output a JSON object.
\end{tcolorbox}
\textbf{\textcolor{purple!80!black}{Few-shot Example:}}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, top=1mm, bottom=1mm]
\textbf{\textcolor{green!60!black}{User Input (Example):}} \texttt{[Video Frames]} with start time \texttt{00:00:04}.
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Assistant Output (Example):}}
\begin{verbatim}
{
"entities": [
{
"name": "Dog_A", "appearance": [...],
"identity": [...],
"first_seen": "00:00:04"
},
...
]
}
\end{verbatim}
\end{tcolorbox}
\end{tcolorbox}
\begin{center}
\Large \textcolor{gray!70}{$\boldsymbol{\downarrow}$}
\end{center}
% --- STEP 2: ENTITY MERGING ---
\begin{tcolorbox}[
width=\linewidth, enhanced, colback=white, colframe=orange!80!black,
title=\textbf{Step 2: Global Entity Merging},
fonttitle=\footnotesize\bfseries, colbacktitle=orange!10, coltitle=black,
top=1mm, bottom=1mm % Reduced padding
]
\textbf{\textcolor{blue!60!black}{System Role \& Instructions:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, top=1mm, bottom=1mm]
You are a professional assistant for merging entities. Your input is a list of entity registries from different clips of the same video. Merge duplicate entities, keep the earliest first seen time, and create a timestamps field to store all occurrence times.
\end{tcolorbox}
\textbf{\textcolor{purple!80!black}{Few-shot Example:}}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, top=1mm, bottom=1mm]
\textbf{\textcolor{green!60!black}{User Input (Example):}} A list of entities, e.g., \\ \texttt{[{"name": "Dog\_X", ...}, {"name": "Dog\_Y", ...}, ...]}
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Assistant Output (Example):}}
\begin{verbatim}
{
"entities": [
{
"name": "Dog_A", ...,
"first_seen": "00:00:00",
"timestamps": ["00:00:00", "..."]
},
{
"name": "frisbee_A", ...,
"first_seen": "00:00:04",
"timestamps": ["00:00:04", "..."]
},
...
]
}
\end{verbatim}
\end{tcolorbox}
\end{tcolorbox}
\end{tcolorbox}
\caption{Prompt used for entity generation and merging in a two-stage manner.}
\label{fig:supp:prompt_entity}
\end{figure*}
% =================================================================
% FIGURE FOR CLIP-LEVEL VIDEO QA PROMPT (DETAILED)
% =================================================================
\begin{figure*}[htbp]
\centering
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Clip-level Video QA Generation},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=4mm, right=4mm, top=2mm, bottom=2mm
]
\footnotesize % Use footnotesize for better readability
% --- SYSTEM ROLE & INSTRUCTIONS ---
\textbf{\textcolor{blue!60!black}{System Role \& Instructions:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
You are a careful dataset constructor that converts one video-clip into high-quality open-ended Q\&A pairs. Given a video clip with multi-level captions and entities, produce a diverse set of Q\&A that fully covers the clip across scales and aspects.
\end{tcolorbox}
% --- GUIDELINES ---
\textbf{\textcolor{purple!80!black}{Guidelines:}}
\begin{tcolorbox}[enhanced, colback=orange!5!white, arc=1mm, boxrule=0.5pt, colframe=orange!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\textbf{Question Composition:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \textbf{Content Granularity:} High-level (gist), Mid-level (attributes, relations), Low-level (fine-grained details).
\item \textbf{Temporal Granularity:} Short (single moment), Medium (sequence), Long (multi-event).
\end{itemize}
\textbf{Timestamp Rules:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Must be within clip's time range.
\item Use float seconds.
\item Prefer ranges for sustained actions.
\item Must be included in the question text.
\end{itemize}
\textbf{Hints (Workflow):}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Parse all captions to collect atomic facts.
\item Resolve facts into a fused, contradiction-free set.
\item Draft questions spanning all granularities.
\item Enforce timestamp rules.
\item Filter for unique, answerable questions.
\item Output JSON only.
\end{enumerate}
\end{tcolorbox}
% --- FEW-SHOT EXAMPLE ---
\textbf{\textcolor{purple!80!black}{Few-shot Example:}}
% --- User Input ---
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\textbf{\textcolor{green!60!black}{User Input (Example):}}
\begin{verbatim}
{
"caption_high_level": "Chefs prepare food in a busy professional kitchen.",
"caption_mid_level": "Inside a bustling commercial kitchen, multiple chefs
clad in white...",
"caption_low_level": "In this dynamic video sequence within a professional
kitchen, chefs...",
"clip_begin_time": "205.0", "clip_end_time": "210.0",
"entities": [{"name": "chef_A", ...}, {"name": "chef_B", ...}]
}
\end{verbatim}
\end{tcolorbox}
% --- Assistant Output ---
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Assistant Output (Example):}}
\begin{verbatim}
{
"qa": [
{
"question": "Between 205.0 - 210.0, what environment is depicted?",
"answer": "A busy professional kitchen."
},
{
"question": "What are the chefs wearing during 205.0 - 210.0?",
"answer": "White shirts and black aprons."
},
...
]
}
\end{verbatim}
\end{tcolorbox}
\end{tcolorbox}
\caption{Prompt for generating clip-level video QA pairs.}
\label{fig:supp:prompt_qa}
\end{figure*}
% =================================================================
% FIGURE FOR IMAGE QA GENERATION PROMPT
% =================================================================
\begin{figure*}[htbp]
\centering
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Image QA Generation},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=3mm, right=3mm, top=2mm, bottom=2mm
]
\footnotesize % Use footnotesize for better readability
% --- SYSTEM ROLE ---
\textbf{\textcolor{blue!60!black}{System Role:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
You are a careful dataset constructor that converts an image into a diverse set of 150-200 high-quality, open-ended Q\&A pairs with a strong emphasis on low-level perception. Base all Q\&A strictly on visual evidence.
\end{tcolorbox}
% --- CORE GUIDELINES & STRATEGIES ---
\textbf{\textcolor{purple!80!black}{Core Guidelines \& Strategies:}}
\begin{tcolorbox}[enhanced, colback=orange!5!white, arc=1mm, boxrule=0.5pt, colframe=orange!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\begin{minipage}[t]{0.48\linewidth}
\setlength{\parskip}{2pt}
\textbf{Perception Priority:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \textbf{Objects:} Presence, categories, parts.
\item \textbf{Attributes:} Colors, materials, textures, states.
\item \textbf{Layout:} Relative positions, depth, orientation.
\item \textbf{Counting:} Total and conditional counts.
\item \textbf{OCR:} Transcribe all visible text exactly.
\item \textbf{Details:} Logos, icons, UI elements, occlusion.
\end{itemize}
\textbf{Perception-Focused Patterns:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \textbf{Presence:} "Is there a [object]...?"
\item \textbf{Attribute:} "What color/material is the [object]?"
\item \textbf{Relation:} "What is to the left of [object]?"
\item \textbf{Counting:} "How many [objects] are there?"
\item \textbf{Part-based:} "Does the [device] have a handle?"
\item \textbf{OCR:} "What text is on the [sign]?"
\item \textbf{Pose:} "Which way is the [person] facing?"
\end{itemize}
\end{minipage}\hfill
\begin{minipage}[t]{0.48\linewidth}
\textbf{Question Diversity:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \textbf{Types:} What, Where, How many, etc.
\item \textbf{Reasoning:} Multi-hop, causal, comparative.
\item \textbf{Variations:} Paraphrase facts into multiple question forms.
\end{itemize}
\textbf{Negative Examples (\textasciitilde15\%):}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \textbf{Format:} Ask a question with a plausible but false premise.
\item \textbf{Answer:} Start with "No," then provide the correct fact.
\item \textbf{Errors:} Wrong attributes, counts, relations, etc.
\end{itemize}
\textbf{Systematic Coverage Strategy:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Extract all entities and their attributes.
\item Extract all actions and spatial relations.
\item Extract all visible text.
\item Cross-reference to create complex questions.
\end{enumerate}
\end{minipage}
\end{tcolorbox}
% --- FEW-SHOT EXAMPLE ---
\textbf{\textcolor{purple!80!black}{Few-shot Example Snippet:}}
\begin{tcolorbox}[enhanced, colback=green!5!white, arc=1mm, boxrule=0.5pt, colframe=green!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\textbf{\textcolor{green!60!black}{User Input:}} An image is provided.
\end{tcolorbox}
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Assistant Output (Example):}}
\begin{verbatim}
{
"qa": [
{ "question": "How many people are visible?", "answer": "Three." },
{ "question": "What color is the sweater on the left?", "answer": "Beige." },
{ "question": "Is the building on the right made of brick?",
"answer": "No, it is made of concrete." },
...
]
}
\end{verbatim}
\end{tcolorbox}
\end{tcolorbox}
\caption{Prompt for generating image QA pairs.}
\label{fig:supp:prompt_image_qa}
\end{figure*}
% =================================================================
% FIGURE FOR VIDEO QA AGENT PROMPT
% =================================================================
\begin{figure*}[htbp]
\centering
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Video-level QA Generation (General Instructions)},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=3mm, right=3mm, top=2mm, bottom=2mm
]
\footnotesize % Use footnotesize for better readability
% --- GOAL ---
\textbf{\textcolor{blue!60!black}{Goal:}}
\begin{tcolorbox}[enhanced, colback=blue!5!white, arc=1mm, boxrule=0.5pt, colframe=blue!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
You are a helpful teacher assistant to teach others to exploring, reasoning and understanding a long video. To achieve this goal, your task is to generate a set of questions with answers, based your comprehensive understanding of the video. You must ensure each question is unique and covers different aspects of the video. Each question you generated should be a multiple choice question with four choices, with only one as correct answer and three distractors.
\end{tcolorbox}
% --- CORE DIRECTIVES ---
\textbf{\textcolor{purple!80!black}{Core Directives:}}
\begin{tcolorbox}[enhanced, colback=orange!5!white, arc=1mm, boxrule=0.5pt, colframe=orange!20!white, left=2mm, right=2mm, top=1mm, bottom=1mm]
\setlength{\parskip}{2pt}
\textbf{Strict Rules:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Plan extensively before each function call, and reflect on outcomes.
\item Only use arguments from user or prior function outputs.
\item Inspect video content with tools; do not guess.
\item Timestamps can be 'HH:MM:SS' or 'MM:SS'.
\end{itemize}
\textbf{Tools:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item \texttt{global\_browse}: Get global information on events/subjects.
\item \texttt{clip\_search}: Search without a specific timestamp.
\item \texttt{temporal\_search}: Search with a timestamp.
\item \texttt{frame\_inspect}: Get fine-grained detail for a time range.
\end{itemize}
\textbf{Suggested Workflow (per question):}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Review previous questions for diversity and coverage.
\item Collect information for the target question using tools.
\item Synthesize a question with 4 options (1 correct, 3 plausible distractors).
\item Confirm the question and choices follow design principles.
\item Assign a difficulty level (easy, medium, hard).
\item Call "finish" tool to output the result.
\end{enumerate}
\end{tcolorbox}
% --- OUTPUT FORMAT ---
\textbf{\textcolor{purple!80!black}{Output Format:}}
\begin{tcolorbox}[enhanced, colback=gray!10!white, arc=1mm, boxrule=0.5pt, colframe=gray!30!white, top=1mm, bottom=1mm]
\textbf{\textcolor{gray!60!black}{Each question must have the following fields:}}
\begin{verbatim}
{
"question": "The question text in plain text format.",
"options": { "A": "Option A", "B": "Option B", ... },
"answer": "A single letter of the correct answer label (A, B, C, or D).",
"clue_duration": [["HH:MM:SS", "HH:MM:SS"], ...],
"distractor_rationales": { "B": "Rationale for why B is wrong.", ... },
"difficulty": "'easy', 'medium', or 'hard'"
}
\end{verbatim}
\end{tcolorbox}
\end{tcolorbox}
\caption{Prompt for generating video level QA.
Note this is only the general instruction prompt. The category-specific instructions are demonstrated in the following figures.}
\label{fig:supp:prompt_video_qa_agent}
\end{figure*}
% =================================================================
% FIGURE FOR CATEGORY-SPECIFIC VIDEO QA PROMPTS
% =================================================================
\begin{figure*}[htbp]
\centering
\scalebox{0.9}{%
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Video-level QA Generation (Category-Specific Instructions)},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=3mm, right=3mm, top=2mm, bottom=2mm
]
\footnotesize % Use footnotesize for better readability
% --- Temporal Grounding Card ---
\begin{tcolorbox}[
enhanced, colback=blue!5!white, arc=2mm, boxrule=1pt, colframe=blue!60!white,
title=\textbf{Category 1: Temporal Grounding},
fonttitle=\footnotesize\bfseries, colbacktitle=blue!15!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm,
% space to next
]
Now, please generate high-quality multiple-choice Q\&A pairs that test the ability of temporal grounding (locating events on the timeline, ordering, and durations).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “What happened at time T?”, “How long did X behavior last?”, “What happened immediately after event A?”, etc.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; grounded in nearby or similar scenes, but differ in crucial timing/action details.
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to build a global outline of the video's events.
\item Use \texttt{clip\_search\_tool} over the whole video to find candidate moments or intervals that instantiate time points, durations, or order (before/after/next).
\item For each candidate, Use \texttt{frame\_inspect\_tool} (possibly multiple times) to verify the exact time(s) and local context. Identify the minimal evidence interval(s) and record them as \texttt{clue\_duration}.
\end{enumerate}
\end{tcolorbox}
% --- Summarization Card ---
\begin{tcolorbox}[
enhanced, colback=green!5!white, arc=2mm, boxrule=1pt, colframe=green!60!white,
title=\textbf{Category 2: Summarization},
fonttitle=\footnotesize\bfseries, colbacktitle=green!15!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm,
% space to next
]
Now, please generate high-quality multiple-choice Q\&A pairs to test the ability of summarization (themes, key event developments and stage-level evolution).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “Which of the following best reflects the main objective/theme of this video?”, “Which of the following best summarizes the core development in section X?”, etc.
\item Time expressions can be in HH:MM:SS or relative ordering/duration.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; distractors should reflect “theme drift / missing a crucial phase / overemphasizing minor details.
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to map the video's storyline/chapters.
\item Use \texttt{clip\_search\_tool} to extract candidate evidence across segments.
\item Use \texttt{frame\_inspect\_tool} within each involved segment to pin down key frames.
\end{enumerate}
\end{tcolorbox}
% --- Reasoning Card ---
\begin{tcolorbox}[
enhanced, colback=orange!5!white, arc=2mm, boxrule=1pt, colframe=orange!70!white,
title=\textbf{Category 3: Reasoning},
fonttitle=\footnotesize\bfseries, colbacktitle=orange!15!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm
]
Now, please generate high-quality multiple-choice Q\&A pairs that test the ability of multi-step reasoning (cause effect, intentions, emotions, or plausible prediction grounded in observed evidence).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “Why did the experiment fail?”, “What most likely motivated X to do Y?”, “Given Z happened, what is the most reasonable immediate follow-up?”, etc.
\item Time expressions can be in HH:MM:SS or relative ordering/duration.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; distractors should be \textbf{plausible but contradicted} by a missing or conflicting logical chain.
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to propose candidate reasoning chains (e.g., cause → intermediate → outcome; intention → action → consequence).
\item Use \texttt{clip\_search\_tool} to locate each chain's key nodes.
\item Use \texttt{frame\_inspect\_tool} to verify each node.
\end{enumerate}
\end{tcolorbox}
\end{tcolorbox}
}
\caption{Category-specific instructions for generating video-level QA pairs. These prompts are used in conjunction with the general instructions shown in Figure~\ref{fig:supp:prompt_video_qa_agent}.}
\label{fig:supp:prompt_video_qa_specific}
\end{figure*}
% =================================================================
% FIGURE FOR MORE CATEGORY-SPECIFIC VIDEO QA PROMPTS
% =================================================================
\begin{figure*}[htbp]
\centering
\scalebox{0.9}{%
\begin{tcolorbox}[
width=\linewidth,
colback=gray!5, colframe=gray!60, arc=2mm, boxrule=1pt,
title=\textbf{Prompt for Video-level QA Generation (Additional Category-Specific Instructions)},
fonttitle=\footnotesize\bfseries, colbacktitle=gray!20, coltitle=black,
left=3mm, right=3mm, top=2mm, bottom=2mm
]
\footnotesize % Use footnotesize for better readability
% --- Entity Recognition Card ---
\begin{tcolorbox}[
enhanced, colback=purple!5!white, arc=2mm, boxrule=1pt, colframe=purple!60!white,
title=\textbf{Category 4: Entity Recognition},
fonttitle=\footnotesize\bfseries, colbacktitle=purple!15!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm,
% space to next
]
Now, please generate high-quality multiple-choice Q\&A pairs that test the ability of entity recognition (identifying, distinguishing, and tracking entities, i.e., people/objects/places).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “What color is the clothing of the person arguing with A in the hallway?”, “Who performs action Z (by jersey number/apparel feature)?”, etc.
\item Time expressions can be in HH:MM:SS or relative ordering/duration.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; distractors should be \textbf{visually similar entities} in the same setting but differing in key attributes (color/number/position).
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to list principal entities (people/objects/locations).
\item Use \texttt{clip\_search\_tool} to find discriminative cues (apparel color, number, accessories, spatial position).
\item Use \texttt{frame\_inspect\_tool} to confirm decisive visual features for the referenced entity.
\end{enumerate}
\end{tcolorbox}
% --- Event Understanding Card ---
\begin{tcolorbox}[
enhanced, colback=teal!5!white, arc=2mm, boxrule=1pt, colframe=teal!60!white,
title=\textbf{Category 5: Event Understanding},
fonttitle=\footnotesize\bfseries, colbacktitle=teal!15!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm,
% space to next
]
Now, please generate high-quality multiple-choice Q\&A pairs that test the ability of event understanding (understanding event-level semantics, i.e., stage/type discrimination, and major scene transitions).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “Which description best characterizes this phase?”, “Which stage follows event A?”, etc.
\item Time expressions can be in HH:MM:SS or relative ordering/duration.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; distractors should be \textbf{nearby types/phases} that look similar but are inconsistent with the confirmed evidence.
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to sketch the event timeline and phase segmentation.
\item Use \texttt{clip\_search\_tool} to focus on turning points or scene switches.
\item Use \texttt{frame\_inspect\_tool} to confirm diagnostic visuals (prop/venue change, audience reaction, scoreboard state, etc.).
\end{enumerate}
\end{tcolorbox}
% --- Key Information Retrieval Card ---
\begin{tcolorbox}[
enhanced, colback=brown!10!white, arc=2mm, boxrule=1pt, colframe=brown!70!white,
title=\textbf{Category 6: Key Information Retrieval},
fonttitle=\footnotesize\bfseries, colbacktitle=brown!20!white, coltitle=black,
left=2mm, right=2mm, top=2mm, bottom=2mm, boxsep=1mm, toptitle=1mm, bottomtitle=1mm
]
Now, please generate high-quality multiple-choice Q\&A pairs that test the ability of key information retrieval (\textbf{extracting precise visual details} such as slides, screens, scoreboards, documents).
\textbf{Question Design Principles:}
\begin{itemize}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Ask in forms such as: “What quarterly revenue growth is shown on the slide?”, “What is the scoreboard at time T?”, etc.
\item Time expressions can be in HH:MM:SS or relative ordering/duration.
\item Ensure the coverage of the question sets; do not generate questions that are too similar to each other.
\item Do not include the answer text in the question.
\item The question must not be answerable by common sense alone.
\item The four options must be abstracted at \textbf{comparable granularity} and similar length; distractors should be \textbf{nearby types/phases} that look similar but are inconsistent with the confirmed evidence.
\end{itemize}
\textbf{Instructions:}
\begin{enumerate}
\setlength{\itemsep}{0pt} \setlength{\parskip}{1pt} \setlength{\parsep}{1pt} \setlength{\leftmargini}{1em}
\item Use \texttt{global\_browse\_tool} to select the target entity or events with rich details.
\item Use \texttt{clip\_search\_tool} to find segments containing visual details, such as on-screen text, digits, tables, charts, scoreboards, or reports.
\item Use \texttt{frame\_inspect\_tool} to ensure readability and confirm the exact detail(s).
\end{enumerate}
\end{tcolorbox}
\end{tcolorbox}
}
\caption{Additional category-specific instructions for generating video-level QA pairs. These prompts supplement those in Figure~\ref{fig:supp:prompt_video_qa_specific}.}
\label{fig:supp:prompt_video_qa_specific_2}
\end{figure*}