HCMRM: A High-Consistency Multimodal Relevance Model for Search Ads
Extracted source text · arXiv 2502.05822
%%
%% This is file `sample-sigconf.tex',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% samples.dtx (with options: `all,proceedings,bibtex,sigconf')
%%
%% IMPORTANT NOTICE:
%%
%% For the copyright see the source file.
%%
%% Any modified versions of this file must be renamed
%% with new filenames distinct from sample-sigconf.tex.
%%
%% For distribution of the original source see the terms
%% for copying and modification in the file samples.dtx.
%%
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%%
%%
%% Commands for TeXCount
%TC:macro \cite [option:text,text]
%TC:macro \citep [option:text,text]
%TC:macro \citet [option:text,text]
%TC:envir table 0 1
%TC:envir table* 0 1
%TC:envir tabular [ignore] word
%TC:envir displaymath 0 word
%TC:envir math 0 word
%TC:envir comment 0 0
%%
%% The first command in your LaTeX source must be the \documentclass
%% command.
%%
%% For submission and review of your manuscript please change the
%% command to \documentclass[manuscript, screen, review]{acmart}.
%%
%% When submitting camera ready or to TAPS, please change the command
%% to \documentclass[sigconf]{acmart} or whichever template is required
%% for your publication.
%%
%%
\documentclass[sigconf]{acmart}
\usepackage[ruled]{algorithm2e}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{balance}
%%
%% \BibTeX command to typeset BibTeX logo in the docs
\AtBeginDocument{%
\providecommand\BibTeX{{%
Bib\TeX}}}
%% Rights management information. This information is sent to you
%% when you complete the rights form. These commands have SAMPLE
%% values in them; it is your responsibility as an author to replace
%% the commands and values with those provided to you when you
%% complete the rights form.
\copyrightyear{2025}
\acmYear{2025}
\setcopyright{acmlicensed}
\acmConference[WWW Companion '25]{Companion Proceedings of the ACM Web Conference 2025}{April 28-May 2, 2025}{Sydney, NSW, Australia}
\acmBooktitle{Companion Proceedings of the ACM Web Conference 2025 (WWW Companion '25), April 28-May 2, 2025, Sydney, NSW, Australia}
\acmDOI{10.1145/3701716.3715231}
\acmISBN{979-8-4007-1331-6/25/04}
%%
%% Uncomment \acmBooktitle if the title of the proceedings is different
%% from ``Proceedings of ...''!
%%
%%\acmBooktitle{Woodstock '18: ACM Symposium on Neural Gaze Detection,
%% June 03--05, 2018, Woodstock, NY}
%%
%% Submission ID.
%% Use this when submitting an article to a sponsored event. You'll
%% receive a unique submission ID from the organizers
%% of the event, and this ID should be used as the parameter to this command.
%%\acmSubmissionID{123-A56-BU3}
%%
%% For managing citations, it is recommended to use bibliography
%% files in BibTeX format.
%%
%% You can then either use BibTeX with the ACM-Reference-Format style,
%% or BibLaTeX with the acmnumeric or acmauthoryear sytles, that include
%% support for advanced citation of software artefact from the
%% biblatex-software package, also separately available on CTAN.
%%
%% Look at the sample-*-biblatex.tex files for templates showcasing
%% the biblatex styles.
%%
%%
%% The majority of ACM publications use numbered citations and
%% references. The command \citestyle{authoryear} switches to the
%% "author year" style.
%%
%% If you are preparing content for an event
%% sponsored by ACM SIGGRAPH, you must use the "author year" style of
%% citations and references.
%% Uncommenting
%% the next command will enable that style.
%%\citestyle{acmauthoryear}
\settopmatter{printacmref=true}
%%
%% end of the preamble, start of the body of the document source.
\begin{document}
%%
%% The "title" command has an optional parameter,
%% allowing the author to define a "short title" to be used in page headers.
\title{HCMRM: A High-Consistency Multimodal Relevance Model for Search Ads}
%%
%% The "author" command and its associated commands are used to define
%% the authors and their affiliations.
%% Of note is the shared affiliation of the first two authors, and the
%% "authornote" and "authornotemark" commands
%% used to denote shared contribution to the research.
\author{Guobing Gan}
\affiliation{%
\institution{Kuaishou Technology}
\city{Beijing}
\country{China}}
\email{ganguobing@kuaishou.com}
\author{Kaiming Gao}
\affiliation{%
\institution{Kuaishou Technology}
\city{Beijing}
\country{China}}
\email{gaokaiming@kuaishou.com}
\author{Li Wang}
\affiliation{%
\institution{Kuaishou Technology}
\city{Beijing}
\country{China}}
\email{wangli18@kuaishou.com}
\author{Shen Jiang}
\affiliation{%
\institution{Kuaishou Technology}
\city{Beijing}
\country{China}}
\email{jiangshen@kuaishou.com}
\author{Peng Jiang}
\affiliation{%
\institution{Kuaishou Technology}
\city{Beijing}
\country{China}}
\email{jiangpeng@kuaishou.com}
%%
%% By default, the full list of authors will be used in the page
%% headers. Often, this list is too long, and will overlap
%% other information printed in the page headers. This command allows
%% the author to define a more concise list
%% of authors' names for this purpose.
\renewcommand{\shortauthors}{Guobing Gan, Kaiming Gao, Li Wang, Shen Jiang, and Peng Jiang}
%%
%% The abstract is a short summary of the work to be presented in the
%% article.
\begin{abstract}
Search advertising is essential for merchants to reach the target users on short video platforms. Short video ads aligned with user search intents are displayed through relevance matching and bid ranking mechanisms. This paper focuses on improving query-to-video relevance matching to enhance the effectiveness of ranking in ad systems. Recent vision-language pre-training models have demonstrated promise in various multimodal tasks. However, their contribution to downstream query-video relevance tasks is limited, as the alignment between the \textbf{pair} of visual signals and text differs from the modeling of the \textbf{triplet} of the query, visual signals, and video text. In addition, our previous relevance model provides limited ranking capabilities, largely due to the discrepancy between the binary cross-entropy fine-tuning objective and the ranking objective.
To address these limitations, we design a \textbf{h}igh-\textbf{c}onsistency \textbf{m}ultimodal \textbf{r}elevance \textbf{m}odel (\textbf{HCMRM}). It utilizes a simple yet effective method to enhance the consistency between pre-training and relevance tasks. Specifically, during the pre-training phase, along with aligning visual signals and video text, several keywords are extracted from the video text as \textbf{pseudo-queries} to perform the triplet relevance modeling. For the fine-tuning phase, we introduce a \textbf{hierarchical softmax} loss, which enables the model to learn the order within labels while maximizing the distinction between positive and negative samples. This promotes the fusion ranking of relevance and bidding in the subsequent ranking stage.
The proposed method has been deployed in the Kuaishou search advertising system for over a year, contributing to a 6.1\% reduction in the proportion of irrelevant ads and a 1.4\% increase in ad revenue.
\end{abstract}
% \begin{abstract}
% Search advertising is essential for merchants to reach the target users on short video platforms. Short video ads aligned with user search intents are displayed through relevance matching and bid ranking mechanisms. This paper focuses on improving query-to-video relevance matching to enhance the effectiveness of ranking in ad systems. Recent vision-language pre-training models have demonstrated promise in various multimodal tasks. However, their contribution to downstream query-video relevance tasks is limited, as the alignment between the pair of visual signals and text differs from the modeling of the triplet of the query, visual signals, and video text. In addition, our previous relevance model provides limited ranking capabilities, largely due to the discrepancy between the binary cross-entropy fine-tuning objective and the ranking objective. To address these limitations, we design a high-consistency multimodal relevance model (HCMRM). It utilizes a simple yet effective method to enhance the consistency between pre-training and relevance tasks. Specifically, during the pre-training phase, along with aligning visual signals and video text, several keywords are extracted from the video text as pseudo-queries to perform the triplet relevance modeling. For the fine-tuning phase, we introduce a hierarchical softmax loss, which enables the model to learn the order within labels while maximizing the distinction between positive and negative samples. This promotes the fusion ranking of relevance and bidding in the subsequent ranking stage. The proposed method has been deployed in the Kuaishou search advertising system for over a year, contributing to a 6.1% reduction in the proportion of irrelevant ads and a 1.4% increase in ad revenue.
% \end{abstract}
%%
%% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
%% Please copy and paste the code instead of the example below.
%%
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10002951.10003317.10003338</concept_id>
<concept_desc>Information systems~Retrieval models and ranking</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10002951.10003227.10003447</concept_id>
<concept_desc>Information systems~Computational advertising</concept_desc>
<concept_significance>300</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[300]{Information systems~Retrieval models and ranking}
\ccsdesc[300]{Information systems~Computational advertising}
%%
%% Keywords. The author(s) should pick words that accurately describe
%% the work being presented. Separate the keywords with commas.
\keywords{Relevance Modeling, Multimodal Pre-training, Search Advertising, Consistency}
%% A "teaser" image appears between the author and affiliation
%% information and the body of the document, and typically spans the
%% page.
% \received{20 February 2007}
% \received[revised]{12 March 2009}
% \received[accepted]{5 June 2009}
%%
%% This command processes the author and affiliation and title
%% information and builds the first part of the formatted document.
\maketitle
\section{Introduction}
\begin{figure*}
\includegraphics[width=\textwidth]{fig/video_introduce_v3.pdf}
\caption{An example of the short video ad, including both video signals and text information. (A) is a diagram of the video image processing procedure. The uniformly sampled frames are blocked and spliced before being input into the model. The upper part of (B) shows that the video text consists of keywords extracted from multiple text fields in the video, arranged in descending order of importance. The lower part of (B) illustrates the process of generating pseudo-queries in the pre-training.}
\Description{An example of the short video ad, including both video signals and text information.}
\label{fig:video_introduce}
\end{figure*}
The consumption of short videos has become popular in our daily lives. Hundreds of millions of users visit short video platforms (e.g., Likee, Instagram Reels, TikTok, and Kuaishou) every day \cite{Xiao2023ExploringTF}. Search advertising is one of the primary ad types on short video platforms, generating substantial commercial prospects. It displays short video ads that align with user search intent and possess high commercial value through relevance matching and bidding ranking mechanisms. The relevance module is crucial in the ad system, as it assesses the degree of relevance between a short video ad and a given query to decide whether the ad should be filtered out. Additionally, relevance scores from this module and bid scores from the bidding module are combined to determine the ad ranking.
To calculate the relevance between a query and a short video ad, it is necessary to integrate the visual signals of the video with its textual information (incl., video title, OCR, ASR, etc). In recent years, vision-language pre-training has achieved remarkable success in various multimodal tasks \cite{Li2023MultimodalFM,Zhang2024MMLLMsRA}. However, most approaches focus on aligning visual signals with video or image descriptions. This pairwise alignment differs from the relevance modeling of the triplet consisting of the query, visual signals, and video text. Consequently, traditional pre-training methods have limited effectiveness for query-video relevance tasks. To enhance query awareness in pre-training, some works \cite{Ye2023QueryawareMB,Zhu2023QueryLIFEQL} in the fields of e-commerce search and long video search suggest leveraging post-click behavioral data for pre-training. However, click data is usually noisy due to exposure bias and clickbait \cite{Zou2021PretrainedLM}.
Moreover, most videos have few impressions or click throughs due to the long-tail distribution, resulting in a limited amount of query-video pairs with high click-through rates and confidence levels available for pre-training.
% Moreover, short video ads make up only a small portion of the videos on short video platforms, and user behavior data is relatively sparse, resulting in a lack of large-scale query-ad click data.
Furthermore, these query-aware methods \cite{Ye2023QueryawareMB,Zhu2023QueryLIFEQL} typically require the introduction of an additional query tower built on mainstream multimodal models like ALBEF \cite{Li2021AlignBF} and BLIP \cite{Li2022BLIPBL}, resulting in substantial modification costs and increases the overall complexity of the architecture.
In the manually annotated data for query-video ad pairs, relevance levels are categorized into four tiers: Bad, Less, Good, and Excellent. To maximize the distinction between positive and negative samples, our previous relevance model employed binary cross-entropy loss (i.e., Bad and Less refer to 0, Good and Excellent refer to 1) for fine-tuning. This approach helps the system better filter out irrelevant query-video ad pair but fails to account for the ordinal nature of the labels, which weakens the ranking ability of relevance models in the subsequent ad ranking stage.
To address these challenges, we propose the high-consistency multimodal relevance model (HCMRM) which aims to enhance the consistency of the pre-training and downstream query-video relevance tasks, as well as the coherence between the relevance module and the ad ranking module.
To improve the computational efficiency, we convert lengthy short video texts into a keyword sequence, with the keywords arranged in descending order of their importance weights, as shown in the upper part of Figure~\ref{fig:video_introduce}(B).
For the query-video relevance model, we leverage the architecture of ALBEF \cite{Li2021AlignBF} with minimal modification cost. This architecture includes a text encoder, a vision encoder, and a fusion encoder, as shown in Figure~\ref{fig:pretrained_model}.
During the pre-training phase, common tasks such as contrastive learning, image-text matching, and masked language modeling are employed to align video images (frames) with video texts (keywords). Additionally, several relatively important keywords on the left side of the video keyword sequence are extracted and used as a pseudo-query, while the remaining portion continues to serve as the video text. The pseudo-query and its corresponding video form a positive sample, while it is paired with other videos as negative samples. A hard negative sampling mechanism is also employed, similar to that used in image-text matching tasks \cite{Li2021AlignBF}. Therefore, using this pseudo-query-video matching mechanism, we can perform training consistent with the downstream relevance task in the pre-training phase.
% The pseudo-query and the remaining video text are separated by a special delimiter and jointly input into the text encoder.
For fine-tuning using human-labeled data, we introduce a symmetric hierarchical softmax loss based on the symmetry of the four-level relevance labels. This relevance loss function consists of three binary cross-entropy objectives: one for distinguishing between positive and negative samples, one for distinguishing between Good and Excellent within the positive samples, and one for distinguishing between Bad and Less within the negative samples. This approach enables the model to maximize the distinction between positive and negative samples while learning the ordinal relationships among the labels, thereby strengthening the ranking capability of the relevance model.
To validate the benefits of the proposed HCMRM, we have conducted extensive experiments both offline and online. The experimental results show that HCMRM achieves significant improvements in metrics such as AUC and Spearman's rank correlation coefficient compared to our previous relevance model and various baseline models. Furthermore, it achieves a 6.1\% reduction in the proportion of irrelevant ads and a $1.4\%$ increase in ad revenue for Kuaishou's search advertising. Our main contributions can be summarized as follows:
\begin{itemize}
\item This is one of the earliest works on query-short video multimodal relevance, particularly in the advertising domain.
\item We propose a multimodal pre-training framework that enhances the consistency between pre-training and downstream relevance tasks, building upon mainstream multimodal models without significant modification costs.
\item We introduce a symmetric hierarchical softmax loss that enhances the ranking capability of relevance models, optimizing the ranking process for advertising systems.
\end{itemize}
\section{Related Work}
\subsection{Vision-Language Pre-training}
Vision-language pre-training has recently achieved remarkable success in various multimodal tasks by learning multimodal representations on large-scale image-text or video-text pairs \cite{Radford2021LearningTV, Fang2022EVAET, Li2021AlignBF, Bain2021FrozenIT, Luo2021CLIP4ClipAE, Mu2021SLIPSM, Ma2022XCLIPEM, Huang2022CloverTA, Xu2021VideoCLIPCP, Chen2023InternVS, Xu2023mPLUG2AM, Liu2023VisualIT, li2024llava}.
Methods like CLIP \cite{Radford2021LearningTV}, ALIGN \cite{Jia2021ScalingUV}, and DeCLIP \cite{Li2021SupervisionEE} leverage contrastive learning for pre-training on large massive noisy web data, achieving promising performance on cross-modal retrieval tasks.
ALBEF \cite{Li2021AlignBF} and FLAVA \cite{Singh2021FLAVAAF} introduce multimodal encoders to perform complex cross-modal interactions, leading to superior performance in multimodal reasoning tasks. SimVLM \cite{Wang2021SimVLMSV}, CoCa \cite{Yu2022CoCaCC}, and BLIP \cite{Li2022BLIPBL} add multimodal decoders, enabling new capabilities in image-conditioned text generation. Recent multimodal large models like BLIP-2 \cite{Li2023BLIP2BL}, LLAVA \cite{li2024llava}, and Qwen-VL \cite{Bai2023QwenVLAV} harness large language models as a cognitive powerhouse, further advancing performance in various multimodal tasks \cite{Zhang2024MMLLMsRA, Yin2023ASO}.
\subsection{Multimodal Models for Relevance}
Early relevance methods focus on the text modality \cite{Yao2021LearningAP, Chang2021ExtremeML, Zou2021PretrainedLM, Liu2021Que2SearchFA}. In recent years, some multimodal relevance methods have emerged. For instance, Query-LIFE \cite{Zhu2023QueryLIFEQL} utilizes a query-based multimodal fusion to effectively incorporate the image and title of products, improving relevance and conversion efficiency in e-commerce search. QUALITY \cite{Ye2023QueryawareMB} introduces a query-aware multimodal model to enhance the ranking relevance of video search. However, these methods necessitate the addition of extra query towers, which complicates both the model structure and the pre-training process. In contrast, our HCMRM can directly reuse existing mainstream multimodal models. Additionally, their pre-training processes rely on constructing query-item pairs from the online click logs. However, click data is often noisy and reliable query-item pairs are typically limited. This limits the effectiveness of the pre-training.
\subsection{Relevance Fine-tuning Loss}
In industrial scenarios, the relevance of query-item pairs is manually divided into multiple levels \cite{Yao2021LearningAP, Zou2021PretrainedLM, Wen2023EnhancingDI, Ye2023QueryawareMB}. For instance, Baidu search ads categorize query-image relevance as \{0, 1, 2\}, with 0 as negative and 1 or 2 as positive, and binary cross-entropy loss is used for relevance fine-tuning training \cite{Wen2023EnhancingDI}. Tencent Video categorizes query-video relevance into four levels \cite{Ye2023QueryawareMB}. They utilize ordinal regression loss for training, which is a good approach to account for the ordinal nature of relevance labels.
\begin{figure*}
\includegraphics[width=0.95\textwidth]{fig/pretrained_model.pdf}
\caption{Overview of HCMRM. It is built based on ALBEF with minimal modifications and pre-trained using four objectives: Image-Text Contrastive Learning (ITC), Image-Text Matching (ITM), Masked Language Modeling (MLM), and Pseudo-Query-Video Matching (PQVM). Note that the downstream relevance task between query and short video ad is consistent with PQVM.}
\Description{Overview of the proposed multimodal relevance model, HCMRM.}
\label{fig:pretrained_model}
\end{figure*}
\section{Methodology}
\subsection{Preliminary}
We first introduce the role of the relevance module in the Kuaishou search advertising system and define the core relevance problem.
\textbf{Role of the Relevance Module.} The relevance module plays a \verb|filtering| and \verb|ranking| role in the search advertising system of short videos. Specifically, it receives a user's query and a set of candidate short video ads retrieved by front-end modules like recall. The relevance module scores the relevance between the query and each short video. On the one hand, different ad slots have specific relevance thresholds, and only ads with relevance scores exceeding these thresholds have the opportunity to be delivered to the corresponding ad slots. On the other hand, during the ad ranking stage, relevance scores are combined with bid scores from the bidding module to determine the final ranking of the ads.
\textbf{Definition of Relevance Problem.}
% We first formulate the search ad relevance problem and our solution as well as the notations used in this paper.
The relevance problem of query-video matching (QVM) can be conceptualized as a scoring function
$\mathcal{F}$. This function takes two inputs: a user-issued query ($Q$) and a short video ad ($V$) triggered by the system. The output is a relevance score ($r$) that quantifies the degree of relevance between the query and the ad.
Since a video contains visual signals (video frames/images) and textual information (i.e., video titles, descriptions, OCR, ASR, etc.), this relevance problem can be formulated as a function involving a \verb|triplet|. It is defined as follows:
\begin{equation}
\begin{aligned}
r = \mathcal{F}(Q, V) = \mathcal{F}(Q, I, T)
\end{aligned}
\end{equation}
where $I$ and $T$ are the visual signals and text information of the given short video ad, respectively.
\subsection{Feature Engineering}
\begin{algorithm}
\SetAlgoNoLine
\caption{Keyword Extraction for Video Text}
\KwIn{\\
\quad A list of $l$ text fields from a video ad: $f=[f_1, \cdots, f_l]$;\\
\quad The predefined weights for each field: $\alpha=[\alpha_1, \cdots, \alpha_l]$;\\
\quad The maximum number of generated keywords: $k$;\\}
\KwOut{\\
\quad The generated keyword sequence: $t$;\\}
Initialize an empty dictionary: $d=\{word:importance\}$; \\
\For{$i=1$ to $l$}
{
$W = Tokenize(f_i)$; \\
\For{each $w \in W$}
{
$\omega = Importance(w, f_i)$; \\
$\omega = \omega * \alpha_i$; \\
$d[w] = d.get(w, 0) + \omega$; \\
}
}
$top\_words =sorted$($d$, key=$d$.get, reverse=True)$[:k]$; \\
$t = "|".join(top\_words)$; \\
return $t$; \\
\label{keyword_extraction}
\end{algorithm}
We introduce the pre-processing of short video ads composed of visual and textual information as follows.
\textbf{Visual Signals.} For the visual signals of a video, $J$ frames are extracted from the video at equal time intervals. These frames are then divided into patches, which are concatenated in chronological order to form a patch sequence for input into the model, as illustrated in Figure~\ref{fig:video_introduce}(A).
\textbf{Textual Information.} The video text contains multiple fields such as title, description, OCR, and ASR, making it very lengthy and redundant. We employ keyword extraction to the video text to improve computational efficiency, as shown in Figure~\ref{fig:video_introduce}(B). The keyword extraction algorithm, detailed in Algorithm~\ref{keyword_extraction}, aims to extract the top $k$ most important words from multiple text fields of a video. Specifically, each text field is assigned a predefined weight $\alpha$. Firstly, initialize an empty dictionary where the keys are words and the values are their importance. Next, perform tokenization on each text field. Then, calculate the importance of each tokenized word. The importance assessment method is based on TF-IDF and named entity recognition. The importance of each word is weighted by the importance of the corresponding text field and summed accordingly. Finally, all words are sorted in descending order of importance, and the top $k$ words are connected to form a sequence of keywords.
\subsection{Model Architecture}
As illustrated in Figure~\ref{fig:pretrained_model}, HCMRM largely reuses the structure of ALBEF \cite{Li2021AlignBF}, consisting of a text encoder, an image encoder, and a multimodal fusion encoder. The image encoder is initialized using the first 6 layers of ViT-B/32 \cite{Dosovitskiy2020AnII} \footnote{ViT's position embeddings are extended to support long patch sequences of videos.}. The image patch sequence from video frames is used as input, resulting in an image feature sequence $[i_{cls}, i_1, \cdots, i_K]$, where $i_{cls}$ is the embedding of the token \verb|[CLS]|. The text encoder is initialized using the first 6 layers of BERT-base-chinese \cite{Devlin2019BERTPO}, and the multimodal fusion encoder is initialized using the last 6 layers of BERT-base-chinese. The text encoder transforms an input text $T$ into a sequence of
embeddings $[t_{cls}, t_1, \cdots, t_L]$. The multimodal fusion encoder takes the image and text features output by the previous two encoders as input and fuses them through the cross-attention mechanism at each of its sub-layers.
\subsection{Pre-training}
HCMRM is pre-trained using the following objectives: Image-Text Contrastive Learning (\textbf{ITC}), Image-Text Matching (\textbf{ITM}), Masked Language Modeling (\textbf{MLM}), and Pseudo-Query-Video Matching (\textbf{PQVM}), as shown in Figure~\ref{fig:pretrained_model}. Online contrastive hard negative mining based on ITC is employed to enhance the performance of ITM and PQVM.
\textbf{Image-Text Contrastive Learning.} ITC aims to align the visual signals and text information of videos before fusion. Based on widely used contrastive learning methods \cite{Radford2021LearningTV, Li2021AlignBF, Li2022BLIPBL, Yu2022CoCaCC}, ITC encourages positive image-text pairs to have similar representations in contrast to negative pairs. We follow the ITC loss of ALBEF \cite{Li2021AlignBF} and BLIP \cite{Li2022BLIPBL}, the normalized \verb|[CLS]| embeddings from the image and text encoders serve as unimodal representations, respectively. Meanwhile, a momentum model is introduced to produce unimodal embeddings and soft image-text similarity targets. For further details on the ITC task, please refer to ALBEF \cite{Li2021AlignBF}.
\textbf{Image-Text Matching.} ITM predicts whether a pair of video images and text is positive (matched) or negative (not matched). The \verb|[CLS]| embedding output from the multimodal fusion encoder serves as a joint representation of the pair. It is projected into a two-dimensional space through a fully-connected layer and trained using binary cross-entropy loss. Similar to ALBEF \cite{Li2021AlignBF}, we utilize the contrastive similarity distribution from ITC to identify in-batch hard negatives. For each image in the mini-batch, text that is more similar to the image within the same batch is sampled with a higher probability to create a negative sample with the current image. Likewise, a hard negative image is also sampled for each text.
\textbf{Masked Language Modeling.} MLM utilizes both video images and video text to predict the masked textual tokens. Similar to BERT \cite{Devlin2019BERTPO}, the special token \verb|[MASK]| is used to randomly replace a portion of the tokens in the video text. However, unlike BERT, we apply masking at the word level rather than at the subword or character level to improve the model's ability to learn entities.
\textbf{Pseudo-Query-Video Matching.} PQVM predicts whether a triplet of query, video images, and video text is positive (matched) or negative (not matched), which is consistent with downstream relevance task. The difference is that the pre-training utilizes synthetic query-video pairs, while the downstream fine-tuning utilizes real human-annotated query-video pairs.
The key to the PQVM pre-training task is the synthesis of query-text-image triplets. Based on the assumption that a query should represent the key content of a short video, we propose a simple method for synthesizing pseudo-queries using video texts. \textit{Since the video text has already been processed into a sequence of keywords ranked by importance, the keywords at the beginning of the sequence can effectively serve as queries.} To simulate real search scenarios, the number of keywords used as pseudo-queries is a random number within the range [a, b] during training. The video text is divided into two parts based on this number, with the remaining portion still serving as the video text, as follows:
\begin{equation}
\begin{aligned}
inx &= \text{randint}(a, \ b) \\
pseudo\ Q_i, \ \ rem. \ T_i &= \text{split}(T_i, \ inx)
\end{aligned}
\label{eq:pseudo_query}
\end{equation}
where $i \in [1, N]$ and $N$ is the batch size. $inx$ is the index for cutting the video text and also the length of the pseudo query.
The online contrastive hard negative
mining is also utilized in PQVM. Let $\mathcal{S}$ be an $N \times N$ similarity matrix, representing the similarity distribution between any two videos in a mini-batch. For HCMRM, $\mathcal{S}$ is the average of the softmax-normalized image-to-text and text-to-image
similarity matrices in the ITC task. The origin of these two matrices refers to ALBEF \cite{Li2021AlignBF}. For the $i$-th video $\{T_i, I_i\}$, its hard negative item is sampled based on the similarity distribution $\mathcal{S}_{i}$. Specifically, a video that is more similar to the $i$-th video within the same batch is sampled with higher probability.
\begin{equation}
\begin{aligned}
x &= \text{multinomial}(\mathcal{S}_{i}) \\
\text{neg}_i &= \{pseudo\ Q_x, \ rem. \ T_i, \ I_i \}, \ x \neq i \\
\text{pos}_i &= \{pseudo\ Q_i, \ rem. \ T_i, \ I_i \}
\end{aligned}
\label{eq:pseudo_sample}
\end{equation}
where multinomial represents a sampling operation from a discrete probability distribution vector, where the return value is the index of the vector. Additionally, $\text{neg}_i$ represents the synthesized $i$-th hard query-video relevance negative sample, while $\text{pos}_i$ is the synthesized $i$-th query-video relevance positive sample. Note that the video text used in PQVM pre-training excludes the pseudo-query portion, and using the complete video text would lead to an information leakage issue.
As illustrated in the right part of Figure~\ref{fig:pretrained_model}, the query and the video text are concatenated with the special tokens \verb|[CLS]| and \verb|[SEP]| and then fed into the text encoder. Meanwhile, the image patch sequence from the same video as the text is prefixed with \verb|[CLS]| and encoded by the vision encoder.
\begin{equation}
\begin{aligned}
z_T &= \text{Text Encoder}\left(\text{concat}\left(\verb|[CLS]|, Q, \verb|[SEP]|, T\right)\right) \\
z_I &= \text{Vision Encoder}\left(\text{concat}\left(\verb|[CLS]|, I\right)\right) \\
z &= \text{Fusion Encoder}\left(z_T, z_I\right)
\end{aligned}
\label{eq:sample_encoding}
\end{equation}
The \verb|[CLS]| embedding $z_{cls}$ output from the multimodal fusion encoder is utilized as a joint representation of the triplet of query, video images, and video text. Similar to ITM tasks, $z_{cls}$ is projected into a two-dimensional space via a linear layer and trained using binary cross-entropy loss. The PQVM loss can be expressed as:
\begin{equation}
\begin{aligned}
\mathcal{L}_{\mathrm{PQVM}} = -\mathbb{E}_{(Q, T, I) \sim D} [log\{P(y(Q, T, I)|(Q, T, I)\}]
\end{aligned}
\end{equation}
where $D$ is a distribution of in-batch samples, and $y(Q, T, I) \in \{0, 1\}$ represents whether the query $Q$ and video $(T, I)$ are matched. $P(y(Q, T, I)|(Q, T, I)$ is the output of the $z_{cls}$ followed by a two-class linear classifier.
Finally, HCMRM is pre-trained using the four tasks mentioned above, as shown in the loss function in Equation~\ref{loss:loss_pretrain}.
\begin{equation}
\begin{aligned}
\mathcal{L}_{\mathrm{pretrain}} = \mathcal{L}_{\mathrm{ITC}} + \mathcal{L}_{\mathrm{ITM}} + \mathcal{L}_{\mathrm{MLM}} + \mathcal{L}_{\mathrm{PQVM}}
\end{aligned}
\label{loss:loss_pretrain}
\end{equation}
\subsection{Fine-tuning}
\begin{figure}[h]
\includegraphics[width=0.9\linewidth]{fig/ordinal_regression_v2.pdf}
\caption{The hierarchical binary structure within the labels.}
\Description{The hierarchical binary structure within the relevance labels.}
\label{fig:ordinary_regression}
\end{figure}
For the downstream relevance task of query-video matching (QVM), HCMRM is fine-tuned on query-video ad data that is manually labeled into four relevance levels: \textbf{Bad (0)}, \textbf{Less (1)}, \textbf{Good (2)}, and \textbf{Excellent (3)}. It is evident that these four levels of relevance are ordinal labels and exhibit a balanced binary tree, as illustrated in Figure~\ref{fig:ordinary_regression}. Therefore, we intuitively model the query-video relevance task as a \textbf{two-layer symmetric hierarchical softmax} problem ~\cite{Tansey2018LeafSmoothedHS}. Specifically, three binary classifiers are used to predict the answers to the questions: \verb|'Is Positive?'|, \verb|'Is Less?'|, and \verb|'Is Excellent?'|, as shown in Equation~\ref{eq:hierarchical-softmax}.
\begin{equation}
\label{eq:hierarchical-softmax}
\begin{aligned}
& p_{pos} = Sigmoid(z_{cls} W^{pos}) \\
& p_{le} = Sigmoid(z_{cls} W^{le}) \\
& p_{ex} = Sigmoid(z_{cls} W^{ex})
\end{aligned}
\end{equation}
where $W^{pos}$, $W^{le}$, and $W^{ex}$ are different linear projections that map the output \verb|[CLS]| embedding of HCMRM ($z_{cls}$) to a scalar. The probabilities for each label can be formulated as follows:
\begin{equation}
\label{eq:predicted-label}
\begin{aligned}
p_0 &= (1-p_{pos})\times(1-p_{le}) \\
p_1 &= (1-p_{pos})\times p_{le} \\
p_2 &= p_{pos}\times(1-p_{ex}) \\
p_3 &= p_{pos}\times p_{ex} \\
\end{aligned}
\end{equation}
where $p_0$, $p_1$, $p_2$, and $p_3$ represent the probabilities of Bad, Less, Good, and Excellent, respectively.
To obtain continuous relevance scores for ad ranking, the final relevance score is defined as the expectation of all labels, as formulated in Equation~\ref{eq:predicted-relevance-score}.
\begin{equation}
\label{eq:predicted-relevance-score}
\begin{aligned}
r &= 0 \times p_0 + 1 \times p_1 + 2 \times p_2 + 3 \times p_3 \\
&= p_1 + 2 \times p_2 + 3 \times p_3
\end{aligned}
\end{equation}
The relevance score
$r$ ranges from [0, 3], with higher values indicating greater relevance.
The loss function for this hierarchical softmax problem is formalized as presented in Equation~\ref{eq:hierarchical-binary-loss}.
\begin{equation}
\label{eq:hierarchical-binary-loss}
\begin{aligned}
\mathcal{L}_{\mathrm{QVM}} = -\frac{1}{N} \sum^{N}log(p_l) - \frac{1}{N} \sum^{N}(r - l)^2
\end{aligned}
\end{equation}
where $l \in \{0, 1, 2, 3\}$ represents the label. $p_l \in \{p_0, p_1, p_2, p_3\}$. This loss combines hierarchical softmax estimation with label regression.
\section{Experiments}
\subsection{Large-scale Industrial Datasets}
There are two datasets utilized for the pre-training and fine-tuning of HCMRM and baseline models. (1) The pre-training dataset consists of 200M Chinese short video materials from the Kuaishou platform, and each is a pair of video text and video images. Since the pre-training of HCMRM does not rely on real queries, we can utilize as much material as possible, making our pre-training dataset much larger than related work \cite{Zhu2023QueryLIFEQL, Ye2023QueryawareMB}.
(2) The fine-tuning dataset contains 2.5M <query, video> labeled data, and 230K samples are selected as the evaluation set. The labels categorize relevance levels from 0 to 3, corresponding to Bad, Less, Good, and Excellent.
\subsection{Evaluation Metrics}
We employ the following metrics for offline and online evaluations.
\textbf{Offline Metrics.} To comprehensively evaluate the model's ability to distinguish between positive and negative samples, as well as its sensitivity to different levels of relevance, we used three offline evaluation metrics: Area Under the Curve (\textbf{AUC}), Spearman's rank correlation coefficient (\textbf{Spearman}), and Pearson Product-Moment Correlation Coefficient (\textbf{Pearson}). For the AUC metric, relevance labels 0 and 1 are considered negative, while labels 2 and 3 are considered positive. The Spearman and Pearson metrics consider the partial order of the labels and assess the consistency between the prediction results and the ground truth.
\textbf{Online Metrics.} We conducted online A/B tests to compare HCMRM with our previous baseline model. We considered three metrics: Irrelevant Ratio, \#Conversions, and Ad Revenue. (1) \textbf{Irrelevant Ratio} refers to the proportion of cases within a sampled dataset that are labeled as "Bad" or "Less" by human annotators, calculated as $(\#\mathrm{Bad} + \#\mathrm{Less}) / (\#\mathrm{Bad} + \#\mathrm{Less} + \#\mathrm{Good} + \#\mathrm{Excellent})$. (2) An advertising conversion occurs when a potential customer views a video ad and subsequently takes an action deemed valuable to the advertiser's business, such as making an online purchase or calling the business from a mobile phone. \textbf{\#Conversions} refers to the total number of advertising conversions. (3) \textbf{Ad Revenue} refers to the revenue of the advertising system after assisting advertisers in achieving conversions.
\subsection{Baselines}
\label{baselines}
We compared HCMRM with the following models, encompassing unimodal, multimodal, and large models.
\textbf{BERT.} This is a 12-layer BERT that is pre-trained on video text data using the same word-level masked language modeling task as HCMRM. Additionally, the processing of the video text follows the same approach as in HCMRM. The BERT model is initialized using BERT-base-chinese \cite{Devlin2019BERTPO}.
\textbf{BERT-VL.} This model is the previous version of the relevance module in our ad system. Its structure is identical to BERT \cite{Devlin2019BERTPO}, except that $J$ frames of video images are encoded into $J$ embeddings by MobileNetV2 \cite{Sandler2018MobileNetV2IR} and then concatenated with text embeddings before being input into the BERT model. For pre-training, we employ the word-level masked language modeling task, where a masked text token is predicted based on the combined representation of the visible tokens and visual features.
\textbf{ViLT.} We replicated ViLT \cite{Kim2021ViLTVT} in the short video advertising scenario. It is a vison-and-language Transformer without convolution or region supervision. The ViLT model is initialized
using ViT-B/32 \cite{Dosovitskiy2020AnII}. The image-text matching and word-level masked language modeling tasks are used for pre-training.
\textbf{ALBEF.} ALBEF can be regarded as a version of HCMRM without the PQVM mechanism, as HCMRM is built on ALBEF \cite{Li2021AlignBF}. The image-text contrastive learning, image-text matching, and masked language modeling tasks are used for pre-training. Unlike the vanilla ALBEF \cite{Li2021AlignBF}, word-level rather than subword-level masked language modeling is used.
\textbf{Query-LIFE.} We replicated Query-LIFE \cite{Zhu2023QueryLIFEQL} in the short video advertising scenario. It is a multimodal model based on a triple-stream architecture, where the query, video text, and video image are independently encoded before fusion. Additionally, it requires two stages of pre-training. The first stage conducts contrastive learning pre-training on video text-image pairs. The second stage utilizes online click data of <query, video> for pre-training. This click data comprises 60M query-video pairs, derived from advertisement delivery data with a click-through rate exceeding 0.2 and at least 5 impressions over the past year. Note that the Query-LIFE we replicated does not include the GenFilt component, as GenFilt is a general trick that can also be effective for other methods.
\textbf{Qwen-VL.} We fine-tuned Qwen-VL \cite{Bai2023QwenVLAV} using the same training objectives as HCMRM. Qwen-VL is a multimodal large language model that supports Chinese. Since Qwen-VL is a decoder-only architecture, the output embedding of the last token of the input sequence is used as the global multimodal representation for relevance estimation. In contrast, the above methods use the first token of the input sequence, i.e., \verb|[CLS]|.
\textbf{MiniCPM-V-2.6.} We fine-tuned MiniCPM-V-2.6 \cite{Yao2024MiniCPMVAG} using the same approach as Qwen-VL. MiniCPM-V-2.6 is also a popular multimodal large language model that supports Chinese.
\subsection{Experimental Setup}
The number of images extracted from a short video was set to 3.
The maximum sequence lengths for the video text and query were set to 64 and 16, respectively. For pre-training, we trained HCMRM and baselines (excluding Qwen-VL and MiniCPM-V-2.6) for 3 epochs with a batch size of 256 and a learning rate of 5e-5 on 8 NVIDIA V100 GPUs. For fine-tuning, we trained HCMRM and the baselines (again excluding Qwen-VL and MiniCPM-V-2.6) on 8 NVIDIA V100 GPUs, while Qwen-VL and MiniCPM-V-2.6 were trained on 8 NVIDIA A800 GPUs. All models were trained for 4 epochs with a batch size of 64. The learning rate for Qwen-VL and MiniCPM-V-2.6 was set to 5e-7, while the learning rate for the other models was 5e-6. For learning rate scheduling, we employed a cosine decay strategy and a linear warmup of 2000 steps. We utilized the AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and a weight
decay rate of 0.02.
\begin{table*}[t]
\centering
\caption{Offline experimental results compared with different baselines on the short video relevance matching task.}
\resizebox{0.8\textwidth}{!}{
\begin{tabular}{llccccc}
\toprule
\textbf{Modality} & \textbf{Model} & \textbf{X-stream} & \textbf{Query-Aware} & \textbf{AUC} & \textbf{Spearman} & \textbf{Pearson} \\
\midrule
\multirow{2}{*}{unimodal}
& BERT & single & \ding{55} & 0.864 & 0.733 & 0.743 \\
& BERT w/ PQVM & single & \ding{51} & 0.868 & 0.742 & 0.752 \\
\midrule
\multirow{6}{*}{multimodal}
& BERT-VL & single & \ding{55} & 0.869 & 0.741 & 0.750 \\
& BERT-VL w/ PQVM & single & \ding{51} & 0.872 & 0.748 & 0.758 \\
\cmidrule(r){2-7}
& ViLT & single & \ding{55} & 0.871 & 0.746 & 0.755 \\
& ViLT w/ PQVM & single & \ding{51} & 0.874 & 0.753 & 0.761 \\
\cmidrule(r){2-7}
& Query-LIFE & triple & \ding{51} & 0.868 & 0.742 & 0.751 \\
& Query-LIFE w/ MLM & triple & \ding{51} & 0.872 & 0.747 & 0.756 \\
\cmidrule(r){2-7}
& ALBEF & dual & \ding{55} & 0.875 & 0.755 & 0.765 \\
& ALBEF w/ PQVM (HCMRM) & dual & \ding{51} & \textbf{0.878} & \textbf{0.760} & \textbf{0.770} \\
\bottomrule
\multicolumn{7}{l}{\footnotesize "X-stream" refers to the architecture. E.g., ALBEF is dual-stream as it uses two encoders (text and image) to process the inputs before fusion.} \\
\multicolumn{7}{l}{\footnotesize "Query-Aware" refers to whether a model implements query awareness during pre-training.}\\
\end{tabular}}
\label{mian_exp}
\end{table*}
\subsection{Offline Experimental Results}
\subsubsection{Relevance Model.} We compared HCMRM with baselines on the short video relevance matching task. These baselines involve text-only unimodal models (BERT), single-stream multimodal models (BERT-VL, ViLT), dual-stream multimodal models (ALBEF), and triple-stream multimodal models (Query-LIFE). We also implemented the pseudo-query-video matching mechanism (PQVM) on baseline methods to verify the effectiveness of the proposed method. All models were pre-trained on domain data of short video ads, and they utilized the same relevance loss of hierarchical softmax for fine-tuning. The experimental results are shown in Table~\ref{mian_exp}.
\textbf{HCMRM achieves the best performance in short video relevance tasks.}
Compared to baselines BERT, BERT-VL, ViLT, Query-LIFE, and ALBEF, the proposed HCMRM achieves relative improvements in the AUC score by 1.6\%, 1.0\%, 0.8\%, 1.2\%, and 0.3\%, respectively. In terms of Spearman's rank correlation coefficient, it achieves relative improvements of 3.7\%, 2.6\%, 1.9\%, 2.4\%, and 0.7\%, respectively. The changes in the Pearson Product-Moment Correlation Coefficient metric are largely consistent with those in the Spearman metric.
In addition, multimodal methods significantly outperform text-only methods. For instance, the most lightweight multimodal method, BERT-VL, surpasses BERT by 0.6\%, 1.1\%, and 0.9\% in the AUC, Spearman, and Pearson metrics, highlighting the importance of multimodal modeling in short video scenario.
Among multimodal methods, the dual-stream method ALBEF outperforms the single-stream methods BERT-VL and ViLT, achieving increases of 0.7\% and 0.5\% in AUC scores, respectively, and enhancements of 1.9\% and 1.2\% in Spearman scores. This may be attributed to the lack of alignment between the visual and textual modalities in the single-stream approaches compared to dual-stream ones. Additionally, the performance of Query-LIFE (triple-stream) is only comparable to that of BERT-VL in short video relevance tasks. By adding the masked language modeling pre-training task, the performance of Query-LIFE has been significantly enhanced. However, there remains a noticeable performance gap compared to HCMRM.
\textbf{The pseudo-query-video matching mechanism can enhance the performance of various baselines.} HCMRM utilizes the PQVM mechanism during pre-training, achieving 0.3\% and 0.7\% improvement in AUC and Spearman metrics, compared to ALBEF. Additionally, the BERT, BERT-VL, and ViLT models that incorporate the PQVM mechanism achieve relative improvements of 0.5\%, 0.3\%, and 0.3\% in the AUC score, respectively, compared to their standard versions. In terms of Spearman's rank correlation coefficient, the relative improvements are 1.2\%, 0.9\%, and 0.9\%, respectively. These results validate the effectiveness of the pseudo-query-video matching mechanism in enhancing the query-video relevance task.
\begin{table}[h]
\centering
\caption{Offline experimental results with different relevance losses.}
\resizebox{0.85\linewidth}{!}{
\begin{tabular}{lccc}
\toprule
\textbf{Relevance loss} & \textbf{AUC} & \textbf{Spearman} & \textbf{Pearson} \\
\midrule
binary cross-entropy & \textbf{0.884} & 0.668 & 0.685 \\
mean squared error & 0.876 & 0.756 & 0.766 \\
ordinal regression & 0.876 & 0.758 & 0.768 \\
hierarchical softmax & 0.878 & \textbf{0.760} & \textbf{0.770} \\
\bottomrule
\end{tabular}}
\label{mian_exp_loss}
\end{table}
\subsubsection{Relevance Loss.} We compared HCMRM combined with different relevance fine-tuning losses. Binary cross-entropy loss, where Bad and Less are considered as 0 and Good and Excellent as 1, is employed in our previous relevance models. Mean squared error loss aims to perform regression prediction of the labels 0, 1, 2, and 3. Ordinal regression \cite{Ye2023QueryawareMB} is a competitive method that accounts for the ordinal information within relevance labels.
As shown in Table~\ref{mian_exp_loss}, the proposed relevance loss of hierarchical softmax achieves the highest scores in both Spearman and Pearson metrics.
% Although the binary cross-entropy loss achieves the highest AUC score, it performs significantly worse than other loss functions in the Spearman and Pearson metrics.
Although hierarchical softmax loss is not as good as binary cross-entropy loss in terms of the AUC metric ($-0.7\%$), it far surpasses it in Spearman ($+13.8\%$) and Pearson metrics ($+12.4\%$).
This indicates that binary cross-entropy loss has a strong discriminative ability but weak ranking capability. Compared to mean squared error and ordinal regression loss, hierarchical softmax outperforms across all three metrics: AUC, Spearman, and Pearson, ultimately leading to a stronger overall capability.
\subsection{Online Experimental Results}
\subsubsection{Relevance Model.}
We conducted online A/B tests to compare HCMRM with our previous baseline, BERT-VL. Both models were fine-tuned using the binary cross-entropy loss. Details of the model's online deployment can be found in Appendix~\ref{serving}.
For the metric of Irrelevant Ratio, we invited annotators to evaluate the relevance performance of both models. Specifically, 10000 query-short video ad pairs were randomly sampled from the experimental buckets of both HCMRM and BERT-VL for relevance annotation. We report the absolute difference in the Irrelevant Ratio between the two test sets. For \#Conversion and Ad Revenue, we report the change of HCMRM relative to BERT-VL. As shown in Table~\ref{abtest_evaluation}, HCMRM achieves a 4\% absolute reduction in the Irrelevant Rate. As ads are more aligned with search intents, HCMRM outperforms BERT-VL by improving the number of advertising conversions (\#Conversions +1.5\%), which further led to a 1\% increase in ad revenue.
\begin{table}[h]
\centering
\caption{Online experimental results of HCMRM compared to BERT-VL.}
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{lccc}
\toprule
& \textbf{Irrelevant Ratio} & \textbf{\#Conversion} & \textbf{Ad Revenue} \\
\midrule
$\Delta$ & -4.0\% & +1.5\% & +1.0\% \\
\bottomrule
\end{tabular}}
\label{abtest_evaluation}
\end{table}
\subsubsection{Relevance Loss.}
We conducted online A/B tests to compare the proposed relevance loss of hierarchical softmax with the previous relevance loss of binary cross-entropy. Both sets of experiments utilized the HCMRM model. As shown in Table~\ref{abtest_evaluation_1}, the proposed relevance loss achieves a $2.1\%$ reduction in Irrelevant Rate, a $0.9\%$ increase in \#Conversion, and a $0.4\%$ improvement in ad revenue.
\begin{table}[h]
\centering
\caption{Online experimental results of hierarchical softmax relevance loss compared to binary cross-entropy loss.}
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{lccc}
\toprule
& \textbf{Irrelevant Ratio} & \textbf{\#Conversion} & \textbf{Ad Revenue} \\
\midrule
$\Delta$ & -2.1\% & +0.9\% & +0.4\% \\
\bottomrule
\end{tabular}}
\label{abtest_evaluation_1}
\end{table}
\subsection{Advanced Topics}
\subsubsection{How do pseudo queries surpass real queries ?}
Related literature \cite{Zhu2023QueryLIFEQL, Ye2023QueryawareMB} primarily uses click data for pre-training to achieve query awareness, thereby improving consistency between pre-training and downstream relevance tasks. We collected a dataset of 60M query-video click instances (with a click-through rate > 0.2 and at least 5 impressions) for the query-aware pre-training of ALBEF. The dataset is also utilized for the Query-LIFE baseline, as mentioned above.
\begin{table}[h]
\centering
\caption{Experimental results compared with ALBEFs pre-trained on the click data.}
\resizebox{1.0\linewidth}{!}{
\begin{tabular}{llcccc}
\toprule
\textbf{Id} & \textbf{Method} & \textbf{Pretrain Data} & \textbf{AUC} & \textbf{Spearman} & \textbf{Pearson} \\
\midrule
1 & ALBEF & material & 0.875 & 0.755 & 0.765 \\
2 & ALBEF w/ QVM & click & 0.869 & 0.743 & 0.753 \\
3 & 1 then 2 & material + click & 0.876 & 0.757 & 0.766 \\
\midrule
4 & HCMRM & material & \textbf{0.878} & \textbf{0.760} & \textbf{0.770}\\
\bottomrule
\end{tabular}}
\label{mian_exp_query}
\end{table}
As shown in Table~\ref{mian_exp_query}, pre-training ALBEF on 60M click data with an additional query-video matching task (2nd row) performs noticeably worse than pre-training it on 200M video materials (1st row). This difference can be attributed to two factors. First, click data is inherently noisy, and a high click-through rate does not necessarily indicate high relevance due to exposure bias and clickbait \cite{Zou2021PretrainedLM}. Second, given the long-tail distribution, most videos receive few impressions or clicks, leading to a limited amount of query-video click data available for effective query-aware pre-training.
Furthermore, we implemented a two-stage pre-training for ALBEF: initially training on 200M video materials, followed by training on 60M click data, incorporating an additional query-photo matching task. The offline metrics of this two-stage pre-trained ALBEF are slightly better than those of the ALBEF pre-trained on 200M videos (3rd row vs. 1st row). However, it still underperforms compared to HCMRM (3rd row vs. 4th row). This indicates that although the pseudo-query mechanism of HCMRM is simple, it is effective.
\subsubsection{How does HCMRM compare to multimodal large language models?}
Multimodal large language models (MLLMs) have recently advanced significantly. We conducted a comparative evaluation of the proposed HCMRM against MLLMs on short video relevance tasks. We fine-tuned two MLLMs that support Chinese, Qwen-VL \cite{Bai2023QwenVLAV} and MiniCPM-V-2.6 \cite{Yao2024MiniCPMVAG}.
\begin{table}[h]
\centering
\caption{Experimental results compared with multimodal large language models.}
\resizebox{0.85\linewidth}{!}{
\begin{tabular}{lccccc}
\toprule
\textbf{Model} & \textbf{MLLM} & \textbf{AUC} & \textbf{Spearman} & \textbf{Pearson} \\
\midrule
Qwen-VL & \ding{51} & 0.872 & 0.754 & 0.765 \\
MiniCPM-V-2.6 & \ding{51} & 0.871 & 0.754 & 0.766 \\
HCMRM & \ding{55} & \textbf{0.878} & \textbf{0.760} & \textbf{0.770} \\
\bottomrule
\end{tabular}}
\label{mian_exp_llm}
\end{table}
As shown in Table~\ref{mian_exp_llm}, MLLMs do not outperform HCMRM on short video relevance tasks. In contrast, HCMRM outperforms Qwen-VL and MiniCPM-V-2.6 by 0.7\% / 0.8\% in AUC scores and 0.8\% / 0.8\% in Spearman scores. This is mainly because HCMRM has been pre-trained on hundreds of millions of short videos, whereas MLLMs lack such domain-specific knowledge. However, MLLMs still outperform multimodal methods like BERT-VL and ViLT which have been pre-trained on domain data, as shown in Table~\ref{mian_exp}. This highlights the immense potential of MLLMs for domain adaptation training when computational resources are not a constraint.
\section{Conclusion}
Mainstream vision-language pre-training methods, such as ALBEF \cite{Li2021AlignBF} and BLIP \cite{Li2022BLIPBL}, have a limited contribution to the relevance task of search ads due to the lack of query awareness. Existing query-aware multimodal methods \cite{Zhu2023QueryLIFEQL, Ye2023QueryawareMB} usually rely on noisy and sparse click data, and the introduction of an additional query tower increases the model's complexity. To address these limitations, we propose HCMRM by building upon ALBEF without significant modification costs. HCMRM enhances the consistency between multimodal pre-training and downstream relevance tasks using a simple pseudo-query-video matching mechanism. Furthermore, the introduction of hierarchical softmax loss enhances the ranking capability of relevance models, optimizing the ranking of ad systems. Ultimately, the proposed method achieves significant improvements to the Kuaishou search advertising system, with a 6.1\% reduction in the rate of irrelevant ads and a 1.4\% increase in ad revenue.
%%
%% The acknowledgments section is defined using the "acks" environment
%% (and NOT an unnumbered section). This ensures the proper
%% identification of the section in the article metadata, and the
%% consistent spelling of the heading.
% \begin{acks}
% To Robert, for the bagels and explaining CMYK and color spaces.
% \end{acks}
%%
%% The next two lines define the bibliography style to be used, and
%% the bibliography file.
\bibliographystyle{ACM-Reference-Format}
\balance
\bibliography{sigconf}
%%
%% If your work has an appendix, this is the place to put it.
\appendix
\section{Online Serving}
\label{serving}
To meet the real-time requirements of the production environment, the relevance model we deploy online is a lightweight multimodal network. This model is based on the BERT-VL architecture introduced in Section~\ref{baselines}. In contrast, it has fewer Transformer layers and a smaller hidden layer dimension. Additionally, it features a larger vocabulary to reduce the length of the input sequences. The visual embeddings generated by MobileNetV2 are precomputed and cached, rather than being calculated in real-time, before they are fed into BERT-VL.
Based on knowledge distillation, an offline pre-trained and fine-tuned relevance model serves as the teacher, while an online lightweight model acts as the student, facilitating the transfer of relevance matching capabilities. Specifically, this is achieved by enabling the student model to learn the relevance scores generated by the teacher model across hundreds of millions of randomly sampled query-video ad pairs.
\end{document}
\endinput
%%
%% End of file `sample-sigconf.tex'.
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{38}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{} and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showURL}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showURL #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Bai et~al\mbox{.}(2023)]%
{Bai2023QwenVLAV}
\bibfield{author}{\bibinfo{person}{Jinze Bai}, \bibinfo{person}{Shuai Bai}, \bibinfo{person}{Shusheng Yang}, \bibinfo{person}{Shijie Wang}, \bibinfo{person}{Sinan Tan}, \bibinfo{person}{Peng Wang}, \bibinfo{person}{Junyang Lin}, \bibinfo{person}{Chang Zhou}, {and} \bibinfo{person}{Jingren Zhou}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond}.
\newblock \bibinfo{journal}{\emph{ArXiv}} (\bibinfo{year}{2023}).
\newblock
\urldef\tempurl%
\url{https://arxiv.org/abs/2308.12966}
\showURL{%
\tempurl}
\bibitem[Bain et~al\mbox{.}(2021)]%
{Bain2021FrozenIT}
\bibfield{author}{\bibinfo{person}{Max Bain}, \bibinfo{person}{Arsha Nagrani}, \bibinfo{person}{G{\"u}l Varol}, {and} \bibinfo{person}{Andrew Zisserman}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval}.
\newblock \bibinfo{journal}{\emph{2021 IEEE/CVF International Conference on Computer Vision (ICCV)}} (\bibinfo{year}{2021}), \bibinfo{pages}{1708--1718}.
\newblock
\href{https://doi.org/10.1109/ICCV48922.2021.00175}{doi:\nolinkurl{10.1109/ICCV48922.2021.00175}}
\bibitem[Chang et~al\mbox{.}(2021)]%
{Chang2021ExtremeML}
\bibfield{author}{\bibinfo{person}{Wei-Cheng Chang}, \bibinfo{person}{Daniel Jiang}, \bibinfo{person}{Hsiang-Fu Yu}, \bibinfo{person}{Choon~Hui Teo}, \bibinfo{person}{Jiong Zhang}, \bibinfo{person}{Kai Zhong}, \bibinfo{person}{Kedarnath Kolluri}, \bibinfo{person}{Qie Hu}, \bibinfo{person}{Nikhil Shandilya}, \bibinfo{person}{Vyacheslav Ievgrafov}, \bibinfo{person}{Japinder Singh}, {and} \bibinfo{person}{Inderjit~S. Dhillon}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Extreme Multi-label Learning for Semantic Matching in Product Search}. In \bibinfo{booktitle}{\emph{Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \& Data Mining}} \emph{(\bibinfo{series}{KDD '21})}. \bibinfo{publisher}{Association for Computing Machinery}, \bibinfo{pages}{2643–2651}.
\newblock
\showISBNx{9781450383325}
\href{https://doi.org/10.1145/3447548.3467092}{doi:\nolinkurl{10.1145/3447548.3467092}}
\bibitem[Chen et~al\mbox{.}(2023)]%
{Chen2023InternVS}
\bibfield{author}{\bibinfo{person}{Zhe Chen}, \bibinfo{person}{Jiannan Wu}, \bibinfo{person}{Wenhai Wang}, \bibinfo{person}{Weijie Su}, \bibinfo{person}{Guo Chen}, \bibinfo{person}{Sen Xing}, \bibinfo{person}{Zhong Muyan}, \bibinfo{person}{Qinglong Zhang}, \bibinfo{person}{Xizhou Zhu}, \bibinfo{person}{Lewei Lu}, \bibinfo{person}{Bin Li}, \bibinfo{person}{Ping Luo}, \bibinfo{person}{Tong Lu}, \bibinfo{person}{Yu Qiao}, {and} \bibinfo{person}{Jifeng Dai}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks}. In \bibinfo{booktitle}{\emph{2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}}. \bibinfo{pages}{24185--24198}.
\newblock
\urldef\tempurl%
\url{https://cvpr.thecvf.com/virtual/2024/poster/30014}
\showURL{%
\tempurl}
\bibitem[Devlin et~al\mbox{.}(2019)]%
{Devlin2019BERTPO}
\bibfield{author}{\bibinfo{person}{Jacob Devlin}, \bibinfo{person}{Ming-Wei Chang}, \bibinfo{person}{Kenton Lee}, {and} \bibinfo{person}{Kristina Toutanova}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding}. In \bibinfo{booktitle}{\emph{Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}}. \bibinfo{publisher}{Association for Computational Linguistics}, \bibinfo{address}{Minneapolis, Minnesota}, \bibinfo{pages}{4171--4186}.
\newblock
\href{https://doi.org/10.18653/v1/N19-1423}{doi:\nolinkurl{10.18653/v1/N19-1423}}
\bibitem[Dosovitskiy et~al\mbox{.}(2021)]%
{Dosovitskiy2020AnII}
\bibfield{author}{\bibinfo{person}{Alexey Dosovitskiy}, \bibinfo{person}{Lucas Beyer}, \bibinfo{person}{Alexander Kolesnikov}, \bibinfo{person}{Dirk Weissenborn}, \bibinfo{person}{Xiaohua Zhai}, \bibinfo{person}{Thomas Unterthiner}, \bibinfo{person}{Mostafa Dehghani}, \bibinfo{person}{Matthias Minderer}, \bibinfo{person}{Georg Heigold}, \bibinfo{person}{Sylvain Gelly}, \bibinfo{person}{Jakob Uszkoreit}, {and} \bibinfo{person}{Neil Houlsby}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale}. In \bibinfo{booktitle}{\emph{International Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=YicbFdNTTy}
\showURL{%
\tempurl}
\bibitem[Fang et~al\mbox{.}(2023)]%
{Fang2022EVAET}
\bibfield{author}{\bibinfo{person}{Yuxin Fang}, \bibinfo{person}{Wen Wang}, \bibinfo{person}{Binhui Xie}, \bibinfo{person}{Quan-Sen Sun}, \bibinfo{person}{Ledell~Yu Wu}, \bibinfo{person}{Xinggang Wang}, \bibinfo{person}{Tiejun Huang}, \bibinfo{person}{Xinlong Wang}, {and} \bibinfo{person}{Yue Cao}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{EVA: Exploring the Limits of Masked Visual Representation Learning at Scale}. In \bibinfo{booktitle}{\emph{2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}}. \bibinfo{publisher}{IEEE Computer Society}, \bibinfo{address}{Los Alamitos, CA, USA}, \bibinfo{pages}{19358--19369}.
\newblock
\href{https://doi.org/10.1109/CVPR52729.2023.01855}{doi:\nolinkurl{10.1109/CVPR52729.2023.01855}}
\bibitem[Huang et~al\mbox{.}(2022)]%
{Huang2022CloverTA}
\bibfield{author}{\bibinfo{person}{Jingjia Huang}, \bibinfo{person}{Yinan Li}, \bibinfo{person}{Jiashi Feng}, \bibinfo{person}{Xiaoshuai Sun}, {and} \bibinfo{person}{Rongrong Ji}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Clover: Towards A Unified Video-Language Alignment and Fusion Model}.
\newblock \bibinfo{journal}{\emph{2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}} (\bibinfo{year}{2022}), \bibinfo{pages}{14856--14866}.
\newblock
\urldef\tempurl%
\url{https://cvpr.thecvf.com/virtual/2023/poster/22766}
\showURL{%
\tempurl}
\bibitem[Jia et~al\mbox{.}(2021)]%
{Jia2021ScalingUV}
\bibfield{author}{\bibinfo{person}{Chao Jia}, \bibinfo{person}{Yinfei Yang}, \bibinfo{person}{Ye Xia}, \bibinfo{person}{Yi-Ting Chen}, \bibinfo{person}{Zarana Parekh}, \bibinfo{person}{Hieu Pham}, \bibinfo{person}{Quoc Le}, \bibinfo{person}{Yun-Hsuan Sung}, \bibinfo{person}{Zhen Li}, {and} \bibinfo{person}{Tom Duerig}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision}. In \bibinfo{booktitle}{\emph{Proceedings of the 38th International Conference on Machine Learning}} \emph{(\bibinfo{series}{Proceedings of Machine Learning Research}, Vol.~\bibinfo{volume}{139})}. \bibinfo{publisher}{PMLR}, \bibinfo{pages}{4904--4916}.
\newblock
\urldef\tempurl%
\url{https://proceedings.mlr.press/v139/jia21b.html}
\showURL{%
\tempurl}
\bibitem[Kim et~al\mbox{.}(2021)]%
{Kim2021ViLTVT}
\bibfield{author}{\bibinfo{person}{Wonjae Kim}, \bibinfo{person}{Bokyung Son}, {and} \bibinfo{person}{Ildoo Kim}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}. In \bibinfo{booktitle}{\emph{Proceedings of the 38th International Conference on Machine Learning}} \emph{(\bibinfo{series}{Proceedings of Machine Learning Research}, Vol.~\bibinfo{volume}{139})}. \bibinfo{publisher}{PMLR}, \bibinfo{pages}{5583--5594}.
\newblock
\urldef\tempurl%
\url{https://proceedings.mlr.press/v139/kim21k.html}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2024)]%
{Li2023MultimodalFM}
\bibfield{author}{\bibinfo{person}{Chunyuan Li}, \bibinfo{person}{Zhe Gan}, \bibinfo{person}{Zhengyuan Yang}, \bibinfo{person}{Jianwei Yang}, \bibinfo{person}{Linjie Li}, \bibinfo{person}{Lijuan Wang}, {and} \bibinfo{person}{Jianfeng Gao}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Multimodal Foundation Models: From Specialists to General-Purpose Assistants}.
\newblock \bibinfo{journal}{\emph{Found. Trends Comput. Graph. Vis.}} \bibinfo{volume}{16}, \bibinfo{number}{1–2} (\bibinfo{year}{2024}).
\newblock
\showISSN{1572-2740}
\urldef\tempurl%
\url{https://doi.org/10.1561/0600000110}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2025)]%
{li2024llava}
\bibfield{author}{\bibinfo{person}{Feng Li}, \bibinfo{person}{Renrui Zhang}, \bibinfo{person}{Hao Zhang}, \bibinfo{person}{Yuanhan Zhang}, \bibinfo{person}{Bo Li}, \bibinfo{person}{Wei Li}, \bibinfo{person}{Zejun Ma}, {and} \bibinfo{person}{Chunyuan Li}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models}. In \bibinfo{booktitle}{\emph{The Thirteenth International Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=oSQiao9GqB}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2023)]%
{Li2023BLIP2BL}
\bibfield{author}{\bibinfo{person}{Junnan Li}, \bibinfo{person}{Dongxu Li}, \bibinfo{person}{Silvio Savarese}, {and} \bibinfo{person}{Steven C.~H. Hoi}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models}. In \bibinfo{booktitle}{\emph{International Conference on Machine Learning}} \emph{(\bibinfo{series}{Proceedings of Machine Learning Research}, Vol.~\bibinfo{volume}{202})}. \bibinfo{publisher}{PMLR}, \bibinfo{pages}{19730--19742}.
\newblock
\urldef\tempurl%
\url{https://proceedings.mlr.press/v202/li23q.html}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2022a)]%
{Li2022BLIPBL}
\bibfield{author}{\bibinfo{person}{Junnan Li}, \bibinfo{person}{Dongxu Li}, \bibinfo{person}{Caiming Xiong}, {and} \bibinfo{person}{Steven Hoi}.} \bibinfo{year}{2022}\natexlab{a}.
\newblock \showarticletitle{{BLIP}: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}. In \bibinfo{booktitle}{\emph{Proceedings of the 39th International Conference on Machine Learning}} \emph{(\bibinfo{series}{Proceedings of Machine Learning Research}, Vol.~\bibinfo{volume}{162})}, \bibfield{editor}{\bibinfo{person}{Kamalika Chaudhuri}, \bibinfo{person}{Stefanie Jegelka}, \bibinfo{person}{Le~Song}, \bibinfo{person}{Csaba Szepesvari}, \bibinfo{person}{Gang Niu}, {and} \bibinfo{person}{Sivan Sabato}} (Eds.). \bibinfo{publisher}{PMLR}, \bibinfo{pages}{12888--12900}.
\newblock
\urldef\tempurl%
\url{https://proceedings.mlr.press/v162/li22n.html}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2021)]%
{Li2021AlignBF}
\bibfield{author}{\bibinfo{person}{Junnan Li}, \bibinfo{person}{Ramprasaath~R. Selvaraju}, \bibinfo{person}{Akhilesh~Deepak Gotmare}, \bibinfo{person}{Shafiq~R. Joty}, \bibinfo{person}{Caiming Xiong}, {and} \bibinfo{person}{Steven C.~H. Hoi}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Align before Fuse: Vision and Language Representation Learning with Momentum Distillation}. In \bibinfo{booktitle}{\emph{Advances in Neural Information Processing Systems}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=OJLaKwiXSbx}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2022b)]%
{Li2021SupervisionEE}
\bibfield{author}{\bibinfo{person}{Yangguang Li}, \bibinfo{person}{Feng Liang}, \bibinfo{person}{Lichen Zhao}, \bibinfo{person}{Yufeng Cui}, \bibinfo{person}{Wanli Ouyang}, \bibinfo{person}{Jing Shao}, \bibinfo{person}{Fengwei Yu}, {and} \bibinfo{person}{Junjie Yan}.} \bibinfo{year}{2022}\natexlab{b}.
\newblock \showarticletitle{Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm}. In \bibinfo{booktitle}{\emph{International Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=zq1iJkNk3uN}
\showURL{%
\tempurl}
\bibitem[Liu et~al\mbox{.}(2023)]%
{Liu2023VisualIT}
\bibfield{author}{\bibinfo{person}{Haotian Liu}, \bibinfo{person}{Chunyuan Li}, \bibinfo{person}{Qingyang Wu}, {and} \bibinfo{person}{Yong~Jae Lee}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Visual Instruction Tuning}. In \bibinfo{booktitle}{\emph{Thirty-seventh Conference on Neural Information Processing Systems}}, Vol.~\bibinfo{volume}{36}. \bibinfo{publisher}{Curran Associates, Inc.}, \bibinfo{pages}{34892--34916}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=w0H2xGHlkw}
\showURL{%
\tempurl}
\bibitem[Liu et~al\mbox{.}(2021)]%
{Liu2021Que2SearchFA}
\bibfield{author}{\bibinfo{person}{Yiqun Liu}, \bibinfo{person}{Kaushik Rangadurai}, \bibinfo{person}{Yunzhong He}, \bibinfo{person}{Siddarth Malreddy}, \bibinfo{person}{Xunlong Gui}, \bibinfo{person}{Xiaoyi Liu}, {and} \bibinfo{person}{Fedor Borisyuk}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Que2Search: Fast and Accurate Query and Document Understanding for Search at Facebook}. In \bibinfo{booktitle}{\emph{Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \& Data Mining}} \emph{(\bibinfo{series}{KDD '21})}. \bibinfo{publisher}{Association for Computing Machinery}, \bibinfo{pages}{3376–3384}.
\newblock
\showISBNx{9781450383325}
\href{https://doi.org/10.1145/3447548.3467127}{doi:\nolinkurl{10.1145/3447548.3467127}}
\bibitem[Luo et~al\mbox{.}(2022)]%
{Luo2021CLIP4ClipAE}
\bibfield{author}{\bibinfo{person}{Huaishao Luo}, \bibinfo{person}{Lei Ji}, \bibinfo{person}{Ming Zhong}, \bibinfo{person}{Yang Chen}, \bibinfo{person}{Wen Lei}, \bibinfo{person}{Nan Duan}, {and} \bibinfo{person}{Tianrui Li}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning}.
\newblock \bibinfo{journal}{\emph{Neurocomput.}} \bibinfo{volume}{508}, \bibinfo{number}{C} (\bibinfo{date}{Oct.} \bibinfo{year}{2022}), \bibinfo{pages}{293–304}.
\newblock
\showISSN{0925-2312}
\href{https://doi.org/10.1016/j.neucom.2022.07.028}{doi:\nolinkurl{10.1016/j.neucom.2022.07.028}}
\bibitem[Ma et~al\mbox{.}(2022)]%
{Ma2022XCLIPEM}
\bibfield{author}{\bibinfo{person}{Yiwei Ma}, \bibinfo{person}{Guohai Xu}, \bibinfo{person}{Xiaoshuai Sun}, \bibinfo{person}{Ming Yan}, \bibinfo{person}{Ji Zhang}, {and} \bibinfo{person}{Rongrong Ji}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval}. In \bibinfo{booktitle}{\emph{Proceedings of the 30th ACM International Conference on Multimedia}} \emph{(\bibinfo{series}{MM '22})}. \bibinfo{publisher}{Association for Computing Machinery}, \bibinfo{pages}{638–647}.
\newblock
\showISBNx{9781450392037}
\href{https://doi.org/10.1145/3503161.3547910}{doi:\nolinkurl{10.1145/3503161.3547910}}
\bibitem[Mu et~al\mbox{.}(2022)]%
{Mu2021SLIPSM}
\bibfield{author}{\bibinfo{person}{Norman Mu}, \bibinfo{person}{Alexander Kirillov}, \bibinfo{person}{David~A. Wagner}, {and} \bibinfo{person}{Saining Xie}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{{SLIP:} Self-supervision Meets Language-Image Pre-training}. In \bibinfo{booktitle}{\emph{Computer Vision - {ECCV} 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part {XXVI}}} \emph{(\bibinfo{series}{Lecture Notes in Computer Science}, Vol.~\bibinfo{volume}{13686})}. \bibinfo{publisher}{Springer}, \bibinfo{pages}{529--544}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1007/978-3-031-19809-0\_30}
\showURL{%
\tempurl}
\bibitem[Radford et~al\mbox{.}(2021)]%
{Radford2021LearningTV}
\bibfield{author}{\bibinfo{person}{Alec Radford}, \bibinfo{person}{Jong~Wook Kim}, \bibinfo{person}{Chris Hallacy}, \bibinfo{person}{Aditya Ramesh}, \bibinfo{person}{Gabriel Goh}, \bibinfo{person}{Sandhini Agarwal}, \bibinfo{person}{Girish Sastry}, \bibinfo{person}{Amanda Askell}, \bibinfo{person}{Pamela Mishkin}, \bibinfo{person}{Jack Clark}, \bibinfo{person}{Gretchen Krueger}, {and} \bibinfo{person}{Ilya Sutskever}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Learning Transferable Visual Models From Natural Language Supervision}. In \bibinfo{booktitle}{\emph{Proceedings of the 38th International Conference on Machine Learning}} \emph{(\bibinfo{series}{Proceedings of Machine Learning Research}, Vol.~\bibinfo{volume}{139})}, \bibfield{editor}{\bibinfo{person}{Marina Meila} {and} \bibinfo{person}{Tong Zhang}} (Eds.). \bibinfo{publisher}{PMLR}, \bibinfo{pages}{8748--8763}.
\newblock
\urldef\tempurl%
\url{https://proceedings.mlr.press/v139/radford21a.html}
\showURL{%
\tempurl}
\bibitem[Sandler et~al\mbox{.}(2018)]%
{Sandler2018MobileNetV2IR}
\bibfield{author}{\bibinfo{person}{Mark Sandler}, \bibinfo{person}{Andrew~G. Howard}, \bibinfo{person}{Menglong Zhu}, \bibinfo{person}{Andrey Zhmoginov}, {and} \bibinfo{person}{Liang-Chieh Chen}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{MobileNetV2: Inverted Residuals and Linear Bottlenecks}. In \bibinfo{booktitle}{\emph{2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition}}. \bibinfo{publisher}{IEEE Computer Society}, \bibinfo{pages}{4510--4520}.
\newblock
\href{https://doi.org/10.1109/CVPR.2018.00474}{doi:\nolinkurl{10.1109/CVPR.2018.00474}}
\bibitem[Singh et~al\mbox{.}(2021)]%
{Singh2021FLAVAAF}
\bibfield{author}{\bibinfo{person}{Amanpreet Singh}, \bibinfo{person}{Ronghang Hu}, \bibinfo{person}{Vedanuj Goswami}, \bibinfo{person}{Guillaume Couairon}, \bibinfo{person}{Wojciech Galuba}, \bibinfo{person}{Marcus Rohrbach}, {and} \bibinfo{person}{Douwe Kiela}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{FLAVA: A Foundational Language And Vision Alignment Model}.
\newblock \bibinfo{journal}{\emph{2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}} (\bibinfo{year}{2021}), \bibinfo{pages}{15617--15629}.
\newblock
\href{https://doi.org/10.1109/CVPR52688.2022.01519}{doi:\nolinkurl{10.1109/CVPR52688.2022.01519}}
\bibitem[Tansey et~al\mbox{.}(2018)]%
{Tansey2018LeafSmoothedHS}
\bibfield{author}{\bibinfo{person}{Wesley Tansey}, \bibinfo{person}{Karl Pichotta}, {and} \bibinfo{person}{James~G. Scott}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Leaf-Smoothed Hierarchical Softmax for Ordinal Prediction}. In \bibinfo{booktitle}{\emph{AAAI Conference on Artificial Intelligence}}, Vol.~\bibinfo{volume}{32}.
\newblock
\href{https://doi.org/10.1609/aaai.v32i1.11754}{doi:\nolinkurl{10.1609/aaai.v32i1.11754}}
\bibitem[Wang et~al\mbox{.}(2022)]%
{Wang2021SimVLMSV}
\bibfield{author}{\bibinfo{person}{Zirui Wang}, \bibinfo{person}{Jiahui Yu}, \bibinfo{person}{Adams~Wei Yu}, \bibinfo{person}{Zihang Dai}, \bibinfo{person}{Yulia Tsvetkov}, {and} \bibinfo{person}{Yuan Cao}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{SimVLM: Simple Visual Language Model Pretraining with Weak Supervision}. In \bibinfo{booktitle}{\emph{International Conference on Learning Representations}}.
\newblock
\urldef\tempurl%
\url{https://openreview.net/forum?id=GUrhfTuf_3}
\showURL{%
\tempurl}
\bibitem[Wen et~al\mbox{.}(2023)]%
{Wen2023EnhancingDI}
\bibfield{author}{\bibinfo{person}{Zhoufutu Wen}, \bibinfo{person}{Xinyu Zhao}, \bibinfo{person}{Zhipeng Jin}, \bibinfo{person}{Yi Yang}, \bibinfo{person}{Wei Jia}, \bibinfo{person}{Xiaodong Chen}, \bibinfo{person}{Shuanglong Li}, {and} \bibinfo{person}{Lin Liu}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Enhancing Dynamic Image Advertising with Vision-Language Pre-training}. In \bibinfo{booktitle}{\emph{Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval}} (Taipei, Taiwan) \emph{(\bibinfo{series}{SIGIR '23})}. \bibinfo{publisher}{Association for Computing Machinery}.
\newblock
\showISBNx{9781450394086}
\href{https://doi.org/10.1145/3539618.3591844}{doi:\nolinkurl{10.1145/3539618.3591844}}
\bibitem[Xiao et~al\mbox{.}(2023)]%
{Xiao2023ExploringTF}
\bibfield{author}{\bibinfo{person}{Lin Xiao}, \bibinfo{person}{Xiaofeng Li}, {and} \bibinfo{person}{Yucheng Zhang}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Exploring the factors influencing consumer engagement behavior regarding short-form video advertising: A big data perspective}.
\newblock \bibinfo{journal}{\emph{Journal of Retailing and Consumer Services}} \bibinfo{volume}{70} (\bibinfo{year}{2023}), \bibinfo{pages}{103170}.
\newblock
\showISSN{0969-6989}
\href{https://doi.org/10.1016/j.jretconser.2022.103170}{doi:\nolinkurl{10.1016/j.jretconser.2022.103170}}
\bibitem[Xu et~al\mbox{.}(2021)]%
{Xu2021VideoCLIPCP}
\bibfield{author}{\bibinfo{person}{Hu Xu}, \bibinfo{person}{Gargi Ghosh}, \bibinfo{person}{Po-Yao Huang}, \bibinfo{person}{Dmytro Okhonko}, \bibinfo{person}{Armen Aghajanyan}, \bibinfo{person}{Florian Metze}, \bibinfo{person}{Luke Zettlemoyer}, {and} \bibinfo{person}{Christoph Feichtenhofer}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{{V}ideo{CLIP}: Contrastive Pre-training for Zero-shot Video-Text Understanding}. In \bibinfo{booktitle}{\emph{Conference on Empirical Methods in Natural Language Processing}}. \bibinfo{publisher}{Association for Computational Linguistics}, \bibinfo{pages}{6787--6800}.
\newblock
\href{https://doi.org/10.18653/v1/2021.emnlp-main.544}{doi:\nolinkurl{10.18653/v1/2021.emnlp-main.544}}
\bibitem[Xu et~al\mbox{.}(2023)]%
{Xu2023mPLUG2AM}
\bibfield{author}{\bibinfo{person}{Haiyang Xu}, \bibinfo{person}{Qinghao Ye}, \bibinfo{person}{Mingshi Yan}, \bibinfo{person}{Yaya Shi}, \bibinfo{person}{Jiabo Ye}, \bibinfo{person}{Yuanhong Xu}, \bibinfo{person}{Chenliang Li}, \bibinfo{person}{Bin Bi}, \bibinfo{person}{Qiuchen Qian}, \bibinfo{person}{Wei Wang}, \bibinfo{person}{Guohai Xu}, \bibinfo{person}{Ji Zhang}, \bibinfo{person}{Songfang Huang}, \bibinfo{person}{Feiran Huang}, {and} \bibinfo{person}{Jingren Zhou}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video}. In \bibinfo{booktitle}{\emph{International Conference on Machine Learning}} \emph{(\bibinfo{series}{ICML'23})}. \bibinfo{publisher}{JMLR.org}, Article \bibinfo{articleno}{1614}, \bibinfo{numpages}{21}~pages.
\newblock
\bibitem[Yao et~al\mbox{.}(2021)]%
{Yao2021LearningAP}
\bibfield{author}{\bibinfo{person}{Shaowei Yao}, \bibinfo{person}{Jiwei Tan}, \bibinfo{person}{Xi Chen}, \bibinfo{person}{Keping Yang}, \bibinfo{person}{Rong Xiao}, \bibinfo{person}{Hongbo Deng}, {and} \bibinfo{person}{Xiaojun Wan}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Learning a Product Relevance Model from Click-Through Data in E-Commerce}. In \bibinfo{booktitle}{\emph{Proceedings of the Web Conference 2021}} \emph{(\bibinfo{series}{WWW '21})}. \bibinfo{publisher}{Association for Computing Machinery}, \bibinfo{pages}{2890–2899}.
\newblock
\showISBNx{9781450383127}
\href{https://doi.org/10.1145/3442381.3450129}{doi:\nolinkurl{10.1145/3442381.3450129}}
\bibitem[Yao et~al\mbox{.}(2024)]%
{Yao2024MiniCPMVAG}
\bibfield{author}{\bibinfo{person}{Yuan Yao}, \bibinfo{person}{Tianyu Yu}, \bibinfo{person}{Ao Zhang}, \bibinfo{person}{Chongyi Wang}, \bibinfo{person}{Junbo Cui}, \bibinfo{person}{Hongji Zhu}, \bibinfo{person}{Tianchi Cai}, \bibinfo{person}{Haoyu Li}, \bibinfo{person}{Weilin Zhao}, \bibinfo{person}{Zhihui He}, \bibinfo{person}{Qi-An Chen}, \bibinfo{person}{Huarong Zhou}, \bibinfo{person}{Zhensheng Zou}, \bibinfo{person}{Haoye Zhang}, \bibinfo{person}{Shengding Hu}, \bibinfo{person}{Zhi Zheng}, \bibinfo{person}{Jie Zhou}, \bibinfo{person}{Jie Cai}, \bibinfo{person}{Xu Han}, \bibinfo{person}{Guoyang Zeng}, \bibinfo{person}{Dahai Li}, \bibinfo{person}{Zhiyuan Liu}, {and} \bibinfo{person}{Maosong Sun}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{MiniCPM-V: A GPT-4V Level MLLM on Your Phone}.
\newblock \bibinfo{journal}{\emph{ArXiv}} (\bibinfo{year}{2024}).
\newblock
\urldef\tempurl%
\url{https://arxiv.org/abs/2408.01800}
\showURL{%
\tempurl}
\bibitem[Ye et~al\mbox{.}(2023)]%
{Ye2023QueryawareMB}
\bibfield{author}{\bibinfo{person}{Chengcan Ye}, \bibinfo{person}{Ting Peng}, \bibinfo{person}{Tim Chang}, \bibinfo{person}{Zhiyi Zhou}, {and} \bibinfo{person}{Feng Wang}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Query-aware Multi-modal based Ranking Relevance in Video Search}. In \bibinfo{booktitle}{\emph{Conference on Empirical Methods in Natural Language Processing}}. \bibinfo{publisher}{Association for Computational Linguistics}.
\newblock
\href{https://doi.org/10.18653/v1/2023.emnlp-industry.31}{doi:\nolinkurl{10.18653/v1/2023.emnlp-industry.31}}
\bibitem[Yin et~al\mbox{.}(2023)]%
{Yin2023ASO}
\bibfield{author}{\bibinfo{person}{Shukang Yin}, \bibinfo{person}{Chaoyou Fu}, \bibinfo{person}{Sirui Zhao}, \bibinfo{person}{Ke Li}, \bibinfo{person}{Xing Sun}, \bibinfo{person}{Tong Xu}, {and} \bibinfo{person}{Enhong Chen}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{A Survey on Multimodal Large Language Models}.
\newblock \bibinfo{journal}{\emph{ArXiv}} \bibinfo{volume}{abs/2306.13549} (\bibinfo{year}{2023}).
\newblock
\urldef\tempurl%
\url{https://arxiv.org/pdf/2306.13549}
\showURL{%
\tempurl}
\bibitem[Yu et~al\mbox{.}(2022)]%
{Yu2022CoCaCC}
\bibfield{author}{\bibinfo{person}{Jiahui Yu}, \bibinfo{person}{Zirui Wang}, \bibinfo{person}{Vijay Vasudevan}, \bibinfo{person}{Legg Yeung}, \bibinfo{person}{Mojtaba Seyedhosseini}, {and} \bibinfo{person}{Yonghui Wu}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{CoCa: Contrastive Captioners are Image-Text Foundation Models}.
\newblock \bibinfo{journal}{\emph{Transactions on Machine Learning Research}} (\bibinfo{year}{2022}).
\newblock
\showISSN{2835-8856}
\urldef\tempurl%
\url{https://openreview.net/forum?id=Ee277P3AYC}
\showURL{%
\tempurl}
\bibitem[Zhang et~al\mbox{.}(2024)]%
{Zhang2024MMLLMsRA}
\bibfield{author}{\bibinfo{person}{Duzhen Zhang}, \bibinfo{person}{Yahan Yu}, \bibinfo{person}{Jiahua Dong}, \bibinfo{person}{Chenxing Li}, \bibinfo{person}{Dan Su}, \bibinfo{person}{Chenhui Chu}, {and} \bibinfo{person}{Dong Yu}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{{MM}-{LLM}s: Recent Advances in {M}ulti{M}odal Large Language Models}. In \bibinfo{booktitle}{\emph{Findings of the Association for Computational Linguistics: ACL 2024}}. \bibinfo{publisher}{Association for Computational Linguistics}, \bibinfo{address}{Bangkok, Thailand}, \bibinfo{pages}{12401--12430}.
\newblock
\href{https://doi.org/10.18653/v1/2024.findings-acl.738}{doi:\nolinkurl{10.18653/v1/2024.findings-acl.738}}
\bibitem[Zhu et~al\mbox{.}(2025)]%
{Zhu2023QueryLIFEQL}
\bibfield{author}{\bibinfo{person}{Hai Zhu}, \bibinfo{person}{Yuankai Guo}, \bibinfo{person}{Ronggang Dou}, {and} \bibinfo{person}{Kai Liu}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Query-LIFE: Query-aware Language Image Fusion Embedding for E-Commerce Relevance}. In \bibinfo{booktitle}{\emph{International Conference on Computational Linguistics}}. \bibinfo{publisher}{Association for Computational Linguistics}.
\newblock
\urldef\tempurl%
\url{https://aclanthology.org/2025.coling-industry.2/}
\showURL{%
\tempurl}
\bibitem[Zou et~al\mbox{.}(2021)]%
{Zou2021PretrainedLM}
\bibfield{author}{\bibinfo{person}{Lixin Zou}, \bibinfo{person}{Shengqiang Zhang}, \bibinfo{person}{Hengyi Cai}, \bibinfo{person}{Dehong Ma}, \bibinfo{person}{Suqi Cheng}, \bibinfo{person}{Shuaiqiang Wang}, \bibinfo{person}{Daiting Shi}, \bibinfo{person}{Zhicong Cheng}, {and} \bibinfo{person}{Dawei Yin}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Pre-trained Language Model based Ranking in Baidu Search}. In \bibinfo{booktitle}{\emph{Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \& Data Mining}} \emph{(\bibinfo{series}{KDD '21})}. \bibinfo{publisher}{Association for Computing Machinery}, \bibinfo{pages}{4014–4022}.
\newblock
\showISBNx{9781450383325}
\href{https://doi.org/10.1145/3447548.3467147}{doi:\nolinkurl{10.1145/3447548.3467147}}
\end{thebibliography}