The Potential of Vision-Language Models for Content Moderation of Children's Videos
Extracted source text · arXiv 2312.03936
\documentclass[conference]{IEEEtran}
% \IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out%.
%\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
% added by SHA begin
\usepackage{algorithm}
%\usepackage{caption}
%\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{multirow}
\usepackage{tabularx}
\usepackage{colortbl}
\definecolor{Gray}{gray}{0.9}
\usepackage{array} %for table fix width
\usepackage{paralist}
\graphicspath{ {figures/} }
%\usepackage[backend=bibtex,bibstyle=ieee,citestyle=numeric-comp]{biblatex}
% \bibliographystyle{plain} % We choose the "plain" reference style
% \bibliography{refs} % Entries are in the refs.bib file
% added by SHA end
% \def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
% T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
%\title{Robust Content Moderation of Children's Videos By Adapting Joint Vision-Language Models}
\title{The Potential of Vision-Language Models for Content Moderation of Children's Videos}
% \author{Paper \#519}
\input{authors}
\maketitle
\input{abstract}
\input{introduction}
% \input{background}
\input{relatedwork}
\input{method}
\input{results}
\input{conclusion}
\input{bibliography}
\end{document}
%\author{\IEEEauthorblockN{1\textsuperscript{st} Syed Hammad Ahmed}
\author{\IEEEauthorblockN{Syed Hammad Ahmed}
% \author{\IEEEauthorblockN{Syed Hammad Ahmed}
\IEEEauthorblockA{\textit{Department of Computer Science} \\
\textit{University of Central Florida}\\
Orlando, USA \\
syed.hammad.ahmed@ucf.edu}
% \and
% \IEEEauthorblockN{2\textsuperscript{nd} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
\and
\IEEEauthorblockN{Shengnan Hu}
% \IEEEauthorblockN{Gita Sukthankar}
\IEEEauthorblockA{\textit{Department of Computer Science} \\
\textit{University of Central Florida}\\
Orlando, USA \\
shengnan.hu@ucf.edu}
\and
\IEEEauthorblockN{Gita Sukthankar}
% \IEEEauthorblockN{Gita Sukthankar}
\IEEEauthorblockA{\textit{Department of Computer Science} \\
\textit{University of Central Florida}\\
Orlando, USA \\
gitars@eecs.ucf.edu}
}
\begin{abstract}
Natural language supervision has been shown to be effective for zero-shot learning in many computer vision tasks, such as object detection and activity recognition. However, generating informative prompts can be challenging for more subtle tasks, such as video content moderation. This can be difficult, as there are many reasons why a video might be inappropriate, beyond violence and obscenity. For example, scammers may attempt to create junk content that is similar to popular educational videos but with no meaningful information. This paper evaluates the performance of several CLIP variations for content moderation of children's cartoons in both the supervised and zero-shot setting. We show that our proposed model (Vanilla CLIP with Projection Layer) outperforms previous work conducted on the Malicious or Benign (MOB) benchmark for video content moderation. This paper presents an in depth analysis of how context-specific language prompts affect content moderation performance. Our results indicate that it is important to include more context in content moderation prompts, particularly for cartoon videos as they are not well represented in the CLIP training data.
%Natural language supervision has demonstrated impressive zero shot performance at many classic computer vision tasks such as object detection and activity recognition. In many of these tasks, it is easy to generate informative prompts. Video content moderation is the problem of evaluating videos to determine whether they contain inappropriate content. Children's videos may be inappropriate for many reasons, beyond violence and obscenity; scammers attempt to create junk content that is similar to popular educational videos but with no meaningful information. This paper evaluates the performance of several CLIP variations for content moderation in both the supervised and zero shot setting. We present an evaluation of different prompt strategies for detecting malicious cartoon videos. Our results indicate that it is important to craft more detailed prompts since cartoons are not well represented in the CLIP training data.
\end{abstract}
\begin{IEEEkeywords}
video content moderation, vision-language models, prompt engineering
\end{IEEEkeywords}
\section{Introduction}
Automated video content moderation is a necessity for protecting children from inappropriate videos uploaded to social media platforms. Every minute YouTube content creators upload around 300 hours of video \cite{ytstats1}. In the United States, more than 80\% children aged 11 years or under watch YouTube videos, out of which almost 50\% have come across inappropriate content as reported by their parents \cite{Auxier_2020}. Cartoon videos are especially problematic because they are attractive to young viewers. Scammers often create junk content that includes familiar cartoon characters in order to monetize their videos~\cite{elsa}. Children younger than 6 years of age are more prone to viewing harmful content as they are unable to distinguish between appropriate and inappropriate. Continuous exposure to malicious video content may have adverse effects on overall cognitive development~\cite{adverse_effect_1}.
YouTube and other video hosting platforms have tried to mitigate this serious problem of content moderation using automated techniques yet have not been successful in providing robust solutions. YouTube Kids was introduced as a means to publish child-appropriate videos, but YouTube acknowledges in \cite{ytk_fail} that they are unable to screen all malicious videos.
%Figure \ref{fig:cartoon} shows an example of a malicious cartoon video published on YouTube Kids depicting violence and aggression by a cartoon character.
Supervised approaches to video content moderation rely on the existence of hand labeled datasets~\cite{sambamulti1,ahmed2023malicious}.
These datasets are typically small and go out of date quickly as scammers mimic newly trending content. However, impressive zero shot performance on a variety of computer vision tasks such as object detection~\cite{vanilla_clip}, action recognition~\cite{actionclip}, and depth estimation~\cite{depthclip} has been achieved by joint vision-language models, such as CLIP (Contrastive Language-Image Pre training)~\cite{vanilla_clip}.
This paper presents an evaluation of the performance of several CLIP variations on the Malicious or Benign benchmark for children's content moderation of cartoon videos~\cite{ahmed2023malicious}. The Malicious or Benign dataset includes challenging cases of malicious junk content videos that were designed to mimic popular educational cartoons. These videos lack explicit obscenity or violence but contain loud sounds, fast motions, and characters with scary appearances. Thus it is challenging to devise prompts that are sufficiently general to describe the disturbing activities. Reynolds and McDonell~\cite{reynolds2021promptprogramming} present an overview of prompt programming strategies for large language models that illustrates how different prompting strategies can yield very different outcomes on downstream tasks.
This paper makes the following contributions:
\begin{compactitem}
\item Introduces a model (Vanilla CLIP with Projection Layer) that outperforms previous content moderation techniques on the MOB benchmark~\cite{ahmed2023malicious};
\item Performs an in-depth analysis of how context-specific language prompts affect the content moderation performance of different CLIP variants;
\item Proposes new benchmark prompt templates for the MOB Dataset.
\end{compactitem}
%Our proposed work focuses on employing joint vision-language models to address the problem of malicious and benign video classification. The CLIP model \cite{vanilla_clip} and other CLIP-based state-of-the-art video classification architectures are evaluated in zero-shot settings on the MOB dataset \cite{ahmed2023malicious}. We also efficiently adapt all of these benchmark models by incorporating fully-tunable projection layers while freezing all parameters of the models, to perform supervised learning of the MOB dataset \cite{ahmed2023malicious}. The choice of text prompts is crucial and affects the performance of vision-language (VL) models. Therefore, we performed Prompt Programming for the MOB Dataset and perform a thorough analysis of various combinations of prompts.
%Various unimodal as well as multimodal methods are proposed to effectively solve the problem. Multimodal solutions with sub-titles and video meta-data as input are presented by Le et al. \cite{sambamulti1}, whereas Chuttur et al. \cite{chuttur_multimodal} proposed a solution with user comments and selected video frames as the 2 input streams. KidGuard \cite{kidsguard} proposed a unimodal architecture where the video classifier is based on Long Short Term Memory (LSTM) and classifies 1-second videos as violent, sexual, both, or safe.
%Malicious cartoon videos include unusual cartoon characters having scary and disgusting appearances as well as characters performing violent or obscene activities \cite{ahmed2023malicious}. Furthermore, the paper also classifies those videos to be malicious as well in which objects or cartoon characters are in fast or repetitive motion. In addition to these visual features, the authors have identified malicious audio features which include loud, fast, repetitive music and sound effects in addition to offensive language use.
%Psychological studies empirically show that children who watch malicious content have problems in doing basic chores, including problems in reading, composition, and understanding written text \cite{psycho1,psycho2, psycho3}. These studies also emphasize that violence in cartoon videos lead to stress and aggressive behavior in children.
%Ahmed et al. \cite{ahmed2023malicious} published a list of intricate audio-visual malicious features and the MOB (Malicious or Benign) dataset which includes annotations of these complex features. The authors addressed the content moderation problem for children's videos using supervised video classification. Furthermore, the paper contributes a set of complex audio and visual features which are often gone undetected by current video hosting platform's algorithms. Table \ref{tab:features} depicts these features which are also annotated as labels of the published MOB dataset. The MOB dataset contributed in the paper comprises of cartoon videos classified as \textit{malicious} or \textit{benign}.
% previous work to address the problem of video classification
% lstm
% clip
% current work how is it different from previous work
% no VL and 0-shot
%\begin{figure}
% \centering
% \includegraphics[scale=.11]{figures/cartoonnetwork.png}
% \caption{Snapshot of a cartoon video available on YouTube Kids showing a violent scene marked as suitable for children 4 years and under.}
% \label{fig:cartoon}
%\end{figure}
\begin{figure*}[htp]
\centering
% \includegraphics[scale=.13]
\includegraphics[width=0.6\textwidth]
%{figures/Fig1.drawio.png}
{figures/architecture.png}
\caption{Proposed architecture for language-assisted supervision of video content moderation. The image encoder produces frame-level embeddings with dimensions 1 x 512 after which Temporal Pooling finds the joint representation of all frames of the input video $v_i$,. Similarly, the encoder in the language branch of our model produces text embeddings, $t_i$ after performing tokenization and positional encoding of the input prompt. Finally, cosine similarity is calculated between each ($v_i$, $t_i$) pair which maximizes the similarity for the video for the class for which the related prompt was learnt.}
% \caption{Proposed architecture for language-assisted supervision of video content moderation.}
\label{fig:model}
\end{figure*}
\section{Related Work}\label{RW}
Previous work on the application of machine learning for video content moderation for children can be categorized into approaches that focus on a single modality including
meta-data \cite{uni1}, video \cite{taxila, kidsguard}, or text (user comments) \cite{uni3} vs.\ approaches that leverage multimodal data~\cite{sambamulti1,chuttur_multimodal}. The Samba system \cite{sambamulti1} uses subtitles and meta-data to classify inappropriate videos, and Chuttur et al. \cite{chuttur_multimodal} combine both user comments and images to detect inappropriate videos.
Natural language supervision models such as CLIP~\cite{vanilla_clip} and ALIGN \cite{jia2021scaling_align} combine video and text in a different way than multimodal models. Rather than using aligned videos and subtitles as the classification input, a natural language prompt is used to describe the instance. The supervised learning process aims to maximize the similarity between the video embedding and prompt text embedding (Figure~\ref{fig:model}), rather than seeking to minimize classification loss. Zero-shot video classification approaches extract visual features in the different frames of a video using a backbone such as Convolutional 3D (C3D) \cite{tran2015learning_c3d}, ResNet \cite{he2015deep_resnet} or ViT \cite{VIT}, before learning a temporal behavior that maps visual representations to semantic embeddings.
Ahmed et al. \cite{ahmed2023malicious} published a hand labeled dataset and a set of video benchmarks for classifying children's cartoon videos as malicious or benign. We demonstrate that our proposed method outperforms these benchmarks on the supervised learning content moderation task. Unlike other datasets, the MOB dataset includes malicious content that parodies children's educational videos. The dataset includes a list of audio and video features (summarized in Table \ref{tab:features}) that were used to characterize junk videos. In this paper we utilize these feature labels for our prompt generation.
\begin{table}[h!]
\centering
\caption{Malicious Video and Audio features identified in \cite{ahmed2023malicious} }
\begin{tabular}{|p{0.15\columnwidth} | p{0.75\columnwidth}|}
\hline
\textbf{Video} & \textbf{fast repetitive motions, scary/disgusting appearance, hurting/destruction/killing activity, obscene/indecent activity} \\
\hline
\textbf{Audio} & \textbf{loud music/noise, screaming or shouting, explosion or gunshot sounds, offensive language} \\
\hline
\end{tabular}
% \caption{Malicious Video and Audio features identified in \cite{ahmed2023malicious} }
\label{tab:features}
\end{table}
% Reynolds and McDonell~\cite{reynolds2021promptprogramming} present an overview of prompt programming strategies for large language models that illustrates how different prompting strategies can yield very different outcomes on downstream tasks.
% We therefore discuss how we systematically selected the prompts for the MOB Dataset \cite{ahmed2023malicious}. The related work includes the concept of Prompt programming \cite{reynolds2021promptprogramming} where the authors suggest techniques of how language models can be exploited in the best way to adapt to solve various downstream tasks.
%a list of intricate audio-visual malicious features and which includes annotations of these complex features. The authors addressed the content moderation problem for children's videos using supervised video classification. Furthermore, the paper contributes a set of complex audio and visual features which are often gone undetected by current video hosting platform's algorithms. Table depicts these features which are also annotated as labels of the published MOB dataset. The MOB dataset contributed in the paper comprises of cartoon videos classified as \textit{malicious} or \textit{benign}.
%However, to the best of our knowledge no one as explored 1) Vision-Language models like CLIP \cite{vanilla_clip} that uses Vision Transformer backbones, as well as 2) Zero-shot classification to solve the problem of content moderation of kids videos.
%Zero-shot video classification approaches initially find out the visual features in the different frames of a video using some typical backbone such as Convolutional 3D (C3D) \cite{tran2015learning_c3d}, ResNet \cite{he2015deep_resnet}, or ViT \cite{VIT}, then learns temporal behavior which maps visual representations to semantic embeddings. Some of the works which discuss Zero-shot video classification include \cite{bishay2019tarn_zs1, Gan_Lin_Yang_Melo_G_Hauptmann_2016_zs2, hahn2019action2vec_zs3, zs4}.
%Similarly, traditional video classification methods which used 3DCNNs \cite{carreira2018quo_vidclassif1, Feichtenhofer_2019_ICCV_vidclassif2} were outperformed by transformer-based approaches \cite{Arnab_2021_ICCV_transformer, Liu_2022_CVPR_swin_tranformer}.
%Recently, models like CLIP \cite{vanilla_clip} and ALIGN \cite{jia2021scaling_align} which provide large-scale pre-training of joint vision and text data have shown promising results. These have been further adapted to solve various computer vision problems including video classification. Some of the CLIP-based models which have demonstrated state-of-the-art results include ViFi \cite{vifi}, Action-CLIP \cite{actionclip}, AIM \cite{yang2023aim}, and X-CLIP \cite{ma2022xclip}. We also perform a detailed analysis of the importance of the selection of text prompts for the performance of CLIP-based models. We therefore discuss how we systematically selected the prompts for the MOB Dataset \cite{ahmed2023malicious}. The related work includes the concept of Prompt programming \cite{reynolds2021promptprogramming} where the authors suggest techniques of how language models can be exploited in the best way to adapt to solve various downstream tasks.
% discuss projection layer work???
% fine-tuning for images/videos??
\section{Methodology}\label{Methodology}
Figure~\ref{fig:model} shows our general architecture for video content moderation. Videos are classified as malicious or benign based on a short time window of input frames. We compare the performance of different video embedding and prompt generation strategies for both the supervised and zero shot setting. This paper evaluates the performance of four variants of vision-language pretraining:
\begin{compactenum}
\item \textbf{Vanilla CLIP~\cite{vanilla_clip}} The simplest way to adapt the CLIP model for video classification is to use temporal pooling to create a combined representation of the $T$ frames input to the model.
\item \textbf{ViFi CLIP~\cite{vifi}} Since CLIP was designed and trained for static images, the temporal representations and object interactions in videos are not explicitly modelled. To address this, ViFi simply performs video fine-tuning on the pre-trained CLIP model. Temporal pooling is used to aggregate representation across frames.
\item \textbf{AIM-CLIP~\cite{yang2023aim}} AIM leverages the CLIP model's video encoders through fully trainable lightweight blocks called \textit{adapters}. Thus, AIM trains on a substantially lower number of parameters. Normally AIM only uses the vision branch of the CLIP model to fine-tune its adapters on video datasets. In this paper, we incorporated the language transformer in the AIM model and then use it to classify malicious or benign cartoon videos.
\item \textbf{ActionCLIP~\cite{actionclip}} ActionCLIP was specifically designed for action recognition. It utilizes CLIP for video-text input and makes use of a multi-level prompt strategy.
\end{compactenum}
\subsection{Supervised Learning}
Since full fine-tuning on videos is computationally expensive, we simply add a projection layer on top of the CLIP model while freezing the latter. Projection layers have been used by various models both in fine tuning scenarios and also for dimensionality reduction \cite{projectionlayer1, projectionlayer2, projectionlayer3}. This helps the model adapt to the downstream task better, as only task-specific parameters are learnt by this layer. Our projection layer has 768 input nodes and 512 output nodes and connects on top of the image encoder which has 768 output nodes. We use ViT \cite{VIT} which uses image patches of size 16 x 16; hence for 3 channels a patch can be fully represented by a vector of size ${16 \times 16 \times 3 = 768}$. We freeze the image and text encoders and leave the projection layer open for tuning as illustrated in Figure \ref{fig:model}.
%This simple improvisation of adding a fully-trainable Projection Layer for supervised learning outperformed other benchmark performances on the MOB dataset with an accuracy of 80.3\%.
\subsection{Zero-shot Classification}
We also performed a zero-shot evaluation on different CLIP-based models. With the exception of Vanilla CLIP, we use respective models pre-trained on the Kinetics-400 dataset \cite{kinetics}.
\subsection{Prompt Engineering and Ensembling} \label{prompteng}
Modifying the input text prompts can have a significant effect on CLIP's performance~\cite{vanilla_clip,reynolds2021promptprogramming}. After performing rigorous prompt engineering we identified seven prompt templates that empirically performed better than other text prompts during the trial-and-error process.
\subsubsection{Adding cartoon context to default prompt}\label{context} The standard prompt used in CLIP is \textit{``a photo of a \{ \}."}. As the reference dataset is cartoon only, based on this intuition we added the token \textit{``cartoon"} and devised two new prompts: 1) \textit{``a \{ \} cartoon."} and 2) \textit{``a photo of a \{ \} cartoon."}.
\subsubsection{Prompt candidate generation with cartoon context tokens}\label{promptgen}
A typical prompt used in CLIP is ``a photo of a \{ \}". As discussed in Section \ref{context}, we add the token ``cartoon" so that the tokens become context-specific as the reference dataset is of cartoons only. A general prompt format after adding context becomes: ``a \textit{clip-token} of a \textit{context-token}". From the prompts published for other datasets we enumerated an initial list of clip-tokens and another list of context-tokens from the cartoon synonyms. Based on each prompt generated, we perform zero-shot performance analysis and selected the tokens contributing to the top performing prompts, to create a candidate list of tokens. The initial and final lists are shown in Table \ref{tab:tokens}.
%%%%%%%%%%%%%%%%%
\begin{table}[h!]
\centering
\caption{Initial and Final Candidate Prompt Template token list used to generate context-specific prompts}
%\caption{Prompt template token lists}
\begin{tabular}{|p{0.35\columnwidth} | p{0.55\columnwidth}|}
\hline
clip-tokens (initial) & `photo', `video', `example', `demonstration', `image' \\
\hline
context-tokens (initial) & `cartoon', `animation', `caricature', `comic', `character' \\
\hline
clip-tokens (candidate) & `image', `example' \\
\hline
context-tokens (candidate) & `cartoon', `caricature', `comic' \\
\hline
\end{tabular}
% \caption{Initial and Final Candidate Prompt Template token list used to generate context-specific prompts}
\label{tab:tokens}
\end{table}
%%%%%%%%%%%%%%%%
% \\
% \textbf{initial:}
% \\
% clip-tokens = [`photo', `video', `example', `demonstration', `image']
% \\
% \hspace{1em}
% context-tokens = [`cartoon', `animation', `caricature', `comic', `character']
% \\
% \textbf{candidate:}
% \\clip-tokens = [`image' `example']
% \\context-tokens = [`cartoon', `caricature', `comic']
% \\
The following formats were used to generate the prompt templates:
\\
\textbf{prompt formats:}
\\`a \textit{clip-tokens$_{i}$} of a \{\} \textit{context-tokens$_{j}$}.',
\\`a \textit{clip-tokens$_{i}$} of a \textit{context-tokens$_{j}$} which is \{\}.'
\\`a \textit{context-tokens$_{j}$} which is \{\}.'
\\
The prompt templates are generated by placing all pairs (\textit{clip-tokens$_{i}$}, \textit{context-tokens$_{j}$}) in the respective placeholders of each of the three prompt formats.
% \begin{algorithm}[H]\label{candidatealgo}
% \caption{Algorithm for generating candidate prompt templates}
% \begin{algorithmic}[1]
% \FOR{\textit{all prompt-formats} $k$}
% \FOR{\textit{all clip-tokens} $i$}
% \FOR{\textit{all context-tokens} $j$}
% \STATE\textit{template \xleftarrow{} promptformats_k}
% % \textit{replace_string(template, clip-tokens$_i$}
% % \textit{replace_string(template, clip-tokens$_i$}
% \ENDFOR
% \ENDFOR
% \ENDFOR
% \RETURN $P$
% \end{algorithmic}
% \end{algorithm}
\subsubsection{Prompt generation using all combinations of cartoon features} \label{featuretokenmethod}
The MOB dataset includes information about the presence or absence of the malicious features shown in Table \ref{tab:features}. Similarly, we devise a list of malicious and benign feature tokens as follows:
\\
\textbf{malicious feature tokens:} ["fast-moving", "scary", "disgusting", "hurting", "destructive", "killing", "obscene", "indecent"]
\\
\textbf{benign feature tokens:} ["good", "friendly", "happy", "joyful", "singing", "enjoying", "loving", "caring", "playing", "funny"]
\\
These lists form a third list of tokens and a candidate list is generated similar to the algorithm described in Section \ref{promptgen}. The prompt format used is:
\\`a \textit{clip-tokens$_{i}$} of a \{\} \textit{context-tokens$_{j}$} which is \{\} and \textit{feature-tokens$_{k}$}.'
\subsubsection{Prompt candidate generation based top-performing frequent token pairs} \label{apriorimethod}
The zero shot evaluation of the exhaustive prompt templates generated from the initial lists described in Section \ref{promptgen} is performed on all pairs (\textit{clip-tokens$_{i}$}, \textit{context-tokens$_{j}$}). The prompt templates with higher than the median performance score (accuracy) are formulated as a list of items in a transaction and given as input to the Apriori rule association algorithm \cite{apriori} which finds the frequent itemsets of size 2. All those frequent pairs become candidate pairs which are then used to generate all prompt templates using the prompt formats discussed in Secion \ref{promptgen}.
\subsubsection{Prompts of other relevant datasets}
We also performed a performance evaluation for the MOB dataset \cite{ahmed2023malicious} using the exact prompt templates as published for other datasets. We evaluated prompts of UCF101 \cite{soomro2012ucf101}, CIFAR10 and CIFAR100 \cite{cifar}, and FER2013 (FacialEmotionRecognition2013) dataset \cite{FER2013}. FER2013 gave the best performance for specific prompt templates for some of the benchmarks.
\section{Evaluation}\label{sec:results}
\begingroup
\renewcommand{\arraystretch}{1.6}
\begin{table*}[!h]
\centering
%\caption{\textbf{Supervised setting:} Evaluation accuracy scores for the benchmarks in the supervised setting where projection layer (PL) is kept fully tunable while video and text encoders are kept frozen. The columns show the different prompts for which we ran our experiments with the different benchmarks on the MOB Dataset. The prompts for the \textit{\textbf{frequent item-set combinations}} outperform other prompts.}
\caption{\textbf{Supervised setting:} The columns show the different prompts for which we ran our experiments with the benchmarks on the MOB Dataset. The prompts for the frequent item-set combinations outperform other prompts.}
%\caption{\textbf{Supervised setting:} Evaluation accuracy scores for the benchmarks in the supervised setting where projection layer (PL) is kept fully tunable while video and text encoders are kept frozen.}
%\caption{Supervised Setting}
\begin{tabularx}{\textwidth} {
>{\raggedright\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X }
\multirow{2}{5em}{\textbf{Model}} & \multicolumn{7}{c |}{\textbf{Text Prompt Templates}}\\
\cline{2-8}
& \textbf{'a photo of a \{\}.'} & \textbf{'a \{\} cartoon.'} & \textbf{'a photo of a \{\} cartoon.'} & \textbf{clip+context token pair combinations} & \textbf{feature-based token combinations} & \textbf{frequent item-set combinations} & \textbf{FER-2013} \\
\hline
\hline
Vanilla (PL) & 78.9 & 78.1 & 77.8 & 76.3 & 75.2 & \textbf{80.3} & 78.5 \\ \hline
ViFi-CLIP (PL) & 71.0 & 71.3 & 69.9 & 69.2 & 71.3 & \textbf{71.7} & \textbf{71.7} \\ \hline
AIM-CLIP (PL) & 68.5 & 65.5 & 67.7 & 66.7 & 66.3 & \textbf{69.9} & 67.7 \\ \hline
\end{tabularx}
% \caption{\textbf{Supervised setting:} Evaluation accuracy scores for the benchmarks in the supervised setting where projection layer (PL) is kept fully tunable while video and text encoders are kept frozen. The columns show the different prompts for which we ran our experiments with the different benchmarks on the MOB Dataset. The prompts for the \textit{\textbf{frequent item-set combinations}} outperform other prompts.}
\label{tab:supervised}
\end{table*}
\endgroup
\begingroup
\renewcommand{\arraystretch}{1.6}
\begin{table*}[!h]
\centering
\caption{\textbf{Zero-shot setting: } The prompts using multiple combinations of context tokens give the best zero-shot classification score.}
%\caption{Zero-shot setting}
\begin{tabularx}{\textwidth} {
>{\raggedright\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X
| >{\centering\arraybackslash}X }
\multirow{2}{5em}{\textbf{Model}} & \multicolumn{7}{c |}{\textbf{Text Prompt Templates}}\\
\cline{2-8}
& \textbf{'a photo of a \{\}.'} & \textbf{'a \{\} cartoon.'} & \textbf{'a photo of a \{\} cartoon.'} & \textbf{clip+context token pair combinations} & \textbf{feature-based token combinations} & \textbf{frequent item-set combinations} & \textbf{FER-2013} \\
\hline
\hline
Vanilla & 62 & 65.6 & 67 & \textbf{68.5} & 67.7 & 67 & 66.3 \\\hline
ViFi-CLIP & 53.4 & 57.3 & \textbf{58.8} & 58.4 & 58.1 & 53.4 & 56.3 \\\hline
AIM-CLIP & \textbf{64.2} & 60.9 & \textbf{64.2} & 62.0 & 54.8 & 60.9 & 60.9 \\\hline
ActionCLIP & 56.6 & 54.8 & 54.8 & 55.6 & 53.8 & \textbf{60.9} & \textbf{60.9} \\ \hline
\end{tabularx}
% \caption{\textbf{Zero-shot setting: }Evaluation accuracy scores for the benchmarks in the zero-shot setting where Vanilla CLIP is adapted for videos whereas all other models are pre-trained on Kinetics-400 dataset. The prompts using multiple combinations of context tokens give the best zero-shot classification score.}
\label{table:zeroshot}
\end{table*}
\endgroup
This section presents results of our evaluation vs.\ the MOB benchmarks~\cite{ahmed2023malicious}. The MOB dataset is annotated with domain-specific features that commonly occur in inappropriate children's cartoon videos. It includes 1875 clips, each 10 seconds in length and frame rate = 25 fps. The dataset has two classes: malicious and benign.
For fair comparison, we used the ViT-B/16 backbone for all CLIP variants discussed in this paper. ViT-B/16 refers to ViT's Base model with an image input resolution of 224 x 224, patch sizes 16 x 16, 12 layers, 12 attention heads, and 12 layers and 8 heads for text input. In the supervised setting, training was performed for 20 epochs with a batch size of 16 and the number of frames set to 16. Adam optimizer with learning rate 1e-4 was used for all experiments. Experiments were run on a 64-bit system with NVIDIA GeForce RTX 3090 GPU and 12th Gen Intel Core i7 CPU with 64 GB of memory.
\subsection{Supervised Learning}
In this section we present results of our proposed supervised language pretraining setting where we use a learnable projection layer to classify videos as malicious or benign. Table \ref{tab:supervised} presents the overview of results for supervised classification with different prompting strategies. We trained and tested three CLIP-based models, out of which the best performing benchmark was our adapted Vanilla CLIP which is labelled with the suffix \textit{(PL)}. ViFi-CLIP (fine-tuned on the Kinetics-400 \cite{kinetics} video dataset) performed the second-best while AIM-CLIP, our adaptation of AIM for CLIP, stood last in terms of accuracy. The best overall supervised classification results for the MOB Dataset are achieved by our proposed Vanilla-CLIP PL with a testing accuracy of 80.3\% which beat all previously published benchmarks from \cite{ahmed2023malicious}. Table \ref{tab:newsota} provides a summary of the comparison vs. the other benchmarks.
\subsection{Prompt Generation Strategies}
The columns in our tables show the results of different prompt generation strategies.
In all our tables, \textit{clip+context token pair combinations} refers to the prompts discussed in Section \ref{promptgen}. \textit{Feature-based token combinations} refers to prompts explained in Section \ref{featuretokenmethod}, and \textit{frequent item-set combinations} refers to text prompt templates generated through the method described in Section \ref{apriorimethod}. Interestingly for all these three benchmarks, the prompt templates generated using the frequent item-set combinations approach give the best results on supervised learning.
We evaluated the zero-shot performance of different prompt generation strategies in Table \ref{table:zeroshot}. Although there is no single dominant prompting strategy, the context-based prompts show better performance during zero-shot inference. The best prompt template is the set of prompts generated using both, clip and cartoon context tokens. For both settings, the top results for each benchmark include prompt templates generated by the techniques which use cartoon context-based tokens.
This is important since the initial CLIP dataset primarily contains natural images rather than cartoons. %Therefore, adding context can generally improve the performance of joint vision-language models.
\begingroup
\begin{table}[h!]
\centering
%%\caption{Comparison of supervised setting for classification of MOB dataset. Our proposed method, Vanilla (PL) where we employed a fully-trainable \textbf{P}rojection \textbf{L}ayer while freezing all other model parameters outperforms the other benchmark accuracy scores on the MOB Dataset.}
\caption{Supervised setting vs.\ other benchmarks}
\begin{tabular}{|c|c|}
\hline
% \rowcolor{Gray}
\textbf{Benchmark} & \textbf{Accuracy (\%)} \\ [0.5ex]
\hline
VTN \cite{vtn} & 77.9 \\
\hline
I3D \cite{i3d} & 72.1 \\
\hline
ConvLSTM \cite{convlstm} & 69.7 \\
\hline
\textbf{Vanilla (PL)} & \textbf{80.3} \\
\hline
\end{tabular}
% \caption{Comparison of supervised setting for classification of MOB dataset. Our proposed method, Vanilla (PL) where we employed a fully-trainable \textbf{P}rojection \textbf{L}ayer while freezing all other model parameters outperforms the other benchmark accuracy scores on the MOB Dataset.}
\label{tab:newsota}
\end{table}
\section{Conclusion and Future Work}
In this paper we discussed how the problem of video content moderation for children of ages 1-5 years can be addressed using state-of-the-art language supervision techniques. We explored the usage of a foundation model for language pre-training, CLIP (Contrastive Language–Image Pre-training), and performed evaluations on several CLIP variants in supervised and zero-shot settings. In supervised settings, we employed projection layers to improve training on the cartoon video dataset. We also highlighted how language prompts which include context-specific tokens can affect performance on different video classification models. Lastly, we propose the benchmark prompt templates for MOB Dataset for training and evaluating joint vision-language models like CLIP.
We believe that less well defined video analysis problems such as content moderation pose a significant challenge to prompting strategies. It is challenging for humans to verbally define what makes a video inappropriate. There is a famous quote about pornography by a Supreme Court Justice: ``I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description, and perhaps I could never succeed in intelligibly doing so. But I know it when I see it.''~\cite{pornography} In the future we plan to apply prompt learning approaches where the prompts are learnable parameters, and also explore the performance on cartoon datasets when fine-tuning is done on other pre-trained video datasets.
% future work:
% apply prompt learning
% fine tune other pre-trained video datasets
% \printbibliography
\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,references}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{ytstats1}
\BIBentryALTinterwordspacing
D.~Donchev, ``{40 Mind Blowing YouTube Facts, Figures and Statistics – 2023},'' accessed 2023-07-24. [Online]. Available: \url{https://fortunelords.com/youtube-statistics/}
\BIBentrySTDinterwordspacing
\bibitem{Auxier_2020}
\BIBentryALTinterwordspacing
B.~Auxier, ``Parenting children in the age of screens,'' Jul 2020. [Online]. Available: \url{https://www.pewresearch.org/internet/2020/07/28/parenting-children-in-the-age-of-screens/}
\BIBentrySTDinterwordspacing
\bibitem{elsa}
D.~D. Placido, ``{YouTube's "Elsagate"} illuminates the unintended horrors of the digital age,'' Forbes, 11 2017.
\bibitem{adverse_effect_1}
\BIBentryALTinterwordspacing
K.~Habib and T.~Soliman, ``Cartoons’ effect in changing children mental response and behavior,'' \emph{Open Journal of Social Sciences}, vol.~03, no.~09, p. 248–264, Jan 2015. [Online]. Available: \url{https://doi.org/10.4236/jss.2015.39033}
\BIBentrySTDinterwordspacing
\bibitem{ytk_fail}
\BIBentryALTinterwordspacing
``Important info for parents about {YouTube Kids},'' accessed 2023-07-25. [Online]. Available: \url{https://support.google.com/youtubekids/answer/6130561}
\BIBentrySTDinterwordspacing
\bibitem{sambamulti1}
L.~Binh, R.~Tandon, C.~Oinar, J.~Liu, U.~Durairaj, J.~Guo, S.~Zahabizadeh, S.~Ilango, J.~Tang, F.~Morstatter, S.~Woo, and J.~Mirkovic, ``Samba: Identifying inappropriate videos for young children on {YouTube},'' in \emph{Proceedings of the ACM International Conference on Information and Knowledge Management}, 2022, p. 88–97.
\bibitem{ahmed2023malicious}
S.~H. Ahmed, M.~J. Khan, H.~M.~U. Qaisar, and G.~Sukthankar, ``{Malicious or Benign? Towards Effective Content Moderation for Children's Videos},'' in \emph{Proceedings of the International FLAIRS Conference}, vol.~36, 2023.
\bibitem{vanilla_clip}
A.~Radford, J.~W. Kim, C.~Hallacy, A.~Ramesh, G.~Goh, S.~Agarwal, G.~Sastry, A.~Askell, P.~Mishkin, J.~Clark, G.~Krueger, and I.~Sutskever, ``Learning transferable visual models from natural language supervision,'' 2021.
\bibitem{actionclip}
M.~Wang, J.~Xing, and Y.~Liu, ``{ActionCLIP}: A new paradigm for video action recognition,'' \emph{arXiv preprint arXiv:2109.08472}, 2021.
\bibitem{depthclip}
R.~Zhang, Z.~Zeng, Z.~Guo, and Y.~Li, ``Can language understand depth?'' in \emph{Proceedings of the ACM International Conference on Multimedia}, 2022, pp. 6868--6874.
\bibitem{reynolds2021promptprogramming}
L.~Reynolds and K.~McDonell, ``Prompt programming for large language models: Beyond the few-shot paradigm,'' 2021.
\bibitem{uni1}
M.~Gkolemi, P.~Papadopoulos, E.~Markatos, and N.~Kourtellis, ``{YouTubers Not MadeForKids: Detecting Channels Sharing Inappropriate Videos Targeting Children},'' in \emph{ACM Web Science Conference}, 2022, p. 370–381.
\bibitem{taxila}
K.~Yousaf and T.~Nawaz, ``\BIBforeignlanguage{en}{A deep learning-based approach for inappropriate content detection and classification of {YouTube} videos},'' \emph{\BIBforeignlanguage{en}{IEEE Access}}, vol.~10, p. 16283–16298, 2022.
\bibitem{kidsguard}
S.~Singh, R.~Kaushal, A.~B. Buduru, and P.~Kumaraguru, ``Kidsguard: fine grained approach for child unsafe video representation and detection,'' in \emph{Proceedings of the ACM/SIGAPP Symposium on Applied Computing}, 2019, p. 2104–2111.
\bibitem{uni3}
S.~Alshamrani, A.~Abusnaina, M.~Abuhamad, D.~Nyang, and D.~Mohaisen, ``Hate, obscenity, and insults: Measuring the exposure of children to inappropriate comments in {YouTube},'' in \emph{Companion Proceedings of the Web Conference}.\hskip 1em plus 0.5em minus 0.4em\relax Association for Computing Machinery, 2021, p. 508–515.
\bibitem{chuttur_multimodal}
M.~Y. Chuttur and A.~Nazurally, ``\BIBforeignlanguage{en}{A multi-modal approach to detect inappropriate cartoon video contents using deep learning networks},'' \emph{\BIBforeignlanguage{en}{Multimedia Tools and Applications}}, vol.~81, no.~12, p. 16881–16900, May 2022.
\bibitem{jia2021scaling_align}
C.~Jia, Y.~Yang, Y.~Xia, Y.-T. Chen, Z.~Parekh, H.~Pham, Q.~V. Le, Y.~Sung, Z.~Li, and T.~Duerig, ``Scaling up visual and vision-language representation learning with noisy text supervision,'' 2021.
\bibitem{tran2015learning_c3d}
D.~Tran, L.~Bourdev, R.~Fergus, L.~Torresani, and M.~Paluri, ``Learning spatiotemporal features with 3d convolutional networks,'' 2015.
\bibitem{he2015deep_resnet}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Deep residual learning for image recognition,'' 2015.
\bibitem{VIT}
A.~Dosovitskiy, L.~Beyer, A.~Kolesnikov, D.~Weissenborn, X.~Zhai, T.~Unterthiner, M.~Dehghani, M.~Minderer, G.~Heigold, S.~Gelly \emph{et~al.}, ``An image is worth 16x16 words: Transformers for image recognition at scale,'' \emph{arXiv preprint arXiv:2010.11929}, 2020.
\bibitem{vifi}
H.~Rasheed, M.~U. Khattak, M.~Maaz, S.~Khan, and F.~S. Khan, ``Fine-tuned {CLIP} models are efficient video learners,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2023, pp. 6545--6554.
\bibitem{yang2023aim}
T.~Yang, Y.~Zhu, Y.~Xie, A.~Zhang, C.~Chen, and M.~Li, ``{AIM}: Adapting image models for efficient video action recognition,'' in \emph{The International Conference on Learning Representations}, 2023.
\bibitem{projectionlayer1}
\BIBentryALTinterwordspacing
P.~Kaliamoorthi, S.~Ravi, and Z.~Kozareva, ``Prado: Projection attention networks for document classification on-device,'' in \emph{Conference on Empirical Methods in Natural Language Processing}, 2019. [Online]. Available: \url{https://api.semanticscholar.org/CorpusID:202785536}
\BIBentrySTDinterwordspacing
\bibitem{projectionlayer2}
C.~Sankar, S.~Ravi, and Z.~Kozareva, ``Proformer: Towards on-device lsh projection based transformers,'' \emph{arXiv preprint arXiv:2004.05801}, 2020.
\bibitem{projectionlayer3}
A.~Frome, G.~S. Corrado, J.~Shlens, S.~Bengio, J.~Dean, M.~Ranzato, and T.~Mikolov, ``Devise: A deep visual-semantic embedding model,'' \emph{Advances in neural information processing systems}, vol.~26, 2013.
\bibitem{kinetics}
W.~Kay, J.~Carreira, K.~Simonyan, and B.~Zhang, ``{The Kinetics Human Action Video Dataset},'' 2017.
\bibitem{apriori}
R.~Agrawal, R.~Srikant \emph{et~al.}, ``Fast algorithms for mining association rules,'' in \emph{Proc of VLDB}, vol. 1215.\hskip 1em plus 0.5em minus 0.4em\relax Santiago, Chile, 1994, pp. 487--499.
\bibitem{soomro2012ucf101}
K.~Soomro, A.~R. Zamir, and M.~Shah, ``{UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild},'' 2012.
\bibitem{cifar}
A.~Krizhevsky, G.~Hinton \emph{et~al.}, ``Learning multiple layers of features from tiny images,'' 2009.
\bibitem{FER2013}
I.~J. Goodfellow, D.~Erhan, P.~L. Carrier, A.~Courville, M.~Mirza, B.~Hamner, W.~Cukierski, Y.~Tang, D.~Thaler, D.-H. Lee, Y.~Zhou, C.~Ramaiah, F.~Feng, R.~Li, X.~Wang, D.~Athanasakis, J.~Shawe-Taylor, M.~Milakov, J.~Park, R.~Ionescu, M.~Popescu, C.~Grozea, J.~Bergstra, J.~Xie, L.~Romaszko, B.~Xu, Z.~Chuang, and Y.~Bengio, ``Challenges in representation learning: A report on three machine learning contests,'' in \emph{Neural Information Processing}, M.~Lee, A.~Hirose, Z.-G. Hou, and R.~M. Kil, Eds., 2013, pp. 117--124.
\bibitem{vtn}
D.~Neimark, O.~Bar, M.~Zohar, and D.~Asselmann, ``Video transformer network,'' in \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, 2021, pp. 3163--3172.
\bibitem{i3d}
J.~Carreira and A.~Zisserman, ``Quo vadis, action recognition? a new model and the kinetics dataset,'' in \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, 2017, pp. 6299--6308.
\bibitem{convlstm}
J.~Donahue, L.~Anne~Hendricks, S.~Guadarrama, M.~Rohrbach, S.~Venugopalan, K.~Saenko, and T.~Darrell, ``Long-term recurrent convolutional networks for visual recognition and description,'' in \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, 2015, pp. 2625--2634.
\bibitem{pornography}
\BIBentryALTinterwordspacing
P.~Lattman, ``The origins of {Justice Stewart's} {I} know it when {I} see it,'' 2007. [Online]. Available: \url{https://www.wsj.com/articles/BL-LB-4558}
\BIBentrySTDinterwordspacing
\end{thebibliography}