Noise-Tolerant Learning for Audio-Visual Action Recognition
Extracted source text · arXiv 2205.07611
%common set
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
\usepackage[linesnumbered, ruled]{algorithm2e}
\SetKwRepeat{Do}{do}{while}%
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{booktabs, multicol, multirow}
\usepackage{stfloats}
\usepackage{cleveref}
\usepackage{url}
\usepackage{verbatim}
\usepackage{titlesec}
\usepackage{graphicx}
\usepackage{cite}
\usepackage{bm}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{float}
\usepackage{amsmath}
\newcommand{\etal}{\textit{et al.}}
\newcommand{\crefrangeconjunction}{--}
\usepackage[T1]{fontenc}
\crefrangeformat{equation}{eq. #3(#1)#4--#5(#2)#6}
\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}
\def\eg{\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot}
\def\ie{\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot}
\def\st{\emph{s.t}\onedot} \def\St{\emph{S.t}\onedot}
\def\cf{\emph{c.f}\onedot} \def\Cf{\emph{C.f}\onedot}
\def\etc{\emph{etc}\onedot} \def\vs{\emph{vs}\onedot}
\def\wrt{w.r.t\onedot} \def\dof{d.o.f\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
\title{Noise-Tolerant Learning for Audio-Visual Action Recognition}
\author{Haochen Han, Qinghua Zheng, Minnan Luo, Kaiyao Miao, Feng Tian and Yan Chen
% <-this % stops a space
\thanks{This study was supported by the National Key Research and Development Program of China (No. 2020AAA0108800), National Nature Science Foundation of China (No. 61872287, No. 62192781, No. 61937001, No. 62050194, No. 62137002), Innovative Research Group of the National Natural Science Foundation of China (61721002), Innovation Research Team of Ministry of Education (IRT\_17R86), Project of China Knowledge Center for Engineering Science and Technology, and Project of Chinese academy of engineering ``The Online and Offline Mixed Educational Service System for ‘The Belt and Road’ Training in MOOC China''.
Haochen Han, Qinghua Zheng, Minnan Luo, Feng Tian and Yan Chen are with the Shaanxi Provincial Key Laboratory of Institute of Multimedia Knowledge Fusion and Engineering, and also with the School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China. (E-mail: hhc2077@outlook.com; qhzheng@xjtu.edu.cn; minnluo@xjtu.edu.cn; fengtian@mail.xjtu.edu.cn; chenyan@mail.xjtu.edu.cn).
Kaiyao Miao is with the Ministry of Education Key Laboratory for Intelligent Networks and Network Security, and also with the School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, 710049, China. (E-mail: miaokaiyao@gmail.com).
}}
% The paper headers
\markboth{Journal of \LaTeX\ Class Files,~Vol.~14, No.~8, August~2021}%
{Shell \MakeLowercase{\textit{et al.}}: A Sample Article Using IEEEtran.cls for IEEE Journals}
\begin{document}
\maketitle
\IEEEpubidadjcol
\IEEEpubid{\begin{minipage}{\textwidth}\ \\[12pt] \centering
0000--0000/00\$00.00~\copyright~2021 IEEE\\
\end{minipage}}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.
\begin{abstract}
Recently, video recognition is emerging with the help of multi-modal learning, which focuses on integrating multiple modalities to improve the performance or robustness of a model. Although various multi-modal learning methods have been proposed and offer remarkable recognition results, almost all of these methods rely on high-quality manual annotations and assume that modalities among multi-modal data provide relevant semantic information. Unfortunately, most widely used video datasets are collected from the Internet and inevitably contain noisy labels and noisy correspondence. To solve this problem, we use the audio-visual action recognition task as a proxy and propose a noise-tolerant learning framework to find anti-interference model parameters to both noisy labels and noisy correspondence. Our method consists of two phases and aims to rectify noise by the inherent correlation between modalities. A noise-tolerant contrastive training phase is performed first to learn robust model parameters unaffected by the noisy labels. To reduce the influence of noisy correspondence, we propose a cross-modal noise estimation component to adjust the consistency between different modalities. Since the noisy correspondence existed at the instance level, a category-level contrastive loss is proposed to further alleviate the interference of noisy correspondence. Then in the hybrid supervised training phase, we calculate the distance metric among features to obtain corrected labels, which are used as complementary supervision. In addition, we investigate the noisy correspondence in real-world datasets and conduct comprehensive experiments with synthetic and real noise data. The results verify the advantageous performance of our method compared to state-of-the-art methods.
\end{abstract}
\begin{IEEEkeywords}
Action recognition, audio-visual learning, noisy labels, noisy correspondence.
\end{IEEEkeywords}
\section{Introduction}
\IEEEPARstart{W}{ith} the growing popularity of mobile devices and online video platforms, people are generating and consuming a huge amount of video content every day. Recent study has shown that over 1 billion hours of video is watched each day on YouTube. This trend has encouraged advanced techniques to accurately recognize actions or events in the videos\cite{liu2018weakly,li2018unified,panda2021adamml}, which can benefit a wide range of applications, including video summarization, video retrieval, and video recommendation.
As an information-intensive media, video is rich in multiple modalities such as frame, motion(optical flow) and audio. Therefore, recent advancements in action recognition have mostly focused on integrating various modalities to improve the performance\cite{panda2021adamml,wang2020makes} or robustness\cite{tian2021can} of supervised model. To utilize the massive unlabeled videos from Internet-scale dataset, self-supervised multi-modal learning methods are proposed to leverage the strong correlation among modalities to obtain pretrained model\cite{XDC,arandjelovic2018objects,rouditchenko2019self}. However, the promising results of most existing methods depend on both clean-annotated and clean-corresponding datasets, which are expensive and time-consuming. Unfortunately, most widely used video datasets such as Kinetics\cite{kay2017kinetics} and YouTube-8M\cite{abu2016youtube} are collected from the Internet, which inevitably contain both noisy labels and noisy correspondence. Specifically, noisy labels are corrupted from the ground-truth labels and thus result in poor generalization performance\cite{han2018co,zhang2021robust}. To alleviate the harmful effects from noisy labels, numerous approaches have been proposed, such as Co-teaching\cite{han2018co}, Meta-Weight-Net\cite{shu2019meta}, and DivideMix\cite{li2020dividemix}. Although these studies have achieved encouraging success, it's challenging to adopt them to video recognition task. On the one hand, they are proposed to tackle the uni-modal situation and cannot simultaneously integrate multiple modalities in video. On the other hand, these studies generally based on the memorization effect of DNNs\cite{arpit2017closer}, and thus distinguish noisy and clean data by the loss difference. However, video media also exist noisy correspondence, which may confuse the loss of noisy and clean data. As shown in Figure \ref{fig_loss}, the existence of noisy correspondence makes the loss difference not as distinguishable as the uni-modal scenarios. Specifically, noisy correspondence is that the multiple modalities in instance provide irrelevant or redundant information leading to sub-optimal performance\cite{panda2021adamml}. Take Kinetics dataset for example, usually the sound tracks of videos are unrelated to the visual content, \eg, human voices masks the sound of instruments, and music montage for a parkour video. Moreover, multi-modal networks are more prone to overfit the noise due to their increased capacity\cite{wang2020makes}. Thus, it's extra challenging and complicated to consider both noisy labels and noisy correspondence simultaneously.
\begin{figure}[H]
\vspace*{-8mm}
\centering
\subfloat[]{\includegraphics[width=0.25\textwidth]{clean.pdf}%
\label{loss_clean}}\hspace{-4mm}
\hfil
\subfloat[]{\includegraphics[width=0.25\textwidth]{noise.pdf}%
\label{loss_nc}}
\caption{Cross-entropy loss $vs.$ epoch on UCF101 dataset under noise for clean-annotated and noisy label samples. (a) Training the late-fusion audio-visual network with 60\% noisy labels and no noisy correspondence. (b) Training the late-fusion audio-visual network with both 60\% noisy labels and 60\% noisy correspondence. From the figure, we can see that the noisy correspondence will confuse the loss of noisy label samples and clean-annotated samples. }
\label{fig_loss}
\end{figure}
Audio and vision information represent the two most essential ways people perceive the world, which are also the two most common modalities in video media. Therefore, we use the audio-visual action recognition task as a proxy to explore the multi-modal learning with both noisy labels and noisy correspondence. Previous works in noisy labels area conduct controlled experiments by injecting different kind of synthetic noises into clean dataset, \eg, Symmetry and Asymmetry. Although these synthetic construction of noisy labels can appropriately simulate the noises in real world, they cannot be applied to simulate the noisy correspondence data. Based on the investigation from the real-world Kinetics dataset, noisy correspondence is more complex and irregular, and unfortunately, there is no benchmark to construct controlled noisy correspondence data. To this end, we annotate the validation set of Kinetics dataset to estimate the noisy correspondence level of each class and then conduct experiments in real-world data with controlled level of noise.
To address the aforementioned problems, we propose a novel noise-tolerant learning framework to find anti-interference model parameters to both noisy labels and noisy correspondence. Our method consists of two phases and aims to rectify noise by the inherent correlation between modalities. A noise-tolerant contrastive training phase is performed prior to the conventional supervised training, aiming to learn robust model parameters unaffected by the noisy labels. To reduce the influence of noisy correspondence, we propose a cross-modal noise estimation component based on the observation shown in the Figure \ref{fig_obv} that the clusters in the feature space by either modality should be the same if the modalities among sample have the related semantic information, which aims to adjust the consistency between different modalities in an instance. Since the noisy correspondence produced by the irrelevant modalities at the instance level, we propose a category-level contrastive loss to further alleviate the interference of noisy correspondence. Then in the hybrid supervised training phase, we calculate the distance metric among the robust features to obtain corrective labels, which are used as complementary supervision for the supervised training.
To our best knowledge, this is the first work to consider both noisy labels and noisy correspondence simultaneously in an audio-visual action recognition task. The main novelties and contributions are summarized as follows:
\begin{itemize}
\item We investigate the noisy correspondence problem in real-world dataset and we propose the first benchmark in audio-visual data with controlled noisy correspondence levels.
\item We propose a noise-tolerant learning framework for audio-visual action recognition with both noisy labels and noisy correspondence that uses the inherent correlation between different modalities to rectify noise.
\item Extensive experiments are conducted with a wide range of noise levels, and demonstrate the advantageous performance of our approach in audio-visual action recognition tasks compared to state-of-the-art methods.
\end{itemize}
%
\begin{figure*}[!t]
\centering
\subfloat[]{\includegraphics[width=0.43\textwidth]{motivation_correct.pdf}%
\label{correct_c}}\hspace{-10mm}
\hfil
\subfloat[]{\includegraphics[width=0.43\textwidth]{motivation_noise.pdf}%
\label{noisy_c}}
\caption{The motivation of cross-modal noise estimation. In this figure, we denote $(\bm{v_i},\bm{a_i})$, $(\bm{v_j},\bm{a_j})$ and $(\bm{v_k},\bm{a_k})$ as the features of audio-visual pairs, and show their similarity in the common space. (a) Correct Correspondence: If $(\bm{v_i},\bm{a_i})$ has correct correspondence, we use the audio modality to find the cluster samples $\bm{a_j}$ and $\bm{a_k}$, and their corresponding visual modality $\bm{v_j}$ and $\bm{v_k}$ also close to $\bm{v_i}$ in common space, vice versa. (b) Noisy Correspondence: If $\bm{a_i}$ is irrelevant to $\bm{v_i}$, the corresponding visual modality $\bm{v_j}$ and $\bm{v_k}$ are far away from $\bm{v_i}$ in common space.}
\label{fig_obv}
\end{figure*}
\section{Related Work}
\subsection{Audio-visual Video Learning}
Video understanding is an essential research areas of computer vision and multimedia. Previous works have gained a significant success in understanding temporal information of video modality \cite{karpathy2014large, qiu2017learning,feichtenhofer2019slowfast,tran2018closer,wang2016actions,xie2017rethinking}. In recent years, the learning trend from single-modality to multi-modality has become active for better performance. Among the rich modalities in video, visual and audio information are the most common modalities thus have been widely developed. One typical technique is to combine the features for different modality and jointly optimize them \cite{petridis2015prediction,wang2020makes}. Alternatively, some works aim to learn effective representation based on the semantic relation between modalities and transfer to downstream tasks. Relja \etal \cite{arandjelovic2018objects} propose a cross-modal self-supervision method to localize the audio object within an image. Humam \etal \cite{XDC} extend Deep Cluster \cite{deepcluster} to multi-modal situation which leverage unsupervised clustering to produce supervisory signal for the other. Andrew \etal \cite{rouditchenko2019self} introduce an approach based on concepts disentanglement and semantic categories assignment for audio-visual co-segmentation. More advancements can be referred in the recent audio-visual learning survey\cite{zhu2021deep}, which also indicates the previous works pay less attention to audio-visual learning under noise.
\subsection{Learning with Label Noise}
Most existing approaches for training DNNS with label noise focus on uni-modal scenarios. One typical direction is to adjust the loss of training samples before updating the DNN, multiplying the predicted outputs by the estimated transition matrix \cite{patrini2017making,xia2019anchor}. To obtain a more precise transition matrix, Hendrycks \etal \cite{hendrycks2018using} use a small dataset with clean labels
as additional information, Yao \etal \cite{yao2020dual} factorize the matrix into the product of two simple-estimated matrices. Moreover, Yang \etal \cite{yang2021estimating} estimate the instance-dependent noise transition probability by Bayesian optimization from distill samples. Another direction focus on re-weighting the contribution of each sample on the loss. Zhang \etal \cite{zhang2021dualgraph} employ graphical model and re-weight samples by the structural relations among labels. Wang \etal \cite{wang2018iterative} perform unsupervised learning to help re-weight the loss by pushing away noisy samples from correct samples in representation space. Recently, meta learning has shown robust learning ability to help infer the weight for loss automatically \cite{ren2018learning,shu2019meta,li2019learning}. However, most prior arts designed for uni-modal scenarios can't directly adopted to multi-modal
cases since the multi-modal data contain data noise that two or more modalities are irrelevant in semantic meaning. Thus, samples with data noise can yield high loss and be misclassified as noisy-label cases. In the multi-modal scenarios, Hu \etal \cite{hu2021learning} combine a robust clustering loss with a multi-modal contrastive loss to address cross-modal retrieval with noisy labels. In recent, Huang \etal \cite{huang2021learning} introduce a new paradigm of noisy labels, termed noisy correspondence, which refers to the mismatch paired in multi-modal samples. Specifically, they use the memorization effect of neural networks to divide the data and then rectifies them via an adaptive prediction model. Contrary to all aforementioned methods, our method designed for the multi-modal situation and consider both noisy labels and noisy correspondence simultaneously.
\subsection{Multi-modal Contrastive Learning}
Contrastive learning can be considered as learning by comparing among the input samples. Most previous works aim to learn single-modal representations by constructing positive and negative pairs through a variety of data augmentation \cite{chen2020simple,van2018representation,wu2018unsupervised,li2021contrastive}. Recently, various works has shown the potential of contrastive learning for multi-modal data. For example, Tian \etal \cite{tian2020contrastive} regard different views(\eg, L and ab channels) of the image as the data augmentation and maximize mutual information between different views of the same sample. In addition to contrasting between different modalities of the same sample, Morgado \etal \cite{morgado2021audio} introduce a method to group together videos with high similarity both in video and audio modality, and these groups are considered as extended positive pairs to directly optimize the visual representations. Yuan \etal \cite{yuan2021multimodal} propose a method combined intra-modal and inter-modal similarity preservation objectives to improve the quality of learned representation. To alleviate the influence of false negatives caused by random sampling, Yang \etal \cite{yang2021partially} propose a noise-robust contrastive loss to simultaneously learn representation and align data in multi-view learning. The existing works heavily rely on the correct correspondence between modalities. However, in practice, such a situation is difficult to satisfy. Different from them, our work aim to learn robust representation with noisy correspondence problem.
\section{Method}
\subsection{Problem Statement}
We are targeting a multi-modal video classification with both noisy labels and noisy correspondence. Contrary to the uni-modal case, multi-modal methods process each modality by a different encoder. After that, they concatenate the features and pass to a classifier. Here we consider two most common modalities in videos, RGB stream and audio track. Let $\mathcal{D} = (\bm{X},Y) = \{ (\bm{v_i},\bm{a_i},y_i) \}_{i=1}^N$ be a set of video data, where for each sample $i$, $\bm{v_i}\in \mathbb{R}^{d_{\mathrm{v}}}$ and $\bm{a_i}\in \mathbb{R}^{d_{\mathrm{a}}}$ are the visual stream and audio stream, and $y_i \in \{ 1,2,...,K \}$ is the label. Let $f_v(\cdot) : \mathbb{R}^{d_{\mathrm{v}}}\rightarrow\mathbb{R}^{d}$ with parameter $\bm{\Theta_v}$ and $f_a(\cdot): \mathbb{R}^{d_{\mathrm{a}}}\rightarrow\mathbb{R}^{d}$ with parameter $\bm{\Theta_a}$ denote the visual and audio encoder, respectively. Let $\bm{\theta}$ denote the multi-modal network's parameters.
The goal of supervised multi-modal classification is to minimize an empirical loss
\begin{equation}
\begin{split}
\label{eq:empirical loss}
\mathcal{L}( \bm{\theta},\bm{X}, Y ) = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L} \left(\mathcal{C} (f_v (\bm{v_i},\bm{\Theta_v} ) \oplus f_a (\bm{a_i},\bm{\Theta_a}) ),y_i \right),
\end{split}
\end{equation}
where $\mathcal{C}$ denotes a classifier and $\oplus$ denotes a fusion operation.
However, when $y_i$ contains noise, the multi-modal network might be overfitting and perform poorly on the clean test set; meanwhile, multi-modal data also contains noisy correspondence, which can be regarded as hard sample and makes uni-modal method perform sub-optimally. To deal with these issues, we propose a noise-tolerant learning framework and the details are elaborated in the following sections.
\begin{figure*}[!t]
\centering
\includegraphics[width=1\textwidth]{Framework_v3.pdf}
\caption{The framework of our proposed method. The noise-tolerant contrastive training phase and hybrid supervised training phase proceed iteratively until converged. The modal encoders are shared weights between these two phases. } \label{fig:framework}
\end{figure*}
\subsection{Overall Framework}
The motivation behind our method is that the inherent correlation between different modalities can aid in the rectification of noise. We achieve this by a novel noise-tolerant learning framework, which consists of two phase: noise-tolerant contrastive training phase and hybrid supervised training phase.
The noise-tolerant contrastive training phase is performed prior to the conventional supervised training, aiming to learn robust model parameters unaffected by the noisy labels. As illustrated in Figure \ref{fig:framework}, this phase is trained by the combination of two jointly learned contrastive loss: instance-level loss $(\mathcal{L}_{ins})$ and category-level contrastive loss $(\mathcal{L}_{cat})$. The instance-level loss is inspired by existing uni-modal contrastive framework SwAV \cite{swAV}. And we extended it to multi-modal situation by maximizing the consistency between modalities instead of data augmentation. Specifically, it clusters data from two different modalities simultaneously and leverage one modality’s cluster assignments as supervisory signal for other modality. However, there is an assumption that modalities among multi-modal data provide relevant semantic information. In practice, the existence of noisy correspondence makes such an assumption difficult to satisfy. Based on the observation shown in Figure \ref{fig_obv}, we address such limitation by designing a cross-modal noise estimation component that assign weights to adjust the consistency between different modalities in an instance. Since the noisy correspondence produced by the irrelevant modalities at the instance level, we propose a category-level contrastive loss as addition to further alleviate the interference of noisy correspondence. Specifically, for each modality, we first obtain the category representations by mapping features to the prototype vectors. Then, we embrace a common semantic autoencoder with tied weights to reconstruct the category representations, which can force the category representations to be closer in semantics and filter noisy information from original features.
In the hybrid supervised training phase, we calculate the distance metric among the robust features to obtain corrective labels. Although the corrected label is more approximate to ground truth label, training using only the corrected labels may lead self-convergence problem and thus yield sub-optimal results. Therefore, original labels and corrected labels are used complementarily as supervisory signal to train the multi-modal network. The proposed two phase trained in an iterative manner to find anti-interference model parameters to both noisy labels and noisy correspondence.
\subsection{Noise-Tolerant Contrastive Training Phase}
\subsubsection{Instance-level Contrastive Loss}
In the noise-tolerant contrastive training phase, model parameters are not affected by noisy labels; to alleviate the influence of noisy correspondence, we propose the cross-modal noise estimation to align the semantic consistency in instance level. The motivation of our method based on the observation illustrated in Figure \ref{fig_obv}. For one modality, we utilize the dense clusters from corresponding modality to excavate the similar samples, which are then used to calculate inter-modal similarity. Given an audio-visual pair $(\bm{v_i},\bm{a_i})$, we denote $\mathcal{J}_i^v$ and $\mathcal{J}_i^a$ as the set of similar samples extended from training samples, respectively,
\begin{equation}
\label{eq:J_v}
\mathcal{J}_i^v = \{ \bm{v_j} \mid \forall{j}: S(f_a(\bm{a_j}),f_a(\bm{a_i})) > {\epsilon}_a \},
\end{equation}
\begin{equation}
\label{eq:J_a}
\mathcal{J}_i^a = \{ \bm{a_j} \mid \forall{j}: S(f_v(\bm{v_j}),f_v(\bm{v_i})) > {\epsilon}_v \},
\end{equation}
where ${\epsilon}_a$ and ${\epsilon}_v$ are threshold hyperparameters to control the quality and quantity of the set, and $S(\cdot)$ is cosine similarity function. For convenience of presentation, we denote $\bm{z_i}$ and $\bm{z_j}$ as two arbitrary features in the common space, and the similarity function between them
is defined as
\begin{equation}
\label{eq:score function}
S(\bm{z_i},\bm{z_j}) = \frac{1}{\tau_1} \frac{\bm{z_i}^\mathrm{T}\bm{z_j}}{\lVert \bm{z_i} \rVert \lVert \bm{z_j} \rVert},
\end{equation}
where $\tau_1$ denotes temperature parameter. For the audio-video pair $(\bm{v_i},\bm{a_i})$, we generate the estimated probability that visual and audio data has correct correspondence as
\begin{equation}
\label{eq:p_hat_v}
\omega_i^v = \frac{1}{\lvert \mathcal{J}_v \rvert } \sum_{j \in \mathcal{J}_v} S(f_v(\bm{v_j}),f_v(\bm{v_i})),
\end{equation}
\begin{equation}
\label{eq:p_hat_a}
\omega_i^a = \frac{1}{\lvert \mathcal{J}_a \rvert } \sum_{j \in \mathcal{J}_a} S(f_a(\bm{a_j}),f_a(\bm{a_i})).
\end{equation}
For audio-visual video data, the strong correlation between visual and audio modality make it possible to leverage one modality's cluster assignments as supervisory for the other modality. Formally, given a mini-batch with $B$ samples $\{ (\bm{v_i},\bm{a_i}) \}_{i=1}^B$ from the training set, we first obtain the marginal cluster probability that each modality belongs to the $k$-th class:
\begin{equation}
\label{eq:prob visual modality}
p(y=k|\bm{v_i}) = \frac{exp(\frac{1}{\tau_2} (f_v(\bm{v_i}))^\mathrm{T} \bm{c_k})}
{\sum_{t=1}^K exp(\frac{1}{\tau_2} (f_v(\bm{v_i}))^\mathrm{T} \bm{c_t})},
\end{equation}
\begin{equation}
\label{eq:prob audio modality}
p(y=k|\bm{a_i}) = \frac{exp(\frac{1}{\tau_2} (f_v(\bm{a_i}))^\mathrm{T} \bm{c_k})}
{\sum_{t=1}^K exp(\frac{1}{\tau_2} (f_a(\bm{v_i}))^\mathrm{T} \bm{c_t})},
\end{equation}
where $\bm{C} = \{\bm{c}_1,...,\bm{c}_K\}$ are the trainable prototypes shared between two modalities and $\tau_2$ is a temperature parameter. Each $\bm{c}_k$ from $\bm{C}$ can be consider as the clustering center vector which representing the $k$-th class. We denote $q(y|\bm{v_i}) \in \{0,1\} ^K$ as the one-hot encoding of the predicted label from visual feature, and $q(y|\bm{a_i}) \in \{0,1\} ^K$ is analogous but uses audio feature. The contrastive learning objective comprises two terms(\ie, video and audio). The video term is defined as the cross-entropy loss between the predicted labels by audio modality and the marginal probability of video features:
\begin{equation}
\label{eq:CMCA entropy loss}
\mathbb{E}_{i}^{v}(p,q) = - \sum_{y=1}^{K} q(y|\bm{a_i})logp(y|\bm{v_i}).
\end{equation}
The formulation of audio term $\mathbb{E}_{i}^{a}(p,q)$ is analogous by exchanging the roles of $\bm{v_i}$ and $\bm{a_i}$. Note that the one-hot vector of predicted label is unknown, following previous work\cite{asano2019self}, it can be learned by transferring the optimization of contrastive learning objective (\ref{eq:CMCA entropy loss}) to optimal transport problem. In order to avoid trivial solution that all samples are predicted to same label, one constraint is added to partition the batch in equal size:
\begin{equation}
\label{eq:constraints}
\sum_{i=1}^{B} q(y|\bm{a_i}) = \frac{B}{K} \mbox{ and } \sum_{i=1}^{B}q(y|\bm{v_i}) = \frac{B}{K}.
\end{equation}
The optimal predicted vector $q(y|\bm{v_i})$ and $q(y|\bm{a_i})$ can be calculated efficiently by employing
the iterative Sinkhorn-Knopp algorithm \cite{sinkhorn}.
Owing to the existence of noisy correspondence, we combine the above noise estimation component with contrasting cluster assignments to adjust the consistency between two modalities in instance level. We thus formulate the instance-level contrastive loss as:
\begin{equation}
\label{eq:instance-level contrastive loss}
\mathcal{L}_{ins} = \frac{1}{B} \sum_{i=1}^{B} \left[ \omega_i^v \mathbb{E}_{i}^{v}(p,q) + \omega_i^a \mathbb{E}_{i}^{a}(p,q) \right].
\end{equation}
\subsubsection{Category-level Contrastive Loss}
Since the noisy correspondence produced by the irrelevant modalities at the instance level, we utilize the semantic information at the category level to further enhance the noise-tolerant ability of networks. Similar to instance-level loss, we use another trainable prototypes $\bm{C^{\prime}} = \{\bm{c}^{\prime}_1,...,\bm{c}^{\prime}_K\}$ to compute the dot-products with each modality in the batch. Formally, given mini-batch with $B$ samples $\{ (\bm{v_i},\bm{a_i}) \}_{i=1}^B$, for $i$-th sample, we denote $\bm{P}_v = \{\bm{p}_1^v,...,\bm{p}_K^v\}$ and $\bm{P}_a = \{\bm{p}_1^a,...,\bm{p}_K^a\}$ be the dot product results with softmax of video feature and audio feature, respectively. For each modality, the $k$-th element $\bm{p}_k^m \in \mathbb{R}^B (m \in \{v,a\})$ measures the similarity between $k$-th class and $B$ samples, and thus can be regarded as the category representation of $k$-th class. To preserve the semantic information both contained in video and audio category representation, a common autoencoder with tied weights is used to reconstruct the category representations:
\begin{equation}
\label{eq:AE_V}
\bm{U}^m = E(\bm{P}_m),
\bm{P}^{\prime}_m = D(\bm{U}_m),m \in \{v,a\},
\end{equation}
where $\bm{U}^m$ denotes the hidden representation for corresponding modality, and $E(\cdot)$ and $D(\cdot)$ denote the encoder and decoder, respectively. The encoder maps the category representation to a lower-dimensional representation and the decoder reconstructs back the original input. The original representation and reconstructed representations of same class from one modality can be regarded as positive pairs. The original representation and reconstructed representations of different class fro both two modalities can be regarded as negative pairs. The visual intra-modal contrastive loss is defined as:
% \begin{equation}
% \label{eq:reconstruction loss v}
% l_{rv} = \frac{1}{K} \sum_{k=1}^K \left\| \bm{p}_k^v - \bm{p}_k^{\prime v} \right\|_2,
% \end{equation}
% \begin{equation}
% \label{eq:reconstruction loss a}
% l_{ra} = \frac{1}{K} \sum_{k=1}^K \left\| \bm{p}_k^a - \bm{p}_i^{\prime a} \right\|_2.
% \end{equation}
\begin{equation}
\label{eq:reconstruction loss v}
l_{rv}\!=\!-\frac{1}{K}\!\sum_{i=1}^K log
\frac{exp( S(\bm{p}_i^{v},\bm{p}_i^{\prime v}))}
{\sum_{j=1}^K [exp( S(\bm{p}_i^{v},\bm{p}_j^{\prime v}))
+ exp( S(\bm{p}_i^{v},\bm{p}_j^{\prime a}))]}.
\end{equation}
The audio intra-modal contrastive loss $l_{ra}$ has the same way but uses audio modality to contrast.
On the one hand, the autoencoder can embed category representations from two modalities more closely in the common space. On the other hand, autoencoder can naturally contain the effective feature information and filter the noisy part. To improve the relevance between modalities at the category level, an inter-modal contrastive loss is proposed, which considers the inter-modal reconstructed representations of same class as positive pairs. The inter-modal contrastive objective also comprises two terms. The video term is defined as:
\begin{equation}
\label{eq:category contrastive loss video}
l_{cv}\!=\!-\frac{1}{K}\!\sum_{i=1}^K log
\frac{exp( S(\bm{p}_i^{\prime v},\bm{p}_i^{\prime a}))}
{\sum_{j=1}^K [exp( S(\bm{p}_i^{\prime v},\bm{p}_j^{\prime v}))
+ exp( S(\bm{p}_i^{\prime v},\bm{p}_j^{\prime a}))]}.
\end{equation}
The audio term $l_{ca}$ is analogous, but uses audio modality to contrast. We thus formulate the category-level contrastive loss as the combination of intra-modal and inter-modal contrastive loss:
\begin{equation}
\label{eq:category contrastive loss}
\mathcal{L}_{cat} = \frac{1}{4} (l_{cv} + l_{ca} + l_{rv} + l_{ra}).
\end{equation}
\subsection{Hybrid Supervised Training Phase}
The final noise-tolerant contrastive learning loss is formulated as:
\begin{equation}
\label{eq:contrastive loss}
\mathcal{L}_{c} = \mathcal{L}_{ins} + \mathcal{L}_{cat}.
\end{equation}
For each epoch, after the noise-tolerant contrastive learning update, we aim to obtain corrective labels from the robust features. To this end, we apply the classical K-Nearest Neighbors approach, computing the nearest neighbors by a distance matrix and filter suspicious samples. It proposes each label by taking a majority voting of similar samples:
\begin{equation}
\label{eq:knn}
\hat{y}_i = \underset{y}{\mathrm{argmax}} \, \sum_{i=1}^N1\cdot\left( y = y_i, \left( \bm{v_i},\bm{a_i} \right) \in N_k(\bm{v},\bm{a}) \right),
\end{equation}
where $\hat{y}_i$ is the corrected label for $i$-th sample, and $N_k(\bm{v},\bm{a})$ is distance matrix. Since $K$-NN may miscalculate some hard samples as noises, we retain the original labels as a portion of supervision, and the hybrid supervised objective loss function is:
\begin{equation}
\label{eq:total loss}
\mathcal{L}_{s}= (1- \gamma) \mathcal{L}\left( \bm{\theta},\bm{X}, Y \right) + \gamma \mathcal{L} ( \bm{\theta},\bm{X} , \hat{Y} ),
\end{equation}
where $\mathcal{L}$ is cross entropy loss as shown in \cref{eq:empirical loss}, $\hat{Y}$ denotes the corrected labels in dataset $D$, and $\gamma \in [0,1]$ is the weight factor that controls the balance between two terms. For initial convergence of the algorithm, we set $\gamma \ = 0 $ to warm up the model for a few epochs. Then the noise-tolerant contrastive training phase and hybrid supervised training phase proceed iteratively. The noise-tolerant contrastive training phase first trains the encoders by using the combination of ${L}_{ins}$ and ${L}_{cat}$. We update the encoder parameters and yield corrected labels by calculating similarity among the robust features. These corrected labels are than used as complementary supervision to train the encoders and classifier. The above procedure proceeds in an iterative manner until converged. The full algorithm is outlined in Algorithm \ref{alg:alg1}.
\begin{algorithm}
\SetKwInOut{Input}{Input}
\Input{Network parameter $\bm{\theta}$, prototypes $\bm{C}$ and $\bm{C^{\prime}}$, training dataset $( \bm{X},Y)$, estimated probability threshold ${\epsilon}_a$ and ${\epsilon}_v$, temperature parameter $\tau_1$ and $\tau_2$, weight factor $\gamma$.}
$\bm{\theta} = WarmUp(\bm{\theta},\bm{X},Y)$\;
\While{e $<$ MaxEpoch}{
Generate the estimated probabilities for each sample
$\{\omega_i^v,\omega_i^a \}_{i=1}^N = Estimate(\bm{\theta},\bm{X},Y,{\epsilon}_v,{\epsilon}_a) $\;
\tcp*[h]{contrastive training phase}
\Repeat{all samples selected}{
Sample a mini-batch $\{ (\bm{v_i},\bm{a_i},y_i) \}_{i=1}^B$\;
Normalize the instance prototypes $\bm{C} = \{\bm{c}_1,...,\bm{c}_K\}$\;
Compute $\mathcal{L}_{ins}$ through \Crefrange{eq:prob visual modality}{eq:instance-level contrastive loss}\;
Normalize the category prototypes $\bm{C^{\prime}} = \{\bm{c}^{\prime}_1,...,\bm{c}^{\prime}_K\}$\;
Compute $\mathcal{L}_{cat}$ through \Crefrange{eq:AE_V}{eq:category contrastive loss}\;
Compute contrastive loss $\mathcal{L}_{c}$ by Eq. (\ref{eq:contrastive loss})\;
Update $\bm{\theta},\bm{C},\bm{C^{\prime}}$ by minimizing $\mathcal{L}_{c}$ with Adam\;
}
\tcp*[h]{rectify labels}
Get the corrected labels $\hat{Y}$ according to Eq. (\ref{eq:knn})\;
\tcp*[h]{supervised training phase}
\Repeat{all samples selected}{
Sample a mini-batch $\{ (\bm{v_i},\bm{a_i},y_i,\hat{y}_i) \}_{i=1}^B$\;
Compute supervised loss $\mathcal{L}_{s}$ by Eq. (\ref{eq:total loss})\;
Update $\bm{\theta}$ by minimizing $\mathcal{L}_{s}$ with Adam\;
}
}
\caption{Iterative Training}
\label{alg:alg1}
\end{algorithm}
\section{Experiments}
To evaluate our proposed method compared with the state-of-the-art methods, we have conducted the proposed method on audio-visual action recognition task with a wide range of noise levels for both noisy labels and noisy correspondence.
\subsection{Noisy Correspondence in Real-World Dataset}
Compared to noisy labels, noisy correspondence problem is more complex and irregular, making it difficult to be simulated by the synthetic construction. To this end, we investigate the real-world audio-visual dataset to conduct experiments under real noisy correspondence. Kinetics \cite{kay2017kinetics} is a large-scale audio-visual video dataset for action recognition, which has 240K training videos and 20K validation videos of 400 classes. As the dataset is harvested from YouTube, the videos inevitably contain noisy correspondence between the two modalities. To quantitatively explore the influence of noisy correspondence problem, we relabel the Kinetics dataset to annotate whether it contains noisy correspondence for each video in validation set. As the training set and validation set follow same distribution, we can consider the statistics as the estimation of noisy correspondence for each class. Figure \ref{fig:statistics} shows the statistics of noisy correspondence in validation set of Kinetics. The classes with high-level noise are usually related to sports(\eg, parkour, windsurfing and snowboarding), which may dub videos with completely unrelated sound tracks. Contrarily, the classes with low-level noise are usually related to sound(\eg, singing and playing harp), which are manifested visually and aurally. To quantitatively study the influence of real-world noisy correspondence, we create 4 different controlled noise-level mini-dataset: \{10\%,20\%,30\%,40\%\}. Each mini-dataset consists of the 50 categories within the specific noise levels; for each category, we randomly sample 400 videos from the training split and 50 videos from the validation split. The full list of each mini-Kinetics is given in appendix.
\begin{figure*}[!t]
\centering
\includegraphics[width=1\textwidth]{statistics}
\caption{Histogram of instance counts over the entire 19,404 validation dataset, sorted by the percentage of noisy correspondence among each class. Here we show the top-20 classes with noisy labels and top-20 clean classes.} \label{fig:statistics}
\end{figure*}
\subsection{Experimental Setup}
\textbf{Datasets.} In this paper, we use two video action classification datasets to evaluate our method. The first one is UCF101 \cite{soomro2012ucf101}. It contains 7K audio-visual videos from 51 classes and the mean video length is about 7 seconds. We adopt split-1 of the 3 official train/test splits to conduct our study. The second main dataset is Kinetics, we use the 4 subsets mentioned above to explore the influence of noisy correspondence.
\textbf{Backbone architecture.} We employ the R(2+1)D-18 \cite{tran2018closer} as our visual backbone and ResNet-22 \cite{kong2020panns} as our audio backbone. For Kinetics, the visual backbone is pretrained on IG65M \cite{ig65m}. For UCF101, the visual backbone is pretrained on Kinetics. And the audio backbone is pretrained on AudioSet \cite{gemmeke2017audio} for both datasets. For fusion, we use the fusion manner as \cite{wang2020makes}. A two-FC-layer network is used to concatenate features from two modalities, and each FC layer follows a ReLU layer withou the last layer.
\textbf{Input preprocessing.}
For visual modality, we use the $8\times112\times112$ clips as input, processed by random crop and center crop for training and test, respectively. For audio modality, we sample 2 seconds audio track and use log-Mel spectrograms as input, the audio preprocessing is followed as \cite{kong2020panns}. Audio and visual are temporally aligned.
\textbf{Implementation Details.} We employ Adam \cite{kingma2014adam} optimizer with default parameters to train our model. The learning rate is initialized with $5\times 10^{-5}$ and the batch size is set as 64. The total training processes contain 60 epochs. The temperature parameters $\tau_1$ and $\tau_2$ are set as 1 and 0.1, respectively. The numbers of similar samples are set as 16 and 32 for UCF101 and Kinetics, respectively. The weight factor $\gamma$ is initialized with 0.6 and iteratively increase to 0.8 after 20 epochs.
\textbf{Evaluation Metrics.} For evaluation, we take the clip-level and video-level accuracy as the measurement. We uniformly sample 10 clips for each testing video and average the predictions as the video-level prediction. In the experiments, we report clip-level top-1, clip-level top-5, and video-level top-1 accuracy for a comprehensive evaluation.
\subsection{Comparisons with State of the Arts}
We compare our method with multiple state-of-the-art methods, including four audio-visual learning methods(\ie, G-Blend\cite{wang2020makes}, XDC\cite{XDC}, AV-R\cite{tian2021can}, AdaMML\cite{panda2021adamml}) and one multi-modal label noise method($i.e.$, MRL\cite{hu2021learning}). Note that for XDC, we also use $K$-NN to rectify labels in order to make it suitable for supervised learning(denoted by XDC*). All methods use the same backbone architecture as ours. To comprehensively evaluate the robustness of the methods, for noisy labels, we set the noise to be symmetric(\ie, noise ratio 20\%, 40\%, 60\%, 80\%) and asymmetric(\ie, noise ratio 20\%, 40\%). For noisy correspondence, we use the mini-Kinetics with different ratios of real noise, \ie, 10\%, 20\%, 30\% and 40\%. Note that the two types of noise exist simultaneously.
\input{table1}
\input{table2}
Table1 and Table2 show the results for symmetric and asymmetric noisy labels, respectively. As shown in these tables, our method is superior to the state-of-the-art methods for all cases. From the experimental results, we can see the following observations:
\begin{itemize}
\item The existence of noisy labels remarkably impact the performance of audio-visual action recognition methods. As the ratio of label noise increases, the performance of these methods decreases rapidly.
\item For the audio-visual action recognition task, it's more challenging to learn under asymmetric noisy labels. But asymmetric label noise has less effect on clip-level top-5 accuracy.
\item The existence of noisy correspondence affects anti-interference performance of audio-visual action recognition methods to noisy labels. On the one hand, it will increase the difficulty of the task, which makes methods more easier to overfit. On the other hand, some existing audio-visual learning methods enforce the agreement between multiple modalities(\eg, XDC* and MRL), which may lead to sub-optimal results.
\item Our approach shows better performance and stability than other audio-visual action recognition methods, especially in the case of high-level noise.
\end{itemize}
\subsection{Progressive Comparison}
Figure \ref{fig:test_train_acc} plots the models' clip-level top-1 accuracy on training data with noisy labels and the corresponding test accuracy on clean test data of UCF101 as training proceeds. We show some representative training processes using 20\% asymmetric label noise, 40\% asymmetric label noise, 60\% symmetric label noise and 80\% symmetric label noise. It can be seen from the Figure \ref{fig:test_train_acc} that during the beginning of training, all methods quickly learn clean data and achieve certain accuracy. However, with the training processing, most met hods progressively overfit the training data and thus decrease the performance in the clean test data. As the noise level increases, most existing methods are more easily to be overfitting in test data. Some methods(\eg, XDC* and MRL) have certain noise-tolerant ability to noisy labels since they designed some anti-interference components such as robust loss function, but they can not deal with both symmetric and asymmetric noisy labels. Compared to the existing state-of-the-art methods, our method does not overfit the train data and has better noise-tolerant ability in all noise cases.
\begin{figure*}[!t]
\centering
\includegraphics[width=1\textwidth]{test_train_acc.pdf}
\caption{Clip-level top-1 accuracy \vs epoch on noisy training dataset and clean test dataset of UCF101. The label noise are set as: 20\% asymmetric, 40\% asymmetric, 60\% symmetric and 80\% symmetric. } \label{fig:test_train_acc}
\end{figure*}
\subsection{Ablation Study}
To evaluate the contribution of the proposed components in our method,\eg instance-level and category-level contrastive loss, we carry out the ablation study on the mini-Kinetics(with noisy correspondence ratio of 40\%) and UCF101, both datasets have 60\% symmetric label noise.
To sufficiently validate the effectiveness of proposed components, we compare our method with three counterparts: 1) None: No noise-tolerant contrastive learning scheme, the corrected labels produced by the features from supervised learning. 2): L-ins: only use instance-level noise-tolerant contrastive loss to update the network. 3): L-cat: only use category-level noise-tolerant contrastive loss to update the network. As shown in Table 3, all these proposed components are important to improve the noise-tolerant ability of the audio-visual network, and the instance-level contrastive loss has the most contribution.
\input{table3}
\subsection{Parameter Analysis}
The weight factor $\gamma$ plays a significant role in the hybrid supervised training phase, which determines the network will focus more on the original labels $Y$ or the corrected labels $\hat{Y}$. For the two extreme cases, $\gamma = 0$ and $\gamma = 1$ denote that the network is trained by only using original labels and corrected labels only, respectively. To evaluate the impact of the trade-off hyper-parameter $\gamma$, we conduct experiments to study the influence of different $\gamma$ ranging from 0.0 to 1.0 under different symmetric label noise rates of 0.2, 0.4, 0.6, 0.8. Note that we set $\gamma$ as constant here for quantitative study.
We plot the clip-level top-1 accuracy versus $\gamma$ on the validation sets of UCF101 and min-Kinetics with 40\% noisy correspondence in Figure \ref{fig_para}. From the figure, we can see that training using only the original labels leads to poor performance for all noise rates. But it's still superior to the some comparing methods, which indicates that the learned parameters in contrastive training phase enhance the noise-tolerant ability in supervised training phase.
Although the corrected labels are more approximate to ground-truth labels, training using only the corrected labels may lead self-convergence problem and thus yield sub-optimal results. Furthermore, the sensitivity of $\gamma$ is also influenced by the noise rates and the difficulty level of datasets (\eg, the number of classes, the existence of noisy correspondence). Specifically, the $\gamma$ is less sensitive to the low-level noise rates and easy-to-learn datasets. For most cases, our method can achieve good performance in a relatively larger range(\ie, 0.4 $\sim$ 0.8). We can also see that when training with only the original labels,
\begin{figure}[H]
\centering
\subfloat[]{\includegraphics[width=0.4\textwidth]{para_UCF101.pdf}%
\label{para_ucf101}}
\hfil
\subfloat[]{\includegraphics[width=0.4\textwidth]{para_K404.pdf}%
\label{para_k404}}
\caption{Clip-level top-1 accuracy $vs.$ different values of $\gamma$ on the validation set of the (a) UCF101 and (b) mini-Kinetics with 40\% noisy correspondence datasets, respectively. The symmetric label noise rates are 0.2, 0.4, 0.6 and 0.8.}
\label{fig_para}
\end{figure}
\begin{figure*}[!t]
\centering
\includegraphics[width=1\textwidth]{visual.pdf}
\caption{Selection results of our cross-modal noise estimation component for correct and noisy correspondence samples.} \label{fig:visual}
\end{figure*}
\subsection{Visualization on the Cross-Modal Noise Estimation}
To further investigate the proposed cross-modal noise estimation on real-world noisy correspondence data, we plot the per-sample weight distribution of clean and noisy correspondence samples on validation set. Note that the networks trained with 40\% symmetric noisy labels on both two mini-Kinetics datasets. As shown in Fig \ref{fig_weight}, our method assigns larger weight to most clean samples than the noisy samples, which implies that our method can distinguish clean and noisy correspondence samples while trained under noisy labels.
\begin{figure}[H]
\centering
\subfloat[]{\includegraphics[width=0.25\textwidth]{0.2_weight.pdf}%
\label{0.2_weight}}\hspace{-4mm}
\hfil
\subfloat[]{\includegraphics[width=0.25\textwidth]{0.4_weight.pdf}%
\label{0.4_weight}}
\caption{Per-sample weight distribution on validation samples. (a) mini-Kinetics with 20\% noisy correspondence. (b) mini-Kinetics with 40\% noisy correspondence.}
\label{fig_weight}
\end{figure}
We also demonstrate the similar samples selected by our cross-modal noise estimation component. specifically, we visualize the selection results of correct samples and noisy correspondence samples under 20\% symmetric noisy labels and 10\% real noisy correspondence in Figure \ref{fig:visual}. It can be seen from Figure \ref{fig:visual} that our method can clearly separate the correct and noisy samples. For correct samples, we use the audio modality to find similar samples that also have high-level similarity to visual modality. For noisy samples, the irrelevant information makes the corresponding visual modality has a low-level similarity. In the classes "answering questions" and "playing monopoly", the audio information of noisy samples interspersed with additional background music. In the class "clay pottery making", the human voices masks the clay making action. In the class "riding camel", the main audio information comes from the noisy road.
\section{Conclusion}
In this paper, we propose a noise-tolerant learning framework for audio-visual action recognition with both noisy labels and noisy correspondence, where a noise-tolerant contrastive training phase is performed prior to the conventional supervised training. The proposed noise-tolerant contrastive training phase aims to learn robust model parameters that unaffected by the noisy labels. Since the existence of noisy correspondence, we also propose a cross-modal noise estimation component to adjust the consistency between different modalities. In the hybrid supervised training phase, we apply $K$-NN approach to obtain corrective labels from the robust features which further used as complementary supervision for the supervised training. The noise-tolerant contrastive training phase and hybrid supervised training phase trained iteratively to find anti-interference model parameters to both noisy labels and noisy correspondence. In addition, we investigate the noisy correspondence in real-world audio-visual dataset and conduct comprehensive experiments with synthetic and real noise data. The results verify the advantageous performance
of our method compared to state-of-the-art methods. To the best of our knowledge, this paper could be the first attempt to reveal the influence that both noisy labels and noisy correspondence exist simultaneously in an audio-visual action recognition task.
\bibliographystyle{plain}
\bibliography{main}
{\appendix[mini-Kinetics]
\begin{itemize}
\item The 50 classes of mini-Kinetics with about \textbf{10\% noisy correspondence} are: air drumming, arranging flowers, balloon blowing, bee keeping, bench pressing, blowing out candles, bobsledding, bowling, brushing teeth, cartwheeling, catching or throwing baseball, clapping, cooking sausages, crying, dancing gangnam style, dancing macarena, dodgeball, drinking beer, feeding fish, feeding goats, giving or receiving award, high jump, holding snake, hoverboarding, juggling fire, opening present, peeling potatoes, petting animal(not cat), petting cat, playing accordion, playing cards, playing cello, playing didgeridoo, playing flute, playing guitar, playing harmonica, playing harp, presenting weather forecast, pushing cart, reading book, salsa dancing, sharpening knives, shining shoes, shooting basketball, singing, spraying, testifying, tobogganing, training dog, trapezing.
\item The 50 classes of mini-Kinetics with about \textbf{20\% noisy correspondence} are: applying cream, archery, bandaging, barbequing, blowing leaves, bouncing on trampoline, brushing hair, catching or throwing softball, cutting nails, deadlifting, drop kicking, flipping pancake, flying kite, golf chipping, headbanging, high kick, hockey stop, jumping into pool, jumpstyle dancing, kicking field goal, knitting, making snowman, making tea, massaging persons head, mopping floor, passing American football(in game), passing American football(not in game), playing cymbals, playing poker, playing volleyball, punching bag, pushing wheelchair, reading newspaper, riding mechanical bull, riding or walking with horse, rock climbing, shearing sheep, shoveling snow, ski jumping, taking a shower, tasting beer, texting, throwing ball, triple jump, walking the dog, washing feet, water skiing, wrestling, yawning, zumba.
\item The 50 classes of mini-Kinetics with about \textbf{30\% noisy correspondence} are: assembling computer, baking cookies, bending metal, blasting sand, bungee jumping, carrying baby, changing wheel, chopping wood, cleaning gutters, cleaning pool, cleaning windows, cooking chicken, cooking egg, cooking on campfire, cutting pineapple, disc golfing, dribbling basketball, folding clothes, getting a tattoo, golf putting, gymnastics tumbling, ice fishing, jetskiing, making a cake, picking fruit, planting trees, plastering, playing paintball, pull ups, riding a bike, roller skating, scrambling eggs, setting table, shaving head, shuffling cards, sled dog racing, smoking, smoking hookah, snorkeling, spinning poi, squat, stomping grapes, sword fighting, tai chi, tying knot(not on a tie), unboxing, unloading truck, washing hair, weaving basket, writing
\item The 50 classes of mini-Kinetics with about \textbf{40\% noisy correspondence} are: bartending , braiding hair , building shed , canoeing or kayaking , carving pumpkin , cleaning floor , cleaning shoes , contact juggling , curling hair , digging , doing nails , drawing , driving car , dunking basketball , dying hair , exercising arm , folding napkins , folding paper , grooming dog , javelin throw , juggling soccer ball , kissing , kitesurfing , making pizza , massaging legs , motorcycling , paragliding , parkour , playing cricket , playing ice hockey , push up , riding mountain bike , sailing , scuba diving , shooting goal(soccer) , skateboarding , skiing(not slalom or crosscountry) , skiing crosscountry , skiing slalom , skydiving , slacklining , snowboarding , snowkiting , surfing water , trimming or shaving beard , trimming trees , using computer , using remote controller(not gaming) , windsurfing , yoga
\end{itemize}}
\end{document}
\begin{table*}[htbp]
\centering
\caption{Comparison with state-of-the-art methods on UCF101 and mini-Kinetics with symmetric label noise.\label{tab:table1}}
\resizebox{\textwidth}{!}{
\begin{tabular}{c|c|ccc|ccc|ccc|ccc|ccc}
\toprule
& & \multicolumn{3}{c|}{\multirow{2}[2]{*}{UCF101}} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c}{mini-Kinetics} \\
& & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{10\% noisy correspondnece} & \multicolumn{3}{c|}{20\% noisy correspondnece} & \multicolumn{3}{c|}{30\% noisy correspondnece} & \multicolumn{3}{c}{40\% noisy correspondnece} \\
\midrule
Noisy Label & Methods & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 \\
\midrule
& G-Blend & 76.90 & 91.46 & 81.52 & 57.24 & 81.99 & 63.27 & 58.66 & 82.63 & 63.85 & 56.01 & 81.65 & 61.27 & 55.21 & 80.49 & 61.27 \\
& MRL & 82.85 & 95.37 & 85.21 & 62.17 & 86.48 & 68.30 & 60.71 & 86.75 & 65.49 & 57.22 & 83.00 & 64.05 & 57.80 & 82.02 & 64.89 \\
& XDC* & 82.66 & 95.06 & 85.91 & 63.11 & 86.84 & 70.18 & 61.21 & 86.58 & 66.63 & 59.68 & 85.54 & 65.17 & 57.82 & 82.28 & 65.59 \\
0.20 & AV-R & 81.33 & 94.44 & 85.12 & 63.61 & 87.11 & 70.30 & 60.06 & 84.29 & 65.88 & 59.92 & 85.69 & 64.56 & 59.39 & 83.49 & 66.42 \\
& AdaMML & 78.34 & 92.28 & 83.77 & 61.16 & 84.47 & 67.59 & 59.03 & 83.14 & 64.41 & 57.59 & 83.00 & 63.27 & 57.22 & 82.12 & 63.37 \\
& Ours & \textbf{83.18} & \textbf{95.47} & \textbf{87.69} & \textbf{65.60} & \textbf{88.64} & \textbf{71.81} & \textbf{63.45} & \textbf{87.31} & \textbf{68.22} & \textbf{63.28} & \textbf{88.19} & \textbf{68.06} & \textbf{62.79} & \textbf{86.22} & \textbf{67.33} \\
\midrule
& G-Blend & 67.44 & 84.57 & 71.66 & 50.36 & 72.80 & 54.65 & 48.79 & 70.96 & 52.55 & 48.03 & 69.67 & 51.43 & 45.70 & 70.83 & 49.81 \\
& MRL & 81.53 & 94.03 & 83.75 & 56.97 & 79.90 & 62.43 & 56.59 & 80.21 & 62.01 & 55.43 & 79.82 & 60.63 & 57.31 & 81.28 & 61.53 \\
& XDC* & 79.78 & 94.75 & 84.28 & 57.57 & 81.85 & 62.30 & 56.57 & 80.00 & 62.46 & 55.06 & 79.73 & 59.48 & 53.76 & 80.17 & 60.79 \\
0.40 & AV-R & 72.48 & 90.59 & 76.58 & 54.70 & 79.46 & 59.82 & 51.58 & 77.65 & 56.13 & 51.15 & 77.56 & 57.16 & 50.06 & 78.75 & 56.09 \\
& AdaMML & 72.43 & 89.92 & 76.59 & 53.63 & 78.04 & 59.29 & 53.75 & 80.41 & 59.35 & 52.17 & 78.36 & 57.91 & 51.06 & 79.46 & 56.64 \\
& Ours & \textbf{82.02} & \textbf{95.32} & \textbf{85.97} & \textbf{62.86} & \textbf{84.53} & \textbf{69.62} & \textbf{60.25} & \textbf{84.61} & \textbf{65.34} & \textbf{58.57} & \textbf{84.31} & \textbf{65.73} & \textbf{58.23} & \textbf{84.17} & \textbf{64.70} \\
\midrule
& G-Blend & 55.56 & 76.39 & 63.12 & 38.79 & 63.45 & 46.02 & 38.75 & 62.94 & 45.32 & 37.84 & 62.37 & 44.77 & 36.68 & 60.73 & 42.33 \\
& MRL & 69.60 & 85.39 & 74.17 & 52.90 & 75.48 & 59.73 & 51.92 & 73.61 & 58.73 & 52.03 & 77.28 & 59.60 & 50.21 & 74.86 & 52.25 \\
& XDC* & 77.57 & 92.54 & 83.59 & 54.21 & 78.06 & 60.75 & 53.52 & 78.26 & 60.97 & 51.19 & 76.65 & 58.45 & 49.04 & 74.13 & 51.99 \\
0.60 & AV-R & 63.53 & 83.08 & 67.08 & 45.59 & 68.38 & 51.25 & 45.69 & 67.76 & 48.89 & 42.85 & 68.50 & 49.44 & 41.56 & 67.25 & 46.51 \\
& AdaMML & 60.75 & 80.71 & 65.64 & 44.00 & 66.66 & 50.04 & 43.91 & 66.67 & 48.80 & 43.87 & 66.95 & 49.10 & 43.37 & 66.91 & 47.70 \\
& Ours & \textbf{80.04} & \textbf{93.98} & \textbf{84.05} & \textbf{58.59} & \textbf{78.79} & \textbf{63.43} & \textbf{60.38} & \textbf{80.48} & \textbf{63.33} & \textbf{55.83} & \textbf{81.91} & \textbf{62.53} & \textbf{54.56} & \textbf{79.86} & \textbf{61.15} \\
\midrule
& G-Blend & 35.94 & 61.27 & 40.46 & 27.39 & 50.73 & 32.65 & 24.62 & 46.07 & 29.89 & 25.76 & 48.77 & 31.41 & 25.72 & 49.63 & 30.50 \\
& MRL & 43.52 & 62.60 & 46.48 & 39.60 & 61.73 & 45.28 & 37.68 & 61.10 & 42.95 & 36.69 & 59.53 & 42.90 & 36.16 & 58.57 & 41.46 \\
& XDC* & 51.95 & 76.34 & 59.11 & 41.55 & 65.49 & 47.46 & 38.53 & 61.78 & 42.98 & 36.20 & 58.78 & 41.77 & 34.16 & 57.26 & 40.56 \\
0.80 & AV-R & 37.96 & 61.16 & 42.16 & 32.38 & 53.57 & 37.28 & 31.66 & 52.67 & 36.09 & 29.45 & 52.64 & 33.68 & 29.15 & 52.74 & 33.73 \\
& AdaMML & 36.68 & 60.39 & 42.33 & 29.68 & 51.53 & 34.85 & 30.48 & 50.72 & 34.95 & 28.06 & 50.70 & 32.44 & 28.07 & 51.78 & 32.55 \\
& Ours & \textbf{59.10} & \textbf{81.12} & \textbf{61.34} & \textbf{45.29} & \textbf{67.60} & \textbf{49.55} & \textbf{44.45} & \textbf{66.76} & \textbf{47.88} & \textbf{43.60} & \textbf{65.97} & \textbf{47.76} & \textbf{43.16} & \textbf{66.22} & \textbf{49.02} \\
\bottomrule
\end{tabular}%
}%
\label{tab:addlabel}%
\end{table*}%
% Table generated by Excel2LaTeX from sheet 'Sheet1'
\begin{table*}[htbp]
\centering
\caption{Comparison with state-of-the-art methods on UCF101 and mini-Kinetics with asymmetric label noise}
\resizebox{\textwidth}{!}{
\begin{tabular}{c|c|ccc|ccc|ccc|ccc|ccc}
\toprule
& & \multicolumn{3}{c|}{\multirow{2}[2]{*}{UCF101}} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c|}{mini-Kinetics} & \multicolumn{3}{c}{mini-Kinetics} \\
& & \multicolumn{3}{c|}{} & \multicolumn{3}{c|}{10\% noisy correspondnece} & \multicolumn{3}{c|}{20\% noisy correspondnece} & \multicolumn{3}{c|}{30\% noisy correspondnece} & \multicolumn{3}{c}{40\% noisy correspondnece} \\
\midrule
Noisy Labels & Methods & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 & C@1 & C@5 & V@1 \\
\midrule
& G-Blend & 73.56 & 92.75 & 77.24 & 54.68 & 80.58 & 59.44 & 53.30 & 79.52 & 58.80 & 53.55 & 78.53 & 57.88 & 52.73 & 77.34 & 57.41 \\
& MRL & 82.66 & 94.96 & 85.17 & 61.57 & 81.92 & 66.56 & 59.69 & 80.86 & 66.02 & 59.62 & 81.68 & 65.83 & 57.43 & 80.60 & 63.93 \\
& XDC* & 80.20 & 94.24 & 83.84 & 58.32 & 80.94 & 64.17 & 56.34 & 79.74 & 63.03 & 55.60 & 79.59 & 61.08 & 53.01 & 78.66 & 57.31 \\
0.20 & AV-R & 77.73 & 92.54 & 81.36 & 57.39 & 80.63 & 62.76 & 57.05 & 79.86 & 63.66 & 56.25 & 80.43 & 62.57 & 55.19 & 79.10 & 60.73 \\
& AdaMML & 77.37 & 94.03 & 81.72 & 57.17 & 80.84 & 62.84 & 56.38 & 80.08 & 63.32 & 54.60 & 79.47 & 60.37 & 54.12 & 78.50 & 58.98 \\
& Ours & \textbf{83.28} & \textbf{95.27} & \textbf{86.79} & \textbf{65.62} & \textbf{87.24} & \textbf{71.06} & \textbf{67.42} & \textbf{88.27} & \textbf{71.26} & \textbf{65.36} & \textbf{86.71} & \textbf{67.41} & \textbf{62.14} & \textbf{85.48} & \textbf{66.88} \\
\midrule
& G-Blend & 51.85 & 91.10 & 55.96 & 35.97 & 69.66 & 40.54 & 35.52 & 68.54 & 40.97 & 36.96 & 69.33 & 41.62 & 34.85 & 68.13 & 39.91 \\
& MRL & 63.32 & 90.90 & 67.08 & 44.80 & 73.39 & 51.25 & 43.86 & 74.02 & 50.98 & 42.04 & 72.72 & 48.91 & 40.99 & 72.27 & 47.46 \\
& XDC* & 60.80 & 90.33 & 64.04 & 43.39 & 72.41 & 49.65 & 40.29 & 71.40 & 47.67 & 39.61 & 70.94 & 47.97 & 38.08 & 70.33 & 46.50 \\
0.40 & AV-R & 56.74 & 90.23 & 61.52 & 41.46 & 72.05 & 49.59 & 41.56 & 71.54 & 48.51 & 40.80 & 72.34 & 48.80 & 40.86 & 72.15 & 48.60 \\
& AdaMML & 51.85 & 89.92 & 55.93 & 39.66 & 70.17 & 45.21 & 38.81 & 70.46 & 44.81 & 38.70 & 70.70 & 45.10 & 37.13 & 69.82 & 44.43 \\
& Ours & \textbf{71.45} & \textbf{92.13} & \textbf{74.39} & \textbf{57.43} & \textbf{81.13} & \textbf{62.43} & \textbf{56.06} & \textbf{80.28} & \textbf{62.27} & \textbf{55.13} & \textbf{79.50} & \textbf{60.65} & \textbf{53.72} & \textbf{78.04} & \textbf{57.61} \\
\bottomrule
\end{tabular}%
}%
\label{tab:addlabel}%
\end{table*}%
% Table generated by Excel2LaTeX from sheet 'Sheet1'
\begin{table}[H]
\caption{Ablation study on the proposed components. \label{tab:table1}}
\centering
\scalebox{1}{
\begin{tabular}{c|l|r|r|r}
\toprule
\multicolumn{1}{l|}{Dataset} & Method & \multicolumn{1}{l|}{C@1} & \multicolumn{1}{l|}{C@5} & \multicolumn{1}{l}{V@1} \\
\midrule
\multirow{4}[2]{*}{UCF101} & None & 65.62 & 80.91 & 69.39 \\
& L-ins & 79.22 & 93.25 & 82.97 \\
& L-cat & 68.21 & 83.93 & 73.53 \\
& Full & \textbf{80.04} & \textbf{93.98} & \textbf{84.05} \\
\midrule
\multirow{4}[2]{*}{mini-Kinetics} & None & 42.91 & 73.07 & 49.30 \\
& L-ins & 52.31 & 79.76 & 58.09 \\
& L-cat & 48.08 & 76.73 & 54.48 \\
& Full & \textbf{54.56} & \textbf{79.83} & \textbf{61.15} \\
\bottomrule
\end{tabular}}%
\end{table}
\begin{thebibliography}{10}
\bibitem{abu2016youtube}
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici,
Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.
\newblock Youtube-8m: A large-scale video classification benchmark.
\newblock {\em arXiv preprint arXiv:1609.08675}, 2016.
\bibitem{XDC}
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem,
and Du~Tran.
\newblock Self-supervised learning by cross-modal audio-video clustering.
\newblock {\em Advances in Neural Information Processing Systems},
33:9758--9770, 2020.
\bibitem{arandjelovic2018objects}
Relja Arandjelovic and Andrew Zisserman.
\newblock Objects that sound.
\newblock In {\em Proceedings of the European conference on computer vision
(ECCV)}, pages 435--451, 2018.
\bibitem{arpit2017closer}
Devansh Arpit, Stanis{\l}aw Jastrz{\k{e}}bski, Nicolas Ballas, David Krueger,
Emmanuel Bengio, Maxinder~S Kanwal, Tegan Maharaj, Asja Fischer, Aaron
Courville, Yoshua Bengio, et~al.
\newblock A closer look at memorization in deep networks.
\newblock In {\em International conference on machine learning}, pages
233--242. PMLR, 2017.
\bibitem{asano2019self}
Yuki~Markus Asano, Christian Rupprecht, and Andrea Vedaldi.
\newblock Self-labelling via simultaneous clustering and representation
learning.
\newblock {\em arXiv preprint arXiv:1911.05371}, 2019.
\bibitem{deepcluster}
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.
\newblock Deep clustering for unsupervised learning of visual features.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, pages 132--149, 2018.
\bibitem{swAV}
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin.
\newblock Unsupervised learning of visual features by contrasting cluster
assignments.
\newblock {\em arXiv preprint arXiv:2006.09882}, 2020.
\bibitem{chen2020simple}
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
\newblock A simple framework for contrastive learning of visual
representations.
\newblock In {\em International conference on machine learning}, pages
1597--1607. PMLR, 2020.
\bibitem{sinkhorn}
Marco Cuturi.
\newblock Sinkhorn distances: Lightspeed computation of optimal transport.
\newblock {\em Advances in neural information processing systems},
26:2292--2300, 2013.
\bibitem{feichtenhofer2019slowfast}
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.
\newblock Slowfast networks for video recognition.
\newblock In {\em Proceedings of the IEEE/CVF international conference on
computer vision}, pages 6202--6211, 2019.
\bibitem{gemmeke2017audio}
Jort~F Gemmeke, Daniel~PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence,
R~Channing Moore, Manoj Plakal, and Marvin Ritter.
\newblock Audio set: An ontology and human-labeled dataset for audio events.
\newblock In {\em 2017 IEEE international conference on acoustics, speech and
signal processing (ICASSP)}, pages 776--780. IEEE, 2017.
\bibitem{ig65m}
Deepti Ghadiyaram, Du~Tran, and Dhruv Mahajan.
\newblock Large-scale weakly-supervised pre-training for video action
recognition.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 12046--12055, 2019.
\bibitem{han2018co}
Bo~Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and
Masashi Sugiyama.
\newblock Co-teaching: Robust training of deep neural networks with extremely
noisy labels.
\newblock {\em Advances in neural information processing systems}, 31, 2018.
\bibitem{hendrycks2018using}
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel.
\newblock Using trusted data to train deep networks on labels corrupted by
severe noise.
\newblock {\em arXiv preprint arXiv:1802.05300}, 2018.
\bibitem{hu2021learning}
Peng Hu, Xi~Peng, Hongyuan Zhu, Liangli Zhen, and Jie Lin.
\newblock Learning cross-modal retrieval with noisy labels.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5403--5413, 2021.
\bibitem{huang2021learning}
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and
Xi~Peng.
\newblock Learning with noisy correspondence for cross-modal matching.
\newblock {\em Advances in Neural Information Processing Systems}, 34, 2021.
\bibitem{karpathy2014large}
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul
Sukthankar, and Li~Fei-Fei.
\newblock Large-scale video classification with convolutional neural networks.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition}, pages 1725--1732, 2014.
\bibitem{kay2017kinetics}
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra
Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et~al.
\newblock The kinetics human action video dataset.
\newblock {\em arXiv preprint arXiv:1705.06950}, 2017.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{kong2020panns}
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark~D
Plumbley.
\newblock Panns: Large-scale pretrained audio neural networks for audio pattern
recognition.
\newblock {\em IEEE/ACM Transactions on Audio, Speech, and Language
Processing}, 28:2880--2894, 2020.
\bibitem{li2018unified}
Dong Li, Ting Yao, Ling-Yu Duan, Tao Mei, and Yong Rui.
\newblock Unified spatio-temporal attention networks for action recognition in
videos.
\newblock {\em IEEE Transactions on Multimedia}, 21(2):416--428, 2018.
\bibitem{li2020dividemix}
Junnan Li, Richard Socher, and Steven~CH Hoi.
\newblock Dividemix: Learning with noisy labels as semi-supervised learning.
\newblock {\em arXiv preprint arXiv:2002.07394}, 2020.
\bibitem{li2019learning}
Junnan Li, Yongkang Wong, Qi~Zhao, and Mohan~S Kankanhalli.
\newblock Learning to learn from noisy labeled data.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5051--5059, 2019.
\bibitem{li2021contrastive}
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey~Tianyi Zhou, and Xi~Peng.
\newblock Contrastive clustering.
\newblock In {\em 2021 AAAI Conference on Artificial Intelligence (AAAI)},
2021.
\bibitem{liu2018weakly}
Jen-Yu Liu, Yi-Hsuan Yang, and Shyh-Kang Jeng.
\newblock Weakly-supervised visual instrument-playing action detection in
videos.
\newblock {\em IEEE Transactions on Multimedia}, 21(4):887--901, 2018.
\bibitem{morgado2021audio}
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra.
\newblock Audio-visual instance discrimination with cross-modal agreement.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 12475--12486, 2021.
\bibitem{panda2021adamml}
Rameswar Panda, Chun-Fu~Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude
Oliva, and Rogerio Feris.
\newblock Adamml: Adaptive multi-modal learning for efficient video
recognition.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 7576--7585, 2021.
\bibitem{patrini2017making}
Giorgio Patrini, Alessandro Rozza, Aditya Krishna~Menon, Richard Nock, and
Lizhen Qu.
\newblock Making deep neural networks robust to label noise: A loss correction
approach.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 1944--1952, 2017.
\bibitem{petridis2015prediction}
Stavros Petridis and Maja Pantic.
\newblock Prediction-based audiovisual fusion for classification of
non-linguistic vocalisations.
\newblock {\em IEEE Transactions on Affective Computing}, 7(1):45--58, 2015.
\bibitem{qiu2017learning}
Zhaofan Qiu, Ting Yao, and Tao Mei.
\newblock Learning spatio-temporal representation with pseudo-3d residual
networks.
\newblock In {\em proceedings of the IEEE International Conference on Computer
Vision}, pages 5533--5541, 2017.
\bibitem{ren2018learning}
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.
\newblock Learning to reweight examples for robust deep learning.
\newblock In {\em International Conference on Machine Learning}, pages
4334--4343. PMLR, 2018.
\bibitem{rouditchenko2019self}
Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio
Torralba.
\newblock Self-supervised audio-visual co-segmentation.
\newblock In {\em ICASSP 2019-2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP)}, pages 2357--2361. IEEE, 2019.
\bibitem{shu2019meta}
Jun Shu, Qi~Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng.
\newblock Meta-weight-net: Learning an explicit mapping for sample weighting.
\newblock {\em Advances in Neural Information Processing Systems},
32:1919--1930, 2019.
\bibitem{soomro2012ucf101}
Khurram Soomro, Amir~Roshan Zamir, and Mubarak Shah.
\newblock Ucf101: A dataset of 101 human actions classes from videos in the
wild.
\newblock {\em arXiv preprint arXiv:1212.0402}, 2012.
\bibitem{tian2021can}
Yapeng Tian and Chenliang Xu.
\newblock Can audio-visual integration strengthen robustness under multimodal
attacks?
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 5601--5611, 2021.
\bibitem{tian2020contrastive}
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
\newblock Contrastive multiview coding.
\newblock In {\em European conference on computer vision}, pages 776--794.
Springer, 2020.
\bibitem{tran2018closer}
Du~Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar
Paluri.
\newblock A closer look at spatiotemporal convolutions for action recognition.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition}, pages 6450--6459, 2018.
\bibitem{van2018representation}
Aaron Van~den Oord, Yazhe Li, and Oriol Vinyals.
\newblock Representation learning with contrastive predictive coding.
\newblock {\em arXiv e-prints}, pages arXiv--1807, 2018.
\bibitem{wang2020makes}
Weiyao Wang, Du~Tran, and Matt Feiszli.
\newblock What makes training multi-modal classification networks hard?
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 12695--12705, 2020.
\bibitem{wang2016actions}
Xiaolong Wang, Ali Farhadi, and Abhinav Gupta.
\newblock Actions\~{} transformations.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition}, pages 2658--2667, 2016.
\bibitem{wang2018iterative}
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le~Song, and
Shu-Tao Xia.
\newblock Iterative learning with open-set noisy labels.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 8688--8696, 2018.
\bibitem{wu2018unsupervised}
Zhirong Wu, Yuanjun Xiong, Stella~X Yu, and Dahua Lin.
\newblock Unsupervised feature learning via non-parametric instance
discrimination.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 3733--3742, 2018.
\bibitem{xia2019anchor}
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo~Han, Chen Gong, Gang Niu, and
Masashi Sugiyama.
\newblock Are anchor points really indispensable in label-noise learning?
\newblock {\em Advances in Neural Information Processing Systems},
32:6838--6849, 2019.
\bibitem{xie2017rethinking}
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy.
\newblock Rethinking spatiotemporal feature learning for video understanding.
\newblock {\em arXiv preprint arXiv:1712.04851}, 1(2):5, 2017.
\bibitem{yang2021partially}
Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi~Peng.
\newblock Partially view-aligned representation learning with noise-robust
contrastive loss.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 1134--1143, 2021.
\bibitem{yang2021estimating}
Shuo Yang, Erkun Yang, Bo~Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu.
\newblock Estimating instance-dependent label-noise transition matrix using
dnns.
\newblock {\em arXiv preprint arXiv:2105.13001}, 2021.
\bibitem{yao2020dual}
Yu~Yao, Tongliang Liu, Bo~Han, Mingming Gong, Jiankang Deng, Gang Niu, and
Masashi Sugiyama.
\newblock Dual t: Reducing estimation error for transition matrix in
label-noise learning.
\newblock {\em arXiv preprint arXiv:2006.07805}, 2020.
\bibitem{yuan2021multimodal}
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire,
Ajinkya Kale, and Baldo Faieta.
\newblock Multimodal contrastive training for visual representation learning.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 6995--7004, 2021.
\bibitem{zhang2021robust}
Chuanyi Zhang, Qiong Wang, Guosen Xie, Qi~Wu, Fumin Shen, and Zhenmin Tang.
\newblock Robust learning from noisy web images via data purification for
fine-grained recognition.
\newblock {\em IEEE Transactions on Multimedia}, 2021.
\bibitem{zhang2021dualgraph}
HaiYang Zhang, XiMing Xing, and Liang Liu.
\newblock Dualgraph: A graph-based method for reasoning about label noise.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 9654--9663, 2021.
\bibitem{zhu2021deep}
Hao Zhu, Man-Di Luo, Rui Wang, Ai-Hua Zheng, and Ran He.
\newblock Deep audio-visual learning: A survey.
\newblock {\em International Journal of Automation and Computing},
18(3):351--376, 2021.
\end{thebibliography}