Audio tagging with noisy labels and minimal supervision
Extracted source text ยท arXiv 1906.02975
% --------------------------------------------------------------------------
% Template for DCASE 2019 paper; to be used with:
% dcase2019.sty - DCASE 2019 LaTeX style file, and
% IEEEbib.bst - IEEE bibliography style file.
% Adapted from spconf.sty and waspaa15.sty
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage[hyphens]{url}
\usepackage{dcase2019,amsmath,graphicx,url,times,booktabs, tabularx}
\usepackage[bookmarks=false, hidelinks]{hyperref}
% Example definitions.
% --------------------
\def\defeqn{\stackrel{\triangle}{=}}
\newcommand{\symvec}[1]{{\mbox{\boldmath $#1$}}}
\newcommand{\symmat}[1]{{\mbox{\boldmath $#1$}}}
\newcommand{\lwlrap}{\emph{l$\omega$lrap}}
% Title.
% --------------------
\title{Audio tagging with noisy labels and minimal supervision}
% Single addresses (uncomment and modify for single-address case).
% --------------------
% \name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
% \address{Author Affiliation(s)}
%
% For example:
% ------------
% \address{School\\
% Department\\
% Address}
% Two addresses
% --------------------
%\twoauthors
% {John Doe\sthanks{Thanks to ABC agency for funding.}}
% {Fictional University\\
%Computer Science Dept., 2133 Long Road\\
% Gotham, NY 10027, USA \\
% john@fictional.edu}
% {Maria Ortega\sthanks{Thanks to XYZ agency for funding.}}
% { University of the Imagination \\
% Big Engineering Building, 8765 Dream Blvd. \\
% New Chicago, IL 60626, USA \\
% maria@imagination.edu}
\name{Eduardo Fonseca$^{1}\sthanks{This work is partially supported by the European Union's Horizon 2020 research and innovation programme under grant agreement No 688382 AudioCommons and a Google Faculty Research Award 2018.}\sthanks{Equal contribution.}$,
Manoj Plakal$^{2}\footnotemark[2]$,
Frederic Font$^{1}$,
Daniel P. W. Ellis$^{2}$,
Xavier Serra$^{1}$
}
% fixed *.sty to allow names on multiple lines
\address{$^1$Music Technology Group, Universitat Pompeu Fabra, Barcelona \{name.surname\}@upf.edu\\
$^2$ Google, Inc., New York, NY, USA \{plakal,dpwe\}@google.com \\
% $^3$ University of the Fantasy, Department of Electronics, Pittsburgh, USA,
% \{john, maria\}@fantasy.edu\\
}
% Authors in two lines, use in case of many authors with many affiliations (uncomment and modify).
% --------------------
% \name{John Doe$^{1}\sthanks{Thanks to ABC agency for funding.}$,
% Maria Ortega$^{1}\sthanks{Thanks to XYZ agency for funding.}$,
% Maria Doe$^{2}$,
% John Ortega$^{2}$,
% John Maria$^{3}$,
% }
% \secondlinename{
% David Smith$^{2}$,
% Maria Smith$^{3}$,
% }
% % fixed *.sty to allow names on multiple lines
% \address{$^1$ Fictional University, Computer Science Dept., Gotham, USA, \{john, maria\}@fictional.edu\\
% $^2$ University of the Imagination, Computer Science Dept., New Chicago, USA, \\
% \{maria, john, david\}@fictional.edu\\
% $^3$ University of the Fantasy, Department of Electronics, Pittsburgh, USA,
% \{john, maria\}@fantasy.edu\\
% }
\begin{document}
\ninept
\maketitle
\begin{sloppy}
\begin{abstract}
This paper introduces Task 2 of the DCASE2019 Challenge, titled ``Audio tagging with noisy labels and minimal supervision''. This task was hosted on the Kaggle platform as ``Freesound Audio Tagging 2019''.
The task evaluates systems for multi-label audio tagging using a large set of noisy-labeled data, and a much smaller set of manually-labeled data, under a large vocabulary setting of 80 everyday sound classes.
In addition, the proposed dataset poses an acoustic mismatch problem between the noisy train set and the test set due to the fact that they come from different web audio sources.
This can correspond to a realistic scenario given the difficulty of gathering large amounts of manually labeled data.
We present the task setup, the FSDKaggle2019 dataset prepared for this scientific evaluation, and a baseline system consisting of a convolutional neural network. All these resources are freely available.
%We believe
\end{abstract}
\begin{keywords}
Audio tagging, sound event classification, audio dataset, label noise, minimal supervision
\end{keywords}
\section{Introduction}
\label{sec:intro}
\input{S1_Intro}
\section{Task Setup}
\label{sec:setup}
%\vspace{-1mm}
The goal of this task is to predict appropriate labels for each audio clip in a test set.
The predictions are to be done at the clip level, i.e., no start/end timestamps for the sound events are required.
Some test clips bear one ground truth label while others bear several labels.
Hence, the task setup is a \emph{multi-label} classification problem and the systems to be developed can be denoted as multi-label audio tagging systems, as illustrated in Fig.~\ref{fig:task_diagram}.
This task was hosted on the Kaggle platform from April 4th to June 10th 2019.
%, and had the participation of more than 800 teams.
The resources associated to this task (dataset download, submission, and leaderboard) can be found on the Kaggle competition page.\footnote{\url{https://www.kaggle.com/c/freesound-audio-tagging-2019}\\Note that the competition name on Kaggle is abbreviated from the full DCASE2019 Challenge task name to ``Freesound Audio Tagging 2019''.\label{footnote_Kaggle_page}}
\begin{figure}[ht]
%\vspace{-1mm}
\centering
\centerline{\includegraphics[width=0.82\columnwidth]{task2_freesound_audio_tagging.png}}
%\vspace{-1mm}
\caption{Overview of a multi-label tagging system.}
\label{fig:task_diagram}
\end{figure}
%\vspace{-1mm}
%3- experimental setup
As described in Section~\ref{sec:dataset}, the audio data for this task consists of a test set and two train sets: a \emph{curated} train set and a \emph{noisy} train set, that allow to experiment with training data of different levels of reliability and coming from different sources.
System evaluation was carried out on Kaggle servers (see Section \ref{ssec:eval_metric}) using the test set, which is further split into two divisions, for the \textit{public} and \textit{private} leaderboards.
% corresponding to the 25% / 75%, respectively.
During the competition, the test subset corresponding to the public leaderboard was used to provide live ranking of all participants.
To compute the final private leaderboard, at the end of the competition, systems were re-evaluated using the unseen private test set, of which neither the audio nor the labels were accessible to participants.
%-old
%Submitted systems were evaluated on Kaggle servers (see below).
%During the competition, neither the audio nor the annotations of the test set were given to participants.
%Submitted systems were evaluated on Kaggle servers (see below) and the leaderboard during the competition was computed using a subset of the test set including 25\% of the total amount of clips equally distributed across labels.
%eval metric + submission
%\vspace{-1mm}
\subsection{Evaluation Metric and Competition Rules}
\label{ssec:eval_metric}
The task was evaluated via label-weighted label-ranking average precision (abbreviated as {\lwlrap} and pronounced ``lol wrap''). This generalizes the mean reciprocal rank (MRR) used in the 2018 challenge~\cite{fonseca2018general} to the case of multiple true labels per test item.
Let $Lab(s, r)$ be the class label at rank $r$ (starting from 1) in test sample $s$, and $Rank(s, c)$ be the rank of class label $c$ in that list, i.e. $Lab(s, Rank(s, c)) = c$.
Then, if the set of ground-truth classes for sample $s$ is $C(s)$, the label-ranking precision for the list of labels up to class $c$ (assumed to be in $C(s)$) is:
\begin{equation}
Prec(s, c) = \frac{1}{Rank(s, c)} \sum_{r=1}^{Rank(s, c)} \mathbf{1}[Lab(s, r) \in C(s)]
\end{equation}
where $\mathbf{1}[\cdot]$ evaluates to 1 if the argument is true, else zero.
% Can eliminate this sentence, but reviewer asked for *insight*
$Prec(s, c)$ is equal to 1 if all the top-ranked labels down to $c$ are part of $C(s)$, and at worst case equals $1/Rank(s, c)$ if none of the higher-ranked labels are correct.
%%
In contrast to plain \emph{lrap}, which averages precisions within a sample then across samples, thereby downweighting labels that occur on samples with many labels, \emph{l$\omega$lrap} calculates the precision for each \emph{label} in the test set, and gives them all equal contribution to the final metric:
\begin{equation}
l\omega lrap = \frac{1}{\sum_s |C(s)|} \sum_s \sum_{c \in C(s)} Prec(s, c)
\end{equation}
where $|C(s)|$ is the number of true class labels for sample $s$.
We use label weighting because it allows per-class values to be calculated, while keeping the overall metric as a simple average of the per-class metrics (weighted by each label's prior in the test set). A Python implementation of {\lwlrap} is provided in \footnote{\url{https://colab.research.google.com/drive/1AgPdhSp7ttY18O3fEoHOQKlt_3HJDLi8}}.
This scientific evaluation was set up as a Kaggle \emph{Kernels-only} competition.
This means that all participants had to submit their systems as inference models in Kaggle Kernels (similar to \emph{Jupyter Notebooks}), to be evaluated on remote servers.
In addition, inference run-time was limited to a maximum of one hour in a Kernel with one GPU, and memory constraints were also imposed.
These constraints aim to discourage the usage of large model ensembles.
Participants could submit a maximum of two submissions per day, and select two \emph{final} submissions to be considered for the private leaderboard ranking.
%Additionally, participants were encouraged to submit a technical report describing their systems.
A detailed description of the task rules can be found in the Rules section of the competition page;\textsuperscript{\ref{footnote_Kaggle_page}} the most important points are summarized in the DCASE Challenge page.\footnote{\url{http://dcase.community/challenge2019/task-audio-tagging\#task-rules}}
%\vspace{-1mm}
%\subsection{Judges' Award}
To complement the leaderboard results of the {\lwlrap} ranking, the task organizers introduced a complementary Judges' Award to promote submissions using novel, problem-specific and efficient approaches.
Details about the Judges' Award rules can be found in the Judges' Award section of \textsuperscript{\ref{footnote_Kaggle_page}}.
%====================================================================================
%====================================================================================
%====================================================================================
\vspace{-4mm}
\section{Dataset}
\label{sec:dataset}
The dataset used is called \emph{FSDKaggle2019}, and it employs audio clips from the following sources:
\begin{itemize}
\item Freesound Dataset (FSD): a dataset under development based on Freesound content organized with the AudioSet Ontology \cite{gemmeke2017audio}. Freesound is a sound sharing site hosting over 400,000 clips uploaded by a community of users, who additionally provide some basic metadata, e.g., tags. \cite{font2013freesound}. These data are used to create the curated train set and the test set.
\item The soundtracks of a pool of Flickr videos taken from the Yahoo Flickr Creative Commons 100M (YFCC100M) dataset \cite{YFCC100M}. These data are used to create the noisy train set.
\end{itemize}
FSDKaggle2019 is freely available from Zenodo\footnote{\url{https://doi.org/10.5281/zenodo.3612637}}, all clips are provided as uncompressed PCM 16 bit 44.1 kHz mono audio files, its ground truth labels are provided at the clip-level (i.e., weak labels), and its partitioning is depicted in Fig.~\ref{fig:data_split}.
\begin{figure}[ht]
%\begin{figure}[ht!]
\vspace{-2mm}
\centering
% \centerline{\includegraphics[width=8.5cm]{image1}}
\centerline{\includegraphics[width=\columnwidth]{FSDKaggle2019_v5.pdf}}
\vspace{-1mm}
\caption{Data split in FSDKaggle2019, including number of clips / duration in hours, and data origin. Colors depict quality of labels: orange, yellow and green correspond to noisy labels, correct but potentially incomplete labels, and exhaustive labels, respectively.}
\label{fig:data_split}
\vspace{-2mm}
\end{figure}
%====================================================================================
\vspace{-1mm}
\subsection{Curated train set and test set}
\label{ssec:curated_train}
% vocab======================
The first step carried out in the creation of FSDKaggle2019 was the definition of a vocabulary of 80 classes drawn from the AudioSet Ontology \cite{gemmeke2017audio}.
This vocabulary was chosen based on the following criteria: \textit{i)} we consider leaf nodes of the AudioSet hierarchy for which there is enough data available in FSD, \textit{ii)} we aim to encompass a diverse range of everyday sounds, and \textit{iii)} we remove few clearly isolated classes (those with the weakest semantic relations with any of the rest), thus promoting confounds between semantically/acoustically similar classes to some extent.
%The main sound \textit{families} (i.e., groups of sound classes) in the resulting vocabulary are shown in Fig.~\ref{fig:vocab} %\footnote{\url{dcase.community/challenge2019/task-audio-tagging}}.
%and
The main sound \textit{families} (i.e., groups of sound classes) in the resulting vocabulary are, in descending order of prevalence, human sounds, domestic sounds, musical instruments, vehicles, animal sounds, natural sounds, materials, and mechanisms.
The full list of 80 classes is available in the Data section of \textsuperscript{\ref{footnote_Kaggle_page}}. %\footnote{\url{https://www.kaggle.com/c/freesound-audio-tagging-2019/data}}.
%%\begin{figure}[t]
%\begin{figure}[ht]
% \vspace{-4mm}
% \centering
%%% \centerline{\includegraphics[width=8.5cm]{image1}}
%% \centerline{\includegraphics[width=1.05\columnwidth]{pie_v3.png}}
% \centerline{\includegraphics[width=1\columnwidth]{pie_chart_sheet_v4.png}}
% \vspace{-27mm}
% \caption{Main sound families in FSDKaggle2019. Numbers in segments correspond to amount of sound classes.}
% \label{fig:vocab}
%\vspace{-3mm}
%\end{figure}
% annotation part 1======================
In a second step, we did a mapping of Freesound clips to the selected 80 class labels.
To this end, a set of keywords was defined connecting the user-provided Freesound tags with the AudioSet labels.
Using this mapping, for every class, we retrieved the audio clips that feature at least one of the defined keywords among their tags.
This process led to a number of automatically-generated \textit{candidate annotations} indicating the potential presence of a sound class in an audio clip (i.e., weak labels, as timing information is not included).
Nonetheless, in some audio clips the target signal fills the clip length almost completely, which can be considered as a strong label.
Subsequently, the candidate annotations were human-validated using a \textit{validation} task deployed in \emph{Freesound Annotator},\footnote{\url{https://annotator.freesound.org}\label{footnote_url_FSDs}} an online platform for the collaborative creation of open audio datasets \cite{Fonseca2017freesound}.
In this task, users verify the presence/absence of a candidate sound class in an audio clip with a \emph{rating} mechanism.
The vast majority of provided labels have inter-annotator agreement but not all of them.
The outcome is a set of clips where the corresponding label(s) are correct; nevertheless, it can happen that a few of these audio clips present additional acoustic material beyond the provided label(s).
% split======================
The resulting data were split into a train set and a test set.
We refer to this train set as \textit{curated} in order to distinguish it from the noisy set described in Section \ref{ssec:noisy_train}.
To mitigate train-test contamination, the split was carried out considering the clip uploaders in Freesound.
We allocated all audio clips uploaded from the same user into either the curated train set or the test set, so that the sets are disjoint at the Freesound user level.
The partition proportion was defined to limit the supervision provided in the curated train set, thus promoting approaches to deal with label noise.
% annotation part 2======================
Finally, the test set was further annotated using a label \textit{generation} tool \cite{favory2018facilitating}, in which \textit{i)} pre-existent labels can be re-validated, and \textit{ii)} potentially missing labels can be added through exploration of the AudioSet Ontology.
The outcome is a set of exhaustively labeled clips where the label(s) are correct and complete considering the target vocabulary; nonetheless, few clips could still present additional (unlabeled) acoustic content out of the vocabulary.
%All the acoustic material present in the test set is labeled, except human error, considering the vocabulary of 80 classes used in the competition.
% TABLE / DATASET CHARACTERISTICS ======
The main characteristics of the curated train set, noisy train set and test set are listed in Table~\ref{tab:dataset_stats}.
The curated train set consists of 4970 clips with a total of 5752 labels.
Labels per clip ranges from 1 to 6 with a mean of 1.2.
The test set consists of 4481 clips with a total of 6250 labels.
Labels per clip ranges from 1 to 6 with a mean of 1.4.
Note the increased number of labels per clip with respect to the curated train set, due to the process of exhaustive labelling.
In both cases, clip length ranges from 0.3s to 30 due to the diversity of the sound classes and the preferences of Freesound users when recording/uploading sounds.
\begin{table}[t!]
\vspace{-1mm}
\caption{Main stats of the sets in FSDKaggle2019. $^*$A few classes have slightly less than 75 clips.}
\vspace{+2mm}
\centering
\begin{tabular}{l|c|c|c}
%\hline
\textbf{Aspect} & \textbf{curated train} & \textbf{noisy train} & \textbf{test} \\
\hline
Clips/class & $\sim$75$^*$ & 300 & $\sim$ 50 - 150 \\
Total clips & 4970 & 19,815 & 4481 \\
Labels/clip & 1.2 & 1.2 & 1.4 \\
Clip length & $\sim$0.3 - 30s & $\sim$15s &$\sim$0.3 - 30s \\
Total duration & $\sim$10.5h & $\sim$80h &$\sim$12.9h \\
Labelling & correct & noisy & exhaustive \\
& (inexhaustive) & & \\
\end{tabular}
\label{tab:dataset_stats}
\vspace{-5mm}
\end{table}
%====================================================================================
%====================================================================================
%====================================================================================
%====================================================================================
%\vspace{-2mm}
\subsection{Noisy train set}
\label{ssec:noisy_train}
The noisy train set was prepared using the YFCC100M dataset \cite{YFCC100M}, which has the advantages of \textit{i)} being a very large and diverse dataset that is not correlated with Freesound in acoustics or domain, and \textit{ii)} offering permissive Creative Commons licenses that allow ease of use, modification, and redistribution. The original dataset contained $\sim$99M photos and $\sim$793k videos from $\sim$581K Flickr users. We dropped videos with licenses that disallowed making derivatives or commercial use, videos that were no longer available, and videos with audio decode errors that we could not transcode, leaving us with $\sim$201K 44.1 kHz mono WAV files. Video length varied with a maximum of 20 minutes, and a mean of $\sim$37s and median of $\sim$20s.
The Flickr video metadata (title, description, tags) proved to be too sparse to meaningfully map to our class vocabulary. Therefore, we used a content-based approach where we generated video-level predictions from a variety of pre-trained audio models: a shallow fully-connected network as well as variants of VGG and ResNet \cite{Hershey2017}, all of which were trained on a large collection of YouTube videos using the AudioSet class vocabulary. We generated sliding windows of $\sim$1s containing log mel spectrogram patches and aggregated the per-window predictions (using either maximum or average pooling) to produce a video-level vector of class scores. For each of our 80 classes, we kept the top 300 videos by predicted score for that class. We browsed the video labels and selected the maximum-pooled VGG-like model as producing a balance between reasonable predictions and a substantial amount of label noise. As a further source of noise, each final clip was produced by taking a random slice of a video of length up to 15 seconds (videos shorter than 15 seconds would be taken in their entirety).
Hence, the label noise can vary widely in amount and type depending on the class, including in- and out-of-vocabulary noises \cite{Fonseca2019learning}.
As listed in Table~\ref{tab:dataset_stats}, the noisy train set consists of 19,815 clips with a total of 24,000 labels (300 * 80). Labels per clip ranges from 1 to 7 with a mean of 1.2.
%and a median of 1.
Clip length ranges from 1s to 15s (by construction), with a mean of 14.5s.
%and a median of 15s,
Therefore, the per-class training data distribution in FSDKaggle2019 is, for most of the classes, 300 clips from the noisy set and 75 clips from the curated set.
This means 80\% noisy / 20\% curated at the clip level, while at the duration level the proportion is more extreme considering the variable-length clips.
Since most of the train data come from YFCC, acoustic domain mismatch between the train and test set can be expected.
We conjecture this mismatch comes from a variety of reasons.
For example, through acoustic inspection of a small sample of both data sources, we find a higher percentage of high quality recordings in Freesound.
In addition, audio clips in Freesound are typically recorded with the purpose of capturing audio, which is not necessarily the case in YFCC.
%====================================================================================
%====================================================================================
%====================================================================================
%\vspace{-2mm}
\section{Baseline System}
\label{sec:baseline}
%\vspace{-1mm}
\subsection{Model Architecture and Training}
The baseline system uses a convolutional neural network that takes log mel spectrogram patches as input and produces predicted scores for 80 classes. We use an efficient MobileNet v1 \cite{howard2017mobilenets} architecture which lets us fit comfortably within the inference time limits of the challenge. Incoming audio (always 44.1 kHz mono) is divided into overlapping windows of size 1s with a hop of 0.5s. These windows are decomposed with a short-time Fourier transform using 25ms windows every 10ms. The resulting spectrogram is mapped into 96 mel-spaced frequency bins covering 20 Hz to 20 kHz, and the magnitude of each bin is log-transformed after adding a small offset to avoid numerical issues. The model consists of 1 convolutional layer followed by 13 separable convolution layers (which give MobileNets their compactness) followed by either max or average pooling, and an 80-way logistic classifier layer. The model contains $\sim$3.3M weights and, by comparison, is $\sim$8x smaller than a ResNet-50 while using $\sim$4x less compute. Detailed documentation is available in the public release of the baseline system code.\footnote{\url{https://github.com/DCASE-REPO/dcase2019_task2_baseline}}
To combat label noise, we use dropout (inserted before the classifier layer) as well as label smoothing \cite{inceptionv3} which replaces each label's target value with a hyperparameter-controlled blend of the original value and 0.5 (representing a uniform probability distribution). To combat the domain mismatch between the test set and noisy train set, we use transfer learning by training a model on the noisy set first to learn a representation, and then use a checkpoint from that run to warm-start training on the curated train set. In addition, we used batch normalization, exponential learning rate decay, and the Adam optimizer. We trained models on the curated data alone, the noisy data alone, the curated and noisy combined, noisy first followed by warm-started curated, as well as a weighted version of warm-started training that we describe next.
%\vspace{-3mm}
\subsection{Results}
Table \ref{baseline_results_table} shows the {\lwlrap} values produced by our various baseline models when evaluated on the entire test set (i.e., including both the public and private splits). Each row lists the best {\lwlrap} obtained from a small grid search (using the public test set for evaluation) that varied maximum vs average pooling, learning rate, label smoothing, dropout, learning rate decay, and whether or not we used batch normalization. The baseline system that we settled on was ``Warm-started curated", which achieved a {\lwlrap} of 0.537 on the public test set (see the publicly released baseline code for hyperparameter choices).
\vspace{-4mm}
\begin{table}[h!]
\caption{Baseline system results on the \textit{entire} test set.}
\vspace{+2mm}
\centering
\begin{tabular}{l|l}
\textbf{Training approach} & \textbf{\lwlrap} \\ \hline
Curated only & 0.542 \\
Noisy only & 0.312 \\
Curated + Noisy & 0.522 \\
\textbf{Warm-started curated} & \textbf{0.546} \\
Weighted warm-started curated & 0.561 \\
\end{tabular}
\label{baseline_results_table}
%\vspace{-1mm}
\end{table}
Comparing ``Noisy only" and ``Curated + Noisy" to ``Curated only" shows a considerable domain mismatch where we hurt our performance when we blindly add more data. A transfer learning approach of warm-starting the curated training with a noisily trained model gives us a small boost in performance.
We conducted a class-based analysis of the results in a public Colab notebook\footnote{\url{https://colab.research.google.com/drive/1vIpAWCGSHyYYLblbtMzAKLNTkHcr1SMK}} where we look at the best and worst classes of each model.
The baseline system attains highest {\lwlrap} values for \emph{Bicycle bell} (0.894) and \emph{Purr} (0.873); lowest values occur for \emph{Cupboard open or close} (0.219) and \emph{Chirp, tweet} (0.127).
We also analyse the correlations between various pairs of models. It becomes evident that, at least for our baseline models, there are classes where using curated data alone is better while there are other classes where the noisy data is better. One simple way to incorporate this in training is to use the ratio of noisy to curated {\lwlrap s} as a per-class weight during noisy training to boost classes that have value and suppress classes that do not. When we warmstart with this weighted noisy model, we get a further boost in performance. This optimization is not included in the released baseline.
%==========================================================================================
%\vspace{-1mm}
\section{Conclusion}
\label{sec:conclusion}
In this paper, we have described the task setup, dataset, and baseline system of DCASE2019 Task 2 ``Audio tagging with noisy labels and minimal supervision''.
This task was hosted on the Kaggle platform as ``Freesound Audio Tagging 2019''.
The goal is to adequately exploit a large set of noisy-labeled data and a small quantity of manually-labeled data, in a multi-label audio tagging setting with a vocabulary of 80 everyday sound classes.
In addition, the dataset poses an acoustic mismatch problem between the noisy train set and the test set due to the fact that they come from different web audio sources.
We believe this can correspond to a realistic scenario given the difficulty in gathering large amounts of manually labeled data.
{\lwlrap} is proposed as evaluation metric. Baseline experiments indicate that leveraging noisy-labeled data with a distribution shift for sound event tagging can be challenging.
The FSDKaggle2019 dataset and the baseline system proposed are freely available and not limited for use within the competition.
%The dataset is currently available on the Kaggle competition page, and future updates of the dataset (including ground-truth data for the test set and extra associated Freesound metadata) will be made publicly available in the Freesound Datasets platform.
%Through FSDKaggle2018 and the provided baseline system, this competition intends to foster open research in sound event recognition.
%\newpage
%\vspace{-1mm}
\section{ACKNOWLEDGMENT}
\label{sec:ack}
%The preferred spelling of the word acknowledgment in America is without an ``e'' after the ``g.''
%Try to avoid the stilted expression, ``One of us (R. B. G.) thanks ...'' Instead, try ``R.B.G.\ thanks ...''
%Put sponsor acknowledgments in the unnumbered footnote on the first page.
We thank Addison Howard and Walter Reade of Kaggle for their invaluable assistance with the task design and Kaggle platform, and everyone who contributed to FSDKaggle2019 with annotations, especially Jordi Pons and Xavier Favory.
Eduardo Fonseca is also grateful for the GPU donated by NVidia.
\newpage
% -------------------------------------------------------------------------
% Either list references using the bibliography style file IEEEtran.bst
\bibliographystyle{IEEEtran}
\bibliography{refs}
%
% or list them by yourself
% \begin{thebibliography}{9}
%
% \bibitem{dcase2016web}
% \url{http://www.cs.tut.fi/sgn/arg/dcase2016/}.
%
% \bibitem{IEEEPDFSpec}
% {PDF} specification for {IEEE} {X}plore$^{\textregistered}$,
% \url{http://www.ieee.org/portal/cms_docs/pubs/confstandards/pdfs/IEEE-PDF-SpecV401.pdf}.
%
% \bibitem{PDFOpenSourceTools}
% Creating high resolution {PDF} files for book production with
% open source tools,
% \url{http://www.grassbook.org/neteler/highres_pdf.html}.
%
% \bibitem{eWilliams1999}
% E. Williams, \emph{Fourier Acoustics: Sound Radiation and Nearfield Acoustic
% Holography}. London, UK: Academic Press, 1999.
%
% \bibitem{ieeecopyright}
% \url{http://www.ieee.org/web/publications/rights/copyrightmain.html}.
%
% \bibitem{cJones2003}
% C. Jones, A. Smith, and E. Roberts, ``A sample paper in conference
% proceedings,'' in \emph{Proc. IEEE ICASSP}, vol. II, 2003, pp. 803--806.
%
% \bibitem{aSmith2000}
% A. Smith, C. Jones, and E. Roberts, ``A sample paper in journals,''
% \emph{IEEE Trans. Signal Process.}, vol. 62, pp. 291--294, Jan. 2000.
%
% \end{thebibliography}
\end{sloppy}
\end{document}
%=====INTRO
Environmental sound recognition has gained attention in recent years, encompassing tasks such as acoustic scene classification, sound event detection or audio tagging \cite{virtanen2018computational}.
The latter is becoming a popular task partly due to the various audio tagging tasks in the DCASE Challenge editions, and its impact on applications such as automatic description of multimedia, or acoustic monitoring.
This paper describes the characteristics, dataset and baseline system of DCASE2019 Task 2 ``Audio tagging with noisy labels and minimal supervision''.
%=====LABEL NOISE AS THE PROBLEM
%PArt 1=========== larger datasets have noise
Everyday sound tagging consists of identifying the sound events present in an audio recording.
The most common approach to create sound event taggers relies on supervised learning through labeled audio datasets.
New released datasets tend to be of increasing size in order to allow exploitation of data-driven approaches, e.g., deep learning.
However, manual labeling of large datasets is expensive and typically a limiting factor in machine listening; hence, creators are often forced to compromise between dataset size and label quality.
Thus, most recent datasets feature larger sizes \cite{gemmeke2017audio,fonseca2018general,Fonseca2019learning}, but their labeling is less precise than that of conventional small and exhaustively labeled datasets \cite{salamon2014dataset,piczak2015esc,foster2015chime}.
We are, therefore, witnessing a transition towards larger datasets that inevitably include some degree of label noise.
%PArt 2============== more categories are difficult to annotate
Likewise, the current trend is moving towards general-purpose sound event recognizers, able to recognize a broad range of everyday sounds.
This is favoured by the appearance of the AudioSet Ontology; a hierarchical tree with 627 classes encompassing the most common everyday sounds \cite{gemmeke2017audio}.
This implies that not only are we interested in recognizing typical sound sources (e.g., \emph{Bark}), but also less usual sound classes (e.g., production mechanisms such as \emph{Fill (with liquid)}).
%, the sound of a container being filled with liquid).
Further, not only are these classes less frequent, but also some can be semantically or acoustically similar to others (e.g. \emph{Trickle, dribble} and \emph{Fill (with liquid)}).
%For instance, \emph{Trickle, dribble}
%, the sound of a liquid flowing slowly in an unsteady stream or drops,
%can be similar to \emph{Fill (with liquid)}, or partially overlapped with it.
%Since these sound categories are more ambiguous, their manual annotation becomes more difficult, and sometimes dependent on the annotator's cultural background.
Manual annotation of these more ambiguous categories becomes more difficult,
which makes them more prone to labeling errors.
%PArt 3=============== web audio
An alternative to gathering data for training general-purpose audio taggers is to retrieve audio and metadata from websites such as Freesound or Flickr.
Labels can be inferred automatically by using automated heuristics applied to the metadata, or applying pre-trained classifiers on the audio material.
This approach supports rapid collection of large amounts of data, but at the cost of a considerable amount of label noise.
%Conclusion
In this context, label noise arises as a challenge in general-purpose sound event recognition, including adverse effects such as performance drop or increased complexity of models \cite{frenay2014classification}, and also hindering proper learning of deep networks~\cite{arpit2017closer,zhang2016understanding}.
Consequently, coping with label noise could open the door to better sound event classifiers, and could allow the exploitation of large amounts of web audio for training, while reducing manual annotation needs.
%=====LABEL NOISE SOTA
The topic of \textit{learning with noisy labels} is a consolidated research area in computer vision \cite{han2018co,malach2017decoupling,jiang2017mentornet,goldberger2016training,veit2017learning,zhang2018generalized,patrini2017making}.
%. probably due to the access of massive amounts of data.
%massive size of some image datasets and the accessibility to even more data in the web.
%Consequently, the literature is extensive, encompassing a wide variety of approaches to deal with label noise \cite{han2018co,malach2017decoupling,jiang2017mentornet,patrini2017making,goldberger2016training,veit2017learning,zhang2018generalized}.
However, in sound recognition it has received little attention, probably due to the conventional paradigm of learning from small and clean datasets; only a few works directly address the analysis and mitigation of label noise \cite{Fonseca2019learning, kumar2018learning, Fonseca2019model}.
%=====TASK2 INTRODUCTION AND RESEARCH PROBLEMS
In this paper, we propose a task, a dataset and a baseline system to foster label noise research in general-purpose sound event tagging.
We follow up on DCASE2018 Task 2 \cite{fonseca2018general}, and propose to investigate the scenario where a small set of manually-labeled data is available, along with a larger set of noisy-labeled data, in a multi-label audio tagging setting, and using a vocabulary of 80 classes of everyday sounds.
The proposed task addresses two main research problems.
The first problem is how to adequately exploit a large quantity of noisy labels, many of which are incorrect and/or incomplete, and how to complement it with the supervision provided by a much smaller amount of reliable manually-labeled data (\textit{minimal} supervision).
The second problem is given by the acoustic mismatch between the noisy train set and the test set.
Distribution shifts between data have been shown to cause substantial performance drops in machine learning, both for vision \cite{recht2019imagenet} and audio \cite{mesaros2018acoustic}.
In our case, the noisy train set comes from a different web audio source than the test set, which is sometimes a real-world constraint.
%, where larger noisy set can be taken from the web.
This paper is organized as follows.
Section \ref{sec:setup} provides more details about the task and its experimental setup.
Section \ref{sec:dataset} presents the dataset prepared for the task, and Section \ref{sec:baseline} describes a baseline system.
Final remarks are given in Section \ref{sec:conclusion}.
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\def\UrlFont{\rmfamily}
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\bibitem{virtanen2018computational}
T.~Virtanen, M.~D. Plumbley, and D.~Ellis, \emph{Computational Analysis of
Sound Scenes and Events}.\hskip 1em plus 0.5em minus 0.4em\relax Springer,
2018.
\bibitem{gemmeke2017audio}
J.~F. Gemmeke, D.~P.~W. Ellis, D.~Freedman, A.~Jansen, W.~Lawrence, R.~C.
Moore, M.~Plakal, and M.~Ritter, ``{Audio Set: An ontology and human-labeled
dataset for audio events},'' in \emph{Proc. IEEE ICASSP 2017}, New Orleans,
LA, 2017.
\bibitem{fonseca2018general}
E.~Fonseca, M.~Plakal, F.~Font, D.~P. Ellis, X.~Favory, J.~Pons, and X.~Serra,
``{General-purpose tagging of Freesound audio with AudioSet labels: task
description, dataset, and baseline},'' in \emph{Proceedings of the Detection
and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)},
2018.
\bibitem{Fonseca2019learning}
E.~Fonseca, M.~Plakal, D.~P.~W. Ellis, F.~Font, X.~Favory, and X.~Serra,
``{Learning Sound Event Classifiers from Web Audio with Noisy Labels},'' in
\emph{Proc. IEEE ICASSP 2019}, Brighton, UK, 2019.
\bibitem{salamon2014dataset}
J.~Salamon, C.~Jacoby, and J.~P. Bello, ``A dataset and taxonomy for urban
sound research,'' in \emph{Proceedings of the ACM International Conference on
Multimedia}.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2014, pp.
1041--1044.
\bibitem{piczak2015esc}
K.~J. Piczak, ``{ESC: Dataset for environmental sound classification},'' in
\emph{Proceedings of the ACM International Conference on Multimedia}.\hskip
1em plus 0.5em minus 0.4em\relax ACM, 2015, pp. 1015--1018.
\bibitem{foster2015chime}
P.~Foster, S.~Sigtia, S.~Krstulovic, J.~Barker, and M.~D. Plumbley,
``{CHiME-home}: A dataset for sound source recognition in a domestic
environment,'' in \emph{Workshop on Applications of Signal Processing to
Audio and Acoustics}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2015.
\bibitem{frenay2014classification}
B.~Fr{\'e}nay and M.~Verleysen, ``Classification in the presence of label
noise: a survey,'' \emph{IEEE transactions on neural networks and learning
systems}, vol.~25, no.~5, 2014.
\bibitem{arpit2017closer}
D.~Arpit, S.~Jastrzebski, N.~Ballas, D.~Krueger, E.~Bengio, M.~S. Kanwal,
T.~Maharaj, A.~Fischer, A.~Courville, Y.~Bengio, \emph{et~al.}, ``A closer
look at memorization in deep networks,'' in \emph{Proceedings of the 34th
International Conference on Machine Learning-Volume 70}.\hskip 1em plus 0.5em
minus 0.4em\relax JMLR. org, 2017, pp. 233--242.
\bibitem{zhang2016understanding}
C.~Zhang, S.~Bengio, M.~Hardt, B.~Recht, and O.~Vinyals, ``Understanding deep
learning requires rethinking generalization,'' \emph{arXiv preprint
arXiv:1611.03530}, 2016.
\bibitem{han2018co}
B.~Han, Q.~Yao, X.~Yu, G.~Niu, M.~Xu, W.~Hu, I.~Tsang, and M.~Sugiyama,
``{Co-teaching: Robust training of deep neural networks with extremely noisy
labels},'' in \emph{Advances in Neural Information Processing Systems}, 2018,
pp. 8527--8537.
\bibitem{malach2017decoupling}
E.~Malach and S.~Shalev-Shwartz, ``{Decoupling" when to update" from "how to
update"},'' in \emph{Advances in Neural Information Processing Systems},
2017, pp. 960--970.
\bibitem{jiang2017mentornet}
L.~Jiang, Z.~Zhou, T.~Leung, L.-J. Li, and L.~Fei-Fei, ``{Mentornet: Learning
data-driven curriculum for very deep neural networks on corrupted labels},''
\emph{arXiv preprint arXiv:1712.05055}, 2017.
\bibitem{goldberger2016training}
J.~Goldberger and E.~Ben-Reuven, ``Training deep neural-networks using a noise
adaptation layer,'' in \emph{Proceedings of the International Conference on
Learning Representations}, 2017.
\bibitem{veit2017learning}
A.~Veit, N.~Alldrin, G.~Chechik, I.~Krasin, A.~Gupta, and S.~J. Belongie,
``Learning from noisy large-scale datasets with minimal supervision.'' in
\emph{Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition}, 2017, pp. 6575--6583.
\bibitem{zhang2018generalized}
Z.~Zhang and M.~Sabuncu, ``Generalized cross entropy loss for training deep
neural networks with noisy labels,'' in \emph{Advances in Neural Information
Processing Systems}, 2018.
\bibitem{patrini2017making}
G.~Patrini, A.~Rozza, A.~Krishna~Menon, R.~Nock, and L.~Qu, ``Making deep
neural networks robust to label noise: A loss correction approach,'' in
\emph{Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition}, 2017, pp. 1944--1952.
\bibitem{kumar2018learning}
A.~Kumar, A.~Shah, A.~Hauptmann, and B.~Raj, ``Learning sound events from webly
labeled data,'' \emph{arXiv preprint arXiv:1811.09967}, 2018.
\bibitem{Fonseca2019model}
E.~Fonseca, F.~Font, and X.~Serra, ``Model-agnostic approaches to handling
noisy labels when training sound event classifiers,'' in \emph{Proceedings of
IEEE Workshop on Applications of Signal Processing to Audio and Acoustics},
New York, US, 2019.
\bibitem{recht2019imagenet}
B.~Recht, R.~Roelofs, L.~Schmidt, and V.~Shankar, ``{Do ImageNet Classifiers
Generalize to ImageNet?}'' \emph{arXiv preprint arXiv:1902.10811}, 2019.
\bibitem{mesaros2018acoustic}
A.~Mesaros, T.~Heittola, and T.~Virtanen, ``{Acoustic scene classification: an
overview of DCASE 2017 challenge entries},'' in \emph{2018 16th International
Workshop on Acoustic Signal Enhancement (IWAENC)}.\hskip 1em plus 0.5em minus
0.4em\relax IEEE, 2018, pp. 411--415.
\bibitem{font2013freesound}
F.~Font, G.~Roma, and X.~Serra, ``Freesound technical demo,'' in
\emph{Proceedings of the ACM International Conference on Multimedia}.\hskip
1em plus 0.5em minus 0.4em\relax ACM, 2013, pp. 411--412.
\bibitem{YFCC100M}
B.~Thomee, D.~A. Shamma, G.~Friedland, B.~Elizalde, K.~Ni, D.~Poland, D.~Borth,
and L.-J. Li, ``{YFCC100M: The New Data in Multimedia Research},''
\emph{Communications of the ACM}, vol.~59, no.~2, pp. 64--73, Feb. 2016.
\bibitem{Fonseca2017freesound}
E.~Fonseca, J.~Pons, X.~Favory, F.~Font, D.~Bogdanov, A.~Ferraro, S.~Oramas,
A.~Porter, and X.~Serra, ``{Freesound Datasets: a platform for the creation
of open audio datasets},'' in \emph{Proceedings of the 18th International
Society for Music Information Retrieval Conference (ISMIR 2017)}, Suzhou,
China, 2017, pp. 486--493.
\bibitem{favory2018facilitating}
X.~Favory, E.~Fonseca, F.~Font, and X.~Serra, ``Facilitating the manual
annotation of sounds when using large taxonomies,'' in \emph{Proceedings of
the 23rd Conference of Open Innovations Association FRUCT}.\hskip 1em plus
0.5em minus 0.4em\relax FRUCT Oy, 2018, p.~60.
\bibitem{Hershey2017}
S.~Hershey, S.~Chaudhuri, D.~P.~W. Ellis, J.~F. Gemmeke, A.~Jansen, C.~Moore,
M.~Plakal, D.~Platt, R.~A. Saurous, B.~Seybold, M.~Slaney, R.~Weiss, and
K.~Wilson, ``{CNN Architectures for Large-Scale Audio Classification},'' in
\emph{Proc. IEEE ICASSP 2017}, New Orleans, LA, 2017.
\bibitem{howard2017mobilenets}
A.~G. Howard, M.~Zhu, B.~Chen, D.~Kalenichenko, W.~Wang, T.~Weyand,
M.~Andreetto, and H.~Adam, ``{MobileNets: Efficient Convolutional Neural
Networks for Mobile Vision Applications},'' \emph{arXiv preprint
arXiv:1704.04861}, 2017.
\bibitem{inceptionv3}
\BIBentryALTinterwordspacing
C.~Szegedy, V.~Vanhoucke, S.~Ioffe, J.~Shlens, and Z.~Wojna, ``{Rethinking the
Inception Architecture for Computer Vision},'' in \emph{Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition,}, 2016. [Online].
Available: \url{http://arxiv.org/abs/1512.00567}
\BIBentrySTDinterwordspacing
\end{thebibliography}