Neural Percussive Synthesis Parameterised by High-Level Timbral Features
Extracted source text · arXiv 1911.11853
% Template for ICASSP-2020 paper; to be used with:
% spconf.sty - ICASSP/ICIP LaTeX style file, and
% IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
\usepackage{amsfonts}
\usepackage{url}
\urlstyle{rm}
\usepackage{hyperref}
% Remove dirty rectagles around urls and refs
\hypersetup{
colorlinks = true, %Colours links instead of ugly boxes
urlcolor = black, %Colour for external hyperlinks
linkcolor = black, %Colour of internal links
citecolor = black %Colour of citations
}
\usepackage{booktabs}
\usepackage{float}
\usepackage{siunitx}
\raggedbottom
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}
% captions
% NOTE: using small caption font in order to make paper fit page limitations (allowed by guidelines, but not default in template)
%\usepackage[font=small,labelfont=bf]{caption}
%\captionsetup[figure]{font=small,labelfont=bf,name={Fig.},labelsep=period}
%\captionsetup[table]{font=small,labelfont=bf,name={Table},labelsep=period}
%\captionsetup{belowskip=-5pt,aboveskip=8pt} % default 0pt,10pt
%\captionsetup{belowskip=-6pt,aboveskip=6pt} % default 0pt,10pt
% Title.
% ------
\title{Neural Percussive Synthesis Parameterised by High-Level Timbral Features}
%https://www.overleaf.com/project/5d821c644d429c000128c06b
% Single address.
% ---------------
\name{Ant\'onio Ramires\textsuperscript{1}, Pritish Chandna\textsuperscript{1}, Xavier Favory\textsuperscript{1}, Emilia G\'omez\textsuperscript{1,2}, Xavier Serra\textsuperscript{1}\thanks{This work is partially funded by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No765068, MIP-Frontiers. This work is partially supported by the Towards Richer Online Music Public-domain Archives (TROMPA) project. The TITANX used for this research was donated by the NVIDIA Corporation. }}
\address{\textsuperscript{1}Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain\\
\textsuperscript{2}Joint Research Centre, European Commission, Seville, Spain}
%
% For example:
% ------------
%\address{School\\
% Department\\
% Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
% {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
% {School A-B\\
% Department A-B\\
% Address A-B}
% {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
% while at ...}}
% {School C-D\\
% Department C-D\\
% Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
We present a deep neural network-based methodology for synthesising percussive sounds with control over high-level timbral characteristics of the sounds.
This approach allows for intuitive control of a synthesizer, enabling the user to shape sounds without extensive knowledge of signal processing.
We use a feedforward convolutional neural network-based architecture, which is able to map input parameters to the corresponding waveform.
%We leverage Freesound, a large archive of publicly sourced sounds to create a dataset of various percussion sounds, and a personal collection of kick sounds.
We propose two datasets to evaluate our approach on both a restrictive context, and in one covering a broader spectrum of sounds.
The timbral features used as parameters are taken from recent literature in signal processing.
We also use these features for evaluation and validation of the presented model, to ensure that changing the input parameters produces a congruent waveform with the desired characteristics.
Finally, we evaluate the quality of the output sound using a subjective listening test.
We provide sound examples and the system's source code for reproducibility.
\end{abstract}
%
\begin{keywords}
Wave-U-Net, Percussive Sound Synthesis, Generative Models, Music Information Retrieval, Creative Interfaces
\end{keywords}
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% INTRO %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\label{sec:intro}
%Percussion is a common part of music and was handled through acoustic percussive instruments
Percussion is one of the main components in music and is normally responsible for a song's rhythm section. Classic percussion instruments create sound when struck or scraped; however new electronic instruments were developed for generating these sounds either through playing prerecorded samples or through synthesising them. These are called drum machines and became very popular for electronic music~\cite{tr808music}. However, these early drum machines did not provide much control over the generation of the sounds. With the developments in digital audio technology and computer music, new drum machines were hand-designed using expert knowledge on synthesis techniques and electronic music production.
With the success of deep learning, several innovative generative methodologies have been proposed in the recent years. These methodologies include Generative Adversarial Networks (GANs)~\cite{gans}, Variational Autoencoders (VAEs)~\cite{vae} and autoregressive networks~\cite{oord2016wavenet,engelNsynth}. In the audio domain, such methodologies have been applied for singing voice~\cite{chandna2019wgansing}, instrumental sounds~\cite{engelNsynth} and drum sound generation~\cite{NeuralDrumMachine}. However, in the case of percussive sounds, the proposed methods only allow the user to navigate in non-intuitive high dimensional latent spaces.
The aim of our research is to create a single-event percussive-sound synthesizer that can be intuitively controlled by users, despite their sound design knowledge. This requires both a back end of a generative model that is able to map the user controls to the output sound and a front end user interface. In this paper, we propose a generative methodology based on the Wave-U-Net architecture~\cite{stoller2018wave}. Our method maps high-level characteristics of sounds to the corresponding waveforms.
%
The use of these features is aimed at giving the end-user intuitive control over the sound generation process. We also present a dataset of \num{10000} percussive one-shot sounds collected from Freesound~\cite{font2013freesound}, curated specially for this study.
The source code for our model is available online\footnote{\url{https://github.com/pc2752/percussive_synth}}, as are sound examples\footnote{\url{https://pc2752.github.io/percussive_synth/}}, showcasing the robustness of the models.
% From the field of , several hand-crafted features have been developed by experts so as to characterise audio in a semantically meaningful way.
% Our purpose is to bridge high-level semantic MIR characteristics together with sound generation to create a novel generative model for drum synthesis. For this, we developed a Wave UNet architecture which uses a set of perceptual characteristics, together with high-level semantic features to generate new drum sounds, allowing a generation based on easy to understand characteristics of the desired sound.
% Specifically, in this paper, our contributions are the following:
% \begin{itemize}
% \item We introduce NeuroDrum, a generative model for creating drum sounds from high-level timbral characteristics.
% \item We present a dataset of around 10k percussive one-shot sounds collected from Freesound \cite{font2013freesound}.
% \item We demonstrate that NeuroDrum is able to model the high-level characteristics of the audio and evaluate perceptually how different training losses affect the synthesis quality.
% \end{itemize}
% The rest of the paper is organised as follows. Section \ref{sec:rel_work} provides a brief overview of related generative models and the state-of-the-art. Section \ref{sec:sem_feat} discusses the semantic features used in our work, while section \ref{sec:dataset} presents the methodology for curating the dataset for this study.
% The following sections, section \ref{sec:system} and section \ref{sec:feats} present our model for drum sound synthesis and the related hyperparameters. We then present an evaluation of the model, in section \ref{sec:results}, before wrapping up with the conclusions of the paper and a discussion of our future direction in section \ref{sec:conclusion}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Related Work %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%history of drum machines and how they affected music production
%deep generative networks
%semantic features used and why they are the coolest thing ever
%UNET with transposed convolutions
\section{Generative Models For Audio}
\label{sec:rel_work}
%Generative Models in the audio domain(Pritish)
%pritish can u say why spectogram suck plz
%describe what was used for speech and stuff
%Generative Models for Musical Note Synthesis (Antonio)
In the audio domain, several generative models have been proposed over the recent years. In the context of music, generative models have shown success specially in creating pitched instrumental sounds, when conditioned on musical notes. A pioneering work on this field was NSynth~\cite{engelNsynth}. This synthesizer is based on the
wavenet vocoder~\cite{shen2018natural} an autoregressive architecture, which, while capable of generating high quality sounds is very resource intensive.
Several alternate architectures have been used for the generation of musical notes, based on GANs~\cite{advaudiosynth,engel2018gansynth}, VAEs~\cite{esling2018bridging,NeuralDrumMachine,manytomany}, Adversarial AEs~\cite{bitton} and AEs with Wavenet~\cite{engelNsynth}.
% These systems have used both the sound waveform as an input~\cite{engelNsynth,advaudiosynth}, magnitude and phase spectograms of the signal \cite{engel2018gansynth} or only the magnitude spectogram \cite{esling2018bridging,bitton,advaudiosynth,NeuralDrumMachine,manytomany}.
For percussive sound synthesis, the most relevant work is the Neural Drum Machine~\cite{NeuralDrumMachine}, which uses a Conditional Wasserstein Auto Encoder~\cite{tolstikhin2017wasserstein}, trained on the magnitude component of the spectogram of percussive sounds coupled with a Multi-Head Convolutional Neural Network for reconstructing the audio from the spectral representation. Principal Component Analysis is used on the low-dimensional representation learned by the AE to select the \num{3} most influential dimensions of the \num{64} dimensions of the embedding. These are provided to the user over a control interface. However these parameters controlled by the user are abstract and are not shown to be perceptually relevant or semantically meaningful.
In our case, we wish to directly map a chosen set of features to the output sound. The WaveNet~\cite{oord2016wavenet} architecture has been shown to generate high quality waveforms conditioned on input features. However, the autoregressive nature of the model makes it resource extensive and the short nature of percussive sounds do not require the use of a long temporal model. Therefore, for our study, we decided to use the Wave-U-Net~\cite{stoller2018wave} architecture, which has been shown to effectively model waveforms in the case of source separation and follows a feedforward convolutional architecture, making it resource efficient. The model takes as input the waveform of the mixture of sources, downsamples it through a series of convolutional operations to generate a low dimensional representation and then upsamples it through linear interpolation followed by convolution to the output dimensions. There are concatinative connections between the corresponding layers of the upsampling and downsampling blocks. In our work, we adapt this architecture to fit the desired use case.
% %falar do wavegan e do specgan que la tem
% However, if no conditioning or regularisation of the latent space is performed, generative networks do not provide an easy control to the user to characterise the desired sound. In \cite{esling2018bridging} uses perceptual based distances between different instrument timbre to regularise the latent space in musically relevant way. Conditioning strategies used in literature are pitch of the note \cite{engelNsynth,engel2018gansynth}, the instrument labels \cite{NeuralDrumMachine,advaudiosynth,manytomany} or the style of playing the instrument \cite{bitton}.
% The existing work for the generation of percussive sounds is not as vast as the one for musical instruments, as these approaches do not use drum sounds in their dataset. In \cite{NeuralDrumMachine}, the authors focus on this task . This latter architecture is used as the authors found that traditional methods such as Griffin-Lin were not reliable for percussive sounds. The control the user has on the timbre of generated sound is the percussive instrument, which is learned when conditioning the network, and a low dimensional sound obtained from the latent space through Principal Component Analysis. However, these latent spaces are learned by the AE and therefore do are not directly related to timbral perception.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Timbral features %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%ToDO: Talk about the gaussianes of the features
%High Level Characterisation of one-shot sounds
\section{Timbral features}
\label{sec:sem_feat}
For our end goal, we require semantically meaningful features that can allow for intuitive control of the synthesizer. In the field of Music Information Retrieval, a strong effort has been put on developing hand crafted features which can characterise sounds. These features enable users to retrieve sounds or music from large audio collections by automatically describing them according to their timbre, their mood, or other characteristics which are easy to understand by users. For our purpose, we need features pertaining to timbre. We understand timbre as pertaining to perceptual characteristics of sounds analogous to colour or quality. Control over such features would enable the user to intuitively shape sounds.
A set of such features have been proposed in~\cite{pearce2017timbral}, where recurrent query terms, related to timbral characteristics, used for searching sounds in large audio databases were identified. Regression models were developed by mapping user-collected ratings to timbral characteristics, which quantify semantic attributes. These are hardness, depth, brightness, roughness, boominess, warmth and sharpness.
The work proposes feature extractors pertaining to these query terms and we use an open-source implementation of the same\footnote{\url{https://github.com/AudioCommons/ac-audio-extractor}}. For the rest of this paper, we refer to the \num{7} features extracted by these extractors as timbral features.
% attributes that users normally search for have been identified from the search queries performed in Freesound.
% The most relevant attributes found were hardness, depth, brightness, roughness, boominess, warmth and sharpness.
% After this identification, a set of models for characterising these features was implemented, as detailed in~\cite{ac52},
% and made available in the Audio Commons Audio Extractor\footnote{\url{https://github.com/AudioCommons/ac-audio-extractor}}.
% Hence, they developed a set of timbral models\footnote{\url{https://github.com/AudioCommons/ac-audio-extractor}} by using regression modelling to predict some user-collected subjective ratings on these attributes. We use these models as feature extractors in our study.
Another relevant characteristic which is commonly present in drum synthesizers and music makers are used to work with is the temporal envelope of the sound. This feature describes the energy of the sounds over time and is normally available to users in drum synthesizers as a set of \textit{attack} and \textit{decay} controls. We use an open-source implementation of the envelope algorithm described in \cite{zolzer2008digital}, present in the Essentia library~\cite{essentia}. An attack time of \SI{5}{\milli\second} and a release time of \SI{50}{\milli\second} was used to generate a smooth curve which matched the sound energy over time.
It must be noted that the timbral features described previously are summary features, i.e. have a single value for each sound while the envelope is time evolving and of the same dimensions as the waveform.
%In some of the more recent synthesizers the user can have full control of the envelope by drawing the desired envelope over time.
% In order to characterise the sounds in our datasets, we used the Envelope function present in the Essentia library~\cite{essentia}. We used the default parameters for the algorithm, except the release and attack time of the first order low-pass filter for the corresponding phase. We empirically found that an attack time of \SI{5}{\milli\second} and a release time of \SI{50}{\milli\second} provided a smooth curve which matched the sound energy over time.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Datasets %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{DATASET CURATION}
\label{sec:dataset}
%Xavier can you do this? Just talk about the dataset we collected from freesound. I think the code is on drive. If you want me to do it I can do it as well
We curated two datasets in order to train our models in different scenarios.
The first consists of sounds taken from Freesound, a website which hosts a collaborative collection of Creative Commons licensed sounds\footnote{\url{https://freesound.org}}~\cite{font2013freesound}.
We performed queries to the database with the name of percussion instruments as keywords in order to retrieve a set of percussive sounds, with a limit on effective duration of \SI{1}{\second}.
We then conducted a manual verification of these sounds\footnote{We developed an annotation tool available at this repository \url{https://github.com/xavierfav/percussive-annotator}.}: to select the ones that were containing one single event, and were of appreciable quality in the context of traditional electronic music production.
% TODO: make the repo open, add license, add info to readme . use footnote
This process created a dataset of around \num{10000} sounds, containing instruments such as kicks, snares, cymbals and bells. The dataset is publicly available in a Zenodo repository\footnote{\url{https://zenodo.org/record/3665275}}. For the rest of this paper, we refer to this dataset as FREESOUND.
% TODO: link to the main repository of the percussive synth, add info in readme.
% This dataset will be made available upon acceptance of the paper.
A second dataset was created by aggregating about \num{5000} kick drum one-shot samples from our personal collections, originating mostly from commercial libraries.
This type of sounds are often of high quality, annotated and contain only one event which makes it very handy to construct a dataset of isolated sounds, suiting our needs for training our model in a restricted context. We refer to this dataset as KICKS.
The aim of creating two datasets was to understand if our method could be applicable for synthesising a wide variety of percussion sounds, or if it was more appropriate to focus on synthesising only one type of sounds, in this case the kick drum.
%The dataset will be made publicly available upon paper acceptance.
%cool looks nice
% I THINK HERE WE CAN SAY WHAT IS THE DIFFERENCE BETWEEN THE SETS - WE WANTED TO EVALUATE BOTH THE NETWORK ON CREATING ONE DRUM SOUND (KICK) AND VARIOUS DRUM SOUNDS.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Methodology %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methodology}
\label{sec:system}
We aim to model the probability distribution of the waveform ${x}$ as a function of the timbral features $fs$ and the time-domain envelope $e$. To this end we use a feedforward convolutional neural network as a function approximator to model $P(x|fs,e)$. We use a U-Net architecture, similar to the one used by \cite{stoller2018wave}, which has been shown to effectively model the waveform of an audio signal. Our network takes the envelope as input and concatenates to it the timbral features, $fs$, broadcast to the input dimensions, as done by \cite{oord2016wavenet}. As shown in Figure \ref{fig:archi}, downsampling is done via a series of convolutions with stride \num{2}, to produce a low-dimension embedding. We use a filter length of \num{5} and double the number of filters after each \num{3} layers, starting with \num{32} filters. A total of \num{15} layers are used in the encoder, leading to an embedding of size \num{512}.
We upsample this low dimensional embedding sequentially to the output $\hat{x}$, using linear interpolation followed by convolution. This mirrors the approach used by \cite{ chandna2019wgansing,stoller2018wave} and has been shown to avoid high frequency artefacts which appear while upsampling with transposed convolutions. As with the U-Net, there are connections between the corresponding layers of the encoder and decoder, as shown in Figure \ref{fig:archi}.
\begin{figure}[ht]
\centering
\includegraphics[width=0.5\textwidth]{perc_arch.pdf}
\caption{The proposed architecture, with $K=15$ layers.}
\label{fig:archi}
\end{figure}
We initially used a simple reconstruction loss function, shown in equation \ref{eq:loss1} to optimise the network.
\begin{equation}
\mathcal{L}_{recon} = \mathbb{E} [\|\hat{x} - x\|_1]
\label{eq:loss1}
\end{equation}
While this resulted in a decent output, we noticed that the network was able to reproduce the low frequency components of the desired sound, but lacked details in high frequency components. To rectify this, we added a short time fourier transform (\textit{STFT}) based loss, similar to \cite{sahai2019spectrogram}.
This loss is shown in equation \ref{eq:loss2}.
\begin{equation}
\mathcal{L}_{stft} = \mathbb{E} [\|STFT(\hat{x}) - STFT(x)\|_1]
\label{eq:loss2}
\end{equation}
The final loss of this network is shown in equation \ref{eq:loss3}.
\begin{equation}
\mathcal{L}_{final} = \mathcal{L}_{recon} + \lambda\mathcal{L}_{stft}
\label{eq:loss3}
\end{equation}
Where $\lambda$ is the weight given to the high frequency component of the reconstruction.
% We also tried adding the full STFT component to the loss, as shown in equation \ref{eq:loss4}
% \begin{equation}
% \begin{aligned}
% \mathcal{L}_{fullstft} = \mathbb{E} [\|STFT(\hat{x}) - STFT(x)\|_1] \\
% \mathcal{L}_{full} = \mathcal{L}_{recon} + \lambda_2\mathcal{L}_{fullstft}
% \end{aligned}
% \label{eq:loss4}
% \end{equation}
% As a result, we have three models, $M1$, which uses just $\mathcal{L}_{recon}$, $M2$, which uses $\mathcal{L}_{high}$ and $M3$, which is optimised to minimise $\mathcal{L}_{full}$. We evaluate all three models on their ability to reconstruct high quality audio, while matching the input timbral features.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Hyperparameters %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Evaluation %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}
\label{sec:results}
\subsection{Data Pre-processing}
\label{sec:feats}
All sound were downsampled to a sampling rate of \SI{16}{\kilo\hertz} and silences were removed from the beginning and end of the sounds. Following this, we calculated the timbral features and envelope described in section \ref{sec:sem_feat} and then zero-padded at the end of the sound to \num{16000} samples. The features were normalised using min-max normalisation, to ensure that the inputs were within the range \SIrange{0}{1}.
\subsection{Training the network}
The network was trained using the Adam optimiser \cite{kingma2014adam} for \num{2500} epochs with a batch size of \num{16}. We use \SI{90}{\percent} of the data for training and \SI{10}{\percent} for evaluation.
The \textit{STFT} used for the $\mathcal{L}_{stft}$ loss function is calculated over \num{1024} samples and a hopsize of \num{512}. With the given sampling rate, this led to a frequency resolution of \SI{16.125}{\hertz} per bin. We evaluate the model with three losses: the $\mathcal{L}_{recon}$ loss, henceforth referred to as WAVE; the $\mathcal{L}_{final}$, referred to as FULL; and a version with only the high frequency components of the \textit{STFT} for the $\mathcal{L}_{stft}$, referred as HIGH. This last model uses \textit{STFT} components above \SI{650}{\hertz} or \num{40} bins as traditional kick synthesizers model a kick sound via a low frequency sinusoid, generally below \SI{650}{\hertz} with some high frequency noise. We use $\lambda=0.5$ for our experiments.
\subsection{Evaluation}
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{outputs_kicks.png}
\caption{A sample of the input envelope and features and the output waveforms for the various models for the KICK dataset}
\label{fig:outputs_kicks}
\end{figure}
The proposed models need to be evaluated in terms of the perceived audio quality and the coherence of timbral features between the input and the output.
A preliminary assessment of the quality of reconstruction can be made by looking at the output waveforms, shown in Figure \ref{fig:outputs_kicks} for a sample from the test set of the KICKS dataset. Although the reconstruction seems to be visually accurate for the three models, the perceived quality of the audio is a subjective metric that cannot be judged by simply looking at the plots.
We can objectively assess the coherence of the timbral features used as input to the model. More importantly, we want to assess that a change in these features leads to a corresponding change in the output.
To this end, we vary each individual timbral feature while maintaining the other features constant. We then check the accuracy of the output waveform via the same feature extractors used for training. For each individual feature, we set values of \textit{low}, corresponding to \num{0.2} over the normalised scale, \textit{mid}, corresponding to \num{0.5} and \textit{high}, corresponding to \num{0.8}. The respective outputs for such models are termed $\hat{x}_{low}^i$, $\hat{x}_{mid}^i$ and $\hat{x}_{high}^i$ and their corresponding features are $fs_{low}^i$, $fs_{mid}^i$ and $fs_{high}^i$ for the $i^{th}$ feature.
For coherent modelling, the models should follow the order
$fs_{high}^i >fs_{mid}^i>fs_{low}^i$. We asses the accuracy of this order in three tests, $E1$, which checks the condition $fs_{high}^i>fs_{low}^i$, $E2$, which checks $fs_{high}^i>fs_{mid}^i$ and $E3$, which checks $fs_{mid}^i>fs_{low}^i$ over all values of $i$. The accuracy of the models over these tests is shown in Table \ref{table:models} and a feature wise summary is shown in Table \ref{table:feats}.
\begin{table}[h]
\centering
\begin{tabular}{l c c c c }
& & & Accuracy & \\
Dataset & Model & E1 & E2 & E3 \\ \hline
& WAVE & 0.601 & 0.569 & 0.552 \\
FREESOUND & HIGH & 0.649 & 0.601 & 0.657 \\
& \textbf{FULL} & \textbf{0.825} & \textbf{0.758} & \textbf{0.780} \\ \hline
& WAVE & 0.805 & 0.722 & 0.722 \\
KICKS & HIGH & 0.876 & 0.789 & 0.769 \\
& \textbf{FULL} & \textbf{0.920} & \textbf{0.814} & \textbf{0.798} \\
\end{tabular}
\caption{Objective verification of feature coherence across models and datasets.}
\label{table:models}
\end{table}
It can be seen that the FULL model, followed by the HIGH, are the most efficient at mapping the input features to the output waveform in terms of feature coherence, but all three models do maintain this coherence to a high degree.
\begin{table}[h]
\centering
\begin{tabular}{l | c c c | c c c }
& \multicolumn{3}{c|}{FREESOUND} & \multicolumn{3}{c}{KICKS} \\
Feature & E1 & E2 & E3 & E1 & E2 & E3 \\
\hline
Boominess & 0.98 & 0.82 & 0.98 & 0.96 & 0.86 & 0.95 \\
Brightness & 0.99 & 0.99 & 1.00 & 0.99 & 0.98 & 0.84 \\
Depth & 0.94 & 0.65 & 0.94 & 0.99 & 0.89 & 0.94 \\
Hardness & 0.64 & 0.66 & 0.59 & 0.85 & 0.61 & 0.79 \\
Roughness & 0.63 & 0.59 & 0.57 & 0.84 & 0.80 & 0.62 \\
Sharpness & 0.63 & 0.77 & 0.45 & 0.90 & 0.91 & 0.54 \\
Warmth & 0.92 & 0.79 & 0.91 & 0.88 & 0.61 & 0.87 \\
\end{tabular}
\caption{Objective verification of the accuracy on feature coherence for the best performing models for each dataset.}
\label{table:feats}
\end{table}
While feature coherence is maintained for features like boominess, brightness, depth and warmth for the full dataset, the models are less consistent in terms of hardness, roughness and sharpness, particularly true for the FREESOUND dataset.
Given the absence of a suitable baseline system, we decided to use an online AB listening test that compared the models amongst themselves and a reference for subjective evaluation of quality. The participants of the test were presented with \num{15} examples each from both datasets. Each example had two options, A and B from two of the models used for the dataset, along with a reference ground truth audio. There were \num{5} examples each from each of the \num{3} pairs. The participant was asked to choose the audio clip which was closest in quality to the reference audio. There were \num{35} participants in the listening test, the results of which are shown in Figure \ref{fig:surv_res}.
\begin{figure}[H]
\centering
\includegraphics[width=0.48\textwidth]{survey_ana.pdf}
\vspace{-1em}
\caption{Results of the listening test, displaying the user preference between loss functions for each of the datasets.}
\label{fig:surv_res}
\end{figure}
A clear preference for the HIGH model can be seen, especially for the KICKS dataset. This can be attributed partly to the choice of cutoff frequency used in the model and partly to the diversity of sounds in the FREESOUND dataset. We note the difficulty in assessing audio quality over printed text and encourage the user to visit our demo page and listen to the audio samples for assessment.
% We performed another rinted text and encourage the userevaluati on our deto listeomo page for assesn by comn to the sounds presentedputing the timbral features on a set of synthesised sounds.
% For both datasets, we selected a set of sounds that were not used during training the models. We analysed them for using their feature values as reference input parameters.
% Then, we synthesised sounds by changing the values of the input parameters individually. We used three types of values for each parameter:
% \textit{high}, \textit{mid} and \textit{low}, corresponding respectively to
% \num{0.8}, \num{0.5} and \num{0.2} for the normalized features.
% This process generated 23k sounds for the dataset of Freesound samples, and 11k kick samples for the other.
% Finally, we analysed these sounds with the timbral model extractor and checked that the values obtained for sounds synthesised with \textit{high} parameters were superior to those of the sounds generated with \textit{mid} and \textit{low} target parameters.
% Table XXX shows the number of time these comparisons were correct for the different datasets and loss functions used when training the models.
% TABLE OR SOMETHING SHOWING THE RESULTS
% Do you know what were the values for the timbral features referring to low, mid, high?
%low: 0.2
%mid: 0.5 or original?
%hight: 0.8
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Conclusion %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusions And Future Work}
\label{sec:conclusion}
In this work, we proposed a method using a feedforward convolutional neural network based on the Wave-U-Net~\cite{stoller2018wave} for synthesising percussive sounds conditioned on semantically meaningful features.
Our final aim is to create a system that can be controlled using high-level parameters, being semantically meaningful characteristics that correspond to concepts casual music makers are familiar with. To this end, we use hand crafted features designed by MIR experts and curate and present a dataset for the purpose of modelling percussive sounds. Via objective evaluation, we were able to verify that the control features do indeed modify the output waveform as desired and quality assessment was done via an online listening test.
Future work will focus on developing an interface for interacting with the synthesizer, which will allow to evaluate the approach in its context of use, with real users.
\vfill\pagebreak
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{refs}
\end{document}
\begin{thebibliography}{10}
\bibitem{tr808music}
Zainab Hasnain,
\newblock ``How the roland tr-808 revolutionized music,'' Apr 2017.
\bibitem{gans}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio,
\newblock ``Generative adversarial nets,''
\newblock in {\em Advances in neural information processing systems}, 2014, pp.
2672--2680.
\bibitem{vae}
Diederik~P. Kingma and Max Welling,
\newblock ``Auto-encoding variational bayes,''
\newblock in {\em 2nd International Conference on Learning Representations,
{ICLR} 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track
Proceedings}, 2014.
\bibitem{oord2016wavenet}
A{\"{a}}ron van~den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol
Vinyals, Alex Graves, Nal Kalchbrenner, Andrew~W. Senior, and Koray
Kavukcuoglu,
\newblock ``Wavenet: {A} generative model for raw audio,''
\newblock in {\em The 9th {ISCA} Speech Synthesis Workshop, Sunnyvale, CA, USA,
13-15 September 2016}, 2016, p. 125.
\bibitem{engelNsynth}
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi,
Douglas Eck, and Karen Simonyan,
\newblock ``Neural audio synthesis of musical notes with wavenet
autoencoders,''
\newblock in {\em Proceedings of the 34th International Conference on Machine
Learning - Volume 70}. 2017, ICML'17, pp. 1068--1077, JMLR.org.
\bibitem{chandna2019wgansing}
Pritish Chandna, Merlijn Blaauw, Jordi Bonada, and Emilia Gomez,
\newblock ``Wgansing: A multi-voice singing voice synthesizer based on the
wasserstein-gan,''
\newblock 2019.
\bibitem{NeuralDrumMachine}
Cyran Aouameur, Philippe Esling, and Ga{\"{e}}tan Hadjeres,
\newblock ``Neural drum machine : An interactive system for real-time
synthesis,''
\newblock 2019.
\bibitem{stoller2018wave}
Daniel Stoller, Sebastian Ewert, and Simon Dixon,
\newblock ``Wave-u-net: {A} multi-scale neural network for end-to-end audio
source separation,''
\newblock in {\em Proceedings of the 19th International Society for Music
Information Retrieval Conference, {ISMIR} 2018, Paris, France, September
23-27, 2018}, 2018, pp. 334--340.
\bibitem{font2013freesound}
Frederic Font, Gerard Roma, and Xavier Serra,
\newblock ``Freesound technical demo,''
\newblock in {\em Proceedings of the 21st ACM international conference on
Multimedia}. ACM, 2013, pp. 411--412.
\bibitem{shen2018natural}
Jonathan Shen, Ruoming Pang, Ron~J Weiss, Mike Schuster, Navdeep Jaitly,
Zongheng Yang, Zhifeng Chen, Yu~Zhang, Yuxuan Wang, Rj~Skerrv-Ryan, et~al.,
\newblock ``Natural tts synthesis by conditioning wavenet on mel spectrogram
predictions,''
\newblock in {\em 2018 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP)}. IEEE, 2018, pp. 4779--4783.
\bibitem{advaudiosynth}
Chris Donahue, Julian~J. McAuley, and Miller~S. Puckette,
\newblock ``Synthesizing audio with generative adversarial networks,''
\newblock 2018.
\bibitem{engel2018gansynth}
Jesse Engel, Kumar~Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue,
and Adam Roberts,
\newblock ``{GANS}ynth: Adversarial neural audio synthesis,''
\newblock in {\em International Conference on Learning Representations}, 2019.
\bibitem{esling2018bridging}
Philippe Esling and Adrien Bitton,
\newblock ``Bridging audio analysis, perception and synthesis with
perceptually-regularized variational timbre spaces.,''
\newblock in {\em Proceedings of the 19th International Society for Music
Information Retrieval Conference (ISMIR 2018)}, 2018.
\bibitem{manytomany}
Adrien Bitton, Philippe Esling, and Axel Chemla{-}Romeu{-}Santos,
\newblock ``Modulated variational auto-encoders for many-to-many musical timbre
transfer,''
\newblock 2018.
\bibitem{bitton}
Adrien Bitton, Philippe Esling, Antoine Caillon, and Martin Fouilleul,
\newblock ``Assisted sound sample generation with musical conditioning in
adversarial auto-encoders,''
\newblock 2019.
\bibitem{tolstikhin2017wasserstein}
Ilya~O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard
Sch{\"{o}}lkopf,
\newblock ``Wasserstein auto-encoders,''
\newblock in {\em 6th International Conference on Learning Representations,
{ICLR} 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track
Proceedings}, 2018.
\bibitem{pearce2017timbral}
Andy Pearce, Tim Brookes, and Russell Mason,
\newblock ``Timbral attributes for sound effect library searching,''
\newblock in {\em Audio Engineering Society Conference: 2017 AES International
Conference on Semantic Audio}, Jun 2017.
\bibitem{zolzer2008digital}
Udo Z{\"o}lzer,
\newblock {\em Digital audio signal processing}, vol.~9,
\newblock Wiley Online Library, 2008.
\bibitem{essentia}
Dmitry~Bogdanov et~al.,
\newblock ``Essentia: an audio analysis library for music information
retrieval,''
\newblock in {\em International Society for Music Information Retrieval
Conference (ISMIR{\textquoteright}13)}, Curitiba, Brazil, 04/11/2013 2013,
pp. 493--498.
\bibitem{sahai2019spectrogram}
Abhimanyu Sahai, Romann Weber, and Brian McWilliams,
\newblock ``Spectrogram feature losses for music source separation,''
\newblock 2019.
\bibitem{kingma2014adam}
Diederik~P. Kingma and Jimmy Ba,
\newblock ``Adam: {A} method for stochastic optimization,''
\newblock in {\em 3rd International Conference on Learning Representations,
{ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings}, 2015.
\end{thebibliography}