TTS-by-TTS 2: Data-selective augmentation for neural speech synthesis using ranking support vector machine with variational autoencoder
Extracted source text ยท arXiv 2206.14984
\documentclass[a4paper]{article}
\usepackage{INTERSPEECH2022}
\PassOptionsToPackage{hyphens}{url}\usepackage{hyperref}
\def\UrlBreaks{\do\/\do-}
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage[dvips]{epsfig}
\usepackage{tablefootnote}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{color}
\usepackage{enumitem}
% green red blue black magenta brown purple
% 2022.06.21 (EW) Let's use those keywords for the final revision (camera-ready)
\newcommand{\feedit}[1]{\textcolor{black}{#1}}
\newcommand{\fmedit}[1]{\textcolor{black}{#1}}
\newcommand{\fredit}[1]{\textcolor{black}{#1}}
\newcommand{\fsedit}[1]{\textcolor{black}{#1}}
\newcommand{\foedit}[1]{\textcolor{black}{#1}}
\newcommand{\fyedit}[1]{\textcolor{black}{#1}}
% do not remove
\newcommand{\reedit}[1]{\textcolor{black}{#1}}
\newcommand{\rmedit}[1]{\textcolor{black}{#1}}
\newcommand{\rredit}[1]{\textcolor{black}{#1}}
\newcommand{\rsedit}[1]{\textcolor{black}{#1}}
\newcommand{\roedit}[1]{\textcolor{black}{#1}}
\newcommand{\ryedit}[1]{\textcolor{black}{#1}}
% please avoid using following commands
%\newcommand{\eedit}[1]{\textcolor{black}{#1}}
%\newcommand{\medit}[1]{\textcolor{black}{#1}}
%\newcommand{\redit}[1]{\textcolor{black}{#1}}
%\newcommand{\sedit}[1]{\textcolor{black}{#1}}
%\newcommand{\oedit}[1]{\textcolor{black}{#1}}
%\newcommand{\yedit}[1]{\textcolor{black}{#1}}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
% \title{TTS-by-TTS 2: \medit{Selective Data} Augmentation for Neural Speech Synthesis Using Ranking Support Vector Machine with Variational Autoencoder}
\title{TTS-by-TTS 2: Data-Selective Augmentation for Neural Speech Synthesis Using Ranking Support Vector Machine with Variational Autoencoder}
%\title{TTS-by-TTS 2: Self-augmented speech synthesis with effective data selection}
\name{Eunwoo Song$^1$, Ryuichi Yamamoto$^{2}$, Ohsung Kwon$^1$, Chan-Ho Song$^1$, Min-Jae Hwang$^1$,\\ Suhyeon Oh$^1$, Hyun-Wook Yoon$^1$, Jin-Seob Kim$^1$, Jae-Min Kim$^1$}
%The maximum number of authors in the author list is twenty. If the number of contributing authors is more than twenty, they should be listed in a footnote or in acknowledgement section, as appropriate.
\address{
$^{1}$NAVER Corp., Seongnam, Korea\\
$^{2}$LINE Corp., Tokyo, Japan}
%\email{author@university.edu, coauthor@company.com}
\begin{document}
%\fontsize{9}{11}\selectfont
%\fontsize{8.7}{10.5}\selectfont
\fontsize{8.7}{10.6}\selectfont
\maketitle
%
\begin{abstract}
Recent advances in synthetic speech quality have enabled us to train text-to-speech (TTS) systems by using synthetic corpora.
%However, it has not been verified whether merely increasing the amount of synthetic data is always advantageous for improving training efficiency.
However, merely increasing the amount of synthetic data is \feedit{not} always advantageous for improving training efficiency.
Our aim in this study is to selectively choose synthetic data that are beneficial to the training process.
In the proposed method, we first adopt a variational autoencoder whose posterior distribution is utilized to extract latent features representing acoustic similarity between the recorded and synthetic corpora.
By using those learned features, we then train a ranking support vector machine (RankSVM) that is well known for effectively ranking relative attributes among binary classes.
By setting the recorded and synthetic ones as two opposite classes, RankSVM is used to determine how the synthesized speech is acoustically similar to the recorded data.
%Then, synthetic TTS data, which are similar to the recorded data, are selected from large-scale synthetic corpora.
Then, synthetic TTS data, \reedit{whose distribution is close} to the recorded data, are selected from large-scale synthetic corpora.
%By using these data for the TTS model training, the performance of the TTS model can be significantly improved.
%By using these data for the TTS model training, the \reedit{synthetic quality} can be significantly improved.
By using these data for \feedit{retraining} the TTS model, the \reedit{synthetic quality} can be significantly improved.
% Objective and subjective evaluation results verify that the proposed TTS model performs better than the original model trained with recorded data alone and the similarly configured system retrained with all the synthetic data without any selection method.
Objective and subjective evaluation results \reedit{show the superiority of the proposed method over the conventional methods}.
\end{abstract}
\noindent\textbf{Index Terms}: Speech synthesis, data augmentation, variational autoencoder, ranking support vector machine
\section{Introduction}
%
As the accuracy of acoustic modeling has increased following the revolution of deep neural networks, the synthetic quality of neural text-to-speech (TTS) systems has improved significantly \cite{yuxuan2017tacotron, jonathan2017natural, li2019close}.
However, these systems still have a major shortcoming in that \feedit{a lot of} training \feedit{corpora are required to learn} the complex \feedit{nature} of speech production \cite{chung2019semi}.
%they require numerous training examples to faithfully represent the complex mechanics of speech production \cite{chung2019semi}.
To overcome this limitation, various studies employing data augmentation techniques have been proposed.
For instance, Huybrechts et al. \cite{huybrechts2021low, ribeiro2022cross} proposed using a well-trained voice conversion model to extend the speaking style of the target speaker's TTS acoustic model;
Wu et al. \cite{wu2021relational} proposed a speaker similarity-based data selection method from other speakers' recordings for enlarging the training corpus of the TTS vocoding model.
While those methods require external datasets to augment target speakers' voices, Sharma et al. \cite{sharma2020strawnet} proposed to generate a large-scale synthetic corpus within the same speaker's model to distill the knowledge from the autoregressive (AR) WaveNet to the non-AR Parallel WaveNet.
Similar to Sharma's work, our previous work proposed a TTS-by-TTS model in which a large-scale synthetic corpus generated by a well-designed TTS model is used to improve the quality of other TTS models \cite{hwang2021tts}.
One of the key ideas of this work was to collect a large number of text scripts while maintaining the recording scripts' phoneme distribution.
This enabled the model to simulate various phoneme combinations, resulting in significantly improving the TTS model's stability with the unseen text.
%However, it has not been verified whether increasing the amount of the corresponding synthetic waveform is always advantageous for improving training efficiency, since this might cause side effects when poorly generated samples are included.
However, increasing the amount of the corresponding synthetic waveform is \feedit{not} always advantageous for improving training efficiency \cite{oh2022effective}, since this might cause \feedit{negative} effects \feedit{if} poorly generated \feedit{waveforms} are included.
%since this might cause side effects when poorly generated samples are included.
To address this problem, we propose a data-selective augmentation method for TTS systems.
From a large-scale synthetic corpus, the proposed method can selectively choose the training data whose acoustic distribution is similar to the recordings.
\feedit{Specifically}, we adopt a \textit{variational autoencoder} (VAE) as a reference encoder for the TTS acoustic model \cite{kingma2014auto}.
VAEs are well known for capturing latent representations of feature distribution and have been employed as style encoders of controllable and/or expressive TTS tasks \cite{akuzawa2018expressive, zhang2019learning, sun2020fully, kim2021conditional}.
%Similarly, our method utilizes the learned VAE's posterior distribution to capture acoustic similarities between the acoustic features extracted from the natural recordings and synthesized by the TTS models.
Similarly, our method utilizes \rredit{a VAE} to capture acoustic similarities between the acoustic features extracted from the natural recordings and synthesized by the TTS models.
We introduce a \textit{ranking support vector machine} (RankSVM) \cite{parikh2011relative}, which is well known for scoring relative attributes between binary classes.
%\feedit{Diverging from our previous work \cite{oh2022effective} that used OpnSMILE features \cite{eyben2010opensmile}}, we set feature vectors extracted from the learned VAE's posterior \rredit{distribution} for recorded and synthetic ones as two opposite classes.
% \feedit{Diverging from our previous work \cite{oh2022effective} that used OpenSMILE features \cite{eyben2010opensmile}, we employ} feature vectors extracted from the learned VAE's posterior \rredit{distribution} for recorded and synthetic ones.
\feedit{Diverging from our previous work \cite{oh2022effective} that used OpenSMILE features \cite{eyben2010opensmile}, we employ} \fredit{latent} feature vectors extracted from the learned VAE's posterior \rredit{distribution} for recorded and synthetic ones.
By setting \feedit{them} as two opposite classes, the RankSVM learns to determine \textit{originalities} that represent how the distribution of the synthetic waveform is acoustically \feedit{close} to that of the natural recordings.
% similar --> close
%On the basis of this originality, it is possible to selectively discard the synthetic samples with dissimilar characteristics to the recordings; therefore, the modeling accuracy of the TTS system retrained with the remaining samples becomes significantly improved.
On the basis of this originality, it is possible to selectively discard the synthetic \feedit{waveforms whose attributes are} dissimilar to the recordings; therefore, the modeling accuracy of the TTS system retrained with the remaining samples becomes significantly improved.
%We investigated the effectiveness of the proposed method by conducting objective and subjective evaluation tasks.
The objective and subjective evaluation results verified the \feedit{superiority of the proposed method over} the original system trained with recorded data alone and the similarly configured system retrained with all the synthetic data without any selection method.
\reedit{In particular, our method achieved 3.89 and 3.74 mean opinion score (MOS) for Tacotron~2 and FastSpeech~2 models, respectively, with 1,000 utterance\rredit{s} of limited training data.}
\section{TTS-driven data augmentation}
\label{sec:txt}
%%%%%%%%%%%%%%%%%%% Fig: block %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\begin{minipage}[b]{1.0\linewidth}
\centerline{\epsfig{figure=fig/block_tts.pdf,width=82mm}}
%\vspace*{-7pt}
\centerline{(a)} \medskip
\end{minipage}
\begin{minipage}[b]{1.0\linewidth}
\centerline{\epsfig{figure=fig/block_svm.pdf,width=80mm}}
\centerline{(b)} \medskip
\end{minipage}
\vspace*{-15pt}
\caption{Block diagrams of (a) the baseline TTS model and (b) the RankSVM model. The symbols $\mu$ and $\sigma^2$ denote the VAE statistics (i.e., the mean and variance of the posterior distribution, respectively) to sample the latent variable $z$.}
%\vspace*{-10pt}
\label{fig:block}
\end{figure}
%%%%%%%%%%%%%%%%%%% Fig: block %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Database}
\label{ssec:data}
The experiments used a phonetically balanced Korean corpus recorded by a Korean female professional speaker.
The speech signals were sampled at 24 kHz with 16-bit quantization.
In total, 1,000 (1.8 hours), 270 (0.5 hours), 130 (0.2 hours) utterances were used for training, validation, and testing, respectively.
\subsection{Baseline TTS model}
\label{ssec:baseline}
%To synthesize high-quality speech corpora, it is very important to design well-structured TTS models.
It is \feedit{crucial} to design \feedit{a} well-structured TTS \feedit{system} to synthesize high-quality speech \feedit{database}.
%Among various state-of-the-art models, we opted to pursue a VAE-Tacotron~2 model \cite{zhang2019learning} with a phoneme alignment approach \cite{okamoto2019tacotron} thanks to its stable generation and competitive synthetic quality, as described in Figure~\ref{fig:block}a.
Among various state-of-the-art models, we opted to pursue a VAE-Tacotron~2 model with a phoneme alignment approach \reedit{(Figure~\ref{fig:block}a)} thanks to its stable generation and competitive synthetic quality \cite{okamoto2019tacotron, song2020neural}.
%Following the previous study \reedit{in Zhang et al.} \cite{zhang2019learning}, the prior and posterior distributions of the VAE are modeled by Gaussian distributions.
Following the previous study \reedit{in Zhang et al.} \cite{zhang2019learning}, the prior and posterior distributions of the VAE are modeled by Gaussian distributions.
Note that the use of VAE is beneficial not only for giving variations to the synthetic sample itself but also for capturing the acoustic characteristics of the natural recordings and the synthetic waveforms.
Specifically, we leverage the statistics of the learned VAE's posterior distribution (i.e., mean and variance) to measure acoustic similarities, which will be discussed in Section~\ref{sec:proposed}.
In the case of the vocoding models, we adopt a Parallel WaveGAN vocoder \cite{ yamamoto2020parallel, song2021improved, yamamoto2021parallel} \feedit{based on a} multi-band harmonic-plus-noise \feedit{model} (MB-HN-PWG \cite{hwang2021high}).
%we adopt a multi-band harmonic-plus-noise Parallel WaveGAN (MB-HN-PWG) vocoder \cite{hwang2021high}. % yamamoto2020parallel, song2021improved, yamamoto2021parallel
\feedit{Specifically}, two separate WaveNet \feedit{generators} are trained to jointly \feedit{learn} the harmonic and noise \feedit{characteristics} of the speech, respectively.
%In this vocoder, two separated harmonic and noise WaveNets are trained to jointly capture the harmonic and noise components of speech signals, respectively.
% In this vocoder --> Specifically
\feedit{By employing} the target speech's periodicity in the multiple frequency bands, it is \feedit{possible to produce} speech \feedit{outputs} as naturally as recordings.
%As each WaveNet is designed to learn the target speech's periodicity in multiple frequency bands, it is able to synthesize speech samples as naturally as recordings.
\subsection{\feedit{Large-scale} data augmentation}
\label{ssec:augmentation}
% To prepare text scripts for data augmentation, we crawled the NAVER website\footnote{\url{https://news.naver.com}} for news articles \cite{hwang2021tts, oh2022effective} and obtained 80,000 text scripts that were 80 times larger than the training utterances.
To \feedit{collect} text scripts for \feedit{synthesizing the large-scale speech waveforms}, we crawled news articles \feedit{from} the NAVER website\footnote{\url{https://news.naver.com}} \cite{hwang2021tts, oh2022effective} and \feedit{prepared} 80,000 text scripts that were 80 times larger than the training utterances.
% Using the prepared text scripts
Using \feedit{these} text scripts, the pretrained TTS model described in the previous section generates\footnote{
In the generation step, we condition the TTS acoustic model on VAE by using the centroid of latent vectors computed over all the training data \cite{um2020emotional}.}
\feedit{the corresponding 80,000} speech waveforms.
% large-scale speech waveforms.
\section{RankSVM-based data selection}
\label{sec:proposed}
Increasing the number of text scripts is beneficial for the target TTS model to learn the distribution of various phoneme combinations, enabling the model to generate more stable speech from the unseen text input \cite{hwang2021tts}.
However, we must carefully increase the number of corresponding synthetic waveforms to exclude poorly synthesized samples before retraining the target TTS model \cite{oh2022effective}.
To achieve this, we propose a RankSVM-based data selection method composed of three sub-steps: VAE fine-tuning, RankSVM training, and data selection.
%%%%%%%%%%%%%%%%%%% Fig: block %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/tsne_before.pdf,width=39mm}}
\vspace*{-5pt}
\centerline{(a)} \medskip
\end{minipage}
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/tsne_after.pdf,width=39mm}}
\vspace*{-5pt}
\centerline{(b)} \medskip
\end{minipage}
\vspace*{-7pt}
%\caption{Distribution of VAE's posterior mean vector extracted from recorded (red dot) and synthetic (blue triangle) samples: (1) before and (2) after applying the VAE fine-tuning process.}
%\caption{Distribution of VAE's posterior mean vector extracted from recorded (red dot) and synthetic (blue triangle) samples: \rredit{(a)} before and \rredit{(b)} after applying the VAE fine-tuning process.}
\caption{Distribution of \fredit{the} VAE's posterior mean vector extracted from recorded (red dot) and synthetic (blue triangle) samples: \rredit{(a)} before and \rredit{(b)} after applying the VAE fine-tuning process.}
\label{fig:vae}
\vspace*{-6pt}
\end{figure}
%%%%%%%%%%%%%%%%%%% Fig: block %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{VAE fine-tuning}
\label{ssec:vae}
To effectively train the RankSVM, it is very important to extract VAE statistics that adequately represent the distributions between the recorded and synthetic waveforms.
%However, the pretrained TTS model learned through the recorded database only, and it is difficult for the VAE model to capture the synthetic database's distribution.
However, the pretrained TTS model \rredit{is} learned through the recorded database only, and it is difficult for the VAE model to capture the synthetic database's distribution.
To address this problem, we fine-tune the pretrained TTS model by using the TTS corpus combined with both recorded and synthetic corpora.
Because both recorded and synthetic acoustic features are now included for training, the fine-tuned VAE model becomes aware of their acoustic distribution\footnote{
The fine-tuned VAE is only used as a feature extractor of the RankSVM model. The large-scale data augmentation described in Section~\ref{sec:txt} is performed by the pretrained model without the fine-tuning process.}.
Figure~\ref{fig:vae} describes the VAE's statistics approximated in the t-distributed stochastic neighbor embedding (t-SNE) space extracted from the test set \cite{van2008visualizing}.
It shows that applying the fine-tuning process (Figure~\ref{fig:vae}b) represents the feature distributions better, which is expected to be beneficial for training the RankSVM model.
%%%%%%%%%%%%%%%%%%% Fig: histogram %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\centering
\includegraphics[width=0.7\linewidth]{fig/histogram.pdf}
\vspace*{-7pt}
\caption{\small
Density histograms of originality determined using RankSVM from the recorded (red boxes) and synthetic (blue boxes) waveforms, respectively.
}
\label{fig:histogram}
\vspace*{-6pt}
\end{figure}
%%%%%%%%%%%%%%%%%%% Fig: histogram %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{RankSVM}
\label{ssec:rankingsvm}
RankSVM is well known for learning relative attributes from two different classes \cite{parikh2011relative}.
As shown in Figure~\ref{fig:block}b, the proposed method sets the VAE feature vectors obtained from recorded and synthetic ones as two opposite classes, respectively.
%Then, the learned ranking function can be used to estimate a real-valued score as the \textit{originality}, which is defined as how the distribution of the synthetic waveform is acoustically similar to that of the natural recordings.
%Then, \feedit{a} ranking function \feedit{is learned} to \feedit{determine} the \textit{originality}, \feedit{a} real-valued score defined as how the distribution of the synthetic waveform is acoustically similar to that of the natural recordings.
Then, \feedit{a} ranking function \feedit{is learned} to \feedit{determine} the \textit{originality}, \feedit{a} real-valued score defined as how the distribution of the synthetic \fredit{waveforms} is acoustically similar to that of the natural recordings.
%\subsubsection{Training criteria}
\subsubsection{Ranking function}
% RY 03/24: we can write more explicit formulations like w = \argmin ... (if we have enough space..)
Assuming we have a training set $T = \{t\}$ represented in $\mathbb{R}^n$ by VAE features $\{\bm{x}_t\}$, and $T = N \cup M$, where $N$ and $M$ denote the recorded and synthetic set, respectively.
The goal of RankSVM is to learn a ranking function defined as follows \cite{deng2014linear}:
\begin{align}
r(\bm{x}_t) = \bm{w}^{T}\bm{\bm{x}_t}, \label{eq:rankingfunction}
\end{align}
with the following constraints:
\begin{align}
\forall (i,j) \in O &: \bm{w}^{T}\bm{\bm{x}_i} > \bm{w}^{T}\bm{\bm{x}_j}, \\
\forall (i,j) \in S &: \bm{w}^{T}\bm{\bm{x}_i} = \bm{w}^{T}\bm{\bm{x}_j},
\end{align}
where $\bm{w}$, $O$, and $S$ denote the weight vector of the ranking function, ordered set, and similar paired set, respectively. Specifically, the ordered set represents pairs of recorded and synthetic samples, and the similar set represents either pairs of synthetic samples or pairs of recorded samples.
Parikh et al. \cite{parikh2011relative} proposed to solve the problem using Newton's method \cite{chapelle2007training}.
However, this is not practical for large-scale data augmentation because of its quadratic computational complexity with regard to the number of training samples. Therefore, we adapted a stochastic gradient decent to accelerate the training \cite{shalev2011pegasos}.
%Once we obtain an optimal $\bm{w}$, the originality of unseen data is calculated by Equation (\ref{eq:rankingfunction}).
Once we obtain an optimal $\bm{w}$, the originality of \rredit{recorded and synthetic samples} is calculated by Equation (\ref{eq:rankingfunction}).
%Note that the originality scores are normalized to $[0, 1]$ for convenience, the results of which are described in Figure~\ref{fig:histogram}.
Note that the originality scores are normalized to $[0, 1]$ for convenience.
%%%%%%%%%%%%%%% Table: analysis %%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[!t]
\begin{center}
\caption{
Distortions with a 95\% confidence interval measured from two different groups: synthetic samples that have high 10\% and low 10\% originalities, respectively.
}
%\vspace*{-8pt}
\label{table:analysis}
{\small
\begin{tabular}{ccc}
\Xhline{2\arrayrulewidth}
Range of originality & F0 RMSE (Hz) & LSD (dB)
\\ \hline
%\medit{\textbf{Higher 10\%}} & \medit{\textbf{26.66$\pm$5.18}} & \medit{\textbf{3.87$\pm$0.18}} \\
High 10\% & 26.66$\pm$5.18 & 3.87$\pm$0.18 \\
Low 10\% & 32.28$\pm$5.96 & 4.01$\pm$0.16 \\
\Xhline{2\arrayrulewidth}
\end{tabular}}
\end{center}
\vspace*{-12pt}
\end{table}
%%%%%%%%%%%%%%% Table: analysis %%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Relationship between originality and acoustic similarity}
To further verify the relationship between the synthetic sample's originality and its acoustic similarity with the natural recordings, we analyzed distortions in the generated acoustic features.
In detail, we collected two different synthetic groups from the test set.
One group consisted of synthetic samples that had high 10\% originalities, whereas the other group consisted of synthetic samples that had low 10\% originalities. Then, objective metrics, such as F0 root mean square error (RMSE) and log spectral distance (LSD), were measured from each group.
Table~\ref{table:analysis} shows the test results.
It can be seen that the first group has much smaller generation distortions, which confirms that the synthetic waveforms with high originality have acoustically close characteristics to the natural recordings.
\subsection{Data selection}
\label{ssec:selection}
On the basis of the synthetic waveform's originality, we selectively discard the samples (e.g., those with low originality) that have dissimilar characteristics with the recordings.
The entire acoustic model described in Section~\ref{ssec:baseline} is then retrained by using the remaining samples (e.g., those with high originality) together with the original recordings.
Note that the proposed task is not applied to the vocoding model since it has been reported that using large sizes of training data is not \feedit{important} for \feedit{training the vocoding model} \cite{okamoto2020realtime}.
%Note that the proposed task is not applied to the vocoding model since it has been reported that using large sizes of training data is not crucial for the vocoder \cite{okamoto2020realtime}.
% --> important
\feedit{Consistently}, our previous work \feedit{also} confirmed \feedit{that including the} augmented database \feedit{to the training process was not that effective to improve the vocoding} quality \cite{hwang2021tts}.
%In addition, in our previous work, we could not confirm quality improvements when the vocoder was trained by an augmented database \cite{hwang2021tts}.
\section{Experiments}
\label{sec:experiment}
\subsection{TTS model details}
\subsubsection{Feature extraction}
% The linguistic features were composed of 330 binary features for categorical contexts (e.g., phoneme identity, etc.) and 24 features for numerical contexts (e.g., the phoneme's position information, etc.), all of which constituted a 354-dimensional feature vector.
The linguistic features \feedit{consisted of} 354-dimensional feature vectors \feedit{containing} 330 binary features for categorical contexts and 24 \feedit{additional} features for numerical contexts.
%The corresponding acoustic features were extracted using the improved time-frequency trajectory excitation vocoder at analysis intervals of 5 ms \cite{song2017effective}, and these features included 40-dimensional line spectral frequencies, F0, gain, binary voicing flag (v/uv), a 32-dimensional slowly evolving waveform, and a 4-dimensional rapidly evolving waveform, all of which constituted a 79-dimensional feature vector.
The corresponding acoustic features were extracted using the improved time-frequency trajectory excitation vocoder at \feedit{every} 5 ms frame intervals \cite{song2017effective}.
\feedit{Note that the feature dimension was 79 containing} 40-dimensional line spectral frequencies, F0, gain, binary voicing flag (v/uv), a 32-dimensional slowly evolving waveform, and a 4-dimensional rapidly evolving waveform.
\subsubsection{Acoustic model}
We adopted a VAE-Tacotron~2 model with a phoneme alignment approach \cite{zhang2019learning, okamoto2019tacotron}.
%As shown in Figure~\ref{fig:block}a, the reference acoustic features were first fed into the VAE reference encoder composed of six convolutional layers followed by a GRU layer \cite{wang2018style}.
As shown in Figure~\ref{fig:block}a, the reference acoustic features were first fed into the VAE reference encoder composed of six convolutional layers followed by a \rredit{gated recurrent unit} layer \cite{wang2018style}.
% Its output vector was then passed through two separate fully connected (FC) layers to generate the VAE statstics.
Its output vector was then passed through two separate fully connected (FC) layers to generate the VAE \feedit{mean and variance, respectively}.
Finally, the latent variables\footnote{
They were simply added to the final layers of the duration model and the Tacotron~2 encoder, respectively. Note that two separated FC layers were used for projecting latent variables to each model.}
%were sampled via the reparameterization method \cite{zhang2019learning}.
were sampled via the reparameterization method \rredit{\cite{kingma2014auto}}.
%Having input linguistic features, the duration model predicted phoneme-level duration through three FC layers followed by a unidirectional LSTM network.
Having input linguistic features, the duration model predicted phoneme-level duration through three FC layers followed by a unidirectional \rredit{long short-term memory (LSTM)} network.
The encoder transformed the same linguistic features into high-level context vectors and these were upsampled to the frame \feedit{resolution}.
% level --> resolution
\feedit{Note that} the encoder \feedit{consisted of} three convolution layers, a \feedit{single} bi-directional LSTM network, and a single FC layer.
%The encoder here was composed of three convolution layers, a bi-directional LSTM network, and a single FC layer.
The decoder was used to generate the output acoustic features.
First, the previously generated acoustic features were fed into the PreNet \cite{jonathan2017natural}, and those features and the context vectors from the encoder were then passed through two unidirectional LSTM layers, followed by two projection layers.
Finally, residual elements of the generated acoustic features \feedit{were passed through the} PostNet \cite{jonathan2017natural} to improve generation accuracy.
%Before training, the input and output features were normalized to have zero mean and unit variance.
\feedit{Both} the input \feedit{linguistic} and output \feedit{acoustic} features were normalized to have zero mean and unit variance.
%The weights were initialized using Xavier initialization, and RAdam optimization was used \cite{xavier2010init, liu2020radam}.
\feedit{Xavier initializer \cite{xavier2010init} and RAdam optimizer \cite{liu2020radam} were used for initializing and updating the weights, respectively.}
More setup details are given in our previous work \cite{hwang2021tts}.
\subsubsection{Vocoding Model}
The setups for training the MB-HN-PWG model followed our previous work in \cite{hwang2021high}.
The harmonic WaveNet \feedit{consisted of} 20 dilated residual blocks with two exponentially increasing dilation cycles.
\feedit{On the other hand}, the noise WaveNet \feedit{consisted of} 10 residual blocks with one exponentially increasing dilation cycle.
% whereas --> on the other hand
In each WaveNet, \feedit{16 band-pass filters parameterized by windowed sinc functions with 255 filter taps were applied to divide the frequency bands.}
The model was trained for 400K steps with RAdam optimizer \cite{liu2020radam}.
%The discriminator was fixed for the first 100K steps, and both the generator and the discriminator were jointly trained afterwards \cite{hwang2021high}.
\fredit{The discriminator weights were fixed} for the first 100K steps, and both the generator and the discriminator were jointly trained \fredit{for the rest 300K steps} \cite{hwang2021high}.
%%%%%%%%%%%%%%% fig: obj %%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/d_f0.pdf,width=42mm}}
\vspace*{-5pt}
\centerline{(a)} \medskip
\vspace*{-5pt}
\end{minipage}
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/d_spec.pdf,width=42mm}}
\vspace*{-5pt}
\centerline{(b)} \medskip
\vspace*{-5pt}
\end{minipage}
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/d_gain.pdf,width=42mm}}
\vspace*{-5pt}
\centerline{(c)} \medskip
\end{minipage}
\begin{minipage}[b]{.49\linewidth}
\centerline{\epsfig{figure=fig/d_vuv.pdf,width=42mm}}
\vspace*{-5pt}
\centerline{(d)} \medskip
\end{minipage}
\vspace*{-10pt}
\caption{Objective evaluation results with respect to various amounts of augmented data used in the retraining process: the dashed red line represents the results of the baseline model learned with recorded data alone.}
\label{fig:obj}
%\vspace*{-10pt}
\end{figure}
%%%%%%%%%%%%%%% fig: obj %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%% Fig: mos %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[!t]
\centering
\includegraphics[width=0.72\linewidth]{fig/mos_all.pdf}
\vspace{-3mm}
\caption{\small
The MOS test results with respect to various amounts of recorded data: baseline model trained with recorded data alone (red square) and augmented models trained without (purple triangle) and with (blue dot) the proposed data selection method.
}
\label{fig:mos}
\vspace*{-12pt}
\end{figure}
%%%%%%%%%%%%%%%%%%% Fig: mos %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Objective \feedit{and subjective} evaluations}
\label{ssec:eval}
% To evaluate the performance of the proposed system, the distortions in the acoustic parameters obtained from the original speech were compared with those estimated by the TTS models.
To evaluate the performance of the proposed system, \feedit{we measured} the distortions \feedit{between} the acoustic \feedit{features} obtained from the original \feedit{recording} and \feedit{generated} by the TTS models.
The metrics for measuring distortion were F0 RMSE, LSD, gain RMSE, and v/uv error rate.
Figure~\ref{fig:obj} shows the objective evaluation results with respect to various amounts of augmented data sets used in the retraining process (10K, 20K, ..., 80K utterances).
The findings can be analyzed as follows:
(1) The distortions decreased when the number of augmentation databases increased to some amount (about 40K utterances) because the modeling accuracy was also improved, depending on the amount of training databases.
(2) However, distortions increased when the model was trained with more than 50K augmented utterances since the amount of poorly generated data increased as well. This implies the importance of data-selective augmentation in the TTS task, which will be further analyzed in the following subjective evaluations.
%To evaluate the perceptual quality of the proposed system, five-scale naturalness MOS listening tests were performed
To evaluate the perceptual quality of the proposed system, \feedit{we performed} five-scale naturalness MOS listening tests\footnote{
Generated audio samples are available at the following URL:\\ \url{https://sewplay.github.io/demos/txt2/}}.
\feedit{In the test}, twenty native Korean listeners were asked to make quality judgments about 20 randomly selected samples from the test set.
% For comparison, we included the FastSpeech (FS)~2 model \cite{ren2020fastspeech2}.
For comparison, we included the FastSpeech (FS)~2\fredit{-based TTS system} \cite{ren2020fastspeech2}.
%The structure was similar to the Tacotron~2-based baseline TTS system, but only the acoustic model was replaced with non-AR transformer networks.
%The structure was similar to the Tacotron~2-based baseline TTS system, but only the acoustic model was replaced with non-AR \rredit{Transformer} networks.
The structure was similar to the Tacotron~2-based baseline TTS system, but only the acoustic model was replaced with \fredit{the} non-AR \rredit{Transformer} networks \cite{ren2019fastspeech}.
The detailed setup for conducting the FS~2 model followed those in our previous work \cite{hwang2021tts}.
Table~\ref{table:subj} shows the MOS evaluation results for the TTS systems with respect to the different training conditions, and the analysis can be summarized as follows:
(1) In both the AR and non-AR models, the systems retrained with the synthetic corpus performed better than those trained with recorded data alone.
%This confirms that the augmentation was beneficial for improving the perceptual quality of the synthesized speech.
\feedit{It means} that \feedit{increasing the training data with the augmented wavefroms} was \feedit{effective} for \feedit{enhancing} the \feedit{synthetic} quality.
(2) Among the systems with augmentations, the proposed systems with the data-selective augmentation method (Tests 3 and 6) performed better than the systems without any selection method (Tests 2 and 5, respectively).
As poorly generated synthetic samples were discarded based on originality, the proposed TTS systems synthesized more natural speech.
(3) Consequently, our system achieved 3.89 and 3.74 MOS for the Tacotron~2 and FS~2 models, respectively, when the recorded data for training the model were limited to 1,000 utterances.
\subsection{\feedit{Additional experiments with enough recordings}}
\label{ssec:add_eval}
%To verify the effectiveness of the proposed method in a large number of training database conditions, additional MOS tests were conducted by changing \textit{the number of recorded datasets} from 1K to 10K utterances.
To \feedit{further confirm} the effectiveness of the proposed method \feedit{under the} condition of \feedit{the enough recordings}, \feedit{we} conducted additional MOS tests by changing \textit{the number of recorded datasets} from 1K to 10K utterances.
% In each case, we trained the baseline VAE-Tacotron~2 model, augmented 80K utterances, selected 40K utterances by using the VAE-RankSVM model, and retrained the target TTS model.
In each case, we trained the baseline VAE-Tacotron~2 model, augmented 80K utterances, selected 40K utterances by using \fredit{\feedit{the} fine-turned VAE with \feedit{the} RankSVM} model, and retrained the target TTS model.
As shown Figure~\ref{fig:mos}, the perceptual quality was improved as the amount of recorded data increased.
Although the gap between the proposed and conventional methods was reduced as the synthetic quality was also improved, the data-selective augmentation method was still effective, even when the size of the source database increased enough.
%%%%%%%%%%%%%%% Table: system %%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[!t]
%\vspace*{-5pt}
\begin{center}
\caption{The MOS test results with a 95\% confidence interval: the system trained with the proposed method is shown in boldface. Note that the number of recorded utterances was 1K, and that of augmented utterances was represented as $M$.}
%\vspace*{-10pt}
\label{table:subj}
{\small
\begin{tabular}{ccccc}
\Xhline{2\arrayrulewidth}
%System & Model & $M$ & Data selection & MOS \\ \hline
\multicolumn{1}{c}{\multirow{2}{*}{System}} & Model & \multicolumn{1}{c}{\multirow{2}{*}{$M$}} & Data & \multicolumn{1}{c}{\multirow{2}{*}{MOS}}\\
\multicolumn{1}{c}{} & type & \multicolumn{1}{c}{} & selection & \multicolumn{1}{c}{} \\ \hline
Test 1 & Tacotron 2 & - & - & 3.35$\pm$0.11 \\
Test 2 & Tacotron 2 & 80K & No & 3.66$\pm$0.11 \\
\textbf{Test 3} & \textbf{Tacotron 2} & \textbf{40K} & \textbf{Yes} &\textbf{3.89$\pm$0.09} \\
Test 4 & FastSpeech 2 & - & - &3.30$\pm$0.12 \\
Test 5 & FastSpeech 2 & 80K & No &3.61$\pm$0.09 \\
\textbf{Test 6} & \textbf{FastSpeech 2} & \textbf{40K} & \textbf{Yes} &\textbf{3.74$\pm$0.08} \\ \hline
Recording & - & - & - & 4.29$\pm$0.08 \\
\Xhline{2\arrayrulewidth}
\end{tabular}}
\end{center}
\vspace*{-18pt}
%\vspace*{-6pt}
\end{table}
%%%%%%%%%%%%%%% Table: system %%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
We proposed a TTS-driven data-selective augmentation \feedit{technique}.
From the \feedit{large-scale synthetic} corpora, \rredit{a} RankSVM with VAE's posterior distribution determined the originality \feedit{that represents how the acoustic characteristics} of the generated speech \feedit{was similar to those of the natural recordings}.
By selectively including the \feedit{synthetic} data \rredit{with} the \feedit{recorded one}, the performance \feedit{of the} retrained TTS system \feedit{has been improved significantly}.
\feedit{Future works include to extend the proposed method to augment other speaking styles for the emotional and/or expressive TTS models.}
%\textbf{Acknowledgements}:
%This work was supported by Clova Voice, NAVER Corp., Seongnam, Korea.
% will be added after acceptance
\section{Acknowledgment}
This work was supported by Clova Voice, NAVER Corp., Seongnam, Korea.
\bibliographystyle{IEEEtran}
\bibliography{mybib}
\end{document}
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{yuxuan2017tacotron}
Y.~Wang, R.~Skerry-Ryan, D.~Stanton, Y.~Wu, R.~Weiss, N.~Jaitly, Z.~Yang,
Y.~Xiao, Z.~Chen, S.~Bengio, Q.~Le, Y.~Agiomyrgiannakis, R.~Clark, and
R.~Saurous, ``Tacotron: {T}owards end-to-end speech synthesis,'' in
\emph{Proc. Interspeech}, 2017, pp. 4006--4010.
\bibitem{jonathan2017natural}
J.~{Shen}, R.~{Pang}, R.~J. {Weiss}, M.~{Schuster}, N.~{Jaitly}, Z.~{Yang},
Z.~{Chen}, Y.~{Zhang}, Y.~{Wang}, R.~{Skerrv-Ryan}, R.~A. {Saurous},
Y.~{Agiomvrgiannakis}, and Y.~{Wu}, ``Natural {TTS} synthesis by conditioning
{WaveNet} on {Mel} spectrogram predictions,'' in \emph{Proc. ICASSP}, 2018,
pp. 4779--4783.
\bibitem{li2019close}
N.~Li, S.~Liu, Y.~Liu, S.~Zhao, M.~Liu, and M.~T. Zhou, ``Neural speech
synthesis with {T}ransformer network,'' in \emph{Proc. AAAI}, 2019, pp.
6706--6713.
\bibitem{chung2019semi}
Y.-A. Chung, Y.~Wang, W.-N. Hsu, Y.~Zhang, and R.~Skerry-Ryan,
``Semi-supervised training for improving data efficiency in end-to-end speech
synthesis,'' in \emph{Proc. ICASSP}, 2019, pp. 6940--6944.
\bibitem{huybrechts2021low}
G.~Huybrechts, T.~Merritt, G.~Comini, B.~Perz, R.~Shah, and J.~Lorenzo-Trueba,
``Low-resource expressive text-to-speech using data augmentation,'' in
\emph{Proc. ICASSP}, 2021, pp. 6593--6597.
\bibitem{ribeiro2022cross}
M.~S. Ribeiro, J.~Roth, G.~Comini, G.~Huybrechts, A.~Gabrys, and
J.~Lorenzo-Trueba, ``Cross-speaker style transfer for text-to-speech using
data augmentation,'' in \emph{Proc. ICASSP (in press)}, 2022.
\bibitem{wu2021relational}
Y.-C. Wu, C.-H. Hu, H.-S. Lee, Y.-H. Peng, W.-C. Huang, Y.~Tsao, H.-M. Wang,
and T.~Toda, ``Relational data selection for data augmentation of
speaker-dependent multi-band {MelGAN} vocoder,'' in \emph{Proc. Interspeech},
2021, pp. 3630--3634.
\bibitem{sharma2020strawnet}
M.~Sharma, T.~Kenter, and R.~Clark, ``Straw{N}et: Self-training {W}ave{N}et for
{TTS} in low-data regimes,'' in \emph{Proc. Interspeech}, 2020, pp.
3550--3554.
\bibitem{hwang2021tts}
M.-J. Hwang, R.~Yamamoto, E.~Song, and J.-M. Kim, ``{TTS-by-TTS}: {TTS}-driven
data augmentation for fast and high-quality speech synthesis,'' in
\emph{Proc. ICASSP}, 2021, pp. 6598--6602.
\bibitem{oh2022effective}
S.~Oh, O.~Kwon, M.-J. Hwang, J.-M. Kim, and E.~Song, ``Effective data
augmentation methods for neural text-to-speech systems,'' in \emph{Proc.
ICEIC}, 2022, pp. 1--4.
\bibitem{kingma2014auto}
D.~P. Kingma and M.~Welling, ``Auto-encoding variational bayes,'' in
\emph{Proc. ICLR}, 2014.
\bibitem{akuzawa2018expressive}
K.~Akuzawa, Y.~Iwasawa, and Y.~Matsuo, ``Expressive speech synthesis via
modeling expressions with variational autoencoder,'' in \emph{Proc.
Interspeech}, 2018, pp. 3067--3071.
\bibitem{zhang2019learning}
Y.-J. Zhang, S.~Pan, L.~He, and Z.-H. Ling, ``Learning latent representations
for style control and transfer in end-to-end speech synthesis,'' in
\emph{Proc. ICASSP}, 2019, pp. 6945--6949.
\bibitem{sun2020fully}
G.~Sun, Y.~Zhang, R.~J. Weiss, Y.~Cao, H.~Zen, and Y.~Wu, ``Fully-hierarchical
fine-grained prosody modeling for interpretable speech synthesis,'' in
\emph{Proc. ICASSP}, 2020, pp. 6264--6268.
\bibitem{kim2021conditional}
J.~Kim, J.~Kong, and J.~Son, ``Conditional variational autoencoder with
adversarial learning for end-to-end text-to-speech,'' in \emph{Proc. PMLR},
2021, pp. 5530--5540.
\bibitem{parikh2011relative}
D.~Parikh and K.~Grauman, ``Relative attributes,'' in \emph{Proc. ICCV}, 2011,
pp. 503--510.
\bibitem{eyben2010opensmile}
F.~Eyben, M.~W{\"o}llmer, and B.~Schuller, ``Open{SMILE} - {T}he {M}unich
versatile and fast open-source audio feature extractor,'' in \emph{Proc.
ACM}, 2010, pp. 1459--1462.
\bibitem{okamoto2019tacotron}
T.~Okamoto, T.~Toda, Y.~Shiga, and H.~Kawai, ``Tacotron-based acoustic model
using phoneme alignment for practical neural text-to-speech systems,'' in
\emph{Proc. ASRU}, 2019, pp. 214--221.
\bibitem{song2020neural}
E.~Song, M.-J. Hwang, R.~Yamamoto, J.-S. Kim, O.~Kwon, and J.-M. Kim, ``Neural
text-to-speech with a modeling-by-generation excitation vocoder,'' in
\emph{Proc. Interspeech}, 2020, pp. 3570--3574.
\bibitem{yamamoto2020parallel}
R.~Yamamoto, E.~Song, and J.~Kim, ``Parallel {W}ave{GAN}: A fast waveform
generation model based on generative adversarial networks with
multi-resolution spectrogram,'' in \emph{Proc. ICASSP}, 2020, pp. 6199--6203.
\bibitem{song2021improved}
E.~Song, R.~Yamamoto, M.-J. Hwang, J.-S. Kim, O.~Kwon, and J.-M. Kim,
``Improved {P}arallel {W}ave{GAN} vocoder with perceptually weighted
spectrogram loss,'' in \emph{Proc. SLT}, 2021, pp. 470--476.
\bibitem{yamamoto2021parallel}
R.~Yamamoto, E.~Song, M.-J. Hwang, and J.-M. Kim, ``Parallel waveform synthesis
based on generative adversarial networks with voicing-aware conditional
discriminators,'' in \emph{Proc. ICASSP}, 2021, pp. 6039--6043.
\bibitem{hwang2021high}
M.-J. Hwang, R.~Yamamoto, E.~Song, and J.-M. Kim, ``High-fidelity {P}arallel
{W}ave{GAN} with multi-band harmonic-plus-noise model,'' in \emph{Proc.
Interspeech}, 2021, pp. 2227--2231.
\bibitem{um2020emotional}
S.-Y. Um, S.~Oh, K.~Byun, I.~Jang, C.~Ahn, and H.-G. Kang, ``Emotional speech
synthesis with rich and granularized control,'' in \emph{Proc. ICASSP}, 2020,
pp. 7254--7258.
\bibitem{van2008visualizing}
L.~van~der Maaten and G.~Hinton, ``Visualizing data using {t-SNE},''
\emph{Journal of Machine Learning Research}, vol.~9, pp. 2579--2605, 2008.
\bibitem{deng2014linear}
W.~Deng, J.~Hu, and J.~Guo, ``Linear ranking analysis,'' in \emph{Proc. CVPR},
2014, pp. 3638--3645.
\bibitem{chapelle2007training}
O.~Chapelle, ``Training a support vector machine in the primal,'' \emph{Neural
computation}, vol.~19, no.~5, pp. 1155--1178, 2007.
\bibitem{shalev2011pegasos}
S.~Shalev-Shwartz, Y.~Singer, N.~Srebro, and A.~Cotter, ``Pegasos: Primal
estimated sub-gradient solver for svm,'' \emph{Mathematical programming},
vol. 127, no.~1, pp. 3--30, 2011.
\bibitem{okamoto2020realtime}
K.~Matsubara, T.~Okamoto, R.~Takashima, T.~Takiguchi, T.~Toda, Y.~Shiga, and
H.~Kawai, ``Investigation of training data size for real-time neural vocoders
on {CPU}s,'' \emph{Acoust. Sci. Tech.}, vol.~42, no.~1, pp. 65--68, 2021.
\bibitem{song2017effective}
E.~Song, F.~K. Soong, and H.-G. Kang, ``Effective spectral and excitation
modeling techniques for {LSTM}-{RNN}-based speech synthesis systems,''
\emph{IEEE/ACM Trans. Audio, Speech, and Lang. Process.}, vol.~25, no.~11,
pp. 2152--2161, 2017.
\bibitem{wang2018style}
Y.~Wang, D.~Stanton, Y.~Zhang, R.-S. Ryan, E.~Battenberg, J.~Shor, Y.~Xiao,
Y.~Jia, F.~Ren, and R.~A. Saurous, ``Style tokens: {U}nsupervised style
modeling, control and transfer in end-to-end speech synthesis,'' in
\emph{Proc. ICML}, 2018, pp. 5180--5189.
\bibitem{xavier2010init}
X.~Glorot and Y.~Bengio, ``Understanding the difficulty of training deep
feedforward neural networks,'' in \emph{Proc. AISTATS}, 2010, pp. 249--256.
\bibitem{liu2020radam}
L.~Liu, H.~Jiang, P.~He, W.~Chen, X.~Liu, J.~Gao, and J.~Han, ``On the variance
of the adaptive learning rate and beyond,'' in \emph{Proc. ICLR}, 2020.
\bibitem{ren2020fastspeech2}
Y.~Ren, C.~Hu, T.~Qin, S.~Zhao, Z.~Zhao, and T.-Y. Liu, ``{FastSpeech} 2:
{F}ast and high-quality end-to-end text-to-speech,'' in \emph{Proc. ICLR},
2021.
\bibitem{ren2019fastspeech}
Y.~Ren, Y.~Ruan, X.~Tan, T.~Qin, S.~Zhao, Z.~Zhao, and T.~Liu, ``Fast{S}peech:
Fast, robust and controllable text to speech,'' in \emph{Proc. NeurIPS},
2019, pp. 3165--3174.
\end{thebibliography}