EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesis
Extracted source text · arXiv 2507.12015
\documentclass{Interspeech}
% 2023-10-21 modified by Simon King (Simon.King@ed.ac.uk)
% 2024-01 modified by TPC Chairs of Interspeech 2024
% 2024-10 modified by Antoine Serrurier for Interspeech 2025
% 2024-12 modified by TPC Chairs of Interspeech 2025
% **************************************
% * DOUBLE-BLIND REVIEW SETTINGS *
% **************************************
% Comment out \interspeechcameraready when submitting the
% paper for review.
% If your paper is accepted, uncomment this to produce the
% 'camera ready' version to submit for publication.
\interspeechcameraready
% **************************************
% * *
% * STOP ! DO NOT DELETE ! *
% * READ THIS FIRST *
% * *
% * This template also includes *
% * important INSTRUCTIONS that you *
% * must follow when preparing your *
% * paper. Read it BEFORE replacing *
% * the content with your own work. *
% **************************************
% title here must exactly match the title entered into the paper submission system
\title{EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesis}
% the order of authors here must exactly match the order entered into the paper submission system
% note that the COMPLETE list of authors MUST be entered into the paper submission system at the outset, including when submitting your manuscript for double-blind review
\author[affiliation={1}]{Haoxun}{Li}
%\author[affiliation={1}]{Leyuan}{Qu}
\author[affiliation={1,*}]{Leyuan}{Qu}
%\author[affiliation={1,\dag}]{Leyuan}{Qu}
\author[affiliation={1}]{Jiaxi}{Hu}
%\author[affiliation={1}]{Taihao}{Li}
\author[affiliation={1,*}]{Taihao}{Li}
%The maximum number of authors in the author list is 20. If the number of contributing authors is more than this, they should be listed in a footnote or the acknowledgement section.
% if you have too many addresses to fit within the available space, try removing the "\\" newlines
\affiliation{Hangzhou Institute for Advanced Study}{University of Chinese Academy of Sciences}{China}
%\affiliation{Second Department}{Second Affiliation}{Second Country}
%\affiliation{}{Just Institute}{And Country}
\email{lihaoxun23@mails.ucas.ac.cn, leyuan.qu@ucas.ac.cn, hujiaxi23@mails.ucas.ac.cn,lith@ucas.ac.cn
}
\keywords{Emotional Speech Synthesis, Emphasis Control, Emotion Expressiveness}
\newcommand{\blue}[1]{\textcolor{blue}{#1}}
\usepackage{comment}
\usepackage{float} % 在导言区添加这一行
\usepackage{subcaption} % 用于子图标注
\usepackage{graphicx} % 插入图片
\usepackage{adjustbox} % 需要在导言区引入 adjustbox 包
\usepackage{hyperref}
\hypersetup{
colorlinks=true,
urlcolor=magenta,
}
%\usepackage{hyperref}
%\usepackage{biblatex}
%\addbibresource{mybib.bib}
\begin{document}
\maketitle
% the abstract here must exactly match the abstract entered into the paper submission system
\begin{abstract}
% In recent years, emotional Text-to-speech (TTS) synthesis and emphasis-controllable speech synthesis have advanced significantly. Previous studies have primarily focused on modeling emotional prosody or developing mechanisms for explicit emphasis control. However, their interaction remains underexplored. We propose Emphasis Meets Emotion TTS (EME-TTS), a novel framework designed to address two key research questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and (2) how to maintain the perceptual clarity and stability of target emphasis across different emotions. EME-TTS employs weakly supervised learning with emphasis pseudo-labels and variance-based emphasis features. Additionally, the proposed Emphasis Perception Enhancement (EPE) block enhances the interaction between emotional signals and emphasis positions. Experimental results show that EME-TTS, when combined with large language models for emphasis position prediction, enables more natural emotional speech synthesis while preserving stable and distinguishable target emphasis across emotions.
%In recent years, emotional Text-to-speech (TTS) synthesis and emphasis-controllable speech synthesis have advanced significantly. Previous studies have primarily focused on modeling emotional prosody or developing mechanisms for explicit emphasis control. However, the interaction between these two aspects remains underexplored. In this paper, we propose Emphasis Meets Emotion TTS (EME-TTS), a novel framework designed to address two key research questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, (2) how to maintain the perceptual clarity and stability of target emphasis across different emotions. EME-TTS employs weakly supervised learning with emphasis pseudo-labels and variance-based emphasis features. Additionally, the proposed Emphasis Perception Enhancement (EPE) block enhances the interaction between emotional signals and emphasis positions. Experimental results show that EME-TTS, when combined with large language models for emphasis position prediction, enables more natural emotional speech synthesis while preserving stable and distinguishable target emphasis across emotions.
\renewcommand{\thefootnote}{*}
\footnotetext{Corresponding author}
\renewcommand{\thefootnote}{\arabic{footnote}}
\noindent In recent years, emotional Text-to-Speech (TTS) synthesis and emphasis-controllable speech synthesis have advanced significantly. However, their interaction remains underexplored. We propose Emphasis Meets Emotion TTS (EME-TTS), a novel framework designed to address two key research questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and (2) how to maintain the perceptual clarity and stability of target emphasis across different emotions. EME-TTS employs weakly supervised learning with emphasis pseudo-labels and variance-based emphasis features. Additionally, the proposed Emphasis Perception Enhancement (EPE) block enhances the interaction between emotional signals and emphasis positions. Experimental results show that EME-TTS, when combined with large language models for emphasis position prediction, enables more natural emotional speech synthesis while preserving stable and distinguishable target emphasis across emotions. Synthesized samples are available on-line\footnote{\url{https://wd-233.github.io/EME-TTS_DEMO/}}.
%Demonstration samples for some of the evaluations are available on an anonymous file-sharing link\footnote{Anonymous file-sharing link: \url{https://www.dropbox.com/scl/fi/7m9ctlhbjfgxn76c4p2r4/EME-TTS_demo.zip?rlkey=ijchg03ipvw0rq3et3lq44lr4&st=s7awj945&dl=0}} on Dropbox.
%GLM4V\footnote{GLM4V: \url{https://open.bigmodel.cn/dev/api}}
% 脚注
%\renewcommand{\thefootnote}{*}
%\footnotetext{Corresponding author}
%\renewcommand{\thefootnote}{\arabic{footnote}}
\end{abstract}
%\href{https://www.dropbox.com/scl/fi/7m9ctlhbjfgxn76c4p2r4/EME-TTS_demo.zip\?rlkey\=ijchg03ipvw0rq3et3lq44lr4\&st\=s7awj945\&dl\=0}{anonymous file-sharing link}
\begin{comment}
\begin{abstract}
% 1000 characters. ASCII characters only. No citations.
In recent years, significant progress has been made in both emotional text-to-speech (TTS) synthesis and emphasis-controllable speech synthesis. However, the interaction between emotion and emphasis remains underexplored. To bridge this gap, we propose Emphasis Meets Emotion TTS (EME-TTS), a novel framework designed to address two key research questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and (2) how to maintain the perceptual clarity and stability of target emphasis across different emotion types. To tackle these challenges, EME-TTS employs a weakly supervised learning approach that leverages emphasis pseudo-labels and introduces variance-based emphasis features. Furthermore, we propose an emphasis adapter built upon the Conditional Cross Attention (CCA) module to guide the model in capturing interactions between emotional signals and emphasis positions. Experimental results demonstrate that EME-TTS, when combined with large language models for emphasis prediction during inference, enables more natural emotional speech synthesis while ensuring that target emphasis remains stable and clearly distinguishable across different emotion types.
\end{abstract}
\end{comment}
\begin{comment}
(1) how to enhance emotional expression through emphasis, and (2) how to ensure that the target emphasis position remains consistently distinguishable under varying emotion types.
\end{comment}
\section{Introduction}
%\begin{equation}
%\mathcal{L}_E = \mathcal{L}_D +\mathcal{L}_P+\mathcal{L}_P^V+\mathcal{L}_D^V
%\end{equation}
With the advancement of deep learning, Text-to-Speech (TTS) systems have significantly improved in terms of quality, clarity, and naturalness, leveraging architectures such as Transformers \cite{waswani2017attention,ren2020fastspeech}, normalizing flows \cite{kim2020glow,kim2021conditional}, and diffusion models \cite{popov2021grad,tan2024naturalspeech}. However, conventional TTS systems often struggle with expressiveness, producing monotonal and mechanical speech. To address this issue, researchers have explored emotionally expressive speech synthesis \cite{ExpressiveFastSpeech2}, employing explicit emotion labels \cite{tits2020exploring,diatlova23_ssw,guo2023emodiff} and reference-based approaches \cite{li2021controllable,lei2022msemotts,zaidi22b_interspeech} to enhance speech expressiveness.
A key aspect of expressive speech is emphasis, which highlights prominent prosodic regions through variations in pitch, phoneme duration, and spectral energy \cite{eriksson2015acoustics,eriksson2016acoustics,eriksson2020lexical}. Several studies have introduced emphasis control in TTS \cite{li2018emphatic,shechtman2021supervised,seshadri22_interspeech}, ranging from handcrafted feature integration in systems based on Hidden Markov Models (HMM) \cite{li2018emphatic} to leveraging intermediate acoustic cues like pitch range \cite{shechtman2021supervised} and variance-based features \cite{seshadri22_interspeech}.
%Despite advancements in emotional TTS and emphasis-controllable synthesis, their interaction remains underexplored. This study aims to develop a speech synthesis model that simultaneously controls both emotion and emphasis, addressing two key questions: (1) how to effectively utilize emphasis to enhance the expressiveness of emotional speech, and (2) how to maintain the perceptual clarity and stability of target emphasis across different emotions.
Despite significant progress in emotional TTS and emphasis-controllable synthesis, the interplay between emotion and emphasis in speech remains largely unexplored. Emphasis and emotion are intrinsically linked—emphasis modulates emotional perception by influencing prosodic patterns, while emotional states naturally determine which words are accentuated in speech. This study develops a speech synthesis model capable of simultaneously controlling both emotion and emphasis. Specifically, we seek to answer two key questions: How can emphasis be effectively leveraged to enhance the expressiveness of emotional speech? And how can emphasis clarity and stability be preserved across different emotional conditions?
To answer the first question, we examine how predefined emphasis positions influence emotional speech synthesis. Emotional speech is shaped by both prosody and semantics, where shifts in emphasis position and intensity can alter an utterance’s emotional interpretation. While prior works such as EE-TTS \cite{zhong23_interspeech} predicts emphasis using textual and grammatical cues, we argue that Large Language Models (LLMs) \cite{zhao2023survey} can infer emphasis more effectively. Instead of focusing on semantic-based emphasis prediction, we investigate how predefined emphasis positions influence emotional speech synthesis. To this end, we annotate emphasis pseudo-labels on an emotional speech dataset and integrate an improved variance-based emphasis modeling approach. During inference, an LLM predicts emphasis positions based on emotion labels and text input. Evaluations show that our model generates more emotionally expressive speech, especially when contextual information is present.
To answer the second question, we introduce the Emphasis Perception Enhancement (EPE) block to improve emphasis clarity and stability across emotions. This block refines the interaction between emotion control signals and emphasis locations, reducing unintended interference from global emotional effects. Additionally, this design mitigates artifacts that often arise at emphasized positions, enhancing both perceptual emphasis clarity and synthesis quality.
%To ensure the perceptual clarity of emphasis across different emotions, we build upon EmoSpeech’s Conditional Cross Attention (CCA) module and introduce an Emphasis Adapter (EA). This mechanism refines the interaction between emotional control signals and emphasis locations, reducing the interference of global emotion signals on emphasis. Additionally, this approach helps mitigate artifacts that may arise at emphasized positions, further improving the overall speech quality. Experimental results confirm that this method enhances both the clarity of emphasis perception across different emotions and the naturalness of synthesized speech.
To summarize, the main contributions of this work are as follows:
\begin{itemize}
\item Emotionally Controllable Emphasis Modeling: We incorporate variance-based emphasis features into an emotional TTS framework, and use weakly supervised learning with EmphaClass \cite{seyssel-etal-2024-emphassess} pseudo-labeling on the ESD dataset \cite{zhou2022emotional}.
\item Refined Emotion-Emphasis Interaction: We propose EPE block to modulate emphasis prominence effectively, which ensures stable and clear emphasis across emotions.
\item To the best of our knowledge, this study is the first to systematically investigate the relationship between emotion and emphasis in speech synthesis. EME-TTS enhances emotional expressiveness and preserves emphasis clarity with predefined emphasis positions and emotion labels.
\end{itemize}
%1. We annotate emphasis pseudo-labels on the ESD dataset [24] using EmphaClass [23] and integrate controllable emphasis modeling into an emotional speech synthesis framework. By incorporating LLM-based emphasis prediction during inference, we validate that our proposed model, Emphasis Meets Emotion TTS (EME-TTS), enhances emotional expressiveness through emphasis control.
%2. We introduce an Emphasis Adapter that adjusts the influence of global emotional signals at target emphasis positions, thereby enhancing their perceptual prominence and ensuring that the intended emphasis remains clearly identifiable across different emotional categories.
%3. To the best of our knowledge, this study is the first to systematically explore the relationship between emotion and emphasis in speech synthesis. Experimental results demonstrate that EME-TTS can synthesize high-quality emotional speech with controllable emphasis, given predefined emphasis positions and emotion labels as model inputs.
\begin{comment}
\begin{figure*}[htbp]
\setlength{\abovecaptionskip}{-5pt}
\setlength{\belowcaptionskip}{-10pt}
\centering
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=2.06in]{fig4.png}
\caption{f1.}
\label{figure4}
\end{minipage}%
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=2.06in]{fig2.png}
\caption{f2.}
\label{figure2}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=1.76in]{fig3.png}
\caption{f3.}
\label{figure3}
\end{minipage}
\end{figure*}
\end{comment}
\begin{comment}
\begin{figure*}[htbp]
\setlength{\abovecaptionskip}{5pt} % 调整标题与图片之间的间距
\setlength{\belowcaptionskip}{5pt} % 调整标题与图片之间的间距
\centering
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=2.06in]{fig4.png}
\caption{f1.} % 添加小标题
\label{figure4}
\end{minipage}%
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=\linewidth]{fig2.png}
\caption{f2.} % 添加小标题
\label{figure2}
\end{minipage}
\begin{minipage}[t]{0.33\linewidth}
\centering
\includegraphics[width=\linewidth]{fig3.png}
\caption{f3.} % 添加小标题
\label{figure3}
\end{minipage}
\caption{Overall Figure: The three subfigures (a), (b), and (c).} % 大标题
\label{figure1}
\end{figure*}
\end{comment}
\begin{figure*}[htbp]
\setlength{\abovecaptionskip}{5pt} % 调整标题与图片之间的间距
\setlength{\belowcaptionskip}{5pt} % 调整标题与图片之间的间距
\centering
% (a) 图 - 变小
\begin{minipage}[t]{0.22\linewidth} % 变小
\centering
\mbox{\adjustbox{valign=m}{\includegraphics[width=1.5in]{aaa.pdf}}} % 仅让图片居中
\vspace{28pt} \\ % 让子标题靠下
\subcaption{Overall Architecture}\label{figure4} % 子标题(a)
\end{minipage}
% (b) 图 - 变大
\begin{minipage}[t]{0.43\linewidth} % 变大
\centering
\mbox{\adjustbox{valign=m}{\includegraphics[width=3.0in]{bb.pdf}}} % 仅让图片居中
\vspace{-12pt} \\ % 让子标题靠下
\subcaption{Variance Adapter}\label{figure2} % 子标题(b)
\end{minipage}
% (c) 图 - 变大
\begin{minipage}[t]{0.34\linewidth} % 变大
\centering
\mbox{\adjustbox{valign=m}{\includegraphics[width=2.4in]{cc.pdf}}} % 仅让图片居中
\vspace{1pt} \\ % 让子标题靠下
\subcaption{EPE Block}\label{figure3} % 子标题(c)
\end{minipage}
\caption{The entire framework of our proposed model. (a) is the overall architecture diagram. (b) and (c) show the detailed structure of variance adapter and Emphasis Perception Enhancement (EPE) block, respectively.} % 大标题
\label{figure1}
\vspace{-16pt}
\end{figure*}
\begin{comment}
%上一版本图片
\begin{figure*}[htbp]
\setlength{\abovecaptionskip}{5pt} % 调整标题与图片之间的间距
\setlength{\belowcaptionskip}{5pt} % 调整标题与图片之间的间距
\centering
% (a) 图 - 变小
\begin{minipage}[t]{0.20\linewidth} % 变小
\centering
\includegraphics[width=1.4in]{aaa.pdf} % 缩小 width，保持原始比例
\subcaption{Overall Architecture}\label{figure4} % 子标题(a)
\end{minipage}
% (b) 图 - 变大
\begin{minipage}[t]{0.45\linewidth} % 变大
\centering
\includegraphics[width=3.0in]{bb.pdf} % 增大 width，保持原始比例
\subcaption{Variance Adapter}\label{figure2} % 子标题(b)
\end{minipage}
% (c) 图 - 变大
\begin{minipage}[t]{0.33\linewidth} % 变大
\centering
\includegraphics[width=2.2in]{cc.pdf} % 增大 width，保持原始比例
\subcaption{EPE Block}\label{figure3} % 子标题(c)
\end{minipage}
\caption{The entire framework of our proposed model. (a) is the overall architecture diagram. (b) and (c) show the detailed structure of variance adapter and Emphasis Perception Enhancement (EPE) block, respectively.} % 大标题
\label{figure1}
\vspace{-16pt}
\end{figure*}
\end{comment}
\begin{comment}
\begin{figure*}[htbp]
\setlength{\abovecaptionskip}{5pt} % 调整标题与图片之间的间距
\setlength{\belowcaptionskip}{5pt} % 调整标题与图片之间的间距
\centering
\begin{minipage}[t]{0.30\linewidth}
\centering
\includegraphics[width=1.9in]{图片101.png}
\subcaption{Overall Architecture}\label{figure4} % 子标题(a)
\end{minipage}%
\begin{minipage}[t]{0.39\linewidth}
\centering
\includegraphics[width=2.65in]{图片44.png}
\subcaption{Variance Adapter}\label{figure2} % 子标题(b)
\end{minipage}
\begin{minipage}[t]{0.30\linewidth}
\centering
\includegraphics[width=1.96in]{图片122.png}
\subcaption{EPE Block}\label{figure3} % 子标题(c)
\end{minipage}
\caption{The entire framework of our proposed model. (a) is the overall architecture diagram. (b) and (c) show the detailed structure of variance adapter and Emphasis Perception Enhancement (EPE) block respectively.} % 大标题
\label{figure1}
\end{figure*}
\end{comment}
\begin{comment}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{overview.png}
\caption{Schematic diagram of speech production.}
\label{fig:speech production}
\end{figure}
\end{comment}
\section{Proposed Method}
\subsection{Overview}
The overall architecture of EME-TTS is shown in Figure 1a. We use EmoSpeech \cite{diatlova23_ssw} as the base architecture of the acoustic model, using the embedding of emotions and the position of the emphasis as conditions. These conditions are obtained through the emotion label and the emphasis pseudo label respectively. EME-TTS consists of a phoneme encoder, a variance adapter, and a frame decoder, working in conjunction to generate emotionally expressive speech with controllable emphasis. The phoneme encoder processes the input phoneme sequence extracted from text, encoding phonetic features that serve as the foundation for speech synthesis. The variance adapter models prosodic variations through duration, pitch, and energy while integrating variance-based emphasis features for explicit emphasis control. The resulting frame-level hidden representations are then converted into a mel-spectrogram by the frame decoder.
%Both the encoder and decoder are composed of multiple stacked Emphasis Perception Enhancement (EPE) blocks, the details of which are provided in Section 2.4.
\begin{comment}
tecture of the model proposed in this paper is shown in Figure 1a. The FastSpeech2[] architecture has been widely adopted for emphasis control tasks due to its explicit modeling of fundamental prosodic features, and its advantages in inference speed also make it suitable for high-load environments. Therefore, this study adopts EmoSpeech [9], which is built on the FastSpeech2 architecture, as its foundation. This architecture introduces emotional embeddings on top of FastSpeech2 and incorporates several improvements, including Conditional Cross Attention, eGeMAPS predictor, and adversarial training, to enhance the model's suitability for lightweight emotional speech synthesis tasks. We further introduce emphasis pseudo-labels as local control signals and make adjustments to the phoneme encoder, variance adapter, and frame decoder.
\end{comment}
%\subsubsection{Page layout}
\subsection{Weakly Supervised Emphasis Pseudo-Labeling}%\label{sec:figures}
Determining whether a word is emphasized within a sentence involves significant subjectivity, as multiple valid emphasis positions may exist. Consequently, collecting large-scale labeled emphasis data is both challenging and costly. Unlike EE-TTS \cite{zhong23_interspeech}, which employs a wavelet-based prosody toolset \cite{suni2017hierarchical} to compute prominence scores via Continuous Wavelet Transform (CWT) using pitch, energy, and duration signals, we leverage the EmphAssess dataset and the EmphaClass emphasis recognizer \cite{seyssel-etal-2024-emphassess}. EmphaClass fine-tunes a pre-trained Self-Supervised Learning (SSL) speech model for frame-level classification, and aggregates these scores to determine word-level emphasis with high accuracy. By utilizing EmphaClass to annotate emphasis in the ESD dataset \cite{zhou2022emotional}, which underpins our TTS experiments, we obtain highly reliable emphasis pseudo-labels without the need for extensive manual annotation.
\subsection{Variance Adapter}
In order to integrate both fundamental prosodic predictors and variance-based emphasis modeling, the variance adapter is designed to model and regulate prosodic variations. As shown in Figure 1b, it consists of predictors for pitch, duration, and energy, which predict their respective prosodic features from encoded phonemes. Additionally, it includes variance-based pitch and duration predictors, which refine these predictions by capturing local deviations in emphasized regions. The upsampling mechanism ensures that phoneme-level prosodic features are aligned with the frame-level hidden sequence before being passed to the decoder.
To explicitly control emphasis, we incorporate variance-based prosodic features as local modulation signals. Following the assumption in \cite{seshadri22_interspeech} that pitch and duration are the primary indicators of emphasis, with energy having a lesser impact, we focus on modeling only pitch and duration variance features, omitting energy. The computation of these variance features is formulated as follows:
\begin{equation}
\text{Pitch Variance} = W_{F_0} - S_{F_0}
\end{equation}
\begin{equation}
\text{Duration Variance} = W_{\text{dur}} - S_{\text{dur}}
\end{equation}
where \( W_{F_0} \) and \( W_{\text{dur}} \) represent the average pitch and duration of phonemes in emphasized regions, while \( S_{F_0} \) and \( S_{\text{dur}} \) denote the sentence-level averages. These values are derived within the Variance Pitch Predictor and Variance Duration Predictor using the emphasis pseudo-labels, which provide the ${start, end}$ positions of emphasized words.
\begin{comment}
The calculations are as follows:
\begin{equation}
W_{F_0} = \frac{1}{N_{\text{emp}}} \sum_{i \in \text{emp}} F_0(i), \quad
S_{F_0} = \frac{1}{N_{\text{all}}} \sum_{i \in \text{all}} F_0(i)
\end{equation}
\begin{equation}
W_{\text{dur}} = \frac{1}{N_{\text{emp}}} \sum_{i \in \text{emp}} D(i), \quad
S_{\text{dur}} = \frac{1}{N_{\text{all}}} \sum_{i \in \text{all}} D(i)
\end{equation}
where \( N_{\text{emp}} \) and \( N_{\text{all}} \) denote the number of phonemes in the emphasized regions and the entire sentence, respectively. The indices \( i \in \text{emp} \) correspond to phonemes within the ${start, end}$ positions specified by the emphasis pseudo-labels.
\end{comment}
During training, the predicted pitch feature \( P_{\text{pre}} \) is used to compute the pitch variance feature \( P^V_{\text{pre}} \) via Equation (1), and its loss is calculated against the target pitch feature \( P_{\text{tar}} \):
\begin{equation}
\mathcal{L}_{P} = \text{MSE}(P_{\text{pre}} + P^V_{\text{pre}}, P_{\text{tar}})
\end{equation}
where Mean Squared Error (MSE) measures the average squared difference between predicted and target values. Similarly, the pitch variance loss is:
\begin{equation}
\mathcal{L}_{P}^{V} = \text{MSE}(P^V_{\text{pre}}, P^V_{\text{tar}})
\end{equation}
For duration modeling, we adopt the same approach:
\begin{equation}
\mathcal{L}_{D} = \text{MSE}(D_{\text{pre}} + D^V_{\text{pre}}, D_{\text{tar}})
\end{equation}
\begin{equation}
\mathcal{L}_{D}^{V} = \text{MSE}(D^V_{\text{pre}}, D^V_{\text{tar}})
\end{equation}
where the variance features \( P^V_{\text{pre}} \) and \( D^V_{\text{pre}} \) are only applied to emphasized regions, with non-emphasized regions set to zero. Since direct variance calculations may introduce extreme values or negative emphasis scores, we normalize the features to the range of [0,2] based on data distribution and apply regularization for stability. The aforementioned losses, along with the energy loss, are included in the final total loss calculation.
\begin{comment}
During inference, an emphasis degree coefficient \( \lambda \) is introduced to adjust the level of emphasis:
\begin{equation}
P_{\text{adj}} = P_{\text{pre}} + \lambda \cdot P^V_{\text{pre}}
\end{equation}
\begin{equation}
D_{\text{adj}} = D_{\text{pre}} + \lambda \cdot D^V_{\text{pre}}
\end{equation}
where \( P_{\text{adj}} \) and \( D_{\text{adj}} \) are the adjusted pitch and duration features. By modifying \( \lambda \), the model dynamically controls the emphasis degree, enabling flexible adaptation across different contexts while maintaining learned emphasis patterns.
\end{comment}
\begin{comment}
We introduce variance-based prosodic features to represent the degree of emphasis and integrate them as local prosodic control signals into the variance adapter, as shown in Figure 1b. Similar to the assumption in [17], we hypothesize that emphasis is mainly determined by pitch and duration, with energy having a lesser impact on emphasis. Therefore, we calculate only the pitch and duration variance features, without modeling energy.
Since our dataset is not a specialized emphasis speech dataset, as in [18], the emphasized words marked by pseudo-labels may not show a significant emphasis effect. Comparing with the EmphAssess dataset[], we found that our training data, especially in terms of pitch prominence, shows a notable difference. To better represent the pitch variation trend of emphasized words within the overall sentence and reduce errors due to unclear emphasis features in the dataset, we adjusted the calculation method for pitch variance features. Instead of relying on the logarithmic pitch span (i.e., the difference between the 95th and 5th percentiles), we adopted a method similar to the calculation of duration variance features. Specifically, the pitch and duration variance features are calculated as follows:
\begin{equation}
\text{Pitch Variance} = W_{F_0} - S_{F_0}
\end{equation}
\begin{equation}
\text{Duration Variance} = W_{\text{dur}} - S_{\text{dur}}
\end{equation}
where \( W_{F_0} \) and \( W_{\text{dur}} \) denote the average pitch and duration of phonemes corresponding to emphasized words, while \( S_{F_0} \) and \( S_{\text{dur}} \) represent sentence-level averages. Our model predicts phoneme-level prosodic features, and the final frame-level sequence is obtained by upsampling according to the predicted durations.
During training, the predicted pitch feature \( P_{\text{prediction}} \) is used to compute the pitch variance \( P^V_{\text{prediction}} \) via Equation (1), and its loss is calculated against the target pitch feature \( P_{\text{target}} \) as:
\begin{equation}
\text{Loss (Pitch)} = \text{MSE}(P_{\text{prediction}} + P^V_{\text{prediction}}, P_{\text{target}})
\end{equation}
where MSE (Mean Squared Error) measures the average squared difference between predicted and target values. Similarly, the pitch variance loss is:
\begin{equation}
\text{Loss (Pitch Variance)} = \text{MSE}(P^V_{\text{prediction}}, P^V_{\text{target}})
\end{equation}
For duration modeling, we adopt the same approach:
\begin{equation}
\text{Loss (Duration)} = \text{MSE}(D_{\text{prediction}} + D^V_{\text{prediction}}, D_{\text{target}})
\end{equation}
\begin{equation}
\text{Loss (Duration Variance)} = \text{MSE}(D^V_{\text{prediction}}, D^V_{\text{target}})
\end{equation}
where variance features \( P^V_{\text{prediction}} \) and \( D^V_{\text{prediction}} \) are only applied to emphasized regions, with non-emphasized regions set to zero. Since direct variance calculations may introduce extreme values or negative emphasis scores, we normalize the features to [0,2] based on data distribution and apply regularization for stability.
During inference, an emphasis degree coefficient \( \lambda \) is introduced to adjust the level of emphasis:
\begin{equation}
P_{\text{adjusted}} = P_{\text{predict}} + \lambda \cdot P^V_{\text{predict}}
\end{equation}
\begin{equation}
D_{\text{adjusted}} = D_{\text{predict}} + \lambda \cdot D^V_{\text{predict}}
\end{equation}
where \( P_{\text{adjusted}} \) and \( D_{\text{adjusted}} \) are the adjusted pitch and duration features. By modifying \( \lambda \), the model dynamically controls the emphasis degree, enabling flexible adaptation across different contexts while maintaining learned emphasis patterns.
\end{comment}
\begin{comment}
where \( W_{F_0} \) and \( W_{\text{dur}} \) represent the average pitch and average duration of phonemes corresponding to the emphasized pseudo-label words, while \( S_{F_0} \) and \( S_{\text{dur}} \) represent the sentence-level average pitch and average duration of phonemes respectively.
Our model predicts the basic prosodic features at the phoneme level, and after prediction, the output frame-level hidden sequence is obtained by upsampling according to the predicted phoneme duration.
During training, the predicted pitch feature \( P_{\text{prediction}} \) first computes the predicted pitch variance feature \( P^V_{\text{prediction}} \) using Formula 1, and then the loss is calculated by comparing it with the target pitch feature \( P_{\text{target}} \) as follows:
\begin{equation}
\text{Loss (Pitch)} = \text{MSE}(P_{\text{prediction}} + P^V_{\text{predict}}, P_{\text{target}})
\end{equation}
where MSE (Mean Squared Error) measures the average squared difference between predicted and target values. \( P^V_{\text{prediction}} \) is only added in the emphasized regions, i.e., in the phonemes corresponding to emphasized words. For non-emphasized regions, the value of \( P^V_{\text{prediction}} \) is zero and does not participate in the calculation. Meanwhile, the pitch variance feature also needs to calculate the loss with the target pitch variance feature \( P^V_{\text{target}} \):
\begin{equation}
\text{Loss (Pitch Variance)} = \text{MSE}(P^V_{\text{prediction}}, P^V_{\text{target}})
\end{equation}
Similarly, the predicted duration \( D_{\text{prediction}} \) and duration variance feature \( D^V_{\text{prediction}} \) are calculated using similar formulas for the loss comparison with the target true values \( D_{\text{target}} \) and \( D^V_{\text{target}} \):
\begin{equation}
\text{Loss (Duration)} = \text{MSE}(D_{\text{prediction}} + D^V_{\text{prediction}}, D_{\text{target}})
\end{equation}
\begin{equation}
\text{Loss (Duration Variance)} = \text{MSE}(D^V_{\text{prediction}}, D^V_{\text{target}})
\end{equation}
where \( D^V_{\text{prediction}} \) is also only added in the emphasized regions. Furthermore, since the directly calculated variance features may have negative values (i.e., some emphasized positions might have negative emphasis feature values), and there may be extreme feature values affecting the model's training stability, we normalize the features by mapping them to the [0,2] range using the data distribution and add a regularization term to better fit the emphasis features.
In the inference phase, we introduce an emphasis degree coefficient \( \lambda \) to control the level of emphasis in the generated speech. The calculation is as follows:
\begin{equation}
P_{\text{adjusted}} = P_{\text{predict}} + \lambda \cdot P^V_{\text{predict}}
\end{equation}
\begin{equation}
D_{\text{adjusted}} = D_{\text{predict}} + \lambda \cdot D^V_{\text{predict}}
\end{equation}
where \( P_{\text{adjusted}} \) and \( D_{\text{adjusted}} \) represent the adjusted pitch and duration features, and \( \lambda \) is the emphasis degree coefficient. During inference, we can freely adjust the value of \( \lambda \) based on different contextual requirements to achieve varying levels of emphasis control. In this way, our model can not only learn the inherent emphasis patterns in the dataset but also flexibly adjust the emphasis degree during inference, making the generated speech more controllable.
\end{comment}
\subsection{Emphasis Perception Enhancement Block}
The phoneme encoder and frame decoder in EME-TTS are composed of multiple stacked Emphasis Perception Enhancement (EPE) blocks, replacing the original feed-forward transformer blocks, as illustrated in Figure 1c. Each EPE block refines the modeling of emphasis perception by integrating Multi-Head Attention (MHA), Conditional Cross Attention (CCA), and Emphasis Adapter (EA), ensuring that the synthesized speech maintains clear and stable emphasis across emotions. The input hidden sequence is processed through self-attention and conditional normalization layers before being further refined by convolutional layers. Simultaneously, the emotion embedding and emphasis position can serve as external conditions.
Among these components, MHA captures global dependencies within the hidden sequence by computing multiple attention heads, allowing the model to extract contextual relationships between phonemes. CCA re-weights self-attention by incorporating emotional cues, allowing attention distributions to be adjusted based on the given emotion embedding $c$. CCA re-weights self-attention using:
\begin{equation}
Q = W_q \cdot h, \quad K = W_k \cdot c, \quad V = W_v \cdot c
\end{equation}
\begin{equation}
w = \text{softmax} \left( \frac{Q \cdot K^T}{\sqrt{d}} \right)
\end{equation}
\begin{equation}
\text{CCA} = w \cdot V
\end{equation}
where \( h \) represents the input feature, and \( w \) denotes computed attention weights. In expressive speech, the prominence of certain words naturally varies with emotion, causing inconsistencies in emphasis perception. This can lead to unintended shifts in emphasis positions or even the suppression of intended emphasis under strong emotional conditions. Additionally, increasing the prominence of emphasized words may introduce artifacts and degrade synthesis quality.
To mitigate these issues, we introduce EA that refines attention distributions in predefined emphasis regions. EA explicitly enhances emphasis perceptibility while minimizing interference from emotional variations. Given an initial attention weight \( w \), the adjusted weight is computed as:
\begin{equation}
w_{\text{adjusted}} = w + \Delta w, \quad \Delta w = {strength} \cdot {mask}({start, end})
\end{equation}
where \({mask(start, end)}\) identifies the designated emphasis positions, and ${strength}$ scales the emphasis intensity. This enhancement serves two key purposes: first, it ensures that emphasized words remain perceptually distinct, even in highly expressive speech; second, it refines emphasis representation through attention modulation rather than direct energy or pitch manipulation, thereby reducing synthesis artifacts.
For Conditional Layer Normalization (CLN), we adopt a design similar to AdaSpeech4 \cite{wu22f_interspeech} to integrate emotional context into the normalization process, ensuring adaptive prosodic control across different emotional conditions.
\begin{comment}
The Phoneme Encoder and Frame Decoder are both composed of multiple stacked feed-forward Transformer (FFT) modules. Each FFT module includes a Multi-Head Attention mechanism (MHA), Conditional Cross Attention (CCA), and a 1D convolutional layer, as shown in Figure 1c. The multi-head attention module follows the original design in FastSpeech2 and is used to extract cross-position information. For CCA, the speaker emotion embedding is denoted as the conditional embedding $c$, which is used to re-weight the hidden layers via the self-attention layers with the matrices $W_q$, $W_k$, and $W_v$:
\begin{equation}
Q = W_q \cdot h, \quad K = W_k \cdot c, \quad V = W_v \cdot c
\end{equation}
\begin{equation}
w = \text{softmax}\left( \frac{Q \cdot K^T}{\sqrt{d}}, \, \text{dim} = 1 \right)
\end{equation}
\begin{equation}
\text{cca} = w \cdot V
\end{equation}
where $h$ is the self-attention output, $Q$ represents the result calculated by the query vector $h$, and $K$ and $V$ are the results obtained by multiplying the conditional embedding $c$ with the weight matrices $W_k$ and $W_v$, respectively. The attention weights $w$ are computed via softmax, and then multiplied by $V$ to obtain the final weighted representation. Then, through a emphasis adapter, the weights are adjusted according to the positional information provided by the emphasize pseudo-labels. Specifically, let us assume we have a ``strength'' to represent the degree of emphasis, a ``start'' to indicate the starting position of emphasis, and an ``end'' to indicate the ending position of emphasis. The emphasis adapter's calculation formula is as follows:
\begin{equation}
w_{\text{adjusted}} = w + \Delta w
\end{equation}
where $w$ is the original weight, and $\Delta w$ is the incremental weight adjusted by the emphasis positional information. We can calculate $\Delta w$ as follows:
\begin{equation}
\Delta w = \text{strength} \cdot \text{mask}(\text{start}, \text{end})
\end{equation}
where mask(start, end) is a binary mask matrix used to identify which positions should be emphasized. The purpose of this mask matrix is to add an additional component, ``strength'', to the weights in the emphasized region.
For Conditional Layer Normalization (CLN), we adopt a design similar to that of AdaSpeech4 [28] in our model. The main function of CLN is to combine conditional information during normalization, replacing the original LayerNorm:
\begin{equation}
y = f(c) \cdot \frac{x - \text{mean}}{\text{var}} + f(c)
\end{equation}
where $f$ is a linear transformation, $c$ is the conditional embedding vector, $x$ is the input feature, mean and var represent the mean and variance of $x$, respectively, computed across the feature dimension. This design enhances the model's ability to generate emotionally salient speech at emphasized positions, ensuring that these positions remain consistently perceptible.
\end{comment}
\section{Experiments}
\subsection{Experimental Setup}
For our experiments, EME-TTS utilizes the English portion of the Emotional Speech Database (ESD) \cite{zhou2022emotional}, which comprises recordings from 10 speakers across five emotions: \textit{angry, happy, sad, surprise, and neutral}. Each speaker contributes 350 utterances per emotion, resulting in approximately 1,750 utterances and 1.2 hours of speech per speaker. We follow the train/validation/test splits established in EmoSpeech \cite{diatlova23_ssw}, where the validation and test sets consist of 19 and 31 utterances per emotion per speaker, leading to a total of 950 and 1,550 utterances, respectively. Emphasis positions in the training data are labeled using EmphaClass \cite{seyssel-etal-2024-emphassess}. %For consistency during training, utterances with multiple emphasis positions were classified as non-emphasized.
The model utilizes iSTFTNet \cite{kaneko2022istftnet} as the vocoder, which is trained on the English subset of the ESD dataset \cite{zhou2022emotional}. The strength in EA is set to 0.2. Training is conducted on 2 Nvidia A100 GPUs and 8 RTX 4090 GPUs, with a batch size of 64 for 100,000 steps. The Adam optimizer \cite{diederik2014adam} is used with a learning rate of 0.0001 and ($\beta_1$, $\beta_2$) = (0.5, 0.9).
%The phoneme embedding, encoder, and decoder hidden dimensions are set to 512, while the encoder and decoder Conv1D filter sizes are also 512. Both the encoder and decoder consist of 9 layers each, with an emotion embedding hidden dimension of 256. Each block in Variance Adapter comprises 2 1D convolutional layers with a kernel size of 3, stride of 1, followed by a ReLU activation and a dropout rate of 0.5. The mel spectrograms are extracted from waveforms using a filter length of 48 ms, a hop length of 12 ms, and 80 mel channels.
\subsection{Evaluation Metrics}
The evaluation framework comprises both objective and subjective assessments. The objective evaluation includes audio quality assessment and emotional accuracy measurement. The subjective evaluation consists of four tasks:
\begin{itemize}
\item Emphasis Accuracy Test (EAT): Measures how well predicted emphasis positions match listener perception.
\item Emotion Accuracy Test (EAT-EMO): Assesses emotion recognition accuracy by comparing perceived and intended emotions.
\item Emotional Expressiveness Preference Test (EEPT): Listeners select the most emotionally expressive sample from multiple outputs, with preference scores indicating expressiveness strength.
\item Mean Opinion Score (MOS) Rating: Evaluates naturalness and quality on a five-point scale.
\end{itemize}
A total of 11 listeners participated in all evaluations, with each participant completing all four tasks. For all tasks, higher scores indicate better performance.
%Demonstration samples for some of the evaluations are available on an anonymous file-sharing link on Dropbox.
\subsubsection{Emphasis Accuracy}
To demonstrate EME-TTS's ability to preserve the perceptual clarity and stability of target emphasis across different emotions, we designed a more challenging subjective test as task 1 (EAT), distinct from previous studies on emphasis control \cite{seshadri22_interspeech}. Instead of rating the degree of emphasis at a predefined position, participants were asked to identify the emphasized words in 80 randomly shuffled speech samples. These samples were generated from texts in the test set using EME-TTS w/o EPE and EME-TTS. The results presented in Table 1 indicate that the emphasis produced by our proposed model was clearly perceivable across different emotions.
Notably, compared to other emotions, the \textit{surprise} emotion posed a greater challenge for listeners in accurately identifying the emphasis position. This is attributed to the frequent pitch rise at the end of \textit{surprise} speech, which often led participants to mistakenly perceive the emphasized word as being at the sentence’s end. The integration of the EPE effectively mitigated this issue by enhancing the prominence of the intended emphasis position. At the same time, it improved the distinctiveness of emphasis recognition across all emotions.
\begin{table}[ht]
\caption{Emphasis Recognition Accuracy of Different Models.}
\label{tab:emphasis_recognition}
\centering
\renewcommand{\arraystretch}{1} % 增加行间距，使字体看起来更大
\resizebox{0.48\textwidth}{!}{
\fontsize{18pt}{24pt}\selectfont % 调整字体大小
\begin{tabular}{ l c c c c c c }
\toprule
\textbf{Model} & \textbf{Mean} & \textbf{Neutral} & \textbf{Angry} & \textbf{Happy} & \textbf{Sad} & \textbf{Surprise} \\
\midrule
EME-TTS w/o EPE & 0.73 & 0.77 & 0.75 & \textbf{0.82} & \textbf{0.75} & 0.55 \\
\midrule
\textbf{EME-TTS} & \textbf{0.78} & \textbf{0.80} & \textbf{0.82} &\textbf{0.82} & \textbf{0.75} &\textbf{0.64} \\
\bottomrule
\end{tabular}
}
\vspace{-8pt}
\end{table}
\begin{comment}
\begin{table}[ht]
\caption{Emphasis Recognition Accuracy of Different Models. }
\label{tab:emphasis_recognition}
\centering
\resizebox{0.48\textwidth}{!}{ % Resize the table to 50% of the text width
\begin{tabular}{ l c c c c c c }
\toprule
\textbf{Model} & \textbf{Mean} & \textbf{Neutral} & \textbf{Angry} & \textbf{Happy} & \textbf{Sad} & \textbf{Surprise} \\
\midrule
EME-TTS w/o EPE & 0.73 & 0.77 & 0.75 & \textbf{0.82} & \textbf{0.75} & 0.55 \\
\midrule
\textbf{EME-TTS} & \textbf{0.78} & \textbf{0.80} & \textbf{0.82} &\textbf{0.82} & \textbf{0.75} &\textbf{0.64} \\
\bottomrule
\end{tabular}
}
\end{table}
\end{comment}
\subsubsection{Emotion Accuracy}
To demonstrate that controlling emphasis in EME-TTS enhances the accuracy of perceived emotions in synthesized speech, we conducted both objective and subjective evaluations. For comparison, we included EmoSpeech \cite{diatlova23_ssw} and CosyVoice2-0.5B-Instruct (CosyVoice2) \cite{du2024cosyvoice} as baseline models. CosyVoice2 provides multiple inference modes; To ensure fairness, all CosyVoice2-generated samples were conditioned on a \textit{neutral} reference speaker’s audio and a textual emotion prompt, ensuring that only the speaker's identity and emotion labels were provided as input. During inference, our proposed model consistently utilized a large language model \cite{achiam2023gpt} to predict suitable emphasis positions, which were then used as input for testing.
%\begin{comment}
% ---- 客观评价表 (Objective Evaluation) ----
\begin{table}[ht]
\vspace{-4pt}
\caption{Objective Evaluation of Different Models on Emotion Accuracy. }
\label{tab:objective_evaluation}
\centering
%\setlength{\abovecaptionskip}{3pt} % 减少标题上方的间距
%\setlength{\belowcaptionskip}{3pt} % 减少标题下方的间距
\resizebox{0.46\textwidth}{!}{ % Resize to fit within the page width
\fontsize{18pt}{24pt}\selectfont % 调整字体大小
\begin{tabular}{ l c c c c c c }
\toprule
\textbf{Model} & \textbf{Mean} & \textbf{Neutral} & \textbf{Angry} & \textbf{Happy} & \textbf{Sad} & \textbf{Surprise} \\
\midrule
CosyVoice2 \cite{du2024cosyvoice} & 0.68 & \textbf{0.99} & 0.51 & \textbf{0.73} & 0.52&0.36 \\
EmoSpeech \cite{diatlova23_ssw} & 0.72 & \textbf{0.99} & \textbf{0.91} & 0.69 & 0.54 & \textbf{0.48} \\
EME-TTS w/o EPE & \textbf{0.74} & \textbf{0.99} & 0.90 & 0.71 & 0.60 & 0.47 \\
\midrule
\textbf{EME-TTS} & 0.73 & \textbf{0.99} & 0.87 & 0.70 & \textbf{0.61} & 0.47 \\
\bottomrule
\end{tabular}
}
\vspace{-4pt}
\end{table}
%\end{comment}
\begin{comment}
% ---- 客观评价表 (Objective Evaluation) ----
\begin{table}[!htb]
\vspace{-0pt} % 减少表格前的空白
\caption{Objective Evaluation of Different Models on Emotion Accuracy.}
\label{tab:objective_evaluation}
\centering
\renewcommand{\arraystretch}{1.0} % 增加行距，提升可读性
\setlength{\abovecaptionskip}{3pt} % 减少标题上方的间距
\setlength{\belowcaptionskip}{3pt} % 减少标题下方的间距
\resizebox{0.46\textwidth}{!}{
\fontsize{18pt}{24pt}\selectfont % 调整字体大小，增强可读性
\begin{tabular}{ l c c c c c c }
\toprule
\textbf{Model} & \textbf{Mean} & \textbf{Neutral} & \textbf{Angry} & \textbf{Happy} & \textbf{Sad} & \textbf{Surprise} \\
\midrule
CosyVoice2 \cite{du2024cosyvoice} & 0.68 & \textbf{0.99} & 0.51 & \textbf{0.73} & 0.52 & 0.36 \\
EmoSpeech \cite{diatlova23_ssw} & 0.72 & \textbf{0.99} & \textbf{0.91} & 0.69 & 0.54 & \textbf{0.48} \\
EME-TTS w/o EPE & \textbf{0.74} & \textbf{0.99} & 0.90 & 0.71 & 0.60 & 0.47 \\
\midrule
\textbf{EME-TTS} & 0.73 & \textbf{0.99} & 0.87 & 0.70 & \textbf{0.61} & 0.47 \\
\bottomrule
\end{tabular}
}
\vspace{-0pt} % 减少表格后的空白
\end{table}
\end{comment}
From an objective perspective, we utilized the Emotion2vec-plus-large model \cite{ma-etal-2024-emotion2vec} to evaluate the emotional accuracy of 1,550 synthesized audio samples from each model in the test set. The recognition outcome was assigned a score of 1 for correct classifications and 0 for incorrect ones, from which the overall accuracy was computed. The results shown in Table 2 indicate that while local emphasis control did not introduce substantial changes in global prosody, it notably improved the recognition of \textit{sad} emotions in objective evaluation. This improvement is attributed to the increased duration of emphasized regions, which effectively enhanced emotion perception.
On the subjective side, in task 2 (EAT-EMO), participants were asked to identify the emotions of 80 randomly shuffled audio samples. Similarly, scores of 1 and 0 were assigned for correct and incorrect classifications respectively, to assess the emotional accuracy of the synthesized speech. However, as shown in Table 3, the subjective evaluation revealed unexpected shifts in emotion perception due to the introduction of emphasis. First, emphasis increased the likelihood of \textit{neutral} speech being perceived as emotional, leading to a slight decrease in the accuracy of \textit{neutral} emotion expression. Second, unlike CosyVoice2 \cite{du2024cosyvoice}, which struggled to synthesize \textit{angry} and \textit{sad} emotions in this inference mode, EME-TTS produced speech that made these emotions more perceptible, outperforming both CosyVoice2 \cite{du2024cosyvoice} and EmoSpeech \cite{diatlova23_ssw}. For \textit{happy} and \textit{surprise} emotions, EME-TTS achieved accuracy levels comparable to other models. These results demonstrate that EME-TTS achieves higher emotion accuracy in synthesized speech compared to baseline models, highlighting its overall effectiveness in generating emotionally expressive speech.
% ---- 主观评价表 (Subjective Evaluation) ----
\begin{table}[ht]
\vspace{-1pt}
\caption{Subjective Evaluation of Different Models on Emotion Accuracy. }
\label{tab:subjective_evaluation}
\centering
\resizebox{0.46\textwidth}{!}{ % Resize to fit within the page width
\fontsize{18pt}{24pt}\selectfont % 调整字体大小
\begin{tabular}{ l c c c c c c }
\toprule
\textbf{Model} & \textbf{Mean} & \textbf{Neutral} & \textbf{Angry} & \textbf{Happy} & \textbf{Sad} & \textbf{Surprise} \\
\midrule
CosyVoice2 \cite{du2024cosyvoice} & 0.48 & \textbf{0.93} & 0.07 & 0.34 & 0.36 & \textbf{0.70} \\
EmoSpeech \cite{diatlova23_ssw} & 0.58 & 0.86 & 0.48 & \textbf{0.36} & 0.50 & \textbf{0.70} \\
EME-TTS w/o EPE & 0.58 & 0.68 & 0.57 & 0.27 & 0.80 & 0.59 \\
\midrule
\textbf{EME-TTS} & \textbf{0.67} & 0.80 & \textbf{0.75} & 0.32 & \textbf{0.82} & 0.68 \\
\bottomrule
\end{tabular}
}
\vspace{-8pt}
\end{table}
\subsubsection{Improvement of Emotional Expressiveness Through Emphasis}
To assess how emphasis enhances emotional expressiveness in EME-TTS, we conducted a ranking experiment as task 3 (EEPT). Participants evaluated 30 sets of speech samples, each containing outputs from four models, based on perceived emotional expressiveness. Among them, 10 sets were derived from a short passage with contextual information. Within each set, samples were ranked from 1 (least expressive) to 4 (most expressive). Figure 2 illustrates the ranking distribution of emotional expressiveness across different TTS models. Participants evaluated speech samples based on perceived emotional expressiveness, with (a) assessing individual sentences and (b) ranking sentences within a contextualized passage. The results indicate that EME-TTS consistently received the highest rankings, especially in the contextualized setting. This suggests that surrounding linguistic context strengthens the semantic foundation for emphasis, further enhancing emotional expressiveness.
\begin{figure}[htbp]
\vspace{-7pt}
\setlength{\abovecaptionskip}{5pt} % 调整标题与图片之间的间距
\setlength{\belowcaptionskip}{5pt} % 调整标题与图片之间的间距
\centering
\begin{minipage}[t]{0.5\linewidth}
\centering
\includegraphics[width=\linewidth]{myplot333.png}
\subcaption{Isolated Sentences}\label{figure4} % 子标题(a)
\end{minipage}%
\begin{minipage}[t]{0.5\linewidth}
\centering
\includegraphics[width=\linewidth]{myplot444.png}
\subcaption{Contextualized Sentences}\label{figure2} % 子标题(b)
\end{minipage}
\caption{Effect of Emphasis on Emotional Expressiveness in TTS Models.} % 大标题
\label{figure1}
\vspace{-20pt}
\end{figure}
\subsubsection{Speech Quality and Naturalness}
We assess the quality and naturalness of the synthesized speech through both objective and subjective evaluations. Objectively, we utilize the NISQA library \cite{mittag20_interspeech} to predict naturalness scores ratings on a 5-point scale. Subjectively, participants completed task 4 (MOS Rating), in which they rated 100 randomly shuffled speech samples for overall audio quality and naturalness, ranging from 1 (bad) to 5 (excellent). These 100 utterances were selected from the test set, ensuring an equal distribution of emotions. Table 4 indicates that EME-TTS mitigates artifacts introduced by emphasis control, enhancing synthesis quality.
\begin{table}[ht]
\vspace{-2pt}
\caption{Comparison of Models for MOS and NISQA Scores. }
\label{tab:model_comparison}
\centering
\resizebox{0.350\textwidth}{!}{ % Resize the table to 50% of the text width
%\fontsize{18pt}{20pt}\selectfont % 调整字体大小
\begin{tabular}{ l c c }
\toprule
\textbf{Model} & \textbf{MOS (\( \uparrow \))} & \textbf{NISQA (\( \uparrow \))} \\
\midrule
Original & 4.94 $\pm$ 0.03 & 4.17 $\pm$ 0.57 \\
Reconstructed & 4.86 $\pm$ 0.08 & 4.11 $\pm$ 0.58 \\
\midrule
EmoSpeech \cite{diatlova23_ssw} & 4.14 $\pm$ 0.20 & 3.71 $\pm$ 0.74 \\
EME-TTS w/o EPE & 3.98 $\pm$ 0.32 & 3.66 $\pm$ 0.68 \\
\midrule
\textbf{EME-TTS} & \textbf{4.22 $\pm$ 0.28} & \textbf{3.76 $\pm$ 0.60} \\
\bottomrule
\end{tabular}
}
\vspace{-14pt}
\end{table}
\section{Conclusion}
This paper presents EME-TTS, a framework that explores how emphasis enhances emotional expressiveness and how to maintain its perceptual clarity and stability across emotions. By leveraging variance-based emphasis features, weakly supervised learning, and EPE, EME-TTS demonstrate its effectiveness in generating emotionally expressive speech with clear and controllable emphasis. For future work, we aim to further investigate the role of different emphasis strategies in enhancing emotional expressiveness, particularly in achieving improvements for specific emotions where the current model's enhancements remain limited.
\section{Acknowledgements}
This work was supported in part by the Scientific Research Staring Foundation of Hangzhou Institute for Advanced Study (2024HIASC2001), in part by Zhejiang Provincial Natural Science Foundation of China (No. LQN25F020001), and in part by the Key R\&D Program of Zhejiang (2025C01104).
\bibliographystyle{IEEEtran}
\bibliography{mybib}
\end{document}
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{waswani2017attention}
A.~Waswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~Gomez, L.~Kaiser, and I.~Polosukhin, ``Attention is all you need,'' in \emph{NIPS}, 2017.
\bibitem{ren2020fastspeech}
Y.~Ren, C.~Hu, X.~Tan, T.~Qin, S.~Zhao, Z.~Zhao, and T.~Liu, ``Fastspeech 2: Fast and high-quality end-to-end text to speech,'' in \emph{9th International Conference on Learning Representations, {ICLR} 2021, Virtual Event, Austria, May 3-7, 2021}.\hskip 1em plus 0.5em minus 0.4em\relax OpenReview.net, 2021.
\bibitem{kim2020glow}
J.~Kim, S.~Kim, J.~Kong, and S.~Yoon, ``Glow-tts: A generative flow for text-to-speech via monotonic alignment search,'' \emph{Advances in Neural Information Processing Systems}, vol.~33, pp. 8067--8077, 2020.
\bibitem{kim2021conditional}
J.~Kim, J.~Kong, and J.~Son, ``Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,'' in \emph{International Conference on Machine Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2021, pp. 5530--5540.
\bibitem{popov2021grad}
V.~Popov, I.~Vovk, V.~Gogoryan, T.~Sadekova, and M.~Kudinov, ``Grad-tts: A diffusion probabilistic model for text-to-speech,'' in \emph{International Conference on Machine Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2021, pp. 8599--8608.
\bibitem{tan2024naturalspeech}
X.~Tan, J.~Chen, H.~Liu, J.~Cong, C.~Zhang, Y.~Liu, X.~Wang, Y.~Leng, Y.~Yi, L.~He \emph{et~al.}, ``Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,'' \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence}, 2024.
\bibitem{ExpressiveFastSpeech2}
\BIBentryALTinterwordspacing
{K. Lee}. {"Expressive-FastSpeech2"}. [Online]. Available: \url{https://github.com/keonlee9420/Expressive-FastSpeech2}
\BIBentrySTDinterwordspacing
\bibitem{tits2020exploring}
N.~Tits, K.~El~Haddad, and T.~Dutoit, ``Exploring transfer learning for low resource emotional tts,'' in \emph{Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 1}.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2020, pp. 52--60.
\bibitem{diatlova23_ssw}
D.~Diatlova and V.~Shutov, ``Emospeech: guiding fastspeech2 towards emotional text to speech,'' in \emph{12th ISCA Speech Synthesis Workshop (SSW2023)}, 2023, pp. 106--112.
\bibitem{guo2023emodiff}
Y.~Guo, C.~Du, X.~Chen, and K.~Yu, ``Emodiff: Intensity controllable emotional text-to-speech with soft-label guidance,'' in \emph{ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2023, pp. 1--5.
\bibitem{li2021controllable}
T.~Li, S.~Yang, L.~Xue, and L.~Xie, ``Controllable emotion transfer for end-to-end speech synthesis,'' in \emph{2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 1--5.
\bibitem{lei2022msemotts}
Y.~Lei, S.~Yang, X.~Wang, and L.~Xie, ``Msemotts: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis,'' \emph{IEEE/ACM Transactions on Audio, Speech, and Language Processing}, vol.~30, pp. 853--864, 2022.
\bibitem{zaidi22b_interspeech}
J.~Zaïdi, H.~Seuté, B.~{van Niekerk}, and M.-A. Carbonneau, ``Daft-exprt: Cross-speaker prosody transfer on any text for expressive speech synthesis,'' in \emph{Interspeech 2022}, 2022, pp. 4591--4595.
\bibitem{eriksson2015acoustics}
A.~Eriksson and M.~Heldner, ``The acoustics of word stress in english as a function of stress level and speaking style,'' in \emph{16th Annual Conference of the International Speech Communication Association (INTERSPEECH 2015), Dresden, Germany, September 6-10, 2015}, 2015, pp. 41--45.
\bibitem{eriksson2016acoustics}
A.~Eriksson, P.~M. Bertinetto, M.~Heldner, R.~Nodari, and G.~Lenoci, ``The acoustics of lexical stress in italian as a function of stress level and speaking style,'' in \emph{Interspeech 2016}.\hskip 1em plus 0.5em minus 0.4em\relax The International Speech Communication Association (ISCA), 2016, pp. 1059--1063.
\bibitem{eriksson2020lexical}
A.~Eriksson, R.~Nodari, J.~{\v{S}}imko, A.~Suni, and M.~Vainio, ``Lexical stress perception as a function of acoustic properties and the native language of the listener,'' in \emph{The 10th International Conference on Speech Prosody: Communicative and Interactive Prosody}.\hskip 1em plus 0.5em minus 0.4em\relax ISCA, 2020, pp. 449--453.
\bibitem{li2018emphatic}
R.~Li, Z.~Wu, Y.~Huang, J.~Jia, H.~Meng, and L.~Cai, ``Emphatic speech generation with conditioned input layer and bidirectional lstms for expressive speech synthesis,'' in \emph{2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2018, pp. 5129--5133.
\bibitem{shechtman2021supervised}
S.~Shechtman, R.~Fernandez, and D.~Haws, ``Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis,'' in \emph{2021 IEEE Spoken Language Technology Workshop (SLT)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 431--437.
\bibitem{seshadri22_interspeech}
S.~Seshadri, T.~Raitio, D.~Castellani, and J.~Li, ``Emphasis control for parallel neural tts,'' in \emph{Interspeech 2022}, 2022, pp. 3378--3382.
\bibitem{zhong23_interspeech}
Y.~Zhong, C.~Zhang, X.~Liu, C.~Sun, W.~Deng, H.~Hu, and Z.~Sun, ``Ee-tts: Emphatic expressive tts with linguistic information,'' in \emph{Interspeech 2023}, 2023, pp. 4873--4877.
\bibitem{zhao2023survey}
W.~X. Zhao, K.~Zhou, J.~Li, T.~Tang, X.~Wang, Y.~Hou, Y.~Min, B.~Zhang, J.~Zhang, Z.~Dong \emph{et~al.}, ``A survey of large language models,'' \emph{arXiv preprint arXiv:2303.18223}, 2023.
\bibitem{seyssel-etal-2024-emphassess}
M.~de~Seyssel, A.~D{'}Avirro, A.~Williams, and E.~Dupoux, ``{E}mph{A}ssess : a prosodic benchmark on assessing emphasis transfer in speech-to-speech models,'' in \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, 2024, pp. 495--507.
\bibitem{zhou2022emotional}
K.~Zhou, B.~Sisman, R.~Liu, and H.~Li, ``Emotional voice conversion: Theory, databases and esd,'' \emph{Speech Communication}, vol. 137, pp. 1--18, 2022.
\bibitem{suni2017hierarchical}
A.~Suni, J.~{\v{S}}imko, D.~Aalto, and M.~Vainio, ``Hierarchical representation and estimation of prosody using continuous wavelet transform,'' \emph{Computer Speech \& Language}, vol.~45, pp. 123--136, 2017.
\bibitem{wu22f_interspeech}
Y.~Wu, X.~Tan, B.~Li, L.~He, S.~Zhao, R.~Song, T.~Qin, and T.-Y. Liu, ``Adaspeech 4: Adaptive text to speech in zero-shot scenarios,'' in \emph{Interspeech 2022}, 2022, pp. 2568--2572.
\bibitem{kaneko2022istftnet}
T.~Kaneko, K.~Tanaka, H.~Kameoka, and S.~Seki, ``istftnet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform,'' in \emph{ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2022, pp. 6207--6211.
\bibitem{diederik2014adam}
P.~K. Diederik, ``Adam: A method for stochastic optimization,'' \emph{ICLR}, 2015.
\bibitem{du2024cosyvoice}
Z.~Du, Y.~Wang, Q.~Chen, X.~Shi, X.~Lv, T.~Zhao, Z.~Gao, Y.~Yang, C.~Gao, H.~Wang \emph{et~al.}, ``Cosyvoice 2: Scalable streaming speech synthesis with large language models,'' \emph{arXiv preprint arXiv:2412.10117}, 2024.
\bibitem{achiam2023gpt}
J.~Achiam, S.~Adler, S.~Agarwal, L.~Ahmad, I.~Akkaya, F.~L. Aleman, D.~Almeida, J.~Altenschmidt, S.~Altman, S.~Anadkat \emph{et~al.}, ``{GPT-4} technical report,'' \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem{ma-etal-2024-emotion2vec}
Z.~Ma, Z.~Zheng, J.~Ye, J.~Li, Z.~Gao, S.~Zhang, and X.~Chen, ``emotion2vec: Self-supervised pre-training for speech emotion representation,'' in \emph{Findings of the Association for Computational Linguistics: ACL 2024}, 2024, pp. 15\,747--15\,760.
\bibitem{mittag20_interspeech}
G.~Mittag and S.~Möller, ``Deep learning based assessment of synthetic speech naturalness,'' in \emph{Interspeech 2020}, 2020, pp. 1748--1752.
\end{thebibliography}