Is the Ideal Ratio Mask Really the Best? -- Exploring the Best Extraction Performance and Optimal Mask of Mask-based Beamformers
Extracted source text · arXiv 2309.12065
\documentclass[conference,a4paper]{APSIPA2021}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{threeparttable}
\usepackage[backend=biber,style=ieee,]{biblatex}
\addbibresource{paperpile.bib}
%\setlength{\voffset}{-2.0cm}
%\setlength{\hoffset}{-1cm}
\usepackage{geometry}
\geometry{a4paper, top=19mm, bottom=43mm, right=13mm, left=13mm}
% \geometry{a4paper, top=19.1mm, bottom=43.1mm, right=13mm, left=13mm, columnsep=0.241in}
% Optional setting if the PDF file couldn't pass the IEEE Xplore format examination.
\usepackage{fancyhdr}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\fancypagestyle{firststyle}{
\fancyhf{}
\fancyhead[C]{2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)}
}
% My modification
\usepackage{bm}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\vecw}{\bm{w}}
\newcommand{\vecs}{\bm{s}}
\newcommand{\vecn}{\bm{n}}
\newcommand{\vecx}{\bm{x}}
\newcommand{\phix}{\bm{\Phi}_{x}}
\newcommand{\phis}{\hat{\bm{\Phi}}_{s}}
\newcommand{\phin}{\hat{\bm{\Phi}}_{n}}
\newcommand{\maskset}{\mathcal M}
%\newcommand{\matphi}{\bm{\Phi}}
\newcommand{\htp}[1]{{#1}^{\rm H}}
\newcommand{\tp}[1]{{#1}^{\rm T}}
\newcommand{\conj}[1]{\overline{#1}}
\newcommand{\average}[1]{\left\langle#1\right\rangle}
\newcommand{\abs}[1]{\left|#1\right|}
%\newcommand{\alr}[1]{\left\langle#1\right\rangle}
\newcommand{\reffig}[1]{Fig.~\ref{#1}}
\newcommand{\reftable}[1]{Table~\ref{#1}}
\newcommand{\refeqn}[1]{(\ref{#1})}
\newcommand{\refsec}[1]{Section~\ref{#1}}
\newcommand{\nop}[1]{}
\begin{document}
\title{
Is the Ideal Ratio Mask Really the Best? --- Exploring the Best Extraction Performance and Optimal Mask of Mask-based Beamformers
}
\author{
\authorblockN{
Atsuo Hiroe\authorrefmark{1},
Katsutoshi Itoyama\authorrefmark{1}\authorrefmark{2}, and
Kazuhiro Nakadai\authorrefmark{1}
}
\authorblockA{
\authorrefmark{1}
Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, Tokyo, Japan}
\authorblockA{
\authorrefmark{2}
Honda Research Institute Japan Co., Ltd., Saitama, Japan\\
E-mail: \{hiroe,itoyama,nakadai\}@ra.sc.e.titech.ac.jp}
}
\maketitle
%\thispagestyle{firststyle}
\pagestyle{fancy}
%\pagestyle{empty}
\begin{abstract}
This study investigates mask-based beamformers (BFs), which estimate filters to extract target speech using time-frequency masks. Although several BF methods have been proposed, the following aspects are yet to be comprehensively investigated. 1) Which BF can provide the best extraction performance in terms of the closeness of the BF output to the target speech? 2) Is the optimal mask for the best performance common for all BFs? 3) Is the ideal ratio mask (IRM) identical to the optimal mask? Accordingly, we investigate these issues considering four mask-based BFs: the maximum signal-to-noise ratio BF, two variants of this, and the multichannel Wiener filter (MWF) BF. To obtain the optimal mask corresponding to the peak performance for each BF, we employ an approach that minimizes the mean square error between the BF output and target speech for each utterance. Via the experiments with the CHiME-3 dataset, we verify that the four BFs have the same peak performance as the upper bound provided by the ideal MWF BF, whereas the optimal mask depends on the adopted BF and differs from the IRM. These observations differ from the conventional idea that the optimal mask is common for all BFs and that peak performance differs for each BF. Hence, this study contributes to the design of mask-based BFs.
\end{abstract}
%%%%%
\section{Introduction}\label{sec:intro}
Target speech extraction is effective in improving speech intelligibility in telecommunication systems and the performance of automatic speech recognition systems~\cite{Chen2018-rl}. Therefore, beamformers (BFs) are utilized to avoid nonlinear distortions such as musical noises and spectral distortions~\cite{Mizumachi2016-jd, Mizumachi2019-hi}. In the last decade, combined frameworks comprising BFs and deep neural networks (DNNs), referred to as mask-based BFs, have been proposed \cite{Heymann2016-eb, Heymann2016-sy, Erdogan2016-jc}. In these frameworks, DNNs generate one or two time-frequency (TF) masks corresponding to the target, interferences, or both to inform the BF of the sound to be enhanced or suppressed. Afterward, the BF estimates a filter for extracting the target using masks. For filter estimation, the following BF methods are adopted:
1) maximum signal-to-noise ratio (max-SNR) or generalized eigenvalue (GEV) BF~\cite{Heymann2016-eb, Heymann2016-sy, Drude2019-so},
~2) minimum variance distortionless response (MVDR) BF~\cite{Heymann2016-sy, Erdogan2016-jc, Souden2010-kp}, and
3) multichannel Wiener filter (MWF) BF~\cite{Stenzel2013-hw, Nugraha2016-rn, Pfeifenberger2017-zp}.
Several mask types have been examined to achieve improved extraction performance.
Initially, ideal binary masks (IBMs) were employed to train DNNs for mask generation \cite{Heymann2016-eb, Heymann2016-sy}. Subsequently, ideal ratio masks (IRMs) have been utilized \cite{Erdogan2016-jc,Wang2018-yy,Pfeifenberger2017-zp}.
Overviewing these studies, we concluded that the following aspects are yet to be comprehensively investigated:
\begin{enumerate}
\item Which BF can achieve the best performance if an optimal mask is provided for each BF method?
\item Is an optimal mask common for all BF methods?
\item Are conventional ideal masks such as IRMs identical to an optimal mask?
\end{enumerate}
Regarding these aspects, several studies such as \cite{Heymann2016-sy,Boeddeker2018-ww}, and \cite{Wang2018-tl} considered that the mask optimal for the single-channel TF masking \cite{Wang2014-gj,Wang2018-yy} should commonly be optimal for all BF methods. However, this assumption was not verified in these studies. Moreover, they compared multiple BF methods that employed the same mask in terms of extraction performance. However, these results do not answer the first aspect unless the optimal mask is common.
The remainder of this paper is organized as follows. Sections~\ref{sec:related-works} and \ref{sec:framework} explain the related work and BF methods presented in this study, respectively. \refsec{sec:optimal-mask} examines the method for obtaining the optimal mask for each BF method. \refsec{sec:experiments} verifies the aforementioned points experimentally while \refsec{sec:discussion} discusses the experimental results. Finally, \refsec{sec:conclusions} concludes the study.
%%%%%
\section{Related work} \label{sec:related-works}
First, we overview the history of mask-based BFs. The max-SNR, MVDR, and MWF BFs employ the statistics of the target, interferences, or both. The statistics are called target (or speech) and interference (or noise) covariance matrices. Given that the accuracy of both statistics affects the extraction performance, estimating them accurately is a fundamental issue \cite{Araki2007-yp, Warsitz2007-fn}.
In \cite{Heymann2016-eb, Heymann2016-sy}, both statistics were computed using two binary masks, each of which represents periods when only the target or interferences are present, and the masks were generated using a properly trained DNN. This idea was first adopted for the max-SNR and MVDR BFs \cite{Heymann2016-eb, Heymann2016-sy, Erdogan2016-jc}, then applied to the MWF BF \cite{Heymann2018-mf}.
Second, we mention the studies that compare mask-based BFs. At least three methods have been adopted as aforementioned, and several studies have compared two or three of them, as presented in \reftable{tab:comparison}. However, the best-performing method depends on experimental setups. Moreover, it is reported in \cite{Heymann2018-mf} that the difference between the max-SNR and MWF BFs is marginal and the performance depends on the number of microphones used.
\begin{table}[b]
\centering
\caption{Studies comparing multiple BF methods. Methods in bold type represent those that performed the best in each study. In \cite{Heymann2018-mf}, the performance depends on the number of microphones. (SNR: Signal-to-noise ratio, PESQ: Perceptual evaluation of speech quality, WER: Word error rate)}
\label{tab:comparison}
\begin{tabular}{l|l|l}
\hline
& Metric & Methods compared \\
\hline
Heymann+16 \cite{Heymann2016-sy} & SNR & {\bf Max-SNR}, MVDR \\
Boeddeker+18\cite{Boeddeker2018-ww} & SNR,PESQ & {\bf Max-SNR}, MVDR \\
Wang+18 \cite{Wang2018-tl} & WER & Max-SNR, MVDR, {\bf MWF}\\
Heymann+18 \cite{Heymann2018-mf} & WER & Max-SNR, MWF \\
Shimada+18 \cite{Shimada2018-uy} & WER & MVDR, {\bf MWF} \\
\hline
\end{tabular}
\end{table}
Third, we mention the mask types considered to be optimal for the mask-based BFs.
Initially, the IBMs were considered to be optimal \cite{Heymann2016-eb, Heymann2016-sy}; then, the IRMs were considered optimal \cite{Erdogan2016-jc,Tu2017-ey,Wang2018-yy,Tu2019-qp,Chakrabarty2019-th}.
These ideas were based on the findings in the single-channel TF masking \cite{Wang2014-gj,Wang2018-yy}. However, these studies did not investigate whether these findings really apply to the mask-based BFs or the optimal mask is common for any BF method.
%%%%%
\section{Framework of mask-based BFs} \label{sec:framework}
In this study, we define the best extraction performance as the BF output closest to the target in the TF domain, considering that a significant goal of BFs is to extract (or estimate) the target. Moreover, similar to conventional TF masks, we have the constraint that mask values are nonnegative and real-valued.
\reffig{fig:mask_curve} illustrates our idea. The vertical and horizontal axes indicate the closeness of the BF output to the target and the variation of the mask values, respectively. Although the mask values vary multidimensionally, this figure conceptually represents the variation as a single axis. In the mask-based BFs, the extraction performance should depend on the variation and exhibit the peak on a particular mask. We refer to this mask as the optimal one.
Considering that multiple BF methods and mask types are employed, we can rephrase the questions mentioned in \refsec{sec:intro} as follows:
\begin{description}
\item[\bf Issue 1:] \quad Which BF method has the highest peak, or are all peaks of the same height?
\item[\bf Issue 2:] \quad Is the optimal mask common for all BF methods, or dependent on each one?
\item[\bf Issue 3:] \quad Can the mask considered to be ideal achieve peak performance? More particularly, is the IRM optimal?
\end{description}
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{mask_curve.pdf}
\caption{Conceptual plot of the relationship between the closeness of the BF output to the target and mask values}
\label{fig:mask_curve}
\end{figure}
We assumed that the studies presented in \reftable{tab:comparison} considered these issues as in \reffig{fig:conventional_image}. This indicates that the mask mentioned in each study is optimal for multiple BF methods such as BFs 1 and 2, whereas each BF method demonstrates a different height of the peak. However, this idea is yet to be verified.
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{conventional_image.pdf}
\caption{Conventional ideas on the peak performance and optimal mask for multiple BF methods}
\label{fig:conventional_image}
\end{figure}
Considering that this study is the first to examine these issues, we focus on the following methods for simplicity:
\begin{itemize}
\item Mask-based MWF BF, which employs a single mask
\item Mask-based max-SNR BF, which utilizes two masks, and its variants that employ a single mask
\end{itemize}
We do not include the MVDR BF in this study, given that the method has another issue regarding the accuracy of the estimation of the steering vector \cite{Tu2018-ts}.
\subsection{Signal models}
In this study, all signals are in the TF domain. For simplicity, the frequency index is omitted, whereas the frame index $t$ is adopted. Let $\vecx(t)=\tp{[x_{1}(t),\ldots ,x_{N}(t)]}$ be an observation vector obtained with $N$ microphones. The observation $\vecx(t)$ can be expressed as the following summation:
\begin{equation}
\vecx(t) = \vecs(t) + \vecn(t), \label{eqn:components-of-x}
\end{equation}
where $\vecs(t)=\tp{[s_1(1), \ldots, s_N(t)]}$ denotes the components arriving from the target and $\vecn(t)=\tp{[n_1(1), \ldots, n_N(t)]}$ represents the residuals called interferences. Using the observation $\vecx(t)$ and BF filter $\vecw$, the BF output $y(t)$ is expressed as
\begin{equation}
y(t) = \htp{\vecw} \vecx(t). \label{eqn:bf-output}
\end{equation}
We use the following three covariance matrices:
\begin{eqnarray}
\phix &=& \average{\vecx(t)\htp{\vecx(t)}}_{t},
\label{eqn:phix}\\
\phis &=& \average{m_{s}(t)\vecx(t)\htp{\vecx(t)}}_{t},
\label{eqn:phis}\\
\phin &=& \average{m_{n}(t)\vecx(t)\htp{\vecx(t)}}_{t},
\label{eqn:phin}
\end{eqnarray}
where $m_{s}(t)$ and $m_{n}(t)$ denote TF masks for the target and interferences, respectively, and $\average{\cdot}_{t}$ computes the average over $t$. Each mask comprises nonnegative real values. We refer to these matrices as observation, target, and interference covariance matrices, respectively. Unlike $\phix$, both $\phis$ and $\phin$ are estimated matrices computed from the masks and observations without using $\vecs(t)$ and $\vecn(t)$.
Moreover, consider ${\rm GEV_{max}}(\bm{A},\bm{B})$ and ${\rm GEV_{min}}(\bm{A},\bm{B})$ to be the eigenvectors corresponding to the maximum and minimum eigenvalues in the following GEV problem, respectively:
\begin{equation}
\bm{Aw} = \lambda \bm{Bw}.
\end{equation}
\subsection{Mask-based and ideal MWF BFs}
\label{subsec:mwf}
The MWF BF is formulated as a problem of minimizing the mean square error (MSE) between the BF output and the corresponding reference $p(t)$ \cite{Wang2018-tl}:
\begin{eqnarray}
\vecw &=& \argmin_{\vecw} \average{\left|p(t) - \htp{\vecw}\vecx(t)\right|^{2}}_{t}
\label{eqn:mwf-def} \\
&=& \phix^{-1} \average{\vecx(t) \conj{p(t)}}_{t},
\end{eqnarray}
where $\conj{p(t)}$ denotes the conjugate of $p(t)$. The mask-based MWF BF employs the masked observation as the reference, and thus, corresponds to the case $p(t)=m_{s}(t)x_{k}(t)$ as
\begin{eqnarray}
\vecw_{\rm mwf} &=& \argmin_{\vecw} \average{\left|m_{s}(t)x_{k}(t) - \htp{\vecw}\vecx(t)\right|^{2}}_{t}
\label{eqn:mask-mwf-def} \\
&=& \phix^{-1} \average{m_{s}(t) \vecx(t) \conj{x_{k}(t)}}_{t},
\label{eqn:mask-mwf}
\end{eqnarray}
where $x_{k}(t)$ denotes the observation obtained with the $k$th microphone.
Another significant variant is the ideal MWF BF, which provides the upper-bound extraction performance for all BFs \cite{Malek2020-nw}.
When $\vecs(t)$ in \refeqn{eqn:components-of-x} is known, the upper-bound extraction performance can be achieved using the MWF with $p(t)=\vecs_{k}(t)$.
The corresponding filter is obtained as
\begin{eqnarray}
\vecw_{\rm ideal} &=& \argmin_{\vecw} \average{\left|s_{k}(t) - \htp{\vecw}\vecx(t)\right|^{2}}_{t}
\label{eqn:ideal-mwf-def} \\
&=& \phix^{-1} \average{\vecx(t) \conj{s_{k}(t)}}_{t}.
\label{eqn:ideal-mwf}
\end{eqnarray}
Note that the ideal MWF BF is not a particular case of the mask-based one. This is because a mask value $m_{s}(t)$ constrained to be real-valued and nonnegative cannot render \refeqn{eqn:mask-mwf-def} equivalent to \refeqn{eqn:ideal-mwf-def}, whereas $m_{s}(t)$ that can take any complex values can. Therefore, it is not evident whether the mask-based MWF BF can achieve the same extraction performance as the ideal alternative.
\subsection{Max-SNR BF and its variants}
\label{subsec:max-snr}
The mask-based max-SNR BF is formulated as the following maximization problem \cite{Heymann2016-eb,Heymann2016-sy}:
\begin{eqnarray}
\vecw_{\rm snr}
&=& \argmax_{\vecw} \frac{\htp{\vecw}\phis\vecw}{\htp{\vecw}\phin\vecw}
\label{eqn:max-snr-ratio} \\
&=& {\rm GEV_{max}}\left(\phis, \phin\right).
\label{eqn:max-snr-gev}
\end{eqnarray}
Although this method originally utilizes two masks, we can derive two different variants that adopt a single mask by assuming the following relationship:
\begin{equation}
\phis + \phin = \phix .
\label{eqn:cov_summation}
\end{equation}
One variant utilizes a mask for the interferences \cite{Warsitz2007-fn, Hiroe2022-rd}:
\begin{eqnarray}
\vecw_{\rm nor}
&=& \argmin_{\vecw} \frac{\htp{\vecw}\phin\vecw}{\htp{\vecw}\phix\vecw}
\label{eqn:min-nor-ratio} \\
&=& {\rm GEV_{min}}\left(\phin, \phix\right).
\label{eqn:min-nor-gev}
\end{eqnarray}
We refer to this as the minimum noise-to-observation ratio (min-NOR) BF. Similarly, we can derive the other, which employs a mask for the target and is expressed as
\begin{eqnarray}
\vecw_{\rm sor}
&=& \argmax_{\vecw} \frac{\htp{\vecw}\phis\vecw}{\htp{\vecw}\phix\vecw}
\label{eqn:max-sor-ratio} \\
&=& {\rm GEV_{max}}\left(\phis, \phix\right).
\label{eqn:max-sor-gev}
\end{eqnarray}
We refer to this as the maximum signal-to-observation ratio (max-SOR) BF.
Furthermore, even without assuming \refeqn{eqn:cov_summation}, the min-NOR and max-SOR BFs can be equivalent to each other in terms of the filter estimation if one mask is computed from the other using \refeqn{eqn:sum-is-constant}.
\begin{equation}
m_{s}(t) + m_{n}(t) = \alpha,
\label{eqn:sum-is-constant}\\
\end{equation}
where $\alpha$ is a nonnegative value, such that all the mask values are nonnegative. For example, assigninig $m_{s}(t) = \alpha - m_{n}(t)$ to \refeqn{eqn:max-sor-ratio} and \refeqn{eqn:phis} results in the problem represented by \refeqn{eqn:min-nor-ratio} if $m_{n}(t)$ is nonnegative for all $t$.
For the max-SNR BF and its variants, the scales of the filter and BF outputs are uncertain, unlike the MWF BF. Hence, a post-process for determining the proper scales is required \cite{Araki2007-yp, Warsitz2007-fn}.
Considering that the formulation of these BFs differs from that of the MWF BF, it is not evident whether these BFs can achieve the same extraction performance as the ideal MWF BF.
%%%%%
\section{Obtaining the optimal mask} \label{sec:optimal-mask}
To explore the peak performance for each BF method, this study employs a bottom-up approach that obtains the optimal mask for each mixture of the target and interferences, instead of a priori deciding whether a particular mask type is optimal.
Let $\maskset$ be a set of mask values adopted in a BF. This set comprises $m_{s}(t)$, $m_{n}(t)$, or both for all $t$, depending on the BF method employed. We can formulate $\maskset$ as the solution to the problem of minimizing the following MSE:
\begin{equation}
\maskset = \argmin_{\maskset} \average{\left|s_{k}(t) - y(t)\right|^{2}}_{t},
\label{eqn:optimal-mask-def}
\end{equation}
where $s_{k}(t)$ and $y(t)$ denote the target included in the observation of the $k$th microphone and BF output, respectively.
To render the mask values nonnegative and to avoid both diverging the mask values and converging them to zero, we impose the following constraints on \refeqn{eqn:optimal-mask-def}:
\begin{eqnarray}
m(t) &\geq& 0 \quad \mbox{\rm for all $t$}, \label{eqn:constraint-nonnegative} \\
\average{m(t)^{2}}_{t} &=& 1, \label{eqn:constraint-unitvariance}
\end{eqnarray}
where $m(t)$ denotes $m_{s}(t)$ or $m_{n}(t)$.
In \refeqn{eqn:optimal-mask-def}, $y(t)$ is computed as follows. First, the BF filter $\vecw(t)$ is computed depending on the adopted BF method, then the $y(t)$ is computed using \refeqn{eqn:bf-output}. To determine the ideal scale of $y(t)$ independent of the BF method, we apply the following post-filtering process, referred to as the {\it ideal scaling}:
\begin{eqnarray}
\gamma &=& \frac{\average{s_{k}(t)\overline{y(t)}}_{t}} {\average{|y(t)|^{2}}_{t}},
\label{eqn:ideal-scaling}\\
y(t) &\leftarrow& \gamma y(t) \label{eqn:apply-ideal-scale}.
\end{eqnarray}
Owing to the ideal scaling, the constraint represented as \refeqn{eqn:constraint-unitvariance} does not affect the scale of $y(t)$.
Because $\maskset$, $\vecw(t)$, and $y(t)$ depend on each other, $\maskset$ cannot be obtained explicitly. In contrast, we employ the iterative algorithm based on gradient descent.
Given that the mask estimation process adopts no explicit association between the masks and sources, the obtained masks such as $m_{s}(t)$ and $m_{n}(t)$ do not necessarily correspond to the sources such as the target and interferences.
Equations \refeqn{eqn:optimal-mask-def}, \refeqn{eqn:constraint-nonnegative}, and \refeqn{eqn:constraint-unitvariance} might seem to solve the same problem as the ideal MWF BF, which is represented as \refeqn{eqn:ideal-mwf-def}, based on the iterative algorithm and consequently achieve the same extraction performance as the ideal MWF BF. However, such a perspective is incorrect, as mentioned in Sections~\ref{subsec:mwf} and \ref{subsec:max-snr}.
Rather, the objective of this study is to examine to what extent the output of the mask-based BF can approach that of the ideal MWF BF if the mask best fits \refeqn{eqn:optimal-mask-def}.
%%%%%
\section{Experiments} \label{sec:experiments}
To clarify Issues 1--3 mentioned in \refsec{sec:framework}, we conducted the following experiments:
\begin{enumerate}
\item Exploring the peak performance for each BF method
\item Verifying whether the optimal mask is common for all the BF methods
\item Examining whether the IRM can achieve the peak performance
\end{enumerate}
The following subsections first mention the dataset and system for these experiments, and then demonstrate the experimental results in order.
\subsection{Dataset and experimental system}
We employed the CHiME-3 simulated test set \cite{Barker2017-tr}, which comprises both 330 utterances from four speakers and four background (BG) noises. In this dataset, sound data were recorded at 16~kHz with six microphones attached to a tablet device. We generated the TF domain signals using the short-time Fourier transform with window and shift lengths of 1024 and 256, respectively.
\reffig{fig:system} illustrates the experimental system.
The modules labeled {\it Absolute value}, {\it Normalization}, and {\it Ideal scaling} correspond to \refeqn{eqn:constraint-nonnegative}, \refeqn{eqn:constraint-unitvariance}, and \refeqn{eqn:apply-ideal-scale}, respectively.
The observation data were generated by mixing clean speech and BG noise. To represent multiple scenarios in different SNRs, three multipliers such as 1, 2, and 4 were applied to the BG noise. We refer to these values as {\it BG multipliers}.
The BF output $y(t)$ was generated as explained in \refsec{sec:optimal-mask};
one or two masks were employed and the BF filter was estimated depending on the BF method, such as \refeqn{eqn:mask-mwf}, \refeqn{eqn:max-snr-gev}, \refeqn{eqn:min-nor-gev}, and \refeqn{eqn:max-sor-gev}.
Given that microphone \#5 is the closest to the speaker, we set $k$ to 5 in \refeqn{eqn:optimal-mask-def} and \refeqn{eqn:ideal-scaling}.
The backpropagation based on the gradient descent algorithm was utilized only in the first experiment to obtain the optimal mask for each utterance. The estimation of the BF filter and output was implemented in PyTorch \cite{Paszke2019-dm}, which supports the backpropagation of matrix operations in the complex number domain.
\begin{figure}[t]
\centering
\includegraphics[width=0.9\linewidth]{system.pdf}
\caption{Configuration of the system obtaining or employing the optimal mask(s) for each utterance}
\label{fig:system}
\end{figure}
\reftable{tab:obs-snr} presents average SNR scores of the observation of microphone \#5 for each BG multiplier.
\begin{table}[b]\begin{center}
\caption{SNR scores {[\rm dB]} of the observation of microphone \#5}
\label{tab:obs-snr}
\begin{tabular}{rrr} \hline
\multicolumn{3}{c}{BG multiplier} \\
\multicolumn{1}{c}{$\times$1} & \multicolumn{1}{c}{$\times$2} & \multicolumn{1}{c}{$\times$4} \\\hline
7.540 & 1.536 & -4.421 \\\hline
\end{tabular}
\end{center}\end{table}
For the evaluation score, we adopted the source-to-distortion ratio (SDR) \cite{Vincent2006-pg}, which is provided as a performance score in this dataset, considering that this score basically represents the closeness of the BF output to the target.
\subsection{Experiment 1: Exploring the peak performance for each BF method}
First, to explore the peak performance of each BF method, we obtained the optimal mask for each utterance, which is the solution to \refeqn{eqn:optimal-mask-def} under the constraints represented as \refeqn{eqn:constraint-nonnegative} and \refeqn{eqn:constraint-unitvariance}, using the backpropagation of 500 iterations.
\reftable{tab:potential-peak} presents SDR scores of the mask-based BFs, including those of the ideal MWF BF using $k=5$ in \refeqn{eqn:ideal-mwf}.
All the mask-based BFs practically demonstrated identical scores comparable to the ideal MWF BF. These are remarkable results, given that they were not theoretically evident, as mentioned in Sections~\ref{subsec:mwf} and \ref{subsec:max-snr}.
\begin{table}[b]\begin{center}
\caption{Peak performance for each method and scenario in SDR {\rm [dB]} compared with that of the ideal MWF.}
\label{tab:potential-peak}
\begin{tabular}{c|c|rrr} \hline
BF & Mask(s) & \multicolumn{3}{c}{BG multiplier} \\
Method & used & \multicolumn{1}{c}{$\times$1} & \multicolumn{1}{c}{$\times$2} & \multicolumn{1}{c}{$\times$4} \\\hline\hline
Max-SNR & $m_{s},m_{n}$ & 19.430 & {\bf 14.276} & {\bf 9.451} \\
Max-SOR & $m_{s}$ & 19.426 & {\bf 14.276} & {\bf 9.451} \\
Min-NOR & $m_{n}$ & 19.434 & {\bf 14.276} & {\bf 9.451} \\
MWF & $m_{s}$ & 19.438 & 14.268 & 9.430 \\\hline
Ideal MWF & & {\bf 19.441} & {\bf 14.276} & {\bf 9.451} \\\hline
\end{tabular}
\end{center}\end{table}
\subsection{Experiment 2: Verifying whether the optimal mask is common for all the BF methods}
Furthermore, to verify whether the optimal mask is common for all the BF methods, we applied the optimal mask obtained in a BF method to another one.
If the same score is achieved after applying the mask, we can claim that the optimal mask is the same between these two BF methods.
Although 12 permutations are possible from four methods, we examined six considered to be nontrivial.
In the cases that $m_{s}(t)$ was applied to a BF method using $m_{n}(t)$, and vice versa, we converted the mask values using the following rules to satisfy \refeqn{eqn:sum-is-constant}:
\begin{eqnarray}
m_{s}(t) &=& \max_{t}\left\{m_{n}(t)\right\} - m_{n}(t),
\label{eqn:ms-from-mn}\\
m_{n}(t) &=& \max_{t}\left\{m_{s}(t)\right\} - m_{s}(t),
\label{eqn:mn-from-ms}
\end{eqnarray}
where $\max_{t}\left\{\cdot\right\}$ denotes the maximum value over $t$.
The obtained results are presented in \reftable{tab:performance-across-bf-methods}.
The first and second rows indicate that the optimal masks obtained in the max-SNR BF were applied to the max-SOR and min-NOR BFs, respectively. These two rows present lower scores than Experiment 1. This suggests that the optimal mask obtained in the max-SNR BF is not necessarily optimal for the max-SOR or min-NOR BFs.
The third row indicates that the optimal mask $m_{s}(t)$ obtained in the max-SOR BF was applied to the min-NOR BF using \refeqn{eqn:mn-from-ms}, and the fourth row indicates that the mask $m_{n}(t)$ obtained in the min-NOR BF was applied to the max-SOR BF using \refeqn{eqn:ms-from-mn}.
The two rows practically demonstrated the same scores as the max-SOR and min-NOR BFs in \reftable{tab:potential-peak}.
This suggests that the optimal mask can be converted between the max-SOR and min-NOR BFs, despite the fact that the two masks are not the same.
The fifth and sixth rows indicate that the optimal mask $m_{s}(t)$ obtained in a method was applied to the other for the max-SOR and MWF BFs. These two rows demonstrated considerably lower scores than the results of the max-SOR and MWF BFs in \reftable{tab:potential-peak}. These results suggest that the optimal mask is not common between these two BF methods.
\begin{table}[b]\begin{center}
\caption{Extraction performance in SDR {[\rm dB]} after applying the optimal mask to another BF method}
\label{tab:performance-across-bf-methods}
\begin{tabular}{cc|c|rrr} \hline
BF for mask & Applied & Mask & \multicolumn{3}{c}{BG multiplier} \\
estimation & to & used & \multicolumn{1}{c}{$\times$1} & \multicolumn{1}{c}{$\times$2} & \multicolumn{1}{c}{$\times$4} \\\hline\hline
Max-SNR & Max-SOR & $m_{s}$ & 17.084 & 13.493 & 9.165 \\
& Min-NOR & $m_{n}$ & 18.399 & 14.032 & 9.381 \\\hline
Max-SOR & Min-NOR & $m_{s}$ \& \refeqn{eqn:mn-from-ms} & 19.426 & {\bf 14.276} & {\bf 9.451} \\
Min-NOR & Max-SOR & $m_{n}$ \& \refeqn{eqn:ms-from-mn} & {\bf 19.434} & 14.274 & {\bf 9.451} \\\hline
Max-SOR & MWF & $m_{s}$ & 15.465 & 10.259 & 5.337 \\
MWF & Max-SOR & $m_{s}$ & 14.230 & 12.092 & 8.458 \\\hline
\end{tabular}
\end{center}\end{table}
\subsection{Experiment 3: Examining whether the IRM can achieve the peak performance}
The third experiment is for examining if the IRM can achieve the peak performance for each BF method.
Considering that the term IRM was ambiguously employed in the related studies \cite{Erdogan2016-jc,Tu2017-ey,Wang2018-yy,Tu2019-qp,Chakrabarty2019-th}, we define several masks that utilize the ratios of the target and interferences based on \cite{Wang2018-yy}.
The IRMs for the target and interferences are defined as
\begin{eqnarray}
m_{s}(t) &=& \left(\frac{\abs{s_{k}(t)}^{2}}{\abs{s_{k}(t)}^{2}+\abs{n_{k}(t)}^{2}}\right)^{\beta}, \label{eqn:def-irm-ms} \\
m_{n}(t) &=& \left(\frac{\abs{n_{k}(t)}^{2}}{\abs{s_{k}(t)}^{2}+\abs{n_{k}(t)}^{2}}\right)^{\beta}, \label{eqn:def-irm-mn}
\end{eqnarray}
where $k=5$ and $\beta=1$ or $0.5$. As another type of ratio mask, we employed the spectral magnitude masks (SMMs) defined as
\begin{eqnarray}
m_{s}(t) &=& \frac{\abs{s_{k}(t)}}{\abs{s_{k}(t) + n_{k}(t)}}, \label{eqn:def-smm-ms} \\
m_{n}(t) &=& \frac{\abs{n_{k}(t)}}{\abs{s_{k}(t) + n_{k}(t)}}. \label{eqn:def-smm-mn}
\end{eqnarray}
In the single-channel TF masking, these masks are ideal in terms of the magnitudes of the target and interferences \cite{Wang2018-yy}.
The obtained results are presented in \reftable{tab:performance-irm-and-smm}. In this table, the combination of the max-SOR and IRM with $\beta=0.5$ demonstrated the best score for all the scenarios. However, these scores were lower than that in Experiment 1. The results suggest that the IRM does not achieve the peak performance for any BF method examined in this study; hence, it differs from the optimal mask for each method.
In addition, \reftable{tab:performance-irm-and-smm} indicates that the max-SNR, max-SOR, and min-NOR BFs present the same scores for the IRM with $\beta=1$. This is because this mask type evidently satisfies both \refeqn{eqn:cov_summation} and \refeqn{eqn:sum-is-constant}. Hence, these three BFs are equivalent in this case, as mentioned in \refsec{subsec:max-snr}.
%\nop{
\begin{table}[b]\begin{center}
\caption{Extraction performance in SDR {[\rm dB]} when the IRM ($\beta=1$, $0.5$) and SMM were employed for each BF method.}
\label{tab:performance-irm-and-smm}
\begin{tabular}{c|l|c|rrr} \hline
BF & Type of & Mask(s) & \multicolumn{3}{c}{BG multiplier} \\
method & ideal mask & used & \multicolumn{1}{c}{$\times$1} & \multicolumn{1}{c}{$\times$2} & \multicolumn{1}{c}{$\times$4} \\\hline\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Max-SNR & IRM ($\beta=0.5$) & $m_{s},m_{n}$ & 18.313 & 13.790 & 9.203 \\
& SMM & & 17.973 & 13.529 & 9.060 \\\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Max-SOR & IRM ($\beta=0.5$) & $m_{s}$ & {\bf 18.725} & {\bf 13.948} & {\bf 9.275} \\
& SMM & & 13.640 & 11.447 & 8.147 \\\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Min-NOR & IRM ($\beta=0.5$) & $m_{n}$ & 18.267 & 13.747 & 9.160 \\
& SMM & & 17.372 & 12.735 & 8.103 \\\hline
& IRM ($\beta=1$) & & 17.316 & 12.788 & 8.375 \\
MWF & IRM ($\beta=0.5$) & $m_{s}$ & 16.228 & 11.799 & 7.477 \\
& SMM & & 17.109 & 12.316 & 7.734 \\\hline
\end{tabular}
\end{center}\end{table}
%}
\nop{
\begin{table}[b]\begin{center}
\caption{Extraction performance in SDR {[\rm dB]} when the IRM ($\beta=1$, $0.5$) and SMM were used for each BF method.}
\label{tab:performance-irm-and-smm}
\begin{tabular}{c|l|c|rrr} \hline
BF & Type of & Mask(s) & \multicolumn{3}{c}{BG multiplier} \\
method & ideal mask & used & \multicolumn{1}{c}{$\times$1} & \multicolumn{1}{c}{$\times$2} & \multicolumn{1}{c}{$\times$4} \\\hline\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Max SNR & IRM ($\beta=0.5$) & $m_{s},m_{n}$ & 18.313 & 13.790 & 9.203 \\
& SMM & & 17.973 & 13.529 & 9.060 \\\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Max SOR & IRM ($\beta=0.5$) & $m_{s}$ & {\bf 18.725} & {\bf 13.948} & 9.275 \\
& SMM & & 13.640 & 11.447 & 8.147 \\\hline
& IRM ($\beta=1$) & & 18.642 & 13.889 & 9.226 \\
Min NOR & IRM ($\beta=0.5$) & $m_{n}$ & 18.267 & 13.747 & 9.160 \\
& SMM & & 17.372 & 12.735 & 8.103 \\\hline
& IRM ($\beta=1$) & & 17.316 & 12.788 & 8.375 \\
MWF & IRM ($\beta=0.5$) & $m_{s}$ & 16.228 & 11.799 & 7.477 \\
& SMM & & 17.109 & 12.316 & 7.734 \\\hline\hline
& IRM ($\beta=1$) & & 17.293 & 13.570 & {\bf 10.239} \\
Masking & IRM ($\beta=0.5$) & $m_{s}$ & 16.179 & 12.201 & 8.549 \\
& SMM & & 17.767 & 13.251 & 9.024 \\\hline
\end{tabular}
\end{center}\end{table}
}
%
\section{Discussion} \label{sec:discussion}
In this section, we discuss the experimental results for each issue mentioned in \refsec{sec:framework}.
\subsection{Discussion on Issue 1}
\label{subsec:discuss-issue1}
The results of Experiment 1 provide the answer to Issue 1. The peak performance of each BF method is practically identical and comparable to the upper bound given by the ideal MWF BF. These findings differ from the conventional idea presented in \reffig{fig:conventional_image} and may deprive the meaning of the discussion on the BF that can achieve the best extraction performance. However, it is an open question whether these findings are applicable to any BF method and dataset. Therefore, further investigation is required.
\subsection{Discussion on Issue 2}
\label{subsec:discuss-issue2}
The results of Experiment 2 provide the answer to Issue 2. Although the optimal mask is not common, it depends on the BF method adopted.
This finding differs from the conventional concept presented in \reffig{fig:conventional_image}, similar to the discussion on Issue 1.
However, we consider this to be natural because the optimal mask is formulated as the solution to a different problem for each BF method, as explained in \refsec{sec:optimal-mask}.
In addition, \reftable{tab:performance-across-bf-methods} indicates several remarkable points. Although both the max-SOR and min-NOR BFs are derived from the max-SNR BF, as mentioned in \refsec{subsec:max-snr}, the top two rows suggest that $m_{s}(t)$ and $m_{n}(t)$ are optimal for the max-SNR BF, while not for the max-SOR or min-NOR BFs. We consider that the reason for this is that the optimal masks obtained in the max-SNR BF do not satisfy \refeqn{eqn:cov_summation} or \refeqn{eqn:sum-is-constant}. Hence, the max-SNR BF is not equivalent to the other two BFs in this case. In contrast, the third and fourth rows in this table suggest that the max-SOR and min-NOR BFs are equivalent if both $m_{s}(t)$ and $m_{n}(t)$ satisfy \refeqn{eqn:sum-is-constant}. Moreover, the bottom two rows exhibited lower scores than the other rows. Although $m_{s}(t)$ has commonly been interpreted as the mask for the target, the facts suggest that the optimal mask $m_{s}(t)$ for the max-SOR BF differs significantly from that of the MWF BF and vice versa.
From the discussion on Issues 1 and 2, we have obtained a novel idea for the peak performance and optimal mask among multiple BF methods, as illustrated in \reffig{fig:discovered_image}. This indicates that the optimal mask depends on the BF method adopted, while the peak performance is the same among multiple BF methods and comparable to the upper bound achieved by the ideal MWF.
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{discovered_image.pdf}
\caption{Obtained concept of the peak performance and optimal mask among multiple BF methods}
\label{fig:discovered_image}
\end{figure}
\subsection{Discussion on Issue 3}
\label{subsec:discuss-issue3}
The results of Experiment 3 provide the answer to Issue 3.
Although these masks have been considered to be optimal for any BF method, \reftable{tab:performance-irm-and-smm} suggests that the IRM and SMM do not achieve the upper-bound performance and differ from the optimal mask for any BF method.
These findings can explain why the studies presented in \reftable{tab:comparison} mentioned a different BF method as the best.
\reftable{tab:performance-irm-and-smm} indicates that the BF method that appears the best depends on the mask type employed. For example, focusing on the IRM with $\beta=0.5$ in this table, we can determine that the max-SOR BF performs the best. However, this result only suggests that this mask type is the closest to the optimal mask for the max-SOR in this dataset.
Furthermore, the fact that the IRM is not optimal imposes another issue on us. This is how the optimal mask is interpreted and represented as a formula. However, this is also an open question.
\section{Conclusions} \label{sec:conclusions}
In this study, we investigated mask-based BFs such as the max-SNR, max-SOR, min-NOR, and MWF BFs.
To explore the peak performance for each BF method, we obtained the optimal mask for each utterance by minimizing the MSE between the BF output and target.
We experimentally verified that these four methods have the same peak performance as the upper bound provided by the ideal MWF BF.
Via additional experiments that applied the optimal mask across BF methods, we determined that the optimal mask differed for the BF method used. However, the mask values can be converted between the max-SOR and min-NOR BFs.
These findings differed from the conventional idea that the optimal mask would be common and the peak performance would depend on the BF method.
We verified that the IRM did not achieve the peak performance for these four BFs. Hence, this mask type was not optimal.
We expect that these findings will contribute to the improvement of mask-based BFs.
Given that these findings are currently experimental, in the future, we would attempt to establish their theoretical background and investigate whether these apply to other BF methods and datasets.
The experimental system has been shared in \url{https://github.com/hreshare/optimal_beamformers/}.
\printbibliography
\end{document}
% $ biblatex auxiliary file $
% $ biblatex bbl format version 3.2 $
% Do not modify the above lines!
%
% This is an auxiliary file used by the 'biblatex' package.
% This file may safely be deleted. It will be recreated by
% biber as required.
%
\begingroup
\makeatletter
\@ifundefined{ver@biblatex.sty}
{\@latex@error
{Missing 'biblatex' package}
{The bibliography requires the 'biblatex' package.}
\aftergroup\endinput}
{}
\endgroup
\refsection{0}
\datalist[entry]{none/global//global/global}
\entry{Chen2018-rl}{article}{}
\name{author}{4}{}{%
{{hash=cddf966305bb6b03378c1dd066ce8016}{%
family={Chen},
familyi={C\bibinitperiod},
given={Szu\bibnamedelima Jui},
giveni={S\bibinitperiod\bibinitdelim J\bibinitperiod}}}%
{{hash=3416da1acca3e6f7c52b0fbb716f4b77}{%
family={Subramanian},
familyi={S\bibinitperiod},
given={Aswin\bibnamedelima Shanmugam},
giveni={A\bibinitperiod\bibinitdelim S\bibinitperiod}}}%
{{hash=cc7ada14a85aa4644c6f419954e5745c}{%
family={Xu},
familyi={X\bibinitperiod},
given={Hainan},
giveni={H\bibinitperiod}}}%
{{hash=ae898cb785c2eecb4cda42d862e0d5d4}{%
family={Watanabe},
familyi={W\bibinitperiod},
given={Shinji},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{aa820d39bf9710f260c0bf0f3856a972}
\strng{fullhash}{d1f353edf64198a1f8c24f2d40f90896}
\strng{bibnamehash}{d1f353edf64198a1f8c24f2d40f90896}
\strng{authorbibnamehash}{d1f353edf64198a1f8c24f2d40f90896}
\strng{authornamehash}{aa820d39bf9710f260c0bf0f3856a972}
\strng{authorfullhash}{d1f353edf64198a1f8c24f2d40f90896}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This paper describes a new baseline system for automatic speech recognition (ASR) in the CHiME-4 challenge to promote the development of noisy ASR in speech processing communities by providing 1) state-of-the-art system with a simplified single system comparable to the complicated top systems in the challenge, 2) publicly available and reproducible recipe through the main repository in the Kaldi speech recognition toolkit. The proposed system adopts generalized eigenvalue beamforming with bidirectional long short-term memory (LSTM) mask estimation. We also propose to use a time delay neural network (TDNN) based on the lattice-free version of the maximum mutual information (LF-MMI) trained with augmented all six microphones plus the enhanced data after beamforming. Finally, we use a LSTM language model for lattice and n-best re-scoring. The final system achieved 2.74\% WER for the real test set in the 6-channel track, which corresponds to the 2nd place in the challenge. In addition, the proposed baseline recipe includes four different speech enhancement measures, short-time objective intelligibility measure (STOI), extended STOI (eSTOI), perceptual evaluation of speech quality (PESQ) and speech distortion ratio (SDR) for the simulation test set. Thus, the recipe also provides an experimental platform for speech enhancement studies with these performance measures.}
\field{journaltitle}{Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH}
\field{title}{Building state-of-the-art distant speech recognition using the {CHiME-4} challenge with a setup of speech enhancement baseline}
\field{volume}{2018-Septe}
\field{year}{2018}
\field{pages}{1571\bibrangedash 1575}
\range{pages}{5}
\keyw{LSTM language modeling; Lattice-free MMI; Mask-based beamforming; Noise robustness; Speech recognition}
\endentry
\entry{Mizumachi2016-jd}{article}{}
\name{author}{2}{}{%
{{hash=e1e1161d790ceea2b5a03bd14767d533}{%
family={Mizumachi},
familyi={M\bibinitperiod},
given={Mitsunori},
giveni={M\bibinitperiod}}}%
{{hash=6f727d6a709cb7716c92e17243ee928c}{%
family={Origuchi},
familyi={O\bibinitperiod},
given={Maya},
giveni={M\bibinitperiod}}}%
}
\strng{namehash}{47f6ee202b32d63809c1c15e312319ce}
\strng{fullhash}{47f6ee202b32d63809c1c15e312319ce}
\strng{bibnamehash}{47f6ee202b32d63809c1c15e312319ce}
\strng{authorbibnamehash}{47f6ee202b32d63809c1c15e312319ce}
\strng{authornamehash}{47f6ee202b32d63809c1c15e312319ce}
\strng{authorfullhash}{47f6ee202b32d63809c1c15e312319ce}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Signal enhancement can be achieved by spatial filtering with multiple microphones, that is, a microphone array. A traditional delay-and-sum beamformer enhances the target signal without spectral distortion, but could not control its beam-pattern, except for the arrival direction of the target signal. The beam-patten is flexibly designed by introducing channel-dependent weights or filters in delay-and-sum beamforming. The channel weights can be optimized using a neural network in order to achieve superdirectivity. In this study, an advanced delay-and-sum beamformer is proposed to make the main lobe much sharper and decrease grating lobes. The advanced beamformer substitutes a deep neural network for the conventional three-layered neural network. The proposed method also adopts a non-equally-spaced microphone arrangement and sub-band beamforming to prevent spatial aliasing. It is confirmed that the proposed beamformer has significant advantages both in sharpening the main-lobe and getting rid of grating lobes over conventional beamformers.}
\field{journaltitle}{22nd International Congress on Acoustics (ICA)}
\field{title}{Advanced delay-and-sum beamformer with deep neural network}
\field{year}{2016}
\keyw{Delay-and-sum beamformer; channel-dependent weight; deep neural network; non-equally-spaced microphone arrangement}
\endentry
\entry{Mizumachi2019-hi}{article}{}
\name{author}{1}{}{%
{{hash=e1e1161d790ceea2b5a03bd14767d533}{%
family={Mizumachi},
familyi={M\bibinitperiod},
given={Mitsunori},
giveni={M\bibinitperiod}}}%
}
\list{publisher}{1}{%
{ICA 2019}%
}
\strng{namehash}{e1e1161d790ceea2b5a03bd14767d533}
\strng{fullhash}{e1e1161d790ceea2b5a03bd14767d533}
\strng{bibnamehash}{e1e1161d790ceea2b5a03bd14767d533}
\strng{authorbibnamehash}{e1e1161d790ceea2b5a03bd14767d533}
\strng{authornamehash}{e1e1161d790ceea2b5a03bd14767d533}
\strng{authorfullhash}{e1e1161d790ceea2b5a03bd14767d533}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{Proceedings of International Congress on Acoustics (ICA 2019)}
\field{month}{9}
\field{title}{Neural Network-based Broadband Beamformer with Less Distortion}
\field{year}{2019}
\field{pages}{2760}
\range{pages}{1}
\endentry
\entry{Heymann2016-eb}{article}{}
\name{author}{4}{}{%
{{hash=e516c4bc1fa1598297c1e5129475435c}{%
family={Heymann},
familyi={H\bibinitperiod},
given={Jahn},
giveni={J\bibinitperiod}}}%
{{hash=17c701486d60a85d0f1d41319b7f2744}{%
family={Drude},
familyi={D\bibinitperiod},
given={Lukas},
giveni={L\bibinitperiod}}}%
{{hash=ef16bb91274b001b205d2d0014ced54c}{%
family={Chinaev},
familyi={C\bibinitperiod},
given={Aleksej},
giveni={A\bibinitperiod}}}%
{{hash=27394268ec19154143123cbcd93f5788}{%
family={Haeb-Umbach},
familyi={H\bibinithyphendelim U\bibinitperiod},
given={Reinhold},
giveni={R\bibinitperiod}}}%
}
\strng{namehash}{9dabd49c97aa276754d510205911c1cc}
\strng{fullhash}{fb4452bcbb6a3376513cdf6cf0a08283}
\strng{bibnamehash}{fb4452bcbb6a3376513cdf6cf0a08283}
\strng{authorbibnamehash}{fb4452bcbb6a3376513cdf6cf0a08283}
\strng{authornamehash}{9dabd49c97aa276754d510205911c1cc}
\strng{authorfullhash}{fb4452bcbb6a3376513cdf6cf0a08283}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We present a new beamformer front-end for Automatic Speech Recognition and apply it to the 3rd-CHiME Speech Separation and Recognition Challenge. Without any further modification of the back-end, we achieve a 53\% relative reduction of the word error rate over the best baseline enhancement system for the relevant test data set. Our approach leverages the power of a bi-directional Long Short-Term Memory network to robustly estimate soft masks for a subsequent beamforming step. The utilized Generalized Eigenvalue beamforming operation with an optional Blind Analytic Normalization does not rely on a Direction-of-Arrival estimate and can cope with multi-path sound propagation, while at the same time only introducing very limited speech distortions. Our quite simple setup exploits the possibilities provided by simulated training data while still being able to generalize well to the fairly different real data. Finally, combining our front-end with data augmentation and another language model nearly yields a 64 \% reduction of the word error rate on the real data test set.}
\field{journaltitle}{2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015 - Proceedings}
\field{number}{June 2016}
\field{title}{{BLSTM} supported {GEV} beamformer front-end for the {3RD} {CHiME} challenge}
\field{year}{2016}
\field{pages}{444\bibrangedash 451}
\range{pages}{8}
\keyw{Beamforming; Feature Enhancement; Neural Networks; Robust Speech Recognition}
\endentry
\entry{Heymann2016-sy}{inproceedings}{}
\name{author}{3}{}{%
{{hash=e516c4bc1fa1598297c1e5129475435c}{%
family={Heymann},
familyi={H\bibinitperiod},
given={Jahn},
giveni={J\bibinitperiod}}}%
{{hash=17c701486d60a85d0f1d41319b7f2744}{%
family={Drude},
familyi={D\bibinitperiod},
given={Lukas},
giveni={L\bibinitperiod}}}%
{{hash=27394268ec19154143123cbcd93f5788}{%
family={Haeb-Umbach},
familyi={H\bibinithyphendelim U\bibinitperiod},
given={Reinhold},
giveni={R\bibinitperiod}}}%
}
\strng{namehash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\strng{fullhash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\strng{bibnamehash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\strng{authorbibnamehash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\strng{authornamehash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\strng{authorfullhash}{b60ecd81dc2121ae45dbcb69f4fc27a6}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{2016 {IEEE} International Conference on Acoustics, Speech and Signal Processing ({ICASSP})}
\field{title}{Neural network based spectral mask estimation for acoustic beamforming}
\field{year}{2016}
\field{pages}{196\bibrangedash 200}
\range{pages}{5}
\endentry
\entry{Erdogan2016-jc}{article}{}
\name{author}{5}{}{%
{{hash=ead22159a811327ebaf2ce764be2c2d2}{%
family={Erdogan},
familyi={E\bibinitperiod},
given={Hakan},
giveni={H\bibinitperiod}}}%
{{hash=450bfea3ade066caf1895eb1faead5ef}{%
family={Hershey},
familyi={H\bibinitperiod},
given={John},
giveni={J\bibinitperiod}}}%
{{hash=ae898cb785c2eecb4cda42d862e0d5d4}{%
family={Watanabe},
familyi={W\bibinitperiod},
given={Shinji},
giveni={S\bibinitperiod}}}%
{{hash=ffda50f9ce7d55739ef488df4c0a7d61}{%
family={Mandel},
familyi={M\bibinitperiod},
given={Michael},
giveni={M\bibinitperiod}}}%
{{hash=b6e189cc6d3a52cc6fb00d42470971a1}{%
family={Le\bibnamedelima Roux},
familyi={L\bibinitperiod\bibinitdelim R\bibinitperiod},
given={Jonathan},
giveni={J\bibinitperiod}}}%
}
\strng{namehash}{d2fe2261bbdd3645e67aa342663d8edb}
\strng{fullhash}{930984f945abfc52d1223d1fe959f38b}
\strng{bibnamehash}{930984f945abfc52d1223d1fe959f38b}
\strng{authorbibnamehash}{930984f945abfc52d1223d1fe959f38b}
\strng{authornamehash}{d2fe2261bbdd3645e67aa342663d8edb}
\strng{authorfullhash}{930984f945abfc52d1223d1fe959f38b}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Recent studies on multi-microphone speech databases indicate that it is beneficial to perform beamforming to improve speech recognition accuracies, especially when there is a high level of background noise. Minimum variance distortionless response (MVDR) beamforming is an important beamforming method that performs quite well for speech recognition purposes especially if the steering vector is known. However, steering the beamformer to focus on speech in unknown acoustic conditions remains a challenging problem. In this study, we use singlechannel speech enhancement deep networks to form masks that can be used for noise spatial covariance estimation, which steers the MVDR beamforming toward the speech. We analyze how mask prediction affects performance and also discuss various ways to use masks to obtain the speech and noise spatial covariance estimates in a reliable way. We show that using a single mask across microphones for covariance prediction with minima-limited post-masking yields the best result in terms of signal-level quality measures and speech recognition word error rates in a mismatched training condition.}
\field{journaltitle}{Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH}
\field{title}{Improved {MVDR} beamforming using single-channel mask prediction networks}
\field{volume}{08-12-Sept}
\field{year}{2016}
\field{pages}{1981\bibrangedash 1985}
\range{pages}{5}
\keyw{LSTM; MVDR beamforming; Microphone arrays; Neural networks; Speech enhancement}
\endentry
\entry{Drude2019-so}{inproceedings}{}
\name{author}{3}{}{%
{{hash=17c701486d60a85d0f1d41319b7f2744}{%
family={Drude},
familyi={D\bibinitperiod},
given={Lukas},
giveni={L\bibinitperiod}}}%
{{hash=e516c4bc1fa1598297c1e5129475435c}{%
family={Heymann},
familyi={H\bibinitperiod},
given={Jahn},
giveni={J\bibinitperiod}}}%
{{hash=27394268ec19154143123cbcd93f5788}{%
family={Haeb-Umbach},
familyi={H\bibinithyphendelim U\bibinitperiod},
given={Reinhold},
giveni={R\bibinitperiod}}}%
}
\list{publisher}{1}{%
{International Speech Communication Association}%
}
\strng{namehash}{7a84ea3827612abc5323d0fc6e40f344}
\strng{fullhash}{7a84ea3827612abc5323d0fc6e40f344}
\strng{bibnamehash}{7a84ea3827612abc5323d0fc6e40f344}
\strng{authorbibnamehash}{7a84ea3827612abc5323d0fc6e40f344}
\strng{authornamehash}{7a84ea3827612abc5323d0fc6e40f344}
\strng{authorfullhash}{7a84ea3827612abc5323d0fc6e40f344}
\field{sortinit}{4}
\field{sortinithash}{9381316451d1b9788675a07e972a12a7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We present an unsupervised training approach for a neural network-based mask estimator in an acoustic beamforming application. The network is trained to maximize a likelihood criterion derived from a spatial mixture model of the observations. It is trained from scratch without requiring any parallel data consisting of degraded input and clean training targets. Thus, training can be carried out on real recordings of noisy speech rather than simulated ones. In contrast to previous work on unsupervised training of neural mask estimators, our approach avoids the need for a possibly pre-trained teacher model entirely. We demonstrate the effectiveness of our approach by speech recognition experiments on two different datasets: one mainly deteriorated by noise (CHiME 4) and one by reverberation (REVERB). The results show that the performance of the proposed system is on par with a supervised system using oracle target masks for training and with a system trained using a model-based teacher.}
\field{booktitle}{Proceedings of the Annual Conference of the International Speech Communication Association, {INTERSPEECH}}
\field{title}{Unsupervised training of neural mask-based beamforming}
\field{volume}{2019-September}
\field{year}{2019}
\field{pages}{1253\bibrangedash 1257}
\range{pages}{5}
\keyw{Beamforming; Deep learning; Multi-channel; Robust speech recognition; Unsupervised learning}
\endentry
\entry{Souden2010-kp}{article}{}
\name{author}{3}{}{%
{{hash=472c8998023fbc2d1bbcb406b0c4886b}{%
family={Souden},
familyi={S\bibinitperiod},
given={Mehrez},
giveni={M\bibinitperiod}}}%
{{hash=50f152a58e4a694e44bf07fc68be4636}{%
family={Benesty},
familyi={B\bibinitperiod},
given={Jacob},
giveni={J\bibinitperiod}}}%
{{hash=5ff17be7aabd14fa97ad58183db10c1c}{%
family={Affes},
familyi={A\bibinitperiod},
given={Sofiène},
giveni={S\bibinitperiod}}}%
}
\list{publisher}{2}{%
{Institute of Electrical}%
{Electronics Engineers (IEEE)}%
}
\strng{namehash}{641673b02b5857a5e7903a88e9662289}
\strng{fullhash}{641673b02b5857a5e7903a88e9662289}
\strng{bibnamehash}{641673b02b5857a5e7903a88e9662289}
\strng{authorbibnamehash}{641673b02b5857a5e7903a88e9662289}
\strng{authornamehash}{641673b02b5857a5e7903a88e9662289}
\strng{authorfullhash}{641673b02b5857a5e7903a88e9662289}
\field{sortinit}{5}
\field{sortinithash}{20e9b4b0b173788c5dace24730f47d8c}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. Signal Process.}
\field{month}{9}
\field{number}{9}
\field{title}{A study of the {LCMV} and {MVDR} noise reduction filters}
\field{volume}{58}
\field{year}{2010}
\field{pages}{4925\bibrangedash 4935}
\range{pages}{11}
\endentry
\entry{Stenzel2013-hw}{inproceedings}{}
\name{author}{4}{}{%
{{hash=98ece3583583e33b353d54fa6edb7b38}{%
family={Stenzel},
familyi={S\bibinitperiod},
given={Sebastian},
giveni={S\bibinitperiod}}}%
{{hash=41643f5d62a55ff03171c45ed1a04d33}{%
family={Lawin-Ore},
familyi={L\bibinithyphendelim O\bibinitperiod},
given={Toby\bibnamedelima Christian},
giveni={T\bibinitperiod\bibinitdelim C\bibinitperiod}}}%
{{hash=8ccac6fa8e41ef5d2345bd74ac7a9024}{%
family={Freudenberger},
familyi={F\bibinitperiod},
given={Juergen},
giveni={J\bibinitperiod}}}%
{{hash=dd004cfd0c0feeb261ad0f703e2bbc8c}{%
family={Doclo},
familyi={D\bibinitperiod},
given={Simon},
giveni={S\bibinitperiod}}}%
}
\list{publisher}{1}{%
{unknown}%
}
\strng{namehash}{e38d5076c08782ca3dc1450199d94e05}
\strng{fullhash}{17753a35a6e00af025c9ce1b8750f850}
\strng{bibnamehash}{17753a35a6e00af025c9ce1b8750f850}
\strng{authorbibnamehash}{17753a35a6e00af025c9ce1b8750f850}
\strng{authornamehash}{e38d5076c08782ca3dc1450199d94e05}
\strng{authorfullhash}{17753a35a6e00af025c9ce1b8750f850}
\field{sortinit}{6}
\field{sortinithash}{b33bc299efb3c36abec520a4c896a66d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{In speech enhancement applications, the multichannel Wiener filter (MWF) is widely used to reduce noise and thus improve signal quality. The MWF performs noise reduction by estimating the desired signal component in one of the microphones, referred to as the reference microphone. However, for distributed microphones, the selection of the reference microphone has a significant impact on the broadband output SNR of the MWF, largely depending on the acoustical transfer function (ATF) between the desired source and the reference microphone. In this paper, a multichannel Wiener filtering approach using a soft combined reference is presented. Simulation results show that the proposed scheme leads to a higher broadband output SNR compared to an arbitrarily selected reference microphone, moreover achieving a partial equalization of the overall acoustic system.}
\field{booktitle}{Applications of Signal Processing to Audio and Acoustics ({WASPAA)}, 2013 {IEEE} Workshop on}
\field{month}{10}
\field{title}{A multichannel Wiener filter with partial equalization for distributed microphones}
\field{year}{2013}
\field{pages}{1\bibrangedash 4}
\range{pages}{4}
\endentry
\entry{Nugraha2016-rn}{article}{}
\name{author}{3}{}{%
{{hash=d09eba4564559c6d4234bd023b738a4a}{%
family={Nugraha},
familyi={N\bibinitperiod},
given={Aditya\bibnamedelima Arie},
giveni={A\bibinitperiod\bibinitdelim A\bibinitperiod}}}%
{{hash=a94cac21598af7fd16ff450f667f6ddd}{%
family={Liutkus},
familyi={L\bibinitperiod},
given={Antoine},
giveni={A\bibinitperiod}}}%
{{hash=8c20d285377b0667d451da42b73d2f11}{%
family={Vincent},
familyi={V\bibinitperiod},
given={Emmanuel},
giveni={E\bibinitperiod}}}%
}
\strng{namehash}{c99d0891d7e72c66339ac18504e90a46}
\strng{fullhash}{c99d0891d7e72c66339ac18504e90a46}
\strng{bibnamehash}{c99d0891d7e72c66339ac18504e90a46}
\strng{authorbibnamehash}{c99d0891d7e72c66339ac18504e90a46}
\strng{authornamehash}{c99d0891d7e72c66339ac18504e90a46}
\strng{authorfullhash}{c99d0891d7e72c66339ac18504e90a46}
\field{sortinit}{6}
\field{sortinithash}{b33bc299efb3c36abec520a4c896a66d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This article addresses the problem of multichannel audio source separation. We propose a framework where deep neural networks (DNNs) are used to model the source spectra and combined with the classical multichannel Gaussian model to exploit the spatial information. The parameters are estimated in an iterative expectation-maximization (EM) fashion and used to derive a multichannel Wiener filter. We present an extensive experimental study to show the impact of different design choices on the performance of the proposed technique. We consider different cost functions for the training of DNNs, namely the probabilistically motivated Itakura-Saito divergence, and also Kullback-Leibler, Cauchy, mean squared error, and phase-sensitive cost functions. We also study the number of EM iterations and the use of multiple DNNs, where each DNN aims to improve the spectra estimated by the preceding EM iteration. Finally, we present its application to a speech enhancement problem. The experimental results show the benefit of the proposed multichannel approach over a single-channel DNN-based approach and the conventional multichannel nonnegative matrix factorization-based iterative EM algorithm.}
\field{journaltitle}{IEEE/ACM Transactions on Audio Speech and Language Processing}
\field{number}{9}
\field{title}{Multichannel audio source separation with deep neural networks}
\field{volume}{24}
\field{year}{2016}
\field{pages}{1652\bibrangedash 1664}
\range{pages}{13}
\keyw{Audio source separation; deep neural network (DNN); expectation-maximization (EM); multichannel; speech enhancement}
\endentry
\entry{Pfeifenberger2017-zp}{inproceedings}{}
\name{author}{3}{}{%
{{hash=bbd7c7b30b3de486012e0bcd50960c31}{%
family={Pfeifenberger},
familyi={P\bibinitperiod},
given={Lukas},
giveni={L\bibinitperiod}}}%
{{hash=d00b2f6ad6676b2495700f5f425b954d}{%
family={Zöhrer},
familyi={Z\bibinitperiod},
given={Matthias},
giveni={M\bibinitperiod}}}%
{{hash=384ef564b8aad191a8876c620bc305e3}{%
family={Pernkopf},
familyi={P\bibinitperiod},
given={Franz},
giveni={F\bibinitperiod}}}%
}
\strng{namehash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\strng{fullhash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\strng{bibnamehash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\strng{authorbibnamehash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\strng{authornamehash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\strng{authorfullhash}{0d51a11d0fabb7b87d6a60cd677c7f8b}
\field{sortinit}{6}
\field{sortinithash}{b33bc299efb3c36abec520a4c896a66d}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{In this paper, we present an optimal multi-channel Wiener filter, which consists of an eigenvector beamformer and a single-channel postfilter. We show that both components solely depend on a speech presence probability, which we learn using a deep neural network, consisting of a deep autoencoder and a softmax regression layer. To prevent the DNN from learning specific speaker and noise types, we do not use the signal energy as input feature, but rather the cosine distance between the dominant eigenvectors of consecutive frames of the power spectral density of the noisy speech signal. We compare our system against the BeamformIt toolkit, and state-of-the-art approaches such as the front-end of the best system of the CHiME3 challenge. We show that our system yields superior results, both in terms of perceptual speech quality and classification error.}
\field{booktitle}{2017 {IEEE} International Conference on Acoustics, Speech and Signal Processing ({ICASSP})}
\field{month}{3}
\field{title}{{DNN-based} speech mask estimation for eigenvector beamforming}
\field{year}{2017}
\field{pages}{66\bibrangedash 70}
\range{pages}{5}
\keyw{Speech;Speech enhancement;Noise measurement;Estimation;Microphones;Array signal processing;Geometry;multi-channel speech enhancement;eigenvector beamforming;speech mask estimation}
\endentry
\entry{Wang2018-yy}{article}{}
\name{author}{2}{}{%
{{hash=be523e3be2cd7ae84ae25812787a6138}{%
family={Wang},
familyi={W\bibinitperiod},
given={Deliang},
giveni={D\bibinitperiod}}}%
{{hash=802e815ed5fcef558fc2d859ea49149b}{%
family={Chen},
familyi={C\bibinitperiod},
given={Jitong},
giveni={J\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\strng{namehash}{570e211fdcdc20a189cdac57b2932b27}
\strng{fullhash}{570e211fdcdc20a189cdac57b2932b27}
\strng{bibnamehash}{570e211fdcdc20a189cdac57b2932b27}
\strng{authorbibnamehash}{570e211fdcdc20a189cdac57b2932b27}
\strng{authornamehash}{570e211fdcdc20a189cdac57b2932b27}
\strng{authorfullhash}{570e211fdcdc20a189cdac57b2932b27}
\field{sortinit}{8}
\field{sortinithash}{a231b008ebf0ecbe0b4d96dcc159445f}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Speech separation is the task of separating target speech from background interference. Traditionally, speech separation is studied as a signal processing problem. A more recent approach formulates speech separation as a supervised learning problem, where the discriminative patterns of speech, speakers, and background noise are learned from training data. Over the past decade, many supervised separation algorithms have been put forward. In particular, the recent introduction of deep learning to supervised speech separation has dramatically accelerated progress and boosted separation performance. This paper provides a comprehensive overview of the research on deep learning based supervised speech separation in the last several years. We first introduce the background of speech separation and the formulation of supervised separation. Then, we discuss three main components of supervised separation: learning machines, training targets, and acoustic features. Much of the overview is on separation algorithms where we review monaural methods, including speech enhancement (speech-nonspeech separation), speaker separation (multitalker separation), and speech dereverberation, as well as multimicrophone techniques. The important issue of generalization, unique to supervised learning, is discussed. This overview provides a historical perspective on how advances are made. In addition, we discuss a number of conceptual issues, including what constitutes the target source.}
\field{journaltitle}{IEEE/ACM Trans Audio Speech Lang Process}
\field{month}{10}
\field{number}{10}
\field{title}{Supervised Speech Separation Based on Deep Learning: An Overview}
\field{volume}{26}
\field{year}{2018}
\field{pages}{1702\bibrangedash 1726}
\range{pages}{25}
\keyw{Seech separation; array separation; beamforming; deep learning; deep neural networks; speaker separation; speech dereverberation; speech enhancement; supervised speech separation; time-frequency masking}
\endentry
\entry{Boeddeker2018-ww}{inproceedings}{}
\name{author}{4}{}{%
{{hash=e578b99a66db18a38903ca6cab830a80}{%
family={Boeddeker},
familyi={B\bibinitperiod},
given={Christoph},
giveni={C\bibinitperiod}}}%
{{hash=ead22159a811327ebaf2ce764be2c2d2}{%
family={Erdogan},
familyi={E\bibinitperiod},
given={Hakan},
giveni={H\bibinitperiod}}}%
{{hash=4e513f74051b8871c6d84ebc40e90240}{%
family={Yoshioka},
familyi={Y\bibinitperiod},
given={Takuya},
giveni={T\bibinitperiod}}}%
{{hash=27394268ec19154143123cbcd93f5788}{%
family={Haeb-Umbach},
familyi={H\bibinithyphendelim U\bibinitperiod},
given={Reinhold},
giveni={R\bibinitperiod}}}%
}
\strng{namehash}{3d71b548dd0d89f50ea5ad151e21ba66}
\strng{fullhash}{43aea81f1aca0cf9543b8c167669c6ac}
\strng{bibnamehash}{43aea81f1aca0cf9543b8c167669c6ac}
\strng{authorbibnamehash}{43aea81f1aca0cf9543b8c167669c6ac}
\strng{authornamehash}{3d71b548dd0d89f50ea5ad151e21ba66}
\strng{authorfullhash}{43aea81f1aca0cf9543b8c167669c6ac}
\field{sortinit}{9}
\field{sortinithash}{0a5ebc79d83c96b6579069544c73c7d4}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This work examines acoustic beamformers employing neural networks (NNs) for mask prediction as front -end for automatic speech recognition (ASR) systems for practical scenarios like voice-enabled home devices. To test the versatility of the mask predicting network, the system is evaluated with different recording hardware, different microphone array designs, and different acoustic models of the downstream ASR system. Significant gains in recognition accuracy are obtained in all configurations despite the fact that the NN had been trained on mismatched data. Unlike previous work, the NN is trained on a feature level objective, which gives some performance advantage over a mask related criterion. Furthermore, different approaches for realizing online, or adaptive, NN-based beamforming are explored, where the online algorithms still show significant gains compared to the baseline performance.}
\field{booktitle}{2018 {IEEE} International Conference on Acoustics, Speech and Signal Processing ({ICASSP})}
\field{month}{4}
\field{title}{Exploring Practical Aspects of Neural {Mask-Based} Beamforming for {Far-Field} Speech Recognition}
\field{year}{2018}
\field{pages}{6697\bibrangedash 6701}
\range{pages}{5}
\keyw{Array signal processing;Artificial neural networks;Microphones;Acoustics;Speech recognition;Time-frequency analysis;Training;Far-field speech recognition;acoustic beamforming;neural networks;time-frequency masks;online processing}
\endentry
\entry{Wang2018-tl}{article}{}
\name{author}{4}{}{%
{{hash=3a7cb8e2b17f299e888fd1039696a93c}{%
family={Wang},
familyi={W\bibinitperiod},
given={Ziteng},
giveni={Z\bibinitperiod}}}%
{{hash=8c20d285377b0667d451da42b73d2f11}{%
family={Vincent},
familyi={V\bibinitperiod},
given={Emmanuel},
giveni={E\bibinitperiod}}}%
{{hash=729fa852b3c297ec98ee59039d338508}{%
family={Serizel},
familyi={S\bibinitperiod},
given={Romain},
giveni={R\bibinitperiod}}}%
{{hash=985bd01a03eac016042d3ac2908f599f}{%
family={Yan},
familyi={Y\bibinitperiod},
given={Yonghong},
giveni={Y\bibinitperiod}}}%
}
\strng{namehash}{ac9c690937f124b379e9c173920b22f8}
\strng{fullhash}{64185ba3eab41667586961d026355c2e}
\strng{bibnamehash}{64185ba3eab41667586961d026355c2e}
\strng{authorbibnamehash}{64185ba3eab41667586961d026355c2e}
\strng{authornamehash}{ac9c690937f124b379e9c173920b22f8}
\strng{authorfullhash}{64185ba3eab41667586961d026355c2e}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Multichannel linear filters, such as the Multichannel Wiener Filter (MWF) and the Generalized Eigenvalue (GEV) beamformer are popular signal processing techniques which can improve speech recognition performance. In this paper, we present an experimental study on these linear filters in a specific speech recognition task, namely the CHiME-4 challenge, which features real recordings in multiple noisy environments. Specifically, the rank-1 MWF is employed for noise reduction and a new constant residual noise power constraint is derived which enhances the recognition performance. To fulfill the underlying rank-1 assumption, the speech covariance matrix is reconstructed based on eigenvectors or generalized eigenvectors. Then the rank-1 constrained MWF is evaluated with alternative multichannel linear filters under the same framework, which involves a Bidirectional Long Short-Term Memory (BLSTM) network for mask estimation. The proposed filter outperforms alternative ones, leading to a 40\% relative Word Error Rate (WER) reduction compared with the baseline Weighted Delay and Sum (WDAS) beamformer on the real test set, and a 15\% relative WER reduction compared with the GEV-BAN method. The results also suggest that the speech recognition accuracy correlates more with the Mel-frequency cepstral coefficients (MFCC) feature variance than with the noise reduction or the speech distortion level.}
\field{journaltitle}{Comput. Speech Lang.}
\field{month}{5}
\field{title}{Rank-1 constrained Multichannel Wiener Filter for speech recognition in noisy environments}
\field{volume}{49}
\field{year}{2018}
\field{pages}{37\bibrangedash 51}
\range{pages}{15}
\keyw{Rank-1 Multichannel Wiener Filter; Speech recognition; Residual noise power; Deep neural network}
\endentry
\entry{Wang2014-gj}{article}{}
\name{author}{3}{}{%
{{hash=fc399f85ccdd18f7c16d2cd99a42d132}{%
family={Wang},
familyi={W\bibinitperiod},
given={Yuxuan},
giveni={Y\bibinitperiod}}}%
{{hash=16ed3ce07488851b0a54751f4e9860c6}{%
family={Narayanan},
familyi={N\bibinitperiod},
given={Arun},
giveni={A\bibinitperiod}}}%
{{hash=be523e3be2cd7ae84ae25812787a6138}{%
family={Wang},
familyi={W\bibinitperiod},
given={Deliang},
giveni={D\bibinitperiod}}}%
}
\list{language}{1}{%
{en}%
}
\strng{namehash}{07078876090ac565a7938aeb298c3740}
\strng{fullhash}{07078876090ac565a7938aeb298c3740}
\strng{bibnamehash}{07078876090ac565a7938aeb298c3740}
\strng{authorbibnamehash}{07078876090ac565a7938aeb298c3740}
\strng{authornamehash}{07078876090ac565a7938aeb298c3740}
\strng{authorfullhash}{07078876090ac565a7938aeb298c3740}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Formulation of speech separation as a supervised learning problem has shown considerable promise. In its simplest form, a supervised learning algorithm, typically a deep neural network, is trained to learn a mapping from noisy features to a time-frequency representation of the target of interest. Traditionally, the ideal binary mask (IBM) is used as the target because of its simplicity and large speech intelligibility gains. The supervised learning framework, however, is not restricted to the use of binary targets. In this study, we evaluate and compare separation results by using different training targets, including the IBM, the target binary mask, the ideal ratio mask (IRM), the short-time Fourier transform spectral magnitude and its corresponding mask (FFT-MASK), and the Gammatone frequency power spectrum. Our results in various test conditions reveal that the two ratio mask targets, the IRM and the FFT-MASK, outperform the other targets in terms of objective intelligibility and quality metrics. In addition, we find that masking based targets, in general, are significantly better than spectral envelope based targets. We also present comparisons with recent methods in non-negative matrix factorization and speech enhancement, which show clear performance advantages of supervised speech separation.}
\field{journaltitle}{IEEE/ACM Trans Audio Speech Lang Process}
\field{month}{12}
\field{number}{12}
\field{title}{On Training Targets for Supervised Speech Separation}
\field{volume}{22}
\field{year}{2014}
\field{pages}{1849\bibrangedash 1858}
\range{pages}{10}
\keyw{Deep neural networks; speech separation; supervised learning; training targets}
\endentry
\entry{Araki2007-yp}{inproceedings}{}
\name{author}{3}{}{%
{{hash=85b79982c3abbb65052e7e88cb76c256}{%
family={Araki},
familyi={A\bibinitperiod},
given={Shoko},
giveni={S\bibinitperiod}}}%
{{hash=f36d23ed71036f57ac3c16aca2de8658}{%
family={Sawada},
familyi={S\bibinitperiod},
given={Hiroshi},
giveni={H\bibinitperiod}}}%
{{hash=c1f668f6669cb6de57b3f8679b388d9c}{%
family={Makino},
familyi={M\bibinitperiod},
given={Shoji},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{25ac1036560f4f9f3a0181b95aa6d54c}
\strng{fullhash}{25ac1036560f4f9f3a0181b95aa6d54c}
\strng{bibnamehash}{25ac1036560f4f9f3a0181b95aa6d54c}
\strng{authorbibnamehash}{25ac1036560f4f9f3a0181b95aa6d54c}
\strng{authornamehash}{25ac1036560f4f9f3a0181b95aa6d54c}
\strng{authorfullhash}{25ac1036560f4f9f3a0181b95aa6d54c}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{2007 {IEEE} International Conference on Acoustics, Speech and Signal Processing - {ICASSP} '07}
\field{title}{Blind Speech Separation in a Meeting Situation with Maximum {SNR} Beamformers}
\field{volume}{1}
\field{year}{2007}
\field{pages}{I--41--I\bibrangedash 44}
\range{pages}{-1}
\endentry
\entry{Warsitz2007-fn}{article}{}
\name{author}{2}{}{%
{{hash=12b9069ebf9d835e4e0062ef0302f131}{%
family={Warsitz},
familyi={W\bibinitperiod},
given={Ernst},
giveni={E\bibinitperiod}}}%
{{hash=27394268ec19154143123cbcd93f5788}{%
family={Haeb-Umbach},
familyi={H\bibinithyphendelim U\bibinitperiod},
given={Reinhold},
giveni={R\bibinitperiod}}}%
}
\strng{namehash}{936460967dac3a08beece09aa9b111d7}
\strng{fullhash}{936460967dac3a08beece09aa9b111d7}
\strng{bibnamehash}{936460967dac3a08beece09aa9b111d7}
\strng{authorbibnamehash}{936460967dac3a08beece09aa9b111d7}
\strng{authornamehash}{936460967dac3a08beece09aa9b111d7}
\strng{authorfullhash}{936460967dac3a08beece09aa9b111d7}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Maximizing the output signal-to-noise ratio (SNR) of a sensor array in the presence of spatially colored noise leads to a generalized eigenvalue problem. While this approach has extensively been employed in narrowband (antenna) array beamforming, it is typically not used for broadband (microphone) array beamforming due to the uncontrolled amount of speech distortion introduced by a narrowband SNR criterion. In this paper, we show how the distortion of the desired signal can be controlled by a single-channel post-filter, resulting in a performance comparable to the generalized minimum variance distortionless response beamformer, where arbitrary transfer functions relate the source and the microphones. Results are given both for directional and diffuse noise. A novel gradient ascent adaptation algorithm is presented, and its good convergence properties are experimentally revealed by comparison with alternatives from the literature. A key feature of the proposed beamformer is that it operates blindly, i.e., it neither requires knowledge about the array geometry nor an explicit estimation of the transfer functions from source to sensors or the direction-of-arrival.}
\field{journaltitle}{IEEE Trans. Audio Speech Lang. Processing}
\field{month}{7}
\field{number}{5}
\field{title}{Blind Acoustic Beamforming Based on Generalized Eigenvalue Decomposition}
\field{volume}{15}
\field{year}{2007}
\field{pages}{1529\bibrangedash 1539}
\range{pages}{11}
\keyw{Array signal processing;Eigenvalues and eigenfunctions;Sensor arrays;Microphone arrays;Signal to noise ratio;Acoustic distortion;Narrowband;Broadband antennas;Antenna arrays;Transfer functions;Blind beamforming;generalized eigenvalue decomposition;microphone arrays;speech distortion}
\endentry
\entry{Heymann2018-mf}{inproceedings}{}
\name{author}{3}{}{%
{{hash=e516c4bc1fa1598297c1e5129475435c}{%
family={Heymann},
familyi={H\bibinitperiod},
given={Jahn},
giveni={J\bibinitperiod}}}%
{{hash=ed39ca669f9aeda3bcbf13427965f5f5}{%
family={Bacchiani},
familyi={B\bibinitperiod},
given={Michiel},
giveni={M\bibinitperiod}}}%
{{hash=28a3d000e5cd60656250014708db8ec1}{%
family={Sainath},
familyi={S\bibinitperiod},
given={Tara\bibnamedelima N},
giveni={T\bibinitperiod\bibinitdelim N\bibinitperiod}}}%
}
\strng{namehash}{03e60d8efb8772e7abf07d8dd7e9662c}
\strng{fullhash}{03e60d8efb8772e7abf07d8dd7e9662c}
\strng{bibnamehash}{03e60d8efb8772e7abf07d8dd7e9662c}
\strng{authorbibnamehash}{03e60d8efb8772e7abf07d8dd7e9662c}
\strng{authornamehash}{03e60d8efb8772e7abf07d8dd7e9662c}
\strng{authorfullhash}{03e60d8efb8772e7abf07d8dd7e9662c}
\field{sortinit}{1}
\field{sortinithash}{4f6aaa89bab872aa0999fec09ff8e98a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Mask based statistical beamforming, where signal statistics for the target and the interference gained from masking are used for beamforming, has shown great effectiveness in the two recent CHiME challenges. This idea has sparked interest in the research community and resulted in numerous proposed approaches based on the idea. At the same time, the advent of voice controlled smart home devices, such as Google Home and Amazon Alexa, has strengthened the need for robust far-field automatic speech recognition. In this paper, we evaluate if mask based beamforming can live up to the expectations created by the CHiME challenges and provide similar gains in a smart home scenario. To this extend, we pinpoint the main differences between the scenarios, review the recent developments and conduct extensive experiments on large scale data. These experiments show that, while a 10 \% relative reduction of the word error rate can be achieved, the gains are not as high as those seen in the CHiME challenge. We also show that approaches where the frontend and back-end is trained jointly do not reach the performance level of their independently trained counterparts. On the plus side, we see a 20 \% relative improvement for an evaluation set with crosstalk.}
\field{booktitle}{2018 {IEEE} International Conference on Acoustics, Speech and Signal Processing ({ICASSP})}
\field{month}{4}
\field{title}{Performance of Mask Based Statistical Beamforming in a Smart Home Scenario}
\field{year}{2018}
\field{pages}{6722\bibrangedash 6726}
\range{pages}{5}
\keyw{Array signal processing;Microphones;Smart homes;Training;Reverberation;Eigenvalues and eigenfunctions;Acoustic beamforming;multi-channel ASR;noise robust ASR;smart home}
\endentry
\entry{Shimada2018-uy}{inproceedings}{}
\name{author}{6}{}{%
{{hash=ce735b493d672ab5cd116bcf096a8e95}{%
family={Shimada},
familyi={S\bibinitperiod},
given={Kazuki},
giveni={K\bibinitperiod}}}%
{{hash=475ec54232e09f4506abf0333441d78f}{%
family={Bando},
familyi={B\bibinitperiod},
given={Yoshiaki},
giveni={Y\bibinitperiod}}}%
{{hash=23fe367eb667990ff5e3fd43ef5a6d01}{%
family={Mimura},
familyi={M\bibinitperiod},
given={Masato},
giveni={M\bibinitperiod}}}%
{{hash=a745c1030859eef5c5c86e892f0eef38}{%
family={Itoyama},
familyi={I\bibinitperiod},
given={Katsutoshi},
giveni={K\bibinitperiod}}}%
{{hash=c1db20d77ea05c54ee0ecbc2e7c74533}{%
family={Yoshii},
familyi={Y\bibinitperiod},
given={Kazuyoshi},
giveni={K\bibinitperiod}}}%
{{hash=2d4e75deaaf504f1988520976f2ab00e}{%
family={Kawahara},
familyi={K\bibinitperiod},
given={Tatsuya},
giveni={T\bibinitperiod}}}%
}
\strng{namehash}{c4f89484793ea79ac1ef81ffcba21f4f}
\strng{fullhash}{19c9a2a4c40bfcf75885f9f09c90fa0f}
\strng{bibnamehash}{19c9a2a4c40bfcf75885f9f09c90fa0f}
\strng{authorbibnamehash}{19c9a2a4c40bfcf75885f9f09c90fa0f}
\strng{authornamehash}{c4f89484793ea79ac1ef81ffcba21f4f}
\strng{authorfullhash}{19c9a2a4c40bfcf75885f9f09c90fa0f}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This paper presents unsupervised multichannel speech enhancement for noisy speech recognition. Time-frequency (TF) mask estimation has actively been studied for estimating the steering vectors and spatial covariance matrices of speech and noise used for beamforming. The state-of-the-art approach to mask estimation is to use deep neural networks (DNN s) for classifying the TF bins of observed signals into speech and noise. Such a supervised approach, however, does not work well in an unknown environment. To accurately estimate the spatial covariance matrices in an unsupervised manner, we perform blind source separation (BSS) based on multichannel nonnegative matrix factorization (MNMF) for decomposing each TF bin into the components of speech and the other sources (noise). To clarify a suitable type of beamforming for MNMF, we tested both time- invariant and time-varying versions of the minimum variance distortionless response (MVDR) beamforming in addition to standard multichannel Wiener filtering (MWF). The experimental results showed that our MNMF-based beamforming approach outperformed the state-of-the-art DNN-based beamforming method in unknown environments that do not match the training data.}
\field{booktitle}{2018 {IEEE} International Conference on Acoustics, Speech and Signal Processing ({ICASSP})}
\field{month}{4}
\field{title}{Unsupervised Beamforming Based on Multichannel Nonnegative Matrix Factorization for Noisy Speech Recognition}
\field{year}{2018}
\field{pages}{5734\bibrangedash 5738}
\range{pages}{5}
\keyw{Array signal processing;Covariance matrices;Estimation;Speech enhancement;Matrix decomposition;Noise measurement;Noisy speech recognition;speech enhancement;multichannel nonnegative matrix factorization;beamforming}
\endentry
\entry{Tu2017-ey}{inproceedings}{}
\name{author}{5}{}{%
{{hash=b3e75905ad08cf7b4058059896f90c27}{%
family={Tu},
familyi={T\bibinitperiod},
given={Yan-Hui},
giveni={Y\bibinithyphendelim H\bibinitperiod}}}%
{{hash=f60521cb1b3dc6ad645440ca7844bd28}{%
family={Du},
familyi={D\bibinitperiod},
given={Jun},
giveni={J\bibinitperiod}}}%
{{hash=48c4637a04d7042f2b0d3daec079d61e}{%
family={Sun},
familyi={S\bibinitperiod},
given={Lei},
giveni={L\bibinitperiod}}}%
{{hash=8621feb862cd029dbdd26654858ea3df}{%
family={Ma},
familyi={M\bibinitperiod},
given={Feng},
giveni={F\bibinitperiod}}}%
{{hash=26883d806fdb5a0cdc88e4154acb0773}{%
family={Lee},
familyi={L\bibinitperiod},
given={Chin-Hui},
giveni={C\bibinithyphendelim H\bibinitperiod}}}%
}
\list{location}{1}{%
{ISCA}%
}
\list{publisher}{1}{%
{ISCA}%
}
\strng{namehash}{9d48aacdaaf051399fffe29101bec24d}
\strng{fullhash}{5a32ee447bf7da8f60e17f072e885ffd}
\strng{bibnamehash}{5a32ee447bf7da8f60e17f072e885ffd}
\strng{authorbibnamehash}{5a32ee447bf7da8f60e17f072e885ffd}
\strng{authornamehash}{9d48aacdaaf051399fffe29101bec24d}
\strng{authorfullhash}{5a32ee447bf7da8f60e17f072e885ffd}
\field{extraname}{1}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{We design a novel deep learning framework for multi-channel speech recognition in two aspects. First, for the front-end, an iterative mask estimation (IME) approach based on deep learning is presented to improve the beamforming approach based on the conventional complex Gaussian mixture model (CGMM). Second, for the back-end, deep convolutional neural networks (DCNNs), with augmentation of both noisy and beamformed training data, are adopted for acoustic modeling while the forward and backward long short-term memory recurrent neural networks (LSTM-RNNs) are used for language modeling. The proposed framework can be quite effective to multi-channel speech recognition with random combinations of fixed microphones. Testing on the CHiME-4 Challenge speech recognition task with a single set of acoustic and language models, our approach achieves the best performance of all three tracks (1channel, 2-channel, and 6-channel) among submitted systems.}
\field{booktitle}{Interspeech 2017}
\field{month}{8}
\field{title}{On design of robust deep models for {CHiME-4} multi-channel speech recognition with multiple configurations of array microphones}
\field{year}{2017}
\endentry
\entry{Tu2019-qp}{article}{}
\name{author}{3}{}{%
{{hash=b3e75905ad08cf7b4058059896f90c27}{%
family={Tu},
familyi={T\bibinitperiod},
given={Yan-Hui},
giveni={Y\bibinithyphendelim H\bibinitperiod}}}%
{{hash=f60521cb1b3dc6ad645440ca7844bd28}{%
family={Du},
familyi={D\bibinitperiod},
given={Jun},
giveni={J\bibinitperiod}}}%
{{hash=26883d806fdb5a0cdc88e4154acb0773}{%
family={Lee},
familyi={L\bibinitperiod},
given={Chin-Hui},
giveni={C\bibinithyphendelim H\bibinitperiod}}}%
}
\strng{namehash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\strng{fullhash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\strng{bibnamehash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\strng{authorbibnamehash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\strng{authornamehash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\strng{authorfullhash}{85acc98ddbf4ea0bdc9eadcfc22a6eeb}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{In this paper, we propose a novel teacher-student learning framework for the preprocessing of a speech recognizer, leveraging the online noise tracking capabilities of improved minima controlled recursive averaging (IMCRA) and deep learning of nonlinear interactions between speech and noise. First, a teacher model with deep architectures is built to learn the target of ideal ratio masks (IRMs) using simulated training pairs of clean and noisy speech data. Next, a student model is trained to learn an improved speech presence probability by incorporating the estimated IRMs from the teacher model into the IMCRA approach. The student model can be compactly designed in a causal processing mode having no latency with the guidance of a complex and noncausal teacher model. Moreover, the clean speech requirement, which is difficult to meet in real-world adverse environments, can be relaxed for training the student model, implying that noisy speech data can be directly used to adapt the regression-based enhancement model to further improve speech recognition accuracies for noisy speech collected in such conditions. Experiments on the CHiME-4 challenge task show that our best student model with bidirectional gated recurrent units (BGRUs) can achieve a relative word error rate (WER) reduction of 18.85\% for the real test set when compared to unprocessed system without acoustic model retraining. However, the traditional teacher model degrades the performance of the unprocessed system in this case. In addition, the student model with a deep neural network (DNN) in causal mode having no latency yields a relative WER reduction of 7.94\% over the unprocessed system with 670 times less computing cycles when compared to the BGRU-equipped student model. Finally, the conventional speech enhancement and IRM-based deep learning method destroyed the ASR performance when the recognition system became more powerful. While our proposed approach could still improve the ASR performance even in the more powerful recognition system.}
\field{journaltitle}{IEEE/ACM Transactions on Audio, Speech, and Language Processing}
\field{month}{12}
\field{number}{12}
\field{title}{Speech Enhancement Based on {Teacher--Student} Deep Learning Using Improved Speech Presence Probability for {Noise-Robust} Speech Recognition}
\field{volume}{27}
\field{year}{2019}
\field{pages}{2080\bibrangedash 2091}
\range{pages}{12}
\keyw{Speech enhancement;Speech recognition;Noise measurement;Adaptation models;Computational modeling;Training;Teacher-student learning;improved minima controlled recursive averaging;improved speech presence probability;deep learning based speech enhancement;noise-robust speech recognition}
\endentry
\entry{Chakrabarty2019-th}{article}{}
\name{author}{2}{}{%
{{hash=12e65489d8ed14f67cde68a27e5a82e6}{%
family={Chakrabarty},
familyi={C\bibinitperiod},
given={Soumitro},
giveni={S\bibinitperiod}}}%
{{hash=931f512188e698674079e7d17ee399a7}{%
family={Habets},
familyi={H\bibinitperiod},
given={Emanuël\bibnamedelimb A\bibnamedelima P},
giveni={E\bibinitperiod\bibinitdelim A\bibinitperiod\bibinitdelim P\bibinitperiod}}}%
}
\strng{namehash}{969b1925a968918a3e14535308dc0a8d}
\strng{fullhash}{969b1925a968918a3e14535308dc0a8d}
\strng{bibnamehash}{969b1925a968918a3e14535308dc0a8d}
\strng{authorbibnamehash}{969b1925a968918a3e14535308dc0a8d}
\strng{authornamehash}{969b1925a968918a3e14535308dc0a8d}
\strng{authorfullhash}{969b1925a968918a3e14535308dc0a8d}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This paper presents a time-frequency masking based online multi-channel speech enhancement approach that uses a convolutional recurrent neural network to estimate the mask. The magnitude and phase components of the short-time Fourier transform coefficients for multiple time frames are provided as an input such that the network is able to discriminate between the directional speech and the noise components based on the spatial characteristics of the individual signals as well as their spectro-temporal structure. The estimation of two different masks, namely, ideal ratio mask (IRM) and ideal binary mask (IBM), along with two different approaches for incorporating the mask to obtain the desired signal are discussed. In the first approach, the mask is directly applied as a real valued gain to a reference microphone signal, whereas in the second approach, the masks are used as an activity indicator for the recursive update of power spectral density (PSD) matrices to be used within a beamformer. The performance of the proposed system with the two different estimated masks utilized within the two different enhancement approaches is evaluated with both simulated as well as measured room impulse responses, where it is shown that the IBM is better suited as an indicator for the PSD updates while direct application of IRM as a real valued gain leads to a better improvement in terms of short term objective intelligibility. Analysis of the performance of the proposed system also demonstrates the robustness of the system to different angular positions of the speech source.}
\field{journaltitle}{IEEE J. Sel. Top. Signal Process.}
\field{month}{8}
\field{number}{4}
\field{title}{{Time--Frequency} Masking Based Online {Multi-Channel} Speech Enhancement With Convolutional Recurrent Neural Networks}
\field{volume}{13}
\field{year}{2019}
\field{pages}{787\bibrangedash 799}
\range{pages}{13}
\keyw{Microphones;Speech enhancement;Estimation;Time-frequency analysis;Feature extraction;Recurrent neural networks;Array signal processing;Speech enhancement;microphone array;convolutional recurrent neural networks;time-frequency masking}
\endentry
\entry{Tu2018-ts}{article}{}
\name{author}{4}{}{%
{{hash=a0bcdc2dd4aa0ac41f12e9c289d5bf50}{%
family={Tu},
familyi={T\bibinitperiod},
given={Yan\bibnamedelima Hui},
giveni={Y\bibinitperiod\bibinitdelim H\bibinitperiod}}}%
{{hash=f60521cb1b3dc6ad645440ca7844bd28}{%
family={Du},
familyi={D\bibinitperiod},
given={Jun},
giveni={J\bibinitperiod}}}%
{{hash=48c4637a04d7042f2b0d3daec079d61e}{%
family={Sun},
familyi={S\bibinitperiod},
given={Lei},
giveni={L\bibinitperiod}}}%
{{hash=92e828b394670304316f6ff648bc64b4}{%
family={Lee},
familyi={L\bibinitperiod},
given={Chin\bibnamedelima Hui},
giveni={C\bibinitperiod\bibinitdelim H\bibinitperiod}}}%
}
\strng{namehash}{cbff8492898c4d79b7d320523be939a0}
\strng{fullhash}{25a5bb70e7676e9a434501d8e9925033}
\strng{bibnamehash}{25a5bb70e7676e9a434501d8e9925033}
\strng{authorbibnamehash}{25a5bb70e7676e9a434501d8e9925033}
\strng{authornamehash}{cbff8492898c4d79b7d320523be939a0}
\strng{authorfullhash}{25a5bb70e7676e9a434501d8e9925033}
\field{extraname}{2}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{Recently, we propose an iterative mask estimation (IME) approach to improve the conventional complex Gaussian mixture model (CGMM) based beamforming and yield the best multi-channel speech recognition accuracy in CHiME-4 challenge. In this study, we focus on multi-channel speech enhancement and present a novel approach via long short-Term memory (LSTM) based IME and post-processing. First, an LSTM is adopted to estimate the ideal ratio mask (IRM) to improve the mask estimated by a CGMM. Then, the improved mask is used to derive a beamformer. Finally, the IME-based beamformed speech is processed by the LSTM-based regression model. Experiments on the CHiME-4 simulation data show that LSTM-based IME approach can improve the PESQ performance comparing to unprocessed signals, with relative PESQ improvements of 17.33\% and 13.89\%, and the LSTM-based post-processing can further yield performance gains based on the IME approach, with relative PESQ improvements of 11.42\% and 10.00\% for 6-channel and 2-channel cases, respectively.}
\field{journaltitle}{Proceedings - 9th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017}
\field{number}{December}
\field{title}{{LSTM-based} iterative mask estimation and post-processing for multi-channel speech enhancement}
\field{volume}{2018-Febru}
\field{year}{2018}
\field{pages}{488\bibrangedash 491}
\range{pages}{4}
\endentry
\entry{Malek2020-nw}{article}{}
\name{author}{3}{}{%
{{hash=138d40d47b9e82fd1b386a17f4c9ba34}{%
family={Malek},
familyi={M\bibinitperiod},
given={Jiri},
giveni={J\bibinitperiod}}}%
{{hash=2f4bcf6b24ef7571f64e3e78caecb2c4}{%
family={Koldovský},
familyi={K\bibinitperiod},
given={Zbynĕk},
giveni={Z\bibinitperiod}}}%
{{hash=5820c85a42bb08369a8576700c515041}{%
family={Bohac},
familyi={B\bibinitperiod},
given={Marek},
giveni={M\bibinitperiod}}}%
}
\list{publisher}{2}{%
{Institution of Engineering}%
{Technology}%
}
\strng{namehash}{b529e662849b3c83b831ac353147d086}
\strng{fullhash}{b529e662849b3c83b831ac353147d086}
\strng{bibnamehash}{b529e662849b3c83b831ac353147d086}
\strng{authorbibnamehash}{b529e662849b3c83b831ac353147d086}
\strng{authornamehash}{b529e662849b3c83b831ac353147d086}
\strng{authorfullhash}{b529e662849b3c83b831ac353147d086}
\field{sortinit}{2}
\field{sortinithash}{8b555b3791beccb63322c22f3320aa9a}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This work addresses the problem of block-online processing for multi-channel speech enhancement. Such processing is vital in scenarios with moving speakers and/or when short utterances are processed, e.g. in voice assistant applications. We consider several variants of a system that performs beamforming supported by deep neural network-based voice activity detection followed by post-filtering. The speaker is targeted through estimating relative transfer functions between microphones. Each block of the input signals is processed independently to make the method applicable in highly dynamic environments. Due to short processed blocks, the statistics required by the beamformer are estimated less precisely. The influence of this inaccuracy is studied and compared to batch processing regime, when recordings are treated as one block. The experimental evaluation is performed on large datasets of CHiME-4 and another dataset featuring moving target speaker. The experiments are evaluated in terms of objective and perceptual criteria. Moreover, word error rate (WER) of a speech recognition system is evaluated, for which the method serves as a front-end. The results indicate that the proposed method is robust for short length of the processed block. Significant improvements in terms of the criteria and WER are observed even for the block length of 250 ms.}
\field{journaltitle}{IET Signal Proc.}
\field{month}{5}
\field{number}{3}
\field{title}{Block-online multi-channel speech enhancement using deep neural network-supported relative transfer function estimates}
\field{volume}{14}
\field{year}{2020}
\field{pages}{124\bibrangedash 133}
\range{pages}{10}
\endentry
\entry{Hiroe2022-rd}{article}{}
\name{author}{1}{}{%
{{hash=c893ce10dc28a2bcf4d4b85e356a06f3}{%
family={Hiroe},
familyi={H\bibinitperiod},
given={Atsuo},
giveni={A\bibinitperiod}}}%
}
\strng{namehash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\strng{fullhash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\strng{bibnamehash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\strng{authorbibnamehash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\strng{authornamehash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\strng{authorfullhash}{c893ce10dc28a2bcf4d4b85e356a06f3}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Open Journal of Signal Processing}
\field{title}{{Similarity-and-Independence-Aware} Beamformer With Iterative Casting and Boost Start for Target Source Extraction Using Reference}
\field{volume}{3}
\field{year}{2022}
\field{pages}{1\bibrangedash 20}
\range{pages}{20}
\endentry
\entry{Barker2017-tr}{article}{}
\name{author}{4}{}{%
{{hash=f0b6d95e6a2fe67942c7057d273ba2c5}{%
family={Barker},
familyi={B\bibinitperiod},
given={Jon},
giveni={J\bibinitperiod}}}%
{{hash=ae7cd7efe9563d2779d099d6578041c0}{%
family={Marxer},
familyi={M\bibinitperiod},
given={Ricard},
giveni={R\bibinitperiod}}}%
{{hash=8c20d285377b0667d451da42b73d2f11}{%
family={Vincent},
familyi={V\bibinitperiod},
given={Emmanuel},
giveni={E\bibinitperiod}}}%
{{hash=ae898cb785c2eecb4cda42d862e0d5d4}{%
family={Watanabe},
familyi={W\bibinitperiod},
given={Shinji},
giveni={S\bibinitperiod}}}%
}
\strng{namehash}{91627b5528634b4a4bef7510daca3e3c}
\strng{fullhash}{52c55efb6925d2891704de7dcad637f6}
\strng{bibnamehash}{52c55efb6925d2891704de7dcad637f6}
\strng{authorbibnamehash}{52c55efb6925d2891704de7dcad637f6}
\strng{authornamehash}{91627b5528634b4a4bef7510daca3e3c}
\strng{authorfullhash}{52c55efb6925d2891704de7dcad637f6}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{abstract}{This paper presents the design and outcomes of the CHiME-3 challenge, the first open speech recognition evaluation designed to target the increasingly relevant multichannel, mobile-device speech recognition scenario. The paper serves two purposes. First, it provides a definitive reference for the challenge, including full descriptions of the task design, data capture and baseline systems along with a description and evaluation of the 26 systems that were submitted. The best systems re-engineered every stage of the baseline resulting in reductions in word error rate from 33.4\% to as low as 5.8\%. By comparing across systems, techniques that are essential for strong performance are identified. Second, the paper considers the problem of drawing conclusions from evaluations that use speech directly recorded in noisy environments. The degree of challenge presented by the resulting material is hard to control and hard to fully characterise. We attempt to dissect the various `axes of difficulty' by correlating various estimated signal properties with typical system performance on a per session and per utterance basis. We find strong evidence of a dependence on signal-to-noise ratio and channel quality. Systems are less sensitive to variations in the degree of speaker motion. The paper concludes by discussing the outcomes of CHiME-3 in relation to the design of future mobile speech recognition evaluations.}
\field{journaltitle}{Comput. Speech Lang.}
\field{title}{The third {`CHiME'} speech separation and recognition challenge: Analysis and outcomes}
\field{year}{2017}
\keyw{Microphone array; Noise-robust ASR; `CHiME' challenge}
\endentry
\entry{Paszke2019-dm}{incollection}{}
\name{author}{21}{}{%
{{hash=56bf0b340039cf8594436a624ff548a9}{%
family={Paszke},
familyi={P\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=4ba5062e5919c814aceec188d54c01f2}{%
family={Gross},
familyi={G\bibinitperiod},
given={Sam},
giveni={S\bibinitperiod}}}%
{{hash=e5dfae4582081d649e3a0d5342050016}{%
family={Massa},
familyi={M\bibinitperiod},
given={Francisco},
giveni={F\bibinitperiod}}}%
{{hash=b5815e1692fa2d0c1f44eecf509bd7c4}{%
family={Lerer},
familyi={L\bibinitperiod},
given={Adam},
giveni={A\bibinitperiod}}}%
{{hash=b75383e6b48c8360c7a60031424c85cf}{%
family={Bradbury},
familyi={B\bibinitperiod},
given={James},
giveni={J\bibinitperiod}}}%
{{hash=f897ed422c34d95af2e22778dfc2607e}{%
family={Chanan},
familyi={C\bibinitperiod},
given={Gregory},
giveni={G\bibinitperiod}}}%
{{hash=046269e070246feb6f394141db80ed87}{%
family={Killeen},
familyi={K\bibinitperiod},
given={Trevor},
giveni={T\bibinitperiod}}}%
{{hash=c40352c194e60a3ef458ee7e8685afb5}{%
family={Lin},
familyi={L\bibinitperiod},
given={Zeming},
giveni={Z\bibinitperiod}}}%
{{hash=6e45f49ec618e619efad90c8e8a61f0c}{%
family={Gimelshein},
familyi={G\bibinitperiod},
given={Natalia},
giveni={N\bibinitperiod}}}%
{{hash=f65a80959d520337ae99a0798515036c}{%
family={Antiga},
familyi={A\bibinitperiod},
given={Luca},
giveni={L\bibinitperiod}}}%
{{hash=954cf7680b6ce14813973eccdca3c4bc}{%
family={Desmaison},
familyi={D\bibinitperiod},
given={Alban},
giveni={A\bibinitperiod}}}%
{{hash=c1b8f8db68d6667b9f2f9a9a3567721b}{%
family={Kopf},
familyi={K\bibinitperiod},
given={Andreas},
giveni={A\bibinitperiod}}}%
{{hash=b9e701339e56fd0b171145b08288a1b7}{%
family={Yang},
familyi={Y\bibinitperiod},
given={Edward},
giveni={E\bibinitperiod}}}%
{{hash=3f9535be511fd2fa346093e63b8e61a0}{%
family={DeVito},
familyi={D\bibinitperiod},
given={Zachary},
giveni={Z\bibinitperiod}}}%
{{hash=d814afaa50b9e22ab92cc9f8f9a9e43a}{%
family={Raison},
familyi={R\bibinitperiod},
given={Martin},
giveni={M\bibinitperiod}}}%
{{hash=3feeeebee8583ecc208f7fb3e0a55068}{%
family={Tejani},
familyi={T\bibinitperiod},
given={Alykhan},
giveni={A\bibinitperiod}}}%
{{hash=e18536d5cb7543731fbf2ca1a4908732}{%
family={Chilamkurthy},
familyi={C\bibinitperiod},
given={Sasank},
giveni={S\bibinitperiod}}}%
{{hash=0a0b028c6b85c46f368317d0c5bfe3a0}{%
family={Steiner},
familyi={S\bibinitperiod},
given={Benoit},
giveni={B\bibinitperiod}}}%
{{hash=998a001f16bb57c079c1d5afb1cb02c8}{%
family={Fang},
familyi={F\bibinitperiod},
given={Lu},
giveni={L\bibinitperiod}}}%
{{hash=3f19c633bbfb847db6a0e71d3659eacd}{%
family={Bai},
familyi={B\bibinitperiod},
given={Junjie},
giveni={J\bibinitperiod}}}%
{{hash=8ef51a0906e47d2b4472c4e714ed598f}{%
family={Chintala},
familyi={C\bibinitperiod},
given={Soumith},
giveni={S\bibinitperiod}}}%
}
\list{publisher}{1}{%
{Curran Associates, Inc.}%
}
\strng{namehash}{932e06a08a48d67a815ffec45fdea7aa}
\strng{fullhash}{ba1e2da270d08cb8de2856498a028fed}
\strng{bibnamehash}{932e06a08a48d67a815ffec45fdea7aa}
\strng{authorbibnamehash}{932e06a08a48d67a815ffec45fdea7aa}
\strng{authornamehash}{932e06a08a48d67a815ffec45fdea7aa}
\strng{authorfullhash}{ba1e2da270d08cb8de2856498a028fed}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{booktitle}{Advances in Neural Information Processing Systems 32}
\field{title}{{PyTorch}: An Imperative Style, {High-Performance} Deep Learning Library}
\field{year}{2019}
\field{pages}{8024\bibrangedash 8035}
\range{pages}{12}
\endentry
\entry{Vincent2006-pg}{article}{}
\name{author}{3}{}{%
{{hash=aaf9e64cd99753a5b1aa54d54ce7474f}{%
family={Vincent},
familyi={V\bibinitperiod},
given={E},
giveni={E\bibinitperiod}}}%
{{hash=5f955dd01e77e5928bea760e72b0081d}{%
family={Gribonval},
familyi={G\bibinitperiod},
given={R},
giveni={R\bibinitperiod}}}%
{{hash=ba52ecd4a5d2a6c1223ab913c8927a32}{%
family={Fevotte},
familyi={F\bibinitperiod},
given={C},
giveni={C\bibinitperiod}}}%
}
\strng{namehash}{d197b757b505bd8d0ebe9f4946b54d64}
\strng{fullhash}{d197b757b505bd8d0ebe9f4946b54d64}
\strng{bibnamehash}{d197b757b505bd8d0ebe9f4946b54d64}
\strng{authorbibnamehash}{d197b757b505bd8d0ebe9f4946b54d64}
\strng{authornamehash}{d197b757b505bd8d0ebe9f4946b54d64}
\strng{authorfullhash}{d197b757b505bd8d0ebe9f4946b54d64}
\field{sortinit}{3}
\field{sortinithash}{ad6fe7482ffbd7b9f99c9e8b5dccd3d7}
\field{labelnamesource}{author}
\field{labeltitlesource}{title}
\field{journaltitle}{IEEE Trans. Audio Speech Lang. Processing}
\field{number}{4}
\field{title}{Performance measurement in blind audio source separation}
\field{volume}{14}
\field{year}{2006}
\field{pages}{1462\bibrangedash 1469}
\range{pages}{8}
\endentry
\enddatalist
\endrefsection
\endinput