Over-Tightening-Aware Pseudo-Labeling for
Tight-Boundary Speaker Diarization

In most end-to-end speaker diarization frameworks, speaker diarization is formulated as a fully supervised classification problem. Let $X\in\mathbb{R}^{F\times L}$ be an input sequence of length $L$, where each time step has $F$ dimensions. Here, $F=1$ corresponds to a raw waveform and $F>1$ corresponds to arbitrary spectral features. A diarization model based on a typically non-causal neural network $f(\cdot)$ transforms the input into frame-wise posterior probabilities as

where $\mbox{\boldmath$q$}_{t}\coloneqq[q_{1,t},\dots,q_{C,t}]^{\mathsf{T}}\in(0,1)^{C}$ represents posteriors for $C$ classes at $t$-th frame with $1\leq t\leq T$. In this study, we use a powerset classification formulation, where each class represents a possible set of active speakers [42]. Thus, $C=\sum_{m=0}^{M}\binom{S}{m}$, where $S$ is the maximum number of speakers per input and $M$ is the maximum number of speakers who speak simultaneously at each frame. These powerset-based posteriors $Q$ can be transformed to the speaker-wise posteriors $P\coloneqq[\mbox{\boldmath$p$}_{1},\dots,\mbox{\boldmath$p$}_{T}]\in(0,1)^{S\times T}$ by using the function $\mathsf{Q2P}:Q\mapsto P$ defined as

where $\mbox{\boldmath$p$}_{t}\coloneqq[p_{1,t},\dots,p_{S,t}]^{\mathsf{T}}$, and $\mathcal{S}_{c}$ is the set of speakers corresponding to powerset class $c$.

The model $f(\cdot)$ is optimized using the cross-entropy loss. Since the output speaker order is arbitrary, the speaker order of the model output must be aligned with that of the reference label beforehand:

where $Y\coloneqq(y_{s,t})\in\{0,1\}^{S\times T}$ denotes the reference label, with $y_{s,t}=1$ if speaker $s$ is speaking at frame $t$ and $0$ otherwise, $\Pi(P)$ denotes the set of all possible speaker-permuted $P$, and $\left\lVert\cdot\right\rVert_{F}$ denotes the Frobenius norm. The speaker-aligned powerset posteriors $q^{(Y)}_{c,t}$ can be obtained accordingly using the optimal permutation computed in (3). Then the loss to be minimized is computed as

where $z_{c,t}$ is the following powerset-based reference label:

Most end-to-end neural diarization studies treated annotated labels $\tilde{Y}\coloneqq(\tilde{y}_{s,t})\in\{0,1\}^{S\times T}$ as the reference, i.e., $\tilde{Y}=Y$. However, the original annotations in some corpora are not sufficiently tight for speaker diarization, mainly because they were developed for ASR and thus avoid excessive segmentation that would break semantic continuity. If a model is trained using such loose annotations, it tends to produce similarly loose speech segments [28]. Motivated by this issue, tight-boundary speaker diarization has recently been proposed as a framework for training a diarization model using only loose annotations while enabling it to predict tight speech boundaries [25]. It assumes one-sided label noise in the speech class, like

Such noise arises from onset or offset padding and pause filling.

In tight-boundary speaker diarization, the loss in (4) cannot be computed directly because the label $y_{s,t}$ required for computing the loss is unknown. To address this issue, as explained in Section I, previous work exploits the properties that causal and anticausal models cannot perform onset padding and offset padding, respectively, and that neither model can perform pause filling perfectly [25]. Specifically, tight pseudo-labels are generated by combining the outputs of causal and anticausal models trained on loose annotations, which are used to compute the loss. The overall procedure for training a non-causal model that can output tight predictions is summarized below.

• •

  Step 1: Train causal and anticausal models independently using the original loose labels $\tilde{Y}$.

• •

  Step 2: Co-train causal and anticausal models from the pretrained weights. At each training iteration, pseudo-labels are generated on the fly from the outputs of the causal and anticausal models, and both models are updated using these pseudo-labels.

• •

  Step 3: Train a non-causal model using the final pseudo-labels.

For later discussion, we describe the pseudo-label generation process, which is used in Step 2 and Step 3, in more detail using Algorithm 1. First, the posterior outputs of the causal and anticausal models over powerset classes, $\causal{Q}$ and $\anticausal{Q}$, are converted into speaker-wise posteriors (Algorithm 1). Next, since the outputs of these models are invariant to speaker permutation, they are aligned with $\tilde{Y}$ (Algorithm 1). These model outputs may contain speaker confusion, and using them directly as pseudo-labels can degrade performance. To mitigate this effect, the prior study proposed two correction strategies: a conservative method based on voice activity detection (VAD) and a more aggressive method based on speaker counting (SC). In this paper, we adopt the SC-based method, which refers to the loose label $\tilde{Y}$ and, at each frame, swaps the posteriors of missed speakers with those of falsely detected speakers (Algorithm 1). Finally, we obtain the tightened pseudo-label $Y_{\mathrm{tight}}$ by first applying the element-wise thresholding function $\phi_{\tau}(\cdot)$ with threshold $\tau$ to the average of $\causal{P}$ and $\anticausal{P}$, and then using the resulting binary mask to mask the loose label $\tilde{Y}$ (Algorithm 1). Finally, since the tightened labels are derived from model predictions, they may be overly tightened. Assuming that speech should not completely disappear within a loose segment, we restore such disappeared segments (Algorithm 1). The prior study adopts a more relaxed criterion, restoring a loose segment only when more than $50\text{\,}\mathrm{\%}$ of the segment has disappeared.

The prior work pioneered tight-boundary speaker diarization by formulating it as a new problem and proposing the first solution [25]. However, its method still has room for improvement. One of its major drawbacks is over-tightening, which increases missed detections and harms downstream tasks. This paper aims to improve this approach by reducing missed detection from multiple perspectives.

Large-scale pretrained models, like WavLM [10] used in DiariZen [22], usually do not support causal or anticausal inference. The prior study employs a multi-speaker embedding extractor as a pretrained temporal encoder instead, with the backend consisting of a bidirectional long short-term memory (LSTM) and a linear layer to obtain frame-wise posteriors [24]. This paper follows the same strategy and uses the ReDimNet-B2-based architecture for the pretrained encoder [59]. For causal and anticausal models, each convolutional kernel is replaced with a causal and anticausal variant of the same kernel size, and self-attention modules in Transformer encoders are applied with causal or anticausal masks, respectively. The bidirectional LSTM is also replaced with a unidirectional LSTM in the left-to-right and right-to-left directions, respectively.

In the pipeline of the prior study, as described in Sec. II-B, causal and anticausal models are first co-trained in Step 2 and then used to generate pseudo-labels for training the final non-causal model in Step 3. However, pseudo-labels from the best causal and anticausal models do not necessarily maximize the performance of the final non-causal model, potentially yielding outputs that are either over-tightened or insufficiently tight. This concern is consistent with findings in knowledge distillation, where the best-performing teacher model does not always lead to the best student model [12, 40, 54, 16], and with pseudo-labeling studies showing that noisy or overconfident pseudo-labels can induce confirmation bias and noisy training [1, 50].

To address this issue, the proposed pipeline makes the co-training process aware of the final non-causal model by training it in parallel and using its validation performance for early stopping. We replace the procedure described in Sec. II-B with the following procedure for training a non-causal model capable of producing tight outputs:

• •

  Step 1: Train causal, anticausal, and non-causal models independently using loose labels.

• •

  Step 2: Starting from their respective pretrained weights, co-train the causal and anticausal models while training the non-causal model in parallel. At each training iteration, pseudo-labels are generated on the fly from the outputs of the causal and anticausal models and are used to update all three models.

Algorithm 3shows the detailed co-training procedure. The bold-faced symbols, e.g., $\causal{\mathbf{Q}}$, denote batched versions of the corresponding original symbols, e.g., $\causal{Q}$. As in the prior method, each iteration starts with a minibatch (Algorithm 3), generates tight pseudo-labels using the causal and anticausal models (lines 3–3), and updates these models using the generated pseudo-labels (lines 3–3). The key difference is that the proposed method also updates the non-causal model, initialized from loose-label training, using the pseudo-labels (lines 3–3). This avoids introducing a separate learning schedule for training the final model from scratch.

In addition, the stopping criterion is based on the validation performance of the non-causal model (Algorithm 3). In the prior method, early stopping was determined by the convergence of the causal and anticausal models. However, pseudo-labels generated using the best causal and anticausal models do not necessarily maximize the performance of the resulting non-causal model. Therefore, the proposed pipeline directly monitors the final non-causal model during the training loop and selects the training point that is better suited to the model used for final inference.

Our experiments are based on the EEND-vector clustering framework, which performs local speaker diarization using a sliding window and then aggregates the window-level results by clustering speaker embeddings [31]. The proposed method is applied to the local speaker diarization model used in the former step. In the latter step, we extracted speaker embeddings using the ResNet34 extractor trained with VoxCeleb [37]¹¹ 1 https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM, and then clustered them via VBx clustering [32, 39] using the corresponding PLDA model²² 2 https://huggingface.co/BUT-FIT/diarizen-wavlm-base-s80-md/tree/main/plda. The VBx clustering parameters, i.e., $F_{A}$ and $F_{B}$, are tuned on the validation set of each corpus using the pyannote.audio toolkit [7].

For local speaker diarization, we followed the prior study [25] and used the architecture based on a multi-speaker embedding extractor [24], as described in detail in Sec. II-C. The non-causal, causal, and anticausal ReDimNet encoders were first pretrained on VoxCeleb-based simulated mixtures following the same pretraining procedure [24]. The backend, consisting of an LSTM and a linear layer, was then attached to the encoder and trained as a diarization model using the compound set. The model and training configurations follow the corresponding prior studies and are summarized in Table I.

A summary of the datasets used in the experiments is provided in Table II. We used the compound set of five corpora for model training: AMI mixed headset microphones (AMI-MHM) and single distant microphones (AMI-SDM) [9], AliMeeting [60], MSDWild few-talker set (MSD) [35], and VoxConverse (VoxC) [13]. Note that AMI-MHM, AMI-SDM, and AliMeeting are ASR corpora, so they have loosely annotated labels, whereas MSD and VC are diarization corpora with tightly annotated labels.

Although the proposed method does not require tight labels for training on the training set, validation and evaluation still require tight labels. We therefore used AMI and AliMeeting, for which publicly available tight labels obtained via forced alignment are available.³³ 3 https://github.com/nttcslab-sp/diar-forced-alignment Specifically, we assume that these tight labels are available on the validation sets for monitoring training convergence, which is common practice in noisy-label training [49, 52]. We then evaluate each system on the test portion of each corpus in terms of diarization error rate (DER) without collar tolerance, using tight labels as references.

Table IIIreports the DERs of conventional and proposed approaches on each corpus. Because speaker confusion depends in part on the accuracy of clustering across chunks, we mainly focus our discussion on missed detection and false alarms. First, $\mathtt{L1}$ shows the DER of the loose labels themselves. AMI has approximately twice the DER of AliMeeting, which, together with the analysis in Sec. III-A, suggests that AMI contains roughly twice as much pause filling.

When evaluated against forced-aligned tight labels, $\mathtt{C1}$, trained with loose labels, performed substantially worse than the ideal tight-label topline $\mathtt{C2}$ on corpora with originally loose annotations. This degradation was mainly reflected in high false-alarm rates, indicating that the model produced loose speech segments. This observation is consistent with prior studies [28, 25]. The pseudo-labeling methods in the prior study [25], shown in $\mathtt{C3}$ and $\mathtt{C4}$, achieved performance close to $\mathtt{C2}$ without tight supervision, although a substantial gap still remained. Although the SC-based method ($\mathtt{C4}$) outperformed the VAD-based method ($\mathtt{C3}$) in DER, its increased missed detection and speaker confusion have been reported to affect downstream tasks negatively [25]. From $\mathtt{C2}$ to $\mathtt{C4}$, missed detection and false alarm increased to a similar extent on AMI, whereas the degradation on AliMeeting was dominated by missed detection. This indicates that over-tightening and insufficient tightening occur to a similar extent on AMI, while over-tightening is dominant on AliMeeting. As discussed in Sec. III-A, this can be attributed to the fixed threshold used in loose segment restoration, despite the utterance-dependent appropriate threshold. Specifically, since AMI contains more pause filling, a smaller threshold can be used. In contrast, since AliMeeting contains less pause filling, using a small threshold tends to make the model learn overly tight outputs.

Focusing on pause filling, the proposed method ($\mathtt{P1}$) improved the DER on average and substantially mitigated this asymmetry. On AMI, the method reduced false alarms while maintaining a similar level of missed detection. In contrast, on AliMeeting, it substantially reduced missed detection with a smaller accompanying increase in false alarms, indicating that over-tightening is effectively alleviated. This is also evident from the average onset and offset boundary shifts shown in Table V. Compared with $\mathtt{C2}$, the $\mathtt{C4}$ system showed a larger amount of tightening at onsets, indicating over-tightening. In contrast, all methods with pause-filling-focused tightening, i.e., $\mathtt{P1}$–$\mathtt{P3}$, mitigated such excessive boundary shifts compared with $\mathtt{C4}$, especially on AliMeeting. This behavior is consistent with the design of the proposed tightening, which suppresses pause filling without aggressively shifting loose onset and offset boundaries.

Introducing the burn-in phase to the causal and anticausal models further reduced missed detection, at the cost of a comparable increase in false alarms ($\mathtt{P2}$). To examine the source of this effect, Fig. 2 shows the position-wise DER within each chunk with and without the burn-in phase. Without the burn-in phase, misses increased by 2–3 percentage points near the beginning of the context available to each unidirectional model, i.e., $01\text{\,}\mathrm{s}$ for the causal model and $910\text{\,}\mathrm{s}$ for the anticausal model. With the burn-in phase, this degradation was substantially mitigated, resulting in approximately uniform DER across positions within chunks.

Finally, $\mathtt{P3}$ in Table III shows the result of non-causal-model-aware co-training, where the final non-causal model was trained in parallel with the co-training of the causal and anticausal models. Although the improvement is modest because this method does not directly improve the pseudo-label generation process itself, it reduced missed detection at the cost of increased false alarms.