SignRefine: Adapting Foundational Video Models for Sign Language Generation

Anton Pelykh Affiliation: Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

https://cogvis-cvssp.github.io/papers/signrefine/ E-mail {a.pelykh,edward.fish,o.mercanoglusincan,r.bowden}@surrey.ac.uk Edward Fish Affiliation: Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

https://cogvis-cvssp.github.io/papers/signrefine/ E-mail {a.pelykh,edward.fish,o.mercanoglusincan,r.bowden}@surrey.ac.uk Ozge Mercanoglu Sincan Affiliation: Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

https://cogvis-cvssp.github.io/papers/signrefine/ E-mail {a.pelykh,edward.fish,o.mercanoglusincan,r.bowden}@surrey.ac.uk Richard Bowden Affiliation: Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, UK

https://cogvis-cvssp.github.io/papers/signrefine/ E-mail {a.pelykh,edward.fish,o.mercanoglusincan,r.bowden}@surrey.ac.uk

Abstract

Sign language video generation demands precise hand and facial articulation, yet modern video diffusion models, trained predominantly on spoken-language video, produce artifacts that render signing unintelligible. We propose SignRefine, a sign language video generation model that produces comprehensible signing from 2D keypoint conditioning alone, generalizing across appearances and visual conditions. Our approach builds on a pretrained video diffusion transformer and introduces local adapters with spatial grounding to selectively refine hand and face regions, steering the strong base model’s prior toward accurate articulation. To enable this work and support broader sign language research, we present DATASET, a large-scale dataset of video content natively produced in sign language, offering diverse signer appearances, environments, and natural conversational settings. Trained on this data, our model shows up to 30% improvement in hand pose precision metrics over the strongest baseline and is preferred by sign language users for visual quality and comprehensibility in more than 80% of comparisons.

Keywords:

Video Generation Sign Language Diffusion Models

1 Introduction

Refer to caption — Figure 1: Qualitative comparison of sign language video generation approaches from 2D keypoint conditioning. From left to right: input keypoint condition, Our model, Wan2.1-1.3B-Fun-Control [60], Wan2.1-1.3B-VACE [23].

Sign languages are the primary means of communication for Deaf communities worldwide. Lacking a widely adopted written form, video is the natural medium for sign language content creation, accessibility, and communication. Consequently, there is a critical need for automatic sign language video generation. While modern Video Diffusion Models (VDMs) produce impressively realistic general-purpose video, they consistently fail to produce plausible sign language outputs (see Fig. 1). This is because the complex articulation of hands and nuanced facial expressions required for sign language are largely out-of-distribution for the training data of these models. Furthermore, hands and face, which carry the core linguistic meaning, are fine-grained structures occupying a small fraction of the frame. In the patchified latent space where VDMs operate, these regions are aggressively compressed spatially and their structural details are overwhelmed by the global reconstruction objective. As a result, standard full-body conditioning cannot recover this lost definition, resulting in corrupted or under-articulated hand and face regions.

Currently, the field of research faces a stark duality between specialized and generalized video generation. Existing sign language-specific video models [54, 45, 43, 50, 38, 11, 61] are typically trained from scratch on small, constrained datasets. While they can produce plausible results on in-distribution data, they overfit to specific signer appearances and fail to generalize to new individuals, scales, or visual environments, producing restrictive, low-resolution outputs. This is amplified by the prevalence of interpreted content in existing datasets, which lacks the spatio-temporal complexity and fluidity of natural signing. In contrast, foundational human video models [19, 72, 83, 68, 64, 65, 66, 12] showcase exceptional generalizability and visual quality but cannot handle the significant domain shift required for continuous signing. To date, no approach successfully resolves these issues to produce high-fidelity, generalizable sign language video that is actually comprehensible to Deaf viewers.

Modern sign language production approaches typically consist of two stages: (1) translating spoken language into a discrete intermediate representation such as skeletal data or latent motion tokens [76, 61, 44, 84], (2) generating videos using these representations [43, 45, 61]. In this work, we tackle the second part of the pipeline – generating anatomically precise and expressive sign language video across diverse appearances given the driving skeleton sequence. To achieve this, we adapt a powerful prior of a large-scale video Diffusion Transformer (DiT) [35] for sign language using our proposed local adapters with spatial grounding. Aligning with sign language linguistics, where manual and non-manual signals operate as distinct communicative channels, we isolate the conditioning signals for each region (face, left and right hands) and process them with separate spatial condition encoders at the increased resolution. To reinforce the spatial alignment of the extracted regional conditions, we also encode the locations of each region in the frame using the CoordConv approach [32]. The extracted regional features are then integrated into the DiT backbone with our local cross-attention adapters to selectively refine the quality of the hand and face areas. To spatially ground the refinements, we employ an attention masking strategy with learned sink tokens, ensuring that background queries are routed away from the foreground.

To address visual sterility and linguistic biases prevalent in existing sign language datasets, we introduce DATASET, a large-scale dataset of native sign language content. The dataset provides a vast diversity of signer appearances, dynamic visual environments, and camera framings. Beyond its utility for the current work in sign language video generation, the dataset’s significant volume of authentic, multi-participant conversation unlocks new research directions for the broader AI community, enabling the study of multi-signer dynamics, turn-taking, and complex non-manual discourse markers.

In summary, our contributions are as follows:

1.

We introduce the first generalizable pose-to-video generation approach designed to improve the comprehensibility of generated sign language content.
2.

We develop a novel conditioning scheme for DiT-based human video models using local adapters with spatial grounding. It enables targeted refinement of hand and face areas without degrading the quality and generalizability of the backbone model.
3.

We introduce DATASET, a large-scale multi-modal natively signed dataset.
4.

Through rigorous quantitative evaluation and a user study with sign language users of different levels, we demonstrate that our approach outperforms the baselines and is capable of generating comprehensible and diverse sign language videos.

2 Related Work

Video Generation Models The field of video generation progressed from GAN-based [13, 59, 7, 57, 42, 67, 53] and traditional Likelihood-based [24, 28, 73, 8, 18] approaches to Video Diffusion Models (VDMs), which have established a new performance frontier in sample quality and scalability. Early VDMs directly extended the 2D DDPM framework [16] to the time dimension in pixel space [17] and later in the latent space of a pre-trained autoencoder [3]. This shift to latent space drastically improved model efficiency and unlocked high-resolution generation. Instead of training from scratch, numerous works [14, 2, 3, 81, 52, 70, 63] expanded existing powerful image Latent Diffusion Models (LDM) [39] to the video domain, sparking a modern trend of adapting strong visual priors for downstream generative tasks.

Alongside generation quality, the research community has heavily focused on the controllability of VDMs. ControlNet-style adapters [78] have been widely adopted to drive video generation using additional visual modalities [80, 20, 31, 36, 62] and camera controls [75, 15]. Recently, the paradigm has shifted away from the traditional denoising U-Net architecture [41] toward DiT [35], which treats video latents as a sequence of spacetime patches and scales more predictably with data and compute. The latest DiT-based video models [4, 34, 74, 26, 56, 60] generate longer, highly consistent videos with complex interactions. Crucially, control mechanisms originally designed for U-Nets can also be successfully adapted to these DiT architectures.

Human Image Animation Advancements in LDMs directly catalyzed developments in human image animation. Several works [19, 72, 83, 68] utilize a frozen LDM as a feature extractor for a reference image. These features are then injected into the backbone video model to produce an animated human video. Other approaches [64, 65] employ distinct encoding and fusion mechanisms for reference images and driving keypoint sequences before passing them into the backbone diffusion model. Progressing alongside foundation models, recent frameworks [66, 12, 6] have adopted DiTs as their backbone. However, while these models excel at general human movement and global pose alignment, they struggle to reliably produce highly articulated hand shapes and nuanced facial expressions crucial for complex communication in sign languages.

Sign Language Video Generation Given the unique anatomical demands of continuous signing, the generation of sign language videos has historically developed as a specialized domain. For several years, pose-conditioned GAN approaches [54, 45, 43] served as the foundational standard for this task, successfully establishing the viability of generating continuous sign language videos. While pioneering, these early models typically overfit to specific appearances, lacked flexible visual conditioning, and struggled to generalise across varying body scales and rotations. Recent works [50, 38, 11, 61] have adopted diffusion models to generate more temporally consistent and expressive sign videos. However, because these models are trained from scratch on small domain-specific datasets, they produce low-resolution outputs and suffer from poor generalisation. Furthermore, some approaches [11, 61] rely on additional dense modalities such as edge maps or posed 3D hand meshes, limiting their practical applicability. In contrast, we adapt a large-scale pretrained video diffusion model, inheriting its strong visual priors and generalisability. Moreover, our method is conditioned on sparse skeleton keypoints, avoiding dense structural modalities to ensure broad real-world utility.

3 DATASET Dataset

Table 1: DATASET is a large-scale sign language dataset containing extensive conversational data over multiple dynamic camera views and signers. The data covers a large number of environments, contexts, and topics, opening new research directions for the community. Conv.: dataset contains multi-party conversational signing. Dynamic Camera: footage involves moving or switching camera angles.

Dataset Lang Source Env Signer Level Conv. Dynamic Camera #Signers #Hours Text Vocab Phoenix-2014T [5] DGS TV Studio Interpreter ✗ ✗ 9 11 3K How2Sign [9] ASL Lab Studio Interp./Native ✗ ✗ 11 79 16K OpenASL [49] ASL Web Varied N/A ✗ ✗ 220 288 33K YouTube-ASL [58] ASL Web Varied N/A ✗ ✗ 2519 984 60K YouTube-SL-25 [55] Varied Web Varied N/A ✗ ✗ 3000 3207 – CSL-News [30] CSL TV Studio Interpreter ✗ ✗ – 1985 5K BOBSL [1] BSL TV Studio Interpreter ✗ ✗ 39 1467 77K CSL-Daily [82] CSL Lab Studio Native ✗ ✗ 10 23 2K BSL Corpus [47] BSL Lab Studio Native ✓ ✗ 249 15^† – MeineDGS [27] DGS Lab Studio Native ✓ ✗ 330 50 18K DATASET BSL Web Varied Native ✓ ✓ 2184 78.1 12K

Progress in sign language modeling is fundamentally constrained by the quality of available data. Existing sign language datasets each leave distinct gaps for training generalisable models. Broadcast resources [5, 1, 30] offer substantial volume, but feature interpreter signing in static environments, carrying structural biases and artifacts from the source spoken language. Web-scraped collections [55, 58] provide scale and visual variety, but lack verified signer profiles, linguistic quality control, and conversational interaction. Lab-recorded datasets [46, 27, 9, 82] include rich linguistic content but are confined to fixed setups. Although some feature conversational interaction [46, 27], they remain visually sterile, failing to prepare models for real-world complexity.

To address these limitations, we introduce DATASET, the first large-scale dataset of content natively produced in sign language. Created by Deaf signers, the dataset captures sign language in its most authentic, natural form. DATASET consists of 44,759 atomic signing segments, sourced from 317 longer videos. This amounts to 78.1 hours of high-resolution video featuring an estimated 2,184 unique signers. Table. 1 compares DATASET to other sign language datasets. A defining feature of DATASET is its significant volume of conversational scenes involving two or more participants (approximately 40% of the dataset). This natural interactivity, lacking in previous datasets, unlocks critical research directions that are impossible to explore with single-signer corpora. They include the analysis of multi-signer dynamics, turn-taking, signer diarisation, and the use of non-manual features as backchanneling and discourse markers. By providing authentic multi-party conversations, DATASET pushes the field past translating isolated sentences towards natural sign language translation and expressive, realistic production. To accelerate this research, we provide a comprehensive set of multi-modal annotations. For each video, we provide meta data including time-aligned English transcripts, extracted 2D human body keypoints, per-signer activity scores, depth maps, SMPL [33] body, MANO [40] hand, and FLAME [29] face parameters. We also provide three data partitions that navigate different trade-offs between signer overlap and content diversity to allow for nuanced model evaluation. A more detailed description of the dataset construction is provided in the supplementary material.

4 Methodology

Given a reference image defining signer appearance and a keypoint sequence specifying sign language motion, our goal is to generate a video depicting the signer faithfully reproducing the motion with precise hand articulations and facial expressions. Our system is built on Wan-Fun-Control-1.3B [60], a pretrained video DiT conditioned on skeleton sequences, selected for its strong generation quality and accessible size. While the full-body skeleton defines global body movements effectively, the model lacks precision for hands and face (regions that are spatially small yet linguistically critical). We introduce trainable local adapters that extract these regional conditions at higher resolution, encode them into compact motion features, and inject them into selected DiT blocks via masked cross-attention. The backbone remains frozen, preserving its generalised video priors while the adapters learn to steer the generation toward faithful and precise articulation. The overall architecture is illustrated in Fig. 2.

4.1 Regional Conditions with Spatial Grounding

Signed communication happens through two primary articulatory channels: manual (hands) and non-manual (face, head and upper body). While these channels work cohesively to convey linguistic meaning, they present vastly different morphological features. The face requires modeling subtle, continuous deformations such as lip shapes and eyebrow movements, whereas the hands involve highly articulated, high-frequency structural changes. To account for this, we process the face, left hand, and right hand through three independent condition encoders, allowing each pathway to specialize its feature representation. Each encoder processes a regional keypoint sequence rendered as a $256\times 256$ video into a sequence of motion tokens. A ResNet-style backbone first downsamples the spatial dimensions to a $16\times 16$ feature grid. To capture motion dynamics, a pair of causal 1D convolutions then compresses the temporal dimension from T frames to T/4. This specific temporal compression aligns the regional condition with the temporal resolution of the backbone’s latent space, producing motion vectors $\mathbf{M}\in\mathbb{R}^{T/4\times S\times d}$ per region, where $S$ is the number of spatial latent patches and $d$ the hidden dimension size.

A fundamental challenge in this regional approach is spatial ambiguity: once extracted, a local crop loses the spatial context of its original location within the video. Because the adapter must project these features back into the correct location of the full frame, the encoder needs global positional awareness. We resolve this by injecting positional information into the input channels of the crop, inspired by the CoordConv approach [32]. For every pixel $(i,j)$ in the crop, we compute its absolute normalized coordinate relative to the original full frame:

x_{i,j}=\frac{2(x_{1}+(x_{2}-x_{1})\cdot j/W)}{W_{\text{orig}}}-1,\quad y_{i,j}=\frac{2(y_{1}+(y_{2}-y_{1})\cdot i/H)}{H_{\text{orig}}}-1,

(1)

where $(x_{1},y_{1},x_{2},y_{2})$ are the bounding box corners in pixel coordinates and $W_{\text{orig}},H_{\text{orig}}$ are the original frame dimensions. This maps each crop pixel to its position in the full frame, normalized to $[-1,1]$ . By providing these dynamic coordinate maps alongside the RGB channels, we directly embed the crop-to-frame geometric mapping into the regional features.

4.2 Localized Feature Injection

The extracted motion vectors must be integrated into the DiT backbone to provide precise and localized control over the articulators without corrupting the background. We achieve this through cross-attention blocks injected at regular intervals across the DiT. The backbone consists of 30 transformer blocks, the face adapter injects cross-attention at blocks $\{0,6,12,18,24\}$ and the hand adapter at blocks $\{2,8,14,20,26\}$ . This offset between face and hand injection prevents any single DiT block from receiving multiple adapter residuals simultaneously, distributing the conditioning load across the network. Furthermore, the left and right hand conditions are concatenated before injection to better resolve inter-hand dependencies and occlusions.

Because the DiT operates in a compressed latent space, each token in the hidden state $\mathbf{x}\in\mathbb{R}^{S\times d}$ maintains a predictable spatial correspondence to a specific area in the original pixel space. This allows us to map the regional bounding box coordinates onto the coarse latent grid, producing a binary spatial mask $\mathbf{m}\in\{0,1\}^{S}$ that separates the target articulator (foreground) from the rest of the frame (background). To reinforce this spatial isolation during feature injection, we append a trainable sink token $\mathbf{s}\in\mathbb{R}^{1\times d}$ to the sequence of motion vectors $\mathbf{M}$ . We compute queries from the DiT hidden states and keys/values from the extended motion sequence:

\mathbf{Q}=W_{Q}\cdot\text{LN}(\mathbf{x}),\quad\mathbf{K},\mathbf{V}=W_{KV}\cdot\text{LN}([\mathbf{M};\mathbf{s}])

(2)

$W_{Q}$ and $W_{KV}$ are the learned linear projection weight matrices for the queries, keys and values, respectively. $\text{LN}(\cdot)$ denotes the Layer Normalization operation. Attention is then routed using the spatial mask:

A_{ij}=\begin{cases}\text{softmax}(\mathbf{q}_{i}\cdot\mathbf{k}_{j}/\sqrt{d})&\text{if }m_{i}=1\text{ and }j\leq N\text{ (foreground}\to\text{content)}\\ \text{softmax}(\mathbf{q}_{i}\cdot\mathbf{k}_{\text{sink}}/\sqrt{d})&\text{if }m_{i}=0\text{ (background}\to\text{sink)}\end{cases},

(3)

where $A_{ij}$ is the computed attention weight mapping the $i$ -th query patch $\mathbf{q}_{i}$ to the $j$ -th key token $\mathbf{k}_{j}$ ; $N=|\mathbf{M}|$ is the length of the motion-vector sequence, so the keys at indices $j\leq N$ are the projected motion tokens, and the key at index $N{+}1$ , denoted $\mathbf{k}_{\text{sink}}$ , is the projection of the appended sink token $\mathbf{s}$ ; $m_{i}\in\{0,1\}$ represents the binary spatial mask value for the $i$ -th latent patch: $m_{i}=1$ indicates the patch falls inside the articulator’s bounding box (foreground), and $m_{i}=0$ indicates its belonging to the background; $\sqrt{d}$ is the scaling factor to stabilize gradients.

This mechanism ensures that the foreground latent patches attend exclusively to the high-resolution regional features, while the background patches are routed to the sink token. The sink token acts as a null target, safely absorbing background queries and preventing the adapter from bleeding hand or face features into unrelated areas.

4.3 Adapter Training

The output projection layers of the regional adapters are initialized with near-zero values [78] to ensure training stability on top of the frozen backbone. The training objective is the mean squared error (MSE) on the predicted noise with spatially non-uniform weighting that emphasizes the regions the adapter is designed to improve. Using the same bounding box masks available from the conditioning pipeline, we partition each frame’s latents into three regions, hands, face, and background, and combine their mean errors into a single objective:

\mathcal{L}=\frac{w_{h}N_{h}\mathcal{L}_{h}+w_{f}N_{f}\mathcal{L}_{f}+w_{b}N_{b}\mathcal{L}_{b}}{w_{h}N_{h}+w_{f}N_{f}+w_{b}N_{b}},

(4)

where $\mathcal{L}_{h},\mathcal{L}_{f},\mathcal{L}_{b}$ are the MSE within the hand, face, and background regions, $N_{h},N_{f},N_{b}$ are the numbers of latent elements they contain, and $w_{h},w_{f},w_{b}$ are their weights. This is equivalent to assigning every latent element the weight of its region and taking a single per-element weighted average, so each region’s influence on the gradient is proportional to both its weight and its area. The background region is defined as the complement of the hand and face masks, so the body (torso and arms outside the articulator boxes) is absorbed into the background term rather than receiving a dedicated weight. In overlapping regions, hands take priority over face, reflecting their common relative position in sign language.

We set $w_{h}=w_{f}=10,w_{b}=1$ in our experiments, motivated by the area imbalance. Although the hands and face regions are critical for sign comprehension, at the latent resolution they occupy only $\sim$ 15% of each frame on average, so under uniform weighting they would receive a correspondingly small share of the training gradient. The chosen weights raise the articulators’ share of the gradient, prioritizing them, while retaining roughly a third of the gradient on the background to preserve global coherence such as body pose and identity.

5 Experiments

5.1 Implementation Details

We use Wan2.1-1.3B-Fun-Control [60] DiT as the frozen backbone, which includes a T5 text encoder, a CLIP image encoder for reference image conditioning, and a 3D VAE for encoding/decoding video latents. As varying the text prompt has a negligible effect on generation quality, we cache the latents for a generic text prompt and load them once to improve training efficiency.

Adapter architecture Each regional condition encoder uses a ResNet-style convolutional backbone with GroupNorm [71] and SiLU [10] activations, downsampling the $256\times 256$ input video into a $16\times 16$ spatial grid. Temporal downsampling is performed by a stack of causal 1D convolutions, compressing $T$ frames to $T/4$ . The final linear layer projects the features from 512 to 1536 dimensions, which matches the hidden size of the Wan backbone. There are two separate adapters, one for face and one for hand regions, each including 6 cross-attention blocks. Each block uses 12 heads with head dimension 128 and RMSNorm [77] on queries and keys.

Training We train the model on $\sim 40,000$ video clips from the DATASET train partition for 30,000 iterations. We use Adam optimizer [25] with learning rate $1e-4$ , gradient accumulation over 8 iterations, the batch size of 1 per-GPU. The model is trained on 4 NVIDIA GH200 GPUs in mixed precision mode.

5.2 Quantitative Evaluation

The evaluation is performed on the DATASET appearance-independent test set, PHOENIX [5] and BSLCP [47] to assess the generalization of the models. We randomly choose 100 videos from each dataset to keep the inference time manageable. As the main goal of this work is to improve the quality of the hand and face regions, we concentrate our evaluation on regional metrics. We use LPIPS [79] with the VGG [51] backbone and SSIM [69] to assess perceptual quality. MPJPE [21] is chosen as a measure of keypoint accuracy and structural precision. For its evaluation, the 133 full-body keypoints from both ground-truth (GT) and generated videos are extracted with RTMPose-X [22]. All regional crops are extracted using the bounding boxes derived from the GT keypoints.

We compare our approach with several state-of-the-art models in video generation. Wan2.1-1.3B-Fun-Control (Wan-FC) and Wan2.1-1.3B-VACE (Wan-VACE) [23] are general DiT video generation models that are driven with a full-body skeleton condition. SignViP [61] is a multi-step video model, specifically trained on sign language data. It encodes the full body skeleton and 3D hand mesh renders as latent tokens, which are then used to condition the video generator. For a fair evaluation, we construct the tokens from the conditions extracted from the GT videos. We use the model checkpoint trained on PHOENIX as it is the only checkpoint made available by the authors. As there is no open-source implementation of SignGAN [43], we use the implementation and the checkpoint obtained directly from the authors. The obtained model was trained on one fixed appearance and does not accept visual conditioning; therefore, the results will not follow the GT visually. However, as we perform the evaluation on regional crops, the influence of the overall appearance is reduced, and the comparison to other models is fair.

The results for all datasets are presented in Table 2 and Fig. 3 shows qualitative examples generated by different models. Our method consistently achieves the strongest performance across all three anatomical regions and all three datasets, with particularly pronounced gains on hand regions. The improvement over the strongest baseline, Wan-FC, is most evident in hand fidelity on DATASET, where our model shows roughly a 30% reduction in MPJPE. Face precision also improves consistently across datasets, by up to 20% on PHOENIX and 15% on BSLCP. The Wan-VACE model, as well as sign-specific SignViP and SignGAN, consistently underperform in all testing scenarios.

A notable pattern emerges when examining the generalization behavior of the prior methods. SignViP, trained exclusively on PHOENIX, suffers a severe performance collapse on DATASET and BSLCP, with hand MPJPE roughly doubling compared to its in-domain results. This indicates that its learned representations are tightly coupled to the visual statistics of its training corpus. Similarly, SignGAN, trained on controlled studio recordings, exhibits the weakest overall performance, particularly on the DATASET data, underscoring the well-known domain gap between laboratory and in-the-wild signing conditions. In contrast, our method maintains stable performance across all datasets without any dataset-specific adaptation, demonstrating that the proposed architecture generalizes robustly across diverse signing environments and visual domains.

Table 2: Quantitative comparison on DATASET (top), PHOENIX (middle) and BSLCP (bottom) datasets. Best results are bolded, and second-best are underlined.

Method Face Right Hand (RH) Left Hand (LH) MPJPE $\downarrow$ LPIPS $\downarrow$ SSIM $\uparrow$ MPJPE $\downarrow$ LPIPS $\downarrow$ SSIM $\uparrow$ MPJPE $\downarrow$ LPIPS $\downarrow$ SSIM $\uparrow$ DATASET Dataset SignGAN [43] 10.424 0.564 0.302 24.774 0.627 0.388 21.852 0.620 0.384 SignViP [61] 10.576 0.534 0.356 23.957 0.581 0.428 24.383 0.588 0.424 Wan-VACE [23] 4.067 0.422 0.367 15.904 0.525 0.380 15.431 0.503 0.407 Wan-FC [60] 1.835 0.197 0.713 9.565 0.364 0.583 9.341 0.349 0.597 Ours 1.770 0.194 0.714 6.602 0.330 0.629 6.638 0.321 0.640 PHOENIX Dataset SignGAN [43] 5.457 0.585 0.343 28.507 0.707 0.469 25.373 0.657 0.470 SignViP [61] 7.194 0.456 0.276 10.345 0.534 0.425 14.405 0.566 0.376 Wan-VACE [23] 2.162 0.383 0.413 11.717 0.551 0.367 13.214 0.546 0.299 Wan-FC [60] 1.151 0.223 0.688 5.754 0.402 0.629 5.719 0.374 0.642 Ours 0.916 0.214 0.701 4.582 0.386 0.660 5.101 0.366 0.668 BSLCP Dataset SignGAN [43] 9.415 0.634 0.330 22.726 0.720 0.406 16.858 0.689 0.390 SignViP [61] 28.222 0.626 0.289 23.085 0.611 0.381 22.111 0.593 0.372 Wan-VACE [23] 3.172 0.436 0.380 17.524 0.567 0.351 15.385 0.557 0.338 Wan-FC [60] 1.790 0.221 0.685 8.232 0.389 0.559 7.002 0.373 0.563 Ours 1.524 0.222 0.690 6.314 0.365 0.608 5.406 0.353 0.607

5.3 User Study

Protocol To strengthen the quality and understandability evaluation of the models, we conduct a perceptual study with 15 participants of varying British Sign Language (BSL) proficiency, from basic to advanced. Each participant completed 17 questions across two question types. In the first, Visual Ranking, participants viewed four videos generated by different models from the same conditioning input and ranked them by naturalness of signing and quality of hand and facial features. These questions include a combination of samples from DATASET and PHOENIX. Although BSL signers can not directly evaluate the linguistic accuracy of DGS (German Sign Language) in PHOENIX videos, they can still assess and compare their visual fidelity. The second type, Text Reference Ranking, is only shown to participants with intermediate or higher BSL proficiency. Given a reference English text sentence, participants are asked to rank four BSL video renditions by semantic fidelity. To ensure a fair comparison among five models using a four-slot display, each question excludes one model according to a balanced rotation schedule that guarantees approximately equal representation. Model-to-label assignments per question and the question order are randomised to prevent positional bias. We analyse the collected preferences through three complementary metrics: (1) Mean Rank aggregates each model’s average position across all participant-question instances, with pairwise significance against our method assessed via Wilcoxon signed-rank tests on co-occurring model pairs, corrected for multiple comparisons using the Holm-Bonferroni procedure; (2) Rank-1 Win Rate measures how often each model was placed first, reported with the Wilson score confidence intervals against a 25% chance baseline; (3) Head-to-Head Preference matrix captures the proportion of co-occurrences in which each model was preferred over every other, indicating detailed dominance relationships beyond aggregate rank.

Table 3: Perceptual Quality study results. Mean rank assigned by participants to each model (1 = best, 5 = worst), reported overall and per evaluation category.

5.4 Ablation Study

Table 5: Ablation study of different adapter configurations and components.

Method	Face		LH (Left Hand)		RH (Right Hand)
Method	MPJPE $\downarrow$	LPIPS $\downarrow$	MPJPE $\downarrow$	LPIPS $\downarrow$	MPJPE $\downarrow$	LPIPS $\downarrow$
Dense conditions	1.692	0.192	6.665	0.319	6.713	0.329
Sparse conditions	1.770	0.194	6.638	0.321	6.602	0.330
Sparse w/o attention masking	1.949	0.214	7.309	0.353	7.270	0.363
Sparse w/o regional coords	1.847	0.202	6.926	0.335	6.888	0.344

We conduct an ablation study on some of the key components of our proposed adapters, which include regional attention masking and CoordConv coordinates. We also experiment with providing dense conditions to the adapters, rendered face normals and 3D MANO meshes, instead of regional keypoint renders. The results are provided in Table 5. Removing masked attention degrades all metrics, with face MPJPE increasing from 1.770 to 1.949 (+10.1%) and hand MPJPE rising by a similar margin (e.g. LH from 6.638 to 7.309). Masked attention focuses the cross-attention on relevant spatial regions, preventing the adapter from concentrating on uninformative background tokens. Removing the CoordConv input encoding yields a smaller but consistent degradation across all metrics (+4.3%), confirming that the absolute spatial position helps the model disambiguate between regions.

When comparing sparse and dense conditions, the dense adapter achieves marginally better face metrics, possibly owing to the richer geometric detail in rendered face normals, but yields no improvement for hands. We attribute this to two factors. Firstly, our region adapters get conditions in $256\times 256$ resolution, which significantly exceeds the area the hands typically occupy in original $832\times 480$ frames. We find that this magnification, rather than condition density, is the primary driver of hand quality: at this resolution, sparse keypoints already provide sufficient structural guidance. Secondly, the pretrained 1.3B-parameter DiT backbone encodes a strong prior over hand appearance and articulation, allowing it to synthesize plausible details from coarse positional cues alone. The additional surface geometry in MANO renders seems largely redundant in this case. Furthermore, sparse keypoints more closely resemble the skeleton-based control signals the backbone was designed to process and are less susceptible to the structured noise present in predicted MANO meshes. Given comparable hand performance, we adopt the sparse adapter as our default to eliminate the inference-time dependency on MANO and face normal prediction and rendering.

6 Limitations

Although our proposed approach produces impressive visual results, it inherits several limitations. Firstly, video diffusion models remain computationally expensive: generating an 81-frame clip takes minutes even on high-end GPUs, precluding real-time applications. Secondly, our model reproduces the motion blur present in the training data, which is typically filmed at conventional frame rates (25 fps). Fast movements are prevalent in sign languages, that results in the model occasionally smearing fine hand details during rapid transitions. Finally, both our conditioning pipeline and keypoint-based evaluation rely on off-the-shelf pose estimators such as RTMPose, WiLoR [37] etc., whose errors propagate into the model. Noisy keypoints in the input condition can mislead generation and reduce the quality. Additionally, inaccuracies in the predicted keypoints frames add noise to the reference-based evaluation metrics, potentially obscuring the true performance differences between methods.

7 Conclusion

In this work, we presented SignRefine, a video generation model that adapts a pretrained Diffusion Transformer for sign language via local adapters with spatial grounding. By injecting the high-resolution regional conditions of the hands and face into the frozen backbone through masked cross-attention, our approach corrects the articulation failures of general-purpose video models without sacrificing their quality and generalisability. Furthermore, we introduced DATASET, a large-scale natively-signed high resolution dataset, providing diverse signer appearances, dynamic visual conditions, and multi-party conversational scenes absent from all prior corpora. SignRefine achieves substantial improvement in hand pose precision over the strongest baseline and is preferred by sign language users in over 80% of pairwise comparisons. This work demonstrates that foundational video models can be effectively used for sign language, opening new possibilities for high-fidelity, generalizable sign language content creation.

Acknowledgements

This work was supported by EPSRC grant APP24554 (SignGPT-EP/Z535370/1), EPSRC grant APP78083 (UMCS UKRI3927) and through funding from Google.org via the AI for Global Goals scheme. The authors acknowledge the use of Isambard-AI National AI Research Resource (AIRR) funded by UK DSIT via UKRI and STFC [ST/AIRR/I-A-I/1023]. This work reflects only the authors’ views and the funders are not responsible for any use that may be made of the information it contains.

References

[1] S. Albanie, G. Varol, L. Momeni, H. Bull, T. Afouras, H. Chowdhury, N. Fox, B. Woll, R. Cooper, A. McParland, et al. (2021) Bbc-oxford british sign language dataset. arXiv preprint arXiv:2111.03635. Cited by: Table 1, §3.
[2] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, V. Jampani, and R. Rombach (2023) Stable video diffusion: scaling latent video diffusion models to large datasets. CoRR abs/2311.15127. External Links: Link Cited by: §2.
[3] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis (2023) Align your latents: high-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563–22575. Cited by: §2.
[4] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh (2024) Video generation models as world simulators. Note: Accessed: 29 June 2026 External Links: Link Cited by: §2.
[5] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden (2018) Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7784–7793. Cited by: Table 1, §3, §5.2.
[6] G. Cheng, X. Gao, L. Hu, S. Hu, M. Huang, C. Ji, J. Li, D. Meng, J. Qi, P. Qiao, et al. (2025) Wan-animate: unified character animation and replacement with holistic replication. arXiv preprint arXiv:2509.14055. Cited by: §2.
[7] A. Clark, J. Donahue, and K. Simonyan (2019) Adversarial video generation on complex datasets. arXiv preprint arXiv:1907.06571. Cited by: §2.
[8] E. Denton and R. Fergus (2018) Stochastic video generation with a learned prior. In International conference on machine learning, pp. 1174–1183. Cited by: §2.
[9] A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i-Nieto (2021) How2sign: a large-scale multimodal dataset for continuous american sign language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2735–2744. Cited by: Table 1, §3.
[10] S. Elfwing, E. Uchibe, and K. Doya (2018) Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, pp. 3–11. Cited by: §5.1.
[11] S. Fang, C. Sui, Y. Zhou, X. Zhang, H. Zhong, Y. Tian, and C. Chen (2025) SignDiff: diffusion model for american sign language production. In 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–11. Cited by: §1, §2.
[12] Q. Gan, Y. Ren, C. Zhang, Z. Ye, P. Xie, X. Yin, Z. Yuan, B. Peng, and J. Zhu (2025) Humandit: pose-guided diffusion transformer for long-form human motion video generation. arXiv preprint arXiv:2502.04847. Cited by: §1, §2.
[13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2020) Generative adversarial networks. Communications of the ACM 63 (11), pp. 139–144. Cited by: §2.
[14] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai (2024) AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.
[15] H. He, Y. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang (2024) CameraCtrl: enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101. Cited by: §2.
[16] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, pp. 6840–6851. Cited by: §2.
[17] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet (2022) Video diffusion models. Advances in neural information processing systems 35, pp. 8633–8646. Cited by: §2.
[18] W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang (2023) CogVideo: large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.
[19] L. Hu (2024) Animate anyone: consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8153–8163. Cited by: §1, §2.
[20] Z. Hu and D. Xu (2023) Videocontrolnet: a motion-guided video-to-video translation framework by using diffusion model with controlnet. arXiv preprint arXiv:2307.14073. Cited by: §2.
[21] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu (2013) Human3. 6m: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence 36 (7), pp. 1325–1339. Cited by: §5.2.
[22] T. Jiang, P. Lu, L. Zhang, N. Ma, R. Han, C. Lyu, Y. Li, and K. Chen (2023) Rtmpose: real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399. Cited by: §5.2.
[23] Z. Jiang, Z. Han, C. Mao, J. Zhang, Y. Pan, and Y. Liu (2025) Vace: all-in-one video creation and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17191–17202. Cited by: Figure 1, Figure 1, §5.2, Table 2, Table 2, Table 2.
[24] N. Kalchbrenner, A. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu (2017) Video pixel networks. In International conference on machine learning, pp. 1771–1779. Cited by: §2.
[25] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §5.1.
[26] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024) Hunyuanvideo: a systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. Cited by: §2.
[27] R. Konrad, T. Hanke, G. Langer, D. Blanck, J. Bleicken, I. Hofmann, O. Jeziorski, L. König, S. König, R. Nishio, A. Regen, U. Salden, S. Wagner, S. Worseck, O. Böse, E. Jahn, and M. Schulder (2020) MEINE dgs – annotiert. Öffentliches korpus der deutschen gebärdensprache, 3. release / my dgs – annotated. public corpus of german sign language, 3rd release. languageresource, Universität Hamburg. External Links: Link, Document Cited by: Table 1, §3.
[28] M. Kumar, M. Babaeizadeh, D. Erhan, C. Finn, S. Levine, L. Dinh, and D. Kingma (2020) VideoFlow: a conditional flow-based model for stochastic video generation. In International Conference on Learning Representations, External Links: Link Cited by: §2.
[29] T. Li, T. Bolkart, Michael. J. Black, H. Li, and J. Romero (2017) Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36 (6), pp. 194:1–194:17. External Links: Link Cited by: §3.
[30] Z. Li, W. Zhou, W. Zhao, K. Wu, H. Hu, and H. Li (2025) Uni-sign: toward unified sign language understanding at scale. In The Thirteenth International Conference on Learning Representations, Cited by: Table 1, §3.
[31] H. Lin, J. Cho, A. Zala, and M. Bansal (2025) Ctrl-adapter: an efficient and versatile framework for adapting diverse controls to any diffusion model. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2.
[32] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Yosinski (2018) An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems 31. Cited by: §1, §4.1.
[33] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black (2015) SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34 (6). External Links: ISSN 0730-0301, Link, Document Cited by: §3.
[34] X. Ma, Y. Wang, X. Chen, G. Jia, Z. Liu, Y. Li, C. Chen, and Y. Qiao (2025) Latte: latent diffusion transformer for video generation. Transactions on Machine Learning Research. Cited by: §2.
[35] W. Peebles and S. Xie (2023) Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195–4205. Cited by: §1, §2.
[36] B. Peng, J. Wang, Y. Zhang, W. Li, M. Yang, and J. Jia (2024) ControlNeXt: powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06070. Cited by: §2.
[37] R. A. Potamias, J. Zhang, J. Deng, and S. Zafeiriou (2025) Wilor: end-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12242–12254. Cited by: §6.
[38] F. Qi, Y. Duan, H. Zhang, and C. Xu (2024) Signgen: end-to-end sign language video generation with latent diffusion. In European Conference on Computer Vision, pp. 252–270. Cited by: §1, §2.
[39] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695. Cited by: §2.
[40] J. Romero, D. Tzionas, and M. J. Black (2017) Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph. 36 (6). External Links: ISSN 0730-0301, Link, Document Cited by: §3.
[41] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234–241. Cited by: §2.
[42] M. Saito, E. Matsumoto, and S. Saito (2017) Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pp. 2830–2839. Cited by: §2.
[43] B. Saunders, N. C. Camgoz, and R. Bowden (2020) Everybody sign now: translating spoken language to photo realistic sign language video. arXiv preprint arXiv:2011.09846. Cited by: §1, §1, §2, §5.2, Table 2, Table 2, Table 2.
[44] B. Saunders, N. C. Camgoz, and R. Bowden (2020) Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision, pp. 687–705. Cited by: §1.
[45] B. Saunders, N. C. Camgoz, and R. Bowden (2022) Signing at scale: learning to co-articulate signs for large-scale photo-realistic sign language production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5141–5151. Cited by: §1, §1, §2.
[46] A. C. Schembri, J. B. Fenlon, R. Rentelis, S. Reynolds, and K. Cormier (2013) Building the british sign language corpus. Language Documentation & Conservation 7, pp. 136–154. External Links: Link Cited by: §3.
[47] A. Schembri, J. Fenlon, R. Rentelis, S. Reynolds, and K. Cormier (2013) Building the british sign language corpus. Language Documentation and Conservation 7, pp. 136–154 (English). External Links: ISSN 1934-5275 Cited by: Table 1, §5.2.
[48] T. Sellam, D. Das, and A. Parikh (2020) BLEURT: learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 7881–7892. External Links: Link, Document Cited by: §5.3.
[49] B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu (2022) Open-domain sign language translation learned from online video. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6365–6379. Cited by: Table 1.
[50] T. Shi, L. Hu, F. Shang, J. Feng, P. Liu, and W. Feng (2024) Pose-guided fine-grained sign language video generation. In European Conference on Computer Vision, pp. 392–409. Cited by: §1, §2.
[51] K. Simonyan and A. Zisserman (2015) Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, Cited by: §5.2.
[52] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y. Taigman (2023) Make-a-video: text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2.
[53] I. Skorokhodov, S. Tulyakov, and M. Elhoseiny (2022) Stylegan-v: a continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3626–3636. Cited by: §2.
[54] S. Stoll, N. C. Camgöz, S. Hadfield, and R. Bowden (2018) Sign language production using neural machine translation and generative adversarial networks. In Proceedings of the 29th British Machine Vision Conference (BMVC 2018), Cited by: §1, §2.
[55] G. Tanzer and B. Zhang (2025) YouTube-sl-25: a large-scale, open-domain multilingual sign language parallel corpus. In The Thirteenth International Conference on Learning Representations, Cited by: Table 1, §3, §5.3.
[56] G. Team (2024) Mochi 1. GitHub. Note: Accessed: 29 June 2026https://github.com/genmoai/models Cited by: §2.
[57] S. Tulyakov, M. Liu, X. Yang, and J. Kautz (2018) Mocogan: decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526–1535. Cited by: §2.
[58] D. Uthus, G. Tanzer, and M. Georg (2023) Youtube-asl: a large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems 36, pp. 29029–29047. Cited by: Table 1, §3.
[59] C. Vondrick, H. Pirsiavash, and A. Torralba (2016) Generating videos with scene dynamics. Advances in neural information processing systems 29. Cited by: §2.
[60] A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. (2025) Wan: open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314 3 (4), pp. 6. Cited by: Figure 1, Figure 1, §2, §4, §5.1, Table 2, Table 2, Table 2.
[61] C. Wang, Z. Deng, Z. Jiang, Y. Yin, F. Shen, Z. Cheng, S. Ge, S. Gan, and Q. Gu (2025) Advanced sign language video generation with compressed and quantized multi-condition tokenization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §1, §2, §5.2, Table 2, Table 2, Table 2.
[62] C. Wang, J. Gu, P. Hu, H. Zhao, Y. Guo, J. Han, H. Xu, and X. Liang (2024) Easycontrol: transfer controlnet to video diffusion for controllable generation and interpolation. arXiv preprint arXiv:2408.13005. Cited by: §2.
[63] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang (2023) Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571. Cited by: §2.
[64] T. Wang, L. Li, K. Lin, Y. Zhai, C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang (2024) Disco: disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9326–9336. Cited by: §1, §2.
[65] X. Wang, S. Zhang, C. Gao, J. Wang, X. Zhou, Y. Zhang, L. Yan, and N. Sang (2025) Unianimate: taming unified video diffusion models for consistent human image animation. Science China Information Sciences 68 (10), pp. 200103. Cited by: §1, §2.
[66] X. Wang, S. Zhang, L. Tang, Y. Zhang, C. Gao, Y. Wang, and N. Sang (2025) UniAnimate-dit: human image animation with large-scale video diffusion transformer. ArXiv abs/2504.11289. External Links: Link Cited by: §1, §2.
[67] Y. Wang, P. Bilinski, F. Bremond, and A. Dantcheva (2020) G3AN: disentangling appearance and motion for video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5264–5273. Cited by: §2.
[68] Z. Wang, Y. Li, Y. Zeng, Y. Guo, D. Lin, T. Xue, and B. Dai (2025) Multi-identity human image animation with structural video diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11937–11947. Cited by: §1, §2.
[69] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), pp. 600–612. Cited by: §5.2.
[70] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou (2023) Tune-a-video: one-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7623–7633. Cited by: §2.
[71] Y. Wu and K. He (2018) Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3–19. Cited by: §5.1.
[72] Z. Xu, J. Zhang, J. H. Liew, H. Yan, J. Liu, C. Zhang, J. Feng, and M. Z. Shou (2024) Magicanimate: temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1481–1490. Cited by: §1, §2.
[73] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas (2021) Videogpt: video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157. Cited by: §2.
[74] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2025) Cogvideox: text-to-video diffusion models with an expert transformer. In International Conference on Learning Representations, Vol. 2025, pp. 83048–83077. Cited by: §2.
[75] S. Yin, C. Wu, J. Liang, J. Shi, H. Li, G. Ming, and N. Duan (2023) Dragnuwa: fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089. Cited by: §2.
[76] Z. Yu, S. Huang, Y. Cheng, and T. Birdal (2024) Signavatars: a large-scale 3d sign language holistic motion dataset and benchmark. In European Conference on Computer Vision, pp. 1–19. Cited by: §1.
[77] B. Zhang and R. Sennrich (2019) Root mean square layer normalization. Advances in neural information processing systems 32. Cited by: §5.1.
[78] L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836–3847. Cited by: §2, §4.3.
[79] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595. Cited by: §5.2.
[80] Y. Zhang, Y. Wei, D. Jiang, X. ZHANG, W. Zuo, and Q. Tian (2024) ControlVideo: training-free controllable text-to-video generation. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2.
[81] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng MagicVideo: efficient video generation with latent diffusion models (2022). arXiv preprint arXiv:2211.11018. Cited by: §2.
[82] H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Li (2021) Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1316–1325. Cited by: Table 1, §3.
[83] S. Zhu, J. L. Chen, Z. Dai, Z. Dong, Y. Xu, X. Cao, Y. Yao, H. Zhu, and S. Zhu (2024) Champ: controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision, pp. 145–162. Cited by: §1, §2.
[84] R. Zuo, R. A. Potamias, E. Ververas, J. Deng, and S. Zafeiriou (2025) Signs as tokens: a retrieval-enhanced multilingual sign language generator. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23806–23816. Cited by: §1.

Metric	Ours	Wan-FC
Keyword Recall $\uparrow$	0.54	0.30
BLEURT $\uparrow$	0.31	0.19
LLM adequacy $\uparrow$	2.07	1.50
Failure Rate $\downarrow$	0.17	0.27