Veda: Scalable Video Diffusion via Distilled Sparse Attention
Extracted source text · arXiv 2605.30325
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage[table]{xcolor}
\usepackage{booktabs} % for professional tables
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2026} with \usepackage[nohyperref]{icml2026} above.
\usepackage{hyperref}
\newcommand{\xjqi}[1]{\textcolor[rgb]{1.0,0,0}{{[\textbf{xjqi}: #1]}}}
\newcommand{\shihao}[1]{\textcolor[rgb]{0,0,1.0}{{[\textbf{shihao}: #1]}}}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
% \usepackage{icml2026}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage[accepted]{icml2026}
\makeatletter
\icmlshowauthorstrue
\renewcommand{\Notice@String}{\ICML@appearing}
\makeatother
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{multirow}
\usepackage{array}
\usepackage{pifont}
\usepackage{float}
\usepackage{cuted}
\usepackage{caption}
\usepackage{capt-of}
\usepackage{dblfloatfix}
\usepackage{setspace}
\definecolor{lightgray}{gray}{0.85}
\newcommand{\cmark}{\ding{51}}
\newcommand{\xmark}{\ding{55}}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
\usepackage[disable,textsize=tiny]{todonotes}
%\usepackage[textsize=tiny]{todonotes}
\newcommand{\statReady}[1]{\textcolor{teal}{\textbf{[READY]} #1}}
\newcommand{\statPending}[1]{\textcolor{orange}{\textbf{[PENDING]} #1}}
\newcommand{\statCritical}[1]{\textcolor{red}{\textbf{[CRITICAL]} #1}}
\newcommand{\statDone}{\textcolor{gray}{\sout{Done}}}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{Veda: Scalable Video Diffusion via Distilled Sparse Attention}
\begin{document}
\twocolumn[
\icmltitle{Veda: Scalable Video Diffusion via Distilled Sparse Attention}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2026
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{intern}{*}
\icmlsetsymbol{lead}{\ensuremath{\dagger}}
\begin{icmlauthorlist}
\icmlauthor{Shihao Han}{comp,hku,intern}
\icmlauthor{Hao Yang}{comp,lead}
\icmlauthor{Xiaofeng Mei}{comp}
\icmlauthor{Xinting Hu}{ustc}
\icmlauthor{Yi Jiang}{comp}
\icmlauthor{Xiaojuan Qi}{hku}
\end{icmlauthorlist}
\icmlaffiliation{comp}{ByteDance Inc.}
\icmlaffiliation{hku}{The University of Hong Kong}
\icmlaffiliation{ustc}{University of Science and Technology of China \textsuperscript{*}Work done during internship at ByteDance. \textsuperscript{\ensuremath{\dagger}}Project Lead}
\icmlcorrespondingauthor{Xinting Hu}{xinting@ustc.edu.cn}
\icmlcorrespondingauthor{Xiaojuan Qi}{xjqi@hku.hk}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
\icmlkeywords{Machine Learning, ICML}
\vskip 0.2in
]
\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
\begin{strip}
\centering
\vspace{-2.1cm}
\includegraphics[width=\textwidth]{visuals/teaser_combined.pdf}
\vspace{-0.2in}\captionof{figure}{\textbf{Veda achieves a 5.1$\times$ end-to-end GPU speedup for high-resolution, long-video generation.} Evaluated on Waver-T2V-12B (720P / 241 frames), Veda at 95\% sparsity reduces the sample time from 19.4 to 3.8 minutes without compromising visual quality.}
\label{fig:teaser}
\end{strip}
% Mentiond, scaling curve is the same as the waver and wan
\begin{abstract}
Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups.
Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-attention speedup, reducing attention overhead from 92\% to 50\%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.
%Scaling diffusion models to high-resolution, long-duration video synthesis is fundamentally constrained by the quadratic computational complexity ($\mathcal{O}(N^2)$) of full self-attention. We identify that existing sparse attention paradigms fail to preserve the underlying attention topology, leading to attention drifts and generation artifacts.
%To bridge this gap, we introduce \textbf{Veda}, a hardware-efficient framework that reformulates sparse token selection as a statistical reconstruction problem. By leveraging flexible tiling to minimize intra-block variance, Veda employs a projector that utilizes sufficient statistics to recover the energy landscape of the oracle distribution, thereby aligning the predicted sparsity mask with the intrinsic attention geometry. This allows Veda to aggressively prune tokens while maintaining generation quality indistinguishable from full-attention baselines.
%To materialize theoretical sparsity into tangible wall-clock latency reductions, we implement a hardware-efficient tile-sparse kernel. Extensive evaluations demonstrate that Veda achieves substantial performance gains at sparsity levels yielding negligible visual discrepancies compared to full attention. On the Waver-T2V-13B model generating 720P videos, Veda delivers a \textbf{5.1$\times$} end-to-end acceleration and a \textbf{10.5$\times$} speedup in attention computation, effectively suppressing the attention overhead from 92\% to 50\%; similarly, a \textbf{2.6$\times$} end-to-end speedup is observed on the 1B variant. These efficiency improvements generalize to Wan2.1-T2V architectures benchmarked against FlashAttention-3: Veda achieves a \textbf{2.6$\times$} end-to-end speedup (\textbf{7.1$\times$} in self-attention) on the 14B model, and a \textbf{1.6$\times$} speedup (\textbf{3.2$\times$} in self-attention) on the 1.3B model.
% To materialize theoretical sparsity into tangible wall-clock latency reductions, we implement a FlashAttention-3\cite{flashattn-3} style block-skipping sparse attention kernel, achieving 80\% of FlashAttention-3 MFU.
% On the Waver-T2V-13B model generating 720P 10 second videos, Veda delivers a \textbf{5.1$\times$} end-to-end acceleration and a \textbf{10.5$\times$} speedup in attention computation, effectively suppressing the attention overhead from 92\% to 50\%. Similarly, for the Waver-T2V-1B\cite{waver} model synthesizing 480P 5s videos, we observe a 2.6$\times$ end-to-end speedup and a 3.5$\times$ attention speedup.
% On Wan2.1-T2V-1.3B, we acheve 1.59x end2end generation speed-up on 480P 21Frame video, and
% Notably, Veda exhibits superior stability at extreme sparsity levels (99\%), eliminating sparse-induced artifacts prevalent in prior works.
% During training, we adopt an attention distillation paradigm supervised by KL divergence. To expedite this process, we leverage TileLang to implement a highly optimized ground-truth attention heatmap reduction operator for teacher computation.
%Consequently, our method halves the training step time—even under full query sequence supervision—compared to FlashAttention-3, while strictly adhering to the convergence trajectory of full attention baselines.
% Our study alleviates the computational bottlenecks inherent to long video generation, rendering complexity manageable at scale, and we hope it will serve as a key enabler for the further scaling of video diffusion models.
\end{abstract}
\input{pages/intro}
\input{pages/related_work}
\input{pages/preliminary}
\input{pages/method}
\input{pages/experiments}
\input{pages/results}
\input{pages/conclusion}
\section*{Impact Statement}
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
% \clearpage{}
\bibliography{paper}
\bibliographystyle{icml2026}
\input{pages/appendix}
\end{document}
\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{visuals/cdp-whiteboard-v3-cropped.png}
\caption{\textbf{Architectural comparison of sparse attention methods.} We compare the proposed Veda (ours) with representative methods that use (i) static sparse patterns (SVG~\cite{svg}) and (ii) dynamically learned sparse masks (VSA~\cite{vsa}). Solid lines indicate the inference data flow, while dashed lines denote auxiliary components used only during training.}
\label{fig:compare_diagram}
\end{figure*}
\section{Introduction}
% xinting - P1：背景 + 把问题“工程化地定义”出来：GPU 上 attention 是按 tile 计算的，所以稀疏化=为每个 query tile 选 top-K 个 key tiles（预测 tile-wise mask） （不用讲 local window）
% 这一段在把“为什么需要 sparse attention”以及“稀疏化在 GPU 上到底意味着什么”说清楚。
%Scaling video diffusion Transformers~\cite{wan,hunyuanvideo-1.5} to higher resolution and longer duration is increasingly limited by the quadratic compute and memory cost of full attention.
Diffusion Transformers (DiTs)~\cite{dit} have emerged as the leading architecture for high-fidelity video synthesis ~\cite{opensora,stepvideo,wan,longcat,cogvideox,hunyuanvideo-1.5}. Nevertheless, their scalability is bottlenecked by the quadratic computational and memory complexity of self-attention ~\cite{attention} with respect to the spatiotemporal token sequence length.
Sparse attention offers a natural path toward alleviating this bottleneck by restricting attention computation to a subset of token pairs.
In practice, however, sparse attention must be realized at the granularity of tiles rather than individual tokens to align with modern GPU hardware, which performs attention through block-wise matrix multiplications \cite{tensorcore,sta}. This reformulates sparse attention as a \textbf{\emph{tile selection}} problem: tokens are grouped into tiles, and each query tile attends to only a small number of key tiles, producing a tile-wise mask that can be directly materialized by a tile-skipping kernel for efficient execution.
%To align with modern GPU hardware, which executes attention as tile-level matrix multiplications, practical acceleration requires skipping entire tiles rather than individual tokens.
%Consequently, video sparse attention can be formulated as \textbf{\emph{tile selection}}: tokens are partitioned into tiles, and each query tile only attends to a small set of key tiles, producing a tile-wise mask that can be directly materialized by a tile-skipping kernel.
% xinting - P2：sparse attention会带来生成视频质量的下降. 为了最大限度保留最有用的attention，现有sparse attn方法怎么做（Static tile Pattern / Learned pattern with tile pooling）+问题
However, translating sparsity into wall-clock acceleration often comes at the cost of generation quality. To preserve spatiotemporal structure, existing approaches construct tile masks with two main paradigms. Methods such as SVG~\cite{svg} and STA~\cite{sta} leverage inductive biases of pretrained models to define candidate spatiotemporal masks and employ online or offline search strategies to select the most effective \textbf{\textit{pre-defined}} patterns (see Fig.~\ref{fig:compare_diagram}, SVG). While these methods are simple and hardware-friendly, they lack adaptability to the highly dynamic and head-specific attention structures learned by DiTs.
In contrast, VMOBA~\cite{vmoba} and VSA~\cite{vsa} adopt dynamic tile selection by estimating tile importance from compressed representations, such as pooled features or low-rank approximations, and ranking tiles to synthesize \textbf{\textit{dynamic}} tile-selection masks. Although more flexible (see Fig.~\ref{fig:compare_diagram}, VSA), these approaches suffer from inaccurate estimation: tile importance is learned only implicitly through the diffusion objective, without explicit supervision to preserve the structural geometry of full attention. Moreover, their reliance on coarse statistics, such as mean pooling, fails to capture salient signal peaks within tiles, causing the estimated tile scores to misrepresent true query-key correlations.
As a result, both paradigms exhibit pronounced quality degradation (Fig.~\ref{fig:sparse_artifacts}) when sparsity is pushed beyond moderate levels, producing structured artifacts including spatial warping, water-ripple patterns, and temporal flickering.
%In contrast, VMOBA~\cite{vmoba} and VSA~\cite{vsa} adopt dynamic tile selection by estimating tile importance from compressed representations, such as pooled features or low-rank approximations, and ranking tiles to synthesize \textbf{\textit{dynamic}} tile-selection masks.
%Although more flexible, these approaches suffer from inaccurate estimation: they are optimized by implicit supervision via the diffusion objective, lacking explicit guidance to preserve the full-attention structure. Furthermore, reliance on coarse statistics such as mean pooling fails to capture signal peaks in tile representation, causing the estimated tile scores to misrepresent query-key correlations (\xjqi{see Fig. \ref{fig:compare_diagram}: VSA}).
%Consequently, both paradigms exhibit pronounced quality degradation (see Fig.~\ref{fig:sparse_artifacts}) when sparsity is pushed beyond moderate levels, producing structured artifacts, including spatial warping, water-ripple patterns, and temporal flickering.
%Although more flexible, these approaches suffer from coarse estimation: simple statistics, such as mean pooling, are insufficient to rank tiles reliably, causing the selected mask to deviate from the optimal one, particularly under extreme sparsity. Consequently, both paradigms exhibit pronounced quality degradation \xjqi{(see Fig. x)} when sparsity is pushed beyond moderate levels, producing structured artifacts, including spatial warping, water-ripple patterns, and temporal flickering. % \xjqi{can we add some active and passive??}
%However, achieving wall-clock efficiency often compromises generation quality.
%To preserve spatiotemporal structure, existing solutions construct the tile mask through two distinct paradigms. STA~\cite{sta} and VORTA~\cite{vorta} rely on inductive biases of the pre-trained model to define candidate spatiotemporal masks, utilizing online or offline search algorithms to select the most effective \textbf{\textit{pre-defined}} ones. These methods benefit from simplicity and hardware friendliness but lack adaptability to the highly dynamic and head-specific attention structures learned by pretrained DiTs. In contrast, Sparge Attention~\cite{spargeattention} and VSA~\cite{vsa} employs dynamic tile selection, estimating tile importance estimating tile importance using compressed representations such as pooled features or low-rank approximations to rank tiles and synthesize \textit{\textbf{dynamic}} tile-selection masks. Although more flexible, these approaches suffer from coarse estimation: simple statistics such as mean pooling fail to reliably identify the tiles that dominate the full attention distribution, especially under extreme sparsity. As a result, both paradigms exhibit pronounced quality degradation when sparsity is pushed beyond moderate levels, producing structured artifacts such as spatial warping, water-ripple patterns, and temporal flickering that are distinct from ordinary generative errors.
% xinting - P3: 为了弄清What matters for good sparse attention， 我们开展了Empirical Study，表明 -- 稀疏 attention 是否能稳定扩展到high spasity，关键在于sparse attention map 与 full attention map 的tile-wise结构对齐
To understand what limits sparse attention in video diffusion, we conduct an empirical study on the relationship between sparsity and generation quality (see Fig.~\ref{fig:oracle_comparison}). We construct an ``Oracle'' sparse mask by max-pooling the full-attention matrix into a tile-level score map and retaining the highest-response tiles, thereby inducing a reference per-query ranking of key tiles. Models using this Oracle mask preserve generation quality even under very high sparsity, demonstrating that sparsity itself is not the primary cause of degradation (Fig.~\ref{fig:oracle_comparison}).
Instead, we find that mask quality, specifically, the degree to which the sparse mask aligns with the tile-wise structure of full attention, dominates performance.
Consistent with this observation, existing methods fail for two complementary reasons: methods with pre-defined sparse patterns suffer from structural mismatch with head-specific attention geometry, while dynamic methods incur estimation error due to implicit supervision and insufficient tile statistics. Furthermore, we observe substantial heterogeneity across attention heads and diffusion timesteps, indicating that a single, uniform tiling strategy is inherently suboptimal. This suggests that sparse attention must control two coupled factors: how tokens are grouped into tile, and which tiles are selected.
% A high-quality score predictor can still be limited if the tile layout splits correlated tokens or merges unrelated ones.
Guided by these findings, we propose Veda (Fig.~\ref{fig:compare_diagram}, Veda), a distilled sparse attention framework that preserves the tile-wise structure and ranking of full attention even under extreme sparsity. Unlike prior dynamic methods in which tile selection emerges implicitly from the diffusion objective, Veda learns tile selection as an explicit target. Specifically, Veda employs a lightweight estimator to distill tile-level attention scores from a full-attention backbone. To reduce estimation error, we enrich tile representations with extrema-aware statistics that preserve salient peaks beyond simple mean pooling.
To further mitigate structural mismatch, Veda introduces a Head-Aware Tiling strategy: under a fixed hardware tile budget, heads that primarily capture local spatial interactions and those that model long-range temporal dependencies are assigned distinct spatiotemporal factorizations of tiles. Together, these components enable aggressive pruning of attention tiles while maintaining alignment with the intrinsic structure of full attention.
Finally, to translate theoretical FLOPs reduction into practical end-to-end acceleration, we implement a tile-skipping sparse attention kernel. This kernel materializes the predicted sparsity through efficient tile skipping and reaches approximately $80\%$ of the MFU achieved by FlashAttention-3~\cite{flashattn-3} while executing sparse attention, ensuring that sparsity yields tangible latency gains rather than overhead-dominated speedups.
%Guided by these findings, we propose \textbf{Veda} (see Fig.~\ref{fig:compare_diagram}, Veda), a distilled sparse attention framework that preserves the tile-wise structure and ranking of full attention even under extreme sparsity. Unlike prior dynamic methods that let tile selection emerge implicitly from the diffusion objective, Veda learns tile selection as an explicit target. Specifically, Veda employs a lightweight estimator to distill the tile-level scores of the full-attention backbone. To reduce estimation error in score prediction, we enrich tile representations with extrema-aware statistics \shihao{Maybe a better name.} that preserve salient peaks beyond mean pooling. Furthermore, to mitigate structural mismatch, Veda introduces a Head-Aware Tiling strategy: under the same hardware tile budget, heads that primarily capture local spatial interactions versus long-range temporal interactions are assigned distinct spatiotemporal factorizations of the tile. Together, these components enable aggressive reduction of attention tiles while preserving alignment with the intrinsic structure of full attention.
%Finally, to translate FLOPs reduction into real end-to-end speedups, we implement a tile-skipping sparse attention kernel.
%This kernel materializes predicted sparsity via efficient tile skipping, achieving $0.8\times$ MFU of FlashAttention-3~\cite{flashattn-3} while executing sparse attention, so that sparsity yields tangible latency gains rather than overhead-dominated speedups.
%Guided by these findings, we propose \textbf{Veda}, a framework that maximizes structural alignment with full self-attention under extreme sparsity. Veda enforces alignment via a tile-wise distillation objective that supervises a sparse model to mimic the attention distribution of a pretrained full attention backbone. To reduce pattern mismatch, we introduce a head-aware tiling strategy that assigns adaptive tile sizes based on head-specific attention structure. To reduce estimation error, we perform tile scoring with richer statistics and a learned projection, preserving fine-grained correlations. %这个tile-wise score estimation 还需要改下
% xinting - P5： 我们实现的 高效tile-skipping kernel 将attention稀疏转化为真实 wall-clock 加速 + 不同模型上的实验结果，特别是高sparsity
% 这里和abs是不是要补充一下，我们在90% sparsity的时候，可以在困难prompt上使用human eval和full attn完全一致？
%We conduct extensive experiments on Waver and Wan2.1 video diffusion models and consistently observe substantial acceleration without sacrificing generation quality. Notably, on Waver-T2V-13B generating 720P videos with 241 frames, Veda achieves a 5.1× end-to-end speedup and a 10.5× self-attention speedup, reducing attention overhead from 92\% to 50\% (Fig.~\ref{fig:waver_wan_accl_compare}).
We conduct extensive experiments on the Waver and Wan2.1 video diffusion models and consistently observe substantial acceleration with no noticeable degradation in generation quality. Across both backbones, Veda maintains stable visual fidelity and temporal coherence even at high sparsity, indicating that the speed gains do not come at the cost of structural artifacts. On Waver-T2V-12B generating 720P 10-second videos, Veda achieves a $5.1\times$ end-to-end speedup and a $10.5\times$ self-attention speedup, reducing attention overhead from 92\% to 50\%. On Wan2.1-T2V-14B 720P 5-second, Veda further delivers a $2.63\times$ end-to-end speedup with a $7.08\times$ acceleration in self-attention. Notably, the speedup increases with sequence length, demonstrating that Veda scales favorably with spatiotemporal resolution across models (Fig.~\ref{fig:waver_wan_accl_compare}).
%\xjqi{We conduct extensive experiments on xxx and xxx and consistently ..... }On Waver-T2V-13B\cite{waver} generating 720P 10-second \shihao{Use frames here better or seconds? For this, we running 24fps w/ 241 frames, but others are 16fps.} videos, Veda delivers a 5.1$\times$ end-to-end acceleration and a 10.5$\times$ attention speedup, reducing attention overhead from 92\% to 50\%.
%On Waver-T2V-1B ~\cite{waver} for 480P 5-second videos, Veda achieves a 2.6$\times$ end-to-end speedup and a 3.5$\times$ attention speedup. Extending to Wan2.1-T2V architectures ~\cite{wan}, Veda demonstrates consistent scalability: on the 14B model, it achieves a 2.63$\times$ end-to-end speedup with self-attention accelerated by 7.08$\times$, while on the 1.3B model, it yields a 1.59$\times$ end-to-end speedup and a 3.15$\times$ reduction in self-attention latency. A breakdown of the inference time across these configurations is presented in Figure~\ref{fig:waver_wan_accl_compare}.
% In summary, our contributions are:
% \begin{itemize}
% \item An empirical diagnosis showing that inaccurate mask prediction (from both fixed and dynamic paradigms) is the primary tileer to scaling sparsity in video diffusion.
% \item The key insight that preserving the original dot-product geometry is necessary for precise oracle mask prediction under extreme sparsity.
% \item \textbf{Veda}, a metric-learning-based sparse attention framework with a tile-wise attention projector and head-aware flexible tiling, enabling aggressive pruning without quality loss.
% \item A tile-skipping sparse attention kernel in the style of FlashAttention-3 that converts sparsity into substantial wall-clock acceleration.
% \end{itemize}
\section{Related Work}
\noindent\textbf{Sparse Attention in Language Modeling}
Sparse attention was first developed and stress-tested at scale in language modeling \cite{llama,qwen,deepseek}. Early works introduced static sparsity constraints like Sliding Window Attention~\cite{neibour,mixattn} and StreamingLLM~\cite{attnsink} that restrict attention to local neighborhoods or specific sinks to maintain efficiency. More recent approaches have evolved toward dynamic sparsity, utilizing either heuristic approximations (\textit{e.g.}, MMInference~\cite{mminference}, MoBA~\cite{moba}, NSA~\cite{nsa}) or predictive mechanisms that estimate significant attention tiles (e.g., SeerAttention~\cite{seer}, DSA~\cite{dsa}). Our approach is closely related to this predictive paradigm. However, direct transfer to video generation is non-trivial: tightly coupled spatiotemporal dependencies can amplify small sparsity errors into structured distortions and temporal flicker, motivating video-specific sparsity designs.
% \subsection{Efficiency in Video Diffusion Transformers}
% Beyond sparse attention, research into accelerating Video Diffusion Transformers (Video DiTs) generally follows three main paradigms.
% \paragraph{Token Merging and Reduction.}
% Approaches such as BSA\cite{bsa} and ToMe\cite{tome} mitigate computational overhead by aggregating or pruning redundant tokens based on visual similarity. However, attaining high sparsity levels solely through these mechanisms remains challenging, as aggressive merging often compromises representational integrity. Crucially, our proposed method is orthogonal to these token reduction techniques, allowing for further enhance efficiency.
% \todo[inline]{
% emperical & inspration
% method ->
%
% \textbf{Rewrite 2.3 and add preliminary here}
% \begin{itemize}
% \item Problem definition
% \begin{itemize}
% \item Attention intro
% \item Sparse attn problem definition
% \end{itemize}
% \item Framework (Basic Formula)
% \item Change under Framework
% \begin{itemize}
% \item Tiling or No tiling (Identity)
% \item Have learnable parameter
% \item Selection or Predict the Pattern
% \item Add Linear Attention?
% \end{itemize}
% \end{itemize}
% }
\noindent\textbf{Sparsity Mechanisms for Video Synthesis}
Sparsifying the attention matrix in Video DiTs offers a pathway to efficient long-video generation.
Works such as SVG~\cite{svg}, Sparse-vDiT~\cite{sparse-vdit}, VORTA~\cite{vorta} and STA~\cite{sta} rely on static sparsity patterns, which are either manually defined or derived from offline statistics.
To produce content-adaptive sparsity at runtime, methods like VSA~\cite{vsa} and VMOBA~\cite{vmoba} rely on the backbone network to implicitly learn the optimal sparse structure alongside the video generation task.
Despite wall-clock gains, the learned selection can drift away from the tile structure implicitly relied upon by the pretrained full-attention model, leading to visible artifacts under high sparsity. This motivates our key stance: learning where to attend benefits from explicit objectives that preserve the tile-wise structure of full attention.
% However, a key insight of our work is that learning \textit{where to attend} is an optimization problem distinct from learning \textit{what to generate}. Without direct supervision or explicit distillation targets, these methods often struggle to converge to an optimal sparsity distribution. Consequently, they tend to exhibit slow training convergence and, at high sparsity levels, frequently produce visible artifacts due to the misalignment between the learned attention mask and the true dependency structure.
% The limitation of this paradigm lies in its rigidity: pre-defined masks often fail to align with the varying motion dynamics and semantic content of different videos. When the static pattern mismatches the underlying video structure, the model is forced to hallucinate missing dependencies, resulting in blocking artifacts.
\noindent\textbf{Efficiency for Video DiTs Beyond Sparse Attention}
Beyond sparse attention, research on accelerating Video DiTs generally follows complementary paradigms.
Methods such as PAB~\cite{pab} and TeaCache~\cite{teacache} reuse intermediate activations across denoising steps, but the locality assumption can break in few-step regimes where large timestep jumps induce substantial feature disparity. In parallel, sampling distillation methods such as CausVid~\cite{causvid} and rCM~\cite{rcm} reduce the number of model evaluations by compressing the diffusion trajectory, yet the per-step spatial cost remains quadratic ($O(N^2)$). Our sparse attention framework is orthogonal to both directions and can be combined with them to maximize end-to-end acceleration.
% To bypass the quadratic term, SANA-Video~\cite{sana-video} replaces standard attention with linear-complexity kernels, while hybrid designs such as SLA~\cite{sla} mix linear and sparse attention, often at the cost of additional overhead \shihao{SLA has overhead over VSA/Ours; SANA is much faster but has very bad quality, maybe just remove this part?}.
\section{What Matters for a Good Sparse Model?}
\subsection{Preliminary}\label{sec:preliminary}
Modern video diffusion DiTs~\cite{dit} model global dependencies over spatiotemporal tokens, yet attention quickly becomes the primary computation and memory bottleneck at high resolution and long duration.
Specifically, a video latent of shape $(T,H,W)$ is flattened into a token sequence of length $N=THW$.
For a single attention head, let $\mathbf{Q},\mathbf{K},\mathbf{V} \in \mathbb{R}^{N\times d}$ denote the query, key, and value matrices. Attention~\cite{attention} is computed as
\begin{equation}
\mathbf{A}=\mathrm{Softmax}\!\left(\frac{\mathbf{QK}^\top}{\sqrt{d}}\right),\qquad \mathbf{O}=\mathbf{AV},
\end{equation}
which incurs $O(N^2)$ computation and memory complexity.
To mitigate the quadratic cost, sparse attention introduces a mask to restrict computation to a subset of token pairs.
Following VSA~\cite{vsa}, this can be written as $\mathbf{A}=\mathrm{Softmax}(\mathbf{QK}^\top/\sqrt{d}+\mathbf{M})$,
where full attention corresponds to $\mathbf{M}=\mathbf{0}$, and sparsity is induced by setting most entries of $\mathbf{M}$ to $-\infty$.
Modern hardware accelerators (e.g., GPUs with Tensor Cores) are optimized for tile-level data movement and computation. Accordingly, we formulate sparse attention at the tile granularity.
Concretely, $N$ tokens are grouped into $N_T$ tiles of size $B$, yielding tiled tensors
$\widetilde{\mathbf{Q}}, \widetilde{\mathbf{K}}, \widetilde{\mathbf{V}}\in\mathbb{R}^{N_T \times B \times d}$. $\widetilde{\mathbf{M}}_{ij}\in\{0,1\}$ denotes a binary tile-selection mask for the $(i,j)$ tile pair; selected tile pairs are evaluated by the sparse kernel, while unselected pairs are equivalently assigned additive mask value $-\infty$ in the softmax.
Under a fixed Top-$k$ budget, each query tile $\widetilde{\mathbf{Q}}_i$ attends to $k$ key tiles and masks out the rest, yielding a tile mask $\widetilde{\mathbf{M}}$ that satisfies
\[
\bigl|\{\, j \mid \widetilde{\mathbf{M}}_{ij}=1 \,\}\bigr| = k, \qquad \forall i \in \{1,\dots,N_T\}.
\]
The sparse attention for query tile $\widetilde{\mathbf{Q}}_i$ is computed as:
\begin{equation}
\begin{aligned}
\hat{\mathbf{K}}_i &= \mathrm{Concat}_{j:\,\widetilde{\mathbf{M}}_{ij}=1}(\widetilde{\mathbf{K}}_j),\quad
\hat{\mathbf{V}}_i = \mathrm{Concat}_{j:\,\widetilde{\mathbf{M}}_{ij}=1}(\widetilde{\mathbf{V}}_j),\\
\mathbf{O}_i &= \mathrm{Softmax}\!\left(\frac{\widetilde{\mathbf{Q}}_i\hat{\mathbf{K}}_i^\top}{\sqrt{d}}\right)\hat{\mathbf{V}}_i.
\end{aligned}
\end{equation}
This enables a tile-skipping kernel to bypass the masked tiles for practical wall-clock speedups.
\subsection{Empirical Study}
\label{sec:empirical}
We first consider two common sparse attention designs that follow the pipeline in \ref{sec:preliminary}:
(i) \textit{Static-pattern} methods that instantiate a tile-structured mask $\mathbf{\widetilde{M}}$ from pre-defined spatiotemporal layouts; and
(ii) \textit{Dynamic} methods that construct tile representations via tile pooling and predict tile importance for per-query Top-$k$ selection.
\begin{figure}[!h]
\centering
\includegraphics[width=1\linewidth]{visuals/water_ripple_zoomed_result.pdf}
\caption{\textbf{Sparsity-Induced Artifacts.} At $90\%$ sparsity, samples generated with average pooling exhibit “water-ripple” distortions and stochastic noise, which are distinct from standard hallucinations.\footnotemark[1]}
\label{fig:sparse_artifacts}
\end{figure}
\footnotetext[1]{Human faces are used because video artifacts are subtle in static images, whereas facial distortions are immediately perceptible to humans.}
\paragraph{Observation 1: structural artifacts emerge at high sparsity.}
At high sparsity ($\ge 90\%$), both static and dynamic baselines exhibit a distinct class of \emph{structural artifacts} that are different from semantic hallucinations of the base model.
Typical failures include water-ripple patterns, local geometric distortions, and temporal flickering across frames (Fig.~\ref{fig:sparse_artifacts}).
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{visuals/mask_compare.pdf}
\caption{\textbf{Impact of Mask Precision.}
At a fixed sparsity level ($90\%$), generation quality is governed by the accuracy of tile selection.
The \emph{Oracle Mask}, selected from full-attention Top-$k$ scores, preserves high-fidelity structure and serves as an upper bound.
In contrast, the \emph{Average Pooling} mask used in VSA~\cite{vsa} misestimates tile importance and causes severe artifacts.
Veda distills the oracle selection into an efficient sparse mask, recovering oracle-like quality under the same budget.}
\label{fig:oracle_comparison}
\end{figure}
\paragraph{Observation 2: mask quality dominates sparsity ratio.}
To disentangle the effect of the sparsity ratio from mask quality, we perform a controlled intervention at a fixed sparsity level (e.g., $90\%$).
Using the ``optimal'' mask derived from full attention scores yields substantially better generations than a pooling-based mask under the same sparsity budget (Fig.~\ref{fig:oracle_comparison}).
This gap indicates that the bottleneck is not the sparsity ratio itself, but whether the tile mask preserves the tile-wise structure of full attention.
\begin{figure}[!h]
\centering
\includegraphics[width=1\linewidth]{visuals/sparse_pattern_recall_axis_legend_text_2x.pdf}
\caption{\textbf{Tile recall analysis against full attention oracle.} Visualization of tile-level selection agreement with the full attention oracle mask (Top-$k$ tiles by aggregated attention scores). Orange indicates true positives (correctly selected tiles), red denotes false positives (incorrectly selected), and blue shows false negatives (missed tiles).}
\label{fig:recall_vs_quality}
\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{visuals/attention_padding_comparison.pdf}
\caption{\textbf{Diversity of Attention Patterns.} Distinct attention heads exhibit varying spatial and temporal footprints that evolve across diffusion timesteps, indicating the need for head-specific adaptation.}
\label{fig:head_diversity}
\end{figure}
\noindent \textbf{Metric: tile recall w.r.t. full attention.}
To quantify tile-wise alignment, we define a simple recall metric against full attention.
For each query tile $i$, let $\widetilde{\mathbf{M}}^{\text{fu}}_i$ be the mask of Top-$k$ key tiles under full attention (computed by aggregating token-level scores into tile scores), and $\widetilde{\mathbf{M}}^{\text{sp}}_i$ be the mask of selected Top-$k$ tiles of a sparse method.
Define the selected key-tile sets as $\mathcal{S}^{\text{fu}}_i=\{j\mid \widetilde{\mathbf{M}}^{\text{fu}}_{ij}=1\}$ and $\mathcal{S}^{\text{sp}}_i=\{j\mid \widetilde{\mathbf{M}}^{\text{sp}}_{ij}=1\}$.
We report \emph{tile recall}:
\begin{equation}
\mathrm{Recall}@k = \frac{1}{N_T}\sum_{i=1}^{N_T}\frac{| \mathcal{S}^{\text{sp}}_i \cap \mathcal{S}^{\text{fu}}_i |}{k}.
\end{equation}
Across sparsity levels, higher tile recall correlates strongly with fewer structural artifacts, making it a reliable indicator of stability under extreme sparsity (Fig.~\ref{fig:recall_vs_quality}).
\paragraph{Observation 3: head-wise heterogeneity challenges uniform tile designs.}
Finally, inspecting attention patterns reveals pronounced head-wise diversity across layers and diffusion timesteps (Fig.~\ref{fig:head_diversity}).
A single, uniform tiling strategy can under-aggregate critical correlations for some heads while over-aggregating others, which degrades tile recall under high sparsity and motivates head-aware tile designs.
% \paragraph{\textit{Discussion with train-from-scratch sparse transformers.}}
% We also evaluate train-from-scratch sparse transformers that learn sparsification jointly with the diffusion objective.
% These methods are not designed to preserve a fixed full attention reference, and they also exhibit noticeable quality degradation and structural artifacts under high sparsity in our experiments. We report the results for completeness and defer a detailed discussion to the Appendix. \shihao{Not sure what this paragraph is talk about. We did not evaluate nor have a good enough train-from-scratch video dit to evaluate.}
\section{Veda}
We introduce Veda, a sparse attention framework designed to distill the intrinsic structure and ranking of full attention. As a lightweight, architecturally modular component, Veda actively aligns tile selection with the full-attention backbone even under extreme sparsity. The overall architecture is shown in Fig.~\ref{fig:compare_diagram}, and this section is organized as follows.
In Sec.~\ref{sec:veda41}, Veda casts tile selection as an explicit alignment objective: a statistics-aware score estimator is supervised to recover full-attention query-key tile scores.
Sec.~\ref{sec:head_aware_search} addresses head-wise heterogeneity with Head-Aware Tiling, assigning head-dependent spatiotemporal tile configurations to reduce structural mismatch.
Finally, Sec.~\ref{sec:hardware_impl} presents a hardware-efficient tile-skipping kernel that materializes the predicted masks into practical end-to-end acceleration.
\subsection{Distilled Tile Scoring for Sparse Attention}
\label{sec:veda41}
Veda optimizes a binary tile mask $\widetilde{\mathbf{M}}$ under a fixed Top-$k$ budget to preserve the tile-wise structure and ranking induced by full attention. Rather than relying on the diffusion objective to shape sparsity implicitly, we cast tile selection as an explicit alignment problem with supervision derived from the full-attention backbone.
\paragraph{Target: Full-Attention Score Construction.}
To guide the sparse selection, we first derive a reference tile ranking from the full-attention backbone. Given the query and key features $\mathbf{Q}, \mathbf{K} \in \mathbb{R}^{N \times d}$, the full attention map is $\mathbf{A}^* = \text{Softmax}(\mathbf{Q}\mathbf{K}^\top / \sqrt{d})$. To map this token-level density to tile-level importance, we apply a max-pooling operation over the corresponding query-key tile regions:
\begin{equation}
\mathbf{S}^{\text{tgt}}_{ij} = \max_{(u,v) \in \text{Tile}(i,j)} \mathbf{A}^*_{uv} .
\end{equation}
We opt for max-pooling rather than averaging because attention distributions are typically sparse and peaky; averaging tends to dilute salient high-frequency signals with background noise, whereas max-pooling preserves the existence of critical dependencies.
\paragraph{Statistic-Aware Estimator.}
We employ a lightweight estimator $\mathcal{E}_{\phi}$ to reconstruct the target scores using compressed tile representations. First, for each query/key tile, we construct a $\text{TripPool}$ descriptor by concatenating {Avg, Max, Min} triplet statistics:
\begin{equation}
\text{TripPool}[\cdot] = \mathrm{Avg}[\cdot] \oplus \mathrm{Max}[\cdot] \oplus \mathrm{Min}[\cdot],
\end{equation}
where $ \oplus $ denotes the concatenation.
To capture heterogeneous dependency patterns, the estimator employs head-specific MLP projections $\phi_q$ and $\phi_k$. For a given attention head, these projections map the statistics into a shared latent space. The predicted score $\mathbf{S}^{\text{pred}}_{ij}$ between query tile $\widetilde{\mathbf{Q}}_i$ and key tile $\widetilde{\mathbf{K}}_j$ is then computed as:
\begin{equation}
\mathbf{S}^{\text{pred}}_{ij} = \frac{\phi_q(\text{TripPool}[\widetilde{\mathbf{Q}}_i]) \cdot \phi_k(\text{TripPool}[\widetilde{\mathbf{K}}_j])^\top}{\sqrt{d'}},
\end{equation}
where $d'$ is the estimator latent dimension. Note that while we omit the head index for brevity, distinct projection weights are learned for each head.
\paragraph{Optimization.}
We align the predicted tile-score distribution to the full-attention reference via a row-wise distillation objective.
Concretely, we apply row-wise normalization on $\mathbf{S}^{\text{tgt}}$ and Softmax on $\mathbf{S}^{\text{pred}}$ over key tiles to obtain per-query attention weights $\mathbf{A}^{\text{tgt}}$ and $\mathbf{A}^{\text{pred}}$, and minimize
\begin{equation}
\mathcal{L}_{\text{distill}} = \mathcal{D}_{\text{KL}}\!\left(\mathbf{A}^{\text{tgt}} \parallel \mathbf{A}^{\text{pred}}\right),
\end{equation}
averaged over query tiles and heads.
By applying per-query Top-$k$ selection on $\mathbf{S}^{\text{tgt}}$ and $\mathbf{S}^{\text{pred}}$, we retain the $k$ highest-scoring key tiles for each query tile and mask out the rest, constructing the oracle and predicted binary tile masks $\widetilde{\mathbf{M}}^*$ and $\widetilde{\mathbf{M}}$.
The resulting mask is materialized by a tile-skipping sparse attention kernel to compute sparse attention outputs.
The backbone is trained under this sparse execution with the standard diffusion denoising objective $\mathcal{L}_{\text{diff}}$, while $\mathcal{L}_{\text{distill}}$ provides explicit supervision to improve tile-score estimation and the induced Top-$k$ selection.
Crucially, to ensure the estimator actively adapts to the backbone without perturbing the pre-trained generative manifold, we apply a stop-gradient (sg) operation on the backbone features feeding into the estimator. This decouples mask learning from feature learning, ensuring stable convergence. In our experiments, we observed that allowing the gradient to propagate back into the base model leads to a noticeable degradation in generation quality. This empirical finding indicates that implicitly coupling sparse mask learning with the diffusion objective is detrimental; forcing the generative backbone itself to learn how to perform sparse attention fundamentally disrupts its pre-trained representation capabilities.
\subsection{Head-aware Tiling Search}
\label{sec:head_aware_search}
Attention heads in Video DiTs exhibit pronounced heterogeneity in their spatiotemporal dependency patterns, so a single uniform tiling can be suboptimal under high sparsity.
Veda therefore assigns a per-layer-per-head tiling configuration $\pi_{l,h} = (p_t, p_h ,p_w)$: For each head $h$ at layer $l$, $\pi_{l,h}$ defines how the $N$ tokens are grouped into $N_T$ tiles over temporal, spatial-height, and spatial-width dimensions.
To remain hardware-efficient, we restrict $\pi$ to factorizations of the hardware tile size $B$ (e.g., $B{=}128$), and the configuration set $\Omega$ is:
\begin{equation}
\Omega \;=\;\{(p_t, p_h ,p_w)\in\mathbb{N}^3 \mid p_t p_h p_w = B\},
\end{equation}
We select $\pi_{l,h}$ offline on a calibration set by directly minimizing the approximation error of full-attention output $\mathbf{O}^\text{fu}$ after applying sparse attention.
For each candidate $\pi\in\Omega$ and calibration sample $x\sim\mathcal{D}_{\mathrm{cal}}$, we first tile the head-specific tokens according to $\pi$ and derive a reference Top-$k$ mask $\widetilde{\mathbf{M}}^*(x;\pi)$ from the full-attention map, following the score pooling and per-query Top-$k$ selection in Sec.~\ref{sec:veda41}.
We then apply $\widetilde{\mathbf{M}}^{*}(x;\pi)$ to execute tile-sparse attention and obtain the corresponding output $\mathbf{O}^{\mathrm{sp}}_{l,h}(x;\pi)$ after mapping tokens back to the original order, while $\mathbf{O}^{\mathrm{fu}}_{l,h}(x)$ denotes the full-attention output of the same head.
The optimal configuration is chosen as
\begin{equation}
\pi^{*}_{l,h}
=
\arg\min_{\pi\in\Omega}
\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{cal}}}
\left[
\left\|
\mathbf{O}^{\mathrm{fu}}_{l,h}(x)
-
\mathbf{O}^{\mathrm{sp}}_{l,h}(x;\pi)
\right\|_F^2
\right],
\end{equation}
which favors tilings that preserve the full-attention output most faithfully under the fixed Top-$k$ budget.
Algorithm~\ref{alg:dynamic_selection} summarizes the procedure.
% 4.3 Head-aware Tiling Search：具体怎么为每个 head 选择更合适的 (pt,ph,pw) 聚合方式
% 4.4 硬件的implementation
\subsection{Hardware Implementation}
\label{sec:hardware_impl}
\paragraph{Hardware-Efficient Tile-sparse Kernel.}
To translate the theoretical reduction in FLOPs into tangible wall-clock acceleration, we implement a highly optimized tile-skipping attention kernel leveraging the ThunderKittens DSL~\cite{thunder}. We address the irregular memory access patterns introduced by sparse attention by exploiting the asynchronous Tensor Memory Access (TMA) and Warp Specialization features of the NVIDIA Hopper~\cite{tensorcore}.
Our kernel decouples data movement from computation via a producer-consumer paradigm.
The \textit{producer} warps orchestrate TMA instructions to fetch only the selected non-contiguous key/value tiles from global memory into a circular shared memory buffer.
Simultaneously, \textit{consumer} warps execute tensor core operations (WGMMA) on the buffered data.
This design effectively hides the latency of sparse memory gathering behind dense matrix math.
Consequently, our kernel reaches approximately $80\%$ of the MFU achieved by the highly optimized FlashAttention-3~\cite{flashattn-3} on 480P, 81-frame resolution sequences (sequence length $L \approx 34\text{K}$).
% Max Pooling Kernel
\paragraph{Efficient Ground-Truth Heatmap Generation.}
Training the sparse predictor requires a ground-truth tile-wise heatmap derived from the full-attention distribution $\mathbf{A}$. Generating exact pooled scores without storing $\mathbf{A}$ is non-trivial because softmax normalization couples all keys within each query row. We address this with a TileLang~\cite{tiellang} kernel that computes pooled tile scores in two passes. The first pass performs tile-wise $\mathbf{Q}\mathbf{K}^\top$ in SRAM and writes per-tile unnormalized maxima together with running row statistics to global memory. The second pass finalizes the row statistics and normalizes the cached maxima to recover exact tile-wise scores. This implementation reaches $\sim0.9\times$ FlashAttention-3 throughput while producing the pooled supervision, and its tile-wise independence further supports \textit{partial query processing} by supervising only a random subset of query tiles. Such sparse supervision dramatically reduces the supervision overhead and accelerates the overall training process, all without substantially compromising the final model performance.
\section{Experiments}
% 我们在开源的state-of-art模型上测试了我们的方法，包括Wan2.1-T2V-1.3B/Wan2.1-T2V-14B，以及SOTA的闭源模型Waver 1.0 Waver-1B/Waver-14B，生成480P 81Frame with total sequence length of 34020 (wan & waver 1b)，以及720P 81Frame (on wan2.1-t2v-14b) with total sequence length of 75600, 720P 241Frame(on waver-14b) with total sequence length of 219600. 这覆盖了目前cross attention，以及mmdit两种主流的video diffusion model架构。对于Veda，我们首先freeze backbone，我们对projector使用per-head的xaiver uniform初始化，同时仅训练projector权重1000步，Lr=6e-4。在Projector完成初始化之后，我们打开全部的权重，继续训练同样的步数。对于both初始化projector和后续sparse微调过程中，我们都直接使用目标预设的sparsity，不存在sparsity warmup的过程。在下面所有的比较中，对于1B/1.3B模型，我们训练2w3步后，使用ema的权重推理，ema=0.9999，使用Waver-bench1.0作为human evaluiton的评测prompt set，共包含304条经过rewrite之后的高动态prompt。从prompt follwing, visual quality, motion qulity与overal qulity四个维度上评估。同时我们也使用VBench prompt set，在vbench上测试我们生成的效果。所有的速度都在同一系列GPU上，使用相同的sparse attention kernel得到。 数据集是内部数据集，包含所有常见的场景，我们使用Full attention作为baseline，full attention也使用完全一样的数据集，训练相同的步骤。对于需要后训练的方法，我们也训练相同的步骤后比较，使用完全一样的lr。除了FA，我们与现在SOTA的固定mask和动态mask，以及混合linear attn和动态mask的方法比较，包括SVG，VSA 与SLA。
\begin{figure*}[!t]
\centering
\includegraphics[width=0.95\linewidth]{visuals/human_eval_main.pdf}
\vspace{-0.2cm}
\caption{\textbf{Human evaluation on Waver-bench 1.0 using Waver-T2V-1B.}
We compare \textbf{Veda} against \textbf{Full Attention} and \textbf{VSA}.
Videos are generated at 480P, 81 frames (sequence length $\approx$ 34k).
Veda achieves a parity win/tie rate against Full Attention at 90\% sparsity.
Notably, Veda at 95\% sparsity yields superior results compared to VSA even when VSA operates at a significantly lower sparsity (87.5\%), and demonstrates a substantial performance margin over VSA at equivalent (95\%) sparsity levels.}
\label{fig:human_eval_main}
\end{figure*}
\paragraph{Implementation Details.}
% We evaluate Veda using the Waver-T2V (1B/12B)~\cite{waver} and Wan2.1-T2V (1.3B/14B)~\cite{wan} architectures. To assess scalability, we conduct experiments across varying resolutions and token counts: 480P (81 frames, 34,020 tokens) for the 1B/1.3B models; 720P (81 frames, 75,600 tokens) for Wan2.1-14B; and 720P (241 frames, 219,600 tokens) for Waver-T2V-12B. All models are trained on an internal dataset encompassing a broad spectrum of general scenes. For Veda, we employ a two-stage training protocol. In the first stage, the backbone is frozen while the projector is optimized for 1,000 steps using Xavier uniform initialization and a learning rate of $6 \times 10^{-4}$ to align the sparse predictor. In the second stage, we unfreeze all parameters and perform sparse fine-tuning at the target sparsity level using a learning rate of $6 \times 10^{-5}$ for backbone and $6 \times 10^{-4}$ for sparse predictor. Across both stages, the sparsity ratio is maintained at a fixed value without warmup. For experiments at the 1B/1.3B scale, models are trained for 23k steps, and we report results using Exponential Moving Average (EMA) weights with a decay of 0.9999. Conversely, for the 14B scale, models are trained for 10k steps, and we report results using the final backbone weights.
We evaluate Veda using the Waver-T2V (1B/12B)~\cite{waver} and Wan2.1-T2V (1.3B/14B)~\cite{wan} architectures. To assess scalability, we conduct experiments across varying resolutions and token counts: 480P (81 frames, 34,020 tokens) for the 1B/1.3B models; 720P (81 frames, 75,600 tokens) for Wan2.1-14B; and 720P (241 frames, 219,600 tokens) for Waver-T2V-12B. All models are trained on an internal dataset encompassing a broad spectrum of general scenes. For Veda, we employ a two-stage training protocol. In the first stage, the backbone is frozen while the projector is optimized for 1,000 steps using Xavier uniform initialization and a learning rate of $6 \times 10^{-4}$ to align the sparse predictor. In the second stage, we unfreeze all parameters and perform sparse fine-tuning at the target sparsity level using a learning rate of $6 \times 10^{-5}$ for backbone and $6 \times 10^{-4}$ for sparse predictor. Unlike prior dynamic sparse methods that require a handcrafted, slow sparsity warm-up to prevent training collapse, Veda's Stage 1 explicitly decouples mask learning from feature learning via a stop-gradient operation. This provides a highly stable initialization, allowing Stage 2 to rapidly adapt to target sparsity levels without fragile scheduling, and we therefore maintain the sparsity ratio at a fixed value without warmup across both stages. For experiments at the 1B/1.3B scale, models are trained for 23k steps, and we report results using Exponential Moving Average (EMA) weights with a decay of 0.9999.
\paragraph{Evaluation Metrics.}
We assess generation performance through a combination of human evaluation and standardized benchmarks. For human evaluation, we utilize Waver-bench 1.0~\cite{waver}, a curated dataset of 304 prompts that cover a wide range of scenarios. Complementary to human ratings, we report quantitative metrics evaluated on the VBench~\cite{vbench} suite. To ensure fair comparison, all latency and speedup measurements are conducted on identical GPU hardware; dense baselines use FlashAttention-3, and sparse methods use the same tile-skipping sparse attention kernel.
\paragraph{Comparison with Existing Baselines.}
We benchmark Veda against Full Attention, Oracle mask and VSA~\cite{vsa}. To ensure a fair comparison, all methods are trained on the identical dataset and number of steps as our approach, utilizing the specific warm-up settings proposed in their respective literature. Similarly, for all post-training sparse baselines, we strictly adhere to an equivalent training budget and learning rate schedule.
\subsection{Generation Quality Comparison}
% We evaluate generation quality primarily through a blinded Side-by-Side (SBS) human preference study on the Waver-bench 1.0 dataset. Evaluators compared pairs of generated videos across four distinct dimensions: Overall Quality, Motion Quality, Visual Quality, and Prompt Following. The results are shown in Fig.~\ref{fig:human_eval_main}.
% We first examine the performance of Veda at 90\% sparsity relative to the Full Attention baseline. The human evaluation results indicate that Veda achieves perceptual parity across all four assessed dimensions. In terms of Visual Quality and Prompt Following, the preference rates between our method and the dense baseline are nearly identical (e.g., 50\% vs. 49\% for Visual Quality). Furthermore, for Motion Quality, we observe a substantial tie rate of 63\% with balanced residual preferences, suggesting that temporal consistency is strictly maintained without introducing artifacts. Consequently, the Overall Quality of samples generated by Veda remains statistically indistinguishable from that of the full model, demonstrating that the proposed method effectively preserves generative capabilities despite aggressive token pruning.
% The effectiveness of our distillation-based approach is further highlighted when comparing Veda against VSA~\cite{vsa}, a state-of-the-art dynamic sparse attention method. We observe that Veda outperforms VSA even when operating under a stricter sparsity constraint. Specifically, Veda at 95\% sparsity achieves a 54\% preference rate in Visual Quality compared to 46\% for VSA at 87.5\% sparsity. When both methods are constrained to 95\% sparsity, the performance gap widens significantly; Veda secures a 76\% preference rate in Visual Quality versus 24\% for VSA. In terms of Overall Quality, Veda (95\%) outperforms VSA (95\%) with a win rate of 39\% against 16\%. These results substantiate that the explicit tile-level distillation allows the model to retain information-dense regions that are typically missed by heuristic pooling strategies used in prior works.
\begin{figure}[!b]
\centering
\includegraphics[width=1.0\linewidth]{visuals/waver_wan_accl_compare_stacked.pdf}
\caption{\textbf{Wall-clock latency decomposition and speedup analysis.} We measure the end-to-end per-layer inference latency of Veda against a FlashAttention-3 baseline on Waver-T2V and Wan2.1-T2V at two resolutions (720p and 480p). Stacked bars break down the runtime into MLP computation, sparse mask preparation (\emph{Prepare Mask}), and attention kernel execution. Veda delivers substantial speedups, reaching up to $4.03\times$ on Waver-T2V and $2.64\times$ on Wan2.1-T2V at 720p.}\label{fig:waver_wan_accl_compare}
\vspace{-0.3cm}
\end{figure}
\begin{figure*}[!t]
\centering
\includegraphics[width=1.0\textwidth]{visuals/qualitative_study.pdf}
\caption{\textbf{Qualitative results of Waver-T2V-12B with 95\% sparsity using Veda.} Two representative samples are shown, each synthesized at 720P resolution with 241 frames. The text prompt for each sample is shown beneath the corresponding image. More qualitative comparisons are included in the supplementary material.}
\label{fig:qualitative_study}
\vspace{0.2cm}
\end{figure*}
\begin{table*}[!h]
\centering
\caption{\textbf{VBench evaluation of Veda against full attention and state-of-the-art sparsification methods on video diffusion models.}
We compare Veda against Full Attention and state-of-the-art sparsification techniques on Waver-T2V and Wan2.1-T2V.
\textbf{Sparsity} indicates the reduction in computation.
``Full Attn.'' serves as the dense baseline.
$\uparrow$ and $\downarrow$ denote better performance.}
\label{tab:vbench_main}
\resizebox{\textwidth}{!}{
\begin{tabular}{lll ccccccccc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Config} & \multirow{2}{*}{Method} &
Subject & Background & Motion & Dynamic & Aesthetic & Image &
\multirow{2}{*}{Sparsity $\uparrow$} & \multirow{2}{*}{FLOPs $\downarrow$} & \multirow{2}{*}{Wall Time $\downarrow$} \\
& & & Cons. $\uparrow$ & Cons. $\uparrow$ & Smooth. $\uparrow$ & Deg. $\uparrow$ & Quality $\uparrow$ & Quality $\uparrow$ & & & \\
\midrule
% --- Waver 1.0 Block ---
\multirow{6}{*}{Waver 1.0} & \multirow{6}{*}{\shortstack{1B\\480P\\81F}}
& Full Attn. & 0.938 & 0.955 & 0.979 & 0.969 & 0.655 & 0.693 & 0\% & $4.83 \times 10^{16}$ & 69.3s \\
& & Oracle Mask & 0.907 & 0.950 & 0.974 & 0.993 & 0.639 & 0.685 & 90\% & -- & 126.3s \\
& & VSA & 0.933 & 0.949 & 0.978 & 0.983 & 0.648 & 0.692 & 87.5\% & $1.56 \times 10^{16}$ & 34.3s \\
& & \textbf{Ours} (S=90\%) & 0.940 & 0.954 & 0.980 & 0.963 & 0.650 & 0.699 & 90\% & $1.47 \times 10^{16}$ & 31.9s \\
& & \textbf{Ours} (S=95\%) & 0.934 & 0.951 & 0.978 & 0.979 & 0.649 & 0.698 & 95\% & $1.28 \times 10^{16}$ & 30.6s \\
% \cmidrule{2-12}
\midrule
% --- Wan2.1 Block ---
\multirow{6}{*}{Wan2.1} & \multirow{6}{*}{\shortstack{1.3B\\480P\\81F}}
& Full Attn. & 0.940 & 0.969 & 0.977 & 0.844 & 0.629 & 0.670 & 0\% & $5.82 \times 10^{16}$ & 58.5s \\
& & Oracle Mask & 0.957 & 0.968 & 0.931 & 0.861 & 0.325 & 0.642 & 90\% & -- & 94.6s \\
& & VSA & 0.911 & 0.950 & 0.975 & 0.795 & 0.549 & 0.661 & 87.5\% & $2.55 \times 10^{16}$ & 42.5s \\
& & \textbf{Ours} (S=90\%) & 0.887 & 0.941 & 0.972 & 0.913 & 0.543 & 0.663 & 90\% & $2.46 \times 10^{16}$ & 37.6s \\
& & \textbf{Ours} (S=95\%) & 0.790 & 0.926 & 0.928 & 0.733 & 0.394 & 0.661 & 95\% & $2.27 \times 10^{16}$ & 35.8s \\
% \cmidrule{2-12}
\bottomrule
\end{tabular}
}
\end{table*}
We evaluate generation quality via a blinded Side-by-Side (SBS) study on Waver-bench 1.0 across four dimensions: \textit{Overall Quality} (OQ), \textit{Motion Quality} (MQ), \textit{Visual Quality} (VQ), and \textit{Prompt Following} (PF). Results are shown in Fig.~\ref{fig:human_eval_main}.
Comparing Veda (90\% sparsity) to the Full Attention baseline, we observe perceptual parity. Preference rates for VQ and PF are nearly identical (e.g., 50\% vs. 49\% for VQ), while a 63\% tie rate in MQ confirms temporal consistency is maintained. Consequently, OQ remains indistinguishable from the full model, proving Veda preserves generative capabilities despite aggressive pruning.
Against VSA~\cite{vsa}, Veda excels even under stricter constraints. Veda (95\% sparsity) surpasses VSA (87.5\% sparsity) in VQ (54\% vs. 46\%). At equal 95\% sparsity, Veda dominates VQ (76\% vs. 24\%) and OQ (39\% vs. 16\%). This validates that our distillation strategy retains information-dense regions better than prior heuristic pooling methods.
In addition to human evaluation, we provide a quantitative analysis using the VBench suite in Table~\ref{tab:vbench_main}. The results demonstrate that Veda effectively retains key generative capabilities despite the significant reduction in computation. On the Waver-T2V architecture, our method maintains subject consistency and motion smoothness scores that are comparable to the Full Attention baseline, even at 95\% sparsity. Across differing model scales, Veda consistently exhibits a favorable efficiency-quality trade-off relative to the VSA baseline, delivering competitive metric performance while achieving lower FLOPs and reduced wall-clock latency.
\subsection{Inference Efficiency Comparison}
We analyze wall-clock latency on NVIDIA Hopper GPUs to quantify practical speedups over FlashAttention-3 (FA3)~\cite{flashattn-3}. We report the end-to-end latency of a single Transformer layer, including both the sparse mask predictor overhead and the sparse attention kernel execution.
In Fig.~\ref{fig:waver_wan_accl_compare}, for Waver-T2V-12B at 720P with 121 frames, a dense Transformer layer takes 315.3 ms, while Veda reduces it to 78.3 ms (4.03$\times$). The Veda breakdown is 39.5 ms for MLP computation, 1.9 ms for sparse mask preparation, and 36.9 ms for sparse attention execution, so the combined sparse-attention overhead remains well below dense attention. We observe a similar trend on Wan2.1-T2V-14B at 720P with 81 frames: layer latency decreases from 583.7 ms to 220.7 ms (2.64$\times$), demonstrating consistent gains across backbones with different architectures and sequence configurations.
We further evaluate scalability with sequence length. As shown in Fig.~\ref{fig:teaser}, at 50,220 tokens (480P, 121 frames), Veda achieves a 2.57$\times$ speedup over FA3 (25.5 ms vs. 65.6 ms). When scaling to 245,760 tokens (720P, 241 frames, SP=8), FA3 grows quadratically to 1576.5 ms, whereas Veda scales near-linearly and remains at 309.1 ms, yielding a 5.1$\times$ speedup. Overall, these results confirm that Veda mitigates the quadratic attention cost and delivers robust wall-clock improvements for high-resolution, long-sequence video generation.
\subsection{Ablation Study}
% Head-aware 3D Block Tiling
% 不同的tilesize和dynamic tilesize ✅
% 可以有一个简单的stats说具体dynamic tilesize怎么选的
%
\paragraph{Ablation on Head-aware Tiling.}
We compare various static tile configurations against our Head-aware Tiling strategy. The static baselines include configurations emphasizing spatial granularity ($[8, 8, 2]$), temporal granularity ($[4, 4, 8]$), and alternative spatial-temporal trade-offs ($[4, 8, 4]$). Among these fixed configurations, $[4, 4, 8]$ demonstrates the strongest performance and serves as the primary static baseline for comparison. We evaluate our Head-aware Tiling method against this optimal static configuration ($[4, 4, 8]$). As shown in Fig.~\ref{fig:dyntile_human_eval}, our approach achieves consistent improvements across all metrics, with notable gains of +7.2\% in \textit{Motion Quality} and +9.6\% in \textit{Overall Quality}. These results demonstrate the superiority of head-aware tiling over uniform static tiling strategies.
% Block score estimator
% 不同的pooling方法（avg， max min only, triplet) TODO（TBD）
\begin{table}[h]
\centering
\captionsetup{font=footnotesize}
\caption{Ablation on tile statistics for the score estimator. Results report the average training loss. \textit{Triplet pooling} (mean, maximum, minimum) best reconstructs the oracle attention structure.}
\label{tab:ablation_pooling}
\vspace{-1mm}
\footnotesize
\setlength{\tabcolsep}{10pt}
\renewcommand{\arraystretch}{0.92}
\begin{tabular}{l c}
\toprule
\textbf{Pooling Strategy} & \textbf{Training Loss ($\downarrow$)} \\
\midrule
MaxMin + Projector & 0.982 \\
Avg + Projector & 0.965 \\
\textbf{Triplet + Projector (Ours)} & \textbf{0.912} \\
\bottomrule
\end{tabular}
\vspace{-2mm}
\end{table}
\paragraph{Ablation on Tile Score Estimator.}
We evaluate three pooling schemes for compressing token-wise features into tile-level representations: (1) \textit{Avg}, (2) \textit{MaxMin} (concatenating maximum and minimum), and (3) \textit{Triplet} (concatenating mean, maximum, and minimum). We train the score estimator with the diffusion backbone frozen and report the mean squared error (MSE) to oracle full-attention tile scores. As shown in Table~\ref{tab:ablation_pooling}, \textit{Triplet} achieves the lowest MSE ($0.912$), outperforming \textit{Avg} and \textit{MaxMin}.
\subsection{Qualitative Results}
Fig.~\ref{fig:qualitative_study} illustrates qualitative generation results of Waver-T2V-12B at 720p resolution for 241-frame sequences under 95\% sparsity. Despite the extreme sparsity (95\%), Veda maintains high visual quality and coherent motion, without noticeable sparse-induced structural artifacts (e.g., warping, water-ripple patterns, or temporal flickering). More qualitative examples are included in the Supplementary Material.
\section{Conclusion and Future Work}
We presented Veda, a hardware-aware sparse attention framework that addresses the scalability bottleneck of high-resolution video Diffusion Transformers. Veda reformulates sparsity as an explicit tile selection problem and supervises tile scoring via full-attention distillation, rather than relying on the diffusion objective to implicitly shape sparse patterns. The induced Top-$k$ masks preserve the tile-wise structure and ranking of full attention even under extreme pruning. By combining a statistics-aware tile score estimator and Head-Aware Tiling with a custom tile-skipping kernel, Veda translates algorithmic sparsity into practical wall-clock gains, achieving up to $5.1\times$ end-to-end speedup on Waver and Wan2.1 with no noticeable degradation in generation quality, while exhibiting favorable scaling as spatiotemporal sequence length increases.
Looking forward, further improvements may come from tighter kernel-level fusion to reduce mask preparation overhead, more advanced scheduling to sustain high MFU at sparsity levels beyond $95\%$, and richer distillation signals or adaptive sparsity strategies that allocate computation across timesteps and heads. Incorporating lightweight temporal caching of tile scores across adjacent diffusion steps could further improve stability.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% APPENDIX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
% Appendi可以补充的内容
% 1. 随不同场景，prompt，pattern的变化 （ "Head-Aware Tiling" 的泛化能力）
% 为了补充更多关于attn head行为的观察，以及对于pattern的繁华能力的评估，我们补充下面的比较实验。下面实验使用相同的预训练模型，相同seed，但是使用四种专门设计的prompt，在这四种prompt的设计中我们尽量保证主体物体完全一致：完全静止；物体运动（场景变化不大）；主体静止，场景移动；转场。
% 下面展示了具体采样的视频帧
% 下面展示了四个有代表性的head采样的结果
% 我们发现，对于softmax attention而言，在video diffusion model里，高速运动的物体并不一定代表着更“dense”的attention行为，反而，由于物体运动，帧前后的连接关系减弱，反而使得attn的heatmap行为更稀疏。但是我们可以发现，在不同场景和prompt下，这些attn heat维持了相同的时空关系的pattern，只是activation有程度的变化，他们的pattern并没有发生根本性改变。
% 2. 具体的训练成本与部署门槛
% 3. Estimator 的开销与延迟掩盖
% 4. 是否gradback loss到backbone
% 5. ablate on监督方法使用KL Loss，MSE Loss TODO （TBD）
% 6. Train from scratch的实验（如退化到VSA）
% 7. Experiment on Dynamic Sparsity
% 8. Partial-Q加速训练
% TODO:
% therorical framework部分要统一符号，以及补充过程
% \textbf{Gradient Isolation.}
% We further investigate whether the diffusion backbone should be jointly optimized with the sparse estimator. Our experiments reveal that back-propagating the mask prediction gradients into the backbone leads to training divergence. We hypothesize that predicting sparse patterns (a discrimination task) and video generation (a denoising task) represent fundamentally conflicting objectives for shared weights. Consequently, forcing the backbone to adapt to the sparsity constraint compromises the stability of the diffusion process. Based on this observation, we employ a stop-gradient operation, freezing the backbone during estimator training to decouple these objectives.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Theoretical Framework for Veda}
In this section, we establish the theoretical foundations of our approach and provide deeper algorithmic and mathematical insights into the nature of sparse attention.
\subsection{The Necessity of Accurate Oracle Masks}
The self-attention mechanism can be formally viewed as an Energy-Based Model (EBM)\cite{hopfield,litman}. Let $\mathbf{u} \in \mathbb{R}^N$ denote the logits (pre-softmax scores), where $u_j = \mathbf{q}^T \mathbf{k}_j$. The generating potential function of the attention distribution is the Log-Sum-Exp (LSE) function\cite{litman}, $\Psi(\mathbf{u})$:
\begin{equation}
\Psi(\mathbf{u}) = \tau \log \sum_{j=1}^N \exp(u_j / \tau),
\end{equation}
where $\tau$ is the temperature parameter. The attention probability distribution $P$ is the gradient of this potential, $P = \nabla \Psi(\mathbf{u})$. The LSE function is dominated by the maximum term, $u_{max}$.
\begin{proposition}[Distributional Shift via Mask Error]
Let $\mathcal{M}^*$ be the set of indices corresponding to the true Top-$k$ keys (Oracle), and let $\hat{\mathcal{M}}$ be an estimated mask such that $\operatorname{argmax}(\mathbf{u}) \notin \hat{\mathcal{M}}$. The perturbed potential $\tilde{\Psi}$ excludes the dominant energy term. This exclusion forces a renormalization of probability mass over the retained keys $\hat{\mathcal{M}}$:
\begin{equation}
\tilde{P}_j = \frac{\exp(u_j/\tau)}{\sum_{r \in \hat{\mathcal{M}}} \exp(u_r/\tau)}
= \frac{Z}{\hat{Z}} P_j > P_j, \quad \forall j \in \hat{\mathcal{M}},
\end{equation}
where $Z=\sum_{r=1}^N \exp(u_r/\tau)$ and $\hat{Z}=\sum_{r \in \hat{\mathcal{M}}} \exp(u_r/\tau)$.
\end{proposition}
This renormalization can assign inflated confidence to noise or irrelevant tokens, a phenomenon we term \textit{structural hallucination}. Unlike linear operations, where errors are additive, the softmax nonlinearity can amplify mask errors through exponentiated logits and row-wise renormalization. Thus, accurate recovery of the Oracle Mask is a structural prerequisite for high-sparsity attention.
\subsection{Limitations of Linear Pooling Representations}
Existing methods like VSA\cite{vsa} or Sparge Attention\cite{spargeattention} often employ Average Pooling to compress tiles. We argue that linear pooling is theoretically insufficient for approximating the bilinear attention operation.
From a spectral perspective, Average Pooling is a low-pass filter equivalent to a projection matrix $W_{avg}$. The approximated score becomes $\hat{u}_{ij} = \mathbf{q}_i^T W_{avg}^T W_{avg} \mathbf{k}_j$.
\begin{lemma}[Loss of Second-Order Statistics]
The inner product $\langle \mathbf{q}, \mathbf{k} \rangle$ depends on both the angle and magnitude of the high-frequency components of the vectors. The projection $W_{avg}$ collapses the vector variance onto the mean, discarding the high-frequency harmonics required to distinguish orthogonal vectors. Mathematically, $\langle \bar{\mathbf{q}}, \bar{\mathbf{k}} \rangle \neq \overline{\langle \mathbf{q}, \mathbf{k} \rangle}$. The loss of orthogonality leads to rank deficiency in the approximated attention matrix, making it impossible to distinguish between distinct but mean-similar tiles.
\end{lemma}
\subsection{Reconstructing Attention via Sufficient Statistics}
To reconstruct the attention distribution accurately with minimal I/O, we propose a statistical estimator based on vector decomposition. We treat the elements of dimension-$d$ vectors $\mathbf{q}$ and $\mathbf{k}$ as samples from a distribution.
\begin{theorem}[Dot Product Decomposition]
The dot product of two vectors $\mathbf{q}, \mathbf{k} \in \mathbb{R}^d$ can be decomposed into a mean product term and a covariance term:
\begin{equation}
\mathbf{q} \cdot \mathbf{k} = d \mu_q \mu_k + d \operatorname{Cov}(\mathbf{q}, \mathbf{k}),
\end{equation}
where $\operatorname{Cov}(\mathbf{q}, \mathbf{k}) = \rho \sigma_q \sigma_k$.
\end{theorem}
While $\mu$ (Avg) captures the centroid, reconstructing the covariance requires information about the spread ($\sigma$) and correlation ($\rho$). We employ the Bhatia-Davis inequality to bound the variance using only extremum statistics.
\begin{lemma}[Variance Bound via Extrema]
For any bounded random variable $X$ with support $[m, M]$ and mean $\mu$, the variance is bounded by:
\begin{equation}
\sigma^2 \le (M - \mu)(\mu - m).
\end{equation}
\end{lemma}
Assuming the vector elements follow a unimodal distribution (e.g., Gaussian or Laplacian, typical in LayerNorm-normalized embeddings), the standard deviation is proportional to the range: $\sigma \approx \frac{M-m}{\alpha}$. By retaining the triplet $\{ \mu, m, M \}$ (Avg, Min, Max), we effectively construct a \textit{hyper-rectangle bounding box} that constrains the manifold of the vectors.
We derive our estimator $\mathcal{S}(\mathbf{q}, \mathbf{k})$ as:
\begin{equation}
\mathcal{S} \approx d \cdot \mu_q \mu_k + \lambda \Delta_q \Delta_k
\end{equation}
where $\Delta_x = \sqrt{(M_x-\mu_x)(\mu_x-m_x)}$ represents the dispersion bound for $x \in \{q, k\}$. Here, the first term anchors the centroid interaction, while the second term approximates the maximal energy potential of the covariance.
\subsection{The Necessity of Intra-Tile Similarity}
We now justify why tokens within a tile should be similar for this approximation to hold. Consider tile-sparse attention where a single statistic represents a tile of size $B$. Let $\boldsymbol{\mu}_{\mathbf{q}}$ and $\boldsymbol{\mu}_{\mathbf{k}}$ denote the tile means, and define the residual vectors $\mathbf{r}^{q}_i=\mathbf{q}_i-\boldsymbol{\mu}_{\mathbf{q}}$ and $\mathbf{r}^{k}_i=\mathbf{k}_i-\boldsymbol{\mu}_{\mathbf{k}}$. The approximation error $E$ from the neglected residual interactions can be written as:
\begin{equation}
E = \left| \sum_{i=1}^B \langle \mathbf{r}^{q}_i, \mathbf{r}^{k}_i \rangle \right|.
\end{equation}
By the Cauchy-Schwarz inequality, this error is bounded by the product of the tile-wise standard deviations:
\begin{equation}
E \le
\left(\sum_{i=1}^B \|\mathbf{r}^{q}_i\|_2^2\right)^{1/2}
\left(\sum_{i=1}^B \|\mathbf{r}^{k}_i\|_2^2\right)^{1/2}
= B \cdot \sigma_{\mathbf{q}} \cdot \sigma_{\mathbf{k}},
\end{equation}
where $\sigma_{\mathbf{q}}^2=\frac{1}{B}\sum_{i=1}^B \|\mathbf{r}^{q}_i\|_2^2$ and $\sigma_{\mathbf{k}}^2=\frac{1}{B}\sum_{i=1}^B \|\mathbf{r}^{k}_i\|_2^2$.
\begin{corollary}[Clustering Condition]
A sufficient condition for the sparse approximation to be robust (i.e., $E \to 0$) is $\sigma_{\mathbf{q}} \to 0$ or $\sigma_{\mathbf{k}} \to 0$, and reducing both variances tightens the bound. This implies that tokens within a tile should lie on a local low-dimensional manifold with minimal variance. If tiles contain diverse tokens (high $\sigma$), the uncertainty bound of the estimator grows, rendering the sparse mask unreliable.
\end{corollary}
\subsection{Token Clustering as Bandwidth Minimization}
To satisfy the condition $\sigma \to 0$, we formulate token clustering as a graph labeling problem. Let $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ be the token graph where edge weights $W_{ij} \propto \exp(\mathbf{q}_i^T \mathbf{k}_j)$. We seek a bijective mapping $\phi: \mathcal{V} \rightarrow \{1, \dots, N\}$ that minimizes the matrix bandwidth.
\begin{definition}[Manifold Embedding under 3D RoPE]
Let tokens $x_i$ reside on a 3D spatio-temporal manifold $\mathcal{M}$ induced by Rotary Positional Embeddings (RoPE). The attention affinity decays with the geodesic distance on this manifold: $W_{ij} \approx f(\|\mathbf{x}_i - \mathbf{x}_j\|_{\Sigma})$. A locality-preserving mapping $\phi$ must satisfy the H\"older continuity condition:
\begin{equation}
|\phi(u) - \phi(v)| \le C \cdot \|\mathbf{x}_u - \mathbf{x}_v\|_2^\alpha.
\end{equation}
\end{definition}
Standard raster scanning ($\phi_{\text{flat}}$) violates this condition for temporal neighbors, yielding a Lipschitz constant $C \propto H \times W$, resulting in a dispersed, high-bandwidth attention matrix. We propose \textbf{Head-aware 3D Tiling}, which creates a mapping $\phi_{\text{tile}}$ where temporal neighbors satisfy $|\phi_{\text{tile}}(u) - \phi_{\text{tile}}(v)| \le p_h p_w$. This effectively folds the 3D manifold into 1D memory while preserving local neighborhoods, implicitly performing graph clustering. Furthermore, we employ \textit{Anisotropic Metric Adaptation} to search for optimal tile shapes $(p_t, p_h, p_w)$, aligning the tiling strategy with the specific induced metric $d_h(\mathbf{x}_i, \mathbf{x}_j)$ of each attention head. This topological optimization minimizes attention bandwidth, effectively clustering high-affinity tokens to lower intra-tile variance $\sigma$. Consequently, the geometric regularization satisfies the spectral approximation conditions, validating the representational power of the compressed statistics $\{\mu, m, M\}$.
\subsection{Projector Optimization via Information Geometry}
Finally, we address the non-differentiability of the Top-$k$ selection required to train the MLP projector. We leverage the framework of Entropic Optimal Transport (EOT) \citep{litman}.
The attention mechanism operates on a statistical manifold equipped with the Fisher-Rao metric. The standard Euclidean gradient is suboptimal for optimization on the probability simplex.
\begin{theorem}[Local KL Geometry]
Let $P_{\mathrm{oracle}}$ be the ground truth distribution derived from full attention, and $P_{\mathrm{pred}}$ be the distribution predicted by the MLP. Around the oracle parameters $\theta_t$, the Kullback-Leibler (KL) divergence admits the local second-order approximation:
\begin{equation}
\mathcal{L} = D_{\mathrm{KL}}(P_{\mathrm{oracle}} \| P_{\mathrm{pred}}) \approx \frac{1}{2} (\theta_t - \theta_s)^T I(\theta_t) (\theta_t - \theta_s).
\end{equation}
Thus, KL supervision locally weights prediction errors by the Fisher Information Matrix $I(\theta_t)$, matching the natural geometry of the probability simplex.
\end{theorem}
By utilizing KL divergence as the training objective, we train the score estimator before the discrete Top-$k$ operation and therefore avoid differentiating through Top-$k$. Instead, we force the MLP to learn the \textit{geometry of the energy landscape} (the LSE potential field). This encourages the predicted logits $\hat{\mathbf{u}}$ to maintain the same ranking and local geometry as the oracle logits $\mathbf{u}$, leading to accurate mask recall during inference.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Detailed Implementation Algorithms for Veda}
We provide the complete algorithmic pseudocode for the core components of Veda, which are omitted from the main text due to space constraints below.
\begin{algorithm}[H]
\caption{Head-Aware Tiling Search}
\label{alg:dynamic_selection}
\begin{algorithmic}[1]
\STATE {\bfseries Input:} Latent dims $(T, H, W)$, Calibration Set $\mathcal{D}_{\mathrm{cal}}$, Tile Size $B=128$, Top-$k$ Budget $k_{\mathrm{top}}$.
\STATE {\bfseries Output:} Optimal Configuration Map $\boldsymbol{\pi}^* \in \Omega^{L \times N_h}$.
\STATE \textcolor{gray}{// Construct hardware-aligned search space}
\STATE $\Omega \leftarrow \{(p_t, p_h, p_w) \in \mathbb{N}^3 \mid p_t p_h p_w = B\}$
\STATE Initialize cumulative error $\mathcal{E} \in \mathbb{R}^{L \times N_h \times |\Omega|}$ to $\mathbf{0}$.
\FOR{sample $x \in \mathcal{D}_{\mathrm{cal}}$}
\STATE Perform inference to cache $\{\mathbf{Q}, \mathbf{K}, \mathbf{V}\}$
\FOR{layer $l \in \{1 \dots L\}$, head $h \in \{1 \dots N_h\}$}
\STATE \textcolor{gray}{// Get Full-Attention Map and Output}
\STATE $\mathbf{A}^{\mathrm{fu}}_{l,h} \leftarrow \text{Softmax}(\mathbf{Q}_{l,h}\mathbf{K}_{l,h}^\top / \sqrt{d})$
\STATE $\mathbf{O}^{\mathrm{fu}}_{l,h} \leftarrow \mathbf{A}^{\mathrm{fu}}_{l,h}\mathbf{V}_{l,h}$
\FOR{$\pi \in \Omega$}
\STATE \textcolor{gray}{// 1. Tiling QKV and Full-Attention Map}
\STATE $\tilde{\mathbf{Q}}, \tilde{\mathbf{K}}, \tilde{\mathbf{V}} \leftarrow \text{Tiling}(\{\mathbf{Q}_{l,h}, \mathbf{K}_{l,h}, \mathbf{V}_{l,h}\}, \pi)$
\STATE $\mathbf{A}^{\pi} \leftarrow \text{Softmax}(\tilde{\mathbf{Q}}\tilde{\mathbf{K}}^\top / \sqrt{d})$
\STATE \textcolor{gray}{// 2. Generate Oracle Mask from Full Attention}
\STATE $\mathbf{S}^{\text{tile}} \leftarrow \text{RowNorm}(\text{MaxPool}(\mathbf{A}^{\pi}))$
\STATE $\widetilde{\mathbf{M}} \leftarrow \text{Top-}k(\mathbf{S}^{\text{tile}}, k_{\mathrm{top}})$
\STATE \textcolor{gray}{// 3. Compute Sparse Output and Error}
\STATE $\mathbf{O}^{\mathrm{sp}}_{l,h} \leftarrow \text{UnTile}(\text{SparseAttn}(\tilde{\mathbf{Q}}, \tilde{\mathbf{K}}, \tilde{\mathbf{V}}, \widetilde{\mathbf{M}}), \pi)$
\STATE $\mathcal{E}_{l, h, \pi} \leftarrow \mathcal{E}_{l, h, \pi} + \| \mathbf{O}^{\mathrm{fu}}_{l,h} - \mathbf{O}^{\mathrm{sp}}_{l,h} \|_F^2$
\ENDFOR
\ENDFOR
\ENDFOR
\STATE \textcolor{gray}{// Select configuration minimizing expected error}
\STATE $\pi^{*}_{l,h} \leftarrow \arg\min_{\pi \in \Omega} \mathcal{E}_{l, h, \pi} \quad \forall l, h$
\STATE \textbf{return} $\boldsymbol{\pi}^*$
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[H]
\caption{Tile Score Estimation \& Mask Generation}
\label{alg:estimator}
\begin{algorithmic}[1]
\STATE {\bfseries Input:} Query $\mathbf{Q}$, Key $\mathbf{K} \in \mathbb{R}^{N_h \times N \times d}$, Tile Size $B$, Top-$k$ Budget $k_{\mathrm{top}}$, Latent Dim $d'$
\STATE {\bfseries Output:} Predicted distribution $\mathbf{A}^{\mathrm{pred}}$ and binary mask $\widetilde{\mathbf{M}} \in \{0, 1\}^{N_h \times T_q \times T_k}$
\STATE \textcolor{gray}{// Tiling}
\STATE $T_q \leftarrow N / B, \quad T_k \leftarrow N / B$
\STATE $\tilde{\mathbf{Q}} \leftarrow \text{Tile}(\mathbf{Q}, \{N_h, T_q, B, d\})$
\STATE $\tilde{\mathbf{K}} \leftarrow \text{Tile}(\mathbf{K}, \{N_h, T_k, B, d\})$
\STATE \textcolor{gray}{// Triplet Statistics Extraction (TripPool)}
\STATE $\mathbf{Z}_q \leftarrow \mathrm{Avg}[\tilde{\mathbf{Q}}] \oplus \mathrm{Max}[\tilde{\mathbf{Q}}] \oplus \mathrm{Min}[\tilde{\mathbf{Q}}]$
\STATE $\mathbf{Z}_k \leftarrow \mathrm{Avg}[\tilde{\mathbf{K}}] \oplus \mathrm{Max}[\tilde{\mathbf{K}}] \oplus \mathrm{Min}[\tilde{\mathbf{K}}]$
\STATE \textcolor{gray}{// Projection \& Score Estimation}
\FOR{each head $h \in \{1, \dots, N_h\}$}
\STATE $\hat{\mathbf{q}}_h \leftarrow \phi_q^{(h)}(\mathbf{Z}_{q, h}), \quad \hat{\mathbf{k}}_h \leftarrow \phi_k^{(h)}(\mathbf{Z}_{k, h})$
\STATE $\mathbf{S}^{\text{pred}}_h \leftarrow \frac{\hat{\mathbf{q}}_h \hat{\mathbf{k}}_h^\top}{\sqrt{d'}}$
\STATE $\mathbf{A}^{\mathrm{pred}}_h \leftarrow \operatorname{Softmax}(\mathbf{S}^{\text{pred}}_h)$ \hfill \textcolor{gray}{// row-wise over key tiles}
\STATE $\widetilde{\mathbf{M}}_h \leftarrow \mathbb{I}(\mathbf{S}^{\text{pred}}_h \ge \text{Top-}k\text{-Threshold}(\mathbf{S}^{\text{pred}}_h, k_{\mathrm{top}}))$
\ENDFOR
\STATE \textbf{return} $\mathbf{A}^{\mathrm{pred}}, \widetilde{\mathbf{M}}$
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[h]
\caption{Tile-Wise Distillation for Tile Score Estimator}
\label{alg:distill}
\begin{algorithmic}[1]
\STATE {\bfseries Input:} Query $\mathbf{Q}$, Key $\mathbf{K} \in \mathbb{R}^{N_h \times N \times d}$, Tile Size $B$, Top-$k$ Budget $k_{\mathrm{top}}$, Latent Dim $d'$
\STATE {\bfseries Output:} Distillation Loss $\mathcal{L}_{\text{distill}}$
\STATE \textcolor{gray}{// Full-Attention Teacher}
\STATE $\mathbf{A}^* \leftarrow \text{Softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right)$
\STATE \textcolor{gray}{// Target Construction (Score \& Weight)}
\STATE $\mathbf{S}^{\text{tgt}} \leftarrow \text{MaxPool}(\mathbf{A}^*, \text{kernel}=B, \text{stride}=B)$
\STATE $\mathbf{A}^{\text{tgt}} \leftarrow \text{RowNormalize}(\mathbf{S}^{\text{tgt}})$
\STATE \textcolor{gray}{// Student Prediction (Estimator with Stop-Gradient)}
\STATE $\mathbf{A}^{\text{pred}}, \widetilde{\mathbf{M}} \leftarrow \text{Algorithm~\ref{alg:estimator}}(\text{sg}(\mathbf{Q}), \text{sg}(\mathbf{K}), B, k_{\mathrm{top}}, d')$
\STATE \textcolor{gray}{// Distribution Alignment}
\STATE $\mathcal{L}_{\text{distill}} \leftarrow \operatorname{Mean}_{h,i}\!\left[\mathcal{D}_{\text{KL}}\!\left(\mathbf{A}^{\text{tgt}}[h,i,:] \parallel \mathbf{A}^{\text{pred}}[h,i,:]\right)\right]$
\STATE \textbf{backward} $\mathcal{L}_{\text{distill}}$
\end{algorithmic}
\end{algorithm}
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Waver-T2V Implementation Details}
\label{sec:implementation_details}
This section provides specifications for both the Waver-T2V-1B and Waver-T2V-12B models~\cite{waver}. The tables below detail the architectural configurations and training hyperparameters. The 1B variant employs a Multimodal Diffusion Transformer (MM-DiT)~\cite{cogvideox} backbone with 8 Dual Flow blocks and 22 Single Flow blocks. The 12B variant scales to 16 Dual Flow layers and 40 Single Flow layers with expanded dimensionality. Both models utilize the Wan-VAE for latent representation and Flan-T5-XXL for text encoding.
\begin{table}[H]
\centering
\caption{Model configuration comparison between Waver-T2V-1B and Waver-T2V-12B. Both architectures utilize a hybrid backbone comprising Dual Flow and Single Flow Multimodal Diffusion Transformer (MM-DiT) blocks.}
\label{tab:model_config_comparison}
\begin{minipage}[t]{0.48\textwidth}
\centering
\textbf{Waver-T2V-1B}
\vspace{0.2cm}
\begin{tabular}{lc}
\toprule
\textbf{Model Config} & \textbf{Value} \\
\midrule
Model Size & 1B \\
Backbone Architecture & MM-DiT \\
Dual Flow Layers & 8 \\
Single Flow Layers & 22 \\
Patch Size & $1 \times 2 \times 2$ \\
Hidden Dimension (Attention Head) & 128 \\
Num Heads & 12 \\
In Channels & 16 \\
Out Channels & 16 \\
Cross Attention Dim & 1536 \\
Text Embedding Dim & 4096 \\
VAE Model & Wan-VAE \\
Activation Function & GELU \\
Normalization & RMS Norm \\
Positional Embedding & RoPE \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\textbf{Waver-T2V-12B}
\vspace{0.2cm}
\begin{tabular}{lc}
\toprule
\textbf{Model Config} & \textbf{Value} \\
\midrule
Model Size & 12B \\
Backbone Architecture & MM-DiT \\
Dual Flow Layers & 16 \\
Single Flow Layers & 40 \\
Patch Size & $1 \times 2 \times 2$ \\
Hidden Dimension (Attention Head) & 128 \\
Num Heads & 24 \\
In Channels & 36 \\
Out Channels & 16 \\
Cross Attention Dim & 3072 \\
Text Embedding Dim & 4096 \\
VAE Model & Wan-VAE \\
Activation Function & GELU \\
Normalization & AdaLayerNormZero \\
Positional Embedding & RoPE \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}
\begin{table}[H]
\centering
\caption{Training hyperparameters comparison between Waver-T2V-1B and Waver-T2V-12B. Both models employ Rectified Flow matching objectives with distinct optimization configurations.}
\label{tab:training_hyperparams_comparison}
\begin{minipage}[t]{0.48\textwidth}
\centering
\textbf{Waver-T2V-1B}
\vspace{0.2cm}
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Learning Rate & $6.0 \times 10^{-5}$ \\
LR Scheduler & Constant \\
Batch Size & 128 \\
Resolution & $480 \times 864$ (161 frames) \\
Optimizer & AdamW \\
AdamW $\beta$ & $(0.9, 0.999)$ \\
Weight Decay & $1.0 \times 10^{-2}$ \\
Gradient Clipping & 1.0 \\
Precision & BF16 \\
Training Objective & Flow Matching \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\textbf{Waver-T2V-12B}
\vspace{0.2cm}
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Learning Rate & $5.0 \times 10^{-6}$ \\
LR Scheduler & Constant \\
Batch Size & 64 \\
Resolution & $720 \times 1280$ (241 frames) \\
Optimizer & AdamW \\
AdamW $\beta$ & $(0.9, 0.999)$ \\
Weight Decay & $1.0 \times 10^{-2}$ \\
Gradient Clipping & 1.0 \\
Precision & BF16 \\
Training Objective & Flow Matching \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Human Evaluation of Dynamic versus Static Tile Size Configurations}
To empirically validate the effectiveness of our Head-aware dynamic tile size selection strategy, we conduct a human evaluation study comparing various static tile configurations against our proposed approach, where human annotators assess the perceptual quality and generation fidelity of the outputs.
\begin{figure}[H]
\centering
\includegraphics[width=0.5\linewidth]{visuals/dyntile_human_eval.pdf}
\caption{\textbf{Human evaluation of dynamic versus static tile size configurations.} We benchmark various static tile size settings alongside our proposed Head-aware dynamic selection method against the strongest static baseline ($[4,4,8]$). While the $[4,4,8]$ configuration emerges as the optimal fixed setting, our dynamic approach consistently outperforms all static alternatives, achieving positive net win rates across evaluations and demonstrating superior generation quality through adaptive tile size allocation.}
\label{fig:dyntile_human_eval}
\end{figure}