Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation
Extracted source text · arXiv 2509.11252
\documentclass[acmsmall,screen]{acmart}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{array}
\usepackage{pifont}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{wrapfig}
\usepackage{caption}
\usepackage{adjustbox}
\usepackage{float}
\usepackage{graphicx}
\usepackage[many]{tcolorbox}
\usepackage{fontawesome5}
\usepackage{colortbl}
\usepackage{xspace}
\def\eg{\emph{e.g.,}\xspace}
\def\Eg{\emph{E.g.,}\xspace}
\def\ie{\emph{i.e.,}\xspace}
\def\Ie{\emph{I.e.,}\xspace}
\AtBeginDocument{%
\providecommand\BibTeX{{%
Bib\TeX}}}
\newtcolorbox{boxK}{
top=2.2pt,
bottom=2.2pt,
left=4.5pt,
right=4.5pt,
boxrule = 0pt,
toprule = 0pt, %
enhanced,
}
\newtcolorbox{boxRQ}{
top=1.2pt,
bottom=1.2pt,
left=3.5pt,
right=3.5pt,
boxrule = 0pt,
toprule = 0pt, %
enhanced,
}
\setcopyright{acmlicensed}
\copyrightyear{2025}
\acmYear{2025}
\acmDOI{XXXXXXX.XXXXXXX}
\acmConference[Conference acronym 'XX]{Make sure to enter the correct
conference title from your rights confirmation email}{June 03--05,
2018}{Woodstock, NY}
\begin{document}
\title{Beyond Autoregression: An Empirical Study of Diffusion Large Language Models for Code Generation}
\author{Chengze Li}
\orcid{0009-0008-0431-0125}
\authornote{Both authors contributed equally to this research.}
\authornote{Working as intern at Tsinghua University.}
\affiliation{%
\institution{College of AI, Tsinghua University}
\city{Beijing}
\country{China}
}
\affiliation{%
\institution{School of Computer Science, Nanjing University}
\city{Nanjing}
\country{China}
}
\email{231220004@smail.nju.edu.cn}
\author{Yitong Zhang}
\orcid{0009-0000-1138-4503}
\authornotemark[1]
\affiliation{%
\institution{College of AI, Tsinghua University}
\city{Beijing}
\country{China}
}
\affiliation{%
\institution{School of Computer Science and Engineering, Beihang University}
\city{Beijing}
\country{China}
}
\email{22373337@buaa.edu.cn}
\author{Jia Li}
\orcid{0000-0002-5579-8852}
\authornote{Jia Li is the corresponding author.}
\affiliation{%
\institution{College of AI, Tsinghua University}
\city{Beijing}
\country{China}
}
\email{jia_li@mail.tsinghua.edu.cn}
\author{Liyi Cai}
\orcid{0009-0002-7848-6007}
\affiliation{%
\institution{School of Computer Science, Peking University}
\city{Beijing}
\country{China}
}
\email{cailiyi@stu.pku.edu.cn}
\author{Ge Li}
\orcid{0000-0002-5828-0186}
\affiliation{%
\institution{School of Computer Science, Peking University}
\city{Beijing}
\country{China}
}
\email{lige@pku.edu.cn}
\renewcommand{\shortauthors}{C. Li, Y. Zhang, J. Li, L. Cai and G. Li}
\input{secs/0_abstract}
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10011007.10011074.10011092</concept_id>
<concept_desc>Software and its engineering~Software development techniques</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Software and its engineering~Software development techniques}
\keywords{code generation, diffusion large language models, non-autoregressive generation, empirical study}
\maketitle
\input{secs/1_introduction}
\input{secs/2_backgroud}
\input{secs/3_design}
\input{secs/4_results}
\input{secs/5_discussion}
\input{secs/6_conclusion}
\input{secs/7_availability}
\bibliographystyle{ACM-Reference-Format}
\bibliography{refs}
\end{document}
\begin{abstract}
Large Language Models (LLMs) have become the mainstream approaches to code generation. Existing LLMs mainly employ autoregressive generation, \ie generating code token-by-token from left to right. However, the underlying autoregressive generation has two limitations in code generation. First, autoregressive LLMs only generate a token at each step, showing low efficiency in practice. Second, programming is a non-sequential process involving back-and-forth editing, while autoregressive LLMs only employ the left-to-right generation order. These two intrinsic limitations hinder the further development of LLMs in code generation.
Recently, diffusion LLMs have emerged as a promising alternative. Diffusion LLMs address the above limitations with two advances, including multi-token prediction (\ie generating multiple tokens at each step) and flexible generation order (\ie flexibly determining which positions to generate tokens). However, there is no systematic study exploring diffusion LLMs in code generation. To bridge the knowledge gap, we present the first empirical study of diffusion LLMs for code generation. Our study involves 9 representative diffusion LLMs and conduct experiments on 4 widely used benchmarks. Based on the results, we summarize the following findings. \ding{182} Existing diffusion LLMs are competitive with autoregressive LLMs with similar sizes. For example, on HumanEval and MBPP, the best-performing diffusion LLM achieves pass@1 scores of 89.6\% and 79.4\%, compared with 84.8\% and 70.8\% for autoregressive LLMs. \ding{183} Diffusion LLMs have a stronger length extrapolation ability than autoregressive LLMs and perform better in long code understanding. \ding{184} We explore factors impacting the effectiveness and efficiency of diffusion LLMs, and provide practical guidance. \ding{185} We discuss several promising further directions to improve diffusion LLMs on code generation. We open-source all source code, data, and results to facilitate the following research.
The code is publicly available at \url{https://github.com/zhangyitonggg/dllm4code}.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
During software development, there is a growing demand for automatic code generation due to the substantial labor required in manual programming~\cite{shin2021survey, li2023skcoder}. Code generation aims to automatically generate executable source code that fulfills natural language requirements. Over the past few years, Large Language Models (LLMs) have shown remarkable capabilities in code generation, achieving strong performance on widely used benchmarks and significantly enhancing developer productivity~\cite{jiang2024aixcoder, o1, li2025aixcoder, li2025structured, zhang2025focused, li2024acecoder}.
Building on this progress, a substantial body of recent work has explored the use of LLMs for code generation~\cite{jiang2024survey}, with representative examples such as Qwen-Coder~\cite{qwencoder}, Seed-Coder~\cite{seed2025seed}, and DeepSeek-Coder~\cite{deepseekcoder, deepseek-coder-v2}.
While these LLMs differ in various technical details, they are all built upon the autoregressive paradigm, where the model generates tokens sequentially from left to right, as illustrated in Figure~\ref{fig:modeling-AR}.
However, there is an open question of \textbf{whether the mainstream autoregressive generation paradigm is truly the best choice for code generation}. In practice, current AutoRegressive LLMs (AR LLMs) reveal two fundamental limitations.
\ding{182} \textbf{Next-Token Prediction $\rightarrow$ Low Efficiency.} Most AR LLMs generate code strictly one token at a time, with each forward pass typically producing only a single token. Our preliminary experiments further indicate that generating each token for AR LLMs requires a substantial amount of computation, measured in billions of FLOPs, which makes the overall process highly inefficient.
This low efficiency constrains the broader application of LLMs in code generation. For instance, generating code at the repository level is exceedingly demanding, often incurring prohibitively high computational costs and significant inference time~\cite{liao20243, wang2025teaching, bi2024iterative}. Such latency is particularly unacceptable in interactive settings such as real-time coding assistants, where real-time feedback is essential to maintain usability and user experience~\cite{liang2024large, sergeyuk2025using, wermelinger2023using}.
\ding{183} \textbf{Left-to-Right Generation Order $\rightarrow$ Misalignment with Programming.}
AR LLMs decode in a strictly left-to-right manner, with each new token fully dependent on the preceding context. This generation paradigm is misaligned with real-world programming, which often involves non-sequential back-and-forth refinement.
For example, as shown in Figure~\ref{fig:code-diff}, a developer may first implement the body of the callee functions and later write the corresponding caller code, which directly contrasts with the rigid sequential nature of autoregressive decoding. In practical, when an AR LLM attempts to generate the caller code before the callee has been defined, it cannot access the necessary information and is therefore prone to errors.
Such limitations are intrinsic to the autoregressive paradigm and restrict the further advancement of code generation. This underscores the need to explore alternative paradigms that better capture the requirements of programming.
\input{figs/modeling}
Recently, a new generation paradigm of large language models, known as diffusion LLMs, has attracted increasing attention. Their generation process is illustrated in Figure~\ref{fig:modeling-diffusion}.
Models such as LLaDA~\cite{llada, llada-v} and Dream~\cite{dream} have demonstrated competitive performance compared to state-of-the-art autoregressive LLMs.
In contrast to AR LLMs, diffusion LLMs offer two notable advantages for code generation.
\ding{182} \textbf{Multi-Token Prediction $\rightarrow$ High Efficiency.} Enabled by parallel decoding, diffusion LLMs can generate multiple tokens within a single forward pass. This significantly reduces the computational cost required for each token and thereby substantially improves inference speed.
Current state-of-the-art diffusion LLMs have already achieved inference speeds far exceeding those of AR LLMs while maintaining performance comparable to leading autoregressive models, offering strong support for both real-time code generation and repository-level tasks~\cite{seeddiffusion, geminidiffusion}.
\ding{183} \textbf{Flexible Generation Order $\rightarrow$ Alignment with Programming.}
Rather than generating tokens strictly from left to right, diffusion LLMs allow the inference process to flexibly select which positions to update at each forward pass. This enables code generation to more closely reflect the non-sequential back-and-forth edit typically employed by human programmers~\cite{diffucoder}. For example, a diffusion LLM may first generate callee functions before producing the corresponding caller functions, or prioritize parts of the code with richer contextual information before filling in those with less context.
However, despite the considerable advantages and potential of diffusion LLMs, a systematic evaluation and analysis of their capabilities for code generation is still missing. This gap leaves many important questions unanswered and hinders their broader adoption in future code generation.
In this paper, we conduct the first empirical study that systematically investigates the potential of diffusion LLMs for code generation. To address the existing gap and to guide future development, we propose four main Research Questions (RQs) together with their corresponding experiments. Building on these experiments, we provide a detailed analysis that uncovers several important findings and offers new insights into the future research directions of diffusion LLMs. In the following, we introduce each research question in turn.
\vspace{-2pt} \begin{boxRQ}
\small \faIcon{search} \textit{\textbf{RQ1:}
How do diffusion LLMs compare with leading AR LLMs in code generation tasks?
}\end{boxRQ} \vspace{-4.5pt}
To address this question, we evaluate 9 popular diffusion LLMs against 4 leading autoregressive baselines on 3 widely used benchmarks.
Our results show that diffusion LLMs are able to solve certain tasks that remain challenging for AR LLMs. They are becoming increasingly competitive in code generation, which highlights their strong potential for advancing code generation.
\vspace{-2pt} \begin{boxRQ}
\small \faIcon{search} \textit{\textbf{RQ2:}
What factors influence the effectiveness of code generation by diffusion LLMs?
}\end{boxRQ} \vspace{-4.5pt}
In this RQ, we investigate how different settings influence the effectiveness of diffusion LLMs. We consider a range of key factors and evaluate several representative models on HumanEval~\cite{humaneval} and MBPP~\cite{mbpp}.
Our analysis yields several interesting findings. For example, very short generation lengths severely reduce the usability of generated code, whereas excessively long lengths tend to produce redundant outputs with only a slight impact on effectiveness.
\vspace{-2pt} \begin{boxRQ}
\small \faIcon{search} \textit{\textbf{RQ3:}
What are the key factors that influence the efficiency of diffusion LLMs?
}\end{boxRQ} \vspace{-4.5pt}
High efficiency is a key advantage of diffusion LLMs. In this RQ, we investigate how inference settings affect their efficiency in code generation by evaluating several representive models on HumanEval and MBPP.
The results show that both increasing the number of diffusion steps and extending the generation length reduce efficiency. These findings provide practical guidance for balancing efficiency and accuracy in real-world code generation scenarios.
\vspace{-2pt} \begin{boxRQ}
\small \faIcon{search} \textit{\textbf{RQ4:}
How well do diffusion LLMs perform on long code understanding?
}\end{boxRQ} \vspace{-4.5pt}
A key challenge for LLMs is understanding code at the repository level. In this RQ, we assess the ability of diffusion LLMs to understanding long code. To this end, we conduct experiments on RepoQA~\cite{repoqa}, evaluating representative diffusion and AR LLMs across a range of context lengths.
The results show diffusion LLMs demomstrate strong performance in long code understanding and exhibit substantially stronger length extrapolation ability than AR LLMs.
Building on the above results and analyses, we outline several promising directions for future research on diffusion LLMs for code generation. These directions span multiple perspectives, such as applying diffusion models to a broader range of suitable coding tasks, integrating the generation process with structural characteristics of code, and exploring other avenues that further enhance their performance and practicality.
To summarize, the key contributions of this paper are as follows:
\begin{itemize}
\item We conduct the first empirical study that explores the potential of emerging diffusion LLMs in the domain of code generation, highlighting their characteristics compared to AR LLMs.
\item We conduct a comprehensive, systematic analysis of the various factors that influence the effectiveness and efficiency of diffusion LLMs in code generation. This analysis uncovers many general patterns that can guide practical usage across different diffusion LLMs.
\item Based on our study results and prior work, we propose multiple promising directions for future research aimed at enhancing the application of diffusion LLMs in code generation.
\end{itemize}
\begin{figure}[t]
\centering
\adjustbox{valign=t}{%
\begin{minipage}[t]{0.59\linewidth}
\centering
\begin{subfigure}[t]{0.99\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/modeling-ar.pdf}
\caption{Autoregressive}
\label{fig:modeling-AR}
\end{subfigure}
\begin{subfigure}[t]{0.99\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/modeling-diffusion.pdf}
\caption{Diffusion}
\label{fig:modeling-diffusion}
\end{subfigure}
\vspace{-0.08in}
\caption{Generation paradigms of AR LLMs and diffusion LLMs.}
\label{fig:modeling}
\end{minipage}}%
\hspace{0.15cm}
\adjustbox{valign=t}{%
\begin{minipage}[t]{0.39\linewidth}
\centering
\includegraphics[height=5.7cm]{assets/code-diff.pdf} %
\caption{An example commit illustrating non-sequential programming.}
\label{fig:code-diff}
\end{minipage}}
\vspace{-0.22in}
\end{figure}
\section{Background and Related Work}
\label{sec:Background}
\subsection{Autoregressive Large Language Models}
\label{sec:arllm}
Autoregressive generation represents the predominant paradigm for large language models, commonly referred to as AR LLMs. It models a text sequence $Y = \{y_1, y_2, \ldots, y_L\}$ typically by producing one token at a time in a left-to-right manner. The generation of each token is conditioned on the sequence of previously generated tokens, a process formalized by the chain rule of probability:
\begin{equation}
p(Y \mid X) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, X),
\label{eq:ar}
\end{equation}
where \( X \) denotes the input context and \( y_{<t} \) represents the tokens generated before step \( t \).
Building on the success of ChatGPT~\cite{achiam2023gpt}, extensive research has applied autoregressive LLMs to code generation, yielding substantial gains in quality and functional correctness~\cite{li2024acecoder, ouyang2025empirical}. This progress has led to specialized state-of-the-art models fine-tuned for programming, such as Qwen-Coder~\cite{qwencoder} and DeepSeek-Coder~\cite{deepseekcoder}, which now serve as the core engines behind many modern coding assistants and agents~\cite{islam2025codesim, mowar2025codea11y}.
Despite their success, the autoregressive paradigm presents inherent limitations that are misaligned with the practical demands of code generation. First, next-token prediction constrains efficiency and creates a latency bottleneck, which poses challenges for repository-level code generation and for real-time applications such as coding assistants and agents. Second, the strictly left-to-right generation process conflicts with human programming practices, which typically involve non-sequential refinement. These fundamental mismatches highlight the need to explore alternative generation paradigms that align more closely with the requirements of code generation.
\subsection{Diffusion Large Language Models}
\label{sec:dllm}
Diffusion language models have recently become a focal point in AI research, offering an alternative to traditional autoregressive models~\cite{sahoo2024simple, li2025survey, discreteVLA, yang2025mmada}. They can be broadly categorized into continuous and discrete formulations. Empirical evidence suggest that discrete diffusion language models scale more effectively to large model sizes~\cite{gong2024scaling}, which has led to the development of diffusion large language models, such as LLaDa~\cite{llada}, Dream~\cite{dream}, and Gemini Diffusion~\cite{geminidiffusion}. Among these, most approaches typically adopt a masked discrete diffusion paradigm~\cite{sahoo2024simple}. To illustrate how diffusion LLMs work in practice, we present a representative example of the training and inference procedure. \textbf{During training}, an original sequence $Y_0=\{y_1^0,\ldots,y_L^0\}$ is corrupted into $Y_t$ along a continuous schedule $t\in[0,1]$ via independent masking, with a simple corruption process:
\begin{equation}
q(y_i^t \mid y_i^0)=
\begin{cases}
1-t & \text{if } y_i^t=y_i^0\\
t & \text{if } y_i^t=\texttt{[MASK]}
\end{cases}
\label{eq:diffusion}
\end{equation}
where \texttt{[MASK]} denotes a special token for model to predict. The model is trained to reverse this corruption process by minimizing a cross-entropy loss computed only on masked positions.
\textbf{During inference}, generation begins from a fully masked sequence of one predefined generation length. At each step, the model first predicts tokens for all masked positions. It then uses specific remask strategy to retain a subset of the predictions, while the remaining positions are remasked for further refinement. This iterative refinement continues until no masks remain, enabling parallel and flexible non-sequential updates across positions.
In practice, several key factors may shape the behavior of diffusion LLMs: (1) \textit{Generation length}, which determines the number of tokens to be generated from an initially fully masked sequence; (2) \textit{Diffusion steps}, which specify how many tokens are updated at each step; (3) \textit{Remasking strategy}, which decides which predicted tokens are retained and which are masked again at each step; (4) \textit{Block diffusion}, supported by certain models such as \textsc{LLaDA-8B-Instruct}~\cite{llada}, where the sequence is divided into blocks decoded sequentially from left to right, with each block processed using the standard diffusion procedure; and (5) \textit{Decoding temperature}, which controls the randomness of outputs, with higher values producing more diverse generations.
Although diffusion LLMs were not originally designed for code generation, their ability to perform parallel and flexible generation aligns well with the iterative refinement workflow inherent in programming, motivating a growing line of code-oriented diffusion models. DiffuCoder~\cite{diffucoder}, Mercury Coder~\cite{mercurycoder}, and Seed Diffusion Preview~\cite{seeddiffusion} represent notable advances in this direction, introducing techniques such as coupled GRPO for reinforcement learning and edit-based corruption to correct errors during inference. TreeDiff~\cite{zeng2025treediff} further incorporates syntax awareness by masking abstract syntax tree subtrees during training, thereby improving syntactic correctness and structured reconstruction. In addition, recent work~\cite{mundler2025constrained} proposes constrained decoding for diffusion LLMs that enforces context-free grammars.
Despite these advances, existing work on diffusion LLMs for code has largely focused on individual training techniques or decoding strategies. To date, there has been no comprehensive evaluation of diffusion LLMs in code generation tasks that systematically assesses their potential and challenges compared to autoregressive baselines.
\section{Study Design}
\label{sec:design}
\input{figs/overview}
\subsection{Overview and Research Questions}
\label{sec:rq}
The goal of this paper is to assess the potential of diffusion LLMs for code generation. We conduct a systematic evaluation across multiple widely used benchmarks and comprehensive experimental settings, comparing popular diffusion LLMs with state-of-the-art AR baselines. Figure~\ref{fig:overview} presents an overview of our study. We next introduce the research questions we aim to investigate.
\textbf{RQ1 Effectiveness of Diffusion LLMs for Code Generation: How do diffusion LLMs compare with state-of-the-art AR LLMs in code generation tasks?} In RQ1, We evaluate popular open-source and closed-source diffusion LLMs against state-of-the-art AR baselines across widely used benchmarks under their optimal settings.
\textbf{RQ2 Impact of Diffusion LLM Settings on Effectiveness: What factors impact the effectiveness of code generation by diffusion LLMs?} The usability and quality of the generated code are critical for real-world applications~\cite{liang2024large}. RQ2 focuses on how different inference settings influence the functional usability of outputs. We select a range of potentially influential factors and evaluate their performance through experiments across multiple settings.
\textbf{RQ3 Impact of Diffusion LLM Settings on Efficiency: What are the key factors that influence the efficiency of diffusion LLMs?} One of the main advantages of diffusion LLMs is their potential for high efficiency due to parallel decoding~\cite{zhang2025survey}. RQ3 aims to explore how various configuration choices affect efficiency. To investigate this, we identify several influential factors and systematically evaluate their impact under different settings.
\textbf{RQ4 Performance of Diffusion LLMs on Long Code Understanding. How well do diffusion LLMs perform on long code understanding?} Understanding and generating code at the repository level remains an open challenge for LLMs~\cite{li2025aixcoder, pan2024enhancing}. RQ4 evaluates the capability of diffusion LLMs in long-context code understanding. To answer it, we compare the performance of several diffusion LLMs with AR baselines that support similar maximum context lengths, using a variety of input lengths to analyze robustness as context length increases.
\subsection{Studied LLMs}
In this section, we introduce the LLMs evaluated in this study. A summary of the selected LLMs is provided in Table~\ref{tab:setting_llm}.
\subsubsection{Diffusion LLMs}
We evaluate nine models drawn from six diffusion LLM families, covering nearly all diffusion LLMs available to us.
\textbf{LLaDA.} The LLaDA models are open-source diffusion LLMs trained entirely from scratch. \textsc{LLaDA-8B-Instruct}~\cite{llada} is the first open-source diffusion large language model of LLaDa series. \textsc{LLaDA 1.5}~\cite{llada15} extends this model by incorporating variance-reduced preference optimization (VRPO) during training. We include both \textsc{LLaDA-8B-Instruct} and \textsc{LLaDA-1.5} in our experiments.
\textbf{Dream.} The Dream family is another line of open-source diffusion LLMs. \textsc{Dream-v0-Instruct-7B}~\cite{dream} is initialized from an autoregressive base and provides the foundation for subsequent variants. \textsc{Dream-Coder-v0-Instruct-7B}~\cite{dreamcoder} targets code generation and adaptively selects its decoding style based on the task. \textsc{DreamOn-v0-7B}~\cite{dream} explores dynamic expansion and contraction of mask tokens during inference. We include \textsc{Dream-v0-Instruct-7B}, \textsc{Dream-Coder-v0-Instruct-7B}, and \textsc{DreamOn-v0-7B} in our experiments.
\textbf{DiffuCoder.} \textsc{DiffuCoder-7B-cpGRPO}~\cite{diffucoder} is post-trained with coupled-GRPO on 21K code examples, yielding substantial gains in code generation performance. We include \textsc{DiffuCoder-7B-cpGRPO} in our experiments.
\textbf{Mercury.} Developed by Inception Labs, \textsc{Mercury-Coder-Small}~\cite{mercurycoder} is the first commercial scale diffusion large language model that demonstrates strong performance on key coding benchmarks. We include \textsc{Mercury-Coder-Small} in our experiments.
\textbf{Gemini-Diffusion.} Developed by DeepMind, \textsc{Gemini-Diffusion}~\cite{geminidiffusion} reports performance comparable to much larger models while offering faster inference. Since no public API is available, we rely on results reported on the official website.
\textbf{Seed-Diffusion.} Developed by the Seed team, \textsc{Seed-Diffusion Preview}~\cite{seeddiffusion} is a code-oriented diffusion language model designed to balance speed and quality. Since no public API is available, we rely on results reported by the authors.
\input{tabs/setting_llm}
\subsubsection{Autoregresive LLMs}
For comparison with diffusion LLMs, we select the following state-of-the-art and representative AR LLMs at similar parameter scales.
\textbf{Qwen.} Developed by the Qwen team, \textsc{Qwen3-8B}~\cite{yang2025qwen3} is the latest generation in the series and shows significant improvements in code generation. We include \textsc{Qwen3-8B} in our experiments.
\textbf{Seed-Coder.} Developed by the Seed team, \textsc{Seed-Coder-8B-Instruct}~\cite{seed2025seed} attains leading results among open source models of comparable size across diverse coding tasks. We include \textsc{Seed-Coder-8B-Instruct} in our experiments.
\textbf{DeepSeek-Coder.} Developed by DeepSeek, the DeepSeek Coder family consists of code models trained from scratch~\cite{deepseekcoder}. We include \textsc{DeepSeek-Coder-6.7B-Instruct} in our experiments.
\textbf{Code Llama.} Developed by Meta, Code Llama provides competitive open source performance for code. We include \textsc{CodeLlama-7b-Instruct-hf}~\cite{codellama} in our experiments.
Additionally, in RQ4, we investigate the long-context capabilities of diffusion LLMs in comparison to AR LLMs. As current diffusion LLMs are still in their early stages of development and typically support context lengths of up to 4k tokens, we include an AR baseline with a comparable context window. Specifically, we select \textsc{Llama-2-7B-chat-hf}~\cite{touvron2023llama} to enable a fair comparison.
\subsection{Benchmark Selection}
\subsubsection{Selection Principles}
Our benchmark selection process was guided by four key considerations to ensure a comprehensive, fair, and reproducible evaluation.
\ding{182} \textbf{Alignment with Research Questions.} We aim to select benchmarks that align closely with our research questions. RQ1 requires breadth and external validity. RQ2 and RQ3 require controllable and inexpensive evaluation. RQ4 requires repository-level long context.
\ding{183} \textbf{Community Adoption and Comparability.} We prioritized well-established, open-source benchmarks that are widely used in the research community. This choice ensures our results are directly comparable with prior studies and enhances the reproducibility of our findings.
\ding{184} \textbf{Task Diversity.} We favor benchmarks that reflect the diversity of real-world code generation scenarios, ranging from fundamental code synthesis and basic programming problems to more complex algorithmic challenges.
\ding{185} \textbf{Evaluation Fidelity.} We favor benchmarks with executable tests or objective oracles so that functional correctness are directly measurable.
\subsubsection{Selected Benchmarks}
Guided by these principles, we selected a specific suite of benchmarks tailored to each of our research questions.
For RQ1, which evaluates overall model performance, we chose three widely used benchmarks: HumanEval~\cite{humaneval}, MBPP~\cite{mbpp}, and LiveCodeBench~\cite{li2024acecoder}. HumanEval is composed of 164 manually scripted Python programming problems, each containing a function signature, a docstring, and multiple unit tests. MBPP offers a larger collection of approximately 974 Python problems, where each problem includes an English task description, a code solution, and three automated test cases. To assess performance on more sophisticated and contemporary scenarios, we employed LiveCodeBench, which continuously curates new problems from prominent competitive programming platforms such as LeetCode and CodeForces.
We report results on all stages from v1 to v6, as well as on the most recent slice v6 (spanning 1 February 2025 to 1 May 2025), in order to prevent potential data contamination during evaluation.
For RQ2 and RQ3, which examine how different settings influence effectiveness and efficiency, we use HumanEval and MBPP. Their lightweight datasets and executable tests enable systematic large scale sweeps over sampling steps, temperature, remasking strategies, and related settings.
For RQ4, which compares how diffusion and AR LLMs handle long-context code scenarios, we use RepoQA~\cite{repoqa}. This benchmark is specifically designed to evaluate a model's ability to comprehend entire code repositories. Our evaluation focuses on its Searching Needle Function task, which challenges a model to retrieve a specific function from a repository based solely on a natural language description. The task consists of 500 problems spanning 50 repositories in five different programming languages, providing a robust evaluation of long-context code understanding.
\subsection{Evaluation Metrics}
We adopt four metrics that are consistent with our research questions and are widely used in prior studies.
\textbf{pass@k.} For HumanEval, MBPP, and LiveCodeBench, we evaluate functional correctness using pass@k. As a widely adopted metric, pass@k calculates the percentage of problems for which at least one of k generated solutions passes all provided test cases.
\textbf{Retrieval Accuracy.} For RepoQA, we follow the metric defined in the original work. Given a repository and a natural language description, the model outputs a target function. A prediction is counted as correct only if the generated function is the most similar to the ground-truth needle function among all functions in the repository under BLEU, and the BLEU score exceeds a fixed threshold. Retrieval accuracy is the proportion of queries that meet both conditions.
\textbf{FLOPs.} To assess computational efficiency, we report the average floating point operations required to generate each output token. This metric reflects the computational cost of the inference process, providing a hardware-agnostic measure that complements throughput by focusing on efficiency rather than latency.
\textbf{Throughput.} To assess time efficiency, we report decoding throughput in tokens per second, calculated as the number of output tokens divided by the total decoding time. Decoding time is measured in wall-clock seconds from the first decoding call to the production of the final output.
\subsection{Implementation}
For closed-source LLMs, we access the models through their official APIs. For open-source models, we load the weights from their official repositories on HuggingFace.
Since acceleration techniques for diffusion LLMs are still underdeveloped, we disable all acceleration techniques including KV cache for all models to ensure fairness in comparison.
We adopt a zero-shot prompting strategy, where each model is provided only with the raw benchmark input as the prompt. All Experiments are conducted on a server equipped with eight NVIDIA A100-40GB GPUs.
\begin{figure*}[t]
\centering
\includegraphics[width=0.91\textwidth]{assets/overview3.pdf}
\vspace{-5pt}
\caption{Overview of our study. DLLM is used as an abbreviation for Diffusion LLM.}
\label{fig:overview}
\vspace{-0.18in}
\end{figure*}
\begin{table}[t]
\centering
\small
\caption{The summary of LLMs used in this study.}
\label{tab:setting_llm}
\vspace{-7pt}
\scalebox{0.81}{
\begin{tabular}{@{}ccccccc@{}}
\toprule
\textbf{Type} & \textbf{Domain} & \textbf{Model} & \textbf{Open-Source} & \textbf{Scale} & \textbf{Release Date} & \textbf{Length} \\ \midrule
\multirow{9}{*}{Diffusion}
& General & \textsc{LLaDA-8B-Instruct} & \ding{51} & 8B & 02/2025 & 4k \\
& General & \textsc{LLaDA-1.5} & \ding{51} & 8B & 05/2025 & 4k \\
& General & \textsc{Dream-v0-Instruct-7B} & \ding{51} & 7B & 04/2025 & 4k \\
& General & \textsc{Gemini-Diffusion} & \ding{55} & -- & 05/2025 & -- \\
& Code & \textsc{Dream-Coder-v0-Instruct-7B} & \ding{51} & 7B & 07/2025 & 4k \\
& Code & \textsc{DreamOn-v0-7B} & \ding{51} & 7B & 07/2025 & 4k \\
& Code & \textsc{DiffuCoder-7B-cpGRPO} & \ding{51} & 7B & 06/2025 & 4k \\
& Code & \textsc{Mercury-Coder-Small} & \ding{55} & -- & 02/2025 & 32k \\
& Code & \textsc{Seed-Diffusion-Preview} & \ding{55} & $\le$15B & 07/2025 & -- \\ \midrule
\multirow{5}{*}{Autoregressive}
& General & \textsc{Qwen3-8B} & \ding{51} & 8B & 04/2025 & 32k \\
& General & \textsc{Llama-2-7B-Chat-hf} & \ding{51} & 7B & 07/2023 & 4k \\
& Code & \textsc{Seed-Coder-8B-Instruct} & \ding{51} & 8B & 05/2025 & 32k \\
& Code & \textsc{DeepSeek-Coder-6.7B-Instruct} & \ding{51} & 6.7B & 08/2024 & 16k \\
& Code & \textsc{CodeLlama-7B-Instruct-hf} & \ding{51} & 7B & 07/2023 & 100k \\
\bottomrule
\end{tabular}%
}
\vspace{-0.2in}
\end{table}
\section{Study Results}
\label{sec:results}
\input{secs/results/rq1}
\input{secs/results/rq2}
\input{secs/results/rq3}
\input{secs/results/rq4}
\subsection{RQ1: Effectiveness of Diffusion LLMs for Code Generation}
\label{sec:rq1}
\textbf{Motivation.}
In this section, we investigate the effectiveness of current diffusion LLMs for code generation and explore their potential to achieve or even surpass the performance of state-of-the-art AR LLMs.
\input{tabs/rq1_acc}
\vspace{4px} \noindent
\textbf{Setting.}
We select the optimal settings for selected LLMs based on the findings from RQ2. Using these settings, we evaluate all models on HumanEval, MBPP, and LiveCodeBench, measuring performance in terms of pass@1.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
The results are shown in Table~\ref{tab:rq1_acc}.
\textit{\textbf{Comparison Among Diffusion LLMs.}}
\ding{182} \textbf{Newer diffusion LLMs demonstrate noticeably better performance in code generation.} For example, \textsc{Dream-v0-Instruct-7B}, released in April 2025, achieves a pass@1 of 13.3\% on LiveCodeBench\textsuperscript{\textcolor{blue}{v1--v6}}, whereas \textsc{LLaDA-8B-Instruct}, released in February 2025, attains only 6.9\%.
These results indicate that diffusion LLMs are improving rapidly.
\ding{183} \textbf{Open-source diffusion LLMs lag behind closed-source counterparts in code generation quality.} For example, \textsc{Mercury-Coder-Small} achieves pass@1 scores of 86.0\% on HumanEval, 76.2\% on MBPP, 31.8\% on LiveCodeBench\textsuperscript{\textcolor{blue}{v1--v6}}, and 22.1\% on LiveCodeBench\textsuperscript{\textcolor{blue}{v6}}, exceeding all open-source diffusion LLMs.
This performance gap may be attributed to the availability of higher-quality training data in commercial settings.
\textit{\textbf{Comparison Between Diffusion and AR LLMs.}}
\ding{182} \textbf{Diffusion LLMs still lag behind AR LLMs in overall performance.}
For example, the average pass@1 scores of diffusion LLMs on HumanEval and LiveCodeBench\textsuperscript{\textcolor{blue}{v6}} are 66.7\% and 14.9\%, respectively, which are lower than the corresponding scores of AR LLMs at 71.3\% and 18.9\%. These results indicate that diffusion LLMs are not yet able to replace AR LLMs at the current stage in code generation.
\ding{183} \textbf{Nevertheless, diffusion LLMs demonstrate strong competitiveness against state-of-the-art AR LLMs in code generation.}
On HumanEval, \textsc{Gemini-Diffusion} achieves a pass@1 of 89.6\%, surpassing the best-performing AR baseline, \textsc{Seed-Coder-8B-Instruct}, which reaches 84.8\%. Moreover, the open-source \textsc{Dream-Coder-v0-Instruct-7B} obtains a pass@1 of 76.2\%, approaching the performance of strong AR models. These findings suggest that diffusion LLMs hold significant potential for practical applications in code generation.
\input{figs/rq1_evolve_veen}
\textit{\textbf{Complementarity Between Diffusion and AR LLMs.}}
Although diffusion LLMs do not yet consistently outperform their AR counterparts in overall accuracy, we further investigate whether they can successfully solve tasks that AR LLMs fail to complete. To this end, we select three representative diffusion LLMs: \textsc{Dream-Coder-v0-Instruct-7B}, \textsc{DiffuCoder-7B-cpGRPO}, and \textsc{Mercury-Coder-Small}, along with three strong AR LLMs: \textsc{Qwen3-8B}, \textsc{Seed-Coder-8B-Instruct}, and \textsc{DeepSeek-Coder-6.7B-Instruct}. For each group, we identify the set of problems that are simultaneously solved across HumanEval, MBPP, and LiveCodeBench\textsuperscript{\textcolor{blue}{v1--v6}}, ensuring that only robustly solved instances are retained. We then compare the two sets using a Venn diagram. As shown in Figure~\ref{fig:rq1_veen}, 367 problems are solved by both paradigms, while 149 are uniquely solved by AR models and 105 are uniquely solved by diffusion models. The existence of a substantial number of problems solved exclusively by diffusion models demonstrates that diffusion LLMs possess complementary strengths. These findings indicate that \textbf{diffusion LLMs are capable of addressing certain code generation tasks that remain challenging for AR models}, highlighting the potential of hybrid approaches that leverage the advantages of both paradigms.
\textit{\textbf{Evolution of Diffusion and AR LLMs.}}
To further compare the developmental trajectories of the two paradigms, we select representative open-source diffusion and AR LLMs of comparable scale released at different points in time. For each model, we record its release date and report its pass@1 performance on HumanEval. The results are illustrated in Figure~\ref{fig:rq1_evolve}. The figure shows that diffusion LLMs demonstrate a rapid upward trend in pass@1 within a relatively short period, whereas the performance improvement of AR LLMs has begun to plateau. This divergence indicates that \textbf{diffusion LLMs are advancing at a significantly faster pace and have the potential to further narrow, or even close, the performance gap with AR approaches in the near future}.
\begin{boxK}
\small \faIcon{pencil-alt} \textbf{Answer to RQ1:}
Although diffusion LLMs have not yet surpassed state-of-the-art AR models in overall performance, they demonstrate increasing competitiveness in code generation and are capable of solving certain tasks that remain challenging for AR LLMs.
Moreover, diffusion LLMs show strong potential to further narrow and even close the performance gap with AR LLMs in the near future.
\end{boxK}
\begin{table}[!t]
\centering
\small
\caption{Effectiveness of Each Model for Code Generation.}
\vspace{-0.12in}
\label{tab:rq1_acc}
\scalebox{0.86}{
\begin{tabular}{@{}cccccc@{}}
\toprule
\textbf{Type} & \textbf{Model} & \textbf{HumanEval} & \textbf{MBPP}
& \textbf{LiveCodeBench}\textsuperscript{\textcolor{blue}{v1--v6}}
& \textbf{LiveCodeBench}\textsuperscript{\textcolor{blue}{v6}} \\ \midrule
\multirow{9}{*}{Diffusion}
& \textsc{LLaDA-8B-Instruct} & 45.1\% & 39.4\% & 6.9\% & 9.2\% \\
& \textsc{LLaDA-1.5} & 43.3\% & 40.4\% & 6.8\% & 9.2\% \\
& \textsc{Dream-v0-Instruct-7B} & 56.7\% & 56.8\% & 13.3\% & 11.5\% \\
& \textsc{Dream-Coder-v0-Instruct-7B} & 76.2\% & 65.8\% & 24.8\% & 18.3\% \\
& \textsc{DreamOn-v0-7B} & 51.2\% & 53.0\% & 10.6\% & 9.2\% \\
& \textsc{DiffuCoder-7B-cpGRPO} & 69.5\% & 64.2\% & 13.0\% & 8.4\% \\
& \textsc{Mercury-Coder-Small} & 86.0\% & 76.2\% & 31.8\% & 22.1\% \\
& \textsc{Gemini-Diffusion\textsuperscript{†}} & \textbf{89.6\%} & 76.0\% & 30.8\% & \textbf{30.9\%} \\
& \textsc{Seed-Diffusion-Preview\textsuperscript{†}} & 82.8\% & \textbf{79.4\%} & \textbf{33.7\%} & -- \\
& \cellcolor{gray!15}Average & 66.7\% & 61.2\% & 19.1\% & 14.9\% \\
\midrule
\multirow{5}{*}{Autoregressive}
& \textsc{Qwen3-8B} & 82.9\% & 68.8\% & \textbf{42.3\%} & \textbf{26.0\%} \\
& \textsc{Seed-Coder-8B-Instruct} & \textbf{84.8\%} & \textbf{70.8\%} & 34.2\% & 22.1\% \\
& \textsc{DeepSeek-Coder-6.7B-Instruct} & 77.4\% & 64.0\% & 16.8\% & 15.3\% \\
& \textsc{CodeLlama-7B-Instruct-hf} & 40.2\% & 39.6\% & 9.9\% & 12.2\% \\
& \cellcolor{gray!15}Average & 71.3\% & 60.8\% & 25.8\% & 18.9\% \\
\bottomrule
\\[-6pt]
\multicolumn{6}{@{}l@{}}{\footnotesize \textsuperscript{†} \textit{
For closed-source models without public APIs, we report the results published on their official websites or in their technical reports.
}}
\end{tabular}
}
\vspace{-0.15in}
\end{table}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.41\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=4.9cm]{assets/veen/veen-all-and.pdf}
}
\vspace{-8pt}
\caption{Venn diagram of tasks solved by diffusion and AR LLMs across HumanEval, MBPP, and LiveCodeBench\textsuperscript{\textcolor{blue}{v1--v6}}.}
\label{fig:rq1_veen}
\end{minipage}
\hspace{0.3cm}
\begin{minipage}[t]{0.52\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=4.9cm]{assets/rq1_evolve.pdf}
}
\caption{
Performance trajectory of representative diffusion LLMs and LLMs on HumanEval, measured by pass@1.
}
\label{fig:rq1_evolve}
\end{minipage}
\hspace{0.6cm}
\vspace{-0.18in}
\end{figure}
\subsection{RQ2: Impact of Diffusion LLM Settings on Effectiveness}
\label{sec:rq2}
\input{figs/rq2_length}
\textbf{Motivation.}
The usability and quality of generated code are essential for real-world applications. In this section, we aim to investigate which factors influence the effectiveness of diffusion LLMs in code generation. We consider several factors that are expected to play an important role, including diffusion-specific factors such as the number of diffusion steps, the remasking strategy, the generation length, and whether to adopt block diffusion, as well as more general factor commonly studied in AR models such as decoding temperature.
\subsubsection{Impact of Generation Length} \label{sec:rq2_length}
Diffusion LLMs typically begin inference from a fully masked sequence of a predefined generation length, which largely determines the number of tokens that will eventually be generated. We believe this factor plays an important role in shaping the usability of the generated code.
\vspace{4px} \noindent
\textbf{Setting.}
We evaluate five diffusion LLMs, including \textsc{LLaDA-1.5}, \textsc{LLaDA-8B-Instruct}, \textsc{Dream-v0-Instruct-7B}, \textsc{Dream-Coder-v0-Instruct-7B}, and \textsc{DiffuCoder-7B-cpGRPO}, under different generation lengths. The values of 64, 128, 256, 512, 1024, and 2048 are considered. All models are assessed on HumanEval and MBPP with pass@1 as the evaluation metric.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
The results are shown in Figure~\ref{fig:rq2_length}.
\textbf{The usability of generated code generally increases rapidly with longer generation lengths and then gradually declines.}
For instance, on HumanEval, the pass@1 score of \textsc{Dream-Coder-v0-Instruct-7B} reaches its peak when the generation length expands from 64 to 128, but then slowly decreases as the length grows from 128 to 2048. Short sequences lack sufficient capacity to produce complete code, which substantially limits usability. Conversely, longer sequences tend to encourage the model to generate additional tokens.
While prior work has suggested that excessively long generation length can reduce output quality~\cite{longllada}, in code generation tasks this effect still exists but is not particularly severe. We attribute this to the fact that the extra content is often benign comments rather than faulty code (as shown in Figure~\ref{fig:case_length}).
\input{figs/case-generaltion-length}
\textbf{The optimal generation length varies across models.} For example, on HumanEval, the best setting for \textsc{LLaDA-8B-Instruct} is 1024, whereas for \textsc{DiffuCoder-7B-cpGRPO} the best length is only 128. This shows that diffusion LLMs differ considerably in their optimal generation length.
\textbf{The optimal generation length also differs across different programming problems.} For most models, the best length on HumanEval is shorter than that on MBPP. For instance, both \textsc{Dream-Coder-v0-Instruct-7B} and \textsc{DiffuCoder-7B-cpGRPO} perform best with a length of 128 on HumanEval, while the optimal settings shift to 256 and 512 on MBPP. We hypothesize that this difference arises from the characteristics of the programming problems. HumanEval provides function signatures and docstrings, which means the model only needs to complete the function body. In contrast, MBPP requires the generation of a full function, which typically demands a larger number of tokens.
\subsubsection{Impact of Diffusion Steps}
\label{sec:rq2_step}
A distinctive feature of diffusion LLMs is the ability to control the number of diffusion steps, which specifies how many tokens are generated in a single forward pass. As the number of steps increases, fewer tokens are produced per pass, allowing the model to perform more thorough refinement, which may lead to higher-quality code.
\input{figs/rq2_step}
\vspace{4px} \noindent
\textbf{Setting.}
The evaluated models, benchmarks, and metric are the same as in Section~\ref{sec:rq2_length}, with diffusion steps set to 8, 16, 32, 64, 128, 256, and 512. The generation length is fixed at 512 for all experiments.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
The results are shown in Figure~\ref{fig:rq2_step}.
\textbf{The usability of generated code generally improves as the number of diffusion steps increases.}
When the number of steps is set to 8, the pass@1 scores of nearly all models are close to zero. In contrast, most models achieve their best performance when the number of steps reaches 512, where each step effectively generates only a single token.
\textbf{When the number of diffusion steps is comparable to the generation length, diffusion LLMs for code generation become relatively insensitive to this parameter.}
We observe that increasing the number of steps from 256 to 512 yields only marginal improvements in effectiveness on HumanEval and MBPP for most models. This finding differs from earlier studies on natural language process, which reported that setting the number of diffusion steps equal to the generation length leads to much stronger performance~\cite{dream}.
We hypothesize that this discrepancy arises from the lower frequency and higher regularity of code tokens, which reduces the benefits of performing additional refinement steps.
\subsubsection{Impact of Remasking Strategy} \label{sec:rq2_remask}
During each inference step, every \texttt{[MASK]} token produces a predicted token. However, only a subset of these predictions is retained, while the remaining positions are remasked for further refinement.
\input{tabs/rq2_remask}
\vspace{4px} \noindent
\textbf{Setting.}
We evaluate two widely used remasking strategies: random remasking and low-confidence remasking. Random remasking assigns mask positions uniformly at random in each step, whereas low-confidence remasking selectively masks tokens with lower prediction confidence.
The evaluated models, benchmarks, and metric are the same as in Section~\ref{sec:rq2_length}, with generation length fixed at 512.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
The results are shown in Table~\ref{tab:rq2_remask}.
\textbf{Low-confidence remasking significantly outperforms random remasking.}
For example, on HumanEval, \textsc{LLaDA-1.5} and \textsc{Dream-Coder-v0-Instruct-7B} achieve performance improvements of 33.0\% and 33.6\%, respectively, when using low-confidence remasking instead of random remasking. Through case analysis, we observe that \textbf{low-confidence remasking encourages the model to first establish a high-level code structure and then progressively refine the implementation details}. For instance, the model tends to generate structural components such as \texttt{"if"} statements or \texttt{"for"} loops in the early steps, followed by filling in the specific content within these constructs.
\subsubsection{Impact of Block Length} \label{sec:rq2_block}
Some diffusion LLMs such as \textsc{LLaDA-8B-Instruct} support block diffusion generation, which can improve output quality by dividing the entire sequence into several blocks and predicting them sequentially from left to right, with each block decoded using the standard diffusion procedure.
\input{figs/rq2_block}
\vspace{4px} \noindent
\textbf{Setting.}
We evaluate \textsc{LLaDA-8B-Instruct} and \textsc{LLaDA-1.5} with generation length fixed at 512, varying block length from 1 to 512. Experiments are conducted on HumanEval and MBPP using pass@1 as the metric.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
The results are shown in Figure~\ref{fig:rq2_block}.
\textbf{Both very short and very long block lengths negatively affect effectiveness.} For example, on HumanEval, \textsc{LLaDA-8B-Instruct} peaks on HumanEval with a block length of 16.
One possible explanation is that programming is often a process of non-sequential back and forth refinement. Very short block lengths may limit the ability of diffusion LLMs to exploit this advantage.
At the same time, because a large portion of the training data for diffusion LLMs consists of natural language text written in a left-to-right order, the models may develop a tendency to perform better under left-to-right generation, which could explain the observed decline in effectiveness when block length becomes excessively long.
\subsubsection{Impact of Decoding Temperature} \label{sec:rq2_temperature}
Temperature is a parameter shared by both AR and diffusion LLMs that controls the level of randomness in the generated outputs. Higher temperatures generally produce more diverse results. For code generation, AR models are often used with temperature set to 0 in order to maximize the usability of the generated code.
\vspace{4px} \noindent
\textbf{Setting.}
We conduct experiments on all available open-source diffusion LLMs, and additionally include two representative AR baselines, \textsc{Seed-Coder-8B-Instruct} and \textsc{CodeLlama-7B-Instruct-hf}.
The temperature is gradually increased from 0 to 2, and models are evaluated on HumanEval using pass@5 and mean pass@1 as metrics.
\input{figs/temperature_step}
\vspace{4px} \noindent
\textbf{Results and Analyses.}
Selected results are shown in Figure~\ref{fig:rq2_temperature}, while complete results are provided in the \textit{Supplementary Materials} due to space limitations.
\textbf{The effectiveness of diffusion LLMs in code generation first increases with temperature and then decreases.} For example, on \textsc{LLaDA-1.5}, pass@1 peaks at a temperature of 0.4, while pass@5 reaches its maximum at 1.2. In contrast, AR models generally achieve their best pass@1 performance at a temperature of 0. These findings suggest that diffusion LLMs benefit from a moderate degree of randomness when applied to code generation. The underlying reasons for this behavior remain unclear and present an interesting direction for future research.
\begin{boxK}
\small \faIcon{pencil-alt} \textbf{Answer to RQ2:}
The code generation ability of diffusion LLMs is impacted by multiple factors. It generally improves with an increasing number of diffusion steps. In contrast, both generation length and decoding temperature show a non-monotonic relationship with performance: effectiveness rises up to a certain point and then declines. Besides, applying low-confidence remasking and adopting block diffusion significantly enhances overall effectiveness in code generation.
\end{boxK}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq2_length_humaneval.pdf}
\caption{HumanEval}
\label{fig:rq2_length_humaneval}
\end{subfigure}
\hspace{0.06cm}
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq2_length_mbpp.pdf}
\caption{MBPP}
\label{fig:rq2_length_mbpp}
\end{subfigure}
\vspace{-0.3cm}
\caption{Impact of different generation length on the effectiveness of diffusion LLMs.}
\label{fig:rq2_length}
\vspace{-0.19in}
\end{figure*}
\begin{wrapfigure}{r}{0.41\textwidth} %
\vspace{-0.9em} %
\centering
\includegraphics[width=\linewidth]{assets/case-length.pdf}
\caption{Case study illustrating the impact of different generation lengths on code usability in diffusion LLMs.}
\label{fig:case_length}
\vspace{-0.3in} %
\end{wrapfigure}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.485\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq2_step_humaneval.pdf}
\caption{HumanEval}
\label{fig:rq2_step_humaneval}
\end{subfigure}
\hspace{0.03cm}
\begin{subfigure}[t]{0.485\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq2_step_mbpp.pdf}
\caption{MBPP}
\label{fig:rq2_step_mbpp}
\end{subfigure}
\vspace{-0.3cm}
\caption{Impact of different diffusion steps on the effectiveness of diffusion LLMs.}
\label{fig:rq2_step}
\vspace{-0.18in}
\end{figure*}
\begin{table}[!t]
\centering
\small
\caption{Impact of different remask strategy on the effectiveness of diffusion LLMs.}
\label{tab:rq2_remask}
\vspace{-0.1in}
\scalebox{0.86}{
\begin{tabular}{@{}c|cc|cc@{}}
\toprule
\multirow{2}{*}{\textbf{Model}} &
\multicolumn{2}{c}{\textbf{HumanEval}} &
\multicolumn{2}{|c}{\textbf{MBPP}} \\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
& \textbf{Random} & \textbf{Low-Confidence} & \textbf{Random} & \textbf{Low-Confidence} \\ \midrule
\textsc{LLaDA-8B-Instruct} & 11.6\% & \textbf{43.9\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}32.3\%)} & 17.4\% & \textbf{38.2\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}20.8\%)} \ \\
\textsc{LLaDA-1.5} & 15.2\% & \textbf{48.2\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}33.0\%)} & 16.8\% & \textbf{40.6\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}23.8\%)} \ \\
\textsc{Dream-v0-Instruct-7B} & 20.7\% & \textbf{42.1\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}21.4\%)} & 28.2\% & \textbf{51.4\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}23.2\%)} \ \\
\textsc{Dream-Coder-v0-Instruct-7B} & 34.1\% & \textbf{67.7\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}33.6\%)} & 31.0\% & \textbf{58.6\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}27.6\%)} \ \\
\textsc{DiffuCoder-7B-cpGRPO} & 40.2\% & \textbf{69.5\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}29.3\%)} & 46.6\% & \textbf{62.2\%} \textcolor[HTML]{8B0000}{(\textcolor[HTML]{8B0000}15.6\%)} \ \\
\bottomrule
\end{tabular}
}
\vspace{-0.12in}
\end{table}
\begin{figure*}[t]
\vspace{-0.3cm}
\centering
\includegraphics[width=0.68\textwidth]{assets/rq2_block.pdf}
\vspace{-0.2cm}
\caption{Impact of different block length on the effectiveness of diffusion LLMs.}
\label{fig:rq2_block}
\vspace{-0.1in}
\end{figure*}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.48\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=3.85cm]{assets/rq2_temperature.pdf}
}
\vspace{4pt}
\caption{Impact of different decoding temperature on the effectiveness of diffusion and AR LLMs.}
\label{fig:rq2_temperature}
\end{minipage}
\hspace{0.2cm}
\begin{minipage}[t]{0.48\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=4.1cm]{assets/step/new-humaneval-diffucoder.pdf}
}
\vspace{-0.11in}
\caption{
Trade-off between time efficiency and effectiveness of \textsc{DiffuCoder-7B-cpGRPO} under different diffusion steps.
}
\label{fig:tradeoff}
\end{minipage}
\hspace{0.6cm}
\vspace{-0.1in}
\end{figure}
\subsection{RQ3: Impact of Diffusion LLM Settings on Efficiency}
\label{sec:rq3}
\textbf{Motivation.}
Efficiency is a critical consideration in code generation, where both real-time responsiveness and low inference cost are often required. A primary advantage of diffusion LLMs is their inherent ability to support parallel decoding. In this section, we investigate how to fully leverage this property to improve efficiency in code generation. We focus on two key factors: the number of diffusion steps and the generation length.
\subsubsection{Impact of Diffusion Steps} \textcolor{black!0}{.}
\label{sec:rq3_step}
\input{figs/rq3_step_efficiency}
\input{figs/rq3_step}
The experimental setting follows Section~\ref{sec:rq2_step}.
Due to space constraints, we present only a subset of the results on HumanEval, while the complete results are provided in the \textit{Supplementary Materials}. The reported results are shown in Figure~\ref{fig:computational_efficiency}, Figure~\ref{fig:time_efficiency}, and Figure~\ref{fig:tradeoff}.
For reference, we also present the best FLOPs and throughput obtained by the four AR baselines.
\textbf{As the number of diffusion steps decreases, both the computational and time efficiency of diffusion LLMs improve substantially.}
For example, when the number of steps is reduced from 512 to 256, the FLOPs required by \textsc{DiffuCoder-7B-cpGRPO} are approximately halved. Similarly, when the steps are reduced from 16 to 8, the throughput of \textsc{Dream-Coder-v0-Instruct-7B} increases from 386 tokens per second to 765 tokens per second.
\textbf{Compared with AR LLMs, diffusion LLMs exhibit substantially greater potential for efficiency improvement.}
For instance, when the number of diffusion steps in \textsc{DiffuCoder-7B-cpGRPO} is reduced from 512 to 8, the FLOPs shrink to only 1.6\% of the original value, while the throughput increases by a factor of 63. These results highlight the promising prospects of diffusion LLMs for advancing efficiency.
\textbf{There exists a clear trade-off between efficiency and effectiveness.}
As shown in Figure~\ref{fig:tradeoff}, when the number of steps in \textsc{DiffuCoder-7B-cpGRPO} is decreased from 512 to 8, its throughput on HumanEval rises from 13 tokens per second to 816 tokens per second, while its pass@1 drops from 61.59\% to 28.66\%. Nevertheless, we find \textbf{an appropriate choice of diffusion steps can balance efficiency and effectiveness}. For example, on MBPP, \textsc{DiffuCoder-7B-cpGRPO} exhibits only a modest decline in pass@1 when the number of steps is set to 64 or 128, while its throughput already surpasses that of the fastest AR baselines.
\subsubsection{Impact of Generation Length} \textcolor{black!0}{.}
\label{sec:rq3_length}
\input{figs/rq3_length}
The experimental setting follows Section~\ref{sec:rq2_length}. Selected results are presented in Figure~\ref{fig:rq3_length_com} and Figure~\ref{fig:rq3_length_time}, with complete results provided in the Supplementary Materials.
\textbf{Longer generation lengths substantially reduce efficiency.}
For example, when the generation length increases from 64 to 2048, the FLOPs of \textsc{DiffuCoder-7B-cpGRPO} on MBPP increase by nearly two orders of magnitude, significantly reducing computational efficiency. This increase in cost also impacts time efficiency, with throughput dropping from 31.1 tokens per second to 4.91 tokens per second. This slowdown occurs because current diffusion LLMs adopt bidirectional attention, where all tokens participate in attention computation during each forward pass, which amplifies the computational cost and lowers efficiency. This suggests that for practical code generation tasks it is beneficial to select a generation length that is as short as possible while still sufficient to produce complete and functional code.
\begin{boxK}
\small \faIcon{pencil-alt} \textbf{Answer to RQ3:}
The computational and time efficiency of diffusion LLMs in code generation is primarily determined by the number of diffusion steps and the generation length. Smaller diffusion steps and shorter generation lengths substantially accelerate inference speed, although they may involve trade-offs with effectiveness.
\end{boxK}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.49\linewidth}
\centering
\begin{subfigure}[t]{0.492\linewidth}
\centering
\includegraphics[width=\linewidth]{assets/step/new-humaneval-diffucoder-com.pdf}
\label{fig:rq2_step_humaneval1}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.492\linewidth}
\centering
\includegraphics[width=\linewidth]{assets/step/new-humaneval-dreamcoder-com.pdf}
\label{fig:rq2_step_humaneval2}
\end{subfigure}
\vspace{-0.25in}
\caption{Impact of different diffusion steps on computational efficiency.}
\label{fig:computational_efficiency}
\end{minipage}
\hspace{0.1cm}
\begin{minipage}[t]{0.49\linewidth}
\centering
\begin{subfigure}[t]{0.492\linewidth}
\centering
\includegraphics[width=\linewidth]{assets/step/new-humaneval-diffucoder-time.pdf}
\label{fig:rq2_step_humaneval1}
\end{subfigure}
\hfill
\begin{subfigure}[t]{0.492\linewidth}
\centering
\includegraphics[width=\linewidth]{assets/step/new-humaneval-dreamcoder-time.pdf}
\label{fig:rq2_step_humaneval2}
\end{subfigure}
\vspace{-0.25in}
\caption{Impact of different diffusion steps on time efficiency.}
\label{fig:time_efficiency}
\end{minipage}
\vspace{-0.4cm}
\end{figure}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.48\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=3.9cm]{assets/rq3_length_com.pdf}
}
\caption{
Impact of different generation length on the computational efficiency of diffusion LLMs.
}
\label{fig:rq3_length_com}
\end{minipage}
\hspace{0.2cm}
\begin{minipage}[t]{0.48\linewidth}
\centering
\adjustbox{valign=t}{%
\includegraphics[height=3.9cm]{assets/rq3_length_time.pdf}
}
\caption{
Impact of different generation length on the time efficiency of diffusion LLMs.
}
\label{fig:rq3_length_time}
\end{minipage}
\hspace{0.6cm}
\vspace{-0.5cm}
\end{figure}
\subsection{RQ4: Performance of Diffusion LLMs on Long Code Understanding}
\label{sec:rq4}
\textbf{Motivation.}
Code generation often requires LLMs to understand the long code context, such as repo-level code generation \cite{EvoCodeBench}. In this RQ, we evaluate the long code understanding ability of diffusion LLMs.
\vspace{4px} \noindent
\textbf{Setting.}
We conduct experiments on a popular long code understanding benchmark - RepoQA \cite{repoqa} and employ retrieval accuracy as a metric. We evaluate four representative diffusion LLMs, including \textsc{LLaDA-1.5}, \textsc{DiffuCoder-7B-cpGRPO}, \textsc{Dream-Coder-v0-Instruct-7B}, and \textsc{Mercury-Coder-Small}. The first three support a 4k context window, while the last supports a 32k context window. For comparison, we also include two AR baselines with matching context windows for a fair comparison: \textsc{Llama-2-7B-chat-hf} with 4k window and \textsc{Qwen3-8B} with 32k window.
\vspace{4px} \noindent
\textbf{Results and Analyses.}
As shown in Figure~\ref{fig:rq4_context}, diffusion LLMs remain relatively robust as context length increases, whereas the performance of AR LLMs declines rapidly. At an input length of 4k tokens, the retrieval accuracy of \textsc{Llama-2-7B-chat-hf} drops below 10\%, while \textsc{DiffuCoder-7B-cpGRPO} still achieves above 30\%. We further observe that diffusion LLMs also perform well under direct length extrapolation, which is consistent with prior findings on their ability to generalize effectively to extended contexts in natural language tasks~\cite{longllada}. For example, when the context length increases from 8k to 64k tokens, which exceeds the model’s context window, \textsc{Mercury-Coder-Small} shows only about a 15\% decrease in performance, whereas \textsc{Qwen3-8B} drops by nearly 30\%. \textsc{Llama-2-7B-chat-hf} experiences a sharp degradation once the input length exceeds its 4k context window, with retrieval accuracy rapidly approaching zero. By comparison, the three diffusion LLMs evaluated under the same setting exhibit only modest declines, maintaining substantially higher accuracy even as the context length continues to grow. These findings suggest that diffusion LLMs are promising in long code understanding.
\input{figs/rq4_context}
\begin{boxK}
\small \faIcon{pencil-alt} \textbf{Answer to RQ4:}
Diffusion LLMs demonstrate strong performance in long code understanding, exhibiting substantially better length extrapolation ability than AR LLMs.
\end{boxK}
\begin{figure*}[t]
\centering
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq4_small.pdf}
\caption{Context Window = 4k}
\label{fig:rq2_step_humaneval}
\end{subfigure}
\hspace{0.12cm}
\begin{subfigure}[t]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{assets/rq4_large.pdf}
\caption{Context Window = 32k}
\label{fig:rq2_step_mbpp}
\end{subfigure}
\vspace{-0.3cm}
\caption{Performance of diffusion and AR LLMs on long code understanding.}
\label{fig:rq4_context}
\vspace{-0.1in}
\end{figure*}
\section{Discussion}
\label{sec:discussion}
\subsection{Future Work}
\label{sec:future}
Based on our study results and prior work, we outline several promising directions for future research on diffusion LLMs for code generation.
\textbf{Integration of AR and Diffusion LLMs.}
Our experiments in Section~\ref{sec:rq1} show that diffusion and AR LLMs exhibit complementary strengths in code generation, with each paradigm capable of solving tasks that the other struggles to complete. This complementarity suggests that future research could explore adaptive frameworks that dynamically select the most suitable paradigm according to the characteristics of a task. Another possibility is to design collaborative systems in which AR models verify or refine the outputs of diffusion models in parallel, thereby improving both accuracy and reliability. Such hybrid approaches may also provide new insights into the broader question of how different generation paradigms can be integrated to achieve stronger overall performance.
\textbf{Structure-Aware Remasking Strategies.}
As discussed in Section~\ref{sec:rq2_remask}, existing remasking strategies mainly include random and low-confidence remasking. While low-confidence remasking has already proven effective in natural language tasks, code generation introduces unique structural information that can be exploited. Future strategies could leverage abstract syntax trees, control-flow graphs, or data-flow dependencies to guide the remasking process. For example, a strategy that prioritizes generating function signatures before their implementations may better reflect natural programming workflows. By incorporating structural signals, such approaches have the potential to substantially improve both syntactic correctness and semantic coherence.
\textbf{Adaptive generation length.}
Existing diffusion LLMs require a predefined generation length, which is not always optimal. As shown in Sections~\ref{sec:rq2_length} and~\ref{sec:rq3_length}, overly short lengths hinder the production of complete programs, while overly long lengths encourage redundant comments and reduce inference speed. Future research could investigate adaptive methods that estimate the appropriate generation length during inference based on task complexity, model confidence, or intermediate validation signals. Such adaptive mechanisms would bring diffusion LLMs closer to practical deployment, where flexibility and efficiency are equally important.
\textbf{Code completion.}
Diffusion LLMs support in-place prompting, where the model is given a prefix, a suffix, and masked segments in between, and is tasked with filling in the masked portions. This capability makes them well suited for code completion naturally. Exploring how to adapt diffusion LLMs for code completion tasks presents a promising research direction. This would align closely with how programmers actually work and has the potential to integrate automated generation more seamlessly into human-in-the-loop development.
\textbf{KV cache for diffusion LLMs.}
Research on KV cache for diffusion LLMs is still at an early stage~\cite{fastdllm, liu2025dllm, wang2025diffusion}, which currently limits the efficiency of long-context code understanding and generation. Effective caching mechanisms could dramatically reduce computational overhead, making diffusion models more practical for repository-level tasks. Investigating how to design KV cache structures that align with the iterative denoising process of diffusion models is a compelling direction, as it would not only improve efficiency but also enable diffusion LLMs to scale more effectively to complex, real-world codebases.
\subsection{Threats to Validity}
\textbf{Potential Data Leakage.}
Since LLMs are trained on vast amounts of data, data leakage poses a serious threat to the validity of our experiments. To mitigate this risk, in addition to reporting the overall results that cover versions v1–v6 of LiveCodeBench, we also provide separate results for its most recent version. This approach helps reduce the possibility of contamination and enhances the reliability of our experimental findings.
\textbf{Replication of Our Experiments.}
The performance of diffusion LLMs is influenced by multiple factors, which introduces controllability but also poses challenges for replication. To ensure reproducibility, we provide detailed experimental settings and links to the model weights in the \textit{Supplementary Materials}. In addition, we make our code repository publicly available. These efforts enhance transparency and help ensure that the results of our experiments can be reproduced with consistency in code generation tasks.
\textbf{Restricted Model Scale.}
Although we have conducted a comprehensive study of nine diffusion LLMs for code generation, almost all the models we evaluated are smaller than 10B parameters. This limitation exists because diffusion LLMs have not yet been scaled to larger sizes. As a result, our findings may not fully generalize to larger diffusion LLMs. We expect future research to investigate diffusion LLMs at substantially greater scales, comparable to models such as GPT-4~\cite{achiam2023gpt}, in order to validate and extend our findings.
\section{Conclusion}
This paper conducts the first comprehensive empirical study that systematically investigates the performance of diffusion large language models for code generation. Our results demonstrate that diffusion models are competitive with state-of-the-art AR LLMs and align well with the process of programming. We further reveal how different factors influence the effectiveness and efficiency of diffusion LLMs, providing useful guidance for their practical use in code generation. In addition, diffusion LLMs show notable advantages when understanding long code. Building on these findings, we outline several promising directions for future research and highlight opportunities to advance the development of more effective diffusion LLMs for code generation. Overall, our study highlights the significant potential of diffusion LLMs in this domain and calls for greater attention from both academia and industry to advance their application across a wider range of code-related tasks.
\section{Data Availability}
The source code and access details for the datasets are provided at: \url{https://github.com/zhangyitonggg/dllm4code}.
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{52}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{} and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showURL}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showURL #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Achiam et~al\mbox{.}(2023)]%
{achiam2023gpt}
\bibfield{author}{\bibinfo{person}{Josh Achiam}, \bibinfo{person}{Steven
Adler}, \bibinfo{person}{Sandhini Agarwal}, \bibinfo{person}{Lama Ahmad},
\bibinfo{person}{Ilge Akkaya}, \bibinfo{person}{Florencia~Leoni Aleman},
\bibinfo{person}{Diogo Almeida}, \bibinfo{person}{Janko Altenschmidt},
\bibinfo{person}{Sam Altman}, \bibinfo{person}{Shyamal Anadkat},
{et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Gpt-4 technical report}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2303.08774}}
(\bibinfo{year}{2023}).
\newblock
\bibitem[Austin et~al\mbox{.}(2021)]%
{mbpp}
\bibfield{author}{\bibinfo{person}{Jacob Austin}, \bibinfo{person}{Augustus
Odena}, \bibinfo{person}{Maxwell Nye}, \bibinfo{person}{Maarten Bosma},
\bibinfo{person}{Henryk Michalewski}, \bibinfo{person}{David Dohan},
\bibinfo{person}{Ellen Jiang}, \bibinfo{person}{Carrie Cai},
\bibinfo{person}{Michael Terry}, \bibinfo{person}{Quoc Le}, {et~al\mbox{.}}}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Program synthesis with large language models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2108.07732}}
(\bibinfo{year}{2021}).
\newblock
\bibitem[Bi et~al\mbox{.}(2024)]%
{bi2024iterative}
\bibfield{author}{\bibinfo{person}{Zhangqian Bi}, \bibinfo{person}{Yao Wan},
\bibinfo{person}{Zheng Wang}, \bibinfo{person}{Hongyu Zhang},
\bibinfo{person}{Batu Guan}, \bibinfo{person}{Fangxin Lu},
\bibinfo{person}{Zili Zhang}, \bibinfo{person}{Yulei Sui},
\bibinfo{person}{Hai Jin}, {and} \bibinfo{person}{Xuanhua Shi}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Iterative refinement of project-level code context
for precise code generation with compiler feedback}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2403.16792}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Chen et~al\mbox{.}(2021)]%
{humaneval}
\bibfield{author}{\bibinfo{person}{Mark Chen}, \bibinfo{person}{Jerry Tworek},
\bibinfo{person}{Heewoo Jun}, \bibinfo{person}{Qiming Yuan},
\bibinfo{person}{Henrique Ponde De~Oliveira Pinto}, \bibinfo{person}{Jared
Kaplan}, \bibinfo{person}{Harri Edwards}, \bibinfo{person}{Yuri Burda},
\bibinfo{person}{Nicholas Joseph}, \bibinfo{person}{Greg Brockman},
{et~al\mbox{.}}} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Evaluating large language models trained on code}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2107.03374}}
(\bibinfo{year}{2021}).
\newblock
\bibitem[Gong et~al\mbox{.}(2024)]%
{gong2024scaling}
\bibfield{author}{\bibinfo{person}{Shansan Gong}, \bibinfo{person}{Shivam
Agarwal}, \bibinfo{person}{Yizhe Zhang}, \bibinfo{person}{Jiacheng Ye},
\bibinfo{person}{Lin Zheng}, \bibinfo{person}{Mukai Li},
\bibinfo{person}{Chenxin An}, \bibinfo{person}{Peilin Zhao},
\bibinfo{person}{Wei Bi}, \bibinfo{person}{Jiawei Han}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Scaling diffusion language models via adaptation
from autoregressive models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2410.17891}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Gong et~al\mbox{.}(2025)]%
{diffucoder}
\bibfield{author}{\bibinfo{person}{Shansan Gong}, \bibinfo{person}{Ruixiang
Zhang}, \bibinfo{person}{Huangjie Zheng}, \bibinfo{person}{Jiatao Gu},
\bibinfo{person}{Navdeep Jaitly}, \bibinfo{person}{Lingpeng Kong}, {and}
\bibinfo{person}{Yizhe Zhang}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{DiffuCoder: Understanding and Improving Masked
Diffusion Models for Code Generation}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2506.20639}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[{Google DeepMind}(2025)]%
{geminidiffusion}
\bibfield{author}{\bibinfo{person}{{Google DeepMind}}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{Gemini Diffusion}.
\newblock
\urldef\tempurl%
\url{https://blog.google/technology/google-deepmind/gemini-diffusion/}
\showURL{%
\tempurl}
\bibitem[Guo et~al\mbox{.}(2024)]%
{deepseekcoder}
\bibfield{author}{\bibinfo{person}{Daya Guo}, \bibinfo{person}{Qihao Zhu},
\bibinfo{person}{Dejian Yang}, \bibinfo{person}{Zhenda Xie},
\bibinfo{person}{Kai Dong}, \bibinfo{person}{Wentao Zhang},
\bibinfo{person}{Guanting Chen}, \bibinfo{person}{Xiao Bi},
\bibinfo{person}{Yu Wu}, \bibinfo{person}{YK Li}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{DeepSeek-Coder: When the Large Language Model Meets
Programming--The Rise of Code Intelligence}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2401.14196}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Hui et~al\mbox{.}(2024)]%
{qwencoder}
\bibfield{author}{\bibinfo{person}{Binyuan Hui}, \bibinfo{person}{Jian Yang},
\bibinfo{person}{Zeyu Cui}, \bibinfo{person}{Jiaxi Yang},
\bibinfo{person}{Dayiheng Liu}, \bibinfo{person}{Lei Zhang},
\bibinfo{person}{Tianyu Liu}, \bibinfo{person}{Jiajun Zhang},
\bibinfo{person}{Bowen Yu}, \bibinfo{person}{Keming Lu}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Qwen2. 5-coder technical report}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2409.12186}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Islam et~al\mbox{.}(2025)]%
{islam2025codesim}
\bibfield{author}{\bibinfo{person}{Md~Ashraful Islam},
\bibinfo{person}{Mohammed~Eunus Ali}, {and} \bibinfo{person}{Md~Rizwan
Parvez}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Codesim: Multi-agent code generation and problem
solving through simulation-driven planning and debugging}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2502.05664}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Jaech et~al\mbox{.}(2024)]%
{o1}
\bibfield{author}{\bibinfo{person}{Aaron Jaech}, \bibinfo{person}{Adam Kalai},
\bibinfo{person}{Adam Lerer}, \bibinfo{person}{Adam Richardson},
\bibinfo{person}{Ahmed El{-}Kishky}, \bibinfo{person}{Aiden Low},
\bibinfo{person}{Alec Helyar}, \bibinfo{person}{Aleksander Madry},
\bibinfo{person}{Alex Beutel}, \bibinfo{person}{Alex Carney},
\bibinfo{person}{Alex Iftimie}, \bibinfo{person}{Alex Karpenko},
\bibinfo{person}{Alex~Tachard Passos}, \bibinfo{person}{Alexander Neitz},
\bibinfo{person}{Alexander Prokofiev}, \bibinfo{person}{Alexander Wei},
\bibinfo{person}{Allison Tam}, \bibinfo{person}{Ally Bennett},
\bibinfo{person}{Ananya Kumar}, \bibinfo{person}{Andre Saraiva},
\bibinfo{person}{Andrea Vallone}, \bibinfo{person}{Andrew Duberstein},
\bibinfo{person}{Andrew Kondrich}, \bibinfo{person}{Andrey Mishchenko},
\bibinfo{person}{Andy Applebaum}, \bibinfo{person}{Angela Jiang},
\bibinfo{person}{Ashvin Nair}, \bibinfo{person}{Barret Zoph},
\bibinfo{person}{Behrooz Ghorbani}, \bibinfo{person}{Ben Rossen},
\bibinfo{person}{Benjamin Sokolowsky}, \bibinfo{person}{Boaz Barak},
\bibinfo{person}{Bob McGrew}, \bibinfo{person}{Borys Minaiev},
\bibinfo{person}{Botao Hao}, \bibinfo{person}{Bowen Baker},
\bibinfo{person}{Brandon Houghton}, \bibinfo{person}{Brandon McKinzie},
\bibinfo{person}{Brydon Eastman}, \bibinfo{person}{Camillo Lugaresi},
\bibinfo{person}{Cary Bassin}, \bibinfo{person}{Cary Hudson},
\bibinfo{person}{Chak~Ming Li}, \bibinfo{person}{Charles de Bourcy},
\bibinfo{person}{Chelsea Voss}, \bibinfo{person}{Chen Shen},
\bibinfo{person}{Chong Zhang}, \bibinfo{person}{Chris Koch},
\bibinfo{person}{Chris Orsinger}, \bibinfo{person}{Christopher Hesse},
\bibinfo{person}{Claudia Fischer}, \bibinfo{person}{Clive Chan},
\bibinfo{person}{Dan Roberts}, \bibinfo{person}{Daniel Kappler},
\bibinfo{person}{Daniel Levy}, \bibinfo{person}{Daniel Selsam},
\bibinfo{person}{David Dohan}, \bibinfo{person}{David Farhi},
\bibinfo{person}{David Mely}, \bibinfo{person}{David Robinson},
\bibinfo{person}{Dimitris Tsipras}, \bibinfo{person}{Doug Li},
\bibinfo{person}{Dragos Oprica}, \bibinfo{person}{Eben Freeman},
\bibinfo{person}{Eddie Zhang}, \bibinfo{person}{Edmund Wong},
\bibinfo{person}{Elizabeth Proehl}, \bibinfo{person}{Enoch Cheung},
\bibinfo{person}{Eric Mitchell}, \bibinfo{person}{Eric Wallace},
\bibinfo{person}{Erik Ritter}, \bibinfo{person}{Evan Mays},
\bibinfo{person}{Fan Wang}, \bibinfo{person}{Felipe~Petroski Such},
\bibinfo{person}{Filippo Raso}, \bibinfo{person}{Florencia Leoni},
\bibinfo{person}{Foivos Tsimpourlas}, \bibinfo{person}{Francis Song},
\bibinfo{person}{Fred von Lohmann}, \bibinfo{person}{Freddie Sulit},
\bibinfo{person}{Geoff Salmon}, \bibinfo{person}{Giambattista Parascandolo},
\bibinfo{person}{Gildas Chabot}, \bibinfo{person}{Grace Zhao},
\bibinfo{person}{Greg Brockman}, \bibinfo{person}{Guillaume Leclerc},
\bibinfo{person}{Hadi Salman}, \bibinfo{person}{Haiming Bao},
\bibinfo{person}{Hao Sheng}, \bibinfo{person}{Hart Andrin},
\bibinfo{person}{Hessam Bagherinezhad}, \bibinfo{person}{Hongyu Ren},
\bibinfo{person}{Hunter Lightman}, \bibinfo{person}{Hyung~Won Chung},
\bibinfo{person}{Ian Kivlichan}, \bibinfo{person}{Ian O'Connell},
\bibinfo{person}{Ian Osband}, \bibinfo{person}{Ignasi~Clavera Gilaberte},
{and} \bibinfo{person}{Ilge Akkaya}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{OpenAI o1 System Card}.
\newblock \bibinfo{journal}{\emph{CoRR}} \bibinfo{volume}{abs/2412.16720}
(\bibinfo{year}{2024}).
\newblock
\showeprint[arXiv]{2412.16720}
\href{https://doi.org/10.48550/ARXIV.2412.16720}{doi:\nolinkurl{10.48550/ARXIV.2412.16720}}
\bibitem[Jiang et~al\mbox{.}(2024b)]%
{jiang2024survey}
\bibfield{author}{\bibinfo{person}{Juyong Jiang}, \bibinfo{person}{Fan Wang},
\bibinfo{person}{Jiasi Shen}, \bibinfo{person}{Sungju Kim}, {and}
\bibinfo{person}{Sunghun Kim}.} \bibinfo{year}{2024}\natexlab{b}.
\newblock \showarticletitle{A survey on large language models for code
generation}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.00515}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Jiang et~al\mbox{.}(2024a)]%
{jiang2024aixcoder}
\bibfield{author}{\bibinfo{person}{Siyuan Jiang}, \bibinfo{person}{Jia Li},
\bibinfo{person}{He Zong}, \bibinfo{person}{Huanyu Liu}, \bibinfo{person}{Hao
Zhu}, \bibinfo{person}{Shukai Hu}, \bibinfo{person}{Erlu Li},
\bibinfo{person}{Jiazheng Ding}, \bibinfo{person}{Yu Han},
\bibinfo{person}{Wei Ning}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{a}.
\newblock \showarticletitle{aixcoder-7b: A lightweight and effective large
language model for code completion}.
\newblock \bibinfo{journal}{\emph{arXiv e-prints}} (\bibinfo{year}{2024}),
\bibinfo{pages}{arXiv--2410}.
\newblock
\bibitem[Labs et~al\mbox{.}(2025)]%
{mercurycoder}
\bibfield{author}{\bibinfo{person}{Inception Labs}, \bibinfo{person}{Samar
Khanna}, \bibinfo{person}{Siddhant Kharbanda}, \bibinfo{person}{Shufan Li},
\bibinfo{person}{Harshit Varma}, \bibinfo{person}{Eric Wang},
\bibinfo{person}{Sawyer Birnbaum}, \bibinfo{person}{Ziyang Luo},
\bibinfo{person}{Yanis Miraoui}, \bibinfo{person}{Akash Palrecha},
{et~al\mbox{.}}} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Mercury: Ultra-Fast Language Models Based on
Diffusion}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2506.17298}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Li et~al\mbox{.}(2025b)]%
{li2025structured}
\bibfield{author}{\bibinfo{person}{Jia Li}, \bibinfo{person}{Ge Li},
\bibinfo{person}{Yongmin Li}, {and} \bibinfo{person}{Zhi Jin}.}
\bibinfo{year}{2025}\natexlab{b}.
\newblock \showarticletitle{Structured chain-of-thought prompting for code
generation}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and
Methodology}} \bibinfo{volume}{34}, \bibinfo{number}{2}
(\bibinfo{year}{2025}), \bibinfo{pages}{1--23}.
\newblock
\bibitem[Li et~al\mbox{.}(2024a)]%
{EvoCodeBench}
\bibfield{author}{\bibinfo{person}{Jia Li}, \bibinfo{person}{Ge Li},
\bibinfo{person}{Xuanming Zhang}, \bibinfo{person}{Yunfei Zhao},
\bibinfo{person}{Yihong Dong}, \bibinfo{person}{Zhi Jin},
\bibinfo{person}{Binhua Li}, \bibinfo{person}{Fei Huang}, {and}
\bibinfo{person}{Yongbin Li}.} \bibinfo{year}{2024}\natexlab{a}.
\newblock \showarticletitle{EvoCodeBench: An Evolving Code Generation Benchmark
with Domain-Specific Evaluations}. In \bibinfo{booktitle}{\emph{Advances in
Neural Information Processing Systems 38: Annual Conference on Neural
Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada,
December 10 - 15, 2024}}, \bibfield{editor}{\bibinfo{person}{Amir
Globersons}, \bibinfo{person}{Lester Mackey}, \bibinfo{person}{Danielle
Belgrave}, \bibinfo{person}{Angela Fan}, \bibinfo{person}{Ulrich Paquet},
\bibinfo{person}{Jakub~M. Tomczak}, {and} \bibinfo{person}{Cheng Zhang}}
(Eds.).
\newblock
\urldef\tempurl%
\url{http://papers.nips.cc/paper\_files/paper/2024/hash/6a059625a6027aca18302803743abaa2-Abstract-Datasets\_and\_Benchmarks\_Track.html}
\showURL{%
\tempurl}
\bibitem[Li et~al\mbox{.}(2023)]%
{li2023skcoder}
\bibfield{author}{\bibinfo{person}{Jia Li}, \bibinfo{person}{Yongmin Li},
\bibinfo{person}{Ge Li}, \bibinfo{person}{Zhi Jin}, \bibinfo{person}{Yiyang
Hao}, {and} \bibinfo{person}{Xing Hu}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Skcoder: A sketch-based approach for automatic code
generation}. In \bibinfo{booktitle}{\emph{2023 IEEE/ACM 45th International
Conference on Software Engineering (ICSE)}}. IEEE,
\bibinfo{pages}{2124--2135}.
\newblock
\bibitem[Li et~al\mbox{.}(2024b)]%
{li2024acecoder}
\bibfield{author}{\bibinfo{person}{Jia Li}, \bibinfo{person}{Yunfei Zhao},
\bibinfo{person}{Yongmin Li}, \bibinfo{person}{Ge Li}, {and}
\bibinfo{person}{Zhi Jin}.} \bibinfo{year}{2024}\natexlab{b}.
\newblock \showarticletitle{Acecoder: An effective prompting technique
specialized in code generation}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and
Methodology}} \bibinfo{volume}{33}, \bibinfo{number}{8}
(\bibinfo{year}{2024}), \bibinfo{pages}{1--26}.
\newblock
\bibitem[Li et~al\mbox{.}(2025c)]%
{li2025aixcoder}
\bibfield{author}{\bibinfo{person}{Jia Li}, \bibinfo{person}{Hao Zhu},
\bibinfo{person}{Huanyu Liu}, \bibinfo{person}{Xianjie Shi},
\bibinfo{person}{He Zong}, \bibinfo{person}{Yihong Dong},
\bibinfo{person}{Kechi Zhang}, \bibinfo{person}{Siyuan Jiang},
\bibinfo{person}{Zhi Jin}, {and} \bibinfo{person}{Ge Li}.}
\bibinfo{year}{2025}\natexlab{c}.
\newblock \showarticletitle{aiXcoder-7B-v2: Training LLMs to Fully Utilize the
Long Context in Repository-level Code Completion}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2503.15301}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Li et~al\mbox{.}(2025a)]%
{li2025survey}
\bibfield{author}{\bibinfo{person}{Tianyi Li}, \bibinfo{person}{Mingda Chen},
\bibinfo{person}{Bowei Guo}, {and} \bibinfo{person}{Zhiqiang Shen}.}
\bibinfo{year}{2025}\natexlab{a}.
\newblock \showarticletitle{A survey on diffusion language models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.10875}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Liang et~al\mbox{.}(2024)]%
{liang2024large}
\bibfield{author}{\bibinfo{person}{Jenny~T Liang}, \bibinfo{person}{Chenyang
Yang}, {and} \bibinfo{person}{Brad~A Myers}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{A large-scale survey on the usability of ai
programming assistants: Successes and challenges}. In
\bibinfo{booktitle}{\emph{Proceedings of the 46th IEEE/ACM international
conference on software engineering}}. \bibinfo{pages}{1--13}.
\newblock
\bibitem[Liang et~al\mbox{.}(2025)]%
{discreteVLA}
\bibfield{author}{\bibinfo{person}{Zhixuan Liang}, \bibinfo{person}{Yizhuo Li},
\bibinfo{person}{Tianshuo Yang}, \bibinfo{person}{Chengyue Wu},
\bibinfo{person}{Sitong Mao}, \bibinfo{person}{Liuao Pei},
\bibinfo{person}{Xiaokang Yang}, \bibinfo{person}{Jiangmiao Pang},
\bibinfo{person}{Yao Mu}, {and} \bibinfo{person}{Ping Luo}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Discrete Diffusion VLA: Bringing Discrete Diffusion
to Action Decoding in Vision-Language-Action Policies}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.20072}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Liao et~al\mbox{.}(2024)]%
{liao20243}
\bibfield{author}{\bibinfo{person}{Dianshu Liao}, \bibinfo{person}{Shidong
Pan}, \bibinfo{person}{Xiaoyu Sun}, \bibinfo{person}{Xiaoxue Ren},
\bibinfo{person}{Qing Huang}, \bibinfo{person}{Zhenchang Xing},
\bibinfo{person}{Huan Jin}, {and} \bibinfo{person}{Qinying Li}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{A 3-codgen: A repository-level code generation
framework for code reuse with local-aware, global-aware, and
third-party-library-aware}.
\newblock \bibinfo{journal}{\emph{IEEE Transactions on Software Engineering}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Liu et~al\mbox{.}(2024)]%
{repoqa}
\bibfield{author}{\bibinfo{person}{Jiawei Liu}, \bibinfo{person}{Jia~Le Tian},
\bibinfo{person}{Vijay Daita}, \bibinfo{person}{Yuxiang Wei},
\bibinfo{person}{Yifeng Ding}, \bibinfo{person}{Yuhan~Katherine Wang},
\bibinfo{person}{Jun Yang}, {and} \bibinfo{person}{Lingming Zhang}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Repoqa: Evaluating long context code
understanding}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.06025}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Liu et~al\mbox{.}(2025a)]%
{longllada}
\bibfield{author}{\bibinfo{person}{Xiaoran Liu}, \bibinfo{person}{Zhigeng Liu},
\bibinfo{person}{Zengfeng Huang}, \bibinfo{person}{Qipeng Guo},
\bibinfo{person}{Ziwei He}, {and} \bibinfo{person}{Xipeng Qiu}.}
\bibinfo{year}{2025}\natexlab{a}.
\newblock \showarticletitle{LongLLaDA: Unlocking Long Context Capabilities in
Diffusion LLMs}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2506.14429}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Liu et~al\mbox{.}(2025b)]%
{liu2025dllm}
\bibfield{author}{\bibinfo{person}{Zhiyuan Liu}, \bibinfo{person}{Yicun Yang},
\bibinfo{person}{Yaojie Zhang}, \bibinfo{person}{Junjie Chen},
\bibinfo{person}{Chang Zou}, \bibinfo{person}{Qingyuan Wei},
\bibinfo{person}{Shaobo Wang}, {and} \bibinfo{person}{Linfeng Zhang}.}
\bibinfo{year}{2025}\natexlab{b}.
\newblock \showarticletitle{dllm-cache: Accelerating diffusion large language
models with adaptive caching}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2506.06295}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Mowar et~al\mbox{.}(2025)]%
{mowar2025codea11y}
\bibfield{author}{\bibinfo{person}{Peya Mowar}, \bibinfo{person}{Yi-Hao Peng},
\bibinfo{person}{Jason Wu}, \bibinfo{person}{Aaron Steinfeld}, {and}
\bibinfo{person}{Jeffrey~P Bigham}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{CodeA11y: Making AI Coding Assistants Useful for
Accessible Web Development}. In \bibinfo{booktitle}{\emph{Proceedings of the
2025 CHI Conference on Human Factors in Computing Systems}}.
\bibinfo{pages}{1--15}.
\newblock
\bibitem[M{\"u}ndler et~al\mbox{.}(2025)]%
{mundler2025constrained}
\bibfield{author}{\bibinfo{person}{Niels M{\"u}ndler}, \bibinfo{person}{Jasper
Dekoninck}, {and} \bibinfo{person}{Martin Vechev}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Constrained Decoding of Diffusion LLMs with
Context-Free Grammars}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.10111}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Nie et~al\mbox{.}(2025)]%
{llada}
\bibfield{author}{\bibinfo{person}{Shen Nie}, \bibinfo{person}{Fengqi Zhu},
\bibinfo{person}{Zebin You}, \bibinfo{person}{Xiaolu Zhang},
\bibinfo{person}{Jingyang Ou}, \bibinfo{person}{Jun Hu}, \bibinfo{person}{Jun
Zhou}, \bibinfo{person}{Yankai Lin}, \bibinfo{person}{Ji-Rong Wen}, {and}
\bibinfo{person}{Chongxuan Li}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Large language diffusion models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2502.09992}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Ouyang et~al\mbox{.}(2025)]%
{ouyang2025empirical}
\bibfield{author}{\bibinfo{person}{Shuyin Ouyang}, \bibinfo{person}{Jie~M
Zhang}, \bibinfo{person}{Mark Harman}, {and} \bibinfo{person}{Meng Wang}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{An empirical study of the non-determinism of
chatgpt in code generation}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and
Methodology}} \bibinfo{volume}{34}, \bibinfo{number}{2}
(\bibinfo{year}{2025}), \bibinfo{pages}{1--28}.
\newblock
\bibitem[Pan et~al\mbox{.}(2024)]%
{pan2024enhancing}
\bibfield{author}{\bibinfo{person}{Zhiyuan Pan}, \bibinfo{person}{Xing Hu},
\bibinfo{person}{Xin Xia}, {and} \bibinfo{person}{Xiaohu Yang}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Enhancing repository-level code generation with
integrated contextual information}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.03283}}
(\bibinfo{year}{2024}).
\newblock
\bibitem[Roziere et~al\mbox{.}(2023)]%
{codellama}
\bibfield{author}{\bibinfo{person}{Baptiste Roziere}, \bibinfo{person}{Jonas
Gehring}, \bibinfo{person}{Fabian Gloeckle}, \bibinfo{person}{Sten Sootla},
\bibinfo{person}{Itai Gat}, \bibinfo{person}{Xiaoqing~Ellen Tan},
\bibinfo{person}{Yossi Adi}, \bibinfo{person}{Jingyu Liu},
\bibinfo{person}{Romain Sauvestre}, \bibinfo{person}{Tal Remez},
{et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Code llama: Open foundation models for code}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2308.12950}}
(\bibinfo{year}{2023}).
\newblock
\bibitem[Sahoo et~al\mbox{.}(2024)]%
{sahoo2024simple}
\bibfield{author}{\bibinfo{person}{Subham Sahoo}, \bibinfo{person}{Marianne
Arriola}, \bibinfo{person}{Yair Schiff}, \bibinfo{person}{Aaron Gokaslan},
\bibinfo{person}{Edgar Marroquin}, \bibinfo{person}{Justin Chiu},
\bibinfo{person}{Alexander Rush}, {and} \bibinfo{person}{Volodymyr
Kuleshov}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Simple and effective masked diffusion language
models}.
\newblock \bibinfo{journal}{\emph{Advances in Neural Information Processing
Systems}} \bibinfo{volume}{37} (\bibinfo{year}{2024}),
\bibinfo{pages}{130136--130184}.
\newblock
\bibitem[Seed et~al\mbox{.}(2025)]%
{seed2025seed}
\bibfield{author}{\bibinfo{person}{ByteDance Seed}, \bibinfo{person}{Yuyu
Zhang}, \bibinfo{person}{Jing Su}, \bibinfo{person}{Yifan Sun},
\bibinfo{person}{Chenguang Xi}, \bibinfo{person}{Xia Xiao},
\bibinfo{person}{Shen Zheng}, \bibinfo{person}{Anxiang Zhang},
\bibinfo{person}{Kaibo Liu}, \bibinfo{person}{Daoguang Zan}, {et~al\mbox{.}}}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Seed-Coder: Let the Code Model Curate Data for
Itself}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2506.03524}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Sergeyuk et~al\mbox{.}(2025)]%
{sergeyuk2025using}
\bibfield{author}{\bibinfo{person}{Agnia Sergeyuk}, \bibinfo{person}{Yaroslav
Golubev}, \bibinfo{person}{Timofey Bryksin}, {and} \bibinfo{person}{Iftekhar
Ahmed}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Using AI-based coding assistants in practice: State
of affairs, perceptions, and ways forward}.
\newblock \bibinfo{journal}{\emph{Information and Software Technology}}
\bibinfo{volume}{178} (\bibinfo{year}{2025}), \bibinfo{pages}{107610}.
\newblock
\bibitem[Shin et~al\mbox{.}(2021)]%
{shin2021survey}
\bibfield{author}{\bibinfo{person}{Jiho Shin}, \bibinfo{person}{Jaechang Nam},
{et~al\mbox{.}}} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{A survey of automatic code generation from natural
language}.
\newblock \bibinfo{journal}{\emph{Journal of Information Processing Systems}}
\bibinfo{volume}{17}, \bibinfo{number}{3} (\bibinfo{year}{2021}),
\bibinfo{pages}{537--555}.
\newblock
\bibitem[Song et~al\mbox{.}(2025)]%
{seeddiffusion}
\bibfield{author}{\bibinfo{person}{Yuxuan Song}, \bibinfo{person}{Zheng Zhang},
\bibinfo{person}{Cheng Luo}, \bibinfo{person}{Pengyang Gao},
\bibinfo{person}{Fan Xia}, \bibinfo{person}{Hao Luo}, \bibinfo{person}{Zheng
Li}, \bibinfo{person}{Yuehang Yang}, \bibinfo{person}{Hongli Yu},
\bibinfo{person}{Xingwei Qu}, {et~al\mbox{.}}}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Seed diffusion: A large-scale diffusion language
model with high-speed inference}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.02193}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Touvron et~al\mbox{.}(2023)]%
{touvron2023llama}
\bibfield{author}{\bibinfo{person}{Hugo Touvron}, \bibinfo{person}{Louis
Martin}, \bibinfo{person}{Kevin Stone}, \bibinfo{person}{Peter Albert},
\bibinfo{person}{Amjad Almahairi}, \bibinfo{person}{Yasmine Babaei},
\bibinfo{person}{Nikolay Bashlykov}, \bibinfo{person}{Soumya Batra},
\bibinfo{person}{Prajjwal Bhargava}, \bibinfo{person}{Shruti Bhosale},
{et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Llama 2: Open foundation and fine-tuned chat
models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2307.09288}}
(\bibinfo{year}{2023}).
\newblock
\bibitem[Wang et~al\mbox{.}(2025b)]%
{wang2025teaching}
\bibfield{author}{\bibinfo{person}{Chong Wang}, \bibinfo{person}{Jian Zhang},
\bibinfo{person}{Yebo Feng}, \bibinfo{person}{Tianlin Li},
\bibinfo{person}{Weisong Sun}, \bibinfo{person}{Yang Liu}, {and}
\bibinfo{person}{Xin Peng}.} \bibinfo{year}{2025}\natexlab{b}.
\newblock \showarticletitle{Teaching code llms to use autocompletion tools in
repository-level code generation}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and
Methodology}} \bibinfo{volume}{34}, \bibinfo{number}{7}
(\bibinfo{year}{2025}), \bibinfo{pages}{1--27}.
\newblock
\bibitem[Wang et~al\mbox{.}(2025a)]%
{wang2025diffusion}
\bibfield{author}{\bibinfo{person}{Xu Wang}, \bibinfo{person}{Chenkai Xu},
\bibinfo{person}{Yijie Jin}, \bibinfo{person}{Jiachun Jin},
\bibinfo{person}{Hao Zhang}, {and} \bibinfo{person}{Zhijie Deng}.}
\bibinfo{year}{2025}\natexlab{a}.
\newblock \showarticletitle{Diffusion llms can do faster-than-ar inference via
discrete diffusion forcing}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.09192}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Wermelinger(2023)]%
{wermelinger2023using}
\bibfield{author}{\bibinfo{person}{Michel Wermelinger}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Using github copilot to solve simple programming
problems}. In \bibinfo{booktitle}{\emph{Proceedings of the 54th ACM Technical
Symposium on Computer Science Education V. 1}}. \bibinfo{pages}{172--178}.
\newblock
\bibitem[Wu et~al\mbox{.}(2025)]%
{fastdllm}
\bibfield{author}{\bibinfo{person}{Chengyue Wu}, \bibinfo{person}{Hao Zhang},
\bibinfo{person}{Shuchen Xue}, \bibinfo{person}{Zhijian Liu},
\bibinfo{person}{Shizhe Diao}, \bibinfo{person}{Ligeng Zhu},
\bibinfo{person}{Ping Luo}, \bibinfo{person}{Song Han}, {and}
\bibinfo{person}{Enze Xie}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Fast-dllm: Training-free acceleration of diffusion
llm by enabling kv cache and parallel decoding}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2505.22618}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Xie et~al\mbox{.}(2025)]%
{dreamcoder}
\bibfield{author}{\bibinfo{person}{Zhihui Xie}, \bibinfo{person}{Jiacheng Ye},
\bibinfo{person}{Lin Zheng}, \bibinfo{person}{Jiahui Gao},
\bibinfo{person}{Jingwei Dong}, \bibinfo{person}{Zirui Wu},
\bibinfo{person}{Xueliang Zhao}, \bibinfo{person}{Shansan Gong},
\bibinfo{person}{Xin Jiang}, \bibinfo{person}{Zhenguo Li}, {and}
\bibinfo{person}{Lingpeng Kong}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \bibinfo{title}{Dream-Coder 7B}.
\newblock
\urldef\tempurl%
\url{https://hkunlp.github.io/blog/2025/dream-coder}
\showURL{%
\tempurl}
\bibitem[Yang et~al\mbox{.}(2025a)]%
{yang2025qwen3}
\bibfield{author}{\bibinfo{person}{An Yang}, \bibinfo{person}{Anfeng Li},
\bibinfo{person}{Baosong Yang}, \bibinfo{person}{Beichen Zhang},
\bibinfo{person}{Binyuan Hui}, \bibinfo{person}{Bo Zheng},
\bibinfo{person}{Bowen Yu}, \bibinfo{person}{Chang Gao},
\bibinfo{person}{Chengen Huang}, \bibinfo{person}{Chenxu Lv},
{et~al\mbox{.}}} \bibinfo{year}{2025}\natexlab{a}.
\newblock \showarticletitle{Qwen3 technical report}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2505.09388}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Yang et~al\mbox{.}(2025b)]%
{yang2025mmada}
\bibfield{author}{\bibinfo{person}{Ling Yang}, \bibinfo{person}{Ye Tian},
\bibinfo{person}{Bowen Li}, \bibinfo{person}{Xinchen Zhang},
\bibinfo{person}{Ke Shen}, \bibinfo{person}{Yunhai Tong}, {and}
\bibinfo{person}{Mengdi Wang}.} \bibinfo{year}{2025}\natexlab{b}.
\newblock \showarticletitle{Mmada: Multimodal large diffusion language models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2505.15809}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Ye et~al\mbox{.}(2025)]%
{dream}
\bibfield{author}{\bibinfo{person}{Jiacheng Ye}, \bibinfo{person}{Zhihui Xie},
\bibinfo{person}{Lin Zheng}, \bibinfo{person}{Jiahui Gao},
\bibinfo{person}{Zirui Wu}, \bibinfo{person}{Xin Jiang},
\bibinfo{person}{Zhenguo Li}, {and} \bibinfo{person}{Lingpeng Kong}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Dream 7B: Diffusion Large Language Models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.15487}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[You et~al\mbox{.}(2025)]%
{llada-v}
\bibfield{author}{\bibinfo{person}{Zebin You}, \bibinfo{person}{Shen Nie},
\bibinfo{person}{Xiaolu Zhang}, \bibinfo{person}{Jun Hu},
\bibinfo{person}{Jun Zhou}, \bibinfo{person}{Zhiwu Lu},
\bibinfo{person}{Ji-Rong Wen}, {and} \bibinfo{person}{Chongxuan Li}.}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Llada-v: Large language diffusion models with
visual instruction tuning}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2505.16933}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Zeng et~al\mbox{.}(2025)]%
{zeng2025treediff}
\bibfield{author}{\bibinfo{person}{Yiming Zeng}, \bibinfo{person}{Jinghan Cao},
\bibinfo{person}{Zexin Li}, \bibinfo{person}{Yiming Chen},
\bibinfo{person}{Tao Ren}, \bibinfo{person}{Dawei Xiang},
\bibinfo{person}{Xidong Wu}, \bibinfo{person}{Shangqian Gao}, {and}
\bibinfo{person}{Tingting Yu}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{TreeDiff: AST-Guided Code Generation with Diffusion
LLMs}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.01473}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Zhang et~al\mbox{.}(2025b)]%
{zhang2025focused}
\bibfield{author}{\bibinfo{person}{Kechi Zhang}, \bibinfo{person}{Ge Li},
\bibinfo{person}{Jia Li}, \bibinfo{person}{Yihong Dong}, {and}
\bibinfo{person}{Zhi Jin}.} \bibinfo{year}{2025}\natexlab{b}.
\newblock \showarticletitle{Focused-dpo: Enhancing code generation through
focused preference optimization on error-prone points}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2502.11475}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Zhang et~al\mbox{.}(2025a)]%
{zhang2025survey}
\bibfield{author}{\bibinfo{person}{Lingzhe Zhang}, \bibinfo{person}{Liancheng
Fang}, \bibinfo{person}{Chiming Duan}, \bibinfo{person}{Minghua He},
\bibinfo{person}{Leyi Pan}, \bibinfo{person}{Pei Xiao},
\bibinfo{person}{Shiyu Huang}, \bibinfo{person}{Yunpeng Zhai},
\bibinfo{person}{Xuming Hu}, \bibinfo{person}{Philip~S Yu}, {et~al\mbox{.}}}
\bibinfo{year}{2025}\natexlab{a}.
\newblock \showarticletitle{A Survey on Parallel Text Generation: From Parallel
Decoding to Diffusion Language Models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2508.08712}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Zhu et~al\mbox{.}(2025)]%
{llada15}
\bibfield{author}{\bibinfo{person}{Fengqi Zhu}, \bibinfo{person}{Rongzhen
Wang}, \bibinfo{person}{Shen Nie}, \bibinfo{person}{Xiaolu Zhang},
\bibinfo{person}{Chunwei Wu}, \bibinfo{person}{Jun Hu}, \bibinfo{person}{Jun
Zhou}, \bibinfo{person}{Jianfei Chen}, \bibinfo{person}{Yankai Lin},
\bibinfo{person}{Ji-Rong Wen}, {et~al\mbox{.}}}
\bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{LLaDA 1.5: Variance-Reduced Preference Optimization
for Large Language Diffusion Models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2505.19223}}
(\bibinfo{year}{2025}).
\newblock
\bibitem[Zhu et~al\mbox{.}(2024)]%
{deepseek-coder-v2}
\bibfield{author}{\bibinfo{person}{Qihao Zhu}, \bibinfo{person}{Daya Guo},
\bibinfo{person}{Zhihong Shao}, \bibinfo{person}{Dejian Yang},
\bibinfo{person}{Peiyi Wang}, \bibinfo{person}{Runxin Xu}, \bibinfo{person}{Y
Wu}, \bibinfo{person}{Yukun Li}, \bibinfo{person}{Huazuo Gao},
\bibinfo{person}{Shirong Ma}, {et~al\mbox{.}}}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Deepseek-coder-v2: Breaking the barrier of
closed-source models in code intelligence}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.11931}}
(\bibinfo{year}{2024}).
\newblock
\end{thebibliography}