FGIT: Fault-Guided Fine-Tuning for Code Generation
Extracted source text · arXiv 2503.16913
\documentclass[sigconf]{acmart}
\AtBeginDocument{%
\providecommand\BibTeX{{%
Bib\TeX}}}
\settopmatter{printacmref=false} %
\renewcommand\footnotetextcopyrightpermission[1]{} %
\pagestyle{plain} %
\setcopyright{acmlicensed}
\usepackage{graphicx} %
\usepackage{algorithm}
\usepackage{appendix}
\usepackage{algpseudocode}
\usepackage{booktabs}
\usepackage{tcolorbox}
\usepackage{amsmath}
\usepackage{enumitem}
\usepackage{arydshln}
\usepackage{xspace}
\newcommand{\boxmargin}{1mm}
\definecolor{lightcyan}{rgb}{0.88, 1.0, 1.0}
\colorlet{mythmback}{lightcyan!40!white}
\newtcolorbox{boxEnv}{
colback=mythmback,coltitle=blue,colframe=mythmback,
center,
width=\linewidth,
boxrule=0.5pt,
left=5pt,right=0pt,
top=2pt,bottom=2pt,
before skip=10pt, after skip=10pt
}
\newtcolorbox{mybox}{
colback=gray!15!white,
arc = 0pt, outer arc = 0pt,
boxsep=0pt, left = 3pt, right = 0pt, top = 0pt, bottom = 0pt,
leftrule=3pt, bottomrule=0pt,toprule=0pt, rightrule=0pt,
left = \boxmargin, right = \boxmargin, top = \boxmargin, bottom = \boxmargin
}
\newcommand{\appname}{{\textsc{Fait}}\xspace}
\title{\appname: \underline{F}ault-\underline{A}ware F\underline{i}ne-\underline{T}uning for Better Code Generation}
\author{Lishui Fan}
\email{flscode@zju.edu.cn}
\affiliation{
\institution{
The State Key Laboratory of Blockchain and Data Security, Zhejiang University}
\country{China}
}
\author{Zhongxin Liu}
\email{liu_zx@zju.edu.cn}
\authornote{Corresponding author}
\affiliation{
\institution{
The State Key Laboratory of Blockchain and Data Security, Zhejiang University}
\country{China}
}
\author{Haoye Wang}
\email{wanghaoye@hzcu.edu.cn}
\affiliation{
\institution{Hangzhou City University}
\country{China}
}
\author{Lingfeng Bao}
\email{lingfengbao@zju.edu.cn}
\affiliation{
\institution{
The State Key Laboratory of Blockchain and Data Security, Zhejiang University}
\country{China}
}
\author{Xin Xia}
\email{xin.xia@acm.org}
\affiliation{
\institution{Huawei}
\country{China}
}
\author{Shanping Li}
\email{shan@zju.edu.cn}
\affiliation{
\institution{
The State Key Laboratory of Blockchain and Data Security, Zhejiang University}
\country{China}
}
\date{February 2025}
\begin{document}
\begin{abstract}
Modern instruction-tuned large language models (LLMs) have made remarkable progress in code generation. However, these LLMs fine-tuned with standard supervised fine-tuning (SFT) sometimes generate plausible-looking but functionally incorrect code variants. This issue likely stems from the limitation of standard SFT, which treats all tokens equally during optimization and fails to emphasize the error-sensitive segments—specific code differences between correct implementations and similar incorrect variants. To address this problem, we propose \underline{F}ault-\underline{A}ware F\underline{i}ne-\underline{T}uning (\appname), a novel fine-tuning technique that enhances LLMs' code generation by (1) extracting multi-granularity (line/token-level) differences between correct and incorrect yet similar implementations to identify error-sensitive segments, and (2) dynamically prioritizing those segments during training via dynamic loss weighting. Through extensive experiments on seven LLMs across three widely-used benchmarks, our method achieves an average relative improvement of 6.9\% on pass@1 with just one epoch of training, with some enhanced 6.7B LLMs outperforming closed-source models, e.g., GPT-3.5-Turbo. Furthermore, our fine-tuning technique demonstrates strong generalization with performance improvements ranging from 3.8\% to 19.1\% across diverse instruction-tuned LLMs, and our ablation studies confirm the contributions of different granularities of differences and loss function components.
\end{abstract}
\maketitle
\input{latex/intro}
\input{latex/Method}
\input{latex/Exp_Setup}
\input{latex/Exp}
\input{latex/Dis}
\input{latex/Threat}
\input{latex/Related}
\input{latex/Conclusion}
\bibliographystyle{ACM-Reference-Format}
\bibliography{IEEEexample}
\end{document}
\section{Introduction}
Recently, fine-tuning LLMs using synthetic datasets generated by teacher models has emerged as a popular paradigm for improving code generation capabilities~\cite{luowizardcoder, wei2024magicoder, zheng2024opencodeinterpreter, gunasekar2023textbooks, yu2024wavecoder}. This paradigm uses stronger teacher models to generate high-quality instruction-response pairs and construct a dataset. These synthetic datasets are then used to fine-tune weaker student models with standard SFT method, which uses instructions to guide LLMs to generate outputs matching reference responses by minimizing cross-entropy loss uniformly across all tokens.
Although these LLMs fine-tuned with standard SFT achieve impressive performance on code generation benchmarks such as HumanEval~\cite{chen2021evaluating}, \textit{they sometimes generate plausible-looking but incorrect code variants}~\cite{huang2025survey,liu2024exploring}.
For example, Llama-3.1-70B-Instruct~\cite{dubey2024llama}, a model fine-tuned from Llama-3.1-70B using standard SFT, is capable of solving 82.3\% of the problems in HumanEval.
However, our analysis shows that 34.5\% of its failed cases are attributed to deviations in error-prone segments of otherwise correct implementations.
As shown in Figure~\ref{fig:motivating}, in a vowel identification task, although the LLM successfully implements the core logic of searching for vowels between consonants from the right, it erroneously initializes the starting index of the loop--a mistake in a critical part of the code that renders the program incorrect.
We refer to such crucial differences between correct implementations and similar incorrect variants as \textit{error-sensitive segments}, such as \textit{len(word) - 1} and \textit{len(word) - 2} in Figure~\ref{fig:motivating}. These \textit{error-sensitive segments} act as critical decision points in code generation, where even slight deviations can determine the correctness of the output.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figure/new_SFT.pdf}
\caption{Llama-3.1-70B-Instruct sometimes makes mistakes in error-sensitive segments in the outputs.}
\label{fig:motivating}
\end{figure}
To address the problem of errors in code generation, we propose \underline{F}ault-\underline{A}ware F\underline{i}ne-\underline{T}uning (\appname) as a finetuning technique specifically designed to guide LLMs to focus on error-sensitive segments and avoid mistakes in these critical regions, thereby improving the reliability and accuracy of code generation.
Implementing this approach involves two main challenges.
The first challenge is to identify these \textit{error-sensitive segments}, as existing instruction-tuning dataset construction methods primarily focus on generating instruction-response pairs without specifically considering these segments~\cite{zhang2023instruction}. To overcome this, we develop a two-phase segment identification component. First, we leverage a teacher model with a carefully designed prompt to generate functionally incorrect yet similar variants of the correct implementations from the existing instruction tuning dataset.
Then, we can identify the differences between these correct implementations and their similar yet incorrect variants, which serve as \textit{error-sensitive segments}.
We annotate the tokens in the segments through a multi-granularity method at both line and token levels. Notably, we annotate both the tokens in correct implementations and their corresponding parts in similar yet incorrect variants.
The second challenge lies in guiding LLMs to focus on these labeled segments during fine-tuning, as standard SFT treats all tokens with equal importance regardless of their criticality.
To address this challenge, we adjust the loss of SFT to prioritize the annotated error-sensitive tokens within correct implementations.
Specifically, \appname processes both correct and incorrect implementations to discriminate the \textit{error-sensitive segments}, and only computes the loss based on correct code implementations. Unlike standard SFT that uniformly weights all tokens during loss computation, we dynamically assign relatively higher weights to those tokens in the correct implementation that correspond to the \textit{error-sensitive segments}. This methodology enhances LLMs' capability to discriminate \textit{error-sensitive segments} when solving programming tasks, thereby increasing the likelihood of generating correct implementation details while suppressing error-prone alternatives.
To implement our method, we construct a refined dataset derived from the original instruction-tuning data. Each data point consists of an instruction, its correct implementation from the original dataset, and an LLM-generated similar incorrect variant. We then develop a multi-granularity error-sensitive segment extraction method and combine it with the refined loss function to enhance LLM's code generation capabilities.
We validate the effectiveness of the \appname through extensive experiments. Notably, through \appname, the selected LLMs undergo only one epoch of training on their original instruction datasets, yet achieve an average relative improvement of 6.9\% on pass@1 (a strict evaluation metric measuring the ratio of first-generated samples that pass all test cases) across three representative code generation benchmarks (Humaneval(+), MBPP(+), and BigCodeBench)~\citep{chen2021evaluating,austin2021program,liu2024your,zhuo2024bigcodebench}. Among these LLMs, SemCoder-S~\cite{dingsemcoder} with 6.7B parameters outperforms closed-source models like GPT-3.5-Turbo~\cite{OpenAI2022} on Humaneval(+) and MBPP(+) benchmarks and MagiCoder$\mathcal{S}$-DS with 6.7B parameters outperforms GPT-3.5-Turbo on HumanEval(+). Our method also demonstrates strong generalization capabilities, showing performance improvements ranging from 3.8\% to 19.1\% across multiple instruction-tuned LLMs, including those trained on closed-source instruction datasets. Moreover, our ablation experiments on \appname confirm the contributions of different granularities of differences in generated code details and loss functions.
\noindent We summarize our contributions as follows.
\begin{itemize}
\item To the best of our knowledge, we are the first to investigate how to enhance LLMs' understanding of \textit{error-sensitive segments} by refining the SFT process to improve LLMs' code generation capabilities.
\item We propose a novel framework, Fault-Aware Fine-Tuning (\appname), to effectively guide LLMs to focus on error-prone parts in code. This is achieved by (1) extracting multi-granularity code differences (token-/line-level) to identify \textit{error-sensitive segments}, and (2) refining SFT to dynamically assign higher weights to these parts during the training process.
\item Through extensive experiments across seven LLMs and three widely-used code generation benchmarks, we demonstrate the effectiveness and generalizability of our approach in effectively boosting LLMs' code generation performance compared to baseline methods.
\end{itemize}
\section{Approach}
\label{sec:3}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{Figure/overall.pdf}
\caption{The overview of \appname, taking one sample for explanation.}
\label{fig:overview}
\end{figure*}
Figure~\ref{fig:overview} illustrates the overview of \appname. This approach takes as input an instruction-tuned LLM and its corresponding instruction-tuning dataset, and outputs an enhanced LLM with improved code generation capabilities that can better discriminate \textit{error-sensitive segments}. It first augments the dataset by generating similar yet incorrect implementations for each correct response. Then, it identifies \textit{error-sensitive segments} between the paired implementations and calculates weights for tokens in the correct implementations that differ from the incorrect variants. During fine-tuning, it only computes loss on the correct implementations, with higher weights assigned to tokens in \textit{error-sensitive segments}, producing an LLM that can better discriminate these \textit{error-sensitive segments}. The methodology consists of two key components:
\begin{enumerate}
\item \textit{Error-Sensitive Segments Identification}: This component creates a refined dataset of paired correct and similar yet wrong code samples from the original dataset, and processes code differences at multiple granularities to identify \textit{error-sensitive segments}.
\item \textit{Dynamic Importance Reweighting}: This component strategically reweights token weights in the loss function to prioritize discriminative elements in correct implementations, building upon the identified error-sensitive segments. This dynamic weighting method enhances the LLM's attention to the key implementation details in correct code, effectively teaching it to distinguish between valid solutions and their similar yet incorrect counterparts, ultimately improving code generation capabilities.
\end{enumerate}
The two components work together to fine-tune LLMs to distinguish between correct implementations and similar yet incorrect alternatives, thereby improving code generation performance. The following subsections detail each component of our methodology.
\subsection{Error-Sensitive Segments Identification}
The input to this component is the instruction-tuning dataset $\mathcal{D} = {(c^{\text{correct}}_i, p^{\text{target}}i)}_{i=1}^{N}$, where $p^{\text{target}}_i$ represents the target problem description and $c^{\text{correct}}_i$ denotes the correct implementation. The output is an enhanced dataset $\mathcal{D} = {(c^{\text{correct}}_i, c^{\text{incorrect}}_i, p^{\text{target}}_i)}_{i=1}^{N}$ with error-sensitive segment information, where $c^{\text{incorrect}}_i$ represents the corresponding similar but incorrect implementation. To generate incorrect code variants, we utilize a stronger teacher LLM with a carefully designed prompt. The prompt template is shown in Figure~\ref{fig:prompt}, which consists of two parts. The first part defines the task for producing incorrect responses corresponding to the target problem and correct answer, explicitly specifying that outputs must be similar to correct solutions, with responses constrained to markdown formatting for consistent post-processing. The second part provides contextual references by presenting the target problem description and the correct solution.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Figure/prompt.pdf}
\caption{The prompt for generating similar yet incorrect response.}
\label{fig:prompt}
\end{figure}
After obtaining pairs of correct and incorrect code, we extract the differences to identify \textit{error-sensitive segments} and process them at different granularity levels to capture both line-level and token-level information. Specifically, we designate $c^{\text{incorrect}}$ as the \emph{pre-change} version and $c^{\text{correct}}$ as the \emph{post-change} version.
\textit{Line-Level Differences}. We align $c^{\text{correct}}_i$ and $c^{\text{incorrect}}_i$ line-by-line using Python's \texttt{difflib} library. For each line, the tool assigns a flag indicating whether it should be deleted (\texttt{-}), added (\texttt{+}), or remain unchanged. We extract the lines marked for deletion from $c^{\text{incorrect}}$ and those marked for addition from $c^{\text{correct}}$.
Let $L_c$ and $L_a$ denote the number of code lines in the correct code $c^{\text{correct}}$ and incorrect code $c^{\text{incorrect}}$, respectively. Based on these extracted lines, we construct the line-level boolean mask vectors $V_{\text{line}}^c$ and $V_{\text{line}}^a$ for $c^{correct}$ and $c^{incorrect}$ as follows:
\begin{small}
\begin{align*}
V_{\text{line}}^c &= [v^c_1, v^c_2, \dots, v^c_{L_c}], \text{where } v^c_i = I(\text{line}_i^c \text{ is added}) \\
V_{\text{line}}^a &= [v^a_1, v^a_2, \dots, v^a_{L_a}], \text{where } v^a_j = I(\text{line}_j^a \text{ is deleted})
\end{align*}
\end{small}
where $I(\cdot)$ is the indicator function that outputs 1 if the condition is true and 0 otherwise.
\textit{Token-Level Differences}. We utilize the Levenshtein distance algorithm~\cite{yujian2007normalized} to identify character-level change information between $c^{\text{incorrect}}$ and $c^{\text{correct}}$. The Levenshtein distance algorithm, also known as the edit distance algorithm, quantifies the minimum number of single-character operations (insertions, deletions, or substitutions) required to transform one string into another. We identify the characters that need to be edited to transform the original string ($c^{\text{incorrect}}_i$) into the modified version ($c^{\text{correct}}$), and record their positions accordingly. For instance, if a character operation is an insertion, we record its position in $c^{\text{correct}}$, as shown in Figure~\ref{fig:overview}. Given that the LLM's embedding layer is tightly coupled with the LLM's tokenizer vocabulary, we map these character-level differences to tokens using the LLM's tokenizer. When character modifications span multiple tokens, all affected tokens are marked.
Let $T_c$ and $T_a$ denote the number of tokens in $c^{\text{correct}}$ and $c^{\text{incorrect}}$. We construct token-level boolean mask vectors $V_{\text{token}}^c$ and $V_{\text{token}}^a$ for $c^{correct}$ and $c^{incorrect}$ as follows:
\begin{small}
\begin{align*}
V_{\text{token}}^c &= [w^c_1, \dots, w^c_{T_c}], \; w^c_k = I(\text{token}_k^c \text{ is added}) \\
V_{\text{token}}^a &= [w^a_1, \dots, w^a_{T_a}], \; w^a_\ell = I(\text{token}_\ell^a \text{ is deleted})
\end{align*}
\end{small}
\begin{algorithm}[t]
\begin{algorithmic}[1]
\Require Code sequence $c=[c_1,c_2,...,c_L]$, where each $c_i$ is a line of code. Line-level mask vector $V_{\text{line}}=[v_1,v_2,...,v_L]$, where $v_i \in \{0,1\}$.
\State Initialize token-level mask vector $V_{\text{line-to-token}} \gets [0,0,...,0]$ of length $T$
\State $t\gets 1$ \Comment{Initialize index}
\For{each line $l_i=1$ to $L$}
\State $N_i \gets \text{number of tokens of } c_{li}$
\If{$v_{l_i} = 1$} \Comment{Line is marked}
\For{each token $j=1$ to $N_i$}
\State Set $V_{\text{line-to-token}}[t] \gets 1$
\State $t \gets t + 1$
\EndFor
\Else \Comment{Line is unmarked}
\For{each token $j=1$ to $N_i$}
\State Set $V_{\text{line-to-token}}[t] \gets 0$
\State $t \gets t +1$
\EndFor
\EndIf
\EndFor
\end{algorithmic}
\caption{Converting Line-Level Mask Vector to Token-Level Mask Vector}
\label{alg:line_to_token}
\end{algorithm}
\textit{Hybrid Level Vectors}.
To create comprehensive representations of \textit{error-sensitive segments}, we combine line-level and token-level masks. However, these two types of masks operate at different granularities and cannot be directly combined. We must first align these representations to the same granularity to enable their integration.
We convert line-level masks to token-level granularity, as illustrated in Algorithm~\ref{alg:line_to_token}. Lines 1-2 initialize a token-level mask vector (size T) and the position index. Lines 3-16 implement the core logic - for each code line (1 to L), when the line is masked $v_i=1$ (marked lines, Lines 5-8), all tokens in that line receive mask value 1; otherwise (unmarked lines, Lines 10-15), tokens receive value 0. For $V_{\text{line}}^c$ and $V_{\text{line}}^a$, we identify the corresponding tokens within that line to get $V_{\text{line-to-token}}^c$ and $V_{\text{line-to-token}}^a$.
With both masks now represented at the token level, we then use an element-wise addition operation to combine $V_{\text{line-to-token}}$ and $V_{\text{token}}$, as follows:
\begin{align*}
V_{\text{hybrid}}^c &= V_{\text{line-to-token}}^c + V_{\text{token}}^c \\
V_{\text{hybrid}}^a &= V_{\text{line-to-token}}^a + V_{\text{token}}^a
\end{align*}
These hybrid vectors precisely identify \textit{error-sensitive segments} at multiple granularities, highlighting critical differences between correct and incorrect implementations. Noted that changed tokens must appear in changed lines, our hybrid representation naturally creates a priority system: 1) tokens that are both in changed lines and are themselves changed will have a value of 2 in the hybrid vector; 2) tokens that are only in changed lines but not directly changed will have a value of 1. This provides a more comprehensive view than either granularity alone, with higher values indicating more critical tokens.
\subsection{Dynamic Importance Reweighting}
With the identified \textit{error-sensitive segments}, we now refine the SFT process to prioritize these critical differences. Based on the constructed dataset $\mathcal{D} = \{(c^{\text{correct}}_i,c^{\text{incorrect}}_i,p^{target}_i)\}_{i=1}^{N}$, the standard SFT loss is computed as:
\begin{equation}
\label{equ:sft}
\begin{aligned}
\mathcal{L}_{SFT} = -\frac{1}{n}\sum_{i=1}^{N}{\sum_{j=1}^{T_c}logP(c^{\text{correct}}_{i,j}|p^{target}_i,c^{\text{correct}}_{i,1:j-1})}
\end{aligned}
\end{equation}
where $N$ denotes the number of samples in a batch. Notably, the standard SFT loss function treats all tokens equally.
In contrast, the \appname introduces dynamic token-level weights $W={w_1,w_2,...,w_j}$ emphasize \textit{error-sensitive segments}:
\begin{equation}
\label{equ:fault}
\begin{aligned}
\mathcal{L}_{Fault} = -\frac{1}{n}\sum_{i=1}^{N}{\sum_{j=1}^{T_c} w_j \cdot logP(c^{\text{correct}}_{i,j}|p^{target}_i,c^{\text{correct}}_{i,1:j-1})}
\end{aligned}
\end{equation}
The weight $W$ is computed as follows: Given input $x = p^{\text{target}}$, outputs $y^c = c^{correct}$ and $y^a=c^{incorrect}$, we first obtain the LLM's prediction probabilities for both correct and incorrect implementations given the same instruction:
\begin{align}
P^c &= f_{\theta}( y^c_{k} \mid y^c_{1:k-1}, x ) \\
P^a &= f_{\theta}( y^a_{l} \mid y^a_{1:l-1}, x )
\end{align}
where $f_{\theta}$ represents the conditional probability function of the LLM that computes the probability of the next token given the input $x$ and previous tokens.
We then apply the hybrid-level vectors to isolate probabilities for \textit{error-sensitive segments}:
\begin{align}
H^c &= P^c \odot V_{\text{hybrid}}^c \\
H^a &= P^a \odot V_{\text{hybrid}}^a
\end{align}
Where $\odot$ denotes element-wise multiplication. Inspired by the Bradley–Terry model~\cite{hunter2004mm}, a pairwise comparison framework widely used in ranking systems~\cite{baker2021modifying,yan2016ranking,menke2008bradley}, we compute dynamic token weights $W$ for differentiating tokens in \textit{error-sensitive segments}:
\begin{small}
\begin{align*}
W = \alpha \ - \vert \frac{\overline{H^c}-\overline{H^a}}{\overline{H^c}+\overline{H^a}} \vert
\end{align*}
\end{small}
Where $\alpha$ is a hyperparameter controlling the weight range, $\overline{H^c}$ denotes the mean probability of tokens in \textit{error-sensitive segments} in $c_{\text{correct},i}$, and $\overline{H^a}$ represents the corresponding value for $c_{\text{incorrect}}$. This formulation ensures that: (1) When the mean probabilities $\overline{H^c}$ and $\overline{H^a}$ are close (indicating the LLM struggles to distinguish the tokens between $c_{\text{correct}}$ and $c_{\text{incorrect}}$), the weights for differentiating tokens in $c_{\text{correct}}$ approach $\alpha$, thereby maximizing emphasis on \textit{error-sensitive segments}. (2) Conversely, when $\overline{H^c}$ and $\overline{H^a}$ diverge significantly (demonstrating the LLM can discriminate the differentiating tokens in $c_{\text{correct}}$ and $c_{\text{incorrect}}$), the weights diminish toward $\alpha-1$, reducing emphasis. For tokens shared between $c_{\text{correct}}$ and $c_{\text{incorrect}}$, we assign fixed weights $\alpha-1$, ensuring the LLM maintains baseline attention to shared elements while prioritizing discriminative features.
This dynamic weighting mechanism guides the LLM to focus on the challenging discriminative aspects of correct implementations, which can improve its code generation capability.
\section{Experiments Setup}
\label{sec:4}
\subsection{Benchmarks}
We conduct experiments on three widely used code generation benchmarks to demonstrate the superiority and generality of proposed \appname.\\
\textit{Humaneval~\cite{chen2021evaluating}:} This benchmark consists of 164 manually crafted programming tasks, created by OpenAI. Each task includes a method signature, a docstring, a method body, and several unit tests. Our work employs both the initial HumanEval and its extended version, HumanEval+\citep{liu2024your}, which expands the test cases of the original with 80$\times$ additional test samples to overcome limitations in test coverage~\citep{liu2024your}.\\
\textit{MBPP~\cite{austin2021program}:} This benchmark contains 974 Python coding tasks spanning core programming concepts, library utilization capabilities, and more. Our study adopts the extended versions proposed by~\citep{liu2024your}, including MBPP and MBPP+. These collections each contain 378 tasks, with the enhanced version incorporating 35 times the number of test samples.\\
\textit{Bigcodebench~\cite{zhuo2024bigcodebench}:} This benchmark presents a rigorous benchmark for code generation, constructed to measure LLMs' capabilities in utilizing programming tools and the following of complex instructions. It contains 1,140 code-generation problems. In the Complete configuration, each problem provides a function signature, a problem description, and a test suite. A small high-quality subset known as BigCodeBench-Hard contains 148 problems that are more user-centric and challenging. Our study uses both the full set and the hard set, namely, BigCodeBench-Full and BigCodeBench-Hard.
\subsection{Metrics}
To evaluate code generation performance, we use the Pass@K metric, which is widely used in prior studies~\cite{chen2021evaluating,mu2023clarifygpt,huang2023competition}. This metric checks whether the generated code passes all test cases successfully within the first K generations. Following prior studies~\cite{fakhoury2024llm,dong2024self,jiang2024self}, our experimental design adopts K=1, focusing exclusively on first-attempt success rates. This metric also aligns with real-world scenarios where developers aim to produce accurate code on the first attempt~\cite{dong2024self}. It should be noted that Pass@1 represents a particularly strict evaluation metric for code generation and improving it is challenging. Higher Pass@1 scores indicate better code generation performance.
\subsection{Implementation Detail}
\subsubsection{Data generation} We use Qwen2.5-Coder-32B-Instruct\footnote{https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct} as the teacher model with temperature=0.8 to generate incorrect code implementations due to its strong coding abilities and good natural language understanding capabilities.
To mitigate the potential threat introduced by errors from the teacher model generation, we manually examined a sample of 50 generated outputs.
Our analysis shows that the LLM could produce \textit{error-sensitive segments} as expected: 96\% of the generated incorrect samples are similar to the correct ones, while the remaining 4\% completely deviate from correct implementations.
Due to space limitations, the checked samples are provided in the replication package.
This manual inspection helps validate the reliability of our training data and supports the soundness of our experimental findings.
\subsubsection{Settings}
All experiments are conducted on a machine with eight Tesla A800 GPUs, each with 80 GB of memory per GPU. $\alpha$ is set to 2, which means the weight range of $W$ is $(1,2)$. All models are trained for 1 epoch. Considering that \appname is designed to enable instruction-tuned LLMs to emphasize \textit{error-sensitive segments} from their original instruction-tuning datasets, we apply relatively low learning rates during training.
Specifically, we use a learning rate of 5e-6 with a linear scheduler and warm-up across all LLMs.
The max sequence length is 1024. For inference evaluation, we use greedy decoding to ensure deterministic outputs, which also aligns with prior studies~\cite{chen2021evaluating,mu2023clarifygpt}.
\section{Results}
\label{sec:5}
In this section, we report and analyze the experimental results to answer the following research questions (RQs):
\begin{itemize}[leftmargin=*]
\item RQ1: How effective is our approach in improving code generation across different benchmarks?
\item RQ2: How do different components of the \appname method contribute to LLMs' performance?
\item RQ3: Does \appname demonstrate generalizability across different LLMs and their corresponding instruction-tuning datasets?
\item RQ4: Does \appname work for instruction-tuned LLMs whose instruction-tuning dataset is closed-source?
\end{itemize}
\subsection{RQ1: Overall Effectiveness}
In this RQ, we evaluate the effectiveness of our approach by applying it to several Instruction-tuned LLMs using their corresponding instruction-tuning datasets and assess their performance against three baselines:\\
\textit{Base Models:} We use the original instruction-tuned LLMs without any additional \appname as our base models. This comparison demonstrates the absolute improvement achieved through \appname. Specifically, we select three representative instruction-tuned LLMs: MagiCoder$\mathcal{S}$-CL~\cite{wei2024magicoder}, MagiCoder$\mathcal{S}$-DS~\cite{wei2024magicoder} and SemCoder-S~\cite{dingsemcoder} as our base models.\\
\textit{Closed-Source Models:} We include GPT-3.5-Turbo~\cite{OpenAI2022} and GPT-4-Turbo~\cite{OpenAI2023} as the closed-source baseline. This comparison illustrates the performance gap between our fault-aware fine-tuned LLMs and advanced closed-source LLMs.\\
\textit{Standard-SFT Models:} We apply standard SFT on the same base models to create this baseline. This comparison serves two purposes: (1) to examine whether further fine-tuning on coarse-grained instruction-response mappings on their existing dataset can improve performance over the original models, and (2) to highlight the superior performance of our approach in learning fine-grained error-sensitive segments.
We choose the instruction-tuning dataset evol-codealpaca-v1~\cite{luo2023wizardcoder} as our training dataset. This dataset is evolved from a seed dataset with GPT-4-Turbo, containing 110K high-quality data points. For MagiCoder$\mathcal{S}$-CL and MagiCoder$\mathcal{S}$-DS, this dataset is their original instruction-tuning dataset. For SemCoder-S, this dataset is a subset of its original instruction-tuned dataset, which is not fully open-sourced.
\input{Table/main-all}
Table~\ref{tab:main-all} presents the performance of LLMs with \appname and the baselines across HumanEval(+), MBPP(+), and BigCodeBench. Overall, LLMs with \appname demonstrate substantial improvements in code generation. We observe that LLMs with \appname show average relative performance improvements of 4.8\% over the base model and 4.9\% over LLMs with Standard-SFT on HumanEval(+), MBPP(+), and BigCodeBench. Notably, with \appname, SemCoder-S with only 7B parameters outperforms the closed-source GPT-3.5-Turbo on HumanEval(+) and MBPP(+), and MagiCoder$\mathcal{S}$-DS outperforms GPT-3.5-Turbo on HumanEval(+). Both LLMs achieve comparable performance to GPT-3.5-Turbo on BigCodeBench, further validating the exceptional effectiveness of \appname in enhancing code generation capabilities. While the improvement on BigCodeBench-Full is modest, our approach shows more gains on BigCodeBench-Hard (e.g., 20.8\% relative improvement for SemCoder-S). This is likely because more challenging problems contain more \textit{error-sensitive segments}, and \appname is designed to guide LLMs to handle these \textit{error-sensitive segments}, thus showing greater effectiveness on difficult programming tasks.
When comparing the effects of \appname versus Standard-SFT on base models, we observe that Standard-SFT provides limited improvements and sometimes even weakens the base models. For example, SemCoder-S with Standard-SFT achieves only 0.8\% relative performance improvement on HumanEval(+) and suffers 1.9\% relative performance decline on MBPP+. This suggests that simply reinforcing the coarse-grained instruction-response mappings on their existing dataset provides minimal benefits, as these models have already captured these general mappings well during their initial instruction tuning. In contrast, \appname goes beyond simply continuing training on the original dataset, enabling LLMs to learn and memorize the mappings between problems and fine-grained \textit{error-sensitive segments} in code. This targeted approach helps models distinguish between correct implementations and similar-looking but incorrect solutions, thereby generating accurate solutions when encountering target problems.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figure/case.pdf}
\caption{A case demonstrating how LLMs after \appname can better focus on \textit{error-sensitive segments} to generate the correct solution.}
\label{fig:dis-case}
\end{figure}
To further figure out the reasons for \appname improving LLMs' ability to generate functionally correct code, we manually inspect the test results. Based on our analysis, \appname demonstrates two main advantages over both the original model and Standard-SFT:
First, \appname can learn diverse \textit{error-sensitive segments} to better recognize and focus on implementation details that are prone to errors, while the original model and Standard-SFT only learn the overall mapping from problem to solution. This method improves the LLM's attention to key implementation choices. For example, Figure~\ref{fig:dis-case} presents a comparison of the results of three versions of SemCoder-S on the HumanEval/135 task. In this example, an error-sensitive segment involves deciding whether to return the index of the target element itself \textit{i} or the index of its previous element \textit{i-1}. This distinction directly impacts the functional correctness of the implementation. Both the original model and the model with Standard-SFT incorrectly return the index of the previous element, while the model with \appname correctly returns the index of the target element.
We manually examine the tasks from HumanEval(+) that are correctly solved after applying \appname but initially incorrect with the base models. We find that among these tasks, 63.6\% of SemCoder-S's improvements result from properly handling error-sensitive segments, with similar rates observed in MagiCoder$\mathcal{S}$-DS (71.4\%) and MagiCoder$\mathcal{S}$-CL (72.7\%). These results further confirm our method's effectiveness in guiding LLMs to recognize error-sensitive segments.
Second, by developing a deeper understanding of critical code segments, \appname also enhances overall code generation capabilities. By strategically emphasizing error-sensitive segments while maintaining appropriate weight for contextual elements, the LLM learns to identify and handle the crucial parts of implementations that determine correctness.
Figure~\ref{fig:dis-case-2} demonstrates this using an example from SemCoder-S on HumanEval/110, which requires determining whether swapping elements between two lists can make all elements in \textit{lst1} even.
This case highlights improvements that go beyond addressing specific error-sensitive segments.
Both the original model and Standard-SFT fail to implement the correct verification logic to determine whether \textit{lst2} contains enough even numbers to replace odd numbers in \textit{lst1}.
In contrast, the Fault-Aware Fine-Tuned model correctly implements this logic, demonstrating enhanced general coding abilities rather than just handling error-sensitive parts.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figure/case1.pdf}
\caption{A case demonstrating how \appname can import overall code generation performance.}
\label{fig:dis-case-2}
\end{figure}
\begin{center}
\begin{mybox} \textbf{RQ1 Summary: } %
\appname delivers consistent and substantial performance improvements across all three benchmarks, with enhanced LLMs even outperforming GPT-3.5-Turbo on certain benchmarks. The results confirm that explicitly learning fine-grained error-sensitive segment mappings is more effective than simply retraining on coarse-grained instruction-response pairs.
\end{mybox}
\end{center}
\subsection{RQ2: Component Analysis}
\input{Table/abl-evalplus}
To understand how different components contribute to the effectiveness of \appname, we conduct ablation studies focusing on the impacts of multi-granularity and the loss function in this RQ.
\textit{Impact of Difference Granularity.} We explore the impact of code difference granularity, which involves synthesizing line-level and token-level code differences to identify \textit{error-sensitive segments}. Specifically, we conduct ablation experiments using MagiCoder$\mathcal{S}$-DS and SemCoder-S as base models and perform evaluation on three selected benchmarks. Table~\ref{tab:abl-evalplus} shows the impact of different granularities of differences on \appname. We can observe that the combination of line-level granularity and token-level granularity yields maximum performance gains. For example, when applied to SemCoder-S, this approach achieves a relative average improvement of 4.9\% across all benchmarks compared to the base model, compared to just 2.9\% for line-level only and 2.4\% for token-level only. This demonstrates that multi-granularity differences enable better \textit{error-sensitive segments}, thereby enhancing LLMs' code generation capabilities.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figure/model_comparison.png}
\caption{Performance of different weights based on SemCoder-S on Humaneval(+)}
\label{fig:abl-weight}
\end{figure}
\textit{Impact of the Loss Function.} To validate the effectiveness of our dynamic loss weighting design, we compare our dynamic weighting approach against a fixed weighting strategy where all error-sensitive tokens receive the same constant weight during training. We experiment with fixed weights in the range of $[0, 1, 2, 3, 5]$ on HumanEval(+). Due to the evaluation time constraints, we select SemCoder-S as the representative LLM for this ablation study, as it demonstrates the best overall performance with \appname. Notably, when the fixed weight equals 1, this configuration is equivalent to standard SFT. Figure~\ref{fig:abl-weight} shows the performance of SemCoder-S with different fixed weights compared to our dynamic weighting approach. Our experimental results reveal two important findings: (1) A fixed weight of 2 yields better performance than other fixed weight values, suggesting that an appropriate constant weight can help the LLM recognize \textit{error-sensitive segments} and enhance code generation capabilities. (2) Our dynamic weighting approach still outperforms the best fixed weighting configuration. This confirms that dynamically adjusting weights based on the LLM's current discrimination ability provides better guidance for the LLM to focus on critical implementation details that differentiate correct solutions from their erroneous variants.
\begin{center}
\begin{mybox} \textbf{RQ2 Summary: } %
All components in \appname contribute to the performance. Combining different levels of granularity of code differences (line + token level) is critical to performance. The loss function with dynamic weighting strategies outperforms that with fixed weighting strategies, highlighting the effectiveness of our weighting method.
\end{mybox}
\end{center}
\subsection{RQ3: The Generalization Capabilities}
\input{Table/gen-eval}
In this RQ, we aim to explore the generalizability of \appname across different instruction-tuned LLMs when using their own instruction-tuning datasets. Specifically, we select two representative LLMs and their corresponding instruction datasets. 1) We select MagiCoder-DS and its corresponding dataset OSS-Instruct. This dataset is generated from open-sourced code by GPT-3.5-Turbo, and contains 75K samples. 2) SemCoder and its corresponding dataset PYX. This dataset consists of 95K samples, including comprehensive reasoning texts with executable code samples. The dataset is constructed with problem descriptions generated by GPT-3.5-Turbo and corresponding responses generated by GPT-4o-mini~\cite{OpenAI2024GPT4o}, creating high-quality instruction-response pairs with detailed reasoning. For each LLM, we process its corresponding dataset through our pipeline and evaluate performance on the same benchmarks used in RQ1. We select Base Models and Standard-SFT Models as our baselines. By using LLMs with different training paradigms and datasets with varying characteristics (open-sourced code versus detailed reasoning with executable samples), we can verify that our approach is not tied to specific LLM series or dataset properties, but rather provides universal benefits.
Table~\ref{tab:gen-eval} shows the performance of LLMs on Humaneval(+), MBPP(+), and BigCodeBench after \appname and Standard-SFT. We can observe that \appname demonstrates robust generalization capabilities to different instruction-tuning LLMs and their corresponding datasets. For MagiCoder-DS and SemCoder, after \appname, the average performances across all benchmarks show relative improvements of 5.3\% and 3.8\% compared to the base models. In contrast, standard SFT yielded modest relative improvements of 1.4\% for MagiCoder-DS and decreased performance by 2.1\% for SemCoder. These results show that \appname's benefits are not tied to specific dataset characteristics or model series. Instead, the approach effectively enhances diverse instruction-tuned LLMs by teaching them to focus on \textit{error-sensitive segments} in correct solutions.
\begin{center}
\begin{mybox} \textbf{RQ3 Summary:}
\appname exhibits strong generalizability across different instruction-tuned LLMs and their corresponding datasets, consistently outperforming standard SFT.
\end{mybox}
\end{center}
\subsection{RQ4: Effectiveness on LLMs with Closed-Source Instruction Data}
\input{Table/rq4-eval}
A key question for the broader adoption is whether \appname can enhance LLMs whose original instruction-tuning datasets are not publicly available. To investigate this, we applied our method to two widely-used LLMs with closed-source training data in this RQ. Specifically, we choose CodeLlama-7B-Instruct~\cite{roziere2023code} and DeepSeek-\\Coder-6.7B-Instruct~\cite{guo2024deepseek} as base models. These LLMs are instruction-tuned on substantial but proprietary datasets - CodeLlama-7B-Instruct underwent instruction tuning on approximately 5B tokens of instruction data, while DeepSeekCoder-6.7B-Instruct is tuned on around 2B tokens. To test our approach without access to these original datasets, we select evol-codealpaca-v1 used in the main experiment as the training dataset. We select Base Models and Base Models with Standard SFT as our baselines and evaluate on Humaneval(+), MBPP(+) and BigCodeBench(+).
Table~\ref{tab:rq4-eval} shows the performance of these two LLMs using Standard-SFT and fault-fine-tuning on Humaneval(+), MBPP(+), and BigCodeBench. We can find that \appname is also applicable to LLMs with closed-source datasets. For CodeLlama-Instruct and DeepseekCoder-Instruct, after \appname, the average relative improvements across all benchmarks are 19.1\% and 5.9\%. By comparison, the standard SFT yield gains of 7.5\% and 0.5\%, respectively. This further demonstrates that fault-fine-tuning is also applicable to LLMs with closed-source datasets, showing strong applicability.
\begin{center}
\begin{mybox} \textbf{RQ4 Summary:}
\appname demonstrates strong applicability to instruction-tuned LLMs with closed-source datasets, delivering particularly dramatic improvements for initially weaker models. This expands the method's application scope to broader scenarios where original training datasets are inaccessible, offering a path to enhance LLMs without requiring access to their proprietary training data.
\end{mybox}
\end{center}
\begin{table}[t]
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
Model & Humaneval(+) & MBPP(+) & \begin{tabular}[c]{@{}c@{}}BCB\\ FUll\end{tabular} & \begin{tabular}[c]{@{}c@{}}BCB\\ HARD\end{tabular} \\ \midrule
\multicolumn{5}{l}{\textit{Closed-Source Models}} \\
GPT-3.5-Turbo (Nov 2023) & 76.8 (70.7) & 82.5 (69.7) & 50.6 & 21.6 \\
GPT-4-Turbo (April 2024) & \textbf{90.2 (86.6)} & \textbf{85.7 (73.3)} & \textbf{58.2} & \textbf{35.1} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: CodeLlama-Python-7B}} \\
MagiCoder$\mathcal{S}$-CL & 70.7 (66.5) & 68.4 (56.6) & 39.7 & 12.8 \\
+Standard-SFT & 69.5 (64.0) & 69.3 (58.7) & 39.3 & 13.5 \\
+\appname & \textbf{73.2 (68.9)} & \textbf{71.7 (59.5)} & \textbf{42.2} & \textbf{15.5} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: DeepseekCoder-6.7B-Base}} \\
MagiCoder$\mathcal{S}$-DS & 76.8 (71.3) & 75.7 (64.4) & 47.6 & 12.8 \\
+Standard-SFT & 75.6 (70.7) & 79.1 (66.4) & 46.9 & 10.8 \\
+\appname & \textbf{77.4 (74.3)} & \textbf{79.6 (69.0)} & \textbf{48.2} & \textbf{15.5} \\ \hdashline
SemCoder-S & 79.3 (74.4) & 79.6 (68.5) & 48.5 & 16.9 \\
+Standard-SFT & 79.9 (75.0) & 80.7 (67.2) & 47.1 & 16.2 \\
+\appname & \textbf{83.5 (78.7)} & \textbf{83.1 (70.6)} & \textbf{48.9} & \textbf{20.3} \\ \bottomrule
\end{tabular}%
}
\caption{Performance of different LLMs using \appname method compared with Standard-SFT on Humaneval(+), MBPP(+) and BigCodeBench, where BCB stands for BigCodeBench.}
\label{tab:main-all}
\end{table}
\begin{table}[t]
\caption{Performance ablation of different granularities of differences on Humaneval(+), MBPP(+) and BigCodeBench based on MagiCoder$\mathcal{S}$-DS and SemCoder-S, where BCB stands for BigCodeBench.}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
Model & Humaneval(+) & MBPP(+) & \begin{tabular}[c]{@{}c@{}}BCB\\ FULL\end{tabular} & \begin{tabular}[c]{@{}c@{}}BCB\\ HARD\end{tabular} \\ \midrule
MagiCoderS-DS & 76.8 (71.3) & 75.7 (64.4) & 47.6 & 12.8 \\ \midrule
+\appname (Line Level) & 75.6 (72.0) & 78.8 (68.3) & 47.1 & 14.2 \\
+\appname (Token Level) & 76.2 (72.6) & 79.1 (68.5) & 47.3 & 14.2 \\
+\appname & \textbf{77.4 (74.3)} & \textbf{79.6 (69.0)} & \textbf{48.2} & \textbf{15.5} \\ \midrule
SemCoder-S & 79.3 (74.4) & 79.6 (68.5) & 48.5 & 16.9 \\ \midrule
+\appname (Line Level) & 80.5 (76.8) & \textbf{83.1} (70.1) & \textbf{49.1} & 18.2 \\
+\appname (Token Level) & 81.1 (76.8) & 82.8 (70.1) & 48.3 & 16.9 \\
+\appname & \textbf{83.5 (78.7)} & \textbf{83.1 (70.6)} & 48.9 & \textbf{20.3} \\ \bottomrule
\end{tabular}%
}
\label{tab:abl-evalplus}
\end{table}
\begin{table}[t]
\caption{Performance of \appname on other instruction-tuned LLMs with their corresponding datasets on Humaneval(+), MBPP(+) and BigCodeBench, where BCB stands for BigCodeBench.}
\resizebox{\linewidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
Model & Humaneval(+) & MBPP(+) & \begin{tabular}[c]{@{}c@{}}BCB\\ FULL\end{tabular} & \begin{tabular}[c]{@{}c@{}}BCB\\ HARD\end{tabular} \\ \midrule
\multicolumn{5}{l}{\textit{(Corresponding Dataset OSS-INSTRUCT)}} \\
MagiCoder-DS & 66.5 (60.4) & 75.4 (61.9) & 43.4 & 12.2 \\
+Standard-SFT & 64.6 (58.5) & 79.1 (66.1) & 43.9 & 12.2 \\
+\appname & \textbf{67.1 (62.2)} & \textbf{79.4 (66.4)} & \textbf{46.2} & \textbf{15.5} \\ \midrule
\multicolumn{5}{l}{\textit{(Corresponding Dataset PYX)}} \\
SemCoder & 73.2 (68.9) & 79.9 (65.3) & 43.5 & 16.9 \\
+Standard-SFT & 71.3 (65.2) & 79.9 (66.4) & 43.4 & 14.2 \\
+\appname & \textbf{73.7 (69.5)} & \textbf{81.0 (67.2)} & \textbf{47.9} & \textbf{21.6} \\ \bottomrule
\end{tabular}%
}
\label{tab:gen-eval}
\end{table}
\begin{table}[t]
\caption{Performance of LLMs trained on closed-source instruction-tuning datasets after using \appname on Humaneval(+), MBPP(+) and BigCodeBench, where BCB stands for BigCodeBench.}
\label{tab:rq4-eval}
\resizebox{\linewidth}{!}{%
\begin{tabular}{@{}lcccc@{}}
\toprule
Model & Humaneval(+) & MBPP(+) & \begin{tabular}[c]{@{}c@{}}BCB\\ Full\end{tabular} & \begin{tabular}[c]{@{}c@{}}BCB\\ Hard\end{tabular} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: CodeLlama-Python-7B}} \\
CodeLlama-Instruct & 36.0 (31.1) & 56.1 (46.6) & 25.7 & 4.1 \\
+Standard-SFT & 39.0 (34.1) & 61.1 (49.7) & 26.5 & 4.1 \\
+\appname & \textbf{47.0 (43.9)} & \textbf{61.9 (51.3)} & \textbf{29.0} & \textbf{4.7} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: DeepseekCoder-6.7B-Base}} \\
DeepseekCoder-Instruct & 73.8 (70.7) & 74.9 (65.6) & 43.8 & 15.5 \\
+Standard-SFT & 75.6 (70.1) & 77.8 (66.9) & 42.0 & 13.5 \\
+\appname & \textbf{81.7 (76.2)} & \textbf{78.6 (66.9)} & \textbf{44.3} & \textbf{16.9} \\ \bottomrule
\end{tabular}%
}
\end{table}
\section{Discussion}
\label{Sec:dis}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{Figure/humaneval_line_comparison.png}
\caption{Performance of different values of $\alpha$ based on SemCoder-S on Humaneval(+).}
\label{fig:abl-alpha-human}
\end{figure}
\textit{Impact of $\alpha$ in Loss}. The parameter $\alpha$ in our loss function controls the emphasis placed on error-sensitive tokens. We further experimentally investigate the performance impact of changing $\alpha$. Specifically, we conduct experiments with $\alpha \in [1, 1.5, 2, 3, 5]$ on Humaneval(+) and observe the performance changes of LLMs. Due to the evaluation time constraints, we select SemCoder-S as the representative LLM for this ablation study, as it demonstrates the best overall performance with \appname. Figure~\ref{fig:abl-alpha-human} illustrates the performance trends as $\alpha$ varies. We can observe that $\alpha=2$ provides an optimal balance between emphasizing error-sensitive tokens and maintaining attention to shared tokens. Additionally, it can be observed that in all cases, after \appname, the LLM's performance matches or exceeds that of Standard-SFT, demonstrating the robustness to hyperparameter choices.
\input{Table/dis-dpo-all}
\textit{Compared to Reinforcement Learning Method}.
Given the increasing popularity of reinforcement learning (RL) methods for improving code generation~\cite{le2022coderl,wang2022compilable}, we believe it's important to compare our approach with these established techniques. As RL methods in code generation typically aim to align model outputs with desired code solutions by increasing the probability of correct implementations while reducing the likelihood of erroneous ones, they share similarities with our work principle of enhancing LLMs' ability to identify \textit{error-sensitive segments} to improve code generation capabilities. Specifically, we compare our approach with the representative DPO method~\cite{rafailov2023direct}, which is widely used and has demonstrated significant advantages in code generation~\cite{yang2024synthesizing,miao2024aligning,gee2024code,zhang2024codedpo,gallego2024refined}.
This method works by training models to directly maximize the likelihood of preferred outputs over non-preferred ones without requiring explicit reward modeling, learning from paired examples of more and less desirable code implementations.
To ensure a fair comparison, we select the same LLMs and use identical experimental settings as stated in RQ1 for DPO training. The training dataset remains consistent across both DPO and \appname, and we evaluate and compare the performance of DPO and \appname on the Humaneval(+), MBPP(+), and BigCodeBench benchmarks.
Table ~\ref{tab:dis-dpo-eval} shows the performance of LLMs trained with different methods. Overall, LLMs with \appname consistently outperform those trained with DPO. We can observe that \appname outperforms DPO by a relative average of 4.2\%, across three selected benchmarks. This advantage stems from fundamental methodological differences: while DPO relies on coarse-grained preference signals that cannot precisely target \textit{error-sensitive segments}, \appname specifically maintains learning across all tokens while strategically emphasizing \textit{error-sensitive segments} within code implementations. This approach ensures the LLM retains general coding knowledge while becoming more attentive to critical details that often determine functional correctness.
Additionally, we note that DPO's ability to differentially reward correct implementations and penalize incorrect ones could be leveraged to enhance the learning of error-sensitive segments. Specifically, a tailored reward function could be designed to strengthen the model's focus on these critical segments, potentially combining the strengths of both approaches. We leave this promising direction for future exploration.
\begin{table}[t]
\caption{Performance of different LLMs using \appname method compared with DPO on Humaneval(+), MBPP(+) and BigCodeBench, where BCB stands for BigCodeBench.}
\label{tab:dis-dpo-eval}
\resizebox{\linewidth}{!}{%
\begin{tabular}{@{}lcccc@{}}
\toprule
Model & Humaneval(+) & MBPP(+) & \begin{tabular}[c]{@{}c@{}}BCB\\ FUll\end{tabular} & \begin{tabular}[c]{@{}c@{}}BCB\\ HARD\end{tabular} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: CodeLlama-Python-7B}} \\
MagiCoder$\mathcal{S}$-CL & 70.7 (66.5) & 68.4 (56.6) & 39.7 & 12.8 \\
+DPO & 66.5 (61.6) & 68.8 (58.7) & 39.8 & 14.2 \\
+\appname & \textbf{73.2 (68.9)} & \textbf{71.7 (59.5)} & \textbf{42.2} & \textbf{15.5} \\ \midrule
\multicolumn{5}{l}{\textit{Base Model: DeepseekCoder-6.7B-Base}} \\
MagiCoder$\mathcal{S}$-DS & 76.8 (71.3) & 75.7 (64.4) & 47.6 & 12.8 \\
+DPO & 76.2 (71.9) & 79.1 (68.3) & 47.8 & 13.5 \\
+\appname & \textbf{77.4 (74.3)} & \textbf{79.6 (69.0)} & \textbf{48.2} & \textbf{15.5} \\ \hdashline
SemCoder-S & 79.3 (74.4) & 79.6 (68.5) & 48.5 & 16.9 \\
+DPO & 81.7 (76.2) & 81.0 (67.7) & 47.9 & 16.2 \\
+\appname & \textbf{82.9 (79.3)} & \textbf{83.1 (70.6)} & \textbf{48.9} & \textbf{20.3} \\ \bottomrule
\end{tabular}%
}
\end{table}
\section{Threats To Validity}
\label{sec:6}
There are three major threats to the validity of our work.
\textit{Threats to external validity} relate to the generalizability of our approach. While we evaluate our approach on multiple instruction-tuned models, there may be concerns about generalization to other LLMs. However, this threat is mitigated by our diverse selection of models with different series. Furthermore, the cross-dataset experiments (RQ3) and closed-source dataset experiments (RQ4) demonstrate robust generalization capabilities across different settings. In addition, due to computational resource constraints, our experiments primarily focus on 7B parameter LLMs rather than larger LLMs. In future work, we plan to explore a broader range of model series to further validate our approach's generalizability.
\textit{Threats to internal validity} involve the impact of the quality of incorrect code and choices of hyperparameters. The effectiveness of our approach depends on the quality of incorrect code variants and the weighting factor $\alpha$. To mitigate threats related to code quality, we prompt a strong teacher model (Qwen2.5-Coder-32B-Instruct) to generate plausible incorrect variants and manually verify whether the generated data meets our expectations. While a small portion of noise data - incorrect implementations that differ completely from correct solutions - remains present, we argue these instances may actually enhance model robustness by preventing overfitting to specific error-sensitive segments~\cite{devries2017improved,xie2020unsupervised}. For hyperparameter-related threats, we conduct extensive sensitivity analysis as shown in Figure~\ref{fig:abl-alpha-human}. In future work, we intend to investigate the use of stronger teacher models, such as GPT-4-Turbo, to generate similar incorrect code and examine their impact.
\textit{Threats to construct validity} relate to the reliability of evaluation metrics. We evaluate our approach using the pass rates metric; however, this metric may inadequately capture the functional correctness of generated code implementations with a limited number of test cases. To address this limitation, we deliberately incorporate the extended versions of some benchmarks, which substantially expand the number of test cases. In future work, we plan to explore additional evaluation approaches, such as LLM-as-a-Judge~\cite{gu2024survey}, to provide a more comprehensive assessment of code quality beyond functional correctness.
\section{Related Work}
\label{sec:2}
\subsection{LLMs for Code Generation}
As a momentous milestone, Codex~\cite{chen2021evaluating} boasting a 12-billion-parameter model demonstrates the extraordinary capability to tackle up to 72\% of Python programming problems. After that, a new wave of code generation models, such as AlphaCode~\cite{li2022competition}, CodeGen~\cite{nijkampcodegen}, InCoder~\cite{friedincoder} and StarCoder~\cite{li2023starcoder} are proposed and have shown promising results in the code generation task. Building upon these foundations, more code-focused LLMs emerged, such as Magicoder~\cite{wei2024magicoder}, SemCoder~\cite{dingsemcoder}, WaveCoder~\cite{yu2024wavecoder} and WizardCoder~\cite{luowizardcoder}. These specialized LLMs are typically based on general LLMs in solving domain-specific coding tasks through instruction tuning.
\subsection{Fine-tuning on Code LLM}
Fine-tuning pre-trained language models has emerged as a dominant paradigm for optimizing performance in code generation. Instruction tuning~\cite{ouyang2022training,achiam2023gpt}, as a form of supervised fine-tuning, aims to align LLMs with instruction through high-quality instruction corpora. For instance, Magicoder~\cite{wei2024magicoder} introduce OSS-Instruct, an instruction tuning dataset generated by a teacher LLM drawing inspiration from open-source code snippets, which effectively enhances code generation capabilities when used for fine-tuning. Furthermore, OSS-Instruct is orthogonal to existing instruction tuning datasets like Evol-Instruct~\cite{luowizardcoder}, enabling the Magicoder$\mathcal{S}$ series LLMs finetuned on this combined data to achieve further performance improvements. Similarly, SemCoder~\cite{dingsemcoder} propose PYX, a dataset created by a teacher LLM simulating human debugging processes. By incorporating data that simulates execution reasoning and captures code execution nuances, LLMs finetuned with PYX understand and articulate the execution process step-by-step, enhancing their reasoning capabilities. Likewise, combining with existing instruction tuning datasets like Evol-Instruct, the resulting SemCoder-S LLM further improves the original LLM's code generation abilities.
To address limitations in preventing untruthful and unexpected outputs, researchers explore reinforcement learning~\cite{ouyang2022training}. To address limitations in undesired outputs, researchers have explored reinforcement learning approaches. For instance, CodeRL~\cite{le2022coderl} utilizes compiler feedback as reward signals combined with REINFORCE~\cite{williams1992simple} to fine-tune CodeT5~\cite{wang2021codet5}, reducing compilation errors. SENSE~\cite{yang2024synthesizing} synthesizes text-to-SQL training data from both strong and weak language models, and leverage Direct Preference Optimization (DPO)~\cite{rafailov2023direct} to learn from correct and incorrect SQL examples, demonstrating state-of-the-art performance on text-to-SQL benchmarks. Similarly, PPOCoder~\cite{shojaeeexecution} trains CodeT5 with proximal policy optimization~\cite{schulman2017proximal}. However, despite these advances, these fine-tuning approaches face a fundamental limitation: they treat all tokens with equal importance during loss calculation, making it difficult for models to distinguish semantically correct implementations from syntactically similar but incorrect ones. In this paper, we aim to address this challenge in code LLMs.
We find recent work Focused-DPO~\cite{zhang2025focused} enhances code generation capability by concentrating preference optimization on error-prone code points through an improved DPO~\cite{rafailov2023direct} methodology. Our method is complementary to this approach - while Focused-DPO operates during post-training reinforcement learning stages, our approach operates during the supervised fine-tuning stage. Additionally, their work has demonstrated that their method can strengthen the capabilities of post-trained models. However, as Focused-DPO is currently under peer review and its implementation is not yet publicly available, we are unable to experimentally validate the potential synergies between our approaches in this work.
\section{Conclusion}
\label{sec:7}
In this paper, we introduce Fault-Aware Fine-Tuning (\appname), a novel fine-tuning technique that enhances code generation capabilities in instruction-tuned LLMs by refining their ability to distinguish between correct implementations and subtly incorrect variants. Unlike conventional supervised fine-tuning that treats all tokens equally, our approach identifies and prioritizes \textit{error-sensitive segments} through two key components: \textit{error-sensitive segments} identification, that captures both line-level and token-level critical differences, and dynamic importance reweighting that dynamically adjusts token weights during training to focus on discriminative elements. Through extensive experiments across seven LLMs and three widely-used benchmarks, we demonstrate that our method achieves an average relative improvement of 6.9\% on pass@1 with just one epoch of training, with certain enhanced 6.7B LLMs even outperforming GPT-3.5-Turbo on selected benchmarks. The technique also exhibits strong generalization capabilities across diverse instruction-tuned LLMs and maintains effectiveness even when applied to LLMs with closed-source instruction datasets, providing a practical solution to improve LLMs' code generation capabilities.
\section*{Data Availability}
Our code is available: https://anonymous.4open.science/r/FAFT-976B.
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{52}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{} and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showURL}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showURL #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Achiam et~al\mbox{.}(2023)]%
{achiam2023gpt}
\bibfield{author}{\bibinfo{person}{Josh Achiam}, \bibinfo{person}{Steven Adler}, \bibinfo{person}{Sandhini Agarwal}, \bibinfo{person}{Lama Ahmad}, \bibinfo{person}{Ilge Akkaya}, \bibinfo{person}{Florencia~Leoni Aleman}, \bibinfo{person}{Diogo Almeida}, \bibinfo{person}{Janko Altenschmidt}, \bibinfo{person}{Sam Altman}, \bibinfo{person}{Shyamal Anadkat}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Gpt-4 technical report}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2303.08774}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Austin et~al\mbox{.}(2021)]%
{austin2021program}
\bibfield{author}{\bibinfo{person}{Jacob Austin}, \bibinfo{person}{Augustus Odena}, \bibinfo{person}{Maxwell Nye}, \bibinfo{person}{Maarten Bosma}, \bibinfo{person}{Henryk Michalewski}, \bibinfo{person}{David Dohan}, \bibinfo{person}{Ellen Jiang}, \bibinfo{person}{Carrie Cai}, \bibinfo{person}{Michael Terry}, \bibinfo{person}{Quoc Le}, {et~al\mbox{.}}} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Program synthesis with large language models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2108.07732}} (\bibinfo{year}{2021}).
\newblock
\bibitem[Baker and Scarf(2021)]%
{baker2021modifying}
\bibfield{author}{\bibinfo{person}{Rose Baker} {and} \bibinfo{person}{Philip Scarf}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Modifying Bradley--Terry and other ranking models to allow ties}.
\newblock \bibinfo{journal}{\emph{IMA Journal of Management Mathematics}} \bibinfo{volume}{32}, \bibinfo{number}{4} (\bibinfo{year}{2021}), \bibinfo{pages}{451--463}.
\newblock
\bibitem[Chen et~al\mbox{.}(2021)]%
{chen2021evaluating}
\bibfield{author}{\bibinfo{person}{Mark Chen}, \bibinfo{person}{Jerry Tworek}, \bibinfo{person}{Heewoo Jun}, \bibinfo{person}{Qiming Yuan}, \bibinfo{person}{Henrique Ponde De~Oliveira Pinto}, \bibinfo{person}{Jared Kaplan}, \bibinfo{person}{Harri Edwards}, \bibinfo{person}{Yuri Burda}, \bibinfo{person}{Nicholas Joseph}, \bibinfo{person}{Greg Brockman}, {et~al\mbox{.}}} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Evaluating large language models trained on code}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2107.03374}} (\bibinfo{year}{2021}).
\newblock
\bibitem[DeVries and Taylor(2017)]%
{devries2017improved}
\bibfield{author}{\bibinfo{person}{Terrance DeVries} {and} \bibinfo{person}{Graham~W Taylor}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Improved regularization of convolutional neural networks with cutout}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1708.04552}} (\bibinfo{year}{2017}).
\newblock
\bibitem[Ding et~al\mbox{.}(2024)]%
{dingsemcoder}
\bibfield{author}{\bibinfo{person}{Yangruibo Ding}, \bibinfo{person}{Jinjun Peng}, \bibinfo{person}{Marcus~J Min}, \bibinfo{person}{Gail Kaiser}, \bibinfo{person}{Junfeng Yang}, {and} \bibinfo{person}{Baishakhi Ray}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning}. In \bibinfo{booktitle}{\emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}}.
\newblock
\bibitem[Dong et~al\mbox{.}(2024)]%
{dong2024self}
\bibfield{author}{\bibinfo{person}{Yihong Dong}, \bibinfo{person}{Xue Jiang}, \bibinfo{person}{Zhi Jin}, {and} \bibinfo{person}{Ge Li}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Self-collaboration code generation via chatgpt}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and Methodology}} \bibinfo{volume}{33}, \bibinfo{number}{7} (\bibinfo{year}{2024}), \bibinfo{pages}{1--38}.
\newblock
\bibitem[Dubey et~al\mbox{.}(2024)]%
{dubey2024llama}
\bibfield{author}{\bibinfo{person}{Abhimanyu Dubey}, \bibinfo{person}{Abhinav Jauhri}, \bibinfo{person}{Abhinav Pandey}, \bibinfo{person}{Abhishek Kadian}, \bibinfo{person}{Ahmad Al-Dahle}, \bibinfo{person}{Aiesha Letman}, \bibinfo{person}{Akhil Mathur}, \bibinfo{person}{Alan Schelten}, \bibinfo{person}{Amy Yang}, \bibinfo{person}{Angela Fan}, {et~al\mbox{.}}} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{The llama 3 herd of models}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2407.21783}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Fakhoury et~al\mbox{.}(2024)]%
{fakhoury2024llm}
\bibfield{author}{\bibinfo{person}{Sarah Fakhoury}, \bibinfo{person}{Aaditya Naik}, \bibinfo{person}{Georgios Sakkas}, \bibinfo{person}{Saikat Chakraborty}, {and} \bibinfo{person}{Shuvendu~K Lahiri}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Llm-based test-driven interactive code generation: User study and empirical evaluation}.
\newblock \bibinfo{journal}{\emph{IEEE Transactions on Software Engineering}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Fried et~al\mbox{.}({[n.\,d.]})]%
{friedincoder}
\bibfield{author}{\bibinfo{person}{Daniel Fried}, \bibinfo{person}{Armen Aghajanyan}, \bibinfo{person}{Jessy Lin}, \bibinfo{person}{Sida Wang}, \bibinfo{person}{Eric Wallace}, \bibinfo{person}{Freda Shi}, \bibinfo{person}{Ruiqi Zhong}, \bibinfo{person}{Scott Yih}, \bibinfo{person}{Luke Zettlemoyer}, {and} \bibinfo{person}{Mike Lewis}.} \bibinfo{year}{[n.\,d.]}\natexlab{}.
\newblock \showarticletitle{InCoder: A Generative Model for Code Infilling and Synthesis}. In \bibinfo{booktitle}{\emph{The Eleventh International Conference on Learning Representations}}.
\newblock
\bibitem[Gallego(2024)]%
{gallego2024refined}
\bibfield{author}{\bibinfo{person}{V{\'\i}ctor Gallego}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Refined direct preference optimization with synthetic data for behavioral alignment of llms}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2402.08005}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Gee et~al\mbox{.}(2024)]%
{gee2024code}
\bibfield{author}{\bibinfo{person}{Leonidas Gee}, \bibinfo{person}{Milan Gritta}, \bibinfo{person}{Gerasimos Lampouras}, {and} \bibinfo{person}{Ignacio Iacobacci}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Code-optimise: Self-generated preference data for correctness and efficiency}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.12502}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Gu et~al\mbox{.}(2024)]%
{gu2024survey}
\bibfield{author}{\bibinfo{person}{Jiawei Gu}, \bibinfo{person}{Xuhui Jiang}, \bibinfo{person}{Zhichao Shi}, \bibinfo{person}{Hexiang Tan}, \bibinfo{person}{Xuehao Zhai}, \bibinfo{person}{Chengjin Xu}, \bibinfo{person}{Wei Li}, \bibinfo{person}{Yinghan Shen}, \bibinfo{person}{Shengjie Ma}, \bibinfo{person}{Honghao Liu}, {et~al\mbox{.}}} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{A Survey on LLM-as-a-Judge}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2411.15594}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Gunasekar et~al\mbox{.}(2023)]%
{gunasekar2023textbooks}
\bibfield{author}{\bibinfo{person}{Suriya Gunasekar}, \bibinfo{person}{Yi Zhang}, \bibinfo{person}{Jyoti Aneja}, \bibinfo{person}{Caio C{\'e}sar~Teodoro Mendes}, \bibinfo{person}{Allie Del~Giorno}, \bibinfo{person}{Sivakanth Gopi}, \bibinfo{person}{Mojan Javaheripi}, \bibinfo{person}{Piero Kauffmann}, \bibinfo{person}{Gustavo de Rosa}, \bibinfo{person}{Olli Saarikivi}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Textbooks are all you need}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2306.11644}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Guo et~al\mbox{.}(2024)]%
{guo2024deepseek}
\bibfield{author}{\bibinfo{person}{Daya Guo}, \bibinfo{person}{Qihao Zhu}, \bibinfo{person}{Dejian Yang}, \bibinfo{person}{Zhenda Xie}, \bibinfo{person}{Kai Dong}, \bibinfo{person}{Wentao Zhang}, \bibinfo{person}{Guanting Chen}, \bibinfo{person}{Xiao Bi}, \bibinfo{person}{Yu Wu}, \bibinfo{person}{YK Li}, {et~al\mbox{.}}} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2401.14196}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Huang et~al\mbox{.}(2025)]%
{huang2025survey}
\bibfield{author}{\bibinfo{person}{Lei Huang}, \bibinfo{person}{Weijiang Yu}, \bibinfo{person}{Weitao Ma}, \bibinfo{person}{Weihong Zhong}, \bibinfo{person}{Zhangyin Feng}, \bibinfo{person}{Haotian Wang}, \bibinfo{person}{Qianglong Chen}, \bibinfo{person}{Weihua Peng}, \bibinfo{person}{Xiaocheng Feng}, \bibinfo{person}{Bing Qin}, {et~al\mbox{.}}} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Information Systems}} \bibinfo{volume}{43}, \bibinfo{number}{2} (\bibinfo{year}{2025}), \bibinfo{pages}{1--55}.
\newblock
\bibitem[Huang et~al\mbox{.}(2023)]%
{huang2023competition}
\bibfield{author}{\bibinfo{person}{Yiming Huang}, \bibinfo{person}{Zhenghao Lin}, \bibinfo{person}{Xiao Liu}, \bibinfo{person}{Yeyun Gong}, \bibinfo{person}{Shuai Lu}, \bibinfo{person}{Fangyu Lei}, \bibinfo{person}{Yaobo Liang}, \bibinfo{person}{Yelong Shen}, \bibinfo{person}{Chen Lin}, \bibinfo{person}{Nan Duan}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Competition-level problems are effective llm evaluators}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2312.02143}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Hunter(2004)]%
{hunter2004mm}
\bibfield{author}{\bibinfo{person}{David~R Hunter}.} \bibinfo{year}{2004}\natexlab{}.
\newblock \showarticletitle{MM algorithms for generalized Bradley-Terry models}.
\newblock \bibinfo{journal}{\emph{The annals of statistics}} \bibinfo{volume}{32}, \bibinfo{number}{1} (\bibinfo{year}{2004}), \bibinfo{pages}{384--406}.
\newblock
\bibitem[Jiang et~al\mbox{.}(2024)]%
{jiang2024self}
\bibfield{author}{\bibinfo{person}{Xue Jiang}, \bibinfo{person}{Yihong Dong}, \bibinfo{person}{Lecheng Wang}, \bibinfo{person}{Zheng Fang}, \bibinfo{person}{Qiwei Shang}, \bibinfo{person}{Ge Li}, \bibinfo{person}{Zhi Jin}, {and} \bibinfo{person}{Wenpin Jiao}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Self-planning code generation with large language models}.
\newblock \bibinfo{journal}{\emph{ACM Transactions on Software Engineering and Methodology}} \bibinfo{volume}{33}, \bibinfo{number}{7} (\bibinfo{year}{2024}), \bibinfo{pages}{1--30}.
\newblock
\bibitem[Le et~al\mbox{.}(2022)]%
{le2022coderl}
\bibfield{author}{\bibinfo{person}{Hung Le}, \bibinfo{person}{Yue Wang}, \bibinfo{person}{Akhilesh~Deepak Gotmare}, \bibinfo{person}{Silvio Savarese}, {and} \bibinfo{person}{Steven Chu~Hong Hoi}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Coderl: Mastering code generation through pretrained models and deep reinforcement learning}.
\newblock \bibinfo{journal}{\emph{Advances in Neural Information Processing Systems}} \bibinfo{volume}{35} (\bibinfo{year}{2022}), \bibinfo{pages}{21314--21328}.
\newblock
\bibitem[Li et~al\mbox{.}(2023)]%
{li2023starcoder}
\bibfield{author}{\bibinfo{person}{R Li}, \bibinfo{person}{LB Allal}, \bibinfo{person}{Y Zi}, \bibinfo{person}{N Muennighoff}, \bibinfo{person}{D Kocetkov}, \bibinfo{person}{C Mou}, \bibinfo{person}{M Marone}, \bibinfo{person}{C Akiki}, \bibinfo{person}{J Li}, \bibinfo{person}{J Chim}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{StarCoder: May the Source be With You!}
\newblock \bibinfo{journal}{\emph{Transactions on machine learning research}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Li et~al\mbox{.}(2022)]%
{li2022competition}
\bibfield{author}{\bibinfo{person}{Yujia Li}, \bibinfo{person}{David Choi}, \bibinfo{person}{Junyoung Chung}, \bibinfo{person}{Nate Kushman}, \bibinfo{person}{Julian Schrittwieser}, \bibinfo{person}{R{\'e}mi Leblond}, \bibinfo{person}{Tom Eccles}, \bibinfo{person}{James Keeling}, \bibinfo{person}{Felix Gimeno}, \bibinfo{person}{Agustin Dal~Lago}, {et~al\mbox{.}}} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Competition-level code generation with alphacode}.
\newblock \bibinfo{journal}{\emph{Science}} \bibinfo{volume}{378}, \bibinfo{number}{6624} (\bibinfo{year}{2022}), \bibinfo{pages}{1092--1097}.
\newblock
\bibitem[Liu et~al\mbox{.}(2024a)]%
{liu2024exploring}
\bibfield{author}{\bibinfo{person}{Fang Liu}, \bibinfo{person}{Yang Liu}, \bibinfo{person}{Lin Shi}, \bibinfo{person}{Houkun Huang}, \bibinfo{person}{Ruifeng Wang}, \bibinfo{person}{Zhen Yang}, \bibinfo{person}{Li Zhang}, \bibinfo{person}{Zhongqi Li}, {and} \bibinfo{person}{Yuchi Ma}.} \bibinfo{year}{2024}\natexlab{a}.
\newblock \showarticletitle{Exploring and evaluating hallucinations in llm-powered code generation}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2404.00971}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Liu et~al\mbox{.}(2024b)]%
{liu2024your}
\bibfield{author}{\bibinfo{person}{Jiawei Liu}, \bibinfo{person}{Chunqiu~Steven Xia}, \bibinfo{person}{Yuyao Wang}, {and} \bibinfo{person}{Lingming Zhang}.} \bibinfo{year}{2024}\natexlab{b}.
\newblock \showarticletitle{Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation}.
\newblock \bibinfo{journal}{\emph{Advances in Neural Information Processing Systems}} \bibinfo{volume}{36} (\bibinfo{year}{2024}).
\newblock
\bibitem[Luo et~al\mbox{.}({[n.\,d.]})]%
{luowizardcoder}
\bibfield{author}{\bibinfo{person}{Ziyang Luo}, \bibinfo{person}{Can Xu}, \bibinfo{person}{Pu Zhao}, \bibinfo{person}{Qingfeng Sun}, \bibinfo{person}{Xiubo Geng}, \bibinfo{person}{Wenxiang Hu}, \bibinfo{person}{Chongyang Tao}, \bibinfo{person}{Jing Ma}, \bibinfo{person}{Qingwei Lin}, {and} \bibinfo{person}{Daxin Jiang}.} \bibinfo{year}{[n.\,d.]}\natexlab{}.
\newblock \showarticletitle{WizardCoder: Empowering Code Large Language Models with Evol-Instruct}. In \bibinfo{booktitle}{\emph{The Twelfth International Conference on Learning Representations}}.
\newblock
\bibitem[Luo et~al\mbox{.}(2023)]%
{luo2023wizardcoder}
\bibfield{author}{\bibinfo{person}{Ziyang Luo}, \bibinfo{person}{Can Xu}, \bibinfo{person}{Pu Zhao}, \bibinfo{person}{Qingfeng Sun}, \bibinfo{person}{Xiubo Geng}, \bibinfo{person}{Wenxiang Hu}, \bibinfo{person}{Chongyang Tao}, \bibinfo{person}{Jing Ma}, \bibinfo{person}{Qingwei Lin}, {and} \bibinfo{person}{Daxin Jiang}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \bibinfo{title}{WizardCoder: Empowering Code Large Language Models with Evol-Instruct}.
\newblock
\bibitem[Menke and Martinez(2008)]%
{menke2008bradley}
\bibfield{author}{\bibinfo{person}{Joshua~E Menke} {and} \bibinfo{person}{Tony~R Martinez}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{A Bradley--Terry artificial neural network model for individual ratings in group competitions}.
\newblock \bibinfo{journal}{\emph{Neural computing and Applications}} \bibinfo{volume}{17} (\bibinfo{year}{2008}), \bibinfo{pages}{175--186}.
\newblock
\bibitem[Miao et~al\mbox{.}(2024)]%
{miao2024aligning}
\bibfield{author}{\bibinfo{person}{Yibo Miao}, \bibinfo{person}{Bofei Gao}, \bibinfo{person}{Shanghaoran Quan}, \bibinfo{person}{Junyang Lin}, \bibinfo{person}{Daoguang Zan}, \bibinfo{person}{Jiaheng Liu}, \bibinfo{person}{Jian Yang}, \bibinfo{person}{Tianyu Liu}, {and} \bibinfo{person}{Zhijie Deng}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Aligning codellms with direct preference optimization}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2410.18585}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Mu et~al\mbox{.}(2024)]%
{mu2023clarifygpt}
\bibfield{author}{\bibinfo{person}{Fangwen Mu}, \bibinfo{person}{Lin Shi}, \bibinfo{person}{Song Wang}, \bibinfo{person}{Zhuohao Yu}, \bibinfo{person}{Binquan Zhang}, \bibinfo{person}{ChenXue Wang}, \bibinfo{person}{Shichao Liu}, {and} \bibinfo{person}{Qing Wang}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification}.
\newblock \bibinfo{journal}{\emph{Proceedings of the ACM on Software Engineering}} \bibinfo{volume}{1}, \bibinfo{number}{FSE} (\bibinfo{year}{2024}), \bibinfo{pages}{2332--2354}.
\newblock
\bibitem[Nijkamp et~al\mbox{.}({[n.\,d.]})]%
{nijkampcodegen}
\bibfield{author}{\bibinfo{person}{Erik Nijkamp}, \bibinfo{person}{Bo Pang}, \bibinfo{person}{Hiroaki Hayashi}, \bibinfo{person}{Lifu Tu}, \bibinfo{person}{Huan Wang}, \bibinfo{person}{Yingbo Zhou}, \bibinfo{person}{Silvio Savarese}, {and} \bibinfo{person}{Caiming Xiong}.} \bibinfo{year}{[n.\,d.]}\natexlab{}.
\newblock \showarticletitle{CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis}. In \bibinfo{booktitle}{\emph{The Eleventh International Conference on Learning Representations}}.
\newblock
\bibitem[{OpenAI}(2022)]%
{OpenAI2022}
\bibfield{author}{\bibinfo{person}{{OpenAI}}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \bibinfo{title}{{ChatGPT}}.
\newblock
\urldef\tempurl%
\url{https://openai.com/blog/chatgpt/}
\showURL{%
\tempurl}
\bibitem[{OpenAI}(2024)]%
{OpenAI2024GPT4o}
\bibfield{author}{\bibinfo{person}{{OpenAI}}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{title}{{GPT-4o-mini}}.
\newblock
\urldef\tempurl%
\url{https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/}
\showURL{%
\tempurl}
\bibitem[OpenAI and Josh~Achiam(2024)]%
{OpenAI2023}
\bibfield{author}{\bibinfo{person}{OpenAI} {and} \bibinfo{person}{et~al. Josh~Achiam}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{title}{GPT-4 Technical Report}.
\newblock
\showeprint[arxiv]{2303.08774}~[cs.CL]
\urldef\tempurl%
\url{https://arxiv.org/abs/2303.08774}
\showURL{%
\tempurl}
\bibitem[Ouyang et~al\mbox{.}(2022)]%
{ouyang2022training}
\bibfield{author}{\bibinfo{person}{Long Ouyang}, \bibinfo{person}{Jeffrey Wu}, \bibinfo{person}{Xu Jiang}, \bibinfo{person}{Diogo Almeida}, \bibinfo{person}{Carroll Wainwright}, \bibinfo{person}{Pamela Mishkin}, \bibinfo{person}{Chong Zhang}, \bibinfo{person}{Sandhini Agarwal}, \bibinfo{person}{Katarina Slama}, \bibinfo{person}{Alex Ray}, {et~al\mbox{.}}} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Training language models to follow instructions with human feedback}.
\newblock \bibinfo{journal}{\emph{Advances in neural information processing systems}} \bibinfo{volume}{35} (\bibinfo{year}{2022}), \bibinfo{pages}{27730--27744}.
\newblock
\bibitem[Rafailov et~al\mbox{.}(2023)]%
{rafailov2023direct}
\bibfield{author}{\bibinfo{person}{Rafael Rafailov}, \bibinfo{person}{Archit Sharma}, \bibinfo{person}{Eric Mitchell}, \bibinfo{person}{Christopher~D Manning}, \bibinfo{person}{Stefano Ermon}, {and} \bibinfo{person}{Chelsea Finn}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Direct preference optimization: Your language model is secretly a reward model}.
\newblock \bibinfo{journal}{\emph{Advances in Neural Information Processing Systems}} \bibinfo{volume}{36} (\bibinfo{year}{2023}), \bibinfo{pages}{53728--53741}.
\newblock
\bibitem[Roziere et~al\mbox{.}(2023)]%
{roziere2023code}
\bibfield{author}{\bibinfo{person}{Baptiste Roziere}, \bibinfo{person}{Jonas Gehring}, \bibinfo{person}{Fabian Gloeckle}, \bibinfo{person}{Sten Sootla}, \bibinfo{person}{Itai Gat}, \bibinfo{person}{Xiaoqing~Ellen Tan}, \bibinfo{person}{Yossi Adi}, \bibinfo{person}{Jingyu Liu}, \bibinfo{person}{Romain Sauvestre}, \bibinfo{person}{Tal Remez}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Code llama: Open foundation models for code}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2308.12950}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Schulman et~al\mbox{.}(2017)]%
{schulman2017proximal}
\bibfield{author}{\bibinfo{person}{John Schulman}, \bibinfo{person}{Filip Wolski}, \bibinfo{person}{Prafulla Dhariwal}, \bibinfo{person}{Alec Radford}, {and} \bibinfo{person}{Oleg Klimov}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Proximal policy optimization algorithms}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:1707.06347}} (\bibinfo{year}{2017}).
\newblock
\bibitem[Shojaee et~al\mbox{.}({[n.\,d.]})]%
{shojaeeexecution}
\bibfield{author}{\bibinfo{person}{Parshin Shojaee}, \bibinfo{person}{Aneesh Jain}, \bibinfo{person}{Sindhu Tipirneni}, {and} \bibinfo{person}{Chandan~K Reddy}.} \bibinfo{year}{[n.\,d.]}\natexlab{}.
\newblock \showarticletitle{Execution-based Code Generation using Deep Reinforcement Learning}.
\newblock \bibinfo{journal}{\emph{Transactions on Machine Learning Research}} (\bibinfo{year}{[n.\,d.]}).
\newblock
\bibitem[Wang et~al\mbox{.}(2022)]%
{wang2022compilable}
\bibfield{author}{\bibinfo{person}{Xin Wang}, \bibinfo{person}{Yasheng Wang}, \bibinfo{person}{Yao Wan}, \bibinfo{person}{Fei Mi}, \bibinfo{person}{Yitong Li}, \bibinfo{person}{Pingyi Zhou}, \bibinfo{person}{Jin Liu}, \bibinfo{person}{Hao Wu}, \bibinfo{person}{Xin Jiang}, {and} \bibinfo{person}{Qun Liu}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Compilable Neural Code Generation with Compiler Feedback}. In \bibinfo{booktitle}{\emph{Findings of the Association for Computational Linguistics: ACL 2022}}. \bibinfo{pages}{9--19}.
\newblock
\bibitem[Wang et~al\mbox{.}(2021)]%
{wang2021codet5}
\bibfield{author}{\bibinfo{person}{Yue Wang}, \bibinfo{person}{Weishi Wang}, \bibinfo{person}{Shafiq Joty}, {and} \bibinfo{person}{Steven~CH Hoi}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation}. In \bibinfo{booktitle}{\emph{Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}}. \bibinfo{pages}{8696--8708}.
\newblock
\bibitem[Wei et~al\mbox{.}(2024)]%
{wei2024magicoder}
\bibfield{author}{\bibinfo{person}{Yuxiang Wei}, \bibinfo{person}{Zhe Wang}, \bibinfo{person}{Jiawei Liu}, \bibinfo{person}{Yifeng Ding}, {and} \bibinfo{person}{Lingming Zhang}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Magicoder: Empowering code generation with oss-instruct}. In \bibinfo{booktitle}{\emph{Forty-first International Conference on Machine Learning}}.
\newblock
\bibitem[Williams(1992)]%
{williams1992simple}
\bibfield{author}{\bibinfo{person}{Ronald~J Williams}.} \bibinfo{year}{1992}\natexlab{}.
\newblock \showarticletitle{Simple statistical gradient-following algorithms for connectionist reinforcement learning}.
\newblock \bibinfo{journal}{\emph{Machine learning}} \bibinfo{volume}{8} (\bibinfo{year}{1992}), \bibinfo{pages}{229--256}.
\newblock
\bibitem[Xie et~al\mbox{.}(2020)]%
{xie2020unsupervised}
\bibfield{author}{\bibinfo{person}{Qizhe Xie}, \bibinfo{person}{Zihang Dai}, \bibinfo{person}{Eduard Hovy}, \bibinfo{person}{Thang Luong}, {and} \bibinfo{person}{Quoc Le}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Unsupervised data augmentation for consistency training}.
\newblock \bibinfo{journal}{\emph{Advances in neural information processing systems}} \bibinfo{volume}{33} (\bibinfo{year}{2020}), \bibinfo{pages}{6256--6268}.
\newblock
\bibitem[Yan(2016)]%
{yan2016ranking}
\bibfield{author}{\bibinfo{person}{Ting Yan}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Ranking in the generalized Bradley--Terry models when the strong connection condition fails}.
\newblock \bibinfo{journal}{\emph{Communications in Statistics-Theory and Methods}} \bibinfo{volume}{45}, \bibinfo{number}{2} (\bibinfo{year}{2016}), \bibinfo{pages}{340--353}.
\newblock
\bibitem[Yang et~al\mbox{.}(2024)]%
{yang2024synthesizing}
\bibfield{author}{\bibinfo{person}{Jiaxi Yang}, \bibinfo{person}{Binyuan Hui}, \bibinfo{person}{Min Yang}, \bibinfo{person}{Jian Yang}, \bibinfo{person}{Junyang Lin}, {and} \bibinfo{person}{Chang Zhou}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Synthesizing Text-to-SQL Data from Weak and Strong LLMs}. In \bibinfo{booktitle}{\emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}}. \bibinfo{pages}{7864--7875}.
\newblock
\bibitem[Yu et~al\mbox{.}(2024)]%
{yu2024wavecoder}
\bibfield{author}{\bibinfo{person}{Zhaojian Yu}, \bibinfo{person}{Xin Zhang}, \bibinfo{person}{Ning Shang}, \bibinfo{person}{Yangyu Huang}, \bibinfo{person}{Can Xu}, \bibinfo{person}{Yishujie Zhao}, \bibinfo{person}{Wenxiang Hu}, {and} \bibinfo{person}{Qiufeng Yin}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning}. In \bibinfo{booktitle}{\emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}}. \bibinfo{pages}{5140--5153}.
\newblock
\bibitem[Yujian and Bo(2007)]%
{yujian2007normalized}
\bibfield{author}{\bibinfo{person}{Li Yujian} {and} \bibinfo{person}{Liu Bo}.} \bibinfo{year}{2007}\natexlab{}.
\newblock \showarticletitle{A normalized Levenshtein distance metric}.
\newblock \bibinfo{journal}{\emph{IEEE transactions on pattern analysis and machine intelligence}} \bibinfo{volume}{29}, \bibinfo{number}{6} (\bibinfo{year}{2007}), \bibinfo{pages}{1091--1095}.
\newblock
\bibitem[Zhang et~al\mbox{.}(2024)]%
{zhang2024codedpo}
\bibfield{author}{\bibinfo{person}{Kechi Zhang}, \bibinfo{person}{Ge Li}, \bibinfo{person}{Yihong Dong}, \bibinfo{person}{Jingjing Xu}, \bibinfo{person}{Jun Zhang}, \bibinfo{person}{Jing Su}, \bibinfo{person}{Yongfei Liu}, {and} \bibinfo{person}{Zhi Jin}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Codedpo: Aligning code models with self generated and verified source code}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2410.05605}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Zhang et~al\mbox{.}(2025)]%
{zhang2025focused}
\bibfield{author}{\bibinfo{person}{Kechi Zhang}, \bibinfo{person}{Ge Li}, \bibinfo{person}{Jia Li}, \bibinfo{person}{Yihong Dong}, {and} \bibinfo{person}{Zhi Jin}.} \bibinfo{year}{2025}\natexlab{}.
\newblock \showarticletitle{Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2502.11475}} (\bibinfo{year}{2025}).
\newblock
\bibitem[Zhang et~al\mbox{.}(2023)]%
{zhang2023instruction}
\bibfield{author}{\bibinfo{person}{Shengyu Zhang}, \bibinfo{person}{Linfeng Dong}, \bibinfo{person}{Xiaoya Li}, \bibinfo{person}{Sen Zhang}, \bibinfo{person}{Xiaofei Sun}, \bibinfo{person}{Shuhe Wang}, \bibinfo{person}{Jiwei Li}, \bibinfo{person}{Runyi Hu}, \bibinfo{person}{Tianwei Zhang}, \bibinfo{person}{Fei Wu}, {et~al\mbox{.}}} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Instruction tuning for large language models: A survey}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2308.10792}} (\bibinfo{year}{2023}).
\newblock
\bibitem[Zheng et~al\mbox{.}(2024)]%
{zheng2024opencodeinterpreter}
\bibfield{author}{\bibinfo{person}{Tianyu Zheng}, \bibinfo{person}{Ge Zhang}, \bibinfo{person}{Tianhao Shen}, \bibinfo{person}{Xueling Liu}, \bibinfo{person}{Bill~Yuchen Lin}, \bibinfo{person}{Jie Fu}, \bibinfo{person}{Wenhu Chen}, {and} \bibinfo{person}{Xiang Yue}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Opencodeinterpreter: Integrating code generation with execution and refinement}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2402.14658}} (\bibinfo{year}{2024}).
\newblock
\bibitem[Zhuo et~al\mbox{.}(2024)]%
{zhuo2024bigcodebench}
\bibfield{author}{\bibinfo{person}{Terry~Yue Zhuo}, \bibinfo{person}{Minh~Chien Vu}, \bibinfo{person}{Jenny Chim}, \bibinfo{person}{Han Hu}, \bibinfo{person}{Wenhao Yu}, \bibinfo{person}{Ratnadira Widyasari}, \bibinfo{person}{Imam Nur~Bani Yusuf}, \bibinfo{person}{Haolan Zhan}, \bibinfo{person}{Junda He}, \bibinfo{person}{Indraneil Paul}, {et~al\mbox{.}}} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions}.
\newblock \bibinfo{journal}{\emph{arXiv preprint arXiv:2406.15877}} (\bibinfo{year}{2024}).
\newblock
\end{thebibliography}