Large Language Models as Code Executors: An Exploratory Study
Extracted source text · arXiv 2410.06667
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2024
% ready for submission
% \usepackage{neurips_2024}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2024}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2024}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2024}
\usepackage{caption}
\usepackage{tabularray}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\newcommand{\cyl}[1]{\textcolor{orange}{#1}}
\usepackage{natbib}
\bibliographystyle{abbrvnat}
% % \usepackage[square,numbers]{natbib}
% \usepackage{subcaption} % for subfigures
% \usepackage[export]{adjustbox} % for valign in subfigures
\usepackage{verbatim}
\usepackage{inconsolata}
\usepackage{tabularx}
\usepackage{float}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{amssymb}
% Define a custom color for the background
\definecolor{lightgray}{rgb}{0.95, 0.95, 0.95}
\definecolor{darkgray}{rgb}{0.4, 0.4, 0.4}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\definecolor{myblue}{rgb}{0.2, 0.4, 0.8} % Example blue color
\definecolor{mygreen}{rgb}{0.2, 0.6, 0.2} % Example green color
\lstset{
basicstyle=\ttfamily\small,
backgroundcolor=\color{backcolour},
frame=single,
framerule=0.5pt,
rulecolor=\color{darkgray},
numbers=left,
numberstyle=\tiny\color{darkgray},
xleftmargin=2em,
framexleftmargin=1.5em,
framexrightmargin=1.5em,
breaklines=true,
columns=fullflexible,
escapeinside={(*}{*)}, % Allows LaTeX commands inside the listing
showstringspaces=false,
moredelim=**[is][\color{red}]{@}{@}, % Define delimiters for red text
moredelim=**[is][\color{myblue}]{~}{~}, % Define delimiters for blue text
moredelim=**[is][\color{mygreen}]{*}{*} % Define delimiters for green text
}
\usepackage{amsmath,amssymb,amsfonts}
% \usepackage{tabularx}
% \usepackage{subcaption}
% \usepackage{ascii}
% \usepackage{newunicodechar}
% \usepackage{multirow}
% \usepackage{enumitem}
% \usepackage{arabtex,utf8}
% \usepackage{wrapfig}
\title{Large Language Models as Code Executors: An Exploratory Study }
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
% \author{%
% Lecheng Yan \\
% School of Computer Science and Technology\\
% Urumgi830046,China:Xinjiang University\\
% \texttt{20222210209@stu.xju.edu.cn} \\
% % examples of more authors
% \And
% Chenyang Lyu \\
% Department of Natural Language Processing\\
% Mohamed bin Zayed University of Artificial Intelligence\\
% \texttt{lyuchenyang.dcu@gmail.com} \\
% % \AND
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% }
\author{
Chenyang Lyu$^1$\thanks{Work done while at MBZUAI} \quad
Lecheng Yan$^2$ \quad
Rui Xing$^{1,3}$ \quad
Wenxi Li$^{4,5}$ \\
\textbf{Younes Samih}$^6$ \quad
\textbf{Tianbo Ji}$^7$ \quad
\textbf{Longyue Wang}$^8$ \\
$^1$Mohamed bin Zayed University of Artificial Intelligence \\
$^2$Xinjiang University
$^3$University of Melbourne
$^4$Tsinghua University \\
$^5$Shanghai Jiao Tong University
$^6$IBM Research
$^7$Nantong University
$^8$Alibaba \\
\texttt{lyuchenyang.dcu@gmail.com}
}
\begin{document}
\maketitle
\begin{abstract}
% The capabilities of Large Language Models (LLMs) have significantly evolved, extending from natural language processing to complex tasks like code
% understanding generation. We extend the scope of LLMs' capability to a broader context - where we use LLMs to \textit{execute} code snippets to get the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90\% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting~(IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by 7.22\% in average~(highest 18.96\%) and an absolute average improvement of 3.86\% against CoT prompting~(highest 19.46\%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and completing complex tasks.
The capabilities of Large Language Models~(LLMs) have significantly evolved, extending from natural language processing to complex tasks like code understanding and generation. We expand the scope of LLMs' capabilities to a broader context, using LLMs to \textit{execute} code snippets to obtain the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90\% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting~(IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by an average of 7.22\% (with the highest improvement of 18.96\%) and an absolute average improvement of 3.86\% against CoT prompting (with the highest improvement of 19.46\%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and the completion of complex tasks.
\end{abstract}
\section{Introduction}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\linewidth]{Figures/llms_exe_preprint.pdf}
\caption{An illustration to the use of LLMs for code execution as a proxy for evaluating execution competence as well as the implications for understanding outcomes, and progressing towards real-world causality comprehension.}
\label{fig:intro}
\end{figure}
The rapid advancement of Large Language Models~(LLMs)~\citep{brown2020language_gpt3,openai2023gpt4} has made a transformation of capabilities across diverse domains, ranging from language translation to creative writing~\citep{wang2023document,bang2023multitask_chatgpt_eval,bai2023qwen}. These models, with their remarkable ability to understand and generate human-like text, have found applications that extend well beyond traditional natural language processing tasks such as code understanding and generation~\citep{chen2021evaluating-codex-gpt3,gao2023makes,zhuo2024bigcodebench}. In the area of programming, LLMs have been predominantly utilized for code generation, aiding developers by suggesting code snippets or completing partially written scripts~\citep{wang2024systematicevaluationlargecode}. This utility has significantly enhanced productivity and coding efficiency by providing real-time assistance and reducing the cognitive load on developers~\citep{jiang2024surveylargelanguagemodels}.
Despite these advancements, the exploration of LLMs as code executors remains a less explored area. The ability to not only generate but also execute code opens up a plethora of possibilities, including automated debugging, real-time code validation, and the development of intelligent programming assistants. More importantly, this links to a broader context and higher-level gold of using LLMs to execute and complete complex actions and plans and even for causality understanding in real world~\citep{wang2023codet5opencodelarge,kambhampati2024position}. This paper is pioneering in its approach, as it is the first to systematically examine the feasibility of employing LLMs to execute code directly, providing immediate feedback and results—an evolutionary leap from mere code suggestion to active code execution. An straightforward illustration of this idea is shown in Figure~\ref{fig:intro}, which describes the central concept of using LLMs like OpenAI's o1 to execute code, emphasizing their role in evaluating execution competence, understanding outcomes, and progressing towards real-world causality comprehension. By processing code snippets, LLMs can serve as a benchmark or proxy for assessing their ability to perform tasks and demonstrate understanding of the results and implications of actions. This approach aims to enhance problem-solving capabilities and extend the application scope of LLMs beyond algorithmic tasks, ultimately paving the way for models to grasp the causality of real-world actions.
In our study, we evaluated various LLMs, such as OpenAI's o1~\footnote{\url{https://openai.com/index/openai-o1-system-card/}}, GPT-4o~\footnote{\url{https://openai.com/index/gpt-4o-system-card/}}, GPT-3.5~\citep{ouyangtraining-instructGPT}, DeepSeek~\citep{guo2024deepseekcoderlargelanguagemodel}, and Qwen-Coder~\citep{hui2024qwen25codertechnicalreport}, to assess their performance as code executors. Our experiments reveal that the latest OpenAI o1 model achieves a remarkable execution accuracy of over 90\%, setting a new benchmark in this field. In contrast, other models demonstrate significantly lower accuracy, often falling below the 50\% threshold. This disparity highlights the need for innovative techniques to boost the performance of less accurate models.
To address this challenge for most of the LLMs, we propose an Iterative Instruction Pprompting technique~(IIP) inspired by Chain-of-Thoughts~\citep{wei2022chain_cot} and Tree-of-Thoughts~\citep{yao2024tree_tot} prompting. This method involves feeding code snippets into LLMs line by line, allowing the models to process and execute each segment individually before generating the final output. This approach not only enhances the comprehension and execution accuracy of the models but also results in an average 7.22\% improvement for those with lower baseline performance and an improvement of 3.86\% against Chain-of-Thoughts prompting~\citep{wei2022chain_cot,kojima2022large_cot}. We also analyse the effect of various factors such as coding type, lines of code snippets and the computational complexity to the performance of LLMs. This exploration and analysis could potentially transform current coding practices by enabling more robust and reliable automated code execution.
In this work, we aim to further expand the utility and functionality of LLMs. By demonstrating the potential of LLMs as code executors, we lay the groundwork for future exploration into automated software development, where intelligent programming assistants could revolutionize how code is written, executed, tested, and deployed. This paper seeks to provide fresh insights and ispire further research into leveraging LLMs for more sophisticated programming tasks and complex tasks for action execution and causality comprehension.
\section{LLMs as Code Executors}
In this section, we outline the methodology employed to investigate the capabilities of LLMs as code executors. Our approach is designed to systematically evaluate how effectively LLMs can execute code snippets and return accurate outputs, an interesting application that extends their use beyond traditional code generation tasks. This involves the collection of diverse code snippets and the careful design of prompts that guide the models in executing code. By doing so, we aim to uncover insights into the operational dynamics of LLMs when tasked with direct code execution and to identify strategies that enhance their performance. The following subsections detail the processes of code snippet collection and prompt design, which form the foundation of our experimental framework.
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{Figures/llms_exe_method.pdf}
\caption{Comparison of prompting methods for LLM code execution: vanilla prompting for general guidance, Chain-of-Thought~(CoT) prompting for step-by-step analysis, and iterative instruction prompting for refining outputs based on prior responses.}
\label{fig:method}
\end{figure}
\begin{table}[H]
\centering
\caption{The amount of the main different question types in the dataset, where DP is \textit{Dynamic Programming}.}
\resizebox{\linewidth}{!}{%
\begin{tabular}{lcccccccccc}
\toprule
Source & Array & Greedy & DP & String & Math & Binary Search & Stack & Heap & Recursion & Sorting \\
\midrule
English & 61 & 45 & 25 & 25 & 24 & 22 & 17 & 15 & 15 & 14 \\
Chinese & 60 & 32 & 27 & 19 & 18 & 16 & 15 & 13 & 11 & 10 \\
\bottomrule
\end{tabular}%
}
\end{table}
\subsection{Code Snippets Collection}
% Shall we include an example here? It would be more illustrative.
We collected code snippets from Leetcode~\footnote{\url{https://leetcode.com}}, including 100 examples in both Chinese and English respectively. The platform provides data such as problem description, test cases, standard solutions and problem types corresponding to each problem, and we collect the matching data manually and then analyze it, and finally, the format of each of our metadata is as follows:
\begin{enumerate}
\item Problem Descriptions. For each code snippet, we provide a detailed description of the problem it aims to solve.
\item Input-Output Examples. Each question is accompanied by the corresponding input data and the corresponding expected output, which is used as the LLMs evaluation data
\item Standard solution. We include a standard solution for each problem, describing in detail the idea of solving each problem in order to provide a cross-reference to the collected code snippets.
\item Solution Code. For the standard solution, we have collected the corresponding Python, Java, C, C\texttt{++} four different problem solving code, so as to provide a perfect solution for each code fragment.
\item Problem Type. We have collected the corresponding types of each problem, such as strings, arrays, sorting, math, etc., for targeted analysis and application to understand how LLMs perform in different types of
\item Problem Difficulty and Human Pass Rate. As an objective response to the difficulty of the problem versus the reality of the situation, indicating the percentage of humans who successfully solved each problem on the LeetCode platform.
\item Other. By re-analyzing the existing data, we record the number of lines of code and time complexity of each solution.
\end{enumerate}
In the end, we collected 100 examples in English and 100 examples in Chinese, in total 200 code snippets.
\subsection{Prompting Designation}
%RX: capitalize the term and make them consistent
As illustrated in Figure \ref{fig:method}, three approaches are compared: Vanilla Prompting~(VP), Chain-of-Thought~(CoT) prompting, and our proposed Iterative Instruction Prompting~(IIP). VP serves as a basic interaction model, providing general assistance without specific guidance. In contrast, CoT prompting facilitates a more detailed analysis by encouraging the model to consider the role of each line of code. Finally, IIP makes LLMs to receive code snippets line by line and builds upon previous outputs, allowing the model to refine its predictions based on earlier results. This comparison highlights the progressive enhancement of LLM capabilities in understanding and executing complex code tasks. Below is the vanilla prompting we employed without sophisticated design:
\begin{lstlisting}
~System~:
You are a helpful assistant
*User*:
This is our python code:
@{python_code}@
what is the result/output of this code if the input is @{input_data}@?
\end{lstlisting}
Moreover, in order to clarify the task of LLMs as code executors, while considering improving their performance in this task, we emphasize their role in considering each line of code, ultimately using the following CoT prompt:
\begin{lstlisting}
~System~:
You are a programming expert, good at python code, especially algorithmic
problems, please think step by step about the execution of the code steps,
think about the role of each line of code, get the result.
*User*:
This is our python code:
@{python_code}@
what is the result/output of this code if the input is @{input_data}@?
\end{lstlisting}
Furthermore, we propose an approach that iteratively prompts LLMs with the code snippets line by line, which allows LLMs to think about the output of the previous code line as the input to the next code line, so we use the following prompt:
\begin{lstlisting}
~System~:
You are a programming expert, good at python code, especially algorithmic
problems, please think step by step about the execution of the code steps,
think about the role of each line of code, get the result.
*User*:
begin:
This is our python code:
@{python_code}@
what is the result/output of this code if the input is @{input_data}@?
Now please output the execution process of each line of code in the code,
and output the variable results after each step of the code while executing
each line of code.
process:
My previous analysis process is as follows:
@{response_i}@,
As for the previous analysis process, please help me rethink the output under
this input, and also output the execution process of each line of code in the
code, output the variable results after each step of code while executing each
line of code.
end:
My previous analysis process is as follows:
@{response_n}@,
As for the previous analysis process, please help me rethink the output under
this input, and also output the execution process of each line of code in the
code, output the variable results after each step of code while executing each
line of code, and finally output the most possible result.
\end{lstlisting}
The iterative code execution process involves several steps to compute outputs from a set of code lines and refine these outputs iteratively. Below is a detailed description of this approach:
Let \( C = \{c_1, c_2, \ldots, c_n\} \) be the set of code lines, and let \( I_0 \) represent the initial input data. We aim to compute the output for each line of code and update our inputs iteratively.
The process begins with the generation of the output \( O_i \) for each line \( c_i \) in \( C \). This is achieved by applying a function \( f \), which takes the current line of code and the previous input as arguments:
\[
O_i = f(c_i, I_{i-1})
\]
After the computation, the input for the subsequent line is updated with the output of the current line. Thus, the input update rule is given by:
\[
I_i = O_i
\]
The iterative refinement process involves re-evaluating the outputs. For each iteration \( k \), the outputs are recalculated to enhance accuracy:
\[
O_i^{(k+1)} = f(c_i, I_{i-1}^{(k)})
\]
Finally, after \( n \) iterations, the process converges to a final output \( O_{\text{final}} \), which is represented as:
\[
O_{\text{final}} = O_n^{(k)}
\]
This method ensures that each line's execution is informed by the previous computations, providing a robust framework for code execution with iterative refinement.
\section{Evaluation Results}
In this section, we present the evaluation results of various LLMs on the code snippets we collected.
\subsection{Evaluated LLMs}
\begin{itemize}
%RX: should we use citep{} instead? duplicated citations as in Intro (fixed)
\item \textbf{GPT-3.5:} GPT-3.5~\citep{ouyangtraining-instructGPT} improves on GPT-3~\citep{brown2020language_gpt3} with enhanced language capabilities, supporting zero-shot and few-shot learning. The GPT-3.5 turbo variant balances cost, latency, and quality. The model name used is \textit{gpt-3.5-turbo-0125}.
\item \textbf{GPT-4:} GPT-4~\citep{openai2023gpt4} is known for its advanced language understanding and generation capabilities, which is a further improved version of GPT-3.5. In our experiments, we test \textit{gpt-4-turbo-2024-04-09}.
\item \textbf{GPT-4o:} GPT-4o can work on text and image processing with excellent multimodal capability, supporting 50 languages with enhanced memory capabilities for context retention. The model we used is \textit{gpt-4o-2024-08-06} and \textit{gpt-4o-mini-2024-07-18}.
\item \textbf{OpenAI's o1:} The OpenAI-o1 model has excellent complex reasoning capabilities, especially in complex tasks such as coding and scientific research. The model name used is \textit{o1-preview-2024-09-12}.
\item \textbf{DeepSeek-Coder:} DeepSeek-Coder~\citep{guo2024deepseekcoderlargelanguagemodel}, an open-source Mixture-of-Experts model, matches GPT4-Turbo in code tasks. Pre-trained with an additional 6 trillion tokens, it supports 338 programming languages and extends context length to 128K.
\item \textbf{Qwen-2.5-Coder:} Qwen-2.5-Coder~\citep{hui2024qwen25codertechnicalreport} based on Qwen LLMs~\citep{bai2023qwen} is optimized for coding, supporting 128K tokens and 92 programming languages. Trained on 5.5 trillion tokens, it excels in code generation and reasoning. We also use Qwen-2.5-72B in our experiments.
\end{itemize}
\subsection{Experimental Setup}
Based on the data we collected above, we extract the python code with test cases for each metadata, embed it in the set prompt, set each test case to be asked twice as LLMs input and record the corresponding output; for our IIP approach, we take the last LLMs replies and embed it in the next prompt.
\subsection{Results}
\paragraph{Main Results}
\begin{table}[h]
\centering
\caption{Experimental results for LLMs on CN and EN data, where the highest performance is marked as \textbf{bold} and the second best accuracy is marked with \underline{underscore}.}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccccc}
\toprule
& \textbf{GPT-3.5} & \textbf{GPT-4} & \textbf{GPT-4o} & \textbf{GPT-4o-mini} & \textbf{o1-Preview} & \textbf{Qwen-Coder} & \textbf{Qwen-72B} & \textbf{DeepSeek-Coder} \\
\midrule
\textbf{CN} & 32.2 & 49.7 & \underline{66.7} & 52.3 & \textbf{93.5} & 20.4 & 58.8 & 60.0 \\
\textbf{EN} & 40.0 & 61.3 & \underline{73.8} & 64.3 & \textbf{96.1} & 23.8 & 73.4 & 70.5 \\
\bottomrule
\end{tabular}
}
\label{tab:main_results}
\end{table}
The experimental evaluation was conducted to assess the performance of various LLMs in executing code snippets, sourced both from CN (Chinese) and EN (English) contexts. The LLMs tested include GPT-3.5, GPT-4, GPT-4o, GPT-4o-mini, o1-Preview, Qwen-Coder, Qwen-72B, and DeepSeek-Coder. Each model was tasked with executing code snippets embedded with comments in their respective languages, providing a comprehensive overview of their capabilities across different linguistic and syntactic environments.
The results, as presented in Table \ref{tab:main_results}, highlight several key insights. Notably, the latest OpenAI o1-Preview model consistently outperformed the others, achieving an accuracy of 93.5\% for CN and 96.1\% for EN, suggesting highly excellent ability for this code execution task and a robust capability to handle code execution across diverse linguistic inputs. This indicates the o1's superior ability to process and understand the nuances of code comments and structure, further confirming the capability of solving complex tasks of o1 model. In contrast, models such as GPt-3.5, Qwen-Coder and Qwen-72B demonstrated lower performance, with accuracy around 20\% to 60\%, which substantially lag behind o1's performance. This suggests that these models may lack sufficient training or optimization for code execution tasks, particularly in handling complex code structures or understanding context from comments.
Another interesting observation is the performance disparity between CN and EN across models. While most models showed better performance with English code snippets, the margin varied, indicating potential biases or limitations in handling code semantics when embedded in non-English contexts. This highlights the need for further refinement in multilingual code execution capabilities.
\paragraph{Effect of prompt type}
\begin{table}[h]
\centering
\caption{Comparison of average accuracy for different prompt types (EN and CN), the highest performance and improvements are in bold.}
\begin{tabular}{llccc}
\toprule
\textbf{Model} & \textbf{Source} & \textbf{Vanilla} & \textbf{CoT} & \textbf{IIP} \\
\midrule
\multirow{2}{*}{GPT-3.5} & CN & 27.05 & 32.18 (+5.13) & \textbf{33.92~(+6.87)} \\
& EN & 31.92 & 40.00 (+8.08) & \textbf{40.46~(+8.54)} \\
\midrule
\multirow{2}{*}{Qwen-2.5-Coder} & CN & 18.63 & 20.42~(+1.79) & \textbf{33.70 (+15.07)} \\
& EN & 24.33 & 23.83~(-0.50) & \textbf{43.29~(+18.96)} \\
\midrule
\multirow{2}{*}{Qwen-2.5-72B} & CN & 55.58 & 58.79~(+3.21) & \textbf{63.50~(+7.92)} \\
& EN & 69.45 & 73.38~(+3.93) & \textbf{76.13~(+6.68)} \\
\midrule
\multirow{2}{*}{Deepseek-Coder} & CN & 55.53 & \textbf{60.04~(+4.51)} & 54.03~(-1.50) \\
& EN & 69.75 & \textbf{70.50~(+0.75)} & {64.96~(-4.79)} \\
\bottomrule
\end{tabular}
\label{tab:results_prompt_type}
\end{table}
This experiment examines the impact of different prompting strategies on the performance of various LLMs in executing code snippets. The study evaluates three types of prompts: vanilla, CoT~\citep{wei2022chain_cot,kojima2022large_cot}, and Iterative Instruction Prompting~(IIP), across both EN and CN datasets. The LLMs we used include GPT-3.5, Qwen-2.5-Coder, Qwen-2.5-72B, and Deepseek-Coder.
The Vanilla Prompt serves as the baseline, representing the simplest form of instruction. The CoT prompt involves role-playing as an expert, providing the model with additional context and guidance as well as step-by-step thinking and reasoning. IIP, or iterative prompting, involves breaking down the code snippets line by line to feed into LLMs to improve comprehension and execution. This experimental setup aims to discern how each prompting technique influences model accuracy in code execution tasks.
The results shown in Table~\ref{tab:results_prompt_type} demonstrate varying degrees of effect across models and languages when transitioning from vanilla to more sophisticated prompt types. Notably, the IIP consistently yields the highest accuracy improvements, particularly in the CN dataset except for Deepseek-Coder model (IIP leads to slightly lower performance). For instance, Qwen-2.5-Coder shows a significant accuracy increase from 18.63\% with vanilla prompts to 33.70\% with IIP, highlighting the effectiveness of iterative prompting in enhancing model performance through step-by-step guidance. In the case of GPT-3.5, both CoT and IIP prompts lead to improvements in the CN dataset, with IIP offering a slightly higher boost. However, its performance remains relatively stable in the EN dataset, suggesting that the model may already be optimized for English code snippets, and further prompting variations have limited impact. Interestingly, Deepseek-Coder shows a decrease in accuracy with IIP prompting in both CN and EN datasets. This suggests that while iterative prompting benefits some models, its efficacy may depend on specific LLMs.
\paragraph{Performance across different question types}
\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{Figures/question_type_results_overall.pdf}
\caption{Split of average accuracy of all LLMs across different categories for both EN and CN code.}
\label{fig:question_type_results}
\end{figure}
We further analyze the average performance of LLMs across various selected problem categories~(categories are not disjoint sets) that appeared more than ten times in our dataset of 100 questions. The categories include dynamic programming, array, segment tree, sorting, bit operation, binary search, greedy algorithms, hash table, mathematics, and string manipulation. As shown in Figure~\ref{fig:question_type_results}, the models achieved the highest average accuracy of 0.63 in binary search questions, indicating a strong proficiency in handling structured search algorithms likely due to their deterministic nature. In contrast, bit manipulation and dynamic programming showed lower accuracies of 0.41 and 0.43, respectively, suggesting these areas challenge LLMs possibly due to the complex logic and recursive reasoning required.
Performance in array and string manipulation was moderate, with accuracies of 0.50 and 0.51, indicating that while LLMs handle basic operations, they struggle with more advanced cases. Interestingly, models performed well in mathematics and sorting problems, achieving accuracies of 0.59 and 0.58, which reflects their ability to leverage algorithmic thinking and numerical computation.
\paragraph{Relationship between model accuracy and human pass rate}
\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth]{Figures/average_model_accuracy_vs_pass_rate_new.pdf}
\caption{Relationship between human pass rates and model accuracy for EN~(left) and CN~(right) code snippets, the Spearman’s Correlation for EN and CN data are 0.25 and 0.17 respectively.}
\label{fig:human_pass_rate}
\end{figure}
This section analyses the relationship between the average accuracy of all evaluated LLMs in Table~\ref{tab:main_results} and corresponding human pass rates of each coding question. The fit results are shown in Figure~\ref{fig:human_pass_rate}. Both EN and CN datasets show a positive correlation, indicating that tasks easier for humans generally yield higher model accuracy. The fit lines in the plots suggest that LLMs are more adept at solving problems with higher human pass rates, likely due to shared cognitive processes or data patterns. Notably, the EN dataset displays a steeper correlation, possibly due to more extensive training data or linguistic characteristics favoring English comprehension for the comments in code snippets.
\paragraph{Effect of Computational Complexity}
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\linewidth]{Figures/average_accuracy_of_time_complexity.pdf}
\caption{Average accuracy on CN and EN code snippets divided by corresponding computational complexity.}
\label{fig:results_complexity}
\end{figure}
This section evaluates LLM performance on code snippets by analyzing average accuracy relative to computational complexity across CN and EN datasets. The complexities considered include \(O(n)\), \(O(n \log n)\), \(O(n^2)\), and others as shown in Figure~\ref{fig:results_complexity}, providing insights into model capabilities across varying algorithmic difficulties.
The resutls are shown in Figure~\ref{fig:results_complexity}. For CN questions, LLMs show strong performance in \(O(n \log n)\) tasks with an accuracy of 0.73, indicating proficiency in moderately complex problems like sorting. However, accuracy drops to 0.27 for \(O(n^3)\) tasks, highlighting challenges with highly complex operations. In EN questions, models achieve the highest accuracy of 0.84 for \(O(2^n \cdot n)\) tasks, suggesting strong capabilities in handling exponential growth problems Simpler complexities like \(O(n)\) yield moderate accuracies in both datasets, reflecting efficiency in straightforward tasks but indicating room for improvement in handling nested operations.
\paragraph{Effect of lines of code snippets}
\begin{figure}
\centering
\includegraphics[width=1.0\linewidth]{Figures/average_score_en_cn_per_code_line_count_new.pdf}
\caption{Overall regression analysis illustrating the impact of code snippet length on model accuracy for CN and EN datasets, the Spearman’s Correlation for EN and CN are -0.32 and -0.53 respectively.}
\label{fig:code_line}
\end{figure}
We also the relationship between LLM accuracy and code snippet length in CN and EN datasets. As shown in Figure~\ref{fig:code_line}, quadratic regression analysis reveals a negative correlation between line count and accuracy for both datasets, indicating that longer code snippets generally reduce model performance. The CN dataset shows a steeper decline, suggesting greater challenges in handling complexity compared to the EN dataset, where the impact is less pronounced. These findings underscore the need to enhance LLMs' ability to manage code complexity, particularly in multilingual contexts, by developing more sophisticated code understanding mechanisms to improve robustness against variability in code length.
\subsection{Case Analysis}
We further analyze the performance of each model in our dataset. For example, question 91 of the English dataset: \textit{Booking Concert Tickets in Groups}. As a complex search problem, its standard code answer is more than one hundred lines. In the experiment, all LLM except OpenAI's o1 could not correctly handle the problem conditions and search and assign them, which proved o1's superior ability in dealing with complex problems.
\begin{lstlisting}[language=Python]
class Node:
def __init__(self, start, end):
self.s = start
self.e = end
self.left = None
self.right = None
self.total = 0 # for range sum query
self.mx = 0 # for range max query
class SegTree:
def __init__(self, start, end, val):
def build(l, r):
if l > r:
return None
if l == r:
node = Node(l, r)
node.total = val
node.mx = val
return node
node = Node(l, r)
m = (l + r) // 2
node.left = build(l, m)
node.right = build(m+1, r)
node.mx = max(node.left.mx, node.right.mx)
node.total = node.left.total + node.right.total
return node
self.root = build(start, end)
# update the total remain seats and the max remain seats for each node (range) in the segment tree
def update(self, index, val):
def updateHelper(node):
if node.s == node.e == index:
node.total -= val
node.mx -= val
return
m = (node.s + node.e) // 2
if index <= m:
updateHelper(node.left)
elif index > m:
updateHelper(node.right)
node.mx = max(node.left.mx, node.right.mx)
node.total = node.left.total + node.right.total
return
updateHelper(self.root)
def maxQuery(self, k, maxRow, seats):
def queryHelper(node):
if node.s == node.e:
# check if the row number is less than maxRow and the number of remains seats is greater or equal than k
if node.e > maxRow or node.total < k:
return []
if node.e <= maxRow and node.total >= k:
return [node.e, seats - node.total]
# we want to greedily search the left subtree
........................... more lines of code
\end{lstlisting}
Meanwhile, for question 85 in the Chinese dataset: \textit{Subarrays Distinct Element Sum of Squares II}. It is a combination of Segment Tree, Array, and Math, which contains numerical computation and remainder operation for large integers. Most LLMs fail in the remainder operation, while o1 completes all the steps perfectly, proving that o1 is better than other models in numerical processing and operations.
\begin{lstlisting}[language=Python]
class Solution:
def sumCounts(self, nums: List[int]) -> int:
n = len(nums)
sum = [0] * (n * 4)
todo = [0] * (n * 4)
def do(o: int, l: int, r: int, add: int) -> None:
sum[o] += add * (r - l + 1)
todo[o] += add
# o=1 [l,r] 1<=l<=r<=n
def query_and_add1(o: int, l: int, r: int, L: int, R: int) -> int:
if L <= l and r <= R:
res = sum[o]
do(o, l, r, 1)
return res
m = (l + r) // 2
add = todo[o]
if add:
do(o * 2, l, m, add)
do(o * 2 + 1, m + 1, r, add)
todo[o] = 0
res = 0
if L <= m: res += query_and_add1(o * 2, l, m, L, R)
if m < R: res += query_and_add1(o * 2 + 1, m + 1, r, L, R)
sum[o] = sum[o * 2] + sum[o * 2 + 1]
return res
ans = s = 0
last = {}
for i, x in enumerate(nums, 1):
j = last.get(x, 0)
s += query_and_add1(1, 1, n, j + 1, i) * 2 + i - j
ans += s
last[x] = i
return ans % 1_000_000_007
........................... more lines of code
\end{lstlisting}
In addition, for question 7 of the Chinese dataset: \textit{Alternating Groups III}.
The standard answer to this question lacks the definition of the tree-like structure given in the question. While the rest of the LLMs refused to answer or answered incorrectly due to the lack of conditions, o1 successfully deduced the conditions and answered correctly, which further proved its strong generalization ability.
\begin{lstlisting}[language=Python]
from sortedcontainers import SortedList
class FenwickTree:
def __init__(self, n: int):
self.t = [[0, 0] for _ in range(n + 1)]
def update(self, size: int, op: int) -> None:
i = len(self.t) - size
while i < len(self.t):
self.t[i][0] += op
self.t[i][1] += op * size
i += i & -i
def query(self, size: int) -> (int, int):
cnt = s = 0
i = len(self.t) - size
while i > 0:
cnt += self.t[i][0]
s += self.t[i][1]
i &= i - 1
return cnt, s
class Solution:
def numberOfAlternatingGroups(self, a: List[int], queries: List[List[int]]) -> List[int]:
n = len(a)
sl = SortedList()
t = FenwickTree(n)
def update(i: int, op: int) -> None:
idx = sl.bisect_left(i)
pre = sl[idx - 1]
nxt = sl[idx % len(sl)]
t.update((nxt - pre - 1) % n + 1, -op)
t.update((i - pre) % n, op)
t.update((nxt - i) % n, op)
def add(i: int) -> None:
if not sl:
t.update(n, 1)
else:
update(i, 1)
sl.add(i)
def remove(i: int) -> None:
sl.remove(i)
if not sl:
t.update(n, -1)
else:
update(i, -1)
for i, c in enumerate(a):
if c == a[(i + 1) % n]:
add(i)
ans = []
for q in queries:
if q[0] == 1:
if not sl:
ans.append(n)
else:
cnt, s = t.query(q[1])
ans.append(s - cnt * (q[1] - 1))
else:
i, c = q[1], q[2]
........................... more lines of code
\end{lstlisting}
Below is the solution given by OpenAI-o1, which proved its excellent capability for handling such complex reasoning tasks:
\begin{lstlisting}
First, we need to understand what the code does.
1. Initialization:
- For a given array a = [0, 1, 1, 0, 1], we traverse each position i.
If a[i] == a[(i + 1) % n], we consider this position i as an end position
and add it to SortedList sl.
- In the initialization, the condition is only satisfied when i = 1 since
a[1] == a[2] (1 == 1), so we add 1 to sl.
- We call add(1), and since sl is empty, we call t.update(5, 1)
to update the tree.
2. Processing Queries:
- The first query is [2, 1, 0], which indicates changing the value at
position 1 to 0.
- Before modification, we check and remove affected end positions.
a[1] was 1, and a[2] was also 1, so 1 was an end position.
We call remove(1) and update the tree.
- After modification, we update a[1] to 0. Then, we check for new
possible end positions. Now a[0] == a[1] (0 == 0), so 0 becomes a
new end position. We call add(0) and update the tree.
- The second query is [1, 4], which queries the number of subarrays
satisfying the condition.
- We call t.query(4), obtaining cnt = 1 (the number of satisfying segments),
s = 5 (the total length of these segments).
- The final result is s - cnt * (size - 1) = 5 - 1 * (4 - 1) = 2.
3. Result:
- The returned result is [2].
\end{lstlisting}
\section{Conclusion}
This study evaluates the performance of LLMs in executing code snippets, revealing key insights across different prompt types, problem categories, and computational complexities. Iterative prompting significantly enhances accuracy, particularly in CN datasets, emphasizing the value of detailed guidance. LLMs performs better in moderate complexity tasks but face challenges with dynamic programming and lengthy code snippets, especially in CN. The correlation between code length and accuracy suggests a need for improved handling of complex, extended tasks.
Future work will focus on extending evaluations to a broader range of coding problems beyond algorithm-specific tasks and incorporating additional programming languages. This will further enhance our understanding of LLM capabilities and inform the development of more robust models.
\bibliography{main}
% \section*{References}
% References follow the acknowledgments in the camera-ready paper. Use unnumbered first-level heading for
% the references. Any choice of citation style is acceptable as long as you are
% consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
% when listing the references.
% Note that the Reference section does not count towards the page limit.
% \medskip
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \appendix
% \section{Appendix / supplemental material}
% Optionally include supplemental material (complete proofs, additional experiments and plots) in appendix.
% All such materials \textbf{SHOULD be included in the main submission.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
% \section*{NeurIPS Paper Checklist}
% %%% BEGIN INSTRUCTIONS %%%
% The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
% limit.
% Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% \begin{itemize}
% \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% \item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% % \item {\bf The papers not including the checklist will be desk rejected.}
% \end{itemize}
% {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% IMPORTANT, please:
% \begin{itemize}
% \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS paper checklist"},
% \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% \end{itemize}
% %%% END INSTRUCTIONS %%%
% \begin{enumerate}
% \item {\bf Claims}
% \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the abstract and introduction do not include the claims made in the paper.
% \item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
% \item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
% \item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
% \end{itemize}
% \item {\bf Limitations}
% \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
% \item The authors are encouraged to create a separate "Limitations" section in their paper.
% \item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
% \item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
% \item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
% \item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
% \item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
% \item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
% \end{itemize}
% \item {\bf Theory Assumptions and Proofs}
% \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include theoretical results.
% \item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
% \item All assumptions should be clearly stated or referenced in the statement of any theorems.
% \item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
% \item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
% \item Theorems and Lemmas that the proof relies upon should be properly referenced.
% \end{itemize}
% \item {\bf Experimental Result Reproducibility}
% \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
% \item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
% \item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
% \item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
% \begin{enumerate}
% \item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
% \item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
% \item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
% \item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
% \end{enumerate}
% \end{itemize}
% \item {\bf Open access to data and code}
% \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that paper does not include experiments requiring code.
% \item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
% \item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
% \item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
% \item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
% \item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
% \end{itemize}
% \item {\bf Experimental Setting/Details}
% \item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
% \item The full details can be provided either with the code, in appendix, or as supplemental material.
% \end{itemize}
% \item {\bf Experiment Statistical Significance}
% \item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
% \item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
% \item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
% \item The assumptions made should be given (e.g., Normally distributed errors).
% \item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
% \item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
% \item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
% \item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
% \end{itemize}
% \item {\bf Experiments Compute Resources}
% \item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
% \item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
% \item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
% \end{itemize}
% \item {\bf Code Of Ethics}
% \item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
% \item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
% \item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
% \end{itemize}
% \item {\bf Broader Impacts}
% \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that there is no societal impact of the work performed.
% \item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
% \item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
% \item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
% \item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
% \item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
% \end{itemize}
% \item {\bf Safeguards}
% \item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper poses no such risks.
% \item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
% \item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
% \item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
% \end{itemize}
% \item {\bf Licenses for existing assets}
% \item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not use existing assets.
% \item The authors should citep the original paper that produced the code package or dataset.
% \item The authors should state which version of the asset is used and, if possible, include a URL.
% \item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
% \item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
% \item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
% \item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
% \item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
% \end{itemize}
% \item {\bf New Assets}
% \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not release new assets.
% \item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
% \item The paper should discuss whether and how consent was obtained from people whose asset is used.
% \item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
% \end{itemize}
% \item {\bf Crowdsourcing and Research with Human Subjects}
% \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
% \item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
% \end{itemize}
% \item {\bf Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects}
% \item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
% \item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
% \item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
% \end{itemize}
% \end{enumerate}
\end{document}
\begin{thebibliography}{18}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bai et~al.(2023)Bai, Bai, Yang, Wang, Tan, Wang, Lin, Zhou, and Zhou]{bai2023qwen}
J.~Bai, S.~Bai, S.~Yang, S.~Wang, S.~Tan, P.~Wang, J.~Lin, C.~Zhou, and J.~Zhou.
\newblock Qwen-vl: A frontier large vision-language model with versatile abilities.
\newblock \emph{arXiv preprint arXiv:2308.12966}, 2023.
\bibitem[Bang et~al.(2023)Bang, Cahyawijaya, Lee, Dai, Su, Wilie, Lovenia, Ji, Yu, Chung, et~al.]{bang2023multitask_chatgpt_eval}
Y.~Bang, S.~Cahyawijaya, N.~Lee, W.~Dai, D.~Su, B.~Wilie, H.~Lovenia, Z.~Ji, T.~Yu, W.~Chung, et~al.
\newblock A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
\newblock \emph{arXiv preprint arXiv:2302.04023}, 2023.
\bibitem[Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei]{brown2020language_gpt3}
T.~Brown, B.~Mann, N.~Ryder, M.~Subbiah, J.~D. Kaplan, P.~Dhariwal, A.~Neelakantan, P.~Shyam, G.~Sastry, A.~Askell, S.~Agarwal, A.~Herbert-Voss, G.~Krueger, T.~Henighan, R.~Child, A.~Ramesh, D.~Ziegler, J.~Wu, C.~Winter, C.~Hesse, M.~Chen, E.~Sigler, M.~Litwin, S.~Gray, B.~Chess, J.~Clark, C.~Berner, S.~McCandlish, A.~Radford, I.~Sutskever, and D.~Amodei.
\newblock Language models are few-shot learners.
\newblock In H.~Larochelle, M.~Ranzato, R.~Hadsell, M.~F. Balcan, and H.~Lin, editors, \emph{Advances in Neural Information Processing Systems}, volume~33, pages 1877--1901. Curran Associates, Inc., 2020.
\newblock URL \url{https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf}.
\bibitem[Chen et~al.(2021)Chen, Tworek, Jun, Yuan, Pinto, Kaplan, Edwards, Burda, Joseph, Brockman, et~al.]{chen2021evaluating-codex-gpt3}
M.~Chen, J.~Tworek, H.~Jun, Q.~Yuan, H.~P. d.~O. Pinto, J.~Kaplan, H.~Edwards, Y.~Burda, N.~Joseph, G.~Brockman, et~al.
\newblock Evaluating large language models trained on code.
\newblock \emph{arXiv preprint arXiv:2107.03374}, 2021.
\bibitem[Gao et~al.(2023)Gao, Wen, Gao, Wang, Zhang, and Lyu]{gao2023makes}
S.~Gao, X.-C. Wen, C.~Gao, W.~Wang, H.~Zhang, and M.~R. Lyu.
\newblock What makes good in-context demonstrations for code intelligence tasks with llms?
\newblock In \emph{2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)}, pages 761--773. IEEE, 2023.
\bibitem[Guo et~al.(2024)Guo, Zhu, Yang, Xie, Dong, Zhang, Chen, Bi, Wu, Li, Luo, Xiong, and Liang]{guo2024deepseekcoderlargelanguagemodel}
D.~Guo, Q.~Zhu, D.~Yang, Z.~Xie, K.~Dong, W.~Zhang, G.~Chen, X.~Bi, Y.~Wu, Y.~K. Li, F.~Luo, Y.~Xiong, and W.~Liang.
\newblock Deepseek-coder: When the large language model meets programming -- the rise of code intelligence, 2024.
\newblock URL \url{https://arxiv.org/abs/2401.14196}.
\bibitem[Hui et~al.(2024)Hui, Yang, Cui, Yang, Liu, Zhang, Liu, Zhang, Yu, Dang, Yang, Men, Huang, Ren, Ren, Zhou, and Lin]{hui2024qwen25codertechnicalreport}
B.~Hui, J.~Yang, Z.~Cui, J.~Yang, D.~Liu, L.~Zhang, T.~Liu, J.~Zhang, B.~Yu, K.~Dang, A.~Yang, R.~Men, F.~Huang, X.~Ren, X.~Ren, J.~Zhou, and J.~Lin.
\newblock Qwen2.5-coder technical report, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.12186}.
\bibitem[Jiang et~al.(2024)Jiang, Wang, Shen, Kim, and Kim]{jiang2024surveylargelanguagemodels}
J.~Jiang, F.~Wang, J.~Shen, S.~Kim, and S.~Kim.
\newblock A survey on large language models for code generation, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.00515}.
\bibitem[Kambhampati et~al.(2024)Kambhampati, Valmeekam, Guan, Verma, Stechly, Bhambri, Saldyt, and Murthy]{kambhampati2024position}
S.~Kambhampati, K.~Valmeekam, L.~Guan, M.~Verma, K.~Stechly, S.~Bhambri, L.~P. Saldyt, and A.~B. Murthy.
\newblock Position: {LLM}s can{\textquoteright}t plan, but can help planning in {LLM}-modulo frameworks.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\newblock URL \url{https://openreview.net/forum?id=Th8JPEmH4z}.
\bibitem[Kojima et~al.(2022)Kojima, Gu, Reid, Matsuo, and Iwasawa]{kojima2022large_cot}
T.~Kojima, S.~S. Gu, M.~Reid, Y.~Matsuo, and Y.~Iwasawa.
\newblock Large language models are zero-shot reasoners.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 22199--22213, 2022.
\bibitem[OpenAI(2023)]{openai2023gpt4}
OpenAI.
\newblock {GPT-4 Technical Report}.
\newblock \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem[Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Gray, et~al.]{ouyangtraining-instructGPT}
L.~Ouyang, J.~Wu, X.~Jiang, D.~Almeida, C.~Wainwright, P.~Mishkin, C.~Zhang, S.~Agarwal, K.~Slama, A.~Gray, et~al.
\newblock Training language models to follow instructions with human feedback.
\newblock In \emph{Advances in Neural Information Processing Systems}, 2022.
\bibitem[Wang et~al.(2024)Wang, Gao, Gao, Wang, Chong, Gao, and Lyu]{wang2024systematicevaluationlargecode}
C.~Wang, S.~Gao, C.~Gao, W.~Wang, C.~Y. Chong, S.~Gao, and M.~R. Lyu.
\newblock A systematic evaluation of large code models in api suggestion: When, which, and how, 2024.
\newblock URL \url{https://arxiv.org/abs/2409.13178}.
\bibitem[Wang et~al.(2023{\natexlab{a}})Wang, Lyu, Ji, Zhang, Yu, Shi, and Tu]{wang2023document}
L.~Wang, C.~Lyu, T.~Ji, Z.~Zhang, D.~Yu, S.~Shi, and Z.~Tu.
\newblock Document-level machine translation with large language models.
\newblock \emph{arXiv preprint arXiv:2304.02210}, 2023{\natexlab{a}}.
\bibitem[Wang et~al.(2023{\natexlab{b}})Wang, Le, Gotmare, Bui, Li, and Hoi]{wang2023codet5opencodelarge}
Y.~Wang, H.~Le, A.~D. Gotmare, N.~D.~Q. Bui, J.~Li, and S.~C.~H. Hoi.
\newblock Codet5+: Open code large language models for code understanding and generation, 2023{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2305.07922}.
\bibitem[Wei et~al.(2022)Wei, Wang, Schuurmans, Bosma, Xia, Chi, Le, Zhou, et~al.]{wei2022chain_cot}
J.~Wei, X.~Wang, D.~Schuurmans, M.~Bosma, F.~Xia, E.~Chi, Q.~V. Le, D.~Zhou, et~al.
\newblock Chain-of-thought prompting elicits reasoning in large language models.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 24824--24837, 2022.
\bibitem[Yao et~al.(2024)Yao, Yu, Zhao, Shafran, Griffiths, Cao, and Narasimhan]{yao2024tree_tot}
S.~Yao, D.~Yu, J.~Zhao, I.~Shafran, T.~Griffiths, Y.~Cao, and K.~Narasimhan.
\newblock Tree of thoughts: Deliberate problem solving with large language models.
\newblock \emph{Advances in Neural Information Processing Systems}, 36, 2024.
\bibitem[Zhuo et~al.(2024)Zhuo, Vu, Chim, Hu, Yu, Widyasari, Yusuf, Zhan, He, Paul, et~al.]{zhuo2024bigcodebench}
T.~Y. Zhuo, M.~C. Vu, J.~Chim, H.~Hu, W.~Yu, R.~Widyasari, I.~N.~B. Yusuf, H.~Zhan, J.~He, I.~Paul, et~al.
\newblock Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.
\newblock \emph{arXiv preprint arXiv:2406.15877}, 2024.
\end{thebibliography}