Assessing LLM code generation quality through path planning tasks
Extracted source text · arXiv 2504.21276
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
\usepackage[nonatbib, preprint]{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{booktabs}
\usepackage{array} % Make sure this is in your preamble
\usepackage{tabularx}
\usepackage{makecell}
\usepackage[pdftex]{graphicx}
\bibliographystyle{acm}
\title{Assessing LLM code generation quality through path planning tasks}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Wanyi Chen \\
Duke University\\
Durham, NC 27705 \\
\texttt{wc151@duke.edu} \\
% examples of more authors
\And
Meng-wen Su \\
George Mason University \\
Fairfax, VA 22030 \\
\texttt{msu5@gmu.edu} \\
\And
Mary Cummings \\
George Mason University \\
Fairfax, VA 22030 \\
\texttt{cummings@gmu.edu} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
As LLM-generated code grows in popularity, more evaluation is needed to assess the risks of using such tools, especially for safety-critical applications such as path planning. Existing coding benchmarks are insufficient as they do not reflect the context and complexity of safety-critical applications. To this end, we assessed six LLMs' abilities to generate the code for three different path-planning algorithms and tested them on three maps of various difficulties. Our results suggest that LLM-generated code presents serious hazards for path planning applications and should not be applied in safety-critical contexts without rigorous testing.
\end{abstract}
\section{Introduction}
%Automated code generation is a long-standing topic of interest in computer science and software engineering, as the ability to automatically generate code based on natural language descriptions is highly desirable \cite{wang2023review}. The advancements of Large Language Models (LLMs) provided new possibilities to achieve this goal. Indeed, LLMs have gained significant traction in code generation and have achieved some success. However, most of the current evaluation of LLM-generated code utilizess programming competition questions. Such questions do not reflect the context and complexity of real-world programming tasks.
As code generated by Large Language Models (LLMs) is increasingly adopted in many domains, evaluation is needed to assess the possible risks and problems of using LLM-generated code, especially in safety-critical contexts like path planning. Path-planning algorithms play a crucial role in aircraft and vehicle navigation, especially in autonomous systems such as drones and self-driving cars. For these systems, errors in planning could lead to safety hazards and even deaths.
To this end, we assessed six LLMs's ability to generate code for the task of generating three path-planning algorithms: Dijkstra's algorithm, the Voronoi algorithm, and the Rapidly-exploring Random Tree (RRT) algorithm. We evaluated the generated code as well as how effective the LLMs were in optimizing for the stated objectives. We then discuss the implications of the evaluation results, and hazards that result from LLM-generated code used in safety-critical contexts.
\section{Background and related work}
Most code generation evaluation benchmarks have problem sets similar to programming competitions. The problem setup typically includes a function signature and a docstring, which is natural language text that describes the purpose of the function. The LLMs are expected to automatically complete the function, and the correctness of the generated code is assessed through pre-written unit tests. HumanEval \cite{chen2021evaluating} is a widely used example of this type of benchmark. It includes 164 handwritten programming problems and an average of 7.7 unit tests per problem \cite{chen2021evaluating}. A survey by Jiang et al. \cite{jiang2024survey} provides a more comprehensive list of programming competition-style benchmarks.
However, real-world software engineering tasks are often more complex than the problems addressed in the programming competition-style benchmarks. To evaluate LLMs' ability to complete real-world software engineering tasks, some researchers proposed task-based benchmarks, such as SWE-bench \cite{jimenez2023swe}. SWE-bench utilizes 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories \cite{jimenez2023swe}. Given a codebase and a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue \cite{jimenez2023swe}. However, other researchers found that some solutions were provided directly in the issue report or comments and therefore doubted whether SWE-bench could meaningfully assess LLMs' software engineering abilities \cite{aleithan2024swe}.
This experiment differed from programming competition-style benchmarks in that we did not provide function signatures. The LLMs were expected to determine the function's input and output and generate appropriate function signatures based on natural language descriptions. Furthermore, whereas most programming competition tasks focus on implementing a single function, in our experiment, the LLMs needed to generate multiple functions. The LLMs determined the number of helper functions as well as how the main function should call the helper functions. Our experiment better captured the context and complexity of real-world software engineering tasks, and further avoided the problem present in SWE-bench: no solutions were offered in our prompts.
Previous evaluations of LLM-generated code found several patterns. First, LLM code generation is non-deterministic \cite{ouyang2023llm}. LLMs could return very different code for the same prompt \cite{ouyang2023llm}. In our experiment, we accounted for the non-determinism into consideration by allowing the LLMs to regenerate responses for up to five times given the same prompt. Second, LLM-generated code is error prone. Tambon et al. identified ten distinctive bug patterns in LLM-generated code \cite{tambon2025bugs}, and Liu et al. identified five categories of hallucinations \cite{liu2024exploring}. Their evaluations were primarily based on programming competition-style questions such as HumanEval \cite{liu2024exploring, tambon2025bugs}. In our experiments, we continue to explore and evaluate patterns in LLM-generated code in safety-critical contexts.
\section{Method}
To assess LLM code generation quality, we tasked six LLMs to generate three path planning algorithms. Path planning is essentially an optimization task in that such algorithms can find the best path for the shortest distance, lowest runtime, maximum clearance around obstacles, etc. Our experiments aim to answer two research questions (RQs):
1) RQ 1: If no optimization objectives are specified, how effective is the LLM code, and what are the default LLM objectives, and 2) RQ 2: If an optimization objective is specified, how effective is the LLM code in optimizing for the stated objective?
\subsection{LLMs and Algorithms}
\iffalse
\begin{table}[h]
\centering
\caption{Large Language Models Tested}
\begin{tabular}{l c c}
\toprule
\textbf{Model} & \textbf{Developer} & \textbf{Parameters (approx.)}\\
\midrule
Mistral 7B & Mistral AI & 7.3 billion\\
Gemini & Google DeepMind & 70 billion\\
DeepSeek & DeepSeek & 671 billion \\
GPT-4o & OpenAI & 1.8 trillion \\
Copilot & OpenAI & 1.8 trillion \\
Grok 3 & xAI & 2.7 trillion\\
\bottomrule
\end{tabular}
\label{table:llm_comparison}
\end{table}
\fi
We tested six LLMs of various sizes: Mistral 7B (developed by Mistral AI, 7.3 billion parameters), Gemini (developed by Google DeepMind, 70 billion parameters), DeepSeek (developed by DeepSeek, 671 billion parameters), GPT-4o (developed by OpenAI, 1.8 trillion parameters), Copilot (developed by OpenAI, 1.8 trillion parameters, fine-tuned for code generation), and Grok 3 (developed by xAI, 2.7 trillion parameters). We prompted each LLM to generate the following path-planning algorithms, which are listed in order of increasing complexity:
\begin{itemize}
\item \textbf{Dijkstra's algorithm}: Dijkstra's algorithm is a fundamental method for finding the shortest paths from a source node to all other nodes in a graph. It uses a priority queue to track the minimum cost to reach each node, initially set to infinity, except for the source node which starts at zero. The algorithm selects the node with the lowest cost, updates its neighbors' costs, and adjusts the priority queue accordingly. This continues until the shortest paths to all nodes are determined, utilizing a systematic and greedy approach \cite{DIJKSTRA1959}.
\item \textbf{Voronoi algorithm}: The Voronoi diagram can be used to optimize path planning by maximizing the distance from obstacles. It partitions space into regions, forming a graph with edges that are equidistant from multiple obstacles. The Voronoi algorithm constructs a Voronoi diagram from the obstacles, removes edges that intersect with obstacles, and connects the start and goal to the nearest vertices. A shortest path algorithm, such as A\* or Dijkstra’s algorithm, is then used to find a path that travels along the edges of the Voronoi diagram. The Voronoi algorithm is especially desirable in cases where maintaining maximum clearance from obstacles is crucial \cite{Voronoi}.
\item \textbf{Rapidly-exploring random tree (RRT)}: RRT is a key method in path finding and motion planning for robotics and autonomous vehicles. It can efficiently explore high-dimensional spaces by building a space-filling tree randomly. Starting from an initial configuration, the tree expands toward randomly chosen points while avoiding collisions and meeting movement constraints. This process continues until a path to the goal is found or a maximum iteration limit is reached. RRT is favored for its speed and effectiveness in complex environments, unlike traditional grid-based algorithms that struggle in these situations \cite{RRT}.
\end{itemize}
\subsection{Experimental setup}
\begin{figure}
\centering
\includegraphics[width=1.\linewidth]{maps.png}
\caption{Increasingly difficult path planning maps}
\label{fig:maps}
\end{figure}
Each LLM-generated algorithm was manually tested on three maps of different difficulty levels: easy, medium, and hard, as shown in Figure \ref{fig:maps} and proposed in previous robotic path planning research \cite{maps}. The tasks described to the LLMs using the following prompts for RQ 1 and RQ 2:
\begin{itemize}
\item \textbf{RQ 1: Any-path prompt -} \textit{``Help me write a Python function code to \textbf{plan a path} on a $20 \times 20$ grid using \_\_\_\_ algorithm: \\
1. The input of the function is an array of obstacles and the starting and ending points. The path can start and end at any (x, y) coordinates of the user's choice \\
2. The path can include diagonal movement in addition to horizontal and vertical directions. \\
3. The grid has rectangular obstacles defined by the coordinates of the bottom-left corner and the top-right corner. For example, “((3, 3), (5, 5))” defines an obstacle \\
4. The function should print the grid, the obstacles, and the planned path \\
5. You are allowed to write helper functions. \\
6. Display the grid in an interactive plot, with x and y axes ranging from (0, 0) to (20, 20) and gridlines at every integer interval. Highlight the starting point in green and the endpoint in red. Clearly illustrate the obstacles and the planned path on the grid. \\
7. Obstacles and the planned path should not touch; there should be a 0.5 gap between the planned path and the obstacles. \\
8. If the goal is not reached, show the incomplete path up to the furthest point.``}
\item \textbf{RQ 2: Shortest-path prompt -} The shortest-path prompt was the same as the any-path prompt except that in the first sentence, the phrase ``plan a path" was replaced with ``plan the shortest path."
\end{itemize}
The blank in the first sentence of the prompt was replaced with ``the Dijkstra's," ``the Voronoi," or ``the Rapidly-exploring Random Tree." We submitted the prompt to the LLMs and manually ran the generated code. If the code compiled and ran successfully, we then verified whether the code generated a viable path on at least one map. A viable path means that the path has the correct start and end points and avoids all the obstacles. We also verified that the LLMs attempted to implement the correct type of algorithm. For example, if the prompt specified ``Voronoi" but the LLM attempted to implement the Dijkstra's algorithm, the output was considered non-viable.
If the output was non-viable, we regenerated the code by reattempting the same prompt in a new window. Each LLM was permitted up to five such attempts. If the output was still not viable after five attempts, we selected the least flawed or closest to the expected result of the five for further analysis. Figure \ref{fig:diagram} explains the process of the experiment. Once the trials were completed, we inspected the generated code including default hyperparameter choices and other implementation choices. We also assessed the following aspects of generated code:
\begin{itemize}
\item \textit{Syntax correctness}: We checked whether the generated code contained any syntax errors, such as missing an import statement, that would prevent the code from compiling.
\item \textit{Algorithm correctness }: We assessed whether the generated algorithm could find correct paths on all three maps. ``Correct" is defined as having the correct start and end points, not running through obstacles, and staying within the boundaries of the $20 \times 20$ grid.
\item \textit{Satisfying prompt requirements}: We evaluated whether the LLM-generated code satisfied all specified requirements: whether the paths had a minimum of 0.5 distance to all obstacles and whether the grid, obstacles, and complete paths were correctly plotted. For RRT, if a path failed to reach the goal, we checked whether an incomplete path was plotted. For the Dijkstra's shortest-path prompt, we verified whether the planned paths were the shortest.
\item \textit{Code Style}: For all LLM-generated code, we evaluated the code style using Pylint, an automatic code analyzer that checks for compliance with the PEP 8 Python style guide \cite{PyPI}.
\end{itemize}
\begin{figure}
\centering
\includegraphics[width=1\linewidth]{code_generation_process.png}
\caption{Experiment process diagram}
\label{fig:diagram}
\end{figure}
\section{Results}
%Regarding RQ1, when the prompts were underspecified, the LLMs chose a wide range of default objectives and default hyperparameters. They also made different implementation choices, resulting in different runtimes and different shapes of the paths. For RQ2, the LLMs had varying degrees of success in implementing the specified algorithms, ranging from bug-free implementation of the Dijkstra’s algorithm to completely wrong implementation of the Voronoi algorithm.
The following subsections break down the results by the three algorithms.
\subsection{Dijkstra's algorithm}
%Given the same prompt, the LLMs implemented very different algorithms. They chose different default objectives for Dijkstra's algorithm and different default hyperparameter values for RRT. Some LLMs also made implementation choices that caused the RRT algorithm to behave differently.
In response to the RQ1 any-path prompt, the LLMs chose different default objectives, summarized in Table \ref{tab:dijkstras_default_objectives}. Gemini and Mistral generated a variation of Dijkstra's algorithm that optimized for runtime instead of the shortest path. They used the Manhattan distance as a heuristic to estimate the cost from the current node to the endpoint, which decreased the number of nodes assessed and improved overall speed. However, since heuristics provide estimates rather than precise cost values, their approach did not guarantee the shortest route.
In contrast, GPT, Copilot, DeepSeek, and Grok attempted to implement the classical Dijkstra's algorithm that sought guaranteed shortest-path solutions through exhaustive search. They did not employ heuristics. However, while DeepSeek attempted to find the shortest path, it implemented the wrong cost calculation. It assumed a uniform cost moving from one node to every neighbor node, but the cost of moving horizontally or vertically is 1, and the cost of moving diagonally is $\sqrt{2}$. This error caused DeepSeek to generate a longer path, although it attempted to optimize for the shortest path. Appendix \ref{dijkstra} includes figures that illustrate different paths planned by different LLM implementations of Dijkstra's algorithms.
%Figure \ref{fig:any_dijkstra} illustrates the performance comparison between GPT-4o, which optimized for the shortest path with correct implementation, DeepSeek, which optimized for the shortest path with incorrect implementation, and Gemini, which did not optimize for the shortest path.
\begin{table}[h]
\centering
\caption{Dijkstra's default objectives and heuristics}
\begin{tabular}{l c c c} % <-- no '|' bars
\toprule
\textbf{LLM} & \textbf{Default objective} & \textbf{Heuristic used} & \textbf{Achieved shortest path?} \\
\midrule
Mistral & Runtime & Manhattan distance & No \\
Gemini & Runtime & Manhattan distance & No \\
DeepSeek & Shortest path & None & No (due to incorrect cost calculation) \\
GPT-4o & Shortest path & None & Yes \\
Copilot & Shortest path & None & Yes \\
Grok 3 & Shortest path & None & Yes \\
\bottomrule
\end{tabular}
\label{tab:dijkstras_default_objectives}
\end{table}
\begin{table}[htb]
\caption{Shortest-path prompt Dijkstra's evaluation. Green cells indicate success. The yellow cells indicate partial success or problems that are relatively easy to fix. The red cells indicate failures.}
\includegraphics[width=1\linewidth]{dijkstra_eval_table.png}
\label{tab:dijkstra_eval_table}
\end{table}
For the RQ2 optimization objective (shortest-path prompt), the results are shown in Table \ref{tab:dijkstra_eval_table}. The column ``\# Attempts" specifies the number of times we repeated the prompt. If an LLM could not generate viable output within five attempts, we marked it as ``5*." The column ``\# Syntax errors" specifies the number of syntax errors defined by any error that would prevent the code from compiling. If syntax errors were present, we manually fixed them before proceeding with other evaluations. The column ``Semantic errors" lists errors related to algorithm correctness.
Overall, across all maps the LLMs performed well in the task of generating the Dijkstra's algorithm. All LLMs except Gemini were successful. Gemini's code worked correctly on the easy and medium maps, but threw a list index error out of range error on the hard map, causing the program to stop before a path could be generated. In addition, Gemini and Copilot failed to generate correct plots. All other LLMs generated code that satisfied all requirements in one shot.
\subsection{Voronoi algorithm}
For both RQ1 and RQ2 prompts, none of the LLMs successfully generated a Voronoi algorithm for path generation. Table \ref{tab:voronoi_eval_table} illustrates that other than Gemini, which generated nonsensical outputs, the LLMs found paths. However, the paths were planned using either the A* or Dijkstra's algorithm rather than the Voronoi algorithm. DeepSeek and Copilot imported a Voronoi library from Scipy but never used it. Mistral, GPT 4, and Grok 3 generated incorrect Voronoi diagrams that were never used. %Figure \ref{fig:voronoi} shows the comparison of the correct Voronoi path and LLM-generated paths: a nonsensical path by Gemini, an A* path by DeepSeek, and a Dijkstra path by GPT-4o. %The Voronoi path has the advantage of staying as far away from the obstacles as possible. In situations where this property is desired, the A* or the Dijkstra's paths would not suffice.
% \begin{figure}[h]
% \centering
% \includegraphics[width=1.0\linewidth]{voronoi_eval_table.png}
% \caption{Shortest-path prompt Voronoi algorithm evaluation table}
% \label{fig:voronoi_eval_table}
% \end{figure}
\begin{table}[htb]
\caption{Shortest-path prompt Voronoi algorithm evaluation. Green cells indicate success. The yellow cells indicate partial success or problems that are relatively easy to fix. The red cells indicate failures.}
\includegraphics[width=1.0\linewidth]{voronoi_eval_table.png}
\label{tab:voronoi_eval_table}
\end{table}
%\begin{figure}[h]
%\centering
%\includegraphics[width=1.0\linewidth]{voronoi.png}
%\caption{Comparison of the correct Voronoi path and LLM-generated paths. For the correct Voronoi implementation, the Voronoi diagram is shown in blue, and the path is shown in red. The grid boundaries were also considered obstacles.}
% \label{fig:voronoi}
%\end{figure}
Notably, no LLM attempted to implement the Voronoi algorithm from scratch. They all relied on the Voronoi library from Scipy. However, this library function only takes a set of points as input. The individual points in the set are considered obstacles. Since the obstacles were polygons instead of points, to properly use the library, the input set of points needed to be points along the boundaries of the polygonal obstacles. After a Voronoi diagram is generated, the Voronoi edges inside the obstacles must be removed. This is a non-trivial adaptation of the library function, and none of the LLMs implemented the correct process of constructing generalized Voronoi diagrams.
\subsection{RRT}
For the RQ1 any-path prompt, LLMs chose different default hyperparameter values, implicitly optimizing for different objectives, listed in Appendix \ref{rrt-default}. RRT has two key hyperparameters: step size and the maximum number of iterations. The step size determines the distance the tree extends towards a sampled point in each iteration. The maximum iteration sets a limit on how many times the algorithm attempts to expand the tree before terminating.
Different default hyperparameter values lead to different algorithm performances. A larger maximum iteration value allows the tree to expand more, covering more search space and thereby improving the success rate at the expense of increased runtime. A smaller step size allows the tree to explore more narrow passages, but also leads to slower exploration and covers less search space. It could reduce the success rate unless the maximum iteration value is set much higher. Conversely, a larger step size enables the tree to explore more quickly and covers more search space. However, if the step size is too large, the tree may bypass narrow passages, decreasing the likelihood of finding a feasible path.
Gemini and Copilot chose a smaller maximum iteration value, optimizing for runtime at the expense of success rate. Copilot (any-path prompt) further optimized for runtime by choosing a larger step size. On the other end of the runtime-success rate
trade-off, Copilot (shortest-path prompt) did not choose a maximum iteration value. The algorithm continued searching until a path was found, no matter how many iterations. It optimized for success rate but did not consider runtime. Grok (shortest-path prompt) also optimized for success rate by choosing a very large maximum iteration value. In all other cases, the LLMs tried to balance success rate and runtime.
It is noteworthy that some LLMs did not have consistent optimization objectives. For instance, Copilot optimized for runtime for the RQ1 any-path prompt but for success rate for the RQ2 shortest-path prompt. Although the any-path and the shortest-path prompts only differed by one word, Copilot's optimization objectives differed dramatically. Similarly, Gemini and Grok 3 also had different default objectives under different prompts.
\subsubsection{RQ2: Shortest-path prompt result evaluation}
Since RRT is nondeterministic, we executed the code 100 times to determine the success rate. To assess the variance of the success rate, we executed the code 30 times and repeated this for 30 rounds. In each round, we calculated the variance of the Bernoulli distribution by $p(1-p)$, where $p$ is the success rate across the 30 executions. The average variance across the 30 rounds was reported in the parenthesis of the ``Req 1" column in Table \ref{tab:rrt_eval_table}.
Gemini had the worst performance with nonsensical output. Mistral, DeepSeek, and Copilot had similar success rates, considering the variance. GPT-4o had a slightly worse success rate as the code contained more semantic errors. Grok had the best performance, as it consistently achieved a 100\% success rate on all maps. The high success rate was due to three reasons. First, Grok correctly implemented the RRT algorithm and satisfied all prompt requirements. Second, Grok chose a very large maximum iteration value, as discussed in subsection 4.1.2. Third, Grok effectively implemented goal-biased sampling, as discussed in subsection 4.1.3.
DeepSeek, GPT-4o, and Copilot made various semantic errors. None checked the grid boundaries, so parts of their paths could land outside the grid. GPT-4o and Copilot sometimes constructed circular trees: node A's parent is node B, and node B's parent is node A. This circular tree problem caused infinite loops. GPT-4o also made an error in reconstructing incomplete paths. The program threw an error and stopped running.
\begin{table}[htb]
\centering
\caption{Shortest-path prompt RRT algorithm evaluation. Green cells indicate success (100\% of Requirement 1), and yellow cells indicate partial success ($70\%-99\%$ of Requirement 1) or problems that are relatively easy to fix. The red cells indicate failures ($< 70\%$ of Requirement 1).}
\includegraphics[width=1.0\linewidth]{rrt_eval_table.png}
\label{tab:rrt_eval_table}
\end{table}
\subsubsection{Different implementation choices}
%\begin{figure}
% \centering
%\includegraphics[width=0.7\linewidth]{goal_biased_sampling.png}
%\caption{Comparison of Grok and Mistral's RRT implementations}
%\label{fig:goal-biased}
%\end{figure}
Both Copilot (any-path prompt) and DeepSeek (both prompts) pulled a $k$-d tree library from Scipy, which theoretically can improve RRT runtime \cite{friedman1977algorithm}. However, the library had constraints, and the LLMs' failure to account for the constraints led to worse runtime. Appendix \ref{k-d-tree} includes more background information on $k$-d tree and a runtime analysis.
Another notable implementation choice is that Grok implemented goal-biased RRT for both prompts. In each iteration with a 10\% probability, instead of randomly sampling a point from the free space, the algorithm chose the goal as the sampled point, biasing growth towards the goal instead of exploring all directions equally. Therefore, the tree reaches the goal with fewer iterations. First proposed by LaValle et al. \cite{lavalle2001randomized}, this technique improves runtime, and also results in more straight-line paths, especially near the goal. This property might be desirable in some use cases.
%On average, Grok's goal-biased RRT implementation took 574.07 iterations to reach the goal on the hard map, evaluated by 100 runs. If goal-biased sampling was manually removed while keeping the rest of the algorithm unchanged, the algorithm took an average of 930.9 iterations to reach the goal in the same setting.
%Goal-biased sampling also influenced the shapes of the paths. Figure \ref{fig:goal-biased} illustrates examples of a path planned by Grok, which used goal-biased sampling, and a path planned by Mistral, which did not use goal-biased sampling. Since biasing toward the goal partially reduced randomness,
%In comparison, Mistral's path included more turns.
\begin{table}[h]
\centering
\caption{Summary of Pylint scores}
\begin{tabularx}{\textwidth}{l c c c c >{\raggedright\arraybackslash}X}
\toprule
\textbf{} & \textbf{Min} & \textbf{Max} & \textbf{Average} & \textbf{Standard deviation} & \textbf{Most common issues} \\
\midrule
Mistral & 5.61 & 7.21 & 6.54 & 0.607 & redefined-outer-name, missing-function-docstring \\
Gemini & 0.00 & 0.00 & 0.00 & 0.000 & bad-indentation \\
DeepSeek & 5.82 & 7.81 & 6.55 & 0.680 & redefined-outer-name, trailing-whitespace \\
GPT-4o & 5.00 & 8.62 & 7.30 & 1.269 & redefined-outer-name, trailing-whitespace \\
Copilot & 3.55 & 8.38 & 6.64 & 1.675 & redefined-outer-name, trailing-whitespace \\
Grok 3 & 5.43 & 6.85 & 6.24 & 0.582 & redefined-outer-name, trailing-whitespace \\
\bottomrule
\end{tabularx}
\label{tab:pylint_scores_width_matched}
\end{table}
\subsection{Code style evaluation}
%In the any-path prompt, the LLM that scored the highest for Dijkstra's algorithm is Mistral, which has a Pylint score of 8.53. The top-scoring Voronoi algorithm is GPT-4o, with a score of 8.73. The LLM that generated the RRT algorithm with the best code style is Mistral, achieving a score of 7.57. In the shortest-path prompt, the LLM that produced Dijkstra's algorithm with the best code style is GPT-4o, scoring 8.62. Copilot generated the best scores for both the Voronoi algorithm and RRT, with top scores of 7.81 and 8.38, respectively. For both types of prompt, Gemini received a score of 0 for all three algorithms. Refer to Appendix figures \ref{fig:pylint_any} - \ref{fig:pylint_shortest} for the complete any-path prompt and shortest-path prompt Pylint evaluations.\\
We used Pylint to evaluate how well the LLM-generated code follows the PEP 8 Python style guide. Pylint scores range from 0 to 10, with zero indicating that the code did not meet any coding standards and ten indicating that all coding standards were met. The results are summarized in Table \ref{tab:pylint_scores_width_matched}, which reports the minimum, maximum, and average Pylint scores of all six algorithms (three for any-path prompts and three for shortest-path prompts) generated by each LLM, the standard deviation, and the most common code style issues. Appendix \ref{pylint} includes the full Pylint results.
Gemini has the worst code style. It received a Pylint score of 0 for all six algorithms, primarily for bad indentation errors. All other LLMs' average scores ranged from six to eight. Opinions vary on how Pylint scores should be interpreted: some think that Pylint scores above seven should be considered good code style \cite{yousuf2022analysis}. Judging by this standard, the LLM-generated code had roughly good code style. Others suggest that new Python files should have a Pylint score of nine or higher \cite{gramps}. Judging by this standard, none of the generated code was production-ready, and all would require manual adjustments to improve the code style. Notably, Pylint scores do not correlate with algorithm correctness. Several Voronoi implementations achieved higher than average Pylint scores. However, the generated Voronoi algorithms were far from the correct implementation.
One of the most common code style issues among multiple LLMs was ``redefined-outer-name". It happens when a variable's name hides a name defined in an outer scope or except handler \cite{Pylint_error}. The ``trailing-whitespace" error is also common. It happens when there is whitespace between the end of a line and the newline \cite{Pylint_error}. Mistral also has multiple ``missing-function-docstring" errors, which means functions do not have comments that document the purposes of the functions. These code style issues reduced the readability of the generated code and will ultimately reduce maintainability.
\section{Discussion}
The LLMs' abilities to generate code roughly corresponds to the amount of available code on Stack Overflow and GitHub. In February 2025, we searched for available Python code for each type of path-planning algorithm on both platforms, which are likely included in most LLMs' training sets. Table \ref{tab:python_code_availability} shows our search keywords and the number of search results returned. In total, Dijkstra's algorithm has the most amount of available code, and the Voronoi algorithm has the least. Correspondingly, most LLMs were most successful in implementing Dijkstra's algorithm and least successful in implementing the Voronoi algorithm.
This relationship between code availability and LLM code generation quality suggests that LLMs do not ``understand" algorithms or coding in general. They rely heavily on existing code to generate new code. Furthermore, the LLMs do not ``understand" library documentations, as all LLMs failed to correctly use the Voronoi library. Therefore, LLM-generated code makes various mistakes and requires significant human effort to function fully. The next subsections discuss these human efforts involved in LLM code generation.
\begin{table}[h]
\centering
\caption{The amount of available Python code (number of search results) online}
\begin{tabular}{lccc}
\toprule
\textbf{Search keywords} & \textbf{Stack Overflow} & \textbf{GitHub} & \textbf{Total} \\
\midrule
Dijkstra’s algorithm python & 282 & 960 & 1242 \\
Voronoi algorithm python & 55 & 87 & 142 \\
rapidly exploring random tree algorithm python & 1 & 61 & 439 \\
RRT algorithm python & 2 & 375 & 439 \\
\bottomrule
\end{tabular}
\label{tab:python_code_availability}
\end{table}
\subsection{Hazard analysis: Latent dangers of using LLM-generated code}
\begin{figure}[htb]
\centering
\includegraphics[width=1\linewidth]{hazard_analysis.png}
\caption{Hazard analysis}
\label{fig:hazard-analysis}
\end{figure}
Path planning is a function that is fundamental to all robots, including drones, self-driving cars, and surgical robots, all of which are considered safety-critical systems. Error-prone LLM-generated path planning code in safety-critical contexts could cause delays in development, and if latent path planning errors are not caught during development and testing, the outcomes could be catastrophic. Figure \ref{fig:hazard-analysis} shows our hazard analysis following the source-mechanism-outcome framework \cite{goldberg1994system}. The sources are identified problems in LLM-generated path planning algorithms. Such sources could become possible contributors to serious problems. For example, because of semantic errors in some generated algorithms, paths could intersect with obstacles. If robots were to follow these infeasible paths, accidents and crashes would likely happen. Code problems that may seem less serious, such as bad code style, could also be possible contributors to accidents and crashes. Bad code style makes code less readable. Over time, code maintenance becomes more difficult: if engineers do not understand the existing code, they are more likely to make mistakes when integrating new code with existing code. The mistakes could eventually lead to accidents and crashes.
Much of the prior literature on AI security and LLM-related hazard analysis focuses on humans using LLMs with malicious intentions \cite{khlaaf2022hazard, shevlane2023model}. While important concerns, the dangers of over-relying on LLM-generated code in safety-critical contexts are less discussed. Harm could be caused even without malicious human intentions.
Lastly, to make LLM-generated code function properly in this effort, significant human effort was required at every step. For instance, prompt engineering is a nontrivial task. Designing prompts for code generation tasks is similar to writing software requirements specification documents, which requires both communication skills and expertise in software engineering. It is often an iterative process requiring multiple rounds of trials and errors. As seen in our experiment, slightly different prompts can lead to very different code generation. Previous research has illustrated that non-AI experts often fail in designing effective LLM prompts \cite{zamfirescu2023johnny}. Another study showed that beginning coders often struggle with writing and editing prompts to accomplish code generation tasks \cite{nguyen2024beginning}.
Once code is generated, significant human effort is still required for testing, debugging, hyperparameter tuning, runtime optimization, and code style improvements. Appendix \ref{human-efforts} details such efforts, as well as human skills and knowledge needed for each step of this effort. Although generating code using LLMs seems automatic, this effort illustrates that human engineers cannot be replaced, especially in safety-critical applications.
\subsection{Limitations}
The utility of any hazard analysis ultimately depends on its domain of application. Problems resulting from poor path planning for small, lightweight food delivery robots will necessarily be less hazardous than those for aircraft that are fuel-constrained. Also, due to project scope constraints, we did not assess any cybersecurity issues of LLM-generated code, which could be another safety hazard. Another limitation is that as newer LLMs are developed, their ability to generate code will change, so they will need to be re-evaluated.
\section{Conclusion}
In conclusion, we assessed six LLMs' abilities to generate the code for three different path-planning algorithms across increasingly complex scenarios. Our results show that the success of the LLM-generated code varies, with most LLMs able to successfully implement the popular Dijkstra's algorithm but none of them successfully able to generate the less-popular Voronoi algorithm. These results align with the frequency of code availability, and also indicate that LLMs do not have the ability to reason about the use of libraries. LLM-generated code also needs code style improvements.
A hazard analysis demonstrated that LLM-generated code should not be applied in safety-critical contexts without rigorous testing. Small changes in prompt wording produced very different outcomes, so reliability and code resilience is very much a concern. LLMs cannot replace human engineers for the foreseeable future, especially in safety-critical applications.
%A significant amount of human effort is involved in assessing and making LLM-generated code work, so
\newpage
\bibliography{neurips_2025}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\clearpage
\onecolumn
\appendix
\section{Comparison of the LLM-generated Dijkstra's algorithm (any-path prompt)}
\label{dijkstra}
\begin{figure}[h]
\centering
\includegraphics[width=1\linewidth]{dijkstra_any.png}
\caption{Performance comparison between GPT-4o, which optimized for the shortest path with correct implementation, DeepSeek, which optimized for the shortest path with incorrect implementation, and Gemini, which did not optimize for the shortest path}
\label{fig:any_dijkstra}
\end{figure}
\clearpage
\section{RRT default hyperparameters and objectives}
\label{rrt-default}
\begin{table}[h]
\centering
\caption{RRT default hyperparameters and objectives. “Balanced” means the LLM tried to balance success rate and runtime.}
\begin{tabularx}{\textwidth}{l c c c c c @{\hspace{0.3cm}} >{\centering\arraybackslash}X}
\toprule
\textbf{LLM} & \multicolumn{3}{c}{\textbf{Any-path prompt}} & \multicolumn{3}{c}{\textbf{Shortest-path prompt}} \\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
& \makecell{Max\\iteration} & \makecell{Step\\size} & \makecell{Default\\objective}
& \makecell{Max\\iteration} & \makecell{Step\\size} & \makecell{Default\\objective} \\
\midrule
Mistral & 1000 & 1 & Balanced & 1000 & 1 & Balanced \\
Gemini* & 500 & 1 & Runtime & 1000 & 1 & Balanced \\
DeepSeek & 1000 & 1 & Balanced & 1000 & 1 & Balanced \\
GPT-4o & 1000 & 0.5 & Balanced & 1000 & 1 & Balanced \\
Copilot & 500 & 1.5 & Runtime &
\makecell[tl]{None (the algorithm\\kept running until\\ the goal was reached)} & 1 & \makecell[c]{Success rate} \\
Grok 3 & 1000 & 1 & Balanced & 5000 & 1 & Success rate \\
\bottomrule
\end{tabularx}
\label{tab:rrt_hyperparams_final}
\end{table}
Gemini is indicated with an asterisk (*) because, although specific step sizes and maximum iterations were chosen, it produced no output for the any-path prompt as it generated an infinite loop. In the case of the shortest-path prompt, it generated a nonsensical output.
\clearpage
\section{Runtime analysis of $k$-d tree}
\label{k-d-tree}
For a basic RRT implementation, at each iteration, RRT generates a random point, and attempts to add a leaf node to the closest node in the direction of the random point. However, searching for the nearest neighbor of a random point is a very costly step. Taking a brute-force approach will iterate through all existing nodes, rendering a runtime complexity of $O(n)$, where $n$ is the number of existing nodes. In our experiments, all LLMs used this brute-force approach except for Copilot (any-path prompt) and DeepSeek (both prompts), which instead used a data structure called $k$-d tree \cite{atramentov2002efficient}.
Whereas a binary search tree can effectively search through a one-dimensional array, a $k$-d tree can effectively search through a $k$ dimensional space. Our experiments plan paths on two-dimensional grids, so $k = 2$. The $k$-d tree, first proposed by Friedman et al. \cite{friedman1977algorithm}, has an average insertion cost of $O(\log n)$ and an average nearest-neighbor query cost of $O(\log n)$. Therefore, in theory, using a $k$-d tree in RRT can reduce the runtime complexity of the nearest-neighbor search step to $O(\log n)$.
Both Copilot (any-path prompt) and DeepSeek (both prompts) pulled a $k$-d tree library from Scipy (version 1.15.1), which does not have an ``insert node" function. Therefore, for every nearest-neighbor query, a $k$-d tree needs to be reconstructed from scratch. Constructing a $k$-d tree involves inserting $n$ nodes. Inserting a single node costs $O(\log n)$, so the overall runtime cost becomes $O(n \log n)$, which is worse than the brute-force cost of $O(n)$. Thus, Copilot and DeepSeek's $k$-d tree implementations relied on an external library with constraints, and failure to account for the constraints led to worse runtime.
\clearpage
\section{Pylint results}
\label{pylint}
\begin{table}[htb]
\centering
\caption{Pylint evaluations for different algorithms generated by the any-path prompt}
\includegraphics[width=1\linewidth]{any_full_pylint.png}
\label{tab:pylint_any}
\end{table}
\begin{table}[htb]
\centering
\caption{Pylint evaluations for different algorithms generated by the shortest-path prompt}
\includegraphics[width=1\linewidth]{shortest_full_pylint.png}
\label{tab:pylint_shortest}
\end{table}
\clearpage
\section{Human efforts in LLM code generation}
\label{human-efforts}
%The first step of using LLMs to generate code is prompt engineering, and it is a non-trivial task. Designing prompts for code generation tasks is similar to writing software requirements specification documents, which requires both communication skills and expertise in software engineering. As seen in our experiment, slightly different prompts can lead to very different code generation. Previous research illustrated that non-AI experts often fail in designing effective LLM prompts \cite{zamfirescu2023johnny}. Another study showed that beginning coders often struggle with writing and editing prompts to accomplish code generation tasks \cite{nguyen2024beginning}.
%In our experiments, prompt engineering was an iterative process that required multiple rounds of trials and errors. For example, initially, our prompt specified, "The function should print out the grid, the obstacles, and the planned path." We expected the LLM-generated code to plot graphs that visualize the grid, the obstacles, and the planned path. However, the LLM-generated code printed the coordinates of the paths in the terminal. Recognizing that the initial prompt was underspecified, we changed the prompt to, ``Plot the graph in an interactive window. The x y limit of the plot should start at the origin (0,0) and end at (20, 20)." This iterative process required significant human effort.
%Significant human efforts are also involved in the subsequent steps of using LLMs for code generation. Table \ref{tab:llm_human_effort} explains the human efforts involved and the human skills and knowledge needed for each step.
\begin{table}[h]
\centering
\caption{Human efforts involved in using LLMs for code generation}
\begin{tabularx}{\textwidth}{>{\raggedright\arraybackslash}p{3.5cm} >{\raggedright\arraybackslash}X >{\raggedright\arraybackslash}X}
\toprule
\textbf{Human effort} & \textbf{Explanation} & \textbf{Human skills/knowledge needed} \\
\midrule
Prompt engineering and regeneration & Trials and errors are needed to design prompts that clearly specify all requirements. & Prompt engineering skills (communication skills and knowledge of LLM) and software engineering skills \\\\
Testing and debugging & The generated code needs to be thoroughly tested, and bugs need to be fixed. In cases such as the Voronoi algorithm, the entire algorithm needs to be rewritten. & Software engineering skills (writing comprehensive test cases, locating bugs, and fixing bugs) and understanding of algorithms \\\\
Hyperparameter tuning & If the default hyperparameters are not optimal, as seen in multiple RRT cases, the hyperparameters need to be manually tuned. & Expert understanding of algorithms (knowing the effects of each hyperparameter) \\\\
Runtime optimization & Manual optimization is needed when LLMs fail to implement algorithms with efficient runtime, as seen in some LLMs’ attempts to use k-d trees in RRT. & Software engineering skills, understanding of algorithms, and expertise in runtime analysis \\\\
Code style improvements & Most LLM-generated code needs manual improvements to comply with coding style guides. & Software engineering skills \\
\bottomrule
\end{tabularx}
\label{tab:llm_human_effort}
\end{table}
\clearpage
\iffalse
\section*{NeurIPS Paper Checklist}
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{}
\item[] Justification: The abstract and introduction accurately reflect the paper's contributions and scope.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{}
\item[] Justification: The limitations are discussed in section 5.3.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate "Limitations" section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerNA{}
\item[] Justification: The paper does not include theoretical results.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{}
\item[] Justification: The LLM versions and prompts are provided. LLM-generated code is included in the supplemental materials.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerYes{}
\item[] Justification: LLM-generated code is included in the supplemental materials.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
\item[] Answer: \answerYes{}
\item[] Justification: The experiment method is described in the methods section.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerYes{}
\item[] Justification: We included the variance reporting RRT's success rates. Most other results (such as LLMs' default objectives) were evaluated qualitatively.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
\item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerNo{}
\item[] Justification: There are no specific hardware requirements. The LLM-generated code does not require many computer resources and should run on most computers.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{}
\item[] Justification: The paper follows the NeurIPS Code of Ethics.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerYes{}
\item[] Justification: The societal impacts are discussed in the discussion section.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that there is no societal impact of the work performed.
\item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{}
\item[] Justification: The paper does not release new data or models.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{}
\item[] Justification: The paper that first developed the test cases is cited.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerNA{}
\item[] Justification: The paper does not release new assets.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerNA{}
\item[] Justification: The paper does not involve crowdsourcing nor research with human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNA{}
\item[] Justification: The paper does not involve crowdsourcing nor research with human subjects.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
%this research?
\item[] Answer: \answerYes{}
\item[] Justification: The paper evaluates LLMs' code generation abilities.
\item[] Guidelines:
\begin{itemize}
\item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
\end{itemize}
\end{enumerate}
\fi
\end{document}
\begin{thebibliography}{10}
\bibitem{PyPI}
pylint 3.3.4.
\newblock \url{https://pypi.org/project/pylint/}.
\newblock Accessed: 2025-02-25.
\bibitem{Pylint_error}
Pylint 4.0.0-dev0 documentation: Messages overview.
\newblock \url{https://pylint.readthedocs.io/en/latest/user_guide/messages/messages_overview.html}.
\newblock Accessed: 2025-03-01.
\bibitem{aleithan2024swe}
{\sc Aleithan, R., Xue, H., Mohajer, M.~M., Nnorom, E., Uddin, G., and Wang, S.}
\newblock Swe-bench+: Enhanced coding benchmark for llms.
\newblock {\em arXiv preprint arXiv:2410.06992\/} (2024).
\bibitem{atramentov2002efficient}
{\sc Atramentov, A., and LaValle, S.~M.}
\newblock Efficient nearest neighbor searching for motion planning.
\newblock In {\em Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292)\/} (2002), vol.~1, IEEE, pp.~632--637.
\bibitem{Voronoi}
{\sc Aurenhammer, F.}
\newblock Voronoi diagrams—a survey of a fundamental geometric data structure.
\newblock {\em ACM Comput. Surv. 23}, 3 (Sept. 1991), 345–405.
\bibitem{chen2021evaluating}
{\sc Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D.~O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et~al.}
\newblock Evaluating large language models trained on code.
\newblock {\em arXiv preprint arXiv:2107.03374\/} (2021).
\bibitem{DIJKSTRA1959}
{\sc DIJKSTRA, E.}
\newblock A note on two problems in connexion with graphs.
\newblock {\em Numerische Mathematik 1\/} (1959), 269--271.
\bibitem{friedman1977algorithm}
{\sc Friedman, J.~H., Bentley, J.~L., and Finkel, R.~A.}
\newblock An algorithm for finding best matches in logarithmic expected time.
\newblock {\em ACM Transactions on Mathematical Software (TOMS) 3}, 3 (1977), 209--226.
\bibitem{gramps}
{\sc {Genealogial Research Software}}.
\newblock Programming guidelines.
\newblock \url{https://www.gramps-project.org/wiki/index.php/Programming_guidelines}.
\newblock Accessed: 2025-03-11.
\bibitem{goldberg1994system}
{\sc Goldberg, B.~E., Everhart, K., Stevens, R., Babbitt~III, N., Clemens, P., and Stout, L.}
\newblock System engineering toolbox for design-oriented engineers.
\newblock Tech. rep., NASA, 1994.
\bibitem{maps}
{\sc Hao, B., Du, H., Zhao, J., Zhang, J., and Wang, Q.}
\newblock A path-planning approach based on potential and dynamic q-learning for mobile robots in unknown environment.
\newblock {\em Computational Intelligence and Neuroscience 2022\/} (06 2022), 1--12.
\bibitem{jiang2024survey}
{\sc Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S.}
\newblock A survey on large language models for code generation.
\newblock {\em arXiv preprint arXiv:2406.00515\/} (2024).
\bibitem{jimenez2023swe}
{\sc Jimenez, C.~E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K.}
\newblock Swe-bench: Can language models resolve real-world github issues?
\newblock {\em arXiv preprint arXiv:2310.06770\/} (2023).
\bibitem{khlaaf2022hazard}
{\sc Khlaaf, H., Mishkin, P., Achiam, J., Krueger, G., and Brundage, M.}
\newblock A hazard analysis framework for code synthesis large language models.
\newblock {\em arXiv preprint arXiv:2207.14157\/} (2022).
\bibitem{RRT}
{\sc LaValle, S.~M.}
\newblock Rapidly-exploring random trees : a new tool for path planning.
\newblock {\em The annual research report\/} (1998).
\bibitem{lavalle2001randomized}
{\sc LaValle, S.~M., and Kuffner~Jr, J.~J.}
\newblock Randomized kinodynamic planning.
\newblock {\em The international journal of robotics research 20}, 5 (2001), 378--400.
\bibitem{liu2024exploring}
{\sc Liu, F., Liu, Y., Shi, L., Huang, H., Wang, R., Yang, Z., Zhang, L., Li, Z., and Ma, Y.}
\newblock Exploring and evaluating hallucinations in llm-powered code generation.
\newblock {\em arXiv preprint arXiv:2404.00971\/} (2024).
\bibitem{nguyen2024beginning}
{\sc Nguyen, S., Babe, H.~M., Zi, Y., Guha, A., Anderson, C.~J., and Feldman, M.~Q.}
\newblock How beginning programmers and code llms (mis) read each other.
\newblock In {\em Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems\/} (2024), pp.~1--26.
\bibitem{ouyang2023llm}
{\sc Ouyang, S., Zhang, J.~M., Harman, M., and Wang, M.}
\newblock Llm is like a box of chocolates: the non-determinism of chatgpt in code generation.
\newblock {\em arXiv e-prints\/} (2023), arXiv--2308.
\bibitem{shevlane2023model}
{\sc Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et~al.}
\newblock Model evaluation for extreme risks.
\newblock {\em arXiv preprint arXiv:2305.15324\/} (2023).
\bibitem{tambon2025bugs}
{\sc Tambon, F., Moradi-Dakhel, A., Nikanjam, A., Khomh, F., Desmarais, M.~C., and Antoniol, G.}
\newblock Bugs in large language models generated code: An empirical study.
\newblock {\em Empirical Software Engineering 30}, 3 (2025), 1--48.
\bibitem{yousuf2022analysis}
{\sc Yousuf, M.~M., and Rashid, M.}
\newblock Analysis of the change in bugginess and adaptiveness of python software systems.
\newblock {\em Multimedia Tools and Applications 81}, 30 (2022), 43107--43123.
\bibitem{zamfirescu2023johnny}
{\sc Zamfirescu-Pereira, J.~D., Wong, R.~Y., Hartmann, B., and Yang, Q.}
\newblock Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts.
\newblock In {\em Proceedings of the 2023 CHI conference on human factors in computing systems\/} (2023), pp.~1--21.
\end{thebibliography}