Test Oracle Automation in the era of LLMs
Extracted source text · arXiv 2405.12766
\documentclass[sigconf]{acmart}
\acmConference[SE 2030]{International Workshop on Software Engineering in 2030}{November 2024}{Puerto Galinàs (Brazil)}
\usepackage{graphicx} % Required for inserting images
\usepackage{listings}
\usepackage{pifont}
\usepackage{todonotes}
\title{Test Oracle Automation in the era of LLMs}
\author{Facundo Molina}
\affiliation{
\institution{IMDEA Software Institute}
\city{Madrid}
\country{Spain}
}
\author{Alessandra Gorla}
\affiliation{
\institution{IMDEA Software Institute}
\city{Madrid}
\country{Spain}
}
\date{February 2024}
\newcommand{\fm}[1]{{\color{red}\textbf{[[Facu:} #1\textbf{]]}}}
%%% Listing config for Java
\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}
\definecolor{mygray}{gray}{1}
\renewcommand*\ttdefault{txtt}
\lstset{frame=tb,
language=Java,
aboveskip=0mm,
belowskip=0mm,
showstringspaces=false,
columns=flexible,
basicstyle=\footnotesize\tt,
numbers=none,
numberstyle=\tiny\color{gray},
keywordstyle=\color{blue},
commentstyle=\color{dkgreen},
stringstyle=\color{magenta},
breaklines=true,
keepspaces=true,
breakatwhitespace=true,
tabsize=3% identifierstyle=\color{magenta}
}
%%% end listing
\begin{document}
\begin{abstract}
The effectiveness of a test suite in detecting faults highly depends on the correctness and completeness of its test oracles.
Large Language Models (LLMs) have already demonstrated remarkable proficiency in tackling diverse software testing tasks, such as automated test generation and program repair.
This paper aims to enable discussions on the potential of using LLMs for
test oracle automation, along with the challenges that may emerge during the generation of various types of oracles.
Additionally, our aim is to initiate discussions on the primary threats that SE researchers must consider when employing LLMs for oracle automation, encompassing concerns regarding oracle deficiencies and %privacy-related issues related to
data leakages.
\end{abstract}
\maketitle
\section{Introduction}
The goal of software testing is to find defects.
To actually find defects, though, test suites require relevant test inputs, i.e.,
inputs that can exercise the software under test (SUT)
in realistic scenarios. Furthermore, to ensure that
the SUT exhibits the expected behavior for these inputs,
accurate test oracles are required. The problem
of automating the generation of test oracles, the so-called
oracle problem~\cite{Barr+2015}, has become very relevant
in the last decades, as it has the potential to improve
the oracles used in the testing process, and therefore
contributing to revealing defects in software.
Various approaches have been proposed to address the oracle problem by
automatically deriving different kinds of oracles~\cite{DBLP:journals/scp/ErnstPGMPTX07,autoinfer2011,gassert2020,evospex2021,memo2021,precis2021,toga2022, specfuzzer2022, agora2023}, including
\emph{test assertions}, \emph{contracts} (such as pre/postconditions and invariants) or
\emph{metamorphic relations}.
Generally, these approaches
observe some artifact related to the SUT (documentation, comments,
source code, executions) and then derive oracles that
are consistent with the observations. For example,
TOGA~\cite{toga2022} observes the source code
of a target test and a focal method (method under test)
and infers a test assertion for the given test;
MeMo~\cite{memo2021} extracts metamorphic relations by observing natural language comments in the source code;
Daikon~\cite{DBLP:journals/scp/ErnstPGMPTX07} and related tools \cite{gassert2020,evospex2021,specfuzzer2022} observe the behavior of the SUT
(from a set of tests) in order to infer class invariants and pre/postconditions.
Despite all these efforts, the problem of automatically deriving oracles is still
an open research problem
in software testing~\cite{Barr+2015}.
The main reason is that automatically derived oracles are rarely accurate.
These accuracy issues can result in high false positive rates,
which can lead to false alarms and reduce the trustworthiness of the testing process.
Indeed, recent studies have shown that even state-of-the-art neural based approaches can produce oracles with high false positive rates~\cite{DBLP:conf/sigsoft/HossainFDEV23}.
%For traditional software systems, i.e., software without
%components based on Artificial Intelligence (AI), the concept of test oracle is well
%established and widely accepted.
%A test oracle for a given SUT, is an executable
%property $\alpha$ that can distinguish between the correct
%and incorrect executions of the SUT, under one or more test scenarios. That is, $\alpha$ can recognize as correct
%those executions that comply with the expected behavior,
%and as incorrect those that doesn't. Under this notion,
%many approaches have been proposed to address the oracle
%problem for traditional software systems \cite{DBLP:journals/scp/ErnstPGMPTX07,Molina+2021,gassert2020}.
Large Language Models (LLMs) have already been used
in tackling diverse software testing~\cite{testing-llms-2024},
demonstrating remarkable proficiency, particularly in tasks
such as automated program repair~\cite{DBLP:conf/icse/FanGMRT23, DBLP:conf/icse/XiaWZ23, DBLP:conf/icse/JiangLLT23}
and automated test generation~\cite{cat-lm-2023, codamosa2023, DBLP:journals/tse/SchaferNET24}.
Given the positive results, researchers have also started to explore
the use of LLMs to automatically generate test oracles,
mainly in the form of test assertions~\cite{DBLP:conf/ast/TufanoDSS22, teco2023,DBLP:conf/icse/NashidSM23}.
Although the initial results are promising, showing that generated
assertions can improve test coverage~\cite{DBLP:conf/ast/TufanoDSS22} and
some lexical/functional metrics~\cite{teco2023}, there are still aspects of
LLM-generated oracles that need to be further explored.
This paper aims to enable discussions on the use of LLMs for
test oracle automation in general (not only test assertions)
by discussing:
\begin{itemize}
\item the potential of LLMs for test oracle automation (including oracles that go beyond test assertions, such as contracts or metamorphic relations) and the challenges when using the LLMs through prompt engineering or by pre-training or fine-tuning them; and
\item the main threats that arise from the use of LLMs to generate different
kinds of oracles, including oracle deficiencies and privacy-related issues related to data leakages,
and how we can mitigate such threats.
\end{itemize}
%\section{LLMs for Software Testing}
\section{LLMs for Oracle Automation}
%\begin{figure}[t]
% \begin{center}
% \includegraphics[width=200pt]{img/llm-oracle-gen.png}
% \end{center}
% \caption{Overview of LLM-based Oracle Generation.}
% \label{llm-oracle-gen}
% \end{figure}
The most straightforward application of LLMs for oracle automation involves prompt engineering, wherein a prompt is designed to instruct any state-of-the-art pre-trained model, such as ChatGPT-3.5~\cite{openai} or Llama2~\cite{llama2}, to produce an oracle.
%The prompt engineering process
%can be performed through different strategies.
The majority of studies in the literature that utilize LLMs for software testing tasks, as well as for other applications in general, concentrate on two primary strategies for prompt engineering: zero-shot learning and few-shot learning~\cite{testing-llms-2024}.
Zero-shot learning consists of providing a prompt
asking the model for results, e.g.,
``Generate assertions for the following test: ...''.
Few-shot learning, instead, consists of providing a set
of high-quality examples to the model. For instance, one could provide a set of test-assertion pairs for the model to generate assertions for a given test.
%In fact, two thirds of recent studies that use LLMs for software testing tasks
%employ prompt engineering to communicate with the LLMs
%to steer their behavior for desired outcomes~\cite{testing-llms-2024}.
A more sophisticated approach to oracle automation via LLMs involves \textbf{pre-training or fine-tuning}.
Pre-training entails training the model on a broad distribution of data to predict the subsequent token in a sequence. Conversely, during fine-tuning, the weights of a pre-trained model are adjusted by retraining it on a designated dataset tailored for a specific task.
A clear example of this approach has been proposed by Tufano et al.~\cite{DBLP:conf/ast/TufanoDSS22}, where an LLM is
pre-trained with a large source code and English language corpora,
and then fine-tuned for generating assert statements.
Regardless of the strategy used,
generating oracle via LLMs requires
the use of data related to the expected oracles,
either to build prompts or to pre-train or fine-tune the model.
Various sources of pertinent information (such as source code, test code, documentation, logs, etc.) may be utilized to feed the LLMs.
However, the type of information employed will also vary based on the type of oracle intended to be generated, thereby presenting distinct challenges. For instance, employing a zero-shot learning approach to produce a test assertion for a given test merely requires furnishing a prompt containing the test case and instructing the model to expand it with assertions.
Applying the same methodology to generate broader oracles, such as postconditions for a method, necessitates a more detailed prompt. In addition to incorporating the method code, one must also specify the formalism or language for articulating the contract (e.g., as a code fragment, an assert statement, a logical expression, etc.).
%LLMs that do not accept natural language prompts
%In principle, two types of LLMs can be used
%for oracle automation: code completion models (\fm{cite}), i.e., LLMs
%specifically trained for the code completion task,
%and more general LLMs capable of accepting prompts in
%natural language (such as ChatGPT-3.5~\cite{openai}, Gemini \fm{cite} or Llama 2~\cite{llama2}).
%In the case of code completion models, the prompt will be an incomplete
%code snippet that will be provided as input to the model so that it can
%complete it with the missing part of the code, which in this case is the oracle.
%For more general LLMs, the prompt can include, in addition to the code, a natural language description of the kind of oracle we want to poduce,
%for which program point, etc.
To properly illustrate these challenges,
we now consider the most common types of
automatically inferred oracles:
test assertions, contracts, and metamorphic relations.
\subsection{Test Assertions}
Test assertions in unit tests check
the expected behavior of the SUT in a specific scenario,
and are typically expressed as code.
Given their importance for having high quality test suites,
the automated generation of test assertions has been widely studied,
including approaches that pre-train and fine-tune the models~\cite{DBLP:conf/ast/TufanoDSS22, teco2023}.
Tufano et al.~\cite{DBLP:conf/ast/TufanoDSS22} pre-trained a BART
Transformer model~\cite{DBLP:journals/corr/abs-1910-13461}
with a large corpus of English text and Java code,
and then fine-tuned it on the task
of generating assert statements for unit test cases.
Similarly, TeCo~\cite{teco2023} fine-tunes
the CodeT5~\cite{DBLP:conf/emnlp/0034WJH21} and CodeGPT~\cite{lu2021codexglue}
LLMs specifically for the test completion task
(i.e., predict the next statement in a test case),
which are then evaluated for assert statement generation
from the code under test (including the method under test),
the test method signature, and the prior statements before the
assertion statement.
These approaches can achieve an exact match rate
(percentage of generated assertions that exactly match the
expected assertions) of up to 62\%.
The recent improvements achieved by state-of-the-art LLMs enable
the use of such models for generating test assertions using
a more straightforward approach, such as zero-shot learning.
Consider for example the test in Figure~\ref{stack-test},
which shows a very simple test case for a \texttt{Stack} class,
with a unique test assertion checking that the stack is not empty after
three push operations and one pop operation.
\begin{figure}[h]
\begin{lstlisting}[]
public void testPop() {
Stack<Integer> stack = new Stack<>();
stack.push(2);
stack.push(3);
stack.push(5);
stack.pop();
assertFalse(stack.isEmpty());
}
\end{lstlisting}
\caption{A simple test for a \texttt{Stack} class.}
\label{stack-test}
\end{figure}
If we provide the prompt
``Extend the following Java test just with assert statements: + test-code''
where test-code is the test \texttt{testPop},
ChatGPT-3.5 produces the following assertions:
\newline
\begin{center}
\begin{lstlisting}[frame=none]
assert !stack.isEmpty() : "Stack should not be empty after pop";
assert poppedElement == 5 : "Popped element should be 5";
\end{lstlisting}
\end{center}
\vspace{0.1cm}
also checking that the popped element is 5
(ChatGPT-3.5 edited the test to save
the result of pop in the variable \texttt{poppedElement}).
In fact, Nashid et al~\cite{DBLP:conf/icse/NashidSM23} recently
proposed a technique for prompt creation based
on embedding or frequency analysis, that can achieve
an exact match rate of 76\% in test assertion generation.
\subsection{Contracts}
Contracts~\cite{DBLP:journals/computer/Meyer92} are logical constraints on a specific
software element (method, class, etc.),
and are usually captured in the form of preconditions
and postconditions for methods, or representation invariants for classes.
As opposed to test assertions, contracts are not specific to a particular
test case, but rather express properties that must hold for any execution.
Moreover, contracts can be expressed in different formalisms,
not only in the same language as the SUT.
For instance, JML~\cite{DBLP:conf/fmco/ChalinKLP05} is a
popular library for Java that includes a language for writing
contracts as annotations in the source code.
Daikon~\cite{DBLP:journals/scp/ErnstPGMPTX07},
a well-known tool for dynamic invariant detection,
produces pre/postconditions and class invariants
using its own language (a mix of Java,
and mathematical logic).
Other contract inference tools also use their own
languages~\cite{gassert2020,evospex2021,specfuzzer2022}.
To the best of our knowledge, there are still no studies
using LLMs to generate contracts.
To give an idea on how LLMs could be used for this task,
let us consider an implementation of the \texttt{push}
operation for a \texttt{Stack} class,
shown in Figure~\ref{stack-push}.
\begin{figure}[h]
\begin{lstlisting}[]
public void push(E e) {
if (size == elements.length) {
ensureCapacity();
}
elements[size++] = e;
}
\end{lstlisting}
\caption{Implementation of a \texttt{push} operation for a \texttt{Stack}.}
\label{stack-push}
\end{figure}
The code first ensures that there is enough capacity
in the \texttt{elements} array, and then pushes the element
in the next available position. A contract for this method
could include a postcondition stating that the size of the stack
is increased by one, and that the element is added at the top (\texttt{size-1}) of the stack.
The variety of formalisms for expressing contracts poses a challenge
when using LLMs,
mainly because one would also need to specify the formalism
in which the contract must be expressed.
State-of-the-art pre-trained models have the potential
of producing contracts with a reasonable performance,
using a prompt engineering approach,
at least for well-established formalisms.
For instance, with the zero-shot prompt
``Generate a postcondition in the form of an assert statement for the following method: + method-code'',
where method-code is the \texttt{push} method in Figure~\ref{stack-push},
ChatGPT-3.5 produces the following postcondition:
\newline
\begin{lstlisting}[frame=none]
assert elements[size - 1].equals(e) : "Element was not successfully pushed onto the stack.";
\end{lstlisting}
\vspace{0.3cm}
%To generate a postcondition for this method using an LLM,
%we should also express the formalism in which the postcondition
%must be expressed, e.g., as a JML annotation, or as a Java assert statement.
%If we ask the postcondition to be expressed in JML, ChatGPT-3.5
%will produce the following:
%\newline
%\begin{lstlisting}[frame=none]
%//@ ensures elements[\old(size)] == e;
%\end{lstlisting}
%\vspace{0.3cm}
Note that although the postcondition captures one expected property,
its execution will crash if the pushed element is \texttt{null}.
Similarly, if we ask for the postcondition to be expressed in JML,
the model produces the same postcondition.
However, for less known formalisms or subjects with a more complex API,
using a few-shot prompt,
or even pre-training and fine-tuning the model,
could be more appropriate.
While using a few-shot prompt would
require one to provide a set of examples of the expected contracts
(e.g., a set of methods with their contracts expressed in the desired formalism),
the use of pre-training or fine-tuning would require
one to provide a specific and extensive dataset for contract inference,
including the software element (method or class), the expected contract,
and possibly other contextual information related to the contract
to learn.
%For example, one could provide a prompt including a set of methods
%with their contracts expressed in the desired formalism using
%the following pattern:
%\newline
%\begin{lstlisting}[frame=none]
%METHOD: public E pop() { ... }
%POSTCONDITION: pc_1
%METHOD: public E top() { ... }
%POSTCONDITION: pc_2
%...
%METHOD: public void push(E e) { ... }
%\end{lstlisting}
%\vspace{0.3cm}
%This kind of prompt allows to provide prior examples
%of the expected outcome, and will increase
%the chances of the model to generate the contract
%in the right and desired format.
\subsection{Metamorphic Relations}
Metamorphic relations express
domain-specific properties
of multiple executions of the SUT~\cite{Segura2016}.
Compared to test assertions and contracts,
they are more general oracles that
are easy to define and mantain.
Some examples of metamorphic relations involving two executions are
$p(x,y) = p(y,x)$ for a commutative operation $p$, or
$sort(a) = sort(b)$ for a sorting operation where $a$
is a permutation of $b$.
Since its introduction by Chen et al.~\cite{chen1998},
metamorphic relations
have been successfully used for detecting bugs
in a variety of software systems,
including the search engines Google and Bing~\cite{DBLP:journals/tse/ZhouXC16}
and the Web APIs of Spotify and YouTube~\cite{DBLP:journals/tse/SeguraPTC18}.
Moreover, metamorphic relations have shown to be
complementary to other types of oracles, such as test assertions,
to detect faults in the SUT~\cite{memo2021}.
According to the literature~\cite{Segura2016},
metamorphic relations are expressed through
some formalism that allows to capture the expected
relationship between the inputs and outputs.
Metamorphic relations are typically instantiated in a test case,
where a \emph{source test} (e.g., $a1 = sort([1,3,2])$)
is executed, then a \emph{follow-up test} (e.g., $a2 = sort([2,1,3])$) is executed,
and finally the relation is checked ($a1 = a2$)~\cite{segura2020}.
For the inference of metamorphic oracles,
the use of LLMs has not yet been explored.
Using LLMs to generate metamorphic relations
can be challenging,
mainly because of the domain-specific knowledge required to identify the relations,
and the lack of well-defined formalisms to express them.
A straightforward approach could be enabled in the case that the
SUT already contains a set of test cases, as these
could be used as source tests to ask
an LLM to generate follow-up tests that preserve
some relation with respect to them.
For example, using the zero-shot prompt
``Generate a follow-up test that is equivalent to the following test + test-code''
with the \texttt{testPop} test from Figure~\ref{stack-test},
ChatGPT-3.5 produces a test with exactly the same operations,
and two additional sentences:
\newline
\begin{lstlisting}[frame=none]
stack.push(7);
stack.pop();
\end{lstlisting}
\vspace{0.1cm}
The produced follow-up test is equivalent to the source test in the sense that
both result in the same stack.
It is easy to see that
the metamorphic relation behind states that
for every stack, if we push an element and then pop it, the stack remains the same.
Using both tests, one could easily implement a new test to check the metamorphic relation
by first executing the source test, then the follow-up test, and finally checking that the stacks
are equal.
If a set of test cases is not available,
or one wants to focus on generating the metamorphic
relation itself (instead of its implementation),
one would need to include in the communication with the LLM how the relations should be expressed.
For this scenario, a few-shot prompt or a
pre-training and fine-tuning approach would be necessary.
%
Independently of the kind of oracle we are generating,
it is evident that LLMs have an enormous potential to
assist in oracle automation.
However, as we discuss in the next section,
the use of LLMs for this task not only inherits some
threats from previous techniques
that can affect the quality of the produced oracles,
but also introduce new threats that need to be considered.
\section{Threats to Validity}
\begin{figure*}[t]
\begin{center}
\includegraphics[width=.80\textwidth]{img/assured-oracles.png}
\end{center}
\caption{Overview of LLM-based Oracle Generation.}
\label{llm-oracle-gen}
\end{figure*}
\subsection{Oracle Deficiencies}
The quality of an oracle can be assessed in terms of
its oracle deficiencies: false positives and false negatives~\cite{DBLP:conf/issta/JahangirovaCHT16}.
Intuitively, a \textbf{false positive} is a correct and expected program state
for which the oracle is false, i.e., a false alarm.
A \textbf{false negative} is an incorrect and unexpected program state
for which the oracle is true, i.e., a missed fault.
While false negatives are more tolerable, as one can
still fix the oracle to improve its fault detection capability,
false positives are more critical, as they can lead to
false alarms resulting in unnecessary debugging efforts.
Failing to identify and eliminate false positives may result
in high false positive rates, and raise concerns about
the practical usefulness of the tools \cite{DBLP:conf/sigsoft/HossainFDEV23}.
%and thus
%This aspect needs to be considered when using LLMs
%for oracle automation as well.
%as the produced oracles are not free from oracle deficiencies.
To mitigate this, we consider that assurance mechanisms post-processing
LLM-produced oracles are imperative to provide
guarantees on their quality,
and may be crucial to minimize oracle deficiencies.
Alshahwan et al.~\cite{Alshahwan2024} recently
proposed the notion of Assured LLM-based Software Engeering
(Assured LLMSE) with the aim of providing guarantees on
the output produced by LLMs used for software engineering tasks.
%which is perfectly applicable for LLM-based oracle generation.
Figure~\ref{llm-oracle-gen} shows an overview of
how Assured LLMSE could be applied
to LLM-based oracle generation,
where the LLM-produced oracles are subjected to an assurance process,
possibly analyzing deficiencies.
The assurance process can also provide information to re-prompt the model
with those oracles that do not pass the process.
%As a starting point, it is important to remark that
As LLMs offer absolutely no guarantees on their outcome,
the assurance process could first easily eliminate
syntactically incorrect oracles.
Detecting oracle deficiencies, however,
is more challenging.
False positives detection depends on
the usage context.
If the oracles are for regression testing, i.e.,
to equip the current version of the SUT with oracles
in order to detect regression errors in the future,
to detect false positives one can search for a
reachable program state (test case)
in which the oracle fails.
For example, GAssert~\cite{gassert2020} uses evolutionary
computation to actively search for test cases that
falsify a given assertion oracle.
On the other hand, if the oracles are intended
to test the current implementation, the detection of
false positives may
require a human to decide whether an oracle failure
indicates a false positive or a real bug.
Once false positives are detected,
one can fix the oracle either by removing the parts
that are incorrect, or by weakening the
oracle to make it more general. This could enable
an iterative process, possibly re-prompting the LLM,
in which the oracle is refined
until no more false positives are detected.
This iterative refinement process has already been
applied for postcondition assertions~\cite{gassert2020}.
Although false negatives are less critical,
their detection can help to
strengthen the oracle,
improving its fault detection capability.
As false negatives are essentially faulty program states
missed by the oracle,
mutation analysis can be used to introduce artificial
faults, and then check if the oracle can detect them.
While some approaches use mutation analysis to improve
the oracles during the inference process~\cite{DBLP:conf/issta/FraserZ10, gassert2020,specfuzzer2022},
others use it as an evaluation technique
for fault detection analysis~\cite{memo2021, agora2023}.
Similarly, LLM-based techniques could also benefit from false
negatives analysis.
\subsection{Oracle Leakages}
The use of LLMs for software engineering tasks
has several threats,
including closed-source models,
data leakages between training data and research evaluation,
and reproducibility issues~\cite{Sallou2023}.
Among these, data leakages is particularly
relevant for LLM-based oracle automation,
specifically during the evaluation of these approaches.
As LLMs are trained on very large amounts of data,
often including publicly available code from GitHub~\cite{chen2021evaluating},
there is a risk that the model has memorized some
code samples from the training data~\cite{inan2021training}.
Thus, when evaluating LLM-based techniques for oracle automation,
we need to pay special attention to the data we use
for evaluation, as we may end up obtaining oracles
that are not actually created by the model,
but rather replicated from the training data.
To illustrate this issue, let us consider Defects4J~\cite{defects4j}, one the
most widely-adopted benchmarks in
software testing research.
Many of the projects involved in Defects4J are
publicly available on GitHub. Thus, evaluating LLM-based
oracle generation techniques on Defects4J
can clearly lead to oracle leakages, and make the LLM
provide accurate oracles just because they are
a copy of the oracles from the training data.
%In fact, we asked ChatGPT-3.5 to generate assertions for
%some test prefixes, i.e., the first part of a test case
%without the assertions, from Defects4J, and the model
%replicated the exact test assertions in the original complete test cases.
Figure~\ref{arraystack-collections} shows an example of
a test prefix (i.e., the first part of a test without the
assertions) from the Apache Commons Collections
project in Defects4J, available in revision
\texttt{7c99c62} of the project repository\footnote{\url{https://github.com/apache/commons-collections}},
and the test assertions generated by ChatGPT-3.5.
With the prompt
``Complete the following Java test with test assertions: + test-code'',
ChatGPT-3.5 produces exactly the same assertions
as in the original test case, with the sole difference that
natural language messages to explain the expected
behavior are included.
Moreover, the model uses
the \texttt{search} method from the \texttt{ArrayStack} class
in the assertions,
which availability was not even informed in the initial prompt.
\begin{figure}[t]
\begin{lstlisting}[]
public void testSearch() {
final ArrayStack<E> stack = makeObject();
stack.push((E) "First Item");
stack.push((E) "Second Item");
-----------------------------------------------------------------
// Test searching for existing elements
assertEquals(2, stack.search("First Item")); // First Item is at index 2 from the top
assertEquals(1, stack.search("Second Item")); // Second Item is at index 1 from the top
// Test searching for non-existing element
assertEquals(-1, stack.search("Non-existing Item")); // Non-existing Item is not found in the stack
\end{lstlisting}
\caption{Test from Defects4J, and the assertions produced by ChatGPT-3.5.}
\label{arraystack-collections}
\end{figure}
To mitigate oracle leakages in the evaluation of LLM-based
techniques for oracle automation, one could consider
using evaluation data from multiple sources,
as recommended by Sallou et al.~\cite{Sallou2023}.
SourceForge projects are potentially
a good source of data for evaluation, as they have
been shown that LLMs can have a worse performance
on them compared to GitHub projects~\cite{siddiq2024using}.
Moreover, one could also consider the use of
datasets containing code produced after the models training, to ensure that the evaluation code has not been seen
by the model. For example, GitBug-Java~\cite{silva2024gitbugjava} is a recent benchmark of recent Java bugs
built with 2023 code, which is after the
cut-off date of the training data of most of the notable LLMs,
including OpenAI models.
\section{Conclusion}
Thanks to the ability of LLMs to quickly generate content,
either as code or as specifically formatted text,
they have an enormous potential to improve
software testing tasks.%, including test oracle automation.
In this paper, we discuss the potential of LLMs
for test oracle automation,
%for generating different kinds of oracles typically
%used in software testing,
along with insights to deal with the
challenges present in the use of the LLMs for
inferring different types of oracles.
We also discuss the main threats that arise from automatically
generating oracles using LLMs. Using the LLMs without an
assurance process can result in low quality oracles,
containing a high number of oracle deficiencies, mainly
false positives, which can reduce the trustworthiness of the
testing process. Moreover, new threats can emerge from the use of LLMs,
such as oracle leakages, which can lead to the generation of
oracles that are not actually created by the model, but are rather
replicated from the training data.
We believe that an Assured LLM-based Software Engineering
approach can be a promising direction to mitigate the threats
of LLM-based oracle generation, and to provide guarantees
on the quality of the oracles produced.
\bibliographystyle{ACM-Reference-Format}
\bibliography{main}
\end{document}
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{42}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Alonso et~al\mbox{.}(2023)]%
{agora2023}
\bibfield{author}{\bibinfo{person}{Juan~C. Alonso}, \bibinfo{person}{Sergio
Segura}, {and} \bibinfo{person}{Antonio Ruiz{-}Cort{\'{e}}s}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{AGORA:} Automated Generation of Test Oracles for
{REST} APIs}. In \bibinfo{booktitle}{\emph{Proceedings of the 32nd {ACM}
{SIGSOFT} International Symposium on Software Testing and Analysis, {ISSTA}
2023, Seattle, WA, USA, July 17-21, 2023}},
\bibfield{editor}{\bibinfo{person}{Ren{\'{e}} Just} {and}
\bibinfo{person}{Gordon Fraser}} (Eds.). \bibinfo{publisher}{{ACM}},
\bibinfo{pages}{1018--1030}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3597926.3598114}
\showDOI{\tempurl}
\bibitem[Alshahwan et~al\mbox{.}(2024)]%
{Alshahwan2024}
\bibfield{author}{\bibinfo{person}{Nadia Alshahwan}, \bibinfo{person}{Mark
Harman}, \bibinfo{person}{Inna Harper}, \bibinfo{person}{Alexandru
Marginean}, \bibinfo{person}{Shubho Sengupta}, {and} \bibinfo{person}{Eddy
Wang}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Assured LLM-Based Software Engineering}.
\newblock \bibinfo{journal}{\emph{CoRR}} \bibinfo{volume}{abs/2402.04380}
(\bibinfo{year}{2024}).
\newblock
\urldef\tempurl%
\url{https://doi.org/10.48550/ARXIV.2402.04380}
\showDOI{\tempurl}
\showeprint[arXiv]{2402.04380}
\bibitem[Astorga et~al\mbox{.}(2021)]%
{precis2021}
\bibfield{author}{\bibinfo{person}{Angello Astorga},
\bibinfo{person}{Shambwaditya Saha}, \bibinfo{person}{Ahmad Dinkins},
\bibinfo{person}{Felicia Wang}, \bibinfo{person}{P. Madhusudan}, {and}
\bibinfo{person}{Tao Xie}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Synthesizing contracts correct modulo a test
generator}.
\newblock \bibinfo{journal}{\emph{Proc. {ACM} Program. Lang.}}
\bibinfo{volume}{5}, \bibinfo{number}{{OOPSLA}} (\bibinfo{year}{2021}),
\bibinfo{pages}{1--27}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3485481}
\showDOI{\tempurl}
\bibitem[Barr et~al\mbox{.}(2015)]%
{Barr+2015}
\bibfield{author}{\bibinfo{person}{Earl~T. Barr}, \bibinfo{person}{Mark
Harman}, \bibinfo{person}{Phil McMinn}, \bibinfo{person}{Muzammil Shahbaz},
{and} \bibinfo{person}{Shin Yoo}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{The Oracle Problem in Software Testing: {A}
Survey}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Software Eng.}}
\bibinfo{volume}{41}, \bibinfo{number}{5} (\bibinfo{year}{2015}),
\bibinfo{pages}{507--525}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/TSE.2014.2372785}
\showDOI{\tempurl}
\bibitem[Blasi et~al\mbox{.}(2021)]%
{memo2021}
\bibfield{author}{\bibinfo{person}{Arianna Blasi}, \bibinfo{person}{Alessandra
Gorla}, \bibinfo{person}{Michael~D. Ernst}, \bibinfo{person}{Mauro
Pezz{\`{e}}}, {and} \bibinfo{person}{Antonio Carzaniga}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{MeMo: Automatically identifying metamorphic
relations in Javadoc comments for test automation}.
\newblock \bibinfo{journal}{\emph{J. Syst. Softw.}} \bibinfo{volume}{181}
(\bibinfo{year}{2021}), \bibinfo{pages}{111041}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1016/j.jss.2021.111041}
\showDOI{\tempurl}
\bibitem[Chalin et~al\mbox{.}(2005)]%
{DBLP:conf/fmco/ChalinKLP05}
\bibfield{author}{\bibinfo{person}{Patrice Chalin}, \bibinfo{person}{Joseph~R.
Kiniry}, \bibinfo{person}{Gary~T. Leavens}, {and} \bibinfo{person}{Erik
Poll}.} \bibinfo{year}{2005}\natexlab{}.
\newblock \showarticletitle{Beyond Assertions: Advanced Specification and
Verification with {JML} and ESC/Java2}. In \bibinfo{booktitle}{\emph{Formal
Methods for Components and Objects, 4th International Symposium, {FMCO} 2005,
Amsterdam, The Netherlands, November 1-4, 2005, Revised Lectures}}.
\bibinfo{pages}{342--363}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1007/11804192_16}
\showDOI{\tempurl}
\bibitem[Chen et~al\mbox{.}(2021)]%
{chen2021evaluating}
\bibfield{author}{\bibinfo{person}{Mark Chen}, \bibinfo{person}{Jerry Tworek},
\bibinfo{person}{Heewoo Jun}, \bibinfo{person}{Qiming Yuan},
\bibinfo{person}{Henrique~Ponde de Oliveira~Pinto}, \bibinfo{person}{Jared
Kaplan}, \bibinfo{person}{Harri Edwards}, \bibinfo{person}{Yuri Burda},
\bibinfo{person}{Nicholas Joseph}, {and} \bibinfo{person}{Greg~Brockman et
al.}} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Evaluating Large Language Models Trained on Code}.
\newblock (\bibinfo{year}{2021}).
\newblock
\showeprint[arxiv]{2107.03374}~[cs.LG]
\bibitem[Chen et~al\mbox{.}(1998)]%
{chen1998}
\bibfield{author}{\bibinfo{person}{Tsong~Y Chen}, \bibinfo{person}{Shing~C
Cheung}, {and} \bibinfo{person}{Shiu~Ming Yiu}.}
\bibinfo{year}{1998}\natexlab{}.
\newblock \showarticletitle{Metamorphic testing: a new approach for generating
next test cases}. In \bibinfo{booktitle}{\emph{Technical Report
HKUST-CS98-01, Department of Computer Science, Hong Kong University of
Science and Technology, Hong Kong, 1998.}}
\newblock
\bibitem[Dinella et~al\mbox{.}(2022)]%
{toga2022}
\bibfield{author}{\bibinfo{person}{Elizabeth Dinella}, \bibinfo{person}{Gabriel
Ryan}, \bibinfo{person}{Todd Mytkowicz}, {and} \bibinfo{person}{Shuvendu~K.
Lahiri}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{{TOGA:} {A} Neural Method for Test Oracle
Generation}. In \bibinfo{booktitle}{\emph{44th {IEEE/ACM} 44th International
Conference on Software Engineering, {ICSE} 2022, Pittsburgh, PA, USA, May
25-27, 2022}}. \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{2130--2141}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3510003.3510141}
\showDOI{\tempurl}
\bibitem[Ernst et~al\mbox{.}(2007)]%
{DBLP:journals/scp/ErnstPGMPTX07}
\bibfield{author}{\bibinfo{person}{Michael~D. Ernst}, \bibinfo{person}{Jeff~H.
Perkins}, \bibinfo{person}{Philip~J. Guo}, \bibinfo{person}{Stephen
McCamant}, \bibinfo{person}{Carlos Pacheco}, \bibinfo{person}{Matthew~S.
Tschantz}, {and} \bibinfo{person}{Chen Xiao}.}
\bibinfo{year}{2007}\natexlab{}.
\newblock \showarticletitle{The Daikon system for dynamic detection of likely
invariants}.
\newblock \bibinfo{journal}{\emph{Sci. Comput. Program.}} \bibinfo{volume}{69},
\bibinfo{number}{1-3} (\bibinfo{year}{2007}), \bibinfo{pages}{35--45}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1016/j.scico.2007.01.015}
\showDOI{\tempurl}
\bibitem[Fan et~al\mbox{.}(2023)]%
{DBLP:conf/icse/FanGMRT23}
\bibfield{author}{\bibinfo{person}{Zhiyu Fan}, \bibinfo{person}{Xiang Gao},
\bibinfo{person}{Martin Mirchev}, \bibinfo{person}{Abhik Roychoudhury}, {and}
\bibinfo{person}{Shin~Hwei Tan}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Automated Repair of Programs from Large Language
Models}. In \bibinfo{booktitle}{\emph{45th {IEEE/ACM} International
Conference on Software Engineering, {ICSE} 2023, Melbourne, Australia, May
14-20, 2023}}. \bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{1469--1481}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00128}
\showDOI{\tempurl}
\bibitem[Fraser and Zeller(2010)]%
{DBLP:conf/issta/FraserZ10}
\bibfield{author}{\bibinfo{person}{Gordon Fraser} {and}
\bibinfo{person}{Andreas Zeller}.} \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{Mutation-driven generation of unit tests and
oracles}. In \bibinfo{booktitle}{\emph{{ISSTA}}}. \bibinfo{publisher}{{ACM}},
\bibinfo{pages}{147--158}.
\newblock
\bibitem[Hossain et~al\mbox{.}(2023)]%
{DBLP:conf/sigsoft/HossainFDEV23}
\bibfield{author}{\bibinfo{person}{Soneya~Binta Hossain},
\bibinfo{person}{Antonio Filieri}, \bibinfo{person}{Matthew~B. Dwyer},
\bibinfo{person}{Sebastian~G. Elbaum}, {and} \bibinfo{person}{Willem
Visser}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Neural-Based Test Oracle Generation: {A}
Large-Scale Evaluation and Lessons Learned}. In
\bibinfo{booktitle}{\emph{Proceedings of the 31st {ACM} Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering, {ESEC/FSE} 2023, San Francisco, CA, USA, December 3-9, 2023}},
\bibfield{editor}{\bibinfo{person}{Satish Chandra}, \bibinfo{person}{Kelly
Blincoe}, {and} \bibinfo{person}{Paolo Tonella}} (Eds.).
\bibinfo{publisher}{{ACM}}, \bibinfo{pages}{120--132}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3611643.3616265}
\showDOI{\tempurl}
\bibitem[Inan et~al\mbox{.}(2021)]%
{inan2021training}
\bibfield{author}{\bibinfo{person}{Huseyin~A. Inan}, \bibinfo{person}{Osman
Ramadan}, \bibinfo{person}{Lukas Wutschitz}, \bibinfo{person}{Daniel Jones},
\bibinfo{person}{Victor Rühle}, \bibinfo{person}{James Withers}, {and}
\bibinfo{person}{Robert Sim}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Training Data Leakage Analysis in Language Models}.
\newblock (\bibinfo{year}{2021}).
\newblock
\showeprint[arxiv]{2101.05405}~[cs.CR]
\bibitem[Jahangirova et~al\mbox{.}(2016)]%
{DBLP:conf/issta/JahangirovaCHT16}
\bibfield{author}{\bibinfo{person}{Gunel Jahangirova}, \bibinfo{person}{David
Clark}, \bibinfo{person}{Mark Harman}, {and} \bibinfo{person}{Paolo
Tonella}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Test oracle assessment and improvement}. In
\bibinfo{booktitle}{\emph{Proceedings of the 25th International Symposium on
Software Testing and Analysis, {ISSTA} 2016, Saarbr{\"{u}}cken, Germany, July
18-20, 2016}}, \bibfield{editor}{\bibinfo{person}{Andreas Zeller} {and}
\bibinfo{person}{Abhik Roychoudhury}} (Eds.). \bibinfo{publisher}{{ACM}},
\bibinfo{pages}{247--258}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/2931037.2931062}
\showDOI{\tempurl}
\bibitem[Jiang et~al\mbox{.}(2023)]%
{DBLP:conf/icse/JiangLLT23}
\bibfield{author}{\bibinfo{person}{Nan Jiang}, \bibinfo{person}{Kevin Liu},
\bibinfo{person}{Thibaud Lutellier}, {and} \bibinfo{person}{Lin Tan}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Impact of Code Language Models on Automated Program
Repair}. In \bibinfo{booktitle}{\emph{45th {IEEE/ACM} International
Conference on Software Engineering, {ICSE} 2023, Melbourne, Australia, May
14-20, 2023}}. \bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{1430--1442}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00125}
\showDOI{\tempurl}
\bibitem[Just et~al\mbox{.}(2014)]%
{defects4j}
\bibfield{author}{\bibinfo{person}{Ren{\'{e}} Just}, \bibinfo{person}{Darioush
Jalali}, {and} \bibinfo{person}{Michael~D. Ernst}.}
\bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Defects4J: a database of existing faults to enable
controlled testing studies for Java programs}. In
\bibinfo{booktitle}{\emph{International Symposium on Software Testing and
Analysis, {ISSTA} '14, San Jose, CA, {USA} - July 21 - 26, 2014}},
\bibfield{editor}{\bibinfo{person}{Corina~S. Pasareanu} {and}
\bibinfo{person}{Darko Marinov}} (Eds.). \bibinfo{publisher}{{ACM}},
\bibinfo{pages}{437--440}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/2610384.2628055}
\showDOI{\tempurl}
\bibitem[Lemieux et~al\mbox{.}(2023)]%
{codamosa2023}
\bibfield{author}{\bibinfo{person}{Caroline Lemieux},
\bibinfo{person}{Jeevana~Priya Inala}, \bibinfo{person}{Shuvendu~K. Lahiri},
{and} \bibinfo{person}{Siddhartha Sen}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{CodaMosa: Escaping Coverage Plateaus in Test
Generation with Pre-trained Large Language Models}. In
\bibinfo{booktitle}{\emph{45th {IEEE/ACM} International Conference on
Software Engineering, {ICSE} 2023, Melbourne, Australia, May 14-20, 2023}}.
\bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{919--931}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00085}
\showDOI{\tempurl}
\bibitem[Lewis et~al\mbox{.}(2019)]%
{DBLP:journals/corr/abs-1910-13461}
\bibfield{author}{\bibinfo{person}{Mike Lewis}, \bibinfo{person}{Yinhan Liu},
\bibinfo{person}{Naman Goyal}, \bibinfo{person}{Marjan Ghazvininejad},
\bibinfo{person}{Abdelrahman Mohamed}, \bibinfo{person}{Omer Levy},
\bibinfo{person}{Veselin Stoyanov}, {and} \bibinfo{person}{Luke
Zettlemoyer}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{BART:} Denoising Sequence-to-Sequence Pre-training
for Natural Language Generation, Translation, and Comprehension}.
\newblock \bibinfo{journal}{\emph{CoRR}} \bibinfo{volume}{abs/1910.13461}
(\bibinfo{year}{2019}).
\newblock
\showeprint[arXiv]{1910.13461}
\urldef\tempurl%
\url{http://arxiv.org/abs/1910.13461}
\showURL{%
\tempurl}
\bibitem[Lu et~al\mbox{.}(2021)]%
{lu2021codexglue}
\bibfield{author}{\bibinfo{person}{Shuai Lu}, \bibinfo{person}{Daya Guo},
\bibinfo{person}{Shuo Ren}, \bibinfo{person}{Junjie Huang},
\bibinfo{person}{Alexey Svyatkovskiy}, \bibinfo{person}{Ambrosio Blanco},
\bibinfo{person}{Colin Clement}, \bibinfo{person}{Dawn Drain},
\bibinfo{person}{Daxin Jiang}, \bibinfo{person}{Duyu Tang},
\bibinfo{person}{Ge Li}, \bibinfo{person}{Lidong Zhou},
\bibinfo{person}{Linjun Shou}, \bibinfo{person}{Long Zhou},
\bibinfo{person}{Michele Tufano}, \bibinfo{person}{Ming Gong},
\bibinfo{person}{Ming Zhou}, \bibinfo{person}{Nan Duan},
\bibinfo{person}{Neel Sundaresan}, \bibinfo{person}{Shao~Kun Deng},
\bibinfo{person}{Shengyu Fu}, {and} \bibinfo{person}{Shujie Liu}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{CodeXGLUE: A Machine Learning Benchmark Dataset for
Code Understanding and Generation}.
\newblock (\bibinfo{year}{2021}).
\newblock
\showeprint[arxiv]{2102.04664}~[cs.SE]
\bibitem[Meyer(1992)]%
{DBLP:journals/computer/Meyer92}
\bibfield{author}{\bibinfo{person}{Bertrand Meyer}.}
\bibinfo{year}{1992}\natexlab{}.
\newblock \showarticletitle{Applying "Design by Contract"}.
\newblock \bibinfo{journal}{\emph{Computer}} \bibinfo{volume}{25},
\bibinfo{number}{10} (\bibinfo{year}{1992}), \bibinfo{pages}{40--51}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/2.161279}
\showDOI{\tempurl}
\bibitem[Molina et~al\mbox{.}(2022)]%
{specfuzzer2022}
\bibfield{author}{\bibinfo{person}{Facundo Molina}, \bibinfo{person}{Marcelo
d'Amorim}, {and} \bibinfo{person}{Nazareno Aguirre}.}
\bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Fuzzing Class Specifications}. In
\bibinfo{booktitle}{\emph{44th {IEEE/ACM} 44th International Conference on
Software Engineering, {ICSE} 2022, Pittsburgh, PA, USA, May 25-27, 2022}}.
\bibinfo{publisher}{{ACM}}, \bibinfo{pages}{1008--1020}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3510003.3510120}
\showDOI{\tempurl}
\bibitem[Molina et~al\mbox{.}(2021)]%
{evospex2021}
\bibfield{author}{\bibinfo{person}{Facundo Molina}, \bibinfo{person}{Pablo
Ponzio}, \bibinfo{person}{Nazareno Aguirre}, {and}
\bibinfo{person}{Marcelo~F. Frias}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{EvoSpex: An Evolutionary Algorithm for Learning
Postconditions}. In \bibinfo{booktitle}{\emph{43rd {IEEE/ACM} International
Conference on Software Engineering, {ICSE} 2021, Madrid, Spain, 22-30 May
2021}}. \bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{1223--1235}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE43902.2021.00112}
\showDOI{\tempurl}
\bibitem[Nashid et~al\mbox{.}(2023)]%
{DBLP:conf/icse/NashidSM23}
\bibfield{author}{\bibinfo{person}{Noor Nashid}, \bibinfo{person}{Mifta
Sintaha}, {and} \bibinfo{person}{Ali Mesbah}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Retrieval-Based Prompt Selection for Code-Related
Few-Shot Learning}. In \bibinfo{booktitle}{\emph{45th {IEEE/ACM}
International Conference on Software Engineering, {ICSE} 2023, Melbourne,
Australia, May 14-20, 2023}}. \bibinfo{publisher}{{IEEE}},
\bibinfo{pages}{2450--2462}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00205}
\showDOI{\tempurl}
\bibitem[Nie et~al\mbox{.}(2023)]%
{teco2023}
\bibfield{author}{\bibinfo{person}{Pengyu Nie}, \bibinfo{person}{Rahul
Banerjee}, \bibinfo{person}{Junyi~Jessy Li}, \bibinfo{person}{Raymond~J.
Mooney}, {and} \bibinfo{person}{Milos Gligoric}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Learning Deep Semantics for Test Completion}. In
\bibinfo{booktitle}{\emph{45th {IEEE/ACM} International Conference on
Software Engineering, {ICSE} 2023, Melbourne, Australia, May 14-20, 2023}}.
\bibinfo{publisher}{{IEEE}}, \bibinfo{pages}{2111--2123}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00178}
\showDOI{\tempurl}
\bibitem[{OpenAI}(2024)]%
{openai}
\bibfield{author}{\bibinfo{person}{{OpenAI}}.} \bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{OpenAI}}.
\newblock
\urldef\tempurl%
\url{https://openai.com/}
\showURL{%
\tempurl}
\newblock
\shownote{Accessed on March 13th, 2024}.
\bibitem[Rao et~al\mbox{.}(2023)]%
{cat-lm-2023}
\bibfield{author}{\bibinfo{person}{Nikitha Rao}, \bibinfo{person}{Kush Jain},
\bibinfo{person}{Uri Alon}, \bibinfo{person}{Claire~Le Goues}, {and}
\bibinfo{person}{Vincent~J. Hellendoorn}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{{CAT-LM} Training Language Models on Aligned Code
And Tests}. In \bibinfo{booktitle}{\emph{38th {IEEE/ACM} International
Conference on Automated Software Engineering, {ASE} 2023, Luxembourg,
September 11-15, 2023}}. \bibinfo{publisher}{{IEEE}},
\bibinfo{pages}{409--420}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ASE56229.2023.00193}
\showDOI{\tempurl}
\bibitem[Sallou et~al\mbox{.}(2023)]%
{Sallou2023}
\bibfield{author}{\bibinfo{person}{June Sallou}, \bibinfo{person}{Thomas
Durieux}, {and} \bibinfo{person}{Annibale Panichella}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Breaking the Silence: the Threats of Using LLMs in
Software Engineering}.
\newblock \bibinfo{journal}{\emph{CoRR}} \bibinfo{volume}{abs/2312.08055}
(\bibinfo{year}{2023}).
\newblock
\urldef\tempurl%
\url{https://doi.org/10.48550/ARXIV.2312.08055}
\showDOI{\tempurl}
\showeprint[arXiv]{2312.08055}
\bibitem[Sch{\"{a}}fer et~al\mbox{.}(2024)]%
{DBLP:journals/tse/SchaferNET24}
\bibfield{author}{\bibinfo{person}{Max Sch{\"{a}}fer}, \bibinfo{person}{Sarah
Nadi}, \bibinfo{person}{Aryaz Eghbali}, {and} \bibinfo{person}{Frank Tip}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{An Empirical Evaluation of Using Large Language
Models for Automated Unit Test Generation}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Software Eng.}}
\bibinfo{volume}{50}, \bibinfo{number}{1} (\bibinfo{year}{2024}),
\bibinfo{pages}{85--105}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/TSE.2023.3334955}
\showDOI{\tempurl}
\bibitem[Segura et~al\mbox{.}(2016)]%
{Segura2016}
\bibfield{author}{\bibinfo{person}{Sergio Segura}, \bibinfo{person}{Gordon
Fraser}, \bibinfo{person}{Ana~Bel{\'{e}}n S{\'{a}}nchez}, {and}
\bibinfo{person}{Antonio~Ruiz Cort{\'{e}}s}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{A Survey on Metamorphic Testing}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Software Eng.}}
\bibinfo{volume}{42}, \bibinfo{number}{9} (\bibinfo{year}{2016}),
\bibinfo{pages}{805--824}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/TSE.2016.2532875}
\showDOI{\tempurl}
\bibitem[Segura et~al\mbox{.}(2018)]%
{DBLP:journals/tse/SeguraPTC18}
\bibfield{author}{\bibinfo{person}{Sergio Segura},
\bibinfo{person}{Jos{\'{e}}~Antonio Parejo}, \bibinfo{person}{Javier Troya},
{and} \bibinfo{person}{Antonio~Ruiz Cort{\'{e}}s}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Metamorphic Testing of RESTful Web APIs}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Software Eng.}}
\bibinfo{volume}{44}, \bibinfo{number}{11} (\bibinfo{year}{2018}),
\bibinfo{pages}{1083--1099}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/TSE.2017.2764464}
\showDOI{\tempurl}
\bibitem[Segura et~al\mbox{.}(2020)]%
{segura2020}
\bibfield{author}{\bibinfo{person}{Sergio Segura}, \bibinfo{person}{Dave
Towey}, \bibinfo{person}{Zhi~Quan Zhou}, {and} \bibinfo{person}{Tsong~Yueh
Chen}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Metamorphic Testing: Testing the Untestable}.
\newblock \bibinfo{journal}{\emph{{IEEE} Softw.}} \bibinfo{volume}{37},
\bibinfo{number}{3} (\bibinfo{year}{2020}), \bibinfo{pages}{46--53}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/MS.2018.2875968}
\showDOI{\tempurl}
\bibitem[Siddiq et~al\mbox{.}(2024)]%
{siddiq2024using}
\bibfield{author}{\bibinfo{person}{Mohammed~Latif Siddiq},
\bibinfo{person}{Joanna C.~S. Santos}, \bibinfo{person}{Ridwanul~Hasan
Tanvir}, \bibinfo{person}{Noshin Ulfat}, \bibinfo{person}{Fahmid~Al Rifat},
{and} \bibinfo{person}{Vinicius~Carvalho Lopes}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{Using Large Language Models to Generate JUnit
Tests: An Empirical Study}.
\newblock (\bibinfo{year}{2024}).
\newblock
\showeprint[arxiv]{2305.00418}~[cs.SE]
\bibitem[Silva et~al\mbox{.}(2024)]%
{silva2024gitbugjava}
\bibfield{author}{\bibinfo{person}{André Silva}, \bibinfo{person}{Nuno
Saavedra}, {and} \bibinfo{person}{Martin Monperrus}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \showarticletitle{GitBug-Java: A Reproducible Benchmark of Recent
Java Bugs}.
\newblock (\bibinfo{year}{2024}).
\newblock
\showeprint[arxiv]{2402.02961}~[cs.SE]
\bibitem[Terragni et~al\mbox{.}(2020)]%
{gassert2020}
\bibfield{author}{\bibinfo{person}{Valerio Terragni}, \bibinfo{person}{Gunel
Jahangirova}, \bibinfo{person}{Paolo Tonella}, {and} \bibinfo{person}{Mauro
Pezz\`{e}}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Evolutionary Improvement of Assertion Oracles}. In
\bibinfo{booktitle}{\emph{Proceedings of the 28th ACM Joint Meeting on
European Software Engineering Conference and Symposium on the Foundations of
Software Engineering}} (Virtual Event, USA) \emph{(\bibinfo{series}{ESEC/FSE
2020})}. \bibinfo{publisher}{Association for Computing Machinery},
\bibinfo{address}{New York, NY, USA}, \bibinfo{pages}{1178–1189}.
\newblock
\showISBNx{9781450370431}
\urldef\tempurl%
\url{https://doi.org/10.1145/3368089.3409758}
\showDOI{\tempurl}
\bibitem[Touvron et~al\mbox{.}(2023)]%
{llama2}
\bibfield{author}{\bibinfo{person}{Hugo Touvron}, \bibinfo{person}{Louis
Martin}, \bibinfo{person}{Kevin Stone}, \bibinfo{person}{Peter Albert},
\bibinfo{person}{Amjad Almahairi}, \bibinfo{person}{Yasmine Babaei},
\bibinfo{person}{Nikolay Bashlykov}, \bibinfo{person}{Soumya Batra},
\bibinfo{person}{Prajjwal Bhargava}, \bibinfo{person}{Shruti Bhosale},
\bibinfo{person}{Dan Bikel}, \bibinfo{person}{Lukas Blecher},
\bibinfo{person}{Cristian Canton{-}Ferrer}, \bibinfo{person}{Moya Chen},
\bibinfo{person}{Guillem Cucurull}, \bibinfo{person}{David Esiobu},
\bibinfo{person}{Jude Fernandes}, \bibinfo{person}{Jeremy Fu},
\bibinfo{person}{Wenyin Fu}, \bibinfo{person}{Brian Fuller},
\bibinfo{person}{Cynthia Gao}, \bibinfo{person}{Vedanuj Goswami},
\bibinfo{person}{Naman Goyal}, \bibinfo{person}{Anthony Hartshorn},
\bibinfo{person}{Saghar Hosseini}, \bibinfo{person}{Rui Hou},
\bibinfo{person}{Hakan Inan}, \bibinfo{person}{Marcin Kardas},
\bibinfo{person}{Viktor Kerkez}, \bibinfo{person}{Madian Khabsa},
\bibinfo{person}{Isabel Kloumann}, \bibinfo{person}{Artem Korenev},
\bibinfo{person}{Punit~Singh Koura}, \bibinfo{person}{Marie{-}Anne Lachaux},
\bibinfo{person}{Thibaut Lavril}, \bibinfo{person}{Jenya Lee},
\bibinfo{person}{Diana Liskovich}, \bibinfo{person}{Yinghai Lu},
\bibinfo{person}{Yuning Mao}, \bibinfo{person}{Xavier Martinet},
\bibinfo{person}{Todor Mihaylov}, \bibinfo{person}{Pushkar Mishra},
\bibinfo{person}{Igor Molybog}, \bibinfo{person}{Yixin Nie},
\bibinfo{person}{Andrew Poulton}, \bibinfo{person}{Jeremy Reizenstein},
\bibinfo{person}{Rashi Rungta}, \bibinfo{person}{Kalyan Saladi},
\bibinfo{person}{Alan Schelten}, \bibinfo{person}{Ruan Silva},
\bibinfo{person}{Eric~Michael Smith}, \bibinfo{person}{Ranjan Subramanian},
\bibinfo{person}{Xiaoqing~Ellen Tan}, \bibinfo{person}{Binh Tang},
\bibinfo{person}{Ross Taylor}, \bibinfo{person}{Adina Williams},
\bibinfo{person}{Jian~Xiang Kuan}, \bibinfo{person}{Puxin Xu},
\bibinfo{person}{Zheng Yan}, \bibinfo{person}{Iliyan Zarov},
\bibinfo{person}{Yuchen Zhang}, \bibinfo{person}{Angela Fan},
\bibinfo{person}{Melanie Kambadur}, \bibinfo{person}{Sharan Narang},
\bibinfo{person}{Aur{\'{e}}lien Rodriguez}, \bibinfo{person}{Robert Stojnic},
\bibinfo{person}{Sergey Edunov}, {and} \bibinfo{person}{Thomas Scialom}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Llama 2: Open Foundation and Fine-Tuned Chat
Models}.
\newblock \bibinfo{journal}{\emph{CoRR}} \bibinfo{volume}{abs/2307.09288}
(\bibinfo{year}{2023}).
\newblock
\urldef\tempurl%
\url{https://doi.org/10.48550/arXiv.2307.09288}
\showDOI{\tempurl}
\showeprint[arXiv]{2307.09288}
\bibitem[Tufano et~al\mbox{.}(2022)]%
{DBLP:conf/ast/TufanoDSS22}
\bibfield{author}{\bibinfo{person}{Michele Tufano}, \bibinfo{person}{Dawn
Drain}, \bibinfo{person}{Alexey Svyatkovskiy}, {and} \bibinfo{person}{Neel
Sundaresan}.} \bibinfo{year}{2022}\natexlab{}.
\newblock \showarticletitle{Generating Accurate Assert Statements for Unit Test
Cases using Pretrained Transformers}. In \bibinfo{booktitle}{\emph{{IEEE/ACM}
International Conference on Automation of Software Test, AST@ICSE 2022,
Pittsburgh, PA, USA, May 21-22, 2022}}. \bibinfo{publisher}{{ACM/IEEE}},
\bibinfo{pages}{54--64}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/3524481.3527220}
\showDOI{\tempurl}
\bibitem[Wang et~al\mbox{.}(2024)]%
{testing-llms-2024}
\bibfield{author}{\bibinfo{person}{Junjie Wang}, \bibinfo{person}{Yuchao
Huang}, \bibinfo{person}{Chunyang Chen}, \bibinfo{person}{Zhe Liu},
\bibinfo{person}{Song Wang}, {and} \bibinfo{person}{Qing Wang}.}
\bibinfo{year}{2024}\natexlab{}.
\newblock \bibinfo{title}{Software Testing with Large Language Models: Survey,
Landscape, and Vision}.
\newblock
\newblock
\showeprint[arxiv]{2307.07221}~[cs.SE]
\bibitem[Wang et~al\mbox{.}(2021)]%
{DBLP:conf/emnlp/0034WJH21}
\bibfield{author}{\bibinfo{person}{Yue Wang}, \bibinfo{person}{Weishi Wang},
\bibinfo{person}{Shafiq~R. Joty}, {and} \bibinfo{person}{Steven C.~H. Hoi}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{CodeT5: Identifier-aware Unified Pre-trained
Encoder-Decoder Models for Code Understanding and Generation}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2021 Conference on Empirical
Methods in Natural Language Processing, {EMNLP} 2021, Virtual Event / Punta
Cana, Dominican Republic, 7-11 November, 2021}},
\bibfield{editor}{\bibinfo{person}{Marie{-}Francine Moens},
\bibinfo{person}{Xuanjing Huang}, \bibinfo{person}{Lucia Specia}, {and}
\bibinfo{person}{Scott~Wen{-}tau Yih}} (Eds.).
\bibinfo{publisher}{Association for Computational Linguistics},
\bibinfo{pages}{8696--8708}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.18653/V1/2021.EMNLP-MAIN.685}
\showDOI{\tempurl}
\bibitem[Wei et~al\mbox{.}(2011)]%
{autoinfer2011}
\bibfield{author}{\bibinfo{person}{Yi Wei}, \bibinfo{person}{Carlo~A. Furia},
\bibinfo{person}{Nikolay Kazmin}, {and} \bibinfo{person}{Bertrand Meyer}.}
\bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Inferring better contracts}. In
\bibinfo{booktitle}{\emph{Proceedings of the 33rd International Conference on
Software Engineering, {ICSE} 2011, Waikiki, Honolulu , HI, USA, May 21-28,
2011}}, \bibfield{editor}{\bibinfo{person}{Richard~N. Taylor},
\bibinfo{person}{Harald~C. Gall}, {and} \bibinfo{person}{Nenad Medvidovic}}
(Eds.). \bibinfo{publisher}{{ACM}}, \bibinfo{pages}{191--200}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1145/1985793.1985820}
\showDOI{\tempurl}
\bibitem[Xia et~al\mbox{.}(2023)]%
{DBLP:conf/icse/XiaWZ23}
\bibfield{author}{\bibinfo{person}{Chunqiu~Steven Xia},
\bibinfo{person}{Yuxiang Wei}, {and} \bibinfo{person}{Lingming Zhang}.}
\bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{Automated Program Repair in the Era of Large
Pre-trained Language Models}. In \bibinfo{booktitle}{\emph{45th {IEEE/ACM}
International Conference on Software Engineering, {ICSE} 2023, Melbourne,
Australia, May 14-20, 2023}}. \bibinfo{publisher}{{IEEE}},
\bibinfo{pages}{1482--1494}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/ICSE48619.2023.00129}
\showDOI{\tempurl}
\bibitem[Zhou et~al\mbox{.}(2016)]%
{DBLP:journals/tse/ZhouXC16}
\bibfield{author}{\bibinfo{person}{Zhiquan Zhou}, \bibinfo{person}{Shaowen
Xiang}, {and} \bibinfo{person}{Tsong~Yueh Chen}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Metamorphic Testing for Software Quality
Assessment: {A} Study of Search Engines}.
\newblock \bibinfo{journal}{\emph{{IEEE} Trans. Software Eng.}}
\bibinfo{volume}{42}, \bibinfo{number}{3} (\bibinfo{year}{2016}),
\bibinfo{pages}{264--284}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1109/TSE.2015.2478001}
\showDOI{\tempurl}
\end{thebibliography}