Accelerating Patch Validation for Program Repair with Interception-Based Execution Scheduling
Extracted source text · arXiv 2305.03955
\pdfoutput=1
\documentclass[10pt,journal,compsoc]{IEEEtran}
\usepackage{colortbl}
\usepackage{balance}
\usepackage{xspace}
\usepackage[numbers]{natbib}
\usepackage[colorlinks,allcolors=violet]{hyperref}
\usepackage{booktabs}
\usepackage{bbding} % for \Checkmark
% fuck this `minted` package, i debugged the problem with \balance for hours!
% see https://tex.stackexchange.com/questions/246892/need-help-balancing-the-columns-of-references-using-two-column-ieeetran-on-last for fucking more info
\makeatletter
\@namedef{ver@lineno.sty}{9999/12/31}
\@namedef{opt@lineno.sty}{}
\makeatother
\usepackage[frozencache,cachedir=minted-output]{minted}
%\usepackage{comment} % for \begin{comment}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{multicol, multirow}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[font=footnotesize]{caption,subcaption}
\usepackage{tablefootnote}
\usepackage{tcolorbox}
\usepackage{xcolor}
\usepackage[normalem]{ulem}
%\usepackage{alltt}
%\usepackage[nounderscore]{syntax}
\usepackage{scrextend}
\usepackage{mathtools}
\usepackage{proof}
\usepackage{setspace}
%\newcommand{\todo}[1]{\textcolor{red}{#1}}
\newcommand{\ourtoolraw}{ExpressAPR}
\newcommand{\ourtool}{\ourtoolraw{}\xspace}
\newcommand{\accOverPlain}{137.1}
\newcommand{\accOverSota}{8.8}
\newcommand{\ted}{mutant deduplication\xspace}
\newcommand{\Ted}{Mutant deduplication\xspace}
\newcommand{\oee}{optimized execution engine\xspace}
\newcommand{\Oee}{Optimized execution engine\xspace}
\newcommand{\napp}{execution scheduling\xspace}
\newcommand{\Napp}{Execution scheduling\xspace}
\newcommand{\NApp}{Execution Scheduling\xspace}
\newcommand{\napper}{execution scheduler\xspace}
\newcommand{\NApper}{Execution Scheduler\xspace}
\newcommand{\tree}{state-transition tree\xspace}
\newcommand{\Tree}{State-transition Tree\xspace}
\newcommand{\eins}{interception-based instrumentation\xspace}
\newcommand{\eIns}{interception-based Instrumentation\xspace}
\newcommand{\EIns}{Interception-based Instrumentation\xspace}
\input{macros}
\begin{document}
\title{Accelerating Patch Validation for Program Repair \\{\LARGE with Interception-Based \NApp}}
\author{%
Yuan-An Xiao, % xiaoyuanan@pku.edu.cn
Chenyang Yang, % chenyangy@pku.edu.cn
Bo Wang, % wangbo_cs@bjtu.edu.cn
and Yingfei Xiong % xiongyf@pku.edu.cn
}
%\markboth{(journal)}%
%{Accelerating Patch Validation for Program Repair}
\IEEEtitleabstractindextext{
\begin{abstract}
Long patch validation time is a limiting factor for automated program repair (APR). Though the duality between patch validation and mutation testing is recognized, so far there exists no study of systematically adapting mutation testing techniques to general-purpose patch validation. To address this gap, we investigate existing mutation testing techniques and recognize five classes of acceleration techniques that are suitable for general-purpose patch validation. Among them, mutant schemata and \ted have not been adapted to general-purpose patch validation due to the arbitrary changes that third-party APR approaches may introduce. This presents two problems for adaption: 1) the difficulty of implementing the static equivalence analysis required by the state-of-the-art \ted approach; 2) the difficulty of capturing patches' changes to the system state at runtime.
To overcome these problems, we propose two novel approaches:
1) \napp, which detects the equivalence between patches online, avoiding the static equivalence analysis and its imprecision;
2) \eins, which intercepts patches' changes to the system state, avoiding a full interpreter and its overhead.
Based on the contributions above, we implement \ourtool, a general-purpose patch validator for Java that integrates all recognized classes of techniques suitable for patch validation.
Our large-scale evaluation with four APR approaches shows that \ourtool accelerates patch validation by \accOverPlain{}x over plain validation or \accOverSota{}x over the state-of-the-art approach, making patch validation no longer the time bottleneck of APR. Patch validation time for a single bug can be reduced to within a few minutes on mainstream CPUs.
\end{abstract}
% Note that keywords are not normally used for peerreview papers.
%\begin{IEEEkeywords}
%Computer Society, IEEE, IEEEtran, journal, \LaTeX, paper, template.
%\end{IEEEkeywords}
}
% make the title area
\maketitle
% To allow for easy dual compilation without having to reenter the
% abstract/keywords data, the \IEEEtitleabstractindextext text will
% not be used in maketitle, but will appear (i.e., to be "transported")
% here as \IEEEdisplaynontitleabstractindextext when the compsoc
% or transmag modes are not selected <OR> if conference mode is selected
% - because all conference papers position the abstract like regular
% papers do.
\IEEEdisplaynontitleabstractindextext
% \IEEEdisplaynontitleabstractindextext has no effect when using
% compsoc or transmag under a non-conference mode.
% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle
%\begin{IEEEkeywords}
%\todo{key, words}
%\end{IEEEkeywords}
\input{sections/1_introduction5_yingfei}
\input{sections/3_survey3_yingfei.tex}
\input{sections/5_approach5}
\input{sections/6_exp_setup2}
\input{sections/7_exp_result4}
\input{sections/8_ending}
\renewcommand{\bibfont}{\footnotesize} % idk why the reference font size is not set up correctly
\bibliographystyle{IEEEtranN}
\balance
\bibliography{bib/zotero,bib/base}
\end{document}
\newcommand{\code}[1]{{\small \texttt{#1}}}
\newcommand{\todo}[1]{{\color{red} \bf \{TODO: {#1}\}}}
\def\modify#1#2#3{{\small\underline{\sf{#1}}:} {\color{red}{\small #2}}
{{\color{red}\mbox{$\Rightarrow$}}} {\color{blue}{#3}}}
%\renewcommand{\modify}[3]{{#3}}
\newcommand{\figref}[1]{Figure~\ref{#1}}
\newcommand{\tabref}[1]{Table~\ref{#1}}
\newcommand{\secref}[1]{Section~\ref{#1}}
\newcommand{\tblref}[1]{Table~\ref{#1}}
\newcommand{\propref}[1]{Property~\ref{#1}}
\newcommand{\algoref}[1]{Algorithm~\ref{#1}} % \algref is already defined, fuck
\newcommand{\procref}[1]{Process~\ref{#1}}
\newcommand{\smalltitle}[1]{{\smallskip \noindent \bf {#1}.\ }}
\newcommand{\smalltitlecolon}[1]{{\smallskip \noindent \bf {#1}:\ }}
\newcommand{\variable}[1]{\textit{#1}}
%\newtheorem{property}{Property}
\IEEEraisesectionheading{\section{Introduction}}
%\yxcomment{I tried to unify online equivalent detection and execution scheduler. Please check if everywhere is changed consistently.}
\IEEEPARstart{A}{utomated} program repair (APR) has attracted much attention in the recent decade.
Many approaches~\cite{le2016history,liu2019tbar,weimer2009automatically,mechtaev2018semantic,liu2013r2fix,kim2013automatic,wei2010automated,zhu2021syntaxguided}
have been proposed, and companies like Bloomberg~\cite{kirbas2021introduction}, Meta~\cite{marginean2019sapfix}, and Alibaba~\cite{zhang2020precfix} are already using APR tools to fix bugs in nightly builds.
Though researchers made significant progress in the effectiveness of APR, the efficiency received relatively limited improvement. Efficiency decides the time needed for repairing a bug and is an important limiting factor for using APR in practice. The state-of-the-art APR approaches still require tens of minutes, and even hours to repair a bug, and recent experiments of program repair still set the timeout to multiple hours~\cite{zhu2021syntaxguided,liu2019tbar,noller2022trust}.
The response time of current APR tools is widely beyond users' patience, as reported by recent studies~\cite{noller2022trust,liang2021interactive}, and significantly limits the application scenario of APR approaches.
The execution time of APR is dominated by patch validation~\cite{le2013current,ghanbari2020prf}.
Most APR approaches are test-based and follow the generate-and-validate pattern: they first generate a bunch of patches, and then validate each patch using the test suite.
Executing the test suite can take minutes, and for a single fault, hundreds of patches can be generated~\cite{liu2020efficiency}. To reduce the execution time of
APR tools, we need to reduce the patch validation time.
%Accelerating patch validation directly benefit current APR users, and may also enrich the application scenarios of APR approaches.%promote %APR's usage from merely nightly builds to other scenarios (e.g., during code editting).
The duality between patch validation and mutation testing has been long recognized~\cite{weimer2013leveraging}. In mutation testing, a set of mutation operators is applied to the program to produce mutants, and the test suite is executed on each mutant to calculate the mutation score. %The calculation of mutation score is a dual of patch validation which executes the test suite on each patch to check whether the ``mutation score'' is zero.
Mutation testing is a dual of patch validation as both need to get the testing result for each mutant/patch.
Since many techniques have been proposed to accelerate mutation testing, they could be potentially adapted to accelerate patch validation.
Under this duality, multiple existing attempts have been made to accelerate patch validation by adapting the corresponding mutation testing technique~\cite{weimer2013leveraging,chen2017contract,mechtaev2018test,hua2018practical,chen2021fast,ghanbari2020prf,guo2019speedup,mehne2018accelerating,qi2013efficient,ghanbari2019practical,wong2021varfix,fast2010designing,pacheco2007feedback,mehne2018accelerating,chen2017contract,hua2018practical,ghanbari2019practical,ghanbari2020prf,guo2019speedup,chen2021fast}. However, as will be discussed in related work, each
of them adapts only one or a few classes
of technique. Therefore, it remains unclear in a systematic manner which techniques can be adapted, and the degree to which their collective adaptation can accelerate patch validation. Furthermore, most attempts are special-purpose, depending on a specific APR approach and not generalizing to different ones.
%\emph{The first contribution of this paper is a systematic survey for existing acceleration techniques for mutation testing and analysis for their suitability for patch validation.}
To fill this gap, in this paper we first systematically investigate the existing acceleration techniques for mutation testing and analyze their suitability for patch validation.
We recognize five classes of suitable techniques, namely,
mutant schemata~\cite{untch1993mutation},
\ted~\cite{just2011major},
test virtualization~\cite{bell2014unit},
test prioritization~\cite{zhang2013faster},
and parallelization.
The remaining techniques are either covered by these five classes or unsuitable for the patch validation of current APR approaches.
Among them, the latter three classes are already used by recent general-purpose patch validators~\cite{chen2021fast,ghanbari2020prf}, but mutant schemata and \ted have never been adapted to a general-purpose patch validator as far as we are aware. Mutant schemata weave all mutants into one program and avoid redundant compilation of different mutants; \ted detects mutants that are equivalent to each other, and executes only one mutant among all equivalent mutants.
%A key component in the newest \ted technique~\cite{just2011major} is to capture the change of patches made to the system state for analysis and to replay them later.
Migrating these two techniques to patch validation is not easy. In mutant testing, mutants are produced from pre-defined mutation operators, and their impacts on the system are under control. %In many systems, the mutation operators only change the result of an expression without introducing a side effect.
%, and thus the change can be easily captured by an instrumented statement that records the result of the expression.
However, in patch validation, the patches are produced from third-party APR approaches that arbitrarily change the program.
This imposes two problems:
\textbf{The first problem is that the state-of-the-art \ted approach~\cite{just2014efficient} requires a static analysis} to determine offline, when the executions of two patches deviate from the original program, whether the rest of their executions would be equivalent or not. The precision of this analysis is directly related to the effectiveness of the technique. It is difficult to implement a precise and scalable static equivalence analysis given the possible changes the patches may conduct on the system.
%\emph{The second contribution of this paper is a novel approach to accelerating patch validation, \textbf{\napp}, which subsumes mutant schemata and \ted, and overcomes the limitation of exiting \ted approaches.}
To cope with this problem, we propose a novel approach, \textbf{\napp}, which does not require the offline static equivalence analysis, and subsumes mutant schemata and \ted.
The basic idea is to detect the equivalence between patches online during patch execution. In this way, we do not need static equivalence analysis, and the effectiveness of the system is also not affected by the imprecision of the static analysis. We also introduce a novel data structure, \emph{\tree}, to record the dynamic analysis results and reuse them across the executions of different patches.
% of our approach is to change the static detection into dynamic detection, and detect other patches equivalent to the current one during the validation of one patch. In this way, we validate only one patch per equivalence class, and support detecting equivalence classes whose execution deviates the original program.
Furthermore, to perform the detection, we need to weave all patches into one program, subsuming mutant schemata. This approach is compatible with test prioritization, test virtualization, and parallelization, allowing us to integrate all suitable techniques into one system for patch validation.
\textbf{The second problem is that we and multiple existing approaches rely on a component to capture how a patch changes the system state at runtime,} for analyzing equivalence and replaying the change later. Existing approaches usually implement an interpreter to execute the changed code.
However, unlike mutation testing where only a selected set of operators may be changed and interpreted, a patch may arbitrarily change a statement that may involve method calls or even system calls. As a result, the interpreter has to support all features in the host programming language and interact with the original runtime. This not only requires huge implementation effort but also is technically difficult as the original runtime may be commercial and does not allow modification. Furthermore, interpreting a large chunk of code inevitably incurs a significant overhead, which may nullify the effectiveness of the technique.
To solve this problem, we propose a novel approach for \textbf{\eins}. It executes a patch without an interpreter by instrumenting code before and after the patch to record the possibly changed states and revert the changes. In this way, the patch is executed in the original runtime, avoiding the implementation cost and the runtime overhead of an interpreter.
{
However, realizing \eins is not easy. Here we highlight our approach in three aspects.
1) \textbf{A patch may bring control flow changes} via statements like \code{break} and \code{throw}. Capturing such changes is difficult because the instrumented statement may be skipped. We propose a design process based on the operational semantics of the programming language for reliably capturing and replaying the control flow change of a patch.
2) \textbf{The scope of data changes} for a patch is large, because a patch may call other methods and arbitrarily modify the memory at any location. The benefit of \ted may not pay off, given the cost to record all possible changes. So we analyze the possibly changed locations in a preparation step, and apply equivalence detection only when the change scope is small enough.
3) \textbf{A patch may not compile}, and weaving all patches together may make the whole program uncompilable. We cannot simply detect an uncompilable patch by compiling it, where the benefit of mutant schemata is lost. Instead, we propose the concept of \emph{isolation unit} where compilation errors within one unit would not affect other units. We then separate each patch into an isolation unit and identify all uncompilable patches at once.
}
% However, migrating the two techniques is not easy, and there are two new problems imposed by APRs. The first problem is the \emph{arbitrary changes} made by APR tools. In mutant testing, mutants are produced from dedicated mutation operators, and their impacts to the system are under control. In many systems, the mutation operators only change the result of an expression without introducing a side effect.
%, and thus the change can be easily captured by an instrumented statement that records the result of the expression.
%However, in patch validation, the patches are produced from third-party APR approaches and could make arbitrary changes to the program.
%This imposes challenges in implementing mutant schemata as well as the capture and replay component in \ted techniques.
% \begin{itemize}
% \item {\bf Large scope of changes.} A patch may call other methods and make arbitrary changes to the system state. Recording all possible changes is costly and the benefit of \ted may not pay off.
% \item {\bf Control flow changes.} Besides changing the stack variables and heap objects, a patch may change the control flow via statements such as \code{break} and \code{throw}. Capturing such changes is difficult because the instrumented statement may be skipped.
% \item {\bf Uncompilable patches.} The patches produced by APR tools may not compile, and weaving such uncompilable patches makes the whole program uncompilable. We cannot simply compile each patch to detect such uncompilable patches, either, where the benefit of mutant schemata is lost.
% \end{itemize}
%\emph{The third contribution of this paper is \textbf{patch virtualization}, a methodology for implementing the component for capturing and replaying the change of a patch to the system state.}
% To address these challenges, we propose the following novel techniques and methods to realize \eins:
% \begin{itemize}
% \item For changes to the memory, we propose a technique that combines static analysis and dynamic analysis. The static analysis analyzes the change scope of a patch, and only patches whose change scope is small enough are used for equivalence detection.
% \item For control-flow changes, we propose a design process based on the operational semantics of the target programming language for designing a wrapper that captures the control-flow change of the wrapped patch. We also design a specific wrapper for the Java programming language.
% \item For uncompilable patches, we propose the concept of \emph{isolation unit} where compilation errors within one unit would not affect the compilation of other units. We demonstrate that isolation unit exists in modern compilers and encapsulate the patches into isolation units where we can reply on the compilation message for identifying uncompilable patches.
% \end{itemize}
% A problem of adapting \ted is that the existing approaches rely on static analysis to detect patches, which has limited effectiveness on patch validation.
% Existing approaches either detect mutants that are fully equivalent in all possible executions~\cite{baldwin1979heuristics,papadakis2015trivial,pan1994using}, which are rare in patch generation, or detect mutants that are equivalent with respect to a test using a pre-pass on the original program execution~\cite{just2011major}, which by nature cannot detect equivalencies among mutants whose executions deviate from the original program. As will be shown later, the existing \ted approaches cannot achieve a speedup of 2x or more on our data in theory.
% % A problem of \ted is that the existing approaches have limited effectiveness on patch validation. The
% % state-of-the-art approach~\cite{just2011major} uses a prepass on the original program execution to detect equivalent mutants, which by nature cannot detect equivalencies among mutants whose executions deviate from the original program. However, in program repair, a patch is designed to change the behavior of the failed test, and it is common for the execution of a patch to deviate from the original program. As will be shown later, more than half of the patches whose validation cannot be accelerated through \ted in our evaluation.
%\emph{The fourth contribution of this paper is the implementation of a general-purpose patch validator for Java, \ourtool, which integrates all suitable techniques for patch validation, and a large-scale empirical evaluation that consumes over 17 months of CPU time to understand its performance on mainstream APR approaches.}
\textbf{Based on the contributions above, we implement \ourtool~\cite{expressapr_replication},} a general-purpose patch validator for Java, which integrates all suitable techniques for patch validation. \ourtool provides out-of-the-box support for Defects4J and can be easily configured to support other Java projects. We then conduct a large-scale empirical evaluation that consumes over 17 months of CPU time to understand its performance on mainstream APR approaches. The evaluation leads to multiple findings:
%We chose Java because it is the programming language supported by most APR tools\footnote{\href{https://program-repair.org/}{program-repair.org} lists 23 APR tools for repairing Java programs, which is the largest among all programming languages.}.
\begin{itemize}
\item \ourtool achieves an acceleration of \accOverPlain{}x over plain validation and outperforms the state-of-the-art approach by \accOverSota{}x. Patch validation is now faster than patch generation for the first time.
\item All adapted techniques are effective for patch validation, each contributing to a significant acceleration.
\item The mean repair time per bug can be reduced to a few minutes, meeting the expectation of most developers~\cite{noller2022trust}.
\end{itemize}
In summary, the contributions of this paper are twofold:
\textbf{Technically,} we propose two novel approaches to realize mutation schemata and \ted, which have not been adapted to general-purpose patch validation before. The \napp approach (\secref{section:approach}) subsumes mutant schemata and \ted, avoiding the need for a static equivalence analysis that can be imprecise. The \eins approach (\secref{section:approach_patch_virtualization}) provides the capture-and-replay component required by \napp, avoiding the need for a full-featured interpreter with runtime overhead.
\textbf{Empirically,} we investigate existing techniques for accelerating mutation testing, and recognize five classes of techniques suitable for patch validation (\secref{section:survey}). We systematically integrate the complete set of techniques (\secref{section:approach_impl}), and conduct a large-scale experiment evaluating their performance (\secref{section:exp_setup}), leading to novel findings that can serve as a solid base for future research (\secref{section:exp_result}).
% \begin{itemize}
% \item An investigation of existing techniques for accelerating mutation testing, which identifies five classes of techniques that are suitable to be applied to patch validation, including mutation schemata and \ted that have never been migrated to a general-purpose patch validation before. (Section~\ref{section:survey})
% \item A novel \napp approach that subsumes mutant schemata and \ted, does not require the static equivalence analysis and does not suffer from the imprecision of the static analysis. (Section~\ref{section:approach})
% \item A novel \eins approach for implementing the capture-and-replay component required by \napp, which does not require the implementation of full-featured interpreter and does not suffer from the runtime overhead of the interpreter. (Section~\ref{section:approach_patch_virtualization})
% \item An implemenation of a general-purpose patch validator, \ourtool, that integrates all five classes of techniques, and a large-scale experiments evaluating the performance of the techniques. (Section~\ref{section:approach_impl}, \ref{section:exp_setup}, \ref{section:exp_result})
% \end{itemize}
%Finally, we discuss related work (Section~\ref{section:related}) and conclude the paper (Section~\ref{section:conclusion}).
\section{Investigating Mutation Testing Acceleration}
\label{section:survey}
We review existing mutation testing literature to systematically find acceleration techniques applicable to patch validation. We first collect mutation testing acceleration techniques by reading through the existing surveys~\cite{jia2010analysis,papadakis2019mutation,pizzoleto2019systematic}, searching for publications matching the term ``mutation testing'' after 2019, after which the papers were not covered by the surveys, and studying the web page collecting existing mutation testing tools~\cite{Pitest_java_mutation_testing_systems}. We then analyze the suitability of each technique for accelerating patch validation.
Our analysis identifies a set of techniques that are suitable to be adapted to patch validation. The remaining techniques are either unsuitable for validating the patch generated by current APR tools, or covered by the identified set of techniques. In this section, we present the classification result and briefly introduce each technique to make the paper self-contained.
% In this section, we briefly introduce these techniques to make this paper self-contained. We first introduce techniques adapted in \ourtool (\secref{section:survey_used}), and then the unadapted techniques, which are either unsuitable for patch validation or covered by adapted techniques (\secref{section:survey_not_used}).
\subsection{Techniques Suitable for Patch Validation}
\label{section:survey_used}
\subsubsection{Mutant Schemata}
\label{section:survey_mutation_schemata}
In mutation testing, each generated mutant needs to be compiled. Different mutants share most of their code, which is repetitively compiled.
Mutant schemata \cite{untch1993mutation} is a common technique to avoid repetitive compilation.
Multiple mutants are encoded in a \textit{meta-program} and then dynamically selected during runtime, so the shared code is compiled only once, reducing the redundant compilation.
\figref{fig:motivation_example_mutation_schemata} illustrates an example where the AOR
(replace arithmetic operator) mutation operator is applied to the expression \code{a+b}, as shown in (a).
AOR generates multiple mutants by replacing the plus sign with other operators,
as shown in (b). Normally each mutated program is independently compiled, so other parts of the code are compiled for multiple times.
But with mutant schemata, all mutants at this expression are grouped into a \textit{meta-function} that dynamically selects a mutant based on a runtime flag, as shown in (c).
In this way, all mutants are encoded in one program, and other parts of the code is compiled only once.
Mutant schemata can be generated
by source code (AST) \cite{just2011major} or byte-code \cite{ma2005mujava} transformation.
\begin{figure}
\centering
\begin{subfigure}[b]{0.2\textwidth}
\begin{minted}[fontsize=\scriptsize]{java}
c = a + b;
\end{minted}
\vskip -.75em
\caption{Program to be mutated}
\vskip .5em
\begin{minted}[fontsize=\scriptsize]{java}
// mutant 1:
c = a - b;
// mutant 2:
c = a * b;
// mutant 3:
c = a / b;
\end{minted}
\vskip -.75em
\caption{Generated mutants}
\end{subfigure}
%\hspace{.5em}
\begin{subfigure}[b]{0.25\textwidth}
\begin{minted}[fontsize=\scriptsize]{java}
c = metaFunc(a, b);
int metaFunc(int x, int y) {
switch(Env.getMutId()) {
case 1: return x - y;
case 2: return x * y;
case 3: return x / y;
}
}
\end{minted}
\vskip -.5em
\caption{The mutant schemata}
\end{subfigure}
\vspace{-.5em}
\caption{
Mutants and mutant schemata created by AOR
%generated by the AOR (replace arithmetic operator) mutation operator on a program.
}
\label{fig:motivation_example_mutation_schemata}
\vspace{-1.25em}
\end{figure}
\subsubsection{Mutant Deduplication}
\label{section:survey_dyn_dedup}
Since a mutant is created by mutating only one or a few statements in the original program, it is possible that a mutant is equivalent to the original program or another mutant. In a group of equivalent mutants, only one of them needs to be executed, saving the test execution time. Two types of equivalencies have been considered in existing work. The first one is \emph{full equivalence}, where the two mutants will produce the same test result on any test. For example, the statement \code{x+=2;} and \code{x+=1+1;} are fully equivalent. Given a group of fully equivalent mutants, only one needs to be executed on all tests. The second one is \emph{test-equivalence}, where the two mutants will produce the same test result on a specific test. Test-equivalence is much more common than full equivalence. For example, the statement \code{x+=2;} and \code{x*=2;} are not fully equivalent but are test-equivalent if \code{x} equals 2 before invoking the mutated statement in a test. Furthermore, any two mutants whose mutated statements are not executed by a test are equivalent with respect to the test. Given a group of test-equivalent mutants with respect to test $t$, only one needs to be executed on $t$.
To deduplicate mutants, existing approaches employ an offline procedure to detect equivalent mutants before the test execution, and then select one mutant among each equivalence class to be tested.
\citet{baldwin1979heuristics} and \citet{papadakis2015trivial} utilize complier optimization to detect fully equivalent mutants, while \citet{pan1994using} using constraint solvers to identify the fully equivalent mutants.
%Since the probability for two mutants to be fully equivalent is not high, \todo{xxx} test-equivalence needs to be detected for better acceleration.
Since test-equivalence is more common than full equivalence,
the Major mutation framework~\cite{just2014efficient} identifies test-equivalent mutants by a pre-pass executing the original program and interpreting the mutated expressions in the mutants along the execution. Since the pre-pass is performed on the original program, Major can successfully detect test-equivalence between the original programs and the mutants, but for test-equivalence between mutants, Major requires a static analysis to determine the equivalence for the rest of the executions after they deviate from the original program. More discussion can be found in the next section.
% collecting runtime states on the original program with a pre-pass. It can successfully detect test-equivalent mutants whose execution is the same as the original program, but not those whose execution deviates from the original program when there are multiple mutated statements, or a mutated statement is executed multiple times in a test.
\subsubsection{Test Virtualization}
\label{section:survey_vmvm}
Before executing a test for the mutants, the test suite has to be initialized for each mutant.
The repetitive initialization is costly in languages based on a virtual machine (VM), such as Java, as booting the VM takes non-trivial time.
Test virtualization approaches such as VMVM~\cite{bell2014unit} reduce this cost by reusing the previous VM instance for the next mutant execution.
The global %program state polluted by previous tests
variables changed by the previous test execution are identified and reset before the next round of test execution by instrumentation.
\subsubsection{Test Case Prioritization}
\label{section:survey_test_prio}
The idea of test case prioritization is that some test cases run faster or are more likely to fail,
so that if these test cases run before other test cases, the mutant will be killed earlier when they fail~\cite{zhang2013faster,yoo2012regression}.
Different heuristic policies can be used to determine what tests get prioritized. In particular, in \textit{regression testing}, of which APR patch validation can be seen as an instance, a typical heuristic is to run the previously failed test cases first. Another heuristic is to execute the test cases in the same package of the modified code first, which are most likely to cover the modified code.
\subsubsection{Parallelization}
Many mutation testing tools~\cite{Pitest_java_mutation_testing_systems,schuler2009javalanche} seek parallel test execution on multicore processors. Because tests and mutants are independent of each other, parallelization can be trivially applied by dividing the work into multiple pieces to be consumed by a process pool, e.g., each piece dealing with a small group of mutants or tests. Some early-stage mutation testing approaches also utilize hardware parallelization mechanisms, such as SIMD~\cite{krauser1991high} and MIMD~\cite{offutt1992mutation}.
\subsection{Techniques Unsuitable for Current APR}
A class of acceleration techniques uses an optimized execution engine to execute all mutants at the same time, reducing possible redundant execution when executing each mutant separately. We classify these techniques as unsuitable for general-purpose patch validation for current APR approaches, because they either require a specific platform feature, or incur too much overhead such that the benefit gained is difficult to surpass the overhead:
\textit{Fork-based mutation analysis} tries to accelerate by sharing the same execution among mutants, relying on the fork mechanism of the POSIX systems.
Spilt-stream execution (SSE) begins with one process representing all mutants, and then forks into multiple subprocesses when a mutated location in the program is reached, one subprocess representing one mutated statement~\cite{tokumoto2016muvm,gopinath2016topsy}. AccMut~\cite{wang2017faster} and WinMut~\cite{wang2021faster} are two enhanced versions of SSE. When a mutated location is reached, their engines analyze all mutants for their changes to the system state, and cluster the mutants based on their changes: the mutants whose changes lead to the same system state are in the same cluster. Finally, the engines fork only one process for each cluster, reducing the number of processes compared with SSE. However, all of the fork-based approaches require the fork mechanism, and are not suitable to be used in a general-purpose patch validation approach as we need to support different operating systems and programming languages that may not support the fork mechanism, such as Windows and Java.
\textit{Variational execution}~\cite{wong2018faster} employs a special execution engine allowing a variable to store a conditional value, which is a special data structure that records possible values of the variable in all mutants. Operations in the program, such as plus or minus, directly manipulate conditional values, and thus executing the program with conditional values produces the test result of all mutants. However, executing the statements over conditional values incurs significant overhead, and thus variational execution is effective only when the number of mutants is huge (e.g., exponential). As reported in an existing study~\cite{wong2018testing}, variational execution could slow down the execution when there are only tens of or hundreds of mutants, a scenario commonly encountered in patch validation.
Furthermore, all techniques discussed above are \textit{lossless} acceleration techniques. There are also \textit{lossy} acceleration techniques~\cite{jia2010analysis,mathur1994empirical,zhang2018predictive,howden1982weak,offutt1994empirical,wong1995reducing,offutt1993experimental,ji2009novel}, which approximate the test results on the mutants to produce the mutation score, which is defined as the ratio of mutants that cause a test failure. However, on patch validation, we need to get the accurate test result on each patch to identify a correct patch, rather than calculating a ratio, and thus the lossy techniques do not apply to the patch validation.
% \subsubsection{Dynamic Mutant Deduplication}
% \label{section:survey_dyn_dedup}
% Since mutants share many statements, many statement executions are redundant when executing tests on each mutant.
% During a test, if none of the mutated statements is reached, the test execution of different mutants so far must be redundant because the executed program is the same. Even if a mutated statement is reached, mutants may still result in the same next state (for example \code{x+=2;} and \code{x*=2;} when \code{x} is 2). In this case, we call the mutants \emph{equivalent modulo current state}, or \emph{state-equivalent} in short.
% If two mutants are always equivalent modulo the current state during the execution of a test, they are \emph{equivalent modulo current test}, or \emph{test-equivalent} in short.
% %If two mutants are equivalent modulo any test, they are \emph{equivalent}.
% \xyaadd{Dynamic mutant deduplication accelerates mutation testing by identifying and removing redundant statement execution for state-equivalent mutants.}
% AccMut~\cite{wang2017faster} chooses to identify and remove redundancy on-the-fly during test execution. It initially launches the test with one process representing all mutants, and when a mutated location in the program is reached, it analyzes all mutants for their changes to the system state. Mutants are clustered into state-equivalence classes by the changes, and the process \code{fork}s into a set of processes, each representing the mutants in a cluster. In this way, redundant statements are executed only once.
% Variational execution is the ultimate form of dynamic mutant deduplication, which eliminates all redundancy during test execution. For example, VarexC~\cite{wong2018faster} uses one process to execute all mutants, each value in the program replaced with a \emph{conditional value} that embeds all possible concrete values in all mutants.
% In this way, variational execution even reduces redundant parts among partially state-equivalent mutants, reaching a theoretically better acceleration than other deduplication approaches.
% However, executing the statements over conditional values incurs significant overheads, and thus variational execution is effective only when the number of mutants is huge (e.g., exponential).
% % Since APR approaches usually generate no more than a few thousand patches, the effect of variational execution is limited.
% % While there is an attempt~\cite{wong2021varfix} to adapt variational execution to APR, variational execution is used to explore combinations of multiple patches, not for general-purpose patch validation.
% }
\subsection{Techniques That Are Subsumed}
\label{section:survey_not_used}
\textit{Test case selection} is an optimization used in
several mutation testing tools \cite{Pitest_java_mutation_testing_systems,gligoric2015practical,harrold2001regression} that skips test cases that
do not execute the mutated code. It is subsumed by \ted because
if the test does not execute the mutated code, all mutants should be in a test-equivalence class.
\section{\NApp}
\label{section:approach}
\newtheorem*{theorem_soundness}{Theorem (Soundness)}
\newtheorem*{theorem_efficiency}{Theorem (Efficiency)}
\algrenewcommand\algorithmicindent{1em}
\newcommand{\step}[4]{\ensuremath{{#1} \xrightarrow{{#2}}\langle {#3}, {#4} \rangle}}
\newcommand{\eval}[3]{\ensuremath{{#1} \xrightarrow{{#2}}{#3}}}
\subsection{Overview}
\label{section:approach_overview}
Among all the five classes of suitable acceleration techniques, mutant deduplication and mutant schemata have not been adapted to general-purpose patch validation due to the differences between mutation testing and patch validation. In this section, we propose the \napp approach, which is suitable for general-purpose patch validation and subsumes both acceleration techniques.
We first demonstrate the possible redundancies during patch validation with an example as shown in the code snippet below. In this code snippet are five patches, $P_1$ to $P_5$, each modifying either of the two statements in the function \code{f()} ($S_1$ for the former two patches, $S_2$ for the latter three patches). The test function \code{test()} determines the correctness of \code{f()} by asserting its behavior.
\begin{minted}[fontsize=\scriptsize]{java}
int i=2, j=1;
void f() {
i+=2; // S1
// P1: "i*=2;" P2: "i=j+3;"
j+=2; // S2
// P3: "j*=2;" P4: "j=i-2;" P5: "j=2;"
}
void test() {
f();
assert(i==4 && j==2);
f();
assert(i==6 && j==4);
}
\end{minted}
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{figures/approach-test-exec.pdf}
\vspace{-1.75em}
\caption{State transitions with different patch validation approaches}
\label{fig:approach_test_exec}
\vspace{-1em}
\end{figure}
\figref{fig:approach_test_exec}(a) illustrates the redundant test executions in this example, where each vertical sequence shows the execution of the test on one patch, and each circle indicates the system state at a specific location. $P_1$ and $P_2$ are test-equivalent to the original program, because all of them effectively set \code{i=4} and \code{j=3} at the first call to \code{f()} and thus the first assertion in \code{test()} fails (State C). $P_3$ and $P_4$ are test-equivalent to each other, setting \code{i=4,j=2} at the first call to \code{f()} (State D), and \code{i=6,j=4} at the second call (State F), passing both assertions. $P_5$ is not test-euivalent to any other variant: though $P_5$ sets \code{i=4,j=2} at the first call to \code{f()}, temporarily leading to the same state D as $P_3$ and $P_4$, their states deviate during the second call where $P_5$ sets \code{j=2} instead of \code{4} (State G). Therefore, if we take the plain patch validation approach, where the test iteratively executes over $P_1$ to $P_5$, test-equivalent patches always follow the same path of state transition, leading to redundancies as highlighted in yellow.
As mentioned, the state-of-the-art \ted approach, Major, detects test-equivalence by a pre-pass on the original program, and executes the test on only one patch from each equivalence class. In this example, Major will instrument the original program and execute the instrumented program as a pre-pass. When a location modified by patches is reached, e.g., $S1$, the instrumented code will invoke an interpreter to interpret all the patches based on the system state of the original program, record and compare their changes to the system state. If a patch always makes the same changes to the system state as the original program, it is test-equivalent to the original program, and thus do not need to test. In this example, it detects that $P_1$ and $P_2$ are test-equivalent to the original program. However, it is hard to identify whether $P_3$, $P_4$, and $P_5$ are test-equivalent to each other. Although they make the same change to the system state in their first invocation (state B), their behavior in the second invocation (state E) is unknown, because state E deviates from the pre-pass which follows the original program ($A \to B \to C$). Therefore, Major requires a static analysis to determine the equivalence between the patches whose execution deviates from the original program. How to implement such a static analysis is not discussed in the original publication, and implementing a precise and scalable static analysis of test-equivalence between patches is difficult. In this case, if we cannot statically determine the test-equivalence between $P_3$ and $P_4$, we have to execute the tests on both of them, leading to \figref{fig:approach_test_exec}(b).
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{figures/approach-state-transition-tree.pdf}
\vspace{-1.75em}
\caption{Iteratively building the state-transition tree in \ourtool}
\label{fig:approach_state_transition_tree}
\vspace{-.5em}
\end{figure}
Our \napp approach overcomes this limitation by embedding the detection process into the test execution process instead of in a pre-pass. To achieve that, we record the runtime behavior of patches as a \textit{state-transition tree}, as shown in \figref{fig:approach_state_transition_tree}(a). Each node in the tree indicates an \textit{interesting} program state of the test, where either the next statement to execute is changed by some patches, or the test completes. In the figure, a node annotated with $S_n (v_i, v_j)$ means that the statement $S_n$ will be executed with variables \code{i} and \code{j} set to $v_i$ and $v_j$, and node annotated with ``Failed'' or ``Passed'' means the result of a completed test. Each edge $U \xrightarrow{P} V$ indicates the transition of interesting states: when executing the variant $P$ against the state $U$, the next interesting state will be $V$. For each patch, its execution corresponds to a path from the root node to a leaf node.
The state-transition tree is built iteratively through multiple rounds of test execution. The current analysis result is recorded on the state-transition tree, so that in the next round we can test another patch that is not test-equivalent to any existing patches. As for the above example, $P_1$ to $P_5$ are validated with three rounds of test execution, as shown in \figref{fig:approach_state_transition_tree}(b). {In the first round, we randomly select a patch $P_1$, and throughout its execution, we record other patches' state changes onto the state-transition tree. We observe that $P_3$ to $P_5$ deviate from $P_1$ (with the state-transition path $A \to B \to C$) at $B$, as marked by the $?$ symbol. Therefore, in the second round, we select a patch among $P_3$ to $P_5$ to explore the deviated $A \to B \to \: ?$ path. This process repeats until all paths are explored.
In this way, we can achieve ideal mutant deduplication for this example, as illustrated in \figref{fig:approach_test_exec}(c), without the need for heavy static analysis in a pre-pass.}
\subsection{Problem Definition}
\label{section:approach_test_def}
Before introducing the approach, we define a set of concepts related to patch validation. These definitions abstract away details in different programming languages, such that our algorithm can apply to a wide range of programs that follow this definition.
We view the procedure of executing a test case as stepping through a state machine, as shown as \algoref{alg:exec_single}. The codebase consists of a set of locations and a mapping ($Cb . \text{Stmt}$) from locations to statements, which represents all source code in the project including tests. When a test begins, the location of the initial state $S$ points to the entry point statement of the test case ($Cb . \text{TestEntry}$). Then the statement gets executed, modifying $S$ that includes the current location ($S. \text{Loc}$). This process repeats until the state becomes a termination state that represents a failed or passed test result. Please note this algorithm is conceptual and does not imply an interpreter. In compiler-based language, the algorithm is implemented by the hardware architecture and the language runtime.
\begin{algorithm}
\caption{Test execution}
\label{alg:exec_single}
\begin{algorithmic}[1]
\Statex \textbf{Input:} The codebase $Cb$
\State $S \gets \{\text{Loc: } Cb . \text{TestEntry}\}$
\While{$S$ is not a termination state}
\State $S \gets \textsc{Execute}(S, Cb . \text{Stmt} [ S . \text{Loc} ])$
\EndWhile
\State $\textsc{ReportResult}(S)$
\end{algorithmic}
\end{algorithm}
A patch is defined as a modification to the codebase, which is a modified mapping that replaces one or a few statements in the codebase. Given a set of patches $Ps$, a plain patch validation procedure enumerates through the patch set to execute the test against each modified mapping of statements, as shown as \algoref{alg:exec_plain}. If the codebase contains multiple test cases, this procedure is repeated for each test against patches that survive all previous tests.
\begin{algorithm}
\caption{Plain patch validation}
\label{alg:exec_plain}
\begin{algorithmic}[1]
\Statex \textbf{Input:} The codebase $Cb$, the patch set $Ps$
\For{$P \in Ps$}
\State $S \gets \{\text{Loc: } Cb . \text{TestEntry}\}$
\While{$S$ is not a termination state}
\State $S \gets \textsc{Execute}(S, P [ S . \text{Loc} ])$
\EndWhile
\State $\textsc{ReportResult}(P,\: S)$
\EndFor
\end{algorithmic}
\end{algorithm}
\subsection{The \NApper}
\begin{algorithm}
\caption{Patch validation with \napp}
\label{alg:exec_scheduler}
\begin{algorithmic}[1]
\Statex \textbf{Input:} The codebase $Cb$, the patch set $Ps$
\State $Root \gets \{ \text{Status: "not-visited"}, \text{Patches: } Ps, \text{Edges: } \varnothing \}$
\label{line:exec_scheduler:init}
\While{there are "not-visited" nodes under $Root$}
\label{line:exec_scheduler:mainloop}
\State $Cur \gets Root$
\State $S \gets \{\text{Loc: } Cb . \text{TestEntry}\}$
\While{$S$ is not a termination state}
\State $Pc \gets \{ P \in Cur . \text{Patches} \:|$ \newline
\hspace*{5em} $P [ S . \text{Loc} ] \neq Cb . \text{Stmt} [ S . \text{Loc} ] \}$
\If{$Pc \neq \varnothing$}
\label{line:exec_scheduler:pc_neq_nothing}
\State $Cur, S \gets \textsc{EvalPatches}(Cur, S, Cur . \text{Patches})$
\label{line:exec_scheduler:call_eval_patches}
\Else
\label{line:exec_scheduler:pc_eq_nothing}
\State $S \gets \textsc{Execute}(S, Cb . \text{Stmt} [ S . \text{Loc} ])$
\EndIf
\EndWhile
\State $Cur . \text{Status} \gets \text{"test-finished"}$
\label{line:exec_scheduler:update_test_finished}
\State $\textsc{ReportResults}(Cur . \text{Patches},\: S)$
\label{line:exec_scheduler:test_fin}
\EndWhile
\Statex
\Procedure{EvalPatches}{$Cur, S, Ps$}
\If{$Cur . \text{Status} = \text{"not-visited"}$}
\State $Cur . \text{Status} \gets \text{"visited"}$
\label{line:exec_scheduler:first_visit}
\For{$P \in Ps$}
\State $Ch \gets \textsc{Capture}(S, P [ S . \text{Loc} ])$
\If{$S . \text{Edges} [Ch]$ is defined}
\label{line:exec_scheduler:find_child}
\State $S . \text{Edges} [Ch] . \text{Patches} \gets \newline
\hspace*{5em} S . \text{Edges} [Ch] . \text{Patches} \cup \{ P \}$
\Else
\State $S . \text{Edges} [Ch] \gets \{ \newline
\hspace*{5em} \text{Status: "not-visited"}, \text{Patches: } \{ P \}, \text{Edges: } \varnothing \}$
\EndIf
\EndFor
\EndIf
\State $Ch, Cur \gets \textsc{FindNotVisitedChild}(Cur . \text{Edges})$ \label{line:exec_scheduler:FindNotVisitedSubtree}
\State \Return $Cur, \textsc{Replay}(S, Ch)$
\label{line:exec_scheduler:move_forward}
\EndProcedure
\end{algorithmic}
\vspace{-.5em}
\end{algorithm}
In this subsection we explain how \napp works. Same as in the previous subsection, we shall only give an algorithm describing the process. How to implement it using instrumentation will be explained in \secref{section:approach_patch_virtualization}.
Our approach requires a component to capture the state changes of the patches for analysis.
Concretely, we require that there exist a \textsc{Capture} procedure and a \textsc{Replay} procedure, so that $C \gets \textsc{Capture}(S, T)$ extracts the state change of statement $T$ over the state $S$ into $C$, and $S' \gets \textsc{Replay}(S, C)$ actually applies the state change $C$ to the state $S$. In this way, we can check whether the state change of two patches is the same by comparing the value of $C$. We require that these two procedures are accurate, such that $\textsc{Replay}(S, \textsc{Capture}(S, T)) = \textsc{Execute}(S, T)$ for any state $S$ and any statement $T$.
Capturing and replaying changes will be discussed in \secref{section:approach_patch_virtualization}, and for now, we keep \textsc{Capture} and \textsc{Replay} as abstract procedures in this section.
The \napper is shown as \algoref{alg:exec_scheduler}. It takes as input the codebase of the project and a patch set to be validated.
It maintains a variable $Root$, which corresponds to the root of the state-transition tree (\figref{fig:approach_state_transition_tree}). Each node in the tree is labeled with a status (``visited'', ``not-visited'', or ``test-finished''), a set of patches that belongs to this node, and a mapping ``Edges'' from state changes to the child nodes.
Initially at line~\ref{line:exec_scheduler:init}, the tree has only a root node labeled as ``not-visited'' with all patches belonging to it, indicating that nothing has been explored yet. Then, the loop body starting at line~\ref{line:exec_scheduler:mainloop} explores a path from the root node to a leaf node. The loop body resembles the plain patch validation, except that when there are some patches to the current statement ($Pc \neq \varnothing$, at line~\ref{line:exec_scheduler:pc_neq_nothing}), the \textsc{EvalPatches} procedure analyzes the state change of each patch, and chooses a child corresponding to a group of patches making the same state change to move forward. When a round of test execution finishes, test results for all patches belonging to the current node are reported (at line~\ref{line:exec_scheduler:test_fin}).
\textsc{EvalPatches} is the critical procedure of the \napp. When the current node is visited for the first time (at line~\ref{line:exec_scheduler:first_visit}), it captures the state change of each patch, and inserts a child node under the current node for each unique state change. For each patch, it searches for an existing edge corresponding to the patch's state change $Ch$ (at line \ref{line:exec_scheduler:find_child}): if there is such an edge, the patch merges into the patch set of that sub-node; otherwise, the patch forms a new sub-node on its own. Finally, \textsc{EvalPatches} finds a child whose subtree includes at least one ``not-visited'' node to continue execution (at line~\ref{line:exec_scheduler:FindNotVisitedSubtree}), updating the current node and system state (at line~\ref{line:exec_scheduler:move_forward}).
\subsection{Properties of the \NApper}
\begin{theorem_efficiency}
The number of rounds of test executions in \algoref{alg:exec_scheduler} is equal to the number of test-equivalence classes among patches, i.e., it removes all redundancies caused by test-equivalence.
\end{theorem_efficiency}
\begin{proof}[Proof Sketch]
Because two patches are test-equivalent iff they always make the same state change during the test execution, test-equivalent patches are never separated by \textsc{EvalPatches} into different sub-nodes, and non-test-equivalent patches must be separated by \textsc{EvalPatches} when they cause different state changes. Therefore, each leaf node labeled with ``test-finished'' corresponds to a test-equivalence class. We can see from \algoref{alg:exec_scheduler} that each round in the loop turns one label of a leaf node from ``not-visited'' to ``test-finished'' at line~\ref{line:exec_scheduler:update_test_finished}, so the number of rounds is equal to the number of ``test-finished'' leaf nodes, which is furtherly equal to the number of test-equivalence classes.
\end{proof}
\begin{theorem_soundness}
\algoref{alg:exec_scheduler} reports the same patch validation result as \algoref{alg:exec_plain}.
\end{theorem_soundness}
\begin{proof}[Proof Sketch]
First, we generalize \algoref{alg:exec_scheduler} so that the initial $S$ and $Root$ are read from inputs. Then we can prove that \algoref{alg:exec_scheduler} is equivalent to \algoref{alg:exec_plain} by induction on the depth of $Root$. \textbf{For the base case} when the depth is one (so $Root$ has no children, and thus \textsc{EvalPatches} is never executed), two algorithms are apparently equivalent because both algorithms execute the original program, leading to the same test result for all patches. \textbf{For the induction case}, consider an arbitrary patch $P$ that belongs to a leaf node $L$, which is reached by the path $Tree \to A \to ... \to L$. The test result for $P$ must be reported in the round that explores this path. During this round, let us consider the moment when $Cur$ changes from $Root$ to $A$, which is the moment when \textsc{EvalPatches} is called (at line~\ref{line:exec_scheduler:call_eval_patches}) for the first time. Before this moment, both algorithms always execute the original program, so $S$ will be the same for the two algorithms up to this moment. Then, $S$ is updated to $S' \vcentcolon= \textsc{Execute}(S, Stmt)$ in \algoref{alg:exec_plain} and $S'' \vcentcolon= \textsc{Replay}(S, \textsc{Capture}(S, Stmt))$ in \algoref{alg:exec_scheduler}, where $Stmt \vcentcolon= P [ S . \text{Loc} ]$. Based on the accuracy requirement for \textsc{Capture} and \textsc{Replay}, we must have $S' = S''$. Then we can apply the induction hypothesis with the initial $S$ set to $S'$ and $Root$ set to $A$.
\end{proof}
\section{\EIns}
\label{section:approach_patch_virtualization}
The \napp algorithm can be implemented in different ways. To ensure the performance of the system, we implement this algorithm using instrumentation. That is, the tests and the patches are executed in the original runtime of the programming language, and only at certain locations, the instrumented code is invoked to detect equivalent patches and schedule executions.
An overview of the patch validation process is shown in \figref{fig:approach_overview}. It takes three steps to validate all patches: preparation, patch compilation, and test execution.
In the first step, the codebase as well as the patches are instrumented to ensure the correct execution of the \napper.
In the second step, we compile the instrumented codebase, dropping patches that are uncompilable. In the third step, the \napper iteratively runs the test suite, and stores the test result for each patch.
\begin{figure}
\centering
\includegraphics[width=.95\columnwidth]{figures/approach-overview-scheduler.pdf}
\vspace{-.5em}
\caption{An overview of \ourtool}
\label{fig:approach_overview}
\vspace{-.5em}
\end{figure}
Implementing the overall process of Algorithm~\ref{alg:exec_scheduler} with instrumentation is straightforward. First, a main procedure is added into the codebase that initializes the state-transition tree and starts test executions. Then, at each patched location, the code is instrumented to evaluate the patches and update the tree. The main challenging part is how to implement the \textsc{Capture} and \textsc{Replay} in \algoref{alg:exec_scheduler}.
Existing mutation-testing approaches~\cite{just2011major,wong2018faster} instrument an interpreter at the mutated location for this purpose. However in APR, since a patch may invoke other methods and then execute a large chunk of code, using an interpreter is not only expensive but also difficult to implement. To solve this problem, we propose \eins. Our approach inserts code before and after the patched statement, such that the patch itself executes normally in the original runtime, and the inserted code is responsible for detecting what changes the patch has made to the system state and reverting the changes, i.e., the change is intercepted.
In the preparation step, our approach adds instrumentation code around each patch, turning each patch into a \code{capture} component and a \code{replay} component. All such components are woven together into the codebase, effectively implementing mutant schemata.
Finally, in the test execution step, the \napper calls the corresponding component as the \textsc{Capture} and \textsc{Replay} procedures.
In the rest of this section, we shall first introduce a small programming language, IMP+, for illustration. Then we describe how to intercept the changes based on IMP+. Finally we show how to deal with uncompilable patches.
\subsection{IMP+}
IMP+ is an enhancement of the classic IMP language~\cite{pierce2010software} with control flow constructs. IMP+ contains the commonalities between imperative programming languages, so we use it to illustrate our instrumentation process.
The syntax of IMP+ is shown below:
$$
\small
\begin{array}{lllr}
\tt s &\rightarrow& \tt s_1~s_2 & \textit{(statements)} \\
\tt &|& \tt x:=e;\\
\tt &|& \tt x:=m(e_1, \ldots, e_n);\\
\tt &|& \tt if(e)~s_1~else~s_2 \\
\tt &|& \tt try~s_1~catch(x)~s_2\\
\tt &|& \tt while(e)~s \\
\tt &|& \tt break; \\
\tt &|& \tt continue; \\
\tt &|& \tt return~e; \\
\tt &|& \tt throw~e;\\
\tt &|& \tt \ldots\\
\tt e &\rightarrow& \tt e_1+e_2 & \textit{(expressions)} \\
\tt &|& \tt \ldots\\
\end{array}
$$
The runtime system state of an IMP+ program is a pair $\langle \sigma, \omega \rangle$, where the \textit{data state} $\sigma$ is a function mapping variables to their values, and the \textit{control state} $\omega$ can be one of the following values that reflects the effect of control flow statements:
\begin{itemize}
\item \code{Normal}, indicating that we should normally execute the next statement;
\item \code{Break}, indicating that a loop should break;
\item \code{Continue}, indicating that a loop should skip to the end of its body;
\item \code{Return $v$}, indicating that a method has returned a value $v$;
\item \code{Exception $e$}, indicating that an exception $e$ has been thrown.
\end{itemize}
The main operational semantic rules of IMP+ are shown in \figref{fig:approach_imp_semantic}, where \eval{\sigma}{e}{v} means that expression $e$ evaluates to $v$ under the system state $\langle \sigma, {\tt Normal} \rangle$, and \step{\sigma}{s}{\sigma'}{\omega'} means that statement $s$ changes the system state from $\langle \sigma, {\tt Normal} \rangle$ to $\langle \sigma', \omega' \rangle$. For simplicity, we only show the rules related to control flow change, and omit standard rules such as those for assignments and conditionals.
Since a state includes two parts, we need to intercept the changes to both parts. Below, we discuss how to deal with changes to the control state and the data state respectively.
\subsection{Changes to the Control State}
\label{section:approach_control_state}
Recall that the \code{capture} component is responsible for detecting what changes a patched statement has made, and reverts the change; the \code{replay} component is responsible for replaying the recorded change. For changes to both states, the main challenge for implementing the two components lies in the control state, where the statement following the patched statement in the code file is not always the next statement to be executed at runtime. Therefore, we need a reliable way to detect and revert control state changes after the execution of the patch. Furthermore, since different programming languages have different language constructs for changing the control states, the design of these two components is by nature language-dependent.
To cope with the challenges,
we propose a design process, \procref{proc:capture_control_state}, to systematically design \code{capture} and \code{replay} components for the control state $\omega$ given the semantic rules of the target programming languages.
While this process is not fully automatic, it only needs to be done once for a language.
{
\floatname{algorithm}{Process}
\begin{algorithm}
\newcounter{_alg_counter_bkp}
\setcounter{_alg_counter_bkp}{\value{algorithm}}
\setcounter{algorithm}{0}
\caption{Capturing/replaying control state changes}
\label{proc:capture_control_state}
\setcounter{algorithm}{\value{_alg_counter_bkp}}
\begin{algorithmic}[1]
\Statex \textbf{Input:} Semantic rules $S$
\Statex \textbf{Output:} Capture component $C$, replay component $R$
\State $C \gets \lambda \tt s, {\tt s}$
\State $R \gets {\tt skip}$
\State $T \gets$ possible types of $\omega$ in $S$
\While{$(T \setminus \{ {\tt Normal} \}) \neq \varnothing$}
\State Choose a $t \in (T \setminus \{ {\tt Normal} \})$
\State $T \gets T \setminus \{t\}$
\State \#1: Pick an $R_c \in S$ that can change $\omega$ from $t$ to {\tt Normal}, and an $R_r \in S$ that can change $\omega$ from {\tt Normal} to $t$.
\State \#2: Use $R_c$ to write a capture component $c_t(s)$ for any target statement $s$ and the type $t$.
\State \#3: Refine $c_t(s)$ to ensure that it preserves the semantics of $s$ when it yields a state in $T$.
\State \#4: Use $R_r$ to write a replay component $r_t$ for type $t$.
\State $C \gets \lambda s, c_t(C(s))$
\label{line:capture_control_state:stack_capture}
\State $R \gets r_t ; R$
\label{line:capture_control_state:stack_replay}
\EndWhile
\end{algorithmic}
\end{algorithm}
}
The generated \code{capture} component takes a patch as input, and produces a code snippet for executing the patch and capturing the change in variable {\tt change}. The generated \code{replay} is a code snippet replaying the change captured in {\tt change}. \procref{proc:capture_control_state} works by capturing and reverting each type of abnormal control state at a time. For each type, there are four manual steps (\#1 to \#4) to craft a \code{capture} component and a \code{replay} component for this exact type of control state. Then, these components are stacked together (at line \ref{line:capture_control_state:stack_capture}-\ref{line:capture_control_state:stack_replay}) to cope with the whole control state.
\begin{figure}
\centering
\newcommand{\rowspace}{\\[.5em]}
{\small
$$
\begin{array}{c}
\infer[\textsc{\scriptsize E-Break}]{\step{\sigma}{\tt break;}{\sigma}{\tt Break}
}{}
\rowspace
\infer[\textsc{\scriptsize E-Continue}]{\step{\sigma}{\tt continue;}{\sigma}{\tt Continue}
}{}
\rowspace
\infer[\textsc{\scriptsize E-Return}]{\step{\sigma}{\tt return~e;}{\sigma}{\tt Return~v}}{\eval{\sigma}{\tt e}{v}}
\rowspace
\infer[\textsc{\scriptsize E-Throw}]{\step{\sigma}{\tt throw~e;}{\sigma}{\tt Exception~v}}{\eval{\sigma}{\tt e}{v}}
\rowspace
\infer[\textsc{\scriptsize E-Seq}]{\step{\sigma}{\tt s_1~s_2}{\sigma''}{\omega''}}
{\step{\sigma}{\tt s_1}{\sigma'}{{\tt Normal}} &
\step{\sigma'}{\tt s_2}{\sigma''}{\omega''}}
\rowspace
\infer[\textsc{\scriptsize E-SeqSkip}]{\step{\sigma}{\tt s_1~s_2}{\sigma'}{\omega'}}{\step{\sigma}{\tt s_1}{\sigma'}{\omega'} & \omega' \neq {\tt Normal}}
\rowspace
\infer[\textsc{\scriptsize E-While}]{\step{\sigma}{\tt while(e)~s}{\sigma''}{\omega''}}{
\begin{array}{cc}
\eval{\sigma}{\tt e}{\tt true} &
\step{\sigma}{\tt s}{\sigma'}{\omega'} \\
\omega' \in \{ {\tt Normal}, {\tt Continue} \} &
\step{\sigma'}{\tt while(e)~s}{\sigma''}{\omega''}
\end{array}
}
\rowspace
\infer[\textsc{\scriptsize E-WhileFalse}]{\step{\sigma}{\tt while(e)~s}{\sigma}{\tt Normal}}{
\eval{\sigma}{\tt e}{\tt false}
}
\rowspace
\infer[\textsc{\scriptsize E-WhileBreak}]{\step{\sigma}{\tt while(e)~s}{\sigma'}{\tt Normal}}{
\eval{\sigma}{\tt e}{\tt true} &
\step{\sigma}{\tt s}{\sigma'}{\tt Break}
}
\rowspace
\infer[\textsc{\scriptsize E-WhileSkip}]{\step{\sigma}{\tt while(e)~s}{\sigma'}{\omega'}}{
\begin{array}{cc}
\eval{\sigma}{\tt e}{\tt true} &
\step{\sigma}{\tt s}{\sigma'}{\omega'} \\
\multicolumn{2}{c}{\omega' \in \{ {\tt Return}~v, {\tt Exception}~e \}}
\end{array}
}
\rowspace
\infer[\textsc{\scriptsize E-Call}]{\step{\sigma}{\tt x:=m(e_1, \ldots, e_n)}{\sigma'[x\backslash v]}{\tt Normal}}{
\begin{array}{l}
{\it body}({\tt m})={\tt s} \qquad \eval{\sigma}{\tt e_i}{v_i} \\
\step{\sigma[{\it para}({\tt m},i)\backslash v_i]}{\tt s}{\sigma'}{{\tt Return}~v}
\end{array}
}
\rowspace
\infer[\textsc{\scriptsize E-CallExc}]{\step{\sigma}{\tt x:=m(e_1, \ldots, e_n)}{\sigma'}{{\tt Exception}~e}}{
\begin{array}{l}
{\it body}({\tt m})={\tt s} \qquad \eval{\sigma}{\tt e_i}{v_i} \\
\step{\sigma[{\it para}({\tt m},i)\backslash v_i]}{\tt s}{\sigma'}{{\tt Exception}~e}
\end{array}
}
\rowspace
\infer[\textsc{\scriptsize E-Catch}]{\step{\sigma}{\tt try~s_1~catch(x)~s_2}{\sigma''}{\omega''}}{
\step{\sigma}{\tt s_1}{\sigma'}{{\tt Exception}~e} & \step{\sigma'[x\backslash v]}{\tt s_2}{\sigma''}{\omega''}
}
\rowspace
\infer[\textsc{\scriptsize E-CatchSkip}]{\step{\sigma}{\tt try~s_1~catch(x)~s_2}{\sigma'}{\omega'}}{
\step{\sigma}{\tt s_1}{\sigma'}{\omega'} & (\omega' \neq {\tt Exception}~e)
}
\end{array}
$$
}
\vspace{-1em}
\caption{Semantic rules of IMP+}
\label{fig:approach_imp_semantic}
\vspace{-.75em}
\end{figure}
We illustrate this process with the semantic rules of IMP+ as shown in \figref{fig:approach_imp_semantic}, where \code{Break}, \code{Continue}, \code{Return $v$} and \code{Throw $e$} are abnormal control states that we concern.
\textbf{For the \code{Break} state:} At step \#1, we find that the only possible $R_c$ is \textsc{E-WhileBreak} and the only possible $R_r$ is \textsc{E-Break}. At step \#2, we use this rule to write a \code{capture} component that turns a \code{Break} state into \code{Normal}:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
while(true) \{s\} \\
change := "break";
\end{addmargin}
\vspace{.5em}
Among possible states in $T = \{\allowbreak \code{Normal},\allowbreak \code{Continue},\allowbreak \code{Return}~ v,\allowbreak \code{Exception}~e \}$, the component above does not preserve the semantics of the former two states: if the patch ends up with a \code{Normal} or \code{Continue} state, it causes an endless loop. Therefore at step \#3, We modify the \code{capture} component to fix these problems:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
flag := 0; \\
while(true) \{
\begin{addmargin}[1em]{0em}
flag := flag+1; \\
if(flag>1) break; \\
\{s\} \\
flag := flag+1;\}
\end{addmargin}
if(flag=1) change := "break";\\
if(flag=2) continue;
\end{addmargin}
\vspace{.5em}
The added \code{flag} variable will distinguish \code{Break} from \code{Normal} and \code{Continue}, so that it correctly preserves the semantics of \code{s} when it does not cause a \code{Break} state. Finally at step \#4, we use \textsc{E-Break} to write a \code{replay} component:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
if(change="break") break;
\end{addmargin}
\vspace{.5em}
\textbf{For the \code{Continue} state:} At step \#1, we find $R_c=$ \textsc{E-While} and $R_r=$ \textsc{E-Continue}. At step \#2\textendash\#3, the \code{capture} component is mostly the same as the component for \code{Break}, but the last line should be changed to \code{if(flag=2) change:="continue";}. At step \#4, the \code{replay} component is \code{if(change="continue") continue;}.
\textbf{For the \code{Return $v$} state:} At step \#1, $R_c=$ \textsc{E-Call} and $R_r=$ \textsc{E-Return}. At step \#2, the \code{capture} component wraps the statement into a new method and then calls it:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
def m() \{s\} \\
x := m(); \\
change := "return"; retval := x;
\end{addmargin}
\vspace{.5em}
For each state in $T = \{\allowbreak \code{Normal},\allowbreak \code{Exception}~e \}$, this component fails when the patch ends up with a \code{Normal} state. At step \#3, we fix it by \code{return}ing a unique special value at the end of the wrapping method:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
def m() \{\{s\} return "--special--";\} \\
x := m(); \\
if(x!="--special--") \{
\begin{addmargin}[1em]{0em}
change := "return"; retval := x;\}
\end{addmargin}
\end{addmargin}
\vspace{.5em}
At step \#4, we come up with this \code{replay} component:
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small\tt
if(change="return") return retval;
\end{addmargin}
\vspace{.5em}
\textbf{For the last \code{Exception $e$} state,} we similarly use $R_c=$ \textsc{E-Catch} to write the \code{capture} component as \code{try\allowbreak\{s\} catch(e) \{change:="exc"; exc:=e;\}}, and uses $R_r=$ \textsc{E-Throw} to write the \code{replay} component as \code{if(change="exc") throw exc;}.
In the end, we get the following components that capture and replay all control state changes.
\begin{align*}
C(s) =&\: c_\text{Exception} ( c_\text{Return} ( c_\text{Continue} ( c_\text{Break} (s) ) ) ) \\
R =&\: r_\text{Exception}; r_\text{Return}; r_\text{Continue}; r_\text{Break}; \tt skip
\end{align*}
\subsection{Changes to the Data State}
After the control state is intercepted in \procref{proc:capture_control_state}, further intercepting changes to the data state is easier because we can detect and revert the modified variables in the next statement following the patch -- the next statement is now guaranteed to run. On the other hand, unlike the control state where reverting always sets it to \code{Normal}, here we need an efficient way to detect and revert the data state to its version before executing the patched statement. Simply recording the whole data state is not feasible due to its size.
To cope with this challenge, we use lightweight static analysis to detect the change scope that each patch may bring to the data state, and record only the values within the scope. After executing the patched statement, the new values in the scope can be compared with the recorded ones for detecting and reverting the change. We further bypass the equivalence detection on patches whose change scope is too large: these patches are validated sequentially using the plain method but not with the \napper.
Our approach does not restrict the choice of the static analysis and the definition of the change scope, and the implementation could use any static analysis algorithm that best fits the target programming language and the APR tools. In the following, we describe an implementation of the two components for the IMP+ language, which is close to those used in \ourtool.
In IMP+, the only two types of statement that can change the data state are \code{$\tt x:=e$} and \code{$\tt x:=m(e_1, \ldots, e_n)$}, which change the value of a variable \code{x} to an expression or the return value of a method. As a result, we can define the change scope as a set of variables, and use an inter-procedural static analysis to produce this change scope. This static analysis can be easily implemented by scanning the code and collecting the left variables in all assignments. When a method call is encountered, all variables changed by the target method are also iteratively added until a fixed point is reached.
Given a set of variables \code{$\{\tt x_1, \ldots x_n\}$} returned by the static analysis, we can refine the \code{capture} component in the following way to intercept data state changes.
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small
\code{$\tt old_1 := x_1;\: \ldots;\: old_n := x_n;$}\\
\code{\textit{(original capture component by \procref{proc:capture_control_state})}}\\
\code{$\tt if (old_1 \neq x_1)\: \{change[0] := x_1; \: x_1 := old_1;\}$}\\
\code{$\tt else\: \{change[0] := null;\}$}\\
\ldots \\
\code{$\tt if (old_n \neq x_n)\: \{change[n-1] := x_n;\: x_n := old_n;\}$}\\
\code{$\tt else\: \{change[n-1] := null;\}$}
\end{addmargin}
\vspace{.5em}
We similarly refine the \code{replay} component.
\vspace{.5em}
\begin{addmargin}[2em]{0em}
\small
\code{$\tt if (change[0]\neq null)\: x_1 := change[0];$} \\
\ldots\\
\code{$\tt if (change[n-1]\neq null)\: x_n := change[n-1];$} \\
\code{\textit{(original replay component by \procref{proc:capture_control_state})}}
\end{addmargin}
\vspace{.5em}
This implementation still has two issues in efficiency. First, the change scope may include too many variables, which causes excessive overhead at runtime. To deal with this issue, our implementation simply uses a purity analysis~\cite{bravenboer2009strictly} rather than an inter-procedural change scope analysis. A method is \emph{pure} if it does not change the data state. A purity analysis involves only a Boolean result and can be efficiently conducted. If a patched statement calls any impure method, we consider its change scope too large and it will bypass the equivalence detection.
Second, a variable may store a large data structure where copy and comparison may be costly. To deal with this issue, we keep an allow list of types that are either primitive types or compound types of a small, fixed size. If the change scope includes any variable whose type is not in the allow list, we consider the change scope too large and the patch will bypass the equivalence detection.
\subsection{Properties of Interception-based Instrumentation}
\begin{theorem_efficiency}
The time cost of the designed \code{capture} and \code{replay} components in the above example is at most linear to the syntactic size of the patched statement,
i.e., the overhead of \eins is small even if the patched statement may execute for a long time (e.g., contains a loop or a recursive call).
\end{theorem_efficiency}
\begin{proof}[Proof Sketch]
The final \code{capture} and \code{replay} components consist of statements that deal with changes to the control state and the data state. For the control state, a constant number of statements are added to the component. For the data state, the number of added statements is linear to the number of possibly modified variables, which is no more than the syntactic size of the statement.
\end{proof}
\begin{theorem_soundness}
$\textsc{Replay}(S, \textsc{Capture}(S, T)) = \textsc{Execute}(S, T)$ for any state $S$ and any statement $T$, i.e., calling the \code{capture} component and then the \code{replay} component for any patch is equivalent to executing the patch.
\end{theorem_soundness}
\begin{proof}[Proof Sketch]
The construction of both components is incremental, i.e., we build a component for each aspect of the state change, and combine them together. Each individual component is sound within its own scope by design, and the combining process is also sound: when combining components for control state changes, step \#3 in \procref{proc:capture_control_state} ensures that an added component does not break previous components; when refining the component for data state changes, the control state is already reverted, so that the refined component always executes normally regardless of control flow effects in the patch.
\end{proof}
\subsection{Compilation Isolation}
Unlike mutation testing that assumes mutants always compile, patches generated by an APR approach could be uncompilable, leading to a challenge when all patches are woven into the codebase: if any patch is not compilable, the whole codebase cannot compile. It is not possible to detect and remove uncompilable patches by individually compiling each patch, because it will nullify the acceleration effect of mutant schemata.
An intuition to solve this challenge is to rely on the error messages from the compiler: if the error messages could precisely pinpoint all uncompilable code snippets,
we can compile the codebase once and then remove the uncompilable patches based on the error messages.
However, the error messages are often not perfectly
precise: for example, a patch in the middle of a method may lead to compile errors at the end of this method, and it is often the case that an error stops the compilation of the rest of the method. Therefore, we cannot rely on the error messages to precisely pinpoint all uncompilable patches.
To solve this problem, we propose the concept \textit{isolation unit} to measure the granularity of the preciseness in the error messages of a compiler: a compilation error within an isolation unit will only cause error messages within the isolation unit but not affect the compilation of other parts of the code. An isolation unit must exist for any compiler: in the extreme case, the whole codebase is the isolation unit. However, our observation is that modern compilers usually have more fine-grained isolation units. For example, many compilers compile each file individually into an object file (\code{.java} files into Java bytecode files, and \code{.c} files into assembly files), and thus each file is an isolation unit. Furthermore, a method or a procedure in many imperative programming languages, including Java and C, is also an isolation unit, because these programming languages are carefully designed such that compilers only need to perform intra-procedural analysis when compiling the code (inter-procedural analysis is often applied at a much later stage for optimizing the code where the compilation errors have already been detected).
Therefore, to isolate compile errors, our interception-based instrumentation approach should wrap each patch in an isolation unit.
Let us assume that methods are also isolation units for an IMP+ compiler. Each \code{capture} component produced in \secref{section:approach_control_state} is already in an individual isolation unit because we wrap the patch in a method to deal with the \code{Return $v$} control state.
Then we can compile the codebase woven with all patches with two rounds of compilation: the first round identifies all uncompilable patches from error messages, which will be removed, and the second round compiles the codebase with only the remaining patches.
\section{Implementation}
\label{section:approach_impl}
After the challenges for adapting mutant schemata and mutant deduplication are solved, we build \ourtool, a general-purpose patch validator for Java, which adapts all five classes of acceleration techniques suitable for patch validation. We choose Java as the target programming language because it is supported by most APR tools\footnote{\href{https://program-repair.org/}{program-repair.org} lists 23 APR tools for repairing Java programs, which is the largest among all programming languages.}.
To achieve mutant schemata and mutant deduplication, we implement the \napp algorithm described in \secref{section:approach} and manually followed the design process of interception-based instrumentation described in \secref{section:approach_patch_virtualization}. While our approach is theoretically sound and can be applied to arbitrary patches, there are two limitations at the implementation level, which will be evaluated in our experiment:
\begin{itemize}
\item \textbf{Patch Limitation:} In \secref{section:approach_test_def}, a patch is defined as a modification of statements. Therefore, patches that modify other parts in the program (e.g., the definition of a field) are out of the scope of \ourtool. We detect this case in the preparation step, and fall back to plain validation for affected patches.
\item \textbf{Test Limitation:} Also in \secref{section:approach_test_def}, executing the test is viewed as stepping through a state machine, where the state-transition should be \emph{stable}, i.e., being exactly the same across multiple runs. In practice, most test cases are stable -- otherwise, it would be difficult to troubleshoot a test failure.
We perform runtime checks in \algoref{alg:exec_scheduler} to detect rare unstable test cases\footnote{In the \textsc{EvalPatches} procedure and when the test ends, we assert that the observed system state should be consistent with the record on the state-transition tree. For example, if the test execution finishes but $Cur . \text{Status} = \text{"visited"}$, the test case is considered unstable.}, and fall back to plain validation for them.
\end{itemize}
The \code{capture/replay} component for Java is similar to the IMP+ example as described in \secref{section:approach_patch_virtualization}, because both languages share similar features. For the control state, control flow statements in Java can only be \code{break}, \code{continue}, \code{return}, or \code{throw}~\cite{jls8}, which are all covered by IMP+. For the data state, Java programs similarly change the data state by assignment statements.
The \code{capture/replay} component for Java has three differences compared against IMP+: 1) Java allows to assign a label to a loop and jumps out of a labeled loop. To support this syntax, we detect labeled \code{break/continue} statements in the preparation step, and add the same label in the generated \code{replay} component. 2) In Java, every method declares a list of checked exceptions that can be thrown. Generated components should inherit this list from the original method. 3) A patch may access local variables in the original method and fields in the original class. To make this possible, we put generated methods in the same class as the original patch, and declare auxiliary fields for passing around local variables between the original method and the generated methods.
Adapting the three other acceleration techniques is straightforward:
\textbf{Test virtualization} is adapted by resetting global states (except for the state-transition tree that should be shared across runs) before each round of test execution. Current \ourtool implementation uses an existing library, VMVM~\cite{bell2014unit}, for test virtualization.
\textbf{Test case prioritization} is adapted by sorting test cases in the test suite. Following the two heuristics mentioned in \secref{section:survey_test_prio}, \ourtool first executes the failing test cases, then other test cases in the same package of a patched location, and then test cases in other packages.
\textbf{Parallelization} is adapted by spawning multiple instances of the patch validator with a process pool, one instance dealing with a subset of patches, each subset corresponding to all patches to the same fault location as generated by the APR approach.
\section{Experiment Setup}
\label{section:exp_setup}
\subsection{Research Questions}
Our evaluation aims to answer these research questions.
\begin{enumerate}
\item[RQ1.] \textbf{Overall performance.}
%Will patch validation be fast enough for online scenario if acceleration techniques are systematically used?
How fast patch validation could be with \ourtool?
\item[RQ2.] \textbf{Technique effectiveness.} Does each technique in \ourtool contribute to the final performance?
\item[RQ3.] \textbf{Feasibility.} Will \ourtool fail or report incorrect validation results, affecting the ability of APR?
\end{enumerate}
\subsection{The Benchmark}
% To evaluate the patch validation process,
% a set of faulty programs with a corresponding test suite (\textit{the benchmark})
% and different APR techniques that generate candidate patches for each program (\textit{studied APR techniques})
% need to be chosen.
We used Defects4J~\cite{Just2014defects4j} v1.2 as the benchmark.
Defects4J v1.2 is a widely-used APR benchmark on Java, containing 395 bugs in open-source Java projects.
We chose four representative publicly-available APR tools covering different kinds of APR approaches to be studied in this evaluation:
\begin{itemize}
\item Recoder~\cite{zhu2021syntaxguided}, a neural-network-based tool that fixes 50 bugs in the benchmark.
\item TBar~\cite{liu2019tbar}, a template-based tool that fixes 42 bugs.
\item SimFix~\cite{jiang2018shaping}, a heuristic-based tool that fixes 34 bugs.
\item Hanabi~\cite{xiong2021l2s}, a decision-tree-based tool that fixes 26 bugs.
\end{itemize}
We obtained the replication package for each tool, and collected all generated candidate patches under the original time-out value for evaluation.
We did not evaluate on all 395 bugs in Defects4J because the computational cost would be unacceptable to execute multiple baselines for the patches generated by the four APR tools. %large running multiple baseline patch validation approaches against four APR tools.
Instead, we sampled the dataset in two possible ways: 1) for each studied APR tool, choosing all bugs it can fix as reported by its authors (the \textit{fixable} dataset); 2) choosing 30 bugs randomly from the whole dataset (the \textit{random} dataset). %The fixable sample is consistent with an existing study~\cite{chen2021fast} and matches our goal in \secref{section:motivation}: if the time usage in each fixable bug is short enough, APR can be used online with effectiveness uncompromised.
The fixable dataset helps us understand whether APR can be accelerated: if the time used in each fixable bug is shorter, we can use a shorter time limit without affecting the effectiveness of the tool.
The random dataset is supplemented to avoid the threat of possible selection bias on the fixable dataset. Results on both datasets are reported.
The current evaluation costs 17 months of single-CPU-core time in total, which is of the largest scale among all existing studies on accelerating APR within our knowledge.
\subsection{Experiment Methodology}
\textbf{In RQ1,} the patch validation time per bug using \ourtool is compared with following two baselines. To simulate the degree of parallelization on average hardware, the job is distributed to eight CPU cores for every baseline.
\begin{itemize}
\item \textbf{Plain.}
Each patch is compiled using the \code{defects4j compile} command that ships with the Defects4J dataset, which uses an Ant script to compile changed files and perform necessary user-defined actions. Compilable patches are tested using the \code{defects4j test} command. This reflects the normal practice of evaluating APR techniques.
% represents researchers simply following the Defects4J user guide without optimization efforts.
\item \textbf{State-of-the-Art.}
Each patch is compiled using the OpenJDK compiler by invoking the \code{javac} command to compile the patched class. Then UniAPR~\cite{chen2021fast}, the current state-of-the-art patch validator on Java, is used to validate all patches. UniAPR has incorporated test virtualization, test case prioritization, and parallelization, but not mutant schemata and mutant deduplication, the two techniques brought to patch validation by \ourtool for the first time. %This represents the best possible approach before \ourtool.
\end{itemize}
\textbf{In RQ2,} we measure the individual effectiveness of Mutant Schemata (MS), Mutant Deduplication (MD), Test Virtualization (TV), and Test Case Prioritization (TCP). We do not discuss parallelization here because its effectiveness is straightforward (roughly $N$ times acceleration when running on $N$ cores).
Among these four techniques, Mutant Schemata accelerates the patch compilation step, and the other three techniques accelerate the test execution step. Besides the original ExpressAPR implementation, we add intermediate configurations by removing each technique from \ourtool one by one, and measure their time usage. For the patch compilation step, we add one intermediate configuration (``-MS''); for the test execution step, we add three intermediate configurations (``-TCP'', ``-TCP -MD'', ``-TCP -MD -TV''). In this way, we can understand the effectiveness of the removed technique between two adjacent configurations. Note that this is not an ablation study due to implementation-level dependencies -- it can be hard to individually remove a technique from \ourtool\footnote{For example, Mutant Deduplication requires the state-transition tree to be persisted across executions, which is implemented as part of the Test Virtualization technique. Therefore, we cannot remove TV without breaking MD.}.
% \begin{figure}
% \centering
% \includegraphics[width=.9\columnwidth]{figures/breakdown-methodology.pdf}
% \vspace{-.25em}
% \caption{Compared configurations in RQ2}
% \label{fig:breakdown-methodology}
% \vspace{-1em}
% \end{figure}
\textbf{In RQ3,} we compare the validation result (``plausible'', ``implausible'' or ``fails to validate'') of \ourtool against the Plain baseline. The \ourtool implementation may fail to accelerate the validation of a patch if it detects a violation of two limitations described in \secref{section:approach_impl}. It may also produce an incorrect result if it fails to detect such a violation that affects the validation result.
In this research question, we evaluate how often these cases happen. We count the number of patches that \ourtool fails to validate or reports incorrect results (reported ``plausible'' when should be ``implausible'' or vice-versa), and analyze the reason.
\subsection{Experiment Settings}
\label{section:exp_setup_implementation}
The experiment is run on a Xeon\textsuperscript{\textregistered} 8270 CPU server with eight processes, each using one dedicated CPU core.
% As mentioned in \secref{section:approach}, \ourtool receives candidate patches in the form of patch sets, but existing APR tools generate patches one by one. Therefore we cluster patches into multiple sets based on the statements each patch modified, and validate each set one by one.
% Our current implementation supports only one patch set, and we validate patch sets one by one. This is a limitation of our implementation but not our approach. Though \ourtool does not achieve the optimal performance under this setting, we found that existing APR tools generate many (59 on average) patches per statement and this setting is enough to demonstrate the effectiveness of the techniques.
For patches \ourtool ``fails to validate'', we fall back to the State-of-the-Art baseline in RQ1.
Candidate patches may contain endless loops. Following an existing study~\cite{ghanbari2020prf}, the test timeout is set to 5 seconds plus 1.5 times over the original test execution time.
%The execution time of timed-out tests is still considered in our evaluation, to reflect the real-world behavior.
UniAPR and \ourtool both have offline procedures to measure the original test execution time or analyze the purity of methods. They are excluded from time usage.
\section{Result Analysis}
\label{section:exp_result}
\newtcolorbox{rqbox}{left=1pt,right=1pt,top=1pt,bottom=1pt}
\subsection{RQ1: Overall Performance}
The total patch validation time per bug in both datasets using each approach is shown in \figref{fig:result_overall_scatter}. Note that UniAPR fails to run on 24\% of bugs, hence some points in the SOTA series are missing in the figure.
This result brings the following findings for the APR community:
\begin{figure}
\centering
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-overall-scatter-plausible.pdf}
\vspace{-1.75em}
\caption{fixable bugs}
\end{subfigure}
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-overall-scatter-sample.pdf}
\vspace{-1.75em}
\caption{random bugs}
\end{subfigure}
\vspace{-.5em}
\caption{Total patch validation time (seconds) per bug}
\label{fig:result_overall_scatter}
\vspace{-.75em}
\end{figure}
\textbf{1. \ourtool accelerates patch validation to a new degree in a variety of settings.} From \figref{fig:result_overall_scatter} we can see that in every studied APR tool, patch validation with \ourtool consumes significantly less time compared with the two baselines for almost all bugs. In the fixable dataset, \ourtool shows an acceleration of \accOverPlain{}x over the Plain baseline, or \accOverSota{}x over the State-of-the-Art baseline on bugs where UniAPR successfully runs.
In the random dataset, \ourtool shows a similar acceleration of 108.9x over Plain or 10.3x over State-of-the-Art. This indicates that the performance of ExpressAPR generalizes to different APR tools and different bugs.
\textbf{2. Patch validation is no longer the speed bottleneck of APR if mutation testing techniques are systematically adapted.} The percentage of time spent for patch validation when repairing each bug in the fixable dataset is shown in \figref{fig:result_overall_pvratio}. When using the stock Defects4J command (the Plain baseline), patch validation takes more than 95\% of the repair time. If the previous best patch validation approach (the State-of-the-Art baseline) is used, this portion is reduced to $75\% \sim 95\%$, still the bottleneck. \ourtool has reduced this portion to about 50\%, so patch validation is now as fast as patch generation. This suggests that future APR acceleration work should also consider the patch generation phase.
\begin{figure}
\centering
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-overall-pvratio-plausible.pdf}
\vspace{-1.5em}
\caption{fixable bugs}
\end{subfigure}
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-overall-pvratio-sample.pdf}
\vspace{-1.5em}
\caption{random bugs}
\end{subfigure}
\vspace{-.5em}
\caption{Percentage of time spent for patch validation}
\label{fig:result_overall_pvratio}
\vspace{-.75em}
\end{figure}
%\textbf{3. The speed of APR can be fast enough for a new usage scenario.}
\textbf{3. The speed of APR satisfies the expectation of most users.}
In our experiment with an 8-core configuration, when \ourtool is used, the total repair time for the fixable dataset has greatly reduced to less than 3 minutes for more than half bugs and less than 10 minutes for nearly all bugs, as shown in \figref{fig:result_overall_eachtime}. %This means that current APR users, who used to wait for an overnight response from a nightly build machine, can now get nearly instant response on their own device during code editting.}
This execution time satisfies the expectation of most users in an existing survey~\cite{noller2022trust}.
\begin{figure}
\centering
\includegraphics[width=.95\columnwidth]{figures/result-overall-eachtime-plausible.pdf}
\vspace{-.5em}
\caption{Total repair time (seconds) of fixable bugs using \ourtool}
\label{fig:result_overall_eachtime}
\vspace{-.75em}
\end{figure}
\subsection{RQ2: Technique Effectiveness}
In this RQ, we separately measure the effectiveness of four acceleration techniques in its step (patch compilation for MS; test execution for MD, TC, and TCP). Results on both datasets are shown in \figref{fig:result_speed_plot}. Each point $(x\%, y)$ in the plot indicates that the time usage of this step among $x$\% patch sets is within $y$ seconds.
We have the following findings:
\begin{figure}
\centering
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-speed-breakdown-plausible.pdf}
\vspace{-1.25em}
\caption{fixable bugs}
\end{subfigure}
\vspace{.75em}
\begin{subfigure}[b]{\columnwidth}
\includegraphics[width=\columnwidth]{figures/result-speed-breakdown-sample.pdf}
\vspace{-1.25em}
\caption{random bugs}
\end{subfigure}
\vspace{-.5em}
\caption{Effectiveness of each acceleration technique}
\label{fig:result_speed_plot}
\vspace{-.75em}
\end{figure}
\textbf{1. Acceleration techniques in ExpressAPR are effective on their own.}
\figref{fig:result_speed_plot} clearly shows that every addition of technique contributes to a significant acceleration in its step. The average contribution of each technique is over 3x. Note that the Y-Axis is logarithmic, so the acceleration ratio can be directly read from the distance of two series.
\textbf{2. With all techniques used, patch compilation takes more time than test execution for most patch sets.} This can be confirmed by comparing the ExpressAPR series in (i) and (ii). This is because: 1) When TCP is used, most patches fail the first few unit tests, which costs little time. 2) The portion of patches that do not compile is considerable, precisely 54\%, 51\%, 76\%, and 52\% for the four evaluated APR tools, and test execution is skipped for them. It suggests that future work may spend more effort optimizing patch compilation.
\textbf{3. The effectiveness of Mutant Deduplication depends on the patch space.} We can see that MD performs best with Hanabi, by comparing the ``-TCP -MD'' and the ``-TCP'' series in different figures. This makes sense because Hanabi, as a decision-tree-based APR approach, naturally produces many patches changing Boolean conditions. These patches are more likely to be test-equivalent because Boolean conditions are either true or false.
\subsection{RQ3: Feasibility}
In this RQ, we count the number of cases where \ourtool fails to accelerate and classify them by their reasons. We also compare the validation result reported by \ourtool and the Plain baseline for detecting incorrect results.
The result is shown in \tabref{tab:result_feasibility}, which leads to the following findings:
\newcommand{\feasibilityBecause}[1]{ {\textit{#1}}}
\begin{table}[t]
\centering\footnotesize
\caption{The feasibility of \ourtool}
\label{tab:result_feasibility}
\vspace{-.5em}
\begin{tabular}{@{\hspace{.25em}}l@{\hspace{1em}}r@{\hspace{1em}}r@{\hspace{.25em}}}
\toprule
\multirow{2}{*}{Category} & \multicolumn{2}{c}{\% of patches in ...} \\
& fixable bugs & random bugs \\
\midrule
Correct result & 97.197\% & 98.782\% \\
\midrule
Validation failure \feasibilityBecause{(patch limitation)} & 1.882\% & 0.693\% \\
\phantom{Validation failure} \feasibilityBecause{(test limitation)} & 0.886\% & 0.524\% \\
\midrule
Result misclassified \feasibilityBecause{(as plausible)} & 0.018\% & 0.000\% \\
\phantom{Result misclassified} \feasibilityBecause{(as implausible)} & 0.017\% & 0.001\% \\
\bottomrule
\end{tabular}
\vspace{-.5em}
\end{table}
\textbf{1. The acceleration feasibility of \ourtool is high.} In the fixable dataset and the random dataset, 97.197\% and 98.782\% of patches are correctly validated with acceleration. It indicates that the two implementation-level limitations do not affect the majority of patches.
\textbf{2. \ourtool has negligible negative impact on the effectiveness of APR.} Most limitation violations are detected by \ourtool by reporting a validation failure. In our implementation, the plain validation approach is automatically used for these patches as a fallback, keeping the validation result correct. Only a very small portion of patches (0.035\% of the fixable dataset and 0.001\% of the random dataset) are misclassified because of undetected unstable tests. Among them, some are implausible patches misclassified as plausible, which can be ruled out by a post-check using the fallback approach on all plausible patches. So only plausible patches misclassified as implausible will have a negative impact on the effectiveness of APR. Since this portion is very small (0.017\% and 0.001\%), we believe the impact is negligible.
\subsection{Threats to Validity}
Threats to \textit{internal validity} might come from the possible faults when implementing \ourtool and performing the experiment. To avoid faults in our implementation, we have added assertions and sanity checks in the code. We have also manually inspected misclassified patches in RQ3 and does not find faults in the final implementation. To mitigate timing errors due to varying system load, we used the \code{cgroup} mechanism in Linux to assign one dedicated CPU core and enough RAM resources to each process.
Threats to \textit{external validity} lie in the representativeness of the benchmark. We use Defects4J as the benchmark dataset, which contains defects in multiple open-source projects and is widely used for APR evaluation. We collect candidate patches from four APR tools, covering multiple kinds of approaches, each tool being the current state-of-the-art tool of its kind. Therefore, our results have a high chance to represent the general use cases.
% \todo{not sure whether we should add a `threat to validity' section. seems that some validity problems are discussed in RQ2.}
\section{Related Work}\label{section:related}
\subsection{Accelerating Patch Validation}
Existing approaches that accelerate patch validation can be categorized as \textit{general-purpose} or \textit{special-purpose}. General-purpose approaches~\cite{chen2021fast,ghanbari2020prf,guo2019speedup,mehne2018accelerating,weimer2013leveraging,qi2013efficient} take an arbitrary set of patches as input, while special-purpose approaches~\cite{wong2021varfix,ghanbari2019practical,hua2018practical,mechtaev2018test,chen2017contract,fast2010designing} are designed for and rely on a specific patch generation algorithm. \tabref{tab:related_work} summarizes acceleration techniques used in general-purpose approaches for easy comparison.
Below we compare our work against these existing approaches in three aspects.
\input{figures/related-work}
1. Our empirical contribution is to understand the overall performance of integrating all five techniques and their relative accelerations on top of the other techniques. No existing approach has integrated all these techniques, so these empirical results were previously unknown. %\xyaadd{so they do not provide such empirical insights.}
2. Our technical contribution is a set of novel techniques to overcome the challenges when adapting mutant schemata and mutant duplication to general-purpose patch validation. There are three existing approaches that have adapted mutant schemata within our knowledge. Two of them~\cite{chen2017contract,hua2018practical} are special-purpose: they are designed for a specific patch space that does not have the problem of uncompilable patches. Though the approach by \citet{mehne2018accelerating} is proposed as general-purpose, it does not discuss how to deal with uncompilable patches, which is one of our technical contributions. There are four existing approaches that have employed mutant deduplication within our knowledge. Three of them~\cite{weimer2013leveraging,hua2018practical,chen2017contract} statically detect the equivalence of patches and only reduce the patch validation time for fully equivalent patches. \citet{mechtaev2018test} uses the test-equivalence relationship to prune the patch space to avoid generating test-equivalent patches. However, their approach works for only a few special cases (interchangeable expressions and swappable statements) but cannot detect equivalence or test-equivalence in general (e.g., cannot detect \code{x+=2;} and \code{x++;x++;} are equivalent).
3. Some approaches~\cite{ghanbari2020prf,mehne2018accelerating,ghanbari2019practical,hua2018practical,fast2010designing} have adapted test case selection. We do not adapt test case selection because its effect is subsumed by mutant deduplication, as discussed in \secref{section:survey_not_used}.
\subsection{Mutation Testing}
\label{section:compare_mutation_testing}
There are multiple techniques for mutation testing acceleration, as surveyed in \secref{section:survey_used}. Our technical contribution is to adapt two classes of them, namely mutant schemata and mutant deduplication, to general-purpose patch validation. Below we compare our work against related mutation testing approaches.
1. Existing approaches with mutant schemata~\cite{untch1993mutation,just2011major,ma2005mujava} only allow pre-defined mutation operators against the program, and these operators are designed not to cause compile errors. In comparison, we allow arbitrary changes to statements, so the space of mutation is significantly enlarged, and compile errors are handled by compilation isolation. Thus, \ourtool is more suitable for patch validation against mainstream APR tools, where the patch space is huge, and many patches cannot compile.
2. Among four existing approaches with mutant deduplication~\cite{baldwin1979heuristics,papadakis2015trivial,pan1994using,just2014efficient}, three approaches~\cite{baldwin1979heuristics,papadakis2015trivial,pan1994using} can detect only fully equivalent mutants, so they are weaker than \ourtool, which detects test-equivalent mutants. The Major mutation framework~\cite{just2014efficient} detects test-equivalence by interpreting mutants in a pre-pass. As discussed in \secref{section:approach_overview}, if a mutant's state transition deviates from the original states, it requires a static analysis process, which may be hard to implement precisely, or fail to detect some test-equivalent mutants if the analysis is imprecise. In comparison, our \napp approach does not need such analyses, so it is easier to implement and detects more test-equivalent mutants.
%cannot track all paths in the pre-pass. In both \textit{fixable} and \textit{random} dataset in our evaluation, this is the case for more than 50\% of patch sets. % 2121/3956 for random; 3363/6560 for random
%Therefore, more than a half of patch sets cannot be accelerated by Major, leading to an acceleration ratio of strictly less than 2x, which is worse than \ourtool.
%VarexC has too much overhead when number of patches in a set is not exponentially high, as admitted in their paper~\cite{wong2018faster}. In their evaluation, VarexC performs even worse than plain execution for 4 out of 7 studied projects. The number of mutants are 20, 40, 146, and 940 for these four projects. Therefore, it will perform poorly for mainstream APR tools that emit no more than hundreds or thousands of patches.
\subsection{Improving APR Effectiveness}
While \ourtool mainly focuses on APR efficiency, readers may wonder about its improvement to APR effectiveness (``Can APR tools now fix more bugs in the same time budget?''). While this is an interesting question, it is orthogonal to our work: we can already study this without \ourtool. \citet{vu2021please} has shown that given exponentially more time, the APR effectiveness only increases linearly or not at all. Therefore we do not expect that \ourtool can help APR tools repair significantly more bugs in its original time limit (generally several hours~\cite{noller2022trust}). That being said, \ourtool is useful for this kind of study, because the time required for experiments can be greatly reduced.
\section{Conclusion} \label{section:conclusion}
%Usage scenarios of APR is limited due to excessively long patch validation time.
%While the duality of APR and mutation testing is well recognized and
%existing works for accelerating patch validation can be seen as adapting one or a few mutation testing techniques,
%no systematic work has been done yet on this topic.
We surveyed mutation testing acceleration techniques and selected five classes of applicable techniques for general-purpose patch validation. We proposed two novel approaches, namely \napp and \eins, to overcome technical challenges when adapting two of the techniques for the first time.
Our large-scale empirical experiment has shown that patch validation can be dramatically accelerated and no longer be the time bottleneck. When acceleration techniques are systematically used, thousands of patches can be validated in minutes, satisfying users' expectation.
%\section{Data Availability}
\textbf{
The \ourtool artifact, including the source code with documentation, a command-line interface for APR users, a Docker image to reproduce the experiment, and raw experiment results, is available~\cite{expressapr_replication}.
}
\newcommand{\apsize}{\footnotesize}
\newcommand{\noname}[1]{\apsize\textit{{\kern -1pt}\citeauthor{#1}{\kern -1pt}}}
\begin{table}[t]
\centering\footnotesize
\caption{Acceleration techniques in general-purpose patch validators}
\label{tab:related_work}
\vspace{-.5em}
\addtolength{\tabcolsep}{-5pt}
\begin{tabular}{l|ccccccc}
\toprule
Technique & \apsize \ourtool & \apsize UniAPR & \apsize PRF & DSU & \noname{mehne2018accelerating} & \apsize AE & \apsize FRTP \\
& \textit{(ours)} & \cite{chen2021fast} & \cite{ghanbari2020prf} & \cite{guo2019speedup} & \cite{mehne2018accelerating} & \cite{weimer2013leveraging} & \cite{qi2013efficient} \\
% Language & Java & Java & Java & Java & C & C & C & Java & Java & Java & C & Java & C \\
\midrule
Mutant Schemata & \Checkmark & & & & \#$^1$\hspace{-.5em} & & \\
Mutant Dedup. & \Checkmark & & & & & & \\
Test Virtualization & \Checkmark & \Checkmark & & \#$^2$\hspace{-.5em} & & & \\
Test Case Prio. & \Checkmark & \Checkmark & \Checkmark & \Checkmark & \Checkmark & \Checkmark & \Checkmark \\
Parallelization & \Checkmark & \#$^3$\hspace{-.5em} & \Checkmark & \#$^3$\hspace{-.5em} & \#$^3$\hspace{-.5em} & \Checkmark &\Checkmark \\
\bottomrule
\end{tabular}
\vspace{.25em}
{\raggedright\footnotesize
\#$^1$: Does not discuss how to deal with uncompilable patches. \\
\#$^2$: Reuses JVM without test virtualization, leading to incorrect results. \\
\#$^3$: No built-in parallelization, but is parallelizable. \\
}
\addtolength{\tabcolsep}{+2pt}
\vspace{-1em}
\end{table}
% Generated by IEEEtranN.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{68}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtranN.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem[Le et~al.(2016)Le, Lo, and Le~Goues]{le2016history}
X.~B.~D. Le, D.~Lo, and C.~Le~Goues, ``History driven program repair,'' in
\emph{2016 IEEE 23rd international conference on software analysis,
evolution, and reengineering (SANER)}, vol.~1.\hskip 1em plus 0.5em minus
0.4em\relax IEEE, 2016, pp. 213--224.
\bibitem[Liu et~al.(2019)Liu, Koyuncu, Kim, and Bissyand{\'e}]{liu2019tbar}
K.~Liu, A.~Koyuncu, D.~Kim, and T.~F. Bissyand{\'e}, ``{{TBar}}: {{Revisiting
Template-based Automated Program Repair}},'' in \emph{Proceedings of the 28th
{{ACM SIGSOFT International Symposium}} on {{Software Testing}} and
{{Analysis}}}, Jul. 2019, pp. 31--42.
\bibitem[Weimer et~al.(2009)Weimer, Nguyen, Le~Goues, and
Forrest]{weimer2009automatically}
W.~Weimer, T.~Nguyen, C.~Le~Goues, and S.~Forrest, ``Automatically finding
patches using genetic programming,'' in \emph{2009 IEEE 31st International
Conference on Software Engineering}.\hskip 1em plus 0.5em minus 0.4em\relax
IEEE, 2009, pp. 364--374.
\bibitem[Mechtaev et~al.(2018{\natexlab{a}})Mechtaev, Nguyen, Noller, Grunske,
and Roychoudhury]{mechtaev2018semantic}
S.~Mechtaev, M.-D. Nguyen, Y.~Noller, L.~Grunske, and A.~Roychoudhury,
``Semantic program repair using a reference implementation,'' in
\emph{Proceedings of the 40th International Conference on Software
Engineering}, 2018, pp. 129--139.
\bibitem[Liu et~al.(2013)Liu, Yang, Tan, and Hafiz]{liu2013r2fix}
C.~Liu, J.~Yang, L.~Tan, and M.~Hafiz, ``R2fix: Automatically generating bug
fixes from bug reports,'' in \emph{2013 IEEE Sixth international conference
on software testing, verification and validation}.\hskip 1em plus 0.5em minus
0.4em\relax IEEE, 2013, pp. 282--291.
\bibitem[Kim et~al.(2013)Kim, Nam, Song, and Kim]{kim2013automatic}
D.~Kim, J.~Nam, J.~Song, and S.~Kim, ``Automatic patch generation learned from
human-written patches,'' in \emph{2013 35th International Conference on
Software Engineering (ICSE)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE,
2013, pp. 802--811.
\bibitem[Wei et~al.(2010)Wei, Pei, Furia, Silva, Buchholz, Meyer, and
Zeller]{wei2010automated}
Y.~Wei, Y.~Pei, C.~A. Furia, L.~S. Silva, S.~Buchholz, B.~Meyer, and A.~Zeller,
``Automated fixing of programs with contracts,'' in \emph{Proceedings of the
19th international symposium on Software testing and analysis}, 2010, pp.
61--72.
\bibitem[Zhu et~al.(2021)Zhu, Sun, Xiao, Zhang, Yuan, Xiong, and
Zhang]{zhu2021syntaxguided}
Q.~Zhu, Z.~Sun, Y.-a. Xiao, W.~Zhang, K.~Yuan, Y.~Xiong, and L.~Zhang, ``A
syntax-guided edit decoder for neural program repair,'' in \emph{Proceedings
of the 29th {{ACM Joint Meeting}} on {{European Software Engineering
Conference}} and {{Symposium}} on the {{Foundations}} of {{Software
Engineering}}}.\hskip 1em plus 0.5em minus 0.4em\relax {New York, NY, USA}:
{ACM}, Aug. 2021, pp. 341--353.
\bibitem[Kirbas et~al.(2021)Kirbas, Windels, McBello, Kells, Pagano, Szalanski,
Nowack, Winter, Counsell, Bowes, Hall, Haraldsson, and
Woodward]{kirbas2021introduction}
S.~Kirbas, E.~Windels, O.~McBello, K.~Kells, M.~Pagano, R.~Szalanski,
V.~Nowack, E.~R. Winter, S.~Counsell, D.~Bowes, T.~Hall, S.~Haraldsson, and
J.~Woodward, ``On {{The Introduction}} of {{Automatic Program Repair}} in
{{Bloomberg}},'' \emph{IEEE Softw.}, vol.~38, no.~4, pp. 43--51, Jul. 2021.
\bibitem[Marginean et~al.(2019)Marginean, Bader, Chandra, Harman, Jia, Mao,
Mols, and Scott]{marginean2019sapfix}
A.~Marginean, J.~Bader, S.~Chandra, M.~Harman, Y.~Jia, K.~Mao, A.~Mols, and
A.~Scott, ``{{SapFix}}: {{Automated End-to-End Repair}} at {{Scale}},''
\emph{2019 IEEE/ACM 41st International Conference on Software Engineering:
Software Engineering in Practice (ICSE-SEIP)}, p.~10, 2019.
\bibitem[Zhang et~al.(2020)Zhang, Zhu, Li, Guo, Liu, and Gu]{zhang2020precfix}
X.~Zhang, C.~Zhu, Y.~Li, J.~Guo, L.~Liu, and H.~Gu, ``Precfix: Large-scale
patch recommendation by mining defect-patch pairs,'' in \emph{Proceedings of
the ACM/IEEE 42nd International Conference on Software Engineering: Software
Engineering in Practice}, 2020, pp. 41--50.
\bibitem[Noller et~al.(2022)Noller, Shariffdeen, Gao, and
Roychoudhury]{noller2022trust}
Y.~Noller, R.~Shariffdeen, X.~Gao, and A.~Roychoudhury, ``Trust {{Enhancement
Issues}} in {{Program Repair}},'' in \emph{Proceedings of the 44th
{{International Conference}} on {{Software Engineering}}}, {Pittsburgh
Pennsylvania}, 2022, p.~13.
\bibitem[Liang et~al.(2021)Liang, Ji, Jiang, Zhou, Lou, Xiong, and
Huang]{liang2021interactive}
J.~Liang, R.~Ji, J.~Jiang, S.~Zhou, Y.~Lou, Y.~Xiong, and G.~Huang,
``Interactive patch filtering as debugging aid,'' in \emph{2021 IEEE
International Conference on Software Maintenance and Evolution
(ICSME)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 239--250.
\bibitem[Le~Goues et~al.(2013)Le~Goues, Forrest, and Weimer]{le2013current}
C.~Le~Goues, S.~Forrest, and W.~Weimer, ``Current challenges in automatic
software repair,'' \emph{Software quality journal}, vol.~21, no.~3, pp.
421--443, 2013.
\bibitem[Ghanbari and Marcus(2020)]{ghanbari2020prf}
A.~Ghanbari and A.~Marcus, ``{{PRF}}: A framework for building automatic
program repair prototypes for {{JVM-based}} languages,'' in \emph{Proceedings
of the 28th {{ACM Joint Meeting}} on {{European Software Engineering
Conference}} and {{Symposium}} on the {{Foundations}} of {{Software
Engineering}}}.\hskip 1em plus 0.5em minus 0.4em\relax {New York, NY, USA}:
{ACM}, Nov. 2020, pp. 1626--1629.
\bibitem[Liu et~al.(2020)Liu, Wang, Koyuncu, Kim, Bissyand{\'e}, Kim, Wu,
Klein, Mao, and Traon]{liu2020efficiency}
K.~Liu, S.~Wang, A.~Koyuncu, K.~Kim, T.~F. Bissyand{\'e}, D.~Kim, P.~Wu,
J.~Klein, X.~Mao, and Y.~L. Traon, ``On the efficiency of test suite based
program repair,'' in \emph{Proceedings of the {{ACM}}/{{IEEE}} 42nd
{{International Conference}} on {{Software Engineering}}}.\hskip 1em plus
0.5em minus 0.4em\relax {New York, NY, USA}: {ACM}, Jun. 2020, pp. 615--627.
\bibitem[Weimer et~al.(2013)Weimer, Fry, and Forrest]{weimer2013leveraging}
W.~Weimer, Z.~P. Fry, and S.~Forrest, ``Leveraging program equivalence for
adaptive program repair: Models and first results,'' \emph{2013 28th IEEE/ACM
International Conference on Automated Software Engineering, ASE 2013 -
Proceedings}, pp. 356--366, 2013.
\bibitem[Chen et~al.(2017)Chen, Pei, and Furia]{chen2017contract}
L.~Chen, Y.~Pei, and C.~A. Furia, ``Contract-based program repair without the
contracts,'' in \emph{2017 32nd IEEE/ACM International Conference on
Automated Software Engineering (ASE)}.\hskip 1em plus 0.5em minus 0.4em\relax
IEEE, 2017, pp. 637--647.
\bibitem[Mechtaev et~al.(2018{\natexlab{b}})Mechtaev, Gao, Tan, and
Roychoudhury]{mechtaev2018test}
S.~Mechtaev, X.~Gao, S.~H. Tan, and A.~Roychoudhury, ``Test-equivalence
analysis for automatic patch generation,'' \emph{ACM Transactions on Software
Engineering and Methodology (TOSEM)}, vol.~27, no.~4, pp. 1--37, 2018.
\bibitem[Hua et~al.(2018)Hua, Zhang, Wang, and Khurshid]{hua2018practical}
J.~Hua, M.~Zhang, K.~Wang, and S.~Khurshid, ``Towards practical program repair
with on-demand candidate generation,'' in \emph{Proceedings of the 40th
{{International Conference}} on {{Software Engineering}}}.\hskip 1em plus
0.5em minus 0.4em\relax {New York, NY, USA}: {ACM}, May 2018, pp. 12--23.
\bibitem[Chen et~al.(2021)Chen, Ouyang, and Zhang]{chen2021fast}
L.~Chen, Y.~Ouyang, and L.~Zhang, ``Fast and precise on-the-fly patch
validation for all,'' in \emph{2021 {{IEEE}}/{{ACM}} 43rd {{International
Conference}} on {{Software Engineering}} ({{ICSE}})}.\hskip 1em plus 0.5em
minus 0.4em\relax {IEEE}, May 2021, pp. 1123--1134.
\bibitem[Guo et~al.(2019)Guo, Gu, Yao, Xu, and Ma]{guo2019speedup}
R.~Guo, T.~Gu, Y.~Yao, F.~Xu, and X.~Ma, ``Speedup automatic program repair
using dynamic software updating,'' in \emph{Proceedings of the 11th
{{Asia-Pacific Symposium}} on {{Internetware}}}.\hskip 1em plus 0.5em minus
0.4em\relax {New York, NY, USA}: {ACM}, Oct. 2019, pp. 1--10.
\bibitem[Mehne et~al.(2018)Mehne, Yoshida, Prasad, Sen, Gopinath, and
Khurshid]{mehne2018accelerating}
B.~Mehne, H.~Yoshida, M.~R. Prasad, K.~Sen, D.~Gopinath, and S.~Khurshid,
``Accelerating search-based program repair,'' \emph{Proceedings - 2018 IEEE
11th International Conference on Software Testing, Verification and
Validation, ICST 2018}, pp. 227--238, 2018.
\bibitem[Qi et~al.(2013)Qi, Mao, and Lei]{qi2013efficient}
Y.~Qi, X.~Mao, and Y.~Lei, ``Efficient automated program repair through
fault-recorded testing prioritization,'' \emph{IEEE International Conference
on Software Maintenance, ICSM}, no. March, pp. 180--189, 2013.
\bibitem[Ghanbari et~al.(2019)Ghanbari, Benton, and
Zhang]{ghanbari2019practical}
A.~Ghanbari, S.~Benton, and L.~Zhang, ``Practical program repair via bytecode
mutation,'' in \emph{Proceedings of the 28th {{ACM SIGSOFT International
Symposium}} on {{Software Testing}} and {{Analysis}}}.\hskip 1em plus 0.5em
minus 0.4em\relax {New York, NY, USA}: {ACM}, Jul. 2019, pp. 19--30.
\bibitem[Wong et~al.(2021)Wong, Santiesteban, K{\"a}stner, and
Le~Goues]{wong2021varfix}
C.-P. Wong, P.~Santiesteban, C.~K{\"a}stner, and C.~Le~Goues, ``{{VarFix}}:
Balancing edit expressiveness and search effectiveness in automated program
repair,'' in \emph{Proceedings of the 29th {{ACM Joint Meeting}} on
{{European Software Engineering Conference}} and {{Symposium}} on the
{{Foundations}} of {{Software Engineering}}}, vol.~1.\hskip 1em plus 0.5em
minus 0.4em\relax {New York, NY, USA}: {ACM}, Aug. 2021, pp. 354--366.
\bibitem[Fast et~al.(2010)Fast, Le~Goues, Forrest, and
Weimer]{fast2010designing}
E.~Fast, C.~Le~Goues, S.~Forrest, and W.~Weimer, ``Designing better fitness
functions for automated program repair,'' in \emph{Proceedings of the 12th
annual conference on Genetic and evolutionary computation}, 2010, pp.
965--972.
\bibitem[Pacheco et~al.(2007)Pacheco, Lahiri, Ernst, and
Ball]{pacheco2007feedback}
C.~Pacheco, S.~K. Lahiri, M.~D. Ernst, and T.~Ball, ``Feedback-directed random
test generation,'' in \emph{29th International Conference on Software
Engineering (ICSE'07)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2007,
pp. 75--84.
\bibitem[Untch et~al.(1993)Untch, Offutt, and Harrold]{untch1993mutation}
R.~H. Untch, A.~J. Offutt, and M.~J. Harrold, ``Mutation analysis using mutant
schemata,'' \emph{Proceedings of the 1993 ACM SIGSOFT international symposium
on Software testing and analysis}, 1993.
\bibitem[Just et~al.(2011)Just, Schweiggert, and Kapfhammer]{just2011major}
R.~Just, F.~Schweiggert, and G.~M. Kapfhammer, ``{{MAJOR}}: An efficient and
extensible tool for mutation analysis in a java compiler,'' in \emph{2011
26th {{IEEE}}/{{ACM International Conference}} on {{Automated Software
Engineering}}, {{ASE}} 2011, {{Proceedings}}}, 2011, pp. 612--615.
\bibitem[Bell and Kaiser(2014)]{bell2014unit}
J.~Bell and G.~Kaiser, ``Unit test virtualization with {{VMVM}},''
\emph{Proceedings - International Conference on Software Engineering}, no.~1,
pp. 550--561, 2014.
\bibitem[Zhang et~al.(2013)Zhang, Marinov, and Khurshid]{zhang2013faster}
L.~Zhang, D.~Marinov, and S.~Khurshid, ``Faster mutation testing inspired by
test prioritization and reduction,'' \emph{2013 International Symposium on
Software Testing and Analysis, ISSTA 2013 - Proceedings}, pp. 235--245, 2013.
\bibitem[Just et~al.(2014{\natexlab{a}})Just, Ernst, and
Fraser]{just2014efficient}
R.~Just, M.~D. Ernst, and G.~Fraser, ``Efficient mutation analysis by
propagating and partitioning infected execution states,'' in \emph{2014
{{International Symposium}} on {{Software Testing}} and {{Analysis}},
{{ISSTA}} 2014 - {{Proceedings}}}, 2014, pp. 315--326.
\bibitem[Xiao et~al.(2023)Xiao, Yang, Wang, and Xiong]{expressapr_replication}
\BIBentryALTinterwordspacing
Y.-A. Xiao, C.~Yang, B.~Wang, and Y.~Xiong. (2023) The replication package for
{ExpressAPR}. [Online]. Available:
\url{https://github.com/ExpressAPR/ExpressAPR}
\BIBentrySTDinterwordspacing
\bibitem[Jia and Harman(2010)]{jia2010analysis}
Y.~Jia and M.~Harman, ``An analysis and survey of the development of mutation
testing,'' \emph{IEEE transactions on software engineering}, pp. 1--31, 2010.
\bibitem[Papadakis et~al.(2019)Papadakis, Kintis, Zhang, Jia, Traon, and
Harman]{papadakis2019mutation}
M.~Papadakis, M.~Kintis, J.~Zhang, Y.~Jia, Y.~L. Traon, and M.~Harman,
``Mutation testing advances: An analysis and survey,'' in \emph{Advances in
{{Computers}}}, 2019, vol. 112, pp. 275--378.
\bibitem[Pizzoleto et~al.(2019)Pizzoleto, Ferrari, Offutt, Fernandes, and
Ribeiro]{pizzoleto2019systematic}
A.~V. Pizzoleto, F.~C. Ferrari, J.~Offutt, L.~Fernandes, and M.~Ribeiro, ``A
systematic literature review of techniques and metrics to reduce the cost of
mutation testing,'' \emph{Journal of Systems and Software}, vol. 157, p.
110388, 2019.
\bibitem[Coles(2021)]{Pitest_java_mutation_testing_systems}
\BIBentryALTinterwordspacing
H.~Coles. (2021, Jul.) Mutation testing systems for {{Java}} compared.
[Online]. Available: \url{http://pitest.org/java_mutation_testing_systems/}
\BIBentrySTDinterwordspacing
\bibitem[Ma et~al.(2005)Ma, Offutt, and Kwon]{ma2005mujava}
Y.~S. Ma, J.~Offutt, and Y.~R. Kwon, ``{{MuJava}}: An automated class mutation
system,'' \emph{Software Testing Verification and Reliability}, vol.~15,
no.~2, pp. 97--133, 2005.
\bibitem[Baldwin and Sayward(1979)]{baldwin1979heuristics}
D.~Baldwin and F.~Sayward, ``Heuristics for determining equivalence of program
mutations.'' GEORGIA INST OF TECH ATLANTA SCHOOL OF INFORMATION AND COMPUTER
SCIENCE, Tech. Rep., 1979.
\bibitem[Papadakis et~al.(2015)Papadakis, Jia, Harman, and
Le~Traon]{papadakis2015trivial}
M.~Papadakis, Y.~Jia, M.~Harman, and Y.~Le~Traon, ``Trivial compiler
equivalence: A large scale empirical study of a simple, fast and effective
equivalent mutant detection technique,'' in \emph{2015 IEEE/ACM 37th IEEE
International Conference on Software Engineering}, vol.~1.\hskip 1em plus
0.5em minus 0.4em\relax IEEE, 2015, pp. 936--946.
\bibitem[Pan(1994)]{pan1994using}
J.~Pan, ``Using constraints to detect equivalent mutants,'' Master's thesis,
George Mason University Master’s thesis, 1994.
\bibitem[Yoo and Harman(2012)]{yoo2012regression}
S.~Yoo and M.~Harman, ``Regression testing minimization, selection and
prioritization: a survey,'' \emph{Software testing, verification and
reliability}, vol.~22, no.~2, pp. 67--120, 2012.
\bibitem[Schuler and Zeller(2009)]{schuler2009javalanche}
D.~Schuler and A.~Zeller, ``Javalanche: Efficient mutation testing for java,''
in \emph{Proceedings of the 7th joint meeting of the European software
engineering conference and the ACM SIGSOFT symposium on The foundations of
software engineering}, 2009, pp. 297--298.
\bibitem[Krauser et~al.(1991)Krauser, Mathur, and Rego]{krauser1991high}
E.~W. Krauser, A.~P. Mathur, and V.~J. Rego, ``High performance software
testing on simd machines,'' \emph{IEEE Transactions on Software Engineering},
vol.~17, no.~5, pp. 403--423, 1991.
\bibitem[Offutt et~al.(1992)Offutt, Pargas, Fichter, and
Khambekar]{offutt1992mutation}
A.~J. Offutt, R.~P. Pargas, S.~V. Fichter, and P.~K. Khambekar, ``Mutation
testing of software using a mimd computer,'' in \emph{Proc. ICPP}, 1992.
\bibitem[Tokumoto et~al.(2016)Tokumoto, Yoshida, Sakamoto, and
Honiden]{tokumoto2016muvm}
S.~Tokumoto, H.~Yoshida, K.~Sakamoto, and S.~Honiden, ``{{MuVM}}: Higher order
mutation analysis virtual machine for {{C}},'' \emph{Proceedings - 2016 IEEE
International Conference on Software Testing, Verification and Validation,
ICST 2016}, pp. 320--329, 2016.
\bibitem[Gopinath et~al.(2016)Gopinath, Jensen, and Groce]{gopinath2016topsy}
R.~Gopinath, C.~Jensen, and A.~Groce, ``Topsy-{Turvy}: a smarter and faster
parallelization of mutation analysis,'' in \emph{ICSE}, 2016, pp. 740--743.
\bibitem[Wang et~al.(2017)Wang, Xiong, Shi, Zhang, and Hao]{wang2017faster}
B.~Wang, Y.~Xiong, Y.~Shi, L.~Zhang, and D.~Hao, ``Faster mutation analysis via
equivalence modulo states,'' in \emph{{{ISSTA}} 2017 - {{Proceedings}} of the
26th {{ACM SIGSOFT International Symposium}} on {{Software Testing}} and
{{Analysis}}}, Feb. 2017, pp. 295--306.
\bibitem[Wang et~al.(2021)Wang, Lu, Xiong, and Liu]{wang2021faster}
B.~Wang, S.~Lu, Y.~Xiong, and F.~Liu, ``Faster mutation analysis with fewer
processes and smaller overheads,'' in \emph{2021 36th IEEE/ACM International
Conference on Automated Software Engineering (ASE)}, 2021, pp. 381--393.
\bibitem[Wong et~al.(2018{\natexlab{a}})Wong, Meinicke, Lazarek, and
K{\"a}stner]{wong2018faster}
C.-P. Wong, J.~Meinicke, L.~Lazarek, and C.~K{\"a}stner, ``Faster variational
execution with transparent bytecode transformation,'' \emph{Proceedings of
the ACM on Programming Languages}, vol.~2, no. OOPSLA, pp. 1--30, 2018.
\bibitem[Wong et~al.(2018{\natexlab{b}})Wong, Meinicke, and
K{\"a}stner]{wong2018testing}
C.~P. Wong, J.~Meinicke, and C.~K{\"a}stner, ``Beyond testing configurable
systems: Applying variational execution to automatic program repair and
higher order mutation testing,'' \emph{ESEC/FSE 2018 - Proceedings of the
2018 26th ACM Joint Meeting on European Software Engineering Conference and
Symposium on the Foundations of Software Engineering}, pp. 749--753, 2018.
\bibitem[Mathur and Wong(1994)]{mathur1994empirical}
A.~P. Mathur and W.~E. Wong, ``An empirical comparison of data flow and
mutation-based test adequacy criteria,'' \emph{Software Testing, Verification
and Reliability}, vol.~4, no.~1, pp. 9--31, 1994.
\bibitem[Zhang et~al.(2018)Zhang, Zhang, Harman, Hao, Jia, and
Zhang]{zhang2018predictive}
J.~Zhang, L.~Zhang, M.~Harman, D.~Hao, Y.~Jia, and L.~Zhang, ``Predictive
mutation testing,'' \emph{IEEE Transactions on Software Engineering},
vol.~45, no.~9, pp. 898--918, 2018.
\bibitem[Howden(1982)]{howden1982weak}
W.~E. Howden, ``Weak mutation testing and completeness of test sets,''
\emph{IEEE Transactions on Software Engineering}, no.~4, pp. 371--379, 1982.
\bibitem[Offutt and Lee(1994)]{offutt1994empirical}
A.~J. Offutt and S.~D. Lee, ``An empirical evaluation of weak mutation,''
\emph{IEEE Transactions on Software Engineering}, vol.~20, no.~5, pp.
337--344, 1994.
\bibitem[Wong and Mathur(1995)]{wong1995reducing}
W.~E. Wong and A.~P. Mathur, ``Reducing the cost of mutation testing: An
empirical study,'' \emph{Journal of Systems and Software}, vol.~31, no.~3,
pp. 185--196, 1995.
\bibitem[Offutt et~al.(1993)Offutt, Rothermel, and
Zapf]{offutt1993experimental}
A.~J. Offutt, G.~Rothermel, and C.~Zapf, ``An experimental evaluation of
selective mutation,'' in \emph{Proceedings of 1993 15th international
conference on software engineering}.\hskip 1em plus 0.5em minus 0.4em\relax
IEEE, 1993, pp. 100--107.
\bibitem[Ji et~al.(2009)Ji, Chen, Xu, and Zhao]{ji2009novel}
C.~Ji, Z.~Chen, B.~Xu, and Z.~Zhao, ``A novel method of mutation clustering
based on domain analysis,'' in \emph{Seke}, vol.~9, 2009, pp. 422--425.
\bibitem[Gligoric et~al.(2015)Gligoric, Eloussi, and
Marinov]{gligoric2015practical}
M.~Gligoric, L.~Eloussi, and D.~Marinov, ``Practical regression test selection
with dynamic file dependencies,'' in \emph{Proceedings of the 2015
International Symposium on Software Testing and Analysis}, 2015, pp.
211--222.
\bibitem[Harrold et~al.(2001)Harrold, Jones, Li, Liang, Orso, Pennings, Sinha,
Spoon, and Gujarathi]{harrold2001regression}
M.~J. Harrold, J.~A. Jones, T.~Li, D.~Liang, A.~Orso, M.~Pennings, S.~Sinha,
S.~A. Spoon, and A.~Gujarathi, ``Regression test selection for java
software,'' \emph{ACM Sigplan Notices}, vol.~36, no.~11, pp. 312--326, 2001.
\bibitem[Pierce et~al.(2010)Pierce, Casinghino, Gaboardi, Greenberg,
Hri{\c{t}}cu, Sj{\"o}berg, and Yorgey]{pierce2010software}
B.~C. Pierce, C.~Casinghino, M.~Gaboardi, M.~Greenberg, C.~Hri{\c{t}}cu,
V.~Sj{\"o}berg, and B.~Yorgey, ``Software foundations,'' \emph{Webpage:
http://www.cis.upenn.edu/bcpierce/sf/current/index. html}, 2010.
\bibitem[Bravenboer and Smaragdakis(2009)]{bravenboer2009strictly}
\BIBentryALTinterwordspacing
M.~Bravenboer and Y.~Smaragdakis, ``Strictly declarative specification of
sophisticated points-to analyses,'' in \emph{Proceedings of the 24th ACM
SIGPLAN Conference on Object Oriented Programming Systems Languages and
Applications}, ser. OOPSLA '09.\hskip 1em plus 0.5em minus 0.4em\relax New
York, NY, USA: Association for Computing Machinery, 2009, p. 243–262.
[Online]. Available: \url{https://doi.org/10.1145/1640089.1640108}
\BIBentrySTDinterwordspacing
\bibitem[Gosling et~al.(2015)Gosling, Joy, Steele, Bracha, and Buckley]{jls8}
\BIBentryALTinterwordspacing
J.~Gosling, B.~Joy, G.~Steele, G.~Bracha, and A.~Buckley. (2015) The
{{Java(R)}} language specification: {{Java}} {{SE}} 8 edition. [Online].
Available: \url{https://docs.oracle.com/javase/specs/jls/se8/html/index.html}
\BIBentrySTDinterwordspacing
\bibitem[Just et~al.(2014{\natexlab{b}})Just, Jalali, and
Ernst]{Just2014defects4j}
R.~Just, D.~Jalali, and M.~D. Ernst, ``{{Defects4J}}: A database of existing
faults to enable controlled testing studies for java programs,'' in
\emph{Proceedings of the 2014 {{International Symposium}} on {{Software
Testing}} and {{Analysis}} - {{ISSTA}} 2014}.\hskip 1em plus 0.5em minus
0.4em\relax {New York, New York, USA}: {ACM Press}, 2014, pp. 437--440.
\bibitem[Jiang et~al.(2018)Jiang, Xiong, Zhang, Gao, and
Chen]{jiang2018shaping}
J.~Jiang, Y.~Xiong, H.~Zhang, Q.~Gao, and X.~Chen, ``Shaping program repair
space with existing patches and similar code,'' in \emph{Proceedings of the
27th ACM SIGSOFT international symposium on software testing and analysis},
2018, pp. 298--309.
\bibitem[Xiong and Wang(2021)]{xiong2021l2s}
Y.~Xiong and B.~Wang, ``{{L2S}}: A framework for synthesizing the most probable
program under a specification,'' \emph{TOSEM: ACM Transactions on Software
Engineering and Methodology}, 2021.
\bibitem[Vu et~al.(2021)Vu, Pashchenko, and Massacci]{vu2021please}
D.-L. Vu, I.~Pashchenko, and F.~Massacci, ``Please hold on: more time = more
patches? automated program repair as anytime algorithms,'' in \emph{2021
IEEE/ACM International Workshop on Automated Program Repair (APR)}.\hskip 1em
plus 0.5em minus 0.4em\relax IEEE, 2021, pp. 9--10.
\end{thebibliography}