Impact of Log Parsing on Deep Learning-Based Anomaly Detection
Extracted source text · arXiv 2305.15897
\documentclass{article}
\usepackage{pgfplotstable}
\usepackage{tabularx}
\usepackage{hyphenat}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{paralist}
\usepackage{natbib}
\usepackage{booktabs}
\usepackage{adjustbox}
\usepackage[linesnumbered, ruled, vlined]{algorithm2e}
\usepackage{pifont}\newcommand{\cmark}{\ding{51}}\newcommand{\xmark}{\ding{55}}\usepackage{amsmath,amsfonts}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{subfig}
\usepackage{float}
\usepackage{xcolor}
\usepackage[symbol]{footmisc}
\usepackage{makecell}
\usepackage{tcolorbox}
\usepackage{color, colortbl}
\title{Impact of Log Parsing on Log-based Anomaly Detection }
\begin{document}
\author{
Zanis Ali Khan\\
University of Luxembourg, Luxembourg \\
\texttt{zanis-ali.khan@uni.lu} \\
\and
Donghwan Shin \\
University of Sheffield, United Kingdom \\
\texttt{d.shin@sheffield.ac.uk} \\
\and
Domenico Bianculli \\
University of Luxembourg, Luxembourg \\
\texttt{domenico.bianculli@uni.lu} \\
\and
Lionel Briand\\
University of Luxembourg, Luxembourg \\
University of Ottawa, Canada \\
\texttt{lionel.briand@uni.lu} \\
}
\maketitle
\begin{abstract}
Software systems log massive amounts of data, recording
important runtime information. Such logs are used, for example, for log-based
anomaly detection, which aims to automatically detect abnormal
behaviors of the system under analysis by processing the information
recorded in its logs. Many log-based anomaly detection
techniques based on deep-learning models include a pre-processing step called log parsing. However, understanding
the impact of log parsing on the accuracy of anomaly detection
techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted.
In this paper, we report on a comprehensive empirical study on the
impact of log parsing on anomaly detection accuracy, using 13 log
parsing techniques and five deep-learning-based anomaly detection
techniques on two publicly available log datasets. Our empirical
results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and
anomaly detection accuracy (regardless of the metric used for
measuring log parsing accuracy). Moreover, we experimentally confirm
existing theoretical results showing that it is a property that we refer to as distinguishability in
log parsing results---as opposed to their accuracy---that plays an essential role in achieving accurate anomaly detection.
\noindent\textbf{Keywords:} logs, log parsing, template identification, anomaly detection
\end{abstract}
\section{Introduction}\label{sec:introduction}
Software system execution logs provide valuable information about the runtime
behavior of the system, which is essential for monitoring and troubleshooting.
Among many log analysis approaches, \textit{log-based anomaly detection} has
been actively studied to automatically detect abnormal behaviors of the
system under analysis by processing the information recorded in
logs~\cite{10.1145/3460345}. Recently, anomaly detection techniques based on
Deep Learning (DL) models, such as Long Short-Term Memory
(LSTM)~\cite{du2017deeplog,meng2019loganomaly,zhang2019robust} and Convolutional
Neural Networks (CNNs)~\cite{lu2018detecting}, have shown promising results.
One common aspect of most anomaly detection techniques is having a
pre-processing step called \emph{log parsing} (also known as log template
identification). This step is needed because anomaly detection techniques require structured
logs to automatically process them, whereas input logs are often free-formed or
semi-structured, as generated by logging statements (e.g., \texttt{printf()}
and \texttt{logger.info()}) in the source code. Many log parsing
techniques have also been developed to automatically convert unstructured input
logs into structured logs~\cite{zhu2019tools}.
The frequent combination of log parsing and anomaly detection clearly implies the
importance of the former for the latter. Nevertheless,
assessing in a systematic way the
impact of log parsing on anomaly detection has received surprisingly little
attention so far. Only
recently, \citet{shin2021theoretical} investigated what \textit{ideal} log parsing results
are in terms of accurate anomaly detection, but purely from a theoretical standpoint.
\citet{le2022log} empirically showed that different log parsing techniques,
among other potential factors, can significantly affect anomaly detection
accuracy, but the accuracy of log parsing results was not adequately measured, and
the correlation between log parsing accuracy and anomaly detection accuracy
was not reported. Although \citet{fu2023empirical} attempted to address the
issue by measuring log parsing and anomaly detection accuracy, they used only
one log parsing accuracy metric among those available in the
literature~\cite{khan2022guidelines}; moreover,
the log parsing results used to measure the accuracy of the anomaly
detection techniques represented only a small subset
(less than 1\%) of all logs used by the anomaly detection techniques
considered in the study, thus limiting the validity of the results.
To systematically investigate the impact of log parsing on anomaly detection
while addressing the issues of the aforementioned studies, this paper
reports on an empirical study, in which we performed a
comprehensive evaluation using 13 log parsing techniques and five anomaly detection
techniques on two publicly available log datasets. We considered all three log
parsing accuracy metrics (i.e., grouping accuracy~\cite{zhu2019tools}, parsing
accuracy~\cite{dai2020logram}, and template accuracy~\cite{khan2022guidelines})
proposed in the literature.
Against all assumptions, our results show that there is no strong correlation between log
parsing accuracy and anomaly detection accuracy, regardless of
the metric used for measuring log parsing accuracy. In other words,
accurate log parsing results do not necessarily increase anomaly detection
accuracy. To better understand the phenomenon at play, we investigated another property of log parsing, \emph{distinguishability}, a concept proposed by \citet{shin2021theoretical}) that was theoretically shown to relate to anomaly detection accuracy. Our empirical results confirm that distinguishability in log parsing results is the property that really matters as far as anomaly detection accuracy is concerned and that it should be the key target of log parsing.
In summary, the main contributions of this paper are:
\begin{itemize}
\item the systematic and comprehensive evaluation of the impact of log parsing
on anomaly detection;
\item the investigation of the impact of the distinguishability of log parsing
results on anomaly detection;
\item a discussion of the practical implications for the application
of log parsing in the context of anomaly detection.
\end{itemize}
The rest of the paper is organized as follows. Section~\ref{sec:background}
provides basic information used throughout the paper, including the definitions
of logs, messages, and templates, as well as an overview of log parsing and
anomaly detection. Section~\ref{sec:motivation} motivates our study and
introduces the research questions. Section~\ref{sec:expr-design} describes
the experimental design, including the log datasets, log parsing techniques, and
anomaly detection techniques used in the experiments. Section~\ref{sec:results}
presents the experimental results. Section~\ref{sec:implications} discusses the
practical implications derived from the results.
Section~\ref{sec:conclusion-future-work} concludes the paper and
provides directions for future
work.
\section{Background}\label{sec:background}
In this section, we provide an overview of the main concepts that will
be used throughout the paper.
We first introduce the definitions of logs, messages, and log templates (\S~\ref{sec:pre}).
We then explain the concept of log parsing (also known as log template
identification) and illustrate different log parsing accuracy metrics proposed in the literature (\S~\ref{sec:log-parsing}).
We discuss log-based anomaly detection and the corresponding accuracy
metrics in \S~\ref{sec:anomaly-detection}.
Finally, we summarize the recent theoretical results on ideal log
parsing for accurate anomaly detection, introducing the concept of \emph{distinguishability} for log parsing results (\S~\ref{sec:distinguishability}).
\subsection{Logs, Messages, and Templates}\label{sec:pre}
A \emph{log} is a sequence of log entries\footnote{Note that a log is different
from a log file. In practice, \emph{one} log file may contain \emph{many} logs
representing the execution flows of different components/sessions. For example,
an HDFS (Hadoop Distributed File System) log file contains many logs,
distinguished by file block IDs, each representing an independent execution for
a specific block.}.
A \emph{log entry} contains various information about the event being
logged, including a timestamp, a logging level (e.g., \texttt{INFO}, \texttt{DEBUG}), and a log message.
A \emph{log message} can be further decomposed into fixed and variable parts
since it is generated by executing a logging statement that can have
both fixed (hard-coded) strings and program variables in the source code.
For example, the execution of the logging statement
``\texttt{logger.info("Deleting block " + blkID + " file " + fileName)}'' when the program variables
\texttt{blkID} and \texttt{fileName} evaluate to \texttt{blk-1781} and \texttt{/hadoop/dfs}, respectively,
will generate a log entry ``\texttt{11:22:33 INFO Deleting block blk-1718 file /hadoop/dfs}''
where the log message ``\texttt{Deleting block blk-1718 file /hadoop/dfs}'' can be decomposed into
the fixed parts (i.e., ``\texttt{Deleting block}'' and ``\texttt{file}'') and the variable parts (i.e., ``\texttt{blk-1718}''
and ``\texttt{/hadoop/dfs}'').
A \emph{(log message) template} masks the various elements of each
variable part with a special character ``\texttt{<*>}''; this
representation is widely used in log-based analyses (e.g., log
parsing~\cite{he2017drain,jiang2008abstracting}, anomaly
detection~\cite{zhang2019robust,du2017deeplog}, and log-based
testing~\cite{elyasov2012log, jeong2020log}) when it is important to focus on
the event types captured by a log message.
For instance, the template corresponding to the example log message
``\texttt{Deleting block blk-1178 file /hadoop/dfs}'' is ``\texttt{Deleting block
<*> file <*>}''.
\subsection{Log Parsing (Log Template Identification)}\label{sec:log-parsing}
Although software execution logs contain valuable information about
the run-time behavior of the software system under analysis,
they cannot be directly processed by log-based analysis techniques that require
structured input logs (containing templates) instead of free-formed log messages.
Extracting log templates from log messages is straightforward
when the source code with the corresponding logging statements is available.
However, often the source code is unavailable, for example, due to the
usage of 3rd-party,
proprietary components. This leads to the problem of log parsing (log template identification):
\textit{How can we identify the log templates of log messages without accessing the source code?}
To address this problem, many automated log-parsing approaches,
which take as input log messages and identify their log templates using different
heuristics, have been proposed in the literature (e.g., AEL~\cite{jiang2008abstracting},
Drain~\cite{he2017drain}, IPLoM~\cite{makanju2009clustering},
LenMa~\cite{shima2016length}, LFA~\cite{nagappan2010abstracting},
LogCluster~\cite{7367331}, LogMine~\cite{hamooni2016logmine}, Logram~\cite{dai2020logram},
LogSig~\cite{tang2011logsig}, MoLFI~\cite{messaoudi2018search},
SHISO~\cite{mizutani2013incremental}, SLCT~\cite{vaarandi2003data},
and Spell~\cite{du2016spell}).
Three different accuracy metrics have been proposed to evaluate the accuracy of log parsing
approaches: \textit{Grouping Accuracy} (GA)~\cite{zhu2019tools}, \textit{Parsing
Accuracy} (PA)~\cite{dai2020logram}, and \textit{Template Accuracy}
(TA)~\cite{khan2022guidelines}.
\citet{zhu2019tools} observed that log parsing can be considered as a clustering
process where log messages with the same template are clustered into the same
group. Based on this idea, they proposed the GA metric to assess if log messages
are correctly grouped. Specifically, GA is defined as the ratio of log messages
\textit{correctly parsed} by the log parsing approach under evaluation over the
total number of log messages, where a log message is correctly parsed when its
log message group is the same as the ground truth (i.e., a group generated by
oracle templates).
\citet{dai2020logram} later proposed PA, to address the issue that GA only considers
message groups, not the equivalence between the templates identified by the log
parsing approach under evaluation and the oracle templates. Although having
correctly grouped messages would be enough in some cases (e.g., detecting
anomalies based on the sequence of template IDs without considering the content
of the templates~\cite{du2017deeplog}), correctly identified templates (i.e.,
templates identical to the corresponding oracle ones) matter when the fixed parts of
templates are used (e.g., detecting anomalies based on the semantic information
in the templates~\cite{zhang2019robust}). To this end, PA replaces the
definition of a correctly parsed log message in GA as follows: a log message is
\textit{correctly parsed} when its identified template is identical to the
oracle template.
\citet{khan2022guidelines} recently proposed the TA metric, since both GA and PA are
defined based on the number of correctly parsed log messages and, therefore, can
be misleading, especially when there are many repeated messages (e.g., heartbeat
messages). Specifically, they introduced Precision-TA (PTA) and Recall-TA (RTA),
where PTA is defined as the number of correctly identified templates over the
total number of identified templates and RTA is defined as the number of
correctly identified templates over the total number of oracle templates.
Moreover, FTA (short for ``F1-measure TA'') is the harmonic mean of PTA and RTA.
\subsection{Anomaly Detection}\label{sec:anomaly-detection}
(Log-based) anomaly detection is a technique that aims to identify
anomalous patterns, recorded in input logs, that do not conform to the
expected behaviors of the system under
analysis~\cite{10.1145/3460345}. It takes as input a sequence of log
templates and determines whether the given sequence represents a
normal behavior of the system or not.
With the recent advances in Deep Learning (DL), many anomaly detection
approaches, which leverage DL models to learn various aspects of log template sequences of normal and abnormal behaviors and classify them, have been proposed in the literature; for example,
DeepLog~\cite{du2017deeplog}, LogAnomaly~\cite{meng2019loganomaly}, and
LogRobust~\cite{zhang2019robust} are based on Long Short-Term Memory based (LSTM),
CNN~\cite{lu2018detecting} is based on Convolutional Neural Network,
and PLELog~\cite{yang2021semi} is based on Gated recurrent units (GRUs).
To assess the accuracy of anomaly detection approaches,
it is common practice to use standard metrics
from the information retrieval domain, such as \emph{Precision}, \emph{Recall}, and \emph{F1-Score}.
These metrics are defined as follows:
$\textit{Precision} = \frac{\textit{TP}}{\textit{TP} + \textit{FP}}$,
$\textit{Recall} = \frac{\textit{TP}}{\textit{TP} + \textit{FN}}$, and
$\textit{F1-score} = \frac{2\times \textit{Precision} \times \textit{Recall}}{\textit{Precision} + \textit{Recall}}$
where \textit{TP} (True Positive) is the number of abnormal logs correctly identified by the model,
\textit{FP} (False Positive) is the number of normal logs incorrectly identified as anomalies by the model, and
\textit{FN} (False Negative) is the number of abnormal logs incorrectly identified as normal.
\subsection{Ideal Log Parsing Results for Accurate Anomaly Detection}\label{sec:distinguishability}
Given the dependency of anomaly detection on log parsing,
\citet{shin2021theoretical} presented a theoretical analysis on
\emph{ideal} log parsing results for accurate anomaly detection.
The idea behind the analysis is that log parsing can be regarded as the abstraction
of log messages, where some tokens in the messages are converted to variable parts.
Then, if normal and abnormal logs are over-abstracted by log parsing so that
they are \textit{indistinguishable} from each other,
it is clear that anomaly detection, which takes as input the parsed logs
(i.e., abstracted logs, sequences of templates), cannot distinguish normal from abnormal logs.
Based on this idea, they formally defined the concept of \emph{distinguishability}
for log parsing results and showed that it is an essential condition for ideal log parsing results.
Specifically, let $M$ be a set of log messages and $L$ be a set of
logs where a log $l\in L$ is a sequence of log messages $\langle \mathit{m_1,
m_2}, \dots, m_n \rangle$.
Also, let $L_n \subseteq L$ be a set of normal logs and $L_a \subseteq L$
be a set of abnormal logs such that $L_n \cap L_a = \emptyset$ and $L_n \cup L_a = L$.
Given $M$ and a set of templates (i.e., log parsing results) $T$,
an abstraction function $\tau: M \to T$ that represents a generic log parsing approach can be defined.
Based on $\tau$, an abstraction of a log $l = \langle \mathit{m_1, m_2}, \dots, m_n \rangle$
can be defined as $\tau^*(l) = \langle \mathit{ \tau(m_1), \tau(m_2)}, \dots, \tau(m_n) \rangle$.
Similarly, an abstraction of a set of logs $L$ can be defined as $\tau^{**}(L) = \{\tau^*(l) \mid l \in L\}$.
Notice that $\tau^{**}(L)$ represents a log parsing result for a set of logs $L$.
The notion of distinguishability can be defined as follows:
$\tau$ \emph{distinguishes} $L_n$ and $L_a$ if and only if $\tau^{**}(L_n) \cap \tau^{**}(L_a) = \emptyset$.
In other words, a log parsing approach distinguishes between normal and abnormal logs
if and only if they are still distinguishable after log parsing.
When $\tau$ distinguishes $L_n$ and $L_a$, $\tau^{**}(L)$ for $L = L_n \cup L_a$ is
called \emph{d-maintaining}, meaning that the distinguishability between
$L_n$ and $L_a$ is \emph{maintained} in the log parsing result.
\section{Motivation}\label{sec:motivation}
As discussed in Section~\ref{sec:background}, log parsing converts unstructured
logs into structured ones, which can then be processed by log-based
analysis techniques like anomaly detection.
It is quite natural to speculate that log
parsing results can affect anomaly detection results.
Intuitively, the research literature has assumed that inaccurate log
parsing results leads to inaccurate anomaly detection results.
However, this hypothesis has not been fully investigated in the
literature, except for one empirical study~\cite{le2022log} and one
analytical investigation~\cite{shin2021theoretical}.
\citet{le2022log} recently presented an empirical work
investigating several aspects that can impact Deep Learning (DL)-based
anomaly detection approaches, such as
training data selection, data grouping, class distribution, data noise, and early detection ability.
One of their experiments considering data noise assessed the impact of noise deriving
from log parsing results. Specifically, they used four log parsing techniques (Drain~\cite{he2017drain},
Spell~\cite{du2016spell}, AEL~\cite{jiang2008abstracting}, and IPLoM~\cite{makanju2009clustering})
to generate log parsing results for two log datasets (BGL~\cite{oliner2007supercomputers} and
Spirit~\cite{oliner2007supercomputers}).
Then, for each log dataset, they used the different log parsing
results as input of five anomaly detection approaches (DeepLog~\cite{du2017deeplog}, LogAnomaly~\cite{meng2019loganomaly},
PLELog~\cite{yang2021semi}, LogRobust~\cite{zhang2019robust}, and
CNN~\cite{lu2018detecting}), and measured the accuracy of the latter.
Their experimental results showed that log parsing approaches highly influence
the accuracy of anomaly detection approaches;
for example, the F1-Score of DeepLog on Spirit logs~\cite{oliner2007supercomputers}
decreases from $0.755$ to $0.609$ when Drain is used instead of IPLoM for log parsing.
Although this is the first clear evidence showing the impact of log parsing
results on anomaly detection accuracy, the scope of the underlying
study is limited. For example, it does not quantitatively assess
the accuracy of log parsing; therefore, it is still unclear if inaccurate log parsing results
lead to inaccurate anomaly detection results.
To this end, we define our first research question as follows:
\textbf{\emph{RQ1 - To which extent does the accuracy of log parsing affect the accuracy of anomaly detection?}}
As summarized in Section~\ref{sec:distinguishability}, \citet{shin2021theoretical} recently proposed a theoretical
framework determining the ideal log parsing results for anomaly
detection by introducing the concept of ``distinguishability'' for
log parsing results. It is argued that, rather than accuracy as previously assumed, what really matters is the extent to which log parsing results are distinguishable.
However, to the best of our knowledge, there is no empirical work
assessing quantitatively distinguishability in log parsing results
and its impact on anomaly detection accuracy.
Therefore, we define our second research question as follows:
\textbf{\emph{RQ2 - How does the
accuracy of anomaly detection vary with distinguishability of log parsing
results?}}
Answering the above questions will have a significant impact on both research
and industry in the field of log-based anomaly detection. For example, if the
answer to the first question is that, regardless of the log parsing accuracy
metrics, there is no relationship between log parsing accuracy and anomaly
detection accuracy, then it means that there is no need to use the existing
accuracy metrics to evaluate log parsing results for anomaly detection. This
would completely change the way log parsing tools are evaluated. Similarly, if
the answer to the second question is that the distinguishability of log parsing
results indeed affects anomaly detection, as expected from the recent
theoretical analysis~\cite{shin2021theoretical}, then this must be the focus of log parsing evaluations. As a result, our answers will
provide essential insights on better assessing the quality of log parsing techniques
for more accurate anomaly detection.
\section{Experimental Design}\label{sec:expr-design}
As motivated in Section~\ref{sec:motivation}, we will answer the following research questions:
\begin{enumerate}[\bf RQ1:]
\item To what extent does the accuracy of log parsing affect the accuracy of anomaly detection?
\item What is the relationship between the distinguishability of log parsing results and anomaly
detection accuracy?
\end{enumerate}
\subsection{Datasets}\label{sec:datasets}
To answer the research questions, we used publicly available
datasets based on the LogHub benchmark~\cite{he2020loghub}, which contains
a large collection of log messages from various types of systems including operating systems (Linux, Windows, and Mac), distributed systems (BGL, Hadoop, HDFS, Thunderbird, and OpenStack),
standalone programs (Proxifier and Zookeeper), and mobile systems (Android).
Among them, we initially selected HDFS, Hadoop, and OpenStack datasets because
(1) they have labels for normal and abnormal logs to be used for assessing the accuracy of anomaly detection techniques and
(2) they have publicly available source code (of the exact program
version used to generate the logs) from which we could extract correct oracle templates (i.e., ground truth templates) for each log message.
Although the benchmark provides oracle templates for all log datasets, they are manually generated without accessing the source code and only for 2K log messages randomly sampled for each dataset. As discussed by \citet{khan2022guidelines}, those manually generated oracle templates are error-prone; therefore, we used the logging statements in the source code to extract correct oracle templates.
For HDFS and OpenStack datasets, during the preliminary evaluation, we found two issues.
First, none of the anomaly detection techniques detected any anomalies
in the OpenStack logs due to the very small number of abnormal logs (4
out of 2068). Therefore, we had to exclude OpenStack.
Second, the original HDFS logs were too large to be processed by the
slowest anomaly detection technique (i.e.,
LogAnomaly~\cite{meng2019loganomaly}) when setting a two-day timeout.
Due to the large number of experiments we needed to conduct (i.e., all
combinations of log parsing and anomaly detection techniques with
additional repeats for distinguishable and indistinguishable log
parsing results, see \S~\ref{sec:eval-rq1-method} and \S~\ref{sec:eval-rq2-method}), we decided to reduce the log dataset sizes, especially for HDFS.
Specifically, we randomly selected $n$ log messages ($n\in \{10K, 20K,
\dots, 500K\}$) and checked if the slowest anomaly detection technique
could process the messages and terminate successfully within a reasonable time limit (2 hours).
Through this process, we established that LogAnomaly (i.e., the
slowest log parsing technique) could process up to $n=300K$ messages
within 2 hours. Therefore, we randomly and iteratively removed logs
from the HDFS dataset to build a reduced one until the total number of remaining messages was less than 300K. Notice that we preserved the sequence of log messages in each log.
To summarize, we considered two datasets: HDFS (reduced) and Hadoop.
Table~\ref{table:datasets} provides the size information of these
datasets, in terms of the number of oracle templates (\textit{O}), the
number of all logs ($L_\mathit{all}$), the number of normal logs ($L_\mathit{n}$), the number of abnormal logs ($L_\mathit{a}$), the number of all messages ($M_\mathit{all}$), the number of messages in normal logs ($M_\mathit{n}$), and the number of messages in abnormal logs ($M_\mathit{a}$).
\begin{table}[ht]
\centering
\caption{Log datasets}
\label{table:datasets}
\small
\begin{tabular}{l@{\hspace{2pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}r@{\hspace{5pt}}}
\toprule
Dataset & O & $L_{all}$ & $L_{n}$ & $L_{a}$ & $M_{all}$ & $M_{n}$ & $M_{a}$ \\
\midrule
HDFS (reduced) & 26 & 15295 & 15026 & 269 & 299971 & 292776 & 7195 \\
Hadoop & 175 & 54 & 11 & 43 & 109968 & 14392 & 95576 \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Log Parsing Techniques}\label{sec:log-parsing-techniques}
We aimed to use as many log parsing techniques available in the literature as possible.
Since \citet{khan2022guidelines} recently provided a comprehensive evaluation
of 14 log parsing techniques (i.e.,
AEL~\cite{jiang2008abstracting}, Drain~\cite{he2017drain},
IPLoM~\cite{makanju2009clustering}, LenMa~\cite{shima2016length},
LFA~\cite{nagappan2010abstracting}, LKE~\cite{fu2009execution},
LogCluster~\cite{7367331}, LogMine~\cite{hamooni2016logmine}, Logram~\cite{dai2020logram}, LogSig~\cite{tang2011logsig},
MoLFI~\cite{messaoudi2018search}, SHISO~\cite{mizutani2013incremental},
SLCT~\cite{vaarandi2003data}, and Spell~\cite{du2016spell}),
we decided to reuse their replication package, including all the
aforementioned techniques.
However, we had to exclude LKE since our preliminary evaluation results
showed that it could not complete its run for all of our log datasets
within the 2-day timeout.
As a result, we considered 13 log parsing techniques in our
experiments.
For all the log parsing techniques, we used their default parameters.
\subsection{Anomaly Detection Techniques}\label{sec:anomaly-detection-techniques}
Similar to the case of log parsing techniques, we considered the work
of \citet{le2022log},
a recent empirical study that evaluated five DL-based anomaly detection techniques
(i.e., DeepLog~\cite{du2017deeplog}, LogAnomaly~\cite{meng2019loganomaly},
LogRobust~\cite{zhang2019robust}, PLELog~\cite{yang2021semi}, and
CNN~\cite{lu2018detecting}), and decided to use their replication
package, including all the aforementioned techniques.
These techniques are representative of the state of the art of DL-based anomaly detection techniques.
For all the anomaly detection techniques, we used their default parameters.
\subsection{Methodology for RQ1}\label{sec:eval-rq1-method}
To answer RQ1, for each dataset, we first executed the log parsing techniques to
generate log parsing results and computed their accuracy in terms of GA,
PA, and FTA (see
\S~\ref{sec:log-parsing}). We then executed the anomaly
detection techniques on each of the log parsing results and computed their
accuracy in terms of precision (PR), recall (RE), and F1 score.
By doing so, we obtained a tuple of accuracy values $\langle \mathit{GA,
PA, FTA, PR, RE, F1} \rangle$ for each combination of datasets, log parsing
results, and anomaly detection techniques.
For log parsing, we executed each of the log parsing techniques with a
2-day timeout. Since MoLFI is non-deterministic, we executed it three
times. In total, we obtained 16 log parsing results (three from the
three different executions of MoLFI and 13 from the
remaining log parsing techniques) for each dataset. For each log parsing result,
we computed $\langle \mathit{GA, PA, FTA} \rangle$ using the oracle templates (and the
messages matching them) for the corresponding datasets.
For anomaly detection, we divided the individual log parsing results into two
disjoint sets, i.e., a training set and a test set, using a split ratio of 80:20. We
trained the anomaly detection techniques on each of the training sets with a
2-day timeout, and used the corresponding test sets to compute
$\langle \mathit{PR, RE, F1} \rangle$. To account for the randomness of anomaly detection
techniques, we repeated the train-and-test process five times and used the average
F1 score.
As a result, we obtained 160 tuples $\langle \mathit{GA,
PA, FTA, PR, RE, F1} \rangle$ from the
combinations of two datasets, 16 log parsing results, and five anomaly detection
techniques.
\subsection{Methodology for RQ2}\label{sec:eval-rq2-method}
To answer RQ2, we need distinguishable and indistinguishable log parsing results to compare in terms of anomaly detection accuracy. Although the log parsing results generated for RQ1 are available, they are mostly distinguishable, leading to unbalanced data for RQ2. To systematically assess the impact of the distinguishability of log parsing results on anomaly detection accuracy using well-balanced data, we generate pairs of distinguishable and indistinguishable log parsing results.
Specifically, let $d(R)$ be the distinguishability --- expressed as a Boolean value,
either \emph{true} ($T$) or \emph{false} ($F$) --- of a log
parsing result $R$. For each log parsing result $R$
generated in the context of
RQ1 (i.e., 16 log parsing results for each of the two datasets), we first
created a pair of log parsing results $\langle R, R' \rangle$ by
artificially generating $R'$
from $R$ such that $d(R') = \neg d(R)$ using Algorithms~\ref{alg:indist} and \ref{alg:dist}, detailed further below.
By definition, if the log parsing result $R$ is distinguishable then
$R'$ will be indistinguishable and vice versa.
For the sake of simplicity, we denote the distinguishable result (be
it $R$ or $R'$) as $R_\mathit{dst}$ and the indistinguishable one
(respectively, either $R'$ or $R$) as $R_\mathit{ind}$.
We then executed, for all pairs $\langle R_\mathit{dst}, R_\mathit{ind} \rangle$,
all the considered anomaly detection techniques twice: the
first time using $R_\mathit{dst}$ as input and the second time using $R_\mathit{ind}$ as
input; for each run of each anomaly detection technique we computed its accuracy in terms of
precision, recall, and F1 score. By doing so, we obtained the anomaly detection
accuracy scores for pairs of distinguishable ($R_\mathit{dst}$) and
indistinguishable ($R_\mathit{ind}$) versions of log parsing results,
and then compared them.
For the generation of $R'$ from $R$, it is important to minimize the difference
between $R$ and $R'$ while achieving $d(R') = \neg d(R)$.
This is to ensure that if there is a difference in anomaly detection scores between $R$ and $R'$, it is due to distinguishability and not to other differences between $R$ and $R'$ (e.g., the number of templates or the size of log parsing results).
To do this, we considered the two cases, when $d(R) = T$ and when $d(R) = F$, separately.
When $d(R) = T$ (i.e., $R$ is $R_\mathit{dst}$), it means that
templates for different log messages in $R$ are different enough to distinguish
between normal and abnormal logs in $R$, as explained in
Section~\ref{sec:distinguishability}.
For example, let us consider two logs $l_1 = \langle \mathit{m_1, m_2} \rangle$ and $l_2 = \langle \mathit{m_3, m_4} \rangle$ where the templates of the four messages are identified as $\tau(m_1) = t_1$, $\tau(m_2) = t_2$, $\tau(m_3) = t_3$, and $\tau(m_4) = t_2$, respectively, using a log parsing technique $\tau$.
Figure~\ref{fig:dist-example} shows the logs, messages, and templates.
In this case, the log parsing result of $\tau$ for $\{l_1, l_2\}$ is
\textit{distinguishable}, as highlighted in blue in the figure, since
$\tau^*(l_1) = \langle \tau(m_1), \tau(m_2) \rangle = \langle t_1,
t_2 \rangle$ and $\tau^*(l_2) = \langle \mathit{\tau(m_3), \tau(m_4)}
\rangle = \langle t_3, t_2 \rangle$ are different
(due to $\tau(m_1) \neq \tau(m_3)$, i.e., $t_1 \neq t_3$). However, if the templates of $m_1$ and
$m_3$ were the same, then the log parsing result would be
\textit{indistinguishable}. In other words, as highlighted in red in the figure, we can make the distinguishable log
parsing result of $\tau$ indistinguishable by merging the templates of $m_1$ and $m_3$
(i.e., by introducing a dummy template $m_{13}$ that corresponds to both $m_1$ and $m_3$).
Using this idea, to generate $R' = R_\mathit{ind}$ from $R = R_\mathit{dst}$, we
generated the templates of $R_\mathit{ind}$ by iteratively merging the templates of $R_\mathit{dst}$
until $d(R_\mathit{ind}) = F$. Furthermore, to minimize the difference between $R_\mathit{dst}$ and $R_\mathit{ind}$
in terms of the number of templates (i.e., to minimize the number of templates
being merged), we
start with merging the templates with the highest number of
matching messages in the log. This is based on the intuition that the more
messages affected by merging templates, the more likely normal and abnormal
logs are to become indistinguishable.
Algorithm~\ref{alg:indist} summarizes the above-mentioned idea into
the pseudocode for generating $R_\mathit{ind}$ from
$R_\mathit{dst}$. After initializing $R_\mathit{ind}$
(line~\ref{alg:indist:init}) as a copy of $R_\mathit{dst}$, the algorithm extracts the set of templates $T$ of $R_\mathit{dst}$ (line~\ref{alg:indist:getT}) and sorts the templates in $T$ in ascending order by the number of matching messages (line~\ref{alg:indist:sort}). The algorithm then iteratively merges the last $n$ templates (starting from $n=2$ as initialized at line~\ref{alg:indist:int}) in the sorted templates list $T_s$ (i.e., merging the top-$n$ templates that have the highest number of matching templates) until $R_\mathit{ind}$ becomes indistinguishable (lines~\ref{alg:indist:while-start}--\ref{alg:indist:while-end}). Notice that the while loop does not continue endlessly since $R_\mathit{ind}$ must be indistinguishable when $n$ becomes $|T_s|$ (i.e., all templates are merged into one) by definition. The algorithm ends by returning $R_\mathit{ind}$.
\begin{algorithm}
\SetNoFillComment
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{Distinguishable Log Parsing Result $R_\mathit{dst}$}
\Output{Indistinguishable Log Parsing Result $R_\mathit{ind}$}
\BlankLine
Log Parsing Result (Set of Parsed Logs) $R_\mathit{ind} \gets \textit{copy}(R_\mathit{dst})$\label{alg:indist:init}\\
Set of Templates $T \gets \textit{getTemplates}(R_\mathit{dst})$\label{alg:indist:getT}\\
Sorted List of Templates $T_s \gets \textit{sortByNumMessages}(T)$\label{alg:indist:sort}\\
Integer $n \gets 2$\label{alg:indist:int}\\
\While{$d(R_\mathit{ind}) = \textit{True}$}{\label{alg:indist:while-start}
Set of Templates $T_m \gets \textit{getLastTemplates}(T_s, n)$\label{alg:indist:getT-to-merge}\\
$R_\mathit{ind} \gets \textit{mergeTemplates}(T_m, R_\mathit{dst})$\label{alg:indist:merge}\\
$n \gets n + 1$\label{alg:indist:inc}\\
}\label{alg:indist:while-end}
\textbf{return} $R_\mathit{ind}$\label{alg:indist:return}\\
\caption{Generating an indistinguishable log parsing result from a
distinguishable one}
\label{alg:indist}
\end{algorithm}
\begin{figure*}
\tiny
\begin{tabularx}{\textwidth}{lccc}
\toprule
Log & Template & Parsed Log (original) & Parsed Log (after merging $t_1$ and $t_3$ into $t_{13}$) \\
\midrule
$l1$ = $\langle m_1, m_2 \rangle$ & $\tau(m_1) = t_1$ , $\tau(m_2) = t_2$ & \color{blue}$\tau^*(l_1) = \langle t_1, t_2 \rangle$ & \color{red}$\tau^*(l_1) = \langle t_{13}, t_2 \rangle$ \\
\midrule
$l2$ = $\langle m_3, m_4 \rangle$ & $\tau(m_3) = t_3$ , $\tau(m_4) = t_2$ & \color{blue}$\tau^*(l_2) = {\langle t_3, t_2 \rangle}$ & \color{red}$\tau^*(l_2) = \langle t_{13}, t_2 \rangle$ \\
\bottomrule
\end{tabularx}
\caption{An example of making a \textcolor{blue}{distinguishable} log parsing result \textcolor{red}{indistinguishable} by merging templates}
\label{fig:dist-example}
\end{figure*}
When $d(R) = F$ (i.e., $R$ is $R_\mathit{ind}$), although one
could do the dual of merging templates (i.e., splitting templates), it would be very complex,
requiring to answer the
following questions: Which templates to split? How many templates should be generated by splitting one template? Instead, we
adopted another
heuristic: we removed the normal (or abnormal) logs that are indistinguishable
from abnormal (or normal) logs. This is based on our observation that,
when $d(R) = F$, only a small number of normal and abnormal logs are
indistinguishable. To minimize the impact of removing logs, we removed normal
logs when the total number of normal logs is larger than that of abnormal logs
(as it is the case for the HDFS dataset); otherwise, we removed
abnormal logs (in the case of the Hadoop dataset).
Algorithm~\ref{alg:dist} shows how to generate $R_\mathit{dst}$ from $R_\mathit{ind}$ based on the above idea. It first extracts the set of indistinguishable logs $L_\mathit{ind}$ from $R_\mathit{ind}$ (line~\ref{alg:dist:indist}). It then removes either normal or abnormal logs in $L_\mathit{ind}$ from $R_\mathit{ind}$ to generate $R_\mathit{dst}$ depending on the total number of normal and abnormal logs (lines~\ref{alg:dist:checkSize}--\ref{alg:dist:rma}). Since $R_\mathit{dst}$ is the result of removing indistinguishable (normal or abnormal) logs from $R_\mathit{ind}$, $R_\mathit{dst}$ is distinguishable. The algorithm ends by returning $R_\mathit{dst}$.
\begin{algorithm}
\SetNoFillComment
\SetKwInOut{Input}{Input}
\SetKwInOut{Output}{Output}
\Input{Indistinguishable Log Parsing Result $R_\mathit{ind}$}
\Output{Distinguishable Log Parsing Result $R_\mathit{dst}$}
\BlankLine
Set of Indistinguishable Logs $L_\mathit{ind} \gets \textit{getIndistLogs}(R_\mathit{ind})$\label{alg:dist:indist}\\
\If{$\textit{numNormalLogs}(R_\mathit{ind}) \geq \textit{numAbnormalLogs}(R_\mathit{ind})$}{\label{alg:dist:checkSize}
Set of Parsed Logs $R_\mathit{dst} \gets R_\mathit{ind} \setminus \textit{getNormalLogs}(L_\mathit{ind})$\label{alg:dist:rmn}\\
}
\Else{
Set of Parsed Logs $R_\mathit{dst} \gets R_\mathit{ind} \setminus \textit{getAbnormalLogs}(L_\mathit{ind})$\label{alg:dist:rma}\\
}
\textbf{return} $R_\mathit{dst}$\label{alg:dist:return}\\
\caption{Generating a distinguishable log parsing result from an
indistinguishable one}
\label{alg:dist}
\end{algorithm}
Notice that some of the anomaly detection techniques (i.e., LogRobust~\cite{zhang2019robust}, PLELog~\cite{yang2021semi}, LogAnomaly~\cite{meng2019loganomaly}) use the semantic information of
templates, instead of simply using template IDs, by converting them into
semantic vectors~\cite{jurafsky2019vector}. Therefore, the notion of ``identical'' templates for
determining the distinguishability of log parsing results must be revised in
terms of the semantic vectors used by these anomaly detection techniques. To do
this, for each log parsing result $R$, we applied a clustering algorithm to the
semantic vectors of all templates and considered the
templates in the same cluster to be identical. Specifically, we used
DBSCAN~\cite{backlund2011density} for clustering since it does not require the
number of clusters as an input parameter.
For instance, in the above example of $\tau$ with $m_1$ and $m_3$,
if the semantic vectors of $\tau(m_1)$ and $\tau(m_3)$ belong to the
same cluster, then the templates of $m_1$ and $m_3$ are considered the same.
Then, we followed the same heuristics described above to generate
$R'$ from $R$ based on the clustered templates.
\section{Results}\label{sec:results}
\subsection{RQ1: Relationship between Log Parsing Accuracy and Anomaly Detection Accuracy}\label{sec:eval-rq1-results}
All log parsing and anomaly detection techniques completed their executions on
both datasets. For all tuples $\langle \mathit{GA, PA, FTA, PR, RE, F1} \rangle$ we collected,
Figure~\ref{fig:rq1-results} shows the relationship between
$\langle \mathit{GA, PA, FTA} \rangle$ (x-axis) and $F1$ (y-axis) for
each dataset in the form of a scatter plot. To additionally distinguish the
results for different anomaly detection techniques, we used different shapes and
colors: {\protect\tikz \protect\draw[thick, draw=red, fill=white]
plot[mark=*, mark options={scale=1.2}]
(0,0);} = DeepLog,
{\protect\tikz \protect\draw[thick, draw=teal, fill=white]
plot[mark=diamond, mark options={scale=1.2}]
(0,0);} = LogAnomaly,
{\protect\tikz \protect\draw[thick, draw=black, fill=white]
plot[mark=triangle*, mark options={scale=1.2}]
(0,0);} = LogRobust,
{\protect\tikz \protect\draw[thick, draw=blue, fill=white]
plot[mark=pentagon*, mark options={scale=1.2}]
(0,0);} = CNN, and
{\protect\tikz \protect\draw[thick, draw=purple, fill=white]
plot[mark=square, mark options={scale=1.2}]
(0,0);} = PLELog.
\begin{figure*}
\centering
\begin{tabular}{ccc}
\subfloat[HDFS-GA\label{fig:HDFS-GA}]{\includegraphics[width=0.33\textwidth]{figures/HDFS_GA.eps}} &
\subfloat[HDFS-PA\label{fig:HDFS-PA}]{\includegraphics[width=0.33\textwidth]{figures/HDFS_PA.eps}} \\
\subfloat[HDFS-FTA\label{fig:HDFS-FTA}]{\includegraphics[width=0.33\textwidth]{figures/HDFS_FTA.eps}} &
\subfloat[Hadoop-GA\label{fig:Hadoop-GA}]{\includegraphics[width=0.33\textwidth]{figures/Hadoop_GA.eps}} \\
\subfloat[Hadoop-PA\label{fig:Hadoop-PA}]{\includegraphics[width=0.33\textwidth]{figures/Hadoop_PA.eps}} &
\subfloat[Hadoop-TA\label{fig:Hadoop-FTA}]{\includegraphics[width=0.33\textwidth]{figures/Hadoop_FTA.eps}} \\
\end{tabular}
\caption{\small Relationship between TI accuracy and AD accuracy
{[}
{\protect\tikz \protect\draw[thick, draw=red, fill=white]
plot[mark=*, mark options={scale=1.2}]
(0,0);} = DeepLog,
{\protect\tikz \protect\draw[thick, draw=teal, fill=white]
plot[mark=diamond, mark options={scale=1.2}]
(0,0);} = LogAnomaly,
{\protect\tikz \protect\draw[thick, draw=black, fill=white]
plot[mark=triangle*, mark options={scale=1.2}]
(0,0);} = LogRobust,
{\protect\tikz \protect\draw[thick, draw=blue, fill=white]
plot[mark=pentagon*, mark options={scale=1.2}]
(0,0);} = CNN, and
{\protect\tikz \protect\draw[thick, draw=purple, fill=white]
plot[mark=square, mark options={scale=1.2}]
(0,0);} = PLELog
{]}
}
\label{fig:rq1-results}
\end{figure*}
Table~\ref{table:spearman_correlation_coefficient} additionally shows
the values of the Spearman's
rank correlation coefficient $\sigma \langle \mathit{X, Y} \rangle$ between $X= \langle \mathit{GA,
PA, FTA} \rangle$ and $Y= F1$
for each pair of anomaly detection technique and dataset. The value of $\sigma \langle \mathit{X, Y} \rangle$,
ranging between $-1$ and $+1$, indicates the degree of the monotonic (not
necessarily linear) relationship between $X$ and $Y$; when
$\sigma\langle \mathit{X, Y} \rangle \ge +0.7$ (or $\sigma\langle \mathit{X, Y} \rangle \le -0.7$), there is a
\textit{strong} positive (or negative) correlation between $X$ and $Y$ (i.e.,
$Y$ increases (or decreases) when $X$ increases)~\cite{ali2022spearman}. Note
that, on the Hadoop dataset, $\sigma\langle \mathit{X, Y} \rangle$
could not be computed for DeepLog, LogAnomaly,
LogRobust, and CNN since the F1 score does not vary with
$\langle \mathit{GA, PA, FTA} \rangle$, indicating no relationship between
$\langle \mathit{GA, PA, FTA} \rangle$ and F1.
\begin{table}[tb]
\small
\caption{Spearman correlation coefficients between log parsing accuracy (GA, PA, and FTA)
and anomaly detection accuracy (F1 score)}
\label{table:spearman_correlation_coefficient}
\pgfplotstabletypeset[
col sep=comma,
empty cells with={-}, string replace={---}{\textemdash},
every head row/.style={
before row={\toprule
\multicolumn{1}{c}{} &
\multicolumn{3}{c}{HDFS (reduced)} &
\multicolumn{3}{c}{Hadoop}
\\
\cmidrule(r){2-4}
\cmidrule(r){5-7}},
after row=\midrule,
},
every last row/.style={after row=\bottomrule},
every column/.style={fixed zerofill,precision=3},
columns/AD_metric/.style={column type=l,column name=AD technique,string type},
columns/HDFS_GA/.style={column type=r, column name=GA, precision=3},
columns/HDFS_PA/.style={column type=r, column name=PA, precision=3},
columns/HDFS_FTA/.style={column type=r, column name=FTA, precision=3},
columns/Hadoop_GA/.style={column type=r, column name=GA, precision=3},
columns/Hadoop_PA/.style={column type=r, column name=PA, precision=3, fixed},
columns/Hadoop_FTA/.style={column type=r, column name=FTA, precision=3},
]{data/correlation_results.csv}
\end{table}
Overall, both Figure~\ref{fig:rq1-results} and
Table~\ref{table:spearman_correlation_coefficient} clearly show that there is no
strong correlation between $\langle \mathit{GA, PA, FTA} \rangle$ and $F1$ in all the
cases where \\ $\langle \mathit{GA, PA, FTA, PR, RE, F1} \rangle$ tuples were successfully collected.
For example, in Figure~\ref{fig:HDFS-GA}, LogAnomaly
({\protect\tikz \protect\draw[thick, draw=teal, fill=white]
plot[mark=diamond, mark options={scale=1.2}] (0,0);}) achieved an F1 score
ranging between 0.2 and 0.5 regardless of the GA score. This means that increasing log
parsing accuracy does not necessarily increase (or decrease) anomaly detection accuracy.
This is counter-intuitive since anomaly detection uses log parsing results, and
having ``better'' log parsing results is expected to increase anomaly detection
accuracy. However, this happens because even inaccurate log parsing results can
lead to accurate anomaly detection results, for reasons explained below.
To better understand the reason for the above results, let us consider a normal log
$l_n = \langle \mathit{m_1^n, m_2^n}, \dots \rangle$ and an abnormal log $l_a = \langle
\mathit{m_1^a, m_2^a}, \dots \rangle$, where $m_i^x$ indicates the $i$-th log message in
$l_x$ for $x\in \{n, a\}$. Using oracle templates, we can group the log messages
having the same template and represent $l_n$ and $l_a$ as groups;
specifically, let $g_\mathit{orc}(l_x)$ be a sequence of
message group indices (i.e., the $i$-th element of $g_\mathit{orc}(l_x)$ is
the message group index of $m_i^x$). In this context, let us take two logs from
the Hadoop dataset as a concrete example where $g_\mathit{orc}(l_n) = \langle
\mathit{1, 2, 3, 4, 5}, \dots \rangle$ and $g_\mathit{orc}(l_a) = \langle \mathit{6, 6, 6, 6,
7}, \dots \rangle$. When templates generated by LogCluster are used to group
messages instead of oracle templates, the sequences of message group indices
changes to $g_\mathit{LC}(l_n) = \langle \mathit{1, 2, 3, 4, 4}, \dots \rangle$
and $g_\mathit{LC}(l_a) = \langle \mathit{8, 9, 10, 11, 12}, \dots \rangle$,
clearly different from $g_\mathit{orc}(l_n)$ and $g_\mathit{orc}(l_a)$,
respectively; in particular, $m_4^n$ and $m_5^n$ are incorrectly grouped
together in $g_\mathit{LC}(l_n)$ while
$m_1^a$, $m_2^a$, $m_3^a$, and $m_4^a$ are incorrectly separated
in $g_\mathit{LC}(l_a)$. The incorrect
groupings of LogCluster clearly reduce GA (as well as PA and TA since incorrect groupings
imply incorrect templates). However, even the incorrect $g_\mathit{LC}(l_n)$ and
$g_\mathit{LC}(l_a)$ are still different enough from each other, so that
an anomaly detection technique can distinguish between normal and abnormal patterns.
This example not only shows why anomaly detection accuracy does not necessarily depend on log parsing accuracy,
but also demonstrates the importance of \textit{distinguishability} in log parsing results for anomaly
detection; we will further investigate this aspect in RQ2.
Before we conclude RQ1, one might be curious to know why DeepLog, LogAnomaly, LogRobust,
and CNN result in the same anomaly detection accuracy value on the Hadoop
dataset (as shown in Figures~\ref{fig:Hadoop-GA}-\ref{fig:Hadoop-FTA} and
Table~\ref{table:spearman_correlation_coefficient}). This happens because (1) the
test set of Hadoop contains only 11 logs (1 normal and 10 abnormal logs, although the number of log messages is in the same order of magnitude as HDFS;
see Table~\ref{table:datasets} for more details) and (2)
the four anomaly detection techniques classified all the 11 logs in the test set as
abnormal. We speculate that PLELog shows different results from the other anomaly
detection techniques because PLELog uses a special deep learning model (i.e., an
attention-based GRU~\cite{cho-etal-2014}). Notice that, in all
cases, the results still corroborate that log parsing accuracy and anomaly detection accuracy do not
have any strong relationship.
\begin{tcolorbox}
The answer to RQ1 is that there is no relationship between log
parsing accuracy and anomaly detection accuracy; increasing log parsing
accuracy does not necessarily increase anomaly detection accuracy, regardless of
the metric (GA, PA, or TA) used for measuring log parsing accuracy.
\end{tcolorbox}
\subsection{RQ2: Log Parsing Distinguishability and Anomaly Detection Accuracy}\label{sec:eval-rq2-results}
\begin{table}
\begin{adjustbox}{max width=\textwidth}
\pgfplotstabletypeset[
col sep=comma,
fixed zerofill, fixed,
precision=3,
every head row/.style={
before row={
\toprule
\multicolumn{1}{c}{} &
\multicolumn{3}{c}{DeepLog (F1)} &
\multicolumn{3}{c}{LogAnomaly (F1)} &
\multicolumn{3}{c}{LogRobust (F1)} &
\multicolumn{3}{c}{CNN (F1)} &
\multicolumn{3}{c}{PLELog (F1)}
\\
\cmidrule(r){2-4}
\cmidrule(r){5-7}
\cmidrule(r){8-10}
\cmidrule(r){11-13}
\cmidrule(r){14-16}
},
after row=\midrule,
},
every last row/.style={
before row=\midrule,
after row=\bottomrule,
},
columns/TI/.style={column type=l,column name=Log Parser,string type},
columns/AD_DeepLog_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_DeepLog_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/DeepLog_delta/.style={column type=r, column name=$\Delta$},
columns/AD_LogAnomaly_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_LogAnomaly_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/LogAnomaly_delta/.style={column type=r, column name=$\Delta$},
columns/AD_LogRobust_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_LogRobust_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/LogRobust_delta/.style={column type=r, column name=$\Delta$},
columns/AD_CNN_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_CNN_F1_indist/.style={column type=r,column name=$R_\mathit{ind}$},
columns/CNN_delta/.style={column type=r, column name=$\Delta$},
columns/AD_PLELog_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_PLELog_F1_indist/.style={column type=r,column name=$R_\mathit{ind}$},
columns/PLELog_delta/.style={column type=r, column name=$\Delta$},
]{data/HDFS_RQ2.csv}
\end{adjustbox}
\caption{Impact of the distinguishability log parsing results on
anomaly detection accuracy for the HDFS (reduced) dataset}
\label{table:rq2_hdfs}
\end{table}
\begin{table}
\begin{adjustbox}{max width=\textwidth}
\pgfplotstabletypeset[
col sep=comma,
fixed zerofill, fixed,
precision=3,
every head row/.style={
before row={
\toprule
\multicolumn{1}{c}{} &
\multicolumn{3}{c}{DeepLog (F1)} &
\multicolumn{3}{c}{LogAnomaly (F1)} &
\multicolumn{3}{c}{LogRobust (F1)} &
\multicolumn{3}{c}{CNN (F1)} &
\multicolumn{3}{c}{PLELog (F1)}
\\
\cmidrule(r){2-4}
\cmidrule(r){5-7}
\cmidrule(r){8-10}
\cmidrule(r){11-13}
\cmidrule(r){14-16}
},
after row=\midrule,
},
every last row/.style={
before row=\midrule,
after row=\bottomrule,
},
columns/TI/.style={column type=l,column name=Log Parser, string type },
columns/AD_DeepLog_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_DeepLog_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/DeepLog_delta/.style={column type=r, column name=$\Delta$},
columns/AD_LogAnomaly_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_LogAnomaly_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/LogAnomaly_delta/.style={column type=r, column name=$\Delta$},
columns/AD_LogRobust_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_LogRobust_F1_indist/.style={column type=r, column name=$R_\mathit{ind}$},
columns/LogRobust_delta/.style={column type=r, column name=$\Delta$},
columns/AD_CNN_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_CNN_F1_indist/.style={column type=r,column name=$R_\mathit{ind}$},
columns/CNN_delta/.style={column type=r, column name=$\Delta$},
columns/AD_PLELog_F1_dist/.style={column type=r,column name=$R_\mathit{dst}$},
columns/AD_PLELog_F1_indist/.style={column type=r,column name=$R_\mathit{ind}$},
columns/PLELog_delta/.style={column type=r, column name=$\Delta$},
]{data/Hadoop_RQ2.csv}
\end{adjustbox}
\caption{Impact of the distinguishability log parsing results on
anomaly detection accuracy for the Hadoop dataset}
\label{table:rq2_hadoop}
\end{table}
Table~\ref{table:rq2_hdfs} shows the anomaly detection accuracy values (F1
scores) when different log parsing techniques (rows) and anomaly detection
techniques (columns) are used together on the HDFS (reduced) dataset; under each of the
anomaly detection technique columns, sub-columns $R_\mathit{dst}$ and
$R_\mathit{ind}$ indicate the F1 scores for distinguishable and
indistinguishable log parsing results, respectively, and $\Delta$ indicates the
difference between $R_\mathit{dst}$ and $R_\mathit{ind}$. For example, if we
choose AEL for log parsing and DeepLog for anomaly detection, the F1 score
decreases from $0.747$ to $0.561$ when
$R_\mathit{ind}$ is used instead of $R_\mathit{dst}$.
The same structure applies to Table~\ref{table:rq2_hadoop}, which shows the
results on the Hadoop dataset.
In Table~\ref{table:rq2_hadoop}, except for PLELog , the values for all anomaly detection techniques
are identical due to the reasons explained in the last
paragraph of Section~\ref{sec:eval-rq1-results}.
In all cases, $\Delta$ is positive, ranging from $0.056$
(LFA-LogRobust on the HDFS (reduced) dataset) to
$0.9$ (SHISO-PLELog on the Hadoop dataset). This means that the anomaly detection accuracy decreases up to 90
percentage points (pp) when $R_\mathit{ind}$ is used instead of
$R_\mathit{dst}$. To see if the differences between $R_\mathit{dst}$ and
$R_\mathit{ind}$ are significant, we applied the non-parametric Wilcoxon signed rank
test~\cite{wilcoxon1992individual} for
paired samples to the F1 scores of $R_\mathit{dst}$ and $R_\mathit{ind}$, for
each of the five anomaly detection techniques and the two datasets. The results show
that, for all the anomaly detection techniques and datasets, the differences
between $R_\mathit{dst}$ and $R_\mathit{ind}$ are significant
($p\text{-value} < 0.005$) in terms of anomaly detection accuracy.
Considering the definition of distinguishability for log parsing results, it
is intuitive that indistinguishable log parsing results lead to lower anomaly
detection accuracy. It may even be surprising that the accuracy
decreases, in the best case, by only $0.056$. This happens because an indistinguishable log parsing
result may only have a few logs that are indistinguishable in terms of normal and
abnormal behavior. Recall that we did not explicitly control the number of
indistinguishable logs since we aimed to minimize the difference between
distinguishable and indistinguishable versions of each log parsing result as
described in Section~\ref{sec:eval-rq2-method}. Nevertheless, the results shown
in Tables~\ref{table:rq2_hdfs} and \ref{table:rq2_hadoop} are sufficient to confirm
the strong impact of distinguishability in log parsing results on anomaly detection
accuracy.
\begin{tcolorbox}
The answer to RQ2 is that the impact of the distinguishability
of log parsing results on anomaly detection accuracy is significant
for all anomaly detection techniques.
\end{tcolorbox}
\subsection{Threats to Validity}\label{sec:threats}
The oracle templates used in log parsing accuracy are critical to the accuracy
values. For example, as noted by \citet{khan2022guidelines}, manually extracting
oracle templates by investigating log messages without accessing the
corresponding source code could result in biased, incorrect oracle templates.
This could be a significant threat to the validity of our results. To mitigate
this, we perused the source code (of the exact version that
generated the logs) for each software system and used the templates directly
extracted from the source code. Although this made us exclude a few log datasets
whose source code was unavailable, it was inevitable to ensure the validity of
our results.
Individual log parsing and anomaly detection techniques have distinct
hyper-parameters, which might significantly affect the log parsing and anomaly
detection results. To mitigate this, we used the same hyper-parameter values
proposed by the authors, when available; otherwise, we ran several sanity checks
and used the values that resulted in the same results reported in the corresponding papers.
Using a specific set of log datasets is a potential threat to external
validity. Though the datasets we have considered have been widely used
in existing literature~\cite{le2022log, chen2021experience} on
log-based anomaly detection, they may not capture all diverse
characteristics of log data. Further experiments with different
datasets are required to improve the generalizability of our results.
\section{Findings and Implications}\label{sec:implications}
One of the most surprising results from our evaluation is that, using
all the existing log parsing accuracy metrics in the literature, none of them showed a
strong correlation with anomaly detection accuracy. In other words, more
accurate log parsing results are not necessarily better for anomaly detection
accuracy. This implies that log parsing accuracy
is not a good indicator of the quality of log
parsing results for anomaly detection purposes. As explained with an example in
Section~\ref{sec:eval-rq1-results}, this happens because inaccurate log parsing
results can still be useful for anomaly detection as long as normal and abnormal
logs are distinguishable. At the extreme, a log parsing result $R_{50}$ with
50\% accuracy could be better for anomaly detection than a log parsing result
$R_{100}$ with 100\% accuracy if $R_{50}$ distinguishes normal and abnormal logs
while $R_{100}$ does not.
This surprising finding leads to an important practical implication: when used
for anomaly detection purposes, we can no longer choose a log parsing technique
based on accuracy. Instead, as shown in Section~\ref{sec:eval-rq2-results},
the distinguishability of log parsing results should be the main selection criterion. For
example, normal and abnormal logs are often used for training anomaly detection
models; then, candidate log parsing results should be compared in terms of their
capability to distinguish normal and abnormal logs. If there are multiple
techniques that can distinguish between normal and abnormal logs, then the one
with the lowest number of identified templates would be preferred since reducing
the number of templates would increase the performance of anomaly detection by
reducing dimensionality (i.e., the number of features considered in machine
learning models)~\cite{shin2021theoretical}.
We want to note that our objective here is not to identify the ``best'' log parsing and anomaly
detection techniques. Nevertheless, through our experiments, we found that there
is no single best technique that significantly outperforms the others in all
cases. To develop better log parsing techniques targeting anomaly detection, it would
be of great help to consider distinguishability, which was not the case
in the development of existing log parsing techniques.
\section{Related Work}\label{sec:related-work}
\begin{table*}
\footnotesize
\caption{Comparison with related empirical studies}
\label{table:related_work}
\begin{tabularx}{\linewidth}{XXXX}
\toprule
Category & \citet{le2022log} & \citet{fu2023empirical} & Our work \\
\midrule
\rowcolor{gray!25}
Objective & Investigate different factors that might affect anomaly detection accuracy &
Investigate the impact of log parsing techniques on anomaly detection accuracy &
Evaluate the impact of log parsing accuracy and the distinguishability of log parsing results on anomaly detection accuracy \\
Log parsing accuracy metrics & N/A &PA & PA, GA, and TA \\
\rowcolor{gray!25}
Oracle templates & N/A & Manually generated for 2K sample log messages & Extracted from the corresponding source code \\
Logs used for measuring log parsing accuracy & N/A & Only a small fraction of logs actually used for anomaly detection & All logs used for anomaly detection \\
\rowcolor{gray!25}
Log parsing techniques & Drain, Spell, IPLoM and AEL & Drain, Spell, IPLoM, LFA, Logram, and LenMa & Drain, Spell, IPLoM, AEL, LFA, Logram, LenMa, LogSig, LogCluster, LogMine, SHISO, MoLFI, and SLCT\\
Anomaly detection techniques & DeepLog, LogRobust, LogAnomaly, PLELog, and CNN & DeepLog, LogRobust, Principal Component Analysis (PCA), LogClustering, Logistic Regression (LR), and Decision Tree (DT) & DeepLog, LogRobust, LogAnomaly, PLELog, and CNN \\
\rowcolor{gray!25}
Distinguishability~\cite{shin2021theoretical} & Not considered & Not considered & Considered \\
\bottomrule
\end{tabularx}
\end{table*}
Although individual techniques for log parsing and anomaly detection have been
studied for a long time, systematic studies covering several techniques have
only recently begun to emerge.
For example, the most comprehensive evaluation studies on many log parsing
techniques~\cite{zhu2019tools,dai2020logram,khan2022guidelines} were conducted
over the last four years.
Similarly, the relationship between log parsing and anomaly detection has
received little attention until very recently. Below, we summarize the recent
studies related to this topic.
\citet{shin2021theoretical} presented the first theoretical study considering
the relationship between log parsing and anomaly detection. As described in
Section~\ref{sec:distinguishability}, they established the concept of ideal log
parsing results for anomaly detection. We adopted their theoretical foundation,
especially the notion of \textit{distinguishability} in log parsing results, and
empirically showed that distinguishability is indeed essential for anomaly detection. To the best of our knowledge, our work is the
first empirical study showing the importance of log parsing distinguishability for anomaly detection.
As explained in Section~\ref{sec:motivation}, \citet{le2022log} presented an empirical study on factors that could affect
anomaly detection accuracy.
Although a part of their study investigated the impact of log parsing on anomaly
detection accuracy, they investigated four log parsing techniques but did not
assess log parsing accuracy. As a result, they only showed that using
different log parsing techniques leads to different anomaly detection
accuracy scores. In our
study, on the other hand, we explicitly measured log parsing accuracy, collected
160 pairs of log parsing accuracy and anomaly detection accuracy values using
different combinations of log parsing and anomaly detection techniques, and showed that there is
no strong correlation between log parsing accuracy and anomaly detection
accuracy based on Spearman's correlation.
During the writing of this paper, \citet{fu2023empirical} also presented an
empirical study on the impact of log parsing on anomaly detection performance.
Although their motivation and research questions are close to ours, there are
several key differences.
First, for measuring log parsing accuracy, they used the manually generated,
error-prone oracle templates~\cite{khan2022guidelines} provided with the 2K log
messages randomly sampled by \citet{zhu2019tools}. In other words, only a very
small fraction of the logs used for anomaly detection was used to measure log
parsing accuracy in their study. In our study, however, the same logs used for
anomaly detection are used to measure log parsing accuracy, and the oracle
templates are directly extracted from the corresponding source code.
Second, they considered only one log parsing accuracy metric (GA), whereas we
considered all three log parsing metrics (GA, PA, and TA) since different metrics assess complementary aspects of log
parsing~\cite{khan2022guidelines}.
Third, log parsing distinguishability, which is an essential factor that substantially affects
anomaly detection accuracy (as shown in our RQ2), is only considered
in our study.
Finally, they only considered two deep-learning-based anomaly
detection techniques (DeepLog and LogRobust), and focused also on more
traditional machine learning approaches (such as Principal Component
Analysis, clustering, logistic regression, and decision trees).
Such differences allow us to report new findings and provide concrete recommendations, as
summarized in Section~\ref{sec:implications}.
Table~\ref{table:related_work} summarizes the key differences between
the previous empirical studies (i.e., \citet{le2022log}, \citet{fu2023empirical}) and our work.
\section{Conclusion and Future Work}\label{sec:conclusion-future-work}
In this paper, we reported on a comprehensive empirical study on the
impact of log parsing on anomaly detection accuracy, using 13 log parsing
techniques and five DL-based anomaly detection techniques on two publicly
available log datasets.
Analyzing log parsing results for anomaly detection, surprisingly,
we did not find any significant relationship between log parsing accuracy and anomaly detection
accuracy, regardless of metric used for the former (including GA, PA, and FTA).
This implies that, as opposed to common research practice to date, we can no longer choose a log parsing technique based on its
accuracy when used for anomaly detection.
Instead, we experimentally confirmed existing theoretical results showing that the distinguishability of log parsing results plays
an essential role in achieving accurate anomaly detection. It is therefore highly
recommended that in the future we rely on such distinguishability to evaluate, compare, and select log parsing techniques.
As part of future work, we plan to extend our study with more publicly available
datasets and anomaly detection techniques to increase the generalizability of
our results. We aim to include state-of-the-art few-shot anomaly detection
techniques~\cite{huang2022registration,pang2021explainable}, which require only
a limited amount of training data and could be more effective in practice. We
also plan to provide a more granular analysis of the distinguishability for log
parsing results by defining a new metric that assesses the degree of
distinguishability.
\bibliographystyle{plainnat}
\begin{thebibliography}{37}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ali Abd Al-Hameed(2022)]{ali2022spearman}
Khawla Ali Abd Al-Hameed.
\newblock Spearman's correlation coefficient in statistical analysis.
\newblock \emph{International Journal of Nonlinear Analysis and Applications},
13\penalty0 (1):\penalty0 3249--3255, 2022.
\bibitem[Backlund et~al.(2011)Backlund, Hedblom, and
Neijman]{backlund2011density}
Henrik Backlund, Anders Hedblom, and Niklas Neijman.
\newblock A density-based spatial clustering of application with noise.
\newblock \emph{Data Mining TNM033}, pages 11--30, 2011.
\bibitem[Chen et~al.(2021)Chen, Liu, Gu, Su, and Lyu]{chen2021experience}
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael~R Lyu.
\newblock Experience report: deep learning-based system log analysis for
anomaly detection.
\newblock \emph{arXiv preprint arXiv:2107.05908}, 2021.
\bibitem[Cho et~al.(2014)Cho, van Merri{\"e}nboer, Bahdanau, and
Bengio]{cho-etal-2014}
Kyunghyun Cho, Bart van Merri{\"e}nboer, Dzmitry Bahdanau, and Yoshua Bengio.
\newblock On the properties of neural machine translation: Encoder{--}decoder
approaches.
\newblock In \emph{Proceedings of {SSST}-8, Eighth Workshop on Syntax,
Semantics and Structure in Statistical Translation}, pages 103--111, Doha,
Qatar, October 2014. Association for Computational Linguistics.
\newblock \doi{10.3115/v1/W14-4012}.
\newblock URL \url{https://aclanthology.org/W14-4012}.
\bibitem[{Dai} et~al.(2020){Dai}, {Li}, {Chen}, {Shang}, and
{Chen}]{dai2020logram}
H.~{Dai}, H.~{Li}, C.~S. {Chen}, W.~{Shang}, and T.~{Chen}.
\newblock Logram: Efficient log parsing using n-gram dictionaries.
\newblock \emph{IEEE Transactions on Software Engineering (TSE)}, 48:\penalty0
879 -- 892, 2020.
\newblock \doi{10.1109/TSE.2020.3007554}.
\bibitem[Du and Li(2016)]{du2016spell}
Min Du and Feifei Li.
\newblock Spell: Streaming parsing of system event logs.
\newblock In \emph{2016 IEEE 16th International Conference on Data Mining
(ICDM)}, pages 859--864, Los Alamitos, CA, USA, 2016. IEEE, IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Du et~al.(2017)Du, Li, Zheng, and Srikumar]{du2017deeplog}
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar.
\newblock Deeplog: Anomaly detection and diagnosis from system logs through
deep learning.
\newblock In \emph{2017 ACM Conference on Computer and Communications Security
(SIGSAC)}, CCS '17, page 1285–1298, New York, NY, USA, 2017. Association
for Computing Machinery.
\newblock ISBN 9781450349468.
\newblock \doi{10.1145/3133956.3134015}.
\bibitem[Elyasov(2012)]{elyasov2012log}
Alexander Elyasov.
\newblock Log-based testing.
\newblock In \emph{2012 34th International Conference on Software Engineering
(ICSE)}, pages 1591--1594, Los Alamitos, CA, USA, 2012. IEEE, IEEE.
\bibitem[Fu et~al.(2009)Fu, Lou, Wang, and Li]{fu2009execution}
Qiang Fu, Jian-Guang Lou, Yi~Wang, and Jiang Li.
\newblock Execution anomaly detection in distributed systems through
unstructured log analysis.
\newblock In \emph{2009 IEEE international conference on data mining (ICDM)},
pages 149--158, Los Alamitos, CA, USA, 2009. IEEE, IEEE.
\newblock \doi{10.1109/ICDM.2009.60}.
\bibitem[Fu et~al.(2023)Fu, Yan, Xu, Xia, Zhang, and Yang]{fu2023empirical}
Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang.
\newblock An empirical study of the impact of log parsers on the performance of
log-based anomaly detection.
\newblock \emph{Empirical Software Engineering}, 28\penalty0 (1):\penalty0
1--39, 2023.
\bibitem[Hamooni et~al.(2016)Hamooni, Debnath, Xu, Zhang, Jiang, and
Mueen]{hamooni2016logmine}
Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and
Abdullah Mueen.
\newblock Logmine: Fast pattern recognition for log analytics.
\newblock In \emph{25th ACM International on Conference on Information and
Knowledge Management (CIKM)}, pages 1573--1582, New York, NY, USA, 2016.
Association for Computing Machinery.
\newblock \doi{10.1145/2983323.2983358}.
\bibitem[He et~al.(2017)He, Zhu, Zheng, and Lyu]{he2017drain}
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael~R Lyu.
\newblock Drain: An online log parsing approach with fixed depth tree.
\newblock In \emph{2017 IEEE International Conference on Web Services (ICWS)},
pages 33--40, Los Alamitos, CA, USA, 2017. IEEE, IEEE.
\newblock \doi{10.1109/ICWS.2017.13}.
\bibitem[He et~al.(2020)He, Zhu, He, and Lyu]{he2020loghub}
Shilin He, Jieming Zhu, Pinjia He, and Michael~R. Lyu.
\newblock Loghub: A large collection of system log datasets towards automated
log analytics, 2020.
\newblock URL \url{https://arxiv.org/pdf/2008.06448.pdf}.
\bibitem[He et~al.(2021)He, He, Chen, Yang, Su, and Lyu]{10.1145/3460345}
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael~R.
Lyu.
\newblock A survey on automated log analysis for reliability engineering.
\newblock \emph{ACM Comput. Surv.}, 54\penalty0 (6), jul 2021.
\newblock ISSN 0360-0300.
\newblock \doi{10.1145/3460345}.
\newblock URL \url{https://doi.org/10.1145/3460345}.
\bibitem[Huang et~al.(2022)Huang, Guan, Jiang, Zhang, Spratling, and
Wang]{huang2022registration}
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya~Zhang, Michael Spratling, and
Yan-Feng Wang.
\newblock Registration based few-shot anomaly detection.
\newblock In \emph{European Conference on Computer Vision}, pages 303--319, New
York, NY, USA, 2022. Springer, Springer.
\bibitem[Jeong et~al.(2020)Jeong, Jha, Shin, and Lee]{jeong2020log}
Sooyong Jeong, Ajay~Kumar Jha, Youngsul Shin, and Woo~Jin Lee.
\newblock A log-based testing approach for detecting faults caused by incorrect
assumptions about the environment.
\newblock \emph{IEICE Transactions on Information and Systems}, 103\penalty0
(1):\penalty0 170--173, 2020.
\bibitem[Jiang et~al.(2008)Jiang, Hassan, Flora, and
Hamann]{jiang2008abstracting}
Zhen~Ming Jiang, Ahmed~E Hassan, Parminder Flora, and Gilbert Hamann.
\newblock Abstracting execution logs to execution events for enterprise
applications.
\newblock In \emph{2008 The Eighth International Conference on Quality Software
(QSIC)}, pages 181--186, Los Alamitos, CA, USA, 2008. IEEE, IEEE.
\newblock \doi{10.1109/QSIC.2008.50}.
\bibitem[Jurafsky and Martin(2019)]{jurafsky2019vector}
Daniel Jurafsky and James~H Martin.
\newblock Vector semantics and embeddings.
\newblock \emph{Speech and language processing}, pages 1--31, 2019.
\bibitem[Khan et~al.(2022)Khan, Shin, Bianculli, and
Briand]{khan2022guidelines}
Zanis~Ali Khan, Donghwan Shin, Domenico Bianculli, and Lionel Briand.
\newblock Guidelines for assessing the accuracy of log message template
identification techniques.
\newblock In \emph{Proceedings of the 44th International Conference on Software
Engineering (ICSE’22)}, page 1095–1106, New York, NY, United States,
2022. ACM, ACM.
\bibitem[Le and Zhang(2022)]{le2022log}
Van-Hoang Le and Hongyu Zhang.
\newblock Log-based anomaly detection with deep learning: how far are we?
\newblock In \emph{2022 IEEE/ACM 44th International Conference on Software
Engineering (ICSE)}, pages 1356--1367, New York, NY, USA, 2022. IEEE, ACM.
\bibitem[Lu et~al.(2018)Lu, Wei, Li, and Wang]{lu2018detecting}
Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang.
\newblock Detecting anomaly in big data system logs using convolutional neural
network.
\newblock In \emph{2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure
Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl
Conf on Big Data Intelligence and Computing and Cyber Science and Technology
Congress (DASC/PiCom/DataCom/CyberSciTech)}, pages 151--158, Los Alamitos,
CA, USA, 2018. IEEE, IEEE.
\bibitem[Makanju et~al.(2009)Makanju, Zincir-Heywood, and
Milios]{makanju2009clustering}
Adetokunbo~AO Makanju, A~Nur Zincir-Heywood, and Evangelos~E Milios.
\newblock Clustering event logs using iterative partitioning.
\newblock In \emph{15th ACM SIGKDD international conference on Knowledge
discovery and data mining (SIGKDD)}, pages 1255--1264, New York, NY, USA,
2009. Association for Computing Machinery.
\newblock \doi{10.1145/1557019.1557154}.
\bibitem[Meng et~al.(2019)Meng, Liu, Zhu, Zhang, Pei, Liu, Chen, Zhang, Tao,
Sun, et~al.]{meng2019loganomaly}
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao
Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et~al.
\newblock Loganomaly: Unsupervised detection of sequential and quantitative
anomalies in unstructured logs.
\newblock In \emph{IJCAI}, volume~19, pages 4739--4745, New York, NY, USA,
2019. ACM.
\bibitem[Messaoudi et~al.(2018)Messaoudi, Panichella, Bianculli, Briand, and
Sasnauskas]{messaoudi2018search}
Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and
Raimondas Sasnauskas.
\newblock A search-based approach for accurate identification of log message
formats.
\newblock In \emph{2018 IEEE/ACM 26th International Conference on Program
Comprehension (ICPC)}, pages 167--16710, New York, NY, USA, 2018. ACM,
Association for Computing Machinery.
\newblock \doi{10.1145/3196321.3196340}.
\bibitem[Mizutani(2013)]{mizutani2013incremental}
Masayoshi Mizutani.
\newblock Incremental mining of system log format.
\newblock In \emph{2013 IEEE International Conference on Services Computing
(SCC)}, pages 595--602, Los Alamitos, CA, USA, 2013. IEEE, IEEE.
\newblock \doi{10.1109/SCC.2013.73}.
\bibitem[Nagappan and Vouk(2010)]{nagappan2010abstracting}
Meiyappan Nagappan and Mladen~A Vouk.
\newblock Abstracting log lines to log event types for mining software system
logs.
\newblock In \emph{2010 7th IEEE Working Conference on Mining Software
Repositories (MSR)}, pages 114--117, Los Alamitos, CA, USA, 2010. IEEE, IEEE.
\newblock \doi{10.1109/MSR.2010.5463281}.
\bibitem[Oliner and Stearley(2007)]{oliner2007supercomputers}
Adam Oliner and Jon Stearley.
\newblock What supercomputers say: A study of five system logs.
\newblock In \emph{37th annual IEEE/IFIP international conference on dependable
systems and networks (DSN'07)}, pages 575--584, Edinburgh, UK, 2007. IEEE,
IEEE/IFIP International Conference on Dependable Systems and Networks
(DSN'07).
\bibitem[Pang et~al.(2021)Pang, Ding, Shen, and Hengel]{pang2021explainable}
Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van~den Hengel.
\newblock Explainable deep few-shot anomaly detection with deviation networks.
\newblock \emph{arXiv preprint arXiv:2108.00462}, abs/2108.00462, 2021.
\bibitem[Shima(2016)]{shima2016length}
Keiichi Shima.
\newblock Length matters: Clustering system log messages using length of words,
2016.
\newblock URL \url{https://arxiv.org/abs/1611.03213}.
\bibitem[Shin et~al.(2021)Shin, Khan, Bianculli, and
Briand]{shin2021theoretical}
Donghwan Shin, Zanis~Ali Khan, Domenico Bianculli, and Lionel Briand.
\newblock A theoretical framework for understanding the relationship between
log parsing and anomaly detection.
\newblock In \emph{International Conference on Runtime Verification}, pages
277--287, Cham, 2021. Springer, Springer.
\bibitem[Tang et~al.(2011)Tang, Li, and Perng]{tang2011logsig}
Liang Tang, Tao Li, and Chang-Shing Perng.
\newblock Logsig: Generating system events from raw textual logs.
\newblock In \emph{20th ACM international conference on Information and
knowledge management (CIKM)}, pages 785--794, New York, NY, USA, 2011. ACM.
\newblock \doi{10.1145/2063576.2063690}.
\bibitem[{Vaarandi} and {Pihelgas}(2015)]{7367331}
R.~{Vaarandi} and M.~{Pihelgas}.
\newblock Logcluster - a data clustering and pattern mining algorithm for event
logs.
\newblock In \emph{2015 11th International Conference on Network and Service
Management (CNSM)}, pages 1--7, Los Alamitos, CA, USA, 2015. IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Vaarandi(2003)]{vaarandi2003data}
Risto Vaarandi.
\newblock A data clustering algorithm for mining patterns from event logs.
\newblock In \emph{3rd IEEE Workshop on IP Operations \& Management (IPOM)},
pages 119--126, Los Alamitos, CA, USA, 2003. IEEE, IEEE.
\newblock \doi{10.1109/IPOM.2003.1251233}.
\bibitem[Wilcoxon(1992)]{wilcoxon1992individual}
Frank Wilcoxon.
\newblock Individual comparisons by ranking methods.
\newblock In \emph{Breakthroughs in statistics}, pages 196--202. Springer, New
York, NY, USA, 1992.
\bibitem[Yang et~al.(2021)Yang, Chen, Wang, Wang, Jiang, Dong, and
Zhang]{yang2021semi}
Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and
Wenbin Zhang.
\newblock Semi-supervised log-based anomaly detection via probabilistic label
estimation.
\newblock In \emph{2021 IEEE/ACM 43rd International Conference on Software
Engineering (ICSE)}, pages 1448--1460, Madrid, Spain, 2021. IEEE, IEEE.
\bibitem[Zhang et~al.(2019)Zhang, Xu, Lin, Qiao, Zhang, Dang, Xie, Yang, Cheng,
Li, et~al.]{zhang2019robust}
Xu~Zhang, Yong Xu, Qingwei Lin, Bo~Qiao, Hongyu Zhang, Yingnong Dang, Chunyu
Xie, Xinsheng Yang, Qian Cheng, Ze~Li, et~al.
\newblock Robust log-based anomaly detection on unstable log data.
\newblock In \emph{Proceedings of the 2019 27th ACM Joint Meeting on European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering}, pages 807--817, Tallin, Estonia, 2019. The ACM Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE).
\bibitem[Zhu et~al.(2019)Zhu, He, Liu, He, Xie, Zheng, and Lyu]{zhu2019tools}
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi~Xie, Zibin Zheng, and
Michael~R Lyu.
\newblock Tools and benchmarks for automated log parsing.
\newblock In \emph{2019 IEEE/ACM 41st International Conference on Software
Engineering: Software Engineering in Practice (ICSE-SEIP)}, pages 121--130,
Los Alamitos, CA, USA, 2019. IEEE, IEEE.
\newblock \doi{10.1109/ICSE-SEIP.2019.00021}.
\end{thebibliography}
\end{document}
\begin{thebibliography}{37}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ali Abd Al-Hameed(2022)]{ali2022spearman}
Khawla Ali Abd Al-Hameed.
\newblock Spearman's correlation coefficient in statistical analysis.
\newblock \emph{International Journal of Nonlinear Analysis and Applications},
13\penalty0 (1):\penalty0 3249--3255, 2022.
\bibitem[Backlund et~al.(2011)Backlund, Hedblom, and
Neijman]{backlund2011density}
Henrik Backlund, Anders Hedblom, and Niklas Neijman.
\newblock A density-based spatial clustering of application with noise.
\newblock \emph{Data Mining TNM033}, pages 11--30, 2011.
\bibitem[Chen et~al.(2021)Chen, Liu, Gu, Su, and Lyu]{chen2021experience}
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael~R Lyu.
\newblock Experience report: deep learning-based system log analysis for
anomaly detection.
\newblock \emph{arXiv preprint arXiv:2107.05908}, 2021.
\bibitem[Cho et~al.(2014)Cho, van Merri{\"e}nboer, Bahdanau, and
Bengio]{cho-etal-2014}
Kyunghyun Cho, Bart van Merri{\"e}nboer, Dzmitry Bahdanau, and Yoshua Bengio.
\newblock On the properties of neural machine translation: Encoder{--}decoder
approaches.
\newblock In \emph{Proceedings of {SSST}-8, Eighth Workshop on Syntax,
Semantics and Structure in Statistical Translation}, pages 103--111, Doha,
Qatar, October 2014. Association for Computational Linguistics.
\newblock \doi{10.3115/v1/W14-4012}.
\newblock URL \url{https://aclanthology.org/W14-4012}.
\bibitem[{Dai} et~al.(2020){Dai}, {Li}, {Chen}, {Shang}, and
{Chen}]{dai2020logram}
H.~{Dai}, H.~{Li}, C.~S. {Chen}, W.~{Shang}, and T.~{Chen}.
\newblock Logram: Efficient log parsing using n-gram dictionaries.
\newblock \emph{IEEE Transactions on Software Engineering (TSE)}, 48:\penalty0
879 -- 892, 2020.
\newblock \doi{10.1109/TSE.2020.3007554}.
\bibitem[Du and Li(2016)]{du2016spell}
Min Du and Feifei Li.
\newblock Spell: Streaming parsing of system event logs.
\newblock In \emph{2016 IEEE 16th International Conference on Data Mining
(ICDM)}, pages 859--864, Los Alamitos, CA, USA, 2016. IEEE, IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Du et~al.(2017)Du, Li, Zheng, and Srikumar]{du2017deeplog}
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar.
\newblock Deeplog: Anomaly detection and diagnosis from system logs through
deep learning.
\newblock In \emph{2017 ACM Conference on Computer and Communications Security
(SIGSAC)}, CCS '17, page 1285–1298, New York, NY, USA, 2017. Association
for Computing Machinery.
\newblock ISBN 9781450349468.
\newblock \doi{10.1145/3133956.3134015}.
\bibitem[Elyasov(2012)]{elyasov2012log}
Alexander Elyasov.
\newblock Log-based testing.
\newblock In \emph{2012 34th International Conference on Software Engineering
(ICSE)}, pages 1591--1594, Los Alamitos, CA, USA, 2012. IEEE, IEEE.
\bibitem[Fu et~al.(2009)Fu, Lou, Wang, and Li]{fu2009execution}
Qiang Fu, Jian-Guang Lou, Yi~Wang, and Jiang Li.
\newblock Execution anomaly detection in distributed systems through
unstructured log analysis.
\newblock In \emph{2009 IEEE international conference on data mining (ICDM)},
pages 149--158, Los Alamitos, CA, USA, 2009. IEEE, IEEE.
\newblock \doi{10.1109/ICDM.2009.60}.
\bibitem[Fu et~al.(2023)Fu, Yan, Xu, Xia, Zhang, and Yang]{fu2023empirical}
Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang.
\newblock An empirical study of the impact of log parsers on the performance of
log-based anomaly detection.
\newblock \emph{Empirical Software Engineering}, 28\penalty0 (1):\penalty0
1--39, 2023.
\bibitem[Hamooni et~al.(2016)Hamooni, Debnath, Xu, Zhang, Jiang, and
Mueen]{hamooni2016logmine}
Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and
Abdullah Mueen.
\newblock Logmine: Fast pattern recognition for log analytics.
\newblock In \emph{25th ACM International on Conference on Information and
Knowledge Management (CIKM)}, pages 1573--1582, New York, NY, USA, 2016.
Association for Computing Machinery.
\newblock \doi{10.1145/2983323.2983358}.
\bibitem[He et~al.(2017)He, Zhu, Zheng, and Lyu]{he2017drain}
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael~R Lyu.
\newblock Drain: An online log parsing approach with fixed depth tree.
\newblock In \emph{2017 IEEE International Conference on Web Services (ICWS)},
pages 33--40, Los Alamitos, CA, USA, 2017. IEEE, IEEE.
\newblock \doi{10.1109/ICWS.2017.13}.
\bibitem[He et~al.(2020)He, Zhu, He, and Lyu]{he2020loghub}
Shilin He, Jieming Zhu, Pinjia He, and Michael~R. Lyu.
\newblock Loghub: A large collection of system log datasets towards automated
log analytics, 2020.
\newblock URL \url{https://arxiv.org/pdf/2008.06448.pdf}.
\bibitem[He et~al.(2021)He, He, Chen, Yang, Su, and Lyu]{10.1145/3460345}
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael~R.
Lyu.
\newblock A survey on automated log analysis for reliability engineering.
\newblock \emph{ACM Comput. Surv.}, 54\penalty0 (6), jul 2021.
\newblock ISSN 0360-0300.
\newblock \doi{10.1145/3460345}.
\newblock URL \url{https://doi.org/10.1145/3460345}.
\bibitem[Huang et~al.(2022)Huang, Guan, Jiang, Zhang, Spratling, and
Wang]{huang2022registration}
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya~Zhang, Michael Spratling, and
Yan-Feng Wang.
\newblock Registration based few-shot anomaly detection.
\newblock In \emph{European Conference on Computer Vision}, pages 303--319, New
York, NY, USA, 2022. Springer, Springer.
\bibitem[Jeong et~al.(2020)Jeong, Jha, Shin, and Lee]{jeong2020log}
Sooyong Jeong, Ajay~Kumar Jha, Youngsul Shin, and Woo~Jin Lee.
\newblock A log-based testing approach for detecting faults caused by incorrect
assumptions about the environment.
\newblock \emph{IEICE Transactions on Information and Systems}, 103\penalty0
(1):\penalty0 170--173, 2020.
\bibitem[Jiang et~al.(2008)Jiang, Hassan, Flora, and
Hamann]{jiang2008abstracting}
Zhen~Ming Jiang, Ahmed~E Hassan, Parminder Flora, and Gilbert Hamann.
\newblock Abstracting execution logs to execution events for enterprise
applications.
\newblock In \emph{2008 The Eighth International Conference on Quality Software
(QSIC)}, pages 181--186, Los Alamitos, CA, USA, 2008. IEEE, IEEE.
\newblock \doi{10.1109/QSIC.2008.50}.
\bibitem[Jurafsky and Martin(2019)]{jurafsky2019vector}
Daniel Jurafsky and James~H Martin.
\newblock Vector semantics and embeddings.
\newblock \emph{Speech and language processing}, pages 1--31, 2019.
\bibitem[Khan et~al.(2022)Khan, Shin, Bianculli, and
Briand]{khan2022guidelines}
Zanis~Ali Khan, Donghwan Shin, Domenico Bianculli, and Lionel Briand.
\newblock Guidelines for assessing the accuracy of log message template
identification techniques.
\newblock In \emph{Proceedings of the 44th International Conference on Software
Engineering (ICSE’22)}, page 1095–1106, New York, NY, United States,
2022. ACM, ACM.
\bibitem[Le and Zhang(2022)]{le2022log}
Van-Hoang Le and Hongyu Zhang.
\newblock Log-based anomaly detection with deep learning: how far are we?
\newblock In \emph{2022 IEEE/ACM 44th International Conference on Software
Engineering (ICSE)}, pages 1356--1367, New York, NY, USA, 2022. IEEE, ACM.
\bibitem[Lu et~al.(2018)Lu, Wei, Li, and Wang]{lu2018detecting}
Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang.
\newblock Detecting anomaly in big data system logs using convolutional neural
network.
\newblock In \emph{2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure
Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl
Conf on Big Data Intelligence and Computing and Cyber Science and Technology
Congress (DASC/PiCom/DataCom/CyberSciTech)}, pages 151--158, Los Alamitos,
CA, USA, 2018. IEEE, IEEE.
\bibitem[Makanju et~al.(2009)Makanju, Zincir-Heywood, and
Milios]{makanju2009clustering}
Adetokunbo~AO Makanju, A~Nur Zincir-Heywood, and Evangelos~E Milios.
\newblock Clustering event logs using iterative partitioning.
\newblock In \emph{15th ACM SIGKDD international conference on Knowledge
discovery and data mining (SIGKDD)}, pages 1255--1264, New York, NY, USA,
2009. Association for Computing Machinery.
\newblock \doi{10.1145/1557019.1557154}.
\bibitem[Meng et~al.(2019)Meng, Liu, Zhu, Zhang, Pei, Liu, Chen, Zhang, Tao,
Sun, et~al.]{meng2019loganomaly}
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao
Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et~al.
\newblock Loganomaly: Unsupervised detection of sequential and quantitative
anomalies in unstructured logs.
\newblock In \emph{IJCAI}, volume~19, pages 4739--4745, New York, NY, USA,
2019. ACM.
\bibitem[Messaoudi et~al.(2018)Messaoudi, Panichella, Bianculli, Briand, and
Sasnauskas]{messaoudi2018search}
Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and
Raimondas Sasnauskas.
\newblock A search-based approach for accurate identification of log message
formats.
\newblock In \emph{2018 IEEE/ACM 26th International Conference on Program
Comprehension (ICPC)}, pages 167--16710, New York, NY, USA, 2018. ACM,
Association for Computing Machinery.
\newblock \doi{10.1145/3196321.3196340}.
\bibitem[Mizutani(2013)]{mizutani2013incremental}
Masayoshi Mizutani.
\newblock Incremental mining of system log format.
\newblock In \emph{2013 IEEE International Conference on Services Computing
(SCC)}, pages 595--602, Los Alamitos, CA, USA, 2013. IEEE, IEEE.
\newblock \doi{10.1109/SCC.2013.73}.
\bibitem[Nagappan and Vouk(2010)]{nagappan2010abstracting}
Meiyappan Nagappan and Mladen~A Vouk.
\newblock Abstracting log lines to log event types for mining software system
logs.
\newblock In \emph{2010 7th IEEE Working Conference on Mining Software
Repositories (MSR)}, pages 114--117, Los Alamitos, CA, USA, 2010. IEEE, IEEE.
\newblock \doi{10.1109/MSR.2010.5463281}.
\bibitem[Oliner and Stearley(2007)]{oliner2007supercomputers}
Adam Oliner and Jon Stearley.
\newblock What supercomputers say: A study of five system logs.
\newblock In \emph{37th annual IEEE/IFIP international conference on dependable
systems and networks (DSN'07)}, pages 575--584, Edinburgh, UK, 2007. IEEE,
IEEE/IFIP International Conference on Dependable Systems and Networks
(DSN'07).
\bibitem[Pang et~al.(2021)Pang, Ding, Shen, and Hengel]{pang2021explainable}
Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van~den Hengel.
\newblock Explainable deep few-shot anomaly detection with deviation networks.
\newblock \emph{arXiv preprint arXiv:2108.00462}, abs/2108.00462, 2021.
\bibitem[Shima(2016)]{shima2016length}
Keiichi Shima.
\newblock Length matters: Clustering system log messages using length of words,
2016.
\newblock URL \url{https://arxiv.org/abs/1611.03213}.
\bibitem[Shin et~al.(2021)Shin, Khan, Bianculli, and
Briand]{shin2021theoretical}
Donghwan Shin, Zanis~Ali Khan, Domenico Bianculli, and Lionel Briand.
\newblock A theoretical framework for understanding the relationship between
log parsing and anomaly detection.
\newblock In \emph{International Conference on Runtime Verification}, pages
277--287, Cham, 2021. Springer, Springer.
\bibitem[Tang et~al.(2011)Tang, Li, and Perng]{tang2011logsig}
Liang Tang, Tao Li, and Chang-Shing Perng.
\newblock Logsig: Generating system events from raw textual logs.
\newblock In \emph{20th ACM international conference on Information and
knowledge management (CIKM)}, pages 785--794, New York, NY, USA, 2011. ACM.
\newblock \doi{10.1145/2063576.2063690}.
\bibitem[{Vaarandi} and {Pihelgas}(2015)]{7367331}
R.~{Vaarandi} and M.~{Pihelgas}.
\newblock Logcluster - a data clustering and pattern mining algorithm for event
logs.
\newblock In \emph{2015 11th International Conference on Network and Service
Management (CNSM)}, pages 1--7, Los Alamitos, CA, USA, 2015. IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Vaarandi(2003)]{vaarandi2003data}
Risto Vaarandi.
\newblock A data clustering algorithm for mining patterns from event logs.
\newblock In \emph{3rd IEEE Workshop on IP Operations \& Management (IPOM)},
pages 119--126, Los Alamitos, CA, USA, 2003. IEEE, IEEE.
\newblock \doi{10.1109/IPOM.2003.1251233}.
\bibitem[Wilcoxon(1992)]{wilcoxon1992individual}
Frank Wilcoxon.
\newblock Individual comparisons by ranking methods.
\newblock In \emph{Breakthroughs in statistics}, pages 196--202. Springer, New
York, NY, USA, 1992.
\bibitem[Yang et~al.(2021)Yang, Chen, Wang, Wang, Jiang, Dong, and
Zhang]{yang2021semi}
Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and
Wenbin Zhang.
\newblock Semi-supervised log-based anomaly detection via probabilistic label
estimation.
\newblock In \emph{2021 IEEE/ACM 43rd International Conference on Software
Engineering (ICSE)}, pages 1448--1460, Madrid, Spain, 2021. IEEE, IEEE.
\bibitem[Zhang et~al.(2019)Zhang, Xu, Lin, Qiao, Zhang, Dang, Xie, Yang, Cheng,
Li, et~al.]{zhang2019robust}
Xu~Zhang, Yong Xu, Qingwei Lin, Bo~Qiao, Hongyu Zhang, Yingnong Dang, Chunyu
Xie, Xinsheng Yang, Qian Cheng, Ze~Li, et~al.
\newblock Robust log-based anomaly detection on unstable log data.
\newblock In \emph{Proceedings of the 2019 27th ACM Joint Meeting on European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering}, pages 807--817, Tallin, Estonia, 2019. The ACM Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE).
\bibitem[Zhu et~al.(2019)Zhu, He, Liu, He, Xie, Zheng, and Lyu]{zhu2019tools}
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi~Xie, Zibin Zheng, and
Michael~R Lyu.
\newblock Tools and benchmarks for automated log parsing.
\newblock In \emph{2019 IEEE/ACM 41st International Conference on Software
Engineering: Software Engineering in Practice (ICSE-SEIP)}, pages 121--130,
Los Alamitos, CA, USA, 2019. IEEE, IEEE.
\newblock \doi{10.1109/ICSE-SEIP.2019.00021}.
\end{thebibliography}
\begin{thebibliography}{37}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ali Abd Al-Hameed(2022)]{ali2022spearman}
Khawla Ali Abd Al-Hameed.
\newblock Spearman's correlation coefficient in statistical analysis.
\newblock \emph{International Journal of Nonlinear Analysis and Applications},
13\penalty0 (1):\penalty0 3249--3255, 2022.
\bibitem[Backlund et~al.(2011)Backlund, Hedblom, and
Neijman]{backlund2011density}
Henrik Backlund, Anders Hedblom, and Niklas Neijman.
\newblock A density-based spatial clustering of application with noise.
\newblock \emph{Data Mining TNM033}, pages 11--30, 2011.
\bibitem[Chen et~al.(2021)Chen, Liu, Gu, Su, and Lyu]{chen2021experience}
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael~R Lyu.
\newblock Experience report: deep learning-based system log analysis for
anomaly detection.
\newblock \emph{arXiv preprint arXiv:2107.05908}, 2021.
\bibitem[Cho et~al.(2014)Cho, van Merri{\"e}nboer, Bahdanau, and
Bengio]{cho-etal-2014}
Kyunghyun Cho, Bart van Merri{\"e}nboer, Dzmitry Bahdanau, and Yoshua Bengio.
\newblock On the properties of neural machine translation: Encoder{--}decoder
approaches.
\newblock In \emph{Proceedings of {SSST}-8, Eighth Workshop on Syntax,
Semantics and Structure in Statistical Translation}, pages 103--111, Doha,
Qatar, October 2014. Association for Computational Linguistics.
\newblock \doi{10.3115/v1/W14-4012}.
\newblock URL \url{https://aclanthology.org/W14-4012}.
\bibitem[{Dai} et~al.(2020){Dai}, {Li}, {Chen}, {Shang}, and
{Chen}]{dai2020logram}
H.~{Dai}, H.~{Li}, C.~S. {Chen}, W.~{Shang}, and T.~{Chen}.
\newblock Logram: Efficient log parsing using n-gram dictionaries.
\newblock \emph{IEEE Transactions on Software Engineering (TSE)}, 48:\penalty0
879 -- 892, 2020.
\newblock \doi{10.1109/TSE.2020.3007554}.
\bibitem[Du and Li(2016)]{du2016spell}
Min Du and Feifei Li.
\newblock Spell: Streaming parsing of system event logs.
\newblock In \emph{2016 IEEE 16th International Conference on Data Mining
(ICDM)}, pages 859--864, Los Alamitos, CA, USA, 2016. IEEE, IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Du et~al.(2017)Du, Li, Zheng, and Srikumar]{du2017deeplog}
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar.
\newblock Deeplog: Anomaly detection and diagnosis from system logs through
deep learning.
\newblock In \emph{2017 ACM Conference on Computer and Communications Security
(SIGSAC)}, CCS '17, page 1285–1298, New York, NY, USA, 2017. Association
for Computing Machinery.
\newblock ISBN 9781450349468.
\newblock \doi{10.1145/3133956.3134015}.
\bibitem[Elyasov(2012)]{elyasov2012log}
Alexander Elyasov.
\newblock Log-based testing.
\newblock In \emph{2012 34th International Conference on Software Engineering
(ICSE)}, pages 1591--1594, Los Alamitos, CA, USA, 2012. IEEE, IEEE.
\bibitem[Fu et~al.(2009)Fu, Lou, Wang, and Li]{fu2009execution}
Qiang Fu, Jian-Guang Lou, Yi~Wang, and Jiang Li.
\newblock Execution anomaly detection in distributed systems through
unstructured log analysis.
\newblock In \emph{2009 IEEE international conference on data mining (ICDM)},
pages 149--158, Los Alamitos, CA, USA, 2009. IEEE, IEEE.
\newblock \doi{10.1109/ICDM.2009.60}.
\bibitem[Fu et~al.(2023)Fu, Yan, Xu, Xia, Zhang, and Yang]{fu2023empirical}
Ying Fu, Meng Yan, Zhou Xu, Xin Xia, Xiaohong Zhang, and Dan Yang.
\newblock An empirical study of the impact of log parsers on the performance of
log-based anomaly detection.
\newblock \emph{Empirical Software Engineering}, 28\penalty0 (1):\penalty0
1--39, 2023.
\bibitem[Hamooni et~al.(2016)Hamooni, Debnath, Xu, Zhang, Jiang, and
Mueen]{hamooni2016logmine}
Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Guofei Jiang, and
Abdullah Mueen.
\newblock Logmine: Fast pattern recognition for log analytics.
\newblock In \emph{25th ACM International on Conference on Information and
Knowledge Management (CIKM)}, pages 1573--1582, New York, NY, USA, 2016.
Association for Computing Machinery.
\newblock \doi{10.1145/2983323.2983358}.
\bibitem[He et~al.(2017)He, Zhu, Zheng, and Lyu]{he2017drain}
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael~R Lyu.
\newblock Drain: An online log parsing approach with fixed depth tree.
\newblock In \emph{2017 IEEE International Conference on Web Services (ICWS)},
pages 33--40, Los Alamitos, CA, USA, 2017. IEEE, IEEE.
\newblock \doi{10.1109/ICWS.2017.13}.
\bibitem[He et~al.(2020)He, Zhu, He, and Lyu]{he2020loghub}
Shilin He, Jieming Zhu, Pinjia He, and Michael~R. Lyu.
\newblock Loghub: A large collection of system log datasets towards automated
log analytics, 2020.
\newblock URL \url{https://arxiv.org/pdf/2008.06448.pdf}.
\bibitem[He et~al.(2021)He, He, Chen, Yang, Su, and Lyu]{10.1145/3460345}
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael~R.
Lyu.
\newblock A survey on automated log analysis for reliability engineering.
\newblock \emph{ACM Comput. Surv.}, 54\penalty0 (6), jul 2021.
\newblock ISSN 0360-0300.
\newblock \doi{10.1145/3460345}.
\newblock URL \url{https://doi.org/10.1145/3460345}.
\bibitem[Huang et~al.(2022)Huang, Guan, Jiang, Zhang, Spratling, and
Wang]{huang2022registration}
Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya~Zhang, Michael Spratling, and
Yan-Feng Wang.
\newblock Registration based few-shot anomaly detection.
\newblock In \emph{European Conference on Computer Vision}, pages 303--319, New
York, NY, USA, 2022. Springer, Springer.
\bibitem[Jeong et~al.(2020)Jeong, Jha, Shin, and Lee]{jeong2020log}
Sooyong Jeong, Ajay~Kumar Jha, Youngsul Shin, and Woo~Jin Lee.
\newblock A log-based testing approach for detecting faults caused by incorrect
assumptions about the environment.
\newblock \emph{IEICE Transactions on Information and Systems}, 103\penalty0
(1):\penalty0 170--173, 2020.
\bibitem[Jiang et~al.(2008)Jiang, Hassan, Flora, and
Hamann]{jiang2008abstracting}
Zhen~Ming Jiang, Ahmed~E Hassan, Parminder Flora, and Gilbert Hamann.
\newblock Abstracting execution logs to execution events for enterprise
applications.
\newblock In \emph{2008 The Eighth International Conference on Quality Software
(QSIC)}, pages 181--186, Los Alamitos, CA, USA, 2008. IEEE, IEEE.
\newblock \doi{10.1109/QSIC.2008.50}.
\bibitem[Jurafsky and Martin(2019)]{jurafsky2019vector}
Daniel Jurafsky and James~H Martin.
\newblock Vector semantics and embeddings.
\newblock \emph{Speech and language processing}, pages 1--31, 2019.
\bibitem[Khan et~al.(2022)Khan, Shin, Bianculli, and
Briand]{khan2022guidelines}
Zanis~Ali Khan, Donghwan Shin, Domenico Bianculli, and Lionel Briand.
\newblock Guidelines for assessing the accuracy of log message template
identification techniques.
\newblock In \emph{Proceedings of the 44th International Conference on Software
Engineering (ICSE’22)}, page 1095–1106, New York, NY, United States,
2022. ACM, ACM.
\bibitem[Le and Zhang(2022)]{le2022log}
Van-Hoang Le and Hongyu Zhang.
\newblock Log-based anomaly detection with deep learning: how far are we?
\newblock In \emph{2022 IEEE/ACM 44th International Conference on Software
Engineering (ICSE)}, pages 1356--1367, New York, NY, USA, 2022. IEEE, ACM.
\bibitem[Lu et~al.(2018)Lu, Wei, Li, and Wang]{lu2018detecting}
Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang.
\newblock Detecting anomaly in big data system logs using convolutional neural
network.
\newblock In \emph{2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure
Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl
Conf on Big Data Intelligence and Computing and Cyber Science and Technology
Congress (DASC/PiCom/DataCom/CyberSciTech)}, pages 151--158, Los Alamitos,
CA, USA, 2018. IEEE, IEEE.
\bibitem[Makanju et~al.(2009)Makanju, Zincir-Heywood, and
Milios]{makanju2009clustering}
Adetokunbo~AO Makanju, A~Nur Zincir-Heywood, and Evangelos~E Milios.
\newblock Clustering event logs using iterative partitioning.
\newblock In \emph{15th ACM SIGKDD international conference on Knowledge
discovery and data mining (SIGKDD)}, pages 1255--1264, New York, NY, USA,
2009. Association for Computing Machinery.
\newblock \doi{10.1145/1557019.1557154}.
\bibitem[Meng et~al.(2019)Meng, Liu, Zhu, Zhang, Pei, Liu, Chen, Zhang, Tao,
Sun, et~al.]{meng2019loganomaly}
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao
Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et~al.
\newblock Loganomaly: Unsupervised detection of sequential and quantitative
anomalies in unstructured logs.
\newblock In \emph{IJCAI}, volume~19, pages 4739--4745, New York, NY, USA,
2019. ACM.
\bibitem[Messaoudi et~al.(2018)Messaoudi, Panichella, Bianculli, Briand, and
Sasnauskas]{messaoudi2018search}
Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and
Raimondas Sasnauskas.
\newblock A search-based approach for accurate identification of log message
formats.
\newblock In \emph{2018 IEEE/ACM 26th International Conference on Program
Comprehension (ICPC)}, pages 167--16710, New York, NY, USA, 2018. ACM,
Association for Computing Machinery.
\newblock \doi{10.1145/3196321.3196340}.
\bibitem[Mizutani(2013)]{mizutani2013incremental}
Masayoshi Mizutani.
\newblock Incremental mining of system log format.
\newblock In \emph{2013 IEEE International Conference on Services Computing
(SCC)}, pages 595--602, Los Alamitos, CA, USA, 2013. IEEE, IEEE.
\newblock \doi{10.1109/SCC.2013.73}.
\bibitem[Nagappan and Vouk(2010)]{nagappan2010abstracting}
Meiyappan Nagappan and Mladen~A Vouk.
\newblock Abstracting log lines to log event types for mining software system
logs.
\newblock In \emph{2010 7th IEEE Working Conference on Mining Software
Repositories (MSR)}, pages 114--117, Los Alamitos, CA, USA, 2010. IEEE, IEEE.
\newblock \doi{10.1109/MSR.2010.5463281}.
\bibitem[Oliner and Stearley(2007)]{oliner2007supercomputers}
Adam Oliner and Jon Stearley.
\newblock What supercomputers say: A study of five system logs.
\newblock In \emph{37th annual IEEE/IFIP international conference on dependable
systems and networks (DSN'07)}, pages 575--584, Edinburgh, UK, 2007. IEEE,
IEEE/IFIP International Conference on Dependable Systems and Networks
(DSN'07).
\bibitem[Pang et~al.(2021)Pang, Ding, Shen, and Hengel]{pang2021explainable}
Guansong Pang, Choubo Ding, Chunhua Shen, and Anton van~den Hengel.
\newblock Explainable deep few-shot anomaly detection with deviation networks.
\newblock \emph{arXiv preprint arXiv:2108.00462}, abs/2108.00462, 2021.
\bibitem[Shima(2016)]{shima2016length}
Keiichi Shima.
\newblock Length matters: Clustering system log messages using length of words,
2016.
\newblock URL \url{https://arxiv.org/abs/1611.03213}.
\bibitem[Shin et~al.(2021)Shin, Khan, Bianculli, and
Briand]{shin2021theoretical}
Donghwan Shin, Zanis~Ali Khan, Domenico Bianculli, and Lionel Briand.
\newblock A theoretical framework for understanding the relationship between
log parsing and anomaly detection.
\newblock In \emph{International Conference on Runtime Verification}, pages
277--287, Cham, 2021. Springer, Springer.
\bibitem[Tang et~al.(2011)Tang, Li, and Perng]{tang2011logsig}
Liang Tang, Tao Li, and Chang-Shing Perng.
\newblock Logsig: Generating system events from raw textual logs.
\newblock In \emph{20th ACM international conference on Information and
knowledge management (CIKM)}, pages 785--794, New York, NY, USA, 2011. ACM.
\newblock \doi{10.1145/2063576.2063690}.
\bibitem[{Vaarandi} and {Pihelgas}(2015)]{7367331}
R.~{Vaarandi} and M.~{Pihelgas}.
\newblock Logcluster - a data clustering and pattern mining algorithm for event
logs.
\newblock In \emph{2015 11th International Conference on Network and Service
Management (CNSM)}, pages 1--7, Los Alamitos, CA, USA, 2015. IEEE.
\newblock \doi{10.1109/CNSM.2015.7367331}.
\bibitem[Vaarandi(2003)]{vaarandi2003data}
Risto Vaarandi.
\newblock A data clustering algorithm for mining patterns from event logs.
\newblock In \emph{3rd IEEE Workshop on IP Operations \& Management (IPOM)},
pages 119--126, Los Alamitos, CA, USA, 2003. IEEE, IEEE.
\newblock \doi{10.1109/IPOM.2003.1251233}.
\bibitem[Wilcoxon(1992)]{wilcoxon1992individual}
Frank Wilcoxon.
\newblock Individual comparisons by ranking methods.
\newblock In \emph{Breakthroughs in statistics}, pages 196--202. Springer, New
York, NY, USA, 1992.
\bibitem[Yang et~al.(2021)Yang, Chen, Wang, Wang, Jiang, Dong, and
Zhang]{yang2021semi}
Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and
Wenbin Zhang.
\newblock Semi-supervised log-based anomaly detection via probabilistic label
estimation.
\newblock In \emph{2021 IEEE/ACM 43rd International Conference on Software
Engineering (ICSE)}, pages 1448--1460, Madrid, Spain, 2021. IEEE, IEEE.
\bibitem[Zhang et~al.(2019)Zhang, Xu, Lin, Qiao, Zhang, Dang, Xie, Yang, Cheng,
Li, et~al.]{zhang2019robust}
Xu~Zhang, Yong Xu, Qingwei Lin, Bo~Qiao, Hongyu Zhang, Yingnong Dang, Chunyu
Xie, Xinsheng Yang, Qian Cheng, Ze~Li, et~al.
\newblock Robust log-based anomaly detection on unstable log data.
\newblock In \emph{Proceedings of the 2019 27th ACM Joint Meeting on European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering}, pages 807--817, Tallin, Estonia, 2019. The ACM Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE).
\bibitem[Zhu et~al.(2019)Zhu, He, Liu, He, Xie, Zheng, and Lyu]{zhu2019tools}
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi~Xie, Zibin Zheng, and
Michael~R Lyu.
\newblock Tools and benchmarks for automated log parsing.
\newblock In \emph{2019 IEEE/ACM 41st International Conference on Software
Engineering: Software Engineering in Practice (ICSE-SEIP)}, pages 121--130,
Los Alamitos, CA, USA, 2019. IEEE, IEEE.
\newblock \doi{10.1109/ICSE-SEIP.2019.00021}.
\end{thebibliography}