DyEdgeGAT: Dynamic Edge via Graph Attention for Early Fault Detection in IIoT Systems
Extracted source text · arXiv 2307.03761
\begin{abstract}
% introduction & background
In the Industrial Internet of Things (IIoT), condition monitoring sensor signals from complex systems often exhibit nonlinear and stochastic spatial-temporal dynamics under varying conditions. These complex dynamics make fault detection particularly challenging.
% general situation & challenge
While previous methods effectively model these dynamics, they often neglect the evolution of relationships between sensor signals. Undetected shifts in these relationships can lead to significant system failures. \revtextiot{Furthermore, these methods frequently misidentify novel operating conditions as faults.
Addressing these limitations, we propose DyEdgeGAT~\textit{(Dynamic Edge via Graph Attention}), a novel approach for early-stage fault detection in IIoT systems.
% The main contribution of the method is to ...
DyEdgeGAT's primary innovation lies in a novel graph inference scheme for multivariate time series that tracks the evolution of relationships between time series, enabled by dynamic edge construction.
Another key innovation of DyEdgeGAT is its ability to incorporate operating condition contexts into node dynamics modeling, enhancing its accuracy and robustness.
% evaluation and performance of the proposed solution on the dataset
We rigorously evaluated DyEdgeGAT using both a synthetic dataset, simulating varying levels of fault severity, and a real-world industrial-scale multiphase flow facility benchmark with diverse fault types under varying operating conditions and detection complexities.
The results show that DyEdgeGAT significantly outperforms other baseline methods in fault detection, particularly in the early stages with low severity, and exhibits robust performance under novel operating conditions.
}
\end{abstract}
% TODO add the main contribution and novelty
% keywords can be removed
\begin{IEEEkeywords}
Graph Neural Networks, Graph Learning, Multivariate Time Series, Unsupervised Fault Detection
\end{IEEEkeywords}
% capable of detecting various faults, especially those characterized by relationship changes at a very early stage, and distinguishing actual faults from changes in operating conditions.
%\revtext{DyEdgeGAT} innovatively incorporates the context of operating conditions into node dynamics extraction, enhancing robustness against novel operating conditions.
%A critical aspect of this monitoring, often overlooked, is the detection of early faults which are indicated by subtle yet vital shifts in the functional relationships between sensor signals. These relationship shifts, if undetected, can evolve into major system failures.
%The challenge in early fault detection within IIoT systems lies in accurately identifying these relationship shifts despite the complex dynamics, interdependencies, and varying operating conditions within the system.
%as they typically only focus on either the dynamics or interdependencies but not on the evolving relationships among the signals. Addressing these relationship shifts is key to proactive maintenance and enhanced system reliability.
%The challenge of detecting \textit{dependency anomalies} in multivariate time series data, especially when the change in relations is subtle, remains largely unaddressed, representing a significant gap in the field of anomaly detection.
% proposed solution
%This outperforms the state-of-the-art discrete static graphs in detecting relationship shifts by better capturing the evolving inter-dependencies between time series.
% contribution
%These results emphasize the importance of modeling changing relationships in multivariate time series and the potential of dynamic graph learning in dependency anomaly detection.
%These anomalies are particularly challenging to detect, as individual sensors may appear to function normally, thus masking the abnormal collective behavior across multiple sensors.
%The difficulty of detecting \textit{collective anomalies} due to relationship shifts in multivariate time series represents a significant gap in the field of anomaly detection for sensor networks.
%While this data facilitates condition monitoring and anomaly detection, the increasing complexity and interdependencies within the sensor network pose significant challenges for anomaly detection.
%In the era of digital transformation, systems monitored by the Industrial Internet of Things (IIoTs) generate large amounts of Multivariate Time Series (MTS) data.
%Despite progress in this field, much of the focus has been on \textit{point anomalies} and \textit{contextual anomalies}, with lesser attention paid to \textit{collective anomalies}.
%A less addressed but common variant of \textit{collective anomalies} is when the abnormal collective behavior is caused by shifts in interrelationships within the system. This can be due to abnormal environmental conditions like overheating, improper operational settings resulting from cyber-physical attacks, or system-level faults.
%We evaluate the performance of \revtext{DyEdgeGAT} using both a synthetic dataset with controlled varying fault severity levels and an industrial-scale multiphase flow facility benchmark featuring various fault types with different detection difficulties. Our proposed approach demonstrated superior performance in fault and anomaly detection for sensor networks, showing particular strength in early-stage fault detection and is robust for novel operating conditions.
%Our results indicate that DyEdgeGAT is highly effective in both fault and anomaly detection, particularly excelling in the early detection of faults. Additionally, it demonstrates robust performance under new and varying operating conditions.
%
% paper title
% Titles are generally capitalized except for words such as a, an, and, as,
% at, but, by, for, in, nor, of, on, or, the, to and up, which are usually
% not capitalized unless they are the first or last word of the title.
% Linebreaks \\ can be used within to get better formatting as desired.
% Do not put math or special symbols in the title.
\title{\textcolor{black}{DyEdgeGAT: Dynamic Edge via Graph Attention for Early Fault Detection in IIoT Systems}}
%Capturing Temporally Evolving Interdependencies in IIoTs with Dynamic Graph Attention \\for Early Fault Detection
%\title{Dynamic Graph Attention for Anomaly Detection in Heterogeneous Sensor Networks
%\\ {\Large Detecting Relationship Shifts in Multivariate Time Series of Complex Systems}}
%\title{\revtext{DyGATAD}: Dynamic Graph Attention for \\Dependency Anomaly Detection in \\ Multivariate Sensor Networks}
%\title{Temporal Graph Representation Learning for \\Multivariate Time Series Anomaly Detection}
%
%
% author names and IEEE memberships
% note positions of commas and nonbreaking spaces ( ~ ) LaTeX will not break
% a structure at a ~ so this keeps an author's name from being broken across
% two lines.
% use \thanks{} to gain access to the first footnote area
% a separate \thanks must be used for each paragraph as LaTeX2e's \thanks
% was not built to handle multiple paragraphs
%
\author{Mengjie~Zhao and
Olga~Fink~\IEEEmembership{Memeber,~IEEE}
% <-this % stops a space
\thanks{The authors with the Laboratory of Intelligent Maintenance and Operation
Systems, EPFL, 1015 Lausanne, Switzerland (e-mail: mengjie.zhao@epfl.ch, olga.fink@epfl.ch).}% <-this % stops a space
}
% note the % following the last \IEEEmembership and also \thanks -
% these prevent an unwanted space from occurring between the last author name
% and the end of the author line. i.e., if you had this:
%
% \author{....lastname \thanks{...} \thanks{...} }
% ^------------^------------^----Do not want these spaces!
%
% a space would be appended to the last name and could cause every name on that
% line to be shifted left slightly. This is one of those "LaTeX things". For
% instance, "\textbf{A} \textbf{B}" will typeset as "A B" not "AB". To get
% "AB" then you have to do: "\textbf{A}\textbf{B}"
% \thanks is no different in this regard, so shield the last } of each \thanks
% that ends a line with a % and do not let a space in before the next \thanks.
% Spaces after \IEEEmembership other than the last one are OK (and needed) as
% you are supposed to have spaces between the names. For what it is worth,
% this is a minor point as most people would not even notice if the said evil
% space somehow managed to creep in.
% The paper headers
\markboth{Dynamic Edge via Graph Attention for Early Fault Detection in IIoT Systems}%
{Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for IEEE Journals}
% The only time the second header will appear is for the odd numbered pages
% after the title page when using the twoside option.
%
% *** Note that you probably will NOT want to include the author's ***
% *** name in the headers of peer review papers. ***
% You can use \ifCLASSOPTIONpeerreview for conditional compilation here if
% you desire.
% Update your Headers here
\fancyhead[LO]{Temporal Graph Representation Learning for Multivariate Time Series Anomaly Detection}
% \fancyhead[RE]{Firstauthor and Secondauthor} % Firstauthor et al. if more than 2 - must use \documentclass[twoside]{article}
%% Title
\title{Temporal Graph Representation Learning for Multivariate Time Series Anomaly Detection
%%%% Cite as
%%%% Update your official citation here when published
%\thanks{\textit{\underline{Citation}}:
%\textbf{Authors. Title. Pages.... DOI:000000/11111.}}
}
\author{
Mengjie Zhao, Olga Fink \\
Intelligent Maintenance and Operations Systems (IMOS) \\
Swiss Federal Institute of Technology Lausanne (EPFL) \\
Lausanne, Switerland\\
\texttt{\{mengjie.zhao, olga.fink\}@epfl.ch} \\
%% examples of more authors
% \And
%Author3 \\
%Affiliation \\
%Univ \\
%City\\
%\texttt{email@email} \\
%% \AND
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
%% \And
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
%% \And
%% Coauthor \\
%% Affiliation \\
%% Address \\
%% \texttt{email} \\
}
\section{Introduction}
% 1. Background on FD and AD in IIoT systems
% 2. Sota in modeling relationship shifts (zoom in)
% 3. Problem statement and novelty (Desire and obstacle)
% 5. Methodology overview (how we tackle it)
% 7. Paper outline
% General background on IIoT and condition monitoring
% MAYBE: an interconnected system of control variables and measurement variables with cause-and-effect relationships in this system.
\IEEEPARstart{T}{he} increasing deployment of sensors in industrial systems has enabled the collection of extensive Multivariate Time Series (MTS) data, facilitating the condition monitoring of complex systems to detect the onset of critical faults as early as possible~\cite{mohammadi2018deep}.
The complex dynamics of monitored systems, characterized by interconnected subsystems and components, often result in strong spatial-temporal dynamics of the heterogeneous MTS data \cite{younan2020challenges}~\cite {dong2023iotreview}. Due to the high interdependencies in the data, it becomes more challenging to detect incipient faults in such systems.
% 1. Background on FD in IIoT Systems:
\revtext{
Effective fault detection is crucial for preventing severe system failures and improving system reliability.
Particularly, detecting faults at their early stage can also contribute to extending the useful lifetime of components by preventing too early preventive replacements. Taking preemptive actions based on early fault detection can be instrumental in maintaining optimal system performance and avoiding costly disruptions.
% 1. Zoom in: early fault detection
While fault detection is relatively straightforward at higher severity levels when faults have fully manifested in the sensor measurements, identifying incipient faults, where they have not yet caused noticeable impacts on the system's performance, poses a greater challenge~\cite{wei2020robust}.
The primary difficulty lies in detecting subtle changes, while an additional challenge arises from the risk of overly sensitive algorithms, potentially leading to an excessive number of false alarms~\cite{chao2019hybrid}.
Different types of faults, each with different difficulties to detect, can affect the system. These range from sensor faults that only impact one signal and are relatively straightforward to detect, to complex faults that affect multiple components. The latter often leads to secondary fault impacts and are considerably harder to detect~\cite{lughofer2019predictive}. Since faults are rare, recently unsupervised fault detection methods have been increasingly applied, influenced by advancements in anomaly detection research. One type of faults that current state-of-the-art methods have insufficiently addressed is the detection of changes in relationship between signals and components. Such faults can remain undetected for extended periods because the individual observations may appear to exhibit healthy conditions in terms of their functional behavior~\cite{boyes2018industrial}.}
\input{tabs/overview}
\revtext{
% 2. Sota in modeling relationship shifts (zoom in)
To effectively detect a change in these functional relationships, it is essential to accurately model these relationships.
Traditionally, modeling relationships between MTS in the IIoT context has mainly focused on two aspects: explicit functional relationship modeling and dynamics modeling.
Feedford Neural Networks (FNN)~\cite{ahmed2014automotive} and Autoencoders (AE)~\cite{guo2020unsupervised} have been successful in explicit functional relationship modeling. Recurrent Neural Networks (RNN)~\cite{ergen2019unsupervised} and Convolutional Neural Networks (CNN)~\cite{garcia2022temporal} have been successful at capturing system dynamics. However, these methods often fall short of addressing spatial-temporal dynamics within complex systems.
% Graph-based methods
In response to these limitations, Graph Neural Networks (GNNs) have emerged as a promising alternative~\cite{wu2020comprehensive}\cite{jin2023survey}.
By establishing a graph structure from the MTS data, with each time series represented as a node and edges indicating interactions between different time series, GNNs can learn the spatial-temporal relationships between the sensors~\cite{zawislak2017graph}.
\revtextiot{Table~\ref{tab:intro_overview} provides a comparative analysis of different approaches in modeling these relationships.}
% Example graph-based methods for functional relationship modeling
Several Spatial-Temporal GNN (STGNN)-based methods have been developed specifically for modeling spatial-temporal dynamics within IIoT systems.
For instance, Multivariate Time-series Anomaly Detection via Graph Attention Networks (MTADGAT)~\cite{zhao2020mtadgat} utilizes graph attention networks (GATs) to simultaneously construct a feature-oriented graph and a time-oriented graph, capturing both spatial and temporal dynamics. Additionally, Graph Deviation Network (GDN)~\cite{deng2021gdn} employs node embeddings to capture the unique characteristics of each sensor and utilizes an attention mechanism incorporating these sensor embeddings to better predict the future behavior of sensors, addressing the heterogeneity of sensor data in IIoT.
% 3. Problem Statement and novelty (Desire and obstacle)
% desire
While these methods effectively model spatial-temporal dynamics within MTS, they generally overlook the evolution of functional relationships between system variables. This aspect is crucial for accurately detecting changes in relationships, which are often indicative of system faults. Current spatial-temporal GNNs are unable to capture these dynamic changes.
% obstacle
Tracking relationship shifts within IIoT systems to enable early fault detection poses two main challenges:
\begin{enumerate}
\item \textbf{Static graph relation}. Existing STGNNs typically assume the relationship in the MTS data within a defined observation time period does not change. While these approaches can partially account for the dynamics within the data by constructing a new graph for a new observation window, they cannot capture the temporal evolution of relationships (edge weights) within the data.
\item \textbf{Distinguishing faults from novel operating conditions}. Training data collected under healthy conditions may not contain all operational scenarios. Identifying whether relationship shifts in test time arise from faults or novel operating conditions remains challenging.
\end{enumerate}
}
% Methodology overview:
\revtext{
To address the limitations outlined above, we introduce \textbf{DyEdgeGAT} (\textit{Dynamic Edge via Graph Attention}), a novel framework designed to capture relationship shifts in MTS data for effective early fault detection.
% first challenge
DyEdgeGAT addresses the challenge of static graph inference by dynamically inferring edges between time series, and constructing an aggregated temporal graph for MTS data. This enables us to capture not only \textbf{node dynamics} (i.e., \textit{the intrinsic dynamics of a sensor}) but also \textbf{edge dynamics} (i.e.,\textit{ the evolving relationships between sensors}). The term ``dynamic'' in DyEdgeGAT specifically refers to the evolving sequence of edge weights over time, reflecting the evolution of pairwise relationships between nodes in the temporal graph. This is contrary to the definition used in previous methods that define it as constructing a new static graph per observation window.
%than dynamics relative to the input data.
% second challenge
\revtextiot{To address the challenge of distinguishing fault-induced relationship changes from novel operating conditions}, DyEdgeGAT innovatively differentiates between \textbf{system-dependent} variables (e.g., \textit{control variables} that are explicitly set and \textit{external factors} outside of the system) and \textbf{system-independent} variables (e.g., \textit{measurement variables} of system internal states) during model construction. Faults typically manifest in system-dependent variables, while system-independent variables, which contain information on operating conditions, may remain unaffected by faults but significantly influence system-dependent variables. Incorporating the context of operating conditions into node dynamics extraction enables the model to more accurately separate actual faults from novel operating conditions.
}
\revtextrewrite{
To summarize, our key contributions are as follows:
\begin{itemize}
\item \textbf{Dynamic edge construction for MTS graph inference}: The proposed DyEdgeGAT algorithm dynamically constructs edges between time series signals, enabling the model to capture the evolution of pairwise relationships.
\item \textbf{Operating condition-aware node dynamics modeling}: System-independent variables in the node dynamics extraction and reconstruction are modeled in different ways within DyEdgeGAT, enabling the distinction between faults and novel operating conditions.
\item \textbf{Temporal topology-informed anomaly scoring}: The proposed anomaly score incorporates temporal topology to account for the diverse strengths in sensor dynamics within IIoT systems with heterogeneous signals.
\item \textbf{Comprehensive performance evaluation}: We evaluated the proposed DyEdgeGAT algorithm on synthetic and real-world datasets across varying fault severities, multiple fault types, and novel operating conditions and compared it to a wide range of algorithms.
%\item \textbf{Ablation study for component contribution analysis}: We perform a detailed ablation study demonstrating the contribution of each component of the proposed methodology toward the overall performance of the model.
\end{itemize}
}
\revtextiot{
The remainder of this paper is organized as follows:
Sec.~\ref{sec:rw} reviews related work in unsupervised fault and anomaly detection for time series data, graph learning from multivariate time series, and GNN-based fault and anomaly detection. Sec.~\ref{sec:method} elaborates on DyEdgeGAT's core components. Sec.~\ref{sec:case_study} introduces the case studies along with data statistics. Sec.~\ref{sec:exp_design} outlines the experimental design, including baseline methods and the evaluation metrics. Sec.~\ref{sec:results} presents and discusses our results. Finally, Sec.~\ref{sec:conclusion} provides conclusions and suggests future research directions.}
%This proactive approach is key to identifying faults before they manifest into significant issues.
%Extracting meaningful information from the complex sensor signals for continuous system health monitoring to improve system reliability remains a significant challenge~\cite {dong2023iotreview}.
%textbf{Anomaly detection (AD)} has been widely employed in IIoT. It is a technique for identifying deviations from expected behaviors~\cite{chandola2009anomaly}. However, not all anomalies are indicative of system faults~\cite{neumayer2023fault}, as they can arise from a variety of factors, including changes in operating conditions, sensor malfunctions, or external disturbances.
%\textbf{Fault detection (FD)}, on the other hand, focuses on detecting changes in a system so that it can no longer operate.
%Given the rarity of faults, especially labeled early faults, there is a growing interest in exploring FD and EFD through normal samples in an unsupervised setting, which is similar in methodology to AD.
%To achieve EFD, it is essential to capture functional relationships within the system, as these relationships encode the system's behavior, and relationship shifts can be indicative of early faults~\cite{boyes2018industrial}\cite{li2023deep}.}
% Control Variables vs. Measurement Variables
%\revtext{
%In IIoT systems, variables can be categorized into \textbf{system-dependent} and %\textbf{system-independent} types. Faults are primarily manifested in system-dependent %variables.
%System-independent variables, which include \textit{control variables} and %\textit{external factors}, may not be directly affected by faults but can substantially %influence the \textit{measurement variables}~\cite{boyes2018industrial}.
%A significant challenge arises when there are changes in system-independent variables due to novel operating conditions; such changes often lead to patterns in measurement variables that could be mistakenly identified as anomalies.
%There is a pressing need for methods that are '\textit{operating condition aware}' to enable more robust and accurate fault detection.
%}
% Dynamic Graphs & Research Gaps
%\revtext{
%While existing GNN-based methods claim to construct ``dynamic'' graphs, the term %``dynamic'' in this context refers merely to the update of input data; each new set of input data results in a new single, static graph~\cite{li2023deep}.
%Although this static graph can account for the temporal dynamics within the data, it fails to capture the temporal evolution of relationships between time series.
%To address these limitations, we propose the construction of a continuous temporal graph representation for MTS data by dynamically inferring edges between time series. This enables us not only to capture \textbf{node dynamics} (i.e., \textit{intrinsic dynamics of a sensor}) but also \textbf{edge dynamics} (i.e., \textit{evolving relationships between sensors}).
%}
%It employs attention mechanisms to dynamically construct edges, resulting in a continuous graph representation.
% Instead of splitting by anomalies, discuss the methods' ability to model functional relationships within the system
%\revtext{
% Types of Functional Relationships to Model
%\revtext{In the context of IIoT, capturing functional relationships is essential to understand the system behavior and to enable unsupervised early fault detection~\cite{boyes2018industrial} \cite{li2023deep}. The ideal method should be capable of modeling:}
%1) \textbf{Pairwise relationships}: capturing interdependencies between individual sensors' time series data
%2) \textbf{Temporal dynamics}: modeling both the intrinsic dynamics within individual sensors (\textit{node dynamics}) and the evolving relationships between them (\textit{edge dynamics}).
%3) \textbf{Variable types}: Differentiating between system-state independent variables (e.g. \textit{control variables} which are externally set and \textit{external factors} outside of the system) and system-state dependent variables (e.g. \textit{measurement variables} of system internal states).
%}
%Existing GNN-based methods for IIoT predominantly focus on modeling interdependencies between time series but generally overlook the evolution of functional relationships and the distinction between system-dependent and system-independent variables.
% \cite{li2023deep} mentioned fault detection as a special case of abnormal time series
%Methodologically similar to anomaly detection, unsupervised early fault detection primarily focuses on detecting early faults that arise from shifts in internal functional relationships triggered by system-level faults, a critical issue for ensuring the health and functionality of IIoT systems.
% tracking changes helps to detect as early as possible
%Such a graph representation becomes crucial for early fault detection tasks, where it is essential to model these evolving relationships and identify relationship shifts as early as possible in critical systems, thereby enabling proactive actions to prevent failures. To the best of our knowledge, no existing work has addressed this specific challenge of detecting relationship shifts in MTS fault and anomaly detection.
%Traditional fault detection predominantly relies on supervised learning methods, however, faults are usually rare.
%While existing methods have addressed a range of anomalies—such as point anomalies due to sensor misconfiguration and context anomalies resulting from incorrect operational settings or fluctuating demand patterns~\cite{chalapathy2019deep}\cite{han2022adbench}—there remains a significant gap in the literature concerning unsupervised early fault detection.
%Traditional fault detection methods often rely on supervised learning, which requires labeled data for training. However, faults are often rare, making supervised learning impractical in many industrial settings.
%Therefore, fault detection has been commonly treated as an unsupervised learning problem. The approach typically involves training a model that describes normal or ``healthy'' system behavior and using the learned model to detect anomalies by generating a high deviation score whenever a fault occurs.~\cite{michau2019unsupervised}.
% Our method provides superior performance over static graph representations in \textit{dependency anomaly} detection.
% These complexities drive the need for more advanced anomaly detection strategies - a proactive approach to system maintenance that predicts potential equipment failures before they lead to significant damage~\cite{soleimani2021diagnostics}.
% AD from sensor data (multivariate time series data) is an important step in PM
% detect outliers (could be changing in environmental conditions, new operating conditions) and serves as the first step before fault detection
% further distinguishment between new operation (environmental) conditions, new faults, and other impacting systems
%In complex systems, anomalies can often be obscured by noise or the high variability of operating conditions.
%When an anomaly occurs in a complex system, it often indicates a shift in the system's functional relations (\textit{dependency anomaly}) and frequently exhibits mixed characteristics~\cite{martinez2016fault}.
%When the function shift is significant, it can lead to a signal exceeding its standard range, referred to as a \textit{point anomaly}, or altering the temporal pattern of a signal, known as a \textit{contextual anomaly}~\cite{chalapathy2019deep}.
%The data generated from the heterogeneous sensor networks are often characterized by strong temporal dynamics and inter-dependencies, posing a significant change for anomaly detection~\cite{dong2023iotreview}.
%This challenge becomes more pronounced with increasing complexity in IIoTs characterized by rich interactions between interconnected subsystems and components, significant non-linearity and stochastic dynamics, multiple control loops, and fluctuating operating conditions~\cite{younan2020challenges}. Given the increasing presence of complex systems in industrial and infrastructure domains, there is a growing need for the development of advanced anomaly detection methods capable of capturing those relations.
%With the increasing complexity of systems in industrial and infrastructure domains, there is a pressing need for advanced anomaly detection methods.
% intro anomaly
%an anomaly may indicate the onset of system faults, as well as changes in environmental conditions or new operating conditions.
%once an anomaly is detected, it becomes essential to differentiate between system faults and changes in either the environmental or operating conditions. Early fault detection provides an active measure against potential operational failures.
% AD in such a system is hard because partially the fault is hidden in the changing relations
%However, these methods may fall short as they do not explicitly model temporal dependencies between time series. Their ability to detect \textit{dependency anomalies} may not necessarily arise from their ability at identifying shifts in functional relations, but rather from their capacity in recognizing the ``symptoms'' of these evolving relations in the system.
%Nonetheless, even when the functional relation change is substantial, leading to a significantly abnormal system state, the
%``symptoms'' may not be easily observable. Under such circumstances, existing deep learning methods may fail at detecting such anomalies.
%Conventional fault detection methodologies primarily employ supervised learning, necessitating labeled data that is labor-intensive to generate and may not encapsulate real-world complexities~\cite{michau2019unsupervised}.
%To overcome this limitation, Unsupervised Fault Detection emerges as a compelling alternative.
%Both methods share the goal of ensuring system reliability, but fault detection goes a step further to preemptively identify issues before they manifest into larger, more complex problems.
%By utilizing unsupervised learning, this approach does not rely on labeled data, making it particularly advantageous for real-time monitoring and predictive maintenance in dynamically evolving IIoT environments~\cite{ahmed2016survey}. Thus, the transition from general anomaly detection to unsupervised early fault detection not only refines the problem scope but also offers a more proactive solution for maintaining the health of complex, interconnected systems.
%In this context, Unsupervised Fault Detection emerges as a viable alternative. It specifically targets incipient faults, offering a proactive measure for system reliability without requiring labeled data~\cite{hodge2004survey, ahmed2016survey}. Unlike general anomaly detection, which is less time and labor-intensive but often inconclusive for actionable insights, unsupervised early fault detection refines the problem scope and provides a more tailored solution for real-time monitoring in dynamic IIoT environments.
% Systems are getting more complex, it is important to learn the relation between the sensors as it helps to better understand the system's behavior
%Heterogeneous sensor network data often exhibit strong temporal dynamics and interdependencies, posing significant challenges to anomaly detection~\cite{dong2023iotreview}.
%This complexity is amplified in the IIoT, with rich interactions between interconnected subsystems and components, non-linearity, and stochastic dynamics~\cite{younan2020challenges}.
%This makes unsupervised fault detection extremely challenging, especially early faults.
%Early faults—incipient stages of system degradation—are particularly difficult to detect~\cite{wei2020robust}. They often manifest as subtle shifts in the functional relationships between system variables rather than abrupt deviations~\cite{michau2019unsupervised}.
%While traditional anomaly detection aims to identify any deviation from a 'normal' state, early fault detection focuses more narrowly on capturing the incipient faults that are precursors to severe system failures~\cite{hodge2004survey}.
% intro functional relationship shift
%\revtext{
%While anomaly detection is valuable, it often falls short in addressing the critical need for fault identification, especially for early faults.
% needs literature support
%An early fault is a nascent stage of system degradation, often characterized by a gradual shift in functional relationships between system variables, which if not addressed, could escalate into a full-fledged fault.
%Such early faults are particularly challenging to detect as they may not result in immediate, drastic deviations in system behavior but can progressively degrade system performance over time.
% fault detection often done in a supervised way
%Conventional fault detection methodologies often rely on supervised learning approaches, which require labeled data sets. These datasets are either manually annotated, thus requiring expert knowledge and labor, or synthetically generated, which may not capture the complexities of real-world systems\cite{michau2019unsupervised}
%}
% intro early fault/anomalies
%In the complex landscape of IIoT systems, the traditional supervised methods for fault detection often require extensive labeled data, which is not always feasible or efficient to obtain~\cite{chandola2009anomaly}. To overcome this limitation, Unsupervised Early Fault Detection emerges as a compelling alternative. While traditional anomaly detection aims to identify any deviation from a 'normal' state, early fault detection focuses more narrowly on capturing the incipient faults that are precursors to severe system failures~\cite{hodge2004survey}. Both methods share the goal of ensuring system reliability, but early fault detection goes a step further to preemptively identify issues before they manifest into larger, more complex problems. By utilizing unsupervised learning, this approach does not rely on labeled data, making it particularly advantageous for real-time monitoring and predictive maintenance in dynamically evolving IIoT environments~\cite{ahmed2016survey}. Thus, the transition from general anomaly detection to unsupervised early fault detection not only refines the problem scope but also offers a more proactive solution for maintaining the health of complex, interconnected systems.
%Anomaly detection can be defined as the task of finding these non-conforming, atypical patterns in data~\cite{chandola2009anomaly}. Within the scope of our work, anomaly detection is often achieved using unsupervised learning (e. g., [23]). Unlike fault detection, this makes the process of anomaly detection less time and labor-intensive as no annotated data set is required.
%In contrast, a fault is defined as a change in a system so that it can no longer operate satisfactorily and meet the requirements of its user [24]. Typically, to fix a fault, corrective action is required, such as replacement or maintenance. Based on this notion, we can define fault detection as the problem of finding changes in a system so that it can no longer operate in accordance with the requirements of its user.
%Fault detection typically requires some form of a labeled data set, either providing annotated [25], [26] or synthetically created [27] fault examples or providing reference substations to compare against.
% Anomaly/faults are rare -> unsupervised
%However, unlike normal time series data, anomalies are difficult to characterize for two main reasons. First, they are typically associated with rare events, so collecting and labeling them is often a daunting task. Secondly, establishing the full range of potential anomalous events is generally impossible, spoiling the effectiveness of supervised learning techniques. Consequently, unsupervised detection techniques have been widely explored as a practical solution to real-world problems \cite{jin2023survey}
%While anomaly detection is valuable, it is not always sufficient for practical applications. Specifically, a more targeted approach to identifying faults is often required. A fault is defined as a change in system properties that leads to an inability to meet user requirements. Addressing faults usually necessitates corrective action, such as component replacement or maintenance procedures.
%Anomaly detection is essential in ensuring the overall reliability of the IIoT and mitigating potential risks associated with these deviations~\cite{wu2021graph}.
%Condition monitoring serves as a cornerstone for anomaly detection, an essential task for maintaining system health. According to Chandola et al., anomalies are patterns in data that deviate from a pre-defined notion of what is considered normal behavior [Chandola et al., 2009]. These anomalies can be indicators of faults, inefficient system usage, or even benign changes in operating conditions. One significant challenge in anomaly detection lies in the fuzzy boundaries between what is considered normal and abnormal, often leading to false positives or negatives [Chandola et al., 2009].
%Condition monitoring enables health monitoring of the system.
%Following Chandola et al.~\cite{chandola2009anomaly}, we define anomalies as patterns in data that do not conform to a previously defined notion of expected and normal behavior. Similar or synonymous terms found in our survey are outliers, deviations, atypical behavior, or abnormal patterns.
%While anomalies may indicate faults or inefficient usage, there may also be other reasons, such as changes in operating conditions.
%Also, the boundary between normal and anomalous behavior is often fuzzy and hard to define. Therefore, anomalous observations close to the boundary can nactually be normal and vice-versa~\cite{chandola2009anomaly}.
%Such data is essential for monitoring the health condition of the systems that these networks are monitoring and enables anomaly detection, which aims at identifying deviations from healthy system states~\cite{cook2019anomaly}.
%Anomaly detection is essential in ensuring the overall reliability of the IIoT and mitigating potential risks associated with these deviations~\cite{wu2021graph}.
% The need for methods that capture functional relationships
%While deep learning approaches have been effective at modeling point and context anomalies, they often struggle to capture complex functional relationships in MTS data. This is a significant drawback when early fault detection is concerned, as it relies on understanding these relationships to anticipate system-level faults.
% Sota with a focus on functional relationships between time series
%Deep learning methods like Autoencoders (AE)~\cite{aggarwaloutlier}, Recurrent Neural Networks (RNN)~\cite{malhotra2015lstmad}\cite{ergen2019unsupervised}, and Convolutional Neural Networks (CNN)~\cite{garcia2022temporal} have made strides in learning certain aspects of these functional relationships~\cite{fahim2019anomaly, chatterjee2022iot}. AEs, for instance, perform well in reconstructing data but are limited in capturing high-level dependencies between various time series. Similarly, RNNs and CNNs can model temporal dynamics but often fall short in modeling pair-wise relationships between time series.
% Limitations of existing methods
% Sota focuses on point anomalies and context anomalies
% State of the art of MTS AD methods with Deep learning
%Deep learning methods have shown promising results for anomaly detection in sensor networks~\cite{fahim2019anomaly}\cite{ball2017comprehensive}.
%These methods commonly utilize reconstruction or prediction techniques to identify such anomalies~\cite{chatterjee2022iot}.
%Notably, Autoencoder-based (AE) methods~\cite{aggarwaloutlier} have gained popularity due to their ability to learn the functional relationships within MTS. These techniques detect anomalies by monitoring the reconstruction error, also referred to as residuals~\cite{aggarwaloutlier}.
%Besides feed-forward architectures for such tasks, Recurrent Neural Networks (RNN), including Long Short Term Memory (LSTM) models~\cite{malhotra2015lstmad}\cite{ergen2019unsupervised} and convolutional neural networks (CNN)\cite{garcia2022temporal}, are also widely adopted.
%These models excel at capturing dynamic interactions both within and among MTS, and they identify anomalies by assessing the prediction error~\cite{malhotra2015lstmad}.
%However, deep learning methods have limitations in that they cannot explicitly model the relationships between time series, limiting their ability to detect relationship shifts in sensor networks.
%Graph Neural Networks (GNNs) have demonstrated potential in effectively modeling relationships between nodes~\cite{wu2020comprehensive}, offering a novel way to handle the complexity of MTS data~\cite{dong2023iotreview}.
%While most existing GNN methods are developed for homogeneous sensor networks such as traffic networks and power grids, where the graph structure is explicitly given, for heterogeneous sensor networks a graph must first be constructed from the MTS data.
%Sec.~\ref{sec:related_work} provides a brief overview of the related work in the fields of anomaly detection in complex systems, graph learning in MTS, graph representation with MTS, and anomaly detection with GNNs.In Sec.~\ref{sec:method}, we introduce our proposed framework, Dynamic Graph Attention for Anomaly Detection (\revtext{TimeGAT}), specifically designed to address the dynamic nature of MTS. This section dives into the core components of \revtext{TimeGAT}.Sec.~\ref{sec:exp_design} outlines our experimental design with two case studies and the baseline methods. Section~\ref{sec:results} presents the results of our case studies and provides a thorough discussion of the findings. Finally, Sec.~\ref{sec:conclusion} concludes with a brief summary and potential directions for future research.
%1) \textbf{Pairwise relationships} between time series, capturing inter-sensor dependencies.
%2) \textbf{Temporal dynamics}, not only within individual time series but also in the relationships between them, allowing for the modeling of evolving interdependencies.
%3) \textbf{Variable types}, differentiating between control variables, which are externally set and independent of system state but can significantly influence system dynamics, and measurement variables, which are sensor-derived and reflect the current state of the system.
%Although valuable for anomaly detection, which identifies deviations from expected behaviors, this approach is often insufficient for determining actionable faults. Anomalies can arise due to various factors such as changes in operating conditions, making them poor indicators for immediate action~\cite{neumayer2023fault}.
%While anomaly detection can effectively identify abnormal patterns, it often falls short of determining the underlying cause of these anomalies~\cite{neumayer2023fault}. This is because anomalies can arise from various factors, including changes in operating conditions, sensor malfunctions, or even external disturbances.
%While fault detection is already challenging for a system that exhibits spatial-temporal dynamics, this task becomes even more difficult if the goal is to detect incipient faults,
%These early faults are often marked by subtle shifts in functional relationships between system variables~\cite{ANYONETOCITE?}, and their early detection can anticipate severe system failures.
%making anomaly detection a non-trivial task~\cite{dong2023iotreview}. This complexity intensifies among interconnected subsystems and components in the IIoT~\cite{younan2020challenges}.
%However, effectively extracting meaningful information from this complex data to increase system reliability poses a significant challenge.
\subsection{Fault and Anomaly Detection for Time Series Data}
\label{sec:rw_fd}
Given the rarity of faults, this section focuses on reviewing unsupervised Fault Detection (FD) techniques. Additionally, Anomaly Detection (AD) methods are also reviewed, due to their methodological similarity to unsupervised FD.
\textbf{Unsupervised fault detection}. Traditional unsupervised FD has focused on identifying patterns in data that deviate from its normal condition and shifts in data interdependencies. Classical machine learning methods, such as one-class Support Vector Machines (SVMs), have been commonly applied in FD, effectively identifying outliers by enclosing all positive instances within a hyper-sphere~\cite{yin2014ocsvm}. Autoencoders (AEs) are popular in unsupervised FD, facilitating fault detection through monitoring the deviation in the reconstruction errors~\cite{plakias2022novel}\cite{guo2020unsupervised}. Generative Adversarial Networks (GANs) have also been utilized for FD, with the generator creating positive samples and the discriminator differentiating between normal and abnormal instances~\cite{akcay2019ganomaly}\cite{plakias2019exploiting}.
\textbf{Anomaly detection.} In the IIoT context, AD methods focus on modeling system dynamics and identifying deviations from these as anomalies. Approaches such as Recurrent Neural Networks (RNNs)~\cite{li2020anomaly}, Long Short-Term Memory (LSTM) networks\cite{wu2019lstm}, and Convolutional Neural Networks (CNNs)~\cite{garcia2022temporal} are commonly employed, using prediction or reconstruction errors as anomaly indicators. These anomalies typically manifest as point anomalies, often due to sensor faults, or context anomalies, which often arise from operational misconfigurations or changes in environmental conditions~\cite{cook2019anomaly}\cite{chatterjee2022iot}. Additionally, self-supervised methods like Masked Anomaly Detection (MAD) have been developed, enabled by input sequence masking and estimation~\cite{fu2022mad}.
\textbf{Novel operating conditions.} A common challenge in FD and AD is their inability to distinguish between novel operating conditions and faults, which arises from a limited representation of all normal operating conditions in the training data. To address this, Guo \textit{et al.} proposed to combine clustering with expert input to better account for unknown operating modes. Differently, Michau \textit{et al.}~\cite{michau2019unsupervised} proposed to utilize unsupervised feature alignment to extract features under varying conditions and integrate them for fault detection. Expanding upon this approach, \revtextiot{Rombach \textit{et al.}~\cite{rombach2021contrastive} enhanced the feature representation learning with contrastive learning using triplet loss to achieve invariance to novel operating conditions.}
% limitation
Although state-of-the-art FD and AD methods have demonstrated a good performance in modeling interdependencies and dynamics within the data, they fall short in modeling the evolution of relationships in the system. This deficiency affects their ability to detect faults characterized by relationship shifts, especially incipient faults.
% account for novel operating conditions
In addition, to account for novel conditions, current approaches either require labeled operating conditions in the training phase or require explicit modeling to extract operating condition invariant features. Addressing dynamic modeling that can implicitly model operating conditions remains a challenge.
%For example, Guo~\textit{et al.}~\cite{guo2020unsupervised} employ Deep Autoencoders (DAEs) to develop separate fault-free models for each operating condition, as well as a global model with combined normal data from all conditions. Then an assembly model makes a decision based on the reconstruction error of each model.
%Li \textit{et at.}\cite{li2020anomaly} proposed a sequential variational auto-encoder (SISVAE) model for robust estimation and anomaly detection for heartbeat categorization.
% Check the review paper
%Fault detection is significantly relevant in various disciplines and industries for identifying irregular patterns.
% traditional
%Traditional methods including density-based, distance-based, and distributional techniques often fail to capture complex relationships with spatial and temporal dynamics~\cite{bibinbe2022survey}\cite{ball2017comprehensive}.
% deep learning
%With the recent advance of deep learning, there has been a surge in its application for anomaly detection, notably in point and contextual anomalies in industrial settings~\cite{chalapathy2019deep}.
%These methods can be categorized into two main categories: \textit{prediction-based} and \textit{reconstruction-based}.
% mainly temporal
%\textit{Prediction-based} methods employ Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Convolutional Neural Networks (CNNs) to forecast future data points for anomaly detection.
%For instance, Long Short Time Memory (LSTM) networks have been used in aerospace applications~\cite{malhotra2015lstmad}, and Gated Recurrent Unit (GRU) networks in analyzing flight data~\cite{nanduri2016anomaly}.
% mainly inter-dependencies
%On the other hand, \textit{reconstruction-based} methods utilize models such as autoencoders (AEs) to model data inter-dependencies~\cite{aggarwaloutlier}\cite{li2020anomaly}. Specific applications include the use of AEs in cyber-physical production systems~\cite{eiteneuer2019ae} and multiple deep AEs in liquid rocket engines~\cite{feng2022unsupervised}.
% combined TODO add USAD
%Hybrid models like LSTM-Autoencoder (LSTM-AE)~\cite{malhotra2016encdecad} combine dynamics modeling with interdependency modeling for using an LSTM-based Autoencoder for anomaly detection in mechanical devices. LSTM-Variational Autoencoder (LSTM-VAE) further extends this framework for use in robot arms~\cite{park2018lstmvae}.
\subsection{Graph Neural Networks for Multivariate Time Series}
% TODO: mention aggregated graph vs snapshotted graph
\label{sec:graph_tgnn}
% motivation for graph modeling
\revtext{
Graph-based methods are effective tools for modeling complex spatial-temporal dynamics in MTS. Two crucial aspects underlie this process: \textit{graph inference}, involving the construction of graph structures from MTS, and the application of \textit{graph neural networks} for subsequent analysis such as forecasting, imputation or anomaly detection~\cite{jin2023survey}.
\textbf{Graph inference} is the initial step to transform MTS data into temporal graphs, also referred to as spatial-temporal or dynamic graphs (with no distinction among these terms in our context).
In practice, two types of strategies are utilized to construct temporal graphs from MTS data: \textit{heuristics} or \textit{learned} from data~\cite{jin2023survey}. Heuristic-based methods extract graph structures from data based on heuristics such as spatial connectivity~\cite{li2017diffusion} and pairwise similarity~\cite{chen2023multivariate}.
Learning-based methods directly learn the graph structure from the data in an end-to-end fashion. These methods commonly utilize embedding-based~\cite{wu2020connecting}, attention-based~\cite{zhao2020mtadgat}, and sampling-based methods~\cite{zhang22grelen}. These learning-based approaches enable the discovery of more complex and potentially more informative graph structures compared to heuristic-based methods \cite{zugner2021study}.
\textbf{Graph neutral networks} are employed to process the spatial-temporal dynamics captured in the inferred temporal graphs.
We follow the definition by Gao et al.~\cite{gao2022equivalence}, distinguishing between two primary paradigms: \textit{time-and-graph}, where graph representations derived from GNNs are integrated with sequence models like RNNs to jointly capture the temporal dynamics of node attributes; and \textit{time-then-graph}, where sequences that describe node and edge dynamics are first modeled and then incorporated as attributes in a static aggregated graph representation.}
\revtextrewrite{
% list sota methods
Existing GNN methods for MTS mainly follow the \textit{time-and-graph} approach, dynamically constructing a static graph for each input sequence. For instance, Diffusion Convolutional Recurrent Neural Network (DCRNN)~\cite{li2017diffusion} applies SpectralGCN and GRU to a predefined static graph for traffic forecasting. Extending this, Graph for Time Series (GTS)~\cite{shang2021gts} builds upon DCRNN, employing it to a jointly learned probabilistic global graph.
%Similarly, Graph Deviation Network (GDN)~\cite{deng2021gdn} employs Graph Attention Network (GAT) to a static graph inferred from sensor embedding for forecasting.
In the \textit{graph-then-time} category, a variation of \textit{time-and-graph}, Anomaly Detection via Dynamic Graph Forecasting (DyGraphAD)~\cite{chen2023multivariate} is worth noting. Specifically, it generates a series of dynamic correlation graphs from MTS using dynamic time warping and processes these graphs to create a sequence of latent graph representations for forecasting.
Conversely, \textit{time-then-graph} has mainly been applied to networks with predefined graph structures such as evolving social networks or traffic systems, where simultaneous graph inference is not required. Notable examples of the \textit{time-then-graph} frameworks include Temporal Graph Attention Networks (TGAT)~\cite{xu2020tgat} and Temporal Graph Network (TGN)~\cite{rossi2020tgn}.}
\revtext{
% limitation
Given the complex nature of MTS data in the IIoT context, more expressive modeling techniques are crucial to improve fault and anomaly detection performance. Gao \textit{et al.}~\cite{gao2022equivalence} have demonstrated the superior expressiveness of the \textit{time-then-graph} approach for MTS. However, applying it to IIoT settings introduces a research gap: it requires the extraction of edge dynamics to infer the temporal graph, which is different from the static graph inference common in the existing literature.
While some \textit{time-and-graph} approaches do construct dynamic graphs from windowed MTS data, they typically produce a single static graph per input, failing to track evolving edge dynamics and graph structures.
Conversely, DyGraphAD constructs dynamic graphs based on the correlation of MTS but follows a \textit{graph-then-time} approach, emphasizing graph embedding changes over the evolution of pairwise relationships which makes them less applicable to detect faults characterized by relationship changes.
}
%Graph Relational Learning Network (GRELEN)~\cite{zhang22grelen} also adopts a \textit{time-and-graph} approach, learning a stochastic graph via multi-head attention and using DCGRU for signal reconstruction.
%ach time series represents a node, and edges are created based on pairwise similarities between node embeddings (\textit{learning-based}) or based on prior system knowledge (\textit{heuristic-based}).
% limitation
%Existing methods such as DCRNN and MTADGAT predominantly opt for a time-and-graph approach and frequently employ either structure-based or inferred graphs. However, these methods tend to treat graph construction and temporal representation as disjoint tasks, which leads to a less integrated model. Particularly lacking are works that utilize Time-then-Graph representations in settings where the graph must be inferred from MTS data.
%Notably, the majority of research focuses on predefined graph structures, leaving a gap in exploring simultaneous graph inference and time-then-graph representation.
%GNNs are not directly applicable to MTS data in sensor networks. A graph structure must first be constructed from the MTS data.
%Graph construction in MTS involves representing each time series as a node and modeling interactions using edges. \textit{Structure-based} approaches rely on prior knowledge of the system's structure~\cite{kong2022spatio}, while \textit{data similarity-based} methods construct graphs based on the pairwise similarity between time series.
% similarity based
%\revtextrewrite{
%Notable data similarity-based methods for graph construction from sensor networks include
%Graph for Time Series (GTS)~\cite{shang2021gts}, which learns a probabilistic global graph by optimizing the performance over the graph distribution;
%Graph Deviation Network (GDN)~\cite{deng2021gdn}, which builds a K-Nearest-Neighbor (K-NN) graph with the cosine similarity between pairwise sensor node embeddings;
%Graph Relational Learning Network (GRELEN)~\cite{zhang22grelen}, which incorporates a stochastic sampling module;
%Graph Wavenet~\cite{wu2019graph}, which defines the similarity score as the inner product of randomly initialized node embeddings followed by softmax;
%and Spectral Temporal Graph Neural Network (StemGNN)~\cite{cao2020spectral}, who utilizes the self-attention mechanism to learn the latent graph structure in the spectral domain.
%}
% limitations
%However, \textit{structure-based} methods in graph construction have limitations due to their reliance on domain expertise, while current \textit{similarity-based} methods assume static interactions between time series, restricting their ability to detect changes in the relationships within a sensor network.
%\revtextrewrite{Temporal or dynamic graphs derived from MTS data can effectively incorporate the inherent inter-dependencies and dynamics present in the data. Two common frameworks exist for representing temporal graphs: \textit{discrete-time} and \textit{continuous-time}~\cite{zaki2016comprehensive}. In the discrete-time framework, temporal graphs are represented as a series of static graph snapshots. Existing work primarily focuses on this representation, with methodologies like \textit{graph-then-time} and \textit{time-and-graph}~\cite{gao2022equivalence}.
%Conversely, the continuous-time representation offers a more expressive framework for capturing temporal dynamics within a static graph~\cite{gao2022equivalence}. Existing spatial-temporal GNNs mainly use discrete graph representations, such as DCRNN~\cite{li2017diffusion} and MTADGAT~\cite{zhao2020mtadgat}. Multivariate Time-series Anomaly Detection via Graph Attention Network (MTADGAT)~\cite{zhao2020mtadgat} uses two separate graphs to capture feature and temporal inter-dependencies between the time series. It constructs a feature graph based on statistical similarity and a temporal graph based on autoregressive models. These graphs are then processed through a Graph Attention Network followed by a GRU to capture dynamics.Graph Deviation Network (GDN)~\cite{deng2021gdn} builds a K-Nearest-Neighbor (K-NN) graph based on the cosine similarity between pairwise sensor node embeddings. The graph is dynamic and gets updated to capture changing relationships between the time series. Graph Relational Learning Network (GRELEN)~\cite{zhang22grelen} also models each time series as a node and employs a stochastic sampling module to select a subgraph, reducing computational cost and allowing for dynamic updates to the graph structure.
% Highlight the limitation and introduce the gap in research for continuous-time graph representation. However, Gao \textit{et al.}~\cite{gao2022equivalence} argue that the continuous-time framework is more expressive and thus better suited for anomaly detection in MTS. This opens a research gap, as there is currently no work on constructing graphs from MTS and applying continuous graph representation to it.}
%Notably, Gao et al. \cite{39} argue for the greater expressiveness of the continuous-time framework, specifically for anomaly detection in MTS. This highlights a research gap, as no current work jointly constructs a graph from MTS while utilizing a continuous-time representation.
%\revtext{Attention-based graph inference can be seen as regularization of Transformer-like models; - regularization that comes in the form of the relational inductive biases typical of graph processing methods: namely, the sparsity of the pairwise relationships between nodes and the locality of the learned representations. -- cite the sparse graph paper.}
% Limitation
%Current graph construction methods from sensor networks have several limitations. Structure-based methods require domain expertise and are sensitive to the quality of constructed graphs, which can impact model performance. However, these graphs are static and may not capture system dynamics fully. Also data-similarity-based methods have limitations. Existing approaches construct graphs either from the entire time series (\textit{global}) or from snapshots (\textit{local}). Global methods overlook time series dynamics, while local methods assume static interactions. As a result, existing methods face challenges in dealing with complex and dynamic time series in real environments.
%
%Diffusion Convolutional Recurrent Neural Network (DCRNN) is a \textit{time-and-graph} application in traffic forecasting utilizing spectralGCN and GRU, it uses fixed graph structure of the traffic network.
%Graph for Time Series (GTS)\cite{shang2021gts}, which constructs a probabilistic global graph. This graph is sampled from a distribution learned from node embeddings generated by MLPs, and the method employs DCRNN for temporal forecasting in traffic scenarios.
%Multivariate Time-series Anomaly Detection via Graph Attention Network (MTADGAT) infers graph structure ~\cite{zhao2020mtadgat} using two separate GATs to capture feature and temporal relationships and linear layers and GRU for forecasting and reconstruction, it is graph-and-time.
%Graph Deviation Network (GDN)~\cite{deng2021gdn} infers a knn graph based on the cosine similarity between pairwise sensor node embeddings, uses Linear layer with node embedding to forecasting, also graph-and-time.
%Graph Relational Learning Network (GRELEN)~\cite{zhang22grelen} also models each time series as a node, use multihead attention mechasim to learn graph distribution and employs a stochastic sampling module to learn discrete graph, it use DCGRU as temporal module to reconstruct the signal.
\subsection{GNN-based Time Series Fault and Anomaly Detection}
\label{sec:rw_gnn_ad}
\revtextrewrite{
Recent works have explored GNN for MTS anomaly detection. For instance, Deng \textit{et al.}~\cite{deng2022graph} introduced the Spatio-temporal Graph Convolutional Adversarial Network (STGAN)~\cite{deng2022graph}, employing a spatiotemporal generator and discriminator to address the challenge of traffic anomalies with varying criteria across locations and time, thereby enhancing early detection capabilities.
Following this trend, Constant-Curvature Riemannian Manifolds Change Detection Test (CCM-CDT)~\cite{zambon2018concept} utilizes an adversarially trained graph autoencoder. This autoencoder generates latent space points on Riemannian manifolds where statistical tests are performed to identify stationarity changes in graph streams.
Building upon the concept of special analysis, Graph Wavelet Variational Autoencoder (GWVAE)~\cite{li2023novel} utilizes spectral graph wavelet transform which can realize multiscale feature extraction for FD.
To better capture the complex spatial-temporal dependencies in MTS, Time-series Anomaly Detection via Graph Attention Network (MTATGAT)~\cite{zhao2020mtadgat} leverages feature-oriented and time-oriented graph attention mechanisms in a joint reconstruction and forecast discrepancy framework to simultaneously capture spatial and temporal dependencies for more accurate AD.
Similarly, Graph Learning with Transformer for Anomaly Detection (GTA)~\cite{chen2021learning} is a transformer-based forecasting-based approach for cyber attack detection in IIoT systems, leveraging a new graph convolution called influence propagation to simulate the information flow among the sensors.
To address the heterogeneity in IIoT sensors, GDN~\cite{deng2021gdn} utilized sensor embeddings for graph construction and prediction-based detection.
Extending the learned spatial correlation from GDN, Correlation-aware Spatial-Temporal Graph Learning (CST-GL)\cite{zheng2023correlation} further exploited multi-hop graph convolution as well as dilated Temporal Convolutional Network (TCN) to capture long-range dependence over space and time.
Another notable example of GNN-based AD methods is Graph Representation Learning for Anomaly Detection (GRELEN)~\cite{zhang22grelen}, which was the first to propose AD based on graph relation discrepancy. It utilized VAE to learn probabilistic graph relations.
}
% limitation
\revtext{
While several GNN-based approaches have demonstrated effectiveness in AD within Multivariate Time Series, they face two main limitations that need to be addressed. Firstly, these methods primarily employ \textit{time-and-graph} representations and focus on node dynamics via forecasting or reconstruction. These approaches have been effective in detecting anomalies arising from deviated system dynamics, particularly in scenarios where single sensor behaviors (point anomalies) and temporal patterns (context anomalies) are key indicators.
Secondly, and more crucially, these methods exhibit limitations in capturing early-stage faults characterized by relationship shifts. This limitation stems from discrete static graphs generated for each input sequence, which neglects the temporal evolution of the graph structure and potentially misses subtle relationship shifts. To the best of our knowledge, no existing GNN methods effectively address these relationship shifts within MTS.
%Although \textit{time-then-graph} approaches have been applied to MTS with pre-defined graph structures~\cite{gao2022equivalence}, to the best of our knowledge, there exists no prior work that simultaneously infers the graph structure while implementing the \textit{time-then-graph} representation.
}
%GNN-based techniques leverage graph structures to model dependencies in MTS for fault and anomaly detection. Generally trained on normal data, these methods usually employ either forecasting or reconstruction loss minimization objectives. The inability of models to maintain a low discrepancy when fed anomalous data serves as a basis for fault and anomaly detection~\cite{jin2023survey}.
%DyGraphAD~\cite{chen2023multivariate} combines forecasting and relational discrepancy frameworks for enhanced anomaly detection.
%Existing methods face two primary limitations: 1) they capture node dynamics while assuming static relationships between nodes, thereby ignoring the evolution of relationships between time series over time; and 2) they often ignore the difference between system dependent and independent variables in multivariate time series, thereby affecting the accuracy and effectiveness of fault and anomaly detection.
% While not explicitly designed for MTS data, Zambon \textit{et al.}~\cite{zambon2022graph} introduced Graph iForest. This approach extends the isolation forest method by incorporating an incremental partitioning mechanism for the space of graphs. This adaptation makes the isolation forest method applicable to generic attributed graphs for graph anomaly detections.
%However, a common limitation of existing GNN-based methods is their assumption of static graph structures. These methods often rely on pre-defined graph structures or assume that the graph remains unchanged over time, which may not hold true in real-world scenarios with changing relationships and evolving dynamics.
%This limitation highlights the research gap in effectively adapting the \textit{time-then-graph} representation to handle the dynamics and relationships specific to sensor networks in complex systems. Further exploration is needed to develop GNN-based approaches that can dynamically capture and model the evolving graph structures in multivariate time series anomaly detection.
%The existing landscape is diverse. Deng \textit{et al.} proposed STGAN~\cite{deng2022graph}, which utilizes a spatiotemporal generator and discriminator specifically for traffic anomaly detection. Zhao \textit{et al.} introduced MTATGAT~\cite{zhao2020mtadgat}, which employs feature and time-oriented graph attention mechanisms in a joint prediction-reconstruction setup. GDN~\cite{deng2021gdn} focuses on node embeddings for prediction-based anomaly detection. GRELEN~\cite{zhang22grelen} uses a Variational AutoEncoder (VAE) to form a probabilistic relation graph, targeting anomalies via both latent variables and node degrees.
%Despite the richness of existing methodologies, a pervasive limitation is the assumption of static graph structures. These methods often presume static or predefined graphs, an assumption that may not hold in real-world, dynamically evolving systems. This identifies a research gap for the \textit{time-then-graph} representation, particularly for capturing the evolving dynamics in sensor networks of complex systems. Future work should focus on GNN models capable of dynamically modeling evolving graph structures in MTS anomaly detection.
% In the context of MTS AD, the focus is primarily on the discrete-time representation with the \textit{time-and-graph} structure, as it allows capturing the local dynamics and temporal dependencies.
%Moreover, while the difference between different types of time series (such as \textit{control variables}, \textit{external factors}, \textit{measurement variables} is commonly considered in state space models~\cite{durbin2012time} and in control theory~\cite{brunton2022modern}, existing methods do not consider this difference in the context of MTS fault and anomaly detection.
%GDN generates unique embeddings for each sensor, but it omits the distinctions among different variable types.
%We argue that distinguishing among different types of variables in MTS data is critical for fault and anomaly detection because some variables are independent of the presence of faults, whereas others are directly impacted.
\section{Related Work}
\label{sec:rw}
\revtextiot{
This section reviews relevant areas of our proposed method from both application and methodological perspectives.
We begin with a discussion of general unsupervised fault and anomaly detection methods for time series (Sec.~\ref{sec:rw_fd}), chosen for their foundational relevance in understanding and identifying irregular patterns and behaviors in time-series data.
Next, we review methods for multivariate time series graph construction and temporal graph representation learning (Sec.~\ref{sec:graph_tgnn}). This area is selected due to its importance in effectively representing complex relationships in time series data, a core aspect of our approach.
Lastly, we examine the application of Graph Neural Networks (GNNs) in detecting faults and anomalies (Sec.~\ref{sec:rw_gnn_ad}),
highlighting the recent advances in GNN techniques and their relevance to complex system analysis.
}
\input{chaps/02-rw/02-1-rw-ad}
\input{chaps/02-rw/02-2-graph_tgnn}
%\input{chaps/02-rw/02-3-tgnn}
\input{chaps/02-rw/02-4-gnn_ad}
% TODO: add a table to discuss the advantages and disadvantages (change in relationship)
% This section offers a brief review of time series fault and anomaly detection (Sec.~\ref{sec:rw_fd}), methods for MTS graph construction, and temporal graph representation learning (Sec.~\ref{sec:graph_tgnn}).
\revtextrewrite{
In this paper, we utilize bold uppercase letters (e.g., $\mathbf{X}$), bold lowercase letters (e.g., $\mathbf{x}$), and calligraphic letters (e.g., $\mathcal{V}$) to denote matrices, vectors, and sets, respectively.
\subsection{Problem Statement}
% state how MTS is defined in a sensor network
In an IIoT sensor network, $N$ system-dependent measurement signals at any time $t$ are represented as $\mathbf{x}^{t} = [x_{1}^{t}, \cdots, x_{N}^{t}] \in \R^{N}$, where $x_{i}^{t}$ represents the $i^{th}$ sensor's time series at time $t$.
By employing a sliding window approach of length $W$, we construct samples as:
$\mathbf{X}^{t_w:t} = [\mathbf{x}^{t_w}, \cdots, \mathbf{x}^{t-1}, \mathbf{x}^{t}] \in \R^{N \times W}$, where $t_w = t - W + 1 > 0$.
Similarly, system independent variables (control variables and external variables) are denoted $\mathbf{U}^{t_w:t} \in \R^{N_u \times W}$.
The proposed method, DyEdgeGAT, employs a reconstruction model $f_\theta$ for unsupervised MTS fault and anomaly detection.
The model generates an output $\mathbf{\hat{X}}^{t_w:t} = f_\theta(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t})$ for each $N_{train}$ healthy samples, aiming to minimize the reconstruction discrepancy $||\mathbf{\hat{X}}^{t_w:t} - \mathbf{X}^{t_w:t}||$.
At test time, the model outputs $\mathbf{y} \in \R^{N_{test}}$ for all $N_{test}$ samples based on the reconstruction discrepancy, where each element $\mathbf{y}_i \in \{0, 1\}$ indicates faults.
}
\subsection{Framework Overview}
\begin{figure*}[tbhp]
\centering
\centering
\includegraphics[width=\linewidth]{figs/dyedgegat.pdf}
\caption{Overview of the Dynamic Edge via Graph Attention (\textbf{DyEdgeGAT}).
Starting from raw sensor measurements $\mathbf{X}^{t_w:t}$ and system-independent variables $\mathbf{U}^{t_w:t}$, the process involves: \revtextiot{\textbf{(1) Dynamic edge construction}}, where the model infers and tracks evolving interdependencies between time series. \revtextiot{\textbf{(2) Operating condition aware node dynamics extraction}}, augmented by operating condition context via GRU and Layer Normalization (LN) modules. \revtextiot{\textbf{(3) Dynamic interaction learning}}, with two Graph Isomorphism Network (GIN) layers and a Batch Normalization (BN) layer in between. \revtextiot{\textbf{ (4) Reverse signal reconstruction} augments operating condition context and reconstructs} the original sensor signals in the reversed order. \revtextiot{\textbf{(5) Temporal topology-informed anomaly scoring}}, leveraging the learned temporal graph structure to balance different strengths of dynamics in the heterogeneous signals. In the training phase, the model minimizes reconstruction loss using normal data. During the testing phase, the model employs reconstruction discrepancies, adjusted by interaction strengths among sensor nodes for anomaly scoring.}
%We use Gated Recurrent Unit (GRU) as the node and edge encoder followed by Layer Norms (LNs). We use Graph Isomorphism Network (GIN) \cite{xu2018gin} as the graph module wit stands for Batch Norm (BN) layers in between.
\label{fig:dyedgegat_overview}
\end{figure*}
\revtextrewrite{
%TODO ref back to figure
We propose \textit{Dynamic Edge via Graph Attention} (\textbf{DyEdgeGAT}) for fault detection to overcome the limitations of existing GNN-based methods.
To address the complexity of temporal and spatial dynamics in IIoT systems, DyEdgeGAT employs the \textit{time-then-graph} framework based on the aggregated temporal graph representation.
The \textit{time-then-graph} framework in DyEdgeGAT, as the name suggests, sequentially extracts temporal patterns using a sequence model and captures spatial-temporal relationships using GNNs. These two steps are elaborated in detail below.
Unlike traditional models that treat edge relationships as static and focus mainly on node dynamics, DyEdgeGAT employs dynamic edge construction to adaptively capture evolving temporal interdependencies. This enables the model to recognize relationship shifts at early fault stages.
Another key innovation of DyEdgeGAT is its integration of operating condition context into node dynamics extraction. This integration improves the model's robustness under varying operating conditions as well as helps the model distinguish between faults and novel operating conditions.
An overview of the DyEdgeGAT framework can be found in Fig.~\ref{fig:dyedgegat_overview}.
% key components
DyEdgeGAT consists of five core components, each elaborated in subsequent sections:
% todo: link to the figure
% use different colors to highlight steps
\begin{enumerate}
\item \textbf{Dynamic edge construction} enables a novel graph inference scheme for MTS that dynamically constructs edges to represent and track the evolving relationships between time series at individual time steps (Sec.~\ref{sec:method_graph_constr}).
\item \textbf{Operating condition-aware node dynamics extraction} innovatively incorporates operating condition contexts into node dynamics, mitigating false alarms due to novel operating conditions (Sec.~\ref{sec:method_oc_node}).
\item \textbf{Dynamic interaction modeling} utilizes GNNs on the inferred aggregated temporal graph, integrating both node and edge dynamics to capture evolving interactions (Sec.~\ref{sec:method_graph_repr}).
\item \textbf{Reversed signal reconstruction} reconstructs sensor signals, enhanced with reversed operating condition contexts for robust reconstruction (Sec.~\ref{sec:method_recon}).
\item \textbf{Temporal topology-informed anomaly scoring} leverages the temporal graph topology to normalize the anomaly score, taking into account different intensities of dynamics in the heterogeneous signals (Sec.~\ref{sec:method_ad_score}).
\end{enumerate}
}
%Additionally, DyEdgeGAT incorporates the distinction between system-dependent and system-independent variables, enabling robust anomaly detection even under novel operating conditions.
%In the context of a sensor network, sensor measurements of $N$ signals can be represented in Multivariate Time Series (MTS) at a time $t$: $\mathbf{X}^{t} = [x_{1}^{t}, \cdots, x_{N}^{t}] \in \R^{N}$. A sliding window of length $W$ is utilized to construct samples, denoted as $\mathbf{X}^{t_w:t} = [\mathbf{x}_{1}^{t_w:t}, \cdots, \mathbf{x}_{N}^{t_w:t}] \in \R^{N \times W}$.
%Here, $t$ represents the current time step, and $t_w = t - W + 1 > 0$ is the considered historical time step.
%\revtext{UPDATE CONTROL TO INDEPENDENT VARIABLES}
%Similarly, control variables can be represented as $\mathbf{U}^{t_w:t} = [\mathbf{u}_{1}^{t_w:t}, \cdots, \mathbf{u}_{N}^{t_w:t}] \in \R^{N_u \times W}$, where $N_u$ is the number of control variables.
% anomaly detection task
%The task of reconstruction-based unsupervised MTS AD is to reconstruct $\mathbf{\hat{X}}^{t_w:t}$ utilizing $N_{train}$ healthy training samples to minimize the discrepancy $||\mathbf{\hat{X}}^{t_w:t}-\mathbf{X}^{t_w:t}||$. During the test time,
%the model generates an output vector $\mathbf{y} \in \R^{N_{test}}$ for all $N_{test}$ samples in the test set, where each element $\mathbf{y}_i \in \{0, 1\}$ indicates whether the given sample is an anomaly based on the reconstruction error.
%We propose \revtext{\textit{Time-Evolving Graph Attention Network} (TimeGAT}), a novel approach specifically designed to address the dynamic nature of MTS.
% summary contribution
%Unlike existing methods that model relationships between time series as static edges, \revtext{TimeGAT} implements them as dynamic edges.
%These dynamic edges allow \revtext{TimeGAT} to effectively capture complex, time-varying interdependencies. As a result, \revtext{TimeGAT} is equipped to identify relationship shifts in MTS, which might otherwise pose a significant challenge to detect.
%\begin{itemize}
%\item \textbf{Continuous Temporal Graph Representation:} This component provides a continuous graph representation for MTS, forming the foundation of \revtext{TimeGAT}.
%\item \textbf{Dynamic Edge Construction:} This essential component of \revtext{TimeGAT} models the relationships between time series as dynamic edges, effectively capturing the evolution of temporal dependencies within MTS.
%\item \textbf{Operating Condition-Aware Reconstruction:} This aspect of \revtext{TimeGAT} integrates the operating condition context into the temporal graph representation, facilitating reconstruction-based anomaly detection.
%\item \textbf{Topology-informed Anomaly Score:} Using temporal graph topology as a normalizing strategy for the anomaly score
%\end{itemize}
%a continuous temporal graph representation; a dynamic edge construction; a reconstruction mechanism that takes operating conditions into account; and an anomaly score that factors in the graph topology. Each of these elements collaboratively ensures the effectiveness of \revtext{TimeGAT}.
% structure
% Each of these components synergistically ensures \revtext{TimeGAT}'s effectiveness, the details of which will be further explained through ablation studies in the case studies.
\subsection{Dynamic Edge Construction with Attention Mechanism}
\label{sec:method_graph_constr}
\revtext{
The dynamic edge construction module focuses on inferring and representing MTS as temporal graphs. We aim to leverage the equivalence between two common temporal graph representations to effectively represent the relationship changes in MTS.
Following the terminology of Gao \textit{et al.}~\cite{gao2022equivalence}, the two equivalent temporal graph representations are defined as follows.
The first, \textbf{discrete-time dynamic graph}, represents MTS data as a sequence of attributed graphs over discrete time steps: $\mathcal{G} = \{\mathcal{G}^{t_w}, \ldots, \mathcal{G}^t\}$, where each graph $\mathcal{G}^{t_i}(\mathbf{x}^{t_i}, A^{t_i})$ at time instance $t_i$ is defined by its feature matrix $\mathbf{x}^{t_i} \in \R^{N}$ and adjacency matrix $\mathbf{A}^{t_i}\in \R^{N \times N}$.
The second, \textbf{aggregated temporal graph}, models MTS as a static graph aggregating node and edge attributes over time: $\mathcal{G} = (\mathbf{X}^{t_w:t}, \mathbf{A}^{t_w: t})$, with $\mathbf{X}^{t_w:t} \in \R^{N\times W}$ and $\mathbf{A}^{t_w:t} \in \R^{N\times N \times W}$ representing aggregated node and edge attributes, respectively.
By leveraging the equivalence of both graph representations, we construct discrete-time graph snapshots within sliding windows $[t_i, t_i +\delta t]$ for each input sequence interval $[t_w, t]$ using an edge weight matrix $\mathbf{A}^{t_i}$, and then integrate these into an aggregated graph.
}
\revtext{Prior to graph inference, each input node signal $\mathbf{x}_j \in \mathbf{X}$, is pre-processed using a 1D Convolutional Neural Network (1DCNN). The 1DCNN preserves the dimensionality of the input sequence, outputting denoised feature vectors $\mathbf{h}_j = \text{1DCNN}(\mathbf{x}_j)\in \mathbb{R}^W$ for each node $j$, preserving the sequence length while reducing noise, enhancing the robustness of the edge construction process.}
\revtextrewrite{
Edge weights are inferred using a GATv2-based attention mechanism~\cite{brody2022gatv2}, generating a set of attention coefficients $\{\alpha_{jk}^{t_w}, \dots, \alpha_{jk}^{t}\}$ for each time window $[t_w, t]$. To infer the attention score per timestamp, we
first introduce the edge significance scoring function $e: \R^W\times \R^W\times \R^d \rightarrow \R$ to compute the relevance of node $k$'s features to node $j$ at time step $t_i$, defined as:
\begin{equation}
e_{jk}^{t_i} = \mathbf{a}^T \text{LeakyReLU}\left(\mathbf{W} \cdot \left[\mathbf{h}_{j} \parallel \mathbf{h}_{k} \parallel \text{emb}(t_i)\right]\right), \label{eq:score_fun}
\end{equation}
where $\mathbf{a}\in \R^{2d'}$ and $\mathbf{W}\in \R^{d' \times (2W+d)}$ are learnable parameters, and $\parallel$ denotes vector concatenation.
Additionally, to ensure that the edge scores are comparable across different nodes, we normalize them across all neighbors $k\in \mathcal{N}_j$ using the softmax function to obtain the attention coefficients $\alpha_{jk}^{t_i}$:
\begin{equation}
\alpha_{jk}^{t_i} = \text{softmax}_j \left( e_{jk}^{t_i} \right) = \frac{\exp \left(e_{jk}^{t_i} \right)}{\sum_{k^{\prime} \in \mathcal{N}_j} \exp \left(e_{jk^{\prime}}^{t_i}\right)}.
\end{equation}
Here, $\mathcal{N}_j$ denotes the set of neighbors for node $j$. The attention coefficients $\alpha_{jk}^{t_i}$ are then used as the edge weights $A_{jk}^{t_i}$ for time step $t_i$, thereby, capturing the importance of each edge in the graph.
In contrast to traditional static attention-based graph inference, as used in MTADGAT~\cite{zhao2020mtadgat}, our proposed approach in Eq.~\ref{eq:score_fun} integrates a novel term $\text{emb}(t_i)$, which transforms a real-value timestamp into a vector, providing a context of the temporal position.
The formulation of $\text{emb}(t_i)$ aligns with the temporal encoding used in TGN \cite{rossi2020tgn}, which is defined as:
\begin{equation}
\text{emb}(t_i) = \left[ \cos(\omega_0 t_i), \cos(\omega_1 t_i), \ldots, \cos(\omega_d t_i) \right], \label{eq:time_enc}
\end{equation}
with frequencies $\omega_i = \frac{1}{10^{i/d}}$, where $i$ ranges from $1$ to time embedding dimension $d$.
Although both our method and TGN employ the same cosine-based temporal encoding, the intuition behind them differs. Specifically, We encode absolute timestamps to track attention evolution over time while TGN encodes relative timestamps to learn time-invariant features.
%One of the main differences between our method and previous approaches is encoding the absolute temporal position to track attention evolution over time. This is different from TGAT and TGN's relative time encoding, which aims at learning time-invariant features.
}
\revtext{
Finally, we apply a GRU-based mechanism to model the temporal evolution of edge weights (attention coefficients $\mathbf{\alpha}_{jk}^{t_i}$):
\begin{equation}
\mathbf{h}_{jk}^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{\alpha}_{jk}^{t_i}, {\mathbf{h}}_{jk}^{t_i-1})\right), \forall t_i \in [t_w, t].
\label{eq:edge_gru}
\end{equation}
Here, \({\mathbf{h}}_{jk}^{t_i} \in \mathbb{R}\) is the hidden state for edge $(j, k)$ at time \(t_i\), with the initial state ${\mathbf{h}}_{jk}^{t_w-1}$ set to $\mathbf{0}$. The final state ${\mathbf{h}}_{jk}^{t}$ representing the encoded edge dynamics is then used in Eq.~\ref{eq:gnn} as the edge weight $\mathbf{A}\in \mathbb{R}^{N\times N}$ for the static temporal graph.
}
%GATv2 enhances the original GAT~\cite{velivckovic2017gat} by overcoming its static attention limitation. GATv2 introduces modifications to the operation order, allowing the attention scores to be conditioned on the query node, resulting in a more dynamic
%\footnote{\revtext{The term ``dynamic'' is used in two distinct contexts within our attention mechanisms.
%In GATv2, "dynamic" characterizes the attention mechanism's ability to condition the ranking of attention scores on the query node's representation, thereby enabling variable attention patterns unlike the uniform attention in the original GAT.
%In the context of DyEdgeGAT, ``dynamic'' denotes the generation of a sequence of attention scores over time, reflecting the evolving relationships between nodes in the temporal graph.}
%
%As mentioned in Section \ref{sec:method_graph_repr}, discrete-time dynamic graphs and aggregated temporal graphs offer equivalent representations for dynamic graphs.
%we use GRU to The GRU cell updates the edge features to encode the temporal dynamics of the graph structure.
%Formally, the update process is governed by the following equation:
%\revtext{
%In the dynamic graph, edges are temporally evolving entities whose features need to reflect the changing inter-node relationships. To model these dynamics, we employ a gated recurrent unit (GRU) \cite{cho2014properties}, updated with attention coefficients as dynamic weights. These coefficients are calculated through the attention mechanism previously described, allowing the model to focus on the most significant edges at each time step.
%For every edge connecting nodes $i$ and $j$, we compute a hidden state $\mathbf{h}_{ij}^{t_i}$ for each time step $t_i$. This hidden state is a function of the current attention coefficient $\mathbf{\alpha}_{ij}^{t_i}$, which acts as a dynamic weight capturing the temporal importance of the edge, and the previous hidden state $\mathbf{h}_{ij}^{t_i-1}$, which carries over the historical information. The GRU cell updates the edge features to encode the temporal dynamics of the graph structure. Formally, the update process is governed by the following equation:
%\begin{equation}
% \mathbf{h}_{ij}^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{\alpha}_{ij}^{t_i} \odot \mathbf{h}_{ij}^{t_i-1}, \mathbf{h}_{ij}^{t_i-1})\right), \forall t_i \in [t_w, t]
% \label{eq:edge_gru}
%\end{equation}
%where $\odot$ denotes element-wise multiplication, ensuring that the edge features at each time step are updated proportionally to their current relevance as indicated by the attention mechanism.}
%\revtext{State-of-the-art methods for time series analysis commonly employ static edges to represent relationships between nodes in a graph. However, static edges are insufficient to capture evolving temporal dependencies. In contrast, we propose a methodology to construct dynamic edges that adapt to such temporal changes.
%Dynamic edges in our model are constructed by combining a sliding window approach with an attention mechanism, designed to capture evolving temporal dependencies within MTS data.
%}
%Dynamic edges are constructed from MTS within \revtext{TimeGAT} to capture the evolution of temporal dependencies within the data.
%We leverage a sliding window approach and attention mechanism to achieve this.
% it was misleading to call GATv2 as dynamic attention
%To augment this, \revtext{TimeGAT} introduces a sliding window approach, which produces a unique set of attention coefficients for each time stamp within the window. This allows us to capture temporal dependencies and extends the dynamic capabilities of GATv2 to a true temporal dynamism.
%This modification allows the attention scores to be conditioned on the query node, capturing complex and dynamic relationships between the entities in the time series data.
% first we encode each time series $x_i$ with a 1DCNN encoder to get $h_i$
%To preprocess the raw signals, we apply a 1D Convolutional Neural Network (1DCNN) to smooth the signals before constructing the temporal graph. Each time series $\mathbf{x}_i$ is encoded as follows:
%\begin{equation}
% \mathbf{h}_{i} = \text{1DCNN}(\mathbf{x}_i)
%\end{equation}
% Then we slide through with a local slide window with a temporal positional encoder
%Then, we slide through two time series, $\mathbf{h_i}$ and $\mathbf{h_j}$, using a local sliding window of size $\delta t$ to learn the edge dynamics described in Eq.~\ref{eq:edge_dyn}. In this process, we use a scoring function $e$ that calculates a score for each edge $(i,j)$:
%\begin{equation}
% e_{ij}^{t} = \mathbf{a}^T \text{LeakyReLU}\left(\mathbf{W} \cdot \left[\mathbf{h}_{i}^{t:t+\delta t} || \mathbf{h}_{j}^{t:t+\delta t}||\text{emb}(t)\right]\right), \label{eq:score_fun}
%\end{equation}
%where $\mathbf{a}$ is a vector of learned coefficients, $\mathbf{W}$ represents the learned weights for the attention mechanism, and $||$ denotes vector concatenation.
%To introduce non-linearity, we employ LeakyReLU with a negative slope of 0.2. Additionally, we normalize the attention coefficients $\alpha_{ij}^t$ across all neighbors $j\in \mathcal{N}_i$ using the softmax function:
%\begin{equation}
% \alpha_{ij}^t = \text{softmax}_j \left( e_{ij}^t \right) = \frac{\exp \left(e\left(\mathbf{h}_i, \mathbf{h}_j\right)\right)}{\sum_{j^{\prime} \in \mathcal{N}_i} \exp \left(e\left(\mathbf{h}_i, \mathbf{h}_{j^{\prime}}\right)\right)}.
%\end{equation}
%Here, $\mathcal{N}_i$ represents the set of neighboring nodes for node $i$. We use the attention coefficients as the edge weights.
%In addition, we apply a temporal encoding technique presented in Time2Vec~\cite{kazemi2019time2vec} and used in TGAT~\cite{xu2020tgat} and TGN~\cite{rossi2020tgn} on the nodes to incorporate the relative temporal position of the input sequence in Eq.\ref{eq:score_fun}.
%Specifically, we augment the normalized time step as an additional node feature.
%The concept of time encoding is critical in using attention mechanisms for temporal data, and incorporating this information into our model is essential for accurately capturing the dynamics of the system. The time encoding function is defined as:
%\begin{equation}
% \text{emd}(t) = \lbrack cos(\omega_0 t), cos(\omega_1 t), \dots, cos(\omega_d t) \rbrack
%\end{equation}
%where $d$ is the dimension of the time embedding and
%\begin{equation}
%\omega_i = \frac{1}{10^{i/d}}, \quad \text{for}\quad i \in \{0, 1, \dots d\}
%\end{equation}
%\begin{figure}[tbp]
%\centering
% \centering
% \includegraphics[width=.7\linewidth]{figs/graph_constr.pdf}
% \caption{Temporal graph construction from MTS with dynamic edges.}
% \label{fig:graph_constr}
%\end{figure}
%To introduce non-linearity, we employ LeakyReLU with a negative slope of 0.2 in Eq.~\ref{eq:score_fun}.
%\begin{equation}
%\omega_i = \frac{1}{10^{i/d}}, \quad \text{for}\quad i \in \{0, 1, \ldots, d\},
%\end{equation}
%where the frequencies $\omega_i$ are defined as:
%Here, \(\mathbf{h}_{jk}^{t_i} \in \mathbb{R}^{d_e}\) is the hidden state for edge $(i, j)$ at time \(t_i\), with the initial state \(\mathbf{h}_{jk}^{t_w-1}\) set to $\mathbf{0}$. The final state $\mathbf{h}_{jk}^t$ represents the encoded edge dynamics. To adapt the dimensionality of the hidden state to align with the GNN input requirements, a linear layer is subsequently applied:
%\begin{equation}
% \mathbf{h}_{jk}^{t_i\prime} = \text{ReLU}\left(\mathbf{W}_e \cdot \mathbf{h}_{jk}^{t_i} + \mathbf{b}_e\right), \forall t_i \in [t_w, t].
%\end{equation}
%In this equation, \(\mathbf{W}_e \in \mathbb{R}^{d \times d_e}\) and \(\mathbf{b}_e \in \mathbb{R}^{d}\) are trainable parameters of the linear projection layer, and ReLU is applied element-wise. The projected hidden state \(\mathbf{h}_{jk}^{t_i}\) now lies in \(\mathbb{R}^{d}\), suitable for further processing within the GNN architecture. The final state of all edges forms the hidden edge presentation \(\mathbf{H}^{\text{edge}}\) in Eq.~\ref{eq:gnn}, where \(\mathbf{H}^{\text{edge}}\) is a tensor that aggregates the edge features across the graph for a given time step.
%\revtext{
%Finally, we apply a GRU-based mechanism to model the temporal evolution of edge weights (attention coefficients $\mathbf{\alpha}_{jk}^{t_i}$):
%\begin{equation}
% {\mathbf{h}^{\prime}}_{jk}^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{\alpha}_{jk}^{t_i}, {\mathbf{h}^{\prime}}_{jk}^{t_i-1})\right), \forall t_i \in [t_w, t].
% \label{eq:edge_gru}
%\end{equation}
%Here, \({\mathbf{h}^{\prime}}_{jk}^{t_i} \in \mathbb{R}^{d_e}\) is the hidden state for edge $(j, k)$ at time \(t_i\), with the initial state ${\mathbf{h}^{\prime}}_{jk}^{t_w-1}$ set to $\mathbf{0}$. The final state ${\mathbf{h}^{\prime}}_{jk}^{t}$ representing the encoded edge dynamics is then transformed to ${h}_{jk}^{t} \in \mathbb{R}$ via a learnable linear transformation ${h}_{jk}^{t} = \mathbf{W}_e \cdot {\mathbf{h}^{\prime}}_{jk}^{t} + \mathbf{b}_e$.
%The matrix \(\mathbf{H}^{\text{edge}} \in \mathbb{R}^{N \times N}\), containing the final state \(h_{jk}^{t}\) for all edges, is used in Eq.~\ref{eq:gnn} as the edge weight of the static temporal graph.
%}
\subsection{Operating-Condition-Aware Node Dynamics Extraction}
\label{sec:method_oc_node}
\revtext{
In our approach, we emphasize the importance of incorporating operating condition contexts into the process of node dynamics extraction, a critical aspect for enhancing fault detection and ensuring robustness against novel operating conditions.
Traditional AD algorithms often detect new operating conditions as anomalies due to significant deviations from the known dynamics of the training dataset.
By integrating these operational state contexts, our method aims to address this limitation.
Node dynamics extraction in our framework focuses on extracting the dynamics of each individual sensor.
Our approach distinctively distinguishes between measurements that are dependent on the system $\mathbf{X}$ from system-independent variables ($\mathbf{U}$), such as control inputs and external factors in this process.
While $\mathbf{X}$ reflects the system's current state through sensor measurements, $\mathbf{U}$ presents control inputs and external factors. Unlike measurements, $\mathbf{U}$ representing the system's operating state, expresses a strong influence on the system dynamics but remains largely unaffected by system faults.}
\revtext{
We propose to extract operational state context with a Gated Recurrent Unit (GRU) network, which processes the sequence of \(\mathbf{U}^{t_w:t}\). The GRU updates its cell state at each time step to capture temporal dynamics:
\begin{equation}
\mathbf{h}_c^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{U}^{t_i}, \mathbf{h}_c^{t_i-1})\right), \forall t_i \in [t_w, t],
\label{eq:control_gru}
\end{equation}
where \(\mathbf{h}_c^{t_i} \in \mathbb{R}^{d_h}\) represents the hidden state at time \(t_i\) with \(d_h\) being the hidden state dimensionality.
The initial hidden state is set to zero, $\mathbf{h}_c^{t_w-1} = \mathbf{0}$. The final hidden state $\mathbf{h}_c^t$ encodes the dynamic operational state context and is then used to initialize the node encoder for each node $j$:
\begin{equation}
\mathbf{h}_j^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{x}_j^{t_i}, \mathbf{h}_j^{t_i-1})\right), \forall t_i \in [t_w, t].
\label{eq:node_gru}
\end{equation}
Here, \(\mathbf{h}_j^{t_i} \in \mathbb{R}^{d_h}\) is the hidden state for node \(j\) at time \(t_i\), with the initial state \(\mathbf{h}_j^{t_w-1}\) set to $\mathbf{h}_c^t$. The final state $\mathbf{h}_j^t$ represents the encoded dynamics of node $j$ up to time $t$ with the context of operating conditions. The final states of all nodes form the hidden node presentation $\mathbf{H}\in \mathbb{R}^{N\times d_h}$ in Eq.~\ref{eq:gnn}.
}
Note that the GRU in Eq.~\ref{eq:control_gru} is a multivariate GRU, processing all control variables, whereas the GRU in Eq.~\ref{eq:node_gru} is a univariate GRU with shared weights across all nodes.
%\footnote{
%\revtext{When system-independent variables $\mathbf{U}$ are unidentified, we infer control variables from domain-specific set points following control theory. Similarly, we deduce external factors by their systemic influence; for instance, alongside water flow control, input water temperature is identified as an external factor.}}.
%DyEdgeGAT enables fault and anomaly detection within IIoT environments by leveraging a reconstruction framework aimed at reconstructing the measurement variables ($\mathbf{X}$) and utilizing the deviations at test time as an indicator of anomalies.
%With the operating condition-aware node dynamics obtained from Eq.\ref{eq:node_gru}, together with the edge dynamics obtained from Eq~\ref{eq:edge_gru}, we finally obtain the hidden node representation $\mathbf{z}_i$ after the GNN layers, as shown in Eq.~\ref{eq:gnn}.
%We then use GRU followed by a linear layer to reconstruct the original $\mathbf{X}^{t_w:t}$:
%\begin{equation}
% \mathbf{\hat{X}}^{t_w:t} = \mathbf{W}\cdot(\sigma\cdot\text{GRU}(\mathbf{z}_i, \mathbf{\tilde{h}}_c) + \textbf{b}
%\label{eq:node_gru_new})
%\end{equation}
%Here, $\mathbf{\tilde{h}}_c$ is the last hidden state of the reverse of $\mathbf{U}^{t_w:t}$ in time, which is inserted in Eq.~\ref{eq:control_gru}.
%Here, $\mathbf{Z} = {\mathbf{z}_i^{t_w}, \ldots, \mathbf{z}_i^{t}}$ is the sequence of hidden node representations, and $\mathbf{\tilde{h}}_c^{t_w}$ represents the refined context vector, capturing the system's operating state over the window $[t_w, t]$. This design allows the model to differentiate between anomalies and regular shifts in operating conditions, thereby enhancing the accuracy of anomaly detection.
% TODO: mention the flip
%\revtext{
%\textit{Control variables} are parameters that are explicitly manipulated to influence system behavior, such as actuator settings or system thresholds.
%\textit{External factors} are variables that are not directly manipulated by the IIoT system but have a significant impact on the system's behavior, such as environmental conditions or variables from other systems.
%\textit{Measurement variables}, on the other hand, are observed sensor measurements that are influenced by both control variables and external factors.
%}
%In the domain of IIoT, \textit{control variables} (\textit{C}) and \textit{measurement variables} (\textit{M}) exhibit distinct roles; \textit{C} are parameters manipulated to influence system behavior, while \textit{M} are the resulting sensor measurements. External factors (\textit{E}), though not controlled, significantly impact \textit{M}.
% The novelty of \revtext{TimeGAT} lies in its explicit distinction between \textit{C} and \textit{M} during the reconstruction phase, a step that most anomaly detection models overlook.
%To mitigate false positives, commonly arising from shifts in \textit{C} rather than system faults, \revtext{TimeGAT} dynamically integrates \textit{C} into the temporal graph, enabling an 'operating condition aware' reconstruction. This integration allows the framework to discern changes in sensor relationships attributable to operating state modifications from those due to anomalies.
%To mitigate false positives, commonly arising from shifts in $\mathbf{U}$ rather than system faults, \revtext{TimeGAT} dynamically integrates the context of the operating conditions into the temporal graph, enabling an ``operating condition aware'' reconstruction. This integration allows the framework to discern changes in sensor relationships attributable to operating state modifications from those due to anomalies.
%It is important to distinguish between system-dependent measurements ($\mathbf{X}$) and system-independent variables ($\mathbf{U})$ in IIoT environments.
%\revtext{TimeGAT} enables anomaly detection by employing a reconstruction module, where the aim is to reconstruct the measurement variables and subsequently use deviation at test time as an anomaly score.
%A unique feature of \revtext{TimeGAT}, setting it apart from existing models, is its differentiation between control variables and sensor measurements.
%Most models do not make such a distinction between the reconstructed signals and treat all the signals in the same way.
%Control variables define the operating state of a system, and in many real-world applications, this state can change over time,
%Concentrating only on measurement variables could lead to scenarios where the relationship between two sensor signals has changed, not because of an anomaly, but due to a change in the operating state.
%To accurately identify functional shifts between measurements, whether they are due to a change in the operating state or a system-level fault, it is essential to incorporate the context of the operating state, which is determined by the control variables, into the temporal graph.
%For operational state context extraction, a Gated Recurrent Unit (GRU) network processes the sequence of control variables \(\mathbf{U}^{t_w:t}\). The GRU updates its cell state at each time step to capture the temporal dynamics, with the update succinctly described by:
%\begin{equation}
%\mathbf{H}_t = \text{GRU-Cell}(\mathbf{U}_t, \mathbf{H}_{t-1}),
%\end{equation}
%where \(\mathbf{H}_t\) represents the embedding of the sub-sequence \(\mathbf{U}^{1:t}\) for \(1 \leq t \leq T\), encapsulating the information up to the current time step \(t\). We initialize the sequence with \(\mathbf{H}_0 = 0\) to denote the starting state with no prior information. The final hidden state \(\mathbf{H}_T\) is then passed through a ReLU activation function to yield the operational state context vector:
%\begin{equation}
%\mathbf{h}_c = \text{ReLU}(\mathbf{H}_T).
%\end{equation}
%\revtext{TimeGAT} accomplishes this by effectively integrating the context derived from the control variables.
%To extract the context of the operating state, we apply a GRU to the control variable time series $\mathbf{U}^{t_w:t}$:
%\begin{equation}
% \mathbf{h}_c = \sigma\cdot\text{GRU}(\mathbf{U}^{t_w:t}) \label{eq:control_gru}
%\end{equation}
%Here $\mathbf{h}_c$ is the last hidden layer of GRU.
%Then we use it to initialize, the node encoder for each node $i$ in Eq.\ref{eq:node_dyn}:
%\begin{equation}
% \mathbf{h}_i = \sigma\cdot\text{GRU}(\mathbf{x}_i, \mathbf{h}_c) \label{eq:node_gru}
%\end{equation}
%This approach ensures that \(\mathbf{h}_c\) captures the relevant temporal features from the control variable time series, thus providing a comprehensive view of the operational state context.
%Note that the GRU in Eq.~\ref{eq:control_gru} is a multivariate GRU, where we pass all control variables, whereas the GRU in Eq.~\ref{eq:node_gru} is a univariate GRU that shares weights for all nodes.
%To reconstruct the original sequence of measurement variables $\mathbf{X}^{t_w:t}$, we apply a GRU network followed by a linear transformation:
%These representations incorporate the temporal context of both the node and edge dynamics, allowing for a comprehensive encoding of the system state under varying operating conditions.
%To reconstruct the original sequence of measurement variables $\mathbf{x}_i^{t_w:t}$ from node hidden representations $\mathbf{z}_i$, we utilize again a GRU network for sequence reconstruction.
%To align dimensions for sequence reconstruction from $\mathbf{z}_i$, we apply a dimensionality-matching linear layer before the GRU: $\mathbf{z}_i^{\prime} = \mathbf{W}_z \mathbf{z}_i + \mathbf{b}_z$ such that $\mathbf{z}_i^{\prime} \in \mathbb{R}^{d_h}$. The transformed node features $\mathbf{z}_i^{\prime}$ are then processed with
%the GRU processes the time-evolving hidden node representation $\mathbf{z}_i^{\prime}$,
%\begin{equation}
%\mathbf{{h}}_{o, i}^{t_i} = \text{ReLU}(\text{GRU-Cell}(\mathbf{z}_i^{\prime}, \mathbf{{h}}_{o, i}^{t_{i}-1})), \forall t_i \in [t_w, t]
%\end{equation}
%here GRU is initialized with the last hidden state $\mathbf{\tilde{h}}_c^{t_w}$ of the GRU that processes the reversed sequence of control variables $\mathbf{U}^{t_w:t}$ (introduced in Eq.~\ref{eq:control_gru}), to capture the most recent operating conditions that might affect the reconstruction quality.
%then $\mathbf{\hat{x}}_i^{t_i} = \mathbf{W}_o \mathbf{{h}}_{o, i}^{t_i} + \mathbf{b}_o$, with $\mathbf{W}_o \in \mathbb{R}^{1 \times d_h}$ and $\mathbf{b}_o \in \mathbb{R}$.
% We ensure that the model emphasizes the latest operational conditions, thereby enhancing the accuracy of earlier state reconstructions.
%Here the GRU cells are initialized with $\mathbf{{h}}_{o, i}^{t_w-1} = \mathbf{\tilde{h}}_c^{t_w}$, which is the final hidden state from the GRU processing the reversed sequence of control variables $\mathbf{U}^{t_w:t}$, as introduced in Eq.\ref{eq:control_gru}. This captures the recent operating conditions pertinent to the reconstruction quality.
%These representations, encoding node, and edge temporal context, effectively capture system dynamic states across conditions. To reconstruct the sequence of measurement variables $\mathbf{x}_j^{t_w:t}$, a GRU network transforms $\mathbf{z}_j$ into $\mathbf{z}_j^{\prime} \in \mathbb{R}^{d_h}$ via $\mathbf{z}_j^{\prime} = \mathbf{W}_z \mathbf{z}_j + \mathbf{b}_z$. The GRU then processes transformed node features $\mathbf{z}_j^{\prime}$ in reverse order
%These representations encode the temporal context of node and edge dynamics, effectively capturing the system state under varying operating conditions.
%With the operating condition-aware node dynamics obtained from Eq.~\ref{eq:node_gru}, and the edge dynamics from Eq.~\ref{eq:edge_gru}, we extract the hidden node representations $\mathbf{z}_j \in \mathbb{R}^{d_z}$ via the GNN layers as presented in Eq.~\ref{eq:gnn}.}
%Inspired by the Seq2Seq \cite{cho2014learning} model for neural machine translation\footnote{\revtext{Seq2Seq is an encoder-decoder LSTM based neural machine translation model that reverses the order of the input source sentence to better align the relevant parts of the source and target sentences.}}, our approach reconstructs the sequence in reverse order.
%By reversing the order of the sequence, we potentially shorten the path between the encoder and the decoder because it starts reconstructing from the most recent time step from the encoder. This could facilitate learning by bringing temporally distant but causally related features closer in the sequence, improving GRU's reconstruction quality.
%Inspired by the Seq2Seq model\footnote{\revtext{Seq2Seq is an encoder-decoder LSTM based neural machine translation model that reverses the order of the input source sentence to better align the relevant parts of the source and target sentences.}} \cite{cho2014learning}, this reversal potentially shortens the path between encoder and decoder, facilitating learning by aligning temporally distant but causally related features, thus enhancing GRU's reconstruction quality.
\subsection{Dynamic Interaction Modeling}
\label{sec:method_graph_repr}
\revtext{
Dynamic interaction modeling focuses on modeling dynamic system interactions using GNNs.
To integrate the dynamics in the learning architecture, we propose to employ GNNs on a static weighted aggregated temporal graph $\mathcal{G}(\mathbf{H}, \mathbf{A})$, that reflects system dynamic states across various operating conditions.
This graph aggregates a hidden node representation containing operating condition-aware node dynamics (Eq.~\ref{eq:node_gru}) as node attributes, denoted by $\mathbf{H} \in \mathbb{R}^{N \times N_{d_h}}$, and a hidden edge representation encapsulating edge dynamics (Eq.\ref{eq:edge_gru}) as edge weights, represented by $\mathbf{A} \in \mathbb{R}^{N \times N}$.
The interaction learning process applies multiple GNN layers to extract hidden node representations $\mathbf{z}_j \in \mathbb{R}^{d_z}$:
\begin{align}
\mathbf{Z} &= \text{GNN}^L(\mathbf{H}, {\mathbf{A}}). \label{eq:gnn}
\end{align}
Here, $\mathbf{Z} \in \mathbb{R}^{N\times d_z}$ is the final node representation after $L$ layers of GNN processing, where $d_z$ is the dimensionality of the output space, and $\mathbf{z}^0$ is initialized with $\mathbf{H}$.
}
\revtext{
The GNN layer in Eq. \ref{eq:gnn} follows the Message Passing Neural Network (MPNN) schema~\cite{gilmer2017neural}. At the $l$-th GNN layer, message passing and update steps occur as follows:
\begin{align}
\mathbf{m}^{(l)}_j &= \sum_{j \in \mathcal{N}_j} \text{MSG}^{(l)} (\mathbf{z}^{(l-1)}_j, \mathbf{z}^{(l-1)}_k, {A}_{jk}), \\
\mathbf{z}^{(l)}_j &= \text{UPDATE}^{(l)} (\mathbf{z}^{(l-1)}_j, \mathbf{m}^{(l)}_j).
\end{align}
Here, $\mathcal{N}_j$ denotes the neighborhood of node $j$, $\mathbf{m}^{(l)}_i$ is the message accumulated at node $j$ in layer $l$, and ${A}_{jk}$ represents the edge weights between nodes $j$ and $k$ of the aggregated temporal graph. The functions $\text{MSG}$ and $\text{UPDATE}$ are learnable transformations specific to the GNN architecture, which aggregate information from the neighboring nodes and update the node representation, respectively.
Specifically, we utilize Graph Isomorphism Network (GIN)~\cite{hu2019gine} as the GNN module, which allows the $\text{MSG}$ function to incorporate edge weights.
The $\text{UPDATE}$ function is a two-layer MLP (Multi-Layer Perceptron) with Batch Normalization (BN)
Node $j$'s representation at layer $l$, $\mathbf{z}^{(l)}_j$, updates as:
\begin{equation}
\mathbf{z}^{(l)}_j = \text{MLP}^{(l)}\left( (1 + \epsilon^{(l)}) \cdot \mathbf{z}^{(l-1)}_j + \sum_{k \in \mathcal{N}(j)} {A}_{jk} \cdot \mathbf{z}^{(l-1)}_k \right). \label{eq:gnn_z}
\end{equation}
Here, $\epsilon^{(l)}$ is a learnable parameter at layer $l$ that allows the model to weigh self-connections. BN is employed between GIN layers for normalization.
}
%The aggregated temporal graph contains information on system dynamic states across conditions.
%To address the complexity of temporal and spatial dynamics in IIoT systems, DyEdgeGAT employs the \textit{time-then-graph} framework based on the aggregated temporal graph representation.
%The \textit{time-then-graph} framework in DyEdgeGAT, as the name suggests, sequentially extracts temporal patterns using a sequence model and captures spatiotemporal relationships using GNNs. These two steps are elaborated in detail below.
%\textbf{First time}. During the temporal dynamics extraction phase, Gated Recurrent Units (GRUs) followed by Layer Normalization (LN) and activation functions are utilized for both, nodes and edges to model dynamics over the time window $[t_w, t]$ of length $W$. The dynamics are encoded as:
%\begin{align}
%\mathbf{H}^{node} &= \text{GRU}^{\text{node}}(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t}), \quad \forall i \in \mathcal{V}, \label{eq:node_dyn}\\
%\mathbf{H}^{edge} &= \text{GRU}^{\text{edge}}(\mathbf{A}^{t_w:t}), \quad \forall (i,j) \in \mathcal{E}, \label{eq:edge_dyn}
%\end{align}
%where $\mathbf{H}^{\text{node}} \in \mathbb{R}^{N\times d_n}$ and $\mathbf{H}^{\text{edge}} \in \mathbb{R}^{N \times N }$ denote the latent dynamics representations of nodes and edges, respectively, with $d_n$ representing the hidden node feature dimensions.
%Here, $\mathcal{V}$ is the set of nodes, and $\mathcal{E}$ is the set of edges.
%In Eq.~\ref{eq:node_dyn}, the matrix $\mathbf{X}^{t_w:t} \in \mathbb{R}^{N \times W}$ represents the sliding window of system-dependent signals, while $\mathbf{U}^{t_w:t} \in \mathbb{R}^{N_u \times W}$ represents system-independent signals, capturing external factors or control variables.
%These system-independent signals $\mathbf{U}^{t_w:t}$ reflect the operating conditions of the system and should be treated separately from the dynamics of the system-dependent measurements $\mathbf{X}^{t_w:t}$. In Sec.~\ref{sec:method_oc_recon}, we will elaborate in more detail on how the context of operating conditions is augmented in Eq.\ref{eq:node_dyn}.
%In Eq.~\ref{eq:edge_dyn}, $\mathbf{A}^{t_w:t}\in \mathbb{R}^{N \times N \times W}$ represents a sequence of aggregated edge weights over \( [t_w, t] \), which differing from the conventional spatial-temporal graph approach where $\mathbf{A}$ is assumed to be static for each input $\mathbf{X}^{t_w:t}$. Therefore, it is necessary to construct dynamic edges to capture the evolving interactions, and further details on this construction will be provided in Sec.~\ref{sec:method_graph_constr}.
%The \textit{time-then-graph} framework in DyEdgeGAT extracts temporal patterns via sequence models and spatio-temporal relationships through GNNs. As the name suggests, it operates in two consecutive phases: the extraction of temporal dynamics followed by graph processing. Through continuous temporal representation, it can better capture the evolution of the relationships between sensor signals.
% Another important observation relates to $\mathbf{A}^{t_w:t}$ in Eq.~\ref{eq:edge_dyn}. Traditional GNN-based methods consider $\mathbf{A}$ as static for the sliding MTS sequence (or for the entire time series). However, in order to extract edge dynamics, it is necessary to construct dynamic edges. We will elaborate on this aspect in Sec.~\ref{sec:method_graph_constr}.
% We formally define a graph as $\mathcal{G}=(\mathcal{V}, \mathcal{E})$, where $\mathcal{V}=\{v_i\}$ is the set of nodes and $\mathcal{E}=\{e_{ij}\}$ is the set of edges.
%We represent a temporal graph over a sliding window of $w$ time steps as a single, aggregated static graph $\mathcal{G}=(\mathbf{X}^{t_w:t}, \mathbf{A}^{t_w: t})$, where $\mathbf{X}^{t_w:t}=[\mathbf{X}^{t_w},\cdots,\mathbf{X}^t] \in \R^{|\mathcal{V}|\times W}$ represents the node attributes, and $\mathbf{A}^{t_w:t}=[\mathbf{A}^{t_w},\cdots,\mathbf{A}^t] \in \R^{|\mathcal{V}|\times|\mathcal{V}|\times W} $ represents the edge attributes. These attributes integrate all past node and edge information, as defined in~\cite{gao2022equivalence}.
%is to first represent the evolving relationships of node and edge attributes as sequences and subsequently embed these sequences in a static graph using a GNN.
%Here, $H^{node}$ and $H^{edge}$ represent the evolving node and edge representations, respectively. $E$ denotes the set of all possible edges in an aggregated form, and $Z$ yields the final temporal graph representation encompassing all nodes.
%A temporal encoder $f$ is used to extract the dynamics of node and edge attributes. The resulting node and edge representations are utilized as the node and edge attributes in a new (static) graph, which is subsequently encoded by a Graph Neural Network (GNN). Following the notation of Gao \textit{et al.}~\cite{gao2022equivalence}, we define the aggregated temporal graph as follows:
%\begin{align}
% \mathbf{H}^{node} &= f^{node}(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t}), \quad \forall i \in \mathcal{V}, \label{eq:node_dyn}\\
% \mathbf{H}^{edge} &= f^{edge}(\mathbf{A}^{t_w:t}), \quad \forall (i,j) \in \mathcal{E}, \label{eq:edge_dyn}\\
% \mathbf{Z} &= \text{GNN}^L(\mathbf{H}^{node}, \mathbf{H}^{edge})\label{eq:gnn}.
%\end{align}
%Here, $\mathbf{H}^{node} \revtext{\in \R^{|\mathcal{V}|\times d_n}}$ and $\mathbf{H}^{edge} \revtext{\in \R^{|\mathcal{E}|\times d_e}}$ represent the dynamics of nodes and edges, respectively. $\mathcal{E}$ denotes the set of all possible edges in an aggregated form, and $\mathbf{Z}$ represents the final temporal graph representation of all nodes. $L$ corresponds to the number of GNN layers. For the node encoder $f^{node}$ and edge encoder $f^{edge}$, we use separate GRU models, followed by a Layer Norm (LN) and an activation function to extract dynamics. The GNN model is then employed to learn the graph representation, utilizing the Graph Isomorphism Network (GIN)~\cite{xu2018gin} as the graph module and incorporating Batch Normalization (BN) between the GNN layers.
%Here, \(\mathbf{H}^{node} \in \R^{|\mathcal{V}|\times d_n}\) and \(\mathbf{H}^{edge} \in \R^{|\mathcal{E}|\times d_e}\) depict the dynamics of nodes and edges. \(\mathcal{E}\) signifies all potential edges in aggregated form, with \(\mathbf{Z}\) as the overall temporal graph representation. For node and edge encoders, separate GRU models enhanced with Layer Norm (LN) and activation functions are employed. The GNN employs the Graph Isomorphism Network (GIN) with Batch Normalization (BN) interleaved between its layers.
%Note that in the original formulation, $f^{node}$ in Eq.~\ref{eq:node_dyn} is a function of only $\mathbf{X}^{t_w:t}$. However, we argue that the control variables provide valuable information about the operating conditions of the system and should be treated separately from the dynamics of the measurements. Therefore, they are modeled differently. In Sec.~\ref{sec:method_oc_recon}, we provide a detailed explanation of how we include the control variables in the node encoding function to capture the context of the operating conditions.
\subsection{Reversed Signal Reconstruction}
\label{sec:method_recon}
\revtextrewrite{
DyEdgeGAT adopts an innovative approach for fault detection by reconstructing the measurement variables ($\mathbf{X}$) in a reversed order, inspired by a technique used in the Seq2Seq model for neural machine translation~\cite{cho2014learning}. This novel reversal technique helps to effectively align temporally distant but causally relevant features, thereby facilitating learning in longer sequences where gradient vanishing during weight backpropagation poses a challenge.
The reconstruction begins with the final graph representation (Eq.~\ref{eq:gnn_z}), which encodes system dynamic states across operating conditions.
For reconstructing the original sequence of measurement variables $\mathbf{x}_j^{t_w:t}$, a GRU network is repurposed.
%Since the GRU encoder processes information sequentially, information from the initial timestamp is deeply embedded in the hidden layers. Due to potential gradient vanishing during weight backpropagation, it is challenging to reconstruct the initial data, particularly in longer sequences.
%Drawing inspiration from the Seq2Seq model~\cite{cho2014learning}, which reverses the order of the input source sentence to better align the relevant parts of the source and target sentences for neural machine translation, we construct $\mathbf{x}_j^{t_w:t}$ in a reversed order.
%This reversion potentially shortens the path between encoder and decoder, facilitating learning by aligning temporally distant but causally relevant features.
Intially, $\mathbf{z}_j$ undergoes a linear transformation $\mathbf{z}_j^{\prime} = \mathbf{W}_z \mathbf{z}_j + \mathbf{b}_z$, yielding $\mathbf{z}_j^{\prime} \in \mathbb{R}^{d_h}$.
The GRU then processes the transformed node features $\mathbf{z}_j^{\prime}$ in reverse order:
\begin{equation}
\overleftarrow{\mathbf{h}}_{o, j}^{t_i} = \text{ReLU}\left(\text{GRU-Cell}(\mathbf{z}_j^{\prime}, \overleftarrow{\mathbf{h}}_{o, j}^{t_{i}-1})\right), \forall t_i \in [t, t_w].
\end{equation}
Initialization of the GRU cells uses $\overleftarrow{\mathbf{h}}_{o, j}^{t-1} = \overleftarrow{\mathbf{h}}_c^{t_w}$, the final hidden state from the reversed control variable sequence $\mathbf{U}^{t_w:t}$ (Eq.~\ref{eq:control_gru}) in a similar fashion. The predicted sequence for node $j$ is then reconstructed at each timestep $t_i$:
Similarly, we inverse the operating condition context as well, and use $\overleftarrow{\mathbf{h}}_{o, j}^{t-1} = \overleftarrow{\mathbf{h}}_c^{t_w}$ to initialize the GRU cells, which is the last hidden state of the GRU for the reversed control variable sequence $\mathbf{U}^{t_w:t}$ in Eq.~\ref{eq:control_gru}.
Finally, we reconstruct the predicted sequence for each node $j$ through a linear output layer for each timestep $t_i$:
\begin{equation}
{\hat{x}}_j^{t_i} = \mathbf{W}_o \overleftarrow{\mathbf{h}}_{o, j}^{t_i} + \mathbf{b}_o, \forall t_i \in [t_w, t]
\end{equation}
where $\mathbf{W}_o \in \mathbb{R}^{1 \times d_h}$ and $\mathbf{b}_o \in \mathbb{R}$ form the linear output layer.
}
%Inspired by the Seq2Seq model
%\footnote{\revtext{Seq2Seq is an encoder-decoder LSTM based neural machine translation model that reverses the order of the input source sentence to better align the relevant parts of the source and target sentences.}}
%~\cite{cho2014learning}, this reversal potentially shortens the path between encoder and decoder, facilitating learning by aligning temporally distant but causally close features. This enhances the GRU's reconstruction quality.
\subsection{Training Objective}
\revtext{
The objective of the training is to minimize the discrepancy between the reconstructed sequence \({\hat{x}}_j^{t_i}\) and the true sequence \({x}_j^{t_i}\) across all sensors and the entire sliding window. The \( p \)-norm loss function is defined as:
\begin{equation}
\mathcal{L}= \left(\frac{1}{N \cdot W} \sum_{j=1}^{N} \sum_{t_i=t_w}^{t} \left\| \hat{x}_j^{t_i} - x_j^{t_i} \right\|_p^p \right)^{1/p}, p \geq 1
\end{equation}
where $N$ is the number of nodes, $W$ is the sliding window length, and $p$ is the norm degree. The parameters of the reconstruction model \( f_\theta \) are optimized to minimize this loss across all training samples \( N_{train} \).
}
%The training process seeks to optimize the model parameters to minimize \( L \) using gradient descent.
%For \( p=2 \), the loss becomes the Mean Squared Error (MSE), and for \( p=1 \), it results in the Mean Absolute Error (MAE).
%\subsection{Training Objective}
%The training objective is formulated as the minimization of the reconstruction error across all sensors in the network and across the entire sliding window. This is achieved by employing a \( p \)-norm loss function. Specifically, the loss can be expressed as:
%\begin{equation}
%L(\theta) = \left( \frac{1}{N \cdot W} \sum_{i=1}^{N} \sum_{t_j=t_w}^{t} \left\| \hat{x}_{i}^{t_j} - x_{i}^{t_j} \right\|_p^p \right)^{1/p},
%\end{equation}
%where \( N \) represents the number of sensors, \( W \) is the length of the sliding window, \( \hat{x}_{i}^{t_j} \) is the reconstructed value for sensor \( i \) at time \( t_j \), \( x_{i}^{t_j} \) is the true sensor reading, and \( p \) denotes the order of the norm. The parameters of the reconstruction model \( f_\theta \) are tuned to minimize this loss over the set of \( N_{train} \) healthy samples.
\subsection{Temporal Topology-Based Anomaly Score Design}
\label{sec:method_ad_score}
% Revised explanation of dynamic edge weights and the physical meaning of topology
\revtext{
In heterogeneous IIoT environments, sensors exhibit diverse dynamic behaviors, which can impact signal reconstruction quality and, consequently, have a negative impact on fault and anomaly detection. Anomaly scoring functions derived directly from reconstruction errors are often biased toward sensors with more significant dynamics due to their larger error magnitudes.
To address this, we propose a temporal topology-based anomaly score, where ``temporal topology'' refers to the interaction strengths among sensor nodes over time, encoded in the aggregated temporal graph. This structure not only reflects the strength of these interactions but also their evolution in time. Changes in interaction strength with other signals are indicative of potential faults, offering an effective method for identifying subtle faults.}
\revtextrewrite{
Building on the concept of strength in dynamic interaction, our proposed methodology normalizes the reconstruction error of each sensor signal sequence $\mathbf{x}_j \in \mathbf{X}^{t_w:t}$ by its corresponding node's degrees in the graph, reflecting its interaction strength with other signals:
\begin{equation}
\mathbf{r}_j = \frac{1}{d_j} |\hat{\mathbf{x}}_i-\mathbf{x}_j|
\end{equation}
Here, $d_j$ represents the sum of the weighted in-degree and out-degree of node $j$, capturing the intensity of its interactions across the graph.
The final anomaly score, $s$, is calculated by averaging these topology-normalized reconstruction errors across all sensors and the entire sequence length, given by:
\begin{equation}
s = \frac{1}{N}\frac{1}{W}\sum_{j=1}^N\sum_{t_i=t_w}^t r_j^{t_i}
\end{equation}
Anomalies are then identified based on $s$, using a threshold determined empirically or via statistical analysis of the healthy validation set.
}
\section{Proposed Framework}
\label{sec:method}
\input{chaps/03-fw/03-1-overview}
\input{chaps/03-fw/03-2-dynamic-edge}
\input{chaps/03-fw/03-3-oc}
\input{chaps/03-fw/03-4-gnn}
\input{chaps/03-fw/03-5-recon}
\input{chaps/03-fw/03-6-loss}
\input{chaps/03-fw/03-7-ad_score}
\section{Case Studies}
\label{sec:case_study}
\revtextiot{
We evaluate DyEdgeGAT based on two case studies to assess its efficacy and robustness in fault detection within complex systems.
The first case study employs a synthetic dataset, developed due to the need to fully control faults characterized by relationship shifts. It provides a unique opportunity to access the ground truth of different fault severities and their fault onsets, which a real-world benchmark cannot provide.
The second case study utilizes a benchmark dataset from an industrial multiphase flow facility~\cite{stief2019pronto}. This dataset contains artificially induced faults that introduce changes in functional relationships within the system. It also includes various operating conditions, which allows us to assess the model's robustness against novel operating conditions.
We detail the description of each case study in the following sections.}
\subsection{Case Study 1: Synthetic Dataset}
We generated a synthetic dataset mimicking an interconnected system of control variables and measurement variables to study cause-and-effect relationships in this system, with the ground truth fault severity and fault onset available.
\subsubsection{Data Generation}
\label{sec:app_syn_gen}
We simulated a system with two sinusoidal control signals influencing five measurement signals, replicating complex sensor data through nonlinear trigonometric relationships. Gaussian noise with a signal-to-noise ratio of 35 ($\text{SNR}=35$) is added to approximate realistic conditions. Faults are introduced by modifying the input-output relationships at randomly selected data segments, reflecting potential real-world system-level faults. The time of the first point in the data segment is considered as the onset of the fault.
\subsubsection{Introducing Fault Severity Levels}
To evaluate model sensitivity to faults of varying severities in particular to early subtle faults, we modulate the input-output relationship with scaling factors from 0.5 to 2.0, reflecting a range of fault severities.
The scaling factors modulate the interdependencies between signals in our system model, selectively impacting a subset of measurements, and thereby modifying the overall dynamics of related signals. A scaling factor of 1 implies unchanged system dynamics, representing the standard operational state. Deviations from this value indicate increasingly significant changes in system dynamics, thereby increasing the fault severity. The more the scaling factor diverges from 1, the easier it becomes to detect faults, as differences in system dynamics become increasingly pronounced compared to normal behavior.
\revtextiot{Ratios of fault samples of each fault severity class are listed in Tab.~\ref{tab:fault_rate_toy}}. The impact of fault severity on system behavior and detection complexity is thoroughly evaluated and discussed in Sec.~\ref{sec:res_toy}.
\input{tabs/anomaly_rate}
% probably add some sample data to the appendix.
\subsection{Case Study 2: Industrial Dataset (Pronto)}
The Pronto dataset \cite{stief2019pronto} offers a benchmark for a multiphase flow facility, featuring various process variables such as pressures and flow rates. In our analysis, we selected 17 process variables sampled at a 1 Hz sampling rate, \revtextiot{as detailed in Table~\ref{tab:pronto_data}}. Due to inconsistencies in the dataset, specifically on test day five, we omitted two variables: water tank level (Ll101) and input water density (FT102-D).
Based on the system process scheme, we identify input air flow rate (FT305/302) and input water flow rate (FT102/104) as control variables, as well as input air temperature (FT305-T) and input water temperature (FT102-T) as external variables influencing the operating conditions, based on the dataset's description.
\textbf{Operating conditions}. The dataset contains 20 distinct flow conditions, defined by varying the input rates of air and water. These conditions can be divided into two major flow regimes: stable, referred to as \textit{normal}, and unstable, identified as \textit{slugging}.
The slugging regime contains eight unique flow conditions whereas the normal regime spreads over 12 flow conditions.
In the training phase, only data from normal operating conditions is used. In the test phase, data from the slugging regime is used to evaluate whether the model is robust under new operating conditions.
Fig.~\ref{fig:pronto_dist} illustrates the data distribution exemplarily for the process variable PIC501 (Air outlet valve opening degree in the 3-phase separator). A shift occurs during the slugging condition, which indicates a transition in the system's behavior, demonstrating the challenge of distinguishing faults from novel operating conditions.
\textbf{Fault types}. The dataset contains three types of faults—\textit{air leakage}, \textit{air blockage}, and \textit{diverted flow}, induced under two specific flow conditions within the normal flow regime. For comprehensive fault descriptions, please refer to \cite{stief2019pronto}. \revtextiot{Ratios of fault samples of each fault type are listed in Tab.~\ref{tab:fault_rate_pronto}.}
Figure~\ref{fig:pronto_tsne} presents a t-SNE visualization of the dataset, highlighting the differences in fault detection difficulties among the various fault types.
The visualization indicates that \textit{air leakage} is comparatively simpler to identify, as it forms a well-defined, separate cluster. This distinct clustering suggests a significant dissimilarity from normal conditions. In contrast, \textit{air blockage} and \textit{diverted flow} appear more challenging to detect due to the partial overlap of samples from these two fault types with the cluster representing \textit{normal} conditions.
Furthermore, as shown in Figure~\ref{fig:pronto_dist}, the air outlet valve process variable PIC501 exceeds its normal range between 5 and 35 \% during air leakage into a negative range. Therefore, detecting air leakage can be seen as a ``point anomaly'', making it easier to detect.
\input{tabs/pronto/pronto_data}
\begin{figure}[tbhp]
\centering
\centering
\subfloat[t-SNE embedding space]{
\includegraphics[width=.46\linewidth,valign=t]{figs/pronto/ponto_tsne.pdf}
\label{fig:pronto_tsne}
}
\hfill
\subfloat[Violin plot of PIC501 normalized by width.]{
\includegraphics[width=.48\linewidth,valign=t]{figs/pronto/air_valve_violin_square.pdf}
\label{fig:pronto_dist}
}
\caption{Pronto Dataset Fault Class Statistics: (a) t-SNE embedding space illustrating normal and faulty raw sequence data under two flow conditions.
(b) Violin plot showing the density distribution of the process variable air outlet valve 3-phase separator (PIC501).}
%\caption{t-SNE embedding space for normal and fault types of the raw sequence data under the same two flow conditions.}
\label{fig:pronto_stats}
\end{figure}
%\begin{figure}[tbhp]
%\centering
% \centering
% \includegraphics[width=.85\linewidth]{figs/pronto/air_valve_violin.pdf}
% \caption{Density distribution of the process variable air outlet valve 3-phase separator (PIC501) in the pronto dataset under different fault types.}
% \label{fig:pronto_dist}
%\end{figure}
%It's important to note that these scaling factors are not linearly proportional to the fault's impact on the system. Scaling factors close to 1 indicate a minimal change in the system behavior and are more challenging to detect; whereas factors farther from 1 result in noticeable system changes, simplifying the detection process.
% Additional details on the data distribution and anomaly rate may be included in the Appendix if necessary.
%Slugging is a deviation from standard flow conditions in manufacturing processes, characterized by lower than usual air and flow rates. Although this deviation can adversely impact process efficiency, it is not a fault from the perspective of the system behavior, but rather a novel operating condition.
%\subsection{Case Study 2: Industrial Dataset (Pronto)}
%The Pronto dataset \cite{stief2019pronto} is an industrial-scale multiphase flow facility benchmark.
% data overview, sampling rate
%It includes heterogeneous data sources sampled at different rates. We focus in this case study only on the process variables sampled at 1 Hz.
% what are the variables that are measured
%A total of 17 process variables were recorded in the dataset, covering a wide range of variables such as pressures, temperatures, densities, and flow rates, giving us the ability to examine changes in system conditions.
%Table~\ref{tab:pronto_data} in the Appendix lists the specific process variables used in our experiments.
% specifics
%Note that we excluded two process variables—water tank level (Ll101) and input water density (FT102-D). This decision was made because these two variables are missing in the data for test day 5 (test 11 with only normal data). As normal data is critically important for unsupervised anomaly detection and its absence could lead to data shortage, it was essential to remove these two process variables to maintain consistency in the variables used across all performed tests.
% faults and operating conditions
%The dataset was collected under 20 operating conditions, with three induced faults (\textit{air leakage}, \textit{air blockage}, \textit{diverted flow}) at two different operating conditions (A and B). Detailed information about the faults and operating conditions can be found in \cite{stief2019pronto}.
%It should be noted that the healthy data contains all 20 operating conditions while the induced faults were generated under fault conditions A and B only.
%To assess visually different degrees of difficulties of the fault types present in the dataset, we visualized the 2 components of t-SNE applied to the raw signal sequences containing both control and measurement variables.
%The resulting embedding space is depicted in Fig.~\ref{fig:pronto_tsne}.
%The analysis reveals that \textit{air leakage} is the most readily identifiable anomaly, with its embedding space exhibiting distinct separation from the normal data points. Conversely, \textit{diverted flow} proves more challenging to detect, as it exhibits a substantial overlap with the normal condition data points.
% TODO: add a table about the number of signals
% TODO: add severity if including it in the future.
%\subsubsection{Data Generation}
%\label{sec:app_syn_gen}
%The system is simulated using two sinusoidal input (control) signals to produce five output (measurement) signals. We model signals generated here represent heterogeneous sensor measurements.
%We define the relationships between these control and measurement variables through nonlinear functions such as a set of complex trigonometric functions. This methodology enables us to capture the inherent complexities in the system's behavior.
%Gaussian noise with a signal-to-noise ratio of 35 ($\text{SNR}=35$) is added to all signals to simulate real-world conditions.
% how fault is generated
%Faults are simulated by altering relationships between the input and output variables. These alterations are applied to random segments of the dataset and mimic the unexpected behaviors that might occur due to system faults in a real-world scenario.
%We assess model sensitivity to faults of varying severities by scaling the input-output relationships. These scaling factors range from 0.5 to 2.0, simulating different severities\footnote{Although reflecting fault ``severity'', these scaling factors are not linearly proportional to the fault's impact on the system. Scaling factors closer to 1 represent subtle system changes thus posing greater detection challenges while scaling factors further away from 1 greatly change system responses, facilitating anomaly detection. } of system anomalies. The scaling factoring is applied to several randomly selected nodes, and all functional relationships related to those selected nodes are scaled.
%The severity level's effect on the system's response and the associated challenge in anomaly detection is discussed in detail in Sec.~\ref{sec:result_severity}.
%\subsubsection{Introducing Fault Severity Levels}
%Relationship shifts in complex industrial systems can exhibit varied severities, where early detection is critical for timely interventions. To scrutinize the sensitivity of different models to these shifts, we simulate faults with a spectrum of severity levels, operationalized through scaling factors applied to the input-output relational dynamics. Although termed as 'severity', these scaling factors are not linearly proportional to the fault's impact on the system. A scaling factor close to 1 signifies subtle anomalies, akin to standard operational behavior, and presents significant detection challenges. Scaling factors diverging from 1, both below and above, correspond to more conspicuous system behaviors, facilitating easier anomaly recognition by models. The nuanced relationship between scaling factors and actual fault severity, and how it bears upon detection difficulty, is explored in depth in Sec.~\ref{sec:result_severity}.
%\subsubsection{Introducing Fault Severity Levels}
%Relationship shifts in complex industrial systems can exhibit varied severities, where early detection is critical for timely interventions. To scrutinize the sensitivity of different models to these shifts, we simulate faults with a spectrum of severity levels, operationalized through scaling factors applied to the input-output relational dynamics. Although termed as 'severity', these scaling factors are not linearly proportional to the fault's impact on the system. A scaling factor close to 1 signifies subtle anomalies, akin to standard operational behavior, and presents significant detection challenges. Scaling factors diverging from 1, both below and above, correspond to more conspicuous system behaviors, facilitating easier anomaly recognition by models. The nuanced relationship between scaling factors and actual fault severity, and how it bears upon detection difficulty, is explored in depth in Sec.~\ref{sec:result_severity}.
%\subsubsection{Introducing Fault Severity Levels}
% different fault severity levels
%To evaluate different models' ability to detect such subtle shifts, we induce faults across a spectrum of severity levels.
%These faults are introduced by scaling the relationships between the control and measurement variables. The scaling factor, referred to as the ``severity" of these faults, varies across a \revtext{range from 0.5 to 2.0}, representing the system’s intensity of the reaction to changes in the relationship shifts between the control variables and measurement variables.
%Detecting anomalies becomes particularly challenging when the scaling factor is close to 1, indicating normal system behavior, or less than 1, when the system responds only slightly to changes in the controlled variable. However, if the scaling factor exceeds 1, anomalies become more noticeable due to the increased response of the system, making them easier to identify by the detection models.
%The severity level's effect on the system's response and the associated challenge in anomaly detection is discussed in detail in Sec.~\ref{sec:result_severity}.
\subsection{Baselines}
\label{sec:doe_baselines}
\input{tabs/baselines_tab}
\revtextrewrite{
For a comprehensive evaluation, our proposed method is benchmarked against simple and state-of-the-art fault and anomaly detection methods, detailed in Table~\ref{tab:baseline_methods}.
These methods are classified based on their approach to modeling multivariate time series (MTS): \textit{relationship-focused} methods like Feedforward Neural Networks (FNN) and AutoEncoders (AE), \textit{dynamics-focused} methods including Long Short-Term Memory (LSTM), LSTM-based AutoEncoders (LSTM-AE), and UnSupervised Anomaly Detection (USAD), and \textit{graph-based} methods (GNNs).
}
\revtext{
In the \textit{graph-based} category, our comparison focuses on GNN-based AD methods tailored for IIoT, specifically those that simultaneously infer graphs from MTS.
The focus on AD stems from the lack of unsupervised FD methods based on GNNs in the IIoT context, to the best of our knowledge.
We selected Graph Deviation Network (GDN) and Multivariate Time-series Anomaly Detection with Graph Attention Network (MTAD-GAT) because they employ attention mechanisms in graph learning, which are similar to those proposed in our approach. Additionally, we included Graph Representation Learning for Anomaly Detection (GRELEN) due to its use of graph discrepancy, aligning with our approach in anomaly score construction.
%Our approach, DyEdgeGAT, is distinct; it dynamically constructs graphs per data segment to capture evolving relationships instead of a static graph per data sample. Contrasting with GRELEN's graph discrepancy anomaly score metric, DyEdgeGAT uses the aggregated temporal graph topology to regularize the anomaly score.
}
%In Table~\ref{tab:baseline_methods}, we summarize these baseline methods along with their model inputs and outputs.
%\revtext{We select GDN and MTADGAT as baselines because they utilize attention-based GNNs for MTS AD.
%We choose GRELEN for its unique approach to using learned graphs for AD. Our proposed method, \revtext{DyEdgeGAT}, also employs an attention mechanism, but it constructs a dynamic graph per input data instead of a static graph.
%Unlike GRELEN, which directly employs graph discrepancy as an anomaly score, \revtext{TimeGAT} leverages the aggregated temporal graph, which contains both node and edge dynamics, to enhance anomaly detection performance.}
\revtextiot{
\input{tabs/model_hyper}
Hyperparameter tuning was tailored to each dataset. Model selection was based on the validation loss.
% anomaly score evaluations
For anomaly score calculation, we employ GDN's scoring function across all baselines, except for USAD which retains its dual-reconstruction score. Scores are the normalized residuals between the observed and predicted values, with normalization parameters drawn from the validation set's median and interquartile range. We employed the mean aggregation instead of the original max aggregation to reflect system relationship changes better and apply time-averaging of scores for LSTM-AE, USAD, GRELEN, and MTADGAT.
Hyperparameters were fine-tuned using grid search across plausible values, as detailed in Tab.~\ref{table:hyperparameters}, with optimal settings indicated within parentheses. The first and second values correspond to the first and second case studies, respectively. $^*$ denotes a reference to the text description for further details. Selections were based on the best validation loss.
We varied the number of layers (L) and the dimensions of the hidden layers (H) and tested different normalization techniques, such as Batch Normalization (BN) and Layer Normalization (LN). Table~\ref{table:hyperparameters} also details model-specific parameters and optimal model sizes. For FNN and AE models, the input sequence window size is 1, while for all other models, it is consistently 15 in both case studies.
\begin{itemize}
\item \textbf{FNN}:
We varied the number of layers and their hidden dimensions. The same independent variables used for dyEdgeGAT serve as input, mapped to the measurement variables.
\item \textbf{AE}:
The latent dimension of AE is the size of system-independent variables. The optimal configuration was 20-10-5-2 for the synthetic study and 20-20-20-10-10-4 for the industrial case study.
\item \textbf{USAD}:
We adhered to the original structure but varied the AE's latent dimension, initial training warm-up epochs, and final activation function. Adding batch normalization layers was tested but found non-beneficial.
\item \textbf{GDN}:
We preserved most of the original hyperparameters from the paper, as modifications had minimal impact on results. Nonetheless, the model showed sensitivity to the decoder hidden layer dimension (tested from 100 to 200) and sensor embedding dimension (tested from 20 to 50). We experimented with varying top k values for graph sparsification and discovered that the GDN model achieved optimal performance as a fully connected graph on our task.
\item \textbf{MTADGAT}:
The attention module was upgraded to GATv2 for its enhanced expressivity. We tested the hidden dimensions for feature and temporal attention embeddings between 10 and 40, and for the reconstruction and forecasting modules, we varied them from 10 to 80.
\item \textbf{GRELEN}:
We adjusted the number of graphs from 1 to 4, adhering to similar graph distribution priors as in the original paper. The hidden dimension was varied between 10 and 40. Additionally, we evaluated different anomaly scoring methods for GRELEN, identifying that the point adjustment in the original implementation was not rigorous for evaluation, further detailed in Section~\ref{sec:res_toy}.
\item \textbf{DyEdgeGAT}:
We varied the hidden node dimension of the node between 10 and 20, the hidden edge temporal embedding dimension between 20, 40, and 100, hidden graph dimension between 20 and 40. The dimension of the temporal encoding is between 5 and 10.
\end{itemize}
}
\subsection{Evaluation Metrics}
\label{sec:doe_evaluation}
\revtextiot{For model performance assessment, we utilize a comprehensive set of metrics: AUC, F1 score, best F1, and best Detection Delay. In addition to these established metrics, we propose a novel metric to evaluate the model's ability in distinguishing novel operating conditions from faults.
The metrics employed are detailed as follows:}
\begin{itemize}
\item \textbf{AUC}: The area under the Receiver Operating Characteristic (ROC) curve, reflects discrimination capability over varying thresholds.
% \textbf{PR AUC}: Expresses the precision-recall trade-off.
\item \textbf{F1}: Average of precision and recall, determined by a threshold at the 95th percentile of normal validation anomaly scores.
\item \textbf{Best F1} (F1$^*$): The maximal F1 score obtained from the precision-recall curve.
% \textbf{F1\textsubscript{PA\%K}}: Adjusted F1 score, robust to K selection, as per Kim et al.~\cite{kim2022towards}.
\item \textbf{Best Detection Delay} (Delay$^*$): Measures the time taken to identify faults after their occurrence using the threshold of best F1.
\revtextiot{\item \textbf{Ambiguity Metric (Ambiguity)}: A novel metric defined as \( \text{Ambiguity} = 1 - 2 \cdot \left| \text{AUC} - 0.5 \right| \). It quantifies the model's inability to differentiate between normal operations and novel conditions, minimizing overfitting to specific patterns. A high ambiguity score indicates that the model effectively avoids mistaking novel operating conditions for faults. In contrast, a low score suggests that the model may struggle with this distinction, potentially leading to high false positive alarms in identifying faults. }
\end{itemize}
\revtextiot{AUC, best F1, and Delay$^*$, ambiguity are all threshold-independent and enable unbiased evaluation for any thresholding methods.
Specifically, the ambiguity metric is applied exclusively to the industrial dataset, as detailed in Section~\ref{sec:pronto_novel_oc}. This is due to the unique relevance of the ambiguity metric in contexts where novel operating conditions are present, which is not the case for the synthetic dataset.
}
%We prioritize early fault detection due to its significance in industrial settings, aligning with our datasets containing gradually evolving faults.
%For threshold-dependent metrics, we set the threshold at the 95th percentile of validation anomaly scores, utilizing healthy data for threshold determination to mimic real-world conditions because faulty data might not be pre-identified.
% Which metrics are more important?
%These metrics, especially AUC and F1, are established benchmarks in anomaly detection literature, offering a comprehensive view of model efficacy in both detection accuracy and timeliness.
%We employ six evaluation metrics:
%ROC AUC, PR AUC, F1 score, best F1 score, F1\textsubscript{PA\%K}, and Delay. ROC AUC, PR AUC, and best F1 are threshold-invariant.
%\textit{Area Under the Receiver Operating Characteristic Curve) (ROC AUC)} and \textit{Precision-Recall Area Under the Curve (PR AUC )}are threshold-invariant measures that assess the model's overall discrimination ability and the trade-off between precision and recall, respectively.
%ROC AUC, PR AUC, F1 score, best F1 score, F1\textsubscript{PA}\%K, best Delay.
%ROC AUC, PR AUC, and best F1 are threshold-invariant.
%ROC AUC measures the model's discrimination ability over varying thresholds. PR AUC evaluates precision-recall trade-offs. Best F1 score is derived from the precision-recall curve. F1\textsubscript{PA\%K} is a point-adjusted F1 score introduced by Kim et al.~\cite{kim2022towards}, insensitive to the choice of K. Delay measures the time lag in fault detection.
%AUC and F1 are common metrics adopted in the literature for anomaly detection tasks. In addition, we evaluate the detection delay because the fault is usually gradually developed. Early detection of faults can help operators to take necessary actions in time before drastic consequences occur. Both the synthetic dataset and the industrial dataset include increasing severity levels, which gives us a unique opportunity to study the performance of the model in early fault detection.
%\textit{Area Under the Curve (AUC)} is the area under the Receiver Operating Characteristic (ROC) curve. It measures the overall performance of the anomaly detection model over all choices of thresholds.
%\textit{F1 score} is a measure of the model's performance balancing between precision and recall.
%\textit{Delay} evaluates the model's response time in detecting an anomaly. It is defined as the difference between the time when a fault is induced and the time when the algorithm detects this fault. A fault is detected when for a user-defined length of a window, the prediction is contiguously positive.
% choice of threshold
%For both the F1 score and delay, the threshold is determined as 95\% percentile of the validation anomaly score. In real-world scenarios, faulty data are often not available at the time of training. Therefore, we opt to use only healthy data to determine the threshold rather than setting it as a hyperparameter.
\section{Design of Experiments}
\label{sec:exp_design}
\revtextiot{
In this section, we detail our experimental design to assess DyEdgeGAT's fault detection performance. Specifically, we introduce the baseline methods and their configurations in Sec.~\ref{sec:doe_baselines}, evaluation metrics in Sec.~\ref{sec:doe_evaluation}, training setups in Sec.~\ref{sec:doe_training} and experiment setups in Sec.~\ref{sec:doe_hardware}.}
%todo: address review 1's 3rd comment in the overview
%The first case study employs a synthetic dataset, developed due to the scarcity of suitable real-world benchmarks and the need to control for faults arising from relationship shifts.
%We evaluate the performance of DyEdgeGAT with two case studies:
%1) a synthetic case study for which we know the ground truth system health state and also allows us to evaluate different severity levels of faults
%and 2) an industrial case study demonstrating a real industrial Internet of Things scenario. This industrial case study involves faults of varying degrees of difficulty that impact the performance of the system.
%In the first case study, due to the inherent difficulty in sourcing real-world benchmark datasets primarily arising from dependency anomalies, we opted for the creation of a synthetic dataset. This approach grants us full control and flexibility to modify the functional relations.
%The synthetic dataset lets us evaluate our model's performance across various levels of fault severity.
%In the second case study, we utilize an industrial-scale multiphase flow facility benchmark dataset. This dataset contains manually induced faults that introduce changes in dependencies within the sensor data.
%In the following sections, we detail the design of each case study.
%\revtext{
%TODO: MOTIVATION NEEDS TO BE CHANGED
%Relationship shifts in complex systems can vary in severity, with early detection %being crucial for preventive interventions.
%}
%Our experiments aim to answer the following three research questions:
%\begin{itemize}
%\item Does the integration of graph representation learning with anomaly detection outperform traditional methods in handling multivariate time series data?
%\item How does the severity of the anomaly affect the performance of \revtext{TimeGAT}? How early can we detect faults for different severity levels of faults?
%\item How crucial are the individual components of \revtext{TimeGAT} - temporal graph, temporal encoding, control parameter augmentation, and topology-based anomaly score - in contributing to its overall anomaly detection performance?
%\end{itemize}
% motivation of the synthetic dataset
%\input{chaps/04-doe/04-1-casestudies}
\input{chaps/04-doe/04-2-baselines}
\input{chaps/04-doe/04-3-metric}
\subsection{Training}
\label{sec:doe_training}
All models were trained using the Adam optimizer with a learning rate of 1e-3, the training continued for a maximum of 300 epochs with early stopping at 150 epochs with a patience of 20 steps. Experiments were repeated 5 times with different initializations, and their mean and standard deviation were reported.
% cases study 1
In the first case study, conducted on a small synthetic dataset, a batch size of 64 and L1 loss were used, and no data normalization was needed as the generated data ranged from -1 to 1.
% case study 2
For the second case study, conducted on a larger dataset, a batch size of 256 was employed along with data standardization. Due to substantial differences in the statistical characteristics of the data distribution in the two case studies, L2 loss was used.
Additionally, ReduceLROnPlateau scheduling was applied to decrease the learning rate by a factor of 0.9 after 10 consecutive epochs of non-improving validation loss.
\subsection{Experimental Setup}
\label{sec:doe_hardware}
Our proposed method, its variants, and the baseline methods were all implemented using PyTorch 1.12.1~\cite{paszke2019pytorch} with CUDA 12.0 and the PyTorch Geometric 2.2.0~\cite{fey2019fast}.
For the synthetic dataset, the computations were performed on a server equipped with 4 NVIDIA RTX2080Ti graphic cards.
We used neptune.ai to track the experiments.
\revtext{For the industrial dataset, computations were performed on a GPU cluster equipped with NVIDIA A100 80GB GPUs.}
\subsection{Case Study I: Results on the Synthetic Dataset}
\label{sec:res_toy}
\begin{figure}[bthp]
\centering
\includegraphics[width=.85\linewidth]{figs/toy/fault_factor_vs_auc.pdf}%
\\
\hspace{-5mm}
\includegraphics[width=.78\linewidth]{figs/toy/fault_factor_vs_f1_best.pdf}%
\caption{Comparison of model performance across different scaling factors on the synthetic dataset. A scaling factor closer to 1 indicates lower fault severity. The shaded areas around the performance lines indicate variance in model performance across multiple runs.}
\label{fig:toy_severity}
\end{figure}
\input{tabs/toy/syn_res}
\revtextiot{
\textbf{Performance across severity levels.}} Fig.~\ref{fig:toy_severity} presents the performance of all compared algorithms in terms of the AUC and best F1 score across varying scaling factors. The results demonstrate that DyEdgeGAT has consistently high AUC and F1 scores across all scaling factors, particularly at factors below 1, where the system response is damped and the fault is more challenging to detect. All other compared methods can only effectively detect faults in high fault severity scenarios where the faults have manifested significantly in the measurement signals, leading to a substantial deviation from the signal's faulty state to its normal state. These results highlight that it is essential to capture the time-evolving relationships in order to capture subtle relationship shifts and enable early fault detection.
% Should I comment 1,1 instead of 0.95 being the hardest?
In scenarios where scaling factors are \textbf{approximately 1}, implying minimal changes in system dynamics, fault detection becomes particularly challenging. Methods that globally model relationships (such as FNN) or dynamics (such as LSTM) struggle to detect such faults. Conversely, graph-based methods, with their ability to capture pairwise relationships, are more effective in these cases. In these scenarios, DyEdgeGAT demonstrates its superiority in terms of both AUC and F1$^*$. Its performance is notably superior at scaling factors 0.9, 0.95, and 1.05, DyEdgeGAT outperforms the other models by far. In terms of AUC, MTADGAT performs as the second-best method across most severity levels, and GDN is the second-best method in terms of F1$^*$.
When the scaling factor \textbf{exceeds 1.5}, there is a significant amplification of system dynamics. This amplification facilitates detection by methods that track system-level temporal dynamics. In such a context, LSTM-AE can outperform DyEdgeGAT.
On the contrary, at scaling factors \textbf{below 0.75}, characterized by a damped system response, dynamic-focused methods such as LSTM and LSTM-AE become less effective. The reason is the reduced discrepancy between the predicted and true values compared to cases with amplified dynamics. Moreover, USAD, which aims to amplify anomalies, struggles to identify them in this scenario due to the declining nature of the signal magnitudes.
\revtextiot{
\textbf{Overall superior performance of DyEdgeGAT.}
Table~\ref{tab:syn_res} presents the aggregated results of DyEdgeGAT and the baseline models, averaged over five runs across varying fault severities.} DyEdgeGAT consistently outperforms other comparison methods in AUC, F1, and F1$^*$ scores, highlighting its efficacy in detecting functional relationship changes.
\revtextiot{\textbf{Dynamics focus vs. relationship-focused}.}
% functional vs spatial-temporal
LSTM and LSTM-AE generally outperform AE and FNN, indicating the importance of learning dynamic system relationships.
% fnn vs ae <- functional relationship modeling
Notably, FNN performs better than AE, which suggests that distinguishing control from measurement variables helps the model in learning system functionalities. This observation aligns with the recent findings of Hsu \textit{et al.} \cite{hsu2023comparison}.
% explanation of usad's underperformance
%\revtextiot{\textbf{Suboptimal performance of USAD}.}
The comparatively poor performance of USAD can be attributed to its focus on amplifying anomalies, which is less effective for detecting relationship shifts, as the magnitude of signals in faulty conditions does not significantly deviate from the normal state.
\revtextiot{\textbf{Graph-based}.}
% gnn
A further notable observation is that other graph-based models also demonstrate strong capabilities in signal relationship modeling. Notably, MTADGAT closely follows DyEdgeGAT in terms of the AUC score, while GDN ranks second in terms of the F1 score after DyEdgeGAT.
\revtextiot{
%\textbf{Suboptimal performance of GRELEN}.
GRELEN's underperformance, contrary to previous findings, is due to excluding the non-rigorous point adjustment step in its original anomaly score calculation, as outlined by Kim~\textit{et al.}~\cite{kim2022towards}.
To demonstrate the significant impact of point adjustment, we report the averaged AUC score of GRELEN with and without point adjustment in Tab.~\ref{tab:syn_grelen_result}.
}
% delay detection performance
\revtextiot{\textbf{Delay detection and F1 score performance}.}
DyEdgeGAT exhibits a moderately higher {Delay}$^*$, yet its superior F1$^*$ indicates more accurate timely detection. In practical applications, it is often preferred to have accurate fault detection over those that are early but are potentially false alarms.
% f1 score performance
%\revtextiot{\textbf{F1 score performance}.}
Furthermore, DyEdgeGAT achieves the highest F1 score, which is noteworthy considering that it is derived from a simple and straightforward thresholding method. In this approach, any score exceeding the 95th percentile of the validation set is considered indicative of a fault. Despite its simplicity, this method proves highly effective, enhancing DyEdgeGAT's suitability for real-world applications.
%its high F1 score demonstrates that straightforward thresholding is effective, enhancing its real-world applicability.}
\input{tabs/toy/grelen_syn_res}
% Non-graph based models (FNN, LSTM, AE) showed similar performance patterns across severity levels, with their detection capability improving as the severity level increased.
%Models that do not account for temporal dependencies, such as AE and FNN, generally underperform across most scenarios.
%LSTM, which focuses primarily on temporal dependencies but weakly models inter-dependencies, outperforms FNN and AE at lower severity levels but shows consistent performance overall.
%LSTM-AE, which effectively models both temporal and inter-dependencies, outperforms other non-graph-based methods. These results indicate that modeling either temporal or inter-dependencies alone is insufficient for detecting relationship shifts within the system.
% Graph base models
%Among graph-based models, MTADGAT and GDN demonstrate a better performance across different severity levels compared to non-graph-based models. MTADGAT ranks second in AUC for small scaling factors and even ranks first when the scaling factor exceeds 1.5. GDN ranks second in F1 across nearly all severity levels. These results affirm the effectiveness of leveraging graph structure in capturing complex dependencies between time series, thereby improving the detection of relationship shifts, particularly for low severity levels.
% remark why MTADGAT better than GDN
%Notably, MTADGAT consistently outperforms GDN at almost all severity levels in terms of AUC, which could be attributed to its joint prediction and reconstruction module. In contrast, GDN only incorporates a prediction module.
% remark ziczag line in F1
%An observation regarding the F1 score is that detecting faults with scaling factors further away from 1 can sometimes be more challenging and less accurate than those with scaling factors closer to 1. This could be attributed to the varying duration of faults within an interval during fault generation. When the scaling factor is at 0.5 or 1.5, the anomaly rate is significantly lower compared to neighboring points. Models with high false alarm rates may exhibit considerably lower F1 scores at those points.
% Among the graph-based models, both MTADGAT and GDN showed enhanced performance across different severity levels compared to the non-graph based models. These models effectively leveraged the graph structure to capture complex dependencies and correlations in the data, leading to higher AUC scores. Notably, MTADGAT outperformed GDN at almost all severity levels, possibly due to its joint temporal and feature attention mechanism, which enables it to focus on more relevant information when detecting anomalies.
%The proposed model, \revtext{TimeGAT}, exhibited remarkable performance across all severity levels, achieving the highest AUC score, especially at low severity levels.
%Fig.~\ref{fig:toy_severity} illustrates the performance of various anomaly detection models as a function of the scaling factor, which indicates different severity levels of anomalies in the synthetic dataset. A scaling factor closer to 1 implies higher fault severity.
%Fig.~\ref{fig:toy_severity} illustrates the performance (AUC and best F1 score) of all models across different severity levels. It can be observed that, for all models, the closer the scaling factor is to 1, the more challenging it becomes to detect the fault, which aligns with our expectations. Additionally, faults tend to be easier to detect when the scaling factor exceeds 1 compared to when it is below 1. This is possible because a scaling factor above 1 amplifies system responses, making anomalies more prominent, while a scaling factor below 1 makes anomalies more subtle.
%Regarding AUC, our proposed model, \revtext{TimeGAT}, consistently outperforms almost all models except in regions with scaling factors above 1.5. In those cases, it ranks as the second-best model, slightly underperforming compared to another GNN-based model, MTADGAT. Regarding the F1 score, our model consistently outperforms other models except when the scaling factor is 1.1.
%These results highlight the effectiveness of \revtext{TimeGAT} in modeling time-evolving inter-dependencies, enhancing its capability to detect even subtle anomalies through its dynamic graph attention mechanism.
%As the scaling factor approaches 1, detecting faults becomes more difficult with minimal system response changes.
%Conversely, detectability improves when the scaling factor diverges from 1. With factors above 1, amplified system responses make anomalies more pronounced, while factors below 1 subdue responses, yet anomalies remain noticeably detectable.
%As the scaling factor approaches 1, the difficulty in fault detection increases with minimal system response changes.
%Conversely, detectability improves when the scaling factor diverges from 1.
%With factors above 1, amplified system responses make anomalies more pronounced, while factors below 1 subdue responses, yet anomalies remain noticeably detectable.
%TimeGAT exhibits a consistently high AUC and F1$^*$ across varying scaling factors, especially in the regions around 1, where it is the hardest to detect faults.
%These results highlight the effectiveness and robustness of TimeGAT in modeling time-evolving relationships, enhancing its capability to detect even subtle anomalies through its dynamic graph attention mechanism.
%showing particular resilience in its performance even as system response gets subdued (scaling factor $<$ 1). This suggests that TimeGAT is adept at detecting anomalies across a broad range of severities.
%TimeGAT's performance is exceptional in AUC over most scaling factors and maintains the lead in the best F1 score, demonstrating its robust detection capabilities, particularly for subtle anomalies. Its slight underperformance in high severity scenarios (scaling factor > 1.5) where MTADGAT excels suggests room for further optimization in highly anomalous environments.
%\textbf{Close to 1.} As the scaling factor approaches 1, the system dynamics barely change, making it difficult for methods that globally model relationships or dynamics to detect such faults. In these cases, graph-based methods with their ability to capture pairwise relationships can better capture the relationship shifts. We can see that for scaling factors between 0.95 and 1.1, MTADGAT is the second-best in AUC and GDN is the second-best in F1$^*$.
%TimeGAT exhibited a consistently high AUC and F1$^*$ across varying scaling factors, especially in the regions below 1, where the system response is subdued and harder to detect. These results highlight the effectiveness and robustness of TimeGAT in modeling time-evolving relationships, enhancing its capability to detect even subtle anomalies through its dynamic graph attention mechanism.
%Conversely, detectability improves when the scaling factor diverges from 1. With factors above 1, amplified system responses make anomalies more pronounced, while factors below 1 subdue responses, yet anomalies remain noticeably detectable.
\subsection{Ablation Study and Sensitivity Analysis}
\label{sec:res_ablation_and_sensititiy}
\subsubsection{Ablation Study}
\label{sec:res_ablation}
\revtextrewrite{
To understand how each component of our proposed methodology contributes to the overall fault detection performance, we conducted an ablation study on the synthetic dataset with five setups:
\begin{itemize}
\item \textbf{w/o oc aug}: Omitting the augmentation of operating condition context, considering each node in the temporal graph equally.
\item \textbf{w/o dyn. graph}: Removing dynamic edge construction, opting for static graph construction via MTADGAT's feature attention and GDN's top-k mechanism.
\item \textbf{w/o reverse}: \revtext{Skipping signal reversion and training the decoder to reconstruct the original signal.}
\item \textbf{w/o time}: Omitting temporal encoding, potentially weakening temporal dependency capturing.
\item \textbf{w/o topl}: Eliminating topology-based anomaly score, assigning uniform weights to all node residuals.
\end{itemize}
}
\input{tabs/toy/syn_ablation_results}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.75\linewidth]{figs/toy/ablation_auc.pdf}
\caption{Ablation study showing the AUC of the DyEdgeGAT model on a synthetic dataset with varying fault severity, denoted by the scaling factor. Each curve indicates the model's robustness to the exclusion of specific features, such as dynamic edge construction, operating condition context augmentation, reversed signal reconstruction, temporal encoding, and topology-based anomaly score.}
\label{fig:syn_ablation}
\end{figure}
% ablation overview
\revtextrewrite{
The results of the ablation study, outlined in Table~\ref{tab:syn_ablation} and visualized by Fig.~\ref{fig:syn_ablation}, demonstrate a performance degradation with the removal of any component from the proposed DyEdgeGAT algorithm. Notably, the elimination of topology-based anomaly scoring, signal reversion reconstruction, and temporal encoding results in a modest reduction in performance metrics.
The topology-based anomaly score, while generally exerting a minor influence on the model, becomes more relevant when the scaling factor approaches 1. This increase in influence can be attributed to its mechanism of normalizing anomaly scores by the strength of signal dynamics, proving valuable for detecting subtle shifts in relationships.
In contrast, excluding operating conditions (OC) context augmentation and dynamic edge construction leads to a significant decrease in both AUC and F1$^*$ scores. The impact of OC context augmentation is particularly noticeable when system dynamics undergo substantial changes, as observed at scaling factors near 0.5 or 2.0. Conversely, the importance of the proposed dynamic edge is more pronounced in detecting subtle changes in system dynamics at lower scaling factors, becoming most critical at a scaling factor of 0.95.
}
%\revtextrewrite{
% aug-control
%\textbf{OC context augmentation} stands out as the most important component of DyEdgeGAT, particularly noticeable when system dynamics undergo significant changes, as observed at scaling factors near 0.5 or 2.0. This augmentation provides essential contextual information crucial for understanding the evolution of system dynamics.
% dynamic edge
%\textbf{Dynamic edge} is the second most important component of the proposed algorithm, with its impact becoming more pronounced when the scaling factor is small (at a scaling factor of 0.95, it becomes the most crucial one). This underscores the significance of capturing the evolving relationships between time series.}
% reverse reconstruction
%\revtext{\textbf{Reversed signal reconstruction} consistently demonstrates a moderate improvement across all severity levels.
%The two-level signal reversion, applied to both the operating conditions signals and the reconstructed signals, facilitates a more symmetric reconstruction. This symmetry aids the model in learning dependencies between the signals within the system. }
% temporal encoding
%\textbf{Temporal encoding} exhibits a relatively modest impact on AUC but becomes more critical for the F1$^*$ score. This suggests that the attention mechanism alone is insufficient for the model to grasp the temporal ordering of the time series.
% ad score
%\revtext{\textbf{Topology-based anomaly score} has a minor overall impact on the model but becomes more influential when the scaling factor approaches 1. This is due to its normalization of the anomaly score by the strength of signal dynamics, proving valuable for detecting subtle shifts in relationships.}
% summary
%In summary, each component of DyEdgeGAT contributes to improving the model's performance.
%Table~\ref{tab:syn_ablation} presents the average performance across all severity levels for the ablation study conducted on synthetic datasets. Fig.~\ref{fig:syn_ablation} further illustrates the model's performance across all severity levels.
%The results indicate that removing any component adversely affects the model's performance. Specifically, discarding the topology-based anomaly score, reconstructing reverted signal, and temporal encoding results in a minor decrease in performance, whereas removing operating condition context augmentation and dynamic edge construction leads to a substantial drop in both AUC and F1$^*$ scores.
% overall
%This suggests that every component in the proposed model enhances its performance, with the construction of the dynamic graph and the operating condition augmentation contributing the most in terms of both AUC and F1$^*$. In addition, by removing these two components, the variance of the AUC prediction also got larger.
%The ablation study highlights the significance of each component in the proposed TimeGAT model. Notably, dynamic graph construction and operating condition context augmentation are essential, contributing significantly to AUC and F1$^*$ improvements. The removal of these components not only lowers performance metrics but also increases the variance in AUC predictions.}
%The augmentation of the operating condition is the most important component of TimeGAT. The control variables give the context of how the dynamics should evolve, this is especially valuable when the system dynamics are changed dramatically, such as when the scaling factor is close to 0.5 or 2.0.
\subsubsection{Sensitivity Analysis of the Sliding Window Size}
\label{sec:res_sensitivity}
\revtext{
The sliding window size $\delta t$ is a critical parameter in the DyEdgeGAT algorithm, influencing the construction of dynamic edges and the extraction of edge dynamic features. Our sensitivity analysis, summarized in Tab.~\ref{tab:syn_sensitivity} and illustrated in Fig.~\ref{fig:toy-sensitivity}, demonstrates the effect of $\delta t$ on the algorithm's fault detection performance.
The analysis shows that DyEdgeGAT's performance is robust across different sliding window sizes, exhibiting competitive performance even at suboptimal choices of $\delta t$ compared to optimized baselines.
The sliding window size $\delta t$ determines the granularity of temporal information encoded into the graph structure. With an input sequence length of 15 and a GRU-based edge encoder in Eq.~\ref{eq:edge_gru}, it is essential to find a balance between local temporal resolution and the preservation of sufficient data points to capture the evolution of dynamics. A small $\delta t$ (e.g., 1) may compromise temporal resolution, whereas a too large $\delta t$ (e.g. 7) may not offer sufficient context for the GRU to capture meaningful dynamics, potentially resulting in poorer performance.
Consequently, $\delta t=5$ emerges as the optimal sliding window size. This setting allows the model to effectively capture a comprehensive range of temporal data, ensuring both detailed temporal resolution and a thorough representation of dynamic evolution.
%suggesting that capturing a wider sliding window allows the model to infer valuable information.
%analogous to the first and second derivatives of the signal, similar to finite difference methods in numerical analysis. With three points, one can calculate the first derivative at the center point, and with five points, one can calculate the second derivative. These high-order temporal features help the DyEdgeGAT model to detect anomalies more effectively, even when they are subtle.
}
\input{tabs/toy/sensitivity_results}
\begin{figure}[tbhp]
\centering
\includegraphics[width=.75\linewidth]{figs/toy/sensitity_tw_auc.pdf}
\caption{Sensitivity analysis of DyEdgeGAT performance, measured by ROC-AUC, over a range of scaling factors for different sliding window sizes ($\delta t$). }
\label{fig:toy-sensitivity}
\end{figure}
\subsection{Case Study II: Results on the Industrial Dataset }
\subsubsection{Fault Detection Performance Across Various Fault Types}
\label{sec:res_pronto}
\begin{figure}[tbhp]
\centering
\includegraphics[width=.85\linewidth]{figs/pronto/pronto_auc.pdf}\\
\hspace{-0.4cm}
\includegraphics[width=.84\linewidth]{figs/pronto/pronto_delay_best.pdf}
\caption{Comparison of fault detection performance on the Pronto Dataset under different fault types.}
\label{fig:pronto-result}
\end{figure}
\input{tabs/pronto/pronto_res}
We evaluate the performance of DyEdgeGAT on the Pronto dataset on three distinct fault types: air leakage, air blockage, and diverted flow, and compare it to the baseline methods. Figure~\ref{fig:pronto-result} shows the AUC scores for each fault type, as well as the average performance across all fault types. Notably, DyEdgeGAT is the only method that demonstrates consistently strong performance across all three fault types in terms of both the AUC score and the detection delay, highlighting its superior ability to detect relationship shifts for fault detection.
\revtext{
\textbf{Air Leakage.} Characterized by cyclic behavior in the multiphase flow facility, air leakage creates a noticeable deviation from normal patterns, making its detection relatively straightforward. DyEdgeGAT, though not the best performer on this fault type, still demonstrates a competitive performance of AUC 0.81 and a detection delay of 59s.
\textbf{Air Blockage.} On this fault type, DyEdgeGAT outperforms all baselines, achieving an AUC of 0.847. This score outperforms the second-best model, AE, by a notable margin of 0.119. Furthermore, DyEdgeGAT achieves a very low detection delay of 62s, well below AE's 1777s. The nature of continuous flow in air blockages without obvious pressure drops makes it a more complex fault to detect. The poor performance of FNN with an AUC of 0.631 in this fault, despite its effectiveness in detecting other faults, indicates that it is not only important to model the functional relationships but also to track their evolution over time to enable accurate fault detection.
\textbf{Diverted Flow.} This fault presents the most challenge with an average AUC of 0.543, with only DyEdgeGAT performing adequately in both an AUC score of 0.748 and a detection delay of 62s. Diverted flow leads to noticeable shifts in functional relationships within the system, which explains FNN's high AUC score. The inability of the dynamics-focused method to detect this type of fault suggests that the signals remain within their normal range and exhibit temporal patterns that resemble those observed under normal conditions. The drop in performance of graph-based models may be attributed to their approach of treating all signal nodes uniformly, ignoring the cause-and-effect relationships in the systems, which reveals the disadvantages of a homogeneous node treatment of all signals.}
%In conclusion, DyEdgeGAT consistently outperforms baseline models in detecting a variety of faults, demonstrating its potential for broad applicability in complex industrial systems.}
% overall performance
\revtextrewrite{
To provide a broader perspective, we now evaluate the average performance of all models on all fault types with two additional metrics, F1 and F1$^*$. Tab.~\ref{tab:pronto_results} reveals that DyEdgeGAT on average outperforms all other compared models in terms of AUC, F1, and F1$^*$. Additionally, it exhibits constantly low detection delay across all fault types with a small variance, indicating its capability for timely fault detection. The following two notable observations emerge from the overall performance:}
\revtext{% explain why fnn/ae is better
\textbf{Importance of functional relationship modeling}. Firstly, models that prioritize functional relationship modeling, such as FNN and AE, demonstrate better performance than those focusing on temporal dynamics like LSTM. This suggests that the changes in functional relationships within the system's variables are more pronounced than changes in their dynamics for the Pronto dataset, where faults were incrementally introduced and the fault severity was gradually increased.
% explain why gnn performs poorly
\textbf{Suboptimal performance of GNN models}. Secondly, graph-based models such as GDN and GRELEN exhibit suboptimal performance on the fault detection task across all fault types in the Pronto case study. This is likely due to their inability to distinguish between system-dependent and system-independent variables. In the Pronto dataset, this differentiation is crucial because system-independent variables remain largely unaffected by faults and should not be modeled as system-dependent variables. Without separating system-independent variables, GNN-based methods are more susceptible to mistaking novel operating conditions as faults. The impact of novel operating conditions on the model's fault detection performance is analyzed in the subsequent Sec.~\ref{sec:pronto_novel_oc}.
}
% overall
%The overall superior performance of, DyEdgeGAT on the Pronto dataset demonstrates its effectiveness in detecting real-world faults, which often involve complex scenarios and gradual changes in functional relationships between system variables.}
%Air leakage is the easiest fault to detect, followed by air blockage and diverted flow. This observation is further supported by the performance of all models with relatively high AUC scores for air leakage but relatively low AUC scores for the other two fault types.
%\textbf{Air Leakage.} Air leakage induces a cyclic behavior in the multiphase flow facility~\cite{stief2019pronto}, creating a distinct pattern that deviates significantly from the normal behavior. As shown in Figure~\ref{fig:pronto_dist}, the process variable PIC501 exceeds its normal range during air leakage. Therefore, detecting air leakage can be seen as a ``point anomaly'', making it easier to detect. Even though DyEdgeGAT does not perform the best on air leakage detection, it still demonstrates competitive performance.
%\revtext{
%\textbf{Air Blockage.} For this fault type, DyEdgeGAT outperforms all other baselines by a significant margin. Unlike the air leakage case, air blockage does not cause a reduction in pressure or a drop in the system. The flow remains continuous, making this fault type more challenging to detect.
%In this class, FNN performs poorly, suggesting that modeling the static functional relationship alone is not sufficient. It is important to track the evolution of these relationships over time.}
%\revtextrewrite{
%\textbf{Diverted Flow.}
%Diverted Flow. This fault type is the most challenging for all baselines to detect, with all methods underperforming except for FNN and DyEdgeGAT. Graph-based models perform particularly poorly on this task.
%Diverted flow induces noticeable changes in the flow regime within the multiphase flow facilities, leading to significant shifts in the functional relationships within the system. This explains why FNN performs well, as it can model these functional relationships.
%The inability of other methods, even those that capture both temporal dynamics and spatial interdependencies, to detect this fault type suggests that the time series remain within their normal range and exhibit typical temporal patterns. This further emphasizes the need to model and capture changes in functional relationships within the system.
%The significant drop in the performance of graph-based models on this task demonstrates the limitation of treating all variables as the same type of nodes in modeling relationships. This highlights the need to distinguish between variable types and incorporate operating condition contexts.
% only interdependencies
%In the case of models that primarily capture interdependencies, AE outperforms FNN in the industrial dataset case study, achieving a higher AUC score, despite the two models' comparable performance in the synthetic case study.
%This performance difference compared to the synthetic case study is likely attributed to the increased complexity of modeling dependencies between control variables and measurement variables in real-world scenarios.
%AE, with its capacity to capture complex and nonlinear interdependencies through a low-dimensional bottleneck representation, is better suited to handle this complexity compared to the simpler feedforward architecture of FNN.
% LSTM's high F1
%Interestingly, most methods exhibit relatively low performance in terms of the F1 score and delay compared to their performance in the AUC score, with the notable exception of LSTM. LSTM, which primarily focuses on modeling temporal dependencies, demonstrates superior performance in these metrics.
%The superior performance of LSTM may be attributed to the employed thresholding method for the anomaly score. This method seemingly favors techniques that exhibit a consistent range of anomaly scores for healthy data during testing, compared to the validation anomaly score.
% GDN performs poor
%Surprisingly, GDN demonstrates the poorest performance across all metrics.
%This outcome could likely be attributed to GDN's dependency on a static graph.
%The graph constructed by GDN is global and remains unchanged for all sequences.
%However, during test time, when the relationships between time series change due to faults, the static graph, derived from healthy training data, becomes inadequate for accurate forecasting and constructing a reliable prediction-based anomaly score. It is worth noting that GDN was initially designed to detect point anomalies, where the functional relationships within the system remain stable, making a static graph beneficial in such scenarios.
%In contrast, MTADGAT constructs graphs based on each input sample, thereby overcoming this limitation. This highlights the crucial significance of dynamic graph construction in effectively capturing shifts in functional relationships.
%Table~\ref{tab:pronto_results} presents the performance comparison on the real-world industrial Pronto dataset. The results demonstrate that \revtext{TimeGAT} outperforms all other models on all three key metrics, with a slight exception in delay where it closely ranks as the second best.
%These results confirm the superiority of \revtext{TimeGAT} in detecting collective anomalies in complex systems, showcasing its robustness and effectiveness on complex real-world industrial data.
%Notably, \revtext{TimeGAT} significantly outperforms the second-best model, MTADGAT and LSTM-AE, in terms of delay, highlighting its ability to promptly detect anomalies while maintaining high precision and recall.
%An intriguing observation from the results is the superior performance of models that emphasize functional relationship modeling, such as FNN and AE, compared to those focusing on temporal dynamics like LSTM. This suggests that for the Pronto dataset, in which faults were introduced gradually, the alteration of functional relationships within the system's variables is more pronounced than changes in their dynamics. It implies that TimeGAT's ability to model these changing relationships effectively contributes to its superior performance in this real-world scenario.
%The performance comparison presented in Table~\ref{tab:pronto_results} on the Pronto dataset shows that TimeGAT outperforms baseline models across all three principal metrics, including AUC, F1-score, and F1$^*$. In addition, it has the second-lowest detection delay. This suggests that TimeGAT's ability to model both temporal and functional relationships within the system is effective for detecting real-world faults, which are often complex and can impact multiple system variables in subtle ways.
%Graph-based models such as GDN and GRELEN exhibit suboptimal performance compared to other baselines. This underperformance can be attributed to the models' inability to differentiate between system-dependent and system-independent variables. In the context of the Pronto dataset, this distinction is essential because system-independent variables remain unaffected by faults and therefore should not be modeled in the same way as system-dependent variables.
%Graph-based models inherently capture pairwise relationships, which can be a powerful approach for many datasets. However, in instances where faults are introduced gradually, causing a shift in the functional relationships within the system, these models may struggle. By treating all relationships equally, graph-based models may not effectively discern the subtleties between changes due to novel operational context and those due to actual faults. This lack of distinction can lead to a diluted representation of anomalies, making it more challenging to accurately identify faults.
%In contrast, TimeGAT's architecture is designed to address this shortcoming by incorporating operating condition context. This allows TimeGAT to more precisely isolate the effects of induced faults from those of operating condition changes, leading to its enhanced performance on the Pronto dataset. We discuss the impact of the choice of variable types in Sec.~\ref{sec:pronto-vartype}.
%We demonstrate the effect of novel operating conditions on model's fault detection performance in Sec.~\ref{sec:pronto-novel-oc}.
%\revtext{
% overall
%Generally, methods that effectively model functional relationships within the system achieve higher AUC scores, which is different from the findings from the synthetic case study. This is likely because the Pronto dataset contains real induced faults, which are much more complex and can impact all other signals, whereas in the synthetic dataset, the dependencies between variables are much simpler.}
\subsubsection{Performance on Novel Operating Conditions}
\label{sec:pronto_novel_oc}
%\input{tabs/pronto/pronto_condition_all}
\revtext{
In IIoT systems, fault detection models must be robust to novel operating conditions to reduce false alarms.
We assess this aspect on the novel operating condition of ``slugging'' in the Pronto dataset.
%To better evaluate the model's ability to distinguish novel operating conditions from faults, we propose a new metric, the ambiguity metric.
%An AUC score of 0.5 indicates that a model's performance is no better than random guessing when distinguishing between two classes. Building upon this, we define the ambiguity metric as:
%\begin{equation}
%\text{Ambiguity} = 1 - 2 \cdot \left| \text{AUC} - 0.5 \right|
%\end{equation}
%This metric quantifies the model's inability to distinguish between normal operation and novel operating conditions, without overfitting to specific operational patterns.
%A high ambiguity score indicates that the model effectively avoids mistaking novel operating conditions for faults. In contrast, a low score suggests that the model may struggle with this distinction, potentially leading to high false positive alarms in identifying faults.
Fig.~\ref{fig:pronto_ambiguity} illustrates that our DyEdgeGAT model achieves a relatively high ambiguity score. This indicates that DyEdgeGAT demonstrates robustness and generalizability, accurately distinguishing novel operating conditions from faults.
In contrast, other models with good fault detection performance on the pronto dataset, such as AE, FNN, and LSTM-AE, have significantly lower ambiguity scores.
A change in operating conditions can result in dynamics that are significantly different from those observed, even if the underlying functional relationships remain the same.
This indicates that these models are highly sensitive to changes in operating conditions, leading to misclassification of new system dynamics induced by novel operating conditions as faults.
Conversely, the high ambiguity scores of USAD and GRELEN stem from their overall underperformance in fault detection. These models struggle not only to differentiate between novel operating conditions and faults but also fail to detect faults effectively. This inefficiency is particularly highlighted by their low AUC scores in identifying air blockage and diverted flow.
}
\begin{figure}[htbp]
\centering
\includegraphics[width=.8\linewidth]{figs/pronto/pronto_ambiguity.pdf}
\caption{
Comparative model evaluation on the Pronto dataset under novel operating conditions, specifically for the slugging condition. The ambiguity metric reflects the model's disability to distinguish between normal and novel operating conditions.}
\label{fig:pronto_ambiguity}
\end{figure}
%pronto contains another fault class slugging, which is actually a test of novel flow conditions. The new flow conditions are lower than its normal range, since its behaviour is undesired in the manufacturing process, it is seen as a type of fault. but from the pure system behaviour point of view, it is simply a novel operating condition, thus we treat this type of fault as normal with novel operating condition instead of fault.
%A critical test for our proposed model, DyGATAD, is its performance under such conditions, which we exemplify using the 'slugging' fault class present in the PRONTO dataset. Slugging represents a deviation from standard flow conditions, characterized by lower than usual flow rates. Although this deviation is undesirable in a manufacturing context and typically classified as a fault due to its adverse impact on process efficiency, it is, from a systems behavior perspective, merely a novel operating condition.
%In IIoT systems, the robustness of fault detection models against novel operating conditions is essential to reduce false alarms.
%Our TimeGAT model's robustness to novel operating conditions is demonstrated through its performance on the \textit{slugging} class in the PRONTO dataset. This class represents a deviation from standard flow conditions, characterized by lower than usual air and flow rates. Although this deviation is undesirable in a manufacturing context and typically classified as a fault due to its adverse impact on process efficiency, it is, from a systems behavior perspective, merely a novel operating condition
%We thus categorize slugging not as a fault but as a novel operational condition.
%To evaluate model's robustness under novel operating conditions, we introduce a new metric.
%The ambiguity metric is introduced to quantify the robustness of a model under novel operating conditions, particularly ensuring that the model does not mistakenly classify these conditions as anomalies. It is defined as:
%This condition is ideal for scenarios where the distinction between normal operation and anomalies is not desired, thus a high ambiguity score is sought to denote a model that maintains robustness by not overfitting to specific operational patterns which may not be true anomalies.
%The AUC measures the model's ability to distinguish between classes; a value of 0.5 indicates no discriminative power, equivalent to random guessing.
%The ambiguity metric reaches its maximum value of 1 when the \(\text{AUC}\) is exactly 0.5, embodying perfect non-discrimination.
\subsubsection{Impact of Separating System-Independent Variables}
\label{sec:pronto_vartype}
%\input{tabs/pronto/oc_ablation}
\begin{figure}[tbhp]
\centering
\includegraphics[width=.8\linewidth]{figs/pronto/pronto_sensitivity_auc.pdf}
\caption{AUC scores for DyEdgeGAT with different combinations of system-independent variables across various fault types. The variables include Measurement Variables (MV), Control Variables (CV), and External Factors (EF).}
\label{fig:pronto-vartype}
\end{figure}
\revtext{
In the pronto dataset description, only control variables are used to define an operating condition~\cite{stief2019pronto}. This ablation study aims to demonstrate the importance of incorporating external factors for a more comprehensive description of operating conditions, as reflected in the performance improvement of DyEdgeGAT shown in Fig.~\ref{fig:pronto-vartype}.
We categorize input air and water flow rates as Control Variables (CV) and their corresponding temperatures as External Factors (EF), with the remaining variables classified as Measurement Variables (MV).
The performance of DyEdgeGAT varies depending on how system-independent variables are incorporated into the model. The best performance is achieved when CV and EF are modeled as operating condition contexts to augment the extraction of node dynamics, with MV modeled as system-dependent variables (MV aug. CV+EF).
This highlights that CV alone is not enough to represent the operating condition. When EF is not considered as part of the operating condition but treated as a system-dependent variable, the model's performance is even worse than not augmenting the operating condition context at all.
The MV+CV+EF configuration treats all variables in the same way, as commonly employed by other graph-based methods. Even under this configuration, DyEdgeGAT outperforms all other GNN-based methods with discrete time dynamic graphs on the air blockage and diverted flow fault types, suggesting that the aggregated dynamic graph representation is superior for detecting relationship shifts.
Another notable observation is that the MV-only configuration performs on par with MV aug. CV+EF. The key distinction between these two configurations is the inclusion of operating condition context in the node dynamics extraction for the MV aug. CV+EF.
This implies that the dynamic edge module alone, even without the augmentation of operating conditions in the node dynamics, is effective in capturing temporal relationships.
In conclusion, separating system-independent from system-independent variables is crucial for accurate fault detection. Treating them in the same way can degrade model performance even more than using only a subset of them. It is not sufficient to consider only control variables; rather, it is crucial to identify the external factors associated with them.}
%In addition, the model's performance varies across different anomaly types. For example, treating CV and EF as operating condition contexts appears to be beneficial for detecting air blockage and diverted flow anomalies, but less so for air leakage.
%The MV+CV+EF configuration treats all signals equally, which is the same as other graph-based methods, even DyEdgeGAT outperforms all other GNN-based methods with discrete time dynamic graphs on the air blockage and diverted flow fault types, suggesting the superiority of aggregated dynamic graph representation in detecting relationship shifts.
% It considers three types of variables: Control Variables (CV) and External Factors (EF) with Measurement Variables (MV) under different types of modeling strategies.
%The "MV+CV+EF" approach, where all variables are treated uniformly, does not significantly outperform the strategies where system-dependent and system-independent variables are distinguished. Specifically, the "MV+EF aug. CV" strategy, which models MV and EF as system-dependent and CV as system-independent in consideration of operating conditions, does not consistently yield better results compared to when CV and EF are both used as contextual indicators of operating conditions, as in "MV aug. CV+EF".
%The integration generally enhances the model’s predictive accuracy. Particularly, the combination of MV with EF augments the model's ability to discern anomalies, suggesting that external factors play a significant role in fault manifestation. However, the model's performance does not uniformly improve with the inclusion of CV, indicating that the relevance of control variables may be context-dependent. The variable impact across anomaly types suggests that an 'operating condition aware' approach is beneficial, as system-independent variables can induce patterns in measurement variables that are not necessarily indicative of faults. The varying performance also underscores the need for tailored feature selection in developing models for industrial IoT systems, ensuring robustness against false positives due to operational variances.
%"MV+EF aug. CV" shows that MV and EV are modeled as system-dependent variables and CV as system-independent variables for the operating condition context. Similarly "MV aug CV+EF" shows that both CV and EF are used as operating condition context.
%In the pronto dataset, we consider the input air and water flow as CV and their temperature as EF, the rest as MV.
%"MV+CV+EF" shows all variables are treated in the same way.
%It can be shown in the figure that "MV aug CV+EF" shows an overall best performance, indicating that it is important to include EF for operation condition context augmentation, Interestingly, when EF is not considered as part of the operating condition but treated as a system-dependent variable, its performance is even worse than not augmenting the operating condition context at all. Another interesting observation that excluding operating condition context and only models the relationship between MV performs on par with "MV aug CV+EF", which shows the effectiveness of the dynamic edge module.
%Interestingly, the model performs variably across different anomaly types, with some configurations showing improved performance in certain cases but not others. For instance, treating CV and EF as operating condition contexts appears beneficial for detecting 'Air blockage' and 'Diverted flow' anomalies, but less so for 'Air leakage'. This nuanced performance underscores the importance of contextual variable selection in model training and suggests that the detection efficacy of TimeGAT can be influenced by the interplay between variable types and the nature of the anomalies.
%This suggests that it is important to consider the interplay between variable types and the nature of the anomalies when selecting variables for contextual modeling.
%The MV aug CV+EF configuration shows the best overall performance,
\section{Results and Discussions}
\label{sec:results}
\revtextiot{
This section evaluates the model's performance for the case studies outlined in Sec.~\ref{sec:case_study}.
The baseline methods, their configurations, the evaluation metrics, and training setups were introduced in Sec.~\ref{sec:exp_design}.
Sec.~\ref{sec:res_toy} analyzes the model's performance on the synthetic dataset across various severity levels.
In contrast, Sec.~\ref{sec:res_pronto} focuses on the industrial dataset, examining performance across different fault types and the ability to differentiate novel operating conditions from faults (detailed in Sec.~\ref{sec:pronto_novel_oc}).
Additionally, Sec.~\ref{sec:res_ablation_and_sensititiy} conducts an in-depth examination of the DyEdgeGAT model, including an ablation study (Sec.~\ref{sec:res_ablation}), sensitivity analysis to sliding window size variations (Sec.~\ref{sec:res_sensitivity}), and the effect of separating system-independent variables (Sec.~\ref{sec:pronto_vartype}).
}
\input{chaps/05-results/05-1-1-toy}
\input{chaps/05-results/05-2-1-pronto}
\input{chaps/05-results/05-2-3-pronto-novel_op}
\input{chaps/05-results/05-1-2-toy-ablation}
\input{chaps/05-results/05-1-3-toy-sensitivity}
%\input{chaps/05-results/05-2-2-pronto-fault-types}
\input{chaps/05-results/05-3-sensitivity}
%\input{chaps/05-4-res-ablation}
\section{Conclusions and Future Outlook}
\label{sec:conclusion}
% summary: method contribution
In this study, we propose DyEdgeGAT, an unsupervised framework for early fault detection that utilizes graph attention to dynamically construct edges. This approach effectively captures evolving relationships between MTS, enhancing early fault detection. We incorporate operating condition context into node dynamics extraction, improving thereby robustness against novel operating conditions and mitigating false alarms.
% transition sentence to the result summary
DyEdgeGAT outperforms existing discrete-time graph-based methods on both synthetic and real-world industrial datasets, particularly in detecting early faults with a low severity that are often missed by other methods. Furthermore, it is effective in distinguishing between faults and novel operating conditions, a task where state-of-the-art methods typically struggle.
%It remains effective under novel operating conditions.
% ablation studies
Our ablation study highlights the efficacy of each component in the proposed architecture.
Additionally, we examined the impact of sliding window size and showed the impact of separating system-independent variables to enable robust reliable fault detection. In particular, it is important to identify both control variables and external factors in order to describe the context of operating conditions.
\revtextiot{
% addressing real-world applicability and IIoT integration
In terms of real-world applicability, DyEdgeGAT demonstrates significant potential for straightforward and resource-efficient deployment in industrial environments. This is attributed to its compact model size and short inference time. Such compactness not only facilitates easier integration into existing industrial systems but also ensures efficiency, which is crucial in environments with limited computational resources.
Furthermore, DyEdgeGAT's rapid data processing capability enables timely fault detection, thus enhancing overall industrial safety and productivity.
The model's unsupervised nature, requiring minimal labeled data, further enhances its practicality for industrial use, especially in scenarios where acquiring extensive fault data is challenging.
% outlook
% scalability discussion
While DyEdgeGAT is effective for moderately large systems, scaling it to very large systems presents challenges, particularly due to the method's quadratic complexity in pairwise dynamic edge construction.
Future research should explore the incorporation of physical prior or hierarchical structures to introduce physical biases in graph construction and enhance scalability while maintaining detection accuracy.
% prior information. particular to complex systems
Additionally, further investigation into the integration of operating state context is beneficial, especially in scenarios where information about system-independent variables is limited.
}
%In future research, it would be beneficial to further assess DyEdgeGAT's applicability to larger systems and investigate improved integration of operating state context, especially when information on system-independent variables is limited.
%The construction of dynamic edges using temporal encoding has proven to have a significant impact on the model performance; hence, investigating the use of learnable temporal encodings as opposed to fixed ones could be a valuable direction of study.
%Currently, \revtext{TimeGAT} uses a reconstruction module, with the reconstruction error employed to construct the anomaly score. Future research could explore the use of generative adversarial networks or semi-supervised learning approaches such as contrastive learning, especially when some faulty samples are available.
%In conclusion, advancements in temporal modeling techniques and efficient graph representations could yield improvements for \revtext{TimeGAT} and similar models.
%We hope that our study provides a foundation for future research in graph-based anomaly detection for multivariate time series data of complex systems.
%The continuous dynamic graph representation, compared to conventional discrete static graphs, demonstrated better performance at detecting relationship shifts due to its enhanced ability to model the evolution of interdependencies between time series.
%Furthermore, we introduced a reconstruction-based approach that differentiates control variables from measurements, effectively incorporating the context of operating states into the temporal graph. This enhancement significantly improves the detection of functional relation shifts.
%By incorporating a continuous graph representation and an operating condition-aware reconstruction, \revtext{DyEdgeGAT} can effectively detect relationship shifts in complex systems.
%Notably, \revtext{DyEdgeGAT} outperformed other state-of-the-art methods in terms of AUC, F1, and detection delay
%In this work, we proposed DyEdgeGAT, a novel unsupervised fault and anomaly detection framework that leverages graph attention for dynamic edge construction.
%The dynamic edge enables an aggregated temporal representation of MTS, which allows a more accurate detection of relationship shifts between MTS in the system, which is often indicative of early faults. Additionally, we proposed to distinguish between node types in graph construction, augmenting system-independent variables as operating condition context which helps the model to adapt to novel operating conditions and reduce false alarm rates.
% transition sentence to the result summary
%Through experiments on both synthetic and real-world industrial datasets, DyEdgeGAT demonstrated superior performance over state-of-the-art discrete time graph-based methods in fault and anomaly detection across various fault severity levels and fault types.
%Notably, DyEdgeGAT delivered a robust performance in challenging scenarios with low fault severity and difficult fault types that primarily arise from gradual shift in functional relationships within the system. Furthermore, DyEdgeGAT is robust to novel operating conditions
%In addition, we performed detailed ablation study and sensitivity analysis on the choice of sliding window size and the choice of system-independent variables.
%We have shown that it is important to distinguish between system-dependent and system-independent variables for graph-based methods, mixing them up is worse than using only system-dependent variables.
%While DyEdgeGAT's performance is promising, there are areas where further research could enhance its capabilities.
%In this work, we propose DyEdgeGAT, a novel unsupervised fault and anomaly detection framework that leverages graph attention for dynamic edge construction. The dynamic edge enables an aggregated temporal representation of MTS, which allows for more accurate detection of relationship shifts between MTS in the system, indicative of early faults. Additionally, we distinguish between node types in graph construction, augmenting system-independent variables as operating condition context, which helps the model adapt to novel operating conditions and reduce false alarm rates.
%Through experiments on both synthetic and real-world industrial datasets, DyEdgeGAT demonstrates superior performance over state-of-the-art discrete time graph-based methods in fault and anomaly detection across various fault severity levels and fault types. Notably, DyEdgeGAT delivers a robust performance in challenging scenarios with low fault severity and difficult fault types that primarily arise from gradual shifts in functional relationships within the system. Furthermore, DyEdgeGAT is robust to novel operating conditions.
%We also perform a detailed ablation study and sensitivity analysis on the choice of sliding window size and the choice of system-independent variables. We show that it is important to distinguish between system-dependent and system-independent variables for graph-based methods, as mixing them up can lead to worse performance than using only system-dependent variables.
%In the future, it is worth exploring DyEdgeGAT's performance on larger systems. Further exploration of how to better incorporate the context of the operating state into the temporal graph is also promising, particularly in cases where the information about system-independent variables is unknown.
\appendix
%\subsection{Additional Information on Experiment and Results}
%\subsection{Industrial Data Preprocessing}
%\label{sec:app_pronto}
% mention the train test split (better in normal text or appendix?)
%\subsection{Baseline Configurations and Hyperparameter Tuning}
%\label{sec:app_hyperparameters}
%\input{tabs/model_hyper}
%Hyperparameters were fine-tuned using grid search across plausible values, as detailed in Tab.~\ref{table:hyperparameters}, with optimal settings indicated within parentheses. The first and second values correspond to the first and second case studies, respectively. $^*$ denotes a reference to the text description for further details. Selections were based on the best validation loss.
%We varied the number of layers (L) and the dimensions of the hidden layers (H) and tested different normalization techniques, such as Batch Normalization (BN) and Layer Normalization (LN). Table~\ref{table:hyperparameters} also details model-specific parameters and optimal model sizes. For FNN and AE models, the input sequence window size is 1, while for all other models, it is consistently 15 in both case studies.
%\textbf{FNN}.
%We varied the number of layers and their hidden dimensions. The same independent variables used for dyEdgeGAT serve as input, mapped to the measurement variables.
%\textbf{AE}.
%The latent dimension of AE is the size of system-independent variables. The optimal configuration was 20-10-5-2 for the synthetic study and 20-20-20-10-10-4 for the industrial case study.
%\textbf{USAD}.
%We adhered to the original structure but varied the AE's latent dimension, initial training warm-up epochs, and final activation function. Adding batch normalization layers was tested but found non-beneficial.
%\textbf{GDN}.
%We preserved most of the original hyperparameters from the paper, as modifications had minimal impact on results. Nonetheless, the model showed sensitivity to the decoder hidden layer dimension (tested from 100 to 200) and sensor embedding dimension (tested from 20 to 50). We experimented with varying top k values for graph sparsification and discovered that the GDN model achieved optimal performance as a fully connected graph on our task.
%\textbf{MTADGAT}.
%We tested the hidden dimensions for feature and temporal attention embeddings between 10 and 40, and for the reconstruction and forecasting modules, we varied them from 10 to 80.
%\textbf{GRELEN}.
%We adjusted the number of graphs from 1 to 4, adhering to similar graph distribution priors as in the original paper. The hidden dimension was varied between 10 and 40. Additionally, we evaluated different anomaly scoring methods for GRELEN, identifying that the point adjustment in the original implementation was not rigorous for evaluation, further detailed in Section~\ref{sec:app_grelen_syn_results}.
%\textbf{DyEdgeGAT}.
%We varied the hidden node dimension of the node between 10 and 20, the hidden edge temporal embedding dimension between 20, 40, and 100, hidden graph dimension between 20 and 40. The dimension of the temporal encoding is between 5 and 10.
%\subsection{Suboptimal performance of GRELEN}
%\label{sec:app_grelen_syn_results}
%GRELEN applied a point adjustment technique, which has been shown in a recent paper by Him \textit{et al.}\cite{kim2022towards} that it is not rigorous for evaluation. We report the average AUC across different severity levels of GRELEN on the synthetic dataset with and without point adjustment in Tab.~\ref{sec:app_grelen_syn_results}.
% mention how hyperparameters are set for each model
%It should be noted that the healthy data in the pronto dataset contains all 20 operating conditions while the induced faults were generated under fault conditions A and B only.
%\subsection{Ablation Study}
%Table~\ref{tab:pronto_agg_results} presents the results of the ablation study. As observed, removing a component primarily impacts the AUC while having less influence on the F1 score and the delay. Among all components, the topology-based score has the least effect on the performance of \revtext{TimeGAT}, while the temporal graph construction demonstrates the most impactful, corroborating the findings from the synthetic case study.
%Fig.~\ref{fig:pronto-ablation} illustrates how each component affects the performance of \revtext{TimeGAT} across different fault types.
% diverted flow
%For the \textit{diverted flow} fault type, the most significant drop in AUC performance is observed when the temporal graph construction is removed. This underscores the critical role of temporal graph construction in detecting faults, especially those dominated by shifts in functional relationships.
% air blockage
%Interestingly, for the \textit{air blockage} fault type, the removal of control augmentation is more detrimental than the removal of temporal graph construction. This might indicate a larger change in the operating state for this specific fault type.
%Overall, the removal of any component results in a performance decrease for all fault types, except for the topology-based anomaly score. In the latter case, the AUC drops only for the \textit{air leakage} fault type, with a slight increase noted for the other two fault types. However, the removal of this anomaly score consistently results in a drop in the F1 score across all fault types.
% conclusion
%In conclusion, this ablation study confirms the integral role each component plays in the robust performance of \revtext{TimeGAT} in anomaly detection across diverse fault types and affirms the effectiveness of our design choices.
% if you will not have a photo at all:
%\begin{IEEEbiographynophoto}{Mengjie Zhao}
%Mengjie Zhao is pursuing her Ph.D. in Robotics, Control, and Intelligent Systems at the Laboratory of Intelligent Maintenance and Operations Systems (IMOS), EPFL, Lausanne, Switzerland. She obtained her BSc in Engineering Science and an MSc with Honors track in Computational Mechanics from the Technical University of Munich (TUM) and was affiliated with the Bavarian Graduate School of Computational Engineering (BGCE). Her current research focuses on temporal graph neural network (TGNN)-based predictive maintenance (PM). The key challenges she addresses include effectively transforming raw heterogeneous system data into a graph representation that captures the diverse, dynamic, and hierarchical interactions between time series under varying operating conditions, and designing appropriate network structures to extract system-level information. Her research aims to construct a TGNN-based framework capable of learning the dynamics of intricate industrial systems, thereby enhancing PM.
%\end{IEEEbiographynophoto}
% I copied it from your most recent IEEE publication, hope it's fine?
%\begin{IEEEbiographynophoto}{Olga Fink}(Memeber,~IEEE), received a Diploma
%degree in industrial engineering from the Hamburg University of Technology, Hamburg, Germany, in
%2008, and the Ph.D. degree from ETH Zürich, Zürich, Switzerland, in 2014. She has been an Assistant Professor of Intelligent Maintenance and Operations Systems with EPFL, Lausanne, Switzerland, since March 2022. Before joining EPFL Faculty, she was an Assistant Professor of Intelligent Maintenance Systems with ETH Zürich from 2018 to 2022. From 2014 and
%2018, she was heading the research group “Smart Maintenance” with the Zurich University of Applied Sciences, Winterthur, Switzerland. And, she is also a Research Affiliate with the Massachusetts Institute of Technology, Cambridge, CA, USA. Her research focuses on hybrid algorithms fusing physics-based models and deep learning algorithms, hybrid operational digital twins, transfer learning, self-supervised learning, deep reinforcement learning, and multiagent systems for intelligent maintenance and operations of infrastructure and complex assets. Dr. Fink was awarded the Prestigious Professorship Grant of the Swiss
%National Science Foundation.
%\end{IEEEbiographynophoto}
%\subsection{Modelling Inter-dependencies and Dynamics with Graphs}
Within the discrete-time framework, temporal graphs are represented by a series of static graph snapshots, sequentially arranged over time. This approach incorporates two common methodologies: \textit{graph-then-time} and \textit{time-and-graph}.
Conversely, the continuous-time representation, often termed \textit{time-then-graph}, presents temporal dynamics within a static graph structure. Gao \textit{et al.}~\cite{gao2022equivalence} provide a more thorough definition of these frameworks.
% examples
Existing work primarily focuses on modeling temporal graphs for MTS using a discrete graph representation and aligns with the \textit{time-and-graph} architecture.
Many spatial-temporal GNNs belong to this category, including
Diffusion Convolutional Recurrent Neural Network (DCRNN)~\cite{li2017diffusion} that utilizes SpectralGCN to capture spatial dependency and then GRU to capture the temporal dependency, and Multivariate Time-series Anomaly Detection via Graph Attention Network (MTADGAT)~\cite{zhao2020mtadgat}, which constructs feature and temporal graphs to capture inter-dependencies between time series, followed by a GRU to extract dynamics.
% limitation
Despite the predominance of the discrete-time representation, Gao \textit{et al.}~\cite{gao2022equivalence} have demonstrated in their recent paper that the \textit{time-then-graph} representation is strictly more representative than the other two representations, making it more appropriate for anomaly detection in MTS.
However, implementing this representation remains a challenge, especially in the context of MTS where the graph must be inferred.
This framework has been predominantly applied to graph-structured networks such as evolving social networks, examples of which include Temporal Graph Network (TGN)~\cite{rossi2020tgn} and Temporal Graph Attention Networks (TGAT)~\cite{xu2020tgat}, where the graph topology is provided implicitly.
Currently, there is no existing work on jointly constructing a graph from MTS and applying continuous graph representation to it.
We believe this approach would offer greater expressiveness compared to the discrete graph representation of MTS, thus enhancing its effectiveness in detecting abnormal changes in functional relations within MTS.
% Temporal Graph Neural Networks (TGNNs) are applied to effectively extract the temporal and inter-dependencies from the MTS data.
% in graphs~\cite{zaki2016comprehensive}, making them particularly relevant for MTS anomaly detection.
%\subsubsection{Continuous time representation in GNNs}
%In the continuous-time representation, all past node and edge attributes are integrated into a single static graph. This representation captures the entire history of temporal dynamics. An encoder function is used to encode node attributes and another encoder function is used to encode edge attributes. The encoded representations are then passed through a GNN to model the interactions and dependencies in the continuous-time graph.
% time-then-graph
%An example of the \textit{time-then-graph} framework is TGAT~\cite{xu2020tgat}, which memorizes the edges of past snapshots to create a static heterogeneous graph, used to extract the final temporal graph representations. TGN~\cite{rossi2020tgn} extends this framework by incorporating a memory module. Studies by Gao \textit{at al.}~\cite{gao2022equivalence} have demonstrated the superior expressiveness of GRU-GCN over GCN-GRU in continuous-time representation.
%\subsubsection{Discrete time representation in GNNs}
%In the discrete-time representation, a temporal graph consists of a sequence of static graph snapshots over multiple time steps. Each snapshot is processed by a static GNN to encode the graph structure at that time step. To model the temporal dynamics, an aggregation function, such as an RNN, is used to aggregate the graph encodings across all snapshots.
% examples
%Notable approaches in this framework include EvolveGCN~\cite{pareja2020evolvegcn}, which learns temporal dynamics through GCN parameters and passes them as recurrent states, and Dynamic Self Attention (DySAT)~\cite{sankar2020dysat}, which introduced the first discrete TGNN based solely on attention.
%ROLAND~\cite{you2022roland} extends static GNNs to dynamic settings by updating node embeddings based on newly observed nodes and edges using a GRU update module.
% time-and-graph
%The \textit{time-and-graph} framework extends the \textit{graph-then-time} approach by incorporating an additional GNN layer to encode historical snapshot representations.
% examples
% GCRN-M2~\cite{seo2018structured} which combine SpectralGCN~\cite{defferrard2016convolutional} with LSTM, DCRNN~\cite{li2017diffusion} which uses a combination of SpectralGCN and GRU.
%Many spatial-temporal GNNs belong to this category, auch as T-GCN~\cite{zhao2019tgcn}, STGCN~\cite{yu2018spatio}, MTGNN\cite{wu2020mtgnn} and GTS~\cite{shang2021gts}.
%In the discrete-time representation, two common frameworks are employed: \textit{graph-then-time} and \textit{time-and-graph}.
%In the discrete graph representation, a temporal graph consists of a sequence of static graph snapshots over multiple time steps.
%The continuous-time representation is also referred to as \textit{time-then-graph}, capturing temporal dynamics within a static graph structure.
%For a more comprehensive definition, readers can refer to the work of Gao \textit{et al.}~\cite{gao2022equivalence}.
%Temporal or dynamic graphs derived from MTS data can effectively incorporate the inherent inter-dependencies and dynamics present in the data.
%There are two common frameworks to represent temporal graphs: \textit{discrete-time} and \textit{continuous-time} graph representations~\cite{zaki2016comprehensive}.
\subsection{Dynamic Interaction Modeling }
\label{sec:method_graph_repr}
\revtextrewrite{
%To address the complexity of temporal and spatial dynamics in IIoT systems, DyEdgeGAT employs the \textit{time-then-graph} framework based on the aggregated temporal graph representation.
%The \textit{time-then-graph} framework in DyEdgeGAT, as the name suggests, sequentially extracts temporal patterns using a sequence model and captures spatio-temporal relationships using GNNs. These two steps are elaborated in detail below.
\textbf{First time}. During the temporal dynamics extraction phase, Gated Recurrent Units (GRUs) followed by Layer Normalization (LN) and activation functions are utilized for both, nodes and edges to model dynamics over the time window $[t_w, t]$ of length $W$. The dynamics are encoded as:
\begin{align}
\mathbf{H}^{node} &= \text{GRU}^{\text{node}}(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t}), \quad \forall i \in \mathcal{V}, \label{eq:node_dyn}\\
\mathbf{H}^{edge} &= \text{GRU}^{\text{edge}}(\mathbf{A}^{t_w:t}), \quad \forall (i,j) \in \mathcal{E}, \label{eq:edge_dyn}
\end{align}
where $\mathbf{H}^{\text{node}} \in \mathbb{R}^{N\times d_n}$ and $\mathbf{H}^{\text{edge}} \in \mathbb{R}^{N \times N }$ denote the latent dynamics representations of nodes and edges, respectively, with $d_n$ representing the hidden node feature dimensions.
Here, $\mathcal{V}$ is the set of nodes, and $\mathcal{E}$ is the set of edges.
In Eq.~\ref{eq:node_dyn}, the matrix $\mathbf{X}^{t_w:t} \in \mathbb{R}^{N \times W}$ represents the sliding window of system-dependent signals, while $\mathbf{U}^{t_w:t} \in \mathbb{R}^{N_u \times W}$ represents system-independent signals, capturing external factors or control variables.
These system-independent signals $\mathbf{U}^{t_w:t}$ reflect the operating conditions of the system and should be treated separately from the dynamics of the system-dependent measurements $\mathbf{X}^{t_w:t}$. In Sec.~\ref{sec:method_oc_recon}, we will elaborate in more detail on how the context of operating conditions is augmented in Eq.\ref{eq:node_dyn}.
In Eq.~\ref{eq:edge_dyn}, $\mathbf{A}^{t_w:t}\in \mathbb{R}^{N \times N \times W}$ represents a sequence of aggregated edge weights over \( [t_w, t] \), which differing from the conventional spatial-temporal graph approach where $\mathbf{A}$ is assumed to be static for each input $\mathbf{X}^{t_w:t}$. Therefore, it is necessary to construct dynamic edges to capture the evolving interactions, and further details on this construction will be provided in Sec.~\ref{sec:method_graph_constr}.
\textbf{Then graph}. Upon obtaining the temporal features, the graph processing phase leverages GNNs followed by Batch Normalization (BN) layers to integrate the node and edge dynamics into an aggregated graph representation:
\begin{align}
\mathbf{Z} &= \text{GNN}^L(\mathbf{H}^{\text{node}}, \mathbf{H}^{\text{edge}}), \label{eq:gnn}
\end{align}
where $\mathbf{Z} \in \mathbb{R}^{N\times d_z}$ denotes the output representation of the nodes after $L$ layers of GNN processing, with $d_z$ being the dimensionality of the output space, and $\mathbf{Z}^0$ initialized with $\mathbf{H}^{\text{node}}$.
}
\revtext{
The GNN layer in Eq. \ref{eq:gnn} follows the Message Passing Neural Network (MPNN) schema~\cite{gilmer2017neural}. In this framework, the $l$-th layer of a GNN consists of a step of message passing and update:
\begin{align}
\mathbf{m}^{(l)}_i &= \sum_{j \in \mathcal{N}_i} \text{MSG}^{(l)} (\mathbf{z}^{(l-1)}_i, \mathbf{z}^{(l-1)}_j, \tilde{{a}}_{ij}), \\
\mathbf{z}^{(l)}_i &= \text{UPDATE}^{(l)} (\mathbf{z}^{(l-1)}_i, \mathbf{m}^{(l)}_i).
\end{align}
Here, $\mathcal{N}_i$ denotes the neighborhood of node $i$, $\mathbf{m}^{(l)}_i$ is the message accumulated at node $i$ in layer $l$, and ${\tilde{{a}}}_{jk}$ represents the edge weights between nodes $j$ and $k$ of the aggregated temporal graph. The functions $\text{MSG}$ and $\text{UPDATE}$ are learnable transformations specific to the GNN architecture, which aggregate neighbor information and update the node representation, respectively.
Specifically, we employ Graph Isomorphism Network (GIN)~\cite{xu2018gin} as the GNN module and employ Batch Normalization (BN) between its layers. An overview of the DyEdgeGAT framework can be found in Fig.\ref{fig:dyedgegat_overview}.
}
%The \textit{time-then-graph} framework in DyEdgeGAT extracts temporal patterns via sequence models and spatio-temporal relationships through GNNs. As the name suggests, it operates in two consecutive phases: the extraction of temporal dynamics followed by graph processing. Through continuous temporal representation, it can better capture the evolution of the relationships between sensor signals.
% Another important observation relates to $\mathbf{A}^{t_w:t}$ in Eq.~\ref{eq:edge_dyn}. Traditional GNN-based methods consider $\mathbf{A}$ as static for the sliding MTS sequence (or for the entire time series). However, in order to extract edge dynamics, it is necessary to construct dynamic edges. We will elaborate on this aspect in Sec.~\ref{sec:method_graph_constr}.
% We formally define a graph as $\mathcal{G}=(\mathcal{V}, \mathcal{E})$, where $\mathcal{V}=\{v_i\}$ is the set of nodes and $\mathcal{E}=\{e_{ij}\}$ is the set of edges.
%We represent a temporal graph over a sliding window of $w$ time steps as a single, aggregated static graph $\mathcal{G}=(\mathbf{X}^{t_w:t}, \mathbf{A}^{t_w: t})$, where $\mathbf{X}^{t_w:t}=[\mathbf{X}^{t_w},\cdots,\mathbf{X}^t] \in \R^{|\mathcal{V}|\times W}$ represents the node attributes, and $\mathbf{A}^{t_w:t}=[\mathbf{A}^{t_w},\cdots,\mathbf{A}^t] \in \R^{|\mathcal{V}|\times|\mathcal{V}|\times W} $ represents the edge attributes. These attributes integrate all past node and edge information, as defined in~\cite{gao2022equivalence}.
%is to first represent the evolving relationships of node and edge attributes as sequences and subsequently embed these sequences in a static graph using a GNN.
%Here, $H^{node}$ and $H^{edge}$ represent the evolving node and edge representations, respectively. $E$ denotes the set of all possible edges in an aggregated form, and $Z$ yields the final temporal graph representation encompassing all nodes.
%A temporal encoder $f$ is used to extract the dynamics of node and edge attributes. The resulting node and edge representations are utilized as the node and edge attributes in a new (static) graph, which is subsequently encoded by a Graph Neural Network (GNN). Following the notation of Gao \textit{et al.}~\cite{gao2022equivalence}, we define the aggregated temporal graph as follows:
%\begin{align}
% \mathbf{H}^{node} &= f^{node}(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t}), \quad \forall i \in \mathcal{V}, \label{eq:node_dyn}\\
% \mathbf{H}^{edge} &= f^{edge}(\mathbf{A}^{t_w:t}), \quad \forall (i,j) \in \mathcal{E}, \label{eq:edge_dyn}\\
% \mathbf{Z} &= \text{GNN}^L(\mathbf{H}^{node}, \mathbf{H}^{edge})\label{eq:gnn}.
%\end{align}
%Here, $\mathbf{H}^{node} \revtext{\in \R^{|\mathcal{V}|\times d_n}}$ and $\mathbf{H}^{edge} \revtext{\in \R^{|\mathcal{E}|\times d_e}}$ represent the dynamics of nodes and edges, respectively. $\mathcal{E}$ denotes the set of all possible edges in an aggregated form, and $\mathbf{Z}$ represents the final temporal graph representation of all nodes. $L$ corresponds to the number of GNN layers. For the node encoder $f^{node}$ and edge encoder $f^{edge}$, we use separate GRU models, followed by a Layer Norm (LN) and an activation function to extract dynamics. The GNN model is then employed to learn the graph representation, utilizing the Graph Isomorphism Network (GIN)~\cite{xu2018gin} as the graph module and incorporating Batch Normalization (BN) between the GNN layers.
%Here, \(\mathbf{H}^{node} \in \R^{|\mathcal{V}|\times d_n}\) and \(\mathbf{H}^{edge} \in \R^{|\mathcal{E}|\times d_e}\) depict the dynamics of nodes and edges. \(\mathcal{E}\) signifies all potential edges in aggregated form, with \(\mathbf{Z}\) as the overall temporal graph representation. For node and edge encoders, separate GRU models enhanced with Layer Norm (LN) and activation functions are employed. The GNN employs the Graph Isomorphism Network (GIN) with Batch Normalization (BN) interleaved between its layers.
%Note that in the original formulation, $f^{node}$ in Eq.~\ref{eq:node_dyn} is a function of only $\mathbf{X}^{t_w:t}$. However, we argue that the control variables provide valuable information about the operating conditions of the system and should be treated separately from the dynamics of the measurements. Therefore, they are modeled differently. In Sec.~\ref{sec:method_oc_recon}, we provide a detailed explanation of how we include the control variables in the node encoding function to capture the context of the operating conditions.
\section{Preliminaries}
%\revtextrewrite{
%In this paper, we utilize bold uppercase letters (e.g., $\mathbf{X}$), bold lowercase letters (e.g., $\mathbf{x}$), and calligraphic letters (e.g., $\mathcal{V}$) to denote matrices, vectors, and sets, respectively.
%\subsection{Problem Statement}
% state how MTS is defined in a sensor network
%In an IIoT sensor network, $N$ system-dependent measurement signals at any time $t$ are represented as $\mathbf{x}^{t} = [x_{1}^{t}, \cdots, x_{N}^{t}] \in \R^{N}$, where $x_{i}^{t}$ represents the $i^{th}$ sensor's time series at time $t$.
%By employing a sliding window approach of length $W$, we construct samples as:
%$\mathbf{X}^{t_w:t} = [\mathbf{x}^{t_w}, \cdots, \mathbf{x}^{t-1}, \mathbf{x}^{t}] \in \R^{N \times W}$, where $t_w = t - W + 1 > 0$.
%Similarly, system independent variables (control variables and external variables) are denoted $\mathbf{U}^{t_w:t} \in \R^{N_u \times W}$.
%The proposed method, DyEdgeGAT, employs a reconstruction model $f_\theta$ for unsupervised MTS fault and anomaly detection.
%The model generates an output $\mathbf{\hat{X}}^{t_w:t} = f_\theta(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t})$ for each $N_{train}$ healthy samples, aiming to minimize the reconstruction discrepancy $||\mathbf{\hat{X}}^{t_w:t} - \mathbf{X}^{t_w:t}||$.
%At test time, the model outputs $\mathbf{y} \in \R^{N_{test}}$ for all $N_{test}$ samples based on the reconstruction discrepancy, where each element $\mathbf{y}_i \in \{0, 1\}$ indicates fault or anomaly.
%}
\subsection{Multivariate Time Series as Temporal Graph}
\label{sec:prelim_graph_repr}
\revtext{
%Given the significance of analyzing the evolving relationships in the IIoT systems, it becomes important to understand the underlying structure and relationships of these MTS data.
Temporal graphs, also referred to as spatial-temporal or dynamic graphs (with no distinction among these terms in our context), present a promising data format for MTS analysis in the IIoT context.
%These graph structures effectively capture the temporal dynamics of MTS data, allowing downstream analysis of the relationships within the network over time~\cite{jin2023survey}.
There are primarily two \textit{equivalent} approaches to represent these temporal graphs following the definition by Gao et al.~\cite{gao2022equivalence}.
\textbf{Discrete-Time Dynamic Graph}.
Conceptualized as a sequence of attributed graphs at discrete time steps, a discrete-time dynamic graph within a time window from $t_w$ to $t$ can be represented as:
\[\mathcal{G} = \{\mathcal{G}^{t_w}, \ldots, \mathcal{G}^t\}.\]
At each time instance \( t \), we define \( \mathcal{G}^t = (\mathbf{A}^t, \mathbf{x}^t) \) as the attributed graph, where $\mathbf{A}^t \in \R^{N \times N}$ represents the adjacency matrix and $\mathbf{x}^t \in \R^{N}$ denotes the feature matrix.
\textbf{Aggregated Temporal Graph}.
An alternative representation, as proposed by Gao et al.~\cite{gao2022equivalence}, is the aggregated temporal graph, where the graph is treated as a static graph aggregating information from all time steps:
\[ \mathcal{G} = (\mathbf{X}^{t_w:t}, \mathbf{A}^{t_w: t}), \]
Here, $\mathbf{X}^{t_w:t}=[\mathbf{x}^{t_w},\cdots,\mathbf{x}^t] \in \R^{N\times W}$ represents all node attributes aggregated within the time window, while $\mathbf{A}^{t_w:t}=[\mathbf{A}^{t_w},\cdots,\mathbf{A}^t] \in \R^{N\times N \times W}$ represents aggregated edge attributes as sequences of standalone attributes, with zeros for absent attributes.
}
% In case of IIoT with heterogenous sensors, each sensor is treated as a separate sensor, so $D=1$.
%A temporal graph observed over a sliding window of $w$ time steps can be represented as a sequence of static graph snapshots.
%Formally, it is defined as: $ \{ (\mathbf{X}^{t}, \mathbf{A}^{t}) \}^T_{t=t_w} $
%\textbf{Attributed Graph}.
%An attributed graph is a static graph that associates each node with a set of attributes, representing node features. Formally, an attributed graph is defined as $\mathcal{G}=(\mathbf{X}, \mathbf{A})$,
%where $\mathbf{X} \in \mathbb{R}^{N \times D}$ is the node attribute matrix, where each row represents the attributes of a node and $D$ denotes the dimensionality of the node attributes.
%And \( \mathbf{A} \in \mathbb{R}^{N \times N} \) is the adjacency matrix, indicating the presence (or weight) of edges between nodes. An entry \( A_{ij} \) is non-zero if an edge \( e_{ij} \) exists between nodes \( v_i \) and \( v_j \).
%In light of the attributed graphs, a temporal graph can be described as a series of attributed graphs, which effectively represent (multivariate) time series data in conjunction with evolving structural information over time.
%\textbf{Discrete-Time Dynamic Graph}. A discrete-time dynamic graph can be visualized through a series of attributed graphs at various discrete time steps. This graph can be represented as \( \mathcal{G} = \{\mathcal{G}^1, \mathcal{G}^2, \ldots, \mathcal{G}^T\} \), where for each time point \( t \), \( \mathcal{G}^t = (\mathbf{A}^t, \mathbf{X}^t) \) signifies the attributed graph. Here, \( \mathbf{A}^t \) in \( \R^{N \times N} \) and \( \mathbf{X}^t \) in \( \R^{N \times D} \) are the adjacency and feature matrices respectively. The adjacency matrix \( \mathbf{A}^t \) might transform as time progresses or could remain static, subject to the particular configurations in place. When we're generalizing time series data into this format, we express \( \mathbf{X}^t \) as a column vector \( \mathbf{x}^t \) with its elements from \( \mathbb{R}^N \).
%When time series data is generalized into this format, \( \mathbf{X}^t \) is expressed as a column vector \( \mathbf{x}^t \) with elements from \( \mathbb{R}^N \).##
%A discrete-time dynamic graph can be visualized through a series of attributed graphs at various discrete time steps.
%Given sliding window of $W$ time steps from $t_w$ to $t$,
%This graph is represented as \( \mathcal{G} = \{\mathcal{G}^{t_w}, \ldots, \mathcal{G}^t\} \). For each time point \( t \), \( \mathcal{G}^t = (\mathbf{A}^t, \mathbf{x}^t) \) is defined as the attributed graph. Here, \( \mathbf{A}^t \) \in \( \R^{N \times N} \) and \( \mathbf{x}^t \) \in \( \R^{N} \) are known as the adjacency and feature matrices, respectively. The adjacency matrix \( \mathbf{A}^t \) may be transformed as time progresses or remain static, depending on the particular configurations chosen.
\section{Proposed Framework}
\subsection{Problem Statement}
% statement of the task (should I mention in addition change of relationship in time?)
Multivariate time-series anomaly detection aims to detect anomalies at the entity level. This is especially challenging due to the complex dependencies and temporal dynamics exhibited by multivariate time-series data.
% state how MTS is defined in a sensor network
Sensor measurements can be modeled as a sequence of $N$ Multivariate Time Series (MTS) at a time $\mathbf{X}^{t_w:t} = \lbrack\mathbf{x}_{1}^{t_w:t},\cdots \mathbf{x}_{N}^{t_w:t}\rbrack\ \in \R^{N\times W}$, where $t$ is the current time step and $W=t-t_w+1>0$ is the length of the observation window.
Similarly, control variables can be modelled as: $\mathbf{U}^{t_w:t} = \lbrack\mathbf{u}_{1}^{t_w:t},\cdots \mathbf{u}_{N}^{t_w:t}\rbrack\ \in \R^{N_u\times W}$
% TODO: introduce sliding window
% anomaly detection task
The task of multivariate time-series anomaly detection is to produce an output vector $\mathbf{y} \in \R^n$, where $\mathbf{y}_i \in \{0,1\}$ denotes whether the $i$-th timestamp is an anomaly. The mathematical notations are presented in Table~\ref{tab:notations}.
\input{tabs/notations}
\subsection{Overview}
Our proposed method, \textit{\revtext{TimeGAT}} (Dynamic Graph Attention for Anomaly Detection), is specifically designed to address the dynamic nature of multivariate time series by modeling relationship changes over time. By effectively capturing the complex inter-feature correlations and temporal dependencies, \revtext{TimeGAT} is able to robustly detect anomalies in multivariate time series data.
The key mechanisms that enable this are a temporal graph representation learning scheme and an operating condition-aware reconstruction mechanism.
% TODO: add bullet point summary
In the following subsections, we elaborate on the core components of the \revtext{TimeGAT}.
\subsection{Multivariate Time Series as a Temporal Graph}
We first introduce how we present the Multivariate Time Series (MTS) in the form of a Continuous Time Temporal Graph (CTTG). We represent a temporal graph over $T$ time steps as a single, aggregated static graph $G(\mathbf{X}_{t\leq T}, \mathbf{A}_{t\leq T})$, where $\mathbf{X}_{t\leq T}$ is the raw signal input, and $\mathbf{A}_{t\leq T}$ is the learned dynamic edge.
\subsection{\revtext{TimeGAT} Temporal Graph Construction}
This section explains how $\mathbf{X}_{t\leq T}$ and $\mathbf{A}_{t\leq T}$ are formed and learned.
\subsubsection{Raw Signal Input}
The raw signal input $\mathbf{X}_{t\leq T} = [\mathbf{X}_1,\cdots,\mathbf{X}_T]$ represents the original multivariate time series data integrated over all past time steps.
\subsubsection{Dynamic Edge Learning}
The dynamic edge $\mathbf{A}_{t\leq T} = [\mathbf{A}_1,\cdots,\mathbf{A}_T]$ is learned using an attention-based approach. We employ a sliding window method to capture the temporal dependencies within the multivariate time-series data. Our graph attention mechanism is built upon the GATv2~\cite{brody2022gatv2} algorithm, which enhances the original Graph Attention Networks (GAT) by overcoming its static attention limitation.
\subsection{Temporal Graph Representation Learning}
After forming the raw signal input and dynamic edge, we use a temporal encoder $f$ to extract the dynamics of node and edge attributes. The resulting node and edge representations become the node and edge attributes in a new (static) graph, which is then encoded by a GNN.
\begin{align}
\mathbf{H}^{node} &= f^{node}(\mathbf{X}_{t\leq T}), \forall i \in \mathcal{V} \\
\mathbf{H}^{edge} &= f^{edge}(\mathbf{A}_{t\leq T}), \forall (i,j) \in \mathcal{E} \\
\mathbf{Z} &= \text{GNN}(H^{node}, H^{edge})
\end{align}
where $H^{node}$ and $H^{edge}$ are the representations of node and edge evolutions, $\mathcal{E}$ is the set of all possible edges in aggregated form, and $Z$ is the final temporal graph representation of all nodes. We use a GRU to extract both edge and node dynamics ($f^{node}=\text{GRU}^{node}$ and $f^{edge}=\text{GRU}^{edge}$) and GCN ($GNN=GCN$) to learn the graph representation. The learned feature attention in this continuous time representation is dynamic, unlike static attention in discrete representations.
%In our case, each $\mathbf{x}_i$ is associated with a graph node $v_i$.
%We define an attributed graph as $\mathcal{G}=(\mathbf{X}, \mathbf{A})$, where $\mathbf{X} \in \R^{N\times w}$ are the node attributes and $\mathbf{A} \in \R^{N\times N}$ are the edge attributes.
%A graph structure can be created by representing each time series as a node in the graph and using edges to represent the interactions between the time series. In general,
%A temporal graph is created by representing each time series as a node in the graph, and the edges represent the interactions between the time series. The temporal graph over $T$ time steps is represented as a single aggregated static graph $G(\mathbf{X}_{t\leq T}, \mathbf{A}_{t\leq T})$, where $\mathbf{X}_{t\leq T}$ and $\mathbf{A}_{t\leq T}$ integrate all past node and edge attributes~\cite{gao2022equivalence}. The temporal encoder extracts the dynamics of node and edge attributes, and the resulting representations become the attributes of the new (static) graph. The GNN is then applied to this graph to learn the representations.
%\subsection{Time-then-graph Representation}
% graph learning task
%A graph structure can be created by representing each time series as a node in the graph and using edges to represent the interactions between the time series. In general, a graph is defined $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ as a set of nodes $\mathcal{V}=\{v_i\}$ and edges $\mathcal{E}=\{e_{ij}\}$. In our case, each $\mathbf{x}_i$ is associated with a graph node $v_i$.
%We define an attributed graph as $\mathcal{G}=(\mathbf{X}, \mathbf{A})$, where $\mathbf{X} \in \R^{N\times w}$ are the node attributes and $\mathbf{A} \in \R^{N\times N}$ are the edge attributes.
% Time then graph
%We represent a temporal graph over $T$ time steps as a single, aggregated static graph $G(\mathbf{X}_{t\leq T}, \mathbf{A}_{t\leq T})$, where $\mathbf{X}_{t\leq T}=[\mathbf{X}_1,\cdots,\mathbf{X}_T]$, $\mathbf{A}_{t\leq T}=[\mathbf{A}_1,\cdots,\mathbf{A}_T]$ integrate all past node and edge attributes~\cite{gao2022equivalence}.
%A temporal encoder $f$ extracts the dynamics of node and edge attributes, and the resulting node and edge representations become the node and edge attributes in a new (static) graph, which is then encoded by a GNN.
%The two steps of a continuous TGNN as introduced in~\cite{gao2022equivalence} are:
%\begin{align}
% \mathbf{H}^{node} &= f^{node}(\mathbf{X}_{t\leq T}), \forall i \in \mathcal{V} \\
% \mathbf{H}^{edge} &= f^{edge}(\mathbf{A}_{t\leq T}), \forall (i,j) \in \mathcal{E} \\
% \mathbf{Z} &= \text{GNN}(H^{node}, H^{edge})
%\end{align}
%where Hnode and Hedge are the representations of node and edge evolutions, E is the set of all possible edges in aggregated form and Z is final temporal graph representation of all nodes. We use GRU to extract both edge and node dynamics ($f^{node}=\text{GRU}^{node}$ and $f^{edge}=\text{GRU}^{edge}$) and GCN to learn the graph representation. The learned feature attention in this continuous time representation is dynamic, unlike static attention in discrete representations.
To capture the temporal dynamics and dependencies in multivariate time series data, \revtext{TimeGAT} utilizes the \textit{time-then-graph} representation.
Formally, we define a graph as $\mathcal{G}=(\mathcal{V}, \mathcal{E})$, where $\mathcal{V}=\{v_i\}$ is the set of nodes and $\mathcal{E}=\{e_{ij}\}$ is the set of edges.
We represent a temporal graph over a sliding window of $w$ time steps as a single, aggregated static graph $\mathcal{G}=(\mathbf{X}^{t_w:t}, \mathbf{A}^{t_w: t})$, where $\mathbf{X}^{t_w:t}=[\mathbf{X}^{t_w},\cdots,\mathbf{X}^t] \in \R^{|\mathcal{V}|\times W}$ represents the node attributes, and $\mathbf{A}^{t_w:t}=[\mathbf{A}^{t_w},\cdots,\mathbf{A}^t] \in \R^{|\mathcal{V}|\times|\mathcal{V}|\times W} $ represents the edge attributes. These attributes integrate all past node and edge information, as defined in~\cite{gao2022equivalence}.
A temporal encoder $f$ is used to extract the dynamics of node and edge attributes. The resulting node and edge representations become the node and edge attributes in a new (static) graph, which is then encoded by a Graph Neural Network (GNN). We adopt the notation from Gao \textit{et al.}~\cite{gao2022equivalence} and define the aggregated temporal graph as follows:
\begin{align}
\mathbf{H}^{node} &= f^{node}(\mathbf{X}^{t_w:t}, \mathbf{U}^{t_w:t}), &\forall i \in \mathcal{V}, \label{eq:node_dyn}\\
\mathbf{H}^{edge} &= f^{edge}(\mathbf{A}^{t_w:t}), &\forall (i,j) \in \mathcal{E}, \label{eq:edge_dyn}\\
\mathbf{Z} &= \text{GNN}(\mathbf{H}^{node}, \mathbf{H}^{edge}).
\end{align}
Here, $\mathbf{H}^{node}$ and $\mathbf{H}^{edge}$ represent the node and edge dynamics, respectively. $\mathcal{E}$ is the set of all possible edges in aggregated form, and $\mathbf{Z}$ is the final temporal graph representation of all nodes. We use the two seperate GRU models to extract edge and node dynamics, and employ the GCN model to learn the graph representation. The learned feature attention in this continuous time representation is dynamic, unlike the static attention in discrete representations.
Note that in the original formulation, $f^{node}$ in Eq.~\ref{eq:node_dyn} is a function of only $\mathbf{X}^{t_w:t}$. However, we argue that the control variables provide information about the operating condition of the system and are different from the dynamics of the measurements. Therefore, they should be modeled differently.
\subsection{Baselines}
% TODO: link to the result section, which categorizes methods into three groups
For a comprehensive evaluation of our proposed method, we compare its performance to a range of state-of-the-art novel anomaly detection methods,
which can be classified into two primary categories: prediction-based and reconstruction-based methods.
Specifically, we group the methods with their abilities to model MTS.
Our objective is to examine the effectiveness of each of these techniques in identifying anomalies, specifically those that arise from changing relations within dynamic systems. This evaluation requires an understanding of the interdependencies between signals as they evolve over time.
\input{tabs/baselines_tab}
%\begin{itemize}
\textbf{FNN}: The Feed-forward Neural Network-based anomaly detection method establishes a mapping between control variables and measurement variables, providing a basic model of the system's internal relationships. It should be noted that this approach does not incorporate temporal information, and may potentially miss dynamic anomalies.
\textbf{AE}~\cite{aggarwaloutlier}: Autoencoders, similar to FNN, attempt to capture interdependencies within the data by reconstructing input data and detecting anomalies based on higher reconstruction errors. Similarly to FNN, they do not factor in temporal information.
\textbf{LSTM}~\cite{malhotra2015lstmad}: Long Short-Term Memory Networks (LSTM) capture temporal patterns within the data and identify anomalies through deviations from predicted values. While they effectively incorporate temporal information, LSTMs do not distinguish between the control variables and the measurement variables and may not model complex interdependencies between the sensors as effectively.
\textbf{ENCDEC-AD}~\cite{malhotra2016encdecad}: The LSTM-based encoder-decoder architecture seeks to capture both temporal and inter-dependencies, providing a more comprehensive model of system behavior. It takes sequences of control and measurement variables as input and outputs control and measurement variables
\textbf{MTADGAT}\cite{zhao2020mtadgat}: Multivariate Time-series Anomaly Detection via Graph Attention Network combines feature-oriented GAT\cite{velivckovic2017gat} and time-oriented GAT to handle spatial dependencies and temporal dependencies simultaneously. By performing forecasting and reconstruction concurrently, MTADGAT can model complex relationships and dynamics.
It does not distinguish control from measurement variables and treats them as identical types of nodes in the graph construction.
\textbf{GDN}~\cite{deng2021gdn}: Graph Deviation Network (GDN) has demonstrated to be particularly good in modeling intricate relationships in the data. It constructs a static graph based on the cosine similarity of the training batch data, uses sensor embedding to distinguish sensor types, and employs an attention mechanism for forecasting. Like MTADGAT, control variables are included in the learned graph as measurement variables.
%\end{itemize}
These baseline methods were carefully selected to test their ability to detect different types of anomalies, each with unique strengths in detecting different types of anomalies - those where temporal relationships are crucial and those primarily driven by inter-dependencies between signals. Consequently, they provide a comprehensive comparison for assessing the performance of our proposed method, \revtext{TimeGAT}.
Hyperparameter tuning for these baseline models was performed separately for both the synthetic and the industrial (Pronto) datasets to ensure optimal performance in each case study.
It is important to note that, for MTADGAT, we opted for a modification of the original implementation, adopting the more expressive GATv2~\cite{brody2022gatv2} instead of the original GAT.
The selection of the final model architectures was based on their Mean Absolute Error (MAE) on the validation sets. Detailed information about the hyperparameters used for each model can be found in Appendix~\ref{sec:app_hyperparameters}.
% anomaly score evaluations
For all the baselines, we employ the same anomaly score function introduced by GDN~\cite{deng2021gdn}. To account for the varying characteristics of the residuals of different sensors and prevent any single sensor from dominating the anomaly detection process, the anomaly score $s_i^t$ of a signal $i$ at time $t$ is a normalized residual $r_i^t$, defined as:
\begin{equation}
s_i^t= \frac{ |r_i^t - \Tilde{\mu_i}|}{\Tilde{\sigma_i}}
\end{equation}
where $\Tilde{\mu_i}$ and $\Tilde{\sigma_i}$ are the median and inter-quartile range (IQR) of the validation set error.
Please note that in their case study, the author of the GDN paper \cite{deng2021gdn} utilizes a max aggregator over all nodes, which is suitable for datasets primarily comprising point anomalies.
However, in our study where functional relation changes may affect multiple signals, we employ a mean aggregator to capture the collective impact on the signals.
Additionally, for ENCDEC-AD and MTAD-GAT, we also average the anomaly score over time.
%\subsubsection{Fault Types Specific Performance on Pronto Dataset}
\label{sec:pronto-fault}
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\linewidth]{figs/pronto/pronto_auc.pdf}
\caption{Comparative fault detection performance analysis on the Pronto Dataset under different fault types.}
\label{fig:pronto-result}
\end{figure}
\revtextrewrite{
We evaluate the performance of DyEdgeGAT on the Pronto dataset, considering three distinct fault types: air leakage, air blockage, and diverted flow. Figure~\ref{fig:pronto-result} shows the AUC scores for each fault type, as well as the average performance across all fault types. Baseline models exhibit varying performances for each fault type. However, DyEdgeGAT consistently demonstrates strong performance across all three fault types, highlighting its robust ability to detect relationship shifts for fault detection.
Air leakage is the easiest fault to detect, followed by air blockage and diverted flow. This observation is further supported by the performance of all models with relatively high AUC scores for air leakage but relatively low AUC scores for the other two fault types.
\textbf{Air Leakage.} Air leakage induces a cyclic behavior in the multiphase flow facility~\cite{stief2019pronto}, creating a distinct pattern that deviates significantly from the normal behavior. As shown in Figure~\ref{fig:pronto_dist}, the process variable PIC501 exceeds its normal range during air leakage. Therefore, detecting air leakage can be seen as a ``point anomaly'', making it easier to detect. Even though DyEdgeGAT does not perform the best on air leakage detection, it still demonstrates competitive performance.}
\revtext{\textbf{Air Blockage.} For this fault type, DyEdgeGAT outperforms all other baselines by a significant margin. Unlike the air leakage case, air blockage does not cause a reduction in pressure or a drop in the system. The flow remains continuous, making this fault type more challenging to detect.
In this class, FNN performs poorly, suggesting that modeling the static functional relationship alone is not sufficient. It is important to track the evolution of these relationships over time.}
\revtextrewrite{\textbf{Diverted Flow.}
Diverted Flow. This fault type is the most challenging for all baselines to detect, with all methods underperforming except for FNN and DyEdgeGAT. Graph-based models perform particularly poorly on this task.
Diverted flow induces noticeable changes in the flow regime within the multiphase flow facilities, leading to significant shifts in the functional relationships within the system. This explains why FNN performs well, as it can model these functional relationships.
The inability of other methods, even those that capture both temporal dynamics and spatial interdependencies, to detect this fault type suggests that the time series remain within their normal range and exhibit typical temporal patterns. This further emphasizes the need to model and capture changes in functional relationships within the system.
The significant drop in the performance of graph-based models on this task demonstrates the limitation of treating all variables as the same type of nodes in modeling relationships. This highlights the need to distinguish between variable types and incorporate operating condition contexts.
In conclusion, DyEdgeGAT consistently outperforms baseline models in detecting a variety of faults, demonstrating its potential for broad applicability in complex industrial systems.}
%This could be attributed to the design of its anomaly score, which primarily focuses on the overall deviation of the system state, emphasizing particularly on shifts in relationships between signals.
%In this scenario, the mean deviation arises due to changes in specific sensor values, such as that of pressure. Even though this fault introduces shifts in the relationships within the system, it primarily acts as a point anomaly rather than a collective anomaly.
%In the evaluation of the Pronto dataset, we consider three distinct fault types for detection.
%Figure~\ref{fig:pronto-result} illustrates the AUC score for each fault type, as well as the average performance across all classes on the Pronto dataset.
%Baseline models exhibit varying performances for each fault type, but notably, our proposed TimeGAT model consistently demonstrates strong performance across all fault types, highlighting its robust ability to detect relationship shifts in multivariate time series data.
%As shown in Fig.~\ref{fig:pronto_tsne}, \textit{diverted flow} is the most challenging fault to detect, while \textit{air leakage} is the easiest.
%This observation is further supported by the performance of all models, which achieve relatively high AUC values for the \textit{air leakage} fault type but struggle to attain high scores for the \textit{diverted flow} fault type.
%For this fault type, all methods achieved relatively high AUC scores.
%The air leakage fault induces a cyclic behavior in the multiphase flow facility~\cite{stief2019pronto}, creating a distinct pattern that deviates significantly from the normal behavior.
%As shown in Fig.~\ref{fig:pronto_dist}, the process variable PIC501 exceeds its normal range during air leakage. Therefore, detecting this fault type can be seen as a \textit{point anomaly}, which is easier to detect.
%Even though not performing the best, TimeGAT still demonstrates competitive fault detection performance for this fault type.
%LSTM-AE, MTADGAT, and \revtext{TimeGAT} exhibit similar performance in terms of AUC, with \revtext{TimeGAT} leading by a small margin. However, \revtext{TimeGAT} achieves a higher F1 score compared to these methods, despite having a similar AUC score.
%As a result, all methods exhibit relatively low AUC scores for this fault type.
%This fault type is the hardest on average for all baselines to detect. Except for FNN and TimeGAT, all methods underperform, especially for graph-based methods.
%The diverted flow induces noticeable differences in the flow regime within the multiphase flow facilities, leading to significant shifts in the functional relationships within the system, this explains why FNN is performing well.
%The inability of all other methods, even those capturing both temporal dynamics and spatial interdependencies, to detect this fault type reveals that the time series remain within their normal range and exhibit typical temporal patterns.
%This further emphasizes the necessity of modeling and capturing changes in functional relationships within the system.
%The significant drop in performance of graph-based methods in this class further shows the limitation of treating all variables as same type of nodes in modeling relationships.
%This further emphasizes the necessity of distinguishing variable types and incorporating operating condition contexts.
%In conclusion, \revtext{TimeGAT} consistently demonstrates robust and superior performance across all fault types, demonstrating its effectiveness in detecting a variety of faults. It showed exceptional capability in identifying faults involving shifts in functional relationships within the MTS data. This underscores \revtext{TimeGAT}'s potential for broad applicability in complex industrial systems.
To merge and revise the two parts while ensuring a smooth flow, the structure can be organized to first detail each fault type's specific detection challenges and DyEdgeGAT's performance, followed by a summary of the overall performance. Here's the revised text:
"\textbackslash{}subsubsection\{Fault Detection Performance Across Various Fault Types\}
We assessed DyEdgeGAT's performance on the Pronto dataset across three distinct fault types: air leakage, air blockage, and diverted flow. Figure\~\ref\{fig:pronto-result\} illustrates the AUC scores for each fault type and their average. While baseline models show varied performances for each fault type, DyEdgeGAT consistently exhibits strong performance across all, demonstrating its robustness in detecting relationship shifts.
\textbackslash{}textbf\{Air Leakage.\} Characterized by cyclic behavior in the multiphase flow facility, air leakage creates a noticeable deviation from normal patterns, making its detection relatively straightforward. DyEdgeGAT, though not the top performer in this category, still showcases competitive performance.
\textbackslash{}textbf\{Air Blockage.\} Here, DyEdgeGAT surpasses all baselines significantly. The continuous flow nature of air blockage, without obvious pressure drops, makes it a more complex fault to detect. The poor performance of FNN indicates the necessity of modeling evolving relationships over time for effective detection.
\textbackslash{}textbf\{Diverted Flow.\} This fault presents the most challenge, with only FNN and DyEdgeGAT performing adequately. Diverted flow leads to marked shifts in functional relationships, explaining FNN's success. The struggle of other models, including those adept in capturing temporal and spatial aspects, underscores the need for a nuanced approach in modeling variable types and incorporating operational contexts.
In summary, DyEdgeGAT's consistent outperformance across varied faults underscores its potential in complex industrial systems.
\textbackslash{}subsubsection\{Overall Performance Evaluation\}
As detailed in Tab.\~\ref\{tab:pronto\_results\}, DyEdgeGAT leads in AUC, F1, and F1\$^*\$ scores on the Pronto dataset and has the second-lowest detection delay. This points to its timely fault detection capability. Two key observations stand out:
\textbackslash{}textbf\{Importance of Functional Relationship Modeling\}: Models like FNN and AE, focusing on functional relationships, fare better than those emphasizing temporal dynamics. This suggests that changes in functional relationships are more critical than dynamic shifts in the Pronto dataset's context.
\textbackslash{}textbf\{Suboptimal Performance of GNN Models\}: GDN and GRELEN, the graph-based models, underperform, likely due to their inability to differentiate between system-dependent and independent variables. In the Pronto dataset, this distinction is vital, as faults don't affect system-independent variables in the same manner as system-dependent ones. The failure to make this distinction makes GNN methods prone to false fault detections under novel operating conditions."
\subsection{RQ3: Ablation Study}
The goal of this ablation study is to evaluate the influence of each component of our proposed \revtext{TimeGAT} method on the overall performance. Through this analysis, we validate the methodological choices made in the design of \revtext{TimeGAT}.
Table~\ref{tab:all_ablation} reports the results of the ablation study on both synthetic and Pronto datasets. We consider four different ablation scenarios, indicated by the labels ``- w/o contr aug'', ``- w/o time enc'', ``- w/o att score'', and ``- w/o dyn edge'', which signify the removal of a specific component from the \revtext{TimeGAT} framework. Each label is explained as follows:
\begin{itemize}
\item \textbf{w/o contr aug}: The control parameter augmentation feature is excluded from \revtext{TimeGAT}. Without this component, the framework does not differentiate control parameters from other nodes, and they are treated as standard measurement nodes.
\item \textbf{w/o time enc}: This scenario omits the temporal encoding feature in \revtext{TimeGAT}, which could potentially weaken the model's ability to identify temporal dependencies.
\item \textbf{w/o att score}: In this setting, \revtext{TimeGAT} does not include the attention-based anomaly score, leading to uniform weights being assigned to all normalized node residuals. This absence may affect the effectiveness of anomaly detection.
\item \textbf{w/o dyn edge}: The dynamic edge construction component is removed. Instead, the attention mechanism from MTADGAT is used for feature graph construction, and the top-k mechanism from GDN is implemented to learn a sparse representation of the graph.
\end{itemize}
Examining \revtext{TimeGAT}'s performance under these scenarios allows us to evaluate the individual contributions of each component to the method's overall efficacy.
% pronto dataset
The findings for the Pronto dataset parallel those from the synthetic dataset. \revtext{TimeGAT} achieves the best performance, with the exclusion of the attention-based anomaly score resulting in a minor performance drop. The removal of the temporal encoding, control parameter augmentation, and dynamic edge construction, however, leads to a noticeable decrease in AUC and F1 scores, thus stressing the importance of these components.
% conclusion
In summary, this ablation study confirms that the attention-based anomaly score, temporal encoding, control parameter augmentation, and dynamic edge construction are all vital contributors to the superior performance of \revtext{TimeGAT}. Among these, the dynamic edge construction's absence appears to impact the AUC scores most significantly, emphasizing its role in modeling evolving relationships and intricate dependencies. Hence, this study substantiates the crucial importance of dynamic edge construction in improving the anomaly detection performance of our proposed \revtext{TimeGAT} method.
\section*{Nomenclature}
\addcontentsline{toc}{section}{Nomenclature}
\begin{IEEEdescription}[\IEEEusemathlabelsep\IEEEsetlabelwidth{$W = t-t_w + 1$}]
\item[Notion] Description
\item[$t \in \R$] current time.
\item[$t_w\in \R$] start time of the observation window.
\item[$\mathbf{X}^{t_w:t}\in\R^{N\times W}$] sliced measurement from time $t_w$ to $t$.
\item[$\mathbf{x}^{t_w:t}_{i} \in \mathbb{R}^{W}$] $i$-th measurement from time $t_w$ to $t$.
\item[${x}_{i}^{t}\in \R$] $i$-th sensor recording at time $t$.
\item[$N\in \N$] number of measurement variables.
\item[$\mathbf{U}^{t_w:t}\in\R^{N_u\times W}$] control variables from time $t_w$ to $t$.
\item[$\mathbf{u}^{t_w:t}_i \in \mathbb{R}^{W}$] $i$-th control variables from time $t$ to $t_w$.
\item[$N_u\in \N$] number of control variables.
\item[$W = t-t_w + 1$] length of the observation window.
\item[$\mathcal{V}=\{v_i\}$] set of nodes representing sensors.
\item[$\mathcal{E}=\{e_{ij}\}$] set of edges representing interactions between sensors.
\item[$\mathcal{G}=(\mathcal{V}, \mathcal{E})$] graph with nodes $\mathcal{V}$ and edges $\mathcal{E}$.
\item[$\delta t \in \R$] length of sliding window in dynamic edge construction.
\item[$\mathbf{A}^{t}\in \R^{|\mathcal{V}|\times|\mathcal{V}|}$] dynamic edge attributes at time $t$.
\item[$\mathbf{H}^{node}\in \R^{|\mathcal{V}|\times p}$] hidden node dynamic representation.
\item[$\mathbf{H}^{edge}\in \R^{|\mathcal{E}|\times p}$] hidden edge dynamic representation.
\item[$p\in \N$] dimension of node or edge attributes.
\item[$\mathbf{Z}\in \R^{|\mathcal{V}|\times p}$] hidden temporal graph representation.
\item[$\alpha_{ij}^t\in \R$] attention coefficient between node $i$ and $j$ at time $t$.
\end{IEEEdescription}
\documentclass[12pt,a4paper,roman]{moderncv}
\usepackage[english]{babel}
\moderncvstyle{classic}
\usepackage{xspace}
\usepackage[utf8]{inputenc}
\usepackage[scale=0.8]{geometry}
%\usepackage{hyperref}
\def\eg{e.g.,\ }
\def\Eg{E.g.,\ }
\def\cf{\textit{cf.}\ }
\def\ie{\textit{i.e.,\ }}
\def\hc{\string:\ }
\newcommand{\titleName}{DyEdgeGAT: Dynamic Edge via Graph Attention for Early Fault Detection in IIoT Systems}
% personal data
\name{Prof. Olga}{Fink}
\address{Intelligent Maintenance and Operations Systems\\ IMOS}{EPFL Lausanne}{}% optional, remove / comment the line if not wanted; the "postcode city" and and "country" arguments can be omitted or provided empty
%\phone[mobile]{} % optional, remove / comment the line if not wanted
%\phone[fixed]{+2~(345)~678~901} % optional, remove / comment the line if not wanted
%\phone[fax]{+3~(456)~789~012} % optional, remove / comment the line if not wanted
\email{olga.fink@epfl.ch} % optional, remove / comment the line if not wanted
%\social[linkedin][linkedin.com/in/ID]{Title}
% \homepage{www.johndoe.com} % optional, remove / comment the line if not wanted
%\extrainfo{additional information} % optional, remove / comment the line if not wanted
%\photo[64pt][0.4pt]{picture} % optional, remove / comment the line if not wanted; '64pt' is the height the picture must be resized to, 0.4pt is the thickness of the frame around it (put it to 0pt for no frame) and 'picture' is the name of the picture file
%\quote{Some quote} % optional, remove / comment the line if not wanted
\begin{document}
%----- letter ---------------------------------------------------------
% recipient data
% \def\namerecipient{Liesbeth Venema}
% \recipient{\namerecipient,}{Chief Editor,\\Nature Machine Intelligence}
% \def\journalname{Nature Machine Intelligence\xspace}
\def\namerecipient{Dr. Nei Kato}
\recipient{\namerecipient}{Editor in Chief,\\IEEE Internet of Things Journal}
\def\journalname{IEEE Internet of Things Journal, Special Issue on Prognostics and Health Management using the Internet of Things\xspace}
\date{\today}
\opening{Dear \namerecipient}
% Yours faithfully (to unknown person on business)
% Yours truly (to slight acquaintance)
% Yours very truly (ceremonious but cordial)
% Yours sincerely (in invitations and friendly but not intimate letters)
\closing{Yours sincerely,}
%\enclosure[Enclosed]{CV} % use an optional argument to use a string other than "Enclosure", or redefine \enclname
\makelettertitle
We are pleased to submit the manuscript entitled ``\titleName'' by Mengjie Zhao and Olga Fink for consideration by \journalname.
We believe the manuscript is well suited for the special issue because the paper's focus on early fault detection in Industrial Internet of Things (IIoT) sensor networks aligns well with the special issue's emphasis on novel solutions for the next evolution of PHM technologies.
We propose a novel graph-based framework specifically designed for early fault detection in the IIoT sensor networks. This framework is especially effective in scenarios characterized by relationship changes between sensor signals.
Our work has the potential to significantly impact the field of PHM by providing novel methodologies for improving system reliability and preventing substantial failures.
Fault detection in IIoT systems is challenging due to strong nonlinear and stochastic spatial-temporal dynamics within the complex system. Although previously proposed methods effectively model these dynamics, they often neglect the dynamic evolution of relationships between sensor signals. Undetected shifts in these relationships can lead to significant system failures. Another limitation is that these methods often mistakenly identify novel operating conditions as faults.
To address this problem, we propose \textbf{DyEdgeGAT} (\textit{Dynamic Edge via Graph Attention}), a novel approach capable of detecting various faults, especially those characterized by relationship changes at a very early stage, and distinguishing actual faults from changes in operating conditions.
DyEdgeGAT is a graph-based framework that offers a novel graph inference scheme for multivariate time series that dynamically constructs edges to represent and track the evolution of relationships between time series.
It also addresses a commonly overlooked aspect: the cause-and-effect relationships within the system, such as between control inputs and measurements. By incorporating the system-independent variables as contexts of operating conditions into node dynamics extraction, DyEdgeGAT enhances robustness against novel operating conditions.
Our main contributions are two-fold:
\begin{enumerate}
\item We introduce a new graph inference scheme for multivariate time series, which dynamically constructs edges between time series, unlike the commonly employed one-time graph inference per input sequence. While existing unsupervised fault and anomaly methods focus on changes in the sensor level (\textit{node level}), our method uniquely emphasizes changes between sensors (\textit{edge level}). Consequently, this graph representation allows our model to capture the evolution of relationships between signals, which is indicative of early faults.
\item We are the first to separate system-independent variables from the measurements in graph construction. This allows us to distinguish faults from novel operation conditions, where many state-of-the-art methods mistakenly identify them as faults. We demonstrated this contribution with an additional experiment in the industrial case study.
\end{enumerate}
We evaluated the performance of the proposed method using both a synthetic dataset with controlled varying fault severity levels and an industrial-scale multiphase flow facility benchmark featuring various fault types with different detection difficulties. We introduced a synthetic dataset because it offered a unique opportunity for us to evaluate the model’s performance for different fault severities and early fault detection performance.
Our proposed approach demonstrated superior performance in fault detection for IIoT sensor networks, showing particular strength in early-stage fault detection, even in the case of faults with minimal severity. In particular, our method is capable of distinguishing novel operating conditions from actual faults.
We performed a comprehensive ablation study to demonstrate the contribution of each component of the proposed method and compared the proposed framework to state-of-the-art graph-based anomaly detection methods. Furthermore, we have included a sensitivity analysis on the impact of separating system-independent variables as well as an experiment to demonstrate the robustness of our method to novel operating conditions, which is a common challenge in industrial systems and where previous methods have struggled significantly.
We believe that the proposed framework will be highly valuable in inferring graph structure from IIoT sensor network time series and for detecting faults characterized by relationship shifts.
We confirm that this manuscript is original and has not been published before. Furthermore, we confirm that there are no known conflicts of interest associated with this publication. The manuscript has been read and approved by all the named authors.
Please address all correspondence concerning this manuscript to me at \makeatletter\@email\makeatother.
Thank you for your consideration of this manuscript.
%\vspace{0.2cm}
\makeletterclosing
\end{document}
%DyEdgeGAT addresses the challenge of static graph inference by dynamically inferring edges between time series, and constructing an aggregated temporal graph for MTS data. This enables us to capture not
%only node dynamics (i.e., \textit{the intrinsic dynamics of a sensor}) but also edge dynamics (i.e., \textit{the evolving relationships between sensors}).
% !TEX root = main.tex
%\documentclass[10pt,journal]{IEEEtran}
\documentclass[lettersize,journal]{IEEEtran}
%\input{auxiliary/setup_iot}
% my comments
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\N}{\mathbb{N}}
\usepackage{multirow, booktabs}
\usepackage[export]{adjustbox}
\usepackage[dvipsnames, table, xcdraw]{xcolor}
\newcommand{\revtextiot}[1]{{\color{black}#1}}
%\newcommand{\revtextrewrite}[1]{{\color{OliveGreen}#1}}
%\definecolor{revcolor}{RGB}{0, 0, 255}
\definecolor{revcolor}{RGB}{0, 0, 0}
\newcommand{\revtext}[1]{{\color{black}#1}}
\newcommand{\revtextrewrite}[1]{{\color{black}#1}}
\usepackage{ifpdf}
\usepackage{cite}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{algorithmic}
\usepackage{array}
\usepackage{booktabs} % professional-quality tables
\graphicspath{{figs/}} % organize your images and other figures under
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage{cite}
\hyphenation{op-tical net-works semi-conduc-tor IEEE-Xplore}
\usepackage{textcomp}
\usepackage{stfloats}
%\input{chaps/00-cover}
\begin{document}
\input{chaps/00-cov-abs/00-cover-ieee}
\maketitle
\input{chaps/00-cov-abs/00-abstract}
%\input{chaps/notations}
\input{chaps/01-intro/01-introduction}
\input{chaps/02-rw/02-related_work}
%\input{chaps/03-fw/03-0-preliminaries}
\input{chaps/03-fw/03-framework}
\input{chaps/04-doe/04-1-casestudies}
\input{chaps/04-doe/04-experiment}
\input{chaps/05-results/05-results}
\input{chaps/06-conclusion}
% TODO: change the title to include a graph (probably with oc)
% dynamically inferring graph structure from MTS
\section{Data and Code Availability}
% reproducibility
The script we used to generate this synthetic dataset is available in the associated code repository to ensure reproducibility and facilitate further studies. Our code and data will be made available after acceptance of the
manuscript under \url{https://github.com/MengjieZhao/dyedgegat}.
\section*{Acknowledgments}
This work was supported by the Swiss National Science Foundation under Grant 200021\_200461.
ChatGPT has been used to correct the grammar of the text and for proofreading.
%\input{chaps/07-appendix}
% Bibliography
\bibliographystyle{IEEEtran}
\bibliography{references}
\input{chaps/08-biography}
\end{document}
\begin{table}[tbhp]
{\color{revcolor}
\caption{Variation of Ratios of Fault Samples ($\alpha$) Across Different Scaling Factors in the Synthetic Dataset}
\label{tab:fault_rate_toy}
\centering
\begin{tabular}{lcccccccc}
\toprule
& 0.5 & 0.75 & 0.9 & 0.95 & 1.05 & 1.1 & 1.5 & 2.0 \\
\midrule
\( \alpha \) & 0.354 & 0.407 & 0.301 & 0.302 & 0.334 & 0.378 & 0.362 & 0.425 \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[tbhp]
{\color{revcolor}
\caption{Ratios of Fault Samples ($\alpha$) for Different Types of Anomalies in the Pronto Dataset}
\label{tab:fault_rate_pronto}
\centering
\begin{tabular}{lcccc}
\toprule
& Air Leakage & Air Blockage & Diverted Flow & Slugging \\
\midrule
\( \alpha \) & 0.596 & 0.613 & 0.673 & 0.463 \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table*}[tbh]
\centering
\caption{Summary of baseline methods for multivariate time series anomaly detection and their model input and output.}\label{tab:baseline_methods}
%\begin{tabular}{llp{11cm}p{1.2cm}p{1.5cm}}
\begin{tabular}{p{0.1mm}p{1.8cm}p{11cm}p{1.5cm}p{1.5cm}}
\toprule
& \textbf{Method} & \textbf{Description} & \textbf{Input} & \textbf{Output} \\
\midrule
\multicolumn{3}{l}{\textit{Relationship-focused}} \\
& FNN & A Feed-Forward Neural Network (FNN) based anomaly detection method that maps control variables to measurement variables, providing a basic model of the system’s internal relationships. & $\mathbf{U}^t$ & $\mathbf{X}^t$ \\
& AE~\cite{aggarwaloutlier} & An Autoencoders (AE), similar to FNN, attempts to capture interdependencies within the data by reconstructing input data and detecting anomalies based on higher reconstruction errors & $\left[\mathbf{U}^t\parallel\mathbf{X}^t\right]$ & $\left[\mathbf{U}^t\parallel\mathbf{X}^t\right]$ \\
\midrule
\multicolumn{3}{l}{\textit{Dynamics-focused}} \\
& LSTM\cite{malhotra2015lstmad} & A Long Short-Term Memory Network mainly captures temporal patterns and weakly the inter-dependencies in MTS and identifies anomalies via deviations from predicted values. & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[\mathbf{U} \parallel \mathbf{X}]\right]^{t+1}$\\
& LSTM-AE~\cite{malhotra2016encdecad} & An LSTM-based encoder-decoder architecture seeks to capture both temporal and inter-dependencies in the MTS. & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ \\
& \revtext{USAD~\cite{audibert2020usad}} & Unsupervised \revtext{Anomaly Detection (USAD) consists of two AEs trained adversarially on the MTS data. The output of the first AE is processed by the second AE, aiming to amplify detected anomalies.} & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ \\
\midrule
\multicolumn{3}{l}{\textit{Graph-based}} \\
& GDN~\cite{deng2021gdn} & Graph Deviation Network (GDN) constructs a static graph based on the cosine similarity of the training batch data, using sensor embedding to distinguish sensor types, and employing an attention mechanism for forecasting. & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[\mathbf{U} \parallel \mathbf{X}]\right]^{t+1}$ \\
& MTADGAT~\cite{zhao2020mtadgat} & Multivariate Time-series Anomaly Detection via Graph Attention Network (GAT) combines feature-oriented GAT\cite{velivckovic2017gat} and time-oriented GAT to handle spatial dependencies and temporal dependencies simultaneously. By performing forecasting and reconstruction concurrently, MTADGAT can model complex relationships and dynamics. & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$, $\left[\mathbf{U} \parallel \mathbf{X}]\right]^{t+1}$ \\
& \revtext{GRELEN~\cite{zhang22grelen}} & Graph \revtext{Relational Learning Network (GRELEN) utilizes a variational AE-based reconstruction module to dynamically infer graph structures and is the first approach to leverage learned graphs to detect anomalies from the relational discrepancy}. & $\left[ \mathbf{U} \parallel \mathbf{X} \right]^{t_w:t}$ & $\left[\mathbf{U} \parallel \mathbf{X}]\right]^{t_w+1:t}$ \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table}
\centering
\caption{Ablation study on synthetic and pronto datasets with 5 runs}
\begin{tabular}{llccc}
\toprule
& & \textbf{AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{Delay} $\downarrow$ \\
\midrule
\multirow{5}{*}{\rotatebox[origin=c]{90}{\textbf{Synthetic}}} & \revtext{TimeGAT} & 0.874 ± 0.035 & 0.752 ± 0.045 & 14.3 ± 6.6 \\
& - w/o att score & 0.863 ± 0.036 & 0.762 ± 0.052 & 2.2 ± 6.5 \\
& - w/o time enc & 0.868 ± 0.027 & 0.694 ± 0.145 & 14.6 ± 6.9 \\
& - w/o contr aug & 0.831 ± 0.041 & 0.339 ± 0.165 & 72.2 ± 6.4 \\
& - w/o dyn edge & 0.803 ± 0.061 & 0.598 ± 0.234 & 17.0 ± 24.2 \\
\midrule
\multirow{5}{*}{\rotatebox[origin=c]{90}{\textbf{Pronto}}} & \revtext{TimeGAT} & 0.766 ± 0.049 & 0.790 ± 0.011 & 36.5 ± 9.9 \\
& - w/o att score & 0.758 ± 0.046 & 0.787 ± 0.012 & 30.0 ± 0.0 \\
& - w/o time enc & 0.681 ± 0.090 & 0.786 ± 0.014 & 36.9 ± 12.2 \\
& - w/o contr aug & 0.731 ± 0.065 & 0.785 ± 0.015 & 48.5 ± 9.3 \\
& - w/o dyn edge & 0.677 ± 0.105 & 0.773 ± 0.043 & 44.1 ± 12.0 \\
\bottomrule
\end{tabular}
\label{tab:all_ablation}
\end{table}
% Please add the following required packages to your document preamble:
\begin{table}
\centering
\caption{Overall performance comparison on synthetic and pronto datasets with 5 runs. Best metric score is highlighted by bold.}
\begin{tabular}{llccc}
\toprule
& & \textbf{AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{Delay} $\downarrow$ \\
\midrule
\multirow{7}{*}{\rotatebox[origin=c]{90}{\textbf{Synthetic}}} & AE & 0.774 ± 0.025 & 0.584 ± 0.031 & 11.9 ± 2.4 \\
& FNN & 0.803 ± 0.018 & 0.572 ± 0.016 & 10.6 ± 0.7 \\
& LSTM & 0.814 ± 0.034 & 0.622 ± 0.001 & \textbf{10.0 ± 0.0}\\
& ENCDEC-AD & 0.832 ± 0.045 & 0.631 ± 0.015 & \textbf{10.0 ± 0.0} \\
& GDN & 0.795 ± 0.079 & \underline{0.667 ± 0.124} & 63.3 ± 14.5 \\
& MTADGAT & \underline{0.864 ± 0.018} & 0.621 ± 0.000 & \textbf{10.0 ± 0.0} \\
& \revtext{TimeGAT} & \textbf{0.874 ± 0.035} & \textbf{0.752 ± 0.045} & 24.3 ± 6.6 \\
\midrule
\multirow{7}{*}{\rotatebox[origin=c]{90}{\textbf{Pronto}}} & AE & 0.698 ± 0.019 & 0.562 ± 0.074 & 2928 ± 667 \\
& FNN & 0.646 ± 0.054 & 0.421 ± 0.020 & 3499 ± 71 \\
& LSTM & 0.672 ± 0.078 & \underline{0.765 ± 0.047} & \underline{38 ± 10} \\
& ENCDEC-AD & \underline{0.748 ± 0.034} & 0.569 ± 0.034 & 3399 ± 100 \\
& GDN & 0.498 ± 0.046 & 0.318 ± 0.013 & 4315 ± 559 \\
& MTADGAT & 0.696 ± 0.055 & 0.452 ± 0.056 & 3450 ± 1 \\
& \revtext{TimeGAT} & \textbf{0.766 ± 0.049} & \textbf{0.790 ± 0.011} & \textbf{37 ± 10} \\
\bottomrule
\end{tabular}
\label{tab:all_result}
\end{table}
\begin{table}
\centering
\caption{Ablation study on pronto dataset with 5 runs}
\label{tab:syn_ablation}
\begin{tabular}{llll}
\toprule
& AUC & F1 & Delay \\
model & & & \\
\midrule
\revtext{TimeGAT} & 0.747 ± 0.072 & 0.790 ± 0.011 & 31.0 ± 0.0 \\
- topol ad_score & 0.716 ± 0.081 & 0.769 ± 0.012 & 31.0 ± 0.0 \\
- time enc & 0.621 ± 0.138 & 0.726 ± 0.083 & 31.0 ± 0.0 \\
- contr aug & 0.648 ± 0.075 & 0.776 ± 0.038 & 31.0 ± 0.0 \\
- temp graph & 0.588 ± 0.140 & 0.639 ± 0.126 & 522.4 ± 972.8 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}
\centering
\caption{Performance comparison on pronto dataset with 5 runs. Best metric score is highlighted by bold}
\label{tab:pronto}
\begin{tabular}{llll}
\toprule
& AUC & F1 & Delay \\
model & & & \\
\midrule
AE & 0.691 ± 0.026 & 0.508 ± 0.051 & 3486 ± 15 \\
FNN & 0.633 ± 0.031 & 0.407 ± 0.009 & 3493 ± 0 \\
LSTM & 0.613 ± 0.104 & 0.749 ± 0.065 & \textbf{31 ± 0} \\
ENCDEC-AD & 0.701 ± 0.032 & 0.521 ± 0.048 & 3480 ± 1 \\
GDN & 0.423 ± 0.035 & 0.313 ± 0.014 & 4574 ± 457 \\
MTADGAT & 0.701 ± 0.040 & 0.516 ± 0.060 & 3479 ± 0 \\
\revtext{TimeGAT} & \textbf{0.747 ± 0.072} & \textbf{0.790 ± 0.011} & 31 ± 0 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[tbhp]
\centering
\caption{Performance comparison on pronto dataset with 5 runs. The best metric score is highlighted in bold and the second best model in underscor}
\label{tab:pronto_agg_results}
\begin{tabular}{llccc}
\toprule
& & \textbf{AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{Delay} $\downarrow$ \\
\midrule
\multicolumn{5}{l}{\textit{Only inter-dependencies}} \\
& FNN & 0.633 ± 0.031 & 0.407 ± 0.009 & 3493 ± 0 \\
& AE & 0.691 ± 0.026 & 0.508 ± 0.051 & 3486 ± 15 \\
\midrule
\multicolumn{5}{l}{\textit{Temporal and inter-dependencies}} \\
& LSTM & 0.613 ± 0.104 & \underline{0.749 ± 0.065} & \textbf{30 ± 0} \\
& ENCDEC-AD & \underline{0.701 ± 0.032} & 0.521 ± 0.048 & 3480 ± 1 \\
\midrule
\multicolumn{5}{l}{\textit{Temporal and inter-dependencies with graphs}} \\
& GDN & 0.423 ± 0.035 & 0.313 ± 0.014 & 4574 ± 457 \\
& MTADGAT & \underline{0.701 ± 0.040} & 0.516 ± 0.060 & 3479 ± 0 \\
\midrule
\multicolumn{5}{l}{\textit{Proposed method}} \\
& \revtext{TimeGAT} & \textbf{0.747 ± 0.072} & \textbf{0.790 ± 0.011} & \textbf{30 ± 0} \\
& - topol score & 0.753 ± 0.084 & 0.789 ± 0.012 & 30 ± 0 \\
& - time enc & 0.621 ± 0.138 & 0.726 ± 0.083 & 30 ± 0 \\
& - contr aug & 0.648 ± 0.075 & 0.776 ± 0.038 & 30 ± 0 \\
& - temp graph & 0.588 ± 0.140 & 0.639 ± 0.126 & 522.4 ± 972.8 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htb]
\centering
\caption{Model Performance comparison on the synthetic dataset with 5 runs. The best metric score is highlighted in bold and the second best model in underscore}
\label{tab:syn_results}
\begin{tabular}{clccc}
\toprule
& & \textbf{AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{Delay} $\downarrow$ \\
\midrule
\multicolumn{5}{l}{\textit{Only inter-dependencies}} \\
& FNN & 0.803 ± 0.018 & 0.572 ± 0.016 & 10.6 ± 0.7 \\
& AE & 0.774 ± 0.025 & 0.584 ± 0.031 & 11.9 ± 2.4 \\
\midrule
\multicolumn{5}{l}{\textit{Temporal and inter-dependencies}} \\
& LSTM & 0.814 ± 0.034 & 0.622 ± 0.001 & \textbf{10.0 ± 0.0} \\
& ENCDEC-AD & 0.832 ± 0.045 & 0.631 ± 0.015 & \textbf{10.0 ± 0.0} \\
\midrule
\multicolumn{5}{l}{\textit{Temporal and inter-dependencies with graphs}} \\
& GDN & 0.795 ± 0.079 & \underline{0.667 ± 0.124} & 29.3 ± 16.8 \\
& MTADGAT & \underline{0.864 ± 0.018} & 0.621 ± 0.000 & \textbf{10.0 ± 0.0} \\
\midrule
\multicolumn{5}{l}{\textit{Proposed method}} \\
& \revtext{TimeGAT} & \textbf{0.874 ± 0.035} & \textbf{0.752 ± 0.045} & 24.3 ± 6.6 \\
&- w/o att score & 0.863 ± 0.036 & 0.762 ± 0.052 & 12.2 ± 6.5 \\
&- w/o time enc & 0.868 ± 0.027 & 0.694 ± 0.145 & 14.6 ± 6.9 \\
&- w/o contr aug & 0.831 ± 0.041 & 0.339 ± 0.165 & 62.2 ± 6.4 \\
&- w/o temp graph & 0.803 ± 0.061 & 0.598 ± 0.234 & 17.0 ± 24.2 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}
\centering
\caption{Model Performance comparison on the synthetic dataset with 5 runs. The best metric score is highlighted in bold and the second best model in underscore}
\label{tab:syn_results}
\begin{tabular}{lccc}
\toprule
& \textbf{AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{Delay} $\downarrow$ \\
\midrule
AE & 0.774 ± 0.025 & 0.584 ± 0.031 & 11.9 ± 2.4 \\
FNN & 0.803 ± 0.018 & 0.572 ± 0.016 & 10.6 ± 0.7 \\
LSTM & 0.814 ± 0.034 & 0.622 ± 0.001 & \underline{\textbf{10.0 ± 0.0}} \\
ENCDEC-AD & 0.832 ± 0.045 & 0.631 ± 0.015 & 10.0 ± 0.0 \\
GDN & 0.795 ± 0.079 & \underline{0.667 ± 0.124} & 29.3 ± 16.8 \\
MTADGAT & \underline{0.864 ± 0.018} & 0.621 ± 0.000 & 10.0 ± 0.0 \\
\revtext{TimeGAT} & \textbf{0.874 ± 0.035} & \textbf{0.752 ± 0.045} & 24.3 ± 6.6 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table*}[htbp]
\centering
{\color{revcolor}
\caption{Range of Hyperparameters for the applied models. The optimal hyperparameters are indicated within parentheses.}
\label{table:hyperparameters}
\begin{tabular}{lcccccc}
\toprule
\textbf{Model} & \textbf{L} & \textbf{H} & \textbf{Norm.} & \textbf{Additional Parameters} & \textbf{\revtext{No. Params}} \\ \midrule
FNN & 3-5 ($3\mid7$) & 10-50 ($10\mid 50$) & BN & first hid. dim. 20-50 ($20\mid 50$) & $515 \mid 16811$ \\
AE & 3-$5^*$ & 5-$20^*$ & BN & - & $532 \mid 3319$ \\
LSTM & 1-2 ($1\mid2$) & 10-40 ($20\mid40$) & LN & - & $2507 \mid 22935$ \\
LSTM-AE & 1 & 10-40 ($10\mid40$) & BN/LN (BN) & - & $1737 \mid 22935$ \\
\revtext{USAD} & 2-3 (2) & 10-20-30 ($10\mid30$) & BN/- (-) & warmup epochs 30-50 (50), final act. func. (tanh$\mid$sigmoid)& $21670 \mid 75888$ \\
MTADGAT & 1-2 (1) & 10-80 ($20\mid10$) & BN & att. embed. dim. 10-40 ($20\mid10$) & $8582 \mid 5790$ \\
GDN & - & 100-200 (200) & BN & sensor embed. dim. 20-50 ($20\mid50$) & $5421 \mid 12160$ \\
\revtext{GRELEN} & - & 10-40 ($10\mid40$) & BN & num. graphs 2-4 ($2$), dist. prior ($(0.99, 0.01) \mid (0.97, 0.03)$) & $12181 \mid 63601$ \\
\revtext{DyEdgeGAT} & 2 & 10-20 ($10\mid20$) & LN+BN & temp. embed. ($5\mid10$), edge embed. ($20\mid100$), gnn embed. ($20\mid40$) & $3921 \mid 15921 $ \\
\bottomrule
\end{tabular}
}
\end{table*}
\begin{table}[tb]
\centering
{\color{revcolor}
\caption{{Comparative Analysis of Approaches for Relationship Modelling in Multivariate Time Series}}
\label{tab:intro_overview}
\begin{tabular}{lccc}
\hline
\multirow{2}{*}{\textbf{{Type of approaches}}} & \textbf{{Spatial-}} & \textbf{{Temporal}} & \textbf{{Evolving}} \\
& \textbf{{dependencies}} & \textbf{{dynamics}} & \textbf{{relationships}} \\
\hline
{Relationship-focused} & {$\checkmark$} & {-} & {-} \\
{Dynamics-focused} & {-} & {$\checkmark$} & {-} \\
{Graph-based} & {$\checkmark$} & {$\checkmark$} & {-} \\
{DyEdgeGAT (Proposed)} & {$\checkmark$} & {$\checkmark$} & {$\checkmark$} \\
\hline
\end{tabular}
}
\end{table}
\begin{table}[htbp]
\caption{Different model configurations for ablation study of operating condition context augmentation.}
\centering
\begin{tabular}{ll}
\hline
\textbf{Acronym} & \textbf{Configuration} \\
\hline
MV & Only models the relationship between MV \\
MV+CV+EF & Models all variables in the same way \\
MV aug CV & Models MV as system-dependent and CV as system-independent variables \\
MV aug CV+EF & Models MV and EF as system-dependent variables and CV as system-independent variables \\
\hline
\end{tabular}
\label{your-label-here}
\end{table}
\begin{table}
\centering
\caption{Performance of the models on the Pronto dataset under novel operating conditions.}
\label{tab:pronto_condition}
\begin{tabular}{llllllllll}
\toprule
model & AE & FNN & LSTM & LSTM-AE & USAD & GDN & GRELEN & MTADGAT & TimeGAT \\
\midrule
\textbf{Ambiguity} & 0.56 ± 0.13 & 0.14 ± 0.01 & 0.81 ± 0.12 & 0.84 ± 0.10 & \textbf{0.90 ± 0.06} & 0.31 ± 0.06 & \underline{0.87 ± 0.06} & 0.08 ± 0.01 & 0.84 ± 0.11 \\
\textbf{FPR} & 0.03 ± 0.01 & 0.02 ± 0.01 & \underline{0.06 ± 0.03} & 0.05 ± 0.02 & 0.01 ± 0.01 & 0.00 ± 0.00 & 0.00 ± 0.00 & 0.01 ± 0.01 & \textbf{0.14 ± 0.12} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}
\centering
\caption{Performance of the models on the Pronto dataset under novel operating conditions.}
\label{tab:pronto_condition}
\begin{tabular}{lll}
\toprule
& \textbf{Ambiguity} & \textbf{FPR} \\
\midrule
AE & 0.11 ± 0.03 & 0.02 ± 0.01 \\
FNN & 0.14 ± 0.01 & 0.00 ± 0.01 \\
LSTM & 0.19 ± 0.08 & 0.00 ± 0.05 \\
LSTM-AE & 0.05 ± 0.01 & 0.00 ± 0.01 \\
USAD & \textbf{0.90 ± 0.06} & 0.06 ± 0.01 \\
GDN & 0.31 ± 0.06 & \underline{0.07 ± 0.00} \\
GRELEN & \underline{0.87 ± 0.06} & 0.01 ± 0.00 \\
MTADGAT & 0.08 ± 0.01 & 0.03 ± 0.01 \\ \midrule
TimeGAT & 0.84 ± 0.11 & \textbf{0.10 ± 0.12} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[tbhp]
{\color{revcolor}
\centering
\caption{Process variables in Pronto dataset used for fault detection~\cite{stief2019pronto}. Dark gray highlights Control Variables (CV) and light gray External Factors (EF).}
\label{tab:pronto_data}
\begin{tabular}{lll} \toprule
\textbf{Tag} & \textbf{Description} & \textbf{Unit} \\ \midrule
\rowcolor[HTML]{C0C0C0}
FT305/302 & Input air flow rate & $\mathrm{S} \mathrm{m}^3 \mathrm{~h}-1$ \\
\rowcolor[HTML]{EFEFEF}
FT305-T & Input air temperature & ${ }^{\circ} \mathrm{C}$ \\
PT312 & Air delivery pressure & $\operatorname{bar}(g)$ \\
\rowcolor[HTML]{C0C0C0}
FT102/104 & Input water flow rate & $\mathrm{kg} \mathrm{s}-1$ \\
\rowcolor[HTML]{EFEFEF}
FT102-T & Input water temperature & ${ }^{\circ} \mathrm{C}$ \\
% FT102-D & Input water density & $\mathrm{kg} \mathrm{m}-3$ \\
PT417 & Pressure in the mixing zone & $\operatorname{bar}(g)$ \\
PT408 & Pressure at the riser top & $\operatorname{bar}(g)$ \\
PT403 & Pressure in the 2-phase separator & $\operatorname{bar}(g)$ \\
FT404 & 2-phase separator output air flow rate & $m^3 h^{-1}$ \\
FT406 & 2-phase separator output water flow rate & $\mathrm{kg} \mathrm{s}^{-1}$ \\
PT501 & Pressure in the 3-phase separator & $\operatorname{bar}(g)$ \\
PIC501 & Air outlet valve 3-phase separator & $(\%)$ \\
LI502 & Water level 3-phase separator & (\%) \\
$\mathrm{LISO} 3$ & Water coalescer level & $(\%)$ \\
LVC502 & Water coalescer outlet valve & $(\%)$ \\
% Ll101 & Water tank level & $\mathrm{m}$ \\
\bottomrule
\end{tabular}
}
\end{table}
% \begin{table}
% \centering
% \caption{Process variables in Pronto dataset used for fault detection~\cite{stief2019pronto}.}
% \label{tab:pronto_data}
% \begin{tabular}{llll} \toprule
% \textbf{Tag} & \textbf{Description} & \textbf{Unit} & \textbf{Var. Type} \\ \midrule
% FT305/302 & Input air flow rate & $\mathrm{Sm}^3\mathrm{h}^{-1}$ & CV \\
% FT305-T & Input air temperature & ${ }^{\circ} \mathrm{C}$ & EF \\
% PT312 & Air delivery pressure & $\operatorname{bar}(g)$ & MV \\
% FT102/104 & Input water flow rate & $\mathrm{kg} \mathrm{s}^{-1}$ & CV \\
% FT102-T & Input water temperature & ${ }^{\circ} \mathrm{C}$ & EF \\
% PT417 & Press. in the mixing zone & $\operatorname{bar}(g)$ & MV \\
% PT408 & Press. at the riser top & $\operatorname{bar}(g)$ & MV \\
% PT403 & Press. in the 2-phase separator & $\operatorname{bar}(g)$ & MV \\
% FT404 & 2-phase separator output air & $\mathrm{m}^3\mathrm{h}^{-1}$ & MV \\
% FT406 & 2-phase separator output water & $\mathrm{kg} \mathrm{s}^{-1}$ & MV \\
% PT501 & Press. in the 3-phase separator & $\operatorname{bar}(g)$ & MV \\
% PIC501 & Air out. valve 3-phase separator & $(\%)$ & MV \\
% LI502 & Water level 3-phase separator & $(\%)$ & MV \\
% LISO3 & Water coalescer level & $(\%)$ & MV \\
% LVC502 & Water coalescer outlet valve & $(\%)$ & MV \\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{table}[tbhp]
\centering
\caption{Performance comparison on pronto dataset with 5 runs over three fault classes. The best metric score is highlighted in bold and the second best model in underscore.}
\label{tab:pronto_results}
\begin{tabular}{lcccc}
\toprule
& \textbf{AUC}$\uparrow$ & \textbf{F1}$\uparrow$ & \textbf{F1}$^*\uparrow$ & \textbf{Delay}$^*$ $\downarrow$ \\
\midrule
AE & 0.73 ± 0.04 & 0.40 ± 0.02 & \underline{0.83 ± 0.02} & 905 ± 559 \\
FNN & \underline{0.74 ± 0.02} & 0.63 ± 0.05 & 0.82 ± 0.01 & 1644 ± 83 \\\midrule
LSTM & 0.60 ± 0.09 & \underline{0.70 ± 0.11} & 0.80 ± 0.02 & 511 ± 599 \\
LSTM-AE & 0.73 ± 0.04 & 0.61 ± 0.04 & 0.81 ± 0.01 & 1952 ± 784 \\
USAD & 0.65 ± 0.02 & 0.27 ± 0.04 & 0.80 ± 0.00 & \textbf{51 ± 4} \\\midrule
GDN & 0.52 ± 0.06 & 0.27 ± 0.05 & 0.78 ± 0.01 & 2467 ± 2345\\
GRELEN & 0.47 ± 0.02 & 0.29 ± 0.01 & 0.77 ± 0.00 & 4838 ± 451\\
MTADGAT & 0.72 ± 0.06& 0.40 ± 0.02& 0.82 ± 0.02& 1232 ± 834\\\midrule
DyEdgeGAT & \textbf{0.80 ± 0.05} & \textbf{0.83 ± 0.02} & \textbf{0.86 ± 0.02} & \underline{61 ± 1}
\\
\bottomrule
\end{tabular}
\end{table}
\begin{table*}[tbhp]
\centering
\caption{Performance comparison on pronto dataset with 5 runs. The best metric score is highlighted in bold and the second best model in underscor}
\label{tab:pronto_agg_results}
\begin{tabular}{lcccccc}
\toprule
& \textbf{ROC AUC} $\uparrow$ & \textbf{PR AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{F1}$^*$ $\uparrow$ & \textbf{F1\textsubscript{PA\%50}} $\uparrow$ & \textbf{Delay}$^*$ $\downarrow$ \\
\midrule
AE & 0.726 ± 0.039 & 0.779 ± 0.025 & 0.400 ± 0.016 & \underline{0.825 ± 0.023} & \underline{0.865 ± 0.027} & 905.1 ± 559.1 \\
FNN & \underline{0.745 ± 0.020} & 0.810 ± 0.011 & 0.634 ± 0.049 & 0.817 ± 0.007 & 0.857 ± 0.012 & 1644.1 ± 83.2 \\
LSTM & 0.603 ± 0.087 & 0.728 ± 0.047 & \underline{0.699 ± 0.109} & 0.796 ± 0.019 & 0.839 ± 0.047 & 510.9 ± 598.5 \\
LSTM-AE & 0.725 ± 0.037 & \underline{0.813 ± 0.019} & 0.607 ± 0.040 & 0.813 ± 0.006 & 0.848 ± 0.011 & 1952.4 ± 783.9 \\
USAD & 0.642 ± 0.017 & 0.740 ± 0.013 & 0.230 ± 0.015 & 0.801 ± 0.001 & 0.807 ± 0.003 & \textbf{47.1 ± 3.7} \\
GDN & 0.511 ± 0.062 & 0.674 ± 0.036 & 0.297 ± 0.042 & 0.783 ± 0.009 & 0.786 ± 0.011 & 1039.3 ± 838.7 \\
GRELEN & 0.458 ± 0.013 & 0.635 ± 0.010 & 0.252 ± 0.029 & 0.770 ± 0.000 & 0.770 ± 0.000 & 1892.7 ± 54.7 \\
MTADGAT & 0.680 ± 0.050 & 0.778 ± 0.025 & 0.402 ± 0.013 & 0.814 ± 0.008 & 0.852 ± 0.013 & 1521.7 ± 1.0 \\
TimeGAT & \textbf{0.801 ± 0.049} & \textbf{0.840 ± 0.043} & \textbf{0.826 ± 0.022} & \textbf{0.859 ± 0.017} & \textbf{0.895 ± 0.023} & \underline{60.9 ± 1.5} \\
\bottomrule
\end{tabular}
\end{table*}
\begin{table}[tbhp]
\centering
\caption{Performance evaluation of GRELEN with and without Point Adjustment (PA) on the synthetic dataset.}
\label{tab:syn_grelen_result}
\begin{tabular}{lcc}
\toprule
& \textbf{AUC} & \textbf{AUC\textsubscript{PA}} \\
\midrule
Degree & 0.548 ± 0.051 & 0.970 ± 0.010 \\
Node Scaled & 0.676 ± 0.005 & 0.943 ± 0.003 \\
\bottomrule
\end{tabular}
\end{table}
%\begin{table}
%\centering
%\caption{Performance evaluation of GRELEN with and without Point Adjustment (PA) on the synthetic dataset.}
%\label{tab:syn}
%\begin{tabular}{lccc}
%\toprule
% & \textbf{AUC} & \textbf{AUC\textsubscript{PA}$^*$} & \textbf{AUC\textsubscript{PA\%50}} \\
%\midrule
%Degree & 0.548 ± 0.051 & 0.970 ± 0.010 & 0.551 ± 0.029 \\
%Node Scaled & 0.676 ± 0.005 & 0.943 ± 0.003 & 0.651 ± 0.007 \\
%\bottomrule
%\end{tabular}
%\end{table}
\begin{table}[tbhp]
\centering
\caption{Sensitivity analysis of TimeGAT's performance on the synthetic dataset across different sliding window sizes.}
\label{tab:syn_sensitivity}
\begin{tabular}{ccc}
\toprule
& \textbf{AUC}$\uparrow$ & \textbf{F1}$^*$$\uparrow$ \\
\midrule
$\delta t=1$ & 0.801 ± 0.045 & 0.726 ± 0.030 \\
$\delta t=3$ & 0.790 ± 0.044 & 0.704 ± 0.031 \\
$\delta t=5$ & \textbf{0.832 ± 0.040} & \textbf{0.750 ± 0.039} \\
$\delta t=7$ & 0.792 ± 0.037 & 0.717 ± 0.033 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\caption{Ablation study on the synthetic dataset with 5 runs}
\label{tab:syn_ablation}
\begin{tabular}{lcc}
\toprule
& \textbf{AUC}$\uparrow$ & \textbf{F1}$^*\uparrow$ \\
\midrule
DyEdgeGAT & \textbf{0.832 ± 0.040} & \textbf{0.750 ± 0.039} \\
- w/o topl & 0.828 ± 0.035 & 0.745 ± 0.034 \\
- w/o time & 0.777 ± 0.028 & 0.702 ± 0.026 \\
- w/o reverse & 0.799 ± 0.040 & 0.721 ± 0.026 \\
- w/o dyn. graph & 0.728 ± 0.053 & 0.693 ± 0.035 \\
- w/o control & 0.710 ± 0.067 & 0.666 ± 0.034 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htb]
\centering
\caption{
Model Performance comparison on the synthetic dataset with 5 runs over 8 test cases. The best metric score is highlighted in bold and the second best model in underscore.}
\label{tab:syn_res}
\begin{tabular}{lcccc}
\toprule
& \textbf{AUC}$\uparrow$ & \textbf{F1}$\uparrow$ & \textbf{F1}$^*\uparrow$ & \textbf{Delay}$^*$ $\downarrow$ \\
\midrule
AE & 0.74 ± 0.03 & 0.56 ± 0.03 & 0.63 ± 0.02 & 30.0 ± 9.7 \\
FNN & 0.78 ± 0.02 & 0.55 ± 0.01 & 0.67 ± 0.01 & 21.7 ± 9.0 \\ \midrule
LSTM & 0.78 ± 0.04 & 0.61 ± 0.00 & 0.70 ± 0.03 & 23.3 ± 11.4 \\
LSTM-AE & \underline{0.80 ± 0.05} & 0.61 ± 0.00 & \underline{0.72 ± 0.04} & \textbf{9.2 ± 5.9} \\
USAD & 0.71 ± 0.03 & 0.61 ± 0.00 & 0.65 ± 0.01 & 43.4 ± 5.1 \\ \midrule
GDN & 0.78 ± 0.06 & \underline{0.66 ± 0.07} & 0.72 ± 0.04 & \underline{19.4 ± 10.6} \\
GRELEN & 0.67 ± 0.00 & 0.42 ± 0.01 & 0.64 ± 0.00 & 33.6 ± 3.2 \\
MTADGAT & 0.80 ± 0.05 & 0.65 ± 0.03 & 0.71 ± 0.02 & 25.0 ± 5.7 \\ \midrule
DyEdgeGAT & \textbf{0.83 ± 0.04} & \textbf{0.69 ± 0.08} & \textbf{0.75 ± 0.04} & 21.4 ± 5.6 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table*}[htbp]
\centering
\revtext{MENTION AVERAGE}
\caption{Model Performance comparison on the synthetic dataset with 5 runs. The best metric score is highlighted in bold and the second best model in underscore}
\label{tab:syn_res}
\begin{tabular}{lcccccc}
\toprule
& \textbf{ROC AUC} $\uparrow$ & \textbf{PR AUC} $\uparrow$ & \textbf{F1} $\uparrow$ & \textbf{F1}$^*$ $\uparrow$ & \textbf{F1\textsubscript{PA\%50}} $\uparrow$ & \textbf{Delay}$^*$ $\downarrow$ \\
\midrule
AE & 0.744 ± 0.028 & 0.646 ± 0.025 & 0.562 ± 0.028 & 0.631 ± 0.024 & 0.759 ± 0.048 & 30.0 ± 9.7 \\
FNN & 0.784 ± 0.019 & 0.698 ± 0.038 & 0.552 ± 0.009 & 0.666 ± 0.014 & 0.797 ± 0.040 & 21.7 ± 9.0 \\ \midrule
LSTM & 0.784 ± 0.037 & 0.778 ± 0.047 & 0.605 ± 0.003 & 0.702 ± 0.033 & 0.837 ± 0.070 & 23.3 ± 11.4 \\
LSTM-AE & 0.799 ± 0.052 & 0.746 ± 0.064 & 0.605 ± 0.002 & 0.722 ± 0.040 & 0.818 ± 0.057 & \textbf{9.2 ± 5.9} \\
USAD & 0.738 ± 0.000 & 0.746 ± 0.000 & 0.604 ± 0.000 & 0.673 ± 0.000 & 0.834 ± 0.000 & 39.7 ± 0.0 \\ \midrule
GDN & 0.777 ± 0.058 & 0.714 ± 0.070 & \underline{0.663 ± 0.074} & 0.718 ± 0.045 & 0.779 ± 0.056 & \underline{19.4 ± 10.6} \\
GRELEN & 0.673 ± 0.003 & 0.655 ± 0.003 & 0.421 ± 0.007 & 0.643 ± 0.003 & 0.696 ± 0.014 & 33.6 ± 3.2 \\
MTADGAT & \underline{0.806 ± 0.038} & \underline{0.788 ± 0.031} & 0.604 ± 0.000 & \underline{0.725 ± 0.036} & \underline{0.853 ± 0.045} & 25.2 ± 9.5 \\ \midrule
TimeGAT & \textbf{0.832 ± 0.040} & \textbf{0.806 ± 0.051} & \textbf{0.690 ± 0.076} & \textbf{0.750 ± 0.039} & \textbf{0.859 ± 0.060} & 21.4 ± 5.6 \\
\bottomrule
\end{tabular}
\end{table*}
% TODO: add evaluation of point anomaly (original implementation in the appendix)
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{mohammadi2018deep}
M.~Mohammadi, A.~Al-Fuqaha, S.~Sorour, and M.~Guizani, ``Deep learning for iot big data and streaming analytics: A survey,'' \emph{IEEE Communications Surveys \& Tutorials}, vol.~20, no.~4, pp. 2923--2960, 2018.
\bibitem{younan2020challenges}
M.~Younan, E.~H. Houssein, M.~Elhoseny, and A.~A. Ali, ``Challenges and recommended technologies for the industrial internet of things: A comprehensive review,'' \emph{Measurement}, vol. 151, p. 107198, 2020.
\bibitem{dong2023iotreview}
G.~Dong, M.~Tang, Z.~Wang, J.~Gao, S.~Guo, L.~Cai, R.~Gutierrez, B.~Campbel, L.~E. Barnes, and M.~Boukhechba, ``Graph neural networks in iot: A survey,'' \emph{ACM Transactions on Sensor Networks}, vol.~19, no.~2, pp. 1--50, 2023.
\bibitem{wei2020robust}
Y.~Wei, D.~Wu, and J.~Terpenny, ``Robust incipient fault detection of complex systems using data fusion,'' \emph{IEEE Transactions on Instrumentation and Measurement}, vol.~69, no.~12, pp. 9526--9534, 2020.
\bibitem{chao2019hybrid}
M.~A. Chao, C.~Kulkarni, K.~Goebel, and O.~Fink, ``Hybrid deep fault detection and isolation: Combining deep neural networks and system performance models,'' \emph{International Journal of Prognostics and Health Management}, vol.~10, no.~4, 2019.
\bibitem{lughofer2019predictive}
E.~Lughofer and M.~Sayed-Mouchaweh, \emph{Predictive Maintenance in Dynamic Systems: Advanced methods, Decision Support Tools and Real-world Applications}.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2019.
\bibitem{boyes2018industrial}
H.~Boyes, B.~Hallaq, J.~Cunningham, and T.~Watson, ``The industrial internet of things (iiot): An analysis framework,'' \emph{Computers in industry}, vol. 101, pp. 1--12, 2018.
\bibitem{ahmed2014automotive}
R.~Ahmed, M.~El~Sayed, S.~A. Gadsden, J.~Tjong, and S.~Habibi, ``Automotive internal-combustion-engine fault detection and classification using artificial neural network techniques,'' \emph{IEEE Transactions on vehicular technology}, vol.~64, no.~1, pp. 21--33, 2014.
\bibitem{guo2020unsupervised}
Z.~Guo, Y.~Wan, and H.~Ye, ``An unsupervised fault-detection method for railway turnouts,'' \emph{IEEE Transactions on Instrumentation and Measurement}, vol.~69, no.~11, pp. 8881--8901, 2020.
\bibitem{ergen2019unsupervised}
T.~Ergen and S.~S. Kozat, ``Unsupervised anomaly detection with {LSTM} neural networks,'' \emph{IEEE Transactions on Neural Networks and Learning Systems}, vol.~31, no.~8, pp. 3127--3141, 2019.
\bibitem{garcia2022temporal}
G.~R. Garcia, G.~Michau, M.~Ducoffe, J.~S. Gupta, and O.~Fink, ``Temporal signals to images: Monitoring the condition of industrial assets with deep learning image processing algorithms,'' \emph{Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability}, vol. 236, no.~4, pp. 617--627, 2022.
\bibitem{wu2020comprehensive}
Z.~Wu, S.~Pan, F.~Chen, G.~Long, C.~Zhang, and S.~Y. Philip, ``A comprehensive survey on graph neural networks,'' \emph{IEEE transactions on neural networks and learning systems}, vol.~32, no.~1, pp. 4--24, 2020.
\bibitem{jin2023survey}
M.~Jin, H.~Y. Koh, Q.~Wen, D.~Zambon, C.~Alippi, G.~I. Webb, I.~King, and S.~Pan, ``A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection,'' \emph{arXiv preprint arXiv:2307.03759}, 2023.
\bibitem{zawislak2017graph}
S.~Zawi{\'s}lak and J.~Rysi{\'n}ski, \emph{Graph-based modelling in engineering}.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2017.
\bibitem{zhao2020mtadgat}
H.~Zhao, Y.~Wang, J.~Duan, C.~Huang, D.~Cao, Y.~Tong, B.~Xu, J.~Bai, J.~Tong, and Q.~Zhang, ``Multivariate time-series anomaly detection via graph attention network,'' in \emph{2020 IEEE International Conference on Data Mining (ICDM)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 841--850.
\bibitem{deng2021gdn}
A.~Deng and B.~Hooi, ``Graph neural network-based anomaly detection in multivariate time series,'' in \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, vol.~35, no.~5, 2021, pp. 4027--4035.
\bibitem{yin2014ocsvm}
S.~Yin, X.~Zhu, and C.~Jing, ``Fault detection based on a robust one class support vector machine,'' \emph{Neurocomputing}, vol. 145, pp. 263--268, 2014.
\bibitem{plakias2022novel}
S.~Plakias and Y.~S. Boutalis, ``A novel information processing method based on an ensemble of auto-encoders for unsupervised fault detection,'' \emph{Computers in Industry}, vol. 142, p. 103743, 2022.
\bibitem{akcay2019ganomaly}
S.~Akcay, A.~Atapour-Abarghouei, and T.~P. Breckon, ``Ganomaly: Semi-supervised anomaly detection via adversarial training,'' in \emph{Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part III 14}.\hskip 1em plus 0.5em minus 0.4em\relax Springer, 2019, pp. 622--637.
\bibitem{plakias2019exploiting}
S.~Plakias and Y.~S. Boutalis, ``Exploiting the generative adversarial framework for one-class multi-dimensional fault detection,'' \emph{Neurocomputing}, vol. 332, pp. 396--405, 2019.
\bibitem{li2020anomaly}
L.~Li, J.~Yan, H.~Wang, and Y.~Jin, ``Anomaly detection of time series with smoothness-inducing sequential variational auto-encoder,'' \emph{IEEE Transactions on Neural Networks and Learning Systems}, vol.~32, no.~3, pp. 1177--1191, 2020.
\bibitem{wu2019lstm}
D.~Wu, Z.~Jiang, X.~Xie, X.~Wei, W.~Yu, and R.~Li, ``Lstm learning with bayesian and gaussian processing for anomaly detection in industrial iot,'' \emph{IEEE Transactions on Industrial Informatics}, vol.~16, no.~8, pp. 5244--5253, 2019.
\bibitem{cook2019anomaly}
A.~A. Cook, G.~M{\i}s{\i}rl{\i}, and Z.~Fan, ``Anomaly detection for iot time-series data: A survey,'' \emph{IEEE Internet of Things Journal}, vol.~7, no.~7, pp. 6481--6494, 2019.
\bibitem{chatterjee2022iot}
A.~Chatterjee and B.~S. Ahmed, ``Iot anomaly detection methods and applications: A survey,'' \emph{Internet of Things}, vol.~19, p. 100568, 2022.
\bibitem{fu2022mad}
Y.~Fu and F.~Xue, ``{MAD}: Self-supervised masked anomaly detection task for multivariate time series,'' in \emph{2022 International Joint Conference on Neural Networks (IJCNN)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2022, pp. 1--8.
\bibitem{michau2019unsupervised}
G.~Michau and O.~Fink, ``Unsupervised fault detection in varying operating conditions,'' in \emph{2019 IEEE International Conference on Prognostics and Health Management (ICPHM)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2019, pp. 1--10.
\bibitem{rombach2021contrastive}
K.~Rombach, G.~Michau, and O.~Fink, ``Contrastive learning for fault detection and diagnostics in the context of changing operating conditions and novel fault types,'' \emph{Sensors}, vol.~21, no.~10, p. 3550, 2021.
\bibitem{li2017diffusion}
Y.~Li, R.~Yu, C.~Shahabi, and Y.~Liu, ``Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,'' in \emph{International Conference on Learning Representations (ICLR '18)}, 2018.
\bibitem{chen2023multivariate}
K.~Chen, M.~Feng, and T.~S. Wirjanto, ``Multivariate time series anomaly detection via dynamic graph forecasting,'' \emph{arXiv preprint arXiv:2302.02051}, 2023.
\bibitem{wu2020connecting}
Z.~Wu, S.~Pan, G.~Long, J.~Jiang, X.~Chang, and C.~Zhang, ``Connecting the dots: Multivariate time series forecasting with graph neural networks,'' in \emph{Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery \& data mining}, 2020, pp. 753--763.
\bibitem{zhang22grelen}
W.~Zhang, C.~Zhang, and F.~Tsung, ``Grelen: Multivariate time series anomaly detection from the perspective of graph relational learning,'' in \emph{Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22}, 2022, pp. 2390--2397.
\bibitem{zugner2021study}
D.~Z{\"u}gner, F.-X. Aubet, V.~G. Satorras, T.~Januschowski, S.~G{\"u}nnemann, and J.~Gasthaus, ``A study of joint graph inference and forecasting,'' in \emph{ICML 2021 Time Series Workshop}, 2021.
\bibitem{gao2022equivalence}
J.~Gao and B.~Ribeiro, ``On the equivalence between temporal and static equivariant graph representations,'' in \emph{International Conference on Machine Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2022, pp. 7052--7076.
\bibitem{shang2021gts}
C.~Shang and J.~Chen, ``Discrete graph structure learning for forecasting multiple time series,'' in \emph{Proceedings of International Conference on Learning Representations}, 2021.
\bibitem{xu2020tgat}
D.~Xu, C.~Ruan, E.~Korpeoglu, S.~Kumar, and K.~Achan, ``Inductive representation learning on temporal graphs,'' in \emph{International Conference on Learning Representations (ICLR)}, 2020.
\bibitem{rossi2020tgn}
E.~Rossi, B.~Chamberlain, F.~Frasca, D.~Eynard, F.~Monti, and M.~Bronstein, ``Temporal graph networks for deep learning on dynamic graphs,'' in \emph{ICML 2020 Workshop on Graph Representation Learning}, 2020.
\bibitem{deng2022graph}
L.~Deng, D.~Lian, Z.~Huang, and E.~Chen, ``Graph convolutional adversarial networks for spatiotemporal anomaly detection,'' \emph{IEEE Transactions on Neural Networks and Learning Systems}, vol.~33, no.~6, pp. 2416--2428, 2022.
\bibitem{zambon2018concept}
D.~Zambon, C.~Alippi, and L.~Livi, ``Concept drift and anomaly detection in graph streams,'' \emph{IEEE transactions on neural networks and learning systems}, vol.~29, no.~11, pp. 5592--5605, 2018.
\bibitem{li2023novel}
T.~Li, C.~Suna, R.~Yan, X.~Chen, and O.~Fink, ``A novel unsupervised graph wavelet autoencoder for mechanical system fault detection,'' \emph{arXiv preprint arXiv:2307.10676}, 2023.
\bibitem{chen2021learning}
Z.~Chen, D.~Chen, X.~Zhang, Z.~Yuan, and X.~Cheng, ``Learning graph structures with transformer for multivariate time-series anomaly detection in iot,'' \emph{IEEE Internet of Things Journal}, vol.~9, no.~12, pp. 9179--9189, 2021.
\bibitem{zheng2023correlation}
Y.~Zheng, H.~Koh, M.~Jin, L.~Chi, K.~Phan, S.~Pan, Y.~Chen, and W.~Xiang, ``Correlation-aware spatial-temporal graph learning for multivariate time-series anomaly detection.'' \emph{IEEE Transactions on Neural Networks and Learning Systems}, 2023.
\bibitem{brody2022gatv2}
S.~Brody, U.~Alon, and E.~Yahav, ``How attentive are graph attention networks?'' in \emph{International Conference on Learning Representations (ICLR)}, 2022.
\bibitem{gilmer2017neural}
J.~Gilmer, S.~S. Schoenholz, P.~F. Riley, O.~Vinyals, and G.~E. Dahl, ``Neural message passing for quantum chemistry,'' in \emph{International conference on machine learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2017, pp. 1263--1272.
\bibitem{hu2019gine}
W.~Hu, B.~Liu, J.~Gomes, M.~Zitnik, P.~Liang, V.~Pande, and J.~Leskovec, ``Strategies for pre-training graph neural networks,'' in \emph{International Conference on Learning Representations}, 2019.
\bibitem{cho2014learning}
K.~Cho, B.~Van~Merri{\"e}nboer, C.~Gulcehre, D.~Bahdanau, F.~Bougares, H.~Schwenk, and Y.~Bengio, ``Learning phrase representations using rnn encoder-decoder for statistical machine translation,'' \emph{arXiv preprint arXiv:1406.1078}, 2014.
\bibitem{stief2019pronto}
A.~Stief, R.~Tan, Y.~Cao, J.~R. Ottewill, N.~F. Thornhill, and J.~Baranowski, ``A heterogeneous benchmark dataset for data analytics: Multiphase flow facility case study,'' \emph{Journal of Process Control}, vol.~79, pp. 41--55, 2019.
\bibitem{aggarwaloutlier}
C.~Aggarwal, ``Outlier analysis. data mining, 2015.''
\bibitem{malhotra2015lstmad}
P.~Malhotra, L.~Vig, G.~Shroff, P.~Agarwal \emph{et~al.}, ``Long short term memory networks for anomaly detection in time series.'' in \emph{ESANN}, vol. 2015, 2015, p.~89.
\bibitem{malhotra2016encdecad}
P.~Malhotra, A.~Ramakrishnan, G.~Anand, L.~Vig, P.~Agarwal, and G.~Shroff, ``{LSTM}-based encoder-decoder for multi-sensor anomaly detection,'' \emph{arXiv preprint arXiv:1607.00148}, 2016.
\bibitem{audibert2020usad}
J.~Audibert, P.~Michiardi, F.~Guyard, S.~Marti, and M.~A. Zuluaga, ``Usad: Unsupervised anomaly detection on multivariate time series,'' in \emph{Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery \& data mining}, 2020, pp. 3395--3404.
\bibitem{velivckovic2017gat}
P.~Veli{\v{c}}kovi{\'c}, G.~Cucurull, A.~Casanova, A.~Romero, P.~Li{\`o}, and Y.~Bengio, ``Graph attention networks,'' in \emph{International Conference on Learning Representations}, 2017.
\bibitem{paszke2019pytorch}
A.~Paszke, S.~Gross, F.~Massa, A.~Lerer, J.~Bradbury, G.~Chanan, T.~Killeen, Z.~Lin, N.~Gimelshein, L.~Antiga \emph{et~al.}, ``Pytorch: An imperative style, high-performance deep learning library,'' \emph{Advances in neural information processing systems}, vol.~32, 2019.
\bibitem{fey2019fast}
M.~Fey and J.~E. Lenssen, ``Fast graph representation learning with pytorch geometric,'' \emph{arXiv preprint arXiv:1903.02428}, 2019.
\bibitem{hsu2023comparison}
C.-C. Hsu, G.~Frusque, and O.~Fink, ``A comparison of residual-based methods on fault detection,'' in \emph{Annual Conference of the PHM Society}, vol.~15, no.~1, 2023.
\bibitem{kim2022towards}
S.~Kim, K.~Choi, H.-S. Choi, B.~Lee, and S.~Yoon, ``Towards a rigorous evaluation of time-series anomaly detection,'' in \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, vol.~36, no.~7, 2022, pp. 7194--7201.
\end{thebibliography}