Enhancing Cyber-Resilience in Integrated Energy System Scheduling with Demand Response Using Deep Reinforcement Learning
Extracted source text · arXiv 2311.17941
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{graphicx}
\graphicspath{{./Figures/}}
\usepackage{subfigure}
\usepackage{xcolor,graphicx,float}
\usepackage{amsthm}
\usepackage{amsmath}
\interdisplaylinepenalty=2500
\usepackage{multirow}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{booktabs}
\usepackage{cite}
\usepackage[colorlinks,linkcolor=black,citecolor=black,urlcolor=black]{hyperref}
\usepackage{url}
\usepackage{titlesec}
% updated with editorial comments 8/9/2021
\begin{document}
\title{Advancing Attack-Resilient Scheduling of Integrated Energy Systems with Demand Response via Deep Reinforcement Learning}
% \title{Attack-Resilient scheduling of integrated demand response-enabled integrated energy system using deep reinforcement learning }
\author{{Yang~Li, Wenjie~Ma, Yuanzheng~Li,~\IEEEmembership{Senior Member,~IEEE}, Sen Li, Zhe Chen,~\IEEEmembership{Fellow,~IEEE}}
% <-this % stops a space
\thanks{ This work is supported by the National Natural Science Foundation of China under Grant (No. U2066208).}
\thanks{Y. Li and W. Ma are with the School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China (e-mail: Y. Li, e-mail: liyang@neepu.edu.cn, 2202100232@neepu.edu.cn).}%
\thanks{Y. Z. Li is with the School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China (e-mail: Yuanzheng\_Li@hust.edu.cn).}
\thanks{Sen Li is with the Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology, Hong Kong (e-mail: cesli@ust.hk).}
\thanks{Z. Chen is with the Institute of Energy Technology, Aalborg University, Aalborg DK-9220, Denmark (e-mail: zch@energy.aau.dk).}}
\maketitle
\begin{abstract}
Optimally scheduling multi-energy flow is an effective method to utilize renewable energy sources (RES) and improve the stability and economy of integrated energy systems (IES). However, the stable demand-supply of IES faces challenges from uncertainties that arise from RES and loads, as well as the increasing impact of cyber-attacks with advanced information and communication technologies adoption. To address these challenges, this paper proposes an innovative model-free resilience scheduling method based on state-adversarial deep reinforcement learning (DRL) for integrated demand response (IDR)-enabled IES. The proposed method designs an IDR program to explore the interaction ability of electricity-gas-heat flexible loads. Additionally, a state-adversarial Markov decision process (SA-MDP) model characterizes the energy scheduling problem of IES under cyber-attack. The state-adversarial soft actor-critic (SA-SAC) algorithm is proposed to mitigate the impact of cyber-attacks on the scheduling strategy. Simulation results demonstrate that our method is capable of adequately addressing the uncertainties resulting from RES and loads, mitigating the impact of cyber-attacks on the scheduling strategy, and ensuring a stable demand supply for various energy sources. Moreover, the proposed method demonstrates resilience against cyber-attacks. Compared to the original soft actor-critic (SAC) algorithm, it achieves a 10\% improvement in economic performance under cyber-attack scenarios. %Optimal scheduling of multi-energy flow is an effective way to consume renewable energy sources (RES) and improve the stability and economy of integrated energy systems (IES). The stable demand-supply of IES faces challenges from multi-uncertainties that arise from RES and loads, and the impact of cyber-attacks on the systems is gradually increasing with the adoption of increasingly advanced information and communication technologies within the IES. To address these challenges, an innovative model-free resilience scheduling method based on deep reinforcement learning (DRL) is proposed for integrated demand response (IDR)-enabled IES, where an IDR program is designed to explore the potential interaction ability of electricity-gas-heat flexible loads. Additionally, we propose a state-adversarial Markov decision process (SAMDP) model to characterize the energy scheduling problem of IES under cyber-attack. To mitigate the impact of cyber-attacks on the scheduling strategy, we propose the state-adversarial soft actor-critic (SA-SAC) algorithm. The simulation results show that the proposed scheduling framework properly handles multi-uncertainties from RES and loads, effectively mitigates the impact of cyber-attack on the scheduling strategy, and achieves a stable demand supply for various energy sources. In addition, the proposed method with better learning effect and economic efficiency compared with the original soft actor-critic (SAC) is also verified.
\end{abstract}
\begin{IEEEkeywords}
Integrated energy system, integrated demand response, cyber-attack, deep reinforcement learning, dynamic pricing mechanism, attack-resilient scheduling
\end{IEEEkeywords}
\section{Introduction}
\IEEEPARstart{T}{o} address the increasing conflict between energy demand and environmental protection, renewable energy sources (RES) with zero-carbon characteristics have received widespread attention, and countries worldwide are adjusting their energy systems to lessen their reliance on traditional fossil fuels\cite{wang2018review}. Integrated energy systems (IES) can effectively integrate and utilize various energy units, improving the efficiency of energy use and the consumption capacity of intermittent RES, which provides a new solution for improving the ecological environment and achieving a low-carbon sustainable energy system\cite{liu2020region}. However, the coordination of multiple energy units and the multi-uncertainties of RES and loads present challenges for the economic operation of IES\cite{lu2020hydraulic}. In addition, the dependence on advanced information and communication technologies for energy management systems (EMS) has introduced cyber security threats while improving the efficiency, security and reliability of EMS. False data injection attacks (FDIA) are a common type of cyber-attack that can affect and mislead system operators by integrating fake data into real-time instrumentation measurements, leading to erroneous decisions on policy and control\cite{che2018mitigating}. As an example, a cyber-attack in 2015 resulted in a massive power outage in the Ukrainian power grid\cite{liang20162015}. Therefore, it is urgent to study resilient scheduling under cyber-attacks to mitigate the uneconomic operation and reliability problems caused by such attacks.
In the research on energy dispatch in integrated energy systems, relevant measures have been proposed to address the high degree of uncertainty in the system. In \cite{zheng2019stochastic}, the authors used the probability density function to address the uncertainty of RES. Modeling the stochasticity of RES using scenario analysis is also a common method for dealing with uncertainty\cite{emrani2021optimal}. These methods require accurate modeling of renewable energy sources or enumeration scenarios, while demand response (DR) for flexible loads has been shown to be beneficial in addressing the uncertainty of RES. In \cite{li2021optimal}, an integrated demand response (IDR) program is proposed that considers the interaction among electrical, gas, and thermal loads to enhance system economic efficiency. However, current studies often neglect the IDR program under real-time scheduling scenarios.
Existing literature on FDIA mainly focuses on designing stealthy FDIA, evaluating their impact on systems, and developing strategies to defend against them. A type of cyber-attacks targeting scheduling information in natural gas systems is proposed, and its impact on the operation of the electric and gas energy subsystems is analyzed\cite{zhao2021coordinated}. Ref. \cite{shayan2019network} proposes a network cyber-secured unit commitment model to improve the economic efficiency of system operation under FDIA. Furthermore, in terms of detecting and filtering bad data, state estimation plays a key role. However, some FDIAs have been deliberately designed to evade detection \cite{liu2016false}. Load redistribution (LR) attack is a specific class of FDIA that involves the use of false load data to impact the system's operation and cause economic losses and physical damages to equipments due to incorrect operational decisions \cite{zhao2021cyber}. Ref. \cite{ding2022cyber} proposes a stealthy heat load redistribution (HLR) attack and analyzes its impact on the security and economics of IES. Given that stealthy FDIAs can bypass state estimation and avoid detection, they can lead to uneconomical system operations. Therefore, it is urgent to study IES resilience scheduling against cyber-attacks.
%%However, these methods explicitly include probabilities of uncertainties and rely on accurate modeling of RES devices or enumeration scenarios. Furthermore, demand response (DR) for flexible loads has been shown to be beneficial in addressing the uncertainty of renewable energy.%the IDR under real-time scheduling is often neglected in current studies. %improve the economic efficiency of system operation In terms of cyber-attacks, the existing literature of FDIA focuses on structuring stealthy design FDIA, evaluating their impact on systems, and designing strategies to defend against or mitigate their impact.
%To overcome the aforementioned set of challenges, deep reinforcement learning (DRL) has been introduced into real-time energy management systems in recent years. Compared to other scheduling methods, reinforcement learning provides model-free methods to solve stochastic decision-making problems without requiring prior knowledge of the environment, guarantees real-time requirements, and has strong adaptability to multi-uncertainties interference%
Deep reinforcement learning (DRL) is increasingly used in real-time energy management for stochastic decision-making without needing prior environmental knowledge. It addresses uncertainties in renewable energy sources (RES) and loads using the soft actor-critic (SAC) algorithm\cite{zhang2021soft} and adapts to power fluctuations\cite{yang2021dynamic}. However, these methods don't fully account for cyber-attacks' impact on decision networks. Given the vulnerability of deep neural networks to minor input changes, this study integrates the CROWN-IBP defense training to enhance DRL's robustness against cyber-attacks, supporting the development of resilient scheduling strategies.
%Recently, deep reinforcement learning (DRL) has been implemented in real-time energy management systems as a model-free method for solving stochastic decision-making problems without the need for prior environmental knowledge. In \cite{zhang2021soft}, the authors considered the uncertainties of RES and loads, and used the soft actor-critic (SAC) algorithm to schedule the outputs of energy units. A DRL algorithm is proposed in \cite{yang2021dynamic}, which can adapt to stochastic fluctuations in power sources and loads, improving the scheduling policy's quality. However, these approaches utilize decision-making network directly for scheduling and overlook the potential impact of cyber-attacks on the decision-making network. Unfortunately, deep neural networks are often fragile, and even imperceptibly small input modifications can significantly affect the model. As a result, it is essential to understand the robustness and security of deep neural networks, particularly their resilience against malicious and carefully crafted inputs. To tackle the aforementioned challenges, this study employs the CROWN-IBP certified defense training method to enhance the DRL framework. This significantly improves its resilience to cyber-attacks and aids in formulating a resilient scheduling strategy.
%To tackle the aforementioned challenges, this study utilizes DRL in the IES cyber-attack scenario to formulate a resilient scheduling strategy.
%However, these approaches do not consider the impact of stealthy FDIA on the scheduling strategy. From the operator's perspective, the impact of the attack should be minimized. For instance, LR attacks can cause deviations in load values, which will impact the state observations of the DRL. Although these deviations do not reflect the real environment, they can mislead the agent to make suboptimal strategies, resulting in uneconomical operation. Therefore, to address the above problems and fill the research gap, this study attempts to apply DRL to IES in the cyber-attack scenario to formulate resilient scheduling strategy under attack.
\begin{table*}[ht]
\centering
\caption{Comparison of resilience scheduling methods for electric energy systems under cyber-attack scenarios}\label{tab1}
\begin{tabular}{ccccccc}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
Reference & Cyber-attack & \multicolumn{2}{c}{Muti-uncertainties} & DR & Test system & Solving method \\
& & RES & Load & & & \\
\midrule
\cite{tajalli2020resilient} & DOSA & $\times $ & $\times $ & $\times $ & Microgrid & Average consensus-based algorithm \\ \cite{chu2022mitigating} & LAA & $\times $ & $\times $ & $\times $ & Power grid & \begin{tabular}[c]{@{}c@{}}LAA detection + IBR droop control+ \\ Distributionally robust method\end{tabular} \\
\cite{huang2021distributed} & Noncolluding and colluding attack & $\times $ & $\times $ & $\times $ & IES (electricity-heat) & \begin{tabular}[c]{@{}c@{}}Attack detection and isolation+ \\ Distributionally robust method\end{tabular} \\
\cite{zhao2021cyber} & FDIA (LR) & $\surd $ & $\times $ & $\times $ & MEDS & DRO \\
\cite{zhao2020cyber} & FDIA(LR) & $\surd $ & $\times $ & $\times $ & WES & DRO \\
\cite{hua2019optimal} & $\times $ & $\surd $ & $\surd $ & $\times $ & Microgrid & DRL algorithm \\
Proposed & FDIA(HLR) & $\surd $ & $\surd $ & $\surd $ & IES (electricity-gas-heat) & DRL algorithm \\
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\end{tabular}%
\label{tab:addlabel}%
\end{table*}%
Table \ref{tab1} compares our proposed approach with current resilient scheduling methods for cyber-attack scenarios, emphasizing its distinctive features. In the table, %symbols “$\surd $” and “$\times $” indicate whether a particular feature is considered or not;
“DRO” refers to the distributionally robust optimization; “DOSA” refers to the denial of service attack; “LAA” denotes the load-altering attack; “MEDS” represents the multivector energy distribution system; “WES” denotes the water-energy system; “IBR” refers to the inverter-based resource. %highlighting the unique features of our method
The existing research gaps in electric energy system scheduling considering cyber-attack can be summarized as: (1) Dealing with multi-uncertainties of RES and loads simultaneously is challenging, (2) DR problem and the potential of flexible resources for resilient IES has not been fully exploited, and (3) To the authors' knowledge, few studies use DRL-based methods for IES resilient scheduling against cyber-attacks.
%To sum up, the existing research gaps in power-energy system scheduling considering cyber-attack can be summarized as follows: (1) The uncertainty of RES and loads has a significant impact on the scheduling strategy, but dealing with the multi-uncertainties of RES and loads simultaneously remains challenging. (2) Existing studies rarely consider the demand response problem in resilient scheduling, and the potential of flexible resources on the demand side has not been fully exploited in the scheduling problem. (3) More importantly, to the authors’ best knowledge, existing studies all use model-based optimization methods, and few studies use DRL-based methods for resilient scheduling.
The main contributions of this work are summarized below:
(1) A resilient scheduling method for IES is developed to counter cyber-attacks, addressing IES's vulnerability in such scenarios. It analyzes the role of energy storage and DR in mitigating attacks and introduces a dynamic pricing mechanism that combines time-of-use (TOU) and real-time pricing, considering user thermal comfort through the predicted average voting index. This method, distinct from traditional ones, employs a DRL agent to establish optimal pricing strategies under source-load uncertainties.
%A real-time resilient scheduling method is proposed for IES to address cyber-attacks. The IES achieves optimal energy coordination under normal conditions, but is vulnerable to cyber-attacks. The IES dispatching situation and the mitigation effects of energy storage devices and DR on the attack are analyzed in the cyber-attack scenario. Furthermore, to further explore the potential of demand response, a dynamic pricing mechanism with time-of-use (TOU) and real-time pricing is designed, and considers the thermal comfort requirements of users by introducing the predicted average voting index. Notably, distinct from traditional pricing mechanisms, our approach leverages DRL agent to explore and establish optimal pricing strategies, especially in the face of source-load uncertainties.
%In addition, to further explore the potential of demand response, a dynamic pricing mechanism with time-of-use (TOU) and real-time pricing is proposed, and considers the thermal comfort requirements of users by introducing the predicted average voting index.
%We introduce FDIA into IES, analyze its impact on the scheduling scheme and economy of IES, and propose a real-time resilient scheduling method for IES against cyber-attack. In this method, a dynamic pricing mechanism integrating time-of-use (TOU) and real-time pricing is designed to fully exploit the adjustment potential of the electricity-gas-heat flexible loads. To the authors' best knowledge, this is the first cyber-attack resilient scheduling work using DRL for integrated demand response-enabled IES.
(2) We consider the uncertainty of state observation in reinforcement learning, take the cyber-attack suffered by IES as the uncertain input of state observation, and transform the dynamic energy scheduling problem of IES under cyber-attack into a state adversarial Markov decision process (SA-MDP). The SA-MDP model handles continuous state and action spaces to reflect the nature of the IES problem. %In this paper, the dynamic energy dispatch problem of an IES considering cyber-attack is transformed into a novel state-adversarial Markov decision process (SAMDP) model, which is used to describe an economic dispatch strategy maximizing operator benefit when the system is attacked. The process is handled in continuous state and action spaces, consistent with the nature of the IES dynamic energy dispatch problem.
(3) A proposed DRL algorithm called state-adversarial soft actor-critic (SA-SAC) adds a regularizer to the policy network during training to reflect the hazard degree of cyber-attacks. A certified defense training method is employed to train policy neural network, aiming to identify the optimal learning policy that demonstrates resilience against cyber-attacks. The superiority of the proposed algorithm is verified in the scenario where cyber-attacks are used as state observation adversary. %We propose a novel DRL algorithm, namely state-adversarial soft actor-critic (SA-SAC). Compared with the original SAC algorithm, the proposed algorithm adds a regularizer to the policy network during training that reflects the hazard degree of cyber-attack. This regularizer is added to the policy loss gradient for gradient updating, thus determining the optimal learning policy with resistance to attack.
%(4) Numerical simulation results for the IES in North China demonstrate the resilience of the proposed method against the attack and its superiority. In addition, to evaluate the adaptability of the method to multi-uncertainties, several special scenarios are set up for simulation.
%(4) Numerical simulation results for a real-world IES in North China demonstrate the resilience of the proposed method against the attack and its superiority. In addition, to evaluate the adaptability of the method to multi-uncertainties, the proposed method has been tested under several special scenarios.
\section{Modeling of IES and LR Attacks}
\subsection{Structure Modeling of IES}
The overall structure of an IES is shown in Fig. \ref{CIES}, including micro-gas turbine (MT), wind turbine (WT), electricity storage device (ESD), heat storage device (HSD), electric boiler (EB), power to gas (P2G), electric load, heat load and gas load. The system integrates and distributes electricity, heat, and natural gas to customers via electric power, district heating, and natural gas networks. %The system integrates three forms of energy: electricity, heat and gas. Energy suppliers are mainly power and gas grids. Electricity, heat and natural gas are transmitted to customers through the electric power network, district heating network and natural gas network, respectively.
%\vspace{-4mm}
\begin{figure}[htpb]
\centering
\includegraphics[width=3.2in,height=2.2in]{fig/CIES-Page-1.jpg}
\caption{Schematic diagram of the IES.}
\label{CIES}
\end{figure}
\subsubsection{Modeling of Load}
%According to the nature of the demand-side load, the electric loads are divided into fixed loads and flexible loads.%
\paragraph{Electric Load Model}
The study classifies electrical loads into fixed and flexible loads. Flexible loads, primarily time-shifted electrical loads, have fixed power consumption but flexible power consumption time. Users receive subsidies to compensate for the impact of load shifting. The actual electrical load, time-shifted electrical load, and reimbursed electrical load at period $t$ are %formulated as follows: %to achieve peak shaving and valley filling
%In this study, electrical loads have price-based demand responsiveness to achieve peak shaving and valley filling of the load curve. They are categorized into fixed loads and flexible loads based on their demand characteristics. The flexible loads are mainly time-shifted electrical loads, which are characterized by constant total power consumption and flexible changes in power consumption time. Since load shifting inevitably affects the user experience, a subsidy is provided to the users.
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq1}
{{P}_{load,t}}={{P}_{basic,t}}-{{P}_{TSE,t}}+{{P}_{PBE,t}}\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq2}
{{P}_{TSE,t}}={{P}_{basic,t}}*\sigma _{t}^{TSE}*\varsigma _{t}^{TSE}\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq3}
{{P}_{PBE,t}}=\sum\limits_{i=1}^{t-1}({\omega _{i,t}^{PBE}}*{{P}_{TSE,t}})\quad \forall t
\end{equation} }where, in period $t$, ${{P}_{basic,t}}$ and ${{P}_{load,t}}$ represent initial and actual electrical loads; ${{P}_{TSE,t}}$ and ${{P}_{PBE,t}}$ denote the actual time-shifted load amount and actual load repayment amount. $\varsigma_{t}^{TSE}$ indicates price level, $\sigma_{t}^{TSE}$ reflects load change sensitivity to price. The binary variable $\omega_{i,t}^{PBE}$, determined by the load transfer probability ${{Pr}_{PBE}}$, signifies the repayment of the time-shifted electrical load, with 1 indicating repayment and 0 indicating no repayment\cite{nakabi2021deep}.%where ${{P}_{basic,t}}$ and ${{P}_{load,t}}$ denote the initial and actual electrical loads in period $t$; ${{P}_{TSE,t}}$ and ${{P}_{PBE,t}}$ are the time-shifted electrical load value and electrical load payback value; $\varsigma _{t}^{TSE}$ is the price level; $\sigma _{t}^{TSE}$ is the sensitivity factor, which represents the percentage of electric load change as the price fluctuates; the binary variable $\omega _{i,t}^{PBE}$ represents the repayment of time-shifted electrical load and is determined by the load transfer probability ${{Pr}_{PBE}}$. A value of 1 indicates that the load transfer is repaid, while 0 indicates that it is not repaid\cite{nakabi2021deep}. %$\omega _{i,t}^{PBE}$ is a binary variable indicating whether the time-shifted electrical load is repaid or not, if $\omega _{i,t}^{PBE}=1$ means that the load transfer is repaid otherwise it is not repaid, which is determined by the probability $P$ that the time-shifted electrical load will be reimbursed. %, when its higher time-shifted load is positive, lower time-shifted load is negative
%${{P}_{load,t}}$ is the actual electrical load in period $t$; ${{P}_{TSE,t}}$ is the time-shifted electrical load value; ${{P}_{PBE,t}}$ is the time-shifted electrical load payback value; $\varsigma _{t}^{TSE}$ is the price level;
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq4}
{{Pr}_{PBE,t}}=clip\left( \frac{-\varsigma _{t}^{TSE}*sign\left( {{P}_{TSE,t}} \right)}{2}+\frac{t-i}{{{\eta }_{TSE}}},0,1 \right)\text{ }\forall t\text{ }
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq5}
clip(X,0,1)=\left\{ \begin{array}{*{35}{l}}
\text{ }0, & \text{ if }X<0 \\
X, & \text{ if }0\le X\le 1 \\
\text{ }1, & \text{ if }X>1 \\
\end{array} \right.
\end{equation} }where the probability $Pr_{PBE}$ relies on the price level ${{\varsigma }_{t}}$ and the patience factor $\eta $; $clip()$ limits the load transfer probability. %$\eta $ denotes the number of hours when the shifted load is repaid, the larger it is the higher the probability of the load being repaid; is used to
%modeled in a similar manner to the electric load model, as follows
\paragraph{Gas Load Model}
The actual gas load, time-shifted gas load, and reimbursed gas load at period $t$ are formulated as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq6}
{{Q}_{load,t}}={{Q}_{basic,t}}-{{Q}_{TSQ,t}}+{{Q}_{PBQ,t}}\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq7}
{{Q}_{TSQ,t}}={{Q}_{basic,t}}*\sigma _{t}^{TSQ}*\varsigma _{t}^{TSQ}\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq8}
{{Q}_{PBQ,t}}=\sum\limits_{i=1}^{t-1}({\omega _{i,t}^{PBQ}}*{{Q}_{TSQ,t}})\quad \forall t
\end{equation} }where ${{Q}_{basic,t}}$ and ${{Q}_{load,t}}$ are the initial and actual gas loads in period $t$; ${{Q}_{TSQ,t}}$ is the time-shifted gas load value; ${{Q}_{PBQ,t}}$ is the time-shifted gas load payback value; $\varsigma _{t}^{TSQ}$ is the price level that influences the magnitude of the time-shifted gas load; $\sigma _{t}^{TSQ}$ is the sensitivity factor, which represents the percentage of gas load change as the price fluctuates; $\omega _{i,t}^{PBQ}$ is a binary variable that represents the repayment of the time-shifted gas load and is determined by the load transfer probability ${{Pr}_{PBQ}}$.
%indicating whether the time-shifted gas load is repaid or not, if $\omega _{i,t}^{PBQ}=1$ means that the load transfer is repaid otherwise it is not repaid.${{Q}_{load,t}}$ denotes the actual gas load in period $t$;
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq9}
{{Pr}_{PBQ,t}}=\operatorname{clip}\left( \frac{-\varsigma _{t}^{TSQ}*\operatorname{sign}\left( {{Q}_{TSQ,t}} \right)}{2}+\frac{t-i}{{{\eta }_{TSQ}}},0,1 \right)\quad \forall t\text{ }
\end{equation} }%where $P$ is the probability that the time-shifted gas load will be reimbursed.
%clip()公式
%{
%{\setlength\abovedisplayskip{2pt}
%\setlength\belowdisplayskip{2pt}
%\begin{equation}\label{Eq10}
%\operatorname{clip}(Y,0,1)=\left\{ %\begin{array}{*{35}{l}}
% 0, & \text{ if }Y<0 \\
% Y, & \text{ if }0\le Y\le 1 \\
% 1, & \text{ if }\quad Y>1 \\
%\end{array} \right.
%\end{equation} }
%
%as follows
\paragraph{Heat Load Model}
Considering the thermal inertia of the building in the heat load model, the heating power stored in buildings can be calculated via the first-order thermodynamic model. In addition, users have some flexibility in their temperature perception, which can be utilized to reduce the thermal load while still maintaining their preferred thermal comfort level. Based on this, the actual thermal load and the reduced thermal load are %follows:there is some degree of flexibility in the user's temperature perception, which can be leveraged to reduce the thermal load within the user's acceptable thermal comfort range.
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq12}
{{H}_{load,t}}={{H}_{basic,t}}-{{H}_{CH,t}}\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eqadd3}
H_{C H, t}=\varsigma_t^{CH} H_{C H, t}^{\max } \quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq13}
0\le {{H}_{CH,t}}\le H_{CH,t}^{\max }
\end{equation} }where ${{H}_{load,t}}$ denotes the actual thermal load in period $t$; ${{H}_{basic,t}}$ denotes the initial thermal load in period $t$; ${{H}_{CH,t}}$ denotes the users’ cuttable thermal load in period $t$; $H_{CH,t}^{\max }$is the maximum cuttable thermal power; $\varsigma _{t}^{CH}$ denotes the price level that influences the amount of the cuttable thermal load.% $\varsigma_t^{H C}$ is the price level.
To quantify the acceptable user thermal comfort, the predicted mean vote (PMV) index is introduced as follows\cite{mao2019pmv}:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq14}
P M V=2.43-\frac{3.76\left(T_s-T_{i n, t}\right)}{M\left(I_{c l}+0.1\right)}
\end{equation} }where ${{I}_{cl}}$ stands for the thermal resistance of clothing; ${{T}_{in,t}}$ denotes the indoor temperature. ${{T}_{s}}$ is the average temperature at which a person's skin feels comfortable; $M$ denotes the human body's metabolic rate. Here, the indoor comfort temperature is taken as 21 degrees Celsius.
To ensure indoor thermal comfort, it is recommended that the PMV value stays within the range of [-0.5, 0.5] \cite{ku2014automatic}. In this study, we limit the nighttime PMV to the range of [-0.9, 0.9]. The PMV change range is
%The indoor temperature change range is:
%PMV范围公式
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq15}
\begin{cases}|P M V| \leq 0.9, & {[1: 00-7: 00] \cup [20: 00-24: 00]} \\ |P M V| \leq 0.5, & {[8: 00-19: 00]}\end{cases}
\end{equation} }
Additionally, the transient heat balance equation of the building is employed to depict the impact of heat variations in the heating system on the building temperature \cite{li2021optimal}. This equation allows us to calculate the building's heat load:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq11}
{{H}_{basic,t}}=\frac{\left[ {{T}_{in,t}}-{{T}_{\text{out },t}} \right]+\frac{K\cdot F}{{{c}_{\text{air }}}\cdot {{\rho }_{\text{air }}}\cdot V}\cdot \Delta t\cdot \left[ {{T}_{in,t-1}}-{{T}_{\text{out },t}} \right]}{\frac{1}{K\cdot F}+\frac{1}{{{c}_{\text{air }}}\cdot {{\rho }_{\text{air }}}\cdot V}\cdot \Delta t}
\end{equation} }where ${{T}_{out,t}}$ represents the outdoor temperature in period $t$; $K$ denotes the comprehensive heat transfer coefficient; $F$ and $V$ denote the surface area and volume of the building, respectively; ${{c}_{air}}$ and ${{\rho }_{air}}$ refer to the specific heat capacity and density of the indoor air, respectively.
\subsubsection{Energy Storage Device Model}
This study categorizes the energy storage devices discussed into two types: ESD and HSD. The ESD and HSD models are formulated as:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq16}
\left\{\begin{array}{l}
C_{t+1}^{E S D}=C_t^{E S D}+\left(\eta_{c h}^{E S D} P_{c h, t}^{E S D}-P_{d c, t}^{E S D} / \eta_{d c}^{E S D}\right) \Delta t \\
C_{t+1}^{H S D}=C_t^{H S D}+\left(\eta_{c h}^{H S D} H_{c h, t}^{H S D}-H_{d c, t}^{H S D} / \eta_{d c}^{H S D}\right) \Delta t
\end{array} \forall t\right.
\end{equation} }where $P_{ch,t}^{ESD}$ and $P_{dc,t}^{ESD}$ denote the charging and discharging powers of the ESD in period $t$, respectively; $H_{ch,t}^{HSD}$ and $H_{dc,t}^{HSD}$ denote the HSD heat storage and release powers; $\eta _{ch}^{ESD}$ and $\eta _{dc}^{ESD}$ are the ESD charge and discharge efficiencies, respectively; $\eta _{ch}^{HSD}$ and $\eta _{dc}^{HSD}$ are the HSD heat storage and release efficiencies; $C_{t}^{ESD}$ and $C_{t}^{HSD}$ are the capacity values of the ESD and HSD.
The state of charge (SOC) of the energy storage device is %in period $t$, respectively at time $t$ is defined as follows
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq17}
\left\{\begin{array}{l}
S O C_t^{E S D}=C_t^{E S D} / C_{\max }^{E S D} \\
S O C_t^{H S D}=C_t^{H S D} / C_{\max }^{H S D}
\end{array} \forall t\right.
\end{equation} }where $C_{\max }^{ESD}$ and $C_{\max }^{HSD}$ denote the maximum capacity of the ESD and HSD, respectively.
\subsubsection{Electric Boiler Model}
%The relationship between the input and output power of the EB can be described as:
The operation of the EB is modelled as
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq18}
{{H}_{EB,t}}={{\eta }^{EB}}{{P}_{EB,t}}\quad \forall t
\end{equation} }where ${{P}_{EB,t}}$ denotes the absorbed electrical energy in period $t$; ${{H}_{EB,t}}$ denotes the heat output of the EB in period $t$; ${{\eta }^{EB}}$ is the conversion efficiency of the EB.
\subsubsection{Power to Gas Model}
The natural gas production by P2G conversion is correlated with the electricity consumption in the following manner: %The relationship between the natural gas produced by P2G conversion and the electricity consumption is:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq19}
{{Q}_{P2G,t}}=\frac{{{\eta }_{P2G}}{{P}_{P2G,t}}}{HHV}\quad \forall t
\end{equation} }where ${{Q}_{P2G,t}}$ denotes the natural gas produced by P2G in period $t$; ${{P}_{P2G,t}}$ denotes the absorbed electrical energy in period $t$; $HHV$ is the calorific value of natural gas; ${{\eta }_{P2G}}$ is the conversion efficiency of P2G.
\subsubsection{Micro-Gas Turbine Model}
%The relationship between input power and output power of MT can be described as:
The output powers of the MT are formulated by
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq20}
{{P}_{MT,t}}=(1-\eta _{h}^{MT}-\eta _{loss}^{MT}){{Q}_{MT,t}}HHV\quad \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq21}
{{H}_{MT,t}}=\left( 1-\eta _{e}^{MT}-\eta _{\text{loss }}^{MT} \right){{Q}_{MT,t}}HHV\quad \forall t
\end{equation} }where ${{H}_{MT,t}}$ is the heat output of the MT in period $t$; ${{P}_{MT,t}}$ is the electrical power output of the MT; ${{Q}_{MT,t}}$ is the volume of natural gas consumed by the MT; $\eta _{e}^{MT}$ and $\eta _{h}^{MT}$ are the electric power and heat generation efficiency coefficients of the MT; $\eta _{loss}^{MT}$ is the heat loss coefficient. %$\eta _{e}^{MT}$ is the electric power generation efficiency coefficient of the MT; $\eta _{h}^{MT}$ is the heat generation efficiency coefficient;
%in period $t$ of the MT in period $t$
\vspace{-3mm}
\subsection{Modeling of LR Attack}
%Communication in the IES network is facilitated by a supervisory control and data acquisition (SCADA) system. The SCADA system collects all available real-time measurement data and uploads them to the EMS. In the electric network, node injected power and branch power are the common types of measurements. In the heating network, mass flow and temperature are the common types of measurements. In the natural gas network, node pressure, branch flow and gas source injection flow are the common types of measurements.
LR attack is a type of FDIA that alters load data, leading to incorrect meter readings and misleading system operators in making scheduling decisions \cite{zhao2021cyber}.%As a special type of FDIA, LR attack can affect the operational scheduling of the system by falsifying load data. The meter readings after the attack may deviate from the actual readings, leading the system operator to make decisions based on the falsified load demand \cite{zhao2021cyber}.
\subsubsection{Bad Data Detection in IES State Estimation}
%State estimation is a powerful tool to detect FDIA by processing raw data measurements, but this detection mechanism is only resistant to some common and less stealthy cyber-attacks. For some well-designed attacks, detection vulnerabilities can be exploited to evade state estimation.
%State estimation is powerful in detecting FDIA through processing raw data measurements, but detection vulnerabilities can be exploited by well-designed attacks to evade it.
The state estimation in an IES is expressed as follows\cite{zang2019robust}:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq22}
z=h(x)+e
\end{equation} }where $z\in \left\{ {{z}_{e}},{{z}_{g}},{{z}_{h}} \right\}$ is the measurement variables in the electricity, gas and heat networks; $x\in \left\{ {{x}_{e}},{{x}_{g}},{{x}_{h}} \right\}$ represents the state variables; $h(x)$ denotes the measurement function of $x$; $e$ is the measurement noise.
%During the transmission of measurement data, errors may generate, causing the measurement data to deviate from the actual data, and this measurement data is called bad data. where the goal of state estimation is to
%Bad data can occur during measurement data transmission, resulting in deviation from actual data.
The common bad data detection method is based on the residual test principle, aiming to minimize the measurement residual $r$, which is defined as:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq23}
||r|{{|}_{2}}=||z-h(x)|{{|}_{2}}
\end{equation} }
One can utilize the largest normalized residual to detect and recognize measurement errors, and we evaluate the Euclidean norm of the residual against a predefined threshold value $\tau$:%The largest normalized residual can be used to detect and identify measurement errors, and we compare Euclidean norm of residual with the threshold value $\tau $:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq24}
||r|{{|}_{2}}=||z-h(x)|{{|}_{2}}\le \tau
\end{equation} }
If the residual is less than the threshold, the state estimate is valid; otherwise, it indicates the presence of bad data. % For IES all stealthy FDIAs are designed with the goal of evading state estimation detection.
\subsubsection{HLR Attack Modeling}
The less mature state estimation of the heating system make it a vulnerable spot in the IES. The "barrel principle" means that a single subsystem failure may propagate to other energy subsystems. However, the existing literature lacks research on resilience strategies for FDIAs on heating systems. This study focuses on a stealthy HLR attack, the indoor temperature delay scaling attack (ITDSA), notable for its economic impact. The ITDSA is characterized as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq25}
\tilde{\tau }_{in}^{t}=\left\{ \begin{array}{*{35}{l}}
\tau _{in}^{t} & t\notin {{\Gamma }^{a}} \\
\left( 1+\lambda \left( 1-{{e}^{-a\left( t-{{t}_{0}} \right)}} \right) \right)\tau _{in}^{t} & t\in {{\Gamma }^{a}} \\
\end{array} \right.
\end{equation} }where $\tau _{in}^{t}$ is the indoor temperature; $\lambda $ and $a$ are the parameters of the HLR attack \cite{ding2022cyber}. According to Eq. (\ref{Eq11}), directly manipulating the indoor temperature results in a corresponding alteration in the heat load. %attacking the indoor temperature directly leads to a change in the heat load.
\vspace{-2mm}
\section{Formulation of IES scheduling model}
%This section presents an IES scheduling approach aiming to maximize operators' profit and a dynamic pricing mechanism is designed that integrates TOU and real-time pricing.
In this section, we formulate the IES dispatch model that pursues the maximum net profit for the operator. A dynamic pricing mechanism that integrates TOU and real-time prices is proposed to fully exploit the demand response potential.
\vspace{-3mm}
\subsection{Dynamic Pricing Mechanism}
%In this study, the energy price is dynamically set by the operator in real-time and takes into account the price fluctuations determined by the behavior of the users. The TOU pricing mechanism is used for heat load and the designed dynamic pricing mechanism is used for electric and gas loads. Specifically, based on the strategy learned in Section 4 and the TOU price
%Based on the price mechanism in \cite{nakabi2021deep}, a dynamic pricing mechanism is designed to determine electricity and gas prices that can fluctuate around the mean of the TOU price. To consider the impact of future revenues, we set a continuous price level ${{\varsigma }_{t}}$ that determines the degree of price fluctuation. The specific pricing mechanism is as follows:
%This section proposes a dynamic pricing mechanism for an IES that combines the benefits of TOU and real-time prices. Initially, the energy supplier sets the pricing benchmark based on the TOU price, providing customers with a predictable and incentivized pricing structure that promotes sustainable and efficient energy consumption. Then, the energy supplier adjusts the price in real-time based on feedback from the grid and the status of devices within the IES. This mechanism effectively guides customers to adjust their energy demand, ultimately increasing the net profit of the IES operator. \cite{li2021coordinating}
%Traditional dynamic pricing mechanisms were typically determined by the ratio of renewable energy generation to load demand.
Our dynamic pricing method integrates DRL to manage electricity, gas, and heat prices. Using TOU prices as a benchmark, it offers customers predictable pricing, while allowing the supplier to adjust prices in real-time based on grid feedback and IES device status. This mechanism guides customer demand and increases IES operator profits. Dynamic price limits, tied to TOU price peaks and valleys, ensure balanced prices without extremes. The mechanism is
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq26}
\left\{\begin{array}{l}
\lambda_{rt, t}^P=\varpi_{P, m}+\varsigma_t^{TSE} * k_P \\
\lambda_{rt, t}^Q=\varpi_{Q, m}+\varsigma_t^{TSQ} * k_Q \quad \forall t \\
\lambda_{rt, t}^H=\varpi_{H, m}+\varsigma_t^{CH} * k_H
\end{array}\right.
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq27}
\left\{\begin{array}{l}
\varpi_{P, m}=\zeta_P\left(\varpi_{s t, u}^P+\varpi_{s t, d}^P\right) \\
\varpi_{Q, m}=\zeta_Q\left(\varpi_{s t, u}^Q+\varpi_{s t, d}^Q\right) \quad \forall t \\
\varpi_{H, m}=\varpi_{s t, t}^H
\end{array}\right.
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq28}
\left\{\begin{array}{c}
l_P \varpi_{s t, d}^P \leq \lambda_{r t, t}^P \leq l_P \varpi_{s t, u}^P \\
l_Q \varpi_{s t, d}^Q \leq \lambda_{r t, t}^Q \leq l_Q \varpi_{s t, u}^Q \\
\varpi_{s t, t}^H \leq \lambda_{r t, t}^H \leq l_H \varpi_{s t, t}^H
\end{array} \forall t\right.
\end{equation} }where $\varpi _{st,t}^{P}$, $\varpi _{st,t}^{Q}$ and $\varpi _{st,t}^{H}$ are the TOU prices of electricity, gas and heat subsystems, respectively; $\varpi_{s t, u}^P$, $\varpi_{s t, u}^Q$ and $\varpi_{s t, d}^P$, $\varpi_{s t, d}^Q$ are the peak and off-peak values of TOU prices of electricity and gas subsystems; $\lambda _{rt,t}^{P}$, $\lambda _{rt,t}^{Q}$ and $\lambda _{rt,t}^{H}$ are the real-time electricity, gas and heat prices issued by the operator to users; ${{\varpi }_{P,m }}$, ${{\varpi }_{Q,m }}$, and ${{\varpi }_{H,m }}$ are the benchmark prices for electricity, gas, and heat subsystem; ${{l}_{P}}$, ${{l}_{Q}}$, and ${{l}_{H}}$ denote the threshold coefficients for real-time electricity, gas and heat prices; ${{k}_{P}}$, ${{k}_{Q}}$ and ${{k}_{H}}$ are the parameters that determine the change in the prices of electricity, gas and heat.
%The proposed dynamic pricing mechanism, more adaptable than TOU pricing, allows IES operators to quickly adjust prices based on network changes, offering customers real-time cost feedback and guiding energy demand adjustment. It also enables real-time price optimization during cyber-attacks to reduce their impact.
%The proposed dynamic pricing mechanism is more flexible and responsive than TOU price. Through the dynamic pricing mechanism, IES operators can swiftly adjust prices in response to changes in network conditions. This provides customers with immediate feedback on the cost of electricity, guiding them to adjust their demand for energy accordingly. In addition, even in the event of the cyber-attack, the system can adjust and optimize the pricing structure in real-time to minimize the impact.
\vspace{-2mm}
\subsection{Objective Function}
The objective function aims to maximize the profit of the operator and is given by:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq29}
%maxF=\sum\limits_{t=1}^{T}{(Re{{v}_{t}}-Cos{{s}_{t}}})
\max F=\sum_{t=1}^T\left(Re{{v}_{t}}-Cos{{t}_{t}}\right)
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq30}
\begin{aligned}
Re{{v}_{t}} & =\lambda_{rt, t}^P \sum_{\text {loads }} P_{\text {load }, t}+\lambda_{rt, t}^Q \sum_{\text {loads }} Q_{\text {load }, t}+\lambda_{rt, t}^H \sum_{\text {loads }} H_{\text {load }, t} \\
& +\lambda_{\text {sell }, t}^P P_{\text {sell }, t}+\lambda_{\text {sell }, t}^Q Q_{\text {sell }, t}
\end{aligned}
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq31}
\begin{aligned}
Co{{s}_{t}} & =P_{b u y, t} \lambda_{\text {buy }, t}^P+Q_{b u y, t} \lambda_{b u y, t}^Q+\gamma_{C H} H_t^{C H} \\
& +\gamma_{T S E} \sum_{T S E} \max \left(P_t^{T S E}, 0\right) \\
& +\gamma_{T S Q} \sum_{T S Q} \max \left(P_t^{T S Q}, 0\right)
\end{aligned}
\end{equation} }where $Re{{v}_{t}}$ and $Co{{s}_{t}}$ are the total income and the operating cost of the operator; $\lambda _{buy,t}^{P}$ and $\lambda _{sell,t}^{P}$ are the unit prices of electricity purchased and sold by the IES from and to the power grid; $\lambda _{buy,t}^{Q}$ and $\lambda _{sell,t}^{Q}$ are the unit prices of natural gas purchased and sold by the IES from and to the gas grid; ${{P}_{buy,t}}$ and ${{P}_{sell,t}}$ are the electricity purchased and sold by the IES from and to the power grid, respectively; ${{Q}_{buy,t}}$ and ${{Q}_{sell,t}}$ are the natural gas purchased and sold by the IES from and to the gas grid; ${{\gamma }_{TSE}}$ and ${{\gamma }_{TSQ}}$ are the unit compensation cost of time-shifted electric load and time-shifted gas load; ${{\gamma }_{CH}}$ is the unit compensation cost of cutting heat load.% ${{\lambda }_{H,t}}$ denotes the TOU price of the heat load in period $t$.
%$Cos{{t}_{t}}$ is the operating cost of IES;
\vspace{-4mm}
\subsection{Constraint Conditions}
\subsubsection{Energy Balance Constraints}
The energy balance constraints are given below: %the constraints of power balance, heat balance and natural gas balance can be expressed as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq32}
{{P}_{\text{grid,t }}}+{{P}_{WT,t}}+{{P}_{ESD,t}}+{{P}_{MT,t}}-{{P}_{EB,t}}-{{P}_{P2G,t}}={{P}_{\text{load,t }}}\text{ }\forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq33}
{{H}_{MT,t}}+{{H}_{HSD,t}}+{{H}_{EB,t}}={{H}_{\text{load,t }}}\text{ }\forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq34}
{{Q}_{\text{grid,t }}}+{{Q}_{P2G,t}}-{{Q}_{MT,t}}={{Q}_{\text{load,t }}}\text{ }\forall t
\end{equation} }where ${{P}_{WT,t}}$ is the output of the WT; ${{P}_{ESD,t}}$ and ${{H}_{HSD,t}}$ are the charging/discharging power of ESD and HSD, respectively.
\subsubsection{ESD Operating Constraints}
In order to ensure that the charging and discharging powers of the ESD are within the allowable ranges, the charging and discharging power are constrained as:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq35}
\left\{\begin{array}{l}
0 \leq P_{d c, t}^{E S D} \leq \theta_{d c, t}^{E S D} P_{d c, \text { max }}^{E S D} \\
0 \leq P_{c h, t}^{E S D} \leq \theta_{c h, t}^{E S D} P_{c h, \max }^{E S D}
\end{array} \forall t\right.
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq36}
\theta _{dc,t}^{ESD}+\theta _{ch,t}^{ESD}\le 1
\end{equation} }where $P_{ch,\max }^{ESD}$ and $P_{dc,\max }^{ESD}$ are the maximum charging and discharging powers of the ESD, respectively; $\theta _{ch,t}^{ESD}$ and $\theta _{dc,t}^{ESD}$ are Boolean variables that represent the ESD running states.% of the ESD, respectively. When the values of $\theta _{ch,t}^{ESD}$ and $\theta _{dc,t}^{ESD}$ are 1, the ESD is in a state of charge and discharge; when their values are 0, the ESD is neither charged nor discharged. %that represent the running state of the ESD, respectively. When the values of $\theta _{ch,t}^{ESD}$ and $\theta _{dc,t}^{ESD}$ are 1, the ESD is in a state of charge and discharge; when their values are 0, the ESD is neither charged nor discharged.
To ensure consistent initial conditions and within-range capacity in each cycle, the ESD capacity must satisfy:%To ensure the same initial conditions in each scheduling cycle and the capacity of the ESD are within the allowable ranges, the capacity must meet the following constraints:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq37}
\left\{\begin{array}{l}
C_{\min }^{E S D} \leq C_t^{E S D} \leq C_{\max }^{E S D} \\
C_0^{E S D}=C_{\text {end }}^{E S D}
\end{array} \forall t\right.
\end{equation} }where $C_{\min }^{ESD}$ and $C_{\max }^{ESD}$ refer to the ESD's minimum and maximum capacities; $C_{0}^{ESD}$ and $C_{end}^{ESD}$ are the starting and ending capacities of the ESD, respectively.
%Since the HSD shares similar operating constraints with the ESD, its constraints are not reiterated.
%\iffalse
\subsubsection{HSD Operating Constraints}
Similar to the ESS, the HSD output constraints are
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq38}
\left\{\begin{array}{l}
0 \leq H_{d c, t}^{H S D} \leq \theta_{d c, t}^{H S D} H_{d c, \max }^{H S D} \\
0 \leq H_{c h, t}^{H S D} \leq \theta_{c h, t}^{H S D} H_{c h, \max }^{H S D}
\end{array} \forall t .\right.
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq39}
\theta _{dc,t}^{HSD}+\theta _{ch,t}^{HSD}\le 1
\end{equation} }where $H_{ch,t}^{HSD}$ and $H_{dc,t}^{HSD}$ are the HSD heat storage and release powers, respectively; $\theta _{ch,t}^{HSD}$ and $\theta _{dc,t}^{HSD}$ are Boolean variables that represent the HSD running states. %When the values of $\theta _{ch,t}^{HSD}$ and $\theta _{dc,t}^{HSD}$ are 1, the HSD is in a state of storing and releasing heat, respectively; when their values are 0, the HSD neither stores nor releases heat.
The capacity of HSD must meet the following constraints:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq40}
\left\{\begin{array}{l}
C_{\min }^{H S D} \leq C_t^{H S D} \leq C_{\max }^{H S D} \\
C_0^{H S D}=C_{\text {end }}^{H S D}
\end{array} \forall t\right.
\end{equation} }where $C_{\min }^{HSD}$ and $C_{\max }^{HSD}$ refer to the HSD's minimum and maximum capacities; $C_{0}^{HSD}$ and $C_{end}^{HSD}$ are the starting and ending capacities of the HSD.
%\fi
\subsubsection{Electric Boiler Constraint}
The output power of the EB must meet the constraint:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq41}
0\le {{H}_{EB,t}}\le {{H}_{EB,\max }}\text{ }\forall t
\end{equation} }where ${{H}_{EB,\max }}$ is the maximum heat power output of EB.
\subsubsection{P2G Operating Constraints}
The inputs to P2G obey the following constraints:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq42}
P_{P 2 G, \text { min }} \leq P_{P 2 G, t} \leq P_{P 2 G, \text { max }} \forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq43}
\Delta P_{P 2 G, \min } \leq P_{P 2 G, t}-P_{P 2 G, t-1} \leq \Delta P_{P 2 G, \max } \forall t
\end{equation} }where ${{P}_{P2G,\max }}$ and ${{P}_{P2G,\min }}$ are the maximum and minimum input power of P2G, respectively; $\Delta P_{P 2 G, \max }$ and $\Delta {{P}_{P2G,\min }}$ are the upper and lower limits of the climbing ability of P2G, respectively.
\subsubsection{MT Operating Constraints}
The inputs to MT obey the following constraints:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq44}
{{Q}_{MT,\min }}\le {{Q}_{MT,t}}\le {{Q}_{MT,\max }}\text{ }\forall t
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq45}
\Delta {{Q}_{MT,\min }}\le {{Q}_{MT,t}}-{{Q}_{MT,t-1}}\le \Delta {{Q}_{MT,\max }}\text{ }\forall t
\end{equation} }where ${{Q}_{MT,\max }}$ and ${{Q}_{MT,\min }}$ are the maximum and minimum gas consumption volume of MT, respectively; $\Delta {{Q}_{MT,\max }}$ and $\Delta {{Q}_{MT,\min }}$ are the upper and lower limits of the climbing ability of MT, respectively.
%\subsubsection{Price Constraint}
%To ensure the revenue of the operator, the prices issued by the operator obey the following constraints:
%{\setlength\abovedisplayskip{2pt}
%\setlength\belowdisplayskip{2pt}
%\begin{equation}\label{Eq46}
%{{\lambda }_{\min }}\le {{\lambda }_{t}}\le {{\lambda %}_{\max }}\text{ }\forall t
%\end{equation} }where ${{\lambda }_{\max }}$ and ${{\lambda }_{\min }}$ are the peak and valley prices of the TOU, respectively; ${{\lambda }_{t}}$ is the price issued in period $t$.
\subsubsection{External Energy Transmission Power Constraints}
The energy trading limit between the integrated energy system and the power grid and gas grid can be expressed as:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{add1}
\left\{\begin{array}{l}
P_{\text { min }}^{\text {grid }} \leq P_{t}^{\text {grid }} \leq P_{ \text { max }}^{\text {grid }} \\
Q_{\text { min }}^{\text {grid }} \leq Q_{t}^{\text {grid }} \leq Q_{\text { max }}^{\text {grid }}
\end{array}\right.
\end{equation} }where $P_{\text{max }}^{\text{grid }}$ and $P_{\text{min }}^{\text{grid }}$ are the upper and lower limits of the interactive electric power between IES and the power grid, respectively; $Q_{\text{ max}}^{\text{grid }}$ and $Q_{\text{min }}^{\text{grid }}$ are the upper and lower limits of IES and gas grid interactive natural gas, respectively.
\section{Model solving}
This section presents a SA-MDP model and a DRL algorithm for resilient scheduling decisions. The SA-MDP model aims to maximize operator profits while balancing supply and demand, considering load and RES uncertainties, and the impact of HLR attack.%In this section, we formulate a SAMDP model and propose a DRL algorithm to make resilient scheduling decisions. The objective of our SAMDP model is to maximize operator profits and ensure a balance between supply and demand, while considering multi-uncertainties from power sources and loads, as well as the impact of HLR attacks on scheduling decisions.from RES and loads
\subsection{SA-MDP Model}
To maximize intra-day operator profits and consider cyber-attacks, the IES scheduling problem is formulated as a SA-MDP. It incorporates constraints and uncertainties from different sources like MT, P2G, WT, and fluctuating loads. %The objective of IES scheduling is to determine the output power of the equipment at each period, maximizing the intra-day operator's profit. The optimization problem can be viewed as a sequential decision problem. Considering the potential threat of cyber-attacks on energy systems, we formulate the IES scheduling problem under cyber-attack as an SA-MDP. The model takes into account the operational constraints of MT, P2G, etc., and incorporates the stochastic generation of WT and fluctuating loads.
%A regular Markov decision process (MDP) is usually defined as a tuple $(S,A,P,r,\gamma )$, where $S$ is the state space; $A$ is the action space; $r:S\times A\to r$ is the reward function; $P:S\times A\times S\to [0,1]$ is the state transition probability function; $\gamma $ is the discount factor used to balance the importance of current and future rewards.
Unlike a regular Markov decision process (MDP), the SA-MDP introduces an adversary $v(s)$ into the process. The adversary's aim is to disturb the agent's observation, causing the agent to take actions based on $\pi(a|v(s))$, while the environment still transitions from state $s$ rather than $v(s)$ to the next state. As a result, the agent's action from $\pi(a|v(s))$ may be sub-optimal, leading to a reduction in the agent's earned reward. In the context of IES scheduling, the adversary can be reflected as cyber-attacks.
\iffalse
The SA-MDP is shown in Fig. \ref{SA-MDP}.
\begin{figure}[ht]
\centering
\includegraphics[width=3.6in,height=1.6in]{fig/SA-MDP.jpg}
\caption{State-Adversarial Markov decision process.}
\label{SA-MDP}
\end{figure}
\fi
\subsubsection{State}
%During the IES's operation, the environmental state consists of the initial load demand for electricity, heat and gas, the wind turbine generation, the SOC of the energy storage devices, the price of electricity and gas purchased and sold to the power grid and the gas grid.
The state is the environmental state space, the agent's perceived state at a particular time is denoted by ${{s}_{t}}\in S$. Here, the state is described as
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq47}
s_t=\left[\begin{array}{l}
S O C_t^{E S D}, S O C_t^{H S D}, \lambda_{\text {grid }, t}^P, \lambda_{\text {grid }, t}^Q, \\
P_{W T, t}, P_{\text {basic }, t}, Q_{\text {basic }, t}, H_{\text {basic }, t}, t
\end{array}\right]
\end{equation} }
\subsubsection{Action}
%including the power of P2G unit, the power of EB unit, the gas consumption volume of MT unit, and the issued real-time electricity, gas and heat prices. Therefore
The action denotes the scheduling decision taken by IES. Here, the action at time $t$ can be described as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq48}
{{a}_{t}}=\left[ {{P}_{P2G,t}},{{P}_{EB,t}},{{Q}_{MT,t}},{\lambda_{rt, t}^H},{\lambda_{rt, t}^P},{\lambda_{rt, t}^Q} \right]
\end{equation} }
\subsubsection{Adversary}
The adversary represents realistic cyber-attack. We need to constrain the power of the adversary, i.e. ${{v}_{s}}\in B(s)$. %where $B(s)$ is a small set of states and $s\in B(s)$.
In this study, the adversary is set to HRL attack, as described in (\ref{Eq25}).
%\textbf{Theorem1.}
\newtheorem{thm}{Theorem}
\begin{thm}
For a non-adversarial MDP with a policy $\pi$, and an optimal adversary $v$ in SA-MDP, the following holds for all states $s\in S$:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq49}
\max _{s \in \mathcal{S}}\left\{V^\pi(s)-\tilde{V}_v^\pi(s)\right\} \leq \alpha \max _{s \in \mathcal{S}} \max _{\hat{s} \in B(s)} \mathrm{D}_{T V}(\pi(\cdot \mid s), \pi(\cdot \mid \hat{s}))
\end{equation} }where ${{\text{D}}_{TV}}(\pi (\cdot \mid s),\pi (\cdot \mid \hat{s}))$ denotes the total variation distance between $\pi (\cdot \mid s)$and $\pi (\cdot \mid \hat{s})$ \cite{zhang2020robust}.
\end{thm}
%\textbf{Proof.} See Appendix in\cite{zhang2020robust}.the total variation distance during training
Theorem 1 indicates that if ${{\text{D}}_{TV}}(\pi (\cdot \mid s),\pi (\cdot \mid \hat{s}))$ for any $\hat{s}$ within $B(s)$ is not too large, the performance difference between $\tilde{V}_{\nu }^{\pi }(s)$ in the SA-MDP environment and ${{V}^{\pi }}(s)$ in the MDP environment is bounded. So, we add a state adversarial regularizer to the policy network to generate a resilient scheduling strategy against cyber-attacks.
\subsubsection{Reward Function}
The reward function for maximizing the net profit of IES operator is augmented with a penalty term to maintain energy balance and accelerate the convergence, and is given by: %To maximize the net profit of IES operator, the operator's profit is taken as the reward function to guide the agent to explore. Simultaneously, to accelerate the algorithm convergence and maintain energy balance, the penalty term is added to the reward function. Therefore, the reward function at time $t$ is %described as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq50}
{{r}_{t}}({{s}_{t}},{{a}_{t}})=F({{s}_{t}},{{a}_{t}})-\left( C_{1,t}^{Penalty}({{s}_{t}},{{a}_{t}})+C_{2,t}^{Penalty}({{s}_{t}},{{a}_{t}}) \right)
\end{equation} }where $F$ is the net profit of the IES operator; $C_{1,t}^{Penalty}$ denotes the action change rate overrun penalty:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq51}
C_{1, t}^{\text {Penaly }}=\sum_{a \in A}\left[\begin{array}{l}
\delta_{1, i} \max \left(a_{i, t}-a_{i, t-1}-a_{i, \mathrm{U} \max }, 0\right) \\
+\delta_{2, i} \max \left(a_{i, t-1}-a_{i, t}-a_{i, \mathrm{D} \max }, 0\right)
\end{array}\right]
\end{equation} }where ${{\delta }_{1,i}}$ and ${{\delta }_{2,i}}$ are the penalty coefficients of action change rate exceeding the upper and lower change limits. ${{a}_{i,\text{U}\max }}$ and ${{a}_{i,\text{D}\max }}$ are the upper and lower change limits of the action change rate. Here, $C_{2,t}^{Penalty}$ ensures supply-demand balance by preventing actions that could cause imbalance. %and satisfies the energy balance in Eq. (\ref{Eq33}). respectively
%To ensure the supply-demand balance, it is essential to prevent the execution of certain actions that may cause imbalance. Therefore, $C_{2,t}^{Penalty}$ denotes the energy system penalty and is mainly used to satisfy Eq. (\ref{Eq33}), i.e.,
%, as described in Eq. (\ref{Eq29})
%, which is mainly used to satisfy Eqs. (\ref{Eq43}) and (\ref{Eq45})
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq52}
C_{2,t}^{Penalty}=\omega_t^H L R_t^H+\left(1-\omega_t^H\right) E L_t^H
\end{equation} }where $LR_{t}^{H}$ is the economic penalty on the demand-side of the heating system; $EL_{t}^{H}$ is the economic penalty on the supply-side of the heating system; $\omega_t^H$ is a binary variable, if $\omega_t^H=1$ means that the demand-side economic penalty is at this time, otherwise the supply-side economic penalty is at this time.
%If the sum of heat supply and thermal storage is not sufficient to meet the heat demand of the building, a financial penalty is imposed on the demand side of the heating system for the portion of the heat demand that cannot be met.
The judgment criteria and penalty value of the demand side economic penalty are described as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq53}
({{H}_{EB,t}}+{{H}_{MT,t}}+{{C}_{HSD,t-1}})<{{H}_{load,t}}\text{ }
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq54}
LR_{t}^{H}={{\beta }_{LR}}\left( {{H}_{load,t}}-\left( {{H}_{EB,t}}+{{H}_{MT,t}}+{{C}_{HSD,t-1}} \right) \right)\text{ }\forall t
\end{equation} }where ${{\beta }_{LR}}$ is the demand side economic penalty coefficient.
%On the other hand, if the sum of heat supply and heat storage exceeds the heat demand of the building, the energy system needs to reduce some energy to maintain a balance between supply and demand.
The judgment criteria and penalty value of the supply side economic penalty are described as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq55}
H_{E B, t}+H_{M T, t}>\left(H_{\text {load }, t}+C_{\max }-C_{H S D, t-1}\right)
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq56}
\begin{aligned}
E L_t^H & =\beta_{EL}\left(H_{E B, t}+H_{M T, t}-\left(H_{\text {load }, t}+C_{\max }\right.\right. \\
& \left.\left.-C_{H S D, t-1}\right)\right) \forall t
\end{aligned}
\end{equation} }where ${{\beta }_{EL}}$ is the supply side economic penalty coefficient.
\vspace{-3mm}
\subsection{SA-SAC based Solution Process}
\iffalse
\begin{figure}[ht]
\centering
\includegraphics[width=3.6in,height=2.4in]{fig/SA-SAC-graph.jpg}
\caption{The IES scheduling framework based on the proposed SA-SAC}
\label{SA-SAC-graph}
\end{figure}
\fi
The proposed algorithm adopts a maximum entropy framework and trains the decision-making network using a certified defense training method to make its neural network robust to cyber-attacks perturbations. The objective function is %of the model in the maximum entropy framework is
%can be expressed as follows:
% in each state
%. Unlike other reinforcement learning algorithms that mainly aim to learn the policy to maximize the expected value of the accumulated reward, the maximum entropy learning objective also seeks to maximize the entropy of the optimal policy by adding an entropy term to the objective function
%This helps to enhance the exploration ability of the RL model.
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq57}
\pi^*=\arg \max _\pi \sum_t \mathbb{E}_{\left(s_t, \mathbf{a}_t\right) \sim \rho_\pi}\left[r\left(s_t, a_t\right)+\alpha \mathcal{H}\left(\pi\left(\cdot \mid s_t\right)\right)\right]
\end{equation} }where $\mathcal{H}$ is the entropy term of the action adopted in state ${{s}_{t}}$; $\alpha $ is the temperature parameter. The objective function contains two terms: the sum of cumulative rewards and the entropy term. This design motivates the agent to explore more action spaces while pursuing the maximum cumulative rewards. During the policy evaluation step of soft policy iteration, the modified Bellman equation is applied:
%, which determines the importance of the entropy term relative to the reward, controls the randomness of the strategy, and represents the strength of the exploration ability
%the first is the sum of cumulative rewards and the second is the entropy term.
%, which helps to avoid premature convergence to local optimal solutions
\iffalse
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq58}
Q\left(s_t, a_t\right)=r\left(s_t, a_t\right)+\gamma \mathbb{E}_{\mathrm{s}_{t+1} \sim p_\pi}\left[V\left(s_{t+1}\right)\right]
\end{equation} }{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq59}
V\left(s_t\right)=\mathbb{E}_{\mathbf{a}_t \sim \pi}\left[Q\left(s_t, a_t\right)-\alpha \log \pi\left(a_t \mid s_t\right)\right]
\end{equation} }where $\gamma $ is the discount factor; $V({{s}_{t}})$is the state value function.
\fi
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq58}
\begin{aligned}
Q\left(s_t, a_t\right) & =r\left(s_t, a_t\right)+\gamma \mathbb{E}_{\mathbf{s}_{t+1} \sim p_\pi, \mathbf{a}_t \sim \pi}\left[Q\left(s_t, a_t\right)\right. \\
& \left.-\alpha \log \pi\left(a_t \mid s_t\right)\right]
\end{aligned}
\end{equation} }
The policy distribution is converted into a probability distribution in the form of energy function and is proportional to the exponential form of the Q function, at which point the action value function is called soft Q value.
\iffalse
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq60}
\pi\left(a_t \mid s_t\right) \propto \exp \left[Q\left(s_t, a_t\right)\right]
\end{equation} }
\fi
%In the policy improvement step, the policy is updated to the exponential form of the soft Q-function to guarantee that an improved policy is resulted in terms of soft value. a form of
For each state, the policy is updated using KL divergence:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq61}
\pi_{\text {new }}=\arg \min _{\pi^{\prime} \in \Pi} D_{K L}\left(\pi^{\prime}\left(\cdot \mid s_t\right) \| \frac{\exp \left(\frac{1}{\alpha} Q^{\pi_{\text {old }}}\left(s_t, \cdot\right)\right)}{Z^{\pi_{\text {old }}}\left(s_t\right)}\right)
\end{equation} }where $\Pi $ denotes the set of policy distribution; ${{Z}^{{{\pi }_{old}}}}({{s}_{t}})$ is the partition function that is used to normalize the distribution.
Consider a parameterized soft Q function $Q_\theta\left(s_t, a_t\right)$ and a tractable policy $\pi_\phi\left(a_t \mid s_t\right)$ with $\theta$ and $\phi$ as the parameters of their networks, respectively. The soft Q-function parameters are obtained by minimizing the soft Bellman residual:%to minimize the soft Bellman residual, i.e.,
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq62}
\begin{aligned}
J_Q(\theta) & =\mathbb{E}_{\left(s_t, a_t, s_{t+1}\right) \sim D, a_{t+1} \sim \pi_\phi}\left[1 / 2\left(Q_\theta\left(s_t, a_t\right)-r\left(s_t, a_t\right)\right.\right. \\
& \left.\left.-\gamma Q_{\bar{\theta}}\left(s_{t+1}, a_{t+1}\right)+\gamma \alpha \log \pi_\phi\left(a_{t+1} \mid s_{t+1}\right)\right)^2\right]
\end{aligned}
\end{equation} }%where the soft Q-function parameters are optimized by stochastic gradient as follows:
%{\setlength\abovedisplayskip{2pt}
%\setlength\belowdisplayskip{2pt}
%\begin{equation}\label{Eq63}
%\begin{aligned}
%\hat{\nabla}_\theta J_Q(\theta) & =\nabla_\theta %Q_\theta\left(a_t, s_t\right)\left(Q_\theta\left(s_t, %a_t\right)-r\left(s_t, a_t\right)\right. \\
%& \left.-\gamma Q_{\bar{\theta}}\left(s_{t+1}, %a_{t+1}\right)+\gamma \alpha \log \pi_\phi\left(a_{t+1} %\mid s_{t+1}\right)\right)
%\end{aligned}
%\end{equation} }
where $\overline{\theta }$ is the parameters of the target soft Q-function, and soft update is used for the parameters update of the target soft Q-function in the training, i.e., $\overline{\theta }\leftarrow \tau \theta +(1-\tau )\overline{\theta }$.
%This approach helps to minimize deviation in policy improvement and improve the stability of the training process.
Here, we use two Q-functions and select the one with the lower value during each iteration. For the update of the policy network, the policy parameter can be learned by %minimizing the KL-divergence in Eq. (\ref{Eq61}):
%, multiplying the parameter $\alpha$ and ignoring the constant log-partition function as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq64}
J_\pi(\phi)=\mathbb{E}_{s_t \sim D, a_t \sim \pi_\phi}\left[\alpha \log \left(\pi_\phi\left(a_t \mid s_t\right))-Q_\theta\left(s_t, a_t\right)\right]\right.
\end{equation} }
In addition, we introduce a reparameterization trick ${{f}_{_{\phi }}}({{\varepsilon }_{t}};{{s}_{t}})$, where $\varepsilon $ is the action noise sampled from the standard normal distribution. Based on Theorem 1, to obtain the resilient scheduling strategy under attack, we need to find ${{\max }_{\hat{s}\in B(s)}}{{\text{D}}_{TV}}(\pi (\cdot \mid s),\pi (\cdot \mid \hat{s}))$. To establish an upper bound for the total variation distance of the stochastic policy distribution, we employ the KL divergence as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq65}
\mathrm{D}_{\mathrm{TV}}(\pi(a \mid s), \pi(a \mid \hat{s})) \leq \sqrt{\frac{1}{2} \mathrm{D}_{\mathrm{KL}}(\pi(a \mid s) \| \pi(a \mid \hat{s}))}
\end{equation} }
%where $d=\|\pi (s)-\pi (\hat{s}){{\|}_{2}}$. Therefore, we can use ${{\max }_{\hat{s}\in B(s)}}\|\pi (s)-\pi (\hat{s}){{\|}_{2}}$ as the replacement. In training, our goal is to keep ${{\max }_{\hat{s}\in B(s)}}\|\pi (s)-\pi (\hat{s}){{\|}_{2}}$ as small as possible.
%for the SA-SAC
By ignoring constant terms, regularizing the KL divergence over all $s$ from sampled trajectories and all $\hat{s} \in B(s)$ results in the state-adversarial regularizer:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{KL}
\begin{aligned}
\mathcal{R}_{S A-S A C}\left(\phi_\mu\right) & =\sum_s \max _{\hat{s} \in B(s)}\left(\pi_\phi^\mu(\hat{s})-\pi_\phi^\mu(s)\right)^{\top} \Sigma^{-1}\left(\pi_\phi^\mu(\hat{s})\right. \\
\left.-\pi_\phi^\mu(s)\right) & =\sum_s \max _{\hat{s} \in B(s)}\left\|\pi_\phi^\mu(\hat{s})-\pi_\phi^\mu(s)\right\|_2^2 \\
& =\sum_s \max _{\hat{s} \in B(s)} \mathcal{R}_s\left(\hat{s}, \phi_\mu\right)
\end{aligned}
\end{equation} }
%For $B(s)=\left\{ \hat{s}:{{s}_{l}}\le \hat{s}\le {{s}_{u}},{{s}_{l}}\le s\le {{s}_{u}} \right\}$, we use perturbation analysis tool CROWN-IBP that combines a tight linear relaxation based bound CROWN with the efficient interval bound propagation (IBP) training method to give linear upper and lower bounds of ${{\pi }_{\phi }}\left( {{{\hat{s}}}_{t}} \right)$\cite{zhang2018efficient}, i.e.,
We employ CROWN-IBP, a defensive training method that merges fast interval bound propagation (IBP) with CROWN's tight convex relaxation technique, to ensure quick training and avoid network over-regularization. CROWN-IBP is used to derive linear upper and lower bounds of $\pi_\phi^\mu(\hat{s})$, i.e.,
%We utilize a defensive training method, CROWN-IBP, which combines the fast interval bound propagation (IBP) method with the tight convex relaxation technique of CROWN. This approach ensures fast training speed while preventing over-regularization of the network. Using CROWN-IBP to obtain linear upper and lower bounds of $\pi_\phi^\mu(\hat{s})$, i.e.,
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq66}
l_{\phi_\mu}(s) \leq \pi_\phi^\mu(\hat{s}) \leq u_{\phi_\mu}(s) \quad
\end{equation} }where the bounds $u_{\phi_\mu}$ and $l_{\phi_\mu}$ are calculated as follows:%are functions of $\phi_\mu$.
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq_defend}
\left\{\begin{array}{l}
u_{\phi_\mu}(s)=(1-\beta) \overline{m}_{\mathrm{IBP}}(s, \epsilon)+\beta \overline{m}_{\mathrm{CROWN}}(s, \epsilon) \\
l_{\phi_\mu}(s)=(1-\beta) \underline{m}_{\mathrm{IBP}}(s, \epsilon)+\beta \underline{m}_{\mathrm{CROWN}}(s, \epsilon)
\end{array}\right.
\end{equation} }where $\epsilon$ refers to perturbations caused by cyber-attacks; ${{m}_{\text{IBP}}}(s,\epsilon )$ and ${{m}_{\text{CROWN}}}(s,\epsilon )$ are the tight linear boundaries generated by IBP and CROWN, respectively; the hyper-parameter $\beta$ in CROWN-IBP is scheduled as suggested in \cite{zhang2019towards}.
Thus, we can calculate the upper bound $\overline{\mathcal{R}}_s\left(\phi_\mu\right)$ of $\mathcal{R}_s\left(\hat{s}, \phi_\mu\right)$ and the upper limit of the distance as follows:
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq67}
L_{S A-S A C}(\phi)=\mathbb{E}_{i \sim N}\left\|u_{\phi_\mu}\left(s_i\right)-l_{\phi_\mu}\left(s_i\right)\right\|_2^2=\mathbb{E}_{i \sim N} \overline{\mathcal{R}}_i\left(\phi_\mu\right)
\end{equation} }
Then we add the upper limit $L_{S A-S A C}(\phi)$ of the distance to the policy gradient loss to obtain the approximate gradient of $J_{\pi }^{'}(\phi )$, i.e.,
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq68}
\begin{gathered}
\hat{\nabla}_\phi J_\pi^{\prime}(\phi)=\nabla_\phi \alpha \log \left(\pi_\phi\left(a_t \mid s_t\right)\right)+\left(\nabla_{a_t} \alpha \log \left(\pi_\phi\left(a_t \mid s_t\right)\right)\right. \\
\left.-\nabla_{a_t} Q\left(s_{t+1}, a_{t+1}\right)\right) \nabla_\phi f_\phi\left(\varepsilon_t ; s_t\right)+\kappa \nabla_\phi L_{S A-S A C}(\phi)
\end{gathered}
\end{equation} }where ${{a}_{t}}$ is evaluated at ${{f}_{_{\phi }}}({{\varepsilon }_{t}};{{s}_{t}})$, i.e., ${{a}_{t}}={{f}_{_{\phi }}}({{\varepsilon }_{t}};{{s}_{t}})=\pi _{\phi }^{\mu }({{s}_{t}})+{{\varepsilon }_{t}}\cdot \pi _{\phi }^{\sigma }({{s}_{t}})$; $\kappa $ is the regularization term coefficient.
In addition, for the temperature parameter $\alpha$, we introduce an adaptive update in the form of minimizing $J(\alpha )$, i.e.,
{\setlength\abovedisplayskip{2pt}
\setlength\belowdisplayskip{2pt}
\begin{equation}\label{Eq69}
J(\alpha)=\mathbb{E}_{a_t, \pi_t}\left[-\alpha \log \left(\pi_\phi\left(a_t \mid s_t\right)-\alpha H_0\right]\right.
\end{equation} }where ${{H}_{0}}$ is set to $-\dim(a)$, the dimension of the action.
%Based on the above analysis, the learning process of the final SA-SAC algorithm is summarized in Algorithm \ref{alg:code}. \subsection{Solution Framework} The scheduling framework based on the SA-SAC is shown in Fig. \ref{SA-SAC-graph}.
\iffalse
\begin{figure}[ht]
\centering
\includegraphics[width=3.6in,height=2.5in]{fig/SA-SAC.jpg}
\caption{The IES scheduling framework based on the proposed SA-SAC.}
\label{SA-SAC-graph}
\end{figure}
\fi
To encapsulate the preceding steps, the detailed solution process based on the SA-SAC is outlined in Algorithm \ref{alg:code}.
\begin{algorithm}[!t]\label{code} % ！htbp
\small
\renewcommand{\algorithmicrequire}{\textbf{Initialize:}}
\renewcommand{\algorithmicensure}{\textbf{return}}
\caption{State-Adversarial Soft Actor-Critic (SA-SAC)}
\label{alg:code}
\begin{algorithmic}[1] %[] 里面的数字为行号 [1]就是一行一个数字
\REQUIRE the critic network and actor network with parameters ${{\theta }_{1}}$, ${{\theta }_{2}}$ and $\phi $
\REQUIRE the target network with weights $\overline{{{\theta }_{1}}}\leftarrow {{\theta }_{1}}$ and $\overline{{{\theta }_{2}}}\leftarrow {{\theta }_{2}}$
\REQUIRE the replay pool capacity D
\FOR{each iteration}
\FOR{each environment step}
\STATE Sample action from the policy ${{a}_{t}}\sim{{\pi }_{\phi }}({{a}_{t}}|{{s}_{t}})$
\STATE Sample state transition from the environment $s_{t+1} \sim p\left(s_{t+1} \mid s_t, a_t\right)$
\STATE Store state transition and reward in the replay pool$D\leftarrow D\bigcup \left\{ ({{s}_{t}},{{a}_{t}},r({{s}_{t}},{{a}_{t}}),{{s}_{t+1}}) \right\}$
\ENDFOR
\FOR{each gradient step}
\STATE Update the Q-function ${{J}_{Q}}({{\theta }_{i}})$ parameters, ${{\theta }_{i}}\leftarrow {{\theta }_{i}}-{{\lambda }_{Q}}{{\hat{\nabla }}_{{{\theta }_{i}}}}{{J}_{Q}}({{\theta }_{i}})$for $i\in \left\{ 1,2 \right\}$
\STATE Obtain upper and lower bounds on ${{\pi }_{\phi }}\left( {{s}_{n}} \right)$ using CROWN-IBP: $l_{\phi_\mu}(s) \leq \pi_\phi^\mu(\hat{s}) \leq u_{\phi_\mu}(s)$ for $s\in B({{s}_{n}})$, $n\in \left[ N \right]$
\STATE Upper bound on distance: $L_{S A-S A C}(\phi)=\mathbb{E}_{i \sim N} \overline{\mathcal{R}}_i\left(\phi_\mu\right)$
\STATE Update policy $J_{\pi }^{'}(\phi )={{J}_{\pi }}(\phi )+\kappa {{L}_{SA}}(\phi )$ weight, $\phi \leftarrow \phi -{{\lambda }_{\pi }}{{\hat{\nabla }}_{\phi }}J_{\pi }^{'}(\phi )$
\STATE Update temperature parameter, $\alpha \leftarrow \alpha -\lambda {{\hat{\nabla }}_{\alpha }}J(\alpha )$
\STATE Soft update target network weights, $\overline{{{\theta }_{i}}}\leftarrow \tau {{\theta }_{i}}+(1-\tau )\overline{{{\theta }_{i}}}$for $i\in \left\{ 1,2 \right\}$
\ENDFOR
\ENDFOR
\STATE Output policy network as the resilient scheduling model of integrated energy system
\end{algorithmic}
\end{algorithm}
\iffalse
\subsection{Solution Process}
The detailed solution process can be outlined as follows:
Step 1: Formulate the models for each device involved in the IES, as well as the overall system model. Design an IDR program taking into account flexible loads in electricity, gas, and heat systems.
Step 2: Build the IES scheduling model and convert into a DRL framework.
Step 3: Select appropriate hyperparameters for SA-SAC to start training and let the agent and the environment continuously interact to learn and collect data.
Step 4: Perform decision-making neural network certified defense training using CROWN-IBP and update the actor network, critic network, target network and temperature parameter.
Step 5: Check if the algorithm has reached the maximum reward, and if so, save the decision-making network. Otherwise, move on to step 3 and continue training.
\fi
\section{Case Study}
To validate the efficacy of our approach, we perform a case study on the IES illustrated in Fig. \ref{CIES}. Our proposed method is developed using Python 3.8 and implemented with PyTorch 1.6.0. The simulations are conducted on a computer equipped with an AMD R7-5800H CPU and 16 GB of RAM.
The parameters of the SA-SAC are given as follows: The actor and critic use ReLU as the activation function of all hidden layers, and the Adam optimizer is used to update the network weights. The learning rates of actor network and critic network are 0.0005 and 0.002, respectively. The discount factor is set to 0.95, the mini-batch size is 256, the target network is updated using a soft update rate of 0.005, and the training epoch is set as 1000. In addition, we train the actor network without actor regularizer for the first 500 episodes, then gradually increase the perturbation $\epsilon$ from the cyber-attack on the actor neural network in 300 episodes, eventually maintaining the target $\epsilon$ during 200 episodes of training.
\subsection{Settings in Test Case}
%The key components of the IES include WT, an EB, a P2G device, an MT unit, and energy storage devices.
The length of a scheduling cycle is set to 24 hours, with a one-hour interval for each time step. The main parameters of the devices are shown in Table \ref{tab2}.
\vspace{-4mm}
\begin{table}[h]
\centering
\caption{Main parameters of IES}\label{tab2}
\begin{tabular}{cccc}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
Parameter & Value & Parameter & Value \\
\midrule
${{H}_{EB,\max }}(\operatorname{kW})$ & 300 & ${\varsigma _{t}^{TSE}}$ & $[-2,2]$ \\
${{\eta }^{EB}}$ & 0.99 & $\eta $ & $\operatorname{N}(10,6)$ \\
$Q_{\max }^{grid}({{\operatorname{m}}^{3}})$ & 80 & ${{\sigma }_{t}}(\operatorname{kW})$ & $\operatorname{N}(0.2,0.2)$ \\
${{Q}_{MT,\min }}({{\operatorname{m}}^{3}})$ & 10 & $P_{\max }^{grid}\text{(kW)}$ & 1000 \\
${{Q}_{MT,\max }}({{\operatorname{m}}^{3}})$ & 40 & ${{P}_{P2G,\min }}(\operatorname{kW})$ & 100 \\
$\Delta {{Q}_{MT,\min }}({{\operatorname{m}}^{3}})$ & -10 & ${{P}_{P2G,\max }}(\operatorname{kW})$ & 500 \\
$\Delta {{Q}_{MT,\max }}({{\operatorname{m}}^{3}})$ & 10 & $\Delta {{P}_{P2G,\min }}(\operatorname{kW})$ & -200 \\
$\eta _{e}^{MT}$ & 0.4 & $\Delta {{P}_{P2G,\max }}(\operatorname{kW})$ & 200 \\
$\eta _{h}^{MT}$ & 0.5 & ${{\eta }_{P2G}}$ & 0.6 \\
$\eta _{loss}^{MT}$ & 0.1 & $C_{\max }^{HSD}\text{(kWh)}$ & 180 \\
$C_{\max }^{ESD}\text{(kWh)}$ & 200 & $P_{ch,\max }^{HSD},P_{dc,\max }^{HSD}\text{(kW)}$ & 90 \\
$P_{ch,\max }^{ESD},P_{dc,\max }^{ESD}\text{(kW)}$ & 100 & $\eta _{ch}^{HSD},\eta _{dc}^{HSD}$ & 1 \\
$\eta _{ch}^{ESD},\eta _{dc}^{ESD}$ & 0.9 & ${\varsigma _{t}^{TSQ}}$ & $[-2,2]$ \\
$K\text{(W}\cdot {{\text{m}}^{\text{-2}}}\text{)}$ & 0.5 & ${\varsigma _{t}^{HC}}$ & $[0,1]$ \\
${{c}_{air}}\text{(kJ}\cdot \text{k}{{\text{g}}^{\text{-1}}}\cdot {}^\circ {{C}^{\text{-1}}}\text{)}$ & 1.007 & $F\text{(}{{\text{m}}^{\text{2}}}\text{)}$ & 2400 \\
${{\rho }_{air}}\text{(kg}\cdot {{\text{m}}^{\text{-3}}}\text{)}$ & 1.2 & $V\text{(}{{\text{m}}^{\text{3}}}\text{)}$ & 36000 \\
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\end{tabular}%
\label{tab:addlabel}%
\end{table}%
\iffalse
\subsection{Analysis of Algorithm Training}
%\subsubsection{Analysis of Algorithm Training}
To use the proposed approach for IES resilient scheduling and obtain a decision-making network that is robust for cyber-attack, training is necessary prerequisite. The proposed method's convergence performance is shown in Figs. \ref{SA-SAC_SAC}. The penalty values during the training process is shown in Figs. \ref{Penalty}. As can be observed, the reinforcement learning agent engages in continuous interaction with the environment to achieve convergence. Fig. \ref{Penalty} shows a significant decrease in the number of agent actions that exceed predefined limits, accompanied by a reduction in action amplitude and rate of change within limits during training. Moreover, the reward fluctuates due to daily variations within each episode.
%\vspace{-7.5mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.4cm,width=4.3cm]{fig/SA-SAC_SAC_improve.jpeg}
\caption{Reward curves during the training process.}
\label{SA-SAC_SAC}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[height=3.4cm,width=4.3cm]{fig/penalty.jpeg}
\caption{Penalty curve during training.}
\label{Penalty}
\end{minipage}
\end{figure}
%\vspace{-4mm}
To demonstrate the superiority of our proposed method over the original SAC algorithm, we also compare the policy quality of both methods in Figs. \ref{SA-SAC_SAC}. It can be observed that our method achieves a higher reward value after convergence, highlighting its superior performance. This is primarily attributed to the addition of regularizer to the decision network of the agents, which enhances the quality of the scheduling policy learning.
%The SAC hyperparameters are chosen similarly to the SA-SAC.
%To verify the convergence performance of the proposed method, the reward curves and penalty values during the training process are shown in Fig. \ref{SA-SAC_SAC} and Fig. \ref{Penalty}. As the agent gains experience through continuous interaction with the environment, the reward trend gradually increases while the action penalty decreases. Eventually, the agent learns the optimal scheduling strategy to maximize the operator's profit. The reward fluctuates during training since the training data varies within each episode from day to day. To demonstrate the superiority of the proposed method, the SAC algorithm is compared with the SA-SAC algorithm. The hyperparameters of the SAC are selected in the same way as the SA-SAC.
%It can be observed that the reward received after executing the scheduling decisions are small because the agent is not familiar with the environment at the beginning. As the training process continues, the agent continuously interacts with the environment and accumulates experience, the overall reward trend gradually increases and the action penalty gradually decreases, and eventually the reward curve tends to converge, while the action amplitude and rate of change are constrained within the limits. This indicates that the agent has learned the optimal scheduling strategy to maximize the profit of the IES operator. In addition, to show the superiority of the proposed method, the SAC algorithm is selected for comparison in this study. The comparison of the reward values of the SAC and SA-SAC algorithms is shown in Fig. \ref{SA-SAC_SAC}. Fig. \ref{SA-SAC_SAC} shows a comparison of their reward values.
%Fig. \ref{SA-SAC_SAC} shows that the SA-SAC can converge to a higher reward value compared to the SAC. To verify the performance of the proposed algorithm in the presence of cyber-attack, we tested the SAC and SA-SAC algorithms in both regular MDP and SAMDP environments. Fig. \ref{Penalty} shows that with the progression of the training process, there is a significant reduction in the number of the agent's actions exceed the predefined limits. This decrease is accompanied by a rapid decrease in the amplitude and rate of change of the agent's actions, which become increasingly constrained within the set limits. The comparison of the robustness of SA-SAC and SAC to environmental changes is shown in Fig. \ref{Algorithm_attack}, where we consider the first 50 episodes as the MDP environment, i.e., the test environment that is not under attack, and the last 50 episodes as the SA-MDP environment, i.e., the test environment that is under HLR attack. As shown in Fig. \ref{Algorithm_attack}, the SA-SAC maintains a higher reward value after the attack compared to the SAC.
%\vspace{-4mm}
%\begin{figure}[htbp]
% \centering
% \includegraphics[width=2.3in,height=1.5in]{fig/SA-SAC_SAC_improve.jpeg}
% \caption{Reward curves during the training process}
% \label{SA-SAC_SAC}
%\end{figure}
\fi
\vspace{-4mm}
\subsection{Analysis of Algorithm Robustness}
To assess the robustness of our decision network under cyber-attack scenarios, we compared the performance of SAC and our proposed algorithms in both regular MDP and SA-MDP environments. The SA-MDP environment's new test data set was created by simulating cyber-attacks on the IES. Fig. \ref{Algorithm_attack} illustrates this comparison, with the first 50 episodes in a non-attacked MDP environment and the last 50 in an HLR-attacked SA-MDP environment.%To verify the robustness of the decision network in our proposed method under cyber-attack scenarios. The performances of the SAC and the proposed algorithms are tested in regular MDP and SA-MDP environments to evaluate their resilience to cyber-attacks. The new test data set in the SA-MDP environment is generated by applying cyber-attacks to the IES. The robustness comparison of the proposed algorithm and SAC to cyber-attacks is illustrated in Fig. \ref{Algorithm_attack}. Here, the first 50 episodes represent the non-attacked MDP test environment, while the last 50 episodes represent the HLR-attacked SA-MDP test environment.
\vspace{-3mm}
\begin{figure}[htbp]
\centering
\includegraphics[width=2.3in,height=1.5in]{fig/Algorithm_attack.jpeg}
\caption{Robustness comparison of the SA-SAC and SAC to cyber-attack.}
\label{Algorithm_attack}
\end{figure}
The proposed method, as illustrated in Fig. \ref{Algorithm_attack}, achieves higher rewards post-attack compared to SAC, demonstrating its resilience in scheduling under cyber-attacks. This success is due to the enhanced robustness of its decision-making network, reinforced by CROWN-IBP, which leads to effective scheduling decisions without losses. In contrast, the untrained SAC algorithm shows marked vulnerability to malicious inputs, significantly reducing reward value.
%As shown in Fig. \ref{Algorithm_attack}, the proposed method maintains a higher reward value after the attack than SAC. The results show that the decision-making network trained by our proposed method exhibits high resilience when used for scheduling, producing reasonable scheduling decisions that prevent the IES operator from incurring losses in the presence of cyber-attack. This can be attributed to the strong robustness of the decision-making neural network after CROWN-IBP tightens the neuron boundaries. On the contrary, the untrained SAC algorithm is extremely fragile when subjected to malicious inputs, resulting in a significant drop in the reward value.
%The comparison of economy for different approaches is shown in Table \ref{tab3}.
\iffalse
%\vspace{-5mm}
\begin{table}[ht]
\centering
\caption{Comparison of economy for different approaches}\label{tab3}
\begin{tabular}{ccc}
\toprule
Environments & Algorithms & Profit/¥ \\
\midrule
\multirow{2}*{MDP} & Proposed method & 8982.55 \\
& SAC & 8746.15 \\
\midrule
\multirow{2}*{SA-MDP} & Proposed method & 8980.95 \\
& SAC & 8147.27 \\
\bottomrule
\end{tabular}%
\label{tab:addlabel}%
\end{table}%
%\vspace{-3mm}
\fi
%Table \ref{tab3} shows that the proposed method yields the highest operator profit when not attacked, with an increase of 236.40¥ compared to the SAC-based method. In the SA-MDP environment, the profit of our method is similar to the non-attacked scenario, with a difference of 833.68¥ compared to the SAC-based method. %is not significantly different from that when it is not attacked, which is 833.68¥ higher than the SAC-based method. %the scheduling process is which is 236.40¥ higher than
\vspace{-2mm}
\subsection{Analysis of Scheduling Results Using SA-SAC}
To validate our approach, we used typical daily energy demand and wind turbine output data. Fig. \ref{temperature and energy price}(a) displays the daily temperature curve and indoor-outdoor temperature difference, while Fig. \ref{temperature and energy price}(b) illustrates electricity and gas purchasing prices, with selling prices being half of the purchasing prices. Fig. \ref{WT} shows the load and WT outputs.
\vspace{-6mm}
\begin{figure}[htbp]
\centering %居中
\subfigure[]{ %第一张子图
\begin{minipage}[t]{0.49\linewidth}\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/temperature.jpeg}\end{minipage}}\subfigure[]{ %第二张子图
\begin{minipage}[t]{0.49\linewidth}
\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/ele_gas_price.jpeg}
\end{minipage}
}
\caption{Daily temperature curve and energy price.} %大图名称
\label{temperature and energy price} %图片引用标记
\end{figure}
\vspace{-1mm}
Upon completion of the training process, the policy network is employed for IES scheduling. The energy dispatching schemes for the electrical, natural gas, and heating subsystems are shown in Fig. \ref{ele_scheduling}, Fig. \ref{gas_scheduling}, and Fig. \ref{heat_scheduling}, respectively.%After training, the obtained decision-making network was saved for the economic dispatch of the IES. The scheduling scheme for each subsystem is obtained from the decision-making network as shown in the following figures: Fig. \ref{ele_scheduling} is the energy dispatching scheme for the electrical subsystem, Fig. \ref{gas_scheduling} is the energy dispatching scheme for the natural gas subsystem, Fig. \ref{heat_scheduling} is the energy dispatching scheme for the heating subsystem.
\vspace{-3mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/wt_load.jpeg}
\caption{WT outputs and daily load variations.}
\label{WT}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/ele_scheduling.jpeg}
\caption{Energy dispatching scheme for electrical subsystem.}
\label{ele_scheduling}
\end{minipage}
\end{figure}
\vspace{-6mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/gas_scheduling.jpeg}
\caption{Energy dispatching scheme for natural gas subsystem.}
\label{gas_scheduling}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/heat_scheduling.jpeg}
\caption{Energy dispatching scheme for heating subsystem.}
\label{heat_scheduling}
\end{minipage}
\end{figure}
\subsubsection{Scheduling Schemes Analysis for Each Device}
Figs. \ref{ele_scheduling}-\ref{heat_scheduling} demonstrate that the decision network's scheduling strategy meets the needs of each energy subsystem. Fig. \ref{ele_scheduling} shows priority given to WT output for renewable energy use, grid electricity use mainly during 0:00-7:00, and increasing MT usage with higher loads and prices. Fig. \ref{gas_scheduling} indicates P2G energy conversion mainly during off-peak hours and MT operation during peak hours, illustrating improved system flexibility through P2G and MT integration. Fig. \ref{heat_scheduling} reveals that MT meets daytime heat demands, while at night, EB caters to increased heat needs due to lower temperatures and electricity prices, and higher WT output.
%Figs. \ref{ele_scheduling}-\ref{heat_scheduling} shows that the scheduling strategy formulated by the decision network can meet the needs of each energy subsystems of the scheduling cycle. According to Fig. \ref{ele_scheduling}, the output of the WT has the highest priority to achieve complete consumption of renewable energy output; electricity purchased from the grid is concentrated during 0:00-7:00; as the load demand and power price increases, the MT unit gradually participates in the system operation. As can be seen from Fig. \ref{gas_scheduling}, the P2G absorbs electric energy for energy conversion during 0:00-7:00 and 23:00-24:00, and the MT is participated in system operation during 7:00-22:00. This indicates that coupling different subsystems through P2G and MT can improve the flexibility of system operation. It can be illustrated from Fig. \ref{heat_scheduling} that during 9:00-20:00 when the heat load is low, the heat demand is mainly provided by the MT; when the temperature drops at night and the electric price is in the valley and the WT output is high, the EB provides most of the heat demand.
\subsubsection{Flexible Demand Response Analysis}
Figs. \ref{ele_scheduling} and \ref{gas_scheduling} show that electric and natural gas loads align with peak and valley prices, demonstrating a "peak shaving and valley filling" pattern post-demand response. Fig. \ref{heat_scheduling} indicates a notable thermal load decrease during nighttime due to reduced temperature perception and wider thermal comfort ranges. Additionally, Fig. \ref{DR_cost} compares the amount of electricity and natural gas purchased from external networks before and after demand response implementation, highlighting its economic impact on the IES.
%It is evident from Figs. \ref{ele_scheduling} and \ref{gas_scheduling} that the electric and natural gas loads decrease during peak prices and increase during valley prices, and the load curve exhibits a “peak shaving and valley filling” pattern after demand response. Furthermore, Fig. \ref{heat_scheduling} illustrates that the thermal load notably decreases during 19:00-24:00 and 0:00-7:00 due to customers perceiving temperature less at night, and having a more extensive range of thermal comfort. In addition, to investigate the economic impact of demand response on the IES, Fig \ref{DR_cost} presents a comparison between the amount of electricity and natural gas purchased by the IES from the external supply network before and after the implementation of demand response.
\vspace{-2mm}
\begin{figure}[htbp]
\centering
\includegraphics[width=2.3in,height=1.5in]{fig/SAC_DR_cost.jpeg}
\caption{Comparison of the cost of energy purchased by IES from external energy supply networks before and after demand response.}
\label{DR_cost}
\end{figure}
\vspace{-2mm}
As shown in Fig \ref{DR_cost}, the introduction of demand response has led to a rational allocation of energy use within the IES through time-shifted electrical load and time-shifted gas load. This has resulted in a reduction in the system's electricity and natural gas procurement costs, as well as an increase in the utilization of renewable energy.
%The impact of IDR on scheduling is illustrated in Figs. \ref{ele_scheduling}-\ref{heat_scheduling}. TSE and TSQ achieve time-shifting for the electric and gas supply subsystems, reducing the purchase cost of electricity and natural gas and increasing the utilization of RES. The proposed method reduces a portion of the heating power during the dispatch period for the heating subsystem, while also considering users' thermal comfort range, compensation mechanisms, and TOU prices.%The impact of IDR on scheduling is shown in Figs. \ref{ele_scheduling}-\ref{heat_scheduling}, where the initial load represents the load without demand response and the actual load represents the load with demand response. For the electric supply and gas supply subsystems, TSE and TSQ play a time-shifting role, reducing the load during peak hours and increasing it during non-peak hours, realizing the effect of peak-shaving and valley-filling. This helps to reduce the purchase cost of electricity and natural gas, and better consume RES. For the heating subsystem, a portion of the heating power will be reduced during the dispatch period, considering the thermal comfort range of users, the effects of compensation mechanisms, and TOU prices.
%Figs. \ref{ele_scheduling}-\ref{heat_scheduling} show that the output combination of each device by the decision-making network can properly meet the resource demand of the current scheduling period. According to Fig. \ref{ele_scheduling}, the output of the WT has the highest priority to achieve complete consumption of renewable energy output; electricity purchased from the grid is concentrated during 0:00-7:00; as the load demand and power price increases, the MT unit gradually participates in the system operation. As can be seen from Fig. \ref{gas_scheduling}, the P2G absorbs electric energy for energy conversion during 0:00-7:00 and 23:00-24:00, and the MT is participated in system operation during 7:00-22:00. This indicates that coupling different subsystems through P2G and MT can improve the flexibility of system operation. It can be illustrated from Fig. \ref{heat_scheduling} that during 9:00-20:00 when the heat load is low, the heat demand is mainly provided by the MT; when the temperature drops at night and the electric price is in the valley and the WT output is high, the EB provides most of the heat demand.
\vspace{-4mm}
\subsection{Impact of Cyber-Attack on Scheduling}
%\subsubsection{Impact of Attack on Scheduling}
In this study we use HLR attack as described in Eq. (\ref{Eq25}). The change in indoor temperature and the heat load demand after the attack are shown in Figs. \ref{temperature_attack} and \ref{heat_load_attack}.
As shown in Fig. \ref{temperature_attack}, after the implementation of ITDSA, the indoor temperature gradually rises. As shown in Fig. \ref{heat_load_attack}, the increasing indoor temperature causes a gradual increase in heat load demand.
\vspace{-3mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/temperature_attack.jpeg}
\caption{Indoor temperature.}
\label{temperature_attack}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/heat_load_attack.jpeg}
\caption{Heat load.}
\label{heat_load_attack}
\end{minipage}
\end{figure}
To evaluate the impact of cyber-attacks on untrained decision networks and the resilience of our approach, we compared scheduling in four modes:
Mode 1: SAC-based IES scheduling without cyber-attack.
Mode 2: SAC-based IES scheduling with cyber-attack.
Mode 3: Proposed method for IES scheduling without cyber-attack.
Mode 4: Proposed method for IES scheduling with cyber-attack.
\iffalse
To verify the impact of cyber-attack on the decision network that are not trained for defense, and the resilience of our proposed approach. We compared scheduling results in four modes to evaluate the effectiveness of our method and demonstrate the impact of cyber-attacks on scheduling strategy.
Mode 1: IES scheduling using SAC without cyber-attack.
Mode 2: IES scheduling using SAC with cyber-attack.
Mode 3: IES scheduling using the proposed method without cyber-attack.
Mode 4: IES scheduling using the proposed method with cyber-attack.
\fi
\subsubsection{Impact of Cyber-Attack on Scheduling Using the SAC Algorithm}
Comparing the energy supply of each energy subsystem in Modes 1 and 2, the results are shown in Figs. \ref{mode1_2_ele}-\ref{mode1_2_gas}.
Note from the following figures that there the energy scheduling strategies of Modes 1 and 2 are clearly different. This suggests that decision networks that have not been trained for defense are often highly vulnerable. While they may be able to obtain optimal scheduling policies under normal conditions, they can experience drastic changes in scheduling policies after being subjected to cyber-attacks on the system state. This can lead to a significant decrease in scheduling quality.
\vspace{-6mm}
\begin{figure}[htbp]
\centering %居中
\subfigure[]{ %第一张子图
\begin{minipage}[t]{0.49\linewidth}\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/SAC-ele_scheduling.jpeg}\end{minipage}}\subfigure[]{ %第二张子图
\begin{minipage}[t]{0.49\linewidth}
\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/ele_attack_scheduling.jpeg}
\end{minipage}
}
\caption{Energy scheduling strategy in Modes 1(a) and 2(b) for the electrical subsystem.} %大图名称
\label{mode1_2_ele} %图片引用标记
\end{figure}
\vspace{-8mm}
\begin{figure}[htbp]
\centering %居中
\subfigure[]{ %第一张子图
\begin{minipage}[t]{0.49\linewidth}\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/SAC-heat_scheduling.jpeg}\end{minipage}}\subfigure[]{ %第二张子图
\begin{minipage}[t]{0.49\linewidth}
\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/heat_attack_scheduling.jpeg}
\end{minipage}
}
\caption{Energy scheduling strategy in Modes 1(a) and 2(b) for heating subsystem.} %大图名称
\label{mode1_2_heat} %图片引用标记
\end{figure}
\vspace{-8mm}
\begin{figure}[htbp]
\centering %居中
\subfigure[]{ %第一张子图
\begin{minipage}[t]{0.49\linewidth}\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/SAC-gas_scheduling.jpeg}\end{minipage}}\subfigure[]{ %第二张子图
\begin{minipage}[t]{0.49\linewidth}
\centering %子图居中
\includegraphics[height=3.3cm,width=4.2cm]{fig/gas_attack_scheduling.jpeg}
\end{minipage}
}
\caption{Energy scheduling strategy in Modes 1(a) and 2(b) for natural gas subsystem.} %大图名称
\label{mode1_2_gas} %图片引用标记
\end{figure}
Fig. \ref{mode1_2_ele} shows that HLR attacks significantly increase EB output due to false heat load data. Fig. \ref{mode1_2_heat} reveals that thermal storage's role in the heating subsystem grows to counteract false heat demand, suggesting capacity reserve can partly mitigate the attack, but may cause supply-demand imbalances without regulation, possibly necessitating cooling measures. Fig. \ref{mode1_2_gas} indicates MT output reduction due to EB's excessive output, with EB and HSD handling most heat load. The rise in EB's electricity demand affects IES's prices, particularly during peak periods, leading to price increases to adjust user demand, enhancing IES resilience against cyber-attacks.
%As shown in Fig. \ref{mode1_2_gas}, the output of the MT in the natural gas subsystem also shows a slight variation due to the abnormal output of the EB and thermal storage.
%The above figures illustrate the energy scheduling strategies in Mode 1 and Mode 2 differ significantly. In Mode 1, the HLR attack results in increased EB output, while in Mode 2, thermal storage participation in the heating subsystem increases significantly due to false heat load data. Capacity reserve can mitigate the attack, but an imbalance may occur when energy storage is not regulated. Fig. \ref{mode1_2_gas} also shows slight MT output variation in the natural gas subsystem. The attack-generated false load data increases the heat load, while the actual thermal load stays the same. This results in abnormal scheduling decisions, leading to economic losses and potential physical equipment damage.%The above analysis indicates that the increased heat load is a result of the false load data generated by the attack, while the actual thermal load remains the same. This leads to the generation of abnormal scheduling decisions, resulting in economic losses and potential damage to physical equipment.
%Comparing the output of each equipment of Mode 3 and Mode 4, the results are shown in Fig. \ref{SA-SAC_attack}.
%Fig. 17 compares the output of each equipment in Mode 3 and Mode 4. As shown in Fig. \ref{SA-SAC_attack}, the output of each equipment in Mode 3 and Mode 4 remains almost unchanged. The SAC-based scheduling scheme suffers from economic losses and significant changes when attacked, while the proposed method maintains stable output states for each equipment even after an attack, demonstrating greater resilience to attacks in scheduling strategies.%As can be seen from Fig. \ref{SA-SAC_attack}, the output of each equipment in Model 3 and Model 4 remains almost the same. The SAC-based scheduling scheme incurs economic losses and significant changes when attacked, whereas the proposed method maintains stable output states for each equipment even after an attack. Hence, the proposed method demonstrates greater resilience to attacks in scheduling strategies.
%\subsubsection{Impact of HSD and DR on Cyber-attack}
\subsubsection{Economic Analysis}
To demonstrate the effectiveness of our method, four operational scenarios were compared, differing in HSD participation and IDR application. Scenarios 3 and 4 exclude HSD, while Scenarios 2 and 3 operate without IDR. Table \ref{tab4} shows the IES net profit for each scenario across four scenarios:
Scenario 1: Includes HSD and applies IDR.
Scenario 2: Includes HSD but doesn't apply IDR.
Scenario 3: Excludes HSD but applies IDR.
Scenario 4: Excludes HSD and doesn't apply IDR.
\vspace{-4mm}
\begin{table}[ht]
\centering
\caption{Comparison of economy for different operating Scenarios}\label{tab4}
\begin{tabular}{ccccc}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\multirow{2}{*}{Operating Scenarios} & \multicolumn{4}{c}{Net profit of IES(¥)} \\
\cmidrule{2-5}
& Mode 1 & Mode 2 & Mode 3 & Mode 4 \\
\midrule
Scenario 1 & 8746.15 & 8147.27 & 8982.55 & 8980.95 \\
Scenario 2 & 7912.12 & 7512.22 & 7933.98 & 7933.14 \\
Scenario 3 & 7741.77 & 6796.90 & 8443.16 & 8437.37 \\
Scenario 4 & 7067.60 & 6351.33 & 7378.01 & 7377.91 \\
\specialrule{0.5pt}{0.5pt}{0.5pt}
\specialrule{0.5pt}{0.5pt}{0.5pt}
\end{tabular}
\label{tab:addlabel1}%
\end{table}
\vspace{-2mm}
Table \ref{tab4} reveals that our method yields higher profits across various scenarios, demonstrating resilience and economic efficiency in normal and cyber-attack conditions. In normal operation, profits increase significantly with the implementation of the IDR program, indicating effective user energy usage guidance. Under cyber-attack, our method's defensive training ensures less volatility and increased profits, showcasing the resilience of our decision-making network. Additionally, the involvement of HSD enhances IES's economic efficiency and resilience, mitigating attack damages by consuming excess heat load, as shown by notable profit increases with the attacks.
\iffalse
As shown in Table \ref{tab4}, by comparing the four different scenarios under Modes 1 and 3, it is evident that our proposed method results in higher profits even under normal operating conditions. When comparing the four scenarios under Modes 2 and 4, the defensive training of the decision network in our proposed method ensures less volatility and significantly higher profits, even under the influence of cyber-attack. Specifically, in Scenario 1, the IES operator's net profit improves from 8147.27¥ in Mode 2 to 8980.95¥ in Mode 4, highlighting the resilience of our decision-making neural network to cyber-attack after tightening linear bounds.
Table \ref{tab4} demonstrates that the IDR program can enhance the economic efficiency of system operation while also providing a level of resilience to the IES. By comparing the four different modes in Scenarios 1 and 2, it is evident that the implementation of the IDR program has significantly increased the profits of the IES operators. Specifically, in normal operation, the net profit of Mode 1 increases from 7912.12¥ in Scenario 2 to 8746.15¥ in Scenario 1, and the net profit of Mode 3 increases from 7933.98¥ in Scenario 2 to 8982.55¥ in Scenario 1. This indicates that our proposed pricing mechanism effectively guides users to adjust their energy usage patterns, resulting in a significant increase in the IES operator's net profit. In the case of an cyber-attack, the net profit of Mode 2 increases from 7512.22¥ in Scenario 2 to 8147.27¥ in Scenario 1, and the net profit of Mode 4 increases from 7933.14¥ in Scenario 2 to 8980.95¥ in Scenario 1. This indicates that the resilience of IES in an cyber-attack scenario is enhanced by utilizing flexible load demand response.
Table \ref{tab4} also shows that the participation of HSD can enhance the economic efficiency of the IES, while also offering a certain degree of resilience. When an HLR attack occurs, a false heat load demand is generated, which leads to an increase in EB output. The excess heat load can then be consumed by the HSD, mitigating the damage caused by the attack. Specifically, in the cyber-attack scenario, the net profit of Mode 2 increases from 6796.90¥ in Scenario 3 to 8147.27¥ in Scenario 1, and the net profit of Mode 4 increases from 8437.37¥ in Scenario 3 to 8980.95¥ in Scenario 1.
\fi
\iffalse
\subsubsection{Analysis of Resilient Scheduling Based on the Proposed Method}
To illustrate the effectiveness of our proposed method in an attack scenario, Fig. \ref{SA-SAC_attack} presents a comparison of the power output of each device between Modes 3 and 4.
%\vspace{-4.5mm}
\begin{figure}[htbp]
\centering
\includegraphics[width=2.2in,height=1.5in]{fig/SA-SAC_attack.jpeg}
\caption{Energy dispatching scheme in Modes 3 and 4.}
\label{SA-SAC_attack}
\end{figure}
As shown in Fig. \ref{SA-SAC_attack}, the output of each device in Modes 3 and 4 remains nearly constant, and the level of interaction with the distribution network remains the same as when no attack is received, which demonstrates the high resilience of our proposed method to cyber-attack.
\fi
%The SAC-based scheduling scheme suffers from economic losses and significant changes when attacked, while the proposed method maintains stable output states for each equipment even after an attack, demonstrating greater resilience to attacks in scheduling strategies.
%The above analysis indicates that the scheduling scheme based on SAC undergoes significant changes when attacked, resulting in economic loss. In contrast, the scheduling scheme based on the method proposed in this study can maintain a more stable output state for each equipment even after an attack. Therefore, the proposed method is more resilient for scheduling strategies that consider the attack.
%\vspace{-3.5mm}
\iffalse
\subsection{Analysis of the Ability to Handle Uncertainty}
To test the method's ability to handle uncertainties, we simulated several special scenarios under typical conditions: increased and decreased WT output and increased and decreased electrical, gas, and heat loads. Figs. \ref{WT_rise}-\ref{WT_down} show the strategies for managing changes in WT output, while Figs. \ref{load_rise}-\ref{load_down} show the strategies for handling changes in load demand.
%\vspace{-5.2mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.1cm,width=4.0cm]{fig/SA-SAC_WT_rise.jpeg}
\caption{Dispatching scheme for WT output increase.}
\label{WT_rise}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[height=3.1cm,width=4.2cm]{fig/SA-SAC_WT_down.jpeg}
\caption{Dispatching scheme for WT output decrease.}
\label{WT_down}
\end{minipage}
\end{figure}
%\vspace{-8mm}
\begin{figure}[htbp]
\begin{minipage}[t]{0.49\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/SA-SAC_load-rise.jpeg}
\caption{Dispatching scheme for loads increase.}
\label{load_rise}
\end{minipage}%
\hfill
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[height=3.3cm,width=4.2cm]{fig/SA-SAC_load-down.jpeg}
\caption{Dispatching scheme for loads decrease.}
\label{load_down}
\end{minipage}
\end{figure}
%\vspace{-2mm}
%To test the ability of the proposed method to handle various uncertainties, we performed simulations of four special scenarios under typical scenarios. These scenarios include an increase and decrease in WT output as well as an increase and decrease in electrical, gas, and heat loads. The strategies to manage the changes in the WT output are shown in Figs. \ref{WT_rise}-\ref{WT_down}. The strategies to handle the changes in load demand are shown in Figs. \ref{load_rise}-\ref{load_down}.
Figs. \ref{WT_rise} and \ref{WT_down} demonstrate the impact of increasing and decreasing WT output, respectively. When the WT output increases, P2G and EB power consumption increase while MT output decreases, with excess wind power traded with the grid. Conversely, when the WT output decreases, MT power increases to meet electrical load, with the remaining deficit supplied by the grid. Similarly, Figs. \ref{load_rise} and \ref{load_down} depict the effect of increasing and decreasing electrical, heat, and gas loads on the power system. As the loads increase, P2G power is increased to meet the gas load, and MT power is increased to meet the electrical and heat loads, with the grid fulfilling the remaining power deficit. Conversely, when the loads decrease, the output of MT decreases, leading to a reduction in the power traded with the grid, as well as in the output of P2G and electric boilers to match the decreased loads.%Fig. \ref{WT_rise} shows that as the WT output increases, P2G and EB power consumption increases while the MT output decreases. The excess wind power is traded with the grid. Conversely, in Fig. \ref{WT_down}, as the WT output decreases, MT power increases to meet the electrical load while the remaining deficit is fulfilled by the grid.%As shown in Fig. \ref{WT_rise}, when the WT output increases, the power of P2G and EB is increased to consume wind power, while the MT output decreases because most of the electric load is satisfied by wind power and most of the heat load is satisfied by EB; wind power that cannot be fully consumed is traded with the grid. As shown in Fig. \ref{WT_down}, when the WT output decreases, the power of the MT is boosted to meet the electrical load and the remaining electrical load deficit is satisfied by the grid.
After conducting tests on the special scenarios, it can be observed that our proposed method can be dynamically adjusted to accommodate changes in renewable energy and loads, resulting in a more reasonable scheduling strategy. The results indicate that our proposed method is capable of being resilient to cyber-attacks while effectively managing multi-uncertainties.
\fi
%As shown in Fig. \ref{load_rise}, when the electrical, heat and gas loads are increased, the power of P2G is increased to meet the gas load, and the power of MT is increased to meet the electrical and heat loads; the remaining power deficit is then supplied by the grid. Fig. \ref{load_down} shows that when the loads are reduced, the output of MT is decreased in response to the reduced electrical and heat loads; the power traded with the grid, as well as the output of P2G and electric boilers, are reduced to match the decreased loads.
\vspace{-2mm}
\section{Conclusion}
%To achieve a stable supply of electricity-gas-heat multiple energy demands and strengthen the utilization of RES, a DRL-bsed resilience scheduling method for IES is proposed. The proposed method takes into account the multi-uncertainties of both the RES and loads, as well as potential cyber-attack.
A resilient scheduling method based on DRL for IES effectively addresses multiple uncertainties from RES, loads, and cyber-attacks, ensuring stable supply and enhanced utilization of renewable energy sources. Simulations in North China yielded these conclusions:
(1) The method effectively manages source load uncertainties and resists cyber-attacks, ensuring stable energy supply. It increases renewable energy use via P2G and EB, with a pricing mechanism that guides user DR, particularly during cyber-attacks, and HSD devices that mitigate HLR attack damages.
(2) The SA-MDP model accurately represents scheduling challenges during cyber-attacks. The SA-SAC algorithm, with an action space regularizer, surpasses the SAC in resilience and performance, both under attack and in normal conditions.
(3) Test results show the algorithm's success in creating a resilient IES scheduling strategy with superior economic performance, handling multi-uncertainties and suitable for real-time scheduling across various scenarios.
\iffalse
A DRL-based resilience scheduling method for IES is proposed, considering multi-uncertainties of RES and loads, as well as potential cyber-attacks, to ensure stable supply and strengthen RES utilization. The proposal is simulated on an IES in North China, leading to the following conclusions:
(1) The proposed IES scheduling method presented in this paper effectively handles multi-uncertainties of source loads while being resilient to cyber-attacks, ensuring a stable supply of electricity, gas, and heat. Through the use of P2G and EB, renewable energy consumption is significantly increased. The proposed pricing mechanism guides users to adjust energy use and provides system resilience by guiding user demand response through adjusting prices in real time during cyber-attacks. Moreover, the introduction of HSD device effectively mitigates the damage caused by HLR attacks.
(2) The developed SA-MDP model accurately depicts the scheduling problem in the presence of cyber-attacks. The SA-SAC algorithm generates an optimal learning policy with cyber-attack resilience, outperforming the original SAC algorithm in the attacked scenario and achieving better performance even when not attacked due to the introduction of an action space regularizer.
(3) The test results indicate that the proposed algorithm successfully generates a cyber-attack resilient IES scheduling strategy with better economic performance than the original SAC algorithm. The method can handle multi-uncertainties and is suitable for real-time scheduling in different scenarios.
\fi
%The simulation results demonstrate the effectiveness of the proposed algorithm in obtaining an IES scheduling strategy that is resilient to cyber-attacks. The strategy's economic performance is significantly better than that of the original SAC algorithm in the attack scenario. Furthermore, the method can effectively handle multi-uncertainties and is appropriate for real-time scheduling in various scenarios.
%This paper focuses on IES resilient scheduling with the example of HLR attack on the thermal subsystem. However, cooperative attacks should be considered in more realistic scenarios as they can pose greater security risks to the system. The dynamic pricing mechanism proposed in this paper primarily focuses on the role and impact of energy suppliers in an IES, providing a feasible mechanism example; however, the actual implementation of pricing mechanisms in real-world scenarios may involve various stakeholders, including government entities, regulatory bodies, and market operators.
%It is also interesting to explore coordinating the interests of operator and users in IES scheduling.
% Therefore, more realistic application scenarios should be considered to generate resilient IES scheduling strategies in the collaborative cyber-attack scenario. This paper addresses IES's resilient scheduling problem against cyber-attacks on the thermal subsystem. Besides, this study only considers the maximization of the operator's profit without considering users' interests. on IES
%and economic
\vspace{-2mm}
\bibliographystyle{IEEEtran}
\bibliography{Attack-Resilient.bbl}
\end{document}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{wang2018review}
D.~Wang, L.~Liu, H.~Jia, W.~Wang, Y.~Zhi, Z.~Meng, and B.~Zhou, ``Review of key
problems related to integrated energy distribution systems,'' \emph{CSEE
Journal of Power and Energy Systems}, vol.~4, no.~2, pp. 130--145, 2018.
\bibitem{liu2020region}
L.~Liu, D.~Wang, K.~Hou, H.~Jia, and S.~Li, ``Region model and application of
regional integrated energy system security analysis,'' \emph{Applied Energy},
vol. 260, p. 114268, 2020.
\bibitem{lu2020hydraulic}
S.~Lu, W.~Gu, C.~Zhang, K.~Meng, and Z.~Dong, ``Hydraulic-thermal cooperative
optimization of integrated energy systems: A convex optimization approach,''
\emph{IEEE Transactions on Smart Grid}, vol.~11, no.~6, pp. 4818--4832, 2020.
\bibitem{che2018mitigating}
L.~Che, X.~Liu, and Z.~Li, ``Mitigating false data attacks induced overloads
using a corrective dispatch scheme,'' \emph{IEEE Transactions on Smart Grid},
vol.~10, no.~3, pp. 3081--3091, 2018.
\bibitem{liang20162015}
G.~Liang, S.~R. Weller, J.~Zhao, F.~Luo, and Z.~Y. Dong, ``The 2015 ukraine
blackout: Implications for false data injection attacks,'' \emph{IEEE
Transactions on Power Systems}, vol.~32, no.~4, pp. 3317--3318, 2016.
\bibitem{zheng2019stochastic}
J.~Zheng, Y.~Kou, M.~Li, and Q.~Wu, ``Stochastic optimization of cost-risk for
integrated energy system considering wind and solar power correlated,''
\emph{Journal of Modern Power Systems and Clean Energy}, vol.~7, no.~6, pp.
1472--1483, 2019.
\bibitem{emrani2021optimal}
P.~Emrani-Rahaghi and H.~Hashemi-Dezaki, ``Optimal scenario-based operation and
scheduling of residential energy hubs including plug-in hybrid electric
vehicle and heat storage system considering the uncertainties of electricity
price and renewable distributed generations,'' \emph{Journal of Energy
Storage}, vol.~33, p. 102038, 2021.
\bibitem{li2021optimal}
Y.~Li, B.~Wang, Z.~Yang, J.~Li, and G.~Li, ``Optimal scheduling of integrated
demand response-enabled community-integrated energy systems in uncertain
environments,'' \emph{IEEE Transactions on Industry Applications}, vol.~58,
no.~2, pp. 2640--2651, 2021.
\bibitem{zhao2021coordinated}
B.~Zhao, A.~J. Lamadrid, R.~S. Blum, and S.~Kishore, ``A coordinated scheme of
electricity-gas systems and impacts of a gas system fdi attacks on
electricity system,'' \emph{International Journal of Electrical Power \&
Energy Systems}, vol. 131, p. 107060, 2021.
\bibitem{shayan2019network}
H.~Shayan and T.~Amraee, ``Network constrained unit commitment under cyber
attacks driven overloads,'' \emph{IEEE Transactions on Smart Grid}, vol.~10,
no.~6, pp. 6449--6460, 2019.
\bibitem{liu2016false}
X.~Liu and Z.~Li, ``False data attacks against ac state estimation with
incomplete network information,'' \emph{IEEE Transactions on Smart Grid},
vol.~8, no.~5, pp. 2239--2248, 2016.
\bibitem{zhao2021cyber}
P.~Zhao, Z.~Cao, D.~D. Zeng, C.~Gu, Z.~Wang, Y.~Xiang, M.~Qadrdan, X.~Chen,
X.~Yan, and S.~Li, ``Cyber-resilient multi-energy management for complex
systems,'' \emph{IEEE Transactions on Industrial Informatics}, vol.~18,
no.~3, pp. 2144--2159, 2021.
\bibitem{ding2022cyber}
S.~Ding, W.~Gu, S.~Lu, R.~Yu, and L.~Sheng, ``Cyber-attack against heating
system in integrated energy systems: Model and propagation mechanism,''
\emph{Applied Energy}, vol. 311, p. 118650, 2022.
\bibitem{zhang2021soft}
B.~Zhang, W.~Hu, D.~Cao, T.~Li, Z.~Zhang, Z.~Chen, and F.~Blaabjerg, ``Soft
actor-critic--based multi-objective optimized energy conversion and
management strategy for integrated energy systems with renewable energy,''
\emph{Energy Conversion and Management}, vol. 243, p. 114381, 2021.
\bibitem{yang2021dynamic}
T.~Yang, L.~Zhao, W.~Li, and A.~Y. Zomaya, ``Dynamic energy dispatch strategy
for integrated energy system based on improved deep reinforcement learning,''
\emph{Energy}, vol. 235, p. 121377, 2021.
\bibitem{tajalli2020resilient}
S.~Z. Tajalli, M.~Mardaneh, E.~Taherian-Fard, A.~Izadian, A.~Kavousi-Fard,
M.~Dabbaghjamanesh, and T.~Niknam, ``Dos-resilient distributed optimal
scheduling in a fog supporting iiot-based smart microgrid,'' \emph{IEEE
Transactions on Industry Applications}, vol.~56, no.~3, pp. 2968--2977, 2020.
\bibitem{chu2022mitigating}
Z.~Chu, S.~Lakshminarayana, B.~Chaudhuri, and F.~Teng, ``Mitigating
load-altering attacks against power grids using cyber-resilient economic
dispatch,'' \emph{IEEE Transactions on Smart Grid}, 2022.
\bibitem{huang2021distributed}
B.~Huang, Y.~Li, F.~Zhan, Q.~Sun, and H.~Zhang, ``A distributed robust economic
dispatch strategy for integrated energy system considering cyber-attacks,''
\emph{IEEE Transactions on Industrial Informatics}, vol.~18, no.~2, pp.
880--890, 2021.
\bibitem{zhao2020cyber}
P.~Zhao, C.~Gu, Z.~Cao, D.~Xie, F.~Teng, J.~Li, X.~Chen, C.~Wu, D.~Yu, X.~Xu
\emph{et~al.}, ``A cyber-secured operation for water-energy nexus,''
\emph{IEEE Transactions on Power Systems}, vol.~36, no.~4, pp. 3105--3117,
2020.
\bibitem{hua2019optimal}
H.~Hua, Y.~Qin, C.~Hao, and J.~Cao, ``Optimal energy management strategies for
energy internet via deep reinforcement learning approach,'' \emph{Applied
Energy}, vol. 239, pp. 598--609, 2019.
\bibitem{nakabi2021deep}
T.~A. Nakabi and P.~Toivanen, ``Deep reinforcement learning for energy
management in a microgrid with flexible demand,'' \emph{Sustainable Energy,
Grids and Networks}, vol.~25, p. 100413, 2021.
\bibitem{mao2019pmv}
N.~Mao, J.~Hao, T.~He, M.~Song, Y.~Xu, and S.~Deng, ``Pmv-based dynamic
optimization of energy consumption for a residential task/ambient air
conditioning system in different climate zones,'' \emph{Renewable Energy},
vol. 142, pp. 41--54, 2019.
\bibitem{ku2014automatic}
K.~Ku, J.~Liaw, M.~Tsai, and T.~Liu, ``Automatic control system for thermal
comfort based on predicted mean vote and energy saving,'' \emph{IEEE
Transactions on Automation Science and Engineering}, vol.~12, no.~1, pp.
378--383, 2014.
\bibitem{zang2019robust}
H.~Zang, M.~Geng, M.~Xue, X.~Mao, M.~Huang, S.~Chen, Z.~Wei, and G.~Sun, ``A
robust state estimator for integrated electrical and heating networks,''
\emph{IEEE Access}, vol.~7, pp. 109\,990--110\,001, 2019.
\bibitem{zhang2020robust}
H.~Zhang, H.~Chen, C.~Xiao, B.~Li, M.~Liu, D.~Boning, and C.-J. Hsieh, ``Robust
deep reinforcement learning against adversarial perturbations on state
observations,'' \emph{Advances in Neural Information Processing Systems},
vol.~33, pp. 21\,024--21\,037, 2020.
\bibitem{zhang2019towards}
H.~Zhang, H.~Chen, C.~Xiao, S.~Gowal, R.~Stanforth, B.~Li, D.~Boning, and C.-J.
Hsieh, ``Towards stable and efficient training of verifiably robust neural
networks,'' \emph{arXiv preprint arXiv:1906.06316}, 2019.
\end{thebibliography}