Resilient Multi-Agent Deep Reinforcement Learning for Automatic Generation Control Under Adversarial Perturbations

An efficient method is proposed to improve the cyber-physical resilience of multi-agent deep deterministic policy gradient (MADDPG)-based automatic generation control (AGC) against malicious adversarial perturbations in the agents’ observation space. We first provide a sensitivity analysis demonstrating that the Jacobian of deep neural network (DNN) policies can amplify small perturbations into large and suboptimal control actions that degrade load-frequency control (LFC) performance. Subsequently, three models of perturbations based on adversary knowledge of the control system, namely random attacks, sample-based attacks (SBAs), and gradient-based attacks (GBAs), are introduced to evaluate the controller’s vulnerability to corrupted agents’ observation spaces. Additionally, it is shown that GBAs possess the most destructive impact on controller performance. To mitigate this vulnerability, a resilient MADDPG (RMADDPG) framework is proposed, which incorporates a robust retraining phase. Motivated by principles of robust control theory, this phase injects gradient-based perturbations into the agents’ observation space, thereby improving policy resilience to adversarial disturbances. The effectiveness of the proposed framework is validated on the IEEE 39-bus system against various attack scenarios, including coordinated attacks. The results show that although RMADDPG adopts a more conservative control strategy during normal load disturbances, it successfully prevents frequency oscillations during attacks, unlike the standard MADDPG. Furthermore, the scalability of the approach is demonstrated on the larger IEEE 118-bus system, where it outperforms conventional robust control methods. Note to Practitioners—Recent studies demonstrate the effectiveness of artificial intelligence-based controllers, particularly MADDPG, for real-time regulation of grid frequency. These controllers improve frequency regulation during normal operating conditions, but they are vulnerable to cyberattacks. Such attacks can manipulate system measurements without being detected and subsequently lead AGC to take unsafe control actions, potentially causing frequency oscillations or even loss of stability. This paper shows that DNN-based controllers can unintentionally amplify small perturbations in measurement data into large control deviations. To address this issue, we introduce a resilient retraining framework, as a result of which controllers provide smoother responses during cyberattacks. Although this resilience comes at the cost of conservative behavior under normal conditions, it substantially improves stability when data integrity is compromised. From a practical standpoint, the proposed approach does not require changes to existing control hardware, and it can be integrated as an offline retraining step using simulation models. Limitations include additional training effort and a trade-off between nominal performance and robustness.

Latest RAS news

Q&A: Mars Perseverance Rover Distance Record

On October 11, the Perseverance rover completed a drive that brought its total distance traveled on Mars to

Video Friday: Robot Decommissioning Takes a Fun Turn

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics.

This Disembodied Hand Is All the Robot You Need

A hand attached to the end of an arm is a given for most humans, but what if

HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object Interaction

This White Paper gives robotics researchers and engineers an overview of a new large-scale motion capture dataset built