Reinforcement learning pentesting agent training in realistic network environments

Ensuring the security of network systems and infrastructure is a critical aspect of cybersecurity. Penetration testing (pentesting) is an effective method for evaluating the network security posture.

In recent years, researchers have aimed to develop efficient approaches for conducting the pentesting procedure automatically to address the issues in traditional manual and time-consuming methods. One approach is to use reinforcement learning (RL) techniques, which have been applied to create automated agents that mimic the actions of human pentesters but have enhanced speed, scale, and precision.

Various simulation environments have been introduced as the main method to train these RL agents. However, the heavy reliance on predefined constants and probabilistic values for agent actions and environment states leads to potential inaccuracies in replicating real-world behavior due to factors that were not modeled, thus decreasing agent accuracy and performance. In addition, the simulated network may not accurately represent the configuration and topology of an actual network.

To address this “reality gap,” a team of researchers led by Associate Professor Razvan Beuran, along with his doctoral student Huynh Phuong Thanh Nguyen at the Japan Advanced Institute of Science and Technology (JAIST), and researchers at KDDI Research, has designed and implemented PenGym, an effective and reliable realistic training framework for RL pentesting agents that was developed as part of a joint project with KDDI Research.

PenGym enables RL agents to execute actual actions on realistic hosts in network environments. For this purpose, the framework contains an Action/State Module that implements a set of real pentesting actions for the interaction between the RL agents and the training environment. Moreover, the training environment is based on the cyber range technology used for human cybersecurity training and is created automatically according to several pentesting scenarios.

Several optimization techniques were implemented to enhance the time execution performance of PenGym. As a result, their framework eliminates the need for action modeling, resulting in a more accurate representation of network and security dynamics compared to simulation-based environments. Their study was published in Computers & Security.

The approach of using a real network environment that makes possible the execution of actual pentesting actions, as employed in this research, yields promising results compared to simulated environments. In particular, their experiments demonstrated the advantages and effectiveness of using PenGym as a realistic training environment for RL pentesting agents. Thus, the PenGym-trained agents showed a superior pentesting performance in real networks compared to simulation-trained agents.

Based on the experiment results the researchers obtained, they consider that their research could lead to changes in various network-related research areas, potentially replacing the traditional approach of creating complex logical models to simulate network environments with more realistic methods. Furthermore, realistic training environments can be applied to other research areas.

One important example is automated cyber defense using RL agents, which can be used to enhance the protection mechanisms of real network infrastructure and contribute to the trustworthiness of Society 5.0. To support the potential activities of other researchers in this field, they released PenGym as open source on GitHub.