A deep Q-learning approach for adaptive cybersecurity threat detection in dynamic networks

Abstract

Cybersecurity faces persistent challenges due to the rapid growth and complexity of network-based threats. Conventional rule-based systems and classical machine learning approaches often lack the adaptability required to detect advanced and dynamic attacks in real time. This study introduces a deep Q-learning framework for autonomous threat detection and mitigation within a simulated network environment that reflects realistic traffic, malicious behaviors, and system conditions. The framework incorporates experience replay and target network stabilization to strengthen learning and policy optimization. Evaluation was performed on a synthesized dataset containing benign traffic and multiple attack categories, including distributed denial of service (DDoS), phishing, advanced persistent threats, and malware. The proposed system achieved 96.7% detection accuracy, an F1-score of 0.94, and reduced detection latency to 50 ms. These results surpassed the performance of rule-based methods and traditional classifiers such as support vector machines, random forests, convolutional neural networks, and recurrent neural networks. A central contribution lies in combining dynamic feature selection with reinforcement learning (RL), allowing the agent to adapt to evolving threats and diverse network conditions. The findings demonstrate the potential of deep reinforcement learning (DRL) as a scalable and efficient solution for real-time cybersecurity defense.