A Cybersecurity-Oriented Framework for Windows Malware Detection via Neural Architecture Search Using Advantage Actor-Critic
DOI:
https://doi.org/10.65204/djes.v3i3.855Keywords:
Neural Architecture Search, Advantage Actor-Critic , Deep Learning, Cybersecurity, Windows Malware Detection, Intelligent AgentAbstract
The continuous growth of Windows-based systems has been accompanied by a parallel increase in sophisticated malware threats, posing serious risks to digital infrastructures. Traditional detection mechanisms, particularly signature-based and manually designed learning models, often struggle to cope with rapidly evolving malware variants and complex evasion techniques. This limitation highlights the need for adaptive and automated solutions capable of improving detection performance without relying heavily on expert-driven design. This study proposes a novel framework for Windows malware detection based on neural architecture search guided by an Intelligent Agent using the Advantage Actor-Critic approach. The proposed method formulates architecture design as a sequential decision process, enabling the agent to construct an optimized neural model by selecting appropriate components from a structured search space. This strategy reduces manual intervention while enhancing the model’s ability to generalize across diverse malware patterns. The proposed framework was evaluated on the EMBER dataset, a widely recognized benchmark for static malware analysis. Experimental results demonstrate strong performance, achieving an accuracy of 0.971, along with superior results in AUC, precision, recall, and F1-score compared to existing approaches. These results confirm the effectiveness of the automatically discovered architecture in capturing meaningful feature interactions while maintaining computational efficiency. Overall, the integration of reinforcement learning with neural architecture design provides a scalable and practical solution for modern malware detection challenges.
References
Y. M. Mohialden, M. T. Younis, S. A. Salman, and E. A. W. Hachim, “Challenges and advancements in quantum cryptography: Standardization, security risks, and practical implementations,” Proc. Int. Conf. Applied Innovations in IT (ICAIIT), vol. 13, no. 4, pp. 307–314, 2025.
J. Ferdous, R. Islam, A. Mahboubi, and M. Z. Islam, “A survey on ML techniques for multi-platform malware detection: Securing PC, mobile devices, IoT, and cloud environments,” Sensors, vol. 25, no. 4, p. 1153, 2025.
M. Tanha and S. Kafaie, “A review of explainable machine learning for Android malware detection and analysis,” IEEE Access, 2025.
S. Berrios, D. Leiva, B. Olivares, H. Allende-Cid, and P. Hermosilla, “Systematic review: Malware detection and classification in cybersecurity,” Applied Sciences, vol. 15, no. 14, p. 7747, 2025.
A. Basim, I. M. Ali, R. A. Lateef, and Z. L. Ali, “Enhanced ransomware traffic detection using a hybrid ResNeSt101 and Isolation Forest framework,” in Proc. SPIE, Third Int. Conf. Emerging Trends in AI and Computational Technologies (ICONEST 2025), vol. 14141, pp. 87–96, 2026.
M. Alshoulie and A. Mehmood, “Deep learning approaches for malware detection: A comprehensive review of techniques, challenges, and future directions,” IEEE Access, 2025.
I. K. Abbas, M. S. Mahdi, A. Basim, and K. I. Abbas, “An agent-based reinforcement learning framework for dynamic cryptographic security,” Dijlah J. Eng. Sci., vol. 3, no. 1, pp. 377–391, Mar. 2026.
S. Alzahrani, Y. Xiao, S. Asiri, J. Zheng, and T. Li, “A survey of ransomware detection methods,” IEEE Access, 2025.
S. F. Ali, M. R. Abdulrazzaq, and M. T. Gaata, “Learning techniques-based malware detection: A comprehensive review,” Mesopotamian J. CyberSecurity, vol. 5, no. 1, pp. 273–300, 2025.
W. Almobaideen, O. Abu Alghanam, M. Abdullah, S. B. Hussain, and U. Alam, “Comprehensive review on machine learning and deep learning techniques for malware detection in Android and IoT devices,” Int. J. Inf. Security, vol. 24, no. 3, p. 110, 2025.
Y. Dehfouli and A. Habibi Lashkari, “Memory analysis for malware detection: A comprehensive survey using the OSCAR methodology,” ACM Comput. Surv., vol. 58, no. 4, pp. 1–58, 2025.
X. Huang, L. Ma, W. Yang, and Y. Zhong, “A method for Windows malware detection based on deep learning,” J. Signal Process. Syst., vol. 93, no. 2–3, pp. 265–273, 2020.
E. Amer and I. Zelinka, “A dynamic Windows malware detection and prediction method based on contextual understanding of API call sequence,” Computers & Security, vol. 92, p. 101760, 2020.
F. Çatak, A. Yazı, O. Elezaj, and J. Ahmed, “Deep learning based sequential model for malware analysis using Windows exe API calls,” PeerJ Comput. Sci., vol. 6, p. e285, 2020.
C. Wang, Z. Zhao, F. Wang, and Q. Li, “A novel malware detection and family classification scheme for IoT based on DEAM and DenseNet,” Security Commun. Networks, vol. 2021, pp. 1–16, 2021.
N. Azeez, O. Odufuwa, S. Misra, J. Oluranti, and R. Damaševičius, “Windows PE malware detection using ensemble learning,” Informatics, vol. 8, no. 1, p. 10, 2021.
M. Dener, G. Ok, and A. Orman, “Malware detection using memory analysis data in big-data environment,” Applied Sciences, vol. 12, no. 17, p. 8604, 2022.
V. Ravi and M. Alazab, “Attention-based convolutional neural network deep learning approach for robust malware classification,” Computational Intelligence, vol. 39, no. 1, pp. 145–168, 2022.
F. ALGorain and J. Clark, “Bayesian hyper-parameter optimisation for malware detection,” Electronics, vol. 11, no. 10, p. 1640, 2022.
S. Kudrekar and U. Rani, “Classification of malware using multinomial linked latent modular double Q learning,” Indonesian J. Electr. Eng. Comput. Sci., vol. 28, no. 1, p. 577, 2022.
M. Kim, H. Cho, and J. Yi, “Large-scale analysis on anti-analysis techniques in real-world malware,” IEEE Access, vol. 10, pp. 75802–75815, 2022.
A. Koçak, E. Söğüt, M. Alkan, and O. Erdem, “Detection of different Windows PE malware using machine learning methods,” Politeknik Dergisi, vol. 26, no. 3, pp. 1185–1197, 2023.
I. E. Maulani, T. Herdianto, M. O. Laksana, F. Syawaludin, and K. Komarudin, “Exploring the effectiveness of artificial intelligence in detecting malware and improving cybersecurity in computer networks,” Eduvest: J. Universal Studies, vol. 3, no. 4, 2023.
C. Du et al., “Toward detecting malware based on process-aware behaviors,” Security Commun. Networks, vol. 2023, pp. 1–16, 2023.
D. Syeda and M. Asghar, “Dynamic malware classification and API categorisation of Windows portable executable files using machine learning,” Applied Sciences, vol. 14, no. 3, p. 1015, 2024.
I. Zada et al., “Fine-tuning cyber security defenses: Evaluating supervised machine learning classifiers for Windows malware detection,” Computers, Materials & Continua, vol. 80, no. 2, pp. 2917–2939, 2024.
H. Gururaja, N. Khandige, N. Nayak, R. Srivatsan, and N. S., “Ensemble learning for robust malware detection in the Windows 7 environment,” Int. Res. J. Adv. Engg. Hub, vol. 2, no. 2, pp. 261–270, 2024.
M. Imran, A. Appice, and D. Malerba, “Evaluating realistic adversarial attacks against machine learning models for Windows PE malware detection,” Future Internet, vol. 16, no. 5, p. 168, 2024.
S. Khan and M. Nauman, “Interpretable detection of malicious behavior in Windows portable executables using multi-head 2D transformers,” Big Data Mining and Analytics, vol. 7, no. 2, pp. 485–499, 2024.
M. S. Mahdi, “A deep learning–based hybrid neural network model for malware detection,” Journal of Techniques, vol. 8, no. 1, pp. 62–70, 2026.
“EMBER: Dataset for training static PE malware machine learning models,” Kaggle. [Online]. Available: https://www.kaggle.com/datasets/dhoogla/ember-2018-v2-features. [Accessed: Mar. 14, 2026].
M. S. Mahdi, Z. L. Ali, A. R. Rashid, N. K. Ibrahim, and A. W. Abdulghafour, “A hybrid deep learning model for facial emotion recognition: Combining multi-scale features, dynamic attention, and residual connections,” Proc. Int. Conf. Applied Innovations in IT (ICAIIT), vol. 13, no. 2, pp. 69–77, 2025.
