Energy Aware Reinforcement Learning Approach for Dynamic Production Scheduling Optimization in Sustainable Smart Manufacturing Environments

Yogiek Indra Kurniawan; Krisna Widi Nugraha; Rosyid Ridlo Al-Hakim; Erick Fernando; Rian Ardianto; Genrawan Hoendarto; Mursalim Mursalim

doi:10.61132/ijmicse.v2i4.408

Authors

Yogiek Indra Kurniawan Universitas Jenderal Soedirman
Krisna Widi Nugraha Universitas Harapan Bangsa
Rosyid Ridlo Al-Hakim Universitas Harapan Bangsa
Erick Fernando Universitas Multimedia Nusantara
Rian Ardianto Universitas Harapan Bangsa
Genrawan Hoendarto Universitas Widya Dharma
Mursalim Mursalim Universitas Sugeng Hartono

DOI:

https://doi.org/10.61132/ijmicse.v2i4.408

Keywords:

Energy Efficiency, Production Scheduling, Reinforcement Learning, System Simulation, Throughput Optimization

Abstract

Background: The development of modern manufacturing systems requires production scheduling strategies that not only improve productivity but also optimize energy utilization. Multi-machine production systems with job-shop configurations exhibit high complexity due to dynamic interactions between machines, job queues, and varying processing times, making conventional scheduling methods less effective in handling changing operational conditions. Objective: This study aims to develop and evaluate a reinforcement learning based production scheduling approach to improve production efficiency while reducing energy consumption in multi-machine manufacturing systems. Methods: This research employs a job-shop based multi-machine production simulation model as the experimental environment. The scheduling problem is formulated as a Markov Decision Process, enabling the implementation of reinforcement learning algorithms, namely Q-learning and Deep Q-Network, to learn optimal scheduling policies through interaction with the simulation environment. Energy consumption parameters are incorporated into the reward function so that the learning agent can consider energy efficiency in the scheduling decision-making process. System performance is evaluated using three main metrics, namely energy consumption, throughput, and makespan. Results: The experimental results show that the reinforcement learning based scheduling approach achieves better performance compared to conventional scheduling methods, resulting in lower energy consumption, higher job completion rates, and shorter production completion times within the multi-machine manufacturing system.

References

Q. Liu et al., “Prediction method and prospect of unit consumption of main high energy consuming products based on technical analysis,” Zhongguo Dianli/Electric Power, vol. 56, no. 1, pp. 166–172, 2023, doi: 10.11930/j.issn.1004-9649.202102003.

K. Kaygusuz, “Energy efficiency and renewable energy sources for industrial sector,” in Energy Services Fundamentals and Financing, 2020, pp. 213–238. doi: 10.1016/B978-0-12-820592-1.00009-9.

M. A. Salido, J. Escamilla, F. Barber, A. Giret, D. Tang, and M. Dai, “Energy efficiency, robustness, and makespan optimality in job-shop scheduling problems,” Artif. Intell. Eng. Des. Anal. Manuf., vol. 30, no. 3, pp. 300–312, 2016, doi: 10.1017/S0890060415000335.

M. A. Salido, J. Escamilla, F. Barber, A. Giret, D. B. Tang, and M. Dai, “Energy efficiency, robustness, and makespan optimality in job-shop scheduling problems,” in Sustainable Manufacturing and Remanufacturing Management, 2018, pp. 213–233. doi: 10.1007/978-3-319-73488-0_9.

Y.-L. Chou, J.-M. Yang, and C.-H. Wu, “An energy-aware scheduling algorithm under maximum power consumption constraints,” J. Manuf. Syst., vol. 57, pp. 182–197, 2020, doi: 10.1016/j.jmsy.2020.09.004.

I. Módos, K. Kalodkin, P. Šůcha, and Z. Hanzálek, “Scheduling on dedicated machines with energy consumption limit,” in ICORES 2019 Proceedings, 2019, pp. 53–62. doi: 10.5220/0007307200530062.

L. Wang, Z. Pan, and J. Wang, “A review of reinforcement learning based intelligent optimization for manufacturing scheduling,” Complex Syst. Model. Simul., vol. 1, no. 4, pp. 257–270, 2021, doi: 10.23919/CSMS.2021.0027.

A. Kuhnle, M. C. May, L. Schäfer, and G. Lanza, “Explainable reinforcement learning in production control of job shop manufacturing system,” Int. J. Prod. Res., vol. 60, no. 19, pp. 5812–5834, 2022, doi: 10.1080/00207543.2021.1972179.

S. Wang, J. Li, Q. Jiao, and F. Ma, “Design patterns of deep reinforcement learning models for job shop scheduling problems,” J. Intell. Manuf., vol. 36, no. 6, pp. 3741–3759, 2025, doi: 10.1007/s10845-024-02454-8.

A. S. Rajawat, S. B. Goyal, C. Chauhan, P. Bedi, M. Prasad, and T. Jan, “Cognitive adaptive systems for industrial Internet of Things using reinforcement algorithm,” Electronics, vol. 12, no. 1, p. 217, 2023, doi: 10.3390/electronics12010217.

B. Madaminov, S. Saidmurodov, E. Saitov, D. Jumanazarov, A. M. Alsayah, and L. Zhetkenbay, “Multi-objective optimization framework for energy efficiency and production scheduling in smart manufacturing using reinforcement learning and digital twin technology integration,” Int. J. Ind. Eng. Manag., vol. 16, no. 3, pp. 283–295, 2025, doi: 10.24867/IJIEM-389.

J.-H. Liu, Z.-M. Ke, and W.-H. Zhou, “Reinforcement learning based energy dispatch strategy and control optimization of microgrid,” J. Beijing Univ. Posts Telecommun., vol. 43, no. 1, pp. 28–34, 2020, doi: 10.13190/j.jbupt.2019-040.

Z. Zhao, W. Zheng, W. Yu, C. Luan, X. Jia, and Z. Ma, “Energy efficiency management and automation scheduling strategies in microgrids,” in Proceedings of SPIE, 2025, p. 138111B. doi: 10.1117/12.3085961.

V. Lavanya and M. Saravanan, “Reinforcement learning framework for optimizing trade-offs between energy-aware and makespan workload scheduling in cloud computing,” J. Internet Serv. Inf. Secur., vol. 15, no. 4, pp. 257–270, 2025, doi: 10.58346/JISIS.2025.I4.019.

D. Lei and J. Cai, “Multi-population meta-heuristics for production scheduling: A survey,” Swarm Evol. Comput., vol. 58, p. 100739, 2020, doi: 10.1016/j.swevo.2020.100739.

C. Fiarni, Y. Yonata, and B. Soefian, “Production scheduling heuristics to minimize makespan and tardiness of Indonesian small medium business (SMBs),” in Proceedings of EECSI, 2023. doi: 10.1109/EECSI59885.2023.10295729.

J. Chen, G. Wang, J. Lu, X. Zheng, and D. Kiritsis, “Model-based system engineering supporting production scheduling based on satisfiability modulo theory,” J. Ind. Inf. Integr., vol. 27, p. 100329, 2022, doi: 10.1016/j.jii.2022.100329.

S. I. Ali, “Reinforcement learning for autonomous optimization in intelligent engineering,” in AI-Driven Approaches for Fully Automated Smart Engineering, 2025, pp. 313–344. doi: 10.4018/979-8-3373-4839-1.ch011.

M. Forbes, “Reinforcement learning techniques to support industrial process control,” in TAPPICON 2025, 2025.

M. Li, “The past decade and future of the cross application of Artificial Intelligence and decision support system,” in 2025 IEEE 6th International Seminar on Artificial Intelligence, Networking and Information Technology, AINIT 2025, 2025, pp. 1666–1669. doi: 10.1109/AINIT65432.2025.11036058.

M. Ali, A. Ullah, M. R. Islam, and R. Hossain, “Assessing of software security reliability: Dimensional security assurance techniques,” Comput. Secur., vol. 150, 2025, doi: 10.1016/j.cose.2024.104230.

Y. Yang, W. Cao, L. Guo, C. Gan, and M. Wu, “Reinforcement learning with reward shaping and hybrid exploration in sparse reward scenes,” in IEEE ICPS 2023, 2023. doi: 10.1109/ICPS58381.2023.10128012.

E. Schmidl, E. Fischer, M. Wenk, and J. Franke, “Knowledge-based generation of a plant-specific reinforcement learning framework for energy reduction of production plants,” in IEEE ETFA 2020, 2020. doi: 10.1109/ETFA46521.2020.9211957.

D. Danang, T. Wahyono, I. Sembiring, T. Wellem, and N. H. Dzulkefly, “An Adaptive Framework Integrating ML Blockchain and TEE for Cloud Security,” in 2025 4th International Conference on Creative Communication and Innovative Technology (ICCIT), IEEE, 2025, pp. 1–7.

D. Danang, S. Siswanto, W. Aryani, and P. Wibowo, “Hybrid Federated Ensemble Learning Approach for Real-Time Distributed DDoS Detection in IIoT Edge Computing Environment,” J. Eng. Electr. Informatics, vol. 5, no. 1, pp. 9–17, 2025.

D. Danang, N. D. Setiawan, and E. Siswanto, “Pemanfaatan Teknologi Internet of Things untuk Monitoring Kualitas Air Sungai di Wilayah Perkotaan,” J. New Trends Sci., vol. 2, no. 1, pp. 23–34, 2024.

M. Dai, D. B. Tang, Y. C. Xu, and W. D. Li, “Energy-aware integrated process planning and scheduling for job shops,” in Sustainable manufacturing and remanufacturing management: Process planning, optimization and applications, Springer, 2018, pp. 13–36. doi: 10.1007/978-3-319-73488-0_2.

X. Gong et al., “Energy- and labor-aware production scheduling for sustainable manufacturing: A case study on plastic bottle manufacturing,” Procedia CIRP, vol. 61, pp. 387–392, 2017, doi: 10.1016/j.procir.2016.11.136.

S. Kollmann et al., “Transparent integration of energy awareness into production planning systems,” in IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), 2025. doi: 10.1109/ETFA65518.2025.11205757.

M. De Martino, A. Grassi, G. Guizzi, and V. Popolo, “Optimising energy consumption in reconfigurable manufacturing systems: A mathematical approach,” IFAC-PapersOnLine, vol. 59, no. 10, pp. 1970–1974, 2025, doi: 10.1016/j.ifacol.2025.09.331.

F. G. Nezami, A. Ghazinezami, and K. K. Krishnan, “Sustainable development in manufacturing systems,” in Measuring sustainable development and green investments in contemporary economies, IGI Global, 2017, pp. 201–224. doi: 10.4018/978-1-5225-2081-8.ch008.

Z. Shi, Y. Xie, W. Xue, Y. Chen, L. Fu, and X. Xu, “Smart factory in Industry 4.0,” Syst. Res. Behav. Sci., vol. 37, no. 4, pp. 607–617, 2020, doi: 10.1002/sres.2704.

M. Ryalat, H. ElMoaqet, and M. AlFaouri, “Design of a smart factory based on cyber-physical systems and Internet of Things towards Industry 4.0,” Appl. Sci., vol. 13, no. 4, p. 2156, 2023, doi: 10.3390/app13042156.

N. Teslya and I. Ryabchikov, “Ontology-based semantic models for industrial IoT components representation,” in Advances in Intelligent Systems and Computing, vol. 874, 2019, pp. 138–147. doi: 10.1007/978-3-030-01818-4_14.

M. Nouiri, D. Trentesaux, A. Bekrar, A. Giret, and M. A. Salido, “Cooperation between smart manufacturing scheduling systems and energy providers: A multi-agent perspective,” in Studies in Computational Intelligence, vol. 803, 2019, pp. 197–210. doi: 10.1007/978-3-030-03003-2_15.

D. A. Zakoldaev, A. V Gurjanov, A. V Shukalov, I. O. Zharinov, and O. O. Zharinov, “Multi-agent environment of cyber and physical production for the Industry 4.0 smart factory,” IOP Conf. Ser. Mater. Sci. Eng., vol. 537, no. 3, p. 32045, 2019, doi: 10.1088/1757-899X/537/3/032045.

P. Zorić, M. Musa, and T. M. Kuljanić, “Smart factory environment: Review of security threats and risks,” in Lecture notes of the institute for computer sciences, social-informatics and telecommunications engineering, vol. 382, Springer, 2021, pp. 203–214. doi: 10.1007/978-3-030-78459-1_15.

D. Mourtzis, J. Angelopoulos, and N. Panopoulos, “Industry 4.0 and smart manufacturing,” in Comprehensive Materials Processing, 2nd ed., vol. 3, 2024, p. V3:14--V3:38. doi: 10.1016/B978-0-323-96020-5.00010-8.

V. N. Kulkarni, V. N. Gaitonde, and B. B. Kotturshettar, “Product lifecycle management (PLM): A key enabler in implementation of Industry 4.0,” in Handbook of smart materials, technologies, and devices: Applications of Industry 4.0, Springer, 2022, pp. 349–380. doi: 10.1007/978-3-030-84205-5_14.

O. Iordache, “Industrial systems,” in Studies in systems, decision and control, vol. 92, Springer, 2017, pp. 139–157. doi: 10.1007/978-3-319-52551-8_8.