Hybrid Reinforcement Learning and Robust Adaptive Control Strategy for Autonomous Manufacturing Systems under Uncertain and Dynamic Production Environments

Irlon Irlon; Teguh Muryanto; Sayyid Jamal Al Din; Dwi Utari Iswavigra; Yulaikha Maratullatifah; Very Dwi Setiawan

doi:10.61132/ijmicse.v1i1.403

Authors

Irlon Irlon Institut Teknologi Budi Utomo
Teguh Muryanto Institut Teknologi Budi Utomo
Sayyid Jamal Al Din Institut Teknologi Budi Utomo
Dwi Utari Iswavigra Universitas Sugeng Hartono
Yulaikha Maratullatifah Universitas Sugeng Hartono
Very Dwi Setiawan Universitas Pignatelli Triputra

DOI:

https://doi.org/10.61132/ijmicse.v1i1.403

Keywords:

Autonomous Manufacturing, Hybrid Control, Process Optimization, Reinforcement Learning, System Stability

Abstract

This study explores the integration of hybrid AI control models, combining reinforcement learning (RL) and robust adaptive control, to improve the adaptability, performance, and stability of autonomous manufacturing systems. Traditional control systems, while effective under stable conditions, often struggle to cope with disturbances and varying production demands. Hybrid AI models, which integrate classical control methods such as Proportional Integral Derivative (PID) with machine learning techniques like RL, deep Q-networks (DQN), and deep deterministic policy gradient (DDPG), enhance decision-making capabilities in dynamic production environments. The study develops a hybrid RL robust control framework and tests it in both simulation and real-world scenarios. Performance metrics, including production efficiency, system stability, and adaptability, are assessed under various disturbance conditions, such as machine failures and fluctuating demands. The hybrid model significantly outperforms traditional PID control in terms of efficiency and stability, demonstrating faster convergence and better adaptability in dynamic environments. Statistical analysis confirms the superiority of the hybrid system over standalone RL models and traditional PID control. This model’s scalability and adaptability make it a promising solution for Industry 4.0 applications, addressing key challenges in real-world manufacturing systems by ensuring computational efficiency and the ability to manage large-scale data. The findings contribute to the development of more robust and efficient control strategies for autonomous manufacturing systems in uncertain environments.

References

Amine, M., & Mohamed, Z. (2024). Mixed reinforcement learning and sliding mode controller design for robot application. 10th 2024 International Conference on Control, Decision and Information Technologies, CoDIT 2024, 2395–2400. https://doi.org/10.1109/CoDIT62066.2024.10708572

Boschi, F., Tavola, G., Taisch, M., Gepp, M., Foehr, M., & Colombo, A. W. (2019). PERFoRM: Industrial context and project vision. In Digitalized and harmonized industrial production systems: The PERFoRM approach. https://doi.org/10.1201/9780429263316-1

Desai, K., Sharma, G. S., & Pawar, R. (2024). Understanding the role of machine learning in optimizing manufacturing processes and quality control. 2024 International Conference on Advances in Computing Research on Science Engineering and Technology, ACROSET 2024. https://doi.org/10.1109/ACROSET62108.2024.10743916

Fridman, L., Poznyak, A., & Bejarano, F. J. (2014). Introduction. In Systems and control: Foundations and applications (pp. 1–8). https://doi.org/10.1007/978-0-8176-4962-3_1

Gaham, M., Bouzouia, B., & Achour, N. (2014). An evolutionary simulation-optimization approach to product-driven manufacturing control. Studies in Computational Intelligence, 544, 283–294. https://doi.org/10.1007/978-3-319-04735-5_19

Hassan, A., Triki, H., Trabelsi, H., & Haddar, M. (2024). Literature review of scheduling problems using artificial intelligence technologies based on machine learning. Lecture Notes in Mechanical Engineering, 341–348. https://doi.org/10.1007/978-3-031-67152-4_36

Heik, D., Bahrpeyma, F., & Reichelt, D. (2024). Solving a dynamic scheduling problem for a manufacturing system with reinforcement learning. Lecture Notes in Networks and Systems, 823 LNNS, 413–432. https://doi.org/10.1007/978-3-031-47724-9_28

Hoppe, S., Giftthaler, M., Krug, R., & Toussaint, M. (2020). Sample-efficient learning for industrial assembly using qgraph-bounded DDPG. IEEE International Conference on Intelligent Robots and Systems, 9080–9087. https://doi.org/10.1109/IROS45743.2020.9341390

Hoppe, S., Giftthaler, M., Krug, R., & Toussaint, M. (2023). Stabilizing deep Q-learning with Q-graph-based bounds. International Journal of Robotics Research, 42(9), 633–654. https://doi.org/10.1177/02783649231185165

Jeung, E. T., & Park, H. B. (2014). A survey of robust control in both frequency domain and time domain. Journal of Institute of Control, Robotics and Systems, 20(3), 270–276. https://doi.org/10.5302/J.ICROS.2014.14.9014

Knights, V. A., Petrovska, O., & Kljusurić, J. G. (2024). Nonlinear dynamics and machine learning for robotic control systems in IoT applications. Future Internet, 16(12). https://doi.org/10.3390/fi16120435

Kolhe, K., Somatkar, A. A., Bhandarkar, M. S., Kotangale, K. B., Ayane, S. S., & Shirke, S. I. (2023). Applications and challenges of machine learning techniques for smart manufacturing in Industry 4.0. 2023 7th International Conference On Computing, Communication, Control And Automation, ICCUBEA 2023. https://doi.org/10.1109/ICCUBEA58933.2023.10392071

Kong, X.-Y., Cao, Z.-H., Du, B.-Y., & Luo, J.-Y. (2019). Quality-related multimodal fault detection technique based on partial least squares. Kongzhi Yu Juece/Control and Decision, 34(12), 2547–2557. https://doi.org/10.13195/j.kzyjc.2018.0282

Le, N. H., Huynh, V. Y., Kieu, M. K., Debnath, N. C., & Le, N. B. (2023). A smart manufacturing system through integration of advanced technologies. Smart Innovation, Systems and Technologies, 394 SIST, 485–496. https://doi.org/10.1007/978-981-97-3980-6_42

Lei, J., Hui, J., Ding, K., & Wu, L. (2021). A framework for planning and scheduling shop floor logistics via cloud-edge collaboration. Journal of Physics: Conference Series, 1983(1). https://doi.org/10.1088/1742-6596/1983/1/012109

Liang, S., Rajora, M., Liu, X., Yue, C., Zou, P., & Wang, L. (2018). Intelligent manufacturing systems: A review. International Journal of Mechanical Engineering and Robotics Research, 7(3), 324–330. https://doi.org/10.18178/ijmerr.7.3.324-330

Lynch, L., McGuinness, F., Clifford, J., Rao, M., Walsh, J., Toal, D., & Newe, T. (2019). Integration of autonomous intelligent vehicles into manufacturing environments: Challenges. Procedia Manufacturing, 38, 1683–1690. https://doi.org/10.1016/j.promfg.2020.01.115

Ma, Q., Yin, X., Zhang, X., Xu, X., & Yao, X. (2024). Game-theoretic receding-horizon reinforcement learning for lateral control of autonomous vehicles. IEEE Transactions on Vehicular Technology, 73(10), 14547–14562. https://doi.org/10.1109/TVT.2024.3412530

Mesbah, A., Wabersich, K. P., Schoellig, A. P., Zeilinger, M. N., Lucia, S., Badgwell, T. A., & Paulson, J. A. (2022). Fusion of machine learning and MPC under uncertainty: What advances are on the horizon? Proceedings of the American Control Conference, 2022-June, 342–357. https://doi.org/10.23919/ACC53348.2022.9867643

Sahoo, S., & Lo, C.-Y. (2022). Smart manufacturing powered by recent technological advancements: A review. Journal of Manufacturing Systems, 64, 236–250. https://doi.org/10.1016/j.jmsy.2022.06.008

Sathya, D., Saravanan, G., & Thangamani, R. (2024). Reinforcement learning for adaptive mechatronics systems. In Computational intelligent techniques in mechatronics. https://doi.org/10.1002/9781394175437.ch5

Sevic, M., & Keller, P. (2021). Model of smart factory using the principles of industry 4.0. MM Science Journal, 2021(March), 4238–4243. https://doi.org/10.17973/MMSJ.2021_03_2020067

Shahnooshi, S., Ranjbaran, P., Ebrahimi, J., Bakhshai, A., & Jain, P. (2023). Reinforcement learning-based control of a buck converter: A comparative study of DQN and DDPG algorithms. 2023 25th European Conference on Power Electronics and Applications, EPE 2023 ECCE Europe. https://doi.org/10.23919/EPE23ECCEEurope58414.2023.10264535

Shalini, T. G., Ganeshabu, L., Chemudugunta, P., Yamsani, N., Kumar, C. A., & Athiraja, A. (2024). Exploring innovations in autonomous robotics and mechatronics: Application in manufacturing and healthcare. Proceedings of the 5th International Conference on Smart Electronics and Communication, ICOSEC 2024, 336–342. https://doi.org/10.1109/ICOSEC61587.2024.10722417

Shoaib-ul-Hasan, S., Macchi, M., Pozzetti, A., & Carrasco-Gallego, R. (2018). A routine-based framework implementing workload control to address recurring disturbances. Production Planning and Control, 29(11), 943–957. https://doi.org/10.1080/09537287.2018.1494344

Singh, A., Jadhav, A., & Singh, P. (2024). AI applications in production. In Industry 4.0, smart manufacturing, and industrial engineering: Challenges and opportunities. https://doi.org/10.1201/9781003473886-7

Swathi, B., Polyakov, S. V., Kandavalli, S. R., Singh, D. K., Murthy, M. Y. B., & Gopi, A. (2024). Enhancing hybrid manufacturing with AI-driven real-time adaptive process control: Integrating machine learning models and robotic systems. International Journal of Advanced Manufacturing Technology. https://doi.org/10.1007/s00170-024-14155-w

Tan, H. (2021). Reinforcement learning with deep deterministic policy gradient. Proceedings - 2021 International Conference on Artificial Intelligence, Big Data and Algorithms, CAIBDA 2021, 82–85. https://doi.org/10.1109/CAIBDA53561.2021.00025

Tyagi, A. K., & Richa, R. (2023). Smart manufacturing using internet of things, artificial intelligence, and digital twin technology. In Global perspectives on robotics and autonomous systems: Development and applications. https://doi.org/10.4018/978-1-6684-7791-5.ch008

Vadlamudi, S., & Lakshmi, K. V. (2024). Reinforcement learning for optimal robotic arm control in ball-balancing tasks. Proceedings of International Conference on Communication, Computer Sciences and Engineering, IC3SE 2024, 1324–1328. https://doi.org/10.1109/IC3SE62002.2024.10593154

Vuga, R., Nemec, B., & Ude, A. (2015). Enhanced policy adaptation through directed explorative learning. International Journal of Humanoid Robotics, 12(3). https://doi.org/10.1142/S0219843615500280

Wagner, M., Sousa, F. J. P., Glatt, M., & Aurich, J. C. (2024). Bridging the gap: A conceptual framework for developing and operating hybrid modeled digital twins under limited model input conditions. Procedia CIRP, 121, 31–36. https://doi.org/10.1016/j.procir.2023.09.226

Wang, D., Mu, C., He, H., & Liu, D. (2017). Event-driven adaptive robust control of nonlinear systems with uncertainties through NDP strategy. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(7), 1358–1370. https://doi.org/10.1109/TSMC.2016.2592682

Xia, K., Sacco, C., Kirkpatrick, M., Saidy, C., Nguyen, L., Kircaliali, A., & Harik, R. (2021). A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence. Journal of Manufacturing Systems, 58, 210–230. https://doi.org/10.1016/j.jmsy.2020.06.012

Yang, Y., & Liu, Y. (2024). Adaptive optimization control strategy for intelligent manufacturing systems based on deep reinforcement learning. Proceedings - 2024 International Conference on Interactive Intelligent Systems and Techniques, IIST 2024, 305–309. https://doi.org/10.1109/IIST62526.2024.00044

Zhang, Y., Chang, G., Su, P., & Zou, X. (2024). Research on robust control strategies for diesel engines based on H∞ theory. Proceedings of SPIE - The International Society for Optical Engineering, 13395. https://doi.org/10.1117/12.3049930