All Issue

2022 Vol.1, Issue 2 Preview Page
Deep Reinforcement Learning Methods for Suppressing Horizontal Sway of Construction Cranes in Virtual Physical Environment

가상물리환경에서의 건설 크레인 수평 흔들림 억제를 위한 심층 강화학습 기법

In Won Kim1
,
Nam Kyoun Kim2
,
Min Hyuk Jung3*
,
Chang Bum Ahn4
,
Moon Seo Park5
김 인원1
,
김 남균2
,
정 민혁3*
,
안 창범4
,
박 문서5
1Master’s Student, Department of Architecture and Architectural Engineering, Seoul National University
2Ph. D. Student, Department of Architecture and Architectural Engineering, Seoul National University
3Research Professor, Department of Architecture and Architectural Engineering, Seoul National University
4Professor, Department of Architecture and Architectural Engineering, Seoul National University
5Professor, Department of Architecture and Architectural Engineering, Seoul National University
1서울대학교 대학원 건축학과 석사과정
2서울대학교 대학원 건축학과 박사과정
3서울대학교 건축학과 연구교수
4서울대학교 건축학과 교수
5서울대학교 건축학과 교수
*Corresponding Author
8 July 2022. pp. 19-24
PDF XML Citation
Abstract

In the development of a deep reinforcement learning-based autonomous operation model of cranes, the control of heavy object’s horizontal sway is an important issue that directly affects crane operation safety. In kinematics, however, the motion control of a heavy object with pendulum motion is classified as an underactuated system in which the degree of freedom of movement of objects is larger than the number of manipulable actions of controllers. This increases the variance of rewards expected from action and state samples in reinforcement learning, and raises the problem of sample efficiency, which means the number of samples effective for learning. Therefore, this study analyzes the sample efficiency that occurs when learning the reinforcement learning model for sway control of cranes using Proximal Policy Optimization and Generative Adversarial Imitation Learning (GAIL) techniques. To this end, this study established a virtual physical environment capable of simulating the movement of a construction crane, and expert demonstration data samples were collected for GAIL. Finally, the effect of PPO and GAIL on sample efficiency was analyzed through the experiment. The results show that the reinforcement learning technique applied to the experiment is effective in improving the sample efficiency and learning performance of the crane model.

Keywords
Autonomous Crane
Deep Reinforcement Learning
Sample Efficiency
Generative and Adversarial Imitation Learning

심층강화학습을 이용한 건설 크레인의 자율 운행 기술 개발에 있어 중량물의 수평 흔들림 제어 문제는 크레인 운행 안전성에 직접적으로 영향을 미치는 중요한 문제이다. 하지만 진자운동을 하는 중량물의 움직임 제어는 운동학적 측면에서 크레인의 조작 가능한 동작의 수보다 중량물 움직임의 자유도가 더 큰 과소작동 시스템으로 분류되며, 이것은 강화학습에서 행동과 상태 표본쌍으로부터 기대할 수 있는 보상의 분산을 증가시킴으로써, 표본 효율성이 저하하는 문제를 발생시킨다. 따라서 본 연구에서는 근위 정책 최적화(Proximal Policy Optimization)와 생성적 적대 모방 학습(Generative Adversarial Imitation Learning) 기법을 이용하여 크레인의 수평 흔들림 제어를 위한 강화학습 모형의 학습시 발생하는 표본 효율성에 대해서 분석한다. 이를 위해 Unity를 활용해 건설 크레인의 움직임과 작업수행의 시뮬레이션이 가능한 가상물리환경을 구축하고, GAIL 적용을 위한 전문가 시연 데이터 표본을 수집하였으며, 실험을 통해 GAIL이 표본 효율성 저하에 미치는 영향을 분석하였다. 본 연구의 결과는 GAIL이 표본 효율성을 크게 개선할 수 있는 것을 보여준다. 하지만, GAIL의 내적보상 강도에 따라 표본 효율성의 개선정도가 다양한 양상으로 나타날 수 있음이 확인되었다.

키워드
건설 크레인
심층강화학습
샘플효율성
생성적 적대 모방학습
References
1
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., and Hassabis, D. (2019). Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23(5), pp. 408-422. 10.1016/j.tics.2019.02.00631003893
2
Bhattacharyya, R., Wulfe, B., Phillips, D., Kuefler, A., Morton, J., Senanayake, R., and Kochenderfer, M. (2020). Modeling human driving behavior through generative adversarial imitation learning. arXiv preprint arXiv:2006.06412.
3
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017). CARLA: An open urban driving simulator. In Conference on Robot Learning, (pp. 1-16), PMLR.
4
Fang, Y., and Cho, Y. K. (2017). Effectiveness analysis from a cognitive perspective for a real-time safety assistance system for mobile crane lifting operations. Journal of Construction Engineering and Management, 143(4), 05016025. 10.1061/(ASCE)CO.1943-7862.0001258
5
Ho, J., and Ermon, S. (2016). Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29.
6
Juliani, A., Berges, V. P., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., Mattar, M., and Lange, D. (2018). Unity: A general platform for intelligent agents. arXiv preprint arXiv:1809.02627.
7
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A. A., Yogamani, S., and Pérez, P. (2021). Deep reinforcement learning for autonomous driving: A survey, IEEE Transactions on Intelligent Transportation Systems. 10.1109/TITS.2021.3054625
8
Kober, J., and Peters, J. (2010). Imitation and reinforcement learning. IEEE Robotics & Automation Magazine, 17(2), pp. 55-62. 10.1109/MRA.2010.936952
9
Matsumoto, K., Yamaguchi, A., Oka, T., Yasumoto, M., Hara, S., Iida, M., and Teichmann, M. (2020). Simulation-based Reinforcement Learning Approach towards Construction Machine Automation, In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, IAARC Publications, 37, pp. 457-464. 10.22260/ISARC2020/0064
10
Ramli, L., Mohamed, Z., Abdullahi, A. M., Jaafar, H. I., and Lazim, I. M. (2017). Control strategies for crane systems: A comprehensive review. Mechanical Systems and Signal Processing, 95, pp. 1-23. 10.1016/j.ymssp.2017.03.015
11
Rong, G., Shin, B.H., Tabatabaee, H., Lu, Q., Lemke, S., Možeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., and Agafonov, E. (2020). Lgsvl simulator: A high fidelity simulator for autonomous driving. In 2020 IEEE 23rd International conference on intelligent transportation systems (ITSC) (pp. 1-6), IEEE. 10.1109/ITSC45102.2020.9294422
12
Sallab, A. E., Abdou, M., Perot, E., and Yogamani, S. (2017). Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 19, pp. 70-76. 10.2352/ISSN.2470-1173.2017.19.AVM-023
13
Savva, M., Chang, A. X., Dosovitskiy, A., Funkhouser, T., and Koltun, V. (2017). MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv preprint arXiv:1712.03931.
14
Sawodny, O., Aschemann, H., and Lahres, S. (2002) An automated gantry crane as a large workspace robot. Control Engineering Practice, 10(12), pp. 1323-1338. 10.1016/S0967-0661(02)00097-7
15
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
16
Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2018). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics (pp. 621-635), Springer, Cham. 10.1007/978-3-319-67361-5_40
17
Wu, Z., and Xia, X. (2014). Optimal motion planning for overhead cranes. IET Control Theory & Applications, 8(17), pp. 1833-1842. 10.1049/iet-cta.2014.0069
18
Yang, R., Jiang, C., Miao, Y., Ma, J., Zhang, X., Yang, T., and Sun, N. (2019). A flexible rope crane experiment system. Applications of Modeling and Simulation, 3(1), pp. 11-17.
19
Yu, Y. (2018). Towards Sample Efficient Reinforcement Learning. In IJCAI (pp. 5739-5743). 10.24963/ijcai.2018/820
20
Zhao, W., Queralta, J. P., and Westerlund, T. (2020). Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 737-744), IEEE.24. 10.1109/SSCI47803.2020.9308468
Information
  • Publisher :Korean Society of Automation and Robotics in Construction
  • Publisher(Ko) :(사)한국건설자동화·로보틱스학회
  • Journal Title :Journal of Construction Automation and Robotics
  • Journal Title(Ko) :건설자동화·로보틱스 논문집
  • Volume : 1
  • No :2
  • Pages :19-24
  • Received Date : 2022-06-27
  • Revised Date : 2022-07-08
  • Accepted Date : 2022-07-08
  • DOI :https://doi.org/10.55785/JCAR.1.2.19
Journal Informaiton Journal of Construction Automation and Robotics Journal of Construction Automation and Robotics
Online Submission
  • crossref crossmark
  • crossref cited-by
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close