All Issue

2024 Vol.3, Issue 2

Research Article

Self-learning of Robot Arm using a Modified Soft-Actor-Critic (SAC) Approach: Complex 6D Pose Brick Grasping Test

수정 SAC 알고리즘을 이용한 산업 로봇팔 자율학습: 자유 상태 6차원 벽돌 그리핑 테스트

Mehrotra Abhishek1
,
Hwang Yi2*
아비세크 메로트라1
,
이 황2*
1Ph. D. Student, Department of Architecture, Korea University
2Corresponding Author, Associate Professor, Department of Architecture, Korea University
1고려대학교 건축학과 박사과정
2교신저자․고려대학교 건축학과 부교수
*Corresponding Author
28 June 2024. pp. 1-10
PDF XML Citation
Abstract

Previous studies have not addressed robot arm grasping for objects with complex 6D poses using reinforcement learning (RL) when the pose is known beforehand. This is crucial for automation in architecture and related fields. To address this gap, we aimed to achieve precise grasping of objects with complex poses. We employed the Soft Actor-Critic (SAC) baseline and its two variants (using median and mean Q values) to demonstrate the limitations of standard RL algorithms and the effectiveness of modified algorithms for this complex task. The trained models were evaluated using statistical metrics, including average episodic rewards, steps, and success rates. Our results indicate that both SAC variants effectively trained the agent to achieve the target 6D pose, with the mean variant performing slightly better than the median variant. Test rollouts demonstrate that the mean variant of the SAC algorithm exhibits superior performance in attaining positions and poses that are more proximate to the target object compared to the median variant.

Keywords
Robot Arm
Modified SAC
Complex Pose
Grasping
Reinforcement Learning
Architecture

산업용 로봇팔은 다양한 크기와 형태를 갖는 건설 재료를 인식, 운반, 설치하는 과정에서 도움을 줄 수 있다. 본 연구는 인공지능 (AI) 강화 학습을 활용한 로봇 자동화를 위하여 복잡한 6D 포즈를 가진 객체를 그리퍼로 정확히 잡는 과정의 알고리즘 성능에 대하여 테스트한 결과와 분석이다. 연구자는 표준강화학습 알고리즘의 넓은 분산과 훈련비용의 한계를 극복하기 위해, 두 가지 변형(중앙값 및 평균 Q 값 사용)을 적용하여 수정된 Soft Actor Critic (SAC)을 개발하고 3차원 벽돌 잡기에 적용하였다. 훈련된 모델은 평균 에피소드 보상, 단계 및 성공률을 포함한 통계 지표를 사용하여 평가하였으며, 실험 결과는 변형SAC 모두 평균 변형이 중앙값 변형보다 향상된 성능을 발휘하며, 목표 6D 포즈를 달성하도록 에이전트를 효과적으로 훈련시킬 수 있음을 입증하였다.

키워드
로봇 암
수정된 SAC
복합 포즈
그립
강화 학습
아키텍처
References
1

Al-Shanoon, A., and Lang, H. (2021). Learn to grasp unknown objects in robotic manipulation. Intelligent Service Robotics, 14(4), pp. 571-582.

10.1007/s11370-021-00380-9
2

Cheng, H., Wang, Y., and Meng, M. Q. H. (2021). Grasp pose detection from a single RGB image, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems. IROS, pp. 4686-4691.

10.1109/IROS51168.2021.9636511
3

Choi, C., Schwarting, W., DelPreto, J., and Rus, D. (2018). Learning object grasping for soft robot hands. IEEE Robotics and Automation Letters, 3(3), pp. 2370-2377.

10.1109/LRA.2018.2810544
4

de Oliveira, D. M., Viturino, C. C. B., and Conceição, A. G. S. (2021). 6D Grasping based on lateral curvatures and geometric primitives. 2021 Latin American Robotics Symposium, LARS, 2021 Brazilian Symposium on Robotics, SBR, and 2021 Workshop on Robotics in Education, WRE, pp. 138-143.

10.1109/LARS/SBR/WRE54079.2021.9605382
5

Deng, X., Xiang, Y., Mousavian, A., Eppner, C., Bretl, T., and Fox, D. (2020). Self-supervised 6D object pose estimation for robot manipulation. 2020 IEEE International Conference on Robotics and Automation, ICRA, pp. 3665-3671.

10.1109/ICRA40945.2020.9196714
6

Fu, H., Mei, X., Zhang, Z., Zhao, W., and Yang, J. (2020). Point pair feature based 6D pose estimation for robotic grasping. 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference, ITNEC, pp. 1803-1808.

10.1109/ITNEC48623.2020.9084720
7

Junare, P., Deshmukh, M., Kulkarni, M., and Bartakke, P. (2022). Deep learning based end-to-end grasping pipeline on a lowcost 5-DOF robotic arm. 2022 IEEE 19th India Council International Conference, INDICON, pp. 1-6.

10.1109/INDICON56171.2022.10040180
8

Li, E., Fu, Y., and Feng, H. (2023). 6D grasp pose detection in cluttered environment from a single-view. 2023 IEEE International Conference on Robotics and Biomimetics, ROBIO, pp. 1-6.

10.1109/ROBIO58561.2023.10354876
9

Li, K., Baron, N., Zhang, X., and Rojas, N. (2022). EfficientGrasp: A unified data-efficient learning to grasp method for multi-fingered robot hands. IEEE Robotics and Automation Letters, 7(4), pp. 8619-8626.

10.1109/LRA.2022.3187875
10

Li, Z., Xu, B., Wu, D., Zhao, K., Lu, M., and Cong, J. (2021). A mobile robotic arm grasping system with autonomous navigation and object detection. 2021 International Conference on Control, Automation and Information Sciences, ICCAIS, pp. 543-548.

10.1109/ICCAIS52680.2021.9624636
11

Liang-shan, Z., Jie, L., Xue-min, S., and Jin-song, B. (2019). Research on multi-stage robotic grasping based on object posture. 2019 WRC Symposium on Advanced Robotics and Automation, WRC SARA, pp. 266-271.

10.1109/WRC-SARA.2019.8931943
12

Liu, P., Zhang, Q., and Cheng, J. (2024). BDR6D: bidirectional deep residual fusion network for 6d pose estimation. IEEE Transactions on Automation Science and Engineering, 21(2), pp. 1793-1804.

10.1109/TASE.2023.3248843
13

Liu, X., Yuan, X., Zhu, Q., Wang, Y., Zhang, H., Feng, M., Wu, Z., and Tang, Y. (2023). A robust pixel-wise prediction network with applications to industrial robotic grasping. IEEE Transactions on Industrial Electronics, 70(8), pp. 8203-8214.

10.1109/TIE.2022.3212422
14

Liu, Y., Xu, H., Liu, D., and Wang, L. (2022). A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping. Robotics and Computer-Integrated Manufacturing, 78, 102365.

10.1016/j.rcim.2022.102365
15

Mohammed, M. Q., Kwek, L. C., Chua, S. C., Aljaloud, A. S., Al‐dhaqm, A., Al‐mekhlafi, Z. G., and Mohammed, B. A. (2021). Deep reinforcement learning‐based robotic grasping in clutter and occlusion. Sustainability, 13(24), 13686.

10.3390/su132413686
16

Park, J., Lee, S., Lee, J., and Um, J. (2020). Gadgetarm-automatic grasp generation and manipulation of 4-dof robot arm for arbitrary objects through reinforcement learning. Sensors, 20(21), pp. 1-16.

10.3390/s2021618333143047PMC7662704
17

Shahid, A. A., Piga, D., Braghin, F., and Roveda, L. (2022). Continuous control actions learning and adaptation for robotic manipulation through reinforcement learning. Autonomous Robots, 46(3), pp. 483-498.

10.1007/s10514-022-10034-z
18

Shukla, P., Kumar, H., and Nandi, G. C. (2020). Robotic grasp manipulation using evolutionary computing and deep reinforcement learning. Intelligent Service Robotics, 14(1), pp. 61-77.

10.1007/s11370-020-00342-7
19

Wang, L., Meng, X., Xiang, Y., and Fox, D. (2022). Hierarchical policies for cluttered-scene grasping with latent plans. IEEE Robotics and Automation Letters, 7(3), pp. 2883-2890.

10.1109/LRA.2022.3143198
20

Wang, S., Liu, J., Lu, Q., Liu, Z., Zeng, Y., Zhang, D., and Chen, B. (2023). 6D Pose estimation for vision-guided robot grasping based on monocular camera. 2023 6th International Conference on Robotics, Control and Automation Engineering, RCAE, pp. 13-17.

10.1109/RCAE59706.2023.10398793
21

Zuo, G., Tong, J., Wang, Z., and Gong, D. (2023). A graph-based deep reinforcement learning approach to grasping fully occluded objects. Cognitive Computation, 15(1), pp. 36-49.

10.1007/s12559-022-10047-x
Information
  • Publisher :Korean Society of Automation and Robotics in Construction
  • Publisher(Ko) :(사)한국건설자동화·로보틱스학회
  • Journal Title :Journal of Construction Automation and Robotics
  • Journal Title(Ko) :건설자동화·로보틱스 논문집
  • Volume : 3
  • No :2
  • Pages :1-10
  • Received Date : 2024-06-07
  • Revised Date : 2024-06-19
  • Accepted Date : 2024-06-20
  • DOI :https://doi.org/10.55785/JCAR.3.2.1
Journal Informaiton Journal of Construction Automation and Robotics Journal of Construction Automation and Robotics
Online Submission
  • crossref crossmark
  • crossref cited-by
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close