Pilot Study of Synthetic Data Augmentation for Vision-based Excavator Pose Estimation using Generative Models

Seung Geon Lim; Aram Seo; SangUk Han

doi:10.55785/JCAR.2.2.14

All Issue

2023 Vol.2, Issue 2 Preview Page Next Page

Pilot Study of Synthetic Data Augmentation for Vision-based Excavator Pose Estimation using Generative Models 생성형 모델을 활용한 가상 굴착기 자세 데이터 증강 방법에 대한 선행연구

30 June 2023. pp. 14-20

PDF XML

Abstract

With the advancement of computational power and data utilization techniques, significant research has been undertaken on automation technologies for monitoring construction equipment. Particularly, deep learning and computer vision techniques commonly applied for construction monitoring require a massive amount of onsite data (e.g., site images). To address this issue, this paper presents a novel data generation method that allows for synthesizing images and pose data in a simulation environment and then transforming these images into synthetic images in a real excavator style using a CycleGAN (cycle generative adversarial networks)-based generative model. For validation, experiments were carried out to evaluate the photorealism performance and geometry preservation performance of the proposed model. The results showed that the proposed model achieved the successful generation of synthetic images with style and content information as intended. This study thus may serve as the foundation for exploring a new way to obtain site data more easily, which has been one of the biggest issues for deep learning studies in construction.

Keywords

Deep Learning

Computer Vision

Data Augmentation

Excavator Pose Estimation

최근 건설 분야에서는 연산 능력과 데이터 활용 기술의 발전에 따라 장비 모니터링을 위한 자동화 기술에 대한 관심이 높아지고 있다. 하지만 최근 활발한 연구가 이루어지고 있는 딥러닝과 컴퓨터 비전을 기반으로 한 모니터링 기법의 경우 막대한 양의 현장 데이터가 없다면 건설에 적용 가능한 우수한 성능을 가진 모델을 얻기가 어렵다는 현실적인 문제점을 가지고 있다. 본 논문은 CycleGAN(cycle generative adversarial networks) 기반 생성 모델을 가상의 시뮬레이션 환경에서 추출한 이미지와 자세 데이터에 적용함으로써 가상 이미지를 실제 굴착기 스타일로 변환하는 데이터 생성 방법을 제시한다. 성능 평가를 위한 실험에서 생성 모델의 포토리얼리티 성능은 생성된 이미지에 사전 훈련된 이미지 분류 모델을 적용하여 측정하였고, 컨텐츠 보존 성능은 자세 정보의 변위로 판단하였으며, 그 결과로서 굴착기의 스타일과 컨텐츠 정보를 반영한 이미지를 생성할 수 있음을 확인하였다. 향후 다양한 장비를 대상으로 자세 데이터 보존 성능에 대한 기술 고도화 및 검증이 이루어진다면, 충분한 학습 데이터의 확보를 통해 데이터 기반의 지능형 장비 모니터링 기술이 건설 현장에서 보다 적극적으로 활용될 수 있을 것으로 기대된다.

키워드

딥러닝

컴퓨터 비전

데이터 증강

굴착기 자세 추정

References

Assadzadeh, A., Arashpour, M., Brilakis, I., Ngo, T., and Konstantinou, E. (2022). Vision-based excavator pose estimation using synthetically generated datasets with domain randomization. Automation in Construction, 134, 104089. 10.1016/j.autcon.2021.104089

Kim, H., Kim, H., Hong, Y. W., and Byun, H. (2018). Detecting construction equipment using a region-based fully convolutional network and transfer learning. Journal of computing in Civil Engineering, 32(2), 04017082. 10.1061/(ASCE)CP.1943-5487.0000731

Kim, J., Kim, D., Lee, S., and Chi, S. (2023). Hybrid DNN training using both synthetic and real construction images to overcome training data shortage. Automation in Construction, 149, 104771. 10.1016/j.autcon.2023.104771

Liang, C. J., Lundeen, K. M., McGee, W., Menassa, C. C., Lee, S., and Kamat, V. R. (2019). A vision-based marker-less pose estimation system for articulated construction robots. Automation in Construction, 104, pp. 80-94. 10.1016/j.autcon.2019.04.004

Liu, K., and Golparvar-Fard, M. (2015). Crowdsourcing construction activity analysis from jobsite video streams. Journal of Construction Engineering and Management, 141(11), 04015035. 10.1061/(ASCE)CO.1943-7862.0001010

Liu, S., Zhao, C., Gao, Y., Wang, J., and Tang, M. (2019). Adversarial image generation by combining content and style. IET Image Processing, 13(14), 2716-2723. 10.1049/iet-ipr.2019.0103

Luo, H., Wang, M., Wong, P. K. Y., and Cheng, J. C. (2020). Full body pose estimation of construction equipment using computer vision and deep learning techniques. Automation in construction, 110, 103016. 10.1016/j.autcon.2019.103016

Mahmood, B., Han, S., and Seo, J. (2022). Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images. Automation in Construction, 133, 103996. 10.1016/j.autcon.2021.103996

Memarzadeh, M., Golparvar-Fard, M., and Niebles, J. C. (2013). Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors. Automation in Construction, 32, pp. 24-37. 10.1016/j.autcon.2012.12.002

Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training gans. Advances in neural information processing systems, 29.

Tang, J., Luo, H., Chen, W., Wong, P. K. Y., and Cheng, J. C. (2022). IMU-based full-body pose estimation for construction machines using kinematics modeling. Automation in Construction, 138, 104217. 10.1016/j.autcon.2022.104217

Torres Calderon, W., Roberts, D., and Golparvar-Fard, M. (2021). Synthesizing pose sequences from 3D assets for vision-based activity analysis. Journal of Computing in Civil Engineering, 35(1), 04020052. 10.1061/(ASCE)CP.1943-5487.0000937

Wang, Y., Liao, P. C., Zhang, C., Ren, Y., Sun, X., and Tang, P. (2019). Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety. Advanced Engineering Informatics, 42, 101001. 10.1016/j.aei.2019.101001

Information

Publisher :Korean Society of Automation and Robotics in Construction
Publisher(Ko) :(사)한국건설자동화·로보틱스학회
Journal Title :Journal of Construction Automation and Robotics
Journal Title(Ko) :건설자동화·로보틱스 논문집
Volume : 2
No :2
Pages :14-20
Received Date : 2023-06-10
Revised Date : 2023-06-16
Accepted Date : 2023-06-20
DOI :https://doi.org/10.55785/JCAR.2.2.14

Journal of Construction Automation and Robotics ISSN:2800-0552(Print) 2951-116X(Online) 건설자동화·로보틱스 논문집

All Issue