All Issue

2022 Vol.1, Issue 2
Automated Virtual Image Synthetization for DNN-Powered Visual Scene Understanding in Construction

심층신경망 기반 건설현장 시각적 장면 이해를 위한 가상 학습 이미지 자동 생성

Jinwoo Kim1*
김 진우1*
1Assistant Professor, School of Civil and Environmental Engineering, Nanyang Technological University
1Nanyang Technological University 건설환경공학과 조교수
*Corresponding Author
8 July 2022. pp. 1-6
PDF XML Citation
Abstract

Although deep neural networks(DNNs) have become a key driver of visual scene understanding, a shortage of construction training images holds DNNs back from achieving its maximum performance potential. To unlock its full potential, this research proposes an automated virtual image synthetization framework and examines the effectiveness of synthetic images as DNN training data. This research particularly focuses on a common real-world application scenario wherein there is only a small, biased real training image dataset available. To this end, this research (1) builds a large amount of synthetic image dataset using the proposed framework; (2) integrates the synthetized images into a small, biased real image dataset; and (3) trains a DNN model with the integrated dataset and evaluates its performance in diverse field conditions. Results show that the integrated dataset-trained model always yields a higher performance than the one trained with only a small, biased real dataset. This finding suggests that synthetic construction images have encouraging potential in enhancing a given real training image dataset’s quantity and visual scene diversity. It is also observed that the integration of real and synthetic images can improve the DNN performance by 51.2% while reducing the number of real training instances by 75%(from 2,000 to 500). This finding is more promising granted that the proposed framework can automatically generate and label synthetic construction images without manual inputs. This research can eventually help address the shortage of construction training images and develop a more accurate and scalable DNN model for visual scene understanding.

Keywords
Construction
Visual Scene Understanding
Synthetic Image
Deep Neural Network(DNN)
Object Detection

심층신경망(Deep Neural Network, DNN)이 시각적 장면 이해의 핵심요소가 되었음에도 불구하고 건설분야에서 학습 이미지 데이터의 부족으로 DNN의 잠재력을 최대화하는 데 한계가 있다. 이를 해결하기 위해서 본 연구는 가상의 건설 이미지 자동 생성 프레임워크를 제안하고, 가상 이미지가 DNN 학습 데이터로 사용될 수 있는지 그 잠재력을 평가한다. 특히, 실제 건설현장에서 수집된 학습 이미지 데이터의 개수가 적고 그 시각적 다양성이 한정된, 일반적인 실무 시나리오에서 DNN 모델 학습 및 테스트 실험을 수행한다. 보다 구체적으로, 본 연구는 (1) 대량의 가상 이미지 데이터를 생성하고 (2) 작고 편향된 실제 데이터셋에 가상 이미지를 추가한 뒤 (3) DNN 모델을 학습하고 실제 건설현장에서 그 성능을 평가한다. 실험을 수행한 결과, 실제 데이터와 가상 데이터를 모두 학습한 모델이 실제 데이터만 학습한 모델보다 항상 더 좋은 성능을 보이는 것으로 나타났다. 이는 생성된 가상 학습 이미지가 실제 이미지 데이터의 한정된 양과 시각적 다양성을 효과적으로 보완 ․ 강화할 수 있음을 시사한다. 또한, 실제 데이터와 가상 데이터를 혼합하여 사용한 경우 실제 데이터 개수를 75%만큼 줄이고 DNN 모델의 성능을 51.2% 향상할 수 있었다. 제안된 프레임워크가 건설 이미지를 완전히 자동으로 생성하고 라벨링할 수 있다는 점을 고려하면 이 결과는 더욱 고무적이다. 이 같은 연구결과는 건설분야에서 학습 이미지 데이터가 부족한 문제를 해결하고 실무 적용성이 높은 DNN 모델을 개발하는 데 기여할 것이다.

키워드
건설
시각적 장면 이해
가상 이미지
심층신경망
물체 인식
References
1
Assadzadeh, A., Arashpour, M., Brilakis, I., Ngo, T., and Konstantinou, E. (2022). Vision-based excavator pose estimation using synthetically generated datasets with domain randomization. Automation in Construction, 134, 104089. 10.1016/j.autcon.2021.104089
2
Braun, A., and Borrmann, A. (2019). Combining inverse photogrammetry and BIM for automated labeling of construction site images for machine learning. Automation in Construction, 106, 102879. 10.1016/j.autcon.2019.102879
3
Hong, Y., Park, S., Kim, H., and Kim, H. (2021). Synthetic data generation using building information models. Automation in Construction, 130, 103871. 10.1016/j.autcon.2021.103871
4
Hwang, J., Kim, J., Chi, S., and Seo, J. (2022). Development of training image DB using web crawling for construction site monitoring. Automation in Construction, 135, 104141. 10.1016/j.autcon.2022.104141
5
Kim, D., Liu, M., Lee, S., and Kamat, V. R. (2019). Remote proximity monitoring between mobile construction resources using camera-mounted UAVs. Automation in Construction, 99, pp. 168-182. 10.1016/j.autcon.2018.12.014
6
Kim, J., and Chi, S. (2021). A few-shot learning approach for database-free vision-based monitoring on construction sites. Automation in Construction, 124, 103566. 10.1016/j.autcon.2021.103566
7
Kim, J., Chi, S., and Ahn, C. R. (2021). Hybrid kinematic-visual sensing approach for activity recognition of construction equipment. Journal of Building Engineering, 44, 102709. 10.1016/j.jobe.2021.102709
8
Kim, J., Hwang, J., Chi, S., and Seo, J. (2020). Towards database-free vision-based monitoring on construction sites: A deep active learning approach. Automation in Construction, 120, 103376. 10.1016/j.autcon.2020.103376
9
Kim, Y., Kim, H., Murphy, R., Lee, S., and Ahn, C. R. (2022). Delegation or collaboration: Understanding different construction stakeholders' perceptions of robotization. Journal of Management in Engineering, 38(1), 04021084. 10.1061/(ASCE)ME.1943-5479.0000994
10
Liang, C.-J., Wang, X., Kamat, V. R., and Menassa, C. C. (2021). Human-robot collaboration in construction: Classification and research trends. Journal of Construction Engineering and Management, 147(10), 03121006. 10.1061/(ASCE)CO.1943-7862.0002154
11
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. J. (2015). SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 34(6), pp. 1-16. 10.1145/2816795.2818013
12
Mahmood, B., Han, S. U., and Seo, J. (2022). Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images. Automation in Construction, 133, 103996. 10.1016/j.autcon.2021.103996
13
Siu, C., Wang, M., and Cheng, J. C. P. (2022). A framework for synthetic image generation and augmentation for improving automatic sewer pipe defect detection. Automation in Construction, 137, 104213. 10.1016/j.autcon.2022.104213
14
Soltani, M., Zhu, Z., and Hammad, A. (2016). Automated annotation for visual recognition of construction resources using synthetic images. Automation in Construction, 62, pp. 14-23. 10.1016/j.autcon.2015.10.002
15
Torres Calderon, W., Roberts, D., and Golparvar-Fard, M. (2021). Synthesizing pose sequences from 3D assets for vision-based activity analysis. Journal of Computing in Civil Engineering, 35(1), 04020052. 10.1061/(ASCE)CP.1943-5487.0000937
16
Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M. J., Laptev, I., and Schmid, C. (2017). Learning from synthetic humans. Proc. of 2017 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Honolulu, HI, USA, pp. 109-117. 10.1109/CVPR.2017.492
Information
  • Publisher :Korean Society of Automation and Robotics in Construction
  • Publisher(Ko) :(사)한국건설자동화·로보틱스학회
  • Journal Title :Journal of Construction Automation and Robotics
  • Journal Title(Ko) :건설자동화·로보틱스 논문집
  • Volume : 1
  • No :2
  • Pages :1-6
  • Received Date : 2022-06-20
  • Revised Date : 2022-07-06
  • Accepted Date : 2022-07-06
  • DOI :https://doi.org/10.55785/JCAR.1.2.1
Journal Informaiton Journal of Construction Automation and Robotics Journal of Construction Automation and Robotics
Online Submission
  • crossref crossmark
  • crossref cited-by
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close