Automated 3D Reconstruction of Mechanical Components Using Multimodal Deep Learning

Mechanical, Production and Energy Laboratory (LMPE), National School of Engineering of Tunis (ENSIT), University of Tunis, 1008, Avenue Taha Hussein, Montfleury, Tunisia

^* e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^*** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

This paper presents a multimodal deep learning approach for precise 3D reconstruction of mechanical parts from images and textual descriptions, offering a cost-effective alternative to traditional methods. By combining high-resolution, multi-angle images with technical text data, the model generates accurate 3D representations. A ResNet-based CNN extracts visual features, while BERT encodes textual descriptions, with a depth estimation module enhancing spatial accuracy. The features are fused to produce a 3D point cloud and mesh. The results demonstrate good performance in capturing the overall shape of the mechanical parts; however, further improvements are needed to enhance the precision of the metric parameters.