A cyber-physical system to design 3D models using mixed reality technologies and deep learning for additive manufacturing

PLOS ONE RESEARCH ARTICLE A cyber-physical system to design 3D models using mixed reality technologies and deep learning for additive manufacturing Ammar Malik ID1*, Hugo Lhachemi2, Robert Shorten1,3 1 Department of Electrical and Electronic Engineering, University College Dublin, Dublin, Ireland, 2 L2S, CentraleSupélec, Gif-sur-Yvette, France, 3 Dyson School of Design Engineering, Imperial College London, London, United Kingdom a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Malik A, Lhachemi H, Shorten R (2023) A cyber-physical system to design 3D models using mixed reality technologies and deep learning for additive manufacturing. PLoS ONE 18(7): e0289207. https://doi.org/10.1371/journal. pone.0289207 Editor: Johari Yap Abdullah, Universiti Sains Malaysia, MALAYSIA Received: October 5, 2022 Accepted: July 13, 2023 Published: July 27, 2023 Copyright: © 2023 Malik et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the paper and its Supporting information files. Funding: This publication has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) under grant number 16/RC/3872 and is co-funded under the European Regional Development Fund and by IForm industry partners. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. * Abstract I-nteract is a cyber-physical system that enables real-time interaction with both virtual and real artifacts to design 3D models for additive manufacturing by leveraging mixed-reality technologies. This paper presents novel advances in the development of the interaction platform to generate 3D models using both constructive solid geometry and artificial intelligence. In specific, by taking advantage of the generative capabilities of deep neural networks, the system has been automated to generate 3D models inferred from a single 2D image captured by the user. Furthermore, a novel generative neural architecture, SliceGen, has been proposed and integrated with the system to overcome the limitation of single-type genus 3D model generation imposed by differentiable-rendering-based deep neural architectures. The system also enables the user to adjust the dimensions of the 3D models with respect to their physical workspace. The effectiveness of the system is demonstrated by generating 3D models of furniture (e.g., chairs and tables) and fitting them into the physical space in a mixed reality environment. The presented developmental advances provide a novel and immersive form of interaction to facilitate the inclusion of a consumer into the design process for personal fabrication. Introduction Industry 4.0 is a digital industrial revolution in which numerous emerging technologies are converging to provide digital solutions to achieve mass customisation with increased speed, better quality, and improved productivity [1, 2]. Additive manufacturing (AM), one of the main driving forces in the realisation of this fourth industrial revolution, has emerged during the last decade as a key enabling technology poised to deeply transform manufacturing [3–5]. AM, also known as 3D printing, rapid prototyping, or generative manufacturing, refers to depositing successive thin layers of materials upon each other in precise geometric shapes based on 3D model files to manufacture three-dimensional physical objects [6]. A workflow of AM, depicted in Fig 1, consists of three phases [7]. It starts with the three-dimensional virtual model of the desired product designed via a computer-aided design (CAD) tool or obtained PLOS ONE | https://doi.org/10.1371/journal.pone.0289207 July 27, 2023 1 / 28 PLOS ONE Competing interests: The authors have declared that no competing interests exist. A cyber-physical system to design 3D models for additive manufacturing from 3D scanning in the design phase. Then, during the manufacturing phase, the 3D printer builds the physical object layer upon layer and post-processing is done either to remove support structures or to give the finishing touch to the 3D-printed product. Finally, the manufactured product is inspected for the desired quality and conformance during the testing phase. Therefore, in such a workflow, testing of the designed 3D model for the desired functionality is postponed to the end of the printing process. Hence, the entire loop is reiterated through a trial-error procedure until the desired results are achieved, making the design process costly and time-consuming. Moreover, most CAD design software programs not only require professional training but also restrain the design of 3D virtual models to 2D interfaces, making the design process unintuitive and cumbersome for non-technical consumers and, hence, limiting their involvement in the design phase to facilitate customisation [8, 9]. In this context, innovations in the design of CPS and technological advancements in its supporting tools (IoT, mixed reality, cloud computing, robotics, machine learning) are playing an important role in the widespread adoption of AM by the general public as well as the industry [10]. I-nteract [12] is a CPS that enables the user to interact with both the virtual as well as the physical objects (deformable and non-deformable) simultaneously in a visio-haptic mixed reality (VHMR) environment. The system streamlines the AM process by allowing the user to generate digital twins of the real objects and to test the properties of the designed virtual model in response to human and physical objects stimuli prior to printing. Hence, adding a virtual model testing phase between the design and the manufacturing phase as illustrated in Fig 1. Such innovations in the development of CPS are not only enabling the development of intuitive interfaces for human-machine interactions (human-in-the-loop) [13–15] but also provide innovative monitoring solutions to improve the build quality of the product [16, 17]. With the emergence of Industry 4.0, the horizon of product creation is shifted towards AIenabled human-centred design innovations from merely a physical production perspective [18]. Hence, directing the product design approach towards coordinated product development to achieve customisation and end-user satisfaction enacted through human-centred cyberphysical systems (CPS) [19]. In comparison to traditional (subtractive and formative) manufacturing, AM allows the manufacturing of complex geometries without using traditional dies, molds, milling, and machining which are expensive and time-consuming for mass customization [20]. This advantage over traditional manufacturing makes AM a key enabler in producing moderate to mass quantities of produ (...truncated)