Deep learning-based detection of dental prostheses and restorations

www.nature.com/scientificreports OPEN Deep learning‑based detection of dental prostheses and restorations Toshihito Takahashi1*, Kazunori Nozaki2, Tomoya Gonda1, Tomoaki Mameno1 & Kazunori Ikebe1 The purpose of this study is to develop a method for recognizing dental prostheses and restorations of teeth using a deep learning. A dataset of 1904 oral photographic images of dental arches (maxilla: 1084 images; mandible: 820 images) was used in the study. A deep-learning method to recognize the 11 types of dental prostheses and restorations was developed using TensorFlow and Keras deep learning libraries. After completion of the learning procedure, the average precision of each prosthesis, mean average precision, and mean intersection over union were used to evaluate learning performance. The average precision of each prosthesis varies from 0.59 to 0.93. The mean average precision and mean intersection over union of this system were 0.80 and 0.76, respectively. More than 80% of metallic dental prostheses were detected correctly, but only 60% of tooth-colored prostheses were detected. The results of this study suggest that dental prostheses and restorations that are metallic in color can be recognized and predicted with high accuracy using deep learning; however, those with tooth color are recognized with moderate accuracy. When planning the dental treatment plan for each patient, to grasp the intraoral information is necessary whatever the treatment is. In clinical practices, this procedure is conducted by dentist and it often takes time. Furthermore, the result of this procedure depends on a dentist’s knowledge and experience. Therefore, there is a need for an automated system for grasping the intra-oral situation in the short time. In recent years, artificial intelligence (AI) technologies have been applied in dental medicine, obviating the need for human input in some cases. For instance, AI-based methods and clinical applications have been developed for the diagnosis of dental c aries1,2 and oral c ancer3 with an accuracy that is comparable to that of human beings. These applications have involved the use of deep learning, which is an object detection method that makes predictions given various images of objects. The intraoral information consists of the missing area and occlusion, the shape and location of the residual teeth, restorative or prosthetic situation, periodontal or endodontic situation, risk of caries or periodontal disease, and so on. In addition to above applications, evaluation of p eriodontal4,5 or endodontic situation6 using deep learning were already reported. In our previous report, classification of the missing area was accomplished with a high prediction r ate7. However, evaluation of prosthodontic situation was not conducted yet. In this study, the recognition of prosthodontic situation, that is dental prostheses and the restoration of residual teeth, both types and materials, was performed using deep learning. The aim of this study was to recognize of dental prostheses and restorations using a deep-learning object detection method. Results At least 104 instances of each type of dental prosthesis and restoration were detected in the oral photographic images; the most common type was CMC (2147 instances) and the least common was RFMCBr (104 instances; Fig. 1). The number of prostheses classified as TP and FP are shown Fig. 2. The ratio of TP ranged from 0.67 in CR to 0.96 in CMC (Fig. 3). The APs of each dental prosthesis and restoration are as follows; CMC: 0.93, MIn: 0.92, GCMC: 0.90, GIn: 0.90, RFMCBr: 0.81, RFMC: 0.78, PFMC: 0.78, Br: 0.77, PFMCBr: 0.77, CR: 0.61, and CC: 0.59 (Fig. 4). The mAP and mIoU of this detection system are 0.80 and 0.76, respectively. 1 Department of Prosthodontics, Gerodontology and Oral Rehabilitation, Osaka University Graduate School of Dentistry, 1‑8 Yamadaoka, Suita, Osaka 565‑0871, Japan. 2Division of Medical Information, Osaka University Dental Hospital, 1‑8 Yamadaoka, Suita, Osaka 565‑0871, Japan. *email: toshi‑‑u.ac.jp Scientific Reports | (2021) 11:1960 | https://doi.org/10.1038/s41598-021-81202-x 1 Vol.:(0123456789) www.nature.com/scientificreports/ Figure 1.  Total number of objects of dental prostheses and restorations in all images. CMC: silver-colored complete metal crown, PFMC: porcelain fused to metal crown, CR: composite resin filling, MIn: silver-colored metal inlay restoration, RFMC: resin facing metal crown, CC: ceramic crown, GCMC: gold-colored complete metal crown, GIn: gold-colored metal inlay restoration, Br: fixed partial denture of CMCs, PFMCBr: fixed partial denture of PFMCs, and RFMCBr: fixed partial denture of RFMCs. Figure 2.  Total number of dental prostheses and restorations detected correctly (TP) and those detected as other prostheses (FP). CMC: silver-colored complete metal crown, PFMC: porcelain fused to metal crown, MIn: silver-colored metal inlay restoration, CR: composite resin filling, RFMC: resin facing metal crown, CC: ceramic crown, GCMC: gold-colored complete metal crown, GIn: gold-colored metal inlay restoration, RFMCBr: fixed partial denture of RFMCs, Br: fixed partial denture of CMCs, and PFMCBr: fixed partial denture of PFMCs. Discussion The proposed system will be developed by the computer itself, and basic information about the intraoral condition, like the prosthetic situation of the residual teeth, could be detected by the computer. In this study, the automated system which is identified prosthetic situation of residual teeth and distinguish between sound and restorative teeth from oral images were developed. This system will be a useful tool to make the treatment plans of each patient. On the other hand, further studies focused on caries risk and periodontal and endodontic situation using dental X-ray images will be necessary to grasp all the intraoral information. When evaluating the performance of deep learning-based object detection, two indices, intersection over union (IoU) and mean average precision (mAP) were usually used whereas confusion matrix were generally used. An IoU of more than 0.7 is regarded as good8,9, and the IoU in this study is 0.76. Therefore, the performance of this learning system is high. A mAP is used to measure the accuracy of object detection model and A mAP of more than 0.7 seemed to be regarded as a good value in other s tudies10, but there is no clear criterion. Scientific Reports | Vol:.(1234567890) (2021) 11:1960 | https://doi.org/10.1038/s41598-021-81202-x 2 www.nature.com/scientificreports/ Figure 3.  Ratio of dental prostheses detected correctly (TP) to all detected prostheses. CMC: silver-colored complete metal crown, MIn: silver-colored metal inlay restoration, GCMC: gold-colored complete metal crown, GIn: gold-colored metal inlay restoration, RFMCBr: fixed partial dentures of RFMCs, PFMC: porcelain fused to metal crown, Br: fixed partial denture of CMCs, PFMCBr: fixed partial denture of PFMCs, RFMC: res (...truncated)