Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks

Journal of Intelligent Manufacturing https://doi.org/10.1007/s10845-021-01906-9 Improving automated visual fault inspection for semiconductor manufacturing using a hybrid multistage system of deep neural networks Tobias Schlosser1 · Michael Friedrich1 · Frederik Beuth1 · Danny Kowerko1 Received: 20 April 2021 / Accepted: 24 December 2021 © The Author(s) 2022 Abstract In the semiconductor industry, automated visual inspection aims to improve the detection and recognition of manufacturing defects by leveraging the power of artificial intelligence and computer vision systems, enabling manufacturers to profit from an increased yield and reduced manufacturing costs. Previous domain-specific contributions often utilized classical computer vision approaches, whereas more novel systems deploy deep learning based ones. However, a persistent problem in the domain stems from the recognition of very small defect patterns which are often in the size of only a few µm and pixels within vast amounts of high-resolution imagery. While these defect patterns occur on the significantly larger wafer surface, classical machine and deep learning solutions have problems in dealing with the complexity of this challenge. This contribution introduces a novel hybrid multistage system of stacked deep neural networks (SH-DNN) which allows the localization of the finest structures within pixel size via a classical computer vision pipeline, while the classification process is realized by deep neural networks. The proposed system draws the focus over the level of detail from its structures to more task-relevant areas of interest. As the created test environment shows, our SH-DNN-based multistage system surpasses current approaches of learning-based automated visual inspection. The system reaches a performance (F1-score) of up to 99.5%, corresponding to a relative improvement of the system’s fault detection capabilities by 8.6-fold. Moreover, by specifically selecting models for the given manufacturing chain, runtime constraints are satisfied while improving the detection capabilities of currently deployed approaches. Keywords Computer vision · Pattern and image recognition · Deep learning · Semiconductor manufacturing · Factory automation · Fault inspection Introduction and motivation Automated visual fault inspection processes involve the development and integration of systems for capturing and monitoring of manufacturing results. The related manufacturing processes include a multitude of complex processing B Danny Kowerko Tobias Schlosser Michael Friedrich Frederik Beuth 1 Junior Professorship of Media Computing, Chemnitz University of Technology, 09107 Chemnitz, Germany steps, whereas one of these processing steps is concerned with the separation of the resulting components (Huang and Pan 2015). While incorporating the analysis of the used materials as well as the imaging of the investigated circuits, the utilized mechanical forces induced by the laser cutting process characterize the quality of the resulting components. The laser cutting process itself is defined by the prevailing temperature, pressure, and voltage values, whereas information about the amount of flawless chips is ultimately derived from the nature of the resulting components. The aim is therefore to identify potential manufacturing defects within the manufacturing chain, but also to determine influential factors while avoiding complications in subsequent processing steps. For this purpose, an important quality property is calculated based on the ratio of flawless to total chips, also called yield (Lee et al. 2017). Since a manual inspection can imply a considerable and in particular exhaustive time expenditure, 123 Journal of Intelligent Manufacturing Wafer Chips and streets (cuts) Flawless (top) and faulty (bottom) street segments Fig. 1 Wafer overview (left) with chips and streets (middle) as well as flawless and faulty street segments (right) as a result of the wafer dicing process (reprinted from Schlosser et al. (2019)). In order to protect the intellectual property of the wafer imagery, the shown examples are alienated from the original imagery while retaining a close resemblance the automation of quality control allows manufacturers to benefit from an increase in yield and a reduction of manufacturing costs. We approach the problem of automated visual fault detection and recognition in the field of semiconductor manufacturing (Hooper et al. 2015; Rahim and Mian 2017). Silicon wafer dicing denotes the separation of silicon wafers into single components, whereas the dicing streets describe the scribed regions of interest on the wafer surface. One of the more commonly deployed separation approaches utilizes a dicing saw (Hooper et al. 2015). Another, alternative method is thermal laser separation, where a thermally induced mechanical force results in a cleave on the wafer surface (Rahim and Mian 2017). For this purpose, the cleave is guided along the scribe in this thermally induced separation. This separation process constitutes our quality criterion, as a cleave deviating from the scribe results in faulty chips and therefore a decrease in yield. A particular challenge represents the detection and classification of complex defect patterns in pixel size, which often occur in this application area. Figure 1 illustrates this by means of a wafer segment (left), subdivided into chips (middle) and street cuttings (right) as they are produced through the cutting process along the scribes. While the data acquisition of semiconductor manufacturing processes often results in vast amounts of image data, defect patterns often occur in pixel size within image resolutions of up to 105 × 105 pixels. The detection and classification of such small defect patterns is often a problem for deep learning based approaches. Deep neural networks are able to either process the given input in its native image resolution or utilize a downsampled version of the input imagery. However, both options lead to problems in the considered application area. The first case would therefore result in a larger network, possibly leading to slower processing performances, while a higher network connectivity would lead to additional problems such as data overfitting. In the second case, the input imagery would be downsampled for the network, whereas smaller structures within pixel size would be lost. Therefore, the deployed system’s localization and classification capabilities have to be considered in order to allow a more efficient recognition of defect patterns. Depending on the underlying manufacturing process, a variety of defect patterns can occur on the wafer surface, including “spur”, “break out”, “wrong size hole”, “overetch”, “missing hole”, and “excessive short” as well as “cluster pattern”, “edge ring”, “linear scratch”, and “semicircle” based defect patterns (Moganti and Ercal 1998; Cho and Park 2002), also described also by Chen a (...truncated)