Development of an automatic sorting system for fresh ginsengs by image processing techniques

Jeong et al. Hum. Cent. Comput. Inf. Sci. Development of an automatic sorting system for fresh ginsengs by image processing techniques Seokhoon Jeong Yong‑Min Lee Sangjoon Lee This study was conducted with the objective of implementing a smart IoT (internet of things) factory consisting of an automatic 6‑ year‑ old fresh ginseng grade classification device. Conventionally, washed 6‑ year‑ old ginseng from farmlands is manually sorted into three grades using classification criteria such as weight and shape. However, the cost associated with this classification process has been on the increase. Consequently, to reduce this associated cost, we developed an automatic ginseng sorting device that classifies 6‑ year‑ old ginseng according to weight and shape via image processing and sends the classification results to a factory server over a network. Evaluations conducted of the performance of the developed machine using 100 units of 6‑ year‑ old ginseng showed that it has a high recognition rate, with an accuracy of 94% for Grade 1, 98% for Grade 2, and 90% for Grade 3. Agriculture classification; IoT factory; Image analysis Introduction Ginseng and red ginseng both originate from the Republic of Korea. Red ginseng products manufactured and sold by the Korea Tobacco & Ginseng Corporation are acknowledged around the world for their quality and reliability. Red ginseng has been peeled, heated through steaming at standard boiling temperatures of 100 °C (212 °F), and then dried or sun-dried. However, an objective quality classification is required to grade 6-year-old fresh ginseng for the manufacture of red ginseng rather than depending on the human eye or experience [ 1 ]. This grading has caused controversy with cultivators every year. Reportedly, hundreds of millions of dollars and 1530 skilled inspectors are required to grade raw ginseng each year. Automatic classification for some, if not all, 6-year-old fresh ginseng for the manufacture of red ginseng would significantly reduce inspection costs [ 2, 3 ]. To this end, in this study, an automatic 6-year-old ginseng sorting machine was developed. The developed machine sends the 6-year-old ginseng classification results to a factory server that stores the daily classification results, monitors the current classification status, and effectively performs production management. © The Author(s) 2017. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. Materials and methods Design of automatic ginseng sorting machine Figure 1 illustrates the operation of the automatic ginseng sorting machine. Among its components are an inlet conveyor belt as shown in Fig. 1a  and a secondary classification conveyor belt as shown in Fig. 1b. The weight of the incoming ginseng is first estimated by a camera (Fig.  1(1)). Next, images of the ginseng are captured from four different directions via a 4-way camera, as shown in Fig. 1(2). Here, the 4-way camera module is a mechanism for determining whether three-dimensional image analysis is possible and will be used in future studies. The incoming ginseng is sorted via a weight-estimating algorithm and a shape analysis algorithm into Grades 1, 2, and 3, respectively. In addition, the sorting results are displayed on an liquid crystal display (LCD) monitor and transmitted to the factory server through an intranet. The ginseng is transferred from point A to point B at a speed of 0.5 m/s and image analysis execution speed is approximately 0.1  s/sample. Further, the developed machine can sort two to three units per second. Figure 2 shows the actual developed system. The system comprises a control box for conveyor belt control, a display unit for checking the image processing results, and inlet and sorting conveyor belts. Figure 3a shows the ginseng inlet conveyor belt, which is configured in black for efficient image processing. The four sorting actuators that control the module and the location of the classification conveyor belt are as shown in Fig. 3b. The controller of the automatic ginseng sorting machine The main control box provides efficient control of the ginseng sorting machine. Figure 4 is a block diagram of the control system box, which consists of a power controller, two alternative current (AC) induction motor drivers for the conveyor belt, four pneumatic actuator controllers, a ginseng position sensor on the conveyor, and a light controller for the image-estimating camera. The five cameras are connected to a control server via universal serial bus (USB), as shown in Fig. 4(1). On completion of image analysis, the classification information is sent to the factory server and displayed on an LCD unit. Figure  5 shows the control box for the complete ginseng sorting machine. All of the modules shown in Fig. 4 are located inside this control box. Figure 6a is a snapshot of the actual developed main control printed circuit board (PCB) and communicates with the server and receives the rating results to perform the cylinder control function. Figure 6b shows a light controller for controlling the illumination brightness of a LED array, and Fig. 6c shows an array of light emitting diodes (LEDs) installed in a dark room as a light source of a camera. The ginseng weight estimating algorithm Figure  7 illustrates the ginseng weight estimation procedure. First, the original 1920 × 1080 pixel color image (Fig. 7a) is converted to a 256 level (8 bit) or zero (black) to 255 (white) gray level value based on the differences in light and shade to analogize the correlation of the image with the fresh ginseng weight (Fig. 7b). Then, the gray image is filtered to extract the parameter most closely related to the weight information. The number of pixels ranging from zero to 255 is then counted in the converted gray image, and a histogram that shows the distribution developed. Next, the background region is eliminated from the fresh ginseng image and banalization performed to distinguish the body. Here, the binary image is subjected to partial bright lighting against the background to distinguish the fresh ginseng body part (Fig.  7c). Subsequently, block filtering is performed to eliminate the minute lateral roots that have little influence on the weight of the fresh ginseng, and the remaining blocks are used to calculate the parameters (Fig. 7d). In image processing, binarization refers to the changing of all pixels higher than a given threshold to white and pixels below that threshold to black. Various binarization methods exist, including global fixed thresholding, locally adaptive thresholding, and hysteresis thresholding. As a method for determining the threshold value, there is a typical Otsu method, a histogram locating method, Huang and Wang et  al. [ 4–8 ]. In this study, we used global fixed binarization because we used images acquired in the same environment with limited illumination and camera position through the developed ginseng image acquisition device. Figure  8 shows the result of the binarization process according to the set threshold value. For the gray image (Fig. 8a), the histogram of the brightness values of the pixels (Fig. 8b) shows that the binomial distribution is evenly distributed from zero to 255 and that the area between the ginseng and the background cannot be precisely divided. This is a difficult problem because, if the threshold is too low, as in Fig. 8c, the bright areas of the background are highlighted. Conversely, if it is too high, as shown in Fig. 8d, it will cause the black areas of the body to blend into the dark background, causing image loss. It is expected that it will be helpful to reduce errors by applying binarization method after detecting object using color of image [ 9 ]. A block filter divides regions of the 1920 ×  1080 pixel images into blocks of certain sizes. The blocks are white if the distribution of white pixels is higher than a particular threshold (%) and black if it is lower than the threshold. This filter is used to remove fine roots and show the body region that provides much of the weight. Figure 9 shows the observed results when the length × width dimensions of the block filter were increased to 20 × 20, 40 × 40, and 60 × 60 while the threshold was fixed at 60%. The body shape was shown while the fine roots were removed with the 20 ×  20 blocks (Fig. 9a), but losses occurred owing to the staircase phenomenon and resolution deterioration with increasing block size. Thus, it is important to set the proper block size and threshold. Correlation analysis was also performed to investigate the relationship between the weight of the ginseng and the ginseng image pixel value. The correlation between the weight of the ginseng and the number of blocks remaining after the block filter was analyzed using linear regression analysis. Through this process, we examined the optimal binarization threshold and the size of the block filter with the highest correlation coefficient. First, the binarization threshold was changed from 41 to 50, and the value of the correlation coefficient observed to determine the threshold value with the highest correlation coefficient. Next, the change in the correlation coefficient was observed while varying the block size from 11 to 20 pixels (the width and length were the same) and the threshold value from 86 to 95%, respectively. Using the correlation coefficient used in this experiment, the t distribution was tested on both sides with a significance level of 0.05 for the number of hosiery relations. The test for the number of parental relationships of pixel values and weights of ginseng images in the parent group indicated that there is a correlation between the two variables in the population and a strong positive correlation was found as the correlation coefficient was closer to one. Fresh ginseng is not processed after harvest in the field, it contains about 70% of water and it is difficult to store for a long time, so it is recommended to treat ginseng for ginseng production within 1 week. Therefore, the samples applied to the weight estimation algorithm used the samples within 1 week after cultivation to minimize errors. The ginseng shape analysis algorithm Fresh ginseng specimens have unique appearances. However, the names of certain parts are different. Figure 10 classifies the five main parts of fresh ginseng. Figure 10a shows the fibrous root, which is attached to the body or head region and is not present in every fresh ginseng. Figure 10b shows the rhizome, which is present in every fresh ginseng at the head region. Figure 10c shows the body of fresh ginseng, which is called the taproot. This occupies most of the fresh ginseng and is the most important part when manufacturing red ginseng. Figure 10d shows the leg region, which is called the lateral root; each fresh ginseng has about one to five of these roots. Figure 10e shows the fine roots, which absorb the nutrients from the ground. Table  1 presents the classification criteria used for Grades 1, 2, and 3 ginseng provided by the Korea ginseng corporation (KGC) ginseng research Institute. Decision parameters such as the fresh ginseng weight, taproot length, ratio, and number of lateral roots were used, and the fact that experts with long careers classify specimens with rough judgment without accurate measurement tools was examined. In this study, (1) the height of the taproot, (2) width of the taproot, (3) height/width ratio of the taproot, and (4) width/height ratio of the taproot were selected as the grading parameters of fresh ginseng for pattern recognition training and test data. Experiments and results Weight‑estimation results Regarding the threshold in the banalization in the gray image, the block filter settings were fixed (block size: 10  ×  10, threshold: 90%) and the correlation coefficient from the linear regression analysis was observed while changing the banalization threshold between 41 and 50. A threshold of 43 was determined to have the highest correlation (Table 1). Subsequently, the length × width of the block filter and thresh-old in Tables 2 and 3 were fixed to the banalization threshold of 43. Based on increasing the block filter size from 11 × 11 to 20 × 20 pixels and increasing the threshold from 86 to 95% in the linear regression analysis, the highest settings for the correlation coefficient were set to 18 × 18 pixel block size and 90% threshold. The correlation coefficient for the number of output pixels and fresh ginseng weight had a maximum value of 0.9162 when a block filter was applied with a banalization threshold of 43, size of 18  ×  18, and threshold of 90%. The estimated regression was y = 2.517x + 45.82, and analysis of variance was performed on the estimated regression equation to test the significance, as shown Fig.  11. It was considered to be significant with p = 4.4652E−51 based on an F test with a 0.05 significance level and decision coefficient of r2  =  0.839. This means that 83% of the data are explained by the estimated regression equation. Figure  11 shows the relationship between the weight of the fresh ginseng and the results after applying the block filter. In the figure, the causes of singular values such as points a and b are analyzed: 1. When the number of pixels is relatively large compared to the weight of the ginseng (point a in Fig. 11), the ginseng has many fine roots and a dense distribution, which occupies a relatively large volume. 2. When the number of pixels is relatively small compared to the weight of the ginseng (point b in Fig.  11), then the body is dark because of soil or foreign matter in the ginseng, or the root or fine roots of the ginseng cover the body, it takes up a small volume. The result of shape analysis We used a support vector machine (SVM) for shape pattern classification [ 10–12 ]. Figure 13 shows the grade classification method based on pattern recognition. The training data and test data for pattern recognition were designed such that they would not repeat. The correct match rate (%) of each grade, mean correct match rate, false rejection rate (FRR), and false acceptance rate (FAR) indices were used for performance evaluation. Table 2 presents the recognition rate and performance according to the amount of training data when each of the four parameters was used. The recognition rates were 94% for Grade 1, 98% for Grade 2, and 90% for Grade 3. A high recognition performance comprising 6.0% FRR and 5.4% FAR was obtained with four parameters and 10 training data points, as indicated in Table  2. The average recognition rate decreases to 95, 93.3 and 92.2% as training data increases to 10, 15, and 20, respectively. However, the detection rates in the first and second grades are highest at 96.6 and 100%, respectively, when the training data is 20 samples. As the number of samples increases in the training data selection process, data that are difficult to express characteristics of grade 3 will be selected. Conclusions In this study, an automatic 6-year-old ginseng grade classification machine was developed. The developed machine sends the 6-year-old ginseng classification results to a factory server that stores the daily classification results, monitors the current classification status, and effectively performs production management. The ginseng classification is performed using image processing on the basis of criteria such as estimated weight and analyzed shape pattern. The various grades according to the weight of the ginseng are as follows: Grade 3: 30–40 g; Grade 2: 40–75 g; and Grade 1: 75 g and above. In this paper, the proposed algorithm segmented image sector to estimate the weight of a ginseng and experiments have shown that this method is effective for weight estimation. This image segmentation method is used to recognize the postures and gestures of the hand and body for using computer based vision system [ 13, 14 ]. Following the weight-estimation procedure, the group of candidates determined as being in Grade 1 was again evaluated using shape analysis and each candidate assigned a grade between one and three. Evaluations conducted of the performance of the developed machine using 100 units of 6-year-old ginseng showed that it has a high recognition rate, with an accuracy of 94% for Grade 1, 98% for Grade 2, and 90% for Grade 3. We considered several applicable algorithms for ginseng classification, and compared the three algorithms SVM, MLP, and Inception-v3 image recognition model (created by Google) as shown Table  3, which are suitable for classification, and adopted the SVM algorithm with the best performance among them. But SVM does not possible guarantee excellent recognition performance in general pattern recognition situations. Compared to the automatic classification time of imported ginseng, which is within 0.1  s/ sample, there is a limit of the time required to sort the classification by 2–3 pcs/s using pneumatic cylinder. Shortening the overall classification time is a challenge in the future. Authors’ contributions SL is responsible for the concept of the paper and writing, SL and SJ are responsible for the quantitative analysis of the presented results. Also, Y‑ML contributed to the paper on the overall composition and advisory on the experiment. All authors read and approved the final manuscript. Author details 1 Division of Smart Automotive Engineering, SUN MOON University, Chungnam, Asan, Republic of Korea. 2 School of Mechanical and ICT Convergence Engineering, SUN MOON University, Chungnam, Asan, Republic of Korea. Acknowledgements This paper is an extended version of a conference paper (CUTE2016). Competing interests The authors declare that they have no competing interests. Funding This work was supported by the Human Resource Training Program for Regional Innovation and Creativity through the Ministry of Education and National Research Foundation of Korea (NRF‑2014H1C1A1066998) and supported by KGC ginseng research institute. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. 1. Kang JY , Lee MG , Kim YT ( 1996 ) Automatic decision‑making on the grade of 6‑ year‑ old fresh ginseng (Panax ginseng C.A . Meyer) by and image analyzer‑1. Shape and weight analyses according to the grade of fresh ginseng . J Ginseng Res 20 ( 1 ): 65 - 71 2. Jeong CM , Shin JS ( 2006 ) Comparison of grade of raw and red ginseng on each factor of quality in Korean and American ginseng . Korean J Med Crop Sci 14 ( 4 ): 229 - 233 3. Kang JY , Lee MG ( 2002 ) Automatic decision‑making on the grade of 6‑ year‑ old fresh ginseng (Panax ginseng C.A . Meyer) by an Image Analyzer 1. Shape and weight analyses according to the grade of fresh ginseng . J Ginseng Res 26 ( 1 ): 6 - 9 4. Otsu N ( 1979 ) A threshold selection method from gray‑level histograms . IEEE Trans Syst ManCbernet 9 ( 1 ): 62 - 66 5 . Chang CC , Wang LL ( 1997 ) A fast multilevel thresholding method based on lowpass and highpass filtering . Pattern Recognit Lett 18 ( 14 ): 1469 - 1478 6. Cheng HD , Chen YH , Jiang XH ( 2000 ) Thresholding using two‑ dimensional histogram and fuzzy entropy principle . IEEE Trans Image Process 9 ( 4 ): 732 - 735 7. Chi Z , Yan H , Pham T ( 1996 ) Fuzzy algorithms: with applications to image processing and pattern recognition . World Scientific 8. Huang LK , Wang MJ ( 1995 ) Image thresholding b minimizing the measure of fuzziness . Pattern Recogn 28 : 41 - 51 9. Truong MT , Kim S ( 2017 ) Parallel implementation of color‑based particle filter for object tracking in embedded systems . Hum Centric Comput Inf Sci 7 ( 1 ): 2 10. Cortes C , Vapnik V ( 1995 ) Support‑ vector network . Mach Learn 20 : 273 - 297 11. Burges CJC ( 1998 ) A tutorial on support vector machines for pattern recognition . Data Min Knowl Discov 2 ( 2 ): 121 - 167 12. Prabusankarlal KM , Thirumoorthy P , Manavalan R ( 2015 ) Assessment of combined textural and morphological features for diagnosis of breast masses in ultrasound . Hum Centric Comput Inf Sci 1 ( 1 ): 1 - 17 13. Marfia G , Roccetti M ( 2017 ) A practical computer based vision system for posture and movement sensing in occupational medicine . Multimed Tools Appl . 76 ( 6 ): 8109 - 8129 14. Roccetti M , Marfia G , Semeraro A ( 2012 ) Playing into the wild: a gesture‑based interface for gaming in public spaces . J Vis Commun Image Represent 23 ( 3 ): 426 - 440