Fisheye freshness detection using common deep learning algorithms and machine learning methods with a developed mobile application

European Food Research and Technology https://doi.org/10.1007/s00217-024-04493-0 ORIGINAL PAPER Fisheye freshness detection using common deep learning algorithms and machine learning methods with a developed mobile application Muslume Beyza Yildiz1 · Elham Tahsin Yasin2 · Murat Koklu1 Received: 17 December 2023 / Revised: 26 January 2024 / Accepted: 3 February 2024 © The Author(s) 2024 Abstract Fish is commonly ingested as a source of protein and essential nutrients for humans. To fully benefit from the proteins and substances in fish it is crucial to ensure its freshness. If fish is stored for an extended period, its freshness deteriorates. Determining the freshness of fish can be done by examining its eyes, smell, skin, and gills. In this study, artificial intelligence techniques are employed to assess fish freshness. The author’s objective is to evaluate the freshness of fish by analyzing its eye characteristics. To achieve this, we have developed a combination of deep and machine learning models that accurately classify the freshness of fish. Furthermore, an application that utilizes both deep learning and machine learning, to instantly detect the freshness of any given fish sample was created. Two deep learning algorithms (SqueezeNet, and VGG19) were implemented to extract features from image data. Additionally, five machine learning models to classify the freshness levels of fish samples were applied. Machine learning models include (k-NN, RF, SVM, LR, and ANN). Based on the results, it can be inferred that employing the VGG19 model for feature selection in conjunction with an Artificial Neural Network (ANN) for classification yields the most favorable success rate of 77.3% for the FFE dataset. Graphical Abstract Keywords Classification · Deep learning · Feature extraction · Fisheye · Fish freshness · Machine learning * Murat Koklu Muslume Beyza Yildiz 1 Department of Computer Engineering, Selcuk University, Konya, Turkey 2 Graduate School of Natural and Applied Sciences, Selcuk University, Konya, Turkey Elham Tahsin Yasin Vol.:(0123456789) European Food Research and Technology Introduction Fish consumption is motivated by its taste, health, quality, and freshness. The demand for high-quality and immune fish products has increased over the past few years due to recent changes in consumer lifestyles [23, 53, 54]. There are several factors associated with fish quality parameters that range from harvesting to consumption, such as safety, nutrition, availability, and freshness, which are affected by storage and processing methods [49]. A variety of factors may negatively affect fish quality and freshness, including production, transportation, sale, domestic storage, and final preparation of food [11, 32]. Fish provides essential nutrients, vitamins, and proteins for human wellness [12, 37]. Fish freshness may be determined using a variety of techniques. Furthermore, with greater technical advancements, there have been attempts to create a way of measuring and assessing more dependable freshness. The criteria used to determine freshness include sensory, physical, chemical, and microbiological. Rapid protein liquid chromatography and hyperspectral imaging techniques are also considered [37, 53, 54]. It is thought that a fish's eye region has a strong relationship between its coloration and the period during which they are stored. For fish to maintain its highest quality after harvest, it must be kept at a specific temperature for a specific time [11]. A multi-nutrient food that provides proteins, omega-3 fatty acids, and vitamins (health benefits) [14]. The fresher the fish, the more nutritious it is. Consequently, it can be challenging for most consumers to determine whether a fish is fresh while they are shopping. By touching and squeezing the fish's body to determine its flexibility, you may quickly determine how fresh it is [53, 54]. Normally fresh fish has higher elasticity. Unfortunately, using this method can contaminate food with germs, harming fish and leading to foodborne illnesses [17, 38, 39]. Fish spoilage is the process through which the quality of fish deteriorates, altering its color, odor, smell, flavor, and flesh texture [4, 14]. There are two primary sources of decomposition once it starts: biological spoilage and chemical spoiling [16]. Microorganisms invade the fish's body via the gills and commence the process of decomposition. Due to chemical interactions, chemical spoilage causes a disagreeable odor and also affects the flavor [42]. The changeover from fresh to stale is indicated by a change in the color of the gills from brilliant pink to dark red or yellowish red. A faded color will be detected if the discoloration of the skin is overall. It is easy to tell a fresh fish by the shine and brightness of its skin, as opposed to a not-so-fresh fish by its dullness and faded colors. Depending on the color of the flesh of the fish, freshness can be determined by colored flesh ranging from cream to yellow-orange to brown to blue-orange [45, 53, 54]. A fisheye's appearance was also considered [46]. When a fish gets spoiled, the areas around its pupils become opaque, which creates the illusion that it is spoiled [48]. The evaluation of fish freshness and quality is the study’s key objective. This is achieved by utilizing deep learning models for extracting features. Training and testing of freshness are performed with images by cross-validation. The CNN classification algorithms InceptionV3, SqueezeNet, VGG16, and VGG19 were all utilized. For each algorithm separately machine learning will be applied for predicting the freshness of the image dataset. The algorithms (k-NN, SVM, ANN, LR, and RF) were selected to be examined on the dataset and discuss the obtained results. This study utilized a combination of deep learning techniques and machine learning methods to classify the freshness of fish. The most successful classification algorithm will be determined. After this stage, a mobile application will be developed based on the most successful models. By using pre-trained deep learning models like SqueezeNet for feature extraction, you can often achieve better results than handcrafted feature extraction techniques. This is because deep learning models have learned to extract relevant features automatically from large amounts of data, which can be difficult to do manually. Related works In this section previously implemented methods for fish freshness classification were reviewed. The reviewed areas were models in deep learning, fish freshness, machine learning algorithms, and prediction with a mobile application. The corresponding articles are listed below. Deep learning models for fish freshness classification In their study, Mohammadi Lalabadi et al. proposed the use of color degradation in the various color areas of fisheyes and gills as a basis for the segmentation of fisheyes and gills. This study utilized digital images to ana (...truncated)