Real-time and accurate deep learning-based multi-organ nucleus segmentation in histology images

Journal of Real-Time Image Processing (2024) 21:43 https://doi.org/10.1007/s11554-024-01420-0 RESEARCH Real‑time and accurate deep learning‑based multi‑organ nucleus segmentation in histology images Noha Y. Ahmed1 Received: 14 August 2023 / Accepted: 14 January 2024 / Published online: 29 February 2024 © The Author(s) 2024 Abstract Automated nucleus segmentation is considered the gold standard for diagnosing some severe diseases. Accurate instance segmentation of nuclei is still very challenging because of the large number of clustered nuclei, and the different appearance of nuclei for different tissue types. In this paper, a neural network is proposed for fast and accurate instance segmentation of nuclei in histopathology images. The network is inspired by the Unet and residual nets. The main contribution of the proposed model is enhancing the classification accuracy of nuclear boundaries by moderately preserving the spatial features by relatively d the size of feature maps. Then, a proposed 2D convolution layer is used instead of the conventional 3D convolution layer, the core of CNN-based architectures, where the feature maps are first compacted before being convolved by 2D kernel filters. This significantly reduces the processing time and avoids the out of memory problem of the GPU. Also, more features are extracted when getting deeper into the network without degrading the spatial features dramatically. Hence, the number of layers, required to compensate the loss of spatial features, is reduced that also reduces the processing time. The proposed approach is applied to two multi-organ datasets and evaluated by the Aggregated Jaccard Index (AJI), F1-score and the number of frames per second. Also, the formula of AJI is modified to reflect the object- and pixel-level errors more accurately. The proposed model is compared to some state-of-the-art architectures, and it shows better performance in terms of the segmentation speed and accuracy. Keywords Unet · Residual learning · Convolution layer · Nuclei instance segmentation · Histology images 1 Introduction Detection and segmentation of nuclei in histology images are considered crucial processes for diagnosing, grading and even for prognosis prediction of many diseases, such as most cancer types and Alzheimer [1–4]. In current clinical practices, the examination of hematoxylin and eosin (H&E)stained tissue images (to analyze nuclei density, morphology and shape) is carried out manually, by means of pathologists. However, the manual analysis results in many problems such as inter- and intra- observer variability, inability to assess fine visual features and a huge amount of time to examine Whole Slide Images (WSI) [5, 6]. With the great revolution in computer vision and image processing techniques, many manual assessment problems of histology images have been * Noha Y. Ahmed 1 Radiation Engineering Department, Egyptian Atomic Energy Authority (EAEA), Cairo, Egypt addressed [7–9]. Several traditional image processing methods for automatic nucleus detection and segmentation were presented, such as Otsu thresholding [10], Marker-controlled watershed segmentation [11] and other region growing, morphology, feature extraction and color-based thresholding operations [12–14]. However, such methods are mainly based on predefined colors, shapes, or textures of nuclei that cannot be constant for all cases, such as different types of tissues, different grading of disease or the wide spectrum of tissue morphologies. In addition, due to noise and various staining concentration that appear in H&E stained images, such traditional methods fail to produce accurate and robust nucleus segmentation. Furthermore, those approaches produce either under- or over-segmentation of clustered and overlapping nuclei [15]. Throughout the past few decades, learning-based nucleus detection and segmentation methods have been proposed, where handcrafted features are extracted, such as color histograms, texture, morphology, optical density, geometric and other characteristics of nuclei [16–18]. The extracted features are then fed into Machine Vol.:(0123456789) 43 Page 2 of 16 Learning (ML)-based models, such as Random Decision Forest (RDF) [19], the K-Nearest Neighbors (KNN) [20] and Support Vector Machine (SVM) [21] algorithms to produce the nuclear probability map within the histology images. However, ML approaches depend on predefined features, and the parameters are adapted based on trial-anderror during training in order to achieve the required performance. This makes it difficult to consider them as generalized nucleus segmentation approaches. Recently, many Deep Learning (DL) models have been proposed for nuclear detection and segmentation that address the limitations of the handcraft feature-based models, where most of them are trained, based on pre-annotated datasets, including nuclear and non-nuclear pixels. This enables models to extract more detailed, sophisticated and hidden features that cannot be easily recognized by traditional and standard handcraft feature-based approaches, to achieve a generalized robust nucleus segmentation [22]. Deep learning-based segmentation approaches are divided into semantic segmentation and instance segmentation. In semantic segmentation, the main concern is only distinguishing nuclear from non-nuclear pixels in the histology image. On the other side, instance segmentation is concerned with distinguishing and segmenting individual nuclei (objects), which is very important to help pathologists study the morphology, size and density of various nuclei in the same tissue to achieve accurate diagnosis, grading and prognosis [23–26]. Yet, the main challenge in most introduced instance nuclear segmentation approaches is separating overlapping and clustered nuclei. However, that is mostly at the expense of the computational complexity and processing time. Moreover, most of them are trained based on a certain type of nuclei, so they cannot be applied to various types of tissues. In this work, a real-time architecture for accurate instance nuclear segmentation is proposed. The proposed architecture is inspired by the Unet [27] and residual nets [28]. The model was trained on multi types of nuclei of different kinds of tissues. The main contribution of the proposed model is that it moderately preserves the spatial features while reducing the computational complexity and processing time; minimizing the processing time is crucial since up to 1000 WSIs can be analyzed and diagnosed per day, in one clinical setup. The relative preservation of spatial features is important, especially, to identify the minor-class pixels that are the boundary pixels, generally, and the common pixels among touching and overlapping nuclei, specifically. To achieve that, the size of feature maps is not intensively reduced; instead, the channels of feature maps are compacted into one channel. Then, a proposed 2D convolution layer is used (...truncated)