MAFNet: dual-branch fusion network with multiscale atrous pyramid pooling aggregate contextual features for real-time semantic segmentation

Complex & Intelligent Systems https://doi.org/10.1007/s40747-024-01428-w ORIGINAL ARTICLE MAFNet: dual-branch fusion network with multiscale atrous pyramid pooling aggregate contextual features for real-time semantic segmentation Shan Zhao1 · Yunlei Wang1 · Xuan Wu1 · Fukai Zhang1 Received: 2 January 2024 / Accepted: 9 March 2024 © The Author(s) 2024 Abstract Currently, many real-time semantic segmentation networks aim for heightened accuracy, inevitably leading to increased computational complexity and reduced inference speed. Therefore, striking a balance between accuracy and speed has emerged as a crucial concern in this domain. To address these challenges, this study proposes a dual-branch fusion network with multiscale atrous pyramid pooling aggregate contextual features for real-time semantic segmentation (MAFNet). The first key component, the semantics guide spatial-details module (SGSDM) not only facilitates precise boundary extraction and fine-grained classification, but also provides semantic-based feature representation, thereby enhancing support for spatial analysis and decision boundaries. The second component, the multiscale atrous pyramid pooling module (MSAPPM), is designed by combining dilation convolution with feature pyramid pooling operations at various dilation rates. This design not only expands the receptive field, but also aggregates rich contextual information more effectively. To further improve the fusion of feature information generated by the dual-branch, a bilateral fusion module (BFM) is introduced. This module employs cross-fusion by calculating weights generated by the dual-branch to balance the weight relationship between the dual branches, thereby achieving effective feature information fusion. To validate the effectiveness of the proposed network, experiments are conducted on a single A100 GPU. MAFNet achieves a mean intersection over union (mIoU) of 77.4% at 70.9 FPS on the Cityscapes test dataset and 77.6% mIoU at 192.5 FPS on the CamVid test dataset. The experimental results conclusively demonstrated that MAFNet effectively strikes a balance between accuracy and speed. Keywords Semantic segmentation · Real time · Multiscale · Pyramid pooling · Autonomous driving Introduction Semantic segmentation is an important technique in the field of computer vision, with the objective of assigning each pixel Shan Zhao, Yunlei Wang, Xuan Wu and Fukai Zhang contributed equally to this work. B Yunlei Wang Shan Zhao Xuan Wu Fukai Zhang 1 School of Software, Henan Polytechnic University, 2001 Century Avenue, Jiaozuo 454000, China to a distinct semantic category within an image. Currently, the application scenarios of semantic segmentation include, but are not limited to, some practical application scenarios such as autonomous driving [1] and medical imaging analysis [2]. The advent of the convolutional neural network (CNN) marked the inception of early semantic segmentation networks like fully convolutional networks (FCN) [3] and U-Net [4]. In particular, FCN demonstrated remarkable performance in the field of semantic segmentation, marking a significant breakthrough. However, the need for more high-performance semantic segmentation networks became apparent as technology evolved. Additionally, existing networks fall short of meeting the demands of general scenarios, given that achieving higher performance often requires substantial computational resources, especially when relying on complex backbone networks like ResNet101 [5]. ResNet101 is a deep CNN with 101 layers, which addresses challenges related 123 Complex & Intelligent Systems to gradient vanishing and exploding during deep network training by introducing a residual learning architecture. Similarly, DeepLabV3+ [6] is a powerful semantic segmentation model that enhances segmentation accuracy by employing techniques such as dilated convolutions and atrous spatial pyramid pooling (ASPP). However, the computational cost of these high-accuracy networks, involving hundreds of giga floating-point operations per second (GFLOs), hinders their suitability for general scenarios like autonomous driving and intelligent transportation. However, some studies have already offered potential insights into semantic segmentation issues in practical application scenarios. Employing the repetitive process control in [7] can enhance batch processing efficiency, potentially contributing practicality to real-time semantic segmentation. Bipartite synchronization in neural networks with event-triggered mechanisms in [8] aims to enhance cooperative operations, with the potential to optimize segmentation strategies. Reference [9] presents a hysteresis-quantified control method for switched systems, offering a potential solution for dynamic scenarios in realtime semantic segmentation. To address the challenge of high computational costs and meet the real-world demand for network inference speed, the development of real-time semantic segmentation is gradually gaining attention. ENet [10], designed for low-latency operations, offers comparable or superior accuracy to stateof-the-art models. While ICNet [11] introduced a cascaded feature fusion unit for high-quality segmentation results with efficient inference speed. STDCNet [12] presented a short-term dense concatenate module (STDCM), an efficient network structure that progressively reduces the dimension of feature maps. S2 -FPN [13] proposed a scale-aware strip attention module (SSAM) with low computational overhead to collect remote context along the vertical axis by striping operations and reduce computational cost. While these methods achieve commendable segmentation accuracy and speed, striking a balance between accuracy and speed in real-time semantic segmentation remains a challenging task. To achieve this balance, some models employ lightweight convolution structures to maintain relatively low computational complexity while achieving faster inference speeds. ESPNet [14] introduced a novel convolutional module, decomposing the standard convolution into a spatial pyramid of pointwise and dilated convolutions. EfficientNet [15] systematically explored model scaling, constructing a simple and efficient composite coefficient to regulate the relationship between depth, width, and resolution of the network. Fast-SCNN [16] proposed a shallow learning to downsample module to extract low-level features quickly and efficiently. Although these lightweight convolution structures enhance speed and maintain relatively low computational complexity to some extent, they often fall short of achieving the desired accuracy. Additionally, lightweight structures may lead to the 123 loss of important features in input images, including spatial detail information. Therefore, these issues must be considered when designing and optimizing lightweight semantic segmentation networks. Additionally, the receptive field of the network should be expanded, and richer spati (...truncated)