Marine ship instance segmentation by deep neural networks using a global and local attention (GALA) mechanism

PLOS ONE RESEARCH ARTICLE Marine ship instance segmentation by deep neural networks using a global and local attention (GALA) mechanism Zequn Sun1, Chunning Meng2*, Tao Huang3, Zhiqing Zhang ID1,4*, Shengjiang Chang1 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 1 Institute of Modern Optics, Nankai University, Tianjin City, China, 2 China Coast Guard Academy, Ningbo City, China, 3 717 Research Institute of China Shipbuilding Industry Corporation, Wuhuan City, China, 4 State Key Laboratory of Applied Optics, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun City, China * (CM); (ZZ) Abstract OPEN ACCESS Citation: Sun Z, Meng C, Huang T, Zhang Z, Chang S (2023) Marine ship instance segmentation by deep neural networks using a global and local attention (GALA) mechanism. PLoS ONE 18(2): e0279248. https://doi.org/10.1371/journal. pone.0279248 Editor: Jean-Christophe Nebel, Kingston University, UNITED KINGDOM Received: June 29, 2022 Accepted: December 4, 2022 Published: February 24, 2023 Copyright: © 2023 Sun et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: The datasets and accompanying algorithms can be download via the link, https://github.com/s2120200252/Visible-shipdataset. Funding: This research was funded by the National Natural Science Foundation of China, grant number 61831012 and 61401105, and by the State Key Laboratory of Applied Optics, grant number SKLAO2022001A14. Competing interests: The authors have declared that no competing interests exist. Marine ships are the transport vehicle in the ocean and instance segmentation of marine ships is an accurate and efficient analysis approach to achieve a quantitative understanding of marine ships, for example, their relative locations to other ships or obstacles. This relative spatial information is crucial for developing unmanned ships to avoid crashing. Visible light imaging, e.g. using our smartphones, is an efficient way to obtain images of marine ships, however, so far there is a lack of suitable open-source visible light datasets of marine ships, which could potentially slow down the development of unmanned ships. To address the problem of insufficient datasets, here we built two instance segmentation visible light datasets of marine ships, MariBoats and MariBoatsSubclass, which could facilitate the current research on instance segmentation of marine ships. Moreover, we applied several existing instance segmentation algorithms based on neural networks to analyze our datasets, but their performances were not satisfactory. To improve the segmentation performance of the existing models on our datasets, we proposed a global and local attention mechanism for neural network models to retain both the global location and semantic information of marine ships, resulting in an average segmentation improvement by 4.3% in terms of mean average precision. Therefore, the presented new datasets and the new attention mechanism will greatly advance the marine ship relevant research and applications. 1 Introduction Image segmentation plays an essential role in many visual understanding and object detection systems [1–3]. It involves a process that employs the intensity (brightness) or other information (e.g., edge) of an image to divide the image into independently connected regions. Image segmentation algorithms can be classified into at least two categories, i.e., semantic segmentation and instance segmentation. Semantic segmentation performs pixel-level labelling using a set of colors (object categories), while instance segmentation extends semantic segmentation by simultaneously detecting and delineating each object of interest in an image [3, 4]. PLOS ONE | https://doi.org/10.1371/journal.pone.0279248 February 24, 2023 1 / 22 PLOS ONE Marine ship instance segmentation by GALA Compared to object detection which merely detects the location of an object and places a window over it, instance segmentation performs more like a combination of object detection and semantic segmentation that not only detects the locations of all specific objects but also outlines and classifies individual detected objects [5]. As to marine ship segmentation, semantic segmentation classifies all ships in an image into one category, by labelling all ships with one color, while instance segmentation detects individual ships and classifies them into different categories. The applications of instance segmentation have been launched successfully in scenarios such as unmanned vehicle development [6, 7], human-computer interaction [8, 9], bio-medicine development [10–12], video surveillance [13–15], and marine ship monitoring [16–18]. Since marine ships are the vehicle of ocean-related activities such as marine scientific research and education, transoceanic transport and marine fishing industry, image and video analysis of marine ships including instance segmentation of marine ships has received an increasing attention in the past years [18–20]. Instance segmentation of marine ships is capable of providing important information such as the relative location of a ship with respect to other ships or surrounding obstacles, which is crucial for ship travel safety. Nowadays, various types of imaging techniques such as radar and infrared camera, have been equipped on a modern marine ship or an intelligent unmanned ship [16]. The spatial analysis of the ship imaging data can provide accurate environmental information to assist the ships to autonomously avoid crashing with other ships and natural obstacles in the ocean. Moreover, by segmenting a ship with respect to the background that may contain spatial information (for example, a known island or city), we can obtain the identity of the ship and the absolute location of the ship on earth. Therefore, instance segmentation of marine ships from a complex maritime background is essential for many ocean-related activities. At present, satellite, synthetic aperture radar (SAR), infrared imaging (IR) and visible light (VL) imaging, are the main imaging tools to record marine ships, resulting in several different types of open-source databases for marine ship segmentation. These databases include satellite remote sensing images [21–23], SAR images [24, 54], IR images [25–27], and VL images [28]. The satellite images usually have a very large field of view, covering a wide space, but their image resolution is low, providing no accurate information (e.g. the shape and type) of a ship. SAR imaging can perform regardless of weather conditions, but SAR images usually contain a large amount of scattering noise and do not have rich spectral information, which is not convenient for subsequent ship segmentation purpose.IR imaging has str (...truncated)