Flash communication pattern analysis of fireflies based on computer vision

International Journal of Advances in Intelligent Informatics Vol. 6, No. 1, March 2020, pp. 60-71 ISSN 2442-6571 60 Flash communication pattern analysis of fireflies based on computer vision Thanaban Tathawee a,1, Wandee Wattanachaiyingcharoen a,b,2, Anantachai Suwannakom c,3, Surisak Prasarnpun d,4,* a Department of Biology, Faculty of Science, Naresuan University, Phitsanulok 65000, Thailand Center of Excellence for Biodiversity, Faculty of Science, Naresuan University, Phitsanulok 65000, Thailand c Department of Physics, Faculty of Science, Naresuan University, Phitsanulok 65000, Thailand d School of Medical Sciences, University of Phayao, Phayao 56000, Thailand 1 ; 2 ; 3 ; 4 * corresponding author b ARTICLE INFO Article history Received June 2, 2019 Revised December 13, 2019 Accepted December 24, 2019 Available online March 31, 2020 Keywords Firefly Computer vision Flash pattern High-throughput Software ABSTRACT Previous methods for detecting the flashing behavior of fireflies were using either a photomultiplier tube, a stopwatch, or videography. Limitations and problems are associated with these methods, i.e., errors in data collection and analysis, and it is time-consuming. This study aims to applied a computer vision approach to reduce the time of data collection and analysis as compared to the videography methods by illuminance calculation, time of flash occurrence, and optimize the position coordinate automatically and tracking each firefly individually. The Validation of the approach was performed by comparing the flashing data of male fireflies, Sclerotia aquatilis that was obtained from the analysis of the behavioral video. The pulse duration, flash interval, and flash patterns of S. aquatilis were similar to a reference study. The accuracy ratio of the tracking algorithm for tracking multiple fireflies was 0.94. The time consumption required to analyze the video decreased up to 96.82% and 76.91% when compared with videography and the stopwatch method, respectively. Therefore, this program could be employed as an alternative technique for the study of fireflies flashing behavior. This is an open access article under the CC–BY-SA license. 1. Introduction Fireflies’ bioluminescence behavior is an interesting phenomenon. The wonderful light of adult fireflies plays a role in reproductive species-specific isolation according to the pattern of emitted light [1]. Fireflies have various kinds of communication systems, especially nocturnal fireflies [2]. Different species emit light in different patterns. The characteristics of the flash, for instance, light intensity, lantern size, and pulse duration are used for species-specific reproductive separation [3]. Several species of female Photinus mimic the flash response of the other female species to attract and devour their males [4]. In addition, their bioluminescence is used for illumination during landing and walking, which protects fireflies from the spider’s webs and flooded areas [5]. Therefore, the study of bioluminescence behavior can lead to an understanding of the biology of fireflies. Since the firefly flash is a sophisticated behavior, a variety of methods were used to study flashing behavior. One method was direct human observations using a stopwatch [1][6][7]. The stopwatch technique is limited in that it is prone to inaccuracies because the stopwatch operator has a significant delay in switching the watch on and off. http://dx.doi.org/10.26555/ijain.v6i1.367 http://ijain.org 61 International Journal of Advances in Intelligent Informatics Vol. 6, No. 1, March 2020, pp. 60-71 ISSN 2442-6571 The photomultiplier tube (light sensor) detects and records using a data acquisition system is another technique used for firefly behavior study [8]-[10]. However, the photomultiplier tube is not appropriate for recording several fireflies simultaneously, because it senses all of the light sources at the same time and leads to interference of the signal. In addition, the video recording method is analyzed based on the frame by frame analysis [3][11]. Flashing behavior study by video recording can decrease the limitation of multiple object recording. Normally, one second of video length consists of 25-30 frames. There is a significant time needed for data interpreting, especially during the process of capturing the pictures and analyzing them frame by frame. This method also limited the analyzing capability due to the manual tracking of individual fireflies in each frame during the frame by frame analysis [11]. Computer-assisted techniques are used extensively to improve the performance of many processes in studies such as robotics, automated agriculture, digital devices, as well as automation monitoring. Dankert and colleagues [12] used computer vision to track fruit fly behavior, which gave highthroughput and accurate results. Computer vision was also applied to a variety of biological studies such as taxonomy (automated identification), plant phenotyping, and cell culture [13]-[15]. Computerized image processing is more accurate and takes less time to investigate and analyze data [16]. Due to the advantages of computer vision, the goal of this study is to develop a program to assist in analyzing firefly flashing behavior based on computer vision. The program tracks the firefly, records the flash amplitude and the time of the flash during frame by frame analysis. The developed program also assists flash interpretation by calculating the pulse duration and flash interval. The developed program can enhance the capability of routine tasks of biologists and entomologists for studying insects and animal behavior. 2. Method 2.1. Development of the program: flash data extraction from the video The recorded video is converted from “.MOV” to “.mp4” format before the image-processing process by Movie Maker, Microsoft, 2012. Image processing, all frames of the video file are processed reclusively based upon three steps. In the first step, each frame is converted from RGB (red-green-blue) color space to HSV (hue-saturation-value) color space because it is more suitable for image segmentation than the RGB model [17]. The conversion follows the study of Pekel [18]. Secondly, the firefly flash is extracted from an image of the value channel (an array image of HSV color space) by the multiple-thresholding technique [19]. Then the flash area (pixel2) is calculated to represent the flash illuminance. The coordinates of the position and time of each flash area were also collected. 2.2. Development of the program: firefly tracking process Global object tracking can be classified into two types: probabilistic and deterministic methods [20]. The probabilistic method solves the tracking problem based on Bayesian inference or uncertainty modeling, such as Monte Carlo, Particle Filtering framework [21][22]. Deterministic methods solve the tracking problem by comparison to the region of interest in the present and previou (...truncated)