A novel Gravity-FREAK feature extraction and Gravity-KLT tracking registration algorithm based on iPhone MEMS mobile sensor in mobile environment

RESEARCH ARTICLE A novel Gravity-FREAK feature extraction and Gravity-KLT tracking registration algorithm based on iPhone MEMS mobile sensor in mobile environment Zhiling Hong*, Fan Lin, Bin Xiao Software School, Xiamen University, Xiamen, Fujian, China * a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Hong Z, Lin F, Xiao B (2017) A novel Gravity-FREAK feature extraction and Gravity-KLT tracking registration algorithm based on iPhone MEMS mobile sensor in mobile environment. PLoS ONE 12(10): e0186176. https://doi.org/10.1371/ journal.pone.0186176 Editor: Quan Zou, Tianjin University, CHINA Received: June 27, 2017 Abstract Based on the traditional Fast Retina Keypoint (FREAK) feature description algorithm, this paper proposed a Gravity-FREAK feature description algorithm based on Micro-electromechanical Systems (MEMS) sensor to overcome the limited computing performance and memory resources of mobile devices and further improve the reality interaction experience of clients through digital information added to the real world by augmented reality technology. The algorithm takes the gravity projection vector corresponding to the feature point as its feature orientation, which saved the time of calculating the neighborhood gray gradient of each feature point, reduced the cost of calculation and improved the accuracy of feature extraction. In the case of registration method of matching and tracking natural features, the adaptive and generic corner detection based on the Gravity-FREAK matching purification algorithm was used to eliminate abnormal matches, and Gravity Kaneda-Lucas Tracking (KLT) algorithm based on MEMS sensor can be used for the tracking registration of the targets and robustness improvement of tracking registration algorithm under mobile environment. Accepted: September 26, 2017 Published: October 31, 2017 Copyright: © 2017 Hong et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: All relevant data are within the paper and we have deposited the minimal dataset into the public repository FigShare (https://doi.org/10.6084/m9.figshare.5426464.v2). Funding: The Project was supported by the National Natural Science Foundation of China (Grant No. 31200769). Competing interests: The authors have declared that no competing interests exist. Introduction With the rapid development of image processing and artificial intelligence, the conception can be realized through the combing use of different technologies and the augmented reality technology which focuses on virtual-real fusion emerged [1,36,39]. Different from the virtual reality technologies that focus on introducing users to virtual 3D scenes, the augmented reality technology emphasizes how to accurately integrate the virtual information generalized by computer into the real-world environment so that to realize the simultaneous presentation of virtual information and the real environment for the supplementation and enhancement of the real environment. The relationship between the two parts is show as Fig 1: Generally, the augmented reality system is consisted of three parts: virtual-real fusion, realtime interaction and 3D registration [2]. Among the three parts, 3D registration, the accurate matching between virtual and real environments, is the key restraining factor of wider application of augmented reality technology. Most of the traditional 3D registration methods were PLOS ONE | https://doi.org/10.1371/journal.pone.0186176 October 31, 2017 1 / 28 A Gravity-FREAK feature description algorithm based on MEMS sensor Fig 1. Mixed reality. https://doi.org/10.1371/journal.pone.0186176.g001 designed and proposed on the basis of PC [3,38]. They cannot be applied to mobile augmented reality systems directly as most of the mainstream mobile devices are not equipped with floating point processor (FPP), and the CPU speed and memory capacity are not able to support the devices efficiently to conduct feature extraction and position calculation of the target. Hence, it becomes an urgent matter to search a mobile 3D registration algorithm with better performance and lower resource occupation to popularize mobile augmented reality. Related work As the product of the constant development of virtual reality technology, the appearance of augmented reality can be traced back to the HMD (Head Mounted Display) invented by an American in 1965 [4]. Through the device, the user can visualize the superposition of real environment and 3D image. Until the early 1990s, the concept of augmented reality was first proposed by Caudell and Mizell [5], scientists from Boeing Co. After that, the size of portable device became smaller and smaller, while the computing performance became stronger and stronger, which makes it possible to conduct image rendering and superposition on mobile devices. In 1997, Feiner et al. [6] designed the first prototype of mobile augmented reality system. The system can add 3D travel guide information onto the real built environment. By the end of 1990s, augmented reality became an independent and significant research field which attracted more and more researchers. Many AR related international conferences also emerged, such as IWSAR (International Workshop and Symposium on Augmented Reality), ISMR (International Symposium on Mixed Reality), DARE (Designing Augmented Reality Environments workshop), etc. Among all the research directions, the research on AR tracking registration technology is always the hotspot, which is also the key step in the application of AR. According to the registration method, AR system can be divided into sensor oriented system and machine vision oriented system. Sensor oriented tracking registration Sensor oriented tracking method has long ago been applied to AR registration field by researchers, including mechanical tracking registration, electromagnetic tracking registration, ultrasonic tracking registration, GPS tracking registration and inertial tracking registration, etc. The method replies on the related sensor function of the hardware device. With the accurate real-time data provided by the sensor, the method can obtain the position and direction information of the tracking target. The outdoor AR system designed by Feiner et al. [6] used sensors as GPS and angle instrument for tracking registration. However, the PLOS ONE | https://doi.org/10.1371/journal.pone.0186176 October 31, 2017 2 / 28 A Gravity-FREAK feature description algorithm based on MEMS sensor method has high requirements of hardware and environment. Many sensor oriented tracking registration methods are still in the experimental stage, which cannot be promoted to ordinary users. Computer vision oriented tracking registration Compared with sensor ori (...truncated)